LangSmith - LLM Observability Platform

Development platform for debugging, evaluating, and monitoring language models and AI applications.

When to use LangSmith

Use LangSmith when:

• Debugging LLM application issues (prompts, chains, agents)

• Evaluating model outputs systematically against datasets

• Monitoring production LLM systems

• Building regression testing for AI features

• Analyzing latency, token usage, and costs

• Collaborating on prompt engineering

Key features:

• Tracing: Capture inputs, outputs, latency for all LLM calls

• Evaluation: Systematic testing with built-in and custom evaluators

• Datasets: Create test sets from production traces or manually

• Monitoring: Track metrics, errors, and costs in production

• Integrations: Works with OpenAI, Anthropic, LangChain, LlamaIndex

Use alternatives instead:

• Weights & Biases: Deep learning experiment tracking, model training

• MLflow: General ML lifecycle, model registry focus

• Arize/WhyLabs: ML monitoring, data drift detection

Quick start

Installation

pip install langsmith

# Set environment variables

export LANGSMITH_API_KEY="your-api-key"

export LANGSMITH_TRACING=true

Basic tracing with @traceable

from langsmith import traceable

from openai import OpenAI

client = OpenAI()

@traceable

def generate_response(prompt: str) -> str:

    response = client.chat.completions.create(

        model="gpt-4o",

        messages=[{"role": "user", "content": prompt}]

    )

    return response.choices[0].message.content

# Automatically traced to LangSmith

result = generate_response("What is machine learning?")

OpenAI wrapper (automatic tracing)

from langsmith.wrappers import wrap_openai

from openai import OpenAI

# Wrap client for automatic tracing

client = wrap_openai(OpenAI())

# All calls automatically traced

response = client.chat.completions.create(

    model="gpt-4o",

    messages=[{"role": "user", "content": "Hello!"}]

)

Core concepts

Runs and traces

A run is a single execution unit (LLM call, chain, tool). Runs form hierarchical traces showing the full execution flow.

from langsmith import traceable

@traceable(run_type="chain")

def process_query(query: str) -> str:

    # Parent run

    context = retrieve_context(query)  # Child run

    response = generate_answer(query, context)  # Child run

    return response

@traceable(run_type="retriever")

def retrieve_context(query: str) -> list:

    return vector_store.search(query)

@traceable(run_type="llm")

def generate_answer(query: str, context: list) -> str:

    return llm.invoke(f"Context: {context}\n\nQuestion: {query}")

Projects

Projects organize related runs. Set via environment or code:

import os

os.environ["LANGSMITH_PROJECT"] = "my-project"

# Or per-function

@traceable(project_name="my-project")

def my_function():

    pass

Client API

from langsmith import Client

client = Client()

# List runs

runs = list(client.list_runs(

    project_name="my-project",

    filter='eq(status, "success")',

    limit=100

))

# Get run details

run = client.read_run(run_id="...")

# Create feedback

client.create_feedback(

    run_id="...",

    key="correctness",

    score=0.9,

    comment="Good answer"

)

Datasets and evaluation

Create dataset

from langsmith import Client

client = Client()

# Create dataset

dataset = client.create_dataset("qa-test-set", description="QA evaluation")

# Add examples

client.create_examples(

    inputs=[

        {"question": "What is Python?"},

        {"question": "What is ML?"}

    ],

    outputs=[

        {"answer": "A programming language"},

        {"answer": "Machine learning"}

    ],

    dataset_id=dataset.id

)

Run evaluation

from langsmith import evaluate

def my_model(inputs: dict) -> dict:

    # Your model logic

    return {"answer": generate_answer(inputs["question"])}

def correctness_evaluator(run, example):

    prediction = run.outputs["answer"]

    reference = example.outputs["answer"]

    score = 1.0 if reference.lower() in prediction.lower() else 0.0

    return {"key": "correctness", "score": score}

results = evaluate(

    my_model,

    data="qa-test-set",

    evaluators=[correctness_evaluator],

    experiment_prefix="v1"

)

print(f"Average score: {results.aggregate_metrics['correctness']}")

Built-in evaluators

from langsmith.evaluation import LangChainStringEvaluator

# Use LangChain evaluators

results = evaluate(

    my_model,

    data="qa-test-set",

    evaluators=[

        LangChainStringEvaluator("qa"),

        LangChainStringEvaluator("cot_qa")

    ]

)

Advanced tracing

Tracing context

from langsmith import tracing_context

with tracing_context(

    project_name="experiment-1",

    tags=["production", "v2"],

    metadata={"version": "2.0"}

):

    # All traceable calls inherit context

    result = my_function()

Manual runs

from langsmith import trace

with trace(

    name="custom_operation",

    run_type="tool",

    inputs={"query": "test"}

) as run:

    result = do_something()

    run.end(outputs={"result": result})

Process inputs/outputs

def sanitize_inputs(inputs: dict) -> dict:

    if "password" in inputs:

        inputs["password"] = "***"

    return inputs

@traceable(process_inputs=sanitize_inputs)

def login(username: str, password: str):

    return authenticate(username, password)

Sampling

import os

os.environ["LANGSMITH_TRACING_SAMPLING_RATE"] = "0.1"  # 10% sampling

LangChain integration

from langchain_openai import ChatOpenAI

from langchain_core.prompts import ChatPromptTemplate

# Tracing enabled automatically with LANGSMITH_TRACING=true

llm = ChatOpenAI(model="gpt-4o")

prompt = ChatPromptTemplate.from_messages([

    ("system", "You are a helpful assistant."),

    ("user", "{input}")

])

chain = prompt | llm

# All chain runs traced automatically

response = chain.invoke({"input": "Hello!"})

Production monitoring

Hub prompts

from langsmith import Client

client = Client()

# Pull prompt from hub

prompt = client.pull_prompt("my-org/qa-prompt")

# Use in application

result = prompt.invoke({"question": "What is AI?"})

Async client

from langsmith import AsyncClient

async def main():

    client = AsyncClient()

    runs = []

    async for run in client.list_runs(project_name="my-project"):

        runs.append(run)

    return runs

Feedback collection

from langsmith import Client

client = Client()

# Collect user feedback

def record_feedback(run_id: str, user_rating: int, comment: str = None):

    client.create_feedback(

        run_id=run_id,

        key="user_rating",

        score=user_rating / 5.0,  # Normalize to 0-1

        comment=comment

    )

# In your application

record_feedback(run_id="...", user_rating=4, comment="Helpful response")

Testing integration

Pytest integration

from langsmith import test

@test

def test_qa_accuracy():

    result = my_qa_function("What is Python?")

    assert "programming" in result.lower()

Evaluation in CI/CD

from langsmith import evaluate

def run_evaluation():

    results = evaluate(

        my_model,

        data="regression-test-set",

        evaluators=[accuracy_evaluator]

    )

    # Fail CI if accuracy drops

    assert results.aggregate_metrics["accuracy"] >= 0.9, \

        f"Accuracy {results.aggregate_metrics['accuracy']} below threshold"

Best practices

• Structured naming - Use consistent project/run naming conventions

• Add metadata - Include version, environment, user info

• Sample in production - Use sampling rate to control volume

• Create datasets - Build test sets from interesting production cases

• Automate evaluation - Run evaluations in CI/CD pipelines

• Monitor costs - Track token usage and latency trends

Common issues

Traces not appearing:

import os

# Ensure tracing is enabled

os.environ["LANGSMITH_TRACING"] = "true"

os.environ["LANGSMITH_API_KEY"] = "your-key"

# Verify connection

from langsmith import Client

client = Client()

print(client.list_projects())  # Should work

High latency from tracing:

# Enable background batching (default)

from langsmith import Client

client = Client(auto_batch_tracing=True)

# Or use sampling

os.environ["LANGSMITH_TRACING_SAMPLING_RATE"] = "0.1"

Large payloads:

# Hide sensitive/large fields

@traceable(

    process_inputs=lambda x: {k: v for k, v in x.items() if k != "large_field"}

)

def my_function(data):

    pass

References

• Advanced Usage - Custom evaluators, distributed tracing, hub prompts

• Troubleshooting - Common issues, debugging, performance

Resources

• Documentation: https://docs.smith.langchain.com

• Python SDK: https://github.com/langchain-ai/langsmith-sdk

• Web App: https://smith.langchain.com

• Version: 0.2.0+

• License: MIT