Tracing with LangSmith
Instrument your LangChain application with LangSmith tracing to record every chain step, LLM call, token count, and latency in a searchable trace explorer.
Tracing with LangSmith is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is LangSmith?
LangSmith is an observability platform built specifically for LLM applications. It automatically captures traces of every LangChain run — every chain step, LLM call, tool execution, retrieval, and output parser — and displays them in a searchable, hierarchical trace explorer. You can filter traces by latency, cost, error status, or custom metadata, and replay any trace to debug failures.
# Install: pip install langsmith
import os
# Set environment variables to enable automatic tracing
os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_API_KEY'] = 'lsv2_...your_key_here...'
os.environ['LANGCHAIN_PROJECT'] = 'my-rag-app' # project name in LangSmith UI
# That's all - LangChain now sends traces to LangSmith automatically
# No code changes needed to your chain or agentAutomatic Tracing with Zero Code Changes
The most compelling feature of LangSmith is that once you set the three environment variables, every LangChain operation is automatically traced with no additional code. Every LCEL chain, every ChatOpenAI call, every retriever call, every tool execution gets captured with inputs, outputs, timing, and token counts. You can deploy LangSmith tracing to production with a single environment variable change.
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
# This chain is automatically traced - no extra code needed
llm = ChatOpenAI(model='gpt-4o')
prompt = ChatPromptTemplate.from_template('Answer this question: {question}')
chain = prompt | llm | StrOutputParser()
# This call creates a trace in LangSmith showing:
# - The formatted prompt (with question substituted)
# - The LLM call with model, temperature, token counts
# - The parsed output
# - End-to-end latency and cost
result = chain.invoke({'question': 'What is RAG?'})
print(result)Tracing RAG Pipelines
For RAG applications, LangSmith traces are especially valuable because they capture the entire retrieve-then-generate pipeline. You can see: which documents were retrieved, what their similarity scores were, how the context was formatted in the prompt, and what the LLM generated. This makes it immediately obvious whether a wrong answer was caused by bad retrieval or poor generation.
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.runnables import RunnablePassthrough
embeddings = OpenAIEmbeddings()
vectorstore = Chroma(embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={'k': 5})
rag_chain = (
{'context': retriever, 'question': RunnablePassthrough()}
| ChatPromptTemplate.from_template('Context: {context}\n\nQuestion: {question}\n\nAnswer:')
| ChatOpenAI(model='gpt-4o')
| StrOutputParser()
)
# LangSmith traces EVERY step:
# 1. Retriever: query embedding + vector search + returned documents (with scores)
# 2. Prompt: formatted template with context injected
# 3. LLM: full prompt, response, tokens, latency, cost
# 4. Parser: final string output
answer = rag_chain.invoke('What is the capital of France?')Adding Metadata to Traces
By default, LangSmith traces contain the inputs and outputs of each step. You can enrich traces with custom metadata tags: the user ID, session ID, feature flag values, A/B test variant, or any other context that helps you filter and analyze traces in the UI. Use the RunnableConfig to pass metadata that will appear on every trace from that request.
from langchain_core.runnables import RunnableConfig
def handle_user_request(user_id: str, query: str, ab_variant: str):
config = RunnableConfig(
tags=['production', ab_variant],
metadata={
'user_id': user_id,
'ab_variant': ab_variant,
'feature': 'rag_qa'
}
)
result = rag_chain.invoke(query, config=config)
return result
# In LangSmith UI you can now:
# - Filter traces by user_id to debug a specific user's issue
# - Compare latency between ab_variant='A' and ab_variant='B'
# - See all traces tagged 'production' vs 'staging'Manual Span Creation
For code that does not go through LangChain (custom API calls, database queries, preprocessing steps), you can create manual spans using the LangSmith client directly. This ensures your non-LangChain steps are captured in the same trace as the LangChain steps, giving you a complete picture of each request's execution path.
from langsmith import Client, traceable
client = Client()
# Decorate functions to auto-trace them
@traceable(name='preprocess_query')
def preprocess_query(raw_query: str) -> str:
# This step is now traced even though it doesn't use LangChain
cleaned = raw_query.strip().lower()
cleaned = ' '.join(cleaned.split()) # normalize whitespace
return cleaned
@traceable(name='fetch_user_history')
def fetch_user_history(user_id: str) -> list[str]:
# Database call - also traced
return db.query('SELECT message FROM chat_history WHERE user_id = ? ORDER BY timestamp DESC LIMIT 5', user_id)
# All three steps appear in the same trace
def handle_request(user_id: str, raw_query: str):
query = preprocess_query(raw_query) # traced
history = fetch_user_history(user_id) # traced
result = rag_chain.invoke({'query': query, 'history': history}) # traced by LangChain
return resultEvaluating Traces in LangSmith
LangSmith includes an evaluation framework that lets you run evaluators over your trace dataset. You can select a set of traced examples, run automated evaluators (including LLM-as-judge scorers for correctness and relevance), and compare the results across different pipeline versions. This turns your production traces into a feedback loop for improving your application.
from langsmith.evaluation import evaluate, LangChainStringEvaluator
# Create an evaluator that uses an LLM to judge correctness
correctness_evaluator = LangChainStringEvaluator(
'qa',
config={'llm': ChatOpenAI(model='gpt-4o')}
)
# Run evaluation against a dataset of traced examples
results = evaluate(
rag_chain,
data='my-rag-test-set', # name of dataset in LangSmith
evaluators=[correctness_evaluator],
experiment_prefix='rag-v2-chunking-test'
)
print('Evaluation results:')
print(f'Correctness: {results.results["correctness"].mean():.2f}')
print(f'Average latency: {results.results["latency"].mean():.2f}s')Creating Test Datasets from Traces
One of LangSmith's most powerful features is the ability to create test datasets directly from production traces. When you notice an interesting trace (a failure, an edge case, or a great example), you can add it to a dataset with a single click. Over time, you build a comprehensive regression test suite from real user queries rather than synthetic examples.
from langsmith import Client
client = Client()
# Create a dataset from existing traces
dataset = client.create_dataset('rag-regression-tests')
# Add examples from production traces (by trace ID)
for trace_id in failed_trace_ids:
run = client.read_run(trace_id)
client.create_example(
inputs=run.inputs,
outputs={'answer': run.outputs.get('output', '')},
dataset_id=dataset.id,
metadata={'source': 'production_failure', 'date': run.start_time.isoformat()}
)
print(f'Added {len(failed_trace_ids)} examples to regression test dataset')Filtering and Searching Traces
In production, you will have thousands of traces. LangSmith's UI and API support rich filtering and search: find traces with latency above a threshold, with a specific error type, from a specific user, containing a specific keyword in the output, or with a completion token count above a limit. This makes it practical to investigate specific failure categories or monitor the behavior of specific users.
from langsmith import Client
client = Client()
# Find slow traces (useful for performance investigation)
slow_runs = client.list_runs(
project_name='my-rag-app',
filter='gt(latency, 5)', # latency > 5 seconds
limit=20
)
# Find error traces
error_runs = client.list_runs(
project_name='my-rag-app',
filter='eq(error, true)',
limit=50
)
# Find traces from a specific user
user_runs = client.list_runs(
project_name='my-rag-app',
filter='has(metadata, user_id="user_abc123")',
limit=100
)
for run in slow_runs:
print(f'Slow run: {run.id}, latency: {run.end_time - run.start_time}')Comparing Experiments in LangSmith
LangSmith supports experiment comparison: run the same test dataset through two versions of your pipeline (e.g., chunk size 500 vs chunk size 1000), and compare them side by side on latency, cost, and quality metrics. This makes it easy to validate that a pipeline change is an improvement rather than a regression before deploying to production.
from langsmith.evaluation import evaluate
test_dataset = 'my-rag-eval-set'
# Run experiment A: chunk size 500
results_a = evaluate(
rag_pipeline_v1,
data=test_dataset,
evaluators=[correctness_evaluator, relevance_evaluator],
experiment_prefix='chunk-500'
)
# Run experiment B: chunk size 1000
results_b = evaluate(
rag_pipeline_v2,
data=test_dataset,
evaluators=[correctness_evaluator, relevance_evaluator],
experiment_prefix='chunk-1000'
)
# Compare in LangSmith UI: Experiments tab shows A vs B side by side
# Or compare programmatically:
print(f'Correctness - v1: {results_a.results["correctness"].mean():.2f}, v2: {results_b.results["correctness"].mean():.2f}')LangSmith in Production
LangSmith is available as a hosted SaaS at smith.langchain.com and as a self-hosted option. In production, tracing can be made asynchronous (non-blocking) to avoid adding latency to your critical path. You can also sample traces (e.g., trace only 10% of requests in high-traffic production) to control cost while maintaining visibility. The dashboard shows real-time graphs of request volume, latency, cost, and error rate.
import os
# Production configuration
os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'
os.environ['LANGCHAIN_PROJECT'] = 'production'
# Enable async tracing (non-blocking - does not add latency to requests)
os.environ['LANGCHAIN_CALLBACKS_BACKGROUND'] = 'true'
# Optional: sample 10% of traces to reduce cost in high-traffic scenarios
import random
def should_trace() -> bool:
return random.random() < 0.10 # 10% sampling rate
def handle_request(query):
config = RunnableConfig()
if not should_trace():
config = RunnableConfig(callbacks=[]) # disable tracing for this request
return rag_chain.invoke(query, config=config)LangSmith vs Custom Logging
You could build your own trace logging system, and for some use cases that is the right choice. LangSmith's advantages over custom logging are: zero-code integration with LangChain, a purpose-built UI for exploring LLM traces (not generic Kibana/Grafana dashboards), native support for evaluation and experiment comparison, and automatic token count and cost tracking. The tradeoff is vendor dependency and cost at scale.
Quick Check
Test your understanding of tracing with LangSmith from this lesson.
Lesson Recap
In this lesson you learned: LangSmith enables automatic end-to-end tracing of LangChain applications by setting three environment variables with no code changes, the @traceable decorator extends tracing to non-LangChain steps like database calls and preprocessing, and experiment comparison lets you validate pipeline improvements against a test dataset before deploying. Next up we explore Langfuse as a model-agnostic observability alternative.
Frequently asked questions
Is the “Tracing with LangSmith” lesson free?
Yes — the full text of “Tracing with LangSmith” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Tracing with LangSmith”?
Instrument your LangChain application with LangSmith tracing to record every chain step, LLM call, token count, and latency in a searchable trace explorer. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Tracing with LangSmith” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.