0Pricing
AI Engineering Academy · 课时

使用 LangSmith 进行追踪

使用 LangSmith 追踪为您的 LangChain 应用添加检测功能，记录每个链步骤、LLM 调用、令牌数量和延迟，并在可搜索的追踪浏览器中查看。

使用 LangSmith 进行追踪 是 CoddyKit 上的免费 AI Engineering Academy 课时。 这是第 2 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 AI Engineering Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 AI Engineering Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

What Is LangSmith?

LangSmith is an observability platform built specifically for LLM applications. It automatically captures traces of every LangChain run — every chain step, LLM call, tool execution, retrieval, and output parser — and displays them in a searchable, hierarchical trace explorer. You can filter traces by latency, cost, error status, or custom metadata, and replay any trace to debug failures.

# Install: pip install langsmith
import os

# Set environment variables to enable automatic tracing
os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_API_KEY'] = 'lsv2_...your_key_here...'
os.environ['LANGCHAIN_PROJECT'] = 'my-rag-app'  # project name in LangSmith UI

# That's all - LangChain now sends traces to LangSmith automatically
# No code changes needed to your chain or agent

Automatic Tracing with Zero Code Changes

The most compelling feature of LangSmith is that once you set the three environment variables, every LangChain operation is automatically traced with no additional code. Every LCEL chain, every ChatOpenAI call, every retriever call, every tool execution gets captured with inputs, outputs, timing, and token counts. You can deploy LangSmith tracing to production with a single environment variable change.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# This chain is automatically traced - no extra code needed
llm = ChatOpenAI(model='gpt-4o')
prompt = ChatPromptTemplate.from_template('Answer this question: {question}')
chain = prompt | llm | StrOutputParser()

# This call creates a trace in LangSmith showing:
# - The formatted prompt (with question substituted)
# - The LLM call with model, temperature, token counts
# - The parsed output
# - End-to-end latency and cost
result = chain.invoke({'question': 'What is RAG?'})
print(result)

Tracing RAG Pipelines

For RAG applications, LangSmith traces are especially valuable because they capture the entire retrieve-then-generate pipeline. You can see: which documents were retrieved, what their similarity scores were, how the context was formatted in the prompt, and what the LLM generated. This makes it immediately obvious whether a wrong answer was caused by bad retrieval or poor generation.

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.runnables import RunnablePassthrough

embeddings = OpenAIEmbeddings()
vectorstore = Chroma(embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={'k': 5})

rag_chain = (
    {'context': retriever, 'question': RunnablePassthrough()}
    | ChatPromptTemplate.from_template('Context: {context}\n\nQuestion: {question}\n\nAnswer:')
    | ChatOpenAI(model='gpt-4o')
    | StrOutputParser()
)

# LangSmith traces EVERY step:
# 1. Retriever: query embedding + vector search + returned documents (with scores)
# 2. Prompt: formatted template with context injected
# 3. LLM: full prompt, response, tokens, latency, cost
# 4. Parser: final string output
answer = rag_chain.invoke('What is the capital of France?')

Adding Metadata to Traces

By default, LangSmith traces contain the inputs and outputs of each step. You can enrich traces with custom metadata tags: the user ID, session ID, feature flag values, A/B test variant, or any other context that helps you filter and analyze traces in the UI. Use the RunnableConfig to pass metadata that will appear on every trace from that request.

from langchain_core.runnables import RunnableConfig

def handle_user_request(user_id: str, query: str, ab_variant: str):
    config = RunnableConfig(
        tags=['production', ab_variant],
        metadata={
            'user_id': user_id,
            'ab_variant': ab_variant,
            'feature': 'rag_qa'
        }
    )
    
    result = rag_chain.invoke(query, config=config)
    return result

# In LangSmith UI you can now:
# - Filter traces by user_id to debug a specific user's issue
# - Compare latency between ab_variant='A' and ab_variant='B'
# - See all traces tagged 'production' vs 'staging'

Manual Span Creation

For code that does not go through LangChain (custom API calls, database queries, preprocessing steps), you can create manual spans using the LangSmith client directly. This ensures your non-LangChain steps are captured in the same trace as the LangChain steps, giving you a complete picture of each request's execution path.

from langsmith import Client, traceable

client = Client()

# Decorate functions to auto-trace them
@traceable(name='preprocess_query')
def preprocess_query(raw_query: str) -> str:
    # This step is now traced even though it doesn't use LangChain
    cleaned = raw_query.strip().lower()
    cleaned = ' '.join(cleaned.split())  # normalize whitespace
    return cleaned

@traceable(name='fetch_user_history')
def fetch_user_history(user_id: str) -> list[str]:
    # Database call - also traced
    return db.query('SELECT message FROM chat_history WHERE user_id = ? ORDER BY timestamp DESC LIMIT 5', user_id)

# All three steps appear in the same trace
def handle_request(user_id: str, raw_query: str):
    query = preprocess_query(raw_query)         # traced
    history = fetch_user_history(user_id)       # traced
    result = rag_chain.invoke({'query': query, 'history': history})  # traced by LangChain
    return result

Evaluating Traces in LangSmith

LangSmith includes an evaluation framework that lets you run evaluators over your trace dataset. You can select a set of traced examples, run automated evaluators (including LLM-as-judge scorers for correctness and relevance), and compare the results across different pipeline versions. This turns your production traces into a feedback loop for improving your application.

from langsmith.evaluation import evaluate, LangChainStringEvaluator

# Create an evaluator that uses an LLM to judge correctness
correctness_evaluator = LangChainStringEvaluator(
    'qa',
    config={'llm': ChatOpenAI(model='gpt-4o')}
)

# Run evaluation against a dataset of traced examples
results = evaluate(
    rag_chain,
    data='my-rag-test-set',      # name of dataset in LangSmith
    evaluators=[correctness_evaluator],
    experiment_prefix='rag-v2-chunking-test'
)

print('Evaluation results:')
print(f'Correctness: {results.results["correctness"].mean():.2f}')
print(f'Average latency: {results.results["latency"].mean():.2f}s')

Creating Test Datasets from Traces

One of LangSmith's most powerful features is the ability to create test datasets directly from production traces. When you notice an interesting trace (a failure, an edge case, or a great example), you can add it to a dataset with a single click. Over time, you build a comprehensive regression test suite from real user queries rather than synthetic examples.

from langsmith import Client

client = Client()

# Create a dataset from existing traces
dataset = client.create_dataset('rag-regression-tests')

# Add examples from production traces (by trace ID)
for trace_id in failed_trace_ids:
    run = client.read_run(trace_id)
    client.create_example(
        inputs=run.inputs,
        outputs={'answer': run.outputs.get('output', '')},
        dataset_id=dataset.id,
        metadata={'source': 'production_failure', 'date': run.start_time.isoformat()}
    )

print(f'Added {len(failed_trace_ids)} examples to regression test dataset')

Filtering and Searching Traces

In production, you will have thousands of traces. LangSmith's UI and API support rich filtering and search: find traces with latency above a threshold, with a specific error type, from a specific user, containing a specific keyword in the output, or with a completion token count above a limit. This makes it practical to investigate specific failure categories or monitor the behavior of specific users.

from langsmith import Client

client = Client()

# Find slow traces (useful for performance investigation)
slow_runs = client.list_runs(
    project_name='my-rag-app',
    filter='gt(latency, 5)',  # latency > 5 seconds
    limit=20
)

# Find error traces
error_runs = client.list_runs(
    project_name='my-rag-app',
    filter='eq(error, true)',
    limit=50
)

# Find traces from a specific user
user_runs = client.list_runs(
    project_name='my-rag-app',
    filter='has(metadata, user_id="user_abc123")',
    limit=100
)

for run in slow_runs:
    print(f'Slow run: {run.id}, latency: {run.end_time - run.start_time}')

Comparing Experiments in LangSmith

LangSmith supports experiment comparison: run the same test dataset through two versions of your pipeline (e.g., chunk size 500 vs chunk size 1000), and compare them side by side on latency, cost, and quality metrics. This makes it easy to validate that a pipeline change is an improvement rather than a regression before deploying to production.

from langsmith.evaluation import evaluate

test_dataset = 'my-rag-eval-set'

# Run experiment A: chunk size 500
results_a = evaluate(
    rag_pipeline_v1,
    data=test_dataset,
    evaluators=[correctness_evaluator, relevance_evaluator],
    experiment_prefix='chunk-500'
)

# Run experiment B: chunk size 1000
results_b = evaluate(
    rag_pipeline_v2,
    data=test_dataset,
    evaluators=[correctness_evaluator, relevance_evaluator],
    experiment_prefix='chunk-1000'
)

# Compare in LangSmith UI: Experiments tab shows A vs B side by side
# Or compare programmatically:
print(f'Correctness - v1: {results_a.results["correctness"].mean():.2f}, v2: {results_b.results["correctness"].mean():.2f}')

LangSmith in Production

LangSmith is available as a hosted SaaS at smith.langchain.com and as a self-hosted option. In production, tracing can be made asynchronous (non-blocking) to avoid adding latency to your critical path. You can also sample traces (e.g., trace only 10% of requests in high-traffic production) to control cost while maintaining visibility. The dashboard shows real-time graphs of request volume, latency, cost, and error rate.

import os

# Production configuration
os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'
os.environ['LANGCHAIN_PROJECT'] = 'production'

# Enable async tracing (non-blocking - does not add latency to requests)
os.environ['LANGCHAIN_CALLBACKS_BACKGROUND'] = 'true'

# Optional: sample 10% of traces to reduce cost in high-traffic scenarios
import random

def should_trace() -> bool:
    return random.random() < 0.10  # 10% sampling rate

def handle_request(query):
    config = RunnableConfig()
    if not should_trace():
        config = RunnableConfig(callbacks=[])  # disable tracing for this request
    return rag_chain.invoke(query, config=config)

LangSmith vs Custom Logging

You could build your own trace logging system, and for some use cases that is the right choice. LangSmith's advantages over custom logging are: zero-code integration with LangChain, a purpose-built UI for exploring LLM traces (not generic Kibana/Grafana dashboards), native support for evaluation and experiment comparison, and automatic token count and cost tracking. The tradeoff is vendor dependency and cost at scale.

Quick Check

Test your understanding of tracing with LangSmith from this lesson.

Lesson Recap

In this lesson you learned: LangSmith enables automatic end-to-end tracing of LangChain applications by setting three environment variables with no code changes, the @traceable decorator extends tracing to non-LangChain steps like database calls and preprocessing, and experiment comparison lets you validate pipeline improvements against a test dataset before deploying. Next up we explore Langfuse as a model-agnostic observability alternative.

常见问题解答

「使用 LangSmith 进行追踪」课时是免费的吗？

是的 — 「使用 LangSmith 进行追踪」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 AI Engineering Academy 课程的其余内容，请升级到 CoddyKit PRO。 AI Engineering Academy 课程共包含 4 节课。

「使用 LangSmith 进行追踪」这节课中我会学到什么？

使用 LangSmith 追踪为您的 LangChain 应用添加检测功能，记录每个链步骤、LLM 调用、令牌数量和延迟，并在可搜索的追踪浏览器中查看。 你通过在浏览器中直接运行的动手代码来练习 AI Engineering Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 AI Engineering Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 AI Engineering Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 2 节课，共 4 节。

「使用 LangSmith 进行追踪」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 AI Engineering Academy 课中编写并运行代码吗？

能。每节 AI Engineering Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. LLM 应用为何难以调试
  2. 使用 LangSmith 进行追踪
  3. 使用 Langfuse 实现与模型无关的可观测性
  4. 针对延迟、成本和质量下降设置告警
← 返回 AI Engineering Academy