PricingLoginStart free
LangChain / RAG / Vector DBs · 课时

监控和记录 RAG 应用

实施可靠的监控和记录方案，以跟踪性能、发现问题并深入了解用户交互。

第 1 / 4 课11 个步骤

监控和记录 RAG 应用 是 CoddyKit 上的免费 LangChain / RAG / Vector DBs 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 LangChain / RAG / Vector DBs 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Monitor RAG Applications?

When you build a RAG (Retrieval Augmented Generation) system, it's like building a complex machine. To ensure it runs smoothly and reliably, you need to know what's happening inside!

Monitoring and logging are essential tools for understanding your RAG application's performance, identifying issues, and ensuring a great user experience.

Key RAG Metrics to Monitor

What exactly should you keep an eye on in a RAG system? Here are some crucial metrics:

  • Latency: How long does it take to retrieve documents? How long for the LLM to generate a response?
  • Retrieval Success: Are relevant documents consistently found?
  • LLM Response Quality: Is the LLM generating accurate, coherent, and helpful answers?
  • Token Usage: How many tokens are being consumed by the LLM (for cost tracking)?
  • Error Rates: Are there frequent errors in any part of the RAG pipeline?

Basic Python Logging

Let's start with simple logging using Python's built-in logging module. It allows you to record messages at different severity levels (e.g., INFO, WARNING, ERROR).

This snippet shows how to configure basic logging and record a message during a simulated RAG process.

import logging

# Configure basic logging
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s - %(message)s')

def process_query(query):
    logging.info(f"Received query: {query}")
    # Simulate RAG processing steps here
    response = f"Processed '{query}' successfully."
    logging.info(f"Generated response: {response}")
    return response

if __name__ == "__main__":
    print("--- Starting RAG simulation ---")
    result = process_query("What is a vector database?")
    print(f"Final result: {result}")
    print("--- Simulation finished ---")

The Power of Structured Logging

While basic logs are helpful, structured logging takes it a step further. Instead of plain text, logs are generated in a structured format, often JSON.

Why is this better for RAG?

  • Easier Parsing: Machines can easily read and extract specific data points.
  • Better Search & Analysis: You can query logs by specific fields (e.g., all logs for a certain request_id or query_type).
  • Integration: Works seamlessly with log management systems like Elasticsearch.

Logging User Queries in RAG

One of the first things to log in your RAG application is the user's input query. This helps you understand what users are asking and how your system is interpreting their requests.

It's good practice to associate a unique ID with each user request for easier tracing.

import logging
import uuid
import json

logging.basicConfig(level=logging.INFO,
                    format='%(message)s') # We'll format the output ourselves

def log_user_query(query):
    request_id = str(uuid.uuid4())[:8] # Short unique ID
    log_entry = {
        "timestamp": logging.Formatter().formatTime(logging.LogRecord('', 0, '', 0, '', '', '', '')),
        "level": "INFO",
        "event": "user_query",
        "request_id": request_id,
        "query": query
    }
    logging.info(json.dumps(log_entry))
    return request_id

if __name__ == "__main__":
    print("--- Logging user query ---")
    req_id = log_user_query("Tell me about the history of AI.")
    print(f"Query logged with ID: {req_id}")

Logging Retrieved Documents

After a user query, your RAG system retrieves relevant documents. Logging which documents were selected is crucial for debugging. If the LLM gives a bad answer, you can check if it received poor context.

Logging document IDs or titles helps you pinpoint issues with your retrieval component.

import logging
import json

logging.basicConfig(level=logging.INFO,
                    format='%(message)s')

def retrieve_docs(query, request_id):
    # Simulate document retrieval
    docs = [
        {"id": "doc_101", "title": "Intro to Embeddings"},
        {"id": "doc_105", "title": "Vector DB Architectures"}
    ]
    log_entry = {
        "timestamp": logging.Formatter().formatTime(logging.LogRecord('', 0, '', 0, '', '', '', '')),
        "level": "INFO",
        "event": "documents_retrieved",
        "request_id": request_id,
        "query": query,
        "retrieved_docs": [d['id'] for d in docs]
    }
    logging.info(json.dumps(log_entry))
    return docs

if __name__ == "__main__":
    print("--- Simulating document retrieval ---")
    req_id = "abc123de"
    retrieved = retrieve_docs("What are embeddings?", req_id)
    print(f"Retrieved document IDs for request {req_id}.")

Logging LLM Responses

Finally, you'll want to log the answer generated by the Large Language Model. This helps you review the quality of responses and track how the LLM performs over time.

It's also useful to log the exact prompt sent to the LLM, as prompt engineering heavily influences the output.

import logging
import json

logging.basicConfig(level=logging.INFO,
                    format='%(message)s')

def generate_llm_response(query, retrieved_context, request_id):
    # Simulate LLM generating an answer
    prompt_sent = f"Based on: {retrieved_context}, answer: {query}"
    llm_response = f"The LLM generated an answer about '{query}' based on the provided context."

    log_entry = {
        "timestamp": logging.Formatter().formatTime(logging.LogRecord('', 0, '', 0, '', '', '', '')),
        "level": "INFO",
        "event": "llm_response_generated",
        "request_id": request_id,
        "prompt_sent": prompt_sent,
        "llm_output": llm_response
    }
    logging.info(json.dumps(log_entry))
    return llm_response

if __name__ == "__main__":
    print("--- Simulating LLM response ---")
    req_id = "fgh456ij"
    query = "Explain RAG architecture."
    context = "Documents on RAG components."
    final_answer = generate_llm_response(query, context, req_id)
    print(f"LLM response logged for request {req_id}.")

Common Monitoring Tools

While Python's logging module handles generating logs, you'll need external tools to collect, store, visualize, and alert on them in a production environment:

  • Prometheus: An open-source system for collecting and storing time-series metrics.
  • Grafana: A popular dashboarding and visualization tool that works well with Prometheus.
  • ELK Stack: A suite including Elasticsearch (for log storage and search), Logstash (for log processing), and Kibana (for visualization).
  • Cloud Services: AWS CloudWatch, Google Cloud Logging/Monitoring, Azure Monitor offer integrated solutions.

Debugging with Logs & Metrics

Logs and metrics are your best friends when things go wrong. They provide a breadcrumb trail to follow and diagnose issues:

  • Slow Performance: Check latency metrics for retrieval or LLM generation.
  • Irrelevant Answers: Examine retrieved documents in logs to see if the context was appropriate.
  • Hallucinations: Review LLM outputs and prompts to understand why incorrect information might have been generated.
  • Errors: Filter logs for ERROR or WARNING levels to quickly find where the system is breaking.

Quick Check: RAG Logging

Understanding what to log is key to a healthy RAG system. Which of the following are crucial pieces of information to log within a RAG application to understand its behavior and debug issues?

在交互式课程中回答

Recap & Next Steps

Great job! In this lesson, you learned why monitoring and logging are indispensable for production RAG systems. We covered:

  • The importance of tracking key RAG metrics like latency and token usage.
  • How to implement basic and structured logging in Python.
  • Crucial data points to log at each stage: user queries, retrieved documents, and LLM responses.
  • An overview of common monitoring and logging tools.

By effectively monitoring and logging, you can ensure your RAG application is robust, performant, and reliable.

免费开始

用 AI 导师学习 LangChain / RAG / Vector DBs — 免费

在浏览器中编写并运行真实代码，获得全天候 AI 导师的即时帮助，并在网页或应用中继续学习。

课程
12
课程
48
免费开始学习查看完整课程

常见问题解答

「监控和记录 RAG 应用」课时是免费的吗？

是的 — 「监控和记录 RAG 应用」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 LangChain / RAG / Vector DBs 课程的其余内容，请升级到 CoddyKit PRO。 LangChain / RAG / Vector DBs 课程共包含 4 节课。

「监控和记录 RAG 应用」这节课中我会学到什么？

实施可靠的监控和记录方案，以跟踪性能、发现问题并深入了解用户交互。 你通过在浏览器中直接运行的动手代码来练习 LangChain / RAG / Vector DBs，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 LangChain / RAG / Vector DBs 需要有经验吗？

无需任何先前经验。CoddyKit 上的 LangChain / RAG / Vector DBs 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。

「监控和记录 RAG 应用」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 LangChain / RAG / Vector DBs 课中编写并运行代码吗？

能。每节 LangChain / RAG / Vector DBs 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 监控和记录 RAG 应用
  2. 缓存与性能优化
  3. 云端 RAG 部署策略
  4. 处理并发与速率限制
← 返回 LangChain / RAG / Vector DBs