LLM Apps in Production (RAG + Vector DB + Caching) · 강의

관측 가능성: 로그 기록, 지표, 추적

종합적인 로그 기록, 지표 수집, 분산 추적을 통합하여 LLM 애플리케이션의 동작을 깊이 있게 파악합니다.

관측 가능성: 로그 기록, 지표, 추적은(는) CoddyKit의 무료 LLM Apps in Production (RAG + Vector DB + Caching) 강의입니다. 이것은 4개 중 2번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 LLM Apps in Production (RAG + Vector DB + Caching) 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

What is Observability?

In this lesson, we'll explore observability, a crucial concept for managing complex software systems, especially LLM applications.

Observability means understanding the internal state of a system by examining the data it produces. Think of it as having X-ray vision into your application's behavior.

For LLM apps, this helps us answer critical questions like:

Why is a request slow?
Is the RAG retrieval working as expected?
Are we incurring unexpected costs?

Logs: Recording Events

Logs are timestamped records of events that happen within your application. They are like a diary of your system's activities.

For LLM applications, logs are essential for:

Tracking incoming user prompts.
Storing responses from the LLM.
Recording intermediate steps in a RAG pipeline (e.g., documents retrieved).
Capturing errors or warnings.

They provide detailed contextual information for debugging and post-mortem analysis.

Logging LLM Interactions

Here's a simple Python example demonstrating how to log an LLM interaction. We're using Python's built-in logging module.

This helps you see exactly what prompts were sent and what responses were received, which is vital for debugging and improving your application.

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

def call_llm(prompt):
    logging.info(f"LLM Request: '{prompt[:40]}...' ")
    # Simulate LLM processing
    response = f"Simulated response to: {prompt}"
    logging.info(f"LLM Response: '{response[:40]}...' ")
    return response

if __name__ == "__main__":
    user_prompt = "Explain observability simply."
    result = call_llm(user_prompt)
    print(f"Application output: {result}")

Metrics: Measuring Performance

Metrics are numerical measurements collected over time, providing aggregated insights into your system's health and performance.

Unlike logs, which are individual events, metrics are typically quantitative values that can be visualized as graphs and dashboards. Key metrics for LLM apps include:

Latency: How long it takes for the LLM to respond.
Token Usage: Input/output tokens consumed per request.
Error Rate: Percentage of failed LLM calls or RAG retrievals.
Cache Hit Rate: How often cached responses are used.

Collecting Custom Metrics

You can collect custom metrics to understand specific aspects of your LLM application. This example shows how to track the number of LLM calls and their average latency.

In a real-world scenario, you'd send these metrics to a monitoring system like Prometheus or Datadog.

import time

class LLMMetrics:
    def __init__(self):
        self.total_calls = 0
        self.total_latency = 0.0

    def record_call(self, duration):
        self.total_calls += 1
        self.total_latency += duration

    def get_avg_latency(self):
        if self.total_calls == 0:
            return 0.0
        return self.total_latency / self.total_calls

metrics_store = LLMMetrics()

def call_llm_with_metrics(prompt):
    start_time = time.time()
    # Simulate LLM processing
    time.sleep(0.05) # simulate 50ms work
    response = f"Simulated reply to: {prompt}"
    end_time = time.time()
    metrics_store.record_call(end_time - start_time)
    return response

if __name__ == "__main__":
    print("Collecting LLM call metrics...")
    call_llm_with_metrics("Hi")
    call_llm_with_metrics("How are you?")
    print(f"Total calls: {metrics_store.total_calls}")
    print(f"Avg latency: {metrics_store.get_avg_latency():.3f}s")

Tracing: Following Request Paths

Tracing is about following a single request as it flows through multiple services and components in a distributed system. This is especially vital for RAG applications that involve many steps: user input, embedding generation, vector DB lookup, LLM call, etc.

A trace visualizes the entire journey of a request, showing the exact path it took and the time spent in each operation.

Traces, Spans, and Context

A trace is a complete end-to-end journey of a request. It's composed of multiple spans.

A span represents a single operation or unit of work within a trace (e.g., 'retrieve documents', 'call embedding model', 'invoke LLM').
Spans have a parent-child relationship, forming a tree structure that shows dependencies.
Context propagation ensures that a unique trace ID follows the request across different services, linking all related spans together.

This helps pinpoint bottlenecks or failures across microservices.

OpenTelemetry for Tracing

While implementing tracing from scratch is complex, tools like OpenTelemetry (an open-source observability framework) provide standardized ways to instrument your code.

You'd use OpenTelemetry SDKs to:

Start a new trace when a request comes in.
Create new spans for each significant operation (e.g., a function call to a vector database or an LLM API).
Propagate the trace context to downstream services.

This allows you to visualize the full request flow in a tracing UI.

The Observability Triangle

Logs, metrics, and traces are often called the "observability triangle" because they offer complementary views of your system:

Logs: The granular details and events.
Metrics: The aggregated numbers and trends.
Traces: The end-to-end journey of a request.

Together, they provide a comprehensive understanding of your LLM application's behavior, making it easier to diagnose issues, optimize performance, and ensure reliability in production.

Quick Check: Observability

You've learned about the three pillars of observability. Let's see if you can distinguish their primary uses.

Recap: Deep Insights

Congratulations! You've explored the world of observability for LLM applications.

We defined observability as understanding internal system state from external data.
We learned about logs for detailed event recording.
We covered metrics for aggregated performance measurements.
We understood traces for visualizing end-to-end request flows.

By integrating these three pillars, you gain powerful insights, enabling you to build more reliable, performant, and cost-efficient LLM systems.

자주 묻는 질문

“관측 가능성: 로그 기록, 지표, 추적” 강의는 무료인가요?

네 — “관측 가능성: 로그 기록, 지표, 추적” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 LLM Apps in Production (RAG + Vector DB + Caching) 강의 전체를 잠금 해제할 수 있습니다. LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 총 4개의 강의가 포함되어 있습니다.

“관측 가능성: 로그 기록, 지표, 추적”에서 뭘 배우나요?

종합적인 로그 기록, 지표 수집, 분산 추적을 통합하여 LLM 애플리케이션의 동작을 깊이 있게 파악합니다. 브라우저에서 직접 실행하는 실습 코드로 LLM Apps in Production (RAG + Vector DB + Caching)을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

LLM Apps in Production (RAG + Vector DB + Caching)을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 LLM Apps in Production (RAG + Vector DB + Caching)은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 2번째 강의입니다.

“관측 가능성: 로그 기록, 지표, 추적” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 LLM Apps in Production (RAG + Vector DB + Caching) 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 LLM Apps in Production (RAG + Vector DB + Caching) 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← LLM Apps in Production (RAG + Vector DB + Caching)(으)로 돌아가기