AI Engineering Academy · Lesson

Why LLM Apps Are Hard to Debug

Understand why traditional logging is insufficient for LLM applications, what information you need to diagnose failures in RAG and agent pipelines, and the tracing data model.

Why LLM Apps Are Hard to Debug is a free AI Engineering Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Unique Challenges of LLM Debugging

Traditional software fails deterministically: given the same input, it always produces the same output, and a stack trace points directly to the line that failed. LLM applications break these assumptions. The same prompt can produce different outputs on different calls, failures are often silent (wrong answer instead of exception), and the cause may be buried in a prompt used 5 steps earlier in a chain. Standard logging and debugging tools are simply not designed for this.

Non-Determinism Makes Reproduction Hard

LLM outputs are non-deterministic by default. Even with temperature=0, the same prompt may produce slightly different outputs due to batch processing and numerical precision. This means bugs are intermittent: a prompt that fails 20% of the time will pass your test suite if you only run it once. Reproducing a specific failure requires logging the exact input, model parameters, and output at the time of failure — not just the input.

import json
import time

def logged_llm_call(client, messages, model, temperature, **kwargs):
    request_id = f'{int(time.time() * 1000)}-{id(messages)}'
    
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        **kwargs
    )
    
    # Log EVERYTHING needed to reproduce this exact call
    log_entry = {
        'request_id': request_id,
        'model': model,
        'temperature': temperature,
        'messages': messages,
        'response': response.choices[0].message.content,
        'finish_reason': response.choices[0].finish_reason,
        'usage': response.usage.model_dump(),
        'timestamp': time.time()
    }
    write_to_trace_store(log_entry)
    return response

Silent Failures: Wrong, Not Broken

The most insidious LLM failures are silent failures: the API call succeeds (HTTP 200, no exception), but the answer is wrong, hallucinated, incomplete, or off-topic. Your application happily processes the wrong answer and returns it to the user with no indication anything failed. Traditional monitoring that only watches for exceptions and error codes will never catch these — you need semantic monitoring of output quality.

# This succeeds with HTTP 200 but returns wrong information
response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'What is the boiling point of water at sea level?'}]
)

output = response.choices[0].message.content
# response.status_code: None (not relevant - always 200 if we got here)
# No exception thrown
# But if output is '90 degrees Celsius', it is WRONG and your app will serve bad data

# You need semantic validation:
def validate_boiling_point_answer(text: str) -> bool:
    return '100' in text  # Rough check - real validation is more sophisticated

Multi-Step Chains: Where Did It Go Wrong?

In a RAG pipeline or agent chain, a failure in the final response might trace back to a retrieval step that returned irrelevant chunks, which itself traces back to a chunking strategy that split a key sentence across two chunks, which traces back to an embedding model that did not handle technical jargon well. Without per-step tracing, you see only the wrong final answer with no way to pinpoint which step introduced the error.

# Without tracing: you see only the final wrong answer
def rag_pipeline_naive(query):
    chunks = retrieve(query)         # step 1 - might return bad chunks
    context = format_context(chunks) # step 2 - might truncate key info
    answer = generate(query, context) # step 3 - LLM gets bad context
    return answer  # WRONG - but why?

# With tracing: you can see each step's input and output
def rag_pipeline_traced(query):
    with trace_span('retrieve') as span:
        chunks = retrieve(query)
        span.set_attribute('num_chunks', len(chunks))
        span.set_attribute('top_chunk_score', chunks[0]['score'] if chunks else 0)
    
    with trace_span('format_context') as span:
        context = format_context(chunks)
        span.set_attribute('context_length', len(context))
    
    with trace_span('generate') as span:
        answer = generate(query, context)
        span.set_attribute('answer_length', len(answer))
    
    return answer  # Now you can diagnose: was retrieve the problem?

Token Count and Cost Surprises

Without instrumentation, token counts and costs are invisible until the monthly bill arrives. A system prompt that grew from 500 to 5000 tokens due to an oversight, a retrieval function returning 20 chunks instead of 5, or a loop that calls the LLM 100 times instead of 10 — all of these silently multiply your costs. Instrument every LLM call to log prompt tokens, completion tokens, and estimated cost so anomalies are visible in real time.

COST_PER_1K = {'gpt-4o': {'input': 0.005, 'output': 0.015},
               'gpt-4o-mini': {'input': 0.000150, 'output': 0.000600}}

def compute_cost(model: str, usage) -> float:
    pricing = COST_PER_1K.get(model, {'input': 0.005, 'output': 0.015})
    input_cost = (usage.prompt_tokens / 1000) * pricing['input']
    output_cost = (usage.completion_tokens / 1000) * pricing['output']
    return input_cost + output_cost

def instrumented_call(client, model, messages):
    response = client.chat.completions.create(model=model, messages=messages)
    cost = compute_cost(model, response.usage)
    
    # Alert if single call is unexpectedly expensive
    if cost > 0.10:  # more than 10 cents for one call
        print(f'WARNING: Expensive LLM call: ${cost:.4f} ({response.usage.prompt_tokens} prompt tokens)')
    
    metrics.record('llm_cost_usd', cost, tags={'model': model})
    metrics.record('llm_prompt_tokens', response.usage.prompt_tokens)
    return response

Latency: Which Step Is Slow?

Users experience LLM latency as a single wait time, but it is actually the sum of many individual steps: vector database query, document retrieval, prompt assembly, API network call, token generation, and response parsing. Without per-step timing, you cannot tell whether a slow response is due to a slow retriever or a slow LLM call. Instrument each step with latency measurements to identify your real bottleneck.

import time
from contextlib import contextmanager

@contextmanager
def timed(name: str, metrics_client):
    start = time.monotonic()
    try:
        yield
    finally:
        elapsed_ms = (time.monotonic() - start) * 1000
        metrics_client.histogram(f'step_latency_ms', elapsed_ms, tags={'step': name})
        if elapsed_ms > 2000:  # flag steps taking more than 2 seconds
            print(f'SLOW STEP [{name}]: {elapsed_ms:.0f}ms')

# Usage
def rag_with_timing(query, metrics):
    with timed('embed_query', metrics):
        query_embedding = embed(query)
    
    with timed('vector_search', metrics):
        chunks = vector_db.search(query_embedding, top_k=5)
    
    with timed('llm_generate', metrics):
        answer = generate(query, chunks)
    
    return answer

What Information You Actually Need

To diagnose any LLM application failure, you need to capture and store: the complete input prompt (system + all messages), the model and parameters used (temperature, max_tokens), the complete output, token counts and estimated cost, latency per step, any tool calls and their results, and a session or request ID that ties all steps of one user request together. This is the minimum viable tracing dataset.

from dataclasses import dataclass, field
from typing import Optional
import time

@dataclass
class LLMTrace:
    request_id: str
    session_id: str
    step_name: str
    model: str
    temperature: float
    system_prompt: str
    user_messages: list[dict]
    response: str
    finish_reason: str
    prompt_tokens: int
    completion_tokens: int
    cost_usd: float
    latency_ms: float
    tool_calls: list[dict] = field(default_factory=list)
    error: Optional[str] = None
    timestamp: float = field(default_factory=time.time)

    def is_anomalous(self) -> bool:
        return (
            self.cost_usd > 0.10 or
            self.latency_ms > 10000 or
            self.finish_reason == 'length' or  # was cut off
            self.error is not None
        )

Trace Correlation with Request IDs

A single user request might trigger 10 LLM calls across different services. Without a correlation ID that flows through all of them, you cannot group these calls into a single trace. Inject a unique request ID at the entry point of every user request and pass it in every downstream LLM call, database query, and log message. This makes it possible to reconstruct the complete execution path for any specific user request.

import uuid
from contextvars import ContextVar

# Thread-safe request ID propagation using context variables
request_id_var: ContextVar[str] = ContextVar('request_id', default='unknown')

def handle_user_request(query: str):
    # Set request ID at the entry point
    req_id = str(uuid.uuid4())[:8]
    request_id_var.set(req_id)
    return rag_pipeline(query)

def get_current_request_id() -> str:
    return request_id_var.get()

# Every LLM call logs with the same request_id
def log_llm_call(model, prompt, response):
    logger.info('LLM call', extra={
        'request_id': get_current_request_id(),  # automatically correlates all calls
        'model': model,
        'prompt_length': len(prompt),
        'response_length': len(response)
    })

The LLM Observability Stack

The LLM observability stack has three layers: Logging captures structured records of every LLM call (LangSmith, Langfuse, custom logs). Metrics track aggregated numbers over time: request count, average latency, error rate, daily cost. Tracing records the causal chain of steps within a single request. Together these three pillars give you the visibility to diagnose failures, catch regressions, and optimize performance.

Alerting on Quality Degradation

Unlike traditional software where errors are binary (working/broken), LLM quality degrades gradually. A prompt change might reduce answer quality from 85% to 70% without any exceptions being thrown. Monitor quality by running automated evaluation (LLM-as-judge scoring) on a sample of production responses daily. Alert when the rolling average quality score drops below a threshold, before users start complaining.

Starting Your Observability Practice

Do not wait until you have a production incident to add observability. Start with three minimal steps: (1) log every LLM call with its full input, output, and token counts to a database or file, (2) assign a request ID to every user interaction and include it in all log entries, and (3) add per-step timing to your pipeline. These three things alone will make 80% of debugging tasks solvable in minutes rather than hours.

Quick Check

Test your understanding of why LLM apps are hard to debug from this lesson.

Lesson Recap

In this lesson you learned: non-determinism makes LLM bugs intermittent and hard to reproduce without capturing the full request context, silent failures (wrong answers from successful API calls) bypass traditional error monitoring and require semantic quality checks, and per-step tracing with request ID correlation is the minimum needed to diagnose failures in multi-step RAG and agent pipelines. Next up we implement tracing with LangSmith.

Frequently asked questions

Is the “Why LLM Apps Are Hard to Debug” lesson free?

Yes — the full text of “Why LLM Apps Are Hard to Debug” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Why LLM Apps Are Hard to Debug”?

Understand why traditional logging is insufficient for LLM applications, what information you need to diagnose failures in RAG and agent pipelines, and the tracing data model. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Why LLM Apps Are Hard to Debug” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

Why LLM Apps Are Hard to Debug
Tracing with LangSmith
Langfuse for Model-Agnostic Observability
Alerting on Latency, Cost, and Quality Degradation

← Back to AI Engineering Academy