0Pricing
Production Debugging & Incident Response Playbook · Lesson

Distributed Tracing for Latency Hotspots

Learn to use distributed tracing to follow a single request across services, identify latency hotspots, and correlate traces with logs during production debugging.

Distributed Tracing for Latency Hotspots is a free Production Debugging & Incident Response Playbook lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Production Debugging & Incident Response Playbook learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Distributed Tracing

In a microservice system a single user request can fan out across dozens of services. When it is slow, which service is to blame?

Distributed tracing answers this by attaching a shared trace_id to a request and recording a span for every operation it touches.

  • A trace = the whole request journey
  • A span = one timed unit of work

Anatomy of a Span

Each span carries timing and context so you can reconstruct the call tree.

  • trace_id links all spans of one request
  • span_id identifies the operation
  • parent_id records who called it
  • start/end timestamps give duration
{
  "trace_id": "abc123",
  "span_id": "s2",
  "parent_id": "s1",
  "name": "db.query.users",
  "start_ms": 1042,
  "end_ms": 1310
}

Context Propagation

For spans to join one trace, the trace_id must travel with the request. This is called context propagation.

Most systems inject standard headers like traceparent (W3C Trace Context) into outgoing HTTP calls and message metadata.

If propagation breaks, traces fragment and the call tree falls apart.

GET /orders HTTP/1.1
traceparent: 00-abc123-s1-01

Instrumenting Code

You create spans around the operations you want to measure. Auto-instrumentation covers common libraries; manual spans capture your own logic.

The example below wraps a function in a span using OpenTelemetry conventions.

with tracer.start_as_current_span('charge_card') as span:
    span.set_attribute('amount', 42)
    result = payment.charge(42)
    span.set_attribute('status', result.status)

Reading the Waterfall

Tracing UIs show spans as a waterfall. The widest bar that is NOT just waiting on a child is usually your hotspot.

  • Long bars with no children = local CPU/IO cost
  • Long bars full of children = downstream cost
  • Gaps between spans = queueing or untraced work

Sampling Strategies

Tracing every request is expensive. Sampling keeps volume manageable.

  • Head sampling: decide at the start (e.g. keep 5%)
  • Tail sampling: decide after the trace ends, keeping slow or errored traces

For debugging latency, tail sampling on high duration is invaluable.

Correlating Traces and Logs

A trace tells you where; logs tell you why. Stamp every log line with the active trace_id so you can jump from a slow span straight to its logs.

import logging
logging.info('cache miss', extra={'trace_id': current_trace_id()})

Span Attributes and Events

Attributes are key/value tags on a span (db statement, HTTP status). Events are timestamped points inside a span (retry, lock acquired).

Rich attributes let you filter traces like 'all spans where db.rows > 10000', turning tracing into a query tool.

span.add_event('retry', {'attempt': 2})
span.set_attribute('db.rows', 12044)

Finding the Critical Path

Total latency is not the sum of all spans. Parallel spans overlap. The critical path is the chain of spans that actually determines end-to-end time.

Optimizing a span NOT on the critical path will not make the request faster.

Tracing Async and Queues

Across queues, the consumer runs later than the producer. Propagate the context inside the message so the consumer span links back as a follows-from relationship instead of a parent-child one.

producer:  msg.headers['traceparent'] = inject_context()
consumer:  ctx = extract_context(msg.headers)

A Debugging Workflow

Put it together when an endpoint is slow in production:

  • Filter traces for that endpoint sorted by duration
  • Open the slowest trace and read the waterfall
  • Identify the dominant span on the critical path
  • Jump to that span's logs via trace_id
  • Fix, then re-check the latency distribution

Quick Check

Test your understanding of distributed tracing.

Recap

You learned how distributed tracing reconstructs a request across services using trace_id, spans, and context propagation.

  • Read waterfalls to find hotspots
  • Focus on the critical path, not total span time
  • Use tail sampling to keep slow traces
  • Correlate spans with logs for the full story

Frequently asked questions

Is the “Distributed Tracing for Latency Hotspots” lesson free?

Yes — the full text of “Distributed Tracing for Latency Hotspots” is free to read here on the web, and the Production Debugging & Incident Response Playbook course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Production Debugging & Incident Response Playbook course, upgrade to CoddyKit PRO.

What will I learn in “Distributed Tracing for Latency Hotspots”?

Learn to use distributed tracing to follow a single request across services, identify latency hotspots, and correlate traces with logs during production debugging. You practise Production Debugging & Incident Response Playbook with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Production Debugging & Incident Response Playbook?

No prior experience is required. Production Debugging & Incident Response Playbook on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Distributed Tracing for Latency Hotspots” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Production Debugging & Incident Response Playbook lesson?

Yes. Every Production Debugging & Incident Response Playbook lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Remote Debugging Live Applications
  2. Post-mortem Debugging with Core Dumps
  3. Memory and CPU Profiling Techniques
  4. Distributed Tracing for Latency Hotspots
← Back to Production Debugging & Incident Response Playbook