Distributed Tracing für Latenz-Hotspots

In a microservice system a single user request can fan out across dozens of services. When it is slow, which service is to blame?

Distributed tracing answers this by attaching a shared trace_id to a request and recording a span for every operation it touches.

• A trace = the whole request journey
• A span = one timed unit of work

Each span carries timing and context so you can reconstruct the call tree.

• trace_id links all spans of one request
• span_id identifies the operation
• parent_id records who called it
• start/end timestamps give duration

For spans to join one trace, the trace_id must travel with the request. This is called context propagation.

Most systems inject standard headers like traceparent (W3C Trace Context) into outgoing HTTP calls and message metadata.

If propagation breaks, traces fragment and the call tree falls apart.

You create spans around the operations you want to measure. Auto-instrumentation covers common libraries; manual spans capture your own logic.

The example below wraps a function in a span using OpenTelemetry conventions.

Tracing UIs show spans as a waterfall. The widest bar that is NOT just waiting on a child is usually your hotspot.

• Long bars with no children = local CPU/IO cost
• Long bars full of children = downstream cost
• Gaps between spans = queueing or untraced work

Tracing every request is expensive. Sampling keeps volume manageable.

• Head sampling: decide at the start (e.g. keep 5%)
• Tail sampling: decide after the trace ends, keeping slow or errored traces

For debugging latency, tail sampling on high duration is invaluable.

A trace tells you where; logs tell you why. Stamp every log line with the active trace_id so you can jump from a slow span straight to its logs.

Attributes are key/value tags on a span (db statement, HTTP status). Events are timestamped points inside a span (retry, lock acquired).

Rich attributes let you filter traces like 'all spans where db.rows > 10000', turning tracing into a query tool.

Total latency is not the sum of all spans. Parallel spans overlap. The critical path is the chain of spans that actually determines end-to-end time.

Optimizing a span NOT on the critical path will not make the request faster.

Across queues, the consumer runs later than the producer. Propagate the context inside the message so the consumer span links back as a follows-from relationship instead of a parent-child one.

Put it together when an endpoint is slow in production:

• Filter traces for that endpoint sorted by duration
• Open the slowest trace and read the waterfall
• Identify the dominant span on the critical path
• Jump to that span's logs via trace_id
• Fix, then re-check the latency distribution

Test your understanding of distributed tracing.

You learned how distributed tracing reconstructs a request across services using trace_id, spans, and context propagation.

• Read waterfalls to find hotspots
• Focus on the critical path, not total span time
• Use tail sampling to keep slow traces
• Correlate spans with logs for the full story