การติดตามแบบกระจายด้วย OpenTelemetry

In a microservice or even a single-service backend that talks to other HTTP APIs and a database, a single user request fans out into many operations. When something is slow or fails, logs alone can't show you the causal chain across process boundaries.

Distributed tracing solves this by giving every request a shared trace_id and breaking the work into nested spans:

• A trace = the whole journey of one request.
• A span = one timed unit of work (an HTTP handler, a DB query, an outbound call).
• Spans carry a parent_span_id, forming a tree.

OpenTelemetry (OTel) is the vendor-neutral standard and SDK we use to produce these traces from FastAPI and ship them to a backend like Jaeger, Tempo or an OTLP collector.

Before wiring anything up, understand the core objects you'll configure in code:

• TracerProvider — the factory that creates tracers; you configure it once at startup.
• Tracer — obtained from the provider, used to start spans.
• Span — has a name, start/end time, attributes (key/value tags), events, and a status.
• SpanProcessor — batches finished spans (use BatchSpanProcessor in production).
• Exporter — serializes spans and sends them out (OTLP over gRPC/HTTP).
• Context — the thread-/task-local carrier that holds the currently active span.

The flow is: TracerProvider → Tracer → Span → SpanProcessor → Exporter → backend.

For a FastAPI backend you install the SDK, the OTLP exporter, and the instrumentation packages:

• opentelemetry-sdk, opentelemetry-api
• opentelemetry-exporter-otlp
• opentelemetry-instrumentation-fastapi, -httpx, -sqlalchemy

At startup you build a TracerProvider with a Resource that names your service, attach a BatchSpanProcessor wrapping an OTLP exporter, then register it globally. The service.name attribute is critical — it's how your tracing backend groups spans.

The FastAPIInstrumentor wraps your app so every incoming request automatically becomes a server span. It reads the route, method, and status code, and — crucially — extracts the incoming trace context from request headers so this service's spans attach to the caller's trace.

Call configure_tracing() first, then instrument the app instance right after you create it. Order matters: the provider must be set globally before instrumentation reads it.

The magic that links spans across services is context propagation. OpenTelemetry defaults to the W3C Trace Context standard, which uses a traceparent HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

• 00 — version
• 4bf9...4736 — the 16-byte trace-id (shared across all services)
• 00f0...02b7 — the parent span-id of the caller
• 01 — trace flags (sampled bit)

On the way out, instrumented HTTP clients inject this header. On the way in, the server instrumentation extracts it. That's how a trace stays unbroken across the network.

When your FastAPI handler calls a downstream service, you must use an instrumented HTTP client so the traceparent header is injected automatically. With httpx, enable HTTPXClientInstrumentor once at startup.

Now every outbound request creates a client span that is a child of the current server span, and the downstream service continues the same trace.

Database queries are often the slowest part of a request, so you want them as spans too. For SQLAlchemy, the SQLAlchemyInstrumentor creates a span per statement and records the SQL and DB system as attributes.

You must instrument the engine (pass engine=... for sync, or the sync engine behind an async engine). These DB spans become children of the active request span, so a slow query shows up nested under the handler that triggered it.

Auto-instrumentation covers I/O boundaries, but your own logic is invisible. Add manual spans around meaningful units of work to see where time goes. Get a tracer from the global provider and use it as a context manager.

Because the span is started inside the active request context, it automatically nests under the request span — no manual parent wiring needed.

A span becomes useful when it carries context. Use:

• set_attribute(key, value) for searchable tags (user id, tenant, item count). Follow OTel semantic conventions where they exist.
• add_event(name, attributes) for time-stamped markers (e.g. "cache_miss").
• set_status(Status(StatusCode.ERROR)) and record_exception(exc) when something fails, so the span shows up red in your backend.

Never put secrets or full PII in attributes — traces are widely readable.

Tracing every request at full volume is expensive. Sampling decides which traces to keep. The recommended head-based sampler is ParentBasedTraceIdRatioBased:

• If an incoming request already carries a sampling decision (the 01 flag in traceparent), it is respected — so a trace is kept or dropped consistently across every service.
• For new root requests, it samples a fixed ratio (e.g. 10%).

This consistency is why parent-based sampling matters: you never want service A to keep a span while service B drops its child, leaving a broken trace.

Traces and logs are most powerful together. Inject the current trace_id and span_id into every log line so you can jump from a log entry straight to the full trace.

You read the active span context from trace.get_current_span().get_span_context(). With the logging instrumentation enabled, OTel can also auto-inject these fields into the standard logging record.

You can now instrument a FastAPI backend for distributed tracing end to end:

• Bootstrap a TracerProvider with a Resource (set service.name), a BatchSpanProcessor, and an OTLP exporter.
• Auto-instrument the app with FastAPIInstrumentor so every request is a server span that extracts incoming context.
• Propagate through downstream HTTP (HTTPXClientInstrumentor) and the database (SQLAlchemyInstrumentor) — the W3C traceparent header keeps the trace unbroken.
• Enrich with manual spans, attributes, events, status and recorded exceptions for your business logic.
• Sample with ParentBasedTraceIdRatioBased for consistent, affordable traces, and correlate logs via the active trace_id/span_id.

The result: one click takes you from a slow request to the exact nested span — handler, HTTP call, or query — that caused it.