0Pricing
FastAPI Backend Development Bootcamp · Lesson

Distributed Tracing with OpenTelemetry

Auto-instrument FastAPI and propagate trace context through downstream HTTP and database calls.

Distributed Tracing with OpenTelemetry is a free FastAPI Backend Development Bootcamp lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the FastAPI Backend Development Bootcamp learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Distributed Tracing?

In a microservice or even a single-service backend that talks to other HTTP APIs and a database, a single user request fans out into many operations. When something is slow or fails, logs alone can't show you the causal chain across process boundaries.

Distributed tracing solves this by giving every request a shared trace_id and breaking the work into nested spans:

  • A trace = the whole journey of one request.
  • A span = one timed unit of work (an HTTP handler, a DB query, an outbound call).
  • Spans carry a parent_span_id, forming a tree.

OpenTelemetry (OTel) is the vendor-neutral standard and SDK we use to produce these traces from FastAPI and ship them to a backend like Jaeger, Tempo or an OTLP collector.

The OpenTelemetry Data Model

Before wiring anything up, understand the core objects you'll configure in code:

  • TracerProvider — the factory that creates tracers; you configure it once at startup.
  • Tracer — obtained from the provider, used to start spans.
  • Span — has a name, start/end time, attributes (key/value tags), events, and a status.
  • SpanProcessor — batches finished spans (use BatchSpanProcessor in production).
  • Exporter — serializes spans and sends them out (OTLP over gRPC/HTTP).
  • Context — the thread-/task-local carrier that holds the currently active span.

The flow is: TracerProvider → Tracer → Span → SpanProcessor → Exporter → backend.

Installing and Bootstrapping the SDK

For a FastAPI backend you install the SDK, the OTLP exporter, and the instrumentation packages:

  • opentelemetry-sdk, opentelemetry-api
  • opentelemetry-exporter-otlp
  • opentelemetry-instrumentation-fastapi, -httpx, -sqlalchemy

At startup you build a TracerProvider with a Resource that names your service, attach a BatchSpanProcessor wrapping an OTLP exporter, then register it globally. The service.name attribute is critical — it's how your tracing backend groups spans.

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)


def configure_tracing() -> None:
    resource = Resource.create({
        "service.name": "orders-api",
        "service.version": "1.4.0",
        "deployment.environment": "production",
    })
    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

Auto-Instrumenting FastAPI

The FastAPIInstrumentor wraps your app so every incoming request automatically becomes a server span. It reads the route, method, and status code, and — crucially — extracts the incoming trace context from request headers so this service's spans attach to the caller's trace.

Call configure_tracing() first, then instrument the app instance right after you create it. Order matters: the provider must be set globally before instrumentation reads it.

from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

from .tracing import configure_tracing

configure_tracing()

app = FastAPI(title="orders-api")
FastAPIInstrumentor.instrument_app(app)


@app.get("/orders/{order_id}")
async def get_order(order_id: int):
    # This handler already runs inside an auto-created server span.
    return {"order_id": order_id, "status": "shipped"}

Trace Context Propagation: The W3C traceparent Header

The magic that links spans across services is context propagation. OpenTelemetry defaults to the W3C Trace Context standard, which uses a traceparent HTTP header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

  • 00 — version
  • 4bf9...4736 — the 16-byte trace-id (shared across all services)
  • 00f0...02b7 — the parent span-id of the caller
  • 01 — trace flags (sampled bit)

On the way out, instrumented HTTP clients inject this header. On the way in, the server instrumentation extracts it. That's how a trace stays unbroken across the network.

Propagating Through Outbound HTTP Calls

When your FastAPI handler calls a downstream service, you must use an instrumented HTTP client so the traceparent header is injected automatically. With httpx, enable HTTPXClientInstrumentor once at startup.

Now every outbound request creates a client span that is a child of the current server span, and the downstream service continues the same trace.

import httpx
from fastapi import FastAPI
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

HTTPXClientInstrumentor().instrument()

app = FastAPI()


@app.get("/orders/{order_id}/full")
async def get_full_order(order_id: int):
    async with httpx.AsyncClient(base_url="http://payments") as client:
        # traceparent is injected automatically on this request.
        resp = await client.get(f"/charges/{order_id}")
    return {"order_id": order_id, "payment": resp.json()}

Propagating Through Database Calls

Database queries are often the slowest part of a request, so you want them as spans too. For SQLAlchemy, the SQLAlchemyInstrumentor creates a span per statement and records the SQL and DB system as attributes.

You must instrument the engine (pass engine=... for sync, or the sync engine behind an async engine). These DB spans become children of the active request span, so a slow query shows up nested under the handler that triggered it.

from sqlalchemy.ext.asyncio import create_async_engine
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

engine = create_async_engine("postgresql+asyncpg://app:secret@db/orders")

# For async engines, instrument the underlying sync engine.
SQLAlchemyInstrumentor().instrument(engine=engine.sync_engine)

# Every statement run through this engine now emits a DB span
# nested under the current request span automatically.

Creating Manual Spans for Business Logic

Auto-instrumentation covers I/O boundaries, but your own logic is invisible. Add manual spans around meaningful units of work to see where time goes. Get a tracer from the global provider and use it as a context manager.

Because the span is started inside the active request context, it automatically nests under the request span — no manual parent wiring needed.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)


def price_order(items: list[dict]) -> float:
    with tracer.start_as_current_span("price_order") as span:
        span.set_attribute("order.item_count", len(items))
        subtotal = sum(i["price"] * i["qty"] for i in items)
        tax = round(subtotal * 0.20, 2)
        total = subtotal + tax
        span.set_attribute("order.total", total)
        return total

Enriching Spans with Attributes, Events and Status

A span becomes useful when it carries context. Use:

  • set_attribute(key, value) for searchable tags (user id, tenant, item count). Follow OTel semantic conventions where they exist.
  • add_event(name, attributes) for time-stamped markers (e.g. "cache_miss").
  • set_status(Status(StatusCode.ERROR)) and record_exception(exc) when something fails, so the span shows up red in your backend.

Never put secrets or full PII in attributes — traces are widely readable.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)


def reserve_stock(sku: str, qty: int, available: int) -> None:
    with tracer.start_as_current_span("reserve_stock") as span:
        span.set_attribute("inventory.sku", sku)
        span.set_attribute("inventory.requested_qty", qty)
        if qty > available:
            span.add_event("stock_shortfall", {"available": available})
            exc = ValueError(f"Only {available} of {sku} in stock")
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise exc
        span.set_status(Status(StatusCode.OK))

Sampling: Controlling Trace Volume

Tracing every request at full volume is expensive. Sampling decides which traces to keep. The recommended head-based sampler is ParentBasedTraceIdRatioBased:

  • If an incoming request already carries a sampling decision (the 01 flag in traceparent), it is respected — so a trace is kept or dropped consistently across every service.
  • For new root requests, it samples a fixed ratio (e.g. 10%).

This consistency is why parent-based sampling matters: you never want service A to keep a span while service B drops its child, leaving a broken trace.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import (
    ParentBasedTraceIdRatioBased,
)

# Keep ~10% of root traces; honor upstream sampling decisions.
sampler = ParentBasedTraceIdRatioBased(rate=0.10)
provider = TracerProvider(sampler=sampler)

Correlating Logs with Traces

Traces and logs are most powerful together. Inject the current trace_id and span_id into every log line so you can jump from a log entry straight to the full trace.

You read the active span context from trace.get_current_span().get_span_context(). With the logging instrumentation enabled, OTel can also auto-inject these fields into the standard logging record.

import logging
from opentelemetry import trace

logger = logging.getLogger("orders")


def log_with_trace(message: str) -> None:
    ctx = trace.get_current_span().get_span_context()
    trace_id = format(ctx.trace_id, "032x")
    span_id = format(ctx.span_id, "016x")
    logger.info("%s", message, extra={
        "trace_id": trace_id,
        "span_id": span_id,
    })

Quick Check: Propagation Across Services

Service A (FastAPI) receives a request and calls Service B over HTTP. You want B's spans to appear under the same trace as A's. Which mechanism makes this work?

Recap

You can now instrument a FastAPI backend for distributed tracing end to end:

  • Bootstrap a TracerProvider with a Resource (set service.name), a BatchSpanProcessor, and an OTLP exporter.
  • Auto-instrument the app with FastAPIInstrumentor so every request is a server span that extracts incoming context.
  • Propagate through downstream HTTP (HTTPXClientInstrumentor) and the database (SQLAlchemyInstrumentor) — the W3C traceparent header keeps the trace unbroken.
  • Enrich with manual spans, attributes, events, status and recorded exceptions for your business logic.
  • Sample with ParentBasedTraceIdRatioBased for consistent, affordable traces, and correlate logs via the active trace_id/span_id.

The result: one click takes you from a slow request to the exact nested span — handler, HTTP call, or query — that caused it.

Frequently asked questions

Is the “Distributed Tracing with OpenTelemetry” lesson free?

Yes — the full text of “Distributed Tracing with OpenTelemetry” is free to read here on the web, and the FastAPI Backend Development Bootcamp course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the FastAPI Backend Development Bootcamp course, upgrade to CoddyKit PRO.

What will I learn in “Distributed Tracing with OpenTelemetry”?

Auto-instrument FastAPI and propagate trace context through downstream HTTP and database calls. You practise FastAPI Backend Development Bootcamp with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start FastAPI Backend Development Bootcamp?

No prior experience is required. FastAPI Backend Development Bootcamp on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Distributed Tracing with OpenTelemetry” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this FastAPI Backend Development Bootcamp lesson?

Yes. Every FastAPI Backend Development Bootcamp lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Structured JSON Logging and Correlation IDs
  2. Distributed Tracing with OpenTelemetry
  3. Prometheus Metrics and RED/USE Dashboards
  4. Alerting on SLOs and Error Budgets
← Back to FastAPI Backend Development Bootcamp