Prometheus Metrics and RED/USE Dashboards

Logs tell you what happened in one request; metrics tell you how the whole service behaves over time. A metric is a numeric time series sampled at a fixed interval, which makes it cheap to store and fast to aggregate across millions of requests.

For a FastAPI backend you mainly care about three questions:

• Is it serving traffic? request rate
• Is it failing? error rate
• Is it slow? request latency

Prometheus is a pull-based time-series database: it periodically scrapes an HTTP /metrics endpoint your app exposes, stores the samples, and lets you query them with PromQL. Grafana then turns those queries into dashboards.

Prometheus has four core metric types. Picking the right one is the most important modeling decision.

• Counter — only goes up (or resets to 0 on restart). Use for totals: requests served, errors, bytes sent. You query its rate, not its raw value.
• Gauge — goes up and down. Use for current state: in-flight requests, memory usage, queue depth, connection pool size.
• Histogram — buckets observations (e.g. latency) into pre-defined ranges, plus a _sum and _count. Lets you compute quantiles server-side.
• Summary — like a histogram but computes quantiles client-side; cannot be aggregated across instances. Prefer histograms for latency in distributed services.

To let Prometheus scrape your app, mount an endpoint that renders all registered metrics in the Prometheus text exposition format. The prometheus_client library gives you generate_latest() and the correct content type.

You can wire this by hand, or use prometheus-fastapi-instrumentator which auto-instruments request count and latency. Doing it by hand first makes the mechanics clear.

Prometheus is then configured to hit http://your-app:8000/metrics on a scrape interval (commonly 15s).

Labels turn one metric into many time series. http_requests_total{method="GET", path="/users", status="200"} is a distinct series from the same metric with status="500".

Cardinality is the number of unique label combinations. It is the single biggest way to blow up Prometheus memory.

• Good labels: bounded sets — HTTP method, status code, route template.
• Dangerous labels: unbounded values — user IDs, request IDs, raw URLs with path params, timestamps.

Always label by the route template (/users/{id}) not the resolved path (/users/4827), or every user creates new series.

The cleanest way to instrument every endpoint is one ASGI middleware that records request count and latency, labeled by method, the route template, and status code.

Note how we read request.scope["route"].path (or the matched path template) instead of the raw URL to keep cardinality bounded. The same three labels feed both the Rate, Errors, and Duration views.

The RED method (popularized by Tom Wilkie) is the standard way to monitor request-driven services like a FastAPI API. For every service track:

• Rate — requests per second
• Errors — failed requests per second (or as a fraction)
• Duration — distribution of request latency (p50/p95/p99)

RED is request-centric: it describes the experience of your callers. Three dashboards rows per service — rate, error ratio, latency percentiles — give you a consistent, comparable view across every microservice.

Counters are queried with rate(), which computes the per-second average increase over a time window. The window (e.g. [5m]) should be at least 4x your scrape interval.

Rate — total requests per second across all routes:

• sum(rate(http_requests_total[5m]))

Error ratio — fraction of requests returning 5xx:

• sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

The =~ operator is a regex match, so "5.." captures 500, 502, 503, etc. Use by (path) to break a result down per route.

Because we used a Histogram, Prometheus stores cumulative bucket counters named http_request_duration_seconds_bucket with a le ("less than or equal") label. histogram_quantile() estimates a percentile from those buckets.

p95 latency across the service:

• histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

You must wrap the buckets in rate(...) first and keep the le label in the by() clause, otherwise the quantile is wrong. Percentiles (p95/p99) beat averages because a single slow tail is invisible in a mean.

RED watches the request flow; the USE method (Brendan Gregg) watches the resources that serve those requests. For every resource (CPU, memory, disk, connection pool, worker threads) track:

• Utilization — percent of time the resource was busy
• Saturation — how much work is queued waiting for it (e.g. requests waiting for a DB connection)
• Errors — error events for that resource

USE catches problems RED misses: if your DB connection pool is saturated, latency rises before error rate does. A Gauge for db_pool_in_use vs pool size is a classic saturation signal for a FastAPI app.

In Grafana you add Prometheus as a data source, then build one dashboard per service with panels backed by the PromQL above:

• Rate panel (time series): sum(rate(http_requests_total[5m])) by (path)
• Error ratio panel (stat / gauge): the 5xx ratio expression, formatted as a percent
• Latency panel (time series): p50, p95 and p99 lines from histogram_quantile
• Saturation panel: db_pool_in_use / db_pool_size

Use template variables (e.g. a $path dropdown from label_values(http_requests_total, path)) so one dashboard works for every route. Set thresholds (green/amber/red) on the error and latency panels so health is readable at a glance.

Dashboards are for humans looking; alerts are for being told. Define alert rules in Prometheus (or Grafana) directly on your RED metrics, ideally tied to a Service Level Objective.

A common pattern is a multi-window burn-rate alert: fire when the error ratio over both a short and a long window exceeds your error budget burn rate, which avoids both flapping and slow detection.

Keep alert labels meaningful (severity, service) so Alertmanager can route pages vs. tickets correctly. Alert on symptoms users feel (high error ratio, high p99 latency), not every internal cause.

You are labeling a latency Histogram for a FastAPI endpoint /orders/{order_id}. Which labeling choice keeps cardinality bounded and the dashboards correct?

You now have an end-to-end observability path for a FastAPI service:

• Instrument with prometheus_client — Counters for totals, Gauges for current state, Histograms for latency.
• Expose a /metrics endpoint and scrape it with Prometheus on a fixed interval.
• Control cardinality by labeling with route templates and bounded sets only.
• RED (Rate, Errors, Duration) describes the request experience; USE (Utilization, Saturation, Errors) describes resource health.
• Query with PromQL: rate() for counters, histogram_quantile() for percentiles.
• Visualize in Grafana with templated dashboards and thresholds, and alert on user-facing symptoms tied to SLOs.

Together these give you a consistent, low-overhead view of whether your service is up, failing, or slow.