FastAPI Backend Development Bootcamp · 课时

Prometheus 指标与 RED/USE 仪表板

公开延迟、错误和饱和度指标，并使用 Grafana 仪表板展示服务健康状况。

Prometheus 指标与 RED/USE 仪表板 是 CoddyKit 上的免费 FastAPI Backend Development Bootcamp 课时。这是第 3 节课，共 4 节。你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。这是 FastAPI Backend Development Bootcamp 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 FastAPI Backend Development Bootcamp 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Metrics Matter

Logs tell you what happened in one request; metrics tell you how the whole service behaves over time. A metric is a numeric time series sampled at a fixed interval, which makes it cheap to store and fast to aggregate across millions of requests.

For a FastAPI backend you mainly care about three questions:

Is it serving traffic? request rate
Is it failing? error rate
Is it slow? request latency

Prometheus is a pull-based time-series database: it periodically scrapes an HTTP /metrics endpoint your app exposes, stores the samples, and lets you query them with PromQL. Grafana then turns those queries into dashboards.

The Four Metric Types

Prometheus has four core metric types. Picking the right one is the most important modeling decision.

Counter — only goes up (or resets to 0 on restart). Use for totals: requests served, errors, bytes sent. You query its rate, not its raw value.
Gauge — goes up and down. Use for current state: in-flight requests, memory usage, queue depth, connection pool size.
Histogram — buckets observations (e.g. latency) into pre-defined ranges, plus a _sum and _count. Lets you compute quantiles server-side.
Summary — like a histogram but computes quantiles client-side; cannot be aggregated across instances. Prefer histograms for latency in distributed services.

from prometheus_client import Counter, Gauge, Histogram

REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["method", "path", "status"])
IN_FLIGHT = Gauge("http_requests_in_flight", "Requests currently being served")
LATENCY = Histogram("http_request_duration_seconds", "Request latency in seconds", ["method", "path"])

REQUESTS.labels("GET", "/users", "200").inc()
IN_FLIGHT.inc()
LATENCY.labels("GET", "/users").observe(0.042)
IN_FLIGHT.dec()

print("counter, gauge and histogram updated")

Exposing /metrics in FastAPI

To let Prometheus scrape your app, mount an endpoint that renders all registered metrics in the Prometheus text exposition format. The prometheus_client library gives you generate_latest() and the correct content type.

You can wire this by hand, or use prometheus-fastapi-instrumentator which auto-instruments request count and latency. Doing it by hand first makes the mechanics clear.

Prometheus is then configured to hit http://your-app:8000/metrics on a scrape interval (commonly 15s).

from fastapi import FastAPI, Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

Labels and Cardinality

Labels turn one metric into many time series. http_requests_total{method="GET", path="/users", status="200"} is a distinct series from the same metric with status="500".

Cardinality is the number of unique label combinations. It is the single biggest way to blow up Prometheus memory.

Good labels: bounded sets — HTTP method, status code, route template.
Dangerous labels: unbounded values — user IDs, request IDs, raw URLs with path params, timestamps.

Always label by the route template (/users/{id}) not the resolved path (/users/4827), or every user creates new series.

A Middleware to Capture RED Signals

The cleanest way to instrument every endpoint is one ASGI middleware that records request count and latency, labeled by method, the route template, and status code.

Note how we read request.scope["route"].path (or the matched path template) instead of the raw URL to keep cardinality bounded. The same three labels feed both the Rate, Errors, and Duration views.

import time
from fastapi import FastAPI, Request
from prometheus_client import Counter, Histogram

REQUESTS = Counter("http_requests_total", "Total requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Latency", ["method", "path"])

app = FastAPI()

@app.middleware("http")
async def record_metrics(request: Request, call_next):
    route = request.scope.get("route")
    path = getattr(route, "path", request.url.path)
    start = time.perf_counter()
    response = await call_next(request)
    LATENCY.labels(request.method, path).observe(time.perf_counter() - start)
    REQUESTS.labels(request.method, path, str(response.status_code)).inc()
    return response

The RED Method

The RED method (popularized by Tom Wilkie) is the standard way to monitor request-driven services like a FastAPI API. For every service track:

Rate — requests per second
Errors — failed requests per second (or as a fraction)
Duration — distribution of request latency (p50/p95/p99)

RED is request-centric: it describes the experience of your callers. Three dashboards rows per service — rate, error ratio, latency percentiles — give you a consistent, comparable view across every microservice.

PromQL for Rate and Errors

Counters are queried with rate(), which computes the per-second average increase over a time window. The window (e.g. [5m]) should be at least 4x your scrape interval.

Rate — total requests per second across all routes:

sum(rate(http_requests_total[5m]))

Error ratio — fraction of requests returning 5xx:

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

The =~ operator is a regex match, so "5.." captures 500, 502, 503, etc. Use by (path) to break a result down per route.

PromQL for Latency Percentiles

Because we used a Histogram, Prometheus stores cumulative bucket counters named http_request_duration_seconds_bucket with a le ("less than or equal") label. histogram_quantile() estimates a percentile from those buckets.

p95 latency across the service:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

You must wrap the buckets in rate(...) first and keep the le label in the by() clause, otherwise the quantile is wrong. Percentiles (p95/p99) beat averages because a single slow tail is invisible in a mean.

# Default Histogram buckets are tuned for seconds; override for fast APIs:
from prometheus_client import Histogram

LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency in seconds",
    ["method", "path"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)

The USE Method

RED watches the request flow; the USE method (Brendan Gregg) watches the resources that serve those requests. For every resource (CPU, memory, disk, connection pool, worker threads) track:

Utilization — percent of time the resource was busy
Saturation — how much work is queued waiting for it (e.g. requests waiting for a DB connection)
Errors — error events for that resource

USE catches problems RED misses: if your DB connection pool is saturated, latency rises before error rate does. A Gauge for db_pool_in_use vs pool size is a classic saturation signal for a FastAPI app.

from prometheus_client import Gauge

DB_POOL_SIZE = Gauge("db_pool_size", "Configured max DB connections")
DB_POOL_IN_USE = Gauge("db_pool_in_use", "DB connections currently checked out")
DB_POOL_WAITERS = Gauge("db_pool_waiters", "Requests waiting for a connection")

def snapshot_pool(pool):
    DB_POOL_SIZE.set(pool.size())
    DB_POOL_IN_USE.set(pool.checkedout())
    DB_POOL_WAITERS.set(pool.overflow() if pool.overflow() > 0 else 0)

Building the Grafana Dashboard

In Grafana you add Prometheus as a data source, then build one dashboard per service with panels backed by the PromQL above:

Rate panel (time series): sum(rate(http_requests_total[5m])) by (path)
Error ratio panel (stat / gauge): the 5xx ratio expression, formatted as a percent
Latency panel (time series): p50, p95 and p99 lines from histogram_quantile
Saturation panel: db_pool_in_use / db_pool_size

Use template variables (e.g. a $path dropdown from label_values(http_requests_total, path)) so one dashboard works for every route. Set thresholds (green/amber/red) on the error and latency panels so health is readable at a glance.

Alerting on SLOs

Dashboards are for humans looking; alerts are for being told. Define alert rules in Prometheus (or Grafana) directly on your RED metrics, ideally tied to a Service Level Objective.

A common pattern is a multi-window burn-rate alert: fire when the error ratio over both a short and a long window exceeds your error budget burn rate, which avoids both flapping and slow detection.

Keep alert labels meaningful (severity, service) so Alertmanager can route pages vs. tickets correctly. Alert on symptoms users feel (high error ratio, high p99 latency), not every internal cause.

groups:
  - name: api-slo
    rules:
      - alert: HighErrorRatio
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "5xx error ratio above 2% for 10m"

Quick Check

You are labeling a latency Histogram for a FastAPI endpoint /orders/{order_id}. Which labeling choice keeps cardinality bounded and the dashboards correct?

Recap

You now have an end-to-end observability path for a FastAPI service:

Instrument with prometheus_client — Counters for totals, Gauges for current state, Histograms for latency.
Expose a /metrics endpoint and scrape it with Prometheus on a fixed interval.
Control cardinality by labeling with route templates and bounded sets only.
RED (Rate, Errors, Duration) describes the request experience; USE (Utilization, Saturation, Errors) describes resource health.
Query with PromQL: rate() for counters, histogram_quantile() for percentiles.
Visualize in Grafana with templated dashboards and thresholds, and alert on user-facing symptoms tied to SLOs.

Together these give you a consistent, low-overhead view of whether your service is up, failing, or slow.

常见问题解答

「Prometheus 指标与 RED/USE 仪表板」课时是免费的吗？

是的 — 「Prometheus 指标与 RED/USE 仪表板」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 FastAPI Backend Development Bootcamp 课程的其余内容，请升级到 CoddyKit PRO。 FastAPI Backend Development Bootcamp 课程共包含 4 节课。

「Prometheus 指标与 RED/USE 仪表板」这节课中我会学到什么？

公开延迟、错误和饱和度指标，并使用 Grafana 仪表板展示服务健康状况。你通过在浏览器中直接运行的动手代码来练习 FastAPI Backend Development Bootcamp，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 FastAPI Backend Development Bootcamp 需要有经验吗？

无需任何先前经验。CoddyKit 上的 FastAPI Backend Development Bootcamp 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。这是第 3 节课，共 4 节。

「Prometheus 指标与 RED/USE 仪表板」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 FastAPI Backend Development Bootcamp 课中编写并运行代码吗？

能。每节 FastAPI Backend Development Bootcamp 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

← 返回 FastAPI Backend Development Bootcamp