FastAPI Backend Development Bootcamp · 강의

Prometheus 지표와 RED/USE 대시보드

지연 시간, 오류 및 포화도 지표를 노출하고 Grafana 대시보드로 서비스 상태를 시각화합니다.

Prometheus 지표와 RED/USE 대시보드은(는) CoddyKit의 무료 FastAPI Backend Development Bootcamp 강의입니다. 이것은 4개 중 3번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 FastAPI Backend Development Bootcamp 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Why Metrics Matter

Logs tell you what happened in one request; metrics tell you how the whole service behaves over time. A metric is a numeric time series sampled at a fixed interval, which makes it cheap to store and fast to aggregate across millions of requests.

For a FastAPI backend you mainly care about three questions:

Is it serving traffic? request rate
Is it failing? error rate
Is it slow? request latency

Prometheus is a pull-based time-series database: it periodically scrapes an HTTP /metrics endpoint your app exposes, stores the samples, and lets you query them with PromQL. Grafana then turns those queries into dashboards.

The Four Metric Types

Prometheus has four core metric types. Picking the right one is the most important modeling decision.

Counter — only goes up (or resets to 0 on restart). Use for totals: requests served, errors, bytes sent. You query its rate, not its raw value.
Gauge — goes up and down. Use for current state: in-flight requests, memory usage, queue depth, connection pool size.
Histogram — buckets observations (e.g. latency) into pre-defined ranges, plus a _sum and _count. Lets you compute quantiles server-side.
Summary — like a histogram but computes quantiles client-side; cannot be aggregated across instances. Prefer histograms for latency in distributed services.

from prometheus_client import Counter, Gauge, Histogram

REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["method", "path", "status"])
IN_FLIGHT = Gauge("http_requests_in_flight", "Requests currently being served")
LATENCY = Histogram("http_request_duration_seconds", "Request latency in seconds", ["method", "path"])

REQUESTS.labels("GET", "/users", "200").inc()
IN_FLIGHT.inc()
LATENCY.labels("GET", "/users").observe(0.042)
IN_FLIGHT.dec()

print("counter, gauge and histogram updated")

Exposing /metrics in FastAPI

To let Prometheus scrape your app, mount an endpoint that renders all registered metrics in the Prometheus text exposition format. The prometheus_client library gives you generate_latest() and the correct content type.

You can wire this by hand, or use prometheus-fastapi-instrumentator which auto-instruments request count and latency. Doing it by hand first makes the mechanics clear.

Prometheus is then configured to hit http://your-app:8000/metrics on a scrape interval (commonly 15s).

from fastapi import FastAPI, Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()

@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

Labels and Cardinality

Labels turn one metric into many time series. http_requests_total{method="GET", path="/users", status="200"} is a distinct series from the same metric with status="500".

Cardinality is the number of unique label combinations. It is the single biggest way to blow up Prometheus memory.

Good labels: bounded sets — HTTP method, status code, route template.
Dangerous labels: unbounded values — user IDs, request IDs, raw URLs with path params, timestamps.

Always label by the route template (/users/{id}) not the resolved path (/users/4827), or every user creates new series.

A Middleware to Capture RED Signals

The cleanest way to instrument every endpoint is one ASGI middleware that records request count and latency, labeled by method, the route template, and status code.

Note how we read request.scope["route"].path (or the matched path template) instead of the raw URL to keep cardinality bounded. The same three labels feed both the Rate, Errors, and Duration views.

import time
from fastapi import FastAPI, Request
from prometheus_client import Counter, Histogram

REQUESTS = Counter("http_requests_total", "Total requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Latency", ["method", "path"])

app = FastAPI()

@app.middleware("http")
async def record_metrics(request: Request, call_next):
    route = request.scope.get("route")
    path = getattr(route, "path", request.url.path)
    start = time.perf_counter()
    response = await call_next(request)
    LATENCY.labels(request.method, path).observe(time.perf_counter() - start)
    REQUESTS.labels(request.method, path, str(response.status_code)).inc()
    return response

The RED Method

The RED method (popularized by Tom Wilkie) is the standard way to monitor request-driven services like a FastAPI API. For every service track:

Rate — requests per second
Errors — failed requests per second (or as a fraction)
Duration — distribution of request latency (p50/p95/p99)

RED is request-centric: it describes the experience of your callers. Three dashboards rows per service — rate, error ratio, latency percentiles — give you a consistent, comparable view across every microservice.

PromQL for Rate and Errors

Counters are queried with rate(), which computes the per-second average increase over a time window. The window (e.g. [5m]) should be at least 4x your scrape interval.

Rate — total requests per second across all routes:

sum(rate(http_requests_total[5m]))

Error ratio — fraction of requests returning 5xx:

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

The =~ operator is a regex match, so "5.." captures 500, 502, 503, etc. Use by (path) to break a result down per route.

PromQL for Latency Percentiles

Because we used a Histogram, Prometheus stores cumulative bucket counters named http_request_duration_seconds_bucket with a le ("less than or equal") label. histogram_quantile() estimates a percentile from those buckets.

p95 latency across the service:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

You must wrap the buckets in rate(...) first and keep the le label in the by() clause, otherwise the quantile is wrong. Percentiles (p95/p99) beat averages because a single slow tail is invisible in a mean.

# Default Histogram buckets are tuned for seconds; override for fast APIs:
from prometheus_client import Histogram

LATENCY = Histogram(
    "http_request_duration_seconds",
    "Request latency in seconds",
    ["method", "path"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)

The USE Method

RED watches the request flow; the USE method (Brendan Gregg) watches the resources that serve those requests. For every resource (CPU, memory, disk, connection pool, worker threads) track:

Utilization — percent of time the resource was busy
Saturation — how much work is queued waiting for it (e.g. requests waiting for a DB connection)
Errors — error events for that resource

USE catches problems RED misses: if your DB connection pool is saturated, latency rises before error rate does. A Gauge for db_pool_in_use vs pool size is a classic saturation signal for a FastAPI app.

from prometheus_client import Gauge

DB_POOL_SIZE = Gauge("db_pool_size", "Configured max DB connections")
DB_POOL_IN_USE = Gauge("db_pool_in_use", "DB connections currently checked out")
DB_POOL_WAITERS = Gauge("db_pool_waiters", "Requests waiting for a connection")

def snapshot_pool(pool):
    DB_POOL_SIZE.set(pool.size())
    DB_POOL_IN_USE.set(pool.checkedout())
    DB_POOL_WAITERS.set(pool.overflow() if pool.overflow() > 0 else 0)

Building the Grafana Dashboard

In Grafana you add Prometheus as a data source, then build one dashboard per service with panels backed by the PromQL above:

Rate panel (time series): sum(rate(http_requests_total[5m])) by (path)
Error ratio panel (stat / gauge): the 5xx ratio expression, formatted as a percent
Latency panel (time series): p50, p95 and p99 lines from histogram_quantile
Saturation panel: db_pool_in_use / db_pool_size

Use template variables (e.g. a $path dropdown from label_values(http_requests_total, path)) so one dashboard works for every route. Set thresholds (green/amber/red) on the error and latency panels so health is readable at a glance.

Alerting on SLOs

Dashboards are for humans looking; alerts are for being told. Define alert rules in Prometheus (or Grafana) directly on your RED metrics, ideally tied to a Service Level Objective.

A common pattern is a multi-window burn-rate alert: fire when the error ratio over both a short and a long window exceeds your error budget burn rate, which avoids both flapping and slow detection.

Keep alert labels meaningful (severity, service) so Alertmanager can route pages vs. tickets correctly. Alert on symptoms users feel (high error ratio, high p99 latency), not every internal cause.

groups:
  - name: api-slo
    rules:
      - alert: HighErrorRatio
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "5xx error ratio above 2% for 10m"

Quick Check

You are labeling a latency Histogram for a FastAPI endpoint /orders/{order_id}. Which labeling choice keeps cardinality bounded and the dashboards correct?

Recap

You now have an end-to-end observability path for a FastAPI service:

Instrument with prometheus_client — Counters for totals, Gauges for current state, Histograms for latency.
Expose a /metrics endpoint and scrape it with Prometheus on a fixed interval.
Control cardinality by labeling with route templates and bounded sets only.
RED (Rate, Errors, Duration) describes the request experience; USE (Utilization, Saturation, Errors) describes resource health.
Query with PromQL: rate() for counters, histogram_quantile() for percentiles.
Visualize in Grafana with templated dashboards and thresholds, and alert on user-facing symptoms tied to SLOs.

Together these give you a consistent, low-overhead view of whether your service is up, failing, or slow.

자주 묻는 질문

“Prometheus 지표와 RED/USE 대시보드” 강의는 무료인가요?

네 — “Prometheus 지표와 RED/USE 대시보드” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 FastAPI Backend Development Bootcamp 강의 전체를 잠금 해제할 수 있습니다. FastAPI Backend Development Bootcamp 강의에는 총 4개의 강의가 포함되어 있습니다.

“Prometheus 지표와 RED/USE 대시보드”에서 뭘 배우나요?

지연 시간, 오류 및 포화도 지표를 노출하고 Grafana 대시보드로 서비스 상태를 시각화합니다. 브라우저에서 직접 실행하는 실습 코드로 FastAPI Backend Development Bootcamp을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

FastAPI Backend Development Bootcamp을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 FastAPI Backend Development Bootcamp은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 3번째 강의입니다.

“Prometheus 지표와 RED/USE 대시보드” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 FastAPI Backend Development Bootcamp 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 FastAPI Backend Development Bootcamp 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← FastAPI Backend Development Bootcamp(으)로 돌아가기