SLO와 오류 예산에 대한 알림

Traditional alerts fire on raw symptoms like CPU > 90% or error_count > 100. The problem: these page you for things users never notice, and stay silent during slow degradation that hurts customers.

SLO-based alerting inverts this. You first define what "good service" means to a user, then alert only when you are at risk of breaking that promise.

• SLI (Service Level Indicator): a measured ratio, e.g. fraction of fast, successful requests.
• SLO (Service Level Objective): the target for that SLI, e.g. 99.9% over 30 days.
• Error budget: the allowed failure, i.e. 100% minus the SLO.

In this lesson you will define SLOs for a FastAPI service and wire alerts that fire before users notice degradation.

A good SLI is a ratio of good events to valid events, scaled 0 to 100%. For a FastAPI backend the two workhorse SLIs are:

• Availability: successful responses / all valid responses. Treat 5xx as failures; usually exclude 4xx (client's fault).
• Latency: requests served under a threshold / all requests, e.g. responses faster than 300ms.

Below is a tiny, self-contained calculator that turns raw request logs into these two SLIs.

Once you commit to an SLO, the error budget is simply the failures you are allowed before breaching it.

• SLO 99.9% over 30 days → budget = 0.1% of all requests may fail.
• If you serve 10,000,000 requests/month, that is 10,000 failed requests you can spend.
• Expressed as time at full outage: 30 days × 0.1% ≈ 43 minutes of downtime per month.

The budget is a currency. Every failure spends it. Alerting is about watching the burn rate of that currency.

Burn rate is how fast you are consuming the error budget relative to "on pace." A burn rate of 1 means you will exactly exhaust the 30-day budget in 30 days. A burn rate of 14.4 means you would exhaust it in roughly 2 days.

The formula:

• burn_rate = observed_error_rate / (1 - SLO)
• If SLO = 99.9%, the budget error rate is 0.001. An observed error rate of 1.44% gives a burn rate of 0.0144 / 0.001 = 14.4.

High burn rate over a short window = something is on fire right now. Modest burn over a long window = slow leak. You alert on both.

A single threshold is a trap: alert too eagerly and you page on blips; too late and you miss real outages. The Google SRE-recommended pattern uses multiple burn rates over multiple windows:

• Fast burn: 14.4x over 1 hour (and a 5-min check) → page now, you will burn the whole month's budget in ~2 days.
• Slow burn: 6x over 6 hours → page, sustained leak.
• Trickle: 1x over 3 days → ticket, not a page.

Pairing a long window with a short "is it still happening" window kills false positives: the alert only fires if the burn is both severe and current.

Alerts are only as good as the data. Add Prometheus counters and a histogram via middleware so every request contributes to your availability and latency SLIs.

This is framework code (FastAPI + middleware), so it is not standalone-runnable, but it is the canonical instrumentation pattern.

With the counters in place, express the SLIs as PromQL. A recording rule precomputes the error ratio so alert rules stay cheap and readable.

The good-events ratio over the alert window is what you compare against the burn-rate threshold. Note how the threshold 0.0144 equals 14.4 × (1 - 0.999) — burn rate folded into the comparison.

Now the alert. It fires only when both the 1h and 5m error ratios exceed the 14.4x burn threshold for a 99.9% SLO. The short window guarantees the problem is still happening before paging an on-call engineer.

• 0.0144 = 14.4 × (1 - 0.999)
• for: 2m debounces transient spikes.
• Labels route severity; annotations carry the runbook link.

Before trusting your rules in production, simulate them. This self-contained evaluator replays a window of requests, computes both short and long error ratios, and decides whether a page should fire — exactly mirroring the multi-window rule.

Averages hide tail pain. If 99% of requests take 80ms and 1% take 4s, the mean looks fine while real users rage. Define latency SLOs on percentiles (p95, p99) or as the fraction under a threshold.

With a Prometheus histogram, p99 over 5m is:

• histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Below, a pure-Python p-quantile helper shows the math your histogram approximates.

An alert that no one can act on is noise. Make every page actionable:

• Symptom-based: alert on the SLO breach (users hurting), not on a single cause.
• Runbook attached: each alert links steps to diagnose and mitigate.
• Routed by severity: page wakes someone; ticket waits for business hours.

Tie it to an error-budget policy: when the budget is healthy, ship features fast. When it is exhausted, freeze risky deploys and prioritize reliability. The budget turns reliability from opinion into a shared, data-driven decision.

You learned to alert on user-facing reliability instead of raw symptoms:

• SLI = good events / valid events; SLO = the target; error budget = 100% minus the SLO.
• Burn rate = observed error rate / budget rate; it tells you how fast the budget is being spent.
• Multi-window, multi-burn-rate alerts (e.g. 14.4x over 1h+5m to page, slower burns to ticket) catch real degradation early without paging on blips.
• Instrument FastAPI with Prometheus counters and histograms, express SLIs as recording rules, and define alerts on the error ratio — using percentiles, never averages, for latency.
• Every alert must be actionable: symptom-based, runbook-linked, severity-routed, and backed by an error-budget policy that governs when to ship versus freeze.

Done well, your pages fire before users notice degradation — and stay quiet when nothing is wrong.