지연 시간과 오류 급증 알림

A dashboard only helps when someone is looking. Alerts watch your metrics around the clock and reach out the moment something breaks. 🚨

The goal is to fire before users notice. Catch the slow creep in latency or errors while there is still time to react.

An alert rule is a PromQL expression plus a condition. When the expression stays true long enough, the alert starts firing.

Watch your p99 latency and fire when it crosses a threshold, say half a second, signaling the model is too slow.

Fire when the error ratio climbs above a small fraction, like five percent of requests failing over the last few minutes.

The for clause requires the condition to hold for a sustained period before firing, so a one-second blip never wakes anyone.

Attach labels like severity to each rule. They let you route a critical page differently from a low-priority warning.

Use annotations to write a clear summary and a runbook link, so whoever gets paged knows what broke and what to do.

Prometheus decides when to fire, but Alertmanager handles delivery, grouping, and silencing of those alerts.

Route alerts to a receiver people actually watch, like Slack or PagerDuty, instead of an inbox nobody reads.

Too many false alarms cause alert fatigue and people stop reading. Tune thresholds and for durations until every page is worth waking for.

You want an alert that ignores momentary blips and only fires on sustained problems. What achieves that?

You wrote PromQL alert rules for latency and error spikes, used a for duration to skip blips, and routed firing alerts through Alertmanager to people. ✅