Measuring Blast Radius and Steady-State Hypotheses
Define a measurable steady state, form falsifiable hypotheses, and bound the blast radius so chaos experiments are safe, scientific, and informative.
Measuring Blast Radius and Steady-State Hypotheses is a free Production Debugging & Incident Response Playbook lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Production Debugging & Incident Response Playbook learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Chaos as a Scientific Method
Chaos engineering is not random breakage; it is an experiment. Like any experiment, it needs a hypothesis, a controlled variable, and a measurable outcome.
This lesson focuses on the steady-state hypothesis and bounding the blast radius.
Defining Steady State
Steady state is your system's normal, healthy behavior expressed as measurable output, not internal metrics.
- Good: orders completed per minute, p99 latency
- Weak: CPU usage, memory
Steady state should reflect what users experience.
Forming a Hypothesis
A chaos hypothesis predicts that steady state holds despite a specific fault. It must be falsifiable.
Hypothesis: 'If one payment replica fails,
orders/min stays within 5% of baseline.'What Is Blast Radius
Blast radius is the maximum harm an experiment could cause: which users, services, and data could be affected if it goes wrong.
Controlling it is what separates a safe experiment from an outage you caused yourself.
Starting Small
Begin with the smallest meaningful scope and expand only after success.
- One instance before one zone
- 1% of traffic before 100%
- Staging before production
Setting an Abort Condition
Define in advance when to stop. If steady state degrades past a threshold, the experiment must halt automatically.
abort_if: error_rate > 2% OR orders_per_min < baseline * 0.9Measuring Before, During, After
Record the steady-state metric across three windows: a baseline before the fault, the period during it, and recovery after. Comparing these tells you whether the hypothesis held and how fast the system recovered.
Reading the Result
Two outcomes are both valuable:
- Hypothesis holds: confidence that the system tolerates this fault
- Hypothesis fails: you found a weakness safely, before customers did
A failed hypothesis is a successful experiment.
Quantifying Blast Radius
Estimate worst-case impact numerically before running: affected users, potential revenue at risk, recovery time. This makes the go/no-go decision explicit rather than gut feeling.
max_affected_users = traffic_pct * active_users
# 1% * 50000 = 500 users worst caseCommunicating the Experiment
Even a well-bounded experiment can surprise on-call. Announce the window, scope, and abort plan beforehand so a real incident is not confused with your test, and so help is ready if needed.
An Experiment Design Workflow
Putting it together:
- Define a user-facing steady-state metric
- State a falsifiable hypothesis
- Bound the blast radius and start small
- Set automatic abort conditions
- Measure before/during/after and learn from either result
Quick Check
Test your understanding of experiment design.
Recap
You learned to design safe, scientific chaos experiments.
- Define steady state from user-facing output
- Form falsifiable hypotheses
- Bound and quantify blast radius, start small
- Set abort conditions and learn from any result
Frequently asked questions
Is the “Measuring Blast Radius and Steady-State Hypotheses” lesson free?
Yes — the full text of “Measuring Blast Radius and Steady-State Hypotheses” is free to read here on the web, and the Production Debugging & Incident Response Playbook course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Production Debugging & Incident Response Playbook course, upgrade to CoddyKit PRO.
What will I learn in “Measuring Blast Radius and Steady-State Hypotheses”?
Define a measurable steady state, form falsifiable hypotheses, and bound the blast radius so chaos experiments are safe, scientific, and informative. You practise Production Debugging & Incident Response Playbook with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Production Debugging & Incident Response Playbook?
No prior experience is required. Production Debugging & Incident Response Playbook on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Measuring Blast Radius and Steady-State Hypotheses” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Production Debugging & Incident Response Playbook lesson?
Yes. Every Production Debugging & Incident Response Playbook lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Principles of Chaos Engineering
- Tools and Platforms for Chaos Experiments
- Building Resilience into System Design
- Measuring Blast Radius and Steady-State Hypotheses