Building Runbooks for Common Events

A runbook is a step-by-step guide your team follows during a specific type of incident. Under stress, people forget steps and skip checks. A good runbook removes guesswork.

It turns expert knowledge into a repeatable procedure anyone on call can execute, making responses consistent, fast, and auditable.

The terms overlap, but a useful distinction is:

• A playbook is the high-level strategy — the decision flow and who is involved for a class of incidents.
• A runbook is the detailed, often technical, sequence of actions to carry it out.

Playbooks decide what to do; runbooks say exactly how to do it.

A solid runbook includes a clear trigger, prerequisites, ordered steps, expected outcomes, and rollback guidance.

It names the exact AWS actions to take — which API call, which console screen, which role to assume — and what success looks like at each step. Ambiguity is the enemy when minutes matter.

Build a runbook per common scenario rather than one giant document:

• Exposed access key in a public repository.
• Compromised EC2 instance.
• Publicly exposed S3 bucket.
• Unusual API calls from a new region.

Targeted runbooks are faster to follow than a sprawling catch-all.

A leaked-key runbook might read: confirm the GuardDuty finding, identify the key owner, disable the key, review CloudTrail for actions taken with it, rotate any affected secrets, and notify stakeholders.

Each step references the precise tool and includes a verification check before moving on.

Runbooks start as documents humans follow. As they mature, automate the safe, repeatable steps with AWS Systems Manager Automation documents or Lambda functions.

This is a spectrum: keep judgment calls human while letting machines handle the mechanical steps reliably and identically every time.

AWS Systems Manager (SSM) Automation runbooks are managed documents that perform actions across your resources. AWS ships dozens of pre-built ones, and you can author custom documents.

They can stop instances, change configurations, or chain steps with approvals — turning a written procedure into executable, audited automation.

A stale runbook is dangerous: it sends responders to delete a resource that no longer exists or assume a role that was renamed.

Store runbooks in version control, review them after every incident and architecture change, and validate them during game days so errors surface in practice, not during a real event.

A runbook is useless if the responder cannot perform its steps. Pre-define the IAM roles and permissions each runbook requires.

Use a dedicated break-glass role with elevated access that is normally unused, heavily logged, and alerted on. That way the step "assume the IR role" actually works the moment it is needed.

Technical actions are only half the job. Runbooks should include who to notify, how, and when — incident commander, legal, leadership, and possibly customers.

Bake the communication plan into the runbook so notifications are not forgotten in the heat of containment. Clear, timely updates build trust and reduce panic.

End each runbook with explicit verification steps and a sign-off. After containment, confirm the threat is truly gone, the resource is in a known-good state, and the incident can be closed.

A documented sign-off prevents a half-finished response from being assumed complete, and the verification notes feed directly into the post-incident review.

Pick the best characterization.

Runbooks turn expert response knowledge into repeatable, ordered steps anyone on call can follow. Build one per common event, include trigger, actions, verification, and communication, and reference exact AWS tools and roles. Pre-create the IAM access each runbook needs, mature mechanical steps into SSM Automation or Lambda, and keep everything version-controlled and rehearsed so it never goes stale.