ทดสอบและดูแลคู่มือรับมือเหตุขัดข้อง

Systems change constantly, but playbooks are written once and forgotten. A stale playbook is worse than none: it sends responders down dead ends during a crisis.

This lesson covers keeping playbooks alive through testing and maintenance.

Store playbooks in version control alongside the services they cover. This gives you history, review, and the ability to update a playbook in the same pull request that changes the system.

Every command, dashboard URL, and threshold in a playbook is a potential lie waiting to happen. Track when each playbook was last verified, and surface stale ones.

A tabletop is a low-cost drill: the team walks through a hypothetical incident verbally, following the playbook step by step, and notes anything missing or wrong.

No production impact, high learning value.

A game day goes further: you inject a real (controlled) failure and have on-call respond using only the playbook. This reveals gaps that reading never would.

Schedule them regularly, not just after an outage.

For commands that are safe to run read-only, add a CI check that confirms they still resolve and that referenced dashboards still exist.

The best test is a real incident. After each one, ask: did the playbook help? Where did it mislead? Feed those answers straight back as edits.

Make 'update the playbook' a standard post-mortem action item.

Under stress, dense prose is unreadable. Maintain steps as short, numbered, imperative actions with expected results.

Assign each playbook a single owner and a review interval. Stale playbooks past their interval should appear in a dashboard or block nothing silently.

• Critical paths: review quarterly
• Others: semi-annually

Track whether playbooks actually shorten incidents.

• Time-to-mitigate with vs without an existing playbook
• Percentage of steps that worked as written during drills
• Number of stale playbooks past review date

Putting it together:

• Version playbooks with the code they cover
• Run tabletops and game days on a schedule
• Automate link and command validation in CI
• Update from every real incident
• Assign owners and review cadences

Test your understanding of playbook maintenance.

You learned to keep playbooks trustworthy.

• Version them as code with last-verified metadata
• Drill via tabletops and game days
• Automate validation and update after real incidents
• Own them and measure their effectiveness