Write a Runbook for When Something Breaks
Produces the document someone reads at 2am when a critical system or process fails and you're unreachable. Use it for anything whose failure costs money or customers.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
You are a reliability engineer. Write a runbook for this failure — the document someone reads while it is broken, not a reference manual.
What breaks: {{system_or_process}}
How we find out: {{alert_monitoring_or_customer_reports}}
What it costs per hour of downtime: {{impact}}
Who might be responding: {{their_access_and_experience}}
What has gone wrong before: {{past_incidents}}
Write for a stressed person at 2am who did not build this. That means: short lines, imperative verbs, no background, no theory, and the most likely fix first rather than the most thorough diagnosis.
Structure it as:
1. Is this actually the problem? Two or three checks that confirm the symptom matches this runbook, so nobody follows the wrong one.
2. Severity and first move. Define the levels in terms of customer impact, not internal drama. Say who to wake, at what level, and how. Include the rule for when to stop investigating and escalate — usually a clock, like "if not resolved in 20 minutes, page the second responder," because the failure mode of a lone responder is heads-down for two hours.
3. Stop the bleeding, before diagnosis. What restores service fastest even if it is inelegant: fail over, roll back, disable the feature, switch to the manual process, put up the status message. Diagnosis can happen after customers are served. Say explicitly which of these are safe to do without approval.
4. Diagnosis, as an ordered checklist. Most likely cause first, with the specific command, screen, dashboard, or log line for each, and what a healthy result looks like. Include what changed recently — deployments, config, vendor status, expiry dates — because it usually is that.
5. Fixes, each with: what it does, what it risks, whether it is reversible, and whether the responder can do it alone.
6. Communication. Who tells customers, who tells internal, what the first message says, and how often to update. Point to the customer-facing template rather than making someone write one under pressure.
7. Verify recovery. The specific checks that prove it is actually fixed, not just quiet.
8. Contacts: names, roles, how to reach them out of hours, and the vendor support line with the account number needed to open a priority ticket.
Finish with what the responder should write down while it is happening, so the review afterwards has a real timeline.