Why You Need a Playbook Before an Incident
During an incident, adrenaline is high and clear thinking is hard. A pre-written playbook removes decision-making from the crisis and lets your team execute a proven process under pressure. The time to design your incident response is not at 2 AM during a production outage with customers tweeting at you. Teams with documented playbooks consistently resolve incidents 40-60% faster — not because the playbook is magic, but because it eliminates the wasted minutes spent asking 'wait, who calls customer success?' and 'do we update the status page now or after we know more?'
The Four Phases of Incident Response
Every incident follows a predictable lifecycle. Structure your playbook around these four phases and rehearse them in tabletop exercises before you need them for real.
1. Detection
How do you learn about incidents? FourSight monitoring, customer support tickets, social media, internal alerts? Define every detection channel and ensure they all route to the same on-call system. The fastest detection wins — a 2-minute MTTD vs 35-minute MTTD makes a 10x difference in total customer impact.
2. Triage
Within 5 minutes of detection, determine severity (critical / major / minor), blast radius, and initial response team. Use a clear severity rubric so the on-call engineer doesn't have to invent one mid-crisis.
3. Mitigation
Focus on stopping the bleeding, not finding the root cause. Roll back the last deployment, scale up infrastructure, switch to a backup provider, drain traffic from the affected region. Speed of mitigation matters far more than elegance.
4. Resolution & Post-Mortem
After the incident is fully resolved, update your status page, send a recap email, and schedule a blameless post-mortem within 48 hours. Post-mortems should be done within a week — past that, memories fade and learning value collapses.
Severity Classification
Standardize severity definitions across your team so triage isn't subjective. A clear rubric reduces decision time during the highest-stress phase and ensures consistent customer communication.
| Severity | Definition | Response Time | Notification |
|---|---|---|---|
| SEV-1 (Critical) | Full outage or data loss for >25% of customers | Immediate page | All-hands, status banner, exec notify |
| SEV-2 (Major) | Degraded service or full outage <25% customers | 15 min ack | On-call + manager, status component |
| SEV-3 (Minor) | Single-feature degradation, regional issue | 1 hour ack | On-call + Slack channel |
| SEV-4 (Low) | Cosmetic, single-user, non-blocking | Next business day | Ticket queue |
Roles During an Incident
Define roles before the incident, not during. Three minimum roles are needed for any SEV-1 or SEV-2.
Incident Commander (IC)
Owns the incident end-to-end. Decides on severity, mitigation strategy, and resolution. Coordinates across teams. The IC does not write code — they coordinate the people who do. For small teams, this is often the on-call engineer.
Technical Lead
The engineer actively debugging and fixing the issue. Reports findings to the IC. May be the on-call engineer themselves if no separate IC is available.
Communications Lead
Owns customer-facing communication: status page updates, in-product banners, support team briefing, social media. Usually a customer-success or product manager. Frees engineers to focus on the technical fix.
Building Your Communication Plan
Incident communication is just as important as the technical response. Define who communicates, where, and what they say at each severity level. Write your status-page update templates before you need them — during an incident, you want to fill in blanks, not compose prose under pressure.
Status page update templates:
[INVESTIGATING] T+0 to T+15min
We are investigating reports of [symptom]. Engineers are
looking into it. Next update in [15-30 min].
[IDENTIFIED] T+15 to T+45min
We have identified the cause of [symptom]. [1-sentence
explanation]. Working on a fix, ETA [estimate].
[MONITORING] T+45 to T+60min
A fix has been deployed. We are monitoring to confirm
full resolution.
[RESOLVED] T+60min+
[Symptom] has been fully resolved. Total customer impact:
[duration]. A detailed post-mortem will be published at
[URL] within 5 business days.Monitoring a Commercial SaaS?
FourSight includes 25 commercial-safe monitors with multi-region validation.
Start Monitoring FreeOn-Call Rotation Best Practices
Sustainable on-call requires fair rotation, adequate compensation, and clear escalation paths. Rotations should be no longer than one week and shared among at least 4-5 engineers to avoid burnout. Use FourSight's notification rules to route alerts to the current on-call engineer automatically. Set up secondary escalation if the primary doesn't acknowledge within 10 minutes.
Related Reading
The First 15 Minutes Checklist
The first 15 minutes of an incident set the tone for everything that follows. Print this checklist and tape it next to your monitor — literally.
T+0 (alert received):
[ ] Ack the alert within 5 minutes
[ ] Open incident channel in Slack (#inc-YYYYMMDD-HHmm)
[ ] Assign IC, Tech Lead, Comms Lead
T+5 (assess):
[ ] Confirm real impact (not a monitoring false positive)
[ ] Determine severity using rubric
[ ] Update status page if SEV-1 or SEV-2
T+10 (mitigate):
[ ] Decide: rollback, scale, failover, or hotfix
[ ] Execute mitigation
[ ] Notify affected customer-facing teams
T+15 (communicate):
[ ] Send status page update with ETA
[ ] Brief support team
[ ] If SEV-1: notify exec teamPost-Mortem Template
Every incident deserves a post-mortem. Use a consistent template so they're easy to compare over time.
Summary (1 paragraph)
What happened, when, who was affected, and how it was resolved. This is the only part most readers will read — make it count.
Timeline
Detection time, triage time, mitigation time, resolution time. Include both human actions and system events.
Root Cause
The technical cause, the process cause, and the contributing factors. Use the Five Whys technique — keep asking 'why' until you reach an actual systemic cause.
Impact
Customers affected, duration of impact, financial impact estimate, SLA implications. Be specific with numbers.
Action Items
Concrete changes with owners and due dates. 'Improve monitoring' is not an action item. 'Add a P95 latency alert to /api/checkout by Friday, owned by Alex' is.
Lessons Learned
What worked well? What didn't? What would we do differently? This section is gold for new engineers reading old post-mortems.
Blameless Culture
Post-mortems must be blameless. The moment people fear being singled out, they hide information — and you lose the entire learning value. Frame everything in terms of 'the system allowed X to happen', not 'Person Y did X'. The engineer who pushed the bad deploy is the same engineer who will spot the next one if they trust the post-mortem process.
Practice Before You Need It
Run tabletop incident exercises quarterly. Pick a realistic failure scenario, assemble the on-call rotation, and walk through the playbook in real time. You'll discover gaps in the playbook, missing access permissions, and outdated runbooks — all much better discovered in a tabletop than in production.