--- title: "Building an Incident Response Playbook | FourSight" description: "Create a step-by-step runbook your team can follow under pressure — from detection to post-mortem." lang: en json-ld: | [ { "@context": "https://schema.org", "@type": "Article", "headline": "Building an Incident Response Playbook", "description": "Create a step-by-step runbook your team can follow under pressure — from detection to post-mortem.", "author": { "@type": "Organization", "name": "FourSight" }, "publisher": { "@type": "Organization", "name": "FourSight" }, "url": "https://foursight.cloud/guides/incident-response-playbook", "mainEntityOfPage": "https://foursight.cloud/guides/incident-response-playbook", "datePublished": "2025-07-23", "dateModified": "2025-12-09", "wordCount": 2800 }, { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://foursight.cloud" }, { "@type": "ListItem", "position": 2, "name": "Guides", "item": "https://foursight.cloud/guides" }, { "@type": "ListItem", "position": 3, "name": "Building an Incident Response Playbook", "item": "https://foursight.cloud/guides/incident-response-playbook" } ] }, { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "How small a team needs an incident response playbook?", "acceptedAnswer": { "@type": "Answer", "text": "Even a solo founder needs one. The playbook for a team of one is shorter (no role separation needed) but the structure is the same. The discipline of writing it down forces you to think through the response calmly, before you're stressed at 3 AM." } }, { "@type": "Question", "name": "What's the difference between MTTD and MTTR?", "acceptedAnswer": { "@type": "Answer", "text": "MTTD (Mean Time To Detect) measures from when an incident starts until you're aware of it. MTTR (Mean Time To Resolve) measures from awareness to full resolution. Reducing MTTD has the biggest leverage — going from 30 minutes to 2 minutes effectively cuts overall downtime by 80%." } }, { "@type": "Question", "name": "Should I have separate playbooks for different incident types?", "acceptedAnswer": { "@type": "Answer", "text": "Start with one general playbook. Add specialized runbooks (database failover, deploy rollback, CDN switchover, DNS change) as appendices for specific high-frequency scenarios." } }, { "@type": "Question", "name": "How do I run a tabletop incident exercise?", "acceptedAnswer": { "@type": "Answer", "text": "Pick a realistic failure scenario. Gather your on-call rotation for an hour. Walk through the playbook in real time, with someone playing customer support and someone playing the on-call engineer. Stop and note every gap. Fix all gaps before the next exercise." } }, { "@type": "Question", "name": "What goes in a post-mortem that doesn't go in a status page update?", "acceptedAnswer": { "@type": "Answer", "text": "Status page updates are customer-facing and focus on impact and resolution. Post-mortems include the technical root cause, the timeline with internal actions, the contributing factors, and concrete action items with owners." } }, { "@type": "Question", "name": "How do I handle vendor-caused incidents (Stripe, AWS, etc.)?", "acceptedAnswer": { "@type": "Answer", "text": "Same playbook. The mitigation phase may be 'wait for vendor to recover' but the communication, customer support, and post-mortem phases are still yours to execute. Always have a fallback for any single-vendor dependency where feasible." } }, { "@type": "Question", "name": "Should I publish post-mortems publicly?", "acceptedAnswer": { "@type": "Answer", "text": "Yes for critical incidents that had customer impact. Public post-mortems within 5 business days dramatically improve customer retention after outages. Make them honest, technical, and specific about what's changing." } }, { "@type": "Question", "name": "Who should be the incident commander?", "acceptedAnswer": { "@type": "Answer", "text": "Whoever can coordinate without writing code. Often the on-call engineer for small teams, often a dedicated SRE or manager for larger ones. The IC's job is decisions and coordination — if the IC is also debugging, both jobs suffer." } } ] } ] --- [FourSight ](/) [Features](/#features)[Pricing](/pricing)[Guides](/guides)[Glossary](/glossary)[FAQ](/faq) [Login](/auth)[Start free](/auth?signup=true) [Start free](/auth?signup=true) [All Guides](/guides) Reliability & Infrastructure # Building an Incident Response Playbook Create a step-by-step runbook your team can follow under pressure — from detection to post-mortem. 14 min read Guide Published Jul 23, 2025Updated Dec 9, 2025 ## Why You Need a Playbook Before an Incident During an incident, adrenaline is high and clear thinking is hard. A pre-written playbook removes decision-making from the crisis and lets your team execute a proven process under pressure. The time to design your incident response is not at 2 AM during a production outage with customers tweeting at you. Teams with documented playbooks consistently resolve incidents 40-60% faster — not because the playbook is magic, but because it eliminates the wasted minutes spent asking 'wait, who calls customer success?' and 'do we update the status page now or after we know more?' ## The Four Phases of Incident Response Every incident follows a predictable lifecycle. Structure your playbook around these four phases and rehearse them in tabletop exercises before you need them for real. ### 1\. Detection How do you learn about incidents? FourSight monitoring, customer support tickets, social media, internal alerts? Define every detection channel and ensure they all route to the same on-call system. The fastest detection wins — a 2-minute [MTTD](/glossary/mttd "Glossary: MTTD") vs 35-minute MTTD makes a 10x difference in total customer impact. ### 2\. Triage Within 5 minutes of detection, determine [severity](/glossary/severity "Glossary: severity") (critical / major / minor), [blast radius](/glossary/blast-radius "Glossary: blast radius"), and initial response team. Use a clear severity rubric so the on-call engineer doesn't have to invent one mid-crisis. ### 3\. Mitigation Focus on stopping the bleeding, not finding the root cause. Roll back the last deployment, scale up infrastructure, switch to a backup provider, drain traffic from the affected region. Speed of mitigation matters far more than elegance. ### 4\. Resolution & Post-Mortem After the incident is fully resolved, update your status page, send a recap email, and schedule a [blameless post-mortem](/glossary/blameless-post-mortem "Glossary: blameless post-mortem") within 48 hours. Post-mortems should be done within a week — past that, memories fade and learning value collapses. ## Severity Classification Standardize severity definitions across your team so triage isn't subjective. A clear rubric reduces decision time during the highest-stress phase and ensures consistent customer communication. Severity Definition Response Time Notification SEV-1 (Critical) Full outage or data loss for >25% of customers Immediate page All-hands, status banner, exec notify SEV-2 (Major) Degraded service or full outage <25% customers 15 min ack On-call + manager, status component SEV-3 (Minor) Single-feature degradation, regional issue 1 hour ack On-call + Slack channel SEV-4 (Low) Cosmetic, single-user, non-blocking Next business day Ticket queue ## Roles During an Incident Define roles before the incident, not during. Three minimum roles are needed for any SEV-1 or SEV-2. ### Incident Commander (IC) Owns the incident end-to-end. Decides on severity, mitigation strategy, and resolution. Coordinates across teams. The IC does not write code — they coordinate the people who do. For small teams, this is often the on-call engineer. ### Technical Lead The engineer actively debugging and fixing the issue. Reports findings to the IC. May be the on-call engineer themselves if no separate IC is available. ### Communications Lead Owns customer-facing communication: status page updates, in-product banners, support team briefing, social media. Usually a customer-success or product manager. Frees engineers to focus on the technical fix. ## Building Your Communication Plan Incident communication is just as important as the technical response. Define who communicates, where, and what they say at each severity level. Write your status-page update templates before you need them — during an incident, you want to fill in blanks, not compose prose under pressure. **💡** The first status page update should happen within 5 minutes of confirming a real customer impact. 'We are investigating reports of elevated errors' is enough — don't wait until you have the answer. Customers panic in silence. ``` Status page update templates: [INVESTIGATING] T+0 to T+15min We are investigating reports of [symptom]. Engineers are looking into it. Next update in [15-30 min]. [IDENTIFIED] T+15 to T+45min We have identified the cause of [symptom]. [1-sentence explanation]. Working on a fix, ETA [estimate]. [MONITORING] T+45 to T+60min A fix has been deployed. We are monitoring to confirm full resolution. [RESOLVED] T+60min+ [Symptom] has been fully resolved. Total customer impact: [duration]. A detailed post-mortem will be published at [URL] within 5 business days. ``` ### Monitoring a Commercial SaaS? FourSight's free plan includes 10 commercial-safe monitors with multi-region validation — free forever, no card. [Start Monitoring Free](/auth?signup=true) ## On-Call Rotation Best Practices Sustainable on-call requires fair rotation, adequate compensation, and clear escalation paths. Rotations should be no longer than one week and shared among at least 4-5 engineers to avoid burnout. Use FourSight's notification rules to route alerts to the current on-call engineer automatically. Set up secondary escalation if the primary doesn't acknowledge within 10 minutes. Related Reading - [→ Alerting Without Alert Fatigue](/guides/alerting-without-alert-fatigue) ## The First 15 Minutes Checklist The first 15 minutes of an incident set the tone for everything that follows. Print this checklist and tape it next to your monitor — literally. ``` T+0 (alert received): [ ] Ack the alert within 5 minutes [ ] Open incident channel in Slack (#inc-YYYYMMDD-HHmm) [ ] Assign IC, Tech Lead, Comms Lead T+5 (assess): [ ] Confirm real impact (not a monitoring false positive) [ ] Determine severity using rubric [ ] Update status page if SEV-1 or SEV-2 T+10 (mitigate): [ ] Decide: rollback, scale, failover, or hotfix [ ] Execute mitigation [ ] Notify affected customer-facing teams T+15 (communicate): [ ] Send status page update with ETA [ ] Brief support team [ ] If SEV-1: notify exec team ``` ## Post-Mortem Template Every incident deserves a [post-mortem](/glossary/post-mortem "Glossary: post-mortem"). Use a consistent template so they're easy to compare over time. ### Summary (1 paragraph) What happened, when, who was affected, and how it was resolved. This is the only part most readers will read — make it count. ### Timeline Detection time, triage time, mitigation time, resolution time. Include both human actions and system events. ### Root Cause The technical cause, the process cause, and the contributing factors. Use the [Five Whys](/glossary/five-whys "Glossary: Five Whys") technique — keep asking 'why' until you reach an actual systemic cause. ### Impact Customers affected, duration of impact, financial impact estimate, [SLA](/glossary/sla "Glossary: SLA") implications. Be specific with numbers. ### Action Items Concrete changes with owners and due dates. 'Improve monitoring' is not an action item. 'Add a [P95](/glossary/percentile-latency "Glossary: P95") latency alert to /api/checkout by Friday, owned by Alex' is. ### Lessons Learned What worked well? What didn't? What would we do differently? This section is gold for new engineers reading old post-mortems. Related Reading - [→ Status Page Best Practices for SaaS](/guides/status-page-best-practices) - [→ Uptime SLA Reporting for SaaS Companies](/guides/uptime-sla-reporting-for-saas) ## Blameless Culture Post-mortems must be blameless. The moment people fear being singled out, they hide information — and you lose the entire learning value. Frame everything in terms of 'the system allowed X to happen', not 'Person Y did X'. The engineer who pushed the bad deploy is the same engineer who will spot the next one if they trust the post-mortem process. ## Practice Before You Need It Run tabletop incident exercises quarterly. Pick a realistic failure scenario, assemble the [on-call rotation](/glossary/on-call-rotation "Glossary: on-call rotation"), and walk through the playbook in real time. You'll discover gaps in the playbook, missing access permissions, and outdated runbooks — all much better discovered in a tabletop than in production. Related Reading - [→ The Real Cost of Downtime for Startups](/guides/cost-of-downtime-for-startups) - [→ Monitoring for Solo Founders](/guides/monitoring-for-solo-founders) ## Frequently Asked Questions ### How small a team needs an incident response playbook? ### What's the difference between MTTD and MTTR? ### Should I have separate playbooks for different incident types? ### How do I run a tabletop incident exercise? ### What goes in a post-mortem that doesn't go in a status page update? ### How do I handle vendor-caused incidents (Stripe, AWS, etc.)? ### Should I publish post-mortems publicly? ### Who should be the incident commander? #### Related Guides [What Is Uptime Monitoring? The Complete Guide 12 min ](/guides/what-is-uptime-monitoring)[Multi-Region Monitoring Explained 8 min ](/guides/multi-region-monitoring-explained)[SSL Certificate Expiry Monitoring 10 min ](/guides/ssl-certificate-expiry-monitoring)[How SSL Certificates Expire Silently: The Failure Modes 8 min ](/guides/how-ssl-certificates-expire-silently) #### Compare FourSight [vs UptimeRobot →](/compare/uptimerobot-alternative)[vs StatusCake →](/compare/statuscake-alternative)[vs Pingdom →](/compare/pingdom-alternative) 10 free commercial-safe monitors [View Pricing](/pricing) ## Protect Your SaaS Revenue Start monitoring in under 60 seconds. [Start Monitoring Free](/auth?signup=true)[View Pricing](/pricing) FourSight © 2026 [TetraCore](https://tetracorehq.com/). All rights reserved. FourSight is a TetraCore product — Bowling Green, Ohio. Product [Pricing](/pricing)[Guides](/guides)[Glossary](/glossary)[FAQ](/faq)[About](/about)[For Agencies](/solutions/agencies)[For Startups](/solutions/startups)[Privacy](/privacy)[Terms](/terms) Features [Cron Job & Heartbeat Monitoring](/features/cron-job-monitoring)[SSL Certificate Monitoring](/features/ssl-monitoring)[Status Pages](/features/status-pages)[Domain Expiry Monitoring](/features/domain-expiry-monitoring)[DNS Monitoring](/features/dns-monitoring)[Port Monitoring](/features/port-monitoring) Compare [All comparisons](/compare)[vs UptimeRobot](/compare/uptimerobot-alternative)[vs StatusCake](/compare/statuscake-alternative)[vs Freshping](/compare/freshping-alternative)[vs Pingdom](/compare/pingdom-alternative)[vs Pulsetic](/compare/pulsetic-alternative)[vs Better Stack](/compare/better-stack-alternative)[vs Uptime Kuma](/compare/uptime-kuma-alternative)[vs Cronitor](/compare/cronitor-alternative)[vs Healthchecks.io](/compare/healthchecks-alternative)[vs Hyperping](/compare/hyperping-alternative) Pricing Guides [UptimeRobot Pricing](/compare/uptimerobot-pricing)[StatusCake Pricing](/compare/statuscake-pricing)[Pingdom Pricing](/compare/pingdom-pricing)[Better Stack Pricing](/compare/better-stack-pricing)[Uptime Kuma Pricing](/compare/uptime-kuma-pricing)[Cronitor Pricing](/compare/cronitor-pricing)[Healthchecks.io Pricing](/compare/healthchecks-pricing)[Hyperping Pricing](/compare/hyperping-pricing)[Pulsetic Pricing](/compare/pulsetic-pricing)