Home › Topics › Measuring Detection & Response › MTTR Playbook

Shrinking Repair Time: The MTTR Playbook


The alert has fired and a human knows. From here the only question is the second half of the incident clock: once you know, how fast can you fix? That interval, from detection to verified recovery, is what MTTR (mean time to repair) averages. It is the half of the clock that monitoring cannot shrink for you. Detection time is mostly configuration. Repair time is mostly preparation: what was written down, staged, backed up, rehearsed, and agreed before the incident started. Nearly everything that makes a 3 a.m. repair slow was decided, or not decided, weeks earlier at a comfortable desk. I have been the person at the comfortable desk and the person at 3 a.m. And the second one has strong views about the first.


This is the playbook for that preparation. The move that unlocks it is refusing to treat repair time as one blob. An incident's response is four phases (engage, diagnose, fix, verify). And each phase has its own levers, its own failure smells, and its own price list. Find which phase eats your incidents, pull that phase's levers, re-measure. If the vocabulary is new, the incident clock guide defines the timestamps these phases run between. If your problem is the other clock, finding out too slowly, you want the MTTD playbook first.


In this guide

  • Where repair time goes: the four phases
  • Engage faster: from alert to hands on keyboard
  • Diagnose faster: runbooks, context, and evidence
  • Fix faster: spares, rollbacks, and permission
  • Verify faster: knowing what "fixed" looks like
  • The practice lever: drills and second responders
  • The measurement loop: one phase per quarter
  • The playbook card and readiness checklist

Where repair time goes: the four phases


Decompose before you invest. Between DETECTED and VERIFIED, every incident passes through four phases. First is engage (alert to a human actually working it), then diagnose (working it to knowing the cause). Then come fix (knowing the cause to the corrective action applied) and verify (action applied to proven recovery). Time them separately whenever you can — even roughly, even reconstructed from chat timestamps afterward — because the totals hide everything and the phases reveal everything. A 95-minute repair where diagnosis took 50 needs runbooks, not urgency. A 95-minute repair where engagement took 40 needs paging and access fixes, and better runbooks would have improved nothing at all.


Ask the phase question in every post-incident look-back: which phase ate this one? Patterns emerge embarrassingly fast, and they follow two classic distributions worth knowing in advance. Diagnosis dominates unfamiliar incidents. The first time anything fails a particular way, most of the response is figuring out what is happening. That is why teams facing novel problems live in the diagnose phase. And access and logistics dominate the 3 a.m. incidents. The mind was willing but the VPN (the remote-access tunnel into your network) had opinions, the password had expired, and the spare was in the other building. Daytime incidents flatter your readiness; the after-hours ones audit it.


Timing the phases does not require new tooling — three timestamps beyond what you already log will do. Record when work actually began, when the cause was named, and when the fix was applied. Chat history usually contains all three ("looking now", "found it — controller hung", "restarting it") whether or not anyone was thinking about metrics. Write them into the incident log's notes while filing the row, and after a handful of incidents the phase pattern is sitting in plain sight.


Engage faster: from alert to hands on keyboard


The engage phase should be minutes and has a talent for becoming an hour. Its first killer is the unacknowledged alert — delivered, seen by no one, escalated to no one. The cure is an escalation timer that actually fires. If a critical alert is not acknowledged within a set number of minutes, it automatically re-notifies louder or moves to the next person. The important word is "actually" — many teams configure escalation once and never test it. They discover during a real outage that the second contact left the company. Test the chain quarterly, deliberately: let a test alert go unacknowledged and watch where it goes. On-call rotation design is its own discipline; from the MTTR side, the requirement is that silence never ends the story.


The second killer is the not-actually-ready responder: awake, willing, and twenty minutes from useful. The VPN client wants an update, the admin password expired the day before, and the dashboard link is on the other laptop. The cure is the access-readiness drill. Monthly, on a calm day, each person who might respond goes through the whole path cold — connect, authenticate, open the consoles, reach the runbooks. Each person fixes whatever snagged. Ten minutes of drill converts to real minutes saved on every future incident, because the fumbling happens on a Tuesday afternoon instead of inside your MTTR. The full checklist at the end of this guide is built around exactly this drill.


Round out the phase with the physical on-call kit. Whoever is answerable this week carries a laptop that is charged, updated, and known to connect. They also carry the analog fallbacks that survive the failures that take your conveniences with them. A printed card with the emergency numbers (the ISP's fault line, the hardware vendor, the after-hours facility contact) matters. That is precisely because the incident that kills the network also kills the wiki that stores those numbers. The test of a good kit is simple: could you begin useful work from your kitchen table, with the office network down, inside five minutes?


Diagnose faster: runbooks, context, and evidence


For most small teams, diagnosis is the biggest single lever in this playbook. The largest part of it is the runbook: the diagnosis path thought through in advance, written for the panicking future self. For this alert, check these three things in this order; if A, do X; if B, do Y. A responder with a runbook starts from step one; a responder without one starts from a search engine. The difference on a familiar failure is routinely half an hour of not-rediscovering what the team already learned last time. Writing and maintaining them is a craft of its own. The MTTR-focused advice is brutally simple. Write runbooks for your top five recurring alerts first, because recurrence means the diagnosis is already known. It is just stored in someone's head, which may be asleep.


Second: make the alert arrive carrying context. An alert that says "web server down" starts a scavenger hunt. One that arrives with the related facts attached — what else is alerting, what is downstream of it, recent state changes — starts a diagnosis. Correlation and enrichment are what turn an alert storm into a diagnosis; the alert correlation guides cover the machinery.


Third, the cheapest trick in this guide: the evidence-bundle script — one command that gathers the usual first facts, so diagnosis begins with data instead of with typing. Every experienced responder runs roughly the same opening moves; script them once:


# evidence.ps1 — run first, read while thinking
# (adapt the target and questions to your own usual suspects)
Start-Transcript evidence-fs01.txt        # timestamped record, free
Test-Connection fs01 -Count 4             # reachable at all?
Get-Service -ComputerName fs01            # what stopped running?
Get-WinEvent -ComputerName fs01 -LogName System -MaxEvents 50
Get-CimInstance -ComputerName fs01 -ClassName Win32_LogicalDisk
Stop-Transcript                           # check free disk space first

Thirty seconds of runtime, and the responder is reading symptoms instead of gathering them — with a timestamped transcript that later writes the incident timeline for you. The disk check is in there because it is always the disk. It is not always the disk, but checking costs ten seconds and skipping it once cost me an hour. The last diagnosis lever is pattern memory. That means a signature library, which sounds grand and is a wiki page. It has one line per past incident: symptom, cause, fix, runbook link. Repeat incidents (and most incidents are repeats) collapse from diagnosis to recognition: "queue full plus that exact event ID — it's the stuck spooler thing again, runbook 7."


Fix faster: spares, rollbacks, and permission


Once diagnosed, the fix phase is where physics and logistics take over — and where preparation pays in hardware and paperwork. Start with spares. The economics at small scale are worth doing explicitly. A spare earns its shelf space only when three things line up. The device is a single point of failure that stops real work. Replacement lead time is measured in days, and the hardware is cheap relative to a day of that downtime. The edge switch, the firewall of a one-firewall shop, the branch office's only access point — these qualify easily. The crucial refinement is configured-and-shelved. That is a spare with the current config loaded, a label saying what it replaces and where it plugs in, and a sticky note with the last config date. The difference between that and a boxed spare with factory settings is an hour of console work at the worst possible moment. The difference between it and no spare is a vendor's shipping schedule.


At Bellweather Logistics the warehouse edge switch died at twenty to six on a dispatch morning. There was a spare on the shelf, still in its box, with factory settings and no label. The responder spent seventy minutes on a console cable rebuilding fourteen VLANs from a config backup that turned out to be eight months old. The loading bay ran on paper until half past eight. The spare now sits on the shelf with the current config loaded, a label naming the rack it belongs to, and a sticky note with the date. The console cable has been kept, as a reminder.


Rollback readiness is the software mirror of spares. Half of all fixes are "put it back the way it was" — which requires knowing the way it was. Keep config backups for every network device, refreshed on change, stored somewhere reachable when the network is sick. Keep them off the failed device, reachable without the thing being repaired — the same self-reliance thinking the monitoring-the-monitor guides apply to your tooling. And a tested restore path: a backup you have never restored from is a hope, not a rollback plan. Once a quarter, restore one config to a spare or a lab unit and confirm the procedure still works and someone besides its author can follow it.


The third fix-phase lever costs nothing but a conversation: permission readiness. Real repairs have died waiting for approval — the responder knew the fix at 3:10 and applied it at 8:40 when someone with the authority came online. The cure is the emergency-change authority, pre-agreed and written down. During a declared incident, the responder may restart services, fail over to spares, and roll back recent changes on their own authority, at any hour. In that situation, actions that risk data loss need a second person. Everything gets documented afterward. Deciding this in a calm meeting takes twenty minutes. Deciding it during an outage takes hours and burns trust in both directions. Where the line sits belongs with your incident response roles and rules — the MTTR point is only that it must sit somewhere before the incident.


Verify faster: knowing what "fixed" looks like


The shortest phase, with the most expensive failure mode. Verification drags when nobody defined what "working" means — the service restarted, probably fine, close it? The cost of guessing wrong is the reopened incident, which spends your credibility along with the extra downtime. Two preparations shrink it. First, every runbook ends with verification steps. Those steps provide the specific test that proves this service works for a user — open this file, send this mail, load this page — not just that a process is running. Deciding what "fixed" looks like before you claim it is the whole trick, and it is decided at writing time, not at 4 a.m.


Second, let monitoring do the confirming. When the same checks that detected the failure show green again — and stay green through a quiet period — you have recovery proof. It is timestamped and impartial. Watching the graph recover beats asking "seems okay now?" in chat. How long a quiet period to demand is a judgment call. Ten or fifteen minutes suits most failures, while anything that failed by slowly degrading deserves longer, because that is exactly the pattern that fools a quick glance. This is the all-clear discipline the incident response guides formalize, and it is what makes the VERIFIED timestamp honest rather than hopeful. That is the difference, as the honest-measurement guide puts it, between a fixed clock and a flattering one.


The practice lever: drills and second responders


One lever cuts across all four phases: practice. Practiced hands move faster everywhere. Engagement is quicker because the login path is worn smooth. Diagnosis is quicker because the runbook has been walked before. Fixes are quicker because the failover has been executed by this person previously. Verification is quicker because everyone knows what green looks like. The vehicle is the drill, scaled honestly for a small shop. Start with the tabletop: an hour in a meeting room, one scenario ("the firewall just died — go"). Talk it through step by step with the actual runbooks open, noting every "wait, who has that password?" Each snag found is real MTTR pre-paid. Then climb the ladder to live drills where the stakes allow. Restore a config to the spare switch for real. Fail over to the backup internet link during a maintenance window. Run the evidence script against a lab machine. A modest calendar sustains it: one tabletop per quarter, one live exercise per half-year, each ending with a fix list. And the first drill always produces the longest list, which is precisely the point.


Practice also teaches the second-responder pattern. On bad incidents, two people engaged early beat sequential heroics. One drives the fix while the other reads logs, keeps the timeline, and handles the "any update?" traffic that otherwise interrupts the driver every four minutes. Small teams resist this ("we can't spare two!") until they time it: on a severity-1, the second responder usually shortens the incident by more than their own time in it. Who drives and who supports is a roles question — the command structure in the incident response guides — but the MTTR effect is measurable and quick.


The measurement loop: one phase per quarter


Like detection, repair improves best as a campaign: baseline, work one phase, re-measure. From the incident log, compute the median repair time and — the crucial extra — rough phase splits for the worst few incidents. Say the baseline shows a median MTTR of 95 minutes across a dozen incidents. The phase splits say the median incident spent 10 minutes engaging, 50 diagnosing, 20 fixing, 15 verifying. The diagnosis phase is the obvious first campaign. In this example, spend a quarter writing runbooks for the top five recurring alerts, building the evidence script, and starting the signature library. Re-measure next quarter: median MTTR 55 minutes, diagnosis median down to under 20. Now the numbers point somewhere new, perhaps at the two after-hours incidents where engagement took half an hour. That makes escalation timers and the readiness drill the next quarter's work. One phase at a time, with a before-and-after number each quarter: that cadence is sustainable indefinitely, visible to management, and immune to the everything-at-once burnout that kills improvement projects.


Tip: when you shorten one phase dramatically, expect the total to improve by a little less than the arithmetic promised. Phases overlap in real life, and the log's timestamps are honest approximations. Judge campaigns by the trend across several incidents, not by any single showcase repair.


The playbook card and readiness checklist


PhaseSmell of troubleLeversTypical cost
EngageAlerts acknowledged late; 3 a.m. fumblingEscalation timers (tested), access-readiness drill, on-call kitConfig + 10 min/month
DiagnoseLong "what is happening?" stretches; repeat incidents re-solvedRunbooks for top-5 alerts, enriched alerts, evidence script, signature libraryA quarter of writing evenings
FixWaiting on parts, backups, or permissionConfigured spares, tested config restores, pre-agreed emergency authorityHardware + one calm meeting
VerifyReopened incidents; "seems fine" closesVerification steps in runbooks, monitoring green-again proof, quiet periodMinutes per runbook
All phasesEverything is slower than it should beTabletop and live drills, second-responder patternOne hour/quarter + one exercise/half-year

RESPONSE-READINESS CHECK — monthly, ten minutes, actually do the steps

[ ] VPN and admin logins work RIGHT NOW (log in, don't assume)
[ ] Password vault opens; emergency account tested and current
[ ] Monitoring dashboard reachable from phone and laptop
[ ] Top-5 runbooks open, and still match reality
[ ] Evidence script runs clean against a healthy target
[ ] Spares on the shelf match their labels; config date noted
[ ] Latest config backup restored somewhere this quarter
[ ] You can state the emergency-authority line from memory
[ ] Escalation test: an unacked alert reached the second contact

Where to go next


Repair time is preparation wearing a stopwatch: runbooks, spares, permissions, and practice, revealed one phase at a time by an honest log. Pair this playbook with its detection twin and you hold both levers of the incident clock, plus a quarter-by-quarter story of both numbers falling. That is the raw material Reporting Incident Metrics to Management turns into budget for the next round of spares and drills. The rest of Measuring Detection & Response keeps the loop honest. Do the preparation at the comfortable desk; the person at 3 a.m. is you.


Put this into practice with SysPeep Monitor

Runbooks, spares, and drills are on you — but the moments around them are where monitoring earns its keep in MTTR. SysPeep Monitor alerts the right person through channels that interrupt. It shows the health picture that speeds diagnosis on live dashboards and maps. It then proves recovery when your checks go green again. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep