The First Fifteen Minutes of an Incident
It is ten past two on a Tuesday and the file server has stopped answering. Monitoring has emailed you about it. The front desk has phoned about it. Your manager is standing in the doorway about it. Something is down, your pulse is up, and your hands want to start fixing things before your head has caught up.
Incidents are mostly won or lost in the first quarter-hour. Not because the fix happens that fast; it usually does not. The early decisions set the shape of everything that follows. Confirm sloppily and you spend an hour chasing a ghost. Skip the scoping questions and you wake the wrong people, or nobody. Say nothing and the ticket queue fills with duplicates while your credibility drains out of the room. Reboot the wrong box and you destroy the only evidence of what went wrong. I have done three of those four. I am not saying which three.
The good news is that the first fifteen minutes are the most learnable part of incident response, because they have the same shape every time. It does not matter whether the file server died, the internet dropped, or everything is mysteriously slow. This guide gives you that shape as a playbook: four phases, each with a time band, one question to answer, and one discipline to hold. By the end you can run it from memory, alone, at 2 a.m., with adrenaline doing its best to make you careless. That is exactly the moment it was built for.
In this guide
- Why the first quarter-hour decides the rest
- Minutes 0–3: confirm it is real
- Minutes 3–7: scope the blast radius
- Minutes 7–10: send the first notice
- Minutes 10–15: stabilize or dig
- Running it alone, or with two people
- A rhythm, not a stopwatch
- A worked example, minute by minute
- The printable first-fifteen card
Why the first quarter-hour decides the rest
An incident is any unplanned interruption or serious degradation of a service people rely on. The word matters because saying it flips a switch in your head. You have stopped doing project work and started doing triage. That is the emergency-room discipline of establishing what is wrong and how bad it is before treating anything.
Emergency rooms are the right mental model, because medics face the same trap: the urge to act immediately. Paramedics do not start surgery in the hallway. They confirm the patient is actually in trouble, assess how widespread the damage is, and tell the family and the surgical team. Only then do they follow a known protocol or investigate further. That sequence — confirm, scope, communicate, then stabilize or dig — is the whole playbook. The rest of this guide is detail hung on those four hooks.
The minute bands are there to fight panic, not to grade you. A time box gives you permission to do one job at a time. For the next three minutes you are only confirming. And the fix can wait because its turn is coming.
Figure 1: The first-fifteen playbook. Four phases, one question each, and the fork at the end that decides how the rest of the incident will run.
Minutes 0–3: confirm it is real
The question this phase answers: is this actually happening? It sounds insultingly basic. It is also the step most often skipped, and the source of the most embarrassing war stories in the profession.
The discipline is the second-source check: before treating an alert as an incident, get one more signal that agrees with it, arriving by a different path. If monitoring says a server is down, you have several options. Glance at the dashboard to see the wider picture, or ping the machine, or try the service yourself the way a user would. If a person says "email is down," check whether monitoring agrees before you believe them. One signal is a claim. Two independent signals are a fact.
Why the suspicion? Because every source of alarm can be wrong. Monitoring is wrong when a check flaps, when the monitoring machine's own network hiccups, or when a maintenance window failed to suppress. People are wrong in perfectly good faith: "the internet is down" has meant, in living memory, one slow website, an unplugged dock, and a Wi-Fi toggle. The second source catches the mistake in ninety seconds instead of ninety minutes.
C:\> ping files01 Pinging files01.corp.local [10.0.20.11] with 32 bytes of data: Request timed out. Request timed out. C:\> ping 10.0.20.1 (the switch files01 hangs off) Reply from 10.0.20.1: bytes=32 time=1ms TTL=64 Reply from 10.0.20.1: bytes=32 time=1ms TTL=64
Two answers, one story: the network path is fine, and the server itself is not answering. That is confirmation. And if the second source had said everything was healthy? Congratulations — you have a monitoring problem, not an outage. Take the false-alarm exit: stand down, note the alert for tuning later, and get your evening back. A false alarm caught in minute two costs almost nothing. A false alarm "fixed" with a reboot costs an outage.
At Tollgate Foods the monitoring server lost its own network cable at 07:40 on a Monday. Every check it ran turned red in the same second. The on-call admin, four minutes out of bed, saw forty alerts and reset the database server from its management card. That was because the database was the most important thing on the list. It came back nine minutes later, which was nine minutes longer than it had been down, because it had not been down. The review recorded the cause as "a cable" and the outage as "self-inflicted". The admin was not blamed, the cable was labelled, and the first line on the card at that desk now reads "ping the switch".
The discipline to hold during confirm: resist the fix reflex. Looking is allowed; changing is not. Understanding must come before intervention, and every experienced admin carries a private catalog of shoot-first regrets that taught them this:
- Rebooted a healthy server because the alert was really a fault in the monitoring machine's own network path. The reboot then caused the outage the alert had only imagined.
- Restarted a hung service on sight — destroying the stuck-process state that would have shown why it hung. It hung again the next night, with the same mystery intact.
- Failed over to the standby mid-confusion, not realizing the standby was the half that had failed. One outage became two, plus a resync.
- Spent forty minutes "fixing" DNS while the actual problem was a dead firewall port that a ten-second scope check would have caught.
Tip: write it on the card and mean it: no changes during confirm. Three minutes of looking before touching feels unbearable mid-incident and invisible afterward. Nobody ever complained that an outage ran three minutes long because the admin checked a second source first.
Minutes 3–7: scope the blast radius
The incident is real. The next question: how big is it? Responders call this the blast radius — how far the damage extends — and you can map it with three quick questions. One user or all of them? One service or several? One site or everywhere? Each answer changes what kind of incident you are in. One person locked out of one mailbox is a ticket; everyone locked out of everything is a very different afternoon.
Answering fast is mostly a dashboard skill. Sweep the monitoring view and read both colors: what else is red, and — just as informative — what is still green. If the file server is down but its switch, its neighbors, and the internet link are all green, you have a one-host problem. If forty alerts arrived in two minutes, you almost certainly do not have forty problems. In that case, you have one problem upstream of forty checks. The oldest alert in the flood is usually closest to the cause. Reading a pile of alerts as one event is a skill of its own, covered in alert storms. The short version: find the top of the dependency chain and suspect everything below it. An alert storm is one problem with a great many friends.
Round out the picture with a test from a second vantage point. Try the service from a different network segment, or check the internet link from a phone on mobile data. Location-dependent symptoms point at network gear; everywhere-symptoms point at the service itself.
Scope is also what lets you call a severity — the agreed how-bad label. SEV-1 is for drop-everything, SEV-2 for serious-but-contained, and SEV-3 for minor. That label sets the response machinery moving without a debate. Calling it early beats calling it perfectly. If your team lacks working definitions, the severity model guide gives you a three-level scheme you can adopt this week.
Scope earns four whole minutes because everything downstream keys off it. Who gets told — two affected people, or the whole company? Who gets woken — nobody, a colleague, or the boss? What gets promised — "fixed shortly," or an honest "this will take hours"? Get scope wrong and every one of those calls is wrong with it, in whichever direction embarrasses you most.
Minutes 7–10: send the first notice
Now the step that feels wrong. The cause is still unknown, the fix has not started, and the playbook says: stop diagnosing and spend three minutes on communication. Every instinct votes to fix first, explain later. The instinct loses, for a practical reason: people who know that you know do not flood you. Silence during an outage is never read as focus; it is read as ignorance. Users assume nobody has noticed, so they file tickets, then duplicates, then walk to your desk. Each interruption lands on the person trying to fix the thing. None of them are being unreasonable. Each has a customer on the phone, and until you say something, you are the only person they can tell. One early notice cuts the flood off and buys you quiet to work in.
The first notice is a ninety-second job because it only has three sentences. What is affected, in user terms — "email is down," not "the mail store dismounted." That you know and are working on it. And when the next update will come, as a clock time:
File shares (the F: and G: drives) are down for everyone. We are aware and working on it now. Next update by 14:45 — sooner if it is fixed.
The third line is the quiet hero. A promised update time is the anxiety killer: people refresh less, call less, and forgive more when they know exactly when news is coming. It also disciplines you, because the promise comes due whether or not the fix does. There is a whole craft to outage updates — cadence, audiences, honest ETAs — and the outage communication guide covers it with templates you can steal.
Communication has an internal half too. Open the incident channel — a chat thread, channel, or shared note named for the incident. Post what you know so far, with timestamps. From now on, the response lives in one place: facts, decisions, evidence links. If colleagues join, they read the thread instead of interrupting you for a recap. If it is just you, the thread is your log, and the review afterward will lean on it heavily. How to run that one-place discipline, even with two people, is the subject of incident command for small teams.
Minutes 10–15: stabilize or dig
Confirmed, scoped, communicated. Only now does the playbook let you at the problem — and it opens with a fork. The question: have we seen this failure before?
Known pattern → runbook. A runbook is a written recovery procedure for a known failure — the "when the VPN appliance wedges, do these six steps" document. If the symptoms match one, execute it rather than improvising. 2 a.m. is not the time to freestyle a procedure a calmer version of you already wrote down. Building that library is its own discipline — see Runbooks. Every incident that ends without a matching runbook is a hint about which one to write next.
Unknown → dig, but grab the evidence first. Before your first intervention, spend one deliberate minute capturing the scene. Screenshot the dashboards and graphs. Save the tail of the relevant logs. Note what changed recently (patches last night? a config change? a power blip?). Record process and service states if the machine will still talk to you. The most tempting fixes are also evidence shredders. A reboot destroys memory contents, hung-process state, temp files, and queued spool data. Restarts truncate some logs. A failover abandons the broken half unread. The review afterward will try to rebuild a timeline. And it can only work from what survived. The guide Post-Incident Reviews leans almost entirely on the material this one minute preserves.
Warning: a reboot is irreversible for evidence even when it is harmless for the service. Sixty seconds of capture before you restart anything is nearly always affordable. What it preserves is sometimes the only thing standing between you and meeting the same incident again next month.
There is a third path, and it deserves honest treatment: stabilize first, diagnose later. Restore service by the fastest safe route — fail over to a standby, switch to a workaround, roll back the change. Postpone the cause hunt until users are working again. Service-first wins when severity is high, a known-good path exists, and diagnosis looks slow. But say the trade out loud in the thread, because it is real. Failing over often destroys the evidence, and a cause you never found usually comes back. Treating the symptom now is a legitimate call when made knowingly, a trap when made by reflex. The distinction has its own guide in root cause vs. symptom.
Running it alone, or with two people
Solo response is the reality for many small shops, and the playbook is built for it. There is one addition: the printed card is your co-pilot. Adrenaline reliably deletes steps from memory; the card does not negotiate. Tape it inside a cabinet door, and when the alert lands, put a finger on line one. Narrating your moves into the incident thread substitutes for the teammate you lack: it forces clear thinking now and becomes the timeline later.
With two people, split the work by role, not by taking turns. One drives: hands on keyboards, owning diagnosis and fixes. The other communicates and scribes: sending updates, keeping notes, and absorbing interruptions (including managerial ones) so the driver's concentration survives. The split works because the two jobs sabotage each other in one brain. Every status question yanks the driver out of a fragile train of thought. The full division of labor is in incident command for small teams.
A rhythm, not a stopwatch
Nobody stands over you with a whistle at minute seven. The bands are honest defaults, and real incidents compress or stretch them. When the whole internet connection is down, confirm takes twenty seconds — your own session is dead and the dashboard agrees. When symptoms are vague ("everything is slow-ish?"), confirm and scope may honestly take ten minutes together. Both are fine.
What is not fine is skipping a phase. The confirm–scope–communicate spine survives every incident shape, because each phase feeds the next. You cannot scope what you have not confirmed, or write a truthful notice about what you have not scoped. And if minute fifteen arrives with the problem unsolved — normal, for hard ones — you have still banked the wins. It is real, you know how big it is, users are calm, and you are acting deliberately. From here the response is a loop: dig, update on schedule, re-scope when facts change.
A worked example, minute by minute
Here is the playbook running against a classic: the file server that stops answering on a Tuesday afternoon.
14:02 — Monitoring emails: "files01 unreachable — 3 failed checks." Pulse jumps. Finger on line one of the card.
14:03 — Confirm. Dashboard: files01 red; its switch, its neighbors, everything else green. Second source: you open \\files01\public from your own PC — hangs. Two signals agree. Real.
14:04 — Scope. One host down, everything else green — no storm, no upstream cause to hunt. But that host carries the shared drives for the whole office: broad impact, single service, single site. Severity: middle tier — hurry, but not wake-the-world. Mid-afternoon, so nobody gets woken; everyone gets told.
14:07 — Communicate. The three-sentence notice goes out: shares down for everyone, we know and are on it, next update by 14:45. You open a thread named inc-files01-tue and paste the facts with timestamps. Two tickets arrive anyway; both get a link, and the flood never forms.
14:10 — The fork. Runbook list: there is one for "files01 disk full," but the disk graphs say 71% — no match. Unknown, then: evidence minute. You take screenshots of the latency graph (a spike at 13:55, then flatline). You note that patches installed at 03:15 last night. You take a photo of the console — frozen at the login screen, keyboard dead.
14:12 — Decision, typed into the thread: "Console frozen, state captured. Hard reset via the management card at 14:13." Evidence is saved, so the reset destroys nothing you have not kept.
14:15 — The quarter-hour mark. Server is at its boot screen. Internal note posted: "resetting now; user update at 14:45 as promised." The incident is not over — but it is confirmed, scoped, communicated, and being acted on deliberately, with the evidence in your pocket.
Notice what did not happen: no ghost chase, no forty duplicate tickets, no destroyed evidence, no heroic random rebooting. And note what remains ahead. When files01 comes back, proving it is actually fixed — and saying so — is its own discipline. That is covered in declaring the all-clear.
The printable first-fifteen card
This is the whole guide compressed to one card. Print it, tape it where incidents find you, and let it be the calm one.
| Minutes | Phase | Do | Done when |
|---|---|---|---|
| 0–3 | Confirm | Second-source check: dashboard, ping, or try it yourself. No changes yet. | Two independent signals agree — or false alarm: exit, note for tuning. |
| 3–7 | Scope | One user or all? One service or many? One site or everywhere? Read the alert flood as one event. Call a provisional severity. | You can say who is affected in one sentence. |
| 7–10 | Communicate | Three-sentence notice in user terms, with a next-update time. Open the incident thread; post facts and stamps. | Notice sent; one working channel exists. |
| 10–15 | Stabilize or dig | Known pattern → runbook. Unknown → one evidence minute, then dig. Service-first only by explicit choice, stated in the thread. | A deliberate path chosen, started, and written down. |
Where to go next
The playbook does not make you faster at fixing things. It makes you slower at the three things that go wrong before the fixing starts: believing an alert, guessing the size, and going quiet. Confirm, scope, communicate, then choose a path on purpose. Fifteen minutes in, the server may still be down. But you know it is real and how big it is. The office knows you are on it. And the manager in the doorway has a sentence and a clock time to walk away with. That is a better quarter-hour than most incidents get.
The first fifteen minutes are the front door of incident response; the rooms behind it each have their own guide. Calling how bad it is deserves real definitions — the working severity model. The notices you promised have a craft — outage communication. Running the response with two or five people has a structure — small-team incident command. And ending properly is harder than it sounds — verification and the all-clear. The rest of the pillar lives at Incident Response.
Put this into practice with SysPeep Monitor
The confirm and scope phases go faster when the second source is already on screen. SysPeep Monitor auto-discovers your network and watches it over SNMP and WMI. It uses dependency rules to keep one outage from arriving as forty separate alarms. So minute zero starts with a map instead of a mystery. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep