Home › Topics › Incident Response › The All-Clear

Ending an Incident Properly: Verification and the All-Clear


Every incident ends twice. The first ending is the one that feels wonderful: the fix lands, the graph twitches upward, and the service appears to breathe again. The second ending is the one that counts: the fix is proven. It has been shown to work, across everything that broke, for long enough to trust, by the people who felt it. A team that treats the first ending as the ending keeps a standing appointment with the same incident, usually around dinner time.


The gap between "fixed" and "proven" is where reputations are made and lost. An all-clear that holds builds the quiet credibility an IT team runs on. An all-clear retracted two hours later costs more trust than the outage did, because an outage looks like bad luck and a bounced all-clear looks like nobody checked. I have sent the second kind. The graph had been green for a minute, and I wanted it to be over more than I wanted to be right.


This guide is the discipline of the second ending. It covers why premature all-clears happen to careful people, and the four layers of verification that make "proven" a checklist instead of a feeling. It covers the provisional all-clear for fixes that need time to trust, and what to do when the incident comes back anyway. It covers the administrative close that hands a finished incident to the learning process instead of dropping it on the floor.


In this guide

  • The premature all-clear, anatomized
  • What "proven" means: four layers
  • The direct check
  • The breadth check
  • The duration check and the provisional all-clear
  • The user check
  • Saying it: the final notice and the internal close
  • When it comes back: the re-open protocol
  • The administrative close
  • A worked ending, minute by minute
  • The all-clear card

The premature all-clear, anatomized


The failure has a classic shape, recognizable from any IT war-story collection. A fix is applied. The most obvious symptom disappears. Somebody broadcasts "we're good!" to visible relief. Everyone scatters — to lunch, to the backlog, to bed. And within hours the incident re-erupts. The fix was partial, or the symptom was intermittent and merely between appearances. Or a second broken thing had been hiding behind the first. Now you get to announce the same outage twice, and the second announcement lands on an audience that has stopped believing your announcements.


The double outage is strictly worse than one long one. Users forgive duration far more readily than they forgive whiplash. "Down for four hours" reads as a hard problem. "Fixed, then broken again an hour later" reads as a team that does not know when things are fixed. And internally, the re-eruption catches everyone dispersed and demoralized — the second response starts slower than the first did.


At Northgate Retail the till system lost its link to head office at 11:50 on a Saturday. The on-call admin found a wedged firewall and restarted it. The graph went green, and at 12:04 the all-clear went to every store manager. At 12:15 the tills dropped again, because the firewall had wedged under lunchtime load and lunchtime was still happening. The review timed the first all-clear at eleven minutes, a figure the admin wrote down themselves. Northgate now holds any Saturday all-clear until after the two o'clock peak. The store managers have stopped asking whether it is really fixed, mostly.


Why do capable people declare victory early? Three honest reasons. Relief bias: after hours of adrenaline, the brain grabs the first evidence that the ordeal is over and calls it conclusive. Pressure to close: the boss wants the good news, users deserve the good news, and holding it back feels like pessimism. Verification-as-ceremony: checking a thing you already believe feels like bureaucratic theater, the way re-reading a form feels — right up until the time the form was wrong. The cure for all three is the same: make verification a concrete checklist rather than a mood, so that "are we done?" has a factual answer.


What "proven" means: four layers


Proven is not one check; it is four, layered from narrow to wide. The direct check: the thing that failed demonstrably works. The breadth check: everything else that broke alongside it works too, and monitoring is fully re-armed. The duration check: it has stayed working long enough, relative to how the failure behaved. The user check: someone who felt the problem confirms it is gone. Small incidents may pass all four in ten minutes; nasty ones stretch the duration layer across a day. But an all-clear that skips a layer is a guess wearing a checkmark.


The direct check


The narrow question first: does the failed thing demonstrably work? Two halves, and you want both. The signal that caught it is green again. The monitoring check that first alarmed has returned to OK on its own schedule. It has not been acknowledged or reset. It has actually measured the service healthy. And the action that failed now succeeds: re-run the actual failure. If users could not send email, send an email. If the share would not open, open the share. If the web app threw errors on login, log in.


The trap here is testing a proxy instead of the failure. The server answering pings is not the database accepting connections. The service showing "running" is not the application doing work; the login page rendering is not a login succeeding. Reproduce the reported failure as literally as you can — same action, same path, ideally same account type. The incident was defined by that failure, and only that failure's absence ends it.


The breadth check


Incidents are rarely tidy enough to break exactly one thing. The outage that took the file server down also failed its backup job, knocked over the print spooler that authenticates against it, and left a dozen alarms in its wake. The headline service coming back does not prove the supporting cast recovered. Sweep the blast radius and re-verify everything that alarmed during the incident, not just the star of the show. If the alerts arrived as a storm, each of the storm's children needs its own green. The correlation skills from alert storms run in reverse here. One cause fanned out into many symptoms, so one fix must be confirmed back across each of them.


The breadth layer has a second, sneakier job: the un-suppress sweep. Mid-incident, responders mute things — the flapping alert silenced to think straight, a check paused during a reboot, notifications snoozed on a phone. Every one of those mutes is now a landmine: monitoring that looks calm because it has been told to be quiet. Before the all-clear, walk the list and re-arm everything. Include the one you muted on your phone at 2 a.m. and have no memory of muting. Then confirm the monitoring is genuinely green rather than gagged. This is the unplanned twin of a discipline maintenance work already taught you — post-window verification — and it is the difference between "no alerts" and "no problems."


The duration check and the provisional all-clear


Some fixes prove themselves instantly: the replaced switch either passes traffic or does not. But intermittent and load-dependent failures cannot be verified by a moment — only by a soak: a stability window during which the service must stay healthy before you finalize. The service that crashed under the morning login rush is not proven at midnight; it is proven at 9:30 tomorrow. The memory leak that killed the app every few hours is not proven after twenty quiet minutes.


Size the window to the failure's own rhythm: the issue that recurred hourly needs hours of green; the one that struck daily needs a day. A useful default is one to two full cycles of whatever pattern the failure kept. Include at least one pass through the conditions that triggered it — the load peak, the nightly job, the backup window.


The communication tool for this gap is the provisional all-clear: "Email is restored and working; we are monitoring closely through tomorrow morning before calling this fully resolved." It hands users the good news the moment it is true, while being honest that trust is still accruing. It makes a re-eruption during the soak a continuation you predicted rather than an all-clear you retract. The distinction sounds subtle; to your credibility, it is everything.


One more rule for the soak: a wobble resets the clock. If the symptom flickers back mid-window — even briefly, even "probably nothing" — the stability window restarts from the last clean moment. In that case, the flicker gets treated as evidence for the diagnosis rather than an inconvenience to explain away. A soak you negotiate with is not a soak; it is optimism with a timer.


Tip: during the soak, tighten the watching. Drop the relevant check intervals, add a temporary threshold a notch more sensitive than normal, and route those alerts straight to the responder. If the problem twitches again, you want to know in one minute, not at the next user complaint.


The user check


The last layer is the cheapest and the most skipped: someone who reported the problem confirms it is gone. Message the first reporter, or the loudest one: "Should be fixed — can you try again and tell me?" This closes a loop monitoring cannot close, because users fail in ways probes do not. There is the cached credential, or the mapped drive that needs a fresh login. There is the one client version that still trips over the workaround you forgot to remove. A green dashboard plus one human "works for me now" is a different order of certainty than the dashboard alone.


The user check also pays a quiet dividend: the person who reported the outage becomes the person who helped end it. That tiny bit of participation is how users learn that reporting problems early is worth their time — which makes your next incident's detection faster.


Saying it: the final notice and the internal close


With all four layers green, send the real all-clear. Its anatomy, in user terms, starts with restored ("email is back and verified working as of 17:40"). It includes duration ("the disruption lasted about three hours"). It includes a one-line cause category ("a storage fault on the mail server" — the category, not the forensics). It includes what happens next ("a short summary of what we are changing will follow this week"). Add the line that invites contradiction: "if anything still looks wrong on your side, tell us." Fill-in templates for this message live in outage communication, alongside the cadence that led up to it.


User closure and internal closure are different events, and the second one is where teams fumble. The moment the all-clear goes out, everyone's brain files the incident under "over" and scatters — taking with it the freshest memory of what actually happened. (The incident thread is the only witness that does not go home.) Spend fifteen minutes before the scattering. Snapshot the timeline from the incident thread while the timestamps still make sense. Save the graphs of the incident window before they roll off retention. Export the log excerpts, and note the two or three open questions the response left behind. This quarter-hour of capture is the raw material the review will be built from. Post-Incident Reviews reconstructs the story, but only from what survived. The capture completes the evidence habit that began in the first fifteen minutes.


When it comes back: the re-open protocol


Sometimes the incident returns anyway — verification lowers the odds; nothing zeroes them. What matters then is deciding, calmly and by rule, whether this is the same incident re-opened or a new incident. The working rule: same symptom or same cause within a defined window — a few days is a sensible default — re-opens the original. Re-opening preserves continuity: one timeline, one evidence pile, one eventual review covering both eruptions, and an honest record that the first fix did not hold. A genuinely different failure, or the same symptom weeks later, starts fresh with a link to its ancestor.


Announce a re-open with the same discipline as a first notice, plus one sentence of candor: "The email issue from this morning has recurred; the earlier fix did not hold. We are treating it as the same incident and are on it now." Users handle honesty about a returning problem far better than they handle a second, apparently unrelated outage that smells identical.


Track your re-opens. The re-open rate — how often "resolved" incidents bounce — is one of the most honest quality signals a small team can watch. Rising re-opens mean verification is being skipped or soaks are too short, long before anyone would say so out loud. The re-open rate belongs in the same small family of response measurements — how fast you detect, how fast you truly resolve — covered in MTTD & MTTR.


The administrative close


Last, the paperwork — five unglamorous minutes that keep the learning pipeline fed. Finalize the timestamps: when it started, when it was detected, when service was restored, and when it was verified. That last stamp is the one teams forget, and the one that makes duration honest. An incident is over when it is proven over, not when the fix was typed. (Those clocks are exactly what MTTD & MTTR is measured from; sloppy stamps make flattering, useless numbers.) Confirm or correct the severity in the log: the thing classified SEV-2 in the fog of minute four may have proven to be a SEV-1 in retrospect. And the record should say what it was, per the severity model. Schedule the review the severity calls for — the full session for a SEV-1, the fifteen-minute mini for a SEV-2. Do it while the capture is fresh and calendars still remember the pain.


None of this is bureaucracy for its own sake. The administrative close is the handoff between the incident that just ended and the team that will be better because of it. Skip it, and the incident's tuition was paid for nothing.


A worked ending, minute by minute


The last hour of an email outage, narrated. The morning's incident: the mail server's disk filled — runaway log growth — and the mail services fell over at 14:20. During the response, two alerts were muted (the flapping mail-queue check, the disk alert itself). The storm produced children: webmail down, the SMTP probe red, the nightly backup pre-check failing.


16:05 — The fix lands: logs truncated and rotated, root cause of the growth identified (a debug flag left on), disk back to 40%. Services restarted; the queue begins draining. This is ending number one. Nobody says "we're good."


16:10 — Direct check. The SMTP probe that first alarmed goes green on its own poll. A test message sent from outside arrives; a reply makes the round trip. The literal failure — "mail does not flow" — now demonstrably does not not-flow.


16:15 — Breadth check. Webmail: loads, logs in. Queue: drained to normal. Backup pre-check: re-run manually, passes. Then the un-suppress sweep: both muted alerts re-armed, and the disk check confirmed green by measurement, not by silence.


16:20 — Duration reasoning. The disk filled over roughly six hours; a twenty-minute green streak proves little. Soak decision: hold the final all-clear until the debug flag's old growth pattern has had a full evening and the nightly jobs have run. A tighter temporary disk threshold goes on with alerts straight to the phone.


16:25 — User check. The office manager who reported it first tries a send and a reply: "working." Provisional all-clear goes out: restored and verified, monitoring closely until tomorrow morning, say something if anything still misbehaves.


16:40 — Capture, before scattering. Fifteen minutes: the thread timeline is snapshotted, and disk and queue graphs of the incident window are saved. The offending debug config is noted. Two open questions are written down (why did no one notice the growth for six hours? should log volumes be separate?). Review slotted for Thursday.


Next morning, 09:00 — The evening and the nightly jobs passed clean; the tightened threshold never fired. Final all-clear sent with duration and cause category. Timestamps finalized — including VERIFIED at 09:00, not 16:05. Incident closed, and the incident stayed closed, which is the entire point.


The all-clear card


The four layers and the close, sized for printing:


StepProve it byDone when
DirectThe alarming check green by measurement; the failed action re-run and succeedingThe literal failure no longer reproduces
BreadthEvery alarm from the incident re-verified; all mutes and suppressions re-armedBlast radius green, monitoring fully live
DurationSoak sized to the failure's rhythm; provisional all-clear during the windowOne or two failure cycles pass clean, triggers included
UserA reporter re-tries and confirmsA human "works now" on record
Close outFinal notice with duration and cause category; 15-minute evidence capture; timestamps incl. VERIFIED; severity confirmed; review scheduledThe learning pipeline has everything it needs

Where to go next


An incident is over when it is proven over, not when the graph goes green and you want it to be. Run the four layers in order. The failed action works. Everything that broke alongside it works. It has kept working through the conditions that broke it. Someone who felt it says so. Then send the all-clear, capture the evidence before anyone scatters, and write down the time it was verified rather than the time the fix was typed. Do that and the incident ends once, in the right order. Skip it and you keep the standing appointment, and it is still at dinner time.


The all-clear is the last move of the response and the first move of the improvement. The verified ending feeds the review, and the review feeds the fixes. And the fixes make the next first-fifteen-minutes rarer. If you are building the discipline end to end, start at the first fifteen minutes, and classify with the severity model. Then communicate by the update craft, and structure the room with small-team command. The whole set lives at Incident Response.


Put this into practice with SysPeep Monitor

Verification is mostly a monitoring job. The alarming checks are measured green again, and the muted ones are re-armed. A soak window is watched without anyone staring at a screen all night. SysPeep Monitor gives you the live dashboard for the breadth sweep and the thresholds and alert rules for a tightened soak — from one web console. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep