Counting the Outages You Prevented (Honestly)
"We prevented seven outages this quarter" is the strongest sentence an IT team can say to the people who pay for it. It is also the sentence most likely to be challenged, because everyone in the room knows it cannot be checked the way an outage can. An outage happened; it has a start time and a ticket. A prevented outage did not happen, by definition, and the person claiming it is the same person who benefits from the claim.
That does not make the claim worthless. It makes it a claim that has to be earned, entry by entry, with evidence for each. The rules must be strict enough that a sceptic can find nothing to pull at. I have watched a prevented-outage count get laughed out of a budget meeting because one entry in twelve was a stretch. The other eleven were solid, and nobody remembers them.
By the end of this guide you will be able to state the three criteria a prevented outage must meet. You will be able to apply conservative rules that keep the count defensible. You will be able to tell the classes of catch that usually qualify from those that usually do not. You will have a prevented-incident log you can copy. You will be able to state the count upward without cost figures you cannot support. And you will be able to explain why one inflated entry undoes the whole exercise.
In this guide
- The most powerful claim and the easiest to abuse
- The three criteria
- The conservative rules
- What usually qualifies, and what does not
- The prevented-incident log
- Stating it upward
- Why inflation destroys the exercise
- The entry that was withdrawn
- The checklist
- Where to go next
The most powerful claim and the easiest to abuse
A prevented outage is a service interruption that would have happened, and did not, because the monitoring system showed something and a person acted on it. The claim is powerful because it connects the monitoring spend and the administrator's time directly to a thing the business understands: invoicing did not stop. It is the only claim in this pillar that turns quiet work into a business outcome in a single sentence.
It is easy to abuse because the outage did not happen, so nobody can prove it would have. Every alert that cleared could be described as a save. Every patch could be described as preventing something. A team that wants a big number can have one by lunchtime. And a reviewer who suspects that is what happened will discount the whole figure to zero rather than work out which entries are real. The count is either strict or it is nothing.
So this guide is mostly about restraint. The invisible work problem introduced the work log, which records everything the monitoring system asked you to do. The prevented-incident log described here is a much shorter list drawn from it, containing only the entries that pass three tests. Most weeks it gains nothing. That is what makes it worth reading.
The three criteria
An entry goes in the prevented-incident log only when it has all three of the following, each with evidence you can point to.
A monitoring signal
Something in the monitoring system showed the problem before it became an interruption: an alert, a trend on a graph, a failed check. The evidence is the alert history entry or the graph with the date range. If the problem was noticed by a person walking past a rack, it may still have been a fine piece of work. But in that case it does not belong in this log, because this log is the case for the monitoring.
An action taken because of it
A person did something, on a date, in response to the signal: expanded the volume, replaced the cable, renewed the certificate, re-ran the backup. The evidence is the change record, the ticket, or the acknowledgement timestamp plus the graph showing the intervention. No action, no entry. A signal that was noted and turned out to be fine is a good threshold, not a prevented outage.
A plausible, specific failure that was averted
You can name the service that would have stopped, roughly when, and who would have noticed. "The file server volume would have filled on or around the 18th; the shared drive and the invoicing export would have stopped saving; finance would have noticed within the hour." The evidence is the projection: the growth rate from the graph and the date it crosses the limit. If you cannot name the service and the date, you have a hunch, and hunches go in the work log, not here.
All three, every time. Two out of three is the most common way a log gets an entry that is later withdrawn. A withdrawn entry costs more than it was ever worth.
The conservative rules
The criteria say what an entry needs. These rules say how to apply them when you are tempted, which is most of the time.
- Never count an alert that cleared itself. If the condition went away without a person doing anything, nothing was prevented; the threshold fired and the world moved on. This rule alone removes most of the candidate entries, and it should.
- Never count "might have". "The switch might have failed" is not an entry. "The switch was reporting a failed fan and running at a temperature the vendor lists as the shutdown threshold" is an entry, if you replaced the fan. The difference is whether the failure had a mechanism you can describe.
- Count the disk at 94 per cent and rising, not the one at 60. A volume at 60 per cent with a flat trend that you expanded during a quiet week was good housekeeping. A volume at 94 per cent and rising a point a day that you expanded before it filled was a prevented outage. The entry has to show the failure was close and the trend was heading for it.
- One entry per failure, not per alert. Six alerts about one disk are one entry. Ten flapping-port alerts on one bad cable are one entry.
- Write the entry on the day, and write the doubt with it. If you are not sure the failure would have followed, say so in the entry and leave it out of the count. An entry marked "uncertain" is honest. An entry quietly promoted to certain in December is the one that gets challenged.
- When in doubt, leave it out. A log of five entries that all survive questioning is worth more than a log of fifteen with two soft ones. This is not modesty; it is arithmetic about trust.
The rules feel severe the first month, because they cut a plausible-looking dozen down to three or four. Then someone asks about one of the four. You show the graph and the change record and the projection, and they stop asking. (Nobody has ever asked me to justify an entry twice.)
What usually qualifies, and what does not
Some kinds of catch pass the three criteria naturally, because the failure they avert has a clear mechanism and a date. Others almost never do, however much work they took. The table is a starting point; the criteria still decide each entry.
| Class of catch | Usually qualifies? | Why | Evidence to keep |
|---|---|---|---|
| Capacity catch (disk, memory, address pool, link) near its limit and rising | Yes | The failure mechanism is "it fills"; the date comes from the trend | Usage graph with trend; change record; projection |
| Creep catch (error rate, latency, queue depth climbing towards a limit) | Usually | Needs a stated limit at which users would have noticed | Metric graph; the limit and why; action record |
| Certificate expiry caught before the date | Yes | The failure is certain and dated: browsers warn on that day | Certificate check history; renewal record |
| Failed backup noticed and re-run before it was needed | Yes, with care | Averts "no usable backup"; state the window of exposure honestly | Backup check history; re-run log; verification |
| Dead redundancy found before it was needed (failed second power supply, dead second link, stale replica) | Yes | The next single failure would have been an outage instead of a non-event | Hardware or link check history; replacement record |
| Failing hardware (fan, disk errors, temperature) replaced before failure | Usually | Needs the vendor's failure condition or a clear error trend | Health check history; error counter graph; replacement record |
| Alert that cleared on its own | No | Nothing was prevented; no action was taken | — |
| Routine patching with no specific vulnerability or fault | No | Good practice, but no named failure with a date | Ticket system |
| Restart of a service that was already down | No | That is an outage, and it goes in the availability record | Incident report |
| Threshold tuned so an alert stops firing | No | Improves the monitoring; prevents nothing by itself | Alert review notes |
Capacity and creep catches are the bulk of most logs. They come out of the weekly trend review. That is where a rising line gets noticed while there is still time to act on it. For the capacity ones, the entry is stronger when the action was taken with enough lead time that the purchase was not an emergency. The guide on lead time and buying before the pain covers that timing. Failed backups are a class of their own, because the thing averted is not an outage but the absence of a recovery. The guide on monitoring backups explains what the check should be watching for the entry to hold up.
The prevented-incident log
The prevented-incident log is one row per entry that passes all three criteria, with six columns. It is kept for as long as the organisation might ask. It is short by design. Copy the table and keep it next to the work log. Entries move from the work log into this one only after the criteria have been applied.
| Date | Signal | Action | What would have failed | How you know | Evidence |
|---|---|---|---|---|---|
| (date) | Invoicing database volume at 94%, rising ~1%/day for 30 days | Volume extended by 500 GB on the 12th | Volume full around the 18th; invoicing exports and shared drive stop saving; finance notices within the hour | 30-day trend crosses 100% on the 18th; same volume filled once two years ago with that result | Disk graph, 30 days; change record 231; projection sheet |
| (date) | Customer portal certificate expiring in 14 days | Renewed and installed on the 14th | Browser security warning on the public site from the 28th; every customer sees it | Expiry date is on the certificate; browsers warn on expiry | Certificate check history; renewal record |
| (date) | Core switch reporting one of two power supplies failed | Supply replaced from spares on the 3rd | Next power event on the remaining supply takes every site service down | Redundancy was gone; one supply cannot fail safely | Hardware check history; replacement record |
| (date) | Finance database backup job failed overnight (target full) | Target cleared, job re-run and verified before 08:00 | No usable backup of the finance database; exposure window 9 hours | Backup check showed no successful set; restore test passed after re-run | Backup check history; job log; verification note |
The "how you know" column is the one that separates this log from a list of good deeds. It states the mechanism by which the failure would have followed, in a sentence a sceptic can test. For capacity entries, that sentence is a projection, and a projection is arithmetic you should be able to show. The spreadsheet version is two cells, illustrative and simple on purpose:
Illustrative spreadsheet cells (adapt to your sheet layout) B2 = usage now, per cent (from the monitoring graph) B3 = usage 30 days ago, per cent (from the same graph) Daily growth, points per day: =(B2-B3)/30 Days until full at that rate: =ROUNDDOWN((100-B2)/((B2-B3)/30),0) Rule: enter the catch only if days-until-full was under about 30 and the growth was steady across the period, not one jump.
A projection drawn through one week's data is a guess; one drawn through a month with a steady slope is evidence. If the growth was a single jump — someone copied a large folder — the days-until-full figure is meaningless and the entry should say so. Honest projections are the whole point. A straight line through a spike is the fastest way to lose the argument you were trying to win.
Tip: when an entry rests on a graph, save the graph image with the date range in the file name at the time you write the entry. Monitoring data ages into rollups, and a thirty-day graph that was crisp in March can be a smoothed line by December. The saved image is the evidence; the live graph is a convenience.
Stating it upward
The count is stated as a count, with the top entries named in business terms, and nothing else. "We recorded seven prevented incidents this quarter; three of them would have stopped invoicing, one would have taken the public website down for a weekend, and the log is available." That is the whole statement. It is short because a longer one invites interpretation, and it is specific because "seven" with three named is a fact, while "numerous" is a mood.
Do not attach cost figures unless you have derived them properly. "Seven prevented outages saved the company a fortune" is the sentence that gets the log discounted. That is because everyone in the room knows you made the number up on the way to the meeting. If a cost figure is wanted, build it in front of the reader from stated assumptions — how many people, for how long, doing what. In that case, use the method in risk and cost framing, and present the cost figure as a range with its assumptions attached. Otherwise, state the count and the services, and let the reader who knows what invoicing is worth do the multiplication themselves. They will, and their number will be believed, because it is theirs.
The statement goes in the quiet month report as the "caught early" section. It goes in the quarterly summary as one line. In the annual conversation, it is the second of the three sentences that answering "what do we pay you for?" is built on. It is the same count in all three places, because the log is the only source.
Why inflation destroys the exercise
One challenged entry discredits the whole log, because the log's value is entirely in the reader's belief that every entry would survive a challenge. A reviewer does not have time to check seven entries. They check one, usually the one that looks softest, and generalise. If it holds, the other six are accepted. If it does not, the other six are assumed to be the same. Then the count goes to zero in the reviewer's mind however good the rest were.
This is why the conservative rules are not optional and not modesty. A log with one soft entry in seven is worth less than a log with four hard entries, even though it is "bigger". The reviewer is not measuring the number; they are measuring whether you can be trusted to count. And a single stretch answers that question for the year.
Inflation also destroys the log's second use, which is internal. A team that keeps an honest prevented-incident log learns, over a year, which classes of catch it is good at and which it misses. That is the best input there is to deciding what to monitor next. A padded log teaches nothing, because the pattern in it is what the author wanted to claim, not what the monitoring caught.
The entry that was withdrawn
At Calder Engineering the prevented-incident log had nine entries for the year. The finance director, who read carefully, asked about the seventh: a firewall firmware update described as having prevented "a likely outage". The administrator looked at his own evidence column, which said "vendor advisory". He found that the advisory described a fault in a feature the firm did not use, and said so in the meeting. He withdrew the entry on the spot, wrote "withdrawn on review, not applicable to our configuration" in the note column, and moved on to the eighth. The finance director asked no more questions about the log that year. The following year, the director approved the second administrator's post on the basis of a log with six entries. Every one of those entries had a graph.
The withdrawal was the most persuasive thing in the meeting. It showed the reader that the log was kept by someone who would rather have eight true entries than nine impressive ones. From that moment the other eight were facts. A log that has never had an entry withdrawn has either been kept perfectly or has never been read closely. And the reader can tell which. Withdraw the soft ones yourself, in the note column, before anyone else has to.
The checklist
Apply this to every candidate entry before it moves from the work log into the prevented-incident log, and to the whole log once a quarter.
- The entry has a monitoring signal, with a pointer to the alert history entry or the graph and its date range.
- The entry has an action taken by a person on a date, with a pointer to the change record or ticket.
- The entry names the service that would have failed, roughly when, and who would have noticed.
- The "how you know" column states the mechanism in one sentence a sceptic could test.
- The alert did not clear on its own.
- The failure was close: a capacity entry shows the limit within weeks on a steady trend, not a flat line at 60 per cent.
- Multiple alerts about one failure are one entry.
- Any doubt is written in the entry, and doubtful entries are excluded from the count.
- Graph evidence is saved as an image with the date range in the file name.
- The upward statement is a count plus the top entries in business terms, with no cost figure unless it was derived with stated assumptions.
- Any entry withdrawn on review is kept in the log with a note, not deleted.
Where to go next
The prevented-outage count is the strongest claim in this pillar and the easiest to ruin. It survives only when every entry has a signal, an action, and a specific averted failure, each with evidence. The rules must also be applied conservatively: no self-clearing alerts, no "might have", the disk at 94 and rising rather than the one at 60. State the count, name the top entries in business terms, and withdraw the soft entries yourself. Six entries with graphs will do more for you than fifteen with adjectives.
The log feeds the quiet month report every month and the one-page record in answering "what do we pay you for?" once a year. Uptime reporting as reputation defence is the record of the outages that did happen, kept to the same standard. All of them are part of Making Invisible Work Visible With Monitoring Data.
Put this into practice with SysPeep Monitor
Every entry in a prevented-incident log rests on a signal you can point to afterwards. SysPeep Monitor keeps alert history and the graphs behind it in one web console. It has thresholds and alert rules on the metrics it collects over SNMP, WMI, and SSH. So the disk trend and the acknowledgement timestamp are still there when the entry is questioned. The free trial installs on a Windows machine if you want to see what your own network would have put in the log this month.
Download Free TrialLearn About SysPeep