Home › Topics › Am I Normal? Benchmarking Your Network and Monitoring Practice › Benchmarking Uptime Fairly

Benchmarking Uptime and Incident Counts Fairly


"How many outages did you have last year?" sounds like a simple question. It is the most dishonest question in the trade. That is because no two people answer it the same way. One administrator counts a five-minute blip on a test server. Another counts nothing that happened outside office hours. A third excludes the weekend the firewall was replaced, because that was planned, and a fourth excludes it because it was embarrassing. All four give you a number. None of the numbers can be compared.


This guide is about making that comparison fair. You will fix your definitions before you count anything. You will measure your own network the same way for long enough to trust the result. You will compare against your own history first. Only then will you approach a small group of peers with the definitions written down. You will also learn to adjust for size. That way, a shop with eighty devices can have a sensible conversation with one that has eight hundred. You will also learn to read the answer without envy when theirs is lower or smugness when yours is.


One thing this guide will not do is tell you what a good outage count is. There is no such number, and anyone who quotes one has skipped the definitions step. The reference that matters is your own last year, measured your way, and the peers you can find whose way matches yours.


In this guide

  • Fix your definitions first
  • Measure the same way for a year
  • Your own history is the primary benchmark
  • Building a peer group and how to ask
  • Normalising by size
  • Reading the answer honestly
  • What to do with the result
  • A war story and a checklist
  • Where to go next

Fix your definitions first


Three definitions decide what your uptime figure means, and you must write all three down before you count anything. The first is what "down" means: which services and devices count, and what state they have to be in. A server that answers pings while its application is frozen is up by one definition and down by the one your users would choose. The library's Defining "Down" works through this properly. For benchmarking purposes, the point is that your definition must be written in a sentence a peer could apply to their own network.


The second definition is what counts as an incident. An incident is an unplanned event that made a service unavailable by your definition of down. The word "unplanned" is doing the work. Planned maintenance is not an incident, but it is downtime. The honest approach is to record both and report them separately rather than quietly leaving one out. Decide whether a slow service counts and whether a single-user problem counts. Decide whether two failures on the same afternoon are one incident or two. The decision matters less than writing it down and sticking to it.


The third is which hours count. Availability measured over every hour of the year and availability measured over business hours only are different measures with different uses. Neither is wrong. Comparing one with the other is. Honest Availability Math covers the arithmetic and the traps. The benchmarking rule is that your peer's hours must match yours or the comparison is over before it starts.


If you also want to compare how quickly incidents were noticed and fixed, the clocks need the same treatment. How to Measure MTTD and MTTR Honestly explains where those clocks start and stop. A peer comparison of response times is only fair if both of you start them in the same place.


Tip: record planned downtime in the same log as unplanned, with a flag that says which it was. You can always exclude it from a figure later. You cannot add it back if it was never written down. The day someone asks "how much of that was planned?" you want an answer that is not a shrug.


Measure the same way for a year


A benchmark needs a full year of measurement under one set of definitions before it can be trusted, because a network has seasons. Think of the year-end close or the summer when half the building is empty. Or think of the term start if you serve a school, or the quarter when the audit lands. Each one changes what "normal" looks like for a few weeks. A quarter of data tells you about that quarter. A year tells you about the organisation.


The year also protects you from your own improvements. Suppose you changed the definition of down in the spring, tightened a threshold in the summer, and added forty devices in the autumn. In that case, a comparison of this year with last year is measuring three changes and one network. Fix the definitions, let the monitoring run under them for twelve months, and only then draw the first line. It feels slow. It is faster than arguing for a year about a number nobody trusts.


Where does the record live? Two places, ideally. The monitoring system keeps the availability history per service, which is the source for downtime minutes. A simple incident log is the source for counts. It has one row per incident with a start, an end, the services affected, and a planned-or-unplanned flag. If you already write post-incident reviews, the log is the index of them; Post-Incident Action Items That Get Done covers what goes in each entry. If you do not, a spreadsheet with five columns is enough to start, and it has been enough at every small organisation I have worked with.


Your own history is the primary benchmark


Your own previous year, measured the same way, is the most honest comparison you will ever have. So it goes first, and it is the one you put in the report. Definitions match by construction. Nobody can argue that the reference counted differently. The question it answers is "are we getting better or worse?" That is the question most managers are really asking when they ask whether you are normal.


Here is a worked example. Harbourview Clinic is fictional, and so are all the numbers, but the shape of the table is the one you want. The clinic agreed its definitions, ran under them for two full years, and pulled these figures from the availability report and the incident log.


Measure (Harbourview Clinic, fictional)Previous twelve monthsLatest twelve monthsChange
Monitored devices (from discovery inventory)180205+25
Incidents (unplanned, agreed definition)1416+2
Incidents per 100 devices7.87.8flat
Critical services counted66none
Critical-service downtime, minutes (all hours, planned excluded)410265−145
Downtime minutes per critical service6844−24
Longest single incident, minutes19075−115
Planned downtime, minutes (reported separately)600720+120

Read the table the way a manager would, and then the way an administrator should. The manager sees the incident count went up by two and asks why. The administrator sees the device count went up by twenty-five, so the rate per hundred devices is flat. And the downtime that matters fell by more than a third, with the worst incident less than half as long. That is a network that is getting better at recovering, and a team that has grown its estate without growing its trouble. The raw count on its own would have told the opposite story. (It usually does. Raw counts have a talent for it.)


Quarter-over-quarter comparisons work the same way with more noise. A single quarter can be wrecked by one bad afternoon. So compare the latest quarter against the same quarter last year as well as against the quarter before it. And look for direction across four or more quarters rather than reacting to one. If your incident log is a comma-separated export, a few lines of script will count incidents per quarter for you:


Import-Csv .\incidents.csv |
  Where-Object { $_.Planned -eq "no" } |
  Group-Object { $d = Get-Date $_.Started; "$($d.Year)-Q$([math]::Ceiling($d.Month / 3))" } |
  Sort-Object Name |
  Select-Object Name, Count

The Where-Object line is the definitions step, in code. Remove it and you are counting planned work as incidents. That is exactly the mistake the war story at the end of this guide is about.


Building a peer group and how to ask


A peer group is three to six organisations that resemble yours in size, sector, and operating hours. They are willing to compare notes with the definitions agreed first. Three is the minimum for the comparison to mean anything beyond one other person's opinion. Six is about the most one administrator can keep a relationship with. Bigger groups become surveys, and surveys are where definitions go to die.


Resemblance matters more than friendliness. A peer for a clinic with three sites and a night shift is another clinic, or a care provider. Or it is anything else with sites and a night shift. It is not the design agency down the road with one office and lovely coffee. Match on device count within a factor of two or so. Match on whether you run around the clock, on the number of sites, and on whether you have a similar mix of things that break. That mix might be a lot of printers, a lot of wireless, or a lot of remote users. Sector is a proxy for all of that; it is the hours and the shape that actually matter.


Where do you find them? Try local user groups and professional associations for your sector. Try the administrator at the organisation you share a building or a supplier with. Or try the person you met at a training course who had the same problems. Most administrators are glad to be asked, because they have the same question and no one to ask either.


The ask itself is the delicate part, and the way to make it safe is to lead with definitions rather than numbers. Nobody wants to be the first to reveal a figure that might look bad. Everybody is happy to compare definitions, and once the definitions are agreed the figures follow naturally. This is the email I send, with the brackets filled in:


Subject: Comparing outage figures - definitions first

Hello [name],

We are trying to compare our availability and incident figures with a
small number of organisations of similar size, and [organisation] came
to mind because [similar site count / sector / operating hours].

Before either of us shares a number, I would like to agree what we are
counting, so the comparison is fair. Our definitions are:

- Down: [a monitored critical service failing its check for more than
  N minutes, confirmed by a person]
- Incident: [an unplanned event that caused a service to be down by the
  definition above; planned maintenance recorded separately]
- Hours counted: [all hours / business hours only]
- Period: [the latest twelve months]
- Size: [device count from the monitoring inventory; number of sites;
  team size]

If your definitions differ, I would rather know that than adjust the
numbers afterwards. Happy to share ours first. Nothing will be
attributed outside our two teams without asking.

Thanks,
[name, role, organisation]

Offer to share first, always. It costs you nothing and signals that you are not fishing. The peer who receives your number with its definitions attached learns what a fair comparison looks like before they have to produce one.


Normalising by size


Normalising means dividing a raw count by a measure of size so that organisations of different sizes can be compared. Sixteen incidents at a two-hundred-device clinic and sixteen at a two-thousand-device college are not the same sixteen. Two normalisations cover most needs, and both use numbers you already have.


The first is incidents per hundred devices: the unplanned incident count, divided by the monitored device count, multiplied by a hundred. Use the device count from your monitoring tool's own inventory, because that is the count a peer can produce the same way. The second is downtime minutes per critical service: total unplanned downtime across your critical services, divided by the number of critical services. This one is fairer than a percentage because it does not reward an organisation for having a long list of services that never break.


Incidents per 100 devices  = (unplanned incidents / monitored devices) x 100
Minutes per critical service = critical-service downtime minutes / critical services

Spreadsheet, with incidents in B2 and devices in C2:
=B2/C2*100
Spreadsheet, with downtime minutes in B3 and service count in C3:
=B3/C3

Both normalisations are lenses, not laws. A device count includes printers and desk phones, which rarely cause incidents. So a site with a big fleet of both will look artificially calm per hundred devices. When a peer's normalised number is very different from yours, the first question is what is in their count. The second is what is in their list of critical services. A normalisation that has not been questioned has not been understood.


Reading the answer honestly


A lower number with a looser definition is not a better number. This is the sentence to keep in front of you when the peer figures arrive. Suppose your peer's downtime is half yours. And suppose their definition of down requires a person to confirm the outage before the clock starts. In that case, their number is smaller because their clock is shorter, not because their network is calmer. Go back to the definitions before you feel anything about the result.


The two emotions to watch for are envy and smugness, and both make you stupid in the same way. Envy makes you assume the peer is doing something clever that you should copy, when they may simply have fewer sites. Smugness makes you assume your lower number is skill, when it may be a mild year. The useful response to a difference is a question: what happened in your worst incident, what is in your critical list, how many sites, what hours? The differences almost always turn out to be structural, and structural differences are information, not verdicts.


One more caution about a single year. A network that had one long outage last year and none this year has not necessarily improved; it may have got lucky. The measures that show real improvement are the ones that move steadily across several periods. Examples are the longest incident getting shorter and the count per hundred devices holding flat while the estate grows. Another is the planned minutes rising as the unplanned minutes fall. One good year is weather. Three is climate.


What to do with the result


The result of a fair comparison earns exactly two places to live. One is a sentence in the monthly report. The other, if it points at something, is an item on the roadmap. The sentence is short and modest, in the style the rest of this pillar recommends: "Unplanned downtime on critical services fell from 410 to 265 minutes year on year under the same definition; two peers with matched definitions reported figures in a similar range." Definitions stated, reference stated, no claim it cannot support.


The roadmap item appears when the comparison shows a gap that is yours to close. If every peer's longest incident is shorter than yours, the question is why yours run long. In that case, the answer usually points at detection or at a runbook that does not exist. That is a change you can plan and schedule, and A Twelve-Month Network Monitoring Roadmap is where it goes. A comparison that produces neither a sentence nor an item was either uninteresting or not fair, and it is worth working out which.


Do not build a dashboard for it. A benchmark changes once a year, and a dashboard that changes once a year is furniture.


A war story: two definitions of "outage"


Bellweather Logistics and Acme Fasteners were similar in size and had administrators who knew each other from a course. So they compared outage counts over lunch. Bellweather reported twenty-two for the year and Acme reported nine. Bellweather's administrator went back to the office convinced his network was in trouble. Two weeks of worry later, he asked Acme for the list. He found that Acme counted only outages that had produced a ticket from a user, during business hours. Acme excluded anything that had been scheduled. Bellweather had been counting every alert-confirmed failure on any monitored device, at any hour. That included a planned weekend of switch replacement that alone accounted for six entries. Under Acme's definition, Bellweather had seven. Under Bellweather's, Acme's administrator declined to estimate, which was the most honest thing said all month.


Neither definition was wrong. Comparing them was. The definitions-first email at the top of this guide exists because of lunches like that one.


The checklist


Before you compare an uptime or incident figure with anyone, including yourself last year, work through this list.


  • "Down" is defined in one written sentence, including which services and devices count.
  • "Incident" is defined, planned work is recorded separately with a flag, and the two are never silently merged.
  • The hours counted (all hours or business hours) are stated on every figure.
  • The definitions have been stable for a full twelve months before the first comparison.
  • The primary comparison is against your own previous period, measured the same way.
  • Peers were chosen for similar size, hours, sites, and device mix, and the definitions were exchanged before any number.
  • Counts are normalised per hundred devices and per critical service, using the monitoring inventory as the device count.
  • A difference has been questioned (what is in their count?) before it has been felt.
  • The result is one sentence in the monthly report and, if warranted, one item on the roadmap.

Where to go next


Uptime and incidents are one of the three comparisons that Am I Normal? What Benchmarking Can and Cannot Tell You recommends. The other two, cost and staffing, are worked in Benchmarking Monitoring Spend and Staffing Without Inventing Numbers. For the parts of a monitoring practice that no count captures, Healthy Signs describes what good looks like in words. And when the year-on-year sentence needs to survive a budget meeting, Using Benchmarks in Budget Conversations shows how to present it. All of them sit under Am I Normal? Benchmarking Your Network and Monitoring Practice. If what you actually wanted was a sense of whether a link's latency or loss figures are reasonable, that is a different question. It is answered in Healthy Network Metric Ranges.


Put this into practice with SysPeep Monitor

The year-on-year table in this guide comes straight out of an availability report, provided the monitoring has been keeping one under a steady definition. SysPeep Monitor records availability and alert history per service and shows them as reports in one web console. Its auto-discovery keeps the device count you normalise by honest. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep