Home › Topics › Am I Normal? Benchmarking Your Network and Monitoring Practice › What Benchmarking Can Tell You

Am I Normal? What Benchmarking Can and Cannot Tell You


Sooner or later you will look at your outage count, your alert inbox, or your licence renewal and wonder whether other people have it this bad. It is a fair question. You have no colleagues at the next desk running a different network, so you have nothing to compare against. And the only people who volunteer comparisons are vendors with a product to sell. The question "am I normal?" is the beginning of good judgement, not a sign of insecurity.


The trouble is that most of the comparisons on offer are worthless, and a few are worse than worthless because they look precise. This guide gives you a way to compare honestly. By the end you will know the three kinds of benchmark and how far to trust each. You will know the four traps that make a comparison lie and the three comparisons that are actually worth making. You will also know a one-page worksheet that forces every comparison to declare its definitions before it is allowed to mean anything.


I should say up front what you will not find here: a table of "typical" figures. I do not have one, nobody honest does, and the guides in this pillar are about how to benchmark, not what the answer is. The answer lives in your own data. This guide shows you how to get it out.


In this guide

  • Why "am I normal?" is a reasonable question
  • What a benchmark actually is
  • The three kinds, ranked by trust
  • The four traps
  • What a benchmark can support, and what it cannot
  • The three comparisons worth making
  • The benchmark worksheet
  • A war story and a checklist
  • Where to go next

Why "am I normal?" is a reasonable question


The question comes up because you are making decisions with no reference point. You had four outages last quarter; is that a lot? Your two-person team gets sixty alerts a week; is that a lot? The renewal quote went up; is that a lot? Without something to compare against, every one of these is a guess, and guesses in a budget meeting have a short life expectancy.


Asking the question is also how you find out whether a problem is yours or everyone's. If every organisation your size struggles with the same thing, the fix is probably structural and you can stop blaming yourself. If nobody else has the problem, you have found something specific to your network, and specific problems are the ones you can actually fix. Either answer is useful. The only useless outcome is a comparison that turns out to have been measuring something else, which brings us to what a benchmark is.


What a benchmark actually is


A benchmark is a comparison of one of your measurements against a reference value. The reference can be your own past, a similar organisation, or a figure somebody published. That is the whole definition, and the important word is reference. A benchmark is only as good as the thing on the other side of the comparison. Your own number might be perfectly measured, and the comparison can still be nonsense if the reference was measured differently.


The analogy I use is a bathroom scale. Weighing yourself is a measurement. Comparing this morning's weight to last month's is a benchmark against your own past. Comparing it to a friend's is a peer benchmark. It only means something if you are roughly the same height and you both weighed yourselves in the same state of dress. Comparing it to a number in a magazine is a published benchmark, and you have no idea who was weighed or how. Same scale, three comparisons, three very different levels of trust.


Every benchmark therefore has four parts, and you need all four before it can mean anything. The parts are your value, your definition of what you measured, the reference value, and the reference's definition. Most published comparisons give you one of the four. (The value. Always the value.)


The three kinds, ranked by trust


Internal: you against your own past

An internal benchmark compares this period's number with an earlier period's number from the same network, measured the same way. It is the most honest comparison available to you, because the definitions match by construction. You use the same tool, the same checks, the same idea of what "down" means. If your critical-service downtime this year is lower than last year, that is a real improvement. In that case, nobody can argue it away by questioning the reference.


Its limit is that it cannot tell you whether your starting point was good. You can improve on a bad position for years and still be in a bad position. Internal benchmarks answer "are we getting better?" with real authority and "are we fine?" not at all.


Peer: you against similar organisations

A peer benchmark compares your number with the same number from an organisation of similar size, sector, and operating hours. It is useful exactly to the extent that the definitions match, and the definitions almost never match unless you agree them first. Two administrators who both say "we had six outages" have told each other nothing until they have agreed what an outage is. They must also agree whether planned work counts and whether a slow service counts as down. Benchmarking Uptime and Incident Counts Fairly shows how to build a peer group and how to have that conversation.


Peer benchmarks are worth the effort because they answer the question internal benchmarks cannot: whether your position is unusual. Three or four honest peers with matched definitions will tell you more than any survey.


Published: surveys and vendor reports

A published benchmark is a figure from a survey, an analyst report, or a vendor's slide deck. It is the least trustworthy of the three. That is not because the people who produce them are dishonest. It is because you almost never know who answered or how many answered. You almost never know how "down" or "incident" or "device" was defined, or whether the number was rounded to look good. Without those four facts, a published figure is a decoration. With them, it can be useful, and the honest reports do include them, usually in an appendix nobody reads.


The rule I follow: a published figure is allowed into a conversation only with its source and its definition attached. If I cannot find either, I do not quote it, and I say so when someone else does.


The four traps


Mismatched definitions

The first trap is comparing two numbers that were measured differently. Availability is the classic case: one organisation counts every minute of the day, another counts business hours only, and a third excludes planned maintenance. All three can report the same percentage while describing completely different years. The library already explains how to fix this in Defining "Down" and Honest Availability Math. So the short version here is: definitions first, numbers second, always.


Survivor bias

Survivor bias is the distortion you get when the people with the worst results do not show up in the data. Surveys are answered by administrators who had time to answer a survey. The administrator who spent the month rebuilding a domain controller did not fill in the form. So the reported "typical" experience is better than the real one. The same thing happens at user groups and conferences: the people presenting are the ones with something to be proud of. The reference you are comparing against is not everyone. It is the people who were doing well enough to say so.


Size and sector mismatch

The third trap is comparing against organisations that do not resemble you. Consider a hospital that runs around the clock, a retailer with two hundred small sites, and a design studio with one office and a fast internet line. They have different outage patterns, different alert loads, and different staffing shapes. None of them is wrong. A figure aggregated across all three describes nobody. When you look at a reference, ask what size and what kind of organisation produced it. If the answer is "a mix", treat the number as scenery.


The number that was invented for a slide

The fourth trap is the figure that exists because a slide needed one. Somebody, somewhere, needed a comparison to make a point, estimated a value, and put it on a chart. The chart got copied. The estimate lost its footnote on the second copy and became a fact on the third. You can recognise these numbers by their tidiness and by the total absence of a method. A benchmark with one decimal place and no source has usually travelled a long way from wherever it was born. And it was not born in a measurement.


Tip: when you cannot find a source for a figure, do not argue about it. Ask, politely and in writing, "which report is that from, and how did they define the measure?" You will either get a source you can evaluate or a silence that answers the question.


What a benchmark can support, and what it cannot


A benchmark supports a claim in proportion to how well the definitions match, and no benchmark supports every claim. The table below is the one I keep in my head when somebody says "the benchmark shows". It is worth reading the right-hand column twice, because those are the sentences that get people into trouble.


Kind of benchmarkCan honestly supportCannot honestly support
Internal (own past, same method)"We improved." "We got worse." "This changed when we changed X.""We are fine." "We are better than others."
Peer (matched definitions)"We are not unusual." "This is worth a look." "Our shape is different because of Y.""We need exactly this budget." "Their number proves our approach is right."
Published (source and definition known)"Organisations that answered this survey reported this range.""The industry average is." "We should match this figure."
Published (source or definition unknown)Nothing, until the source and definition are found.Any claim.

Notice what the "can support" column has in common: every claim is modest. "We are not unusual" is a genuinely useful thing to be able to say in a meeting. A peer benchmark lets you say it with a straight face. "We are fine" is not the same claim, because fine depends on what the business needs, and no comparison with outsiders can tell you that. The internal targets your own organisation has agreed, the sort described under Internal SLAs, are what "fine" is measured against.


The budget claim deserves its own sentence. A benchmark can show that your spend is in the same neighbourhood as your peers. It cannot show that you need a particular amount. That is a question about what the money would buy and what happens without it. The evidence for that lives in a budget case, not a comparison. Using Benchmarks in Budget Conversations covers how the two fit together.


The three comparisons worth making


You could benchmark a hundred things, and a hundred things is how a benchmarking exercise dies. Three comparisons cover most of what a manager will ask and most of what you need to know. All three start with your own history, which you already have. The reference for each one is you, last year or last quarter. Peers come later, once the internal number is solid.


1. Availability of your top services, against your own last year

Pick the handful of services the business would notice within an hour (the ones that would generate the phone calls). Compare their downtime this year with last year, measured the same way. Work in minutes of downtime rather than percentages, because minutes are honest and percentages hide the difference between one long outage and thirty short ones. If your monitoring keeps an availability report per service, this is a report you read, not a calculation you do.


The spreadsheet version is one cell. With last year's minutes in B2 and this year's in C2:


=(C2-B2)/B2

A negative result means less downtime this year. That is the whole comparison. It is more defensible than any figure anyone will show you on a slide. That is because both sides were measured by the same tool with the same definition of down.


2. Alerts per person per week, against your own last quarter

Alerts per person per week is the number of alert notifications your monitoring sent in a week, divided by the number of people who received them. It is the fastest honest measure of whether your alerting is getting noisier or quieter. Count from the alert history, not from memory, and count for a whole quarter so one bad week does not set the tone. If your monitoring can export alert history to a comma-separated file, a short script groups it by week:


Import-Csv .\alerts.csv |
  Group-Object { Get-Date $_.Time -UFormat "%Y-W%V" } |
  Sort-Object Name |
  Select-Object Name, Count

Divide each week's count by the number of people on the notification list and you have the measure. Compare this quarter's median week with last quarter's. Rising is worth investigating; Diagnosing Alert Fatigue explains what to look for once you know the number is going the wrong way. Falling is good news, provided nobody achieved it by muting the channel.


3. Cost per monitored device, against your own last renewal

Cost per monitored device is your total monitoring cost for the year divided by the number of devices you actually monitor. The total includes the licence, the hardware or virtual machine it runs on, and an honest fraction of the time people spend keeping it working. Divide by the real monitored count from the tool's own inventory, not the number on the licence or the number you remember. Then compare with the same calculation from the last renewal. Benchmarking Monitoring Spend and Staffing Without Inventing Numbers works through this one in full, hidden costs included.


The benchmark worksheet


The worksheet below is the discipline that makes all of this safe. One row per comparison, and the row is not complete until every column is filled. The column that matters most is "definitions match?" And the honest answer for a published figure is usually "unknown". That is fine, because "unknown" is a fact you can act on. The last column forces you to write down what the comparison actually supports, in the modest language from the table above.


MeasureYour definitionYour valueReferenceReference definitionDefinitions match?What it supports
Critical-service downtime (minutes, twelve months)[services counted, hours counted, planned work in or out][minutes][own previous twelve months / peer / published][as stated by the reference, or "unknown"][yes / partly / no / unknown][one sentence]
Alerts per person per week (quarter median)[notification channels counted, people counted][number][own previous quarter / peer / published][as stated, or "unknown"][yes / partly / no / unknown][one sentence]
Cost per monitored device (annual)[licence + hardware + time fraction; device count source][figure][own previous renewal / peer / published][as stated, or "unknown"][yes / partly / no / unknown][one sentence]
[additional measure]

Keep the completed worksheet with the monthly report rather than in a personal folder. That is because the next person to ask "are we normal?" will be a manager. The worksheet is the answer. It also stops the same comparison being rebuilt from scratch every budget cycle. I have watched that happen at organisations that had done the work twice already and filed it somewhere sensible both times.


A war story: the survey figure that lasted ten minutes


Meridian Parts ran a modest network with two administrators. At a budget meeting, a director produced a figure from a supplier's slide. It showed that organisations "like Meridian" ran monitoring on far fewer staff hours. The senior administrator asked for the definition of a staff hour and the size of the organisations surveyed. The slide had neither. She then showed the fortnight time log the team had kept. The hours spent on monitoring, on the ticket queue, and on the printer fleet were separated out. The monitoring hours turned out to be a small fraction of the total the slide had implied. And the rest of the time was being spent on work the slide had never measured. The figure was withdrawn before the coffee arrived, and the director asked for the time log to be included in the next monthly report.


The lesson is not that the director was foolish. She asked a reasonable question with the only evidence she had been given. The lesson is that an administrator who arrives with definitions and a measured internal number wins every comparison against a figure that has neither. She does so without raising her voice.


The checklist


Run through this before you quote, accept, or act on any comparison.


  • I know which of the three kinds this benchmark is: internal, peer, or published.
  • I have written down my own definition of the measure, including what is excluded.
  • I have the reference's definition, or I have written "unknown" and treated the comparison accordingly.
  • I have checked the reference for survivor bias: who answered, and who would not have.
  • I have checked that the reference organisations resemble mine in size, sector, and hours.
  • I have a source for any published figure, and I can name it if asked.
  • The claim I am making is one from the "can support" column, not the other one.
  • The worksheet row is complete and filed with the monthly report.

Where to go next


With the three comparisons and the worksheet in hand, the rest of this pillar takes each one further. Benchmarking Uptime and Incident Counts Fairly covers the availability comparison and how to find peers whose definitions match yours. Benchmarking Monitoring Spend and Staffing Without Inventing Numbers works the cost and staffing ratios honestly. Healthy Signs describes a good monitoring practice in behaviours rather than numbers, for the questions no comparison can answer. And when a comparison has to survive a finance director, Using Benchmarks in Budget Conversations shows how to present it. All of them live under Am I Normal? Benchmarking Your Network and Monitoring Practice. The wider question of turning monitoring data into something management can use is covered under Translating Monitoring Data for Management.


Put this into practice with SysPeep Monitor

The honest half of every benchmark is your own history, measured the same way twice. SysPeep Monitor keeps availability and alert history as reports in one web console. Its auto-discovery keeps the monitored device count honest. So the per-device figure is divided by what is really there. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep