The Coverage Review: Auditing What Your Monitoring Actually Watches
The day you finished setting up monitoring was the last day it covered everything. Since then the network has moved on. A new server stood up in an afternoon for a project that could not wait. Another switch appeared in the branch office. Someone racked a storage box "temporarily," and a whole network segment materialized for the security cameras. None of these things asked permission, and none of them told your monitoring system that they exist.
This is the quiet truth about monitoring: it rarely rots by breaking. It rots by omission. Every system deployed since your last audit is invisible to your monitoring until the day it breaks. Then it introduces itself with an outage, a confused user, and the most expensive sentence in IT: "wait, weren't we watching that?" I have said that sentence myself. It does not get cheaper with practice.
The fix is a habit, not a product: the coverage review, a recurring, systematic comparison between what exists and what is watched. This guide walks you through the five steps of a review, and the depth question most audits forget. It covers how often to run one and the number worth tracking over time. It also gives a worked example of a small shop's first review. The first one takes an afternoon. Every one after that gets shorter.
In this guide
- Why monitoring rots by omission
- The five steps of a coverage review
- Deeper than "is it monitored?"
- Cadence: the ritual and its triggers
- Coverage as a number you can track
- A worked first review
- Where to go next
Why monitoring rots by omission
Your monitoring configuration only changes when someone changes it. Your network changes constantly, whether anyone updates the monitoring or not. That mismatch is the whole problem. Servers get deployed. Virtual machines get cloned. "Temporary" equipment becomes permanent the moment it works, and companies acquire other companies whose networks arrive like surprise luggage. Every change silently widens the gap between reality and what your dashboards show. Nothing about day-to-day operations ever closes it, because a system with no checks also produces no reminders that it has no checks.
Think of it like smoke detectors. When you moved in, every room had one. Then you built an extension, converted the garage, and put up a shed. Nobody thought about detectors, because building things and protecting things are different activities done in different moods. The rooms most likely to lack a detector are exactly the newest ones, which are also the ones full of fresh wiring.
The rot runs in both directions, and the second direction is easy to forget. Besides real systems with no checks, you accumulate orphan checks: checks still pointed at servers that were retired, renamed, or re-addressed. Orphans are not harmless leftovers. They alert about machines that no longer matter, which trains you to ignore alerts. They get muted, which buries the evidence. And they sit on dashboards padding your device counts, making coverage look better than it is. A good review hunts both species.
Tollgate Foods ran its first coverage review about eighteen months after the monitoring went in. The diff came back with forty-one devices nobody could name. Most turned out to be weighbridge controllers, cold-store sensors, and a rack of boxes. The packaging line's vendor had installed those boxes on a segment IT had never been told about. The one that mattered was fin-app-01, the new finance system, which had gone live six months earlier with a launch party and no checks. Its nightly interface job had been failing on and off for seven weeks. Finance had been quietly re-keying the failed records by hand, because nobody had told them it was a fault. The review log for that quarter reads "41 unknown, 1 genuine miss, 1 difficult conversation."
What a coverage review actually is
A coverage review is a recurring audit with one job. Produce a complete list of what exists. Produce a complete list of what is monitored. Compare them in both directions. Then deal with every difference on purpose — either by adding monitoring or by writing down why not. That is the entire method. It needs no special software, it works with any monitoring system ever made, and its results fit on a page of notes.
If it helps, think of it as a census. A census does not estimate the population from memory; it goes door to door and counts. Your review does the same for the network. And like a census, its value comes from being repeated on a schedule, not performed once in a burst of enthusiasm and then fondly remembered.
Step 1: Pull the inventory — the "what exists" list
Start with reality, not memory. The backbone of the inventory is a discovery scan. Your monitoring or discovery tool sweeps your address ranges and lists everything that answers — servers, switches, printers, mystery boxes and all. A scan finds what is actually on the wire, which is reliably more than anyone remembers buying. If you do not have a discovery habit yet, the guides under Discovery & Inventory cover how to build one.
Then add the things a device scan cannot see, because not everything that "exists" has an IP address of its own. TLS certificates (the files behind the browser padlock) expire. Backup jobs succeed or fail. The internet link has latency and loss. Cloud services have reachability. Business services like "email" span several boxes and deserve a line of their own. Your strategy documents are the source for these — the services and priorities you wrote down when defining your monitoring goals and scope. A review that only counts devices will happily miss the expiring certificate that takes down the VPN portal. That is the sign-in page remote staff use to reach the office network.
Practically: merge both sources into one plain-text list, one target per line. And do not worry that the inventory itself is imperfect — it is, because inventories rot too. The scan grounds it in what is really on the network. Every review you run leaves the inventory a little more honest than it found it.
Step 2: Pull the monitored list — the "what is watched" list
Now export the target of every check from your monitoring system. Most tools can export the device or check list to a CSV file (comma-separated values, the plain-text form a spreadsheet opens). In the worst case you walk the device tree and copy names out by hand, which is tedious exactly once. You want targets, not checks: a server with five checks on it should appear on this list one time.
Before comparing anything, normalize both lists to the same naming form. Pick one canonical style — the lowercase short hostname, or the IP address — and convert both lists to it. That way, each real-world thing has exactly one spelling. I once skipped this step, ran the diff with three spellings of every server, and briefly concluded the network had tripled.
Tip: normalization is where first reviews go sideways. To a comparison tool, FILESRV01, filesrv01.corp.example and 10.0.2.11 are three different strangers, and your diff fills up with false gaps. Ten minutes of cleanup before the compare saves an hour of confusion after it.
Step 3: Diff both directions
With two clean lists, the comparison itself is almost anticlimactic. On Windows, PowerShell's Compare-Object does it in one line:
PS> Compare-Object (Get-Content .\inventory.txt) (Get-Content .\monitored.txt) InputObject SideIndicator ----------- ------------- nas-02 <= cam-sw-01 <= hvac-ctrl <= oldmail-01 =>
Read the arrows like this: <= means the target exists in the inventory but nothing watches it. That is a coverage gap, the thing this whole exercise hunts. => means something is monitored that the inventory has never heard of — usually an orphan check pointing at a machine that was retired or renamed. Anything not shown matched on both sides, which is the boring majority you want.
Treat the two directions with equal respect. Gaps are future surprises; orphans are present-tense noise. A check on a decommissioned server either alerts forever until someone mutes it, or sits there green, quietly inflating your device count and your confidence. Both cost you trust in the tool, and trust is the only currency monitoring has.
Step 4: Classify each gap — the honest ledger
Not every gap is a mistake, and pretending otherwise makes reviews miserable. Sort every difference into one of three piles. Genuine miss: it should be monitored and is not — the new server, the forgotten storage box. Deliberate exclusion: you looked at it and chose not to watch it — the test lab, the printers you have made your peace with. Check these against your scope document; if an exclusion is not written there, it is not deliberate yet, merely unexamined. Unknown: you do not recognize the target at all. Investigate these before anything else, because an unrecognized device on your network is a finding in its own right. That is true whether it turns out to be a vendor's controller, a colleague's experiment, or something you genuinely need to meet.
This classification step is the honest ledger at the heart of the method. A gap with a written reason next to it is a decision. A gap without one is a future outage plus a future argument about whose fault it was. The temptation during a review is to wave things through as "excluded" so the numbers look tidy. Resist it — you are the main person the tidy numbers will mislead.
Step 5: Close it or record it
Every line in the diff now gets exactly one of three treatments. Genuine misses get checks. Batch the work, start with the essential checks rather than aiming for instant perfection, and give each new check sensible starting thresholds you will tune later. Orphans get deleted — deleted, not disabled. A disabled check is clutter with a pension. The check pads every list you scroll through for years, and eventually it confuses someone into re-enabling it. If you want a record of the removal, put a line in the review log instead.
Exclusions get written down. A one-line ledger entry is enough: the target, the reason, who decided, and a revisit date. "Printers — excluded; alerting cannot fix them; revisit next year" is a perfectly professional entry. It will save a future admin an afternoon of wondering whether the gap was a choice or a blunder.
Finally, log the review itself: the date, how many gaps you found, how many you closed, how many you excluded. Three lines in a text file. (Yes, a text file. It will outlive the tool.) This small log turns coverage into a trend you can actually see — more on that shortly.
Deeper than "is it monitored?"
Here is the failure mode the basic diff cannot catch: a server counts as "monitored" because something pings it. A ping proves the machine answers the network — and nothing else. Meanwhile its disk fills, and its backup job has been failing for a month. Its certificate creeps toward expiry, and the one service users care about is stopped. All invisible, all officially "monitored." Existence coverage and depth coverage are different questions, and the second one is where the embarrassing outages hide.
The cure is a minimum-depth standard for each class of device. This is a short list of what "watched" must mean for each kind of thing you run. In the table, UPS is the uninterruptible power supply (the battery that carries a rack through a power cut). The WAN port is the side of the firewall that faces the internet. Something like this:
| Device class | "Monitored" means at least |
|---|---|
| Server | Ping, CPU, memory, disk space, key service status, backup result |
| Switch | Ping, uplink port status, port error counters |
| Firewall / router | Ping, CPU, memory, WAN port status |
| Storage box / NAS | Ping, disk usage, share reachability |
| UPS | On-battery status, battery health, runtime, temperature if available |
| Wi-Fi access point | Up/down status, client count |
| Internet link | Reachability, latency, packet loss |
| Public or internal service | A real request succeeds, certificate days remaining |
You do not need to invent these lists from scratch. The starting checks and thresholds in the first ten checks guide map almost directly onto the table above. Then, during each review, pick one device class and spot-check a handful of its members against the standard. Rotating one class per quarter keeps the effort small and still visits everything within a year.
Cadence: the ritual and its triggers
For most small and mid-size networks, quarterly is the sweet spot: frequent enough that the diff stays short, rare enough to respect your calendar. Book it as a recurring appointment with yourself, because "when things calm down" is not a date and never has been.
On top of the ritual, three events should trigger an extra review regardless of the calendar. After any significant deployment or project, because new systems arrive in batches and so do gaps. After an acquisition, merger, or office move, because other people's networks come with other people's undocumented habits. And after any incident where the honest answer to "how did we not know?" was "we weren't watching it." That incident just performed a tiny, very expensive coverage review on your behalf. The least you can do is formalize the finding and go looking for its siblings while the sting is fresh.
Each review also shrinks the next one if you pair it with an intake rule. The rule is that no system goes live without monitoring. That is a standard line on the deployment checklist. With intake in place, reviews catch the occasional drip instead of a quarterly flood. The guides under Monitoring Scope cover intake rules and scope decisions in more depth.
Coverage as a number you can track
Once gaps are classified honestly, you can compute a genuinely useful number: coverage — monitored in-scope targets divided by all in-scope targets. If the inventory holds 68 things you have decided matter and 61 of them have checks, coverage is about 90%. Write the number in the review log each quarter. It is concrete, it moves when you do the work, and it is one of the few monitoring numbers a non-technical manager understands on first contact.
Now the honest caveats, because this number can be gamed in three classic ways. You can shrink the denominator by excluding whatever is inconvenient — which is why exclusions require written reasons. You can count a lonely ping as "covered" — which is why the depth spot-check exists. And you can let the inventory go stale, at which point coverage drifts toward a proud, meaningless 100%. In fact, treat a perfect score with suspicion: it usually measures the age of your inventory rather than the quality of your monitoring. The number is a trend, not a trophy.
A good companion metric is gaps found per review. If your intake rule is working, it falls over time. If it climbs, systems are leaking past intake, and the review has just told you exactly where your process is torn.
A worked first review
Here is how this plays out in a small shop: one admin, a colleague who helps, and a network everyone "knows" has about sixty devices. The discovery scan returns 71. The scope document contributes nine non-device targets: four certificates, three backup jobs, the internet link, and reachability of the payroll cloud service. Inventory: 80 targets. The monitoring export lists 67 targets. The diff reports 18 inventory lines with no checks, and 5 checks pointing at things the inventory has never heard of.
Three findings tell the story. The first is nas-02, a NAS (network-attached storage — a shared disk box on the network). It was racked during the camera project and never handed to monitoring. It is quietly filling with camera footage. On its current curve the disk fills within months, recording stops, and the failure is discovered the day someone urgently needs yesterday's video. The second is oldmail-01, retired long ago but still monitored. Its "server down" alert has been emailed and deleted every week since, teaching everyone that deleting alerts is normal. The third is the camera VLAN itself. A VLAN is a virtual LAN — a logically separate network segment. The camera VLAN hangs off one switch, cam-sw-01, that nobody ever added. So an entire segment has zero visibility. The diff also offers a bonus mystery, hvac-ctrl, which facilities confirms is the heating and air-conditioning (HVAC) vendor's controller. Of course it is.
| Finding | Classification | Action |
|---|---|---|
| nas-02 — forgotten camera storage | Genuine miss | Add ping, disk usage, share reachability |
| cam-sw-01 and the camera VLAN | Genuine miss | Add ping, uplink status, port error counters |
| VPN portal certificate | Genuine miss (depth) | Add expiry check: warn 30 days, alert at 14 |
| Two branch access points | Genuine miss | Add up/down and client-count checks |
| oldmail-01 — retired, still paging | Orphan check | Delete check and alert rule; note in review log |
| Four other stale checks | Orphan checks | Delete three; one was a renamed server, re-pointed |
| hvac-ctrl — mystery device | Unknown, investigated | Vendor HVAC controller; ping only, noted as vendor-owned |
| Eight printers | Deliberate exclusion | Ledger entry with reason and a revisit date |
| Four lab VMs | Deliberate exclusion | Ledger entry: out of scope per strategy doc |
Total cost: about two hours of list work, plus an afternoon of adding checks. Total value: at least three future outages deleted from the timeline before they happened. Five sources of noise were removed, and twelve exclusions were converted from vague assumptions into written decisions. There is also an inventory that finally matches the wire. The next quarter's review took forty minutes.
Where to go next
Coverage reviews are compound interest. The first one is an archaeology dig. The second is a tidy-up. By the fourth, with the intake rule doing its quiet work, it is a coffee-break ritual that keeps a slowly changing map honest. By then, nothing on the network gets to introduce itself with an outage any more. Very few habits in IT pay back this reliably for this little effort.
One boundary worth knowing: this review asks whether the things on your network are watched. Its sibling exercise asks whether you can see the traffic between them — the segments, paths and conversations your tools are blind to. That is a different hunt with its own method, covered in the blind spot audit. Run both and you know, rather than hope, what your monitoring sees. And for the decisions that determine what belongs in scope in the first place, the rest of Building a Monitoring Strategy picks up exactly where this review leaves off.
Put this into practice with SysPeep Monitor
A coverage review is only as good as its two lists. SysPeep Monitor builds the harder one for you. It auto-discovers the devices and interfaces on your network. So new systems show up in the console instead of hiding until they break. Closing the gaps you find is quick as well, with health checks over SNMP, WMI and SSH managed from a single web console. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep