Monitoring Everything: The Anti-Pattern That Feels Like Diligence
Nobody sets out to monitor everything. It happens to you, one sensible decision at a time. By the time you notice, the console has four thousand checks in it. The one graph you need is somewhere on page nine. Worse, it feels like a virtue. A person who monitors everything cannot be accused of missing anything, and that is exactly the reasoning that gets you there.
This guide is about that anti-pattern: what pushes a team into it and what it actually costs (most of the bill is not disk). It shows how to get out without swinging to the opposite error of monitoring nothing. The cure is not "monitor less". It is "monitor on purpose", and the difference between those two is the whole guide.
By the end you will be able to recognise the three pressures that produce over-monitoring. You will be able to name the five costs and put a rough number on two of them. You will be able to prune a real network class by class using a worked example. You will be able to fill in a worksheet that makes the pruning repeatable and install the one rule that stops it growing back.
In this guide
- How it happens
- What it costs
- The cure is not "monitor less"
- A worked pruning: Northgate Retail
- The pruning worksheet
- Relapse, and the intake rule
- The alert that was three hundred and twelfth
- The pruning checklist
- Where to go next
How it happens
Three pressures produce over-monitoring, and they usually arrive together. None of them is stupid. That is what makes them dangerous.
"Why not" is easier than "why"
Auto-discovery is the feature that scans your network and lists everything it can find: every device, every interface, every disk, every service. It is a genuinely good feature. It also presents you with four thousand things and a button that adds them all. Adding one requires a reason. Adding all of them requires nothing, and it is Friday. So the question asked of each item is "why not?" And "why not?" has no answer, because there is never a reason not to know something. There is only a cost, and the cost is invisible at the moment of clicking.
Fear of the one metric you did not collect
Every admin has a memory of an incident where the one counter that would have explained it was the one not being collected. That memory is doing a lot of work. It means that every un-monitored thing feels like a future regret, and monitoring it feels like insurance. The trouble is that insurance has a premium, and this one is paid in attention, which is the scarcest thing you have. You cannot collect everything just in case. You can only collect what you will look at, plus a deliberate margin.
The licence says you may
Many monitoring products license by device, sensor, or element. A team that has paid for a thousand feels a duty to use a thousand. An unused licence looks like waste, so the licence count becomes a target. It is not a target. It is a ceiling. (A thousand-seat licence does not oblige you to hire a thousand people either.)
Put the three together and you get a monitoring system that grew because growing was free, felt safe, and used the licence. Nobody decided to monitor everything. Everybody decided not to decide.
What it costs
Over-monitoring has five costs, and the one people worry about, disk, is the cheapest. The others are paid by people.
Attention
The important graph is one of nine hundred. When the file server's disk graph sits in a list with every virtual adapter on every host, finding it takes a search. A thing that takes a search does not get looked at on a Tuesday. Attention does not scale with check count. It is fixed, and every check you add divides it further.
Disk
Suppose four thousand sensors, each polled once a minute. That is four thousand samples a minute, two hundred and forty thousand an hour, and about five and three-quarter million a day. At a modest fifty bytes per stored sample, before any rollup, that is roughly two hundred and ninety megabytes a day. It is about a hundred gigabytes a year of raw samples. Rollups shrink it, and storage math for monitoring data shows how to work it out for your own numbers. The point here is only that most of that hundred gigabytes describes things nobody asked about.
Polling load
Every check is a request the collector sends and a device answers. Four thousand checks a minute is a real load on the collector, a real load on the network, and a real load on the devices. Some of those devices are small switches with small processors that would rather be switching. The polling load budget guide shows how to add it up. Over-monitoring is how the budget gets spent on things that did not need buying.
Alerts
Every metric eventually gets a threshold. That does not happen on day one. It happens after an incident, or because a template applied one, or because someone noticed a graph and felt it should have a line. And every threshold eventually fires. A system with four thousand checks is a system with a slow, permanent drizzle of alerts about things nobody chose to watch. That is the origin story of alerting on everything, the next guide in this pillar.
Trust
This is the expensive one. A monitoring system that shows nine hundred graphs and drizzles alerts acquires a reputation for being "noisy". And "noisy" is a polite word for "ignored". Once the team has decided the monitoring is noise, the one real alert arrives in the same inbox as the noise and is treated the same way. Over-monitoring does not just waste the monitoring. It teaches people not to believe it.
The five costs compound. Attention is divided, so alerts are missed, so trust falls. Then the monitoring is checked less, so more is added to "make sure", so attention is divided further. I have watched that loop run for years in a single system, and it never once looked like a problem from inside. It looked thorough.
The cure is not "monitor less"
The tempting fix is to delete half of everything. And it is the wrong fix, because it keeps the same absence of reasoning and merely applies it in reverse. The right fix is to monitor on purpose: every check has a question it answers and a person who will read the answer. A check that has both stays. A check that has neither goes. A check that has a question but no reader is the interesting middle, and it is where most of the four thousand end up.
Monitoring on purpose sorts every check into one of three bins:
- Keep. The check answers a question someone asks, and its failure should interrupt a person. It gets a threshold and an alert.
- Collect but don't alert. The check answers a question someone will ask later (a capacity review, an incident investigation, a report). But nobody needs to be told when it changes. It is stored, graphed, and silent.
- Drop. Nobody asks the question. It is not polled at all.
The middle bin is the one over-monitored teams do not have, and adding it is most of the cure. "Collect but don't alert" lets you keep the insurance without paying the attention premium, because a silent graph costs disk and nothing else. It is the answer to the fear of the one metric you did not collect: collect it, and shut it up.
Two other guides carry the reasoning I will not repeat here. What not to monitor lists the classes of thing that are almost always safe to drop. The guide on criticality tiers gives the method for deciding which devices matter most. That is what tells you where the "keep" bin should be generous. The anti-hoarding guide covers the collector's instinct itself. This guide is the worked practice: what pruning looks like when you actually sit down and do it.
A worked pruning: Northgate Retail
Northgate Retail is fictional, but its monitoring is not. Northgate has a head office, a warehouse, twelve stores, and an auto-discovery run that was accepted in full three years before anyone looked at it. The counts below are illustrative and rounded. The two admins sorted the checks into classes, asked three questions of each class, and recorded a verdict.
The three questions were the same every time:
- What question does this class of check answer?
- Who looks at the answer, by name, and how often?
- If it alerted at three in the morning, what would the on-call person do?
The third question is the one that does the work. If the honest answer is "nothing until morning", the check belongs in the collect-only bin, whatever it is. If the answer is "nothing, ever", it belongs in the drop bin. Only a check whose answer is a real action at three in the morning has earned an alert.
| Class of check | Count | What the questions revealed | Verdict |
|---|---|---|---|
| Traffic and errors on every access switch port | 2,300 | Nobody looked at user-facing ports; uplinks were looked at weekly | Drop user ports; keep uplinks (about 60) |
| Virtual, loopback, and tunnel interfaces on servers | 180 | No question anyone could name | Drop |
| Every running service on every server | 2,600 | About 40 services had a named person who would restart them at night | Keep 40; drop the rest |
| Disk space on every server volume | 140 | Looked at monthly for capacity; alerted on only for the tier-one servers | Keep, with tiered thresholds |
| Printers: ping, toner, page counts | 45 | Facilities wanted page counts for the lease; nobody wanted a 3 a.m. toner alert | Collect only |
| Store tills: ping, CPU, memory | 160 | Store managers wanted a morning list; IT only cared if a whole store went dark | Collect only; one dependency alert per store |
| WAN links and firewalls | 30 | Every outage in two years had started here | Keep |
| Core services: directory, DNS, DHCP, email, stock system | 5 | Ping only; the stock system had been "up" during its last two outages | Keep, and add real service checks |
| Server room temperature and power | 22 | Two incidents, both caught here first | Keep |
The totals tell the story. Roughly five thousand five hundred checks went in. About four hundred came out in the "keep" bin, another two hundred or so were collected silently, and the rest were dropped. The monitoring got smaller by an order of magnitude and better in the one place it had been genuinely dangerous. That place was the five core services, which had been the least-monitored things in the building.
Notice what the pruning did not do. It did not remove a single check that anyone had ever acted on, because the second question would have caught it. It did not lose the printers' page counts, which facilities needed, because collect-only kept them. And it did not require anyone to argue about a specific switch port, because the decision was made per class. Arguing per check is how prunings die. (You will still be asked about one port. There is always one port.)
Before you can sort by class, you need the counts, and the monitoring database usually has them. Table and column names differ between products, so treat this as the shape of the query and check the documentation for yours:
-- Illustrative: how many enabled checks of each type, and how many
-- of them have an alert rule attached. Names differ between products.
SELECT s.sensor_type,
COUNT(*) AS checks,
SUM(CASE WHEN r.rule_id IS NULL THEN 0 ELSE 1 END) AS with_alerts
FROM sensors s
LEFT JOIN alert_rules r ON r.sensor_id = s.sensor_id AND r.enabled = 1
WHERE s.enabled = 1
GROUP BY s.sensor_type
ORDER BY checks DESC;
The first few rows of that result are your classes, in the order you should prune them. The biggest class is almost never the most important one, which is exactly why it is the biggest.
The pruning worksheet
The pruning worksheet is one row per class of check, with the question it answers, who looks at it, and the verdict. Fill it in with a colleague, one class at a time, and do not move to the next row until the current one has a verdict. Leave the count column in; it is the evidence that the pruning happened.
| Class of check | Count | Question it answers | Who looks at it (name, cadence) | 3 a.m. action? | Keep / collect-only / drop |
|---|---|---|---|---|---|
Rules for filling it in, so that the worksheet survives contact with a real network:
- A class is a set of checks that would get the same verdict for the same reason. If half a class would be kept and half dropped, split it into two rows.
- "Who looks at it" must be a person's name or a named report with a named reader. "The team" is not an answer. "Nobody" is an answer, and it means drop.
- The 3 a.m. column is yes or no. "Depends" means no.
- Keep requires a yes in the 3 a.m. column. Collect-only requires a name in the "who looks" column. Neither means drop.
- Date the worksheet and keep it. The next pruning starts from this one.
Tip: when a class lands in "collect-only", set its polling interval to something lazier at the same time. A graph nobody alerts on rarely needs a sample every minute. Every five or ten is usually plenty, and the polling load falls with the alert count. Check the product documentation for how intervals are set per check or per group.
Relapse, and the intake rule
A pruned monitoring system grows back. The three pressures that filled it are still there. Discovery still finds things, the fear of the missing metric still nags, and the licence still has headroom. Without a rule, the next discovery run puts everything straight back, usually within a year. Usually, someone who was not in the room for the pruning runs it.
The intake rule is the defence: nothing is added to monitoring without a row on the worksheet. A new device, a new class of check, a newly discovered interface, all of them get the three questions before they get a poller. The rule is small, it takes a minute per addition, and it converts monitoring from something that accumulates into something that is chosen. New system intake describes the fuller process for bringing a system into monitoring; the intake rule is its smallest possible form.
The rule has one enemy, which is the discovery run that adds by default. Most products let you choose whether discovered items are monitored automatically or merely listed for approval. Choose listed. A discovery result that waits for a human to say "keep" is a list of candidates. One that starts polling on its own is next year's pruning. Check the product documentation for where that setting lives; it is worth the five minutes it takes to find.
Schedule the next pruning when you finish this one. Once a year is enough if the intake rule holds; twice if it does not. Put it on the calendar with the worksheet attached. And treat a pruning that finds nothing to drop as a success, not a wasted hour. It means the rule worked.
The alert that was three hundred and twelfth
Harbourview Clinic monitored every port on every switch, every service on every server, and the toner in every printer. Each of them had a threshold. When the disk holding the patient records system filled one Tuesday morning, the alert arrived at nine minutes past seven. It was the three hundred and twelfth email of the day in the shared inbox. A nurse reported the outage by phone at twenty past eight. The review found that monitoring had "worked as designed", which was true, and was the problem.
The clinic's pruning took two afternoons. The patient records disk is now one of about ninety alerts. And the disk's threshold is the only thing that would have been paged that morning. The printers still report their toner. Nobody is woken up about it.
The pruning checklist
Use this when you run a pruning, and again at the yearly review.
- The check-count query has been run and the result is attached to the worksheet.
- Every class of check has a row on the worksheet with a question, a named reader, a 3 a.m. answer, and a verdict.
- No class marked "keep" lacks a yes in the 3 a.m. column.
- No class marked "collect-only" lacks a named reader.
- Every "drop" class has been disabled or removed, and the count afterwards has been recorded.
- Every "collect-only" class has had its alert rules removed and its polling interval reviewed.
- The top-tier services each have a service-level check, not only a ping.
- Discovery is set to list new items for approval, not to monitor them automatically.
- The intake rule is written down where new devices are added.
- The worksheet is dated and stored, and the next pruning is on the calendar.
Where to go next
Monitoring everything is the anti-pattern that feels most like doing the job well, which is why it needs a guide of its own. The cure is a worksheet, three questions, and a rule at the door. And the result is a monitoring system small enough that the important graph is on page one. What grows in the gaps left by over-monitoring is its twin, alerting on everything. What happens to a pruned system nobody tends is set-and-forget rot. Both are in this pillar, along with the field guide that names the rest.
All of these are part of Common Monitoring Mistakes and Anti-Patterns.
Put this into practice with SysPeep Monitor
Monitoring on purpose needs a tool that finds everything and then lets you choose. SysPeep Monitor auto-discovers devices and interfaces and leaves the decision of what to monitor to you. So a discovery run is a list of candidates rather than four thousand new pollers. Thresholds, dependencies, and alert rules keep the noise down on what you keep. The free trial installs on a Windows machine if you want to run the worksheet against your own network.
Download Free TrialLearn About SysPeep