Set-and-Forget Rot: How Monitoring Decays When Nobody Tends It
A monitoring system does not break when it is neglected. That would be too easy. It keeps running, keeps polling, keeps drawing green tiles, and slowly stops describing the network it was built for. The screen is still on. The alerts still go out. Nothing on the surface says that a third of what it shows is no longer true. The only way to find out is to check, which is the thing nobody has been doing.
This guide is about that decay: what it looks like, specimen by specimen. It explains how to detect each one with a query or a report rather than a feeling. It shows how to stop decay with a maintenance calendar. That calendar turns tending the monitoring into a set of small, boring, scheduled tasks with names on them. Boring is the point. Rot is what happens when the tasks are interesting enough to be optional.
By the end you will be able to define rot and explain why it is invisible from the console. You will be able to recognise seven specimens and detect each one, and run three rot-detection reports against your own system. You will be able to copy a maintenance calendar with weekly, monthly, quarterly, and yearly tasks. And you will be able to connect it to the review rituals you may already have.
In this guide
- Rot, defined
- The seven specimens
- Rot detection reports
- The maintenance calendar
- Tying it to rituals you already have
- The mailbox at Acme
- The rot checklist
- Where to go next
Rot, defined
Rot is the state a monitoring system reaches when it is still running but no longer describes the network. Every part of it is doing what it was configured to do. The configuration is simply about a network that has since changed: devices have been retired, services moved, disks grown, people left. And the monitoring was told about none of it.
Rot is invisible from the console because the console reports on the configuration, not on whether the configuration is right. A retired server that is still polled shows as down. And its alert was suppressed a year ago so that it would stop bothering anyone. So the server shows as a quiet grey tile that everyone has learned to look past. A threshold set for a smaller disk shows as a threshold. A notification list containing three people who have left shows as a notification list. Nothing is red. Nothing is wrong, in the sense the software understands wrong.
It is the upkeep family's defining specimen, and it differs from the other anti-patterns in this pillar in one way. It does not need anyone to make a bad decision. Monitoring everything requires someone to click "add all". Alerting on everything requires someone to say "we should alert on that". Rot requires nothing. It is what a monitoring system does on its own, at its own pace, whenever nobody is holding it. Left long enough, it becomes a museum of the network as it stood on the day the last person who cared about it left.
The seven specimens
Each specimen below has a description and a way to detect it. The detection matters more than the description, because every one of these is easy to describe and easy to deny. A list from a query is harder to deny than a hunch.
Dead devices still polled
The device was retired, physically removed, and never removed from monitoring. It has been down for a year. Its alert was suppressed within a week of it going down, because the alert was annoying and the device was known to be gone. The suppression has outlived everyone's memory of why. The device is polled every minute, fails every minute, and costs a small amount of collector time forever. Detect it with the ninety-day down report below. Keeping inventory current is the upstream fix.
Retired services with live checks
The service was migrated to a new system, and the old one was switched off. But the check against the old address was never deleted. Sometimes it fails quietly and is suppressed. Sometimes, worse, the old address was reused and the check is now passing against something else entirely. Detect it by listing every check whose target address is not in the current inventory, or whose name mentions a system nobody uses.
Thresholds set for a network three sizes ago
The eighty per cent disk warning made sense on the disk it was set for. The disk has been replaced twice since. The bandwidth alert was set when the link was a tenth of its current size, and now fires at a level of traffic that is an ordinary afternoon. Detect it by listing thresholds with their last-modified date and sorting oldest first; anything older than the device it applies to has not been looked at since.
Checks that pass because they check nothing
This is the most dangerous specimen because it is the only one that shows green. An HTTP check requests a web page and passes if the server answers with a success code, usually 200. A web server that has been replaced by a maintenance page still returns 200. So does a login page, a redirect that landed somewhere friendly, and an error page that a developer styled nicely. The check is passing. The service is gone. Detect it by listing checks that have never failed, and then looking at what each one actually fetches. (A check that has never failed is either monitoring something perfect or monitoring nothing. You have fewer perfect things than you think.)
Notification lists with people who left
Every alert goes to a list, and the list was right when it was made. Since then two people have left and one has changed role. The mailbox of one of the leavers still exists and accepts mail because nobody closed it. Alerts are being delivered, successfully, to nobody. Detect it by comparing every address in every notification rule against the current directory; the report below shows how.
The map that stopped matching reality
The network map on the wall shows the site as it was drawn. The site has since gained a floor and lost a building. A map that is wrong is worse than no map, because it is trusted, and keeping maps truthful covers the cure in full. Detect it by walking a colleague through the map and counting the corrections; more than two and it is rotten.
The collector nobody upgraded
The monitoring server itself is a server, and it rots like the others. It has not been patched, its database has not been housekept, and its backups have not been restored. And the person who installed it has left. It works. It will work until the day it does not. On that day it will fail in one of the ways monitoring system failure modes describes, silently, while every tile stays green. Detect it by asking when it was last upgraded and when its backup was last restored, and treating "I'd have to check" as the answer.
Rot detection reports
Three reports find most rot. They are shown as queries against a monitoring database. But the same reports can usually be built from the product's own reporting or exported to a spreadsheet. The query is only the clearest way to say what the report contains. Table and column names differ between products, so treat these as illustrative and check the documentation for yours.
Devices down for more than ninety days
-- Illustrative: every device that has not answered in ninety days.
SELECT device_name, ip_address, last_up_at,
CAST(julianday('now') - julianday(last_up_at) AS INTEGER) AS days_down
FROM devices
WHERE status = 'down'
AND last_up_at < DATE('now', '-90 days')
ORDER BY last_up_at;
Every row is either a device that should be removed from monitoring or an outage that has lasted three months. And the second kind is rarer than you would hope. Work through the list with the inventory beside you. Remove what is gone, fix what is broken, and for anything you cannot identify, ask around for a week and then remove it too.
Checks that have never failed
-- Illustrative: enabled checks older than a year with no failure on record.
SELECT s.sensor_name, s.target, s.created_at
FROM sensors s
LEFT JOIN alert_events e ON e.sensor_id = s.sensor_id
WHERE s.enabled = 1
AND s.created_at < DATE('now', '-365 days')
GROUP BY s.sensor_id
HAVING COUNT(e.event_id) = 0
ORDER BY s.created_at;
Some of these are fine. A check on a core switch that has genuinely never gone down is a good check with a good switch behind it. The ones to look at are the HTTP and service checks. Open each target in a browser and confirm it is the service the check is named for. Then break one on purpose, in a maintenance window, and confirm the check notices. A check that has never been seen to fail has never been proved to work.
Notification recipients not in the directory
This one is a comparison rather than a query. Export every address from every notification rule into one file, one per line. Export every active mailbox from the directory into another. Then list what is in the first and not the second. On Windows, a short PowerShell command does it:
# Illustrative: addresses in recipients.txt that are not in directory.txt Compare-Object (Get-Content recipients.txt) (Get-Content directory.txt) | Where-Object SideIndicator -eq '<=' | Select-Object -ExpandProperty InputObject
Every line of output is an alert going to someone who cannot read it. Replace each with a named current person, and while the export is open, look for rules whose only recipient is a distribution list. Those are alerts nobody owns, and process anti-patterns covers what to do about them.
Once the three reports exist, save them. Most products can store a report definition and run it on a schedule. And a rot report that lands in a named person's inbox on the first of the month is a report that gets read. One that has to be rebuilt from memory each quarter is a report that gets rebuilt twice and then forgotten. If the product cannot schedule it, put the query in a text file next to the monitoring documentation, with the date it was last run written at the top.
The maintenance calendar
The maintenance calendar is the cure for rot. And it is nothing more than a list of small tasks with a cadence and an owner. Its value is not in any one task. It is in the fact that each task has a day on which it is supposed to happen and a person who is supposed to do it. That makes "nobody has looked at this in a year" impossible without someone noticing. Copy it, fill in the owner column with names, and put every row on a real shared calendar.
| Cadence | Task | Owner | Evidence it happened |
|---|---|---|---|
| Weekly | Read the current "down" list and act on anything new | Note in the weekly log | |
| Weekly | Check the collector's own health: service running, disk, database size, last backup | Note in the weekly log | |
| Weekly | Read the ten alert rules that fired most; flag any with no action taken | Flagged rules added to the triage list | |
| Monthly | Run the ninety-day down report; remove or repair every row | Report saved with actions written on it | |
| Monthly | Compare notification recipients against the directory; replace leavers | Comparison output saved, zero lines or all resolved | |
| Monthly | Confirm every change made to the network this month is reflected in monitoring | Change list with a tick per item | |
| Quarterly | Threshold review | Review record with last-reviewed dates updated | |
| Quarterly | Alert triage | Triage table dated and stored | |
| Quarterly | Coverage review against the current inventory | Gaps list with owners | |
| Quarterly | Run the never-failed report; verify each HTTP and service check by hand | Report with a verified tick per row | |
| Quarterly | Walk the network map with a colleague; correct it | Map updated, date on it | |
| Quarterly | Monitoring database housekeeping and retention check | Database size before and after | |
| Yearly | Full pruning worksheet against every class of check | Worksheet dated and stored | |
| Yearly | Upgrade the collector; restore its backup to a test machine | Version recorded; restore test passed | |
| Yearly | Re-read this calendar; remove tasks nobody did, add tasks that were missed | New calendar version dated |
Two rules keep the calendar alive. First, the evidence column is not optional: a task with no evidence did not happen, whatever anyone remembers. The evidence is what you show the next person who asks whether the monitoring can be trusted. Second, every task has one owner, by name. A task owned by "the team" is owned by whoever feels most guilty, and that person leaves eventually.
The weekly tasks take about twenty minutes. The monthly ones take an hour. The quarterly ones take an afternoon between them, and the yearly ones a day. Call it forty hours a year to keep a monitoring system telling the truth. I have spent longer than that on a single incident that a truthful monitoring system would have caught, and so, probably, have you.
Tip: the yearly "re-read this calendar" task is the one people skip and the one that matters most. A calendar that has a task nobody does for two years is teaching the team that the calendar is optional. Delete the task, or find out why it is not done, but do not leave it there being ignored.
Tying it to rituals you already have
Several of the calendar's quarterly rows are rituals that already have guides of their own, and the calendar should point at those rather than duplicate them. The threshold review ritual is the quarterly threshold row, done properly. The coverage review is the quarterly coverage row: the check that everything in the inventory is monitored and everything monitored is in the inventory. The alert triage row is the quarterly pass from alerting on everything, and the yearly pruning is the worksheet from monitoring everything.
If you already do some of these, the calendar's job is to make sure they are on the same calendar, with the same evidence rule. That way, the one ritual that quietly lapses gets noticed. If you do none of them, do not start with all of them. Start with the three weekly tasks and the ninety-day down report. Those four alone remove most of the rot, and the rest can be added a quarter at a time.
The calendar also answers a question that comes up when the person who tends the monitoring goes on leave or leaves for good: what needs doing, and when? A written calendar with owners is the difference between a handover and an archaeology project. Upgrading monitoring safely covers the yearly upgrade row so that it is a task and not an adventure.
The mailbox at Acme
Acme's monitoring had been emailing a mailbox nobody owned for three years. It had belonged to an admin who left; his account was disabled. But the mailbox was kept "in case", and the notification rules that named it were never touched. The rot was found by accident, during a mail system clean-up, when someone opened the mailbox and found forty thousand unread alerts. Among them were eight months of warnings from the warehouse UPS. Its battery had been failing the whole time, and the UPS failed completely, during a power cut, the following month. The review described the notifications as "correctly generated".
Acme now runs the recipient comparison monthly. It takes ten minutes. The first run found eleven addresses.
The rot checklist
Use this at the quarterly review, and once, now, to see where you stand.
- The ninety-day down report has been run this month and every row has been removed, repaired, or assigned.
- No check targets an address that is not in the current inventory.
- Every threshold has a last-reviewed date within the last year.
- The never-failed report has been run this quarter and every HTTP and service check on it has been verified by hand.
- Every notification recipient is a current, named person in the directory.
- The network map has been walked with a colleague this quarter and corrected.
- The collector has been upgraded within the last year and its backup restored to a test machine.
- The maintenance calendar is on a shared calendar with a named owner on every row.
- Every task due this month has evidence that it happened.
- The calendar itself has been re-read within the last year.
Where to go next
Rot is the only anti-pattern in this pillar that arrives without a decision, and so it is the only one that cannot be prevented by deciding better. It can only be prevented by tending: small tasks, on a calendar, with names on them, and evidence that they were done. None of it is interesting. All of it is the difference between a monitoring system that describes your network and one that describes the network your predecessor had.
The upkeep rows on the calendar assume the monitoring was pruned and triaged first; monitoring everything and alerting on everything cover that. The reason calendars lapse is usually a process problem, and process anti-patterns covers the review that never happens and the monitoring nobody owns. The field guide names the rest. All of these are part of Common Monitoring Mistakes and Anti-Patterns.
Put this into practice with SysPeep Monitor
Rot is easier to see when the tool shows you what it is actually polling and who it is actually telling. SysPeep Monitor auto-discovers devices and interfaces so a retired device stands out against the inventory. It keeps thresholds and alert rules where they can be reviewed, and reports on device state and alert history from a single web console. That is where the ninety-day down report comes from. The free trial installs on a Windows machine if you want to run the rot reports against your own network.
Download Free TrialLearn About SysPeep