Home › Topics › Who Watches the Watcher › Monitor Failure Modes

How Monitoring Systems Die (and How You Find Out)


Every failure on your network has a witness. When a switch dies, the monitoring system notices the silence and raises an alarm. When a disk starts filling, a threshold trips days before anything breaks. But there is exactly one failure with no witness at all: the failure of the monitoring system itself. When the watcher dies, the announcement of its death dies with it. No dashboard turns red — the dashboard is gone, and nobody stares at dashboards on a good day anyway. No alert email arrives — and no alert arriving is exactly what a healthy Tuesday looks like. The network goes quiet, and quiet is precisely what "all is well" sounds like too.


That is why this is the cruelest outage in IT. A dead monitoring system costs you nothing on the day it dies. It sends the bill later, on the day something else dies. The file server can crash at 2 AM, and instead of your phone buzzing, the first witness is a user at 8 AM. Then someone asks the awkward question: "why didn't we get an alert?" You discover the monitor has been a decorative web page for three weeks.


The good news: monitoring systems die in a small, predictable set of ways. This guide catalogs the seven failure modes. For each, it covers how it happens and what it looks like from the outside (spoiler: usually a suspicious silence). It also covers the specific check or habit that catches it. By the end you will have a self-monitoring starter kit you can configure in an afternoon, plus a quarterly drill that keeps it honest.


In this guide

  • Why you will not notice on your own
  • The seven failure modes, from hard death to time drift
  • The unifying lesson: no system can be its own witness
  • The self-monitoring starter kit (a table you can copy)
  • The quarterly failure-mode drill

Why you will not notice on your own


A monitoring system communicates in two states. Noise means something is wrong. Silence means everything is fine. Except that is not what silence means. Silence actually means "either everything is fine, or nothing is being checked" — and from the receiving end those two messages are identical. Every quiet day trains you a little more deeply to read silence as health, which is exactly the habit a dead monitor exploits.


So the goal of this guide is one sentence: turn every silent failure mode into a loud one. It is the same trick seven times, wearing seven disguises.


Failure mode 1: the hard death


How it happens: the monitoring server is just a computer, and it dies the way computers die. The monitoring service crashes and does not restart — a bug, a failed update, an out-of-memory condition, a Windows service stuck forever in "stopping." The host itself dies — power supply, failed disk, motherboard. Or, in the virtual world, the VM gets eaten. The hypervisor host fails, the storage behind it has a bad day, or an admin tidying up deletes the small unlabeled VM. It "didn't seem to be doing anything." That last one happens more often than anyone admits.


What it looks like from outside: total silence, instantly. Checks stop, alerts stop, and the console stops loading. But you only discover the console is down when you try to open it, and on a calm week nobody does. Days can pass. The network keeps working, so there are no complaints to tip you off. This is the purest form of the problem: a system whose one job is to report failures cannot report the failure that removed it.


What catches it: nothing on the box, because the box is gone. The only detection that works is something external that notices absence. An independent watcher expects a periodic "I'm alive" signal from the monitor and raises the alarm when the signal stops. That pattern — the heartbeat and the dead-man switch — is important enough that it gets its own guide. Also enable the operating system's automatic service-restart settings. They fix the mildest case (a crashed service on a healthy host), but they are a seatbelt, not a witness.


Failure mode 2: the slow suffocation


How it happens: a monitoring system writes constantly. Every poll of every device produces results that land in its database, plus logs, plus reports. Perhaps data retention was never configured — or was configured back when you had forty checks and you now have four hundred. If so, the database grows until the disk hits 100%. Then writes fail, the database stops, and the service hangs or crashes. The bitter irony: this machine watches everyone else's disk space for a living, and its own disk is often the one nobody monitors. A close cousin is resource starvation. In that scenario, the check count has tripled since the VM was sized. Polling cycles take longer than the polling interval, work queues back up, and the system slowly falls behind reality.


What it looks like from outside: a fade rather than a bang. Graphs develop gaps. The console gets sluggish. Alerts arrive late, then stop. Because the decline takes weeks, each individual symptom is easy to shrug off until the morning the whole thing is simply down.


What catches it: promote the monitoring host to a first-class monitored device — arguably the most important one you own. Its disk space, CPU, memory, and database size must be real checks with real alerts, not something you glance at when you remember. Use tighter thresholds than you would elsewhere. For an ordinary server you might warn at 85% disk. For this one, warn at 75%, because you want extra runway on the machine that provides your visibility. Watch the database's growth trend too, and set retention deliberately — your choice of database backend shapes how you manage that.


Failure mode 3: the silent partial


How it happens: a monitoring system is not one program; it is a bundle of moving parts. There are pollers (the worker processes that actually run checks) and collectors (components that speak a particular protocol or handle a particular device class). There are also schedulers, a database, and a notification engine. Any one part can fail while the rest carry on. A poller thread dies and takes its share of the checks with it. The collector for one device family breaks after a firmware update changes what the devices report. A remote probe at a branch office loses its connection home. The result: some checks stop while the others continue normally.


What it looks like from outside: this is the scariest mode, because the dashboard stays mostly green. The dead slice does not turn red — red means "checked and found broken," and these checks are not running at all. Instead the affected devices show stale data: the last known value, repeated forever. A graph flatlines at whatever the final reading was, or simply stops with a gap at its right edge. Green-with-a-frozen-slice looks almost exactly like green, and nobody investigates green.


What catches it: three habits. First, track the poll success rate — the percentage of scheduled checks that returned a fresh result this cycle — and alert when it dips. Second, set last-successful-check age alarms. The logic is simple enough to sketch:


for each check:
    age = now - time of last successful result
    if age > 3 x polling interval:
        raise alarm: "check is stale - no fresh data"

Third, learn the stale-data smell with your own eyes. Real metrics are jittery: CPU wiggles, traffic breathes, response times bounce around. A normally lively graph that has gone ruler-straight is not calm — it is frozen. Once you have consciously noticed this once, you will spot it across a room forever.


Tip: when a dashboard looks suspiciously peaceful, do a "newest data" sweep. Pick three busy devices and look at the timestamp of their most recent sample. If any timestamp is older than a couple of polling intervals, you are looking at a portrait of the past, not a live picture.


Failure mode 4: credential rot


How it happens: a monitoring system logs into things all day. It holds a service account (a user account owned by software rather than a person) for querying Windows machines over WMI. It holds community strings for SNMP, SSH keys for network gear, and API tokens for appliances. Every one of those credentials can rot. A password policy quietly expires the service account's password. A security review disables it ("what is this svc_monitor account? Nobody owns it? Lock it."). Repeated failed logins trip a lockout. A token is revoked during an integration cleanup. The moment a shared credential dies, everything it touched goes dark at once.


What it looks like from outside: a block flip. Thirty servers change state simultaneously — and here you must learn the difference between down and unknown. Down means the device did not answer at all: the ping failed, the host is unreachable. Unknown means the device is there and answering the doorbell, but it refused to answer the question — an authentication error. Thirty machines going down at once means a network disaster. Thirty machines going unknown at once almost always means one dead credential. An admin who has not internalized this difference can lose an hour of panic — or start rebooting healthy servers — over one expired password.


What catches it: give authentication failures their own alert category, separate from device-down, and make it loud and early. Lockouts are usually preceded by a trickle of auth failures — an alert on the first few buys you days. Put known expirations (service-account password rotations, token lifetimes) on a calendar exactly the way you track certificate expiry. And when a whole device class flips state in one moment, suspect the shared credential before you suspect the devices.


Failure mode 5: the notification death


How it happens: every check runs perfectly; the alarms just cannot leave the building. The mail relay the monitor sends through gets decommissioned or moved, and nobody updates the monitor's settings. The mail server starts requiring authentication or encryption the monitor is not configured for. The chat webhook (a special URL that lets software post messages into a chat channel) is revoked during a workspace cleanup. Or the distribution list that alerts go to is deleted, and the bounces land where nobody reads them.


What it looks like from outside: nothing. This is the most perfect silence of all seven modes, because the monitor itself believes everything worked. It evaluates the check, decides to alert, writes "notification sent" in its own log, and moves on, screaming into the void. Dashboards are green and accurate. The failure is only discovered during a real outage, at the worst possible moment. The discovery comes via the question this guide keeps circling back to: "why didn't we get the alert?"


What catches it: the only honest proof that alerting works is an alert that actually arrives. Schedule one: a weekly notification drill. A test alert deliberately fires and travels the full pipeline — same rules, same relay, same channels as a real emergency. A human confirms it landed. Test every channel you actually rely on, not just email. The fancy version has an independent system expect the test alert and alarm on its absence. That removes the human memory from the loop. It is the dead-man trick again, and the heartbeat guide shows how to wire it.


Failure mode 6: the config self-wound


How it happens: the monitor does exactly what its configuration says, and configuration is edited by humans holding power tools. A bulk edit with a slightly wrong filter disables half your checks instead of the three you meant. A maintenance window gets scoped to "all devices" instead of one site, or is created with no end time. Somebody pauses a group of checks "temporarily" before a migration and never comes back. Nothing crashed. The system is in perfect health, faithfully executing a mistake.


What it looks like from outside: silence for the affected slice — the same smell as the silent partial, but self-inflicted. It is often suspiciously tidy: exactly every device matching some pattern, all quiet from the same minute onward.


What catches it: a change-review habit that takes ten seconds. Before any bulk operation, note the number of active checks. Afterward, look again. If you meant to disable four and the count dropped by four hundred, you just caught the wound while it is still a scratch. Give every pause and every maintenance window an end time, without exception. And version your configuration — a scheduled export, kept and compared over time, turns "what changed?" from archaeology into a one-line answer. The backup guide builds that export-and-diff habit properly.


Failure mode 7: time drift


How it happens: the monitor's own clock goes wrong. NTP was never configured, or points at a server that no longer exists. NTP is Network Time Protocol, the mechanism computers use to keep their clocks synchronized. A VM resumes from a snapshot or a long pause carrying yesterday's clock. A hypervisor host drifts and drags its guests along.


What it looks like from outside: not silence this time, but a subtle scrambling of reality. Events are stamped out of order, so the timeline of an incident stops making sense. The alert appears to fire before the failure it describes. Scheduled checks and reports run at odd hours. Maintenance windows open at the wrong time, suppressing alerts you needed or failing to suppress ones you expected. Day-counting checks, like certificate expiry, quietly miscount. Every one of these symptoms will send you hunting a ghost unless you think to look at the clock.


What catches it: monitor the monitor's own NTP offset — the gap between its clock and a reference — as a first-class check. Alarm when the offset exceeds a couple of seconds. It is the most boring check in this entire guide. It costs nothing. And the day it fires it will save you an afternoon of confusion.


The unifying lesson: no system can be its own witness


Read back through the catalog and a pattern emerges. Naive self-monitoring — the monitor checking itself and emailing you about problems — is defeated by every single mode, each in its own way. The hard death removes the machine that would send the email. The notification death lets the self-checks pass while the email silently evaporates. The silent partial and the config self-wound hide inside the green. Credential rot and suffocation are the kinder modes: they at least offer warnings first, if you have the checks to hear them. But the principle stands, and it is the founding idea of this whole topic. The monitor cannot be the sole witness to its own death. Somebody — something — outside the box has to hold the other end of the rope.


The full answer is layered, and this pillar assembles it one guide at a time. Self-checks on the monitor's host catch the slow degradations before they become deaths. An external heartbeat with a dead-man switch catches every flavor of total failure with one mechanism. Backups and a rebuild runbook turn a dead monitor from a disaster into an afternoon. Out-of-band access covers the day the network itself is the thing that died. And the upgrade guide handles the one time you make the monitor go dark on purpose. Where the monitor physically lives shapes which modes threaten you most — see where the monitoring server should live.


The self-monitoring starter kit


Here is the practical distillation: the dozen checks the monitoring system should run on itself, plus the external minimum that no self-check can replace. Each self-check is an ordinary check, configured in the same tool the same way as for any server; only the target is special.


CheckFailure mode it catchesSuggested starting rule
Own disk spaceSlow suffocationWarn 75%, alert 85% — tighter than normal servers
Own CPUResource starvationWarn at 80% sustained 15 min
Own memoryResource starvationWarn at 85% sustained 15 min
Database size and growthSlow suffocationAlert on unusual growth; review trend monthly
Poll success rateSilent partialAlert when below ~98% of scheduled checks
Newest-sample age per checkSilent partialAlert at 3x the polling interval
Active check countConfig self-woundAlert on an unexplained drop
Authentication failuresCredential rotAlert on any new source of auth errors
NTP offsetTime driftAlert beyond a few seconds
Service watchdog (OS auto-restart)Hard death (mild case)Auto-restart on crash, plus an alert that it happened
Own console certificate expiryPlain embarrassmentWarn 30 days out
Config backup ageRecovery readinessAlert if the newest export is over a week old
External: heartbeat + dead-manHard death, total silenceIndependent watcher expects a signal every few minutes
External: weekly test alertNotification deathFires through the full pipeline; a human (or watcher) confirms arrival
External: reachability from another boxHost deathAny independent machine pings the monitor and complains loudly

The quarterly failure-mode drill


Checks decay, and so does confidence in them — which is why the last piece is a habit, not a configuration. Once a quarter, run a tabletop exercise: a talk-through of a failure, no actual breakage required. Half an hour, with coffee.


  • Step 1. Pick one failure mode from this page. Rotate through them; by the end of two years you will have covered the catalog twice.
  • Step 2. State the scenario out loud, concretely: "the SNMP collector died at 9:14 on a Tuesday morning."
  • Step 3. Trace, honestly, what would notice. Which check fires? Where does the alert go? Who sees it, and when?
  • Step 4. If the honest answer is "a user, eventually" or "nobody," you have found a gap — and finding it today cost nothing.
  • Step 5. Close the gap the same week, while the scenario is still vivid.
  • Step 6. Write one line in your notes: date, mode tested, result.

The advanced version, once the paper drill stops finding gaps: actually do it. On a quiet afternoon, with a colleague warned, stop the monitoring service. Time how long until something — other than your own guilty conscience — tells you it is down. That number is your real detection delay for the hard death, and shrinking it is exactly what the next guide is about.


Where to go next


You now know the seven ways a monitor dies, and the shape of the defense. That means self-checks for the slow modes, external witnesses for the fatal ones, and drills to keep both honest. The single highest-value next step is building that external witness — Heartbeats and Dead-Man Switches walks you through it in an afternoon. The rest of the defense-in-depth lives in the other guides of Who Watches the Watcher.


Put this into practice with SysPeep Monitor

The self-monitoring checks in this guide are ordinary checks — disk, CPU, database growth, data freshness. SysPeep Monitor can point them at its own host the same way it watches any other Windows server, over WMI and SNMP. Its alert rules and email notifications handle the weekly drill, and high-availability clustering blunts the hard-death scenario. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep