Home › Topics › Common Monitoring Mistakes and Anti-Patterns › Process Anti-Patterns

Process Anti-Patterns: Hero Culture, Silent Changes, and the Monitoring Nobody Owns


You can buy your way out of most monitoring problems. A better collector fixes polling load. A better alert engine fixes flapping. A better dashboard fixes the forty-panel screen. What no purchase fixes is the admin praised for two-in-the-morning heroics that a Tuesday afternoon would have prevented. No purchase fixes the change nobody told monitoring about. Nor does it fix the shared inbox where every alert goes to be seen by everyone and acted on by no one. Those live in people and habits, and they are the subject of this guide.


Process anti-patterns are the family that causes the other four. Monitoring everything, alerting on everything, wallpaper, and rot are all things a team with the right habits corrects before they take hold. A team with the wrong habits reinstalls them a year after any cleanup. That is why they come last in the pillar and why, if you only fix one family, it should be this one.


By the end you will be able to recognise seven process anti-patterns by how they look and why they persist. You will be able to apply a cure to each that costs almost nothing. You will be able to fill in an ownership and rituals table that gives "who does this?" an answer. And you will be able to explain to a manager why the cheapest fixes in monitoring are the ones that never get done.


In this guide

  • The mistakes no tool can fix
  • Mistakes about people
  • Mistakes about change
  • Mistakes about ownership
  • The ownership and rituals table
  • The site nobody told monitoring about
  • Hardest to fix, cheapest to fix
  • The process checklist
  • Where to go next

The mistakes no tool can fix


A process anti-pattern is a habit, rather than a configuration, that reliably makes monitoring worse. The distinction matters because the cures are different in kind. A configuration anti-pattern is fixed by changing the configuration, which is work but is contained. A process anti-pattern is fixed by changing what people do. That is cheap and takes a sentence to describe. It is resisted with an energy that no configuration change provokes.


Each of the seven below is described in three parts: how it looks, why it persists, and the cure. The "why it persists" part is the important one. Every one of these habits is a rational response to some pressure. If you remove the habit without removing the pressure, the habit comes back. (It comes back under a different name, and someone will call it a process improvement.)


Mistakes about people


Hero culture

It looks like praise. The admin is woken at two in the morning, drives in, and brings the site back by six. That admin is thanked in the morning meeting, mentioned in the monthly report, and remembered at review time. Three months earlier, another admin spent a Tuesday afternoon fixing the threshold that would have caught the same failure at nine in the evening. That admin is thanked by nobody, because nothing happened. Nothing happening was the achievement. Nobody can see it.


It persists because the reward is real and the alternative is invisible. Managers reward what they can see, and an outage resolved is visible in a way that an outage prevented never is. The organisation does not decide to reward firefighting. It decides to reward what it notices, and then it notices fires. Over time the people who prevent things learn that prevention is unpaid, and the whole team drifts toward the mode that burns people out. Solo admin burnout describes what that does to the one person carrying it.


The cure is to make prevention visible in the same place as incidents. Keep a log of near misses and preventions next to the incident log. One example is the disk expanded at seventy per cent. Another is the alert that fired at nine in the evening and was fixed by ten. A third is the retired switch removed from monitoring before it flapped. Report both lists to the same manager in the same document. The making invisible work visible pillar covers this at length. The short version is that a thing written down can be thanked, and a thing that is not cannot.


Blame in post-incident reviews

It looks like a review that ends with a name. "The outage was caused by a misconfigured threshold" becomes "the outage was caused by the person who set the threshold". The review's action item is, in effect, that the person should be more careful. Careful is not an action item. It is a wish.


It persists because naming a person feels like closure. It also persists because a review that names a system has to go on to fix the system. That is more work than a quiet word. The people being blamed rarely object out loud. They just stop bringing things to the review, and within a year the reviews are shorter, calmer, and useless. Blameless reviews covers the method. The thing it protects is that nobody tells the truth in a review that ends with a name.


The cure is a rule that every action item names a change to a system, a document, a check, or a ritual. An action item must never name a change to a person. If the review cannot find one, it has not finished.


Mistakes about change


Silent changes

It looks like a network that is bigger than the monitoring thinks it is. A new site, a moved server, a re-addressed subnet, a replaced firewall: all are real, all are working, and all are unknown to the monitoring. That is because the change was made by someone whose job was the change. The monitoring is not wrong about anything it knows. It just knows less than it did, and it has no way to tell you.


It persists because telling monitoring is nobody's step. The change process, if there is one, asks whether a change affects the monitoring server. The answer is almost always no, so the box is ticked. The question it does not ask is whether the change affects what monitoring should be watching. That answer is almost always yes. The configuration backup and drift pillar covers the version where the change is to a device's configuration. The version here is a change to the network's shape.


The cure is one line on the change form: "Monitoring told? Yes / No / Not needed, because ___." A change that says "not needed" with a reason is fine. A change that leaves the line blank is not complete. New system intake describes the fuller process for bringing a system into monitoring on purpose. The change-form line is what makes sure intake is triggered at all. Back it with a monthly report of what discovery has seen and monitoring has not:


-- Illustrative: addresses seen by discovery in the last thirty days
-- that have no monitored device. Names differ between products.
SELECT d.ip_address, d.hostname, d.first_seen, d.last_seen
FROM discovered_devices d
LEFT JOIN devices m ON m.ip_address = d.ip_address
WHERE m.device_id IS NULL
  AND d.last_seen >= DATE('now', '-30 days')
ORDER BY d.first_seen;

Every row is a thing that should be monitored, a thing deliberately excluded, or a silent change. Only the last is a problem, and the report finds it before the outage does. It only works if discovery's scan range covers the whole network, so the one change that must never be silent is a new subnet.


Tool sprawl as a process failure

It looks like four monitoring tools. One was there when you arrived. One came with the firewall. One was set up for a project that ended, and one is the good one. Each monitors some of the network. None monitors all of it. An alert about a given device might come from any of them, or two, or none. Monitoring tool sprawl is a pillar of its own. This section makes only one claim about it: sprawl is not a purchasing mistake.


It persists because nobody owns the decision to have one tool. Each tool was adopted by someone with a reason, and removing it means finding that person and having a conversation nobody wants. So the tools stay, and each new project adds one more, because adding is easy and consolidating is a meeting. The process failure is the missing rule, not the surplus software.


The cure is a named owner of "what monitors what" and a written inventory of the tools and the devices each covers. It also needs a rule: no new tool is installed until the inventory says why the existing ones cannot do the job. The rule will be broken. The inventory is what tells you it was.


Mistakes about ownership


The shared inbox

It looks like it-alerts@example.com. Every alert goes to it. Every admin receives it, and every admin assumes one of the others has it. The inbox has thousands of unread messages, which nobody finds embarrassing. That is because it is understood to be where alerts go rather than where they are read. An alert sent to six people has been sent to nobody.


It persists because it is fair. Nobody is singled out. Everybody can say they were told, and it is true, and the alert still went unread. It also persists because the alternative, naming a person, feels like assigning blame in advance. It is the opposite: it is telling one person "this is yours" so that five others can stop half-watching.


The cure is the ownership rule: every alert has exactly one person who is expected to act on it. That person's name is on the rule. The person can be "whoever is on call this week", as long as the rota resolves to a name at every hour of every day. The shared inbox can stay as a copy, an archive, a place to search. It is never the place an alert is sent to be acted on. Humane on-call covers building a rota that people can sustain.


Monitoring nobody owns

It looks like a system everybody uses and nobody is responsible for. The person who built it left. The person who inherited it "keeps an eye on it". Questions about it are answered with "I think", and changes are made by whoever needs one, without telling anyone, because there is no one to tell. When it breaks, the first ten minutes of the incident are spent finding out who can log in.


It persists because ownership of monitoring is unglamorous and unfunded. Nobody's job description says "own the monitoring", so it is done in the gaps by whoever cares most. When that person is busy or gone it is done by nobody. The monitoring bus factor pillar covers what happens when the one person who understands it is unavailable. The anti-pattern here is the step before, where there is not even one.


The cure is a name. One person owns the monitoring system, by written agreement with their manager, with a deputy who can do the weekly tasks. The owner does not have to do everything. They have to be the person who is asked, and who says "that is on the calendar for March". It costs a sentence in a job description, and it is the sentence most organisations have never written.


The review that never happens

It looks like agreement. Everyone agrees the thresholds should be reviewed quarterly, the alerts triaged, the coverage checked against the inventory. It was agreed at the meeting. It is in the minutes, and it has not happened. That is because agreeing that something should happen is not the same as putting it on a calendar with a name next to it.


It persists because the review's cost is immediate and its benefit is invisible. The afternoon spent on the threshold review is an afternoon not spent on the ticket queue. The outage the review prevented never happens, so there is nothing to point to. This is hero culture from the other side: prevention is unpaid, so prevention is postponed. The monitoring standards guide describes writing the rituals down as a standard. The anti-pattern is having the standard and not the calendar.


The cure is the sentence this pillar keeps repeating: calendar it or it does not exist. Every ritual gets a recurring entry, an owner, an attendee, and a place where its evidence goes. A review on the calendar can be missed, and a missed entry is visible, which is the whole point. An agreed review that is not on the calendar cannot even be missed.


The ownership and rituals table


The ownership and rituals table is one row per thing that needs an owner. It records the ritual that keeps it honest, the cadence, and where the evidence goes. It is the smallest document that fixes four of the seven anti-patterns at once. That is because a thing with a named owner, a scheduled ritual, and an evidence column cannot be silently changed, left to nobody, or reviewed never. Copy it, fill in the names, and put every row on a shared calendar.


ThingOwner (name)DeputyRitualCadenceEvidence it happened
The monitoring server  Health check: service, disk, database, backupWeeklyLine in the weekly log
Alert rules  Alert triageQuarterlyTriage table, dated
Thresholds  Threshold reviewQuarterlyReview record with last-reviewed dates
Coverage  Inventory versus monitored devicesQuarterlyGaps list with owners
Discovery scan range  Discovered-but-unmonitored reportMonthlyReport with a decision per row
Notification recipients  Compare against the directoryMonthlyComparison output, all rows resolved
On-call rota  Confirm every hour resolves to a nameMonthlyRota published for the coming month
Dashboards  Panel inventory and pruneQuarterlyPanel inventory, dated
Network map  Walk-through with a colleagueQuarterlyMap updated, date on it
Change-to-monitoring handoff  Every closed change has the "monitoring told?" line filledMonthlyCount of changes with a blank line (target: zero)
Post-incident reviews  Review held; action items name systems, not peoplePer incidentReview document with action items assigned
Prevention log  Near misses and preventions recorded and reportedMonthlySection in the monthly report
Tool inventory  List of tools and what each coversYearlyInventory updated, dated

The deputy column is not decoration. An owner with no deputy is a bus factor of one. The week the owner is on leave is the week the monthly report does not run. The deputy does not need to understand everything. They need to do the weekly tasks and know where the calendar is.


Tip: fill the table in during one meeting with the manager present, and get the names agreed in the room. A table filled in by the monitoring owner alone will have the owner's name in every row, which is the anti-pattern the table is meant to fix. The manager's job in the meeting is to say "no, that one is yours" to someone else.


The site nobody told monitoring about


Carrow Engineering opened a small design office in another town. The network team set it up properly: a firewall, a switch, and a VPN back to head office. The change record read "no impact on monitoring". That was true, because the monitoring server was not touched. Six weeks later the firewall applied a firmware update at six in the morning and the VPN did not come back. Thirty people arrived at eight and could not reach anything. The monitoring, asked afterwards what it had seen, had seen nothing. That was because the site's subnet had never been added to the discovery range. The review recorded that the site "was not in scope". It had never been given the chance to be.


The change form now has the line, and the discovered-but-unmonitored report runs monthly. The first run found a printer, a conference room panel, and a second small office that had opened the previous year.


Hardest to fix, cheapest to fix


Every cure in this guide costs almost nothing. A line on a form. A name on a rule. A recurring calendar entry. A log with two columns. A sentence in a job description. None needs a budget, a project, or a purchase, and all of them together would take a week to put in place.


They are also the hardest, and for the same reason. A purchase is a decision made once. A habit is a decision made every week by people who have other things to do. The habit only holds while somebody is paying attention to it. The threshold review will be skipped the first busy quarter. The fixes are cheap to install and expensive to keep, and the keeping is the job.


I have not found a way round that. What I have found is that the table above holds far longer with real names in it and a manager who has seen it. In that form, it outlasts the same rituals agreed in a meeting and written nowhere. Ownership is a fact about a document. Make the document.


The process checklist


Use this once a quarter, with the ownership table beside it.


  • The prevention log exists, has entries from the last month, and is reported alongside incidents.
  • The last post-incident review's action items name systems, documents, checks, or rituals, not people.
  • The change form has a "monitoring told?" line, and last month's changes have it filled in.
  • The discovered-but-unmonitored report has been run this month and every row has a decision.
  • The discovery scan range includes every subnet the network currently has.
  • Every alert rule names one person or a rota that resolves to one person at every hour.
  • The shared inbox receives copies only; no alert is sent there to be acted on.
  • The monitoring system has a named owner and a named deputy, agreed in writing with their manager.
  • Every ritual in the ownership table is on a shared calendar, and every ritual due last quarter has evidence.
  • The tool inventory lists every monitoring tool in use and what each covers.
  • The ownership table has been reviewed within the last year and the names are current.

Where to go next


Process anti-patterns are where the others come from, and they are fixed with a form, a name, a calendar, and a log. None of it is technical, which is why technical people postpone it. All of it is the difference between a monitoring system that stays fixed and one that needs rescuing every second year.


The field guide names all five families and gives the self-audit. Set-and-forget rot is what the review that never happens produces, and alerting on everything is what the shared inbox hides. For the habits that outlast any particular tool, the monitoring fundamentals that last pillar is the companion to this one. All of these are part of Common Monitoring Mistakes and Anti-Patterns.


Put this into practice with SysPeep Monitor

A tool cannot fix a habit, but it can make the habit easier to keep. SysPeep Monitor auto-discovers devices and interfaces so the discovered-but-unmonitored report is a real report rather than a guess. It sends email and other notifications from its alert rules. It shows device state and alert history from a single web console the owner and the deputy can both read. The free trial installs on a Windows machine if you want to fill in the ownership table against your own network.

Download Free TrialLearn About SysPeep