Home › Topics › Making Monitoring Survive You: Documentation and Bus Factor › Monitoring Bus Factor

The Bus Factor of Your Monitoring System: Could Anyone Else Run It Tomorrow?


Somewhere in your monitoring system there is a threshold that only you understand. Perhaps it is the disk check on the file server. In that example, it is set to 91 per cent instead of 90 for a reason involving a backup job and a Tuesday. Perhaps it is an alert routed to your personal mobile because the team distribution list was broken the week you set it up. In that case, you never went back. You know these things. Nobody else does. That is a bus factor of one, and this guide is about finding out whether you have one and what it is costing you.


This is not a guide about documenting the network. Documentation that survives already covers the bus-factor test for network documentation in general: could a stranger run the network from what is written down. This guide points the same test at a narrower and stranger target, the monitoring system itself. Its configuration, its checks, its thresholds, its credentials, its alert routing, and the reasons behind each of them. Monitoring is the system that tells you when everything else is broken, and it is almost always the least documented thing in the building.


By the end you will be able to say what bus factor means and why monitoring nearly always scores one. You will be able to run a five-question test on your own setup and list what lives only in your head. You will be able to fill in a self-assessment table you can copy. And you will be able to put an honest shape on what a bus factor of one costs, both to the business and to you.


In this guide

  • Bus factor, defined
  • Why monitoring scores one, even in a team of four
  • The five-question test
  • What lives only in your head
  • The self-assessment table
  • Finding the gaps with a query
  • What a bus factor of one costs
  • The checklist
  • Where to go next

Bus factor, defined


The bus factor of a system is the number of people who would have to be unavailable before nobody could run it. The name comes from the grim question "what if she got hit by a bus?", but buses are the least likely cause. Holidays, resignations, a broken leg, a move to another department, a new baby: all of these take the same person out of the building. The system does not care why. If one person's absence leaves nobody who can run the thing, the bus factor is one. If two people would both have to be away, it is two. Higher is better, and one is the number to be afraid of.


"Run the thing" means more than "log in to it". A colleague who can open the console and watch the tiles change colour has not raised the bus factor. Running a monitoring system means changing it safely. That means adding a device, tuning a threshold, knowing why an alert exists before deciding whether to silence it, and rebuilding it after the server dies. The test in this guide measures those, because those are the things that stop happening when the one person leaves.


A bus factor of one is not a character flaw. It is the natural resting state of any system built by one busy person. You built the monitoring because nobody else would. You tuned it because the alerts were wrong at two in the morning and you were the one awake. Each decision was obvious to you at the time, so you did not write it down. Every monitoring system I have inherited was like this, and so was every one I left behind before I learned better. (I learned better by inheriting one.)


Why monitoring scores one, even in a team of four


Monitoring has a bus factor of one in most teams because one person builds it and the others only receive its output. In a four-person team, four people get the alert emails and four can acknowledge them, and that looks like shared ownership. It is not. Acknowledging an alert is reading; building the alert is writing. Ask who last added a check, who last changed a threshold, and who knows where the database lives. Three of the four will point at the same person.


This happens because monitoring is treated as a tool rather than as a system. A tool is something you use; a system is something you maintain. The team uses the monitoring every day and maintains it never. The exception is the one person who set it up, who maintains it constantly and invisibly. Nobody else needs to touch it, so nobody else learns how. It works so well that it hides its own bus factor.


The knowledge that matters is not how to use the tool; it is why the tool is configured the way it is. The manual tells anyone how to set a threshold. Only you know that the threshold on the warehouse switch uplink is 60 per cent because that uplink is oversubscribed on purpose. Only you know that raising it to the default of 80 would hide the one condition the check exists to catch. Configuration is easy to read. Reasons are invisible unless somebody wrote them down.


Alert routing makes it worse, because the routing itself often points at one person. Escalation paths built by one administrator tend to end at that administrator's phone, and sometimes to start there too. A monitoring system whose every alert path leads to one mobile number has a bus factor of one in the most literal way possible. When that number stops answering, the alerts go into the dark.


The five-question test


The test is this: could a competent colleague, tomorrow, with no phone call to you, do the five things a monitoring system needs done? Not a genius. Not someone who has used your monitoring product before. A competent administrator, with your documentation and a login, and you unreachable. Answer each question honestly, and "probably" counts as "no".


  • 1. Add a device. A new switch arrives. Could they get it into the monitoring with the right checks, the right credentials, and the right alert routing? Could they do it without asking you which template to use or where the SNMP community string lives? (SNMP is the Simple Network Management Protocol, the way most network devices answer questions about themselves; a community string is the password for it.)
  • 2. Change a threshold. A disk alert fires every night at 91 per cent. Could they decide whether to raise it, and to what, from something written down, or would they have to guess?
  • 3. Find out why an alert exists. An alert named WH-CORE-UPLINK-60 fires. Could they find out what it protects and who cares, or would they have to reverse-engineer it from the check definition?
  • 4. Restore the server. The monitoring machine's disk dies on a Sunday. Could they rebuild it from a backup they can find, using a procedure that exists, to the point where alerts flow again?
  • 5. Silence a false alarm safely. An alert is wrong. Could they quiet it for the right duration, for the right device, without also silencing the real alert next to it, and without leaving it silenced forever?

Five yeses is a bus factor of at least two. Five nos is a bus factor of one, and it is the common score. The interesting result is the mixed one. Most monitoring systems pass question one, because adding a device is the thing the manual is best at. They fail questions three and four, because the reasons and the restore are the two things no manual can contain. Those are also the two that hurt most.


Take the test with a real colleague if you can. Sit them down, hand them whatever documentation exists, and leave the room. You will find out in twenty minutes what you have been assuming for two years. If there is no colleague, take it yourself, and be honest about how much of each answer comes from memory rather than from the page.


What lives only in your head


The inventory of what only you know is longer than you think, because you cannot see your own knowledge; it feels like common sense. The list below is what I found on my own monitoring, and every item felt obvious until I tried to point at where it was written.


  • The reasons behind thresholds. Every non-default number in the monitoring was set for a reason, and the reason is in your head. Ninety-one per cent because the backup staging area fills nightly and drains by morning. Sixty per cent because of the oversubscribed uplink. Fifteen minutes because the branch VPN drops for ten and recovers on its own. None of it is in the tool.
  • The credential that is "in the usual place". The SNMP community string, the WMI service account, the SSH key for the firewall. You know where they live. "The usual place" is a location on a map only you can read. Discovery credentials covers how to keep them properly; the bus-factor question is only whether somebody else could find them.
  • The dependency nobody wrote down. The monitoring server polls the branch through a firewall rule a previous admin added by hand. The remote probe depends on a VPN tunnel that is not itself monitored. The alert emails leave through a mail relay nobody else knows the monitoring uses.
  • The alert routed to your personal phone. The one you set up "for now". It is still there.
  • The maintenance window that exists because of a job. Every Sunday from one to three the file server checks are suppressed, because that is when the backup runs and the disk goes to 98 per cent. Delete the window and the alerts return, and nobody will know why.
  • The device that is deliberately not monitored. The lab switch, the retired print server, the thing in the cupboard. Excluded from discovery on purpose. The next person will add it and wonder why it alarms.
  • How the monitoring is backed up, and whether it is. You know the database is copied nightly to the file share. You know because you set it up. It is not in any backup schedule anyone else reads.

The common thread is that every item was a sensible decision made under time pressure by someone who intended to write it down later. Later is a place where a great deal of documentation lives.


The self-assessment table


The self-assessment is a table with four columns: the task, whether someone else could do it today, where it is written, and the gap. Fill it in yourself first, honestly, and then have a colleague check your "yes" answers by trying them. The "where is it written" column is where most rows fall over, because the honest entry is so often "nowhere" or "in my head". Copy the table, replace the rows with your own routine tasks, and keep it; it becomes the first page of the handover pack.


TaskCould someone else do it today?Where is it writtenGap
Add a new switch with standard checksYesProduct manual; internal template listNone
Add a new server with WMI checksPartlyService account name in password store; which template to use is not writtenDocument the template choice and why
Change the file server disk thresholdNoNowhere; reason in my headAdd a "why this number" line to the README
Find why alert WH-CORE-UPLINK-60 existsNoNowhereAlert reasons list
Restore the monitoring server from backupNoBackup job exists; no restore procedureWrite and test the restore procedure
Silence a false alarm for one device for four hoursYesProduct manualNone
Find the SNMP credentials for the core switchesPartlyPassword store, but only I have accessGive the on-call group access to the store
Know who is notified for each alert groupNoOnly visible inside the tool, rule by ruleAlert routing table in the README
Know which devices are excluded on purposeNoNowhereExclusions list with reasons
Know which maintenance windows exist and whyPartlyWindows visible in the tool; reasons not writtenAdd a reason to each window

Score it by counting the rows that are not "yes". The count is the size of the job in front of you, and the gap column is its to-do list. Handover-ready monitoring covers closing the gaps in the order that matters; most of them are an afternoon each, not a project.


Finding the gaps with a query


The monitoring tool can help you find undocumented reasons, if it stores a comment or description with each check. Most products let you attach a note to an alert rule or a sensor, and most administrators leave it blank. A list of every check with no note is a list of every reason that lives only in your head. Table and column names differ between products, so treat this as the shape of the query rather than the query. Check the product documentation for yours, and run it against a copy of the database rather than the live one.


-- Illustrative: every check with no description attached
SELECT device_name, check_name, warning_threshold, critical_threshold, description
FROM checks
WHERE description IS NULL OR description = ''
ORDER BY device_name, check_name;

What comes back is the raw material for the inventory. A second useful pass lists every check whose thresholds differ from the template it came from. Those numbers were chosen on purpose, and numbers chosen on purpose have reasons:


-- Illustrative: checks whose thresholds were changed from the template
SELECT c.device_name, c.check_name,
       c.warning_threshold, t.warning_threshold AS template_warning
FROM checks c
JOIN check_templates t ON c.template_id = t.id
WHERE c.warning_threshold <> t.warning_threshold
ORDER BY c.device_name;

If your product has no queryable database, export the alert rules to a file, or print the list, and read it with a colleague. Even a plain list of alert names read aloud will surface the ones nobody can explain. Those are the ones to write down first, while the person who can explain them is still in the room.


Tip: the fastest way to shrink the inventory is to make the tool hold the reason. If your monitoring lets you put a description on a check or an alert rule, write the reason there, in one sentence, every time you change a number. The reason then travels with the check, survives a restore, and is on screen when the alert fires, which is exactly when the next person needs it.


What a bus factor of one costs


To the business, a bus factor of one means the monitoring will be fine until the day you are not there. Then it will degrade in a way nobody notices. Alerts keep going to the wrong place. A device gets added without the right checks. A threshold is loosened to make a noisy alert stop, and the condition it protected against arrives unannounced. Nothing visible breaks. The system simply stops being trusted, and untrusted monitoring is furniture with a login.


The cost has an awkward shape: small every day and enormous on one day. On the day you leave, or the day you are in hospital, the business discovers that its early-warning system for everything has no one who can run it. That is the worst possible time, because the monitoring is needed most exactly when it is being asked questions nobody can answer.


To you, a bus factor of one is the holiday you cannot take without your phone. It is the promotion you cannot accept because nobody can cover your desk. And it is the resignation that turns into six weeks of handover. Surviving vacations covers the mechanics of getting away when you are the only admin. The point here is that the monitoring is usually the thing that makes leaving hardest, because it is the thing that will call you. Being indispensable feels like job security. It is a cage that happens to fit well.


If the self-assessment has more gaps than you can close alone in a quarter, that is a workload signal, not a personal failure. When to get help turns that signal into a case for a second pair of hands, and a table full of "no" is persuasive evidence.


At Meridian Parts the monitoring had been built by one administrator over four years. Every critical alert escalated to his mobile number after fifteen minutes unacknowledged. He left on good terms, handed back the company phone, and the phone was wiped and reissued to a new sales representative. For seven weeks every escalated alert at Meridian went to a device whose new owner had been told to ignore "the IT spam". The team found out when a storage array lost a controller on a Friday night and nobody was told until Monday. The review described the escalation path as "historically configured".


The checklist


Use this now, and again whenever someone joins or leaves the team. The bus factor is not above one until every line is a yes.


  • You know the bus factor of your monitoring and can say the number out loud.
  • The five-question test has been taken, by you or by a colleague, and the answers are written down.
  • The self-assessment table exists, with a row for every routine monitoring task.
  • Every row that is not "yes" has an entry in the gap column.
  • No alert path ends at a personal phone number.
  • Every credential the monitoring uses is in a store that at least two people can open.
  • The list of what is deliberately not monitored exists, with a reason per item.
  • The reasons behind non-default thresholds have been listed, even if only as a first draft.
  • Every maintenance window has a written reason.
  • A restore procedure for the monitoring server exists, or "write it" is the first item on the gap list.
  • A date is on the calendar for the next time the test is taken.

Where to go next


A bus factor of one is not a sign that you did something wrong. It is what happens when one competent person builds something useful and is never given the afternoon to explain it. The cure is not heroics. It is a list of what only you know and a table that says where each thing is written. And it is the discipline to move items from "in my head" to "on the page" one at a time.


The rest of this pillar closes the gaps in order. Handover-ready monitoring covers the four things to write down and gives you a README template. Extracting tribal knowledge is the interview that gets the reasons out of a head and onto the page. Onboarding the next admin is the week-long plan that proves the documentation works. And the handover pack is the checklist you could hand over today. All of them belong to Making Monitoring Survive You: Documentation and Bus Factor.


Put this into practice with SysPeep Monitor

A monitoring system is easier to hand over when everything about it lives in one place. SysPeep Monitor keeps devices, checks, dependencies, and alert rules in a single web console on a central Windows service. It has optional remote probes for other sites and stores its data in SQLite, MySQL/MariaDB, or SQL Server. So the whole setup is something a colleague can back up, restore, and be walked through. For exporting a list of checks or rules, check the product documentation. The free trial is at /download/ if you want to see the console first.

Download Free TrialLearn About SysPeep