Onboarding the Next Admin to Your Monitoring in a Week
A new administrator starts on Monday, and by Friday you would like them to be able to run the monitoring without you. Not to know everything; to do the five things that matter, safely, from the documentation, with you as a fallback rather than a requirement. Most onboarding does not achieve this. It consists of a login, a tour of the console, and "shout if you have questions". After that, the newcomer has questions for a year and shouts about none of them.
This guide is a five-day plan that works, on the condition that the README from handover-ready monitoring exists. Each day has one task, done end to end by the newcomer's own hands, with a piece of evidence at the end that proves it was done. The week finishes with the five bus-factor questions from the bus factor of your monitoring answered "yes" and signed. It also finishes with a monitoring system that two people can run.
By the end you will have the plan and a table you can copy to track it. You will have a restore-test procedure for day four and a sign-off checklist. You will also have advice for the other case, where you are the newcomer and there is no README at all.
In this guide
- Why a week is enough
- Day one: the map, the README, and the morning check
- Day two: add a device end to end
- Day three: change a threshold and handle a false alarm
- Day four: restore the database and walk the alert paths
- Day five: the shadow on-call shift
- The five-day plan template
- The sign-off checklist
- If you are the newcomer and there is no README
- Where to go next
Why a week is enough
A week is enough for a competent administrator to run the monitoring day to day, because day-to-day running is five tasks. Each can be learned by doing it once with someone watching. Adding a device, changing a threshold, understanding an alert, restoring the server, and silencing a false alarm are the whole job at the operational level. Everything beyond them, from redesigning the alert routing to planning a migration, can wait for month two.
The condition is the README. Without it, the newcomer learns by asking you, which takes as long as you are available and stops when you are not. With it, the newcomer learns by reading and then doing, and asks you only when the README is wrong. That is how you find out where the README is wrong. Onboarding is the best test the documentation will ever get, and a newcomer who cannot follow it has found a bug, not failed a test.
Each day follows the same shape. Read the relevant README section, and do the task with your hands off the keyboard. Produce evidence, and write a line in the README's change log. The mentor sits beside the newcomer and answers questions with "where does the README say?" before answering with the answer. This is annoying for about a day. On day three the newcomer stops asking and starts reading, and that is the week's real achievement.
Day one: the map, the README, and the morning check
Day one is reading, and then watching. The newcomer reads the map of the monitoring and the README from top to bottom, with the console open in another window. That way every section can be checked against the real thing. Then they shadow the morning check: the daily walk through the console that a monitoring administrator does before anything else. That walk covers what is red, what is amber, what went red overnight and recovered, and what has stopped reporting.
The evidence for day one is a conversation. Ask the newcomer to explain the map back to you from memory. That includes where the server is, what the probes cover, where the backup goes, and how an alert gets from a check to a phone. Then ask them to name the three services the business would notice first if they broke, from the service-to-checks map. If they can do both, the README has done its job. If they cannot, find out which section failed them and fix the section, not the newcomer.
Give them the runbooks to read on day one as well, if any exist. Runbooks as training makes the case that a runbook is the best onboarding document there is. That is because it describes a real task in the order it is done. The newcomer will follow one on day three, so reading it on day one is preparation, not homework.
Day two: add a device end to end
Day two is the first task with the newcomer's own hands. Add a device to the monitoring, from discovery to alert routing, with nothing typed by the mentor. Use a real device if one is waiting, or a test device that can be removed on Friday. The device gets the standard checks for its class and credentials fetched from the credential store. It gets membership of the right groups and alert routing that matches the routing table in the README.
"End to end" is the point, and it is the part that usually gets skipped. Discovering a device is one click. Getting it monitored correctly means knowing which checks its class gets, which is what monitoring standards defines. It also means knowing where the SNMP community lives, which is the README's credential section. And it means knowing who should be told when it fails, which is the routing table. If any of those three steps sends the newcomer to the mentor instead of the document, note it as a README gap.
The evidence for day two is the device in the console with its checks green. The evidence also includes a line in the README change log saying what was added, by whom, and why. Then, before the end of the day, the newcomer breaks the device on purpose, by disconnecting it or blocking its polling. Before the end of the day, they also confirm the alert arrives where the routing table says it should. A device that is monitored but never tested is a device that is assumed to be monitored, and assumptions are what the week is for removing.
Day three: change a threshold and handle a false alarm
Day three teaches the two decisions the newcomer will make most often and get wrong most expensively: changing a number and silencing an alert. Pick a real threshold that needs adjusting, or a test one, and have the newcomer change it. The change is not finished when the number is saved. It is finished when two things are true. The threshold table in the README has a new row with a "why this number" entry. And the rule's description in the tool says the same thing in one sentence.
Then handle a false alarm together. Every monitoring system produces at least one a day, and day three is the day to wait for it. When it arrives, the newcomer works out why it fired. They decide whether the fix is a threshold change, a dependency, a maintenance window, or a temporary silence. Then they apply the fix with the right scope and the right duration. The mentor's job is to ask "what else does that silence?" before the button is pressed. That is because the most common onboarding mistake is a silence that is one level too wide and covers the real alert next door.
The evidence for day three is two README changes: the threshold row and, if a window or exclusion was created, its reason. If the false alarm turned out to be a real alarm, which happens, the evidence is the incident handled and the runbook followed. In that case, the newcomer has had a better day three than planned.
Day four: restore the database and walk the alert paths
Day four is the one people skip, and it is the one that matters most. The newcomer restores the monitoring database from last night's backup onto a test machine and starts the product against it. They confirm the console comes up showing the same devices and checks as production. Backing up the monitoring server covers what to back up and how. Day four is where the newcomer proves the backup is a backup and not a file with a hopeful name.
The procedure differs by database, so the steps below are the shape, not the script. A small install typically uses SQLite, which is a single file; larger ones use MySQL/MariaDB or SQL Server, which need a restore command. Check the product documentation for the exact steps for your product, and do the whole thing on a machine that cannot send real alerts.
RESTORE TEST: illustrative shape, adapt to your product and database
1. On the test machine, install the product with the same installer
the production server uses. Do not start the service yet.
2. Copy last night's backup from the backup target to the test machine.
3. Restore the database:
SQLite: copy the backed-up .db file over the product's data file
MySQL/MariaDB: mysql -u monitor -p monitoring < monitoring-backup.sql
SQL Server: RESTORE DATABASE monitoring
FROM DISK = 'D:\restore\monitoring.bak' WITH REPLACE;
4. Point the product at the restored database (configuration file or
setup screen, per the product documentation).
5. Disable outbound email on the test machine, or point it at a test relay.
6. Start the service. Open the console.
7. Confirm: device count, check count, and alert rule count match
production. Open three devices and compare their thresholds.
8. Record the date, who did it, how long it took, and anything that
was missing, in README section 7.
Step eight is the reason the test exists. The things that turn out to be missing on day four are the things nobody would have found until the real restore. Examples are the service account password that is not in the store or the licence file that lived only on the old server. Another is the mail relay that the README forgot. Every one of them goes into the README the same afternoon. A restore test that finds nothing missing is good news. A restore test that finds three things missing is better news, because it found them on a Thursday.
The second half of day four is walking the alert paths. For every row in the routing table, the newcomer sends a test alert. They confirm it arrives at the right address, in the right form, within the expected time. They also confirm it escalates to the right place if unacknowledged. Do it from the production system with a clearly labelled test rule, and warn the recipients first. The evidence is a tick beside every row of the routing table and a fixed entry for any row that did not work. There is usually one. (It is usually the escalation.)
Day five: the shadow on-call shift
Day five is the on-call phone, held by the newcomer, with you reachable but not present. A shadow shift means the newcomer is the first responder for everything the monitoring sends for a working day. They handle it from the runbooks and the README, and call you only if stuck. You are at your desk or on your phone, and you do not look at the console unless asked. The point is not to test the newcomer. It is to find out what the documentation does not cover when a real alert arrives and the mentor is not in the chair.
Setting up your first on-call rotation covers what a rotation needs to be humane and workable. The shadow shift is the newcomer's first step into it. At the end of the shift, the newcomer writes a handoff note in the form the on-call handoff describes. It records what fired, what was done, what is still open, and what surprised them. That last item is the day's evidence, and it goes into the README as either a fix or a known issue.
If Friday is quiet and nothing fires, that is not a failed day five. Send a test alert at two in the afternoon and have the newcomer handle it as if it were real. Include the runbook, the acknowledgement, and the note. A quiet shadow shift with one drill is a pass. A shift where the newcomer had to call you three times is also a pass, with three README fixes attached.
The five-day plan template
The plan fits in one table. Copy it, put the newcomer's name at the top, and fill in the sign-off column as the week goes. Every row has evidence that either exists or does not, which keeps the week honest and gives both people something to point at on Friday.
| Day | Task | Evidence of completion | Sign-off |
|---|---|---|---|
| 1 | Read the map and the README against the live console; shadow the morning check | Newcomer explains the map from memory and names the three most important services from the service map | Mentor initials, date |
| 2 | Add a device end to end: discovery, standard checks, credentials from the store, group membership, alert routing; then break it on purpose | Device green in the console; test alert received where the routing table says; README change-log line written | Mentor initials, date |
| 3 | Change a threshold with a "why this number" line; handle one false alarm with the right scope and duration | Threshold table row added; rule description updated; false alarm resolved and its cause recorded | Mentor initials, date |
| 4 | Restore last night's database backup to a test machine; walk every alert path with a test alert | Test console shows matching device, check, and rule counts; every routing table row ticked; restore recorded in README section 7 | Mentor initials, date |
| 5 | Shadow on-call shift with the mentor reachable but not present; write a handoff note | Handoff note filed; every question asked of the mentor recorded as a README fix or known issue | Mentor initials, date |
Keep the finished table with the README. It is the record that the monitoring has been handed over once. It is the template for the next time, when it is the newcomer's turn to be the mentor.
The sign-off checklist
The week is signed off when the five bus-factor questions can be answered "yes" for the newcomer, with the evidence from the plan behind each one. Go through them together on Friday afternoon. A "yes" that both people believe is the point. A "yes" to be polite is a bus factor of one with a signature on it.
- The newcomer can add a device with the right checks, credentials, and routing from the README alone. (Day two.)
- The newcomer can change a threshold and record the reason in the right two places. (Day three.)
- The newcomer can find out why an alert exists from the README or the rule's description without asking. (Days one and three.)
- The newcomer has restored the monitoring database to a test machine and the procedure is written down. (Day four.)
- The newcomer has silenced a false alarm with the right scope and duration, and knows what else the silence covered. (Day three.)
- Every alert path has been tested and every row of the routing table is ticked. (Day four.)
- The newcomer has held the on-call phone for a shift and written a handoff note. (Day five.)
- Every README gap found during the week has been fixed or added to the known-issues list.
- The newcomer is in the credential store's access group and has opened at least one entry.
- The completed plan table is filed with the README.
At Harbourview Clinic the new administrator's onboarding ran four days instead of five, because day four was the restore test and the senior administrator was needed elsewhere. "The backups run every night, you'll be fine." Six weeks later the monitoring server's disk failed on a Saturday. The new administrator found the backup files and found that they held the database only. They also found that nobody had written down which installer, which service account, or which mail relay the server had used. The monitoring was back on Tuesday. The review noted that the restore procedure had been "validated in production".
Tip: run the week again, in reverse, a month later: the newcomer mentors you through days two, three, and four using the README. The parts where you correct them from memory rather than from the page are the parts of the README that still say what you meant instead of what is true.
If you are the newcomer and there is no README
Sometimes you are the newcomer, and the previous administrator has gone. In that case, there is no README and no map. The monitoring system hums along on decisions nobody can explain. The five-day plan still works; it just runs slower and produces the README as it goes rather than consuming one. First week survival covers the wider situation of inheriting an undocumented network. The monitoring-specific advice is to start the README on day one with the ten headings and fill in each section on the day that exercises it.
Day one becomes drawing the map from what you can see. Log on to the server, and list the services and scheduled tasks. Find the database and where it is being backed up to, if it is. Day two's device add tells you which template the previous admin used, because the existing devices of the same class will all have the same checks. Day three's threshold change tells you which numbers are not defaults, because the tool will show you. Day four's restore test tells you whether there is a backup at all. That is the single most important fact about the monitoring and the one to establish first if you are nervous. Day five's on-call shift tells you where the alerts go, because some of them will go somewhere odd.
Write "reason unknown" wherever it is true, and change nothing you do not understand until the week is over. A threshold that looks strange is usually strange for a reason; the previous administrator was not a fool, only silent. By Friday you will have a thin README with honest gaps, and a thin honest README is worth more than the thick confident one your predecessor never wrote.
Where to go next
A week is enough because the operational job is five tasks. Each can be learned by doing it once with the README open and a mentor who answers "where does it say?". Read, add, change, restore, hold the phone. Sign off the five questions on Friday, and file the table. Then the monitoring has two people who can run it. That is the whole aim of this pillar and rarer than it should be.
The week consumes the README that handover-ready monitoring describes and the answers extracting tribal knowledge gathers. The handover pack is where the finished plan table and everything it tested end up, ready for the next time. All of them are part of Making Monitoring Survive You: Documentation and Bus Factor.
Put this into practice with SysPeep Monitor
Day four is simpler when the monitoring is one service and one database. SysPeep Monitor runs as a central Windows service with optional remote probes. It keeps devices, dependencies, and alert rules in a single web console. It stores its data in SQLite, MySQL/MariaDB, or SQL Server. So a restore test is a database you can copy to a test machine and start the service against. For the exact restore steps, check the product documentation. The free trial at /download/ makes a fine test machine.
Download Free TrialLearn About SysPeep