Extracting Tribal Knowledge Before It Leaves the Building
A colleague has handed in their notice. Somewhere in their head is the reason the warehouse switch alerts at 60 per cent. Also in their head is the name of the person to call when the label printers stop. And there is the fact that the Sunday maintenance window exists because of a backup job nobody else has heard of. In four weeks all of that walks out of the door with them. It will not be in the monitoring tool, because tools store settings, not reasons. It will not be in the documentation, because it was never written down. It will be gone, and the first you will know of it is an alert nobody can explain.
This guide is about getting that knowledge onto the page while the person who holds it is still in the building. It is also about getting your own out of your own head before you forget you ever had it. The bus factor of your monitoring is the test that shows how much lives in one head. And handover-ready monitoring is the document it should live in instead. This is the extraction: the interview, the habit that stops the problem recurring, and a twenty-question list you can print.
By the end you will know where tribal knowledge hides in a monitoring system. You will know how to run a console-side interview that gets it out in an afternoon. You will know how to make the "why is this alert here" pass a quarterly habit and how to annotate as you go. And you will know what to do when the person you needed to interview has already gone.
In this guide
- Tribal knowledge, defined
- Where it hides in a monitoring system
- The console-side interview
- The "why is this alert here" pass
- Annotate as you go
- The question list
- When the person has already left
- The checklist
- Where to go next
Tribal knowledge, defined
Tribal knowledge is what people know and nobody wrote down. It is passed by conversation, by watching someone work, and by being the person who was there when the thing happened. It is real knowledge, often the most important knowledge in the department. It has one flaw: it lives in people, and people leave. A team can run for years on tribal knowledge and never notice, because the knowledge is always available. Then one person resigns and the team discovers how much of the monitoring was held together by somebody remembering.
In a monitoring system, tribal knowledge is almost entirely about reasons. The settings are visible to anyone who opens the console: this check, this threshold, this notification group. What is invisible is why. Why this device and not that one? Why 60 rather than 80? Why does the alert go to a named person instead of the team? Why is Sunday from one to three quiet? Every one of those "why"s was a decision, and the decision-maker remembers it, and nobody else does.
Extracting it is not a matter of asking "can you document everything before you go?" That question produces a document written on the last Friday, from memory, in a hurry, by someone whose mind is already on the next job. The technique below sits at the console and walks through the system item by item, because the console is the prompt. People cannot remember what they know in the abstract. They remember perfectly well when the alert rule is on the screen in front of them.
Where it hides in a monitoring system
Tribal knowledge hides where a setting looks arbitrary but is not. Once you know the shapes, you can find them in any monitoring system in an hour.
Alert names that are jokes, initials, or history
An alert called Steve's thing tells you nothing unless you know Steve. An alert called DO NOT DELETE tells you it matters and nothing about why. Names like these were clear to the author on the day, and every day since they have grown a little more opaque. A monitoring system that has been through three administrators is a small archaeological site, and the alert names are the pottery.
Thresholds set after an incident nobody remembers
The most valuable thresholds in the system are the ones tuned after something went wrong. And they are the ones with no explanation, because the explanation was the incident and everyone was busy. A disk check at 91 per cent, a latency check at 180 milliseconds, a queue-length check that exists at all: each is the scar of a bad day. The next person sees a strange number and nothing else.
Devices excluded from discovery on purpose
Auto-discovery finds everything, so a device that is not monitored was excluded by someone, for a reason. The reason might be "it is a lab switch and alarms all day", or "it belongs to the tenant upstairs", or "polling it crashes it". The exclusion survives; the reason does not, and the next administrator, seeing a gap, will helpfully add the device back.
The maintenance window that exists because of a job
A recurring quiet period on a set of checks is nearly always covering for a scheduled job. That might be the backup that fills the staging disk, the batch run that pins the CPU, the nightly reboot of a device that needs one. Delete the window and the alerts come back. Delete the job and the window keeps silencing checks that no longer need it. That is worse, because it hides real failures at exactly the time nobody is watching. Maintenance windows and suppression covers designing them; the tribal-knowledge problem is only that the reason for each one lives in a head.
The contact who is "the printer guy"
Every monitoring system has alerts that go to, or are acted on by, someone outside the team who is known by a role rather than a name. The printer guy. The person at the ISP who actually answers. The warehouse supervisor who wants to know when the scanners drop. These contacts are in one administrator's phone and nowhere else. When that phone leaves, the alert still fires and nobody knows who it was for.
The console-side interview
The console-side interview is two people, one screen, and every alert rule in turn. You sit next to the person who holds the knowledge, open the console, and go through the alert rules from the top. You ask three questions of each. It takes an afternoon for a few hundred rules and gets more out of a person than any form, because the screen does the remembering.
The three questions are the same for every rule, and you ask them in this order:
- Why is this here? What was it added to catch, and was there an incident behind it? If the answer is "I don't know, it came with the template", write that; it is a real answer.
- What happens if it fires? What is actually broken when this goes red, what is the first thing to check, and is there a runbook?
- Who cares? Which service does it protect, which person or department feels it, and who should be told?
Record the answers in the README as you go, not afterwards. One person asks and types; the other talks and points. If your monitoring lets you add a description to a rule, type the "why" straight into the rule while it is on screen. That way, the reason lives with the check and survives a restore. The README gets the longer story: the incident, the person, the job the window is for. The interview is finished when the last rule has three answers or an honest "nobody knows".
Do the same for the parts of the system that are not alert rules: the excluded devices, the maintenance windows, the notification groups, and the credentials. Each has a "why" and a "who". The question list later in this guide covers those. And the interview is the moment to ask them. A leaving colleague in front of a console is the most cooperative source of documentation there is. (A leaving colleague with a form is the least.)
Prioritise if there is not time for everything. Rules that have fired recently matter more than rules that never have. An export of the rule list sorted by last-fired time gives you the order. The query shape is below; names differ between products. So check the product documentation for yours, and query a copy of the database rather than the live one.
-- Illustrative: alert rules, most recently fired first, with their notes
SELECT r.rule_name, r.severity, r.notify_group, r.description,
MAX(a.fired_at) AS last_fired, COUNT(a.id) AS fired_count
FROM alert_rules r
LEFT JOIN alert_history a ON a.rule_id = r.id
GROUP BY r.rule_name, r.severity, r.notify_group, r.description
ORDER BY last_fired DESC;
Rules with a blank description and a recent last-fired date go first. Rules with a blank description that have never fired go last. The interview may conclude they should not exist at all, which is also worth writing down.
The "why is this alert here" pass
The interview should not wait for a resignation. Run it on yourself, once a quarter. Open the rule list and walk down it. Ask the three questions of every rule that has no description or whose description you no longer believe. You are interviewing your past self, who was busy and did not write things down. And the past self is the harder witness because they are not in the room. Do it while you still remember; "still" is shorter than you think.
The pass pairs naturally with reviews other pillars describe. The alert lifecycle covers how an alert is born, tuned, and retired. And the "why is this here" pass is the moment in that lifecycle where an alert with no reason gets one or gets retired. The threshold review ritual looks at the numbers; this pass looks at the reasons behind them. If you already run one of those, add the three questions to it rather than adding a meeting.
Keep a tally of the results, because the trend is the point. First quarter: forty rules with no reason. Second: twelve. Third: two, and both are candidates for deletion. When the tally reaches zero, the pass takes ten minutes. It consists of checking that the rules added since last time have their reasons. They will, because of the habit in the next section. A monitoring system with a reason on every rule is a rare and pleasant thing.
Annotate as you go
The habit that stops the pile building up again is simple. Every change gets a note, at the moment of the change, in the place the next person will look. Where the tool allows a comment or description on a check, a rule, a window, or an exclusion, the note goes in the tool. Where it does not, the note goes in the README. Either way it is written before the console is closed, because a note written "later" is written by the next quarter's pass, if at all.
A good note is one sentence with a reason, and optionally initials. "Raised to 91%: backup staging fills to 89% nightly, drains by 06:00. JJ." "Excluded: lab switch, alarms constantly, owned by the training team." "Window Sun 01:00-03:00: file01 backup job drives disk to 98%." Nobody needs an essay. They need the one fact they could not have guessed from the setting.
The rule generalises to everything the monitoring touches. Runbooks stay alive by the same mechanism, which keeping runbooks alive describes. A runbook updated at the moment it is used stays true, and one updated at review time drifts. The note on a check is the smallest runbook there is, and it lives or dies by the same habit.
Tip: if your tool has a description field on alert rules, make "description is not blank" a condition of the change being finished. It costs fifteen seconds per rule. Six months later, the description is on screen when the alert fires, which is the only moment anyone has ever wanted to know why an alert exists.
The question list
The question list is the printed version of the interview. It covers the parts of the system that are not alert rules. It also covers the questions you will forget to ask when the console is in front of you. Twenty questions, grouped by area. Use it with a leaving colleague, use it on yourself, and file the answers in the README under the matching sections.
HANDOVER INTERVIEW: QUESTION LIST
The system
1. Which machine is the monitoring server, and what else runs on it?
2. Where is the database, what type is it, and where does the backup go?
3. Which probes or collectors exist, where are they, and what do they cover?
4. What does the monitoring depend on that is not itself monitored
(mail relay, VPN tunnel, DNS, a firewall rule, a service account)?
5. Has the server ever been restored from backup? Where is the procedure?
Checks and thresholds
6. Which checks were added after an incident, and which incident?
7. Which thresholds are not defaults, and why is each one the number it is?
8. Which devices are deliberately excluded from discovery or monitoring, and why?
9. Which checks are known to be unreliable, and what is the workaround?
10. Which checks would you delete if you had the time?
Alerts and routing
11. For each alert group, who is notified first, and who after that?
12. Which alerts go to a personal phone or address rather than a group?
13. Which alert names mean something other than what they say?
14. Which maintenance windows exist, and what job or event is each one for?
Credentials and access
15. Where does each credential the monitoring uses live, and who can open it?
16. Which credentials are shared with other systems, so that changing them
breaks something else?
17. Who has administrator access to the console, and who should?
People and history
18. Who outside IT cares about which alerts, and how are they contacted?
19. What is the thing you always meant to fix?
20. What would you tell your replacement on their first day that is not
written anywhere?
Question twenty is the one to ask last and to leave the most time for. The answer is usually the most useful paragraph in the whole handover. And it is nearly always something the person did not think counted as documentation. Question nineteen is a close second; the thing someone always meant to fix is a known issue with a diagnosis attached, which is a gift.
At Bellweather Logistics there was an alert rule called DO NOT DELETE, with no description. It was watching a TCP port on a server whose name nobody recognised. The rule's author had left long before. Three administrators in turn had opened it, read the name, and closed it again. When the unrecognised server was finally decommissioned, the alert fired. It emerged that the port was a forwarder somebody had set up in an emergency. It was the only thing telling the warehouse label printers where their print server was. The label printers stopped for a day and a half. The rule had been correct all along; only its reason had left the building.
When the person has already left
Sometimes there is nobody to interview. The administrator left before anyone thought to ask, or left in a hurry. Or they left long ago and the monitoring has been running on momentum since. The knowledge is gone, and what remains is evidence: the settings, the alert history, the change dates, and whatever the old emails hold. Extracting knowledge from evidence rather than from a person is slower and less certain, but it is possible, and other guides cover it in depth.
Predecessor landmines is the mirror image of this guide. It describes what the next person suffers when the extraction never happened, and how to find the undocumented decisions before they go off. Credential archaeology covers recovering the passwords and keys a departed administrator left behind. For a monitoring system, that usually means the SNMP strings and the service account. Read those if you are the successor. Read the rest of this guide if you would prefer nobody ever had to.
Even without the person, the three questions still work; they just take longer. "Why is this here?" becomes a search of the alert history for the first time it fired and the ticket or email from that week. "What happens if it fires?" becomes a careful look at what the check measures and what depends on it. "Who cares?" becomes asking department heads which alerts they have ever been told about. Write down what you find and what you could not. The note "reason unknown, change with care" is more useful to the person after you than a blank.
The checklist
Use this when someone gives notice, and once a quarter on yourself. It is done when every line is a yes.
- The console-side interview has been scheduled within the first week of the notice period, not the last.
- Every alert rule has been asked "why is this here, what happens if it fires, who cares". The answers are in the README or in the rule's description.
- Every alert rule with a name that is a joke, initials, or history has been renamed or given a description that explains it.
- Every non-default threshold has a reason recorded, or "reason unknown" recorded honestly.
- Every excluded device has a reason recorded.
- Every maintenance window names the job or event it exists for.
- Every alert that goes to a personal phone or address has been re-routed to a group.
- Every outside contact known by role has a name and a way to reach them in the README.
- The twenty questions have been asked and the answers filed under the matching README sections.
- The answers to questions nineteen and twenty have been added to the known-issues list.
- The "why is this alert here" pass is on the calendar for next quarter.
- "Description is not blank" is the rule for every new or changed alert rule from today.
Where to go next
Tribal knowledge is not a failure of the people who hold it. It is what happens when competent people make good decisions faster than anyone can write them down. The fix is not a bigger form. It is a screen, two chairs, three questions, and the habit of writing the reason at the moment of the change. Do the interview once and the backlog is cleared. Keep the habit and it never returns.
The knowledge you extract goes into the README that handover-ready monitoring describes. Onboarding the next admin is the week that tests whether what you extracted is enough. And the handover pack is where the interview answers end up living. All of them are part of Making Monitoring Survive You: Documentation and Bus Factor.
Put this into practice with SysPeep Monitor
The console-side interview is easier when every rule is in one place. SysPeep Monitor keeps devices, checks, dependencies, and alert rules together in a single web console on a central Windows service. It has optional remote probes for other sites. So walking down the whole list with a colleague is one screen rather than several. For exporting alert rules to a file, or adding notes to a rule, check the product documentation. The free trial is at /download/.
Download Free TrialLearn About SysPeep