Your First Five Runbooks
Somewhere in your organisation there is a wiki page called Operational Procedures. It has a heading, a table of contents, and nothing under it. I know, because I have created that page myself, at two different jobs, after being told to "document all your operational procedures." That sentence has produced more empty wikis than any other in IT. "Everything" is not a task you can schedule. It has no first step and no finish line, so it loses to real work every week. And the usual result is zero runbooks and a vague sense of guilt about them. If that describes your team, nothing is wrong with you. The advice was wrong.
Here is the version that works: write five. Five is a task. Five fits in five weeks and covers a surprising share of your real incidents. It builds the habit that produces the sixth and the sixteenth without anyone scheduling them. My first runbook was one line long. It said which modem to power-cycle, and it was used the following week.
This guide shows you how to choose your five (not a generic list, though I will give you strong candidates). It shows you how to write each one fast. Then it explains how to run a five-week sprint that ends with all five tested and wired to their alerts.
In this guide
- What a runbook is (and is not)
- Why five beats "document everything"
- The four selection filters
- The usual suspects: five candidates for a small shop
- How to write one fast
- The five-week sprint
- What five runbooks change
- The first-five toolkit
What a runbook is (and is not)
A runbook is a written, step-by-step guide for handling one specific operational problem. It explains what to check first and what to do for each likely cause. It says when to give up and call someone, and how to confirm the problem is really gone. It is written for one reader — the person responding — at one moment: during trouble, probably under stress, possibly at 3 a.m. Think of it as the difference between an alert and a plan. The alert says "the internet is down." The runbook says "here is what we do about that."
A runbook is not general documentation. Your network diagram, your server inventory, and your "how we set up the VPN" notes are all valuable, but they describe how things are. I have stood in front of a very good network diagram during an outage; it was beautiful, and it was no help at all. A runbook describes what to do, in order, when a specific thing breaks. Keeping that distinction sharp is what makes runbooks short enough to write and usable enough to follow. (For the full internal structure of a good one, see the anatomy of a runbook that works under stress. This guide uses a simplified starter version so you can get moving today.)
Why five beats "document everything"
Incidents are not evenly distributed. A handful of failure types generate most of the responses your team actually performs. Think of the flaky internet link, the disk that keeps filling, and the backup job that dies on Sundays. Write runbooks for that handful and you have covered most of the nights and weekends that matter, for a few hours of writing. The long tail of rare, exotic failures can wait, because by definition it rarely shows up.
Five is also small enough to do well. Each of your first five can be tested by a second person, wired to its alert, and actually trusted. That matters. The first time someone follows a runbook and it works, your team's relationship with documentation changes. And five is how the habit starts. Once a template exists and the value is proven, writing a quick runbook after the next new incident stops being a project. It becomes a reflex. The sixth runbook writes itself. The trick is getting to five.
The four selection filters
Which five? Not the five most interesting problems, and not the five scariest. Run every candidate through four filters. The winners will usually be obvious by the second one.
Filter 1: Frequency
A runbook pays rent every time it is used. The alert that fires monthly beats the alert that fired once, ever, even if the one-timer was more dramatic. Frequency is also the filter you do not have to guess about. Your monitoring system's alert history is a ranked list of what actually happens on your network. If you already hold a periodic alert review, the "fired most often" list from that meeting is literally your shopping guide.
Filter 2: Fumble cost
Some problems are forgiving: if your first move is wrong, you lose ten minutes. Others punish a wrong first move badly. Think of restoring a backup over live data or rebooting the one box that everything else depends on. Another example is power-cycling a RAID array while it is rebuilding a failed disk. A RAID array is a group of disks that act as one, with spare copies of the data. Where a fumble is expensive, a runbook is cheap insurance: it puts the safe order of operations on paper before anyone is improvising near the edge. Without it, the safe order lives in one person's memory, and that person is not always the one holding the phone.
Filter 3: 3 a.m. likelihood
A rested expert at 2 p.m. can improvise. A woken human at 3 a.m. should not have to. Judgment, memory, and typing accuracy all degrade when you are half asleep. That is precisely when networks like to fail. Problems that tend to arrive at night (WAN outages, backup failures, disk-full alerts from overnight jobs) need scripts more than problems that politely wait for business hours.
Filter 4: Delegate value
Ask: "what would let the junior admin, the backup person, or whoever covers my vacation handle this without calling me?" A runbook is the answer for exactly one problem at a time. This filter matters double if you are trying to get a newer teammate through their first on-call rotation. Every runbook you write is one more incident they can own instead of escalating at the first sign of trouble.
The usual suspects: five candidates for a small shop
Run those filters over a typical small or mid-size network and the same candidates surface again and again. Treat this as a menu, not a mandate — your alert history outranks any generic list.
- Internet / WAN down. (WAN — wide area network — is the link between your site and the world.) High frequency at most shops, guaranteed user impact, and it loves the night shift. The response also has real branches — cable, modem, provider — that are easy to fumble in a panic.
- The critical server down. Whichever box would generate an instant flood of complaints — the domain controller, the file server, the main application server. It has high fumble cost (rebooting it wrong, or rebooting the wrong thing) and high delegate value. "The server is down" is the incident a backup person most fears facing alone.
- Disk space critical. Probably the most frequent serious alert in existence. The response is mechanical enough to delegate completely: find what grew, clear the known offenders, confirm the trend. And mechanical responses are what runbooks capture best.
- Backup failure, morning response. The alert arrives with your coffee. The runbook turns "ugh" into a ten-minute routine: which job, which error class, rerun or investigate, and what counts as "we are safe for tonight." Low drama, high frequency, enormous delegate value.
- Phone / VoIP trouble. (VoIP — voice over IP — means your phones are network devices now.) When phones break, management appears at your desk in person. A runbook keeps the first responder moving through the likely causes — handset, switch port, PoE power, provider — while everyone watches.
Now pick your five. Open your alert history, sort by count, and let the fired-most list argue with the menu above. If printer-queue meltdowns genuinely dominate your months, then yes — a printer runbook makes your five. The filters are the judges; the history is the evidence.
One warning about what not to pick: resist the pull of the memorable one-off. Every shop has a legendary outage — the storage array that died spectacularly three years ago. It is tempting to write that runbook first because the memory still stings. But a runbook for a once-ever event will sit unused, untested, and quietly rotting, while the boring monthly disk alert keeps getting improvised. Frequency filters out drama in favor of rent-paying documents.
At Tollgate Foods the first runbook anyone wrote was for the storage array, because the memory of the day it died still stung. It ran to thirty-four pages, had three diagrams, and was signed off by two managers. The array was replaced the following spring and the runbook was never opened again. In the same year the file server's disk-full alert fired eleven times and was handled eleven different ways, one of which cleared the wrong folder. The review recorded that the team "had documentation". Save the legend for runbook eleven.
How to write one fast
The biggest obstacle between you and five runbooks is the blank page, so remove it. Start every runbook from the same starter template — five headings and a metadata line. Then writing becomes filling in blanks instead of composing an essay:
# RUNBOOK: [name of the alert this serves] Owner: [you] Last tested: [date] Applies to: [device / service] 1. VERIFY — is it really this problem? [ ] Check ___ (you should see ___) If not: stop. This is a different incident. 2. DIAGNOSE — which kind is it? [ ] Check ___ -> if X, go to Fix A. If Y, go to Fix B. 3. FIX A. [exact command or click-path] (you should see ___) B. [exact command or click-path] (you should see ___) 4. ESCALATE — after 30 min without progress, or if [condition]: Call ___ (number lives in ___). Say what you already checked. 5. CONFIRM — prove it is fixed [ ] ___ back to normal for 15 minutes. Log what happened in ___.
With the template in hand, three writing methods get you from zero to a usable draft in under an hour each.
Write the next one during (or right after) the next incident
The freshest-memory rule: thirty minutes of writing immediately after an incident beats three hours of writing a month later, cold. Right after the fix, the commands are still in your shell history. The error text is still on screen, and the dead ends you hit are still vivid. The incident just did your research for free. So make it a standing rule: whenever an incident from your five-list gets resolved, the resolver spends thirty minutes filling in the template. That happens before they touch anything else. You paid for that knowledge with an outage; collect it while it is warm.
The interview: veteran narrates, someone types
Some of your five have never been written down because the response lives entirely in one veteran's head. And veterans famously "don't have time to document." They do, however, have time to talk. Book one hour: the veteran narrates how they handle the problem, walking through it out loud. Meanwhile, someone else types into the template and asks the naive questions ("wait, how do I know which modem?" "what does that error actually look like?"). The naive questions are the point — they force out the tribal knowledge the veteran no longer notices they have. One hour, one runbook, and the knowledge finally exists outside one skull.
Good enough now beats perfect later
Version one of a runbook is bullet points that work. Not polished prose, not every edge case, not pretty — just the checks in the right order and the commands that actually fix things. Polish comes from use: every time the runbook runs, someone notices a missing step or an unclear line and fixes it. A rough runbook that exists gets improved; a perfect runbook that is still "in progress" helps nobody at 3 a.m. Ship the bullet points.
The five-week sprint
One runbook a week, for five weeks. A week per runbook is deliberately loose — the writing itself is an hour or two. The schedule has to survive contact with your actual job. And each runbook is not done when it is written. It is done when two more things happen.
First, someone other than the author tests it. Call it the stranger test: a teammate walks through the runbook while the author stays silent. Ideally, choose the least experienced person who might ever face this incident. Test against the real system where safe, as a tabletop read-through where not. Every place the tester hesitates, asks a question, or guesses is a defect to fix. Authors are the worst judges of their own runbooks, because their brains silently fill every gap. The same principle that makes documentation drills work makes this test non-negotiable. A runbook only counts toward your five when a stranger has driven it.
Second, the runbook gets wired to its alert. A runbook nobody can find during the incident it serves is a decoration. As each runbook lands, edit the matching alert so the notification itself carries the link. The "now what?" line of a well-designed alert is exactly where it belongs. The responder's path should be: phone buzzes, alert names the problem, alert links the runbook, runbook opens to step one. No searching, no "where does Dave keep those docs," no wiki archaeology at 3 a.m.
Tip: announce the sprint out loud — "five runbooks, five weeks, here's the list" — even if your team is two people. A public finish line turns a vague good intention into a small project with a scoreboard. It invites the friendly nagging that gets week four's runbook written during week four.
What five runbooks change
Five documents sounds like a small thing. Here is what actually becomes observably different once they exist, are tested, and are linked from their alerts.
- Night responses follow the same path no matter who responds. The 3 a.m. WAN outage gets the same ordered checks from the junior on their second rotation as from the veteran. Both are reading the same page. Outcomes stop depending on who happened to be holding the phone.
- Escalations arrive with context instead of panic. When a runbook's escalate step fires, the senior person gets a call: "I ran the WAN runbook. I verified the link is down and power-cycled the modem twice. No address on the WAN interface." The caller has steps already tried and evidence in hand. That call is ten minutes shorter and infinitely less annoying than "the internet's broken, help."
- Vacation coverage actually works. The backup person covering your week off can genuinely handle the five most likely incidents. They no longer just own a pager they secretly plan to answer by calling you at the beach.
- Responders are calmer, so they make fewer new problems. Stress narrows thinking; a script widens it back out. This is why pilots — who understand their aircraft deeply — still fly emergencies from printed checklists. A responder following steps is a responder not improvising a second outage on top of the first.
The first-five toolkit
Everything above, compressed into three tables you can use this week. First, the selection worksheet — list your candidate incidents (start from the fired-most list in your alert history) and score each filter roughly high/medium/low. You are not computing a formula; you are making the argument visible. Pick the five best rows.
| Candidate incident | How often? | Fumble cost | Arrives at night? | Would a runbook let the backup handle it? | In the five? |
|---|---|---|---|---|---|
| WAN / internet down | Monthly-ish | Medium | Often | Yes | Likely |
| Critical server down | Rare but real | High | Sometimes | Yes | Likely |
| Disk space critical | Monthly or more | Low-medium | Often | Completely | Likely |
| Backup job failed | Monthly or more | Medium | Morning after | Yes | Likely |
| Phones / VoIP trouble | Varies | Medium | Business hours | Mostly | Your call |
| Your recurring special | Check history | ? | ? | ? | Your call |
Second, the starter template as a checklist of what each section must contain. (The full anatomy guide deepens each of these once your five exist.)
| Section | Its job | Done when it contains… |
|---|---|---|
| Header | Maintainability | Owner, last-tested date, what it applies to |
| Verify | Right runbook? | A two-minute check plus "if not X, stop — different incident" |
| Diagnose | Which cause? | Ordered checks with "if X then Fix A" branches |
| Fix | The actions | Exact commands / click-paths, each with what you should see after |
| Escalate | The exit | A time-box, a trigger condition, who to call and what to tell them |
| Confirm | Really fixed? | Proof of normal, how long to watch, where to log what happened |
Third, the sprint plan itself.
| Week | Do | Counts as done when |
|---|---|---|
| 0 (prep) | Pull the fired-most list; run the filters; pick five; announce the sprint | The five are named, each with an owner |
| 1 | Write runbook #1 (start with the most frequent incident) | Stranger-tested and linked from its alert |
| 2 | Write #2; fix anything the week-1 test exposed | Same bar |
| 3 | Write #3 (use the interview method for the one that lives in someone's head) | Same bar |
| 4 | Write #4 | Same bar |
| 5 | Write #5; do a 30-minute retro: what did testing teach you? | All five tested, linked, and announced to the team |
Where to go next
Five weeks from now you can have the thing most shops never get: a tested, findable script for each of your five most likely bad days. They do not have to be long, and they do not have to be dignified. A printer runbook that gets used beats a storage-array epic that does not. Mine started as one line about a modem, and that one line was still better than the empty page it replaced. The natural next step is depth: the anatomy of a runbook that works under stress turns your starter drafts into documents a panicked stranger can follow. And the rest of the Runbooks pillar covers keeping them alive, using them to train juniors, and eventually letting scripts take over the boring steps.
Put this into practice with SysPeep Monitor
Software cannot write your runbooks — but it can hand you the shopping list and then open each one at the right moment. SysPeep Monitor watches your network over SNMP and WMI. Its alert reports show which problems actually recur: the exact list this guide asks you to pick five from. The free trial installs on a Windows machine in minutes and starts discovering your devices on its own.
Download Free TrialLearn About SysPeep