Home › Topics › Runbooks › Runbook to Automation

From Runbook to Automation: When Scripts Replace Steps


Every runbook has a script hiding inside it. The steps a human follows today can become one command the human runs tomorrow. And a few of those can eventually run with nobody awake. The runbook is the specification, and a better one than most, because it has already been tested against live incidents by tired people. Teams that see this stop treating documentation and automation as separate projects and start treating them as the same project at different ages. I did not see it for years. I kept a folder called scripts and a wiki called runbooks, and never noticed that they described the same ten problems.


But the path has a trap at the end of it: not every step should stop being human. And the ones that should not are precisely the ones that look most tempting at 3 a.m. This guide is the judgment piece. You will get the four-stage ladder with honest examples, plus the filters that decide which steps climb and which never do. You will see what happens to the runbook as its steps automate (it does not disappear; it moves up a level). You will learn the discipline that keeps response scripts trustworthy. And you will get a worked example: one disk-space runbook walked up the ladder over about a year, with every decision annotated.


In this guide

  • The four-stage maturity ladder
  • Choosing what climbs: the worthiness filters
  • The runbook does not disappear — it moves up a level
  • Scripts are production now
  • Monitoring the automation itself
  • The honest economics
  • Worked example: the disk-space runbook's year
  • The ladder-decision worksheet

The four-stage maturity ladder


Four stages, and — this matters — they are climbed per step, not per runbook. A mature runbook is usually a mix: automated evidence-gathering feeding human decisions feeding one or two scripted fixes.


Stage 1: WRITTEN — the human follows steps

The baseline: a tested runbook with the full anatomy — verify, diagnose, fix, escalate, confirm — executed entirely by a person. Do not rush past this stage; time spent here is research. I once scripted a runbook after its second use and spent a year maintaining a script for a problem that changed shape on its third. Every run teaches you which steps are purely mechanical (the same commands, in the same order, every time) and which involve actual judgment. You cannot see that split from your desk on a planning day. The runbook's use history reveals it.


Stage 2: ASSISTED — the script does the typing, the human does the deciding

The runbook's mechanical stretches collapse into scripts the responder runs on purpose: one command replacing ten typed ones. The classic first automation — the right first automation for almost everyone — is the diagnostic bundle. This script gathers every piece of evidence the DIAGNOSE section asks for in one shot. That includes disk numbers, top space consumers, recent growth, relevant service states, and the last hour of pertinent log lines. It prints a tidy summary. It touches nothing, so a bug in it costs you a bad report, not an outage. The human still walks the branches and still makes every decision. They just start from assembled facts instead of assembling them, half asleep, one command at a time.


Stage 3: GATED — the system prepares the action, the human approves it

Now the corrective step itself is scripted, but a person holds the trigger. The pattern: the system detects the condition, prepares the fix, and presents it. "App pool hung on WEB-01; restart it? Here is what the checks showed." The responder replies yes or no. The restart-with-confirmation is the archetype. The human contribution shrinks to the one thing humans are still best at: recognizing when tonight is not a normal night. ("Wait — a deployment is mid-flight; do not restart anything.") Gates are cheap to build — often just a script that shows its plan and waits for confirmation before acting. They preserve that judgment at almost no speed cost.


Stage 4: AUTONOMOUS — the system acts and reports

The self-healing response: condition detected, fix applied, humans told afterward. Be honest about where this belongs at small scale, because it does belong somewhere: narrow, well-understood, low-blast-radius recoveries. Think of the known service that hangs in a known way and has been safely restarted by hand fifty times. Or think of clearing a specific application's disposable cache folder when it bloats. The recovery must be boring, proven, and harmless when wrongly triggered. Autonomy does not belong in any action touching data or any action with a wide blast radius (reboots of shared infrastructure). It also does not belong in any action whose failure mode you have not personally watched. And it does not belong in any response to a condition your monitoring sometimes misreads. An autonomous action firing on a false positive is your automation causing an outage — the exact inversion of the point.


StageWho actsWho decidesTypical exampleBug cost
1 · WrittenHumanHumanThe tested runbookA wrong step, caught by the reader
2 · AssistedScript, on requestHumanDiagnostic-bundle scriptA bad report
3 · GatedScript, on approvalHumanRestart-with-confirmationA bad proposal, declined
4 · AutonomousSystemSystemRestart-on-known-hangA wrong action, taken

Read the last column twice. Each rung up the ladder moves the cost of a bug from "annoying" toward "self-inflicted incident." That is not a reason to stay at stage 1; it is the reason the next section exists.


Choosing what climbs: the worthiness filters


Filter 1: frequency × mechanicalness. Automation pays rent per use, so frequent steps win — but only the mechanical ones. A step is mechanical if two competent people following it always do the same thing: same commands, same order, no interpretation. Pure-procedure steps automate beautifully. Judgment steps — "decide whether this growth is expected," "check whether now is a safe time" — do not, no matter how often they occur. The judgment is the content and a script has none.


Filter 2: blast radius and reversibility. Ask: what is the worst thing a bug in this automation does, and can I undo it? Restarting a stateless service (one that holds nothing in memory it would miss) is small and reversible. Deleting files is small and irreversible — unless the script moves them to a holding area first, which is why good automation designs reversibility in. Rebooting the firewall is large. Anything irreversible or large needs, at minimum, a gate; often it should simply stay human.


Filter 3: the diagnostic/corrective split. The cleanest rule in this whole guide: automate evidence-gathering aggressively; automate changes conservatively. Reading state is always safe — a diagnostic script cannot make the incident worse. So every runbook's VERIFY and DIAGNOSE sections are fair game immediately. Changing state is where all the risk lives, so corrective steps climb one rung at a time, earning each promotion. This mirrors the guardrails in automating yourself out of the boring parts: scripts that look win instantly; scripts that touch earn trust slowly.


And the never-automate list, stated plainly, starts with steps requiring situational judgment (the "is tonight normal?" call). It includes irreversible actions without a gate in front of them. And it includes anything you could not debug at 3 a.m. That last one is the sleeper. Every automation eventually misbehaves during an incident, adding a second mystery on top of the first. The on-call human must be able to understand what the script did and why, from its logs, while stressed. Otherwise, you have not removed work from 3 a.m.; you have added a suspect.


The runbook does not disappear — it moves up a level


A common mistake: the script gets written, and the runbook gets deleted ("it's automated now"). Six months later a responder faces a failing automation with no documentation of what it was supposed to do. The right move: the script replaces steps, but the runbook remains — rewritten one level up. Where it used to say "run these ten commands," it now says when to run the script (and when not to). It explains what it does in one honest paragraph and what output to expect. And it covers the part that saves you someday: what to do when the script itself fails. That includes the manual fallback: the original steps, kept in an appendix or a linked older version. The automation gets its own runbook. That is the meta-layer, and it is not bureaucracy; it is the only thing standing between "the script errored" and a genuinely unscripted incident.


In practice the rewrite is pleasant work: the runbook gets shorter and higher-level with each promotion. Ten commands become "run disk-evidence.ps1; you should see a top-20 list ending with a summary line." The freed space is worth spending on what the automation cannot carry — richer interpretation guidance, the judgment branches, the escalation context. A year into the ladder, a good runbook reads less like a keyboard transcript and more like a short field manual. Here is what the machinery does for you. Here is where you must think. Here is the exit. That is the document a 3 a.m. responder actually wants.


At Northgate Retail the disk-space runbook was run eleven times in one quarter. In the fourth month it became a script, which was the right call. The runbook was then deleted, because it was automated now. Eight months later an upgrade moved the log folder. The script carried on reporting success while freeing nothing, and the disk filled on a Saturday night. The on-call admin found a script, no runbook, and a comment at the top of the script that read "see runbook". He cleared the disk by hand in an hour and spent the second hour working out what the script had been for. The runbook is back, one level up, and the script now says how much it freed.


Scripts are production now


The moment a script is part of incident response, it stops being a convenience and becomes production infrastructure. It is arguably more critical than the systems it fixes, because it runs during their worst moments. Three disciplines follow. Version-keeping: response scripts live in one known place with change history. Use a proper version-control system if you have one, or at minimum dated copies and a changelog. "Which version ran last night?" must be answerable. Testing after changes: any edit to a response script gets exercised against a healthy system before the next incident finds the typo for you. That is the same last-tested honesty your runbook maintenance already practices. It has the same change-hook: when the systems a script touches change, the script is flagged for review too. The quiet killer here is drift: the cleanup script that still targets the old log path after a migration. It is "working" — exit code zero, which is the script's way of saying all went well — while freeing nothing. That continues for months, until the disk fills anyway. The responder wastes an hour trusting a tool that had silently stopped mattering. Announcing: when a script changes behavior, the team hears about it before an incident demonstrates it.


Monitoring the automation itself


Every automated response adds two questions your monitoring must now answer: did it run, and did it work? Treat automation events as first-class alert material. A gated or autonomous action should announce itself every time it fires ("auto-restart triggered on APP-01 — recovery confirmed"). It should fire a real alert when it acts but the problem persists. Think of the hang that the restart no longer cures — a human is needed now, with a head start of context. And — the subtle one — the action should make silence detectable. The self-healing that quietly stopped healing is the failure mode nobody notices. Think of the scheduled job disabled by an update, the credential that expired, or the script erroring before its first log line. If a known-flaky service has not triggered its auto-restart in months, that deserves a raised eyebrow. Either the flakiness got fixed (lovely — retire the automation and its runbook, with a tombstone) or the watcher is dead. A periodic "automation is alive" check closes the loop. It is the same heartbeat thinking you apply to backups, where a missing success report is itself a failure.


The honest economics


Automation is not a one-time cost. Every script you promote into the response path is a thing you now tend forever: re-tested after changes, updated when systems move, documented, monitored. That is the build-versus-tend tradeoff, and at small scale it bites hard. A two-person shop that automates twenty things has acquired twenty small production systems and no additional staff. The failure mode is a graveyard of half-trusted scripts nobody quite remembers. That is strictly worse than the runbooks they replaced, because at least a stale runbook cannot execute itself.


So here is the recommendation, stated plainly: for most small shops, the assisted rung is the sweet spot, and it is not a consolation prize. Diagnostic bundles and one-command fixes capture most of the speed. The ten minutes of half-asleep evidence-gathering compressed to one command is the single biggest win on the whole ladder. These scripts come with none of the autonomy risk and a fraction of the tending burden. A script a human consciously runs gets an implicit test on every use. Climb to gated where frequency genuinely justifies it. Reserve autonomous for the two or three recoveries so narrow and so proven that arguing about them feels silly. Ambition is not the goal; sleep is.


Tip: a fine default policy is simple. Every runbook earns a diagnostic-bundle script as soon as it has been used three times. Nothing corrective automates until the runbook's fix steps have gone unchanged through at least a handful of real incidents. Frequency of use is the promotion committee.


Worked example: the disk-space runbook's year


Here is the ladder in real life: the "C: drive critical on the app server" runbook from your first five, walked upward over about a year. The dates are approximate; the decisions are the point.


Months 1–3, WRITTEN. The runbook exists and gets used monthly. Verify the alert, find what grew, clear the known offenders (an app's verbose logs, a temp folder), confirm the trend flattened. Use history teaches the split: finding-what-grew is identical every time (mechanical). Deciding-what-to-delete is not, because sometimes the growth is something new and mysterious (judgment).


Month 4, ASSISTED — the evidence script. The mechanical stretch becomes disk-evidence.ps1, run by the responder as step 2:


# disk-evidence.ps1 — gathers facts, changes NOTHING
Get-Volume C | Select-Object Size, SizeRemaining
Get-ChildItem D:\AppLogs, C:\Temp -Recurse -File |
    Sort-Object Length -Descending |
    Select-Object -First 20 FullName, Length, LastWriteTime

Decision annotated: read-only, so it climbed immediately (filter 3); it runs monthly, so it pays rent (filter 1). The runbook's DIAGNOSE section shrinks to "run the script; interpret the top-20 list." The interpretation guidance — what the usual suspects look like, what counts as surprising — stays in prose, because interpretation is judgment.


Month 7, GATED — the cleanup proposal. After the fix steps survive several incidents unchanged, the known-offender cleanup becomes a script that proposes. It lists exactly which archived log files it intends to remove: those older than the retention window, in the one designated folder. It shows how much space that frees, then waits for the responder's yes. Decision annotated: it changes state, so it needed both a gate and designed-in reversibility. Files move to a holding folder that empties a week later, so a wrong yes is recoverable (filter 2). The mysterious-growth branch stays fully human, forever.


Month 12, one narrow AUTONOMOUS piece. One offender proves utterly predictable: the app's already-archived logs, backed up nightly, in one folder the team has manually approved for cleanup a dozen times. That single action goes autonomous: when disk crosses the warning threshold, purge that folder's expired archives. Then log the action and send the "I acted; here is what I freed; current disk level" report. The alert stays armed: if disk stays critical after the purge, a human is paged with the evidence bundle attached. Decision annotated: narrow (one folder), proven (a dozen supervised runs), low blast radius (backed-up, expired archives), reversible enough, debuggable from its own log at 3 a.m. Everything else in the runbook — the interpretation, the mystery branch, the escalation — is still a human reading a page. That is what a mature runbook looks like: not fully automated, but automated exactly as far as the filters allow.


The ladder-decision worksheet


To run this exercise on one of your own runbooks, list its steps and fill four columns. The worksheet for the disk example, mid-year:


Runbook stepStage todayNext stage?Filters applied
Verify alert is real (check volume stats)Assisted (in script)StaysRead-only: automate aggressively
Gather top space consumersAssisted (in script)StaysMechanical + frequent + read-only
Interpret the list: expected or surprising?WrittenNever climbsJudgment step — never-automate list
Clear known-offender archivesGatedAutonomous (one folder only)Narrow, proven, reversible, low blast radius
Investigate mysterious growthWrittenNever climbsJudgment; irreversible deletes; undebuggable
Confirm trend flattened; log outcomeWrittenAssistedMechanical + read-only; cheap win

Fill this honestly and the argument settles itself. The read-only rows climb now, the judgment rows never climb, and everything else queues for promotion by frequency. Keep the worksheet with the runbook — it is the automation roadmap and its justification in one table.


Where to go next


The ladder only works when the bottom rung is solid. A runbook with the right anatomy, kept honest by real maintenance, is what makes every promotion above it safe. A script with no runbook under it is just a folder called scripts, which is where I started. The rest of the Runbooks pillar covers building that foundation, from choosing your first five to training the team that will one day review your scripts.


Put this into practice with SysPeep Monitor

Every rung of the ladder starts with detection, and that half is squarely a monitoring job. SysPeep Monitor's thresholds, dependencies and alert rules fire on the real problem. The same machinery watches your automation. If the "fixed" state never arrives after a scripted response, you hear about it. Your scripts do the responding; it does the noticing. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep