Safe Candidates for Auto-Remediation: What to Automate First
Auto-remediation starts to look attractive at about two in the morning. The phone has buzzed for the third time this month about the same hung service on the same server. And the fix is what it always is: log in, open the services console, click Restart, go back to bed. You could do it in your sleep. What you would really like is for something to do it while you sleep.
Auto-remediation is exactly that: monitoring detects a problem, and instead of only telling a human, something runs a fix without waiting for one. It is one of the most satisfying things you can build. And it is where careful people slow down. That is because a script is a colleague who never sleeps, never doubts itself, and never notices that today is different. I learned this from the first one I wrote. It restarted a service every ninety seconds, whether the service needed it or not. It did that for seven weeks before anyone asked why the event log was so long. Choose the wrong first candidate and you will learn that the hard way. Choose the right one and you will wonder why you waited.
This guide gives you a way to tell the two apart. It offers five criteria for judging any candidate, and a shortlist of the classic first automations and the check each should run before acting. It also offers a scoring sheet you can apply to your own alert history this afternoon.
In this guide
- What auto-remediation is, and what "carefully" means
- The five criteria for a safe candidate
- The classic first candidates
- Scoring a candidate: the worked example
- Where good candidates come from
- The check-first habit
- The candidate cheat sheet
- Where to go next
What auto-remediation is, and what "carefully" means
A monitoring system watches your servers and network, compares what it sees against thresholds, and raises an alert when something crosses a line. Normally the alert goes to a person, who investigates and fixes it. Auto-remediation removes the person from the middle of that loop for a specific, narrow kind of problem. The alert fires, a script runs a pre-decided fix, and the person hears about it afterwards — or only if the fix did not work.
The word "carefully" is doing a lot of work in the title of this topic. The trouble with automation is not that scripts are unreliable. It is that they are perfectly reliable at doing exactly what you told them, in situations you did not anticipate. A human who sees a service down on a server that is also showing a full disk and a flapping network link will pause and think. A script will restart the service, because that is what it does. So the craft of auto-remediation is choosing actions that are safe even when the script's picture of the world is incomplete. It also means surrounding them with checks that make a wrong action unlikely and its damage small.
Choosing the action well is the half this guide covers. The checks that wrap the action — attempt limits, cooldowns, dry-run mode, circuit breakers — get their own guide in Guardrails: Rate Limits, Circuit Breakers, and Rollback.
The five criteria for a safe candidate
Test every candidate against five questions. A good first automation gets a confident "yes" to all five. A weaker one might be worth doing later, once you have experience and guardrails, but not first. The order matters more than it looks, because the first automation is the one that decides whether you are allowed a second.
1. Well understood
You have fixed this problem by hand many times, and you know why the fix works. Not "restarting it seems to help" but "the service leaks handles under load, the restart releases them, and it will be fine for another week." If you cannot explain the mechanism, you cannot predict the situations where the fix will not work. Those are the situations where an automation does damage. The test: could you write the runbook for this fix from memory, including the two or three times it did not work and what was actually wrong?
2. Reversible
If the automation acts when it should not have, undoing it should be easy and obvious. Starting a service that was deliberately stopped is annoying but trivially reversible: stop it again. Deleting a file is not reversible. Changing a firewall rule is reversible in theory and terrifying in practice. The test: write down, in one sentence, what undoes this action. If the sentence starts with "restore from backup," the candidate fails.
3. Idempotent
Idempotent is a word borrowed from mathematics that simply means: doing it twice has the same effect as doing it once. Think of the difference between a light switch and a light button. A button labeled "on" is idempotent — press it five times and the light is on. A toggle switch is not — flip it twice and you are back in the dark. Automations get run twice far more often than you would expect. The alert fires again before the first fix finished, two triggers overlap, or a scheduled task double-fires after a clock adjustment. "Start the service if it is stopped" is idempotent. "Toggle the service" is not. "Append a line to the config" is not. The test: run the fix twice in a row, by hand, on purpose. Was the second run harmless?
4. Frequent
An automation is a small program you now own, test, document, and keep working through upgrades. That effort only pays off if the problem happens often enough to matter. A fix you apply weekly is worth automating. A fix you applied once, memorably, at three in the morning, is worth a runbook — not a script. The test: count actual occurrences in your alert history or ticket queue over the last few months. Gut feeling exaggerates; the log does not.
5. Low blast radius
Blast radius is how much can break if the action goes wrong. Restarting one service on one host has a blast radius of one host. Restarting a core switch has a blast radius of everything behind it. Clearing a print queue affects the people who were printing. Reloading a firewall affects everyone. Your first automation should have a blast radius you could explain to your manager in one sentence without them going pale. The test: if this action ran on the wrong host, or at the wrong moment, who notices?
Tip: the five criteria are not equally negotiable. "Frequent" is a matter of economics — you can automate a rare problem if you want the practice. "Reversible" and "low blast radius" are matters of safety. Never trade the safety criteria for the economic one.
The classic first candidates
Certain problems show up in nearly every environment, are fixed the same way everywhere, and pass all five criteria comfortably. Begin here. For each one, notice that the fix is only half the story: the check that runs before it is what makes the automation safe.
Restarting a hung service
The archetype. A service is meant to be running and either is not, or is technically running but no longer answers. Why it qualifies: you have done this a hundred times, a start is idempotent, the undo is a stop, and only that service's users are affected. The check first: confirm the service is actually stopped or unresponsive (query its state, or probe the port it should listen on). Confirm nobody stopped it deliberately. And confirm the host is healthy enough that a restart could work. A server with no free memory will hang the service again in a minute, and that case needs a human.
Clearing temp files when a disk fills
A disk-space alert on a server whose temp folder grows until it chokes the machine. Why it qualifies: it is frequent, the mechanism is obvious, and deleting genuinely temporary files is low-risk when the rule is narrow. The check first: this is where the check matters most, because deletion is not reversible. Only touch known temp locations. Only delete files older than a sensible age. Skip files that are open. And before deleting anything, confirm the temp folder is actually why the disk is full. If it holds a few hundred megabytes and the disk is missing a hundred gigabytes, the culprit is something else (a database, a log, a backup dump). In that case, cleaning temp achieves nothing but a false sense of action.
Restarting a stuck print spooler
Printers exist to test the patience of administrators, and the print spooler service makes sure that patience is tested by proxy. A stuck job blocks the queue; restarting the spooler clears it. Why it qualifies: frequent, understood, one-host blast radius, and users would rather reprint than wait. The check first: confirm the queue really is stuck (a job that has not progressed in several minutes) rather than merely busy. Be honest in the runbook that clearing a stuck queue discards the jobs in it. That is a small, reversible-by-reprinting cost, but users should know it.
Rotating a runaway log
An application whose log file grows without limit until it fills the disk. Why it qualifies: rotating means renaming the current log aside, letting the application start a fresh one, and compressing or archiving the old one. Rotating is reversible (the data is still there) and idempotent (rotating an already-small log is harmless). The check first: confirm it is a plain application log and not something that merely looks like one. Database transaction logs, mail queue files, and anything the application holds open with an exclusive lock are not logs to be rotated by a script. Move one of those aside and you will discover how much the application depended on it. Rotate only files you have named explicitly, never "anything ending in .log."
Re-enabling a port shut by port security, after checks
A switch port disabled itself because it saw a MAC address it did not expect, and now a perfectly legitimate desk is dark. Why it qualifies: re-enabling one access port is a one-desk blast radius, and the undo is to shut it again. In many environments the same handful of ports trip on the same handful of innocent causes (a docking station swap, a phone rebooted behind a PC). The check first: this one earns its place only because of the check. Before re-enabling, confirm the MAC that tripped the port is one you already know. Also confirm that the port has not tripped repeatedly in a short window, and that nobody deliberately left it shut. If any of those fail, the automation stops and hands the evidence to a human. The situation port security exists to catch is exactly the one an automation must not paper over. The signals to read are covered in Port Security Signals.
Recycling an application pool
Consider a web application whose worker process gets slow or unresponsive over time. It comes back to life when its application pool — the group of worker processes that serves it — is recycled. Why it qualifies: recycling is designed to be graceful (new workers start before old ones retire). So the blast radius is a brief slowdown rather than an outage, and recycling is idempotent. The check first: confirm the application is actually unresponsive (a request to a known page times out or errors) rather than busy under a legitimate load spike. And confirm it has not been recycled recently, because an application that needs recycling every ten minutes has a problem a recycle will not fix.
Scoring a candidate: the worked example
A scoring sheet turns "this feels safe" into something you can defend and compare. It also replaces the phrase "what could go wrong", which has never once been asked as a question. Give each criterion a score from 0 to 2. A 0 means the candidate clearly fails it, 1 means it passes with conditions, and 2 means it passes cleanly. A total of 9 or 10 is a first automation. Anything with a 0 in reversible or blast radius is off the table regardless of the total.
Here is the sheet filled in for the archetype: a reporting service on one application server that hangs roughly weekly. It is fixed by a restart every time, and has been for months.
| Criterion | Score | Notes |
|---|---|---|
| Well understood | 2 | Restarted by hand many times; the vendor's own guidance is a restart; it leaks resources under load and the restart releases them |
| Reversible | 2 | Undo is stopping the service again; no data is touched |
| Idempotent | 2 | "Start if stopped" is harmless to repeat; the script checks state before acting |
| Frequent | 2 | Alert history shows it most weeks, often out of hours |
| Low blast radius | 2 | One service, one host; the reports it produces are not time-critical |
| Total | 10 | First automation material |
Now the same sheet for a tempting but wrong candidate: a site-to-site VPN tunnel that occasionally drops and comes back when someone reboots the branch firewall.
| Criterion | Score | Notes |
|---|---|---|
| Well understood | 1 | The reboot works, but nobody knows why the tunnel drops; sometimes it is the carrier |
| Reversible | 1 | The reboot itself cannot be undone; the firewall comes back in whatever state it comes back in |
| Idempotent | 0 | Rebooting twice in a row means two outages; there is no "reboot if not already rebooted" |
| Frequent | 2 | Happens often enough to be a real irritation |
| Low blast radius | 0 | The entire branch loses connectivity during the reboot, and if the tunnel does not come back, someone drives there |
| Total | 4 | Not an automation: a runbook and a root-cause investigation |
The scores make the difference explicit. Both problems are frequent, and both have a known "fix," which is why the second one is tempting. But the second fix is a reboot of a device with dozens of people behind it, for a cause nobody understands. Automating that would not remove the pain; it would schedule it.
Where good candidates come from
You do not have to invent candidates. Three places already hold the list, and they agree with each other more often than you would think.
- Your runbooks. A written procedure that has been followed twenty times without variation is the purest source of automation candidates there is. The "well understood" criterion has already been satisfied in writing. The path from a runbook step to a script is a progression with its own stages and its own judgment calls. Those are laid out in From Runbook to Automation. If you do not have runbooks yet, First Runbooks shows how to write the five that matter. In that case, you may find that writing them is what reveals the candidate.
- Your alert history. Sort your monitoring system's alert log by how often each alert fired, and look at the top. Frequent alerts with identical resolutions are candidates. Frequent alerts with varied resolutions need better diagnosis, not automation.
- Your ticket queue. The same sort, applied to tickets. Look for tickets closed routinely with the same one-line resolution note. That note is a runbook in disguise, and possibly a script.
One less official source: your own grumbling. The fix you mutter about while doing it is usually frequent, understood, and small. Trust the mutter, then verify it against the log.
The check-first habit
Every candidate above came with a "check first," and that is not an accident. The habit that separates safe automation from reckless automation is refusing to act on the alert alone. An alert is a claim about the world made by a monitoring system that might be wrong, stale, or looking at a symptom rather than a cause. A precondition is the automation's own independent look before it does anything: is the service really stopped right now, according to this host? Is anything else wrong that would make the fix pointless or harmful? Has this already been tried recently?
When a precondition fails, the automation should do the most valuable thing it can do, which is nothing — loudly. It logs what it checked and what it found, and hands that to a human. An automation that stops itself and explains why is not a failure. It is a diagnostic tool that also fixes things when the situation is simple. The log of "did not act, because" entries will teach you more about your environment than the log of successes.
At Meridian Parts the order-entry service hung every few days, and a script restarted it. The script had no preconditions, so it never noticed that the host had a little less free memory after each restart. Nobody else noticed either, because the alert cleared before anyone read it. Over eight months the restarts went from every few days to every few hours, which the script handled without complaint. The host finally ran out of memory on a Saturday. And the engineer who looked at it found a leak that had been visible in the restart log since the first week. The script had not hidden the leak so much as tidied it away, several hundred times.
A useful way to design the checks: for each candidate, list every occasion when the manual fix did not work. Ask what you noticed that told you it would not. Those observations are your preconditions. A restart that failed because the disk was full becomes "check free disk space." A print queue that was merely slow becomes "check whether the top job has progressed." You are encoding your own judgment, one bad night at a time.
The candidate cheat sheet
The six classic candidates, with the check, the action, and the undo side by side. Copy it into your own documentation and add rows as you find candidates of your own.
| Candidate | Check first | Action | Undo |
|---|---|---|---|
| Hung service | Service stopped or unresponsive on the host itself; host healthy; not deliberately stopped; no recent attempt | Start the service, wait, verify | Stop the service |
| Temp files filling disk | Temp folder is actually large; files older than the age limit; not open | Delete only the matching files in named locations | None — keep the rule narrow |
| Stuck print spooler | Top job has not progressed for several minutes | Restart the spooler service | Users reprint discarded jobs |
| Runaway log | File is on the explicit list; application does not hold it locked | Rename aside, compress, let the application start fresh | Rename it back |
| Port shut by port security | Known MAC; no repeat trips; not deliberately left shut | Re-enable the one port | Shut the port again |
| Unresponsive application pool | Request to a known page fails; no recycle in the last while | Recycle the pool | Nothing to undo — recycling is graceful |
Notice that the "undo" column for temp-file cleanup is honest: there is not one. When an action cannot be undone, the checks have to do all the work of making a mistake unlikely. In that case, the scope has to be narrow enough that a mistake is small. If you cannot make it that narrow, do not automate the deletion; automate the report that tells you what is filling the disk, and delete by hand.
Where to go next
Any recurring fix can now be judged with reasons rather than with a feeling. You have five questions, a score, and a check that runs before the action. The score keeps you honest on the night a problem is annoying enough to tempt you past the safety criteria. And the check is what stops a script from doing the right thing in the wrong situation. Had I owned either when I wrote the ninety-second restart, it would have run once, checked, and stopped. Pick a candidate that scores ten, give it a check it cannot skip, and let it have the two-in-the-morning part.
Choosing the candidate is the beginning. The next guide in this topic, Guardrails: Rate Limits, Circuit Breakers, and Rollback, covers the safety machinery that wraps the action. And Building Your First Auto-Remediation End to End walks through the whole thing for the hung-service archetype. Just as important is knowing where the line is: What Never to Automate (and Why) lists the actions that never qualify, however frequent they are. The full set of guides lives on the Auto-Remediation: Letting Monitoring Fix Things (Carefully) pillar page.
Put this into practice with SysPeep Monitor
Every automation in this guide starts from a good alert — and a good alert starts with monitoring that watches the right things. SysPeep Monitor collects service state, disk space, and interface status over SNMP and WMI. It applies thresholds and dependencies so a real problem stands out from the noise, and sends an email notification your own scripts can act on. The alert history it builds is the list of candidates; the free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep