What Never to Automate (and Why)
The dangerous thought arrives about a week after your first auto-remediation works. You have watched it fix a hung service at three in the morning without you. You have a list of other tedious fixes, and the thought is: what else could I do this to? It is dangerous because it is reasonable. Scripts are cheap, and the monitoring already knows when things break. Somewhere on that list is the one that will take the network down, and it looks exactly like the others.
This guide is the list of things that stay on the human side of the line, however frequent, however boring, however confident you feel. The reasoning is not that automation is scary. It is that some actions cost so much when they are wrong that no amount of frequency pays for them. And every automation is wrong eventually, because the monitoring that triggers it is not perfect and never will be. I once wrote "should be fine" in the risk column of a change ticket for a script that ran unattended. It was fine for eleven days.
By the end you will have the never-list with the reason behind each entry. You will have the story of the automation storm that teaches all of them at once. And you will have the alternative for every tempting case: an automation that gathers the evidence and proposes the fix, but leaves the decision to a person.
In this guide
- The asymmetry that decides everything
- The never list
- The automation storm
- Propose, don't act: handling the tempting cases
- The gray zone: never, for now
- The never / why / instead table
- Where to go next
The asymmetry that decides everything
The safe-candidate criteria — understood, reversible, idempotent, frequent, low blast radius — are laid out in Safe Candidates for Auto-Remediation. This guide is the mirror image: what happens when a candidate fails the safety criteria badly. The reason those failures are disqualifying rather than merely inconvenient comes down to an asymmetry.
A good automation saves you ten minutes, a few times a month. That is the upside, and it is real, but it is bounded. A bad automation causes an outage, a data loss, or a security incident. That is the downside, and it is not bounded. When the upside is "some minutes" and the downside is "the worst day of the year," the math does not care how frequent the problem is. The only question that matters is: when this automation acts on a false alert — not if, when — what is the cost? If the answer is "we restart a service that was fine," automate. If the answer includes the words "restore," "outage," "compliance," or "explain to the executive," do not.
Hold onto that framing while reading the list. Every entry below is a category where the cost-of-being-wrong is unbounded. There is also a quicker test that catches most of them before you have written a line. Certain words in a script, a runbook step, or your own description of the idea are reliable warnings:
- "Force" — any force flag means the normal, safe path was not enough, which means the script is overriding a protection somebody put there.
- "All" or "every" — all hosts, every service, everything matching a pattern. The blast radius just became the fleet.
- "Recursive" or "wildcard" next to a delete — the scope is no longer a list of things you have thought about.
- "Just in case" — an action taken without a confirmed reason is an action taken on a guess.
- "Should be fine" — the action has not been tested in the situation it will run in.
None of those words is forbidden in a script a human runs deliberately. In a script that runs by itself at three in the morning, each one is a reason to stop and reconsider. "Should be fine" is the one to listen for in your own voice, because its honest translation is "I have not tested this."
The never list
Anything destructive or irreversible
Deleting data, wiping configurations, decommissioning servers, dropping databases, purging queues that hold real work. These fail the "reversible" criterion completely, and the failure mode of a false alert is permanent. Notice that "clear temp files" from the safe list is technically a deletion too. It qualifies only because the scope is so narrow (named locations, old files, not open) that a mistake is trivial. The moment a deletion's scope grows beyond "things nobody could possibly want," it moves to this list.
At Harbourview Clinic a disk-space remediation on the appointments server acted whenever the disk crossed ninety per cent. It removed anything ending in .log or .dmp and older than an hour. One Thursday morning the appointments application crashed. It wrote a very detailed error log and a set of crash dumps while it failed, and filled the disk. The alert fired, the script found several gigabytes of matching files, deleted them, and cleared the alert. The post-incident review was at two o'clock. It established that something had crashed and that the disk had filled. It also established that the only record of why had been deleted by the remediation an hour before the meeting. That deletion was in accordance with its configuration.
Core network configuration on the strength of an alert
Routing changes, VLAN changes, firewall rule changes, spanning-tree adjustments, anything on a core switch or edge router. The blast radius is everything behind the device. The reversibility is theoretical: the change is easy to undo only if you can still reach the device after making it. And the alert that triggers it is often a symptom several layers away from the cause. An automation "fixes" a routing problem by pushing a config. One day, that automation will push a config in response to a monitoring host's own network hiccup. Re-enabling a single access port after checks is the narrow exception, and it is narrow precisely because it touches one desk, not a segment.
Security containment without human review
Isolating a host from the network, disabling a user account, blocking an address at the firewall, quarantining a device. These are tempting because the security logic is real: a compromised machine should be cut off fast. But an ad-hoc script that isolates hosts on the strength of an alert will one day isolate the file server because a scanner tripped a rule. Or it will lock out the finance team on payroll day because a password spray hit their accounts. Containment is also a decision with consequences beyond IT — evidence preservation, notification obligations, business continuity.
The honest nuance: there are scenarios, ransomware being the obvious one, where fast isolation is the right call and waiting for a human costs real damage. The answer is not "a script someone wrote on a Tuesday." It is a deliberately designed, documented, and drilled containment process, agreed with the people who own the business risk. That process has clear triggers, clear scope, and a rehearsed way to undo it. That is a security program, not an auto-remediation, and it deserves the planning a security program gets. Until you have that, containment stays with a human.
Mass actions across many hosts at once
Restarting a service on one host is a safe candidate. Restarting it on two hundred hosts at the same moment is not the same action scaled up. It is a different action with a different blast radius. Simultaneous restarts drop every session at once and hammer whatever those services depend on when they all reconnect. If the trigger was a false alert, the restarts turn a non-problem into a fleet-wide outage in seconds. An automation may be able to act on more than a handful of targets per run. Any such automation needs a hard cap, a stagger, and a fleet-wide circuit breaker. Even then it should start its life acting on one.
Anything you cannot test safely
There may be no test host, no lab, no way to make the trigger fire on purpose and watch the script react without touching production. In that case, you cannot know what the script does. "It worked when I ran it by hand once" is not a test of an automation. The automation's job is precisely the cases you did not run by hand. If you cannot test it, you cannot trust it, and if you cannot trust it, it does not run unattended.
Anything with a business decision inside it
Some actions look technical but contain a judgment that belongs to someone else. Disabling an account is technical; disabling an executive's account during a board meeting is a business event. Paying for burst capacity, extending a license, accepting a certificate, closing a ticket as resolved, sending a customer-facing notice — each has a "should we?" inside it. A script cannot answer that question and should not be allowed to guess at it. The tell: if a person doing the action by hand would pause to ask someone, the script must pause too, which means it must stop.
Actions the monitoring system itself depends on
These dependencies include the monitoring server, its database, its network path, and the DNS it uses to resolve hosts. They also include the mail relay it sends alerts through and the account it authenticates with. An automation might restart the monitoring database "because monitoring says it is slow." That can leave you with no monitoring, no alerts, and no record of what the automation did. It is the one automation that can arrange its own alibi. Worse, a remediation that acts on the monitoring system's own infrastructure can cause the exact false alerts that then trigger it again. These components are watched by a separate, simpler mechanism and fixed by people.
Tip: the never list is not a list of things that can never be scripted. Almost all of them make excellent scripts for a human to run, with the evidence in front of them and a finger over the key. The list is about what runs unattended, triggered by an alert, with nobody watching.
The automation storm
Every entry on the never list can be learned from a story, and there is one story that teaches most of them at once. It happens in some form in every environment that automates enthusiastically before it automates carefully. Here it is, with the details generic because the details are always the same.
A team builds a good remediation: when monitoring reports a service down, restart it. It works. Encouraged, they extend it to every service on every server. Still works. One night, a switch upstream of the monitoring server reboots for a firmware update — forty seconds of lost connectivity, entirely on the monitoring side. To the monitoring system, every host in the building has just gone dark. Every service check fails. Every alert fires.
The remediation does what it was built to do. Across two hundred servers, it restarts every monitored service, at once. Database connections drop mid-transaction. Web sessions vanish. Application servers lose their connections to databases that are themselves restarting, and fail to reconnect because the database is not back yet. File shares disappear under the users still in the building. Now the alerts are real. Monitoring — back online after its forty seconds — sees a fleet of genuinely broken services, and the remediation restarts them again, into the same chaos. The thing that turns a blip into an outage is the fix.
A human is eventually paged, not by the automation, which believes it is doing splendidly, but by a user. The human's first job is to find and stop the script, which was never designed to be stopped. Then the real recovery starts, and it takes hours, because services that were restarted mid-write need attention that a clean shutdown would not have.
Count the lessons. The trigger was a monitoring failure, not a host failure, and the automation had no way to tell the difference. A precondition on the monitoring system's own health would have caught it. It acted on hundreds of hosts at once (a mass action). It had no fleet-wide circuit breaker, so nothing noticed that "everything is down" is not a situation one restart fixes. It could not be stopped quickly. And it restarted things that other things depended on, in the wrong order, because dependency was not part of its picture. Each of those is one guardrail from Guardrails: Rate Limits, Circuit Breakers, and Rollback, and each was absent.
The deeper lesson is the one this whole guide rests on: an alert is a claim, not a fact. Monitoring can be wrong in ways that make everything look wrong at once. And any automation that trusts alerts without limit will one day act on all of them at once. Your remediations should be designed on the assumption that the storm will happen, and that when it does, they will refuse to participate.
Propose, don't act: handling the tempting cases
Being told "never" about the most annoying items on your list is unsatisfying, so here is the constructive version. For every entry on the never list, there is an automation that is safe to build: one that does everything except the last step. It gathers the evidence, works out what it thinks the fix is, and drafts the ticket or the change. Then it stops and waits for a person. The principle is propose, don't act.
Consider the port-security case. A port has shut itself because of an unexpected MAC address. Re-enabling it after checks is on the safe list; deciding that the unknown device is fine is not. The propose-don't-act automation runs when the alert fires. It collects the port, the switch, the offending MAC, and the vendor prefix of that MAC. It also collects whether the MAC has ever been seen elsewhere on your network. It collects the last known device on that port and how many times this port has tripped this month. It writes all of that into a ticket with a one-line proposed action. That action is "re-enable, this is the same docking station as always" or "leave shut, this MAC is new to the network." The automation then assigns the ticket. The human reads a complete picture and clicks once. The fifteen minutes of digging have been done; the decision has not been taken away.
The same shape works for the others. A firewall rule appears to be blocking legitimate traffic. In that case, the automation pulls the deny-log entries and identifies the rule. It drafts the change with the exact addresses and ports, and attaches it to a ticket for review. For a host that looks compromised, the automation snapshots the process list, the network connections, and the recent logons. It packages them and pages the on-call person with the package attached — and does not isolate. A disk is full of something that is not temp files. In that case, the automation lists the largest folders and files, ages them, and reports, instead of deleting. In a mass-restart situation, the automation notes that many hosts are alerting at once. It checks whether the monitoring host itself can reach the network. Then it reports "this looks like a monitoring-side problem" rather than acting on anything.
Propose-don't-act automations have a wonderful property: they are safe to be wrong. A bad proposal costs a human a minute of reading. They are also how you earn the confidence to move something from the never list toward the safe list, if it ever belongs there. A proposal that has been correct every time for six months is evidence. A hunch is not.
The gray zone: never, for now
Honesty requires admitting that the never list is "never for a new automation built by one person on the strength of a monitoring alert." Some entries can graduate under conditions. It is worth knowing the conditions so you recognize when they are met rather than when they merely feel met.
- Security containment can become a drilled process: written with the people who own the risk, scoped precisely, tested in exercises, with a rehearsed undo. At that point it is not an auto-remediation any more; it is an incident-response capability that happens to be automated.
- Mass actions can become staged rollouts: one host, verify, five hosts, verify, and a hard stop the moment a verification fails. The safe version of "restart everything" is "restart one thing, look, then decide about the next."
- Configuration changes can become automated when the change is generated from an authoritative source and reviewed before it is pushed. That is configuration management, not remediation, and is triggered by an approved change rather than an alert.
What never graduates: destruction of data on the strength of an alert, business decisions, and touching the monitoring system's own foundations. If you find yourself building any of those, stop and ask who asked for it.
The never / why / instead table
| Never automate | Why | Instead |
|---|---|---|
| Deleting data, wiping configs, decommissioning | Irreversible; a false alert costs something permanent | Report what would be removed and by whom it should be confirmed; delete by hand |
| Routing, VLAN, firewall, or core switch changes | Blast radius is everything behind the device; alert is usually a symptom, not the cause | Draft the exact change with evidence attached; a person reviews and applies it |
| Isolating hosts or disabling accounts | Consequences beyond IT; false positives lock out real people | Package the evidence and page a person; build a drilled containment process separately if the risk warrants it |
| Simultaneous action on many hosts | Amplifies a false alert into a fleet outage; overloads dependencies | One target per run, staggered, capped, with a fleet-wide circuit breaker |
| Anything untestable | Unknown behaviour in exactly the cases that matter | Build a test host first, or keep it as a manual runbook |
| Actions containing a business decision | The "should we?" is not yours to script | Prepare the action and the context; the decision-maker confirms |
| The monitoring system's own dependencies | Can blind you and trigger itself | Separate, simpler watchdog; human-only fixes |
Where to go next
One question draws the line between what to automate and what never to automate: when this acts on a false alert, what does it cost? Unbounded cost means the action stays with a person, no matter how frequent the problem is. It stays with a person no matter how much it looks like the others on your list. The automation storm is what that cost looks like in practice, and every guardrail in this topic exists because someone lived through it. For the tempting cases, build the automation that proposes rather than acts; you keep almost all of the time saved and none of the risk. The risk column on the change ticket can then say "tested", which has aged better than "fine" every time I have compared them.
With the boundary clear, Building Your First Auto-Remediation End to End walks through a candidate that sits comfortably on the safe side. And Measuring, Auditing, and Retiring Automations covers keeping what you build honest over time. The complete set of guides is on the Auto-Remediation: Letting Monitoring Fix Things (Carefully) pillar page.
Put this into practice with SysPeep Monitor
The automation storm starts with monitoring that cannot tell a blip from an outage. SysPeep Monitor's dependency rules suppress the flood of downstream alerts when an upstream switch drops. Its thresholds and alert rules are built to keep noise down. So the notification email that reaches your scripts is one worth acting on. It watches; your people and your carefully guarded scripts decide. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep