Measuring, Auditing, and Retiring Automations
An automation you built six months ago is running right now, somewhere, doing something. Do you know what? Do you know whether it worked last night or whether it has quietly stopped? Do you know whether the service it restarts still exists, or whether last month's upgrade changed the port it probes? If the honest answer is "probably fine," this guide is for you. "Probably fine" is what every automation says until the day it is not.
The first four guides in this topic are about building remediations carefully. This one is about owning them afterwards, which turns out to be the longer job. An automation is not a task you finished. It is a small system you now run, and like any system it needs measuring, auditing, reviewing, and eventually retiring. Done well, that upkeep is a few minutes a month. Done badly, or not at all, you end up with a folder of scripts nobody understands. They run on servers nobody remembers, fixing problems that may or may not still exist. I have inherited that folder. It held eleven scripts, and the one with the most privileges was called test2.
By the end you will know what to count and what not to, and what an audit trail must contain. You will know how to run a monthly review that takes fifteen minutes, and what every automation's documentation page looks like. You will also know how to recognize the automation that has become a bug report, and how to retire it cleanly.
In this guide
- What to measure (and what not to invent)
- The audit trail
- The monthly review
- Ownership and the one-page record
- Retiring an automation
- Version control for scripts
- The automation register
- The review checklist
- Where to go next
What to measure (and what not to invent)
You measure an automation for two reasons: to know whether it is still doing its job, and to know whether the job still needs doing. Four counts, taken straight from the log, answer both.
- Attempts. How many times the automation actually acted (not counting dry runs or "nothing to do" exits). This is the frequency of the underlying problem, measured honestly.
- Successes. How many attempts were followed by a passed verification. Attempts minus successes is the number of times the fix did not work.
- Escalations. How many times the automation handed off to a human, and why — a precondition failed, the attempt limit was hit, the verification failed. The "why" breakdown is more useful than the total.
- Human actions. How often a person still had to fix this problem by hand, whether because the automation escalated or because it never fired. This one is not in the automation's log; it comes from tickets, or from asking the team.
Time-to-fix matters too, but resist turning it into a number you did not measure. What you can say honestly is qualitative. Before the automation, a hang at two in the morning waited until someone woke up. Now it is fixed within five minutes and nobody wakes. That account is worth more than an invented average. If you want a real figure, the log has timestamps for the alert and the success, and you can compute it. But only claim what the log actually shows.
Pulling the counts from a log in the format used elsewhere in this topic is a one-screen script. It reads a month of lines, counts by outcome, and prints a summary you can paste into the review notes:
param([string]$LogFile, [int]$Days = 30)
$since = (Get-Date).AddDays(-$Days)
$lines = Get-Content $LogFile | Where-Object {
$_ -match '^(\S+ \S+) ' -and [datetime]$matches[1] -ge $since
}
$counts = [ordered]@{
'ACTION' = ($lines | Where-Object { $_ -match '\] ACTION:' }).Count
'SUCCESS' = ($lines | Where-Object { $_ -match '\] SUCCESS:' }).Count
'ESCALATE' = ($lines | Where-Object { $_ -match '\] ESCALATE:' }).Count
'DRY RUN' = ($lines | Where-Object { $_ -match '\] DRY RUN:' }).Count
'REFUSED' = ($lines | Where-Object { $_ -match '\] REFUSED:' }).Count
}
"Summary for $LogFile, last $Days days:"
$counts.GetEnumerator() | ForEach-Object { " $($_.Key): $($_.Value)" }
"Escalation reasons:"
$lines | Where-Object { $_ -match '\] ESCALATE: (.+)$' } |
ForEach-Object { $matches[1] } | Group-Object | Sort-Object Count -Descending |
ForEach-Object { " $($_.Count) x $($_.Name)" }
The script assumes each log line starts with a date and a time in the format the scripts elsewhere in this topic write. Adjust the pattern to match yours. The script's output is deliberately plain, because the numbers are for a person to read once a month, not for a dashboard to display forever.
What not to measure is just as important. Do not report "hours saved" unless you actually timed the manual fix and multiplied honestly. A made-up figure will be repeated in a meeting and become a fact. Do not report a success percentage on five events; say "five attempts, five successes." And do not compare automations against each other on numbers alone. A spooler restart that fires daily and a sync restart that fires monthly are both doing exactly their jobs. The counts exist to answer "is this still working and still needed," nothing grander.
The audit trail
An audit trail is the record that lets someone reconstruct what an automation did and why. That someone could be you, a colleague, an auditor, or you in a year. It is not the same as the operational log, though it is built from it. The audit trail answers five questions for every action taken:
| Question | What to record | Example |
|---|---|---|
| Who | The identity the automation ran as, and which automation | svc-remediation running Restart-SyncAgent |
| What | The exact action and target | Graceful restart of service SyncAgent on APP01 |
| When | Timestamp of the decision and of the action | Decided 02:31:10, acted 02:31:12 |
| Why | The trigger and every check that passed | Port 8443 not answering; 3.9 GB RAM and 41 GB disk free; no attempt in the last 60 minutes; no maintenance flag |
| Result | The verification outcome, and any handoff | Port answering after 45 seconds; SUCCESS; no escalation |
If your log lines already carry all five, the audit trail is simply the log, kept. Keeping is the part people skip. Decide a retention period that outlasts the longest gap between anyone asking "what happened that night?" and the night in question. In practice, that means long enough to cover your incident reviews and any compliance obligation your organization has. And make sure the log is not on the same disk the automation cleans up. Copy it somewhere central on a schedule, so that a host being rebuilt does not take the record with it.
One more thing belongs in the trail: changes to the automation itself. Who enabled -Execute, who added a host to the allowlist, who reset the circuit breaker and why. Version control covers changes to the script; a line in the review notes covers the operational changes. The goal is that no action, by the script or on the script, is untraceable.
The monthly review
Once a month, for each automation, sit with the summary numbers and the log and ask three questions. It takes fifteen minutes for a handful of automations. And it is the single practice that keeps the folder of scripts from turning into a folder of mysteries.
Is it still needed?
Look at the attempt count. Zero attempts for three months means one of two things: the problem went away, or the automation has stopped firing. Both are worth knowing. If the problem went away — a patch fixed the leak, the server was replaced — the automation is a candidate for retirement. If it has stopped firing while the problem persists, the heartbeat should have told you. In that case, the fact that it did not is a finding of its own.
Is it still safe?
Look at the escalations and failures. A rise in "restart failed" lines means the fix is losing its grip on the problem. A rise in precondition failures means the host is unhealthy in ways the restart cannot help. Any refusal or circuit-breaker event is an incident to understand. And look at the allowlist: has anything been added that you would not have added in a calm moment?
Is it still correct after upgrades?
This is the question most reviews forget. Every upgrade to the application, the operating system, or the monitoring tool can silently break an assumption in the script. The service may have been renamed, or the port may have changed. The log format the mailbox parser reads may be different, or the command the script runs may behave differently. Check the change log for the month against the automation's assumptions. Better still, re-run the guardrail tests after any upgrade to anything the script touches. A script that was right last month is not automatically right this month. (The script will not tell you the port changed. It will tell you, every hour, that the port is not answering.)
The review produces one short, dated note per automation. It contains the four counts and one line answering each of the three questions. It also records any action taken (a host added, a test re-run, a retirement started, a bug chased). Six months of those notes, read in order, are the history of the automation and the history of the problem it works around. They are what you hand to whoever inherits it.
Tip: put the review in the calendar as a recurring appointment with a checklist attached, not as a good intention. Automations are designed to be forgotten, and they are very good at it.
Ownership and the one-page record
Every automation has exactly one owner. That is a named person who answers for it, reviews it, and is the one to ask when it does something odd. Not a team, not "IT," a person — because a team owns nothing in particular. When the owner leaves or changes role, ownership transfers explicitly, as part of the handover, and the transfer is written down.
Each automation also has one page of documentation. And the page has a fixed shape so that anyone can read any of them in a minute. This is the page the two-in-the-morning escalation links to:
AUTOMATION: Restart-SyncAgent
Owner: (name) Last reviewed: (month)
Status: ENABLED on APP01 Dry-run on: APP02
TRIGGER: Scheduled self-check every 5 minutes on the host.
Monitoring alert "SyncAgent port not responding" is the
human-facing second opinion, delayed 10 minutes.
ACTION: Graceful restart of the SyncAgent service.
Never a process kill. One attempt per hour.
GUARDRAILS: Port probe before acting; host RAM/disk sanity check;
attempt limit 60 min; maintenance flag honoured;
circuit breaker file; dry-run default.
OFF SWITCH: Create C:\Remediation\state\MAINTENANCE, or disable
the scheduled task "Remediation - SyncAgent".
UNDO: Stop-Service SyncAgent.
LOGS: C:\Remediation\logs\SyncAgent.log, copied nightly to
the central log share. Retained per the log policy.
ROOT CAUSE: Vendor ticket open for the resource leak. This
automation retires when the fix ships and is verified.
Notice the last block. Every automation that works around a bug should say so, and should name what would make it unnecessary. That line is what turns "we have a restart script" into "we have a workaround with an expiry condition."
Retiring an automation
Here is the uncomfortable thing about a restart automation that fires every week: it is a bug report, not a solution. The service hangs because something is wrong with it — a leak, a deadlock, a dependency that times out. The restart hides the symptom without touching the cause. That is a perfectly reasonable trade while the cause is being fixed. It becomes a problem when the automation is so effective that the cause never gets fixed, because nobody feels the pain any more. The solo-admin version of this trap, where a script quietly becomes a permanent system you own, is described in Solo Admin Automation.
At Bellweather Logistics the dispatch service hung every few days. A restart script handled it so well that the vendor ticket, opened the same month, was closed by the vendor for inactivity. Nobody noticed, because nobody felt the hang any more. The script ran for three years, through two server rebuilds, with its register row reading "retire when vendor fix verified" the entire time. When somebody finally chased the vendor, the fix turned out to have shipped in an upgrade eighteen months earlier. The service had not hung since. The script had spent that time probing a port that always answered. The retirement review was short. The action count for the final eighteen months was zero, which the minutes recorded as "working as intended".
So the goal for most remediations is retirement. When the vendor ships the fix, when the server is rebuilt, when the leaking component is replaced — the automation should go. And it should go deliberately:
- Confirm the root cause is fixed. Not "the patch is installed" but "the service has run for a month without the automation acting." Switch the automation back to dry-run and watch: if it keeps logging "would restart," the fix did not take.
- Disable, do not delete. Turn off the trigger. Leave the script and its page in place with the status changed to RETIRED and the date and reason noted. Keep it for a further review cycle in case the problem returns.
- Keep the log. The audit trail outlives the automation; retention applies to it regardless.
- Update the register and the runbook. The manual runbook should note that the automation no longer exists, so nobody waits for a fix that is not coming.
- Archive after a clean cycle. Once a review has passed with no recurrence, move the script to an archive folder in version control. At that point, remove the scheduled task or mailbox rule entirely.
A retired automation is a success story: a problem that was frequent enough to automate, and then fixed properly. Count them. A register with a healthy number of RETIRED rows is the sign of a team that uses automation as a bridge, not a permanent floor.
Version control for scripts
A remediation script is production code. It runs unattended, with privileges, against real servers. And it deserves the basic discipline production code gets. That does not require a build pipeline; it requires a version control repository and four habits.
- Every change is committed with a reason. "Raise settle time to 60 seconds; service takes longer to bind the port after the upgrade" is the message that saves the next reviewer twenty minutes.
- Someone else reads changes to guardrails. The action can be edited freely. The preconditions, attempt limit, allowlist, and dry-run default get a second pair of eyes. That is because those are the lines a well-meaning simplification tends to remove.
- What is deployed is tagged. You should be able to say which version is running on which host, and diff it against what is in the repository. A script edited in place on a server is a script whose history you have lost.
- The tests live next to the script. The guardrail tests from the build guide are a file in the same folder, run after every change.
If your team has no repository, a simple one on a file share is enough to start. The point is not sophistication; it is that a change to something with privileges is never invisible.
The automation register
The register is the single list of every automation that exists, one row each, kept somewhere the whole team can see. It is what you consult when something restarts and nobody knows why. You also consult it when a server is being decommissioned and you need to know what runs on it. And you consult it when the monthly review starts. A worked example, with the columns that earn their place:
| Automation | Target | Trigger | Status | Owner | Last review | Retire when |
|---|---|---|---|---|---|---|
| Restart-SyncAgent | APP01 | Scheduled self-check, 5 min | Enabled | (name) | (month) | Vendor leak fix verified |
| Restart-SyncAgent | APP02 | Scheduled self-check, 5 min | Dry-run | (name) | (month) | Same as APP01 |
| Clear-TempFiles | FILE01 | Alert mailbox: disk C: below threshold | Enabled | (name) | (month) | Application temp cleanup fixed upstream |
| Restart-Spooler | PRINT01 | Alert mailbox: stuck queue | Enabled | (name) | (month) | Never expected; no single root cause to fix |
| Rotate-AppLog | APP03 | Scheduled, daily | Retired | (name) | (month) | Done: application now rotates its own log |
| Propose-PortReenable | Access switches | Alert mailbox: port security violation | Enabled (proposes only) | (name) | (month) | Not applicable; advisory |
The "retire when" column is the one that keeps the register honest. Every row either names its exit condition or admits, like the spooler row, that none is expected. (The spooler row's reason, in full, is "printers".) A row that has said "vendor fix" for a year is a prompt to chase the vendor. Or you can accept that this is now a permanent workaround and document it as one.
The review checklist
| Check | Looking for | If found |
|---|---|---|
| Attempts this month | Zero, or a sharp rise | Zero: confirm the heartbeat, consider retirement. Rise: the root cause is getting worse; escalate the bug |
| Success rate | Attempts that were not followed by success | Read each failure; if the fix is losing effectiveness, revisit the candidate |
| Escalation reasons | Any new reason, or one reason dominating | A precondition that always fails needs a human look at the host, not a tweak to the script |
| Human actions | People still fixing this by hand | Find out why the automation did not fire or did not help |
| Refusals and breaker events | Any at all | Each one is an incident; understand it before resetting anything |
| Upgrades this month | Changes to anything the script touches | Re-run the guardrail tests; check names, ports, formats |
| Allowlist changes | Additions since last review | Confirm each had its own dry-run period |
| Log retention | Logs present, copied centrally, within policy | Fix the copy job before anything else |
| Heartbeat | The "still running" signal is current | Silence means the automation has died; treat as an outage of the automation |
| Page and register | Owner, status, last-review date current | Update now, while the facts are in front of you |
| Retire condition | Has it been met? | Start the retirement steps |
Where to go next
Owning an automation is the long half of building one. Count what the log actually shows, and keep the audit trail somewhere the automation cannot delete it. Ask the three review questions monthly. Give every script one owner and one page. Put the code under version control, and plan for its retirement from the day it is enabled. A remediation that fires weekly is a workaround for a bug. And the best thing that can happen to it is that it becomes unnecessary. Do all of that and "probably fine" turns into "fine, and here is the log", which is the only version of fine worth saying out loud.
If you are arriving here before building anything, start at Safe Candidates for Auto-Remediation. Then work through Building Your First Auto-Remediation End to End. The guardrails that make the log worth reading are in Guardrails: Rate Limits, Circuit Breakers, and Rollback. The full set of guides is on the Auto-Remediation: Letting Monitoring Fix Things (Carefully) pillar page.
Put this into practice with SysPeep Monitor
The clearest sign that an automation should retire is the alert that stops firing — and that is a story monitoring tells best. SysPeep Monitor's reports and dashboards show how often a service really went down, before and after your fix. So the register's "retire when" column is backed by evidence rather than memory. SysPeep Monitor watches and records; your scripts and your review do the rest. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep