Building Your First Auto-Remediation End to End
There is a service on one of your application servers that hangs about once a week, and you restart it. You have restarted it so often that you could do it from the car park. You have read about criteria, guardrails, and the things never to touch, and now it is time to build one. This guide takes that single, ordinary problem and walks it all the way from "that is annoying" to "that fixes itself, and I trust it." Nothing here is clever. It is the boring, careful version, which is the version that works. The clever version is shorter, and it is also the one that restarts the service during the backup.
Along the way you will document the manual fix and write a guarded script that does nothing real until you tell it to. You will choose how the script gets triggered without pretending your monitoring tool does things it does not. You will run it in dry-run mode against real alerts for a week. Then you will switch it on for one host and watch. By the end you will have a working automation and, more usefully, a repeatable procedure for building the next one.
In this guide
- Step 1: Pick the candidate
- Step 2: Document the manual fix
- Step 3: Write the script, guardrails first
- Step 4: Decide the trigger
- Step 5: A week in dry-run against real alerts
- Step 6: Enable for one host, watch, expand
- Monitoring the automation itself
- The rollout checklist
- Where to go next
Step 1: Pick the candidate
The candidate for this walkthrough is a Windows service — call it SyncAgent — on an application server called APP01. Roughly weekly, it stops responding. The service still shows as Running. But the port it listens on stops answering, the sync jobs it runs pile up, and users notice within an hour. A restart fixes it every time. The vendor's guidance is a restart. It has been like this for months.
Run it through the five criteria from Safe Candidates for Auto-Remediation. It is well understood: the restart releases whatever it has leaked; you have done it dozens of times. It is reversible: the undo is to stop it. It is idempotent: a restart of a healthy service is harmless, and the script will check before acting anyway. It is frequent: weekly, often out of hours. And it has a low blast radius: one service, one host, jobs that can wait ten minutes. Full marks. The one thing worth noting is that "hung but Running" is a slightly harder problem than "Stopped." That is because the script has to detect the hang itself rather than just read the service state. That is fine; it just means the precondition is a probe rather than a status check.
Step 2: Document the manual fix
Before writing any code, write the runbook — the step-by-step procedure a colleague could follow at three in the morning. If it already exists, re-read it; if not, this is the moment, and First Runbooks shows the shape. The point is not bureaucracy. The runbook is the specification for the script, and writing it forces you to notice the checks you do without thinking.
RUNBOOK: SyncAgent hang on APP01
Symptom: Monitoring reports "SyncAgent port not responding" on APP01.
Service shows Running in the services console.
Check 1: Is the port really dead? Test the TCP port from the server.
If it answers, the alert is stale - do nothing, close the alert.
Check 2: Is the server otherwise healthy? Free memory, free disk,
no reboot in progress, no patching window open.
Check 3: Has anyone restarted it in the last hour? If so, do not
restart again - something else is wrong. Escalate.
Fix: Restart the SyncAgent service (graceful restart, not a kill).
Wait: Give it 45 seconds to come up.
Verify: Test the TCP port again. If it answers, done.
If not, do NOT loop - escalate with what you found.
Undo: Stop the service (if a restart was somehow the wrong call).
Note: Log the time and outcome in the ticket.
Read the runbook back and notice how much of it is not the fix. Three checks, a wait, a verification, an explicit "do not loop," and an undo. Those are your guardrails, already designed, in plain language. The fix is one line of the twelve, which is the right proportion. Now the script is a translation, not an invention.
Step 3: Write the script, guardrails first
The script below is the runbook, line for line, with the guardrails from Guardrails: Rate Limits, Circuit Breakers, and Rollback built in. It runs on the host itself. It does nothing real without -Execute. It uses a graceful restart, never a process kill, and if the graceful restart fails it escalates rather than trying harder.
param(
[string]$ServiceName = 'SyncAgent',
[int]$Port = 8443,
[int]$AttemptWindowMinutes = 60,
[int]$SettleSeconds = 45,
[switch]$Execute # omit for dry run (the default)
)
$Computer = $env:COMPUTERNAME
$StateFile = "C:\Remediation\state\$ServiceName.last"
$LogFile = "C:\Remediation\logs\$ServiceName.log"
$WindowFlag = 'C:\Remediation\state\MAINTENANCE'
function Write-Log {
param([string]$Message)
$line = "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') [$Computer/$ServiceName] $Message"
Add-Content -Path $LogFile -Value $line
Write-Output $line
}
function Test-ServicePort {
$r = Test-NetConnection -ComputerName 'localhost' -Port $Port -WarningAction SilentlyContinue
return $r.TcpTestSucceeded
}
# Guardrail: stand down during maintenance
if (Test-Path $WindowFlag) {
Write-Log "NO ACTION: maintenance flag present."
exit 0
}
# Check 1: is the port really dead? (independent of the alert)
if (Test-ServicePort) {
Write-Log "NO ACTION: port $Port answers. Alert is stale."
exit 0
}
# Check 2: is the host healthy enough for a restart to help?
$os = Get-CimInstance Win32_OperatingSystem
$freeMB = [int]($os.FreePhysicalMemory / 1024)
$sys = Get-CimInstance Win32_LogicalDisk -Filter "DeviceID='C:'"
$freeGB = [int]($sys.FreeSpace / 1GB)
if ($freeMB -lt 512 -or $freeGB -lt 2) {
Write-Log "ESCALATE: host unhealthy ($freeMB MB RAM, $freeGB GB disk free)."
exit 2
}
# Check 3: recent attempt?
if (Test-Path $StateFile) {
$ago = ((Get-Date) - [datetime](Get-Content $StateFile -Raw)).TotalMinutes
if ($ago -lt $AttemptWindowMinutes) {
Write-Log "ESCALATE: already restarted $([int]$ago) min ago."
exit 2
}
}
# Record the attempt before acting
(Get-Date).ToString('o') | Set-Content $StateFile
# Fix (or dry run)
if (-not $Execute) {
Write-Log "DRY RUN: would run Restart-Service '$ServiceName' now."
exit 0
}
try {
Write-Log "ACTION: graceful restart."
Restart-Service -Name $ServiceName -ErrorAction Stop
} catch {
Write-Log "ESCALATE: graceful restart failed: $($_.Exception.Message)"
exit 2
}
# Wait, then verify
Start-Sleep -Seconds $SettleSeconds
if (Test-ServicePort) {
Write-Log "SUCCESS: port $Port answers after restart."
exit 0
}
Write-Log "ESCALATE: port $Port still not answering after restart."
exit 2
A few design notes. The precondition is a real probe of the port, not a reading of the service status. That is because the status is exactly what lies in this scenario. The host-health check is deliberately crude — two numbers. Its job is to catch the obvious cases where a restart cannot help, not to diagnose the server. The attempt is recorded before the restart. The restart is inside a try block so that a hung service which refuses to stop gracefully produces an escalation, not a force-kill. Deciding to kill a stuck process is a human's call. And every exit writes a log line first.
Store the script in version control from the first line. Run it as an account that can restart that one service and write to those two folders and nothing else. Put the runbook's undo step in a comment at the top. Then run the guardrail tests: a fake recent timestamp in the state file, the maintenance flag present, the script run while the port is healthy. Each should produce the right log line and touch nothing.
Step 4: Decide the trigger
A script needs something to run it when the problem occurs. This is the point where people tend to assume their monitoring tool will do it, and sometimes it can. Some monitoring systems can launch a script when an alert fires. If yours does, the guardrails above still apply in full. If it does not, or you would rather keep the automation independent, there are two generic patterns that work with any monitoring tool. Both are worth understanding.
Pattern A: the alert email as the trigger
Nearly every monitoring system sends alert notifications by email. Point those notifications at a dedicated mailbox, and run a small scheduled script every minute or two. The script reads that mailbox and looks for unread messages whose subject matches the alert you care about. The host name and the check name are usually in the subject. For a matching message, the script launches the remediation for that host. Mark the message read. Log what was launched. The remediation itself then re-checks everything, so a stale or duplicate email costs nothing. The advantages: it works with any monitoring tool, and the mailbox is a visible, auditable queue. The disadvantages: a delay of a minute or two, and a dependency on mail flow. That dependency means the automation goes quiet if the mail relay does — something you will want to notice.
Pattern B: the scheduled self-check
Alternatively, skip the alert entirely as a trigger and run the remediation on a schedule — every five minutes, on the host, from a scheduled task. Because the script's first act is to probe the port and do nothing if it answers, running it constantly is harmless. It is idempotent by design. Monitoring then becomes the second opinion: it still alerts if the service is down. If you see an alert that the remediation did not resolve, that is your escalation. The advantages: no parsing, no mail dependency, fastest reaction. The disadvantages: the script now runs hundreds of times a day. So its logging needs to be quiet on the "nothing to do" path or the log becomes noise. (A log that says "nothing to do" every five minutes is not a log. It is a diary.)
For a first automation, Pattern B is simpler and has fewer moving parts. Pattern A becomes valuable once you have several remediations and want one place where "something fired" is visible. Whichever you choose, the monitoring system's job is unchanged: it detects, it alerts, and its dashboards show whether the fix held. The script is yours.
Tip: whichever trigger you use, make the alert that reaches humans slower than the remediation. If the remediation runs within five minutes of the hang and the human notification waits ten, a successful fix means nobody is woken. If the fix fails, the human still hears about it, with the remediation's log to read.
Step 5: A week in dry-run against real alerts
Deploy the script with its trigger, but without -Execute. For a week, it will run exactly as it will in production, checking everything, recording attempts, and logging "would restart" instead of restarting. Meanwhile, you keep fixing the hang by hand as before. At the end of the week, read the log next to the alert history and ask three questions.
- Would it have acted when I did? Every time you restarted the service by hand, there should be a "DRY RUN: would restart" line within a few minutes. If there is not, the trigger or the probe missed it.
- Would it have acted when I would not have? Any "would restart" line with no corresponding real hang is a false positive. Find out why — a probe timeout too short, a port that briefly closes during a legitimate job — and fix the precondition.
- Did the guardrails fire correctly? Look for "port answers, alert is stale," "already restarted," and "maintenance flag" lines. Each should correspond to a situation you recognize.
A healthy dry-run week, on the scheduled-self-check pattern, looks something like this once the quiet "port answers" lines are filtered out. Dates are shortened to weekdays for width:
Tue 02:31:10 [APP01/SyncAgent] DRY RUN: would run Restart-Service 'SyncAgent' now. Tue 02:36:11 [APP01/SyncAgent] ESCALATE: already restarted 5 min ago. Tue 02:41:12 [APP01/SyncAgent] ESCALATE: already restarted 10 min ago. Tue 03:05:40 [APP01/SyncAgent] NO ACTION: port 8443 answers. Alert is stale. Sat 09:00:02 [APP01/SyncAgent] NO ACTION: maintenance flag present.
That tells a complete story. The hang at 02:31 was caught. The attempt limit correctly refused to "restart" again on the following runs. The port came back at 03:05 — which is when you restarted it by hand. And the Saturday patching window was respected. If your log reads like that, the automation is ready.
If the week is quiet and the hang did not happen, extend the dry run rather than enabling on faith. One real event observed in dry-run is worth more than any amount of reasoning. If the log has surprises, fix them and run another week. The dry-run period is cheap; the alternative is discovering the surprise with -Execute on.
Step 6: Enable for one host, watch, expand
Add -Execute for APP01 only. Tell the people who would normally get the alert what is changing and how to turn it off. Removing the switch or creating the maintenance flag turns it off. Both are one-minute actions anyone can do. Then wait for the next hang and read the log. You are looking for the sequence: probe failed, host healthy, no recent attempt, action, success. If you see it, and the users did not notice anything, the automation has done its job once.
The off-switch deserves a sentence of its own. Anyone on the team should be able to stop the automation in under a minute without reading the script. Creating the maintenance flag file does it, and so does disabling the scheduled task. Write both on the runbook page, in bold, above everything else. An automation nobody knows how to stop is the one that runs all the way through an incident.
At Tollgate Foods the warehouse-sync service went down properly one evening, with a corrupt queue file underneath it. The on-call engineer began by stopping the service so she could move the file aside. Forty-five seconds later the service was running again. She stopped it again, and forty-five seconds later it was running again. It did this nine times while she looked for whoever was doing it. Nobody on shift knew that a remediation existed, let alone where it lived. The person who had written it had left the company. The remediation had no attempt limit, no off-switch anyone had been told about, and no entry in the runbook. It was found two hours in, as a scheduled task on the host, running as a service account named after a project nobody remembered. The fix took ten minutes, once she was allowed to perform it.
Once is not proof. Let it handle several real events before you consider it trustworthy, and read the log after each one, including the boring successes. Expansion, when it comes, is deliberate: add one more host at a time to the allowlist. Each needs its own dry-run period if its behaviour differs. Never enable a new host on the same day as a change to the script. A hung service on a second server is a new candidate that happens to share a script, not an automatic extension of the first.
Monitoring the automation itself
An automation is a small system, and small systems fail in three ways you have to watch for. They fail loudly, they fail quietly, and they stop running altogether. The loud failures are the easy ones — the escalation lines in the log reach a person by design. The quiet ones are worse. A script might keep logging "port answers" because the probe target was changed, while the service actually hangs. Or it might log success every time because the settle time is long enough that the port answers briefly before the service hangs again. Read the log against reality, not against itself.
The most dangerous failure is silence. A scheduled task got disabled during a patch, a mailbox's password expired, or a log folder filled up. Each produces an automation that has stopped and a log that simply ends, and nothing tells you. Give every automation a heartbeat: a line it writes on every run, even the "nothing to do" ones. A single daily summary line is enough for a frequent schedule. Add a separate check that complains if that line is older than it should be. The same "no system can be its own witness" principle that applies to the monitoring server applies here. And Monitoring System Failure Modes is the guide to the ways a watcher goes quiet.
Finally, count. How many attempts this month, how many successes, how many escalations, how many times a human still had to act on this service. You are not looking for a target number; you are looking for the trend. A restart automation whose attempt count climbs from weekly to daily is telling you the underlying problem is getting worse. That story continues in Measuring, Auditing, and Retiring Automations.
The rollout checklist
| Stage | Done when | Evidence |
|---|---|---|
| Candidate chosen | Scores 9 or 10 on the five criteria; no 0 in reversible or blast radius | Scoring sheet in the automation's documentation |
| Manual fix documented | Runbook has checks, fix, wait, verify, undo, and an explicit "do not loop" | Runbook page, followed by hand at least once from the text alone |
| Script written | Every runbook step has a line; dry-run is the default; graceful actions only; every exit logs | Script in version control with the undo step in a header comment |
| Guardrails tested | Each guardrail made to fire on purpose, on a test host | Test notes listing each guardrail and the log line it produced |
| Trigger chosen | Mailbox poll or scheduled self-check in place; human alert slower than the remediation | Scheduled task or mailbox rule documented; a test run logged |
| Dry-run week complete | At least one real event observed; no false "would act" lines unexplained | Log excerpt beside the alert history for the same week |
| Enabled for one host | Colleagues told; off-switch documented; first real fix reviewed in the log | Log sequence: probe failed, healthy, no recent attempt, action, success |
| Heartbeat in place | Something complains if the automation goes silent | A deliberately disabled task produced a complaint |
| Expansion | One host at a time, never on the same day as a script change | Allowlist history in version control |
Where to go next
A weekly annoyance is now a script that checks before it acts, acts gently, verifies, and hands off when it cannot help. And you proved it in dry-run before it touched anything. The procedure is the reusable part: pick, document, script with guardrails, choose a trigger, dry-run against reality, enable one host, watch, expand. Run the next candidate through the same steps and it will take half the time. It will still not be clever. That is the point, and it is also why you will be able to find it, stop it, and explain it in a year.
Before the next candidate, though, check it against What Never to Automate (and Why), because the second automation is where confidence starts to outrun caution. And plan for the day the automation should be retired. A restart that fires every week is a bug report with a workaround, not a permanent fixture. The full set of guides lives on the Auto-Remediation: Letting Monitoring Fix Things (Carefully) pillar page.
Put this into practice with SysPeep Monitor
The trigger for the script in this guide is an alert you can trust. That is what SysPeep Monitor is built to send. It watches service and host health over WMI, SNMP, and its other collectors. It applies thresholds and dependency rules so the email it sends means a real problem. Its dashboards show at a glance whether the service came back after your restart. Your script does the fixing; SysPeep does the watching. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep