A Change Workflow That Keeps Backups, Diffs, and Alerts in Sync
You have nightly backups, a diff that runs every morning, and a golden config per role. Then the drift alerts start arriving, and the reply to almost every one of them is "oh, that was me". A colleague changed a port VLAN on Tuesday and mentioned it in passing. You yourself added a route on Thursday during a call and forgot. The contractor updated the firewall and emailed someone who was on leave. Every alert is explained, eventually, and after a month of explaining, everyone has quietly stopped reading them. The detector is fine. What is missing is the habit that makes its output meaningful.
That habit is a change workflow. It is a short, fixed sequence that every configuration change goes through. By the time the diff runs, the change already has a record and a before-and-after pair of backups. The change also has a declared window in the monitoring system and a verification that what changed is what was meant to change. When planned changes reliably go through the loop, the only diffs left without an explanation are the ones that genuinely need one. A drift alert then becomes something people read again.
This guide lays out the loop step by step and gives you a change record template you can copy. It explains what the loop buys you and shows how to handle the emergency change that cannot wait for paperwork. It describes the version that works for a team of two and suggests how to tell whether it is working without inventing numbers. It draws on the sibling guides in this topic and assumes their pieces exist: automated backups, the daily diff, and golden configs.
In this guide
- Why detection needs a workflow
- The loop, step by step
- The change record template
- What the loop gives you
- Emergency changes
- The small-team version
- Measuring whether it works
- Where the loop usually breaks
- Where to go next
Why detection needs a workflow
A drift alert is only as useful as the answer to one question: should this have happened? The diff cannot answer it; it only knows that lines changed. The answer lives in whether a person wrote the change down before making it. If nobody writes anything down, every diff is a mystery and every alert is noise. If everybody writes everything down, every diff has a matching record. The alerts that remain are then the real drift — the contractor, the forgotten script, the change nobody on the team made at all.
So the workflow is not bureaucracy layered on top of detection. It is the other half of detection. A change record is the thing the diff gets compared against. A backup taken immediately before a change is the thing you roll back to. A backup taken immediately after is the thing that lets you verify the change. You can do that before the nightly run blends it with anything else that happened that day. A declared maintenance window is what stops the monitoring system paging you about the outage you are deliberately causing. Each step exists because a specific later step needs its output.
The loop, step by step
Here is the whole loop. It reads longer than it is; for a routine change on one device, the entire sequence takes a few minutes beyond the change itself.
| Step | What happens | What it produces | Which later step needs it |
|---|---|---|---|
| 1. Record the change | Write down what, why, when, and who, before touching anything | A change record with an identifier | The diff (step 6) is matched against it; the audit trail depends on it |
| 2. Declare the window | Tell the monitoring system which devices will be worked on and for how long | A suppression window on the affected devices | Keeps step 4 from paging the on-call person |
| 3. Back up before | Run the on-demand backup for the affected device(s) | A committed "before" configuration | Rollback; the "before" side of step 6 |
| 4. Make the change | Do the work, save the configuration on the device | The changed running and startup configs | Everything after |
| 5. Back up after | Run the on-demand backup again | A committed "after" configuration | The "after" side of step 6 |
| 6. Diff verifies intent | Compare before and after; confirm every changed line is one you meant to change and nothing else moved | A diff attached to the record | Closes the record; nightly drift alert already has its explanation |
| 7. Update the golden if the standard moved | If the change was a new standard rather than a one-off, change the role's golden too, with its own record | An updated golden and a commit | Tomorrow's compliance run does not flag the fleet |
| 8. Close the record | Mark it done, note the before and after commits, end the window | A closed record with links to both backups | Future investigations; the measurement in this guide |
A few of the steps deserve a note. Declaring the window (step 2) is the monitoring system's side of the story. A planned change should not generate an outage alert, and a maintenance window is how the system is told. Whether you schedule windows in advance or declare them at the moment of the change is a question with its own tradeoffs. Those are covered in the guide on scheduled versus ad-hoc suppression. For the loop, what matters is that the window exists and ends when the work does.
Backing up before (step 3) is the step people skip because "last night's backup is fine". It usually is — unless something else changed on that device today, in which case rolling back to last night undoes that too. The on-demand backup takes seconds and removes the doubt. Backing up after (step 5) is the one that pays off in step 6.
Diff verifies intent (step 6) is the heart of the loop and the part that is genuinely new for most teams. You meant to change one port's VLAN. The diff shows that line — and, if you fat-fingered a command, also the line you did not mean to change. Reading the before-and-after diff while still logged in is the cheapest possible review. It catches the "I meant port 17 but typed 7" class of mistake before a user does. If the diff shows exactly what the record says, paste it into the record and move on. If it shows more, fix it now, back up again, and diff again.
The change record template
The record does not need a system. It needs a consistent set of fields, kept somewhere everyone can read and search. A ticketing tool is fine if you have one; a shared document or a text file in the backup repository is fine if you do not. The template below fits on a screen and takes a minute to fill in for a routine change.
CHANGE RECORD CR-0412
What: sw-floor2-01, port Gi1/0/17 access VLAN 50 -> 30 (guest)
Why: phone vendor migration, request in ticket 4471
When: Tue 17:30-17:45 window declared in monitoring: yes
Who: made by: A. Admin approved by: n/a (routine, single port)
Risk: phone gateway loses internal reachability if VLAN is wrong;
the phone system's owner would notice within minutes
Rollback: restore the "before" commit; one line
Backup before: commit 3f9c2a1 17:28
Backup after: commit 8b04e77 17:41
Diff matches intent: yes (one line changed, as above)
Golden updated: n/a
Closed: Tue 17:46 by A. Admin
Two fields earn their place more than the others. Risk forces the ten-second thought "what breaks if I get this wrong, and who notices?", which is most of what a formal change review provides. Rollback is nearly always "restore the before commit". Writing it down every time means that at two in the morning nobody has to think about it.
What the loop gives you
Drift alerts you can trust. When every planned change has a record, the diff script can look for one before alerting. In that case, a commit that falls inside a declared window for that device, with a record that names it, is logged and annotated automatically. The alert that reaches a human is the one with no record, and that is a rare event worth a human's attention. The signal-to-noise ratio of the whole drift practice depends on this single effect.
An audit trail. Every change now has, in one place, what was done, why, by whom, and when. It also has the exact lines that changed and what the device looked like on either side. An auditor, a manager, or your future self may ask "why does this firewall have that rule?" The answer is a search of the records rather than an archaeology project. When an incident starts and the first question is "what changed recently?", the records and the commit history answer it in a minute. The guide on change correlation covers using that trail under pressure.
Faster rollback. A change that goes wrong is undone by restoring a specific, known-good commit taken minutes before. You do not have to guess which of last night's backups predates the problem. The rollback field in the record already says which commit. That difference — between "restore the before commit" and "figure out what state we were in" — is the difference between a five-minute blip and a bad evening.
There is a quieter benefit too. A team that reads its own before-and-after diffs learns its own network. Patterns emerge — the same fix on the same kind of device, the same forgotten line. Those patterns become golden-config rules. That is how a standard grows from practice rather than from a document nobody consulted.
Emergency changes
Sometimes the network is down and the fix is a line on a router, and nobody is going to fill in a form first. Good. The loop is not a gate; it is a record. The rule for emergencies is simple: do it, then paper it the same day.
In practice that means: make the change and restore service. As soon as things are stable, run the on-demand backup so the "after" state is captured before anything else moves. Then write the record with the timestamps you actually have. The "before" backup is last night's, and the record says so. Mark the record as an emergency. Declare the window retroactively if your monitoring system allows it, so that the outage alerts are annotated rather than left as unexplained noise. Diff last night's backup against the emergency "after" backup. Confirm that the only changes are the ones you made in the heat of the moment. Emergencies are exactly when a stray extra line gets typed. Close the record.
"The same day" is the important part. An emergency change papered the same afternoon is a normal record with an unusual order of steps. An emergency change papered next week has already produced a drift alert, been explained by memory, and possibly been half-forgotten. Count emergency changes separately in the measurement below. A rising count is a sign that too much work is being done in a hurry, which is a planning problem, not a workflow one.
Tip: give the emergency path a name and a one-line shortcut. One example is a script that takes a backup and creates a pre-filled record marked "emergency" in one command. The easier the paperwork is after the fact, the more reliably it gets done.
The small-team version
Everything above scales down to a team of two, or one, without losing its point. The full version needs three things, and none of them is a product.
- A shared form. Keep the change record template above as a document everyone can append to. Or keep one text file per change in a
changes/folder in the backup repository. The repository version has a pleasant property: the record and the backups it references are committed together, so a single history contains both. - A repository. The same one the backups live in. Records, goldens, exceptions, and configurations in one place, with one access control and one history.
- A nightly script and an on-demand one. The nightly job pulls, diffs, checks compliance, and alerts. The on-demand script is the same code with a hostname argument, and it is what steps 3 and 5 call. A two-person team runs the on-demand script by habit. "I am about to touch the core switch" and "done with the core switch" each mean running it once.
What the small team skips is approval. A single-port change made by the only network person does not need a second signature, and pretending it does breeds a workflow people route around. Keep the record, keep the backups, keep the diff; drop the ceremony. If the team grows, approval can be added to the template as a field. That field is filled in for changes above a risk line you agree on — core devices, firewall policy, anything touching authentication. It is left as "n/a" for the rest.
The other thing a small team can afford to do is read every diff. On a network of a few dozen devices, the nightly diff output is usually empty and occasionally three lines. Reading those three lines each morning, with coffee, is the cheapest change review process ever devised and catches more than most formal ones.
Measuring whether it works
You do not need a dashboard for this, and you should not invent targets. Two counts, kept weekly in the simplest way you can manage, tell you everything.
- Changes with records. The number of change records closed this week. This comes from the records themselves.
- Drift alerts without records. The number of diffs this week that could not be matched to a record and had to be explained after the fact. This comes from the diff loop's explanations file.
What you want is for the second number to fall toward zero and stay there while the first stays roughly level. If both are zero, either the network is genuinely static or the backup job has stopped and nobody noticed; check which. If the second number is large and the explanations are all "that was me", the workflow is not being followed. In that case, the fix is habit, not tooling. If the second number is small but the explanations include names you do not recognize, the workflow is working exactly as intended. It has found the thing it exists to find.
Two secondary counts are worth a glance. Emergency changes per week, as discussed above. And time from drift alert to explanation: a rough sense of whether alerts are answered the same day or a week later. If the answer is drifting toward a week, the alerts are going to the wrong person or the wrong inbox.
Where the loop usually breaks
The loop fails in a small number of predictable ways, and it is easier to watch for them than to discover them by accident.
- The record is written afterwards, from memory. The "what" field ends up vague and the diff ends up doing the record's job. Fix: make the record the first step, and make it fast enough that there is no reason to defer it.
- The window is declared but never ended. Suppression that outlives the change hides the next real outage on that device. Fix: the close step ends the window, and the monitoring system should alert on windows that run long.
- The "after" backup is skipped because the nightly run will get it. Two changes on one day then blend into one diff, and verifying intent becomes guesswork. Fix: the on-demand script is one command; run it.
- The golden is not updated when the standard moves. The next compliance run flags every device in the role, and the report loses credibility. Fix: step 7 is part of closing the record, not a separate project.
- Tooling made changes nobody recorded. A configuration-management tool, a controller pushing a template, an automated remediation script. These are legitimate changes with no human record. Fix: the tool's runs should write a record too, or the diff script should recognize the tool's account in the device logs and annotate accordingly.
Where to go next
The loop is eight small steps. Record the change, declare the window, back up before, and make the change. Back up after, and diff to verify intent. Update the golden if the standard moved. Close the record. Each step exists because a later step needs its output. Together they turn drift detection from a source of noise into a source of the one signal it was built for: the change nobody announced. Emergencies go through the same loop in a different order, the same day. A team of two runs it with a shared form, a repository, and two scripts. And two weekly counts — changes with records, drift alerts without — tell you whether the habit has taken.
This guide closes the topic. The reasons for all of it are in why configuration backups belong next to monitoring. The mechanics are in automating config backups, diffing configs, and golden configs. If the idea of a versioned, reviewable loop appeals to you for the monitoring configuration as well, the Monitoring as Code topic takes that further. The full set lives under Config Backup and Drift Detection Alongside Monitoring.
Put this into practice with SysPeep Monitor
Step 2 of the loop is where the monitoring system joins the workflow. SysPeep Monitor does not manage configuration changes or diffs. Those are the companion practice this guide describes. But its alert rules and dependencies are built to keep noise down. So a declared window means the change you are making does not page the on-call person. The SNMP and SSH metrics it collects show you the moment the device settles afterwards. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep