Home › Topics › Monitoring as Code › Testing and Rollout

Testing and Rolling Out Monitoring Changes Safely


Every admin has a story about a monitoring change that went wrong. There was the threshold that was meant to apply to one busy server and instead applied to all of them. So at three in the morning, every on-call phone lit up at once. Or take the quieter, nastier one: the template edit that dropped the uplink check from an entire site's switches. It was discovered a month later when an uplink failed and nobody was told. Monitoring changes are small and look harmless. They have a blast radius out of all proportion to their size.


Application developers solved this problem for their own code long ago. Test it somewhere that is not production, apply it to a few users first, watch, then widen. None of that needs a large team or expensive tooling. It needs a second small monitoring install and a script that refuses obviously bad files. It needs a handful of devices nominated to go first and the discipline to wait a day before doing the rest. This guide walks through each of those, then compresses the whole thing into the version a team of one or two can actually run.


It assumes your configuration lives in files in a repository, because rollback through version control is the safety net under everything else. If it does not yet, start with Version-Controlling Your Checks and Thresholds and come back.


In this guide

  • Why monitoring changes deserve testing
  • Too loud and too quiet: the two failure modes
  • A staging instance
  • Validation before apply
  • Canary devices
  • Rollout waves and watching the alert volume
  • Rollback through version control
  • Change windows for risky changes
  • The pre-apply checklist
  • The small-team version
  • Where to go next

Why monitoring changes deserve testing


A monitoring system is unusual among the things you administer because its failures are felt by people rather than by machines. A bad change to a file server makes a file share slow. A bad change to monitoring wakes up three colleagues or trains them to ignore the pager. Or it leaves a site unwatched while everyone believes it is fine. The cost lands on trust, and trust is slow to rebuild.


The second reason is leverage. In a templated setup, one line in a role file changes every device of that role. One line in a routing rule changes who is paged for everything. That leverage is exactly why templates are worth having. And it is also why a typo in one is not a typo in one place. The more of your configuration is generated, the more each change deserves a look before it is applied everywhere.


The third reason is that monitoring changes are easy to test and cheap to stage compared with almost anything else in IT. You do not need a copy of the production network, only a second small instance of the monitoring software pointed at a slice of it. The excuse "we cannot test that" is rarely true here.


Too loud and too quiet: the two failure modes


Bad monitoring changes fail in one of two directions, and the two need different tests.


Too loud is a change that produces alerts that should not exist. Examples are a threshold lowered too far or a check added to devices that cannot answer it. Another is a dependency removed so that one router outage becomes forty device alerts. Loud failures are unpleasant but self-announcing. Someone notices within hours, and the fix is usually a rollback.


Too quiet is a change that removes alerts that should exist. Examples are a check dropped from a template or a threshold raised beyond any value the device will ever reach. Another is a routing rule that sends alerts to a group with no members. So is a credential name misspelled so that collection silently fails. Quiet failures announce nothing. They are discovered when something breaks and the alert that should have fired did not, which can be weeks later. Every test in this guide is designed to catch quiet failures as well as loud ones, because loud ones would be caught anyway.


A staging instance


A staging instance is a second, small installation of your monitoring software, fed from the same repository as production. It monitors a representative slice of the network — one site, or one device of each role — using read-only credentials. Its notifications go to a test inbox rather than to anyone's phone. If you have rehearsed a rebuild as described in Reproducible Setups, you already have one. The lab VM (virtual machine) you rebuilt on is the staging instance. It is kept current by applying the repository to it.


Staging is where a change is applied first, before any canary. It catches an entire class of problems that cannot page anyone but would stop a production rollout dead. Examples are files the tool refuses to import or a reference to a group or credential that does not exist. Others are a generator that produces one file too few or a dashboard export that no longer loads. It also lets you see the change from the console's point of view. The diff says a check was added. Staging shows whether the check actually returns a value on a real device of that role.


Be honest about what staging cannot do. It cannot tell you how loud a threshold will be across two hundred devices, because it has ten. It cannot tell you whether the after-hours routing works, because it routes to a test inbox. Those answers come from canaries and waves. Staging answers "does this apply cleanly and do the checks work at all?", which is the question you most want answered before touching production.


Keep staging small and keep it boring. Use a modest VM, an embedded database, and one site's worth of devices. Add a scheduled job that applies the main branch of the repository every night so that staging never drifts far from production. Check the product documentation for licensing or configuration of a second instance. Many products allow a small secondary install for exactly this purpose.


Validation before apply


Before anything reaches even staging, a script should refuse the mistakes that a human reviewer is worst at spotting. Those are the missing field, the transposed number, and the threshold that is impossible rather than merely wrong. This is linting — an automated read-through that checks structure and sanity, not intent.


A useful lint for monitoring configuration checks several things, at minimum. It checks that every file parses as valid YAML, JSON, or CSV (the plain-text formats the configuration is written in). It checks that every required field is present in every role and every inventory row. It checks that warning thresholds are below critical ones, that percentages are between zero and one hundred, and that polling intervals are sane. It checks that every dependency names a device that exists in the inventory. It checks that every credential and notification group named in a role exists in the list of known names. And it checks that nothing that looks like a secret value appears anywhere. Here is a sketch of the shape such a script takes. The helper functions are yours to write for your own file formats.


# scripts/lint.py -- refuse to apply anything that would page people by mistake
import sys
from pathlib import Path
from helpers import load_role, inventory_rows, known_credentials, known_groups

problems = []

for path in sorted(Path("roles").glob("*.yaml")):
    role = load_role(path)                      # parses the file; raises on bad syntax
    device = role["device"]
    for name, check in device["checks"].items():
        warn, crit = check.get("warn"), check.get("critical")
        if warn is not None and crit is not None and warn >= crit:
            problems.append("%s: %s: warn %s is not below critical %s" % (path, name, warn, crit))
        if name.endswith("_percent") and crit is not None and not 0 < crit <= 100:
            problems.append("%s: %s: critical %s is not a percentage" % (path, name, crit))
        if check.get("interval_minutes", 1) < 1:
            problems.append("%s: %s: interval below one minute" % (path, name))
    for target in device["alert_routing"].values():
        if "$" not in target and target not in known_groups():
            problems.append("%s: routing target %s does not exist" % (path, target))

names = set(row["name"] for row in inventory_rows())
for row in inventory_rows():
    if row["uplink"] and row["uplink"] not in names:
        problems.append("inventory: %s depends on unknown device %s" % (row["name"], row["uplink"]))
    if not Path("roles/" + row["role"] + ".yaml").exists():
        problems.append("inventory: %s has unknown role %s" % (row["name"], row["role"]))

if problems:
    print("\n".join(problems))
    sys.exit(1)
print("lint: OK")

Run the lint in two places. Run it automatically before every commit, so that a broken file never enters the history. Run it again at the start of the apply script. That way, nothing broken reaches a monitoring instance no matter how it got there. A lint that fails is not an inconvenience; it is a page that did not happen. Add a rule every time a mistake gets past the existing ones. The script grows into a record of everything that has ever gone wrong, and it never forgets.


Canary devices


A canary is a device that receives a change before the rest. The name comes from the caged birds that miners carried underground: if the air went bad, the bird showed it first. In monitoring terms, you apply the new template or threshold to a handful of real production devices. Let them run under real conditions, and watch what happens before applying the change to the fleet.


Choose canaries for representativeness, not importance. Choose one device per affected role, from a site that is neither the quietest nor the busiest. Prefer devices whose behaviour you know well enough to recognise "different." The core switch at headquarters is a terrible canary: if the change is wrong, you have made your most important device wrong first. A branch access switch in a mid-sized office is a good one. Write the canary list down in the repository so that the same devices go first every time and their history is easy to compare.


Let the canaries run for at least one full day, including a night. Many bad thresholds only show themselves at backup time or under the overnight quiet. While they run, look for both failure modes. Look for any new alert from a canary that you cannot explain. And make the quiet check: every check the change touched should still be returning values on every canary. If a check went from "green" to "no data," that is not a pass. For changes to alert routing, trigger a test alert on a canary deliberately and confirm it arrives where the new rule says it should.


Rollout waves and watching the alert volume


After the canaries, the change goes out in waves: successively larger groups of devices, with a pause after each to look at the effect. The pause is the whole point. A wave with no pause is just a rollout with extra steps.


WaveScopeWait before the nextGo / no-go signal
0. StagingThe staging instanceOne polling cycle, plus a look at the consoleImports without error; every touched check returns data
1. CanaryOne or two devices per affected roleA full day including a nightNo unexplained alerts; touched checks still report; test alert routed correctly
2. One siteAll affected devices at one representative siteOne to two daysAlert count per rule for the site within its normal range
3. Half the fleetRoughly half of all affected devices, mixed sitesTwo daysFleet-wide alert count per rule within normal range; no "no data" checks
4. EverythingAll remaining devicesWatch for a weekSame signals; then tag the commit as known-good

The signal you watch between waves is alert volume per rule, compared with what it was before. Before you start, write down how many times each affected rule fired in the previous week. Most tools can report this, and if not, counting the notification emails works. After each wave, count again for the devices in the wave. A rule that fired twice a week and now fires twenty times a day is too loud. A rule that fired daily and now never fires is worth a suspicious look. The world rarely gets that much better overnight. That is the quiet failure, and volume comparison is one of the few ways to see it.


Waves do not need to be large or numerous. For a change that touches ten devices, "canary, then the rest" is two waves and entirely adequate. For a change that touches every device in the fleet or every routing rule, all five rows earn their place. Match the number of waves to the blast radius, not to a rule.


Tip: a change to a dependency or a suppression rule deserves an extra test that most people skip: make sure alerts still happen. After applying it to a canary, pick a downstream device. Confirm that its alert fires when its parent is fine and is suppressed when its parent is down. Dependency changes fail quiet more often than any other kind.


Rollback through version control


When a wave shows a problem, the response is to roll back. Rollback in a monitoring-as-code setup is two steps: revert the commit, then run the apply step again. The revert creates a new commit that undoes the old one, so the history records both the change and its reversal. Nothing is lost and nothing is hidden.


$ git log --oneline -3
b8e0d12 Raise CPU warning on file servers to 90% after the threshold review (ticket 4488)
7b2d44f Nightly export
3e9c1a0 Lower disk thresholds on file servers after the FS02 full-disk incident (ticket 4471)
$ git revert b8e0d12
[main e1c9a07] Revert "Raise CPU warning on file servers to 90% after the threshold review (ticket 4488)"
 1 file changed, 1 insertion(+), 1 deletion(-)
$ scripts/apply.ps1 -Scope file-server
lint: OK
generated 14 device definitions
imported 14 devices, 0 errors

Two habits make rollback reliable. First, before any rollout, write the name of the commit or tag you would roll back to on the change record. Deciding this in advance takes ten seconds and saves a panicked search later. Second, roll back the whole change, not part of it. Patching a half-rolled-out change with a second hurried change is how you end up with three states in the fleet. Then you have no idea which device is in which. Revert to known-good, let things settle, then try again properly.


Rollback is not an admission of failure. It is the reason you can afford to make changes at all. A team that can undo a change in two minutes will make more improvements, more often, than a team that treats every change as a one-way door.


Change windows for risky changes


Most monitoring changes can go out during the working day, because a working day is when people are awake to notice a loud failure. A few deserve a change window: a scheduled time, announced in advance. During that time, the person making the change is present and watching, and someone else knows it is happening. The on-call rota has been warned that alerts may be noisy for an hour.


The risky changes are the ones with fleet-wide leverage or the ones that touch the path an alert takes to a human. Examples include anything to notification routing, escalation, or schedules, and a template change that affects every device of a common role. Others include removing or restructuring dependencies, changes to the notification channels themselves, and, biggest of all, upgrading the monitoring software. For these, the window should be during working hours, not overnight. The instinct to make risky changes at midnight "when nobody is affected" is exactly backwards for monitoring. A quiet failure made at midnight has the whole night to go unnoticed.


Tell the on-call person before the window opens and after it closes. If the change turns out to be loud, they know why and can wait for the rollback rather than investigating. If it turns out to be quiet, they know to be a little more suspicious of silence for the next few days.


The pre-apply checklist


Run through this before applying any change that affects thresholds, templates, dependencies, or routing. It takes five minutes, which is less than one false page costs.


  • The diff matches the message. The change touches only the files and devices the commit message claims. Nothing else moved.
  • Lint passes. Syntax, required fields, warn below critical, references resolve, no secrets.
  • A second person has read the diff for anything that pages people. For a team of one, you have read it aloud.
  • Staging imported it cleanly and every touched check returns data there.
  • Canaries are named. The list of devices going first is written down.
  • The baseline is recorded. Alert count per affected rule for the past week, so "louder" and "quieter" mean something.
  • The rollback target is written down. The commit or tag to revert to, and the apply command to run afterward.
  • The test notification target is verified if routing changed: you know where a test alert should land and you will send one.
  • The on-call person knows, and a change window is booked if the change is on the risky list.
  • The wave plan is chosen to match the blast radius: two waves for ten devices, five for the fleet.

The small-team version


Everything above scales down further than it looks. If you are one or two people, here is the entire practice. Use a lab VM running the monitoring software with one site's devices and a test inbox. Use a lint script with five rules. Choose one canary device per role. Write it on a sticky note that later becomes a file. And use the checklist. Apply the change to the lab VM and look at it. Apply it to the canaries and sleep on it. Look at the alert list in the morning. Apply it everywhere. If the alert list looks wrong at any point, revert and re-apply.


That is perhaps an extra hour per risky change and no extra hours for trivial ones. Against it, weigh one three-in-the-morning page for the whole team, or one site unmonitored for a month. A lab VM and a checklist are not sophisticated. They are enormously better than nothing, and nothing is what most small teams are currently using.


Where to go next


Testing and staged rollout is the top rung of the monitoring-as-code ladder, and it rests on everything below it. That includes the repository that makes rollback a command and the templates that make one change apply consistently. It also includes the reproducible build that gives you a staging instance for free. If any of those is missing, the guides in Monitoring as Code cover them in order, starting with what monitoring as code means. With all of them in place, a monitoring change becomes what it should have been all along. It becomes a small, reviewed, reversible step that nobody has to lose sleep over.


Put this into practice with SysPeep Monitor

Staged rollout is a habit you can practise with any monitoring tool. How a second instance is set up and how configuration is imported are product-specific. So check the product documentation for its import and backup options. The things you will be testing in SysPeep Monitor are its thresholds, dependencies, and alert rules. Those are the settings designed to keep alert noise down. They are the ones a canary and a day of watching will tell you the truth about. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep