The Alert Parity Checklist
There is a specific kind of quiet that follows a monitoring cutover, the day the new system becomes the one people act on. The old system has been silenced. The new one is live. Nothing is alerting. Is that because nothing is wrong, or because the new system would not tell you if something were? Until you have deliberately broken something and watched the new system notice, you do not know. "We haven't had any alerts" is not reassurance. It is the sound of a smoke detector with no battery in it.
Alert parity is the answer. Every alert that mattered in the old system must exist in the new one and must be proven to fire correctly. It must be detected, at the right severity, and delivered to the right person by the right channel, within the expected time. All of that must be proven before the old system stops being the truth. Not configured. Proven. The distance between an alert rule that exists and an alert that arrives is a chain of things that can each be wrong. And the only way to test a chain is to pull on it. I have pulled on chains that came apart at the third link. It is much better to find that out in daylight, with the old system still watching.
This guide walks through building the list of alerts that need proving, and safe ways to trigger each kind on purpose. It explains what "correctly" means beyond simply firing. It provides a sign-off template that turns testing into a record, and covers what to do about the alerts you cannot safely test. It ends with a filled-in checklist you can copy.
In this guide
- The principle: exists and fires
- Building the parity list
- Testing each alert deliberately
- Parity includes routing and timing
- The sign-off template
- Alerts that cannot be tested safely
- The checklist, filled in
- The gaps parity testing usually finds
- Where to go next
The principle: exists and fires
Break the claim "this alert works" into the pieces that can independently fail:
- The check exists and is polling the right thing on the right device.
- The threshold is set to the intended value, including any hand-tuned adjustment carried over from the old system.
- The condition, when it occurs, is detected within the polling interval plus any confirmation delay.
- An alert rule matches that check and severity, and is enabled.
- The rule's action points at the right contact or channel, and the contact details are current.
- The delivery path works: the mail relay accepts the message, the webhook endpoint answers, the paging service allows the new system's sender.
- Dependencies and maintenance windows do not wrongly suppress it.
- The recipient recognizes the alert for what it is. The subject line, device name, and severity make sense to a person at three in the morning.
Eight links. Parallel running means both systems watch the same devices while only the old one is trusted. It is described in Parallel Running: Two Monitors, One Truth. Parallel running tests the first few for whatever conditions happen to occur naturally. Parity testing tests all eight, for the alerts you choose, on a schedule you control. Together they are the evidence that cutover is safe.
None of this is a comment on either tool. The same eight links exist in every monitoring system ever built. A migration in any direction — to or from a commercial product, to or from open source — resets all of them at once. It is the closest a monitoring system comes to a clean slate, and a clean slate has never paged anyone.
Building the parity list
Start with the inventory from Inventorying What You Monitor Before You Move It: every row whose alert-rule column is not empty is a candidate. That list is usually far too long to test one by one. So it gets prioritized using a second source, the old system's alert history for the last year. Export it, sort by check, and classify:
- Alerts that fired and led to action. Someone responded, fixed something, closed a ticket. These are the ones that mattered. Every one goes on the parity list.
- Alerts that fired and were ignored or acknowledged without action. Candidates for tuning or pruning, not for parity testing, unless someone argues for them.
- Alerts that never fired in a year. Ask why. Some are the most important alerts in the building — core switch down, data-center temperature, UPS on battery — and they never fired because the disaster never happened. Those go on the list marked "must test deliberately." Others never fired because they cannot: a wrong threshold, a disabled rule. That is worth knowing too.
Then add something the history will not show you: the alerts that represent a whole class. If there are two hundred interface-down alerts, one per uplink, you do not test two hundred. You test one per device type, treat the rest as covered by the class, and spot-check a few. The parity list ends up as a few dozen alerts, each representing itself or a class, each with a named stakeholder who cares about it.
Ashgrove Housing built its parity list carefully, and one row on the never-fired list read "boiler house pump: comms lost". Nobody could say what it was for, so it was marked "probably legacy" and left off the list. It was not legacy. The first cold week after cutover, the pump controller in the largest estate's boiler house dropped off the network. The new system, which had no rule for it, said nothing. The old system, kept read-only in its sunset period, still had the rule, pointing at a heating contractor whose contract had been renewed every year for exactly this. The reason nobody could remember turned out to be on page two of the contractor's invoice.
Testing each alert deliberately
The rule is: make the condition real, safely, on purpose, and watch what happens. Not "set the threshold to zero so it fires" — that tests the notification path but not the detection. And it is easy to forget to set the threshold back. Wherever possible, cause the thing the alert is for. Tell the on-call person before each test, and record the time, because you are about to look for it in two systems' logs.
Network alerts
Interface down: pick a test port — a spare access port with a laptop on it, or a lab switch — and unplug it. Device unreachable: power off a lab device, or block the poller's address on a test device's management access list. Do not pull the uplink of a production switch to see what happens. The dependency behavior you want to see is one alert for the parent, not forty for the children. You can test that with a lab switch that has a child device behind it.
Server and storage alerts
Disk full: on a test volume, create a large file until the threshold is crossed. Most operating systems have a command that allocates a file of a given size in seconds. Watch the alert, delete the file, and watch the recovery. Service stopped: stop a non-critical service on a test server, or a service on a production server during a maintenance window with the owner's agreement. High CPU: a stress tool or a deliberately silly loop on a test machine. Memory and swap follow the same pattern: cause it on something that does not matter.
Application and availability alerts
Web check failing: point a test check at a page you control and take it down, or make it return an error status. Certificate expiry: most tools let you set the warning window. Temporarily widen it until it includes the real certificate that expires soonest. Confirm the alert and set it back. Application-specific checks: stage the failure in a test environment that the new system also monitors.
Environmental and power alerts
UPS on battery: many UPS units have a self-test that briefly switches to battery and reports it over SNMP; use that rather than pulling the plug. Temperature: warming a sensor with your hand is a legitimate test. Door contact, water sensor: trigger it physically. These are the alerts most likely to be forgotten and least likely to have fired naturally, so they deserve the most attention here.
The notification path on its own
Every tool has some way to send a test notification. It could be a button on the contact, a test action on the rule, or simply a temporary check that fires immediately. Use it for every distinct contact and channel, not just one. The contact that is wrong is never the one you tested.
Keep a plain log as you go. One line per event, with the time, is enough to reconstruct what happened when the sign-off sheet is filled in later:
# Parity test log 09:14 TEST-07 Interface down, sw-lab-01 Gi1/0/8 unplugged cable 09:15 new: CRIT "sw-lab-01 Gi1/0/8 down" -> #netops-quiet delay 0:52 09:15 old: CRIT "sw-lab-01 port 8 down" -> #netops delay 0:40 09:22 plugged in; both cleared within one minute 09:30 TEST-08 Disk full, fs-test-01 D: created 40 GB file 09:37 new: WARN "fs-test-01 D: 91% used" -> srv-team mail delay 5:10 (interval 300 s) 09:37 old: WARN "fs-test-01 D: free 9%" -> srv-team mail delay 4:55 09:40 deleted file; both cleared 10:05 TEST-09 UPS on battery, ups-closet-b self-test from web UI 10:07 new: CRIT "ups-closet-b on battery" -> facilities-OLD delay 2:10 ** wrong alias ** 10:07 old: CRIT "ups-closet-b on battery" -> facilities delay 1:50
Parity includes routing and timing
An alert that fires is not the same as an alert that reaches someone. For each test, record three more things.
- Right person. Did it reach the contact the old system would have used? Migrations often import contacts from a stale list. The test that lands in the migration team's inbox instead of the on-call engineer's phone has found a real gap. So has the one in the log above that went to a facilities alias nobody reads any more.
- Right channel and severity. A critical that arrives as an email when it used to page is a downgrade someone will notice at the worst possible time. A warning that pages is the opposite problem, and it will train people to ignore pages.
- Right time. How long from the condition to the notification? Compare against the old system's delay for the same condition and against what the stakeholder expects. A longer polling interval, a confirmation count ("must fail three times"), or an escalation delay can each add minutes. Sometimes the new delay is fine; sometimes it is not. Either way it is a decision to record, not a surprise to discover later.
Also check the content. Open the notification the way the recipient would, which is on a phone, in the dark, with one eye. Does the subject line say which device and what is wrong? Does the link go to the right place in the new console? Is the device name the one people use, or an internal identifier? Recipients who cannot understand an alert at a glance will treat it as noise, and noise gets muted.
Tip: Test the escalation, not just the first notification. Let one test alert go unacknowledged and confirm that the second-level contact is told after the configured delay. Escalation chains are configured once and tested never, and a migration is the one time you have an excuse.
The sign-off template
Each test produces a row in the sign-off sheet. The columns are deliberately few, so that people will actually fill them in.
| Column | What it records |
|---|---|
| Alert | Its name in the new system, and the old system's name if different |
| Condition | What has to be true for it to fire, in plain words, with the threshold |
| Tested how | The exact method: unplugged port X, filled test volume, ran UPS self-test, sent test notification |
| Fired in new | Yes, no, or partial, with the delay from condition to notification |
| Routed correctly | Yes or no: right contact, right channel, right severity, readable content |
| Signed by | The stakeholder who cares about this alert, not the person who ran the test |
| Date | When it was tested; retest if the rule changes afterwards |
| Notes | Anything odd, and any accepted difference from the old system's behavior |
The "signed by" column is what makes this a parity proof rather than a testing log. The person who signs is the person who would be woken by the alert, or who owns the service it protects. Their signature means "I have seen this work and I accept it." When cutover happens and something is missed, the sheet shows whether the alert was tested, how, and who agreed it worked. That turns a blame conversation into an engineering one.
Alerts that cannot be tested safely
Some alerts cannot be triggered without doing real damage: the core switch down, the storage array degraded, the internet link at head office down, the primary domain controller unreachable. Do not test these by causing them, not even briefly, not even to settle an argument. Do this instead.
- Test the class on a safe instance. If "switch down" works on a lab switch with the same check type and alert rule, you have evidence the mechanism works. Record that the production instance is covered by class.
- Test the path separately. Send a test notification through the exact rule and contact the untestable alert uses, so at least the routing is proven.
- Test during planned maintenance. If the storage array has a scheduled firmware update, the migration team is invited, and the alert that fires during the reboot is the test. Record it like any other.
- Document the reasoning. A row that says "not safely testable; class tested via TEST-03; path tested via TEST-22; verify on first real occurrence" is an honest, defensible record.
- Plan first-real-occurrence verification. Somebody owns the job of confirming the alert behaved correctly the first time the condition really happens after cutover. The row stays open until they do. Keep that list short and visible.
What is not acceptable is the untested row that is simply assumed. The whole point of parity is replacing assumption with observation; where observation is impossible, the substitute is a written argument, not silence.
The checklist, filled in
Here is what the sheet looks like partway through a migration, with the date column dropped to fit. The names are invented; the shape, and the fact that one row found a real problem, are the point.
| Alert | Condition | Tested how | Fired in new | Routed correctly | Signed by | Notes |
|---|---|---|---|---|---|---|
| Core switch unreachable | No ping or SNMP reply for three polls | Class: lab switch powered off (TEST-03); path: test notification via NET-CRIT (TEST-22) | Class: yes, 1:40 | Yes, on-call page | Network lead | Production instance: verify on first real occurrence |
| Uplink interface down | Interface operational status down | Unplugged sw-lab-01 Gi1/0/8 (TEST-07) | Yes, 0:52 | Yes, chat and email | Network lead | Class covers all uplinks; two spot-checked |
| File server D: nearly full | Free space below ten percent warn, five percent critical | Filled test volume on fs-test-01 (TEST-08) | Yes, 5:10 | Yes, server team mail | Server lead | Delay accepted at a five-minute interval |
| Mail relay service stopped | Service not running | Stopped service on test relay | Yes, 1:05 | Yes, page | Server lead | — |
| UPS on battery, closet B | Battery status reports on battery | Self-test from UPS web interface (TEST-09) | Yes, 2:10 | No: went to a stale facilities alias | Facilities | Contact fixed, retested, now yes |
| Public website down | Two consecutive non-success responses | Returned an error status from a test page | Yes, 2:30 | Yes | Web owner | — |
| Certificate expiring | Fewer than thirty days remaining | Widened window to include the soonest real certificate | Yes | Yes, email | Web owner | Window restored afterwards |
| Server room temperature high | Above the agreed limit | Warmed the sensor by hand | Yes, 2:00 | Yes, page | Facilities | — |
| Storage array degraded | Array health not OK | Not safely testable; path tested via TEST-22 | — | Yes (path only) | Storage lead | Verify at next planned firmware update |
| Escalation chain | Critical unacknowledged for twenty minutes | Left TEST-03 unacknowledged | Yes, 20:15 | Yes, second-level contact | Network lead | — |
Read the UPS row again. The alert existed. It fired. It fired quickly. And it would have gone to a mailbox nobody reads, on the night the closet lost power. That is what parity testing is for.
The gaps parity testing usually finds
The same handful of gaps turn up in almost every migration, in every direction, so look for them on purpose.
- Rules built but never enabled.
- Contacts imported with old details, or mapped to a group with no members.
- The new system's sender not allowed through the mail relay or the paging service.
- Severity mapping mismatch: the old "critical" became the new "high," which routes as a warning.
- Confirmation counts left at default: the new system waits three failures where the old waited one, tripling the delay.
- Dependencies the wrong way round, suppressing a child when the parent is fine, or failing to suppress when the parent is down.
- Maintenance windows in the wrong time zone, muting the daytime instead of the night.
- Recovery notifications missing, so nobody learns that the problem cleared.
Each of these, found during parity testing, costs a few minutes. Each one, found after cutover, costs whatever the alert was there to prevent.
Where to go next
Alert parity is the gate that makes cutover safe. Build the list from the inventory and the last year's real alerts, plus the alerts that never fired because the disaster never came. Trigger each one deliberately and safely. Unplug a test port, fill a test volume, stop a test service, run the UPS self-test, and send a test notification to every contact. Record not just whether it fired but whether it reached the right person, by the right channel, at the right severity, within the expected time. Get the person who would be woken by it to sign. For the alerts you cannot cause, test the class and the path, write down the reasoning, and verify on first real occurrence. Then the quiet after cutover means what you hope it means. You have pressed the button on the smoke detector, and it beeped.
This checklist sits inside the larger plan described in Why Monitoring Migrations Fail, between the parallel run and cutover. Once cutover is done, the old system's history still needs looking after; that is Data Export Realities. Everything in this topic is collected at Migrating Between Monitoring Platforms Without Losing Your History.
Put this into practice with SysPeep Monitor
Parity testing goes faster when you can see, in one place, which threshold a check has, which rule it triggers, and where that rule sends. SysPeep Monitor keeps thresholds, dependencies, and alert rules together in a single web console with email and other notifications. So a test on a trial installation can be watched from condition to inbox. If it is the candidate in your parallel run, the checklist above is the one to hold it to.
Download Free TrialLearn About SysPeep