The Anatomy of a Good Alert
Every alert you configure is an interruption on a timer. You set it up today, calmly, with coffee in hand. It goes off months from now, for a person you cannot see, having a day you cannot predict. Maybe that person is you at 3 a.m. Maybe it is the new hire in their second week, covering alone because you are finally on holiday. Maybe it is future-you in the middle of an outage, with four other alerts already open. A good alert is one that was designed for that moment — not for the quiet afternoon when it was created.
Here is the uncomfortable truth: most alerts are never designed at all. They are merely enabled — a checkbox in a monitoring tool, a vendor default left in place, a copy of whatever the last device got. That is how inboxes fill with messages that technically say something and practically say nothing. This guide is about the alternative: treating each alert as a small product with a user, and building it deliberately. By the end you will have a test that filters out alerts that should not exist. You will have a four-question contract every alert must honor and a worksheet that captures a complete alert design in one row. You will also have a worked example that goes from blank page to finished alert.
In this guide
- The actionability test
- The alert contract: four questions
- The six design inputs and the worksheet
- Choosing your failure mode
- A worked example: the WAN-degraded alert
- The ten-point design checklist
The actionability test
First, a definition, because the word gets stretched. In this guide an alert is an interruption: a message pushed at a human. It carries the expectation that they stop what they are doing and pay attention. A number on a dashboard is not an alert. A line in a weekly report is not an alert. An alert grabs someone by the sleeve. That is a real cost, paid in attention. And it is the reason every alert must pass one test before it earns the right to exist:
When this fires, a specific person can take a specific action.
Both halves matter. A specific person: someone receives it, owns it, and knows it is theirs. An alert broadcast to a mailbox nobody owns fails the test before it even fires. A specific action: the receiver can actually do something about the condition, now or on a known schedule. Run three candidates through the test and you will feel how it works.
Candidate 1: disk nearly full — passes
"Drive D: on the file server is 92% full." A specific person (whoever runs that server) can take a specific action (delete or archive data, extend the volume, chase the log file that grew). The condition will not fix itself, it gets worse on a predictable slope, and acting early prevents an outage. This is the model citizen of alerting: clear owner, clear move, real stakes.
Candidate 2: CPU briefly high — fails
"CPU on the application server touched 97% for one minute." What would you even do? By the time you log in, it is over. Short CPU spikes are what CPUs are for — a busy minute is the machine doing its job. There is no action, so there should be no interruption. That does not mean you ignore CPU: record it and graph it. Let a different, better-designed alert catch the condition that is actionable — CPU pinned high for fifteen minutes while response times degrade. Same metric, different condition, opposite verdict.
Candidate 3: the informational alert — the trap
"Backup completed successfully." "Seventeen updates are available." "A device rejoined the network." Each is genuinely useful information, and none of them is an action. These are FYIs, and the trap is that they feel harmless. But every FYI delivered as an interruption teaches its recipients, one message at a time, that interruptions can be ignored. That training is exactly how the alert that mattered gets deleted unread. The whole sad mechanism has its own set of guides under alert fatigue. FYIs belong in a daily digest, a dashboard, or a report — places you visit on your schedule, not places that visit you.
Apply the test ruthlessly and something pleasant happens: the alerts that remain are all, by construction, worth reading. That is the foundation. Everything else in this guide is about making those survivors excellent.
The alert contract: four questions
When an alert fires, its reader should be oriented before they log in to anything. That is the alert contract: on its face — in the notification itself, not three clicks away — every alert answers four questions. What happened? Where? So what? Now what? Miss one and you have transferred work from the calm person who designed the alert to the stressed person who received it. That is precisely the wrong direction.
1. WHAT happened
The check and its finding, in plain words. Not the internal name of the check, not a status code — the thing a colleague would say out loud.
Bad: Status change on chk_0447: state 2 -> state 0 Good: Mail server stopped answering on port 25 — inbound mail is queuing
The bad version is true and useless; it describes the monitoring system's bookkeeping instead of the world. The good version tells you what broke without requiring you to remember what chk_0447 was configured to do two years ago.
2. WHERE it happened
Device, service, and site — enough to orient instantly, even for a reader who does not hold the whole network in their head.
Bad: High memory on host Good: High memory on DB01 (finance database server, rack 2, HQ server room)
"Which host?" should never be a question the reader has to answer by logging in. Include the friendly role, not just the hostname: DB01 means nothing to the new hire, but "finance database server" tells anyone how nervous to be.
3. SO WHAT — the impact statement
Who or what is affected, in user-visible terms. This is the field that separates signal from trivia. And it is the one most alerts omit — because it is the one the monitoring tool cannot fill in for you. It requires a human to have decided, at design time, why this check exists.
Bad: Interface Gi0/1 utilization 96%
Good: Office-to-warehouse link at 96% — warehouse scanners and
terminals will be slow until this clears
The impact line also works as a design filter. Suppose you sit down to write it and honestly cannot — the truthful version is "so nothing much, probably." Then the alert just failed the actionability test in disguise. In that case, you should demote it to a dashboard before it ever fires.
4. NOW WHAT — the first move
The reader's next sixty seconds: a link to the runbook (the written procedure for handling exactly this situation — a topic of its own), or at minimum a suggested first check. You knew the likely first move when you designed the alert; hand it over.
Bad: (nothing — the message just ends)
Good: First check: is the nightly archive job still running?
Runbook: internal wiki > Runbooks > mail-queue-growth
Even one sentence of "start here" collapses the worst part of receiving an alert into a single click. That worst part is the blank few minutes of deciding where to begin. For the 3 a.m. reader, that sentence is the difference between acting and flailing.
Tip: write the SO WHAT and NOW WHAT lines at design time, while the reasons are fresh. It takes two minutes, and it is the cheapest documentation you will ever produce. That documentation is delivered automatically, to exactly the right person, at exactly the moment they need it.
The six design inputs
So where does an alert come from? Not from a checkbox. A complete alert is the product of six explicit decisions, and naming them is what turns "enabling notifications" into design:
- A monitored condition. The precise circumstances that make the alert true — usually a metric crossing a threshold for some duration. Picking the value deserves real thought; static thresholds done right covers how to choose numbers you can defend out loud.
- Noise guards. The mechanisms that stop one wobbly reading from becoming a page: confirmation counts, sustained-for timers, and separate trigger and clear levels. The anti-noise toolkit walks through each with worked numbers.
- A severity. The promise about how fast a human must act — which is a policy question, not a feeling. Designing severity levels that mean something is its own guide.
- An audience and a channel. Who receives it, and by what medium — a page, an email, a chat message, a digest. Matching urgency to medium is the subject of the notification channels guides.
- A response expectation. What the receiver is supposed to do about it, and by when: "act within minutes," "handle within the business day," "no response owed."
- The message. The four-question contract text from the previous section, written for a tired stranger — there is a whole guide on writing alert messages humans can act on.
Six decisions per alert sounds heavy until you see it as a worksheet — one row of a table you fill in per alert. Here is the worksheet with an everyday example filled in:
| Design input | The question you are answering | Example: file-server disk alert |
|---|---|---|
| Condition | What exactly is true when this fires? | Drive D: on FILE01 above 90% used |
| Guards | What stops noise from firing it? | Sustained across 3 polls; clears below 85% |
| Severity | How fast must a human act? | Warning — business-day response |
| Audience & channel | Who hears it, and where? | Infrastructure inbox, email |
| Response expectation | What is the receiver meant to do? | Free or extend space within one business day |
| Message | What does it say on its face? | What / where / so what / now what, per the contract |
Keep the filled-in rows — a spreadsheet is fine — and you have an alert catalog: a living record of what you decided and why. It pays rent twice. When an alert misbehaves, the row tells you what its designer intended. And when you run a weekly alert review, the catalog is the thing you review against.
Choosing your failure mode
Every alert can fail in exactly two directions. A false positive fires when nothing is actually wrong — noise. A false negative stays silent while something is genuinely wrong — a miss. Here is the part nobody tells beginners: you cannot minimize both at once. Every guard you add — another confirmation poll, a longer sustained-for timer — buys quiet by adding delay. At the margin, it also adds the chance of missing a short-lived real problem. Every guard you remove buys speed by admitting noise. Tuning is not finding a magic setting with zero of each; it is deciding, per alert, which failure this alert can afford.
Two devices from ordinary life show the two ends of the dial. A smoke detector is tuned to never, ever miss a real fire. In exchange, everyone accepts that it will occasionally scream at burnt toast. A motion-sensor porch light flicks on for cats and falling leaves all night. Nobody minds, because both its false alarms and its misses cost almost nothing. Neither device is badly tuned; each is tuned for its stakes.
Give each alert the same treatment, in one written sentence. "Core switch down" is a smoke detector: a missed real outage is unacceptable. So it gets light guards and you accept the rare false page when the switch is merely slow to answer. "Disk trending toward full in three weeks" can afford to be slow and skeptical: heavy confirmation, generous timers. A few hours of detection delay on a three-week problem costs nothing, while noise would erode trust. And notice the special case where a false negative hides by default: "backup report missing." Silence is the failure, so the alert must fire on absence — no success report by 7:00 AM is itself the alarm. Writing the budget down ("this alert may cost us about one false alarm a month; zero tolerated misses") sounds bureaucratic and takes fifteen seconds. It is the sentence that settles every future tuning argument about that alert.
A worked example: the WAN-degraded alert
Let us design one alert end to end, through the worksheet, so you can see the decisions connect. The subject is the office's WAN link — the connection from your site to the internet or to other sites. It is degraded: not down, which is a different and easier alert, but limping. Degraded links are worth their own alert because they produce the vaguest tickets in IT ("everything is kind of slow?"). Meanwhile, a simple up/down check sees nothing wrong.
Step 1 — condition. The clearest signal of a limping link is packet loss: the percentage of test messages that never come back. We choose: packet loss to the ISP gateway above 5% sustained for 10 minutes. Why 5? Below a couple of percent is background weather on many links; above 5, interactive traffic visibly suffers. That is a defensible starting value of the kind the threshold guide teaches. And it is a starting value, to be tuned against your link's reality.
Step 2 — guards. The 10-minute window is already a guard: one lost ping is nothing. We add a clear level of 2%, so the alert does not resolve-and-refire while loss dances around 5%. That is the trigger/clear gap from the anti-noise toolkit. Failure-mode budget: this alert leans smoke-detector. A degraded link affects everyone, so we accept the occasional false alarm from a brief internet storm rather than risk missing a real degradation.
Step 3 — severity. Impact is real but the building is not on fire, so we choose Warning, with an escalation rung. If loss passes 20%, the same condition is promoted to Critical, because at that point the link is effectively unusable. Two severities, one ladder; the severity guide covers why that pattern beats a single line.
Step 4 — audience and channel. The network admin (you, and the shared infrastructure inbox as backstop), by an interrupting channel during business hours. The Critical rung is allowed to page after hours. The Warning is not — a mildly lossy link at 2 a.m., with nobody using it, can wait for morning. Channel-to-urgency matching is its own topic.
Step 5 — response expectation. Warning: investigate within the hour during business time. Critical: act within 15 minutes, any time.
Step 6 — the message. Now the contract, written once at design time and filled in by the monitoring platform at fire time:
WARNING | HQ WAN link degraded — 8% packet loss to ISP gateway Impact: internet, cloud apps and site-to-site VPN slow for all of HQ Evidence: loss 8% (threshold 5%), sustained 12 min; latency 38 -> 74 ms First move: check firewall WAN interface errors, then ISP status page Runbook: wiki > Runbooks > wan-degraded Site: HQ Alert: WAN-03
What happened, where, so what, now what — all four answered in five lines, readable in a preview pane, actionable by whoever is on duty. Total design time for all six steps: perhaps twenty minutes. The alert will run for years. Few twenty-minute investments in IT pay better.
The ten-point alert design checklist
Everything above, compressed into the list to run before any new alert goes live — and, just as usefully, over any old alert that has been annoying you:
- 1. It passes the actionability test: a specific person can take a specific action.
- 2. It is an interruption on purpose — it is not an FYI that belongs in a digest or dashboard.
- 3. The condition is precise, and you can explain the threshold's value out loud without blushing.
- 4. Noise guards (confirmation, sustained-for, trigger/clear gap) were chosen, not defaulted.
- 5. The failure-mode budget is written down: which is worse here, a false alarm or a miss?
- 6. The severity matches your written severity policy, not the designer's mood.
- 7. The channel matches the severity — interrupting media for interrupt-worthy news only.
- 8. The message answers all four contract questions on its face.
- 9. A runbook link or first-move hint is included.
- 10. The alert has a row in the catalog, so future reviews can judge it against its intent.
Ten yeses and the alert has earned its interruption. Anything less, and you now know exactly which part to fix.
Where to go next
This guide gave you the skeleton; the rest of the pillar adds the flesh. Severity design turns the "how urgent?" decision into policy. Alert message writing goes deep on the contract text, with templates and rewrites. The alert lifecycle follows the alert after it fires — acknowledgement, resolution, and learning. And the weekly alert review is the ritual that keeps every design honest after contact with reality. They all live in Alert Design.
Put this into practice with SysPeep Monitor
Designing the alert is your job; running it faithfully is the software's. SysPeep Monitor gives every check the working parts this guide covers. Those include thresholds, confirmation and dependency rules built to keep alert noise down, severities, and email and other notifications. They all come from one web console. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep