Alerting on Everything: When Every Alert Is Critical, None Is
Somewhere in your monitoring system there is one alert that has saved you. It fired at the right moment, said the right thing, and somebody fixed a real problem before a user noticed. That alert is why the system exists. It is also, in a way that nobody intended, the reason there are now four hundred others that have never saved anyone. It is also the reason the person carrying the on-call phone has stopped reading them.
This guide is the origin story of that anti-pattern and the cure for it. It is not a guide to alert fatigue as a condition; diagnosing alert fatigue covers the symptoms in people. And the rest of that pillar covers the treatment in depth. This is the guide for the moment before you open those. It covers how you got here, and how to prove it with numbers from your own alert history. It also covers the order in which to do the work so that it sticks.
By the end you will be able to tell the story of how one alert became four hundred. You will be able to pull four numbers from your alert history that show how bad it is. You will be able to run a five-step cure in the only order that works. You will be able to fill in a triage table for every alert you have. And you will be able to handle the awkward conversation that follows when you delete an alert someone asked for.
In this guide
- The road from one alert to four hundred
- Why every alert is critical
- The diagnosis, in numbers
- The cure sequence
- The alert triage table
- The political cost of deleting
- A quiet Sunday at Bellweather Logistics
- The alert triage checklist
- Where to go next
The road from one alert to four hundred
Nobody builds four hundred alerts on purpose. They are built one at a time, in the same sequence, by a process that looks like learning from experience. Here is the sequence.
- An incident happens. A disk fills, a link drops, a service stops.
- The review asks "could we have seen this coming?" and the answer is usually yes, because in hindsight everything has a warning sign.
- Someone says "we should have alerted on that", and everyone agrees, because disagreeing would sound like arguing for the next outage.
- A new alert is created. It is marked critical, because the incident was.
- The alert fires, and mostly it fires when nothing is wrong. That is because the warning sign that was obvious in hindsight is a thing that also happens on ordinary days.
- Nobody deletes it, because it was created after an incident, and deleting it would feel like inviting the incident back.
- Repeat for every incident, for two years.
Each step is reasonable. The review is right to ask the question. The person suggesting the alert is trying to help. Marking it critical is a natural response to having just been through something painful. The only broken step is the one that never happens. Nobody ever asks whether the alert did what it was created to do, and so nothing ever leaves. The alert count is a ratchet. It turns one way.
The road has a second lane, which is templates. A device template that applies twenty alert rules to every device of a type adds twenty alerts per device without anyone looking at one of them. A template is efficient. It is also how a hundred switches acquire two thousand alert rules in an afternoon, none of which was chosen for any specific switch.
Why every alert is critical
A severity is the label on an alert that tells the reader how urgently to act. Typically the labels are something like critical, warning, and informational. The top level means "stop what you are doing". In an alerting-on-everything system, most rules carry the top label, and the reason is not carelessness. It is the absence of an argument.
When an alert is created, the person creating it wants it noticed, and the way to be noticed is to mark it critical. The person who might object, the one who will be woken by it, is usually not in the room. If they are, arguing that a colleague's alert is "only a warning" sounds like arguing that their problem does not matter. So nobody argues. Critical wins by default, every time, and after two years the top severity is where most of the alerts live.
The cost is that the label stops carrying information. If four hundred alerts are critical, then "critical" means "an alert". In that case, the reader has to open each one to find out whether it is the disk or the toner. Severity is a signal. When the signal is always on, it is not a signal.
The same thing happens to notification lists. Each new alert is sent to everyone who might care, because nobody wants to be the person who left someone off. So every alert goes to every admin. That is how six people receive the same warning about a printer and all six assume one of the other five has it. (Six people is the ideal number for that. Fewer and someone acts; more and someone complains.)
The diagnosis, in numbers
You do not have to argue about whether you alert on everything. Your alert history already knows, and four numbers will settle it. For each week over the last quarter:
- Alerts per week. The total count of alert events. A number nobody could read is a number nobody reads.
- Share acknowledged within an hour. An acknowledgement is a person marking an alert as seen. If most alerts are never acknowledged, or acknowledged hours later, they are not being treated as interruptions, whatever their severity says.
- Share that led to an action. An action is anything your team records against an alert: a note, a change, a ticket number in whatever system you use. This is the only number that measures whether an alert was useful.
- Share that cleared themselves. An alert that fired and then returned to normal with no acknowledgement and no action was noise, by definition. It interrupted nobody usefully.
If your monitoring database is one you can query, the four numbers come from one query. Table and column names differ between products, and some products expose alert history only through their own reports. Treat this as the shape of the thing and check the documentation for yours:
-- Illustrative: alert volume and outcome per week for the last quarter.
-- Assumes one row per alert event with fired, acknowledged, and cleared
-- timestamps and an optional action reference. Names differ by product.
SELECT strftime('%W', fired_at) AS week,
COUNT(*) AS alerts,
ROUND(100.0 * SUM(CASE WHEN acknowledged_at IS NOT NULL
AND (julianday(acknowledged_at) - julianday(fired_at)) * 24 <= 1
THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_acked_1h,
ROUND(100.0 * SUM(CASE WHEN action_ref IS NOT NULL
THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_actioned,
ROUND(100.0 * SUM(CASE WHEN cleared_at IS NOT NULL
AND acknowledged_at IS NULL AND action_ref IS NULL
THEN 1 ELSE 0 END) / COUNT(*), 1) AS pct_self_cleared
FROM alert_events
WHERE fired_at >= DATE('now', '-91 days')
GROUP BY week
ORDER BY week;
Read the result honestly. If alerts per week is in the hundreds, the acknowledged share is low, the actioned share is in single figures, and the self-cleared share is most of the total, you alert on everything. You do not need a second opinion. You need the next section.
One more number is worth pulling: alerts per rule. Sort every rule by how many times it fired last quarter, and look at the top ten. In every system I have done this on, the top ten rules produced most of the volume, and most of the top ten had never led to an action. That list is where the cure starts.
The cure sequence
The cure has five steps, and the order is not optional. Teams that start at step five, adding the alerts they are missing, make the problem worse, because they add signal to a channel nobody is listening to. Teams that start at step four, tuning, spend a quarter polishing alerts that should have been deleted. Do them in this order.
1. Stop adding
Declare a freeze: no new alert rules for one quarter, except by written exception from one named person. The freeze is not permanent and it is not a policy. It is a tourniquet. Its job is to stop the ratchet turning while you work on the rest. It is also to make the team notice how often "we should alert on that" gets said. (It gets said a lot. Most of the time, a week later, nobody remembers what "that" was.)
2. Delete what nobody acted on
Take every rule that fired in the last quarter and led to no action, and delete it. Not disable, not mute, not "review later": delete, with a one-line record of what it was and why. This is the step that feels like negligence and is the opposite. An alert that fired thirty times and was acted on zero times has proved, thirty times, that it does not need a human. Deleting alerts covers the mechanics and the safety net. The one rule I will repeat here is that anything deleted can be recreated in five minutes. So the risk of deleting is small and the risk of keeping is the on-call phone.
3. Demote the rest
Every surviving rule gets the severity it deserves, not the one it was given. The test is the 3 a.m. question: if this fired at three in the morning, should a person get out of bed? Yes means the top severity. "It can wait until morning" means the middle one. "Somebody should know eventually" means the bottom one. The bottom one does not notify anybody. It sits in the console until someone looks. Severity design is the guide for building the scale; this step is applying it to what you have.
4. Add hysteresis and confirmation
The survivors that still fire too often are usually firing on the edge of a threshold. Hysteresis means the alert clears at a different level from the one it fired at. So a value hovering at the line does not fire and clear every minute. Confirmation means the condition must hold for several polls before the alert fires, so a single bad sample does not become a page. Hysteresis and confirmation covers both properly. Related alerts that always arrive together should also be grouped, which deduplication and grouping covers.
5. Then, and only then, add the missing ones
Now that the channel is quiet, an alert that arrives will be read. This is the moment to add the alerts you genuinely lacked. Each one has to pass the bar that anatomy of a good alert sets. The bar requires a named condition, a named owner, a severity chosen by the 3 a.m. test, and a line saying what the reader should do. If you cannot write that line, you do not have an alert. You have a graph, and graphs are fine. Keep it as a graph.
Expect the five steps to take a quarter. Expect the alert count to fall by more than half and the actioned share to rise sharply. That is not because the network got better but because the alerts that remain are the ones that were ever going to be acted on. That is not a trick. It is the definition of a useful alert, applied.
The alert triage table
The triage table is one row per alert rule, with its firing count for the quarter, how many of those were acted on, and the decision. It is the working document for steps two and three, and it is also the record that makes the deletions defensible later. Fill it in from the alerts-per-rule query, not from memory. Memory remembers the alert that saved you and forgets the forty that did not.
| Alert rule | Severity now | Fired last quarter | Acted on | Decision | Reason |
|---|---|---|---|---|---|
| Core switch uplink down | Critical | 2 | 2 | Keep | Both were real; both actioned within minutes |
| Any interface utilisation over 80% | Critical | 412 | 0 | Delete | Never actioned; keep the graph for capacity review |
| File server disk over 85% | Critical | 31 | 1 | Demote to warning | Growth is slow; morning is soon enough |
| Backup job still running at 06:00 | Critical | 58 | 0 | Merge | Fold into "backup not complete by 08:00", warning |
| Printer offline (any) | Critical | 140 | 0 | Delete | Facilities has its own list; nobody in IT acts |
| Stock system HTTP check failing | Warning | 3 | 3 | Promote to critical | Users noticed before we did each time |
| Ping loss to branch router, single poll | Critical | 96 | 4 | Keep, add confirmation | Real four times; needs three polls before firing |
| Interface flapping, retired switch | Critical | 209 | 0 | Delete | Device retired; remove from monitoring entirely |
The decision column has five values, and each has a rule. The first three are keep (fired rarely, acted on when it did), demote (real but not urgent), and delete (never acted on). The others are merge (several rules that always fire together become one), and promote (acted on every time, but at the wrong severity). Promote is rare and important. The alert that was demoted to warning years ago because it was "noisy" is sometimes the one the business most needs to hear.
Keep the reason column short and factual. It is the sentence you will read back to the person who asks, six months from now, why their alert is gone.
Tip: do the triage in two passes. The first pass, alone, takes an hour and produces a provisional decision for every row from the numbers. The second pass, with one colleague and the on-call rota, takes another hour and changes about a fifth of them. The second pass is where you find the alert that fired once, was acted on once, and prevented something nobody else remembers.
The political cost of deleting
Every alert you delete was asked for by someone, and some of them will notice. The request was made in good faith, often after a bad day. Telling the requester that their alert has been removed can sound like telling them that their bad day did not matter. It did matter. That is the thing to say first.
A few habits make the conversation easier. Tell the requester before they find out, with the row from the triage table: "this fired forty times and was acted on zero times, so it was costing attention without buying anything". Offer the graph in place of the alert: the condition is still collected, still visible, still there for the next investigation. It just no longer interrupts anyone. And keep a retired-alerts log, so that "can we have it back?" is answered with "yes, in five minutes" rather than a debate.
The deletion that is hardest is the one created after a serious incident, because the alert has become a small memorial to it. Be plain: the incident was real. The alert did not catch a repeat because there was no repeat. The thing that would catch a repeat is probably a different alert, at a different severity, which step five will add. Nobody is deleting the lesson. They are deleting a rule that fired at the toner instead.
Sometimes you lose the argument, and the alert stays. That is fine. Demote it, route it to the person who asked for it and to nobody else, and put it on the list for the next triage. An alert that only bothers the person who wanted it is a problem that solves itself.
A quiet Sunday at Bellweather Logistics
The on-call phone at Bellweather Logistics received a hundred and forty notifications one Sunday on which nothing was broken. Most came from a backup job's hourly "still running" alert and an interface on a decommissioned switch that had been flapping, at critical severity, since the spring. The on-call admin turned the phone face down at eleven in the morning, which everyone later agreed was reasonable. The warehouse's WAN link dropped at half past two. Its alert was read at seven, when he turned the phone back over to check the time. The review recorded that "the alert was sent".
The triage that followed deleted a hundred and ten rules in an afternoon and demoted sixty more. It left the WAN link alert as one of nineteen critical rules. The following Sunday the phone received two notifications. Both were read.
The alert triage checklist
Use this once a quarter, starting now.
- The four diagnosis numbers have been pulled for the last quarter and recorded.
- The alerts-per-rule list has been pulled and sorted, and the top ten are on the triage table.
- The new-alert freeze is in place, with one named person able to grant exceptions.
- Every rule that fired last quarter has a row on the triage table with a decision and a reason.
- Every "delete" decision has been carried out and logged in the retired-alerts list.
- Every surviving rule has a severity chosen by the 3 a.m. test.
- The bottom severity notifies nobody.
- Every rule that fires on the edge of a threshold has hysteresis or confirmation set.
- Every alert rule has a named owner, not a distribution list.
- Every requester of a deleted alert has been told, with the row from the table.
- Missing alerts have been added only after the steps above, and each has an action line.
- The next triage is on the calendar.
Where to go next
Alerting on everything is what monitoring turns into when the only question ever asked is "should we add one?" The cure is to ask the other question, "did it help?", and to ask it of every rule, with numbers, once a quarter. Do that and the on-call phone becomes something a person can carry. The alert count falls, the useful share rises, and the one alert that saved you stops being lost among four hundred that did not.
Its twin in the scope family is monitoring everything, which is where most of the four hundred came from. What happens when nobody runs the triage again is set-and-forget rot, and the shared inbox that hid the WAN alert is in process anti-patterns. All of these are part of Common Monitoring Mistakes and Anti-Patterns.
Put this into practice with SysPeep Monitor
The triage needs alert history you can actually read, and rules that can be demoted and tuned without a fight. SysPeep Monitor keeps thresholds, dependencies, and alert rules designed to keep noise down. It sends email and other notifications from those rules. It reports on alert history from a single web console. That is where the four diagnosis numbers come from. The free trial installs on a Windows machine if you want to run the triage against your own alerts.
Download Free TrialLearn About SysPeep