The Weekly Alert Review: Keeping the System Honest
Nobody decides to have bad alerting. It happens the way a garden becomes a jungle: by itself, while you were busy. The network changes and the alerts do not. Thresholds that were right for last year's traffic quietly become wrong for this year's. Alerts added during one bad week never get removed. One by one, messages start arriving that nobody acts on, and everyone develops the small, fatal habit of deleting first and reading… eventually. This is decay by default: an alerting system left alone does not stay the same — it gets worse, silently, until the day it matters.
The antidote is not a heroic cleanup (that jungle grows back) but a small standing ritual: the weekly alert review. It takes fifteen minutes, with one agenda, four questions, and one improvement made every single week. It is the operational pulse of your alerting: frequent, shallow, and relentless. Its deep-dive sibling already has its own guide. The quarterly threshold review re-derives your numbers from a quarter of evidence. The two are designed to feed each other. The weekly pulse notices and patches, and its accumulated notes become the quarterly review's raw material. This guide covers the pulse: the format, the metrics, and how to run it alone or with a team. It also covers where the findings flow, and a worked example week to make it concrete.
In this guide
- Why alerting decays by default
- The fifteen-minute format
- The numbers that keep it honest
- Running it solo versus as a team
- Where the findings go
- Keeping the review itself alive
- A worked week
- The weekly review kit
Why alerting decays by default
It helps to see that the decay has mechanisms, because each mechanism is something the review will specifically counter. The world moves. Traffic grows, servers take on new roles, and that "temporary" subnet becomes permanent. Every change quietly invalidates some assumption baked into some alert. Alerts only accumulate. Creating one takes a minute and feels responsible. Deleting one takes courage and feels risky, so nobody ever does. Additions minus zero removals equals sprawl. Incidents add, calm never subtracts. After every rough patch someone adds alerts and promotes severities; no quiet month ever triggers the reverse. And attention adapts. Humans are magnificent at learning to ignore repeated stimuli — the mechanism the alert fatigue guides dissect. So a noisy alert does not just waste time. It trains the team to skim, and skimming is fatal precisely once.
Notice what all four mechanisms have in common: they are invisible day to day. No single week of drift is noticeable; the sum of fifty-two of them is a monitoring system nobody trusts. The review works because it is the one moment each week when someone looks at the alerting itself. They look not through it at the network, but at it, and ask whether it is still telling the truth.
The fifteen-minute format
The whole ritual is a standing agenda run against one artifact: the week's alert list with dispositions. This is the fired/acked/resolved record. Each alert carries its one-word verdict (fixed, self-healed, false-positive, known-issue) from the alert lifecycle. The list comes from your monitoring platform's alert history or your ticket system. It can also come from the shared alert log if you run the minimum-viable version. Wherever it lives, the facilitator pulls it before the review starts. Fifteen minutes of review should never begin with ten minutes of report-wrangling. Then walk five steps:
- 1. Pull the week (2 min). The alert list with dispositions, sorted by alert name so repeats clump together.
- 2. What fired most? (3 min). The top offenders list: the three to five alerts with the most firings. Volume is not guilt — a link that genuinely flapped all week deserved its noise. But the top of this list is always where the leverage is. Fixing one frequent alert improves more weeks than fixing five rare ones.
- 3. What was noise? (3 min). The false-positive tally, per alert — plus the softer count of alerts nobody acted on at all. An alert that fired six times and was actioned zero times is telling you it wants to be a dashboard, a digest line, or a memory.
- 4. What did we miss? (3 min). The inverted question, and the one nobody thinks to ask: problems that happened without an alert. The mechanic is the tickets-versus-alerts cross-check. Skim the week's helpdesk tickets and incidents for anything monitoring should have caught first. Examples include the outage a user reported and the full disk found by accident. Every miss is a coverage gap or a threshold set too politely, and it goes on the list.
- 5. What's still open? (2 min). The forgotten-alert sweep: anything unresolved and older than a few days. Some are legitimately long-running (waiting on parts); the rest are the cracks the lifecycle warned about — acked and abandoned, or auto-resolved in name only. Each gets an owner or gets closed honestly.
Then the step that makes the ritual compound: take one action, every week, minimum. Tune one threshold. Demote one severity. Delete one alert that has never once been acted on. Draft one runbook stub for the thing you fixed manually twice. One is deliberately small — small enough that no busy week can excuse skipping it. It is enough, because the arithmetic is on your side. Fifty-two small fixes a year is a transformed alerting system, achieved without a single cleanup project ever appearing on a roadmap. Teams that try to fix everything in one purge relapse; teams that fix one thing a week never stop improving.
Tip: resist the urge to fix things during the review — that way lie ninety-minute reviews that soon stop happening. The review decides; the fix happens afterwards, as a normal task. The only in-review work allowed is writing the action down with a name on it.
The numbers that keep it honest
Memory is a flatterer. Ask anyone how alerting felt this week and you will hear about the one dramatic day; only numbers remember the whole week. The review therefore keeps a small scorecard — six numbers, one row per week, tracked in the same sheet as the action log:
| Metric | What it tells you | Healthy direction |
|---|---|---|
| Total alerts fired | Overall interruption load on the team | Falling, then stable and low |
| By severity (crit / warn) | Whether the severity pyramid holds — few criticals atop many warnings | Small critical share |
| Acked within target % | Whether response promises are being kept | Rising toward "always" |
| False-positive % | How often the system cried wolf | Falling |
| Actioned % | Share of alerts that led a human to do something | Rising — the truest signal-quality number |
| After-hours pages | The human cost: interrupted evenings and sleep | Falling, and every one justified |
Two honest cautions. First, resist inventing targets: there is no universal correct false-positive percentage, and the healthy absolute numbers depend entirely on your fleet's size and stakes. What is universal is the direction — week over week, a healthy system trends fewer, truer, calmer. That means fewer total interruptions, a higher fraction of them real and actioned, and fewer of them at dinner time. The trend line is the health story; any single week is just weather. Second, never let the scorecard become a performance review of people. The moment "acked within target" is used to grade a human, the numbers will be gamed into meaninglessness. The metrics judge the alerting; keep it that way and they stay true.
A word on collection, because this is where good intentions go to die. The scorecard must be cheap to fill or it will not be filled. Most of the numbers fall straight out of your platform's alert history or reports; the rest come from a quick count over the shared log. If a metric takes more than a minute to produce, approximate it or drop it. A six-number scorecard kept every week beats a twelve-number dashboard abandoned by spring. This is a coffee-break tally, not a data warehouse project. Its whole value is in the unbroken weekly rhythm rather than in any individual figure's precision.
Running it solo versus as a team
Solo — and many readers of this site are an IT department of one — the review is the Friday-coffee version. It uses the same table, same four questions, same one action: fifteen minutes with the week's list before the weekend. The discipline matters more alone, not less, because there is no colleague to notice the drift for you. The scorecard is your second opinion. Write the one action down even though nobody else will read it — especially because nobody else will read it. Future-you, staring at a weird alert in six months, will find the note "demoted this to warning — fired 11 times, zero impact." Future-you will silently thank past-you.
As a team, add three pieces of machinery. A rotating facilitator means whoever ran last week's on-call runs the review. That spreads ownership and guarantees the person with the freshest scar tissue is holding the pen. A blameless tone, structurally enforced: the review judges alerts, never people. "This alert false-alarmed four times" is on the agenda; "why did you ignore this alert" is not. That is because the honest answer is usually "because the alert taught me to," and that is the alert's fault. The moment reviews feel like trials, people stop reporting the very ignores and misses the review runs on. And the two-minute timebox: no single alert gets more than two minutes of discussion. If it needs more, it becomes an action item with an owner ("investigate WAN-03's flapping") and the review moves on. The timebox is what keeps fifteen minutes true — and what keeps attendance from decaying the way the alerts did.
Where the findings go
A review that only admires the week is a book club. The findings have destinations, and knowing them turns each observation into a routed work item.
- Threshold evidence → the quarterly review. Weekly notes like "disk warning fired at 85, nobody acted, filled to 88 and stopped" are exactly the evidence the quarterly threshold review runs on. The weekly log means the quarterly session starts with a dossier instead of a shrug.
- Repeat offenders → redesign. An alert that tops the offenders list three weeks running has outgrown tuning. It needs its worksheet revisited — condition, guards, severity, audience, the works — per the anatomy of a good alert. Sometimes the honest redesign is deletion.
- Fake criticals → demotion. A critical that fired after hours and turned out to be plan-work ammunition feeds the severity audit from severity design. That is the wake-someone bar, applied with the week's receipts in hand.
- Repeated manual fixes → runbooks. The disposition log saying "fixed, same way, third time" is a runbook writing itself. The procedure is already proven; it just is not written down yet.
- Fatigue signals → load decisions. Rising totals, falling actioned %, growing after-hours pages: those trends are alert fatigue arriving in the data before it arrives in a resignation. The review is where you catch it early enough to matter.
Keeping the review itself alive
Fair warning about week one: the first review will not take fifteen minutes, because it inherits the whole backlog at once. That means months of accumulated noise, a dozen obvious offenders, and several alerts nobody can explain. Do not try to clear it. Run the same agenda, pick the same single action, and let the backlog drain at one fix a week like everything else. The first review's real job is smaller and stranger: producing the first scorecard row, so that next week has something to be compared against. Trend lines have to start somewhere.
The review is a system too, and it decays by the same physics. Three defenses. First, treat skipped weeks as the alarm. One skip is life; two in a row is the beginning of the end. That is because the jungle does not pause while you are busy — busy weeks are when it grows fastest. The fifteen-minute size exists precisely so that "too busy" is never quite true. Second, defend the calendar slot: same day, same time, a recurring appointment with the same standing as patching. A review that floats to "whenever things calm down" is a review that has already ended, since things do not calm down. Third, keep the review log — the one-line-per-week record of scorecard numbers and the action taken. It is your tuning history: the answer, forever after, to "why is this threshold 92?" and "when did we demote that?" It turns your monitoring from a pile of settings into a system with a memory. When a new colleague eventually joins, the log is the fastest possible education in why the alerting is the way it is.
A worked week
Here is a plausible Friday, fifteen minutes, coffee in hand. The week's pull, condensed:
alert fired dispositions notes WAN-03 link degraded 9 7 self-healed, 2 fixed flapping 02:00-04:00 FS-D-01 disk warning 1 fixed archived old logs UPS-HQ on battery 1 false-positive poller glitch CERT-VPN expires 21d 1 open (renewal ordered) fine DB01 CPU high 4 4 self-healed each ~5 min, no impact tickets check: printer outage Tue had NO alert (out of scope, Tier 3 - OK) open sweep: switch port-errors alert, acked 9 days ago, silent since
Walk the questions. What fired most? WAN-03, nine times, mostly self-healing in the small hours — that is a flap pattern, and two real fixes hiding among seven ghosts. What was noise? The UPS false positive (logged, not actionable this week) and arguably DB01's four five-minute CPU blips, which nobody has ever acted on. What did we miss? Nothing in scope — the printer stays gloriously unmonitored, as decided in your coverage policy. What's still open? The port-errors alert, acked and forgotten for nine days: it gets an owner and a Monday deadline on the spot.
Now the one action — and the discipline of choosing. The tempting action is DB01, because deleting noise feels good. But the leverage is WAN-03: nine firings, sleep-adjacent hours, and a self-healed pile that smells like a real intermittent fault plus over-eager triggering. The action: add a sustained-for guard to WAN-03 so sub-ten-minute blips stop firing. Also open a ticket to review the link's error counters for the underlying fault. Written in the log with an owner. DB01 goes on next week's shortlist — and that is the whole trick: one week, one deliberate fix, no heroics, fifty-one more coming.
The scorecard row takes another thirty seconds. It records sixteen total, one critical, fifteen warnings, one false positive, roughly a third actioned, and zero after-hours pages that reached a human. Next Friday, if the WAN-03 guard worked, the total should drop by half a dozen. In that case, seeing that drop on the trend line is the small, real reward that keeps the ritual funded with your own motivation. Note the certificate alert, too, doing quiet exemplary work. It fired once, three weeks out, and somebody ordered the renewal. Nobody will ever remember there could have been an outage. That is what "the system working" looks like in the data: boring.
The weekly review kit
Everything above as a copy-paste kit — agenda, scorecard row, action log. One sheet with three tabs, or one text file, whichever your team will actually keep:
AGENDA (15 min, standing)
1. Pull week's alerts + dispositions .............. 2 min
2. Top offenders: what fired most? ................ 3 min
3. Noise: false positives + never-actioned ........ 3 min
4. Misses: tickets/incidents without alerts ....... 3 min
5. Open sweep: unresolved > 3 days ................ 2 min
6. Choose ONE action, assign owner, log it ........ 2 min
SCORECARD (one row per week)
week | total | crit | warn | acked-in-target % | false-pos %
| actioned % | after-hours pages | note
ACTION LOG (one line per week)
week | action taken | owner | result seen (filled in later)
Fifteen minutes, six numbers, one action. It is almost insultingly simple. It is the difference between an alerting system that erodes into background noise and one that, a year from now, is quieter, truer, and trusted by everyone it wakes.
Where to go next
The review keeps honest what the rest of the pillar builds. Those are alerts designed through the anatomy worksheet and labeled by severity design. They are worded per alert message writing and tracked through the alert lifecycle that produces the dispositions this ritual feeds on. Start anywhere; these guides close the same loop. All five live in Alert Design.
Put this into practice with SysPeep Monitor
A weekly review needs a week's alert history at your fingertips. SysPeep Monitor keeps the record — what fired, when, at what severity. Its reports and dashboards let you pull your Friday numbers. Its thresholds, dependencies and alert rules are built to keep the noise you are reviewing down in the first place. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep