The Quarterly Threshold Review: Tuning as a Habit
Every threshold you set is a guess. A careful, reasoned, defensible guess if you have done the work, but a guess. It is a prediction that this line, on this metric, will separate trouble from normal life. Reality then spends three months grading the prediction, in the most honest medium there is: the alert history. Every alert that fired and was acted on is a pass. Every alert that fired and was deleted unread is a fail. Every incident that arrived with no alert at all is a question you did not know was on the exam.
Most teams never collect the grades. The alerts pile up, and ignoring them becomes a reflex. And the thresholds, which are the actual adjustable causes of all of it, sit untouched for years. I once inherited a fleet where the newest threshold change was older than the newest server. The threshold review is the fix: a short, scheduled, evidence-driven session where you read the quarter's grades and move the numbers that earned moving. It is the least glamorous ritual in monitoring and one of the highest-leverage hours in your calendar.
This guide gives you the whole ritual. It explains why it must be scheduled, what evidence to gather, the standard move for each kind of finding, and the one-hour agenda. It provides a worked example that walks a realistic quarter of alerts to five concrete decisions. None of it needs software you do not already have.
In this guide
- Why tuning needs a calendar, not a mood
- Gathering the evidence: dispositions and silent failures
- The review moves, finding by finding
- The register: every change gets a paper trail
- The one-hour agenda
- A worked review, and the ritual checklist
Why tuning needs a calendar, not a mood
Without a schedule, thresholds get touched at exactly one kind of moment: during or immediately after an incident, under adrenaline, with an audience. Call it tuning by trauma. The outage that slipped past a threshold gets that threshold yanked tight. The alert that annoyed someone important gets loosened into silence. And every change is a reaction to the most recent painful event rather than a reading of the whole record. Trauma-tuned systems drift toward a signature shape: hair-trigger alerts on whatever failed most recently, and dangerous silence everywhere attention has not recently visited.
A calendar fixes the sampling problem. Reviewing on a schedule means you judge thresholds against everything that happened — the boring weeks included — instead of against the one night everyone remembers. Quarterly is the sweet spot for most small teams. A quarter is long enough to accumulate a statistically meaningful pile of alerts and at least a few real incidents. It is short enough that the estate has not changed under your feet and the incidents are still in living memory. Monthly reviews are fine if your alert volume is high; yearly ones are archaeology.
Northgate Retail's first review was mostly archaeology. It spent its longest twenty minutes on a single check: the till server's CPU warning, set at 55% sustained for two minutes. The warning had fired on forty-one afternoons that quarter and been acted on once. Nobody in the room knew why it was 55. The monitoring tool's change log eventually gave it up. The threshold had been dropped from 85 during a checkout outage two years earlier, by an engineer who had since left. Nobody had touched it since. The outage it was tightened for had been caused by a failed disk, not by CPU. The review put it back to 85 sustained for fifteen minutes and wrote the reason in the register with a name and a date. And the check has fired twice since, both times about something real.
The quarterly cadence gets one supplement: the event-triggered mini-review. After any big change — a server migration, a new circuit, a major application rollout — spend ten minutes asking which thresholds the change just invalidated. Do that rather than letting the next quarter discover it via false alarms. The classic example: the WAN (wide area network) upgrade that quadruples a link's capacity. Its utilization thresholds, set as absolute traffic rates for the old circuit, quietly become four times too tight. The link spends a month "alerting" at a quarter of its real capacity. Minis handle change; the quarterly handles drift; together they mean no threshold is ever more than a few months from a pair of skeptical eyes.
Gathering the evidence: dispositions and silent failures
The review runs on two lists, and the first is the easy one: the quarter's alert history, which your monitoring tool already keeps. Export or view every alert from the period, grouped by check, with counts. The raw counts alone are revealing — the top five checks usually account for a comfortable majority of all noise. But counts only become grades when you attach a disposition to each alert: what did a human actually do about it? Three buckets cover reality:
- Acted on: someone investigated or fixed something because this alert fired. A true positive. The alert earned its interruption.
- Acknowledged-ignored: someone saw it, shrugged, and moved on — "yeah, it does that." The most important bucket, because every entry is a documented case of the system training people not to listen.
- False / self-cleared: nobody needed to do anything and nothing was wrong; it resolved by itself or reported a non-problem.
Where do dispositions come from? Anywhere durable. Some monitoring tools let you attach a comment when acknowledging an alert — one word there ("fixed", "known", "noise") is a disposition. If alerts open tickets, the ticket outcome is the disposition. Failing both, the spreadsheet tally below does the job. The mechanism matters far less than the existence of any record connecting "it fired" to "and here is what a human did."
An honest note: if you are not recording dispositions today, you are in the majority, and the first review is where you start. You do not need workflow software. A weekly two-minute tally works: skim the week's alerts every Friday and mark each one A, I, or F in a spreadsheet. Rough is fine; the signal you are after (this check fired 30 times, action count zero) survives any amount of sloppy bookkeeping. Building this habit into a lightweight weekly alert glance is part of the broader review discipline covered in Alert Design.
The second list is the one nobody's tool exports: the silent-failure list — every incident from the quarter that monitoring did not announce. Mine it from the ticket queue, the helpdesk log, and honest memory: what broke that we learned about from a human? Each entry is the other half of the grade — the exam question you missed. Each is worth more than a hundred noisy alerts, because misses are where outage minutes actually come from. If the quarter's list is empty, either you had a genuinely great quarter or your ticket search was too gentle; check twice.
The review moves, finding by finding
With both lists in hand, the review itself is close to mechanical: each finding matches one of four patterns, and each pattern has a standard move.
Noisy alerts: fired often, seldom acted on
The signature: high fire count, action count near zero. The moves, in order of preference. Raise the threshold, with a written reason. If disk warnings at 85% produced twelve alerts and zero actions because the volume just lives there, then 85% is not a meaningful line on this volume. Add confirmation or dwell if the firings were short-lived transients — the mechanics live in the anti-noise toolkit. Demote severity if the information is worth keeping but not worth an interruption — let it land on a dashboard instead of a pager. And when a check protects nothing and informs no decision, delete it honestly rather than muting it into a zombie. The discipline of un-monitoring — deciding what deserves silence — is a running theme of Monitoring Scope. Deleting a useless alert is not lowering your standards; it is raising the signal.
Missed incidents: it broke, nothing fired
For every silent failure, run a tiny postmortem — four questions, in order: Was there a check on the thing that failed at all? If not, the move is a new check. Was there a check, with a threshold the incident never crossed? Then tighten the threshold, with the incident as your calibration point. Did it cross, but too late to matter? Then the move is faster detection: shorter polling, quicker confirmation. Did an alert fire and get ignored? Then the problem is not this threshold — it is the noise burying it, and the fix is the noisy-alert moves above. Misses convert directly into next quarter's coverage; they are the most valuable finding a review produces.
Boundary-dancers: trigger, clear, trigger, clear
The signature: the history shows the same check raising and resolving in tight pairs, minutes apart, dozens of times — a metric camped on its threshold, tap-dancing. The move is almost always hysteresis: separate trigger and clear levels with a gap wider than the metric's normal wiggle. If the dancer is an up/down check rather than a numeric one, the move is flap damping instead — and a ticket. A flapping object is a finding, not just noise.
Validated alerts: fired, mattered, acted on
The signature: fire count modest, action count close to it. These are your winners — and the move is to leave them alone. This deserves saying explicitly, because reviews breed a temptation to demonstrate value by adjusting things. Resist it. A threshold that caught real problems and produced no noise is a calibrated instrument; "improving" it is how calibrated instruments die. The do-no-harm rule: a review that changes nothing about a working check is a successful review of that check. Not every quarter must move every dial.
The register: every change gets a paper trail
Every move the review makes goes into the threshold register — the same value/reason/source/owner/date record introduced in Static Thresholds Done Right. Record it as a change entry: old value, new value, why, date. Ten seconds per change while the reasoning is fresh. The payoff compounds. Next quarter, you are reviewing both thresholds and your own past decisions with outcomes attached. "We raised this in the spring review; did the noise actually stop?" The register turns tuning from a sequence of disconnected nudges into an experiment log.
The register is also where you track the only two numbers that summarize whether all of this is working — the two metrics of alerting health, checked quarter over quarter:
- Are we getting quieter? Total alerts per week, and the share of them acted on. Falling volume with a rising action share means the noise is dying and the signal is not.
- Are we catching more? Of the quarter's real incidents, how many did monitoring announce before a human did? This is the miss rate inverted — the coverage grade.
Improving on both at once is the entire game. Quieter but blinder means you over-pruned. Louder but sharper may be a fine trade for a growing estate. Quieter and sharper, quarter after quarter, is what healthy tuning looks like. It is also the direct antidote to the slow-motion failure described in Alert Fatigue, where noise teaches a team to stop believing its own monitoring.
The one-hour agenda
The ritual fits in one timeboxed hour — and the timebox is load-bearing: an open-ended review becomes a philosophical debate about monitoring; a one-hour review produces decisions. The agenda:
| Time | Item | Output |
|---|---|---|
| 10 min | The two health metrics vs last quarter: alert volume, action share, incidents caught first by monitoring | The trend, stated aloud |
| 30 min | The top ten noisiest checks, one by one: disposition pattern, then the standard move | A tuning decision (or an explicit "leave it") per check |
| 15 min | The silent-failure list: the four miss-postmortem questions per incident | New or tightened checks |
| 5 min | Register update: every change logged old/new/why/date | The paper trail |
Who attends, in a small team: whoever answers the alerts, and one person who sees the ticket queue. Often that is the same human wearing both hats, and that is fine. A solo review works. Speaking the findings out loud to a colleague works better. "This fires daily and we have never once acted on it" is a sentence that triggers action when said aloud and triggers nothing as a private thought. If you are a department of one, the register is your colleague — write the sentences down. Preparation is deliberately light: pull the alert summary and the ticket search the day before, and resist doing any actual tuning until the hour itself. That way, the decisions get made against the whole quarter rather than whichever alert annoyed you most recently.
Tip: put the review on the calendar as a recurring appointment now, before the first one happens. A ritual that lives in someone's intentions dies inside two quarters. A ritual that lives in the calendar merely gets rescheduled occasionally, which it survives.
A worked review: one quarter, five decisions
Here is the method applied to a realistic quarter. The alert summary, straight from the monitoring tool plus the Friday tallies:
| Check | Fired | Acted on | Pattern in the history |
|---|---|---|---|
| DC01 disk C: warn 85% | 22 | 0 | Hovers 84–86%; warns, clears after nightly log rotation, repeats |
| Switch port 14 up/down | 41 | 1 | Down/up pairs in bursts, a few seconds each |
| App response time (avg-based) | 15 | 1 | Fires ~02:10 during maintenance; once at midday, self-cleared |
| WAN latency | 9 | 8 | Real congestion events, each investigated |
| Backup job result | 3 | 3 | Two failed jobs, one missing report — all real |
And the silent-failure list, mined from tickets: the mail queue backed up for four hours one Thursday. Users noticed; monitoring said nothing — no check exists on queue depth. One branch office lost its VPN (the encrypted tunnel back to head office) overnight. A check exists, but at 15-minute polling with 3 confirmations, users beat the alert by twenty minutes... and the alert then drowned among port 14's flapping.
Walking the method through the table produces five decisions. One of them comes with a ticket attached, because damping is anesthetic, not treatment:
- 1. DC01 disk — noisy: the volume simply lives near 85 and grows glacially. Raise warn to 92% with the lead-time math written down (at ~1% growth per month, 92% still leaves a quarter's warning). Add a clear level at 89% so log rotation stops bouncing it.
- 2. Port 14 — boundary-dancer, and a finding: enable flap damping (one "flapping" alert instead of forty transitions) and open a ticket to replace the patch lead and transceiver. The flapping itself is a hardware finding.
- 3. App response — noisy and blind: the average-based rule alerts on maintenance arithmetic and slept through daytime slowness complaints. Convert it to a p95-over-15-minutes rule with two-window confirmation, per percentile thinking.
- 4. The misses — coverage: add a mail-queue-depth check (baseline it for two weeks, then threshold with margin). Fix the VPN check's delay budget: 5-minute polling, 2 confirmations — worst case drops from ~45 minutes to 10.
- 5. WAN latency and backups — validated: leave both exactly alone, and say so in the register ("reviewed, no change, earning its keep"). Do no harm.
This quarter's health metrics tell their own story: ninety alerts fired and thirteen were acted on. That is an action share around one in seven, dominated by two checks that produced sixty-three firings and one action between them. And two real incidents reached the team through humans before monitoring said a word. Both numbers now have somewhere to go, and next quarter's ten-minute metrics slot will say whether decisions one through four moved them. That is the feedback loop closing.
One hour, five entries in the register, and next quarter starts with two fewer noise factories, two closed coverage gaps, and a written record of why.
The ritual checklist
- 1. Recurring calendar slot, quarterly, one hour — plus a ten-minute mini after any big change.
- 2. Dispositions recorded all quarter (the Friday two-minute tally counts).
- 3. Evidence in hand before the hour starts: alert summary by check, and the silent-failure list from tickets.
- 4. Health metrics first: quieter? catching more?
- 5. Standard moves: noisy → raise/confirm/demote/delete; missed → the four postmortem questions; dancers → hysteresis or damping; validated → hands off.
- 6. Every change in the register: old, new, why, date.
- 7. Stop at the hour. Unfinished items open next quarter's agenda.
And the reason to keep the appointment, quarter after quarter: tuning debt never holds still. Every threshold is aging toward wrongness as the network changes underneath it. The only question is whether the drift is being paid down on a schedule or accumulating toward the incident that will force the issue. Each review makes the next one easier: less noise to wade through, fewer misses to postmortem, a register that already explains itself. Teams that keep the ritual for a year are routinely startled by the "before" numbers in their own register. The compounding is real; it just compounds in whichever direction you choose by keeping, or skipping, the appointment.
Where to go next
The review assumes there is something worth reviewing: defensible starting values (Static Thresholds Done Right), noise mechanics on the right checks (the anti-noise toolkit), and honest choices about moving lines (dynamic thresholds). The whole craft, end to end, lives in Setting & Tuning Thresholds. Reality will grade your thresholds whether or not you read the marks. The review is the one hour a quarter where you do.
Put this into practice with SysPeep Monitor
A quarterly review is only as good as the history it reads. SysPeep Monitor keeps your alert history and metric data in one place, with per-check thresholds and alert rules you can adjust as the review decides. Its dashboards and reports make the quarter's story easy to pull up. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep