Static Thresholds Done Right: Picking Numbers You Can Defend
A static threshold is a fixed line, drawn in advance, that a metric must not cross. "Alert me when disk usage passes 85%." "Warn me when CPU stays above 90%." It is the simplest idea in monitoring and the least fashionable one. Vendors would rather talk about machine learning. Meanwhile static thresholds quietly run most of the world's real alerting. A fixed number chosen well is fast to set up, easy to understand at 3 a.m., and completely predictable. Those are not small virtues. Every threshold in the first monitoring system I ran was 80%. I had not chosen a single one of them.
The mechanism is not the problem. The problem is how the numbers get picked: copied from a screenshot, inherited from a predecessor, or invented under pressure and never questioned again. A default ships with the tool, and nobody has a reason to move it. And a year later it is on every disk in the fleet, from the 60 GB system drive to the archive array. Nobody can say why, because there was never a why.
This guide is about doing the humble fixed threshold properly. By the end you will be able to set a starting value for every common metric family and explain why that value and not another. You will be able to write it down so that the next person can defend it too. No statistics degree required. The only habit you need is asking "what would actually happen at this number?"
In this guide
- What makes a threshold defensible
- Starting values, metric family by metric family
- The two-threshold pattern: warning, critical, and the gap
- Writing it down: the threshold register
- Four common static-threshold sins
- The defensibility checklist
What makes a threshold defensible
Here is the test this whole guide builds toward. For every threshold you set, you should be able to say its reason out loud to another human without feeling silly. Try it on your current alerts. "We alert at 85% disk because at this server's growth rate that leaves us roughly two weeks to free space or order storage" survives being said aloud. "We alert at 80% because 80 felt right" does not — and you can hear the difference immediately. A threshold nobody can explain is not a decision. It is a superstition with an email address.
A defensible threshold has three parts. First, a reason anchored to a consequence. The number exists because something specific happens past it: users start waiting, a queue starts building, hardware starts throttling, or your remaining time-to-react shrinks below what your team needs. The best reasons contain a "so that": warn at this level so that we still have time to do X calmly.
Second, a source, and sources have a ranking. At the bottom is the generic starter value — the kind you find in a reference table of healthy ranges for common network metrics. Starter values are honest and useful on day one; they are simply not yours. Above them sits your own baseline: what this metric actually does on this device across normal weeks. A baseline beats a generic number every time, because "normal" varies wildly between networks — the whole discipline of measuring yours is covered in Baselining & Knowing Normal. And for one family we will get to, vendor engineering limits outrank everything. The practical rule: start from a reference, replace with a baseline as soon as you have one, and defer to the vendor where physics is involved.
Third, an owner and a date. Somebody chose this number, on some day, knowing what they knew then. Recording who and when sounds bureaucratic until the first time you stare at an alert and wonder whether 70% was wisdom or a typo. A threshold with a name and a date attached can be questioned, updated, and trusted. An anonymous one can only be obeyed or ignored.
Starting values, metric family by metric family
Different metrics fail differently, so each family gets its own reasoning pattern — not just its own number. Learn the pattern and you can re-derive the number for any device you meet. (If you are still deciding which checks to run at all, start with the first ten checks and come back.)
Disk space: think in lead time, not percentages
A disk threshold is not really about fullness. It is about how much time you have left to act once the alert fires. The right question is never "what percentage is scary?" but "at this volume's growth rate, how long does the remaining space last?" If a log volume grows 2% a week, an 85% warning gives you seven or eight weeks of calm. If a database volume grows 2% a day, that same 85% gives you one nervous week. Same number, wildly different meaning — which is why the percentage alone is never the reason.
Beware the big-volume fallacy: 90% used sounds alarming until you notice it is 90% of 10 TB. That leaves a full terabyte of runway — possibly years at that server's growth rate. On very large volumes, alerting on free space remaining (say, below 200 GB) or on days-until-full at current growth is far more honest than any percentage. Starting point: warn at 85%, alert at 95%. Then immediately sanity-check both against growth rate and absolute free space, and adjust until the warning buys the lead time you actually need.
Ashgrove Housing learned the big-volume fallacy from a forty-terabyte archive array. The fleet-wide disk threshold was 80%, so the array crossed it one Friday evening with eight terabytes free and paged the on-call engineer. The engineer worked out that eight terabytes was about four years of growth and went back to bed. Nobody wanted to be the person who loosened a disk alert, so it was acknowledged instead of changed. It re-opened after every monitoring restart for the next five months. The review described the alert as "technically accurate". The array now warns when free space falls below two terabytes, which still leaves a year of lead time. The threshold register says who decided that and why.
CPU and memory: the duration is part of the threshold
A CPU spike to 100% is not a problem; it is a computer doing its job. A CPU pinned at 95% for twenty minutes is a different animal. For utilization metrics, the threshold is really two numbers: a level and a duration — "above 90% sustained for 15 minutes." Leave out the duration qualifier and you will be paged for every backup job and virus scan on the fleet. (Duration qualifiers and their cousins get a full guide in the anti-noise toolkit.)
Per-role normals matter just as much. A batch or render server that runs flat-out all night is healthy at 100% CPU — that is what you bought it for. A domain controller at 100% is an emergency. Starting point: warn at 85% CPU or 90% memory sustained for 15 minutes on general-purpose servers. Then create explicit exceptions per role rather than loosening the fleet-wide rule to accommodate the busiest machine.
Interface utilization: the queuing-pain zone
Network links do not degrade politely from 0% to 100%. They feel fine, fine, fine — and then queues form. As sustained utilization climbs past roughly 70–80%, packets increasingly wait in buffers, latency climbs, and users feel "slowness" long before the link is technically full. That queuing-pain zone, not 100%, is what your threshold should guard. Starting point: warn at 70% sustained for 10 minutes, alert at 85–90%.
And remember that direction matters: a link has separate inbound and outbound counters, and they fail independently. An internet connection with plenty of download headroom can be choking on uploads from an offsite backup. Threshold each direction on its own, and expect their normals to differ.
Latency and packet loss: your normal beats the reference
Latency thresholds are the clearest case of the reference-versus-your-normal split. A generic table can tell you that under 100 ms feels snappy for interactive work and that sustained loss above 1–2% ruins calls — true and useful. But whether 60 ms is fine or alarming depends entirely on the path. A reading of 60 ms to the office next door means something is badly wrong. A reading of 60 ms across an ocean is excellent. So latency thresholds must be set per path, anchored to that path's own measured baseline. Alert when a path runs well above its normal. A common starting rule is sustained latency at double the baseline, or loss above 1% over ten minutes. Do not alert just because it crosses a number borrowed from a chart. Reference values are for day one, before your baseline exists.
Service response times: anchored to human patience
For a web app, a database query, or a login, the consequence that matters is a human waiting. So anchor the threshold to patience, not to the server's comfort. Interactions that feel instant sit under a few hundred milliseconds. "Noticeably working" is a second or two. Somewhere past several seconds users assume it is broken and start clicking things twice. Starting point: warn when a key transaction takes over 2 seconds, alert over 5. Then tighten to what your users and any promises you have made actually require. That promise might be a formal SLA (a service level agreement, the written commitment about how fast something will be) or a hallway expectation. Averages hide most of this pain, which is why response times are the natural home of percentile thresholds.
Temperature and environment: where vendor numbers rule
Here the hierarchy flips. Your opinion about a comfortable intake temperature is worth nothing next to the manufacturer's operating range. The consequence — throttling, shutdown, shortened hardware life — is set by the engineering, not by your traffic patterns. The pattern for this family: take the vendor's documented maximum and subtract a safety margin large enough to act in. If the datasheet says the device tolerates up to 40°C ambient, warn around 30°C and alert around 35°C. That way, the alert arrives while the air conditioning is failing rather than after it has failed. The same logic covers UPS (uninterruptible power supply) load, humidity, and fan status: vendor limit, minus room to react. The hardware has read its own datasheet. You should too.
| Metric family | Starting warn | Starting critical | The reasoning anchor |
|---|---|---|---|
| Disk space | 85% used | 95% used | Lead time at this volume's growth rate |
| CPU / memory | 85–90% for 15 min | 95% for 15 min | Sustained, not spiking; per-role normals |
| Interface utilization | 70% for 10 min | 85–90% | Queuing pain starts before "full" |
| Latency / loss | 2× path baseline | Loss >1% / 10 min | Per-path normal, not a borrowed chart |
| Response time | >2 s | >5 s | Human patience and promises made |
| Temperature | Vendor max −10° | Vendor max −5° | Engineering limit minus reaction margin |
Tip: every value in that table is a starting point, not a verdict. The column that ages well is the last one — the anchors stay true even as the numbers move. When a threshold and its anchor disagree, believe the anchor and change the number.
The two-threshold pattern: warning, critical, and the gap
Most metrics deserve two lines, not one, and the two mean different things. Warning means: nothing is broken, but start planning — investigate during business hours, order the disk, schedule the cleanup. Critical means: act now, whatever the hour. The distance between them is not decoration; the gap is your planning window. Warn at 85% disk and go critical at 95%, and the 10 points between are literally the time you gave yourself to respond calmly instead of heroically.
That means the gap should be sized per metric, by asking how fast the metric moves and how long the fix takes. A slow-growing file server can live with a narrow gap — it will take weeks to cross. A volume that fills fast needs a wide one, or the critical will arrive while you are still reading the warning. If procurement takes three weeks in your organization, your disk warning must fire at least three weeks of growth before critical. That single sentence justifies more threshold values than any reference chart ever will.
A few metrics are honestly one-threshold metrics, because no meaningful "start planning" state exists. A device is reachable or it is not; a backup job succeeded or it failed; a certificate handshake works or it does not. Inventing a warning level for a binary condition just manufactures a second notification with no distinct action attached. Every notification without its own action is a small deposit in the alert fatigue account.
Write it down: the threshold register
The defensibility test — reason, source, owner, date — only works if those four things are recorded somewhere. Memory does not count: in six months you will not remember why the mail server warns at 78%, and neither will the person who replaces you. The fix is a threshold register: one small table, anywhere durable. Use a wiki page, a shared spreadsheet, even comments in the monitoring config next to each check. The habit must survive contact with a busy week.
Four columns carry all the weight. Here is the shape, with realistic rows:
| Check | Value | Reason | Source / owner / date |
|---|---|---|---|
| FILESRV01 D: disk | Warn 80%, crit 92% | Grows ~1%/week; warn leaves ~3 months, procurement takes 3 weeks | Baseline / J.M. / spring register review |
| WAN link out | Warn 70% / 10 min | Queuing latency measurable past ~75% on this circuit | Own testing / J.M. / after circuit upgrade |
| SQL01 CPU | Warn 90% / 20 min | Nightly index job pins CPU ~12 min; 20 min clears it, real runaways do not | Baseline / A.R. / post-incident 41 review |
| Server room temp | Warn 30°C, crit 35°C | Vendor ambient max 40°C; margin covers a weekend AC failure response | Vendor datasheet / A.R. / install |
Notice what the reason column does: every row survives being read aloud. Notice also what the register makes possible. When a threshold misfires, you can see what its author knew and decide what changed, instead of arguing with a bare number. The register is also the raw material for the single best habit in this whole topic, the scheduled threshold review, where these rows get graded against a quarter of real alerts.
Four common static-threshold sins
Sin one: copied defaults, fleet-wide. The tool shipped with 80/90, so every check on every device runs 80/90 forever. Defaults are a vendor's guess about the average network; yours is not average, because no network is. The default was chosen by someone who has never met your network, and it is doing its best anyway. Defaults are fine scaffolding for week one — the sin is never replacing them.
Sin two: round-number worship. Thresholds cluster at 80, 90, and 95 for no reason except that humans like tidy numbers. The metric does not care about base ten. If the queue pain on your WAN link starts at 73%, then 73% is the honest warning. An oddly specific number in the register is a quiet signal that somebody actually measured something.
Sin three: set during an incident, never revisited. Mid-outage, someone drops a threshold to 60% "so we never miss this again." Understandable — and now that check fires every Tuesday afternoon during the weekly export, teaching everyone to ignore it. Incident-driven changes are legitimate inputs; the sin is letting a decision made at 2 a.m., under adrenaline, with one data point, stand unreviewed forever.
Sin four: the same number for every server. One disk threshold across the fleet ignores everything this guide is about. The 10 TB archive and the 60 GB system drive, the batch server and the domain controller, do not share a normal. Uniformity feels like tidiness but is really just sin one wearing a policy costume. Group similar roles, by all means — then let each group's number have its own reason.
Tip: all four sins share one root: a number that outlived its reasoning — or never had any. You do not need to fix the whole fleet this week. Fix each threshold as it next fires, and write the reason down as you go; within a quarter the loudest offenders will all have been through your hands anyway.
The defensibility checklist
Before a new threshold goes live — or when an old one comes up for review — make it survive these five questions:
- 1. Can I say the reason out loud? One sentence, containing a consequence: what starts going wrong, or what lead time disappears, past this line.
- 2. What is the source — and is a better one available? Reference table, own baseline, or vendor limit. If it is still a generic number and you now have weeks of history, the baseline is waiting to be promoted.
- 3. What action happens when it fires? A warning implies a planning task; a critical implies an immediate one. If you cannot name the action, you have designed noise, not an alert — the actionability test from Alert Design applies to every line you draw.
- 4. Does the warning-to-critical gap buy enough time? At this metric's realistic speed, could you actually complete the fix between the two lines? If not, widen the gap or admit it is a one-threshold metric.
- 5. Is it written down, with a name and a date? Value, reason, source, owner, date — in the register, where the next review (and the next admin) can find it.
A threshold that passes all five is a number you can defend — to your team, to your manager, and to yourself at 3 a.m. When it wakes you, you need to trust that it was right to.
Where to go next
Static thresholds done right will carry most of your alerting most of the way. Their one real weakness is that a fixed line cannot follow a metric with a strong daily rhythm. And that is exactly where dynamic and baseline-relative thresholds earn their keep. The values you picked here only stay defensible if they are re-graded against reality on a schedule, which is the whole point of the quarterly threshold review. Both guides, and the rest of the tuning craft, live in Setting & Tuning Thresholds. Every threshold in my first system was 80%. None of them is now, and I can tell you why for each one.
Put this into practice with SysPeep Monitor
Every idea in this guide needs a monitoring tool that lets you set warning and critical levels per check, with duration qualifiers, instead of one blunt number. SysPeep Monitor does exactly that: per-device thresholds over SNMP, WMI and SSH, designed to keep alert noise down. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep