Home › Topics › Baselining & Knowing Normal › Baselines in Alerting

Using Baselines to Set Smarter Alerts


Ask an admin why a CPU alert is set at 80%. You will usually get some version of "that's what it was set to when I got here." The number came from a forum post, a vendor default, or folklore. It came from anywhere, in fact, except the one place it should have come from: what this metric on this device normally does. Thresholds set by folklore fail in both directions at once. They fire on servers whose normal happens to live near the magic number, training everyone to ignore them. And they sleep through disasters on servers whose normal sits far below it, because "way above normal" can still be under 80%.


This guide is the payoff piece of the baselining series: turning documented normal into alert rules that catch change instead of noise. If you have a baseline register — built in collecting your first baselines — you already own everything the method needs. If the word baseline is still hazy, start with what a baseline really is; this guide leans on that vocabulary throughout.


You will learn one core move (threshold equals baseline plus a reasoned margin) and how to reason about the margin. You will learn what deviation-band alerting does and honestly does not do. You will also learn how to make alerts time-aware, and about the underrated two-sided alert. Finally, you will learn a review habit that makes every false alarm improve the system. At the end, a worksheet walks three real metrics from register row to finished alert rule.


In this guide

  • The core move: threshold = baseline + reasoned margin
  • Reasoning about the margin
  • Deviation bands: alerting on "unusual for this hour"
  • Alerts that know what time it is
  • The two-sided alert: too quiet is also wrong
  • The baseline-first alert review
  • The worksheet: three metrics, end to end
  • Where to go next

The core move: threshold = baseline + reasoned margin


Here is the whole idea in one sentence: a good threshold sits a reasoned distance above your documented normal, not at a number somebody once liked. Two worked contrasts show why this beats folklore in both directions.


Contrast one: the folklore number that alerts too late

An application server's register row reads: busy-hour CPU 45–60%, p95 58%, brief peaks to 65%. The inherited alert is at the folklore 80%. Now something goes wrong at 9 a.m. — a runaway process, a stuck query loop — and CPU climbs steadily. At 70% the server is already far outside anything it has done in six months. Users are already feeling the sluggishness. The 80% alert is still asleep and stays asleep for another hour of climb. Reset the threshold from the baseline instead — warn at 75%, sustained ten minutes. The alert then fires roughly an hour earlier, while the problem is one process to kill instead of an outage to explain. Same tool, same metric; the only change is that the threshold now knows what normal is.


Contrast two: the folklore number that cries nightly

Now the batch server. Its register row says: overnight processing runs CPU at 88–95% from 23:00 to 04:00, by design; daytime normal 20–35%. The inherited 80% alert fires every single night, and every single night someone glances, sighs, and deletes it. That is the classic training course in ignoring alerts. The baseline justifies a different rule, with a written reason. Overnight, alert only above 98% sustained thirty minutes (the machine is supposed to work hard; pegged-and-stuck is the abnormal state). In daytime, warn at 50% — because for this server, half-loaded at noon is the weird event worth a look. The baseline does not always lower thresholds. It puts each one where abnormal actually begins, which on some machines is far above the folklore number.


Tip: write the reason next to every threshold you set — one sentence in the register: "warn 75%: busy-peak is 60–65%, so 75% means something new is happening." A threshold with a written reason can be reviewed, defended, and safely changed later. A bare number becomes folklore the day you stop remembering why.


Reasoning about the margin


Two definitions before the reasoning, because the words carry weight. A warn is "look at this when convenient" — it goes to a dashboard or a daily digest and interrupts nobody. An alert is "stop what you are doing" — it earns the right to buzz a phone. Every threshold decision below is really deciding which of those two promises a deviation deserves. Keeping the two honest is what keeps either one meaningful.


Also decide what number the margin measures from. Anchor it to the top of documented normal — the busy-hour peak or the p95 — never to the average. The average is dragged down by nights and weekends. A margin measured from it is secretly re-counting the rhythm you already documented. It lands the threshold somewhere in the middle of an ordinary Tuesday. "Fifteen points above the highest level this metric routinely reaches" means something; "fifteen points above the average" mostly means "Monday."


So how far above normal should the line sit? The honest answer: it depends on what an interruption costs and what missing the change costs. You are choosing, in advance, how unusual something must be before it is allowed to interrupt a human. So the margin should be consequence-weighted. Three questions do the work:


  • How bad is a miss? For a Tier-1 service (the ones the whole company feels), a tight margin is worth some false alarms. For a lab box, a generous margin — or no alert — is fine.
  • How jumpy is the metric? A metric whose normal band is narrow and steady (DNS response time on a healthy LAN) can carry a tight margin. A naturally spiky one (CPU on a build server) needs headroom plus a sustained-for duration so single spikes never page anyone.
  • How fast does trouble develop? Disk space creeps over days — a wide margin with early warning is perfect. A saturating WAN link hurts in minutes — the margin must be closer to the action.

A practical starting pattern, to tune from — expressed as distance above the documented busy-hour normal, always paired with a sustained-for duration so momentary spikes stay silent:


TierWarnAlertSustained for
Tier 1: company-wide pain if it degrades~10–15 points above busy-peak~20–25 points above busy-peak5–10 min
Tier 2: a team feels it~20 points above busy-peak~30 points above busy-peak10–15 min
Tier 3: an admin feels it, eventuallyOptionalWell clear of any recorded peak30+ min

One more margin worth setting: the recovery margin. If a rule fires at 75% and clears at 74.9%, a metric hovering around the line will fire, clear, and fire again all afternoon. Admins call this flapping, and it is as fatiguing as any false alarm. Most tools let the clear condition sit lower than the fire condition. Set it meaningfully lower — back inside the documented normal band, not just under the threshold. That way, an alert only declares recovery when the metric has actually come home to normal. At that point, the metric has not merely ducked below the line to catch its breath.


Treat the numbers as scaffolding, not law — the register and the three questions outrank any table, including this one. Then record the outcome: add a threshold column and a reason note beside each row of the baseline register from the collection guide. Normal, threshold, and reason living on one line is the whole system in one glance. That makes the review habit at the end of this guide almost effortless.


Deviation bands: alerting on "unusual for this hour"


Everything so far uses fixed lines. Some monitoring tools offer a second mode, worth understanding in plain words: deviation-band alerting, sometimes called dynamic or adaptive thresholds. The tool computes, from weeks of history, what this metric typically does at this hour of this day — a rolling baseline. It draws a band of normal variation around that typical value. No fixed number anywhere: the alert fires when the live value leaves the band. Fifty percent utilization at 10 a.m. Tuesday sits mid-band and is silence. The same fifty percent at 3 a.m. Sunday is far outside the band and fires. The threshold has, in effect, learned the rhythm.


Used well, band alerting is genuinely powerful — it catches "wrong for the context" events no fixed line can see. But it is not magic, and it has three honest limits. First, it needs good history. Bands computed from two days of data, or from a period containing an outage, are bands around garbage. Second, it chases drift if unsupervised. A slow creep — utilization climbing a percent a month — stays inside the band the whole way up, because the band politely moves with it. Slow problems get absorbed into "normal" unless a human reviews the trend. Third, novelty is not badness. Bands flag unusual, and plenty of unusual is fine — a big legitimate file transfer, the first day of a new app. Expect band alerts to mean "look at this," not "this is broken." The practical stance: use bands where rhythm is strong and fixed lines are awkward. Keep fixed safety-net thresholds on the metrics that must never be wrong (disk full is disk full at any hour). And keep a human reviewing trends either way. For the deeper mechanics of tuning all of this, the threshold tuning pillar is the dedicated deep dive.


Alerts that know what time it is


Between one fixed line and full deviation bands sits a sturdy middle option most tools support: schedule-based thresholds. These are different limits for different times, driven by the rhythms you mapped in weekly rhythms and seasonal patterns. The pattern starts with a business-hours threshold set snug above the daytime normal, and a separate overnight threshold. That overnight threshold loosens for known heavy windows (backup night is supposed to be busy). Or it tightens dramatically because a quiet network should stay quiet. Even implementing this pattern on just your WAN link and Tier-1 servers removes a surprising fraction of routine false alarms. And where scheduling is not available, the bluntest tool still works: maintenance-window suppression. That means telling the monitoring system that known peaks like the backup window or patch night are expected. Their alerts are then muted rather than sent and sighed at. Suppression is its own small discipline with its own traps, covered in the maintenance windows and suppression pillar.


The two-sided alert: too quiet is also wrong


Folklore thresholds only ever look up: alert when the number is too high. But a baseline gives you a floor as well as a ceiling. The floor unlocks the most underrated alert in monitoring: the too-quiet alert. Abnormally low is also a deviation from normal, and it is very often the signature of something that died silently rather than loudly.


Consider what "suspiciously quiet" means against a documented normal. A DNS server whose query rate drops to zero during business hours has not become efficient. Either the service died, or clients are asking someone else, or your monitoring lost sight of it. All three are worth knowing about within minutes. Consider a WAN link that normally hums at 35–50% on a Tuesday morning. If it is suddenly flat near its overnight floor, that means the office lost its path out, even if the interface still reports "up." Backup traffic that fails to appear during the backup window is a failed backup announcing itself twelve hours before the failure report does. In each case nothing is "over threshold" — the classic alert sleeps soundly through the outage. Here is the rule of thumb: for every Tier-1 metric whose normal has a healthy busy-hours floor, add a low-side alert. The rule is "below the floor, during hours when it never is, sustained ten minutes." It is cheap, it is quiet in practice, and the day it fires it is almost always right.


The baseline-first alert review


Every alerting system produces false alarms; good ones get better because each false alarm triggers the same short ritual. When an alert fires and turns out to be nothing, the first question is not "how do I make this stop?" That is the road to muted alerts and missed outages. Instead, ask: what does the baseline say? Two diagnoses cover nearly every case, and they call for opposite fixes.


If the metric was inside its documented normal when the alert fired, the threshold was wrong. In that case, it was set too tight, or blind to a rhythm the register clearly shows (firing every backup night is a classic). Fix the rule: widen the margin, add a schedule, extend the duration, and update the written reason. But if the metric genuinely was outside its documented normal — the alert was "correct" and yet nothing is wrong — then normal has moved. In that case, the threshold was faithfully defending a baseline that no longer exists. That is not a threshold problem; it is a rebaselining trigger. Re-measure, confirm the new normal has an explanation, update the register row, re-derive the threshold from the new baseline. The full discipline — including the crucial rule about never rebaselining over an unexplained change — is in when normal changes. Run this ritual for a few minutes after every false alarm and the alert set converges toward trustworthy. Skip it and every fix is a guess.


The worksheet: three metrics, end to end


Here is the whole method as a repeatable sequence, then applied to three rows of a real-shaped register. The steps, per metric:


  • 1. Pull the register row: busy-hour normal, p95, peak pattern, quiet floor.
  • 2. Decide the consequence tier: who hurts, how fast, if this goes wrong?
  • 3. Set warn and alert as baseline + margin, each with a sustained-for duration.
  • 4. Add time-awareness where the peak pattern demands it (schedule or suppression).
  • 5. Add the low side if the metric has a meaningful floor.
  • 6. Write the reason beside the numbers, date it, and calendar a review after two weeks of live fire.

Metric one — application server CPU (Tier 1; normal 45–60%, p95 58%, peaks 65%). Warn at 75% sustained 10 minutes; alert at 85% sustained 5. Reason: "15 points above any recorded peak = something new." No schedule needed — its rhythm is mild. No low side — an idle CPU is not an incident. Review date set.


Metric two — WAN utilization, download (Tier 1; business-hours normal 35–50%, p95 58%, Monday login peak 78%, backup window 55–60% at 01:00–03:00, floor 3%). Business hours: warn at 85% sustained 15 minutes — above every recorded peak, close enough to saturation to matter. Overnight: suppress utilization alerts inside the backup window, and warn above 75% outside it. That is because a heavy night that is not the backup is a question. Low side: alert if below 10% for 10 minutes during business hours — the office has effectively lost its internet even if every interface shows green. Reasons written; review date set.


Metric three — DNS response time, primary internal server (Tier 1; normal 10–30 ms, p95 35 ms, brief 60 ms bursts at 09:00). Warn at 100 ms sustained 5 minutes — roughly triple the p95, safely above the login-burst noise. Alert at 250 ms or two consecutive failed lookups. Low side on the companion query-rate metric: near-zero queries during business hours, sustained 5 minutes, alerts — a silent DNS server is a loud problem. Reasons written; review date set.


Notice what just happened: nine judgment calls, and not one of them was folklore. Every number traces back to a measured normal plus a stated reason. That means every one can be reviewed, defended, and improved when the register changes.


Roll the method out the same way: three metrics this week, live for two weeks, reviewed, then the next three. Converting the whole register in one heroic afternoon produces a wall of untested rules and a noisy fortnight that erodes exactly the trust you are building. Alerting earns credibility one honest threshold at a time.


Where to go next


This is the loop that makes monitoring trustworthy. Baselines inform thresholds, and thresholds fire. Every firing — true or false — feeds the review. And the review updates either the rule or the baseline. Two companion guides complete the loop. The guide to rebaselining when normal changes handles the "normal moved" branch. The threshold tuning pillar goes deep on the rule-mechanics branch. For the rest of the baselining series, head back to Baselining & Knowing Normal.


Put this into practice with SysPeep Monitor

Every rule in this guide needs two ingredients: history to read your normal from, and flexible thresholds to encode it. SysPeep Monitor collects metrics over SNMP, WMI, and SSH. Its thresholds, dependencies, and alert rules are designed to keep noise down. So the margins you reasoned out here fire as email or other notifications only when something real changes. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep