Dynamic and Baseline-Relative Thresholds, Minus the Magic
Somewhere in your monitoring tool, or in the brochure for the next one, there is a checkbox promising thresholds that "learn". Turn it on, the story goes, and the system works out what normal looks like and alerts on anything unusual, with no numbers required. It sounds like the end of threshold tuning. It is not. It is also not a trick. Underneath the marketing sits a genuinely useful idea that solves a real problem fixed numbers cannot. I have watched it catch a 3 a.m. outage that no static line would have seen. I have also watched it page a colleague because it was Monday.
The real problem is rhythm. Your internet traffic at 2 p.m. on a Tuesday and at 2 a.m. on a Sunday are different worlds. Any single fixed line must be wrong for at least one of them. It is too high to catch a night-time anomaly, or too low to survive a normal afternoon. A dynamic threshold (also called adaptive or baseline-relative) is a threshold that moves with the clock and the history, so "too high" means "too high for now." That is the entire mechanism. Everything else in this guide is detail about how the line gets moved, and what happens when it moves the wrong way.
This guide explains how these mechanisms work in plain words and where they shine. It shows where the humble static number remains the honest choice, and how the "learning" fails when it fails. It does fail, in predictable ways, and each failure has a known guard. By the end you will be able to decide, check by check, which of your alerts deserves a moving line.
In this guide
- The family tree: three kinds of "dynamic"
- What dynamic thresholds need to work
- Where dynamic shines
- Where static stays king
- The honest failure modes — and their guards
- A pragmatic mixed design, plus the bake-off
The family tree: three kinds of "dynamic"
Vendors use one word for at least three different mechanisms. They differ in complexity, in what they need from you, and in how easy they are to trust. So it pays to know which one a given checkbox actually enables. (The checkbox does not say.)
Time-banded static: different numbers by hour and day
The simplest member of the family barely deserves the word dynamic: you keep ordinary fixed thresholds, but define different ones for different time windows. Business hours, the WAN link (the wide area network connection out to other sites or the internet) warns at 80% utilization. Overnight, when the only legitimate traffic is backups, it warns at 40%. Weekends get their own numbers. Nothing learns anything — you looked at a week of graphs and encoded what you saw into two or three bands.
Do not let the lack of glamour fool you. Time-banded static thresholds capture most of the value of dynamic alerting with none of the opacity. Every trigger is still a fixed number a human chose and can explain. For a small network, two bands — business hours and everything else — on a handful of rhythmic metrics is frequently all the "dynamic" you need.
Rolling-statistics bands: typical-for-this-hour, plus margin
The workhorse of real dynamic alerting. The tool keeps several weeks of history for a metric. For each time slot — say, Tuesdays around 10 a.m. — it computes what the metric typically does then. It finds a center value, plus a measure of how much it normally wanders. From those it builds a band: typical-for-this-slot, plus-or-minus a margin. Live values inside the band are normal; values outside it, in either direction, raise an event.
Made concrete: suppose the tool has six weeks of history for internet traffic, and looks at the samples taken near 10 a.m. on Tuesdays. It might see something like this:
Tuesdays ~10:00, download traffic (six weeks of samples) 112 98 124 105 131 118 (Mbps) center (typical value) ~115 Mbps normal wander ~ 25 Mbps either way band for Tue 10:00 ~ 90 to 140 Mbps live value right now 338 Mbps --> outside band: event
Repeat that little computation for every slot of the week and you have a corridor traced around the shape your graph makes in a normal week. It is wide where the metric is naturally jumpy, narrow where it is steady, and re-drawn as new history arrives. The important demystification: there is no intelligence here, just arithmetic on history. The brochure calls it learning. It is a calculator with a long memory. That is exactly why it works — and why it inherits every flaw of the history it is fed, a point we will return to under failure modes.
Anomaly scoring: ranking strangeness
The most ambitious branch. Instead of a visible band per metric, the system computes how unusual current behavior is — often across several metrics at once. It produces a strangeness score, alerting when the score runs high. Approaches vary by product and the details are usually proprietary. Described honestly at the concept level, they are all attempts to answer "how far from every normal pattern is this?" rather than "did this number cross this line?"
These systems can catch oddities no per-metric band would. And they can also raise an event because this Tuesday was mildly unlike the last six. Treat any claim beyond "it ranks deviations from learned patterns" with polite skepticism until you have watched it run on your own network for a while. More on that evaluation below.
What dynamic thresholds need to work
All three branches feed on the same three inputs, and starve without them.
History, and enough of it. A band computed from two weeks of samples is a guess wearing a confidence it has not earned — it has seen each weekday twice. Most rhythms need four to six weeks of clean history before "typical for Tuesday 10 a.m." means anything. Monthly patterns (payroll runs, month-end reporting) need several months. Until then, the band is folklore.
A stable rhythm to learn. Dynamic thresholds assume tomorrow will rhyme with last month. Metrics driven by office life — traffic, logins, request rates — usually do rhyme. A metric dominated by irregular events, or a network in the middle of reorganization, migration, or rapid growth, gives the math nothing repeatable to grip. The tool will still draw a band; the band will just be wide, jumpy, and useless. Whether your metrics have a dependable rhythm is precisely what a baselining effort tells you — the groundwork is covered in Baselining & Knowing Normal.
Sane handling of the learning period. Between enabling the feature and having real history, the system knows nothing. Good tools admit this and stay quiet or advisory during the warm-up. Naive setups alert enthusiastically off three days of data and burn the team's trust in week one — trust the feature never gets back. Know what your tool does before you flip the switch, and tell the team what to expect.
Where dynamic shines
Dynamic earns its keep on metrics with a strong, reliable rhythm — the ones where any single static line is guaranteed to be wrong twice a day. Traffic volumes on user-facing links. Connection counts through the firewall. Request rates, email flow, VPN (virtual private network) session counts, authentications per minute. On these, a band gives you something a static line never can. A 3 a.m. traffic surge gets caught even though it is far below the daytime threshold. And there is not a single false alarm at the daily peak.
The other genuine superpower is too-quiet detection. A band has a floor as well as a ceiling, so it notices when a metric goes abnormally low. Quiet is one of the most sinister failure shapes in monitoring. Traffic at 10% of a normal Tuesday morning does not trip any "too high" rule, but it absolutely means something. It could mean a dead upstream service, a broken integration, users who cannot reach you at all. The classic version of the story is the web shop whose order rate went to zero after a botched payment-gateway change. Every server was green, every utilization low and lovely. And the only metric telling the truth was the one that had gone suspiciously calm. Static thresholds are almost never written for "suspiciously fine"; bands get it for free, on both sides of normal, around the clock.
Where static stays king
Plenty of checks should never be handed to the math, and the reasons are worth being crisp about. "It learns" is the argument you will hear for moving everything. The strongest reason is the last one below. When a postmortem asks why the page fired, "the value exceeded 85, which we set for this reason" is an answer. "The model found it anomalous" is a shrug in a suit.
- Hard physical limits. Disk full is full at any hour of any day. The consequence of 100% does not care what is typical for Tuesday. Capacity metrics with absolute ceilings want fixed lines placed by lead-time reasoning, as covered in Static Thresholds Done Right.
- Compliance and SLA lines. If a contract or policy says a response over 2 seconds is a breach, the threshold is 2 seconds. A learned band that quietly relaxes as performance degrades would be a machine for hiding breaches from yourself.
- Weak-rhythm metrics. Temperature, disk growth, error counters, fan status — no meaningful daily shape, so a moving line adds motion without adding information.
- Anything you must explain afterward. Suppose the question "why exactly did this page fire?" will be asked in a postmortem or an audit. Then "the value exceeded 85, which we set for this reason" is an answer. "The model found it anomalous" is not. Where auditability matters, fixed lines rule.
The honest failure modes — and their guards
Dynamic thresholds fail in four characteristic ways. None is disqualifying; all are manageable — but only if you know to look, because each one fails quietly.
Drift-chasing. The bands learn from history — including bad history. Consider a memory leak that worsens a little each week, or a link creeping toward saturation over a quarter. The band absorbs the deterioration and re-labels it normal, week by week. The alert you were counting on recedes like a horizon. The guard: a periodic human look at the trend itself — is this metric's center moving month over month? — which slots naturally into the quarterly threshold review. And one hard rule: never let bands keep learning across an open incident, or the system will study the outage and conclude outages are normal. Most tools let you exclude a time range from learning; use it.
Meridian Parts turned on rolling bands for the warehouse application's response time and let them learn for six weeks. Unknown to anyone, the application had a slow memory leak, and response times were climbing a little every week. The band climbed with them. By the end of the learning period, a four-second page load sat comfortably inside "normal for Thursday afternoon". The static two-second line that would have caught it had been switched off as redundant. The warehouse staff reported the slowness by telephone, which is how the team learned that the band had learned the leak. The bands now re-learn only from weeks the quarterly review has marked as clean, and the two-second static line is back beside them.
Novelty panic. The mirror image. Anything legitimately new is, by definition, unlike history, so the system flags all of it. That includes a rollout, a migration, a new office coming online, a marketing campaign that doubles web traffic. A change-heavy week becomes an alert storm of true statements ("this is unusual!") with zero useful content. The guard: change annotation — when you change something on purpose, tell the monitoring system. Suppress or acknowledge the affected checks for the window. Let the bands re-learn on the new normal deliberately rather than letting them thrash.
Holiday confusion. Rhythm learning assumes the calendar repeats cleanly, and human calendars do not. A public holiday on a Tuesday looks like a Sunday wearing a Tuesday costume: traffic vanishes, and every band built from ordinary Tuesdays screams "too quiet" all day. The day after, back-to-work surges can trip the ceilings. The guard: calendar awareness where the tool supports it (holiday schedules, special-day profiles). Where it does not, use tolerance — expect band noise on exceptional days, and read it accordingly rather than re-tuning anything in response.
The black-box problem. The worst failure mode is not a wrong alert — it is an alert nobody can interrogate. If the tool cannot show you the band it drew and the value that escaped it, every alert becomes an argument between you and an oracle. In that case, on-call engineers stop trusting the oracle fast. The guard is a purchasing-and-configuration rule, the explainability test. Prefer tools and modes that display the expected range on the same graph as the actual value. "Alert because traffic was 340 Mbps against a typical 80–140 for this hour" is explainable; a bare anomaly score is not. If you cannot see the band, you cannot debug the band — and someday you will need to.
Tip: all four guards share a theme — dynamic thresholds remove the chore of moving the line, not the responsibility for supervising it. Budget a little recurring attention for the bands themselves, or drift and novelty will quietly spend your alert credibility for you.
A pragmatic mixed design
Nobody should run an all-dynamic estate, and after this far you can see why an all-static one leaves value on the table. The practical answer is an explicit allocation: each check runs the cheapest mechanism that can honestly do its job. Here is a worked allocation for a typical small estate:
| Check | Mode | Why this mode |
|---|---|---|
| Disk space, all servers | Static | Hard ceiling; lead-time reasoning; no rhythm |
| Certificate / backup / service status | Static | Binary conditions; auditability |
| Server CPU & memory | Static + duration | Sustained-level logic beats rhythm logic here |
| Temperature, UPS | Static | Vendor engineering limits rule |
| WAN utilization | Time-banded static | Two bands (office / overnight) capture the rhythm explainably |
| Internet traffic volume | Rolling band | Strong rhythm; too-quiet detection genuinely valuable |
| Firewall connection count | Rolling band | Rhythmic; surges and silences both meaningful |
| App request rate / logins | Rolling band | Rhythmic; low side signals user-facing breakage |
| Latency / packet loss | Static per path | Baseline-informed fixed lines; explainable afterward |
The pattern to steal is the ratio. Most checks stay static, a couple of rhythmic ones get hand-made time bands, and a handful of high-rhythm, high-value metrics justify real rolling bands. Dynamic thresholds are seasoning, not the meal. And the allocation itself is a decision worth writing down. Keep a one-line "why this mode" note per check, wherever you keep threshold reasoning. Then the next person (or next-year you) inherits the logic and not just the configuration.
The bake-off: evaluate before you trust
Never hand a production alert to a dynamic mechanism on faith. Run a shadow-mode bake-off first — the monitoring equivalent of hiring on a trial contract:
- 1. Pick two or three candidate metrics from the rolling-band rows above — rhythmic, valuable, currently static.
- 2. Enable the dynamic mechanism alongside the existing static threshold, with its notifications routed to a log or a folder nobody is paged by. The static alert keeps running production exactly as before.
- 3. Let both run for a month — long enough to include ordinary weeks and at least a little real trouble.
- 4. Compare the two records side by side. For every real problem in the period: which fired, and which fired first? For every firing: would it have been worth an interruption? Count the catches, count the noise, and note anything the band caught that static never could — the too-quiet events especially.
- 5. Promote the dynamic check to production only where it demonstrably won — earlier catches, new catches, or equal catches with less noise. Where it lost or tied, keep static.
The comparison at the end wants to be embarrassingly concrete. A good bake-off verdict reads like: "Real events this month: three. Static caught two. The band caught all three — including the Sunday-night traffic collapse static had no rule for. It caught the link congestion forty minutes earlier. Noise: static fired 4 times without cause, the band 11, of which 9 landed on the public holiday." That verdict writes its own decision: promote the band, add holiday handling. Where static won or tied, keep static and feel no shame about it; finding that out was the point of the month. A month of shadow data converts the discussion from faith versus cynicism into a comparison of two logs. That is the only discussion worth having about any alerting mechanism.
The decision in one table
Strip away the vendor language and dynamic thresholds are one old idea, automated: baseline plus margin — know your normal, alert on meaningful departure. The automation moves the line for you. It does not absolve you from knowing why the line is where it is, supervising what the math learns, and choosing the right mechanism per metric. That choice compresses to a few questions:
| The metric… | Run it as |
|---|---|
| Has a hard ceiling or a contractual line | Static, always |
| Must be explainable in audits and postmortems | Static (or time-banded static) |
| Has weak or no daily rhythm | Static, baseline-informed |
| Has a strong rhythm, simple shape | Time-banded static first |
| Has a strong rhythm, and "too quiet" matters | Rolling band — after a bake-off |
| Is still being learned (< 4–6 weeks history) | Static, until the history exists |
Where to go next
The order of work is fixed. Get the static foundation defensible first; Static Thresholds Done Right is the companion piece. Then layer bands onto the few metrics that earn them, and put both kinds under the same recurring scrutiny in the quarterly threshold review. Whichever mechanism draws the line, the craft of choosing and tending it lives in Setting & Tuning Thresholds. The checkbox that promised to learn is still there, and on a handful of rhythmic metrics it is the best thing in the tool. It has stopped paging anyone about Mondays, because nobody asks it to watch things that have no rhythm to learn.
Put this into practice with SysPeep Monitor
Trying the mixed design is easier with a tool that supports it. SysPeep Monitor lets you set per-check thresholds with schedules and duration conditions. It watches your metrics over SNMP, WMI and SSH, and keeps the history you need to judge what normal looks like. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep