Synthetic Test Frequency: The Cost, Noise, and Coverage Triangle
Every monitoring tool eventually asks you a small, innocent-looking question: check interval? There is a default in the box, often five minutes. Most admins accept it for everything, forever. The other camp reasons that faster must be better and sets everything to thirty seconds. It discovers the consequences in stages. The application team asks why the login page is being hammered all night. The alert channel fills with false alarms, and nothing gets fixed any faster. I have been in both camps. The second one is louder.
Both camps are answering by instinct a question that has real arithmetic behind it. This is short, friendly arithmetic, the kind you can do on a napkin. How often a check runs is a three-way trade. It balances how fast you find out, what the checking itself costs, and how much false noise you are willing to put up with. This guide gives you the napkin: what an interval can and cannot detect. It explains how to match frequency to how much a service can hurt you and what your probes cost the thing they test. It also covers the confirmation trick that beats raw speed at its own game. At the end there is a decision table you can apply to your whole check list in ten minutes.
In this guide
- The triangle: coverage, cost, noise
- The detection math made simple
- Which outages can an interval even see?
- Tiering: match the interval to the damage
- What your probes cost the target
- Confirmation beats raw frequency
- Trend data vs. alerting
- The frequency decision table
The triangle: coverage, cost, noise
Every frequency decision balances three corners. Coverage is what you buy by checking more often. Outages are detected sooner, and shorter outages become visible at all. Cost is what you pay for it — and not mostly in money. Each check consumes a little of the target's capacity. A synthetic login is real work for the application. Each check also consumes a little of your monitoring host. If you use a hosted checking service, it often requires a bigger service tier, too. Noise is the sneaky third corner: every check executed is one more lottery ticket for a false alarm. Probes fail for their own reasons — a lost packet, a brief hiccup on the probe host. A check that runs 1,440 times a day gets 1,440 chances to be wrong.
You cannot maximize all three corners at once. You can only choose a sensible point inside the triangle per service. That last phrase is the escape hatch. The mistake is not picking a wrong interval; it is picking one interval for everything. The payroll application and the warehouse label printer do not deserve the same answer. Giving them the same answer overpays for one and underprotects the other. (The label printer will not be offended.)
The detection math made simple
Here is the entire theory. Suppose a check runs every N minutes, and an outage begins at some random moment. The next check lands — on average — half an interval later. So detection takes N/2 on average, and up to N in the worst case. An outage that starts one second after a passing check waits nearly the full interval for the next one. A one-minute check finds trouble in about thirty seconds, worst case a minute. A five-minute check: about two and a half minutes, worst case five. A fifteen-minute check: about seven and a half, worst case fifteen. Add whatever confirmation delay your alert rule imposes. A "3 failures in a row" rule adds two more intervals — more on that soon. With that delay added, you have your true time-to-alert.
One refinement worth knowing: the check's timeout rides on top of all this. A check does not report failure the instant it starts. It waits out its timeout first. So a one-minute check with a thirty-second timeout has a true worst case of a minute and a half. In that worst case, the first red result does not even exist until then. Keep timeouts realistic: a little above the service's slowest honest response. Never make them longer than the interval itself. Otherwise, slow checks begin stacking up behind each other and your tidy schedule turns into a queue.
Before you race to shrink N, hold it against the other clock: your response time. Suppose nobody will act on a warehouse printer alert until someone is standing in the warehouse anyway. Detecting its death in thirty seconds instead of seven minutes then changes nothing except your load and your noise. Fast detection only matters where fast reaction follows.
Which outages can an interval even see?
Average detection time is only half the story. The other half is stranger: some outages are invisible to your checks entirely. An outage that lasts L minutes is guaranteed to be seen only when L is at least N. A check must land inside it. When the outage is shorter than the interval, catching it is literally a matter of luck. The chance is L divided by N. Otherwise, it begins and ends in the quiet gap between two passing checks, leaving no trace. The arithmetic looks like this:
| Outage length | Checked every 1 min | Every 5 min | Every 15 min |
|---|---|---|---|
| 30-second blip | A coin flip | About 1 in 10 | About 1 in 30 |
| 2 minutes | Always seen | About 2 in 5 | About 1 in 7 |
| 5 minutes | Always seen | Just barely always | 1 in 3 |
| 10 minutes | Always seen | Always seen | 2 in 3 |
| 30 minutes | Seen within 1 min | Seen within 5 min | Seen within 15 min |
Read the fifteen-minute column and a truth falls out. A fifteen-minute check is an honest instrument for "is this thing up, generally," and nearly blind to brief trouble. Suppose users complain about two-minute brownouts (spells when the service is up but barely usable) and your checks run every five minutes. Your monitoring will confirm their experience about two times in five. In practice, that reads as "monitoring says it's fine," repeated until nobody on either side trusts the dashboard. The system is not lying. It is sampling, and the sample keeps missing. This sampling blindness is one of active monitoring's built-in gaps. The Blind Spots of Active and Passive Monitoring tours the full collection. The cure for repeated short blips is either a faster check on that one service or a passive data source that never blinks. Interface error counters are one example.
Tiering: match the interval to the damage
The right question for each service is not "how fast could I detect a failure?" but "what does an undetected minute cost?" — in money, in idle staff, in screams. Sort your services by that cost and the intervals assign themselves into tiers. A worked example for a typical small organization:
Tier 1, every minute: the internet edge, the firewall, the identity or directory service everything authenticates against, and the phones. The payroll or ERP application also belongs here in its busy season. ERP means enterprise resource planning: the system that runs orders, stock, and invoices. A minute of outage here idles the whole organization, and someone will genuinely drop what they are doing to respond. Fast detection converts directly into shorter outages.
Tier 2, every five minutes: the file server, the intranet, print services, the Wi-Fi controller, per-site checks at the branches. Failures matter and get fixed within the hour. A five-minute interval costs you an extra two minutes of average detection time. That extra time disappears inside the walk to the server room.
Tier 3, every fifteen minutes: the warehouse label printer, the training-room gear, test systems, the dashboard screen in the lobby. Run the honest experiment: the label printer dies at 6:41 AM. A fifteen-minute check reports it by 6:56 at the latest, typically around 6:48. The first person who needs a label arrives at 7:00. Nothing about anyone's morning changes if you had known at 6:41. But checking it every minute would have cost 1,440 daily probes and fifteen times the false-alarm exposure, for zero changed outcomes. Fifteen minutes is not neglect. It is the interval that matches the damage.
Tiering also has a time dimension. Some tools can run a service on a faster schedule during business hours and relax it overnight. That fits things like payroll perfectly. Use it where it exists; do not mourn it where it does not. Getting the tiers right matters far more than refining them hourly.
What your probes cost the target
A ping is close to free for everyone involved. A synthetic transaction is not. A scripted login exercises the directory service, spins up a session, and evaluates authentication policies. It touches lockout counters, writes audit-log rows, warms caches with fake activity, and may occupy a license seat. All of that is real work, performed by the very application you are trying to protect. Now the napkin again: one synthetic login per minute is 1,440 logins a day. Run it from five vantage points (the places a probe runs from; a sensible placement plan). That is 7,200 a day, over 50,000 a week. At some point the security team opens a very reasonable investigation into who is methodically hammering the ERP at 3 AM. The answer is: you are.
You can keep the coverage and shrink the footprint with a few habits of politeness:
- Stagger your vantage points. Five probes on the same schedule will fire in the same instant. Offset them across the minute so the target sees a steady trickle rather than a single pulse.
- Add jitter. A small random offset each cycle stops your probes from marching in step with cron jobs, backups, and every other on-the-minute event. It also avoids aliasing, a sampling trap. In that trap, a probe that always fires at second :00 keeps hitting (or keeps missing) the same recurring spike. It reports a distorted picture of the service.
- Reuse sessions where it is safe. Log in fully once an hour and run lighter authenticated checks between. Keep your eyes open about the trade: the light checks are no longer testing login itself. Test what you mean to test; reuse where the login is not the point.
- Use a dedicated, flagged account. Give the probe its own account. Tell the application owner and the security team it exists, and label its noise so it is recognizable in audit trails. Resist exempting it from the controls real users face. An exempted probe tests a road nobody else is allowed to drive.
- Stay read-only, and clean up. Prefer transactions that change nothing. Where the realistic test must write (a test order, an uploaded file), make deleting it part of the probe.
These habits are covered properly in Designing Synthetic Probes That Mimic Real Users. So is the deeper craft of making a probe behave like a person instead of a battering ram. For this guide, the frequency-shaped conclusion is this: heavyweight synthetics usually belong on a slower schedule than your instinct says. In that setup, a cheap fast check covers the gap.
Confirmation beats raw frequency
The naive way to detect faster is to check faster and alert on the first failure. The math above says that works. The noise corner of the triangle says you will regret it, because a single failed check is weak evidence. One lost packet produces a red result. So does one garbage-collection pause (a moment when an application stops to tidy its own memory) or one momentary stumble of the probe host. Alerting on first failure means every such twitch pages a human. The fix is a confirmation rule: require several failures before alerting — "2 of the last 3 checks failed." Then the alert fires on a pattern, not a twitch.
Here is the part worth writing on the whiteboard: fast checks with confirmation beat slow checks on speed and noise at the same time. Compare two configurations watching the same service. Configuration A checks every five minutes and alerts on the first failure. Its average time-to-alert is two and a half minutes, worst case five — and every stray lost packet pages you. Configuration B checks every minute and alerts when 2 of 3 fail. The first failure lands about thirty seconds in. The confirming failure lands a minute later. Configuration B's average time-to-alert is about a minute and a half, worst case about two. A lone blip cannot page you, because one failure out of three passing neighbors never satisfies the rule. B is faster than A and quieter than A. You spent frequency and bought both.
The rule of thumb that falls out: think of interval × confirmations as your real detection time. Choose the pair together, spending confirmation checks to buy calm. Even the strict version — every minute, 3 of 3 to alert — averages about two and a half minutes to alarm. Its worst case is near three. The strict version is still ahead of the five-minute single-shot check, while shrugging off two consecutive blips. For services checked from several vantage points, the same idea works in space as well as time. Require two vantages to agree before declaring an outage. You have filtered out most probe-side and path-side false alarms in one move.
Tip: confirmation rules are for down alerts. Slowness alerts need a different shape: response times twitch by nature. So alert on a sustained condition — "average response above threshold for 15 minutes" — rather than on any single slow sample. A one-off slow reading is weather. A quarter hour of slow readings is climate, and climate is what deserves a ticket.
Trend data vs. alerting
One more distinction quietly saves both load and disk. The frequency you need for alerting is not the frequency you need for trends. Alerting cares about freshness — how stale can your knowledge be before it embarrasses you. Trend data cares about history — enough samples, over enough months. Those samples can show that the portal has been gaining forty milliseconds a quarter or that Monday mornings are the seasonal storm. A response-time sample every ten or fifteen minutes builds beautiful long-term curves. That is thousands of points over a season, plenty for any capacity conversation. That sampling rate is useless for fast detection — and that is fine, because detection was never its job.
In practice one check often serves both masters. The fast tier feeds alert rules, and the same results, rolled up, become the trend record. Most tools summarize old data into coarser averages automatically. The point of the distinction is what it removes: the urge to crank frequency "for better graphs." The graphs do not need it. Only the pager does, and only for services whose tier justifies it.
Bellweather Logistics learned that from its own flow collector. The dispatch portal had a synthetic login running every two seconds from three vantage points. It was set that way for the smoothness of the response-time graph. Because the three probes shared a schedule they arrived in the same instant, like a tiny synchronized raid. The portal would much rather have seen a gentle drizzle. The week the flow collector went live, the monitoring server was the second-largest talker on the network. It was behind the nightly backup and ahead of every human in the building. The portal's audit log had grown to the point where the security team had stopped reading it. The review described the probe as "thorough". It now runs once a minute with a two-of-three rule, offset by twenty seconds per vantage point. The response-time graph looks exactly the same.
The frequency decision table
All of the above, folded into one table you can apply across a check list in an afternoon:
| Service class | Interval | Confirmation rule | Why |
|---|---|---|---|
| Org-wide lifelines: internet edge, firewall, identity, phones | 1 min | 2 of 3 | Minutes of outage idle everyone; blips must not page |
| Customer-facing services: website, portal, mail | 1 min | 2 of 3, agreed by two vantage points | Outsiders judge you by these; vantage agreement kills path-side false alarms |
| Key internal apps: ERP, file server, intranet | 1–5 min | 3 of 3 (at 1 min) or 2 in a row (at 5) | Response is within the hour; calm matters as much as speed |
| Heavyweight synthetic transactions | 5–15 min | 2 in a row | Each run costs the app real work; pair with a cheap 1-min reachability check |
| Convenience gear: printers, test systems, lobby screens | 15 min | 2 in a row | Detection speed changes no outcome; keep it cheap and quiet |
| Trend-only response-time logging | 10–15 min | None — no alert attached | History for capacity talks, not alarms |
Notice the heavyweight-synthetic row leans on a pairing. The expensive scripted login ambles along every five or fifteen minutes. Meanwhile, a nearly free reachability check on the same service sprints every minute. The fast check catches the dead service in seconds. The slow one catches the subtler "up but broken" failures the cheap check cannot see. Layered like that, you get most of the coverage of a fast synthetic at a small fraction of its cost. That is the triangle bent as far in your favor as it goes.
Treat the table as a starting point, then let real incidents tune it. After every genuine outage, ask two questions: did the alert beat the first user report? Would a faster interval actually have changed what you did? Promote a service to a faster tier only when the second answer is yes. And glance occasionally at the failures that were detected but never alerted — the lone blips your confirmation rules absorbed. A service quietly collecting blips every day is not producing noise. It is producing a warning, at a frequency your rules were wise enough not to page you about. You are wise enough not to ignore it.
Where to go next
Frequency is one of the three design decisions every active check needs. Another is what the probe should actually do — see Designing Synthetic Probes That Mimic Real Users. The third is where it should run from. A check's vantage point changes what it can see at any frequency. The rest of the Active vs. Passive Monitoring pillar covers those. It also covers blending active tests with passive collection into one budget-shaped design. And if you are still building your first check list and frequency feels premature, start with the first ten checks. The tiers in this guide will be waiting when you get back. And the next time a tool asks you that small, innocent-looking question, you will have an answer for each service instead of one for everything. You will have a napkin to show for it.
Put this into practice with SysPeep Monitor
Intervals and confirmation rules only pay off if your monitoring system makes them easy to apply per service. SysPeep Monitor is built around exactly that. It offers checks over SNMP, WMI, and SSH on the schedules you choose. Its thresholds, dependencies, and alert rules are designed to keep noise down. That way, a blip stays a blip and a real outage finds you fast. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep