Shrinking Detection Time: The MTTD Playbook
The file server had been frozen for forty-nine minutes before anything told me. I was at my desk the whole time, drinking coffee in perfect innocence. The dashboard was green, because the only check on that server was a ping. The server answered pings beautifully while every share on it was locked solid. Those forty-nine minutes are detection time, and their average across incidents is MTTD, mean time to detect. It is the cruelest number in the incident log. Repair time at least buys something, because while it runs someone is working the problem. Blind time buys nothing.
MTTD is also the more mechanical of the two clocks to improve. Repair time depends on humans: skill, documentation, sleep. Detection time depends almost entirely on configuration: whether a check exists and how often it runs. It also depends on how the check decides to alert and how fast the alert reaches a person. Those are levers you can pull on purpose, and most of them are cheap.
This guide is the playbook. First comes the diagnostic habit that tells you which lever your network needs, then the levers in rough cost order. Then comes a campaign format that turns it all into one focused quarter with a before-and-after number. The guide assumes you know what MTTD measures; if not, start with the incident clock and come back.
In this guide
- Diagnose first: how did we learn of this?
- Lever 1: coverage — the check that didn't exist
- Lever 2: sensitivity — the check that waited for catastrophe
- Lever 3: intervals — the polling clock you can't beat
- Lever 4: routing — detected by the system, unknown to humans
- Lever 5: event channels — hearing between the polls
- The outer perimeter: synthetic user checks
- The MTTD campaign: one focused quarter
- The playbook card
Diagnose first: how did we learn of this?
Before touching any lever, adopt one habit: after every incident, ask a single question and write the answer down. How did we first learn of this? The answers sort into a four-rung taxonomy, and the rung tells you everything about where your detection time is going. Users are wonderful people, and as a detection layer they are a sensor with feelings and no timestamp.
- An alert. Good — monitoring did its job. The follow-up question is how fast: compare the alert time against your best estimate of the fault time. A ten-minute gap is the system working. A ninety-minute gap means a check existed but was slow, blunt, or badly routed — levers 2 through 4.
- A user report. The classic monitoring gap. If users are your detection layer, your MTTD is however long it takes an annoyed human to give up and call. In that case, add however long the problem brewed before it annoyed anyone. Every user-detected incident is a free, precise answer to "what check are we missing?" That is lever 1.
- Accident. Someone happened to be logged in and noticed. This is worse than a user report, because it means even the users had not found it yet. Pure luck stood between you and a much longer outage.
- Never directly. The silent degradation discovered later: the backup that had been failing for weeks, unearthed during a restore; the dead redundant link found during maintenance. This is the worst rung, because the blind time is unbounded — and because whatever it was, it is the kind of thing that fails again.
Log the answer per incident. The detected-by column of the six-column incident log exists precisely for this. One ratio falls out for free: the detection-source ratio, the percentage of incidents monitoring caught before anyone else. Trended over quarters, this is the health metric of your entire monitoring investment, arguably more useful than MTTD itself. MTTD says how fast you find out. The ratio says whether finding out is by design or by luck. If you run post-incident reviews, the detection question slots straight into the template. It is the review question with the highest yield-per-minute in this whole discipline.
Lever 1: coverage — the check that didn't exist
The biggest MTTD wins come from the humblest source: adding checks for things that broke without one. Work it gap-driven: every user-reported or accidentally-found incident names a missing check with perfect precision. The thing that just failed unwatched is telling you exactly what to watch. File shares froze while the ping stayed green? Add a share-access check. Mail queued silently for three hours? Add a queue-depth check. The rule is mechanical and worth making policy. No incident closes on the "user report" or "accident" rung without either adding the check that would have caught it, or writing one line about why not. A few honest "why nots" exist — some failures are genuinely one-off — but they should be the exception that gets written down, not the default.
Gap-driven additions are reactive by nature. The systematic complement is the coverage review. This means periodically walking your inventory against your check list and asking what could break unwatched. That way, the gap gets found before the incident does. That discipline has its own guide, and pays into MTTD directly. While you are there, look for depth gaps as well as absence gaps. The server that is pinged but whose actual service is never exercised is half-covered. It is alive at the network layer, unwatched at the layer users touch. Our frozen-file-server example is the canonical case. Climbing from "it answers pings" to "it does its job" is the ladder the uptime and availability guides teach rung by rung.
Lever 2: sensitivity — the check that waited for catastrophe
The next class of blind time comes from checks that existed but alerted late. The usual shape: a threshold set for catastrophe when the incident was a slide. The disk alert at 95% full that fired twenty minutes before the database stopped — technically a detection, practically a starting gun. The latency alert set at one full second while users suffered at 400 ms for two days. In the incident log this appears as an alert-rung detection with a suspiciously long fault-to-alert gap. The check watched the whole slide and spoke only at the cliff edge.
The cure is the warning tier: a lower, earlier threshold that flags degradation while it is still cheap to investigate. It sits beneath the critical threshold that means drop-everything. Warn at 85% disk, alert at 95%; warn when latency doubles its normal, alert when it is genuinely unusable. The warning catches the slide days earlier, often converting a would-be incident into a Tuesday-afternoon task. That is the best MTTD of all: the incident that never started.
The other sensitivity dial is the confirmation delay — requiring, say, three failed checks in a row before alerting. That way, one dropped packet does not page anyone. Confirmation is good engineering, but it is purchased with detection time. On some checks it overpays: three confirmations at five-minute spacing adds ten minutes to every real detection. That is a fine price on a flappy Wi-Fi access point and a terrible one on the core firewall. Rebalance per check: tight confirmation on noisy, low-stakes checks; minimal confirmation on stable, critical ones. This is the classic noise-versus-speed trade, and it interlocks with how you tier and schedule alerts. And the tiering guide covers the noise side of the same coin.
Lever 3: intervals — the polling clock you can't beat
A hard floor hides under everything above: detection cannot be faster than the polling clock. A check that runs every five minutes cannot notice anything in under zero-to-five minutes of it happening. For a failure landing at a random moment, the average wait is half the interval. The arithmetic compounds quietly. A 15-minute interval with two-failures confirmation means an average of about 7.5 minutes to the first failed poll. Add 15 more to the confirming one. That is over twenty minutes of built-in blindness before routing even begins, with the check working exactly as configured.
The lever is selective tightening. Identify the handful of checks where minutes genuinely matter: the internet link, the firewall, the services whose outage stops the whole office. Poll them at one-minute intervals with lean confirmation. Leave the printer fleet at fifteen. Nobody's quarter was saved by detecting a printer jam ninety seconds sooner. And the printer was lying about its status anyway. Interval choices ripple into device load, data volume, and alert noise, so do the tightening with the full math in hand. And the interval guide works the trade-offs properly. For MTTD purposes the summary is: know your intervals, because they are a term in every detection time you will ever measure.
Lever 4: routing — detected by the system, unknown to humans
Detected-by-system is not known-by-human. The gap between them is pure, preventable blind time, because for all practical purposes an alert nobody has seen is a detection that has not happened yet. The monitoring system does everything right and the last hop fails. In the log this shows as a healthy alert timestamp followed by a long crawl to responded. That is why it pays to record both. It is also why teams that define detection as "when a human knew" find this lag sitting inside their MTTD itself.
The fixes are plumbing, and they are quick wins. Critical alerts go to a channel that interrupts (push notification, SMS, a phone that rings), not one that waits to be checked. Warning-tier alerts can keep the inbox. Test the interrupting channel monthly, because an untested path fails on the night it is finally needed. And when the human does not respond, the system should escalate rather than wait. An unacknowledged critical re-notifies or escalates to the next person after a set timer. Channel choices, delivery reliability, and their failure modes are the notification channel guides' whole subject. From the MTTD side, the requirement is one sentence: the alert's journey ends at a human who is now awake, and the clock runs until it does.
At Harbourview Clinic the backup server's disk filled on a Friday evening, and the monitoring system noticed within five minutes and sent an email. The email went to backups@example.com, a shared mailbox created by an administrator who had since left, and nobody had opened it since. The disk was found on Monday morning by the person restoring a file, two days after the alert. The incident went into the log as detected by accident. The alert had fired on time, into a mailbox that had been retired without anyone telling the monitoring system. Critical alerts at Harbourview now go to a phone, and the phone is tested on the first of every month.
Lever 5: event channels — hearing between the polls
Polling asks devices "are you okay?" on a schedule. Event channels are the reverse: the device speaks up the moment something happens. The classic is the SNMP trap — a small unsolicited message a device sends when a state changes. For example, a port went down, a power supply failed, or a temperature crossed its line. Syslog, the standard stream of log messages network gear can send to a collector, plays the same role in prose form. The message is sent at the moment of the event. Because of that, detection time for that class of failure drops from "half the polling interval, on average" to seconds. That is the fastest detection you will ever get. It arrives without tightening a single interval or adding any polling load.
There are honest caveats. Traps require configuring each device to send them and something to receive and alert on them. They travel unacknowledged, so a lost trap is simply lost. That is why traps complement polling rather than replace it. The poll remains the safety net that eventually notices anything the trap missed. And an unfiltered syslog stream will happily bury you in noise. Start with the events that matter most — link state, hardware failure, failover — on your most critical devices. Setup lives in the SNMP guides. As an MTTD lever, event channels give you the fastest detection you will get for the failures they cover. They cost no polling at all.
The outer perimeter: synthetic user checks
One lever catches the failures all the others miss: the synthetic user check. This is a check that does what a user does, end to end, rather than asking components how they feel. Log in to the web app and load a page. Send a mail through the system and confirm it arrives. Open a file from the share. Resolve a name, fetch a URL, complete the transaction. Component checks can all be green while the combination is broken. Examples are a wrong DNS record, an expired certificate, or a dependency between two healthy-looking services. That combination failure is precisely the kind users report and dashboards deny.
Synthetic checks are your detection perimeter's outermost ring: they detect what users feel, minutes before users call. They cost more thought than a ping. You must script the transaction and keep it maintained. So deploy them sparingly: one for each of the three or four services whose failure generates the fastest phone calls. If the detection post-mortem keeps saying "users noticed a broken experience while every component check was green," this is the lever it is asking for.
The MTTD campaign: one focused quarter
The levers work best run as a campaign with a start date and an end date. The format is four steps and one quarter long — long enough to gather real incidents, short enough to hold focus.
- 1. Baseline. From the incident log, compute the current numbers: median detection time and the detection-source ratio. Say a baseline quarter shows eleven incidents, five caught by monitoring (45%), with median detection 24 minutes. It also shows one never-directly horror: a backup job dead for nine days.
- 2. Post-mortem every detection. For one quarter, ask the how-did-we-learn question religiously and keep the answers. The pattern emerges fast. In the example, the six missed incidents break down as four user reports, one accident, and one silent backup. Those user reports involved three missing checks — share access, mail queue, VPN session capacity — and one threshold that waited for catastrophe.
- 3. Work the top gap class. Resist fixing everything; fix the biggest class properly. Here that is coverage: add the three named checks. Add a backup-success check where silence itself alerts. Add the warning tier to the catastrophic threshold. A few evenings of configuration, total.
- 4. Re-measure. Next quarter: nine incidents, eight caught by monitoring (89%), median detection 7 minutes, worst 25. It is not perfection: one user report remains, naming next quarter's check. But the blind hours are gone. The before-and-after pair is now a story that justifies the time spent, in exactly the shape management reporting needs.
Tip: improving detection has a side effect nobody warns you about: for a while, your measured incident count goes up. You are not breaking more; you are finally seeing the failures that used to pass silently. Expect it, and say it out loud before the numbers do. A rising count with a falling detection time is the signature of monitoring getting better, not the network getting worse.
The playbook card
| Lever | Typical cost | What it shrinks | Reach for it when |
|---|---|---|---|
| Coverage | Minutes per check | The unbounded blind times — hours to days | Incidents arrive via users or luck |
| Sensitivity | Threshold tuning time | The long slide before a late alert | Alerts fire only at the cliff edge |
| Intervals | Config time + polling load | The built-in floor (interval ÷ 2 + confirmation) | Tier-1 checks poll slower than minutes matter |
| Routing | Channel setup + monthly test | The alert-to-awake-human lag | Alerts sit unseen in mailboxes |
| Event channels | Per-device config + a receiver | Poll-gap blindness — down to seconds | State-change failures on critical gear |
| Synthetic checks | Scripting + upkeep | The green-dashboard, broken-experience gap | Users feel what components deny |
And the habit that powers all of it, as a card to pin beside the incident log:
DETECTION POST-MORTEM — five minutes, every incident 1. How did we first learn? alert / user / accident / never 2. Fault time vs alert time: how long were we blind, and why? 3. If an alert: could it have fired earlier? (threshold too high? interval too slow? confirmation too fat?) 4. If not an alert: what check would have caught it? Add it now, or write one line on why not. 5. When did a human actually see it? If that lag is the story, fix the channel, not the check.
Where to go next
Detection is half the incident clock, and for most small networks it is the half with the cheaper wins. A quarter of gap-driven checks does more for user-experienced downtime than any amount of speed after the alert. That is because it removes the minutes nobody was working on. Once the alerts arrive fast, the other clock starts to dominate: how long to fix once you know. That is the MTTR playbook's territory: runbooks, access, spares, and practice. Keep feeding the log that told you which lever to pull; the rest of Measuring Detection & Response turns its numbers into steering. The file server has a share-access check now, and I drink my coffee with a clear conscience.
Put this into practice with SysPeep Monitor
Most of this playbook is configuration, and configuration goes faster with the right tool. SysPeep Monitor auto-discovers the devices and interfaces you forgot to watch. It runs checks over SNMP, WMI and SSH at the intervals you choose. It supports warning and critical thresholds per check and pushes alerts through channels that actually interrupt someone. The free trial installs in minutes.
Download Free TrialLearn About SysPeep