Home › Topics › Measuring Detection & Response › Honest Measurement

Measuring Response Times Without Lying to Yourself


Incident metrics are easy to game, and most of the gaming is accidental. The ticket gets opened an hour into the firefight. The incident gets closed at "I rebooted it" rather than "it stayed up." The ugly outage quietly becomes "not really an incident." Nobody decides to cheat. The numbers drift toward flattery anyway, because at every small step a flattering number meets less resistance than an honest one. Sometimes it is deliberate: a team told to "get MTTR down" finds that redefining when the clock stops is quicker than fixing things faster.


Gamed metrics are worse than no metrics. No metrics leave you ignorant and appropriately humble. Gamed metrics leave you confident and wrong, celebrating a detection time that is really a bookkeeping artifact. Meanwhile, users learn that things break more often than the dashboard admits. If you have read the incident clock guide, you know what MTTD (mean time to detect) and MTTR (mean time to repair) are supposed to measure. This guide is about making sure yours do.


Honesty here is an engineering problem, and a small one. It comes down to knowing where trustworthy timestamps come from and recognizing the five classic distortions by their tells. It also means running a measurement system light enough that a busy team keeps it up. I have kept both kinds of log, and the honest one costs about fifteen minutes per incident.


In this guide

  • Where the data comes from: sources ranked by trust
  • The gaming catalog: four clock distortions
  • The aggregation gloss: statistics that hide disasters
  • The tells-and-fixes summary table
  • Measurement-lite: the six-column incident log
  • The culture line: metrics never judge people
  • The honest-measurement checklist

Where the data comes from: sources ranked by trust


Every timestamp in your incident log came from somewhere, and the somewheres are not equally trustworthy. Ranking them explicitly is the first honesty tool, because it tells you which numbers to believe when sources disagree — and they will disagree.


1. Alert timestamps: automatic and honest

The best source you have. When a monitoring check fails and an alert fires, the system records the moment without opinion, embarrassment, or hindsight. It was not busy, it was not flustered, and it does not round to the nearest comfortable quarter hour. Alert timestamps are the gold standard for the DETECTED clock, and often help bound FAULT too. The last successful check gives you "it was still fine at 06:50." A simple rule follows: wherever an alert timestamp exists, it wins over anything a human wrote down later.


2. Ticket timestamps: human-entered and drifty

Ticket systems look authoritative. Every ticket has official-looking created and closed times. But those record when someone did paperwork, not when things happened. The classic failure: the outage starts at 7:00, everyone dives in, and the ticket gets opened at 8:10 by whoever surfaces first. Your "detection" timestamp is now seventy minutes late, and it flatters MTTD beautifully. Closing drifts the same way, in the same flattering direction, when tickets get closed during cleanup the next morning — or punitively inflates MTTR instead. Two hygiene habits tame this. First is an open-the-ticket-first culture: thirty seconds before diving in. This also timestamps RESPONDED for free. Second is a reconciliation pass afterward, correcting ticket times against alert times while writing the incident log entry.


3. Chat and log archaeology: the forensic fallback

When nothing was recorded deliberately, you reconstruct. Use the "is the file server down for anyone else?" message in the team chat. Or use the first failed transaction in an application log or the event log's last healthy entry. This is real evidence and often surprisingly precise — chat messages are timestamped whether or not anyone was thinking about metrics. It is also exactly the timeline-reconstruction skill that post-incident reviews run on; the post-incident review guides make a discipline of it. The cost is time: archaeology takes twenty minutes where a glance at the alert history takes one. Use it to fill gaps, not as the primary system.


4. Memory: the worst source, used anyway

"It was maybe half an hour?" Human memory of stressful events is confidently wrong. It compresses boring waits, stretches frantic stretches, and — most dangerously for metrics — edits toward the story we prefer. You will still use it, because sometimes there is nothing else. The bound that keeps it survivable: write timelines down within a day, while memory still resembles events, and mark memory-derived timestamps as estimates. A week later you are not remembering the incident; you are remembering your retelling of it.


When sources disagree, the rank order settles it. Suppose the alert says 07:41, the ticket says 08:10, and the responder remembers "around half seven." The higher source on this list wins. The disagreement itself goes in the notes column. Those notes are quietly valuable. A log where tickets routinely lag alerts by an hour has just measured its own paperwork culture. That is a fixable finding in its own right.


The gaming catalog: four clock distortions


Each of the four distortions below can be committed innocently or cynically. The mechanics are identical; only the intent differs. Each has a tell you can spot in your own data and a fix that closes the loophole. Read these as a self-audit, not an accusation. Most teams find at least two happening already, by accident.


The late-started clock

The move: detection gets recorded at ticket-open rather than at the alert, so the blind time before paperwork silently vanishes and MTTD deflates. The tell: your detection times look suspiciously tidy — lots of zero-and-five-minute values. Or MTTD is implausibly short for a network you know has coverage gaps. Cross-check a few incidents: if the alert fired at 07:41 and the log says detected 08:10, you have found it. The fix: the rule from the sources section, made policy: the alert timestamp wins, always. The log's detected column cites its source (alert / user report / found-by-accident) so a reviewer can spot substitutions.


The early-stopped clock

The move: the incident ends at "fix applied" instead of "verified working." Reboot the switch and close the ticket. MTTR looks great. And when the switch wedges again forty minutes later, that is logged as a fresh, separate, also-short incident. Two flattering numbers out of one continuing outage. The tell: the reopen pattern. In that pattern, incidents on the same system cluster within hours of each other. Or users say "it's down again" about things your log says were resolved. The fix comes in two parts. The first is a written verified-stop definition: checks green, real test passed, quiet period observed. The incident clock guide gives one. The second is reopen tracking: count reopens as their own metric. A rising reopen count is an early-stopped clock confessing.


The severity shuffle

The move: your measurement policy sensibly sets a severity floor — only sev 1 and sev 2 incidents get timed. The shuffle reclassifies painful incidents to sev 3 so they fall below the floor and out of the statistics. Sometimes this is deliberate laundering; just as often it is optimistic scoring in the moment ("only some users were affected, right?"). The tell: the incident count drifts down while lived pain does not. There are fewer measured incidents, the same number of awful weeks, and users grumbling about outages the metrics say never happened. A shrinking numerator with a constant misery level is the smell. The fix: write severity criteria in advance, in observable terms (how many users, which services, work stoppage or not). Also run a quarterly audit: pull five closed sev 3 tickets and check they really were minor. Severity schemes worth borrowing live in the incident response guides.


The exclusion creep

The move: "that one doesn't count because..." — the ISP caused it, it was during a change window, it was DNS, it was a one-off. Each exclusion sounds reasonable alone. (The one-off has usually happened before.) Compounded, they curate a highlight reel: the metrics describe only the incidents you were good at. The tell: the excluded pile grows quarter over quarter, and the excluded incidents are consistently the long ones. The fix: an exceptions ledger. Excluding an incident from the headline numbers is allowed — sometimes genuinely fair — but every exclusion gets a ledger line: date, duration, reason excluded. The ledger is reported alongside the metrics. Exclusions that must be written down and shown stay rare and defensible; exclusions that vanish silently multiply.


The aggregation gloss: statistics that hide disasters


The fifth distortion deserves its own section, because it happens after every timestamp was recorded honestly. You can collect immaculate data and still lie with the summary line — usually by letting a mean do the talking on skewed data.


Northgate Retail had a wall tile labelled "Avg MTTR" that read 41 minutes for most of a year. The tile was never wrong. Behind it sat thirty incidents of around twenty minutes and one Saturday on which the point-of-sale system was down for nine hours. The tile had absorbed that without changing colour. The service desk knew about the Saturday. The tile did not mention it. The monthly report copied the tile. The nine hours were finally discussed when a store manager asked why nobody had. The tile now shows the median and the longest incident side by side. The review described the old tile as "accurate", which it was.


Work one example, because the arithmetic is the argument. A quarter brings ten incidents with response times, in minutes, of 8, 9, 11, 12, 14, 15, 18, 20, 22 — and one afternoon-eater at 360. The mean (sum divided by count) is 489 ÷ 10 ≈ 49 minutes. The median (sort the list, take the middle; here, halfway between 14 and 15) is about 14 minutes. Now watch what each number does. The mean, 49 minutes, describes not a single incident you actually had. It makes your nine quick fixes look three times slower than they were, and hides a six-hour disaster entirely. The median says truthfully: a typical incident here is fixed in about a quarter of an hour. Neither mentions the disaster, so you report it explicitly: median 14 minutes, worst case 6 hours. Two numbers, whole truth.


When your incident count grows, a percentile earns its place as the "bad day" number. The p90 is the value nine-tenths of your incidents come in under. So "p90 response of 25 minutes" means even your bad days mostly wrapped inside half an hour. In the example above, nine of ten incidents finished within 22 minutes. That is the p90 flavor of honesty. It is far more informative than the mean about what a rough week feels like.


Your incident count should be small for a healthy small network. When it is small, the most counterintuitive honesty rule of all applies: report the list, not just the statistics. With six incidents in a quarter, "N" is small. That is the count your averages are built on. It is so small that one weird incident moves every summary number. A median of six values is barely a statistic at all. The honest artifact at that scale is the incident table itself — six rows, one per incident, with severity, duration, and cause. Anyone can read six rows in thirty seconds. No summary can distort them. And a leadership audience trusts them more than any average. Statistics are a compression format; below about a dozen incidents, you do not need compression.


The tells-and-fixes summary table


The distortionThe tellThe fix
Late-started clockDetection times tidy or implausibly short; alert times disagree with the logAlert timestamp always wins; log the detection source per incident
Early-stopped clockReopens: same system failing again within hours of "resolved"Verified-stop definition in writing; track reopens as a metric
Severity shuffleIncident count falls while pain does notWritten severity criteria; quarterly audit of downgraded tickets
Exclusion creepGrowing "doesn't count" pile, mostly the long onesExceptions ledger, reported alongside the metrics
Aggregation glossOne mean summarizing a quarter; disasters invisibleMedian + worst case; the full list when N is small

Measurement-lite: the six-column incident log


None of this needs a heavyweight process. The honest system for a small shop is one table: a spreadsheet, a wiki page, whatever your team will actually keep. It has six columns and one row per incident above the severity floor. Anything with more columns is a form, and forms go unfilled.


INCIDENT LOG — one row per sev 1/2 incident

DATE     | WHAT HAPPENED        | SEV | DETECTED BY | TIMESTAMPS              | NOTES
---------+----------------------+-----+-------------+-------------------------+---------------
(date)   | File shares frozen,  |  2  | alert       | F 06:52e D 07:41        | ping stayed
         | RAID controller hang |     | (share chk) | R 07:49 X 08:51 V 09:07 | green; F est.
(date)   | Mail queue stalled,  |  2  | user report | F ~13:30e D 14:55       | no queue
         | disk full            |     |             | R 15:02 X 15:20 V 15:41 | check existed

F=fault  D=detected  R=responded  X=fixed  V=verified  e=estimated

The columns earn their places. The date and one-line description make the log readable as history. Severity enforces the measurement floor. And detected-by is the column that powers the entire MTTD playbook. The ratio of alert-detected to user-reported incidents is the single best health metric your monitoring has. The timestamps are the raw material for every interval. And notes hold the honesty flags — estimates marked, oddities admitted. If you keep an uptime evidence file, you will recognize the species: same one-table ledger, same write-it-while-fresh habit. The two share rows: every sev 1 in this log is an outage in that one.


The operating budget is deliberately small. Per incident: fifteen minutes, within a day. Pull the alert times, reconcile the ticket, write the row. Per quarter: one roll-up ritual — an hour with coffee. In that hour, compute the medians and worst cases and read the whole list. Check the tells from the catalog above (any reopens? severity drift? a fat exclusions ledger?). And write three sentences about what the quarter's incidents had in common. That hour is where measurement turns into improvement, because patterns visible across ten rows are invisible inside any single firefight. The hour's output slots straight into the quarterly report if anyone above you wants numbers.


Tip: log the boring incidents too — the ones fixed in eight minutes where everything worked. A log containing only disasters reads like a horror novel and skews every number pessimistic. The quick saves are real data: they prove your median, and they are the evidence that monitoring and runbooks are paying for themselves.


The culture line: metrics never judge people


One rule sits underneath everything in this guide. If you skip it, none of the fixes above will save you: incident metrics exist to improve systems, never to grade individuals. Suppose MTTR appears in someone's performance review. A specific person gets asked why their incidents run long. At that moment, every distortion in the catalog stops being an accident and becomes a survival strategy. People are resourceful; threatened with a number, they will fix the number. Clocks start late, tickets close early, severities shuffle, and your measurement system converts, quietly and permanently, into a fiction-generation system. This is not a flaw in your colleagues; it is what measurement under threat does to every profession that has tried it.


So state the rule out loud and keep it. Numbers are computed at team level, discussed at team level, and used to justify investments — never to rank responders. The moment of maximum temptation is a bad quarter with an identifiable name attached. Hold the line precisely there: the question is never "why was Sam slow?" but "what would have made that incident fast for whoever caught it?" One question produces defensiveness and creative bookkeeping; the other produces runbooks. And say it to management too, before the spreadsheet reaches them without the rule attached. The reporting guide covers how to share these numbers upward without turning them into weapons.


The honest-measurement checklist


Run this against your own practice once a quarter — it takes five minutes and catches drift while it is still small:


  • Definitions written: severity floor, clock start, clock stop — on one page, agreed before measuring.
  • Alert timestamps win over ticket timestamps everywhere both exist.
  • Every log row records how the incident was detected (alert / user / accident).
  • Incidents stop at verified, and reopens are counted where anyone can see them.
  • Severity assigned by written criteria; a handful of downgrades audited each quarter.
  • Exclusions live in a ledger that is reported, not in silence.
  • Summaries lead with median and worst case; the full list is reported whenever N is small.
  • Timelines written within 24 hours; estimates marked as estimates.
  • Numbers discussed at team level only — never attached to a person.

Where to go next


Honest numbers are the license for everything else in this pillar. Once the log stops flattering you, it starts steering you, and the steering is the point. If detection times are the ugly column, the MTTD playbook turns your detected-by ratio into a repair plan for your monitoring. If response times are, the MTTR playbook does the same for runbooks, access, and spares. When the trend finally bends the right way, Reporting Incident Metrics to Management shows how to spend that credibility. The rest of Measuring Detection & Response is the map. Fifteen minutes an incident is the cheapest honesty you will ever buy.


Put this into practice with SysPeep Monitor

The most trustworthy timestamps in your incident log are the ones no human wrote down. SysPeep Monitor records exactly when every check failed, every alert fired, and every recovery was seen. That gives you an automatic, honest DETECTED clock, plus the alert history to reconcile your tickets against. The free trial runs on any Windows machine.

Download Free TrialLearn About SysPeep