Why Intermittent Problems Are So Hard (and How Monitoring Helps)
The ticket says the network went down for about a minute yesterday afternoon and is fine now. You check the graphs: green. You check the logs: nothing. You walk over, test everything the user touches, and all of it works. You close the ticket as "could not reproduce", with the small guilty feeling that phrase always brings. Three days later the same user is back, politely furious, because it happened again.
That is an intermittent fault, and it is the special torment of this job. A hard failure is at least honest. The switch is dead, the disk is full, the cable is cut, and the fixing can begin. An intermittent fault is real enough to hurt people and brief enough to be gone before anyone with a laptop bag arrives. You cannot beat it by direct assault, because it does not happen while you are looking. It waits until you leave.
This guide explains why these faults are so good at dodging you. It covers the observer effect, the blind spots between monitoring polls, and the bias built into user reports. It also explains why "it's just random" is almost never true. Most of all, it walks you through the shift that wins. You stop trying to catch the fault in the act, and you build instruments that catch it for you. The other guides in this topic build on that foundation, one trap at a time.
In this guide
- What makes a problem "intermittent"
- The observer effect: why it hides when you watch
- Polling gaps: how short events slip between the frames
- Survivor bias: why user reports mislead you
- Intermittent does not mean random
- The mindset shift: from chasing to trapping
- Your first week with a ghost
- Where to go next
What makes a problem "intermittent"
An intermittent problem is any fault that appears and disappears on its own schedule instead of yours. Between episodes, the system passes every test you can throw at it. During an episode — which you usually hear about afterward — something real and painful happens. That combination is the defining trait: real when it strikes, invisible when examined.
You will recognize the family resemblance in complaints like these:
- The Wi-Fi in one conference room drops everybody "sometimes, usually in the mornings, but not always."
- An application freezes for thirty seconds a few times a week, then behaves for days.
- Voice calls turn robotic for a couple of minutes at a time, on no schedule anyone can name.
- A switch port logs a burst of errors in the small hours, then runs clean all day.
- The overnight batch job fails one night in ten and succeeds on every manual retry.
These faults are expensive out of all proportion to their duration. They burn hours in useless "come look at it now" sprints. They corrode trust in IT faster than a full outage does. An outage at least gets acknowledged and fixed. A ghost makes it look like nothing is being done. (Something is being done. It is just not working yet.) Worst of all, they corrode users' willingness to report. After a second "could not reproduce," many people quietly give up and live with it. That robs you of your best source of evidence, a problem we will come back to.
The symptom is what users experience — the frozen screen, the dropped call. The fault is the underlying misbehavior causing it, and the trigger is whatever condition sets the fault off. Intermittent problems are hard precisely because you only ever meet the symptom, secondhand, after the fact; the fault and its trigger stay offstage. Everything in this topic is about dragging those two into the light.
None of this means the fault is unknowable. It means the standard walk-up-and-test method is the wrong tool for this class of problem. To see why, start with the ghost's most famous defensive trick.
The observer effect: why it hides when you watch
In physics, the observer effect is the principle that measuring a system disturbs the system. In troubleshooting it takes a more irritating form: the fault seems to know when you are watching. Every admin jokes about it, and the joke is worth taking seriously, because none of it is magic. The fault does not know you have arrived. It only behaves, every time, as if it did, and there are three concrete reasons why.
First, your visit changes the conditions. The fault lives inside a specific situation: a particular load level, a particular mix of traffic, a particular time. You, meanwhile, test when it suits you — mid-afternoon, network quiet, one laptop in an empty conference room. The fault needed the morning rush, forty devices waking up at once, and the backup still running late. Your test may not even follow the same path through the network that the user's traffic takes. You are examining the stage after the play has ended.
Second, your examination resets the system. To test "properly," you reseat the patch cable and reboot the access point. In doing so you have re-made a marginal connection. You have also emptied whatever slowly-leaking memory or slowly-filling table was the real cause. State that took days to accumulate is gone, so the fault will now take days to return. You have learned nothing, and worse, you have manufactured a false "fixed it," which makes the recurrence feel like a brand-new mystery.
Third, your attention is a tiny sample. Suppose the fault is active two minutes per day and you watch for twenty minutes. Most days, those windows simply do not overlap. It is the rattle in your car that never rattles at the mechanic: the ten-minute test drive samples the wrong road, the wrong speed, the wrong temperature. The rattle is real. The sample is wrong. I once watched a switch port for most of a week because it logged errors "in the evenings". I watched it in the afternoons, because that was when I was there. The port, reasonably, waited.
Put together, this means your physical presence is a terrible detection instrument: rare, brief, biased toward business hours, and disruptive to the thing it examines. The winning move is to replace presence with instrumentation — recording instruments that watch continuously, identically, and without touching anything. Keep that word in mind; the entire strategy hangs on it.
Polling gaps: how short events slip between the frames
"But I already have monitoring," you may object, "and it saw nothing." The uncomfortable part is that standard monitoring has blind spots exactly where intermittent faults live. Most monitoring works by polling. The monitoring server asks each device a question on a fixed schedule ("are you up?", "what do your counters say?"). It records the answer. Every one to five minutes is a typical rhythm. Between questions, the system is not looking at all.
Think of a security camera that snaps one photo every five minutes instead of recording video. A burglar who is in and out in ninety seconds has excellent odds of never appearing in a single frame. The arithmetic is plain: a thirty-second outage occupies one-tenth of a five-minute polling cycle. So most cycles, the poll lands on a healthy moment and writes down "fine." And even when a poll does land inside the outage, many alert rules deliberately require two or three failed polls in a row before alerting. That is a sensible defense against false alarms. It also guarantees a short, real outage never fires anything. That is not a defect; it is a tradeoff that is usually correct and occasionally blinding.
Averaging hides events too. Many metrics are collected as an average over the whole interval. A ten-second burst that pins a link at full capacity barely nudges a five-minute average; a violent CPU spike disappears into the mean. The graph is telling the truth about the average and staying silent about the moment.
The full anatomy of these blind spots — and what fills them — is covered in what your polling interval cannot see. For this hunt, the takeaway fits in one sentence: "the graphs show nothing" and "nothing happened" are different claims. The graphs show nothing at the moments they looked, which is the most anyone can ask of a graph. Later guides in this topic will shrink the gaps. They cover faster polling on suspects, event-driven evidence like device logs, and retention that preserves peaks instead of averaging them away. Start with long-term trending.
Survivor bias: why user reports mislead you
Survivor bias is the error of studying only the cases that made it through some filter, and forgetting the filter exists. The classic story comes from wartime aircraft engineering. Analysts mapped the bullet holes on returning bombers and proposed armoring the most-hit areas. Then a statistician pointed out that they were only looking at planes that made it home. The areas that looked unhit were the deadly ones; planes hit there never returned to be counted.
User reports of an intermittent fault pass through a stack of filters just like that. A report only reaches you when someone was present, noticed, cared enough to say something, and still believes reporting does any good. Consider what that filters out:
- Events at night and on weekends go almost entirely unreported, because nobody is there to suffer them.
- Brief blips get shrugged off — people retry, it works, they move on and forget by lunch.
- Your most-affected users may have stopped reporting entirely after the earlier "could not reproduce." Their silence is not recovery.
- The pattern of reports maps your users' working hours and temperaments, not the fault's actual behavior.
The consequence is nasty. A fault that fires around the clock — say, roughly every eight hours — gets reported at nine in the morning and occasionally at five. The small-hours occurrence never meets a witness. On paper it now looks like a business-hours problem, and you will spend a week hunting a load-related cause that does not exist. The reported pattern is the shadow of a shadow.
Harbourview Clinic had a booking system that froze for about a minute "in the late morning, when reception is busy". Every report in the ticket said so, because every report came from reception. The admin spent a fortnight looking for a load problem that was not there. A one-minute check on the database link, added almost as an afterthought, showed the freeze at 03:12, 11:12, and 19:12 every day. It was a site-to-site tunnel rekeying on an eight-hour timer and dropping the connection each time. The evening strike had a witness all along, the on-call nurse, who had stopped reporting after the second "could not reproduce". Both closures, strictly speaking, had been accurate.
This does not make reports worthless — far from it. Each one is a sighting with a timestamp and an eyewitness description, and building timelines shows how to put them to work. Just never mistake the collection of sightings for a map of the fault. For the map, you need a witness with no filters: an instrument that samples three in the morning exactly as faithfully as three in the afternoon.
Intermittent does not mean random
The word "random" does a lot of dishonest work in troubleshooting. Computers are relentlessly deterministic machines: given the same state and the same inputs, they do the same thing. When a fault looks random, that is a statement about your visibility, not about the machine. Somewhere there is a trigger — a condition, or a coincidence of conditions, present every single time the fault fires.
This is genuinely hopeful news, because triggers leave fingerprints. They show up as patterns in time, in load, and in what else was happening at that moment. Most intermittent faults belong to a small set of trigger families, and the shape of the recurrence points at the family:
| Pattern clue | Usual trigger family | Typical example |
|---|---|---|
| Same clock time each occurrence | Scheduled tasks | Backup, antivirus scan, or sync job saturating a link or a disk |
| Certain weekdays or month-ends | Calendar cycles | Weekly full backup, month-end batch run, patch-night reboots |
| Only under heavy use | Capacity and queue limits | Login storm overrunning a link, buffer exhaustion, duplex mismatch showing itself under load |
| Follows temperature or weather | Thermal and physical layer | Server-closet air conditioning set back on weekends, a failing fan, a marginal optic that drops when warm |
| After long uptime, then clean after reboot | Slow accumulation | Memory leak, table filling to its limit, counter wrap, lease or rekey cycle |
| After someone was physically nearby | Physical disturbance | A bumped patch cable, rack vibration, a device on a timer outlet |
Even genuinely failing hardware fails with structure — heat, vibration, and load are its triggers. And the most maddening cases, the ones that pass every pattern test, are usually two innocent cycles aligning. Consider a lease renewal that only hurts when it lands in the same minute as a tunnel rekey. It will strike on a rhythm set by the arithmetic of the two timers. That is regular in principle, random-looking in practice, like two cars' turn signals drifting into sync and out again. I have never met a random fault. I have met several with two clocks.
Spotting a trigger's fingerprint requires knowing what normal looks like, which is exactly what a baseline gives you. And if your particular ghost is dropped packets, there is a symptom-specific companion in tracking down intermittent packet loss.
The mindset shift: from chasing to trapping
Chasing is trying to be present when the fault happens. That means camping at the user's desk, staring at a live graph, asking to be called "the moment it happens again." Everything above explains why chasing loses — you cannot be everywhere, your presence changes the system, and your samples are tiny and biased.
Trapping is the opposite: arranging for evidence to be captured automatically, so the fault documents itself no matter when it strikes. Nobody photographs a snow leopard by wandering the mountains hopefully. They strap camera traps to the trees along the trails and let the animal take its own portrait. Your network can do the same, because unlike the leopard, it is entirely made of things that can be told to keep records.
The rest of this topic is a tour of the four traps, and it helps to see up front how they fit together:
- Long-term trending runs graphs for weeks and months so time-of-day, day-of-week, and calendar patterns become visible to the eye. No human shift could ever notice those patterns live.
- Timelines merge user sightings, alerts, device logs, and change records into one ordered story, so you can see what always precedes the fault.
- Tripwires are fast, targeted watchdogs aimed at the exact symptom, armed around the clock. They record and react at three in the morning so you do not have to be awake.
- Reproduction is the endgame: once the traps suggest a trigger, you force it deliberately, on your schedule, and prove the diagnosis and the fix.
The shift also changes what "success" means, and this matters for morale. While chasing, every recurrence is a defeat. While trapping, every recurrence is a gift: one more data point, one more log entry, one tighter turn of the net. You will find yourself almost looking forward to the next strike, because this time the cameras are rolling. And a fault that stops before you have found its cause has not died; the ghost is on vacation.
Warning: beware the fault that "goes away on its own." Untriggered disappearance usually means the trigger's schedule moved — the backup finishes earlier, the office is quieter — not that the fault died. If you did not find a cause, assume the fault is dormant rather than gone, and leave your traps armed.
It changes what you tell users, too. "Could not reproduce" reads as "did not believe you." Compare: "I believe you, I cannot force it to happen yet, and I have set four traps — every time it recurs, we will learn more. Please keep telling me the moment you see it." That is honest, it is accurate, and it keeps your witnesses reporting instead of giving up.
Your first week with a ghost
Here is a concrete opening sequence for any intermittent fault, before you know anything about its cause:
1. Start a ghost log. One shared document. Every sighting gets a row. Record the date, time to the minute (seconds if you can get them), who saw it, and where. Include what it looked like and how long it lasted. Interview reporters the same day while memory is fresh, and always ask what they did next — a retry or reboot is part of the record.
2. Stop the reflex fixes. No rebooting, reseating, or swapping "just to see." Every unrecorded intervention destroys accumulated state — which is to say, evidence — and may suppress the fault for days while teaching you nothing. Interventions come later, deliberately, one at a time, written down.
3. Cover the blast radius with basic monitoring. The blast radius is every device and server anywhere near the symptom. Put up/down checks, interface error counters, CPU, memory, and disk on all of them. If monitoring already exists, verify the suspects are actually covered — ghosts love the one unmonitored switch. This also starts building the baseline of normal you will compare everything against.
4. Tighten the sampling around the suspects. Shorten polling to one minute or faster on the suspect scope only, so short events have somewhere to land. Watch that the extra polling does not burden small devices — it rarely does at this scale, but check.
5. Arm one simple tripwire at the symptom. Use the simplest scripted test that measures what the user actually feels — a repeated fetch, a scripted login, a timed file open. Run it every ten to thirty seconds, logging every result with a timestamp, success and failure alike.
6. Hold a weekly correlation review. Once a week, lay the ghost log against the graphs and device logs. Look for what the sightings have in common — and especially for what always comes just before them.
Tip: write every time down with seconds and the time zone, and note whose clock it came from. Half of all correlation pain is clocks that disagree; the timeline guide shows how to sort that out before it bites.
Where to go next
You now know the enemy. It is a deterministic fault hiding behind the observer effect, polling gaps, and biased reports. And the counter is instrumentation that never sleeps, never reboots anything to see what happens, and does not care whether you are at your desk. The fault will still wait until you leave. From now on, that is its problem. The natural next step is the patient workhorse of the whole hunt: long-term trending, which turns weeks of quiet measurement into visible patterns. The full set of guides lives in Catching Intermittent Problems.
Put this into practice with SysPeep Monitor
An intermittent fault never performs while you are watching — so the answer is a witness that never blinks. SysPeep Monitor discovers your devices automatically. It polls health and performance over SNMP, WMI and SSH around the clock. And it keeps the graphs, dashboards and alert history that let you read last night's ghost story over morning coffee. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep