Guess, Test, Learn: Hypothesis-Driven Diagnosis
Every troubleshooter guesses. The difference between the admin who finds the fault in twenty minutes and the one still rebooting things at midnight is not that the first one guesses less. It is that they guess out loud, in ranked order, with a plan to prove each guess wrong. The slow version keeps its guesses secret, even from itself. A vague feeling that "it might be the switch" turns into an hour of fiddling with the switch. There is no clear result, and no memory of why the switch was suspicious in the first place.
There is a name for the fast version: the scientific method, wearing work boots. You form a small set of explicit hypotheses from the symptoms and rank them. Design the cheapest test that could kill each one. Run the tests, and let the survivors guide you. It sounds like ceremony. In practice it is faster than winging it, because every test either convicts a suspect or permanently crosses one off — nothing is wasted.
This guide teaches the full loop. Turn symptoms into a shortlist, rank by likelihood and test cost, and design killer tests. Dodge the mental traps (confirmation bias has ended more troubleshooting sessions than hardware failure) and keep a diagnosis log. And — because we live in the real world — learn when it is acceptable to ship a fix without ever confirming the cause.
In this guide
- What counts as a hypothesis
- From symptoms to a shortlist
- Ranking: likely first, cheap first
- Designing the killer test
- The traps: bias and its friends
- The random-reboot anti-pattern
- Keep a diagnosis log
- Fixing without a confirmed cause
- The whole loop, in one small story
- Where to go next
What counts as a hypothesis
A hypothesis is a guess precise enough to be proven wrong. That last part is the whole game. The formal word is falsifiable, meaning there exists some test result that would force you to abandon the idea. "The network is flaky" is not a hypothesis; no possible measurement kills it, so it can absorb an infinite amount of your afternoon. "The uplink between the access switch and the core is dropping frames under load" is a hypothesis. It predicts things you can check, like error counters on that specific port climbing while the problem happens. It also predicts problems clustering among users behind that uplink.
A useful hypothesis has three parts. A cause says what is broken. A mechanism explains how that brokenness produces what users see. And a prediction says what else must be true if this is right. The mechanism is the part beginners skip and veterans lean on. If you cannot explain how your suspect would produce these exact symptoms, you do not have a hypothesis yet — you have a hunch with a badge. Hunches are allowed; they just have to grow up before they get tested.
From symptoms to a shortlist
Resist testing anything for the first few minutes. Collect instead: what exactly fails, for whom, since when, and — the underrated half — what still works. The fastest tool for organizing that is a two-column table you can scribble anywhere:
| Still works | Broken |
|---|---|
| Web browsing, from every desk | Shared drives, for about half the office |
| Email, everywhere | Printing, for the same half |
| Shared drives, for the other half | — |
This little table is a hypothesis-generating machine, because every candidate cause must explain both columns. "The file server is down" fails instantly — half the office is using it right now. "The internet link is congested" fails too — broken services are internal ones. But "the affected half of the office sits behind one access switch whose uplink is sick" explains every row. Internal traffic from those desks suffers, and their web traffic… wait, their web browsing works fine. Good catch — so refine it: perhaps only some traffic paths are affected, or the working services live somewhere the broken ones do not. The point is the table pushed back, and hypotheses that survive its pushback are already better than anything you would have tested on reflex.
Aim for three to five candidates and write them down. Actually write them, because the list is about to become your test plan and, later, your log. If you are struggling to generate candidates at all, walk the layers of the network stack. Ask "what at this layer could produce exactly this?" The ladder from Layer by Layer doubles as a brainstorming checklist.
Ranking: likely first, cheap first
You cannot test everything at once, so rank the shortlist along two independent axes. The first is likelihood. Three things raise it. First, a recent change touching the suspect is the single strongest signal in all of troubleshooting — see What Changed?. Second, this exact suspect has a history of misbehaving. Third, ordinary base rates matter. Cables, name resolution, and full disks break constantly; exotic protocol bugs mostly live in war stories. The second axis is test cost: how long, how disruptive, and how reversible is the test that would settle it?
Then order your tests with a simple bias: cheap tests of likely causes first. The interesting cases are the diagonal ones, and a small table settles them:
| Cheap to test | Expensive to test | |
|---|---|---|
| Likely | Do it now | Prepare it while cheap tests run |
| Unlikely | Do it while the coffee brews | Only when everything else is dead |
The lower-left box surprises people: yes, sometimes you test an unlikely cause early. Ten seconds is ten seconds, and a surprise conviction there saves the whole afternoon. What you must never do is the upper-right first — launching the two-hour packet capture before anyone has pinged the gateway. When no hypothesis clearly leads the ranking, that is your cue to stop testing favorites and split the field instead. The technique in Binary Search for Networks exists for exactly that moment.
Designing the killer test
Here is the mental flip that separates this method from ordinary poking: for each hypothesis, do not ask "what would show I am right?" Ask "what is the cheapest observation that would prove me wrong?" A test that can only agree with you is a mirror, not a test. Suppose your hypothesis is "the DNS record for the file server is missing." Looking up the name and getting a correct answer kills it dead. That is wonderful, because a dead hypothesis shrinks the search space just as surely as a confirmed one.
Before running any test, say the prediction out loud or write it down: "if I am right, this lookup fails; if I am wrong, it answers in milliseconds." Committing to the prediction before seeing the result is the cheapest anti-bias device ever invented. It removes the wiggle room your brain will otherwise exploit ("well, it answered, but slowly, so I am probably still right…"). A few more rules of thumb for good tests:
- One hypothesis per test, where possible. A test whose failure could implicate three different suspects leaves you nearly as confused as before.
- Read-only first. Looking at counters, logs, and status never makes anything worse; restarting things can. Save state-changing tests for last — the discipline in Troubleshooting Under Pressure covers why.
- Prefer tests that produce a number. "Ping times jumped from 1 to 90 ms" beats "it feels slower" forever.
- Believe the result. If the test was well designed and the result says your favorite is dead, it is dead. Re-run once to rule out a fluke, then move on.
The traps: bias and its friends
The method exists because human brains cut corners, and troubleshooting punishes every one of those corners. Four traps account for most of the damage. Confirmation bias is the tendency to seek, notice, and remember evidence that agrees with what you already believe. It is why an admin convinced the firewall is guilty can stare at ten innocent log lines and quote you the one ambiguous entry. Anchoring is the first idea's unfair grip: whatever cause you thought of first feels more likely for no reason beyond being first. The streetlight effect is searching where the light is good. You test what is easy to test rather than what is informative. It is like the drunk in the old joke looking for his keys under the lamp because the light is better there. And sunk cost keeps you testing a pet theory because you have already spent an hour on it, which is exactly backwards.
The antidotes are mechanical, which is good news — you do not have to be wise, just procedural. Written predictions before every test. The standing question "what would I expect to see right now if I am wrong?" A hard rule that a hypothesis twice contradicted is retired, no appeals. And when you are stuck, explain the problem out loud to a colleague — or to anything, really. The venerable trick of explaining your problem to a rubber duck works. Forcing the story into words exposes the step where the story cheats. The duck has solved more outages than several commercial tools we could name, and its licensing terms are excellent.
Tip: your ego is not on the suspect line. The goal is not for your first guess to be right; it is for the fault to be found in the fewest tests. An admin whose first three hypotheses die in nine minutes is beating the one whose first guess is "confirmed" after ninety.
The random-reboot anti-pattern
Now for the elephant in the server room. Rebooting things at random — the switch, the access point, the server, whatever is nearest — sometimes works, and that is precisely what makes it poisonous. A reboot clears accumulated state: caches, leaked memory, hung sessions, wedged negotiations, stuck queues. A decent fraction of all faults are bad accumulated state somewhere. So rebooting boxes in sequence fixes a lot of problems. It never tells you which state, in which box, was bad. You get your evening back and lose the diagnosis.
The costs arrive later. The fault returns, because its cause was never found — a memory leak refills, a flapping negotiation re-wedges. The evidence is gone, because reboots wipe exactly the counters, tables, and process states that would have convicted the real culprit. And the habit spreads: "we reboot it weekly and it behaves" is how superstitions get institutionalized, one hopeful power cycle at a time.
Used deliberately, though, a reboot is a legitimate test. State it as the hypothesis it is: "something in this device's accumulated state is corrupt; if I clear the state and the symptom stops, that class of cause is implicated." Then capture the evidence first: screenshot the counters, export the logs, note the uptime. The moment of restart destroys it all. The case for evidence-before-restart is made minute by minute in The First Fifteen Minutes. If the reboot works, you have not finished; you have learned where to dig for the leak or the wedge. (Printers are exempt from all of this. Rebooting a printer is not a diagnostic act; it is a religious rite, and skepticism only angers them.)
Keep a diagnosis log
Memory under stress is a sieve, and troubleshooting sessions routinely outlive it. The fix costs one open text file. Every entry gets a timestamp, what you believed, what you tested, what you predicted, what actually happened, and what you decided next. Terse is fine — this is a lab notebook, not literature:
14:02 Symptom confirmed: \\files01 unreachable by name from 2 PCs, fine by IP
14:05 H1: DNS record wrong/missing (likely - server re-addressed recently?)
H2: client DNS settings broken H3: file service down (unlikely - IP works)
14:07 Test H1: nslookup files01 against 10.0.0.5. Predict: wrong/no answer.
14:08 Result: correct address returned. H1 DEAD.
14:11 Test H2: ipconfig /all on affected PC. Predict: odd DNS server listed.
14:12 Result: DNS points at 10.0.0.9 (decommissioned). H2 CONFIRMED.
14:20 Fixed adapter to DHCP; name resolves. Root cause: stale static setting.
Ten lines, and look what they buy. You never re-run a test you already ran. The log remembers that H1 is dead even when, an hour deeper, your tired brain suggests "maybe check DNS again?" A colleague can pick up the case cold, mid-investigation, and know exactly where things stand — which is also what makes clean handoffs and escalations possible. The vendor's support line takes you seriously, because you can read them a timeline instead of a vibe. And after the dust settles, the log becomes raw material twice over. It is the factual backbone of any post-incident review. And if this failure ever recurs, the log is already half of a runbook. The next person can then follow your trail at reading speed.
Where the log lives matters less than that it exists: a ticket, a text file, a paper notebook with coffee rings. The only rules are timestamps on every line and no retroactive editing. A log you tidy afterwards to look smarter is a novel, and novels do not help the next outage.
Fixing without a confirmed cause
Purists hate this section, but you run a network for a living: sometimes service comes back before understanding does, and that is the right call. Users bleeding productivity outrank your curiosity. Getting the restore time down is a discipline of its own, covered from the other side in Improving MTTR. Failing over to the backup link, restarting the wedged service, or rolling back to the last good config can be honorable moves. That holds even with no root cause in hand — if you do them like a professional rather than a gambler. The difference is a short checklist:
- 1. Capture evidence first. Logs, counters, screenshots, a quick packet capture if it is cheap. Thirty seconds of collection keeps the case solvable after the restart wipes the scene.
- 2. Make the move reversible and log it. One line in the diagnosis log: what you did, when, how to undo it.
- 3. Say the honest words. Report "mitigated, cause not yet confirmed" — never "fixed." The distinction changes what everyone does next, including you.
- 4. Schedule the real diagnosis while the trail is warm: review the captured evidence within a day or two, while logs still cover the window.
- 5. Set a tripwire. Point a monitoring check or alert at the symptom so a recurrence announces itself instead of hiding for weeks.
Skip the checklist and the unconfirmed fix quietly rots into ritual. You get the weekly reboot and the "just fail it over when it acts up" folklore. You get the switch nobody dares to patch because nobody knows why it works. Every one of those started life as a reasonable emergency mitigation that never got its follow-up. Mitigate boldly; close the loop stubbornly.
The whole loop, in one small story
Symptom: Wi-Fi in the conference room drops during big meetings — and only then. Laptops work fine in that room on quiet days, and fine everywhere else always. The two-column table practically writes itself: works — that room when empty, every other room always; broken — that room, full. Any surviving hypothesis must explain why fullness matters.
The shortlist, ranked: H1 — the room's access point saturates when thirty people pile onto it. Likely; the mechanism fits fullness perfectly, and it is free to check from the controller. H2 — the guest address pool runs out when a crowd of visitor phones joins. Plausible; meetings are when guests appear, and their phones join by themselves from inside pockets. H3 — interference from the room's wireless conferencing gadget (possible; it is only powered on for meetings). All three predict different, checkable things — and all three tests are read-only.
Test H1 during the next big meeting: the controller shows the AP at a comfortable client count and airtime. Prediction was "near its limits"; result says healthy. H1 dead, and honestly mourned, because it was the favorite. Test H2 in the same minute: the guest scope shows zero addresses free. Prediction confirmed — and the mechanism explains everything, including why staff laptops (on a different pool) mostly survived while anything joining fresh failed. Widen the guest pool, shorten its lease time, and the next full-house meeting passes quietly. H3 never even needed its test.
Notice the shape of the win: the favorite died in one look, and the winner was the second-ranked idea. The whole diagnosis cost two read-only checks during one meeting. It also produced a log entry that will make this a five-minute fix the next time some other pool fills. That is the method working exactly as designed. Being wrong quickly is the skill.
Where to go next
The loop is small enough to memorize. Put symptoms in two columns. Write three to five explicit hypotheses, ranked by likelihood and test cost. Each faces the cheapest test that could kill it. Log everything and write every prediction before its result. Run it on trivial problems until it is reflex, and it will hold your thinking together on the terrible days. It slots neatly into the rest of the method. Use the ladder for generating suspects, halving for when no hypothesis leads, and change-hunting for the likeliest lead of all. The whole toolkit lives together at A General Troubleshooting Methodology.
Put this into practice with SysPeep Monitor
Hypotheses die fastest against real data. Suppose the graphs already show whether latency jumped, a disk filled, or an interface flapped — and when, exactly. Then half your shortlist is settled before you run a single test. SysPeep Monitor collects that history from your network over SNMP, WMI and SSH and keeps it in dashboards ready for exactly these arguments. The trial is free on any Windows machine.
Download Free TrialLearn About SysPeep