Home › Topics › A General Troubleshooting Methodology › What Changed?

What Changed? Correlating Faults with Changes


The most valuable question in troubleshooting is also the most boring one: what changed? Networks that worked yesterday do not stop working today out of spite. Something is different — a setting, a certificate, an address, a cable, a version of something, the phase of some vendor's update cycle. Find the difference and you have usually found the fault, or at least its front door. Skip the question and you can spend a day rediscovering, test by painstaking test, a fact that was sitting in an update log the whole time.


Ask the question out loud, though, and you will hear the universal answer: "nothing changed." It is delivered with complete sincerity by people who updated a driver Tuesday and "just tweaked" a firewall rule Wednesday. Their devices patched themselves overnight without asking anyone. Nobody is lying. "Nothing changed" really means "I did not change anything that I remember and that I count as a change" — and both of those qualifiers hide multitudes.


This guide makes you the person who finds the change anyway. You will learn why networks are never actually static and the catalog of changes that do not feel like changes. You will learn where the evidence hides even when nobody kept records, and how to reconstruct a timeline that makes the culprit jump out. You will also learn the lightweight habits that make next time faster. No enterprise change-management bureaucracy required — just a notepad's worth of discipline.


In this guide

  • Why "nothing changed" is almost never true
  • The hidden-change catalog
  • Time is your search key
  • Where change evidence hides
  • Asking humans so they can actually answer
  • A worked timeline
  • Correlation is not conviction
  • Building the change-tracking habit
  • When the change was not yours
  • Where to go next

Why "nothing changed" is almost never true


Three forces guarantee that something always changed. First, people forget and discount. A change that took ten seconds does not register as a change; neither does one made "temporarily," nor one made to a system that feels unrelated. The colleague who swears nothing changed genuinely believes it. The firewall rule he added was "just for the new scanner," and the scanner works fine, so what could it have to do with the VPN?


Second, machines change themselves. Modern equipment updates its own firmware, patches its own operating system, renews its own certificates, rotates its own keys, and expires its own leases. It reboots itself at scheduled hours to do all of the above. A network in which no human touches anything for a month still experiences dozens of self-inflicted changes. "Untouched" and "unchanged" stopped being synonyms a long time ago.


Third, absence of action is also a change. A certificate nobody renewed expires on schedule — the change was configured at issue time and merely detonated today. A trial license lapses. A disk that filled one percent per week finally reaches full. Even hardware aging is a slow change: capacitors, fans, and flash cells drift until one morning the switch that "just worked for years" does not. The question is never whether something changed. It is only which change matters. Finding it is a search problem, the kind this pillar's other guides solve with layers and halving. Here the search space is history instead of hardware.


The hidden-change catalog


When you ask "what changed?", you and everyone you ask will instinctively scan for deliberate, human, recent changes. The catalog below is the rest of the iceberg — the changes that answer to nobody's memory. Skim it during any confusing incident and ask which rows could touch the symptom:


Hidden change sourceWhat actually changesHow it bitesFirst place to look
OS auto-updatesPatches, drivers, reboots overnightFeature that worked at logoff is gone at logonThe machine's update history
Firmware auto-updatesRouter, AP, firewall internalsSettings reset to defaults; behavior shifts subtlyDevice's update log and schedule
Security software pushesDefinitions, agent versions, policiesSuddenly blocks traffic it allowed yesterdayThe security console's change history
Certificate lifecyclesRenewal replaces cert; failure to renew expires itPadlock warnings; services silently refusing connectionsCertificate validity dates
DHCP churnDevices receive different addresses over timeAnything pinned to an old address stops matchingLease table; reservation list
DNS record edits and TTL expiryName-to-address mappings shift, sometimes hours laterHalf the office reaches the new address, half the oldRecord change times; TTL values
Scheduled jobsBackups, scans, syncs kick inNightly slowness that is really a backup windowJob schedules vs. symptom times
ExpirationsLicenses, trials, subscriptions, domains lapseFeatures degrade or vanish on their anniversaryExpiry dates in vendor portals
Leftover "temporary" fixesA static setting, a test rule, a borrowed cableOutlives its purpose, then fails or conflictsAnything labeled "temp" or "test"
Physical worldCleaning crews, new furniture, new devices plugged inUnplugged gear; duplicate addresses; the occasional loopEyes, on site
Clock driftA machine's time wanders from everyone else'sAuthentication and certificates fail mysteriouslyCompare clocks across devices

Notice how many rows are delayed-fuse changes: the triggering decision happened weeks ago and only detonated now. That matters for the next section. It means "what changed at the moment of failure" is sometimes the wrong window to search. The fuse was lit earlier. Firmware deserves special respect here; updates that arrive uninvited and reset settings are common enough that firmware lifecycle planning is its own topic.


The sneakiest delayed fuse of all is the humble reboot. Plenty of changes do nothing until the next restart: a pending patch, an edited startup config, a driver staged for install. That restart may come months later courtesy of a power blip. The device then comes back subtly different, and everyone blames the power blip, because "it broke when the power flickered" feels like a complete story. It is not: the blip was only the detonator. When a device misbehaves right after any restart, ask what had been staged on it, not just what was done to it that day. Check whether its running configuration had ever been saved. An unsaved config change dies in a reboot and takes its improvements with it.


Time is your search key


Change correlation is a join between two timelines: the timeline of symptoms and the timeline of changes. The sharper you can draw the first, the less of the second you must search. You need two timestamps, and people routinely get both wrong.


When did it actually break? Not when it was reported — when it broke. Users report failures when they next need the thing, which can lag the break by hours or days; the Monday-morning complaint flood often describes a Saturday-night casualty. Your monitoring system is the honest witness here: the first failed check pins the break to the minute. The gap between when things break and when humans notice is precisely what the detection-time guides are about. When did it last provably work? Not "it always works" — the last timestamped success: the last green check, the last log line showing a completed transfer, the last successful backup of the thing.


Between those two timestamps lies the break window. Every change inside the window is a suspect; everything outside is mostly cleared. A tight window is devastatingly effective — a five-minute window during a quiet weekend usually contains exactly one event. And it is your culprit taking a bow. And when the window will not tighten from testimony alone, bisect it with evidence. Check the middle of the window in your monitoring history or logs, see whether things were still good, and halve again. This is the same move, applied to time, that Binary Search for Networks applies to paths. Just remember the delayed fuses: for expirations and TTL-style changes, the decision may sit far outside the window even though the detonation is inside it.


Where change evidence hides


No formal change log? You still have one — it is just scattered across a dozen systems that each remember their own slice. When the break window is drawn, sweep these in rough order of effort:


  • 1. Update histories. Every OS keeps one; so do most firewalls, APs, and NAS boxes. On a Windows machine, Get-HotFix or the update history screen answers "did you patch yourself last night?" in seconds.
  • 2. System event logs. The system log remembers reboots, service installs, driver changes, and time jumps — each entry timestamped, none of them relying on anyone's memory.
  • 3. Config backups, diffed. If you back up device configurations, compare the current config against the last good one. A diff — a line-by-line comparison showing only what differs — turns two thousand-line configs into a three-line answer.
  • 4. The DHCP lease table. Did the troubled machine's address change recently? Did something new join the network in the window?
  • 5. Certificate dates. Issue and expiry timestamps are changes with sub-second precision, printed right on the certificate.
  • 6. Mailbox archaeology. Search for "maintenance," "scheduled," and "update" across the window — ISPs and vendors announce changes in emails everyone deletes unread.
  • 7. Tickets, chats, and calendars. "Oh right, the electrician was in Saturday" has closed more cases than packet analysis ever will.

The diff deserves a demonstration, because it is the closest thing this job has to a magic trick. Two config snapshots, one command's worth of comparison:


> fc /n fw-config-friday.txt fw-config-monday.txt

***** fw-config-friday.txt
  114:  allow tcp any -> 10.0.20.15 port 445   # file server
***** fw-config-monday.txt
(line missing)

<- the rule admitting file-share traffic vanished over the weekend

Whoever "cleaned up unused rules" on Saturday now has some explaining to do — but notice you did not need their confession. The backups testified. This is why scheduled config backups are the highest-value habit in this entire guide: they turn every future "what changed?" into a thirty-second diff. If you keep no other records, keep those.


Asking humans so they can actually answer


You still need testimony, and how you ask determines what you get. "Did anything change?" is a yes/no question aimed at memory and pride at the same time; it reliably returns "no." Ask narrow, concrete, blame-free questions instead, and watch the answers improve:


  • "What were you working on Thursday and Friday?" — projects, not confessions. The change surfaces as a side detail of an innocent story.
  • "Did anything ask you to restart, update, or log in again recently?" — users experience changes as prompts, not as changes.
  • "Has anything new been plugged in, installed, or delivered lately?" — catches the new printer, the visitor's laptop, the smart TV someone gifted the break room.
  • "Was anyone in the server room or wiring closet last week?" — electricians, cleaners, and cabling contractors count, and nobody thinks to mention them.

Tone is half the technique. The instant "what changed?" sounds like "who broke it?", memories fog over in self-defense. Say the quiet part out loud — "nobody is in trouble; I just need the timeline" — and mean it, because you will be asking these same people next month.


A worked timeline


Monday, 08:20: tickets pour in — "VPN is down," meaning the remote-access portal every hybrid worker uses. Nobody in IT touched anything, obviously. Here is the reconstruction, assembled in about fifteen minutes from sources that were all sitting there waiting to be asked:


TimeFactSource
Fri 17:12Last confirmed VPN login before the weekendAuthentication log
Sat 03:00Firewall's weekly maintenance window opensDevice settings
Sat 03:07Firewall installs a firmware update and rebootsDevice update log
Sat 03:14Monitoring's check of the VPN portal goes red — and stays redMonitoring history
Sun 19:41One user emails "VPN acting up?"; nobody sees itSupport mailbox
Mon 08:20Ticket flood begins as the workweek startsTicket system

Laid out this way, the story tells itself: portal healthy until minutes after a self-installed firmware update. The update log confirms the mechanism — the upgrade restored a handful of settings to factory defaults, among them the one that enabled the portal. Re-apply the setting, service returns, and the fix took five minutes once the timeline existed. Without it, Monday would have opened with reboots and guesswork aimed at a perfectly healthy VPN server three hops behind the actual culprit.


Take the two quiet lessons with you. The break happened Saturday 03:14, but detection happened Monday 08:20. The monitoring system knew all weekend, and its history is what made the timeline trivial. A check that also alerted someone would have made the whole incident a Saturday footnote. And the culprit change was made by no human at all. That is exactly why "did anyone change anything?" nearly sent everyone down the wrong road. The question got asked at 08:25 and was answered honestly with "no".


Correlation is not conviction


One caution before you get too fond of the trick. "It broke right after X" makes X a prime suspect, not a convict — correlation (two things happening together) is not causation (one causing the other). Changes travel in herds: patch nights touch a dozen systems at once, and the maintenance window that updated the firewall may also have rebooted three servers. Pick the wrong herd member and you will "fix" something innocent, watch the problem persist, and lose the trail.


So treat the correlated change as what it really is: the top-ranked hypothesis, entitled to the first test but not to blind faith. Confirm the mechanism: can you explain, step by step, how this change produces this symptom? Or confirm by controlled reversal. Roll the change back and watch whether the symptom leaves with it, ideally as a deliberate, reversible, one-at-a-time move. The full discipline of ranked guesses and killer tests is in Guess, Test, Learn; the timeline just hands that process its best lead.


Building the change-tracking habit


Everything so far was archaeology — reconstructing history nobody recorded. The upgrade is to record it as it happens. And the trap is over-engineering: a heavyweight change-approval process for a three-person shop dies in a month, and a dead process records nothing. The habit that survives is almost embarrassingly small. Write one line per change in one agreed, searchable, timestamped place: a shared document, a dedicated chat channel, a plain text file. Each line: when, who, what, why, and how to undo it.


Tue 16:40  dana  fw1: allowed tcp/8443 to 10.0.20.31 (new HR app)   undo: remove rule 118
Wed 09:15  sam   sw-floor2 port 14: moved to VLAN 30 (intern desk)  undo: back to VLAN 10
Thu 22:05  dana  files01: NIC driver updated during patch window    undo: roll back driver

Three deliberate habits multiply the value. Log the temporary changes with special care. They are the ones that outlive memory and ambush you at month six, like the stale setting in this pillar's worked ladder example. Schedule automatic config backups of every device that supports them, because backups plus diff equal a change log that writes itself and never forgets. And prefer doing risky work inside announced maintenance windows. When something breaks at 03:07 Saturday, the window itself is documentation. There is a whole guide to running those windows without drowning in your own alerts.


Tip: the change log is for reading during incidents, not for auditing people. The moment a logged change gets someone scolded, the log starts lying by omission. And an incomplete change log is worse than none, because you will trust it. Keep it blameless and it stays true.


When the change was not yours


Sometimes the timeline points cleanly at a change on the far side of a fence you cannot see over. Maybe the ISP "improved" something upstream, a cloud application changed its login flow, or a vendor rotated the certificates on their service. You cannot read their change logs — but you can still correlate. Your evidence is your own monitoring history. "Our checks of your service went red at 03:14, from two locations, while everything else stayed green." That turns a support call from a plea into a briefing. Providers move noticeably faster for callers with timestamps. The art of demonstrating that the problem is beyond your edge — politely, with graphs — is its own genre, covered in proving it is not the network.


Where to go next


Make "what changed?" your opening move, not your last resort. It costs one minute: draw the break window, sweep the hidden-change catalog, and check the update logs and the diffs. Only then start climbing ladders and splitting paths. A fault with a fresh change behind it is the easy kind. And most faults are that kind. Then spend one afternoon on the habits — the one-line log and automatic config backups — and next quarter's mystery outage becomes a thirty-second diff. The rest of the toolkit lives at A General Troubleshooting Methodology.


Put this into practice with SysPeep Monitor

Half of change correlation is knowing exactly when things broke — and that is what a monitoring history gives you. SysPeep Monitor timestamps the first failed check and keeps the graphs that show healthy-until-when. It alerts you at the break instead of at the Monday flood. So you can line the fault up against your change log while the trail is fresh. The free trial runs on any Windows machine.

Download Free TrialLearn About SysPeep