Home › Topics › Catching Intermittent Problems › Building Timelines

Building Timelines: Correlating Symptoms with Logs and Changes


Every intermittent fault leaves fingerprints. The trouble is that it leaves them in five different drawers. There is a line in a firewall's log, a spike on a graph, and a grumble in a ticket. There is a service restart in a Windows event log, and a change somebody made and half-remembers. Each drawer on its own tells you nothing. Laid side by side in strict time order, they tell you nearly everything. Somewhere in that merged record something happens right before the fault, every single time.


That merged record is a timeline, and building one is the closest thing this job has to detective work. You gather the witnesses, get their stories, check their watches, and find out who was always at the scene just before the crime. It is not glamorous. It is a spreadsheet. It is also, for a recurring fault, the single highest-yield hour you can spend. I have spent that hour clearing suspects I had already half-decided to rebuild.


This guide shows you the craft end to end. It covers where the evidence lives and how each source lies to you, and why synchronised clocks come before everything else. It covers how to assemble and read the timeline itself. It shows how the "what always precedes it" test narrows a mystery down to a suspect. It closes with a worked example where the method catches a tunnel that kept dying at dawn.


In this guide

  • Why a timeline beats memory and hunches
  • The evidence sources — and how each one lies
  • Clocks first: time synchronization as a prerequisite
  • Building the timeline, step by step
  • The narrowing game: what always precedes the fault
  • A worked example: the tunnel that died at dawn
  • Keeping the habit lightweight
  • Where to go next

Why a timeline beats memory and hunches


Human memory is a terrible evidence locker. It rounds "11:42" to "around lunch," it reorders events to fit whatever story feels right, and it welds together things that happened on different days. Ask three people when the outage started and you will get three confident, incompatible answers. None of them are lying; memory just does not store timestamps.


Causality, meanwhile, is ruthless about order: causes come before effects. That single physical fact is the entire engine of this method. If you can put every recorded event in true order, then whatever caused your fault must appear somewhere upstream of every occurrence. You no longer need a brilliant theory; you need an accurate sequence and the patience to read it. The theory falls out of the sequence.


A timeline also disciplines your own hunches, which is more valuable than it sounds. A hunch says "it is probably the new switch." The timeline replies, "the fault predates the new switch by two weeks" — and just saved you a weekend of rebuilding the wrong thing. Half the wins from this method are exonerations: suspects cleared cheaply, before you spent real effort on them. And unlike live-chasing the ghost, timeline work happens at your desk, at your pace, with coffee — the fault already happened; you are simply reading its diary.


The evidence sources — and how each one lies


A good timeline draws on every recording system you have. Each source contributes something the others cannot, and each has a characteristic way of misleading you. Knowing the quirks is what separates correlation from confusion:


SourceWhat it gives youHow it lies
User reports / ghost logWhat the fault feels like; sightings with rough timesRecords when someone noticed, not when it happened; filtered by who was around to see it
Monitoring alerts and graphsObjective timestamps, before-and-after valuesPolling gaps miss short events; alerts fire minutes late by design (confirmation rules)
Device syslogLink flaps, spanning-tree changes, rekeys, errors from switches, routers, firewallsDevice clocks drift; logs vanish on reboot unless shipped to a collector
Windows event logsService crashes, disk timeouts, unexpected restarts, authentication troubleBuries the signal in noise; mixes local-time display with UTC storage
Application logsErrors and slowness in the app users actually feelEvery app invents its own timestamp format and time zone
Change recordsWhat humans deliberately didThe most important change is often the one nobody wrote down

The ghost log — your running record of user sightings — is genuinely valuable evidence despite its biases. Both why reports mislead and why you should collect them anyway are explained in the first guide in this topic. Device logs, meanwhile, are only usable for this game if they still exist when you need them. That means shipping them off the device to a central collector. The syslog and log collection guides cover building that habit properly. A switch that reboots takes its uncollected memory with it — exactly the memory you wanted.


Do not stop at the obvious drawers, either. UPS (uninterruptible power supply) and power-management logs testify about blips and brownouts nobody felt. DHCP server logs record which device asked the address-handout service for an address, and when. They sometimes reveal the mystery guest that joins the network right before every incident. Backup and job schedulers keep precise start and end times for the nocturnal machinery. Even the building can testify: door-badge times and cleaning schedules have closed more than one case involving a bumped cable. The rule of thumb: anything with a clock and a memory is a potential witness.


Ashgrove Housing lost its estate-office network for a minute or two "every few days, at random" for most of a spring. The timeline showed, before every occurrence, a DHCP request from the same unregistered address, which appeared in no inventory. The reception sign-in sheet, added to the sheet mostly to be thorough, showed the same visitor arriving twenty minutes before each one. This was a contractor surveying the building for a lift upgrade. The contractor's laptop still carried a DHCP server from a previous job and cheerfully offered addresses to anyone who asked. The disconnects correlated perfectly with the contractor's calendar, which was the one calendar nobody in IT had. The laptop took ten minutes to fix. The sign-in sheet is now a standing row in every timeline Ashgrove builds.


Warning: about that last table row. When a recurring fault starts on a specific date, something changed on or near that date. But the change that matters is frequently informal. It might be a "quick" config tweak, a new device someone plugged in, or a job someone rescheduled. Ask "what changed around then?" with genuine amnesty, not blame. You want the confession; make it cheap to give.


Clocks first: time synchronization as a prerequisite


The failure mode that quietly ruins timelines is that the sources disagree about what time it is. A firewall running four minutes fast puts its log lines four minutes into the future. Suddenly the "effect" appears to happen before its cause. Your beautifully sorted timeline is scrambled nonsense that feels authoritative. Bad clocks do not just add noise — they invert causality, which sends you hunting in exactly the wrong direction.


The cure is NTP (Network Time Protocol), the standard mechanism by which devices continuously nudge their clocks toward an agreed source. Point every server, switch, router, firewall, and the monitoring system itself at the same time source. Then the whole estate keeps time within fractions of a second — more than good enough for this work. On a Windows machine you can check the time service's health and measure its offset against a reference like this:


C:\> w32tm /query /status
    (shows the sync source, stratum, and last successful sync time)

C:\> w32tm /stripchart /computer:dc1.corp.local /samples:3
    (measures this machine's clock offset against dc1, in seconds)

Network devices have equivalents. Most can display their clock, their configured NTP source, and whether synchronization is actually working. That is worth confirming rather than assuming. A device with a wrong clock is a witness who arrived after the ambulance and insists it was there first. Beyond "turn on NTP," three habits keep timelines honest. The second one mentions UTC, which is Coordinated Universal Time: the reference zone that never shifts for daylight saving. Here are those habits:

  • Audit every source's clock at the start of the hunt. For any device you cannot fix immediately, measure and write down its offset ("FW2 runs 6 min 10 s fast"). Apply the correction arithmetic when you harvest that device's lines.
  • Pick one time zone for the timeline and convert everything into it. Mixed local-time and UTC sources are the classic silent one-hour error, and daylight-saving transitions add a twice-a-year trap on top. Note which zone the timeline uses right in its header.
  • Keep seconds. Minute-level times are often too coarse to order a fast chain of events; the difference between 05:10:02 and 05:11:30 can be the whole story.

Give one clock special status: the monitoring server's. It stamps every alert and every graph point. It is usually the best-synchronized machine you own, and its records span the whole network. That makes it the natural reference everything else gets corrected toward. When two sources disagree and you cannot tell which drifted, the one that agrees with the monitoring server usually wins the argument.


Clock hygiene for logs is a discipline of its own, and the log collection guides go deeper. For timeline purposes, the rule is simply: verify before you trust, correct what you can, and record the offsets you could not.


Building the timeline, step by step


The tooling is deliberately humble: a spreadsheet. Fancier options exist, but a sheet is fast, sortable, shareable, and attachable to the ticket afterward. I have built timelines in three tools designed for the purpose. I keep going back to the sheet, because the ticket can hold a sheet and the tools cannot hold the ticket. The procedure:


1. Define the windows. For each known occurrence of the fault, take the best-known symptom time and pad it generously — thirty to sixty minutes before, fifteen after. The padding before matters most; that is where causes live. Slow-burn suspects (a backup that started hours earlier) will still show up as long-running entries that span the window.


2. Create the master sheet. Four columns: Timestamp (one zone, with seconds), Source, Event (short and factual), Notes (confidence, clock corrections applied, links to raw evidence).


3. Harvest each source for each window. Pull the matching slices of alert history, syslog, event logs, application logs, the ghost log, and change records. Copy in only lines that say something — state changes, errors, starts and stops — not every heartbeat. When in doubt, include it; you can ignore rows later, but you cannot read rows you never collected.


4. Normalize the clocks. Convert time zones, apply the measured offsets from your clock audit, and flag any row whose time you cannot trust. An honest "about 08:00, per user memory" belongs in the sheet — clearly marked as soft.


5. Sort, then mark the symptom. Sort by timestamp and visually mark the rows where the fault itself shows — bold them, shade them, anything. These are the crime scenes; everything else is context around them.


6. Read upstream. For each marked row, read backward and list every event in the preceding half hour. This per-occurrence list of precursors is the raw material for the narrowing game below.


7. Repeat per occurrence and keep everything. One occurrence gives you a story; three give you a pattern. The sheet grows with each recurrence, and each recurrence makes the next step sharper.


8. Write the story in one paragraph. After each pass, force yourself to summarize what the timeline currently says, in plain words, including what it rules out. For example: "Fault follows the 05:10 rekey, but only on nights X; new switch cleared; user reports lag the fault by hours." If you cannot write the paragraph, you do not understand the sheet yet — which is itself useful to know. The paragraph is also what you hand your boss instead of the spreadsheet.


The narrowing game: what always precedes the fault


With precursor lists from at least three occurrences, play the intersection game: which events appear before every occurrence? Anything present before some occurrences but not others drops down the suspect list. What survives the intersection is usually a very short list — often one or two items — and the mystery suddenly has names in it.


Two traps to dodge while playing. The first is the periodic-event trap: something that happens every five minutes precedes everything, including your fault, and proves nothing by itself. The counter is the control test — check the quiet periods. A true suspect is present before occurrences and absent (or different) when the fault stays away. "The rekey ran at 05:10" means little if the rekey runs at 05:10 on good days too. But "the rekey ran at 05:10 while the link was saturated" may be the whole answer. That combination exists only on bad days.


While you are reading upstream, also watch for what is missing: the event that should be in the window and is not. Look for the heartbeat that skipped, the poll that returned nothing, or the hourly job with no completion line. Look for the switch that logged nothing at all during a minute when its neighbors were shouting. A switch that says nothing while its neighbours shout is not calm. It is unconscious. Absences sort just as informatively as events once you know the normal rhythm. That is one more reason the quiet-day rows and your baseline knowledge belong in the sheet.


The second trap is treating correlation as a verdict. A precursor that survives the intersection is a suspect, not a conviction — two events can share a hidden common cause, or simply coincide. What upgrades suspicion to a case is mechanism plus prediction. Mechanism: can you tell a concrete story of how the precursor produces the symptom? Prediction: does the theory say when the fault will strike next — and does reality comply? The cleanest way to run the prediction test is to aim an automated watcher at the precursor and the symptom together. That is precisely the business of setting tripwires. Forcing the prediction on your own schedule instead of waiting is the business of reproducing rare failures.


A worked example: the tunnel that died at dawn


The complaint concerns a branch office's site-to-site VPN tunnel — the encrypted link carrying all traffic between branch and headquarters. It drops for a couple of minutes at around ten past five in the morning, a few mornings a week, never the same days. The early warehouse shift loses their scanners mid-shift; by the time anyone senior arrives, everything works. Weeks of occasional chasing have produced nothing.


First move: the clock audit. The firewall turns out to be running six minutes slow — and this alone explains why earlier attempts went nowhere. In the raw logs, the firewall's own entries appeared to come after the monitoring alert. It looked as if the tunnel died for no reason and the firewall reacted late. With the offset measured and corrected, the merged timeline for one occurrence reads:


Time (corrected)SourceEvent
22:01:14Backup server event logNightly offsite copy job started
22:00–05:30WAN utilization graphBranch link pinned at its ceiling all night
05:10:02Firewall syslogScheduled tunnel rekey begins (fresh encryption keys negotiated)
05:10:40Firewall syslogRekey retransmissions — replies not arriving in time
05:11:30Firewall syslogPeer declared dead; tunnel torn down
05:13:00MonitoringTunnel-down alert (two failed polls confirmed)
05:14:20Firewall syslogTunnel re-established
05:15Ghost logWarehouse lead: "scanners froze again, few minutes" (reported at 08:40)

Now the narrowing game. The rekey happens every morning at 05:10 — including quiet mornings — so by itself it fails the control test. The backup job runs every night, so by itself it fails too. But the overlap passes perfectly. On every bad morning, the offsite copy was still running — and still saturating the link — at 05:10. On every quiet morning, it had finished by then. The mechanism writes itself. The rekey is a brief, time-critical handshake. On a saturated link its packets queue and drop. The firewall concludes the far end is dead and tears the tunnel down, exactly as designed. The fault needed two innocent schedules to collide, which is why the bad days looked random — they were the days the backup ran long.


The theory predicts: no drop on any morning the backup finishes early, a drop whenever it overruns past 05:10. Four more mornings confirm it, ticket by quiet ticket. The fix — capping the backup's bandwidth so the link never fully starves — is boring, as the best fixes are. The same monitoring that supplied the timeline's rows then confirms the tunnel stays up through the next overrun. Case closed on evidence, not on a hunch.


Keeping the habit lightweight


Timeline-building fails when it turns into a ceremony, so keep it small. A spreadsheet and an hour is the standard kit. Centralized logs make harvesting a five-minute job instead of a scavenger hunt across a dozen devices. That is one more reason to collect logs centrally before you need them. Save the finished timeline into the ticket. The next ghost in the same neighborhood starts half-solved. The write-up doubles as the incident's story for your future self, who forgets things at the same alarming rate as everyone else's future self.


Tip: add rows for silence, too. "05:10:02 rekey — completed normally, link idle" from a quiet day sits next to the bad day's version. That comparison makes the difference jump off the screen. Contrast is evidence.


Where to go next


A timeline tells you what the fault does and what always walks in just ahead of it. The kit has not changed since the top of this guide: a spreadsheet, an hour, and clocks that agree. Skip the clocks and the sheet will still sort beautifully, in the wrong order, and send you to rebuild something innocent. The natural next move is to stop harvesting evidence after the fact and start capturing it automatically at the moment of the crime. That means fast checks, threshold traps, and log-pattern triggers armed around the clock. That is setting tripwires, the next guide in Catching Intermittent Problems. And if your timeline needs richer raw material over time, long-term trending keeps the graph rows flowing.


Put this into practice with SysPeep Monitor

Half the rows in a good timeline come straight out of your monitoring system. They include alert history with clean timestamps, and graphs that show what every link and server was doing at the fatal moment. SysPeep Monitor collects those around the clock over SNMP, WMI and SSH. It keeps the history and lets you read any suspect's past from one web console. The free trial installs in minutes on a Windows machine.

Download Free TrialLearn About SysPeep