Home › Topics › Syslog & Log Collection › Timestamp Sanity

Timestamp Sanity: NTP, Time Zones, and Log Correlation


Picture a bad morning, assembled from real ones. Overnight, someone got into the network through the VPN, and you are rebuilding the story from five devices. They are the firewall, the VPN gateway, the core switch, the file server, and the domain controller. The logs say something impossible. The file server shows sensitive folders being opened at 02:15 — but the VPN gateway insists the attacker did not log in until 02:24. Effect before cause. For one ugly hour the working theory is a second, undiscovered way in, and the incident doubles in size.


Then someone checks the clocks. The VPN gateway's clock is running eleven minutes fast — it was never pointed at a time source. Subtract eleven minutes and the login really happened at 02:13, two minutes before the file access: cause, then effect after all. Meanwhile the firewall logs its lines in UTC (universal time) while everything else uses local time. The firewall entries seemed to describe a quiet, unrelated hour. They were actually the same minutes as everything else, several hours of label apart. Reconciled — drift subtracted, zones aligned — the timeline finally reads forward: VPN login, then lateral movement, then file access. The attack was ordinary. The evidence was scrambled, by two clocks nobody had ever bothered to set.


Every lesson in this guide is in that story. A log entry is a claim that something happened at a time; when clocks lie, your archive still looks authoritative — it just quietly stops being true. This guide covers the four ways timestamps go wrong and the two-part fix (NTP everywhere, plus a written time zone policy). It shows how to verify sync instead of assuming it. It covers the correlation habit that turns five devices' logs into one trustworthy timeline.


In this guide

  • The four ways timestamps lie
  • The fix, part one: NTP everywhere
  • Verify the sync — then keep verifying
  • The fix, part two: pick a time zone policy
  • Correlation in practice: one incident, three devices
  • The maintenance ritual
  • The timestamp-hygiene checklist
  • Where to go next

The four ways timestamps lie


1. Clock drift, and the January-first mystery

Computer clocks are not precision instruments. Each device keeps time with a cheap crystal oscillator that runs slightly fast or slightly slow — a little more so as temperature changes. Left unsupervised, the error compounds. A clock that gains just one second a day drifts six minutes in a year; the eleven-minutes-fast VPN gateway from the opening story is no hypothetical. Drift is the slow version of the lie: every unsynced device wanders off in its own direction, at its own speed. So any two of them disagree by an amount that grows forever.


The fast version is the reset. Many network devices — especially cheaper switches and embedded gear — have no battery-backed clock. Lose power, and on boot the device's clock restarts at a default date, often January 1 of whatever year its firmware considers the beginning of time. Until something sets the clock, every log line it emits is stamped years in the past. This produces one of IT's classic head-scratchers: a log search turns up entries apparently written decades ago, neatly interleaved into your archive. It is not haunted. It is a switch that rebooted after a power blip and spent twenty minutes logging from the distant past before it resynced. Or it never resynced at all.


2. Time zone chaos

Now suppose every clock is perfectly accurate — and the timestamps still disagree, because the devices are speaking different time languages. Device A logs in local time. Device B logs in UTC, the global reference zone that many network devices default to. And your collector, when it writes a message to disk, can stamp the arrival in its own configured zone. That is whatever the box happens to be set to. One moment, three honest clocks, three different numbers on paper. Unlike drift, nothing here is broken, which makes it more treacherous: no amount of clock-checking reveals it, because the clocks agree. Only the labels differ.


Daylight saving time adds two special traps for zones that observe it. On the fall change, one local hour happens twice — a log stamped 01:30 could be either pass, and events from the repeated hour interleave into nonsense. On the spring change, an hour never happens at all, and a naive reader wonders why every device went silent. Any incident that straddles a DST change in local-time logs is a small forensic puzzle you did not need.


3. The two timestamps on every line

A subtlety that eventually bites everyone: most collected log lines have two times attached. The first is the device-stated time — the timestamp the sender wrote inside the message, from its own clock, in its own zone. The second is the arrival time — when the collector received the message, stamped by the collector's clock. They differ for boring reasons (a millisecond of network transit) and for interesting ones. The sender's clock may be wrong. The sender may have buffered messages during a network hiccup and delivered them late in a batch. Or the message waited out a device reboot and arrived minutes after the event it describes.


Which to trust? If the device is NTP-synced, its stated time is the truth about the event — that clock was present when it happened. The arrival time is the truth about delivery, and it earns its keep twice. It is the tie-breaker when a device's clock is known-bad. The arrival stamp on that January-first line tells you when the reboot really was. It is also a diagnostic in its own right. A large or growing gap between stated and arrival time means clock error or delivery delay, both worth investigating. This is why a good collector setup keeps both rather than overwriting one with the other.


4. The limits of precision

Classic syslog timestamps resolve to one second — and within a single second, a busy network can emit dozens of events. Their order inside that second is simply not recorded; the sequence you see is an accident of delivery and file position. So do not over-read it. If the firewall's deny and the server's error carry the same second, the logs alone cannot say which came first. Building a causal story on that ordering is guesswork wearing a lab coat. The modern syslog format adds sub-second precision and an explicit zone offset. That is one of the quiet upgrades covered in syslog fundamentals. But plenty of gear still sends the classic format. At small-network scale the honest posture is: trust ordering across seconds and treat ordering within a second as unknown. Lean on message content and sequence context when one second matters.


The fix, part one: NTP everywhere


The cure for drift has existed for decades and costs nothing. NTP (Network Time Protocol) is a standard service by which a device periodically compares its clock against a reference server. The device measures the offset while accounting for network delay. It nudges its clock into agreement — gradually, so time never visibly jumps backward. A device syncing over a LAN stays within fractions of a second of its reference, indefinitely, with no human attention. Every switch, firewall, server, and NAS you own almost certainly speaks it; the entire fix is pointing them all at the same place.


The small-network architecture is two sentences. One or two internal machines sync to the outside world; everything else syncs to them. Pick stable boxes for the internal sources — the firewall/router and a server are the usual pair. If you run a Windows domain, the domain controller is already playing this role for domain members. Point each internal source at a set of public NTP servers (your vendor's default pool is fine). Then configure every other device with the internal sources as its NTP servers. Two internal sources beat one, because a time source that fails silently leaves its clients adrift for months before anyone notices; with two, devices cross-check. The payoff of the hierarchy: your devices agree tightly with each other — which is what correlation needs. They would agree even if the whole site were briefly a second off from the world.


Handle the leftovers deliberately: gear with no NTP support (some UPSes, printers, embedded oddities) gets its clock set by hand at intake and checked at each maintenance pass. Or it gets mentally flagged "timestamps decorative." For devices flagged this way, the collector's arrival stamp is the real record. Either is fine; what is not fine is not knowing which category a device is in.


Verify the sync — then keep verifying


Configured is not synced. NTP fails quietly: a firewall rule blocks the port, an upstream source goes stale, a device silently falls back to its own drifting crystal. Since the whole point is trusting timestamps without thinking about them, verification has to be a standing check, not a hope. Two layers:


The spot check — ask a device for its clock and its sync status, and compare against a known-good clock. Most gear answers directly:


# Cisco-style CLI            # Linux server
show clock                   timedatectl
show ntp status              #   look for: "System clock synchronized: yes"
# Windows server
w32tm /query /status         #   look for a small offset and a real source

Read for two things. The clock should roughly match your reference (phone clocks are synced and make a fine pocket reference). The status line should confirm an active source rather than "unsynchronized." Off by seconds or claiming no source: fix it now, while it is a config chore instead of an investigation footnote.


The monitored check — make drift page you. Your monitoring system may be able to query a device's reported time (many can, over SNMP or an agent) or check NTP health directly. If it can, alert when any device's offset exceeds a few seconds. That threshold is deliberately loose — NTP keeps healthy devices within fractions of a second, so multiple seconds of offset never happens by accident. It means sync is broken and the slow wander has begun. This turns clock failure from a thing you discover mid-incident into a ticket like any other.


Tip: your log archive doubles as a free drift detector. The gap between each message's device-stated time and the collector's arrival stamp should be near zero for a LAN. Scan for hosts where it is not, and you have a list of broken clocks without touching a single device. A one-line habit after any power event pays for itself the first time it catches a switch that came back on the default date.


The fix, part two: pick a time zone policy


NTP makes the clocks agree; it does not make the labels agree — a synced device can still log in any zone it likes. That is a policy decision, and the strong recommendation is: store and log in UTC, translate for humans at read time. UTC has no daylight saving — no doubled hour, no missing hour, ever. It matches the default on much network gear. It stays coherent the day you add a site in another zone or ship logs to someone else's analysis tool. The cost is real but small: mentally converting "17:42Z" to local wall-clock time when you read raw logs. Search platforms and dashboards do that translation for you; grep does not, and you will genuinely get used to it within weeks.


The honest alternative: an all-local policy — every device logs in the site's one zone. That is defensible for a small, single-site shop with no expansion plans, because timestamps then match the wall clock and human memory ("it broke just after lunch"). You accept the DST puzzles twice a year as the price. What is not defensible is the accidental third option most networks actually run: mixed — some devices local, some UTC, nobody sure which is which. Mixed is the opening story. It combines the conversion cost of UTC with the DST cost of local and adds a per-device guessing game, during incidents, forever. Pick one policy, write it down next to your retention policy, and make the zone setting part of device intake. The choice matters less than its uniformity.


Correlation in practice: one incident, three devices


Correlation is the payoff skill: lining up entries from multiple devices into one causal timeline. Here is the whole method on a realistic mini-incident — users at a branch office dropped off the VPN for half a minute. There are three sources, and (realistically) not yet a uniform zone policy. The core switch logs UTC, the firewall logs UTC, and the file server logs local time, which for this site is UTC minus five. The raw evidence, as found:


sw-core-01 (UTC):    18:42:11 %LINK-3-UPDOWN: Interface Gi1/0/1 (uplink to fw), changed state to down
fw-a (UTC):          18:42:13 WAN monitor: primary link loss detected
fw-a (UTC):          18:42:40 IPsec: tunnel to branch-office DOWN, dead peer detected
srv-file (UTC-5):    13:42:52 SMB session from 10.8.20.14 (branch) disconnected
sw-core-01 (UTC):    18:43:05 %LINK-3-UPDOWN: Interface Gi1/0/1, changed state to up

A naive reading has the server event happening five hours before everything else — or, if you assume all-local, five hours after. The method, in four steps:


  • 1. List each source's zone and offset before reading a single line. (This is the step the written zone policy makes instant.)
  • 2. Normalize everything to one zone — UTC by convention. Here only the server needs converting: 13:42:52 local + 5 hours = 18:42:52 UTC.
  • 3. Sort and read. 18:42:11 uplink down → :13 firewall sees WAN loss → :40 VPN declares its peer dead → :52 server drops the branch session → 18:43:05 link restored.
  • 4. Check the story against causality. Each event now follows from the one before, with believable gaps — the two-second detection lag, the tunnel's dead-peer timeout doing its patient waiting. Cause upstream, effects downstream, nothing reversed.

Verdict: one flapping uplink explains everything; the VPN and the file server were victims, not causes — and the fix is a cable, not a firewall change. Notice what made thirty seconds of confidence possible: trusted clocks (NTP), known zones (policy), and one normalization pass. Without them, this same evidence supports three different stories, and this was a friendly incident — nobody was hiding. When you suspect an intruder, the timeline is often all you have, and it is exactly then that an eleven-minutes-fast clock does its damage. This normalize-then-sort habit is also what makes multi-device alert patterns and parsed-field queries trustworthy. Every "N events within M minutes" rule silently assumes the clocks agree.


The maintenance ritual


Time hygiene is not a project; it is a small recurring habit with three triggers. Quarterly: run the spot check across your core devices — ten minutes with the commands above, or one glance at your drift monitoring if you built it. After any power event: a site-wide outage or a rack losing power is exactly when battery-less clocks reset to their default date. So add "verify clocks" to your post-outage checklist, right after "verify everything came back." At device intake: every new device gets NTP pointed at your internal sources, its zone set per policy, and a one-line verification before it enters production. A device that starts life synced tends to stay synced. One that starts adrift is a mystery deferred to the worst possible moment. If you followed the collector build guide, "confirm NTP everywhere" was already on the checklist; this ritual is what keeps that checkbox true.


The timestamp-hygiene checklist


The whole guide as a working checklist — print it, run it, and your timeline problem is permanently boring:


CheckWhyWhen
One or two internal NTP sources, synced to outside poolsEverything agrees with each other, with redundancySetup; verify quarterly
Every device points at the internal sourcesDrift never startsDevice intake
Sync status verified, not assumedNTP fails silentlyQuarterly; after power events
Drift alert in monitoring (offset beyond a few seconds)Clock failure becomes a ticket, not a forensic surpriseSetup
Time zone policy written: UTC (or documented all-local)One time language; no mixed-zone guessingOnce; enforce at intake
Collector keeps arrival stamps alongside device timesTie-breaker and delivery diagnosticSetup
Stated-vs-arrival gap scanned for outliersFree fleet-wide drift detectorQuarterly; after power events
Logs dated far past or future treated as clock incidentsCatches default-date resets earlyWhenever seen
No device in production without NTP + zone configuredThe archive stays trustworthy by defaultEvery intake

Where to go next


Timestamps are the spine of everything else this library builds. Collection gives you the evidence, parsing makes it queryable, and alerting makes it tap your shoulder. None of it can be trusted a minute further than the clocks behind it. An afternoon of NTP configuration, one written zone policy, and a small quarterly ritual buy you the thing the opening story lacked. You get an archive where the order of events is simply true. The rest of the Syslog & Log Collection guides build on that foundation.


Put this into practice with SysPeep Monitor

Good clocks make your logs a trustworthy timeline; monitoring writes the other half of the record. SysPeep Monitor watches device health, interface status, and reachability over SNMP and WMI. Its alert history gives you a second account of when things happened — recorded on the monitoring server's clock, not the device's. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep