NTP and Clock Drift: The Weirdest Failure Modes in IT
Somewhere in your building right now, a clock is wrong. It is not dramatically wrong — computers do not usually leap to next Thursday. But it is wrong by a growing number of seconds, because every computer clock drifts, the way every cheap wristwatch drifts. Most of the time a service called NTP quietly corrects this and you never think about it. And then one day the correction stops working on one machine. Nobody notices. A few weeks later you are staring at a ticket that reads like a prank. A user can log into everything except one server, from one PC, except after a reboot, when it works for an hour.
Time problems produce the strangest symptoms in IT. Not the biggest — a dead switch is bigger — but the strangest, because almost nothing reports "the clock is wrong." Instead the wrongness leaks out through everything that silently assumed the clocks matched: logins, logs, certificates, backups, scheduled jobs. Admins who have been burned once develop a reflex; admins who have not can lose days.
This guide gives you the reflex without the burn. You will learn why computers need clock agreement even more than clock correctness. You will learn the four families of weird symptoms drift produces, and how NTP actually works in plain language. You will also learn a sane sync architecture for a small org. Then come the four checks that catch drift while it is still a graph instead of an outage.
In this guide
- Agreement matters more than correctness
- The four families of weird symptoms
- NTP in plain language
- A sync hierarchy for a small org
- Monitoring time: the four checks
- Virtual machines and their clock quirks
- Fixing drift with architecture, not whack-a-mole
- Where to go next
Agreement matters more than correctness
Here is the counterintuitive core of the whole subject: your systems care less about knowing the true time than about agreeing with each other. If every machine in the org were exactly four minutes fast, almost nothing would break. Meetings would start oddly early, but logins, logs, and jobs would hum along. That is because every timestamp comparison happens between machines that agree. Now instead make just one server four minutes fast while the rest are correct, and things start failing in ways that look supernatural.
The reason is that distributed systems constantly compare timestamps across machines. A login system stamps a credential "valid starting now, for the next few hours," and another machine checks that stamp against its own clock. A log collector interleaves events from fifty sources by their timestamps. A replication process decides which change is newer. Every one of those comparisons silently assumes the two clocks involved are close. The technical term for the difference between two clocks is skew (or offset), and the slow accumulation of skew is drift. Relative skew — machine against machine — is what breaks things first, which is why "but the time looks right on my screen" proves nothing. Right compared to what?
Absolute correctness does still matter. Certificates are checked against real-world validity dates, and evidence you hand to an ISP or auditor needs true timestamps. But if you remember one thing, remember this: time sync is an agreement protocol. Your job is keeping the whole org inside a tight window together.
The four families of weird symptoms
Logins fail for no visible reason
Modern authentication (Kerberos in Windows domains being the everyday example) is built on time-stamped tickets, precisely so that an intercepted ticket cannot be replayed later. The system tolerates a small amount of skew between machines — commonly five minutes by default — and rejects anything outside it. The failure mode is deliciously confusing: the user's password is correct, but the error says access denied, invalid credentials, or something equally misleading. Only the drifted machines fail, so it looks random. Reboots sometimes "fix" it briefly because many machines sync on startup, then drift again. If you have ever seen a login problem that follows certain computers around, you have probably met this one.
Logs that lie
When clocks disagree, your logs stop telling a coherent story. The firewall records a connection at 09:04 that the server swears it received at 09:01. Effects appear to precede their causes. During an incident this is poison — you reconstruct timelines from logs, and a lying timeline sends you chasing the wrong lead. It also quietly corrupts your monitoring history, since graphs are timestamp-ordered too. Making log timestamps trustworthy is its own small discipline, covered in the guide to timestamp sanity. The short version is that it begins and ends with NTP.
Certificates get confused
A certificate carries two dates — valid-from and valid-until — and every machine checks them against its own clock. A machine whose clock has fallen into the past will reject a perfectly good certificate as "not yet valid." A machine running fast will declare it expired early. The resulting errors blame the certificate, so admins chase renewals and re-issues while the actual problem ticks away on the client. Any time exactly one machine complains about a certificate the rest of the org accepts, check its clock before you touch the cert.
Scheduled jobs go feral
Backups, syncs, report generation, and cleanup scripts all fire on the clock. A bad clock can get corrected in one jump — say a machine that drifted twenty minutes finally reaches a time server. When that happens, the local time can lurch forward or backward. Schedulers react badly to time moving in ways physics does not allow. Jobs run twice, or not at all, or overlap a run still in progress. The tickets this produces ("the backup report is missing… sometimes") are maddening precisely because the scheduler is working exactly as designed, around a clock that teleported.
NTP in plain language
NTP, the Network Time Protocol, is one of the oldest services still running the internet, and its design is genuinely elegant. A machine running NTP asks one or more time sources "what time do you have?" It measures how long the question and answer took to travel. It uses that to estimate what the source's clock read at the exact moment of the exchange. Ask repeatedly, keep statistics, and you can figure out both your offset from the source and how fast your own clock runs. Then you can correct for both. Crucially, NTP corrects by slewing: nudging the clock gradually rather than jumping it, so time never visibly runs backward on a healthy system.
The one piece of jargon you must know is stratum. And it is simpler than it sounds: stratum is distance from the reference clock, counted in hops. At stratum 0 are the reference clocks themselves — atomic clocks and GPS receivers, which are not on the network. A server directly attached to one is stratum 1. A server that syncs from a stratum 1 server is stratum 2, and so on down the tree. Lower numbers mean closer to the source, but stratum is a distance, not a grade. A well-run stratum 3 server is more than accurate enough for an office network. There is one special value worth memorizing: stratum 16 means "not synchronized at all." A device reporting stratum 16 is not far from the reference clock; it is off the tree entirely, freewheeling on its own drift.
One more design detail pays off later. An NTP client with several sources compares them against each other and can vote out a source whose answers disagree with the majority. That source is a "falseticker," in the protocol's charming vocabulary. That only works with enough voters, which is why the old advice holds. One source is a leap of faith. Two is a tie you cannot break. Three or more is a quorum.
A sync hierarchy for a small org
You do not want every device in the building independently talking to time servers on the internet. It works, mostly, but it means dozens of firewall conversations. There is no internal consistency (each device lives at the mercy of its own path and sources). And there is no single place to look when time goes wrong. The standard architecture for a small or mid-size org is a two-layer pyramid:
- Layer one: two internal time servers. Pick two stable, always-on machines — a pair of domain controllers, or the firewall plus a core server. Configure them to sync from several reputable internet sources each. These two become the org's official clocks.
- Layer two: everything else syncs internally. Point every server, switch, access point, camera, and controller at those two internal servers — and nothing else. Windows machines that belong to a domain largely handle themselves. The domain forms its own time hierarchy, with one domain controller holding the time role at the top. You only configure that one machine's upstream sources, and members follow the domain.
In configuration terms the client side is two lines of intent, whatever the device's syntax looks like:
# On everything internal: use the two org time servers, nothing else server time1.corp.example iburst server time2.corp.example iburst # On time1/time2 only: sync outward to several public sources # (three or more, so a lying source can be outvoted)
The payoff of the pyramid is agreement by construction. Even if the internet path gets flaky and your internal pair ends up a hair off true time, the whole org drifts together. As we established, that is the thing that actually matters day to day. It also gives your monitoring a beautifully small surface: watch two servers hard, and spot-check that everyone else is following them.
Monitoring time: the four checks
Time is the purest example of a service where "the box is up" tells you nothing. The service can be running, reachable, and confidently wrong. These four checks ask the questions that matter.
Check 1: offset per device. Measure each important machine's clock offset against your internal time servers and record it over time. The resulting drift graph is the single most useful time artifact you can own. A healthy machine wobbles within a few milliseconds to tens of milliseconds. A machine whose graph shows a steady ramp is drifting and will eventually cross a tolerance line. The graph tells you weeks before users feel it. Alert when offset exceeds a threshold measured in seconds (one second is a generous line for office infrastructure; anything approaching a minute is a fire).
Check 2: source reachability. Your internal time servers are only as good as their upstream connection. Verify, on a schedule, that each one is successfully exchanging with its internet sources — not "can ping them," but "is actually syncing." Firewall changes are the classic silent killer here: someone tightens outbound rules, blocks the time protocol's port, and the time servers begin their long, quiet free fall.
Check 3: stratum changes. A device's stratum should be boring and constant. If your stratum 3 file server suddenly reports stratum 5, its path to the reference changed. In that case, something rewired itself, and while it may still work, you want to know why. If anything reports stratum 16, treat it as down: it is not syncing at all.
Check 4: the quietly free-running server. The nastiest failure in the whole domain: an internal time server that lost its upstream sources long ago but keeps answering clients from its own drifting clock. Clients see a working time server and follow it faithfully. The whole org marches in step, gradually away from the real world, until something that checks absolute time (a certificate, an external service) snaps. This is why check 2 exists, and why the server's own opinion of itself needs reading, not just its reachability.
| Check | Healthy looks like | Alert when |
|---|---|---|
| Offset per device | Milliseconds, wobbling flat | Above 1 second, or steadily climbing |
| Source reachability | Time servers syncing upstream every interval | Any internal time server misses several syncs in a row |
| Stratum | Constant, small, boring | Any change; stratum 16 anywhere |
| Free-run detection | Servers report valid upstream sources | A time server answering clients with no sources of its own |
The status commands make all four checks scriptable. On the ubiquitous Unix-style daemon, ntpq -p lists each source with its stratum, reachability, and offset. On Windows, w32tm answers the same questions:
$ ntpq -p
remote refid st t when poll reach delay offset jitter
==================================================================
*time1.corp.exa .GPS. 1 u 34 64 377 1.204 -0.512 0.087
+time2.corp.exa 10.0.0.9 2 u 29 64 377 1.891 0.344 0.101
C:\> w32tm /query /status (look at Stratum, Source, Last Sync)
C:\> w32tm /stripchart /computer:time1.corp.example /samples:3
Reading the ntpq lines: the * marks the source currently being followed. The + marks a good candidate. The st field is the source's stratum. The reach field is a rolling record of the last eight sync attempts. The value 377 means all eight succeeded, and anything less means recent misses worth investigating. offset is your distance from that source in milliseconds. A monitoring system can run these commands (or query the same data over SNMP where devices expose it) and alert on exactly the four conditions above.
Tip: alert on the trend, not just the threshold. An offset that doubles every day is a machine whose sync has silently stopped. Catching it at half a second costs you a two-minute fix. Catching it at five minutes costs you a morning of authentication weirdness first.
Virtual machines and their clock quirks
Virtualization deserves one honest paragraph, because VMs are where clock trouble breeds. A virtual machine does not own a real clock; it borrows slices of the host's. When the host is busy, a VM can be paused mid-tick. Time literally stands still for it, then lurches to catch up. Snapshots, live migrations, and resume-from-suspend can hand a VM a clock that is minutes or hours stale. On top of that, most platforms offer their own guest time-sync tools, which can fight with NTP inside the guest. Two authorities disagreeing about the time, on the same machine, is exactly the pathology this whole guide exists to prevent. The practical rules: give every VM exactly one source of time truth. Your platform's documentation will say whether that should be the host tools or in-guest sync against your internal servers. Follow it; the answer varies. Be extra suspicious of clocks right after snapshot restores and migrations. And monitor VM offsets exactly like physical ones, because "it should be handled automatically" is a hypothesis, and your drift graph is the test.
Fixing drift with architecture, not whack-a-mole
When drift surfaces, the tempting fix is to log into the offending box and force a sync. Do it during the incident, sure — but recognize it for what it is: whacking one mole. The machine drifted because its sync arrangement is broken or absent, and it will drift again on the same schedule. Fixing time for good is an architecture project, and a mercifully small one:
- Stand up the pyramid. Two internal servers syncing out to three or more sources; everything else pointed at the pair. If this exists already, verify it is what devices actually use. Audits regularly turn up devices still aimed at a server decommissioned years back, or at nothing.
- Sweep the strays. Printers, cameras, door controllers, lab equipment, and appliances all have their own time settings, set once at install and never again. Point them internally as you touch them. The drifters you cannot configure, note in your documentation as known liars.
- Watch the graph flatten. Turn on the four checks, then enjoy one of monitoring's rare aesthetic pleasures: a page of offset graphs settling into flat lines. Any line that will not flatten is a machine with a story — a blocked port, a fighting hypervisor tool, a failing hardware clock. Now you have the evidence to chase it.
There is a satisfying inversion at the end of this. Time, the service that produces IT's weirdest and most scattered symptoms, is also one of the cheapest to make boring. Two servers, four checks, one sweep of the strays — and an entire genre of impossible ticket quietly leaves your life.
Where to go next
Clock agreement underpins the other services in this pillar. The directory rejects logins outside its skew tolerance (see LDAP and Active Directory health). When authentication breaks org-wide, clocks are one of the first hops in the authentication-path playbook. For how time fits into the whole family of invisible foundations, start at the dependency iceberg. It also explains why watching them pays off so disproportionately. Or browse everything in Core Services: NTP, RADIUS, LDAP/AD.
Put this into practice with SysPeep Monitor
Your time servers are ordinary machines until the day they quietly are not. SysPeep Monitor auto-discovers them along with the rest of your network. It watches their health and reachability over SNMP, WMI, and SSH. It lets you set thresholds that turn slow, silent drift into an early, actionable alert — before the login tickets start.
Download Free TrialLearn About SysPeep