Detecting Path Changes Before Users Notice
Here is an uncomfortable fact about networks: the road your traffic takes can change completely, at any hour, without a single alarm going off. The destination still answers. Every status light stays green. And yet your packets are suddenly riding a backup circuit, or a detour through a distant city. The first person to find out is a user who wonders why the video call sounds like a robot underwater.
Paths change silently because nothing in the ordinary monitoring stack watches them. Up/down checks ask "does it answer?" — not "which road did the answer take?" This guide closes that gap with a simple discipline. Capture the path on a schedule, compare each capture against your recorded normal, and raise a hand the moment the road diverges. You will see why paths shift in the first place and what a change looks like from your side of the wire. You will learn how to build a scripted detector in an afternoon and how to keep it from crying wolf. And you will learn what to actually do when it fires.
One prerequisite, and it is non-negotiable: change detection compares against a recorded normal, so you need one. If you have not yet built a path baseline and register, start with know your normal. Everything below assumes those captures and that register exist.
In this guide
- Why paths change when "nothing changed"
- What a path change looks like from your side
- The detection loop: capture, normalize, compare
- A scripted detector you can build in an afternoon
- Taming the false alarms
- When the alert fires: a response checklist
- Same road, worse ride: the latency dimension
- Where to go next
Why paths change when "nothing changed"
Nobody on your team touched anything, and yet the road moved. That is normal, because most of the road does not belong to you — and even the part that does can rearrange itself by design. The usual suspects:
WAN failover. If your edge has a primary circuit and a backup, it is built to switch over automatically when the primary falters. That backup may be a second provider, a broadband line, or a cellular link. The automatic switch is the feature working. The catch: failover is designed to be seamless, so it is also silent. A brief wobble on the primary at three in the morning, and by breakfast the whole office is riding the backup. It is slower, maybe metered, maybe fine — and invisible unless something watches the path. The evil twin is the silent non-failback. On many setups the return to the primary needs a healthier signal, a timer, or a human. So you can run on the backup for weeks after the primary recovered. Everyone notices the outage; nobody notices the recovery that never happened.
Provider rerouting. Your internet provider and the carriers beyond it re-engineer constantly. There are maintenance windows, fiber cuts routed around, congestion shifted, and peering arrangements made and unmade. Peering arrangements are the agreements about where two networks exchange traffic. Any of that re-engineering can move your packets onto a different road through the middle of the internet. Often you gain or lose a few milliseconds and nobody cares. Sometimes the detour adds real latency or crosses a congested exchange. Then your application quietly degrades while every device you own reports perfect health.
Routing flaps. When a link somewhere is unstable, routers around it may switch between alternatives repeatedly — path A, path B, path A again, minutes apart. Each individual road might be fine; the constant lane-changing is not. Flapping shows up as captures that disagree with each other run after run. It deserves attention even when each variant looks healthy, because somewhere a link is sick.
Your own network. A colleague edits a static route, a firewall policy steers traffic through a different tunnel, a VPN renegotiates to a secondary gateway. Change detection does not care that the cause was in-house; it catches the unannounced internal change just as happily. Those are often the fastest mysteries to solve, once you know the road moved.
The far end moved. Sometimes every hop you own and every provider in the middle stays put. In that case, the change is at the destination. A hosted service shifted your sessions to a different data center, or its content network began answering from a node in another region. The tail of the trace changes — new final hops, a different destination address entirely if you trace by name. Latency moves with the geography. It is worth recognizing this shape, because no amount of shouting at your own provider fixes it. The road changed because the building at the end of it moved.
What a path change looks like from your side
You cannot see a provider's routing tables, and you do not need to. Every change that matters leaves fingerprints in an ordinary traceroute, and each fingerprint suggests its own story:
| Signal in a scheduled capture | What it usually means | How urgent |
|---|---|---|
| The first external hop changed | WAN failover — you are leaving through a different door | High: confirm the primary's state and the failback plan |
| New provider domain in mid-path DNS names | Provider rerouted, or a peering change moved your traffic to another carrier | Medium: check what the ride costs now |
| Hop count jumped noticeably | A longer detour is in place somewhere | Medium: detours often add latency |
| Same road, sustained latency step | Congestion or a slower link inside an unchanged sequence of hops | Medium: users feel this one directly |
| Loss begins at a hop and persists to the end | A fault, not a reroute | High: switch to the localization playbook |
| Hops alternate between two values run to run | Load-balanced parallel links (ECMP) — usually normal | Low: record as known variation in the register |
Notice that the strongest signals are about identity, not performance: which addresses, which provider names, how many hops. That is what makes path-change detection scriptable — comparing two lists of addresses is something a computer does perfectly, every fifteen minutes, forever.
The detection loop: capture, normalize, compare
The whole mechanism is one loop with four steps, and every implementation — from a shell script to a commercial platform — is a variation on it:
- 1. Capture on a schedule. Run a traceroute to each register destination from the same source machine the register names. Every fifteen minutes is a sensible default: frequent enough to beat most users to the discovery, gentle enough to be polite to the routers answering you.
- 2. Normalize the output. Strip everything that legitimately varies between healthy runs: the per-hop latencies, the occasional unanswered probe. Keep the part that defines the road: hop numbers and addresses. Comparing raw output means alerting on noise; comparing normalized output means alerting on roads.
- 3. Compare against a golden path. The golden path is a normalized capture from a healthy day, blessed by a human as "this is the road we expect." It is the executable form of your register entry: the register explains normal to people, the golden file states it for scripts.
- 4. Raise a hand on divergence. When the current capture does not match, tell someone — with the diff itself in the message. The diff is the news: which hops disappeared, which appeared, where the road forked.
The elegance of the loop is that it never needs to understand routing. It cannot tell a failover from a peering change — but it does not have to. It only has to notice that today's road is not yesterday's road and wake up a human who can read the fingerprints in the table above.
A scripted detector you can build in an afternoon
Here is the loop as a small shell script, suitable for a Linux machine or monitoring probe. It captures one destination, normalizes to hop-and-address, compares with the golden file, and emails the diff on divergence:
# path-watch.sh - alert when the road to a destination changes
DEST=203.0.113.30
BASE=/var/lib/pathbase
CUR=$(mktemp)
# capture and normalize: keep hop number and address only
traceroute -n -w 2 -q 1 "$DEST" | awk 'NR>1 {print $1, $2}' > "$CUR"
if ! diff -q "$BASE/golden-$DEST.txt" "$CUR" >/dev/null; then
{
echo "Path to $DEST diverged from golden at $(date)"
diff -u "$BASE/golden-$DEST.txt" "$CUR"
} | mail -s "PATH CHANGE: $DEST" netops@corp.example
fi
rm -f "$CUR"
A quick tour of the choices. -n skips name resolution so the output is stable and fast. -q 1 sends a single probe per hop, plenty when you only want addresses. The awk line drops the header and keeps hop number plus address, which is the normalization step. And diff -u produces the compact before/after listing that lands in the email. You create the golden file once, on a healthy day, by running the same capture-and-normalize pipeline and copying the result to golden-$DEST.txt. Wrap the script in a loop over your register's destinations, schedule it every fifteen minutes, and you have working change detection. The same idea ports to Windows with tracert -d, a few lines of PowerShell to strip the latency columns, and Compare-Object for the diff.
One easy upgrade before you call it done: have the script append every normalized capture to a dated archive, not just the ones that diverge. The alert tells you that the road changed. The archive tells you when, to the quarter hour. That is exactly the question a provider will ask, and exactly the evidence that pairs with your users' "it got weird around lunchtime." Text files this small cost nothing to keep for months, and a timeline you can quote beats a memory you have to defend.
When it fires, the email reads like this — and after a little practice you will interpret one at a glance:
Path to 203.0.113.30 diverged from golden at Tue 07:15 --- golden-203.0.113.30.txt +++ current 1 10.0.1.2 -2 198.51.100.1 <- fiber handoff gone... -3 198.51.100.77 +2 192.0.2.65 <- ...replaced by the backup provider +3 192.0.2.9 4 203.0.113.5
Minus lines are the road you expected; plus lines are the road you got. In this particular shape, the first external hops are swapped for a different provider's addresses. It is the classic signature of a WAN failover. The failover is caught at seven in the morning instead of discovered at nine by thirty slow-motion complaints.
Taming the false alarms
Run the naive version for a week and you will meet its two failure modes: alerts about healthy variation, and alert storms from flapping. Both have simple cures.
Healthy variation comes mostly from ECMP — equal-cost multipath, the provider habit of spreading traffic across parallel links. So consecutive captures alternate between sibling routers at the same positions. Your baseline work already identified these alternates as known variation. Teach the detector the same tolerance. Keep one golden file per known-good variant — golden-a, golden-b. Alert only when the capture matches none of them. Two or three variants cover most destinations, and each new legitimate variant gets blessed into the collection the first time a human confirms it.
Flap storms are the opposite problem: the road really is changing, sixty times an hour, and your inbox pays for it. The cure is to alert on transitions, not states. Remember whether the path was already known-divergent. Send one message when it first leaves the golden set, and one more when it returns. Between those two events, stay quiet and keep logging. A related dampener is worth the extra minute: on first divergence, wait briefly and capture again before mailing. That way, a single mid-reroute snapshot — caught while the internet was literally changing its mind — does not page anyone.
Tip: updating a golden file is a human act, not an automated one. The moment a script "helpfully" re-blesses whatever road it currently sees, your detector stops detecting and starts agreeing. Bless new goldens deliberately, date the change in the register, and keep the old file. A history of golden paths is a history of your network's detours.
Finally, prove the whole contraption works before you trust it, because a detector that has never fired is a hope, not a control. During a planned maintenance window, make a path change happen on purpose. Trigger your edge's failover to the backup circuit. Most dual-WAN devices offer a test action, and unplugging the primary works where they do not. Then watch for four things. The alert arrives, and the diff correctly shows the backup provider's addresses. The ride numbers over the backup get recorded (you want to know what failover mode feels like before a real one). And the return-to-golden message appears after you restore. Fifteen minutes of drill buys you something rare: certainty that the silent failure mode this whole guide exists for will actually make noise.
When the alert fires: a response checklist
A path-change alert is a fact, not a verdict: the road moved, and someone should decide whether that matters. Walk the same short list every time and the decision takes minutes:
- 1. Confirm it is real and current. Run a fresh capture. If the road already snapped back, note the blip — blips that repeat are a flap in the making.
- 2. Measure the ride, not just the road. Ping the destination and compare latency and loss against the register's ranges. A detour with identical performance is trivia; a detour that doubled latency is an incident.
- 3. Classify from the fingerprint. First external hop changed — failover: check the primary circuit's interface status and alarms at your edge. New provider names mid-path — reroute out in the world. Alternating captures — instability, yours or theirs.
- 4. If intended or harmless, bless it. Update the register with the date, add or replace the golden file, and close the loop. Ten quiet minutes, and your normal is current again.
- 5. If it hurts, escalate with the evidence you now hold. A before/after diff plus latency numbers from both eras is precisely the package a provider ticket wants. And if the new road also carries loss, the localization playbook tells you where along it to point.
- 6. Watch for the road back. Failbacks are as silent as failovers. Keep the destination on a mental watch list until the path returns to golden or the new road is declared the new normal.
One more habit that pays: when an alert coincides with a maintenance window — yours or a provider's announced one — say so in the ticket or the team channel. Half the value of change detection is being able to write "path change at 02:10, matches the carrier's maintenance notice, performance unchanged" before anyone asks.
Same road, worse ride: the latency dimension
Address-diffing has one blind spot, and you should build for it deliberately: the road can stay identical while the ride degrades. Same hops, same providers — and forty extra milliseconds, because a link along the way is congested or a carrier's internal capacity moved. Your diff-based detector will sleep straight through it.
The companion measurement is continuous latency and loss tracking against the baseline's ranges. A long-running MTR session shows the per-hop shape of a step-change when you are investigating. For standing coverage, a SmokePing-style latency grapher draws every destination's ride around the clock. A sustained shift above the register's range is your alert. Pair the two detectors and the whole space is covered. The diff catches identity changes, the grapher catches performance changes, and between them the road cannot move or rot without a hand going up. A monitoring platform's latency thresholds can stand in for the grapher's alerting role. The principle is the same: watch the number against a known range, not against hope.
And remember the road you cannot see from your side. That is the return path, which can change independently of the outbound one and carry its own detours. Detecting that is a job for two-ended measurements, covered in asymmetric routing.
Where to go next
The discipline in one breath: paths change silently. So watch them on purpose with scheduled captures, normalized and diffed against human-blessed golden paths. Use tolerance for known variation and transition-based alerting. Add a response checklist that ends in "bless it or escalate it," and a latency grapher covering the blind spot. None of it is exotic. All of it turns "the internet felt weird last week" into a dated, diffable record of exactly when the road moved and what it cost.
If you arrived here without a baseline, build one first. The detector is only as good as the golden it compares against. And when a detected change turns out to hurt, the rest of the Path & Hop-by-Hop Analysis guides carry the investigation from "the road moved" to "here is the segment, here is the owner, here is the evidence."
Put this into practice with SysPeep Monitor
A diff script tells you the road changed; monitoring tells you what the change did to the ride. SysPeep Monitor watches latency, packet loss, and WAN interface health around the clock. It has thresholds and dependency-aware alerts that stay quiet until something real moves. Put it beside your path detector and the free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep