Route Stability: Catching Flaps and Churn
Some network problems announce themselves. A dead switch is a fire alarm: loud, unambiguous, over quickly one way or another. A route flap is the opposite — a mumbler. Users report ghosts: "it froze for a bit, then it was fine," "the branch office dropped for a minute this morning," "the call got weird around ten." You check the graphs, the graphs look clean, and you are left investigating a crime with no body. Few things burn more admin hours than a problem that heals itself right before you look at it.
What is actually happening, more often than people expect, is that a road keeps appearing and disappearing from the network's map. A route — one entry in the routing tables your routers share — is being learned, withdrawn, learned again, withdrawn again. Every cycle forces every router to redraw its map, and every redraw is a brief moment of network-wide confusion. When these changes become a steady background hum instead of rare events, admins call it churn. It is one of the most user-visible, monitoring-invisible conditions a network can be in.
This guide makes it visible. You will learn what a flap is and why it costs so much more than it seems to. You will see why users feel it long before a ping check does and learn the four usual causes. Then come the three signals: route counts, recalculation counters, and log patterns. These turn "vague weirdness" into a timestamped, diagnosable event. Along the way we will demystify damping, the routers' own crotchety response to flapping roads.
In this guide
- What a route flap actually is
- What users feel (and why ping barely notices)
- The four usual causes
- Three signals that expose churn
- Damping: when routers hold a grudge
- Correlating churn with circuit events
- Baselines and starting thresholds
- Where to go next
What a route flap actually is
Quick grounding, with the full story in the plain-language routing guide: every router keeps a routing table — its map. On a dynamically routed network, entries in that map appear when a router learns a path from a neighbor. They are withdrawn when the source of that path goes away. A route flap is one of those entries cycling: installed, withdrawn, installed, withdrawn, sometimes for hours. Picture a road crew that opens and closes the same bridge every few minutes, all day, and dutifully updates every map in the region each time. The bridge is technically fine most of the time. The region is not.
The expensive part is that no flap is local. When a route is withdrawn, the router that lost it tells its neighbors, who tell theirs. Every router in the network recalculates its map — a process called convergence. This takes real time and has real side effects (its own guide covers them). A single flapping route in a small network means the entire network is redrawing its maps continuously. The routers spend their afternoons doing arithmetic, and your users spend theirs experiencing the arithmetic.
Flaps come in two sizes. Sometimes one route flaps — a single subnet behind one unstable link blinks in and out while everything else holds steady. Sometimes an entire neighbor flaps. Every route learned from it — possibly hundreds — vanishes and returns as a block, over and over. The neighbor-sized version is covered from the session angle in monitoring neighbor state. This guide watches the same disease from the map's point of view. To users, both sizes feel identical: intermittent, unexplainable weird.
It is tempting to shrug at this — each individual flap heals in seconds, so how bad can forty of them be? Worse than forty times a small thing. The recalculations burn router CPU, which on a busy device can push it toward the overload that causes more flapping. Each redraw window is a fresh chance for a dropped call or a reset connection, so user pain accumulates linearly while goodwill drains exponentially. And a route that flaps toward the wider world can get itself formally punished by upstream routers. It can be suppressed for a stretch even after it stabilizes. We will meet that mechanism properly in the damping section. Flapping is not a series of small outages. It is one long, low-grade outage wearing a disguise.
What users feel (and why ping barely notices)
Route instability produces a very particular flavor of complaint, and learning to recognize the flavor is half the diagnostic battle. The destinations behind the flapping road are intermittently unreachable; everything else is fine. The outages are short — seconds to a couple of minutes — and end on their own. And the pattern repeats. Specific shapes to listen for:
- "It dropped for like thirty seconds, then came back." Repeatedly. Usually about the same branch, subnet, or service each time — the one behind the unstable route.
- Long-lived connections suffer most. VPN tunnels re-establish over and over, database connections reset, remote desktop sessions freeze and resume. Quick web requests often squeak through between redraws, which makes the reports even more confusing: "browsing is fine but the app keeps disconnecting."
- Brief stalls during each redraw. While routers converge, some traffic takes a moment of loss or a detour. Users describe it as a hiccup, if they describe it at all.
- The retry always works. By the time anyone investigates, the road is back on the map. Trouble that evaporates under observation is churn's signature move.
Now consider your ping checks. A check that runs once a minute against one address straddles most flap windows entirely. When it does land inside one, it fails once, then recovers before the confirmation retry. It either never alerts or self-resolves with a cheery "back to normal." Repeated often enough, that trains you to ignore that device. This is not a flaw in ping monitoring; it is the wrong instrument for the job. Reachability checks watch destinations. Churn lives in the map, so the map is what we will watch.
One interviewing trick while you are here: when a vague report arrives, ask two questions. "When it happens, is it everything, or just certain things?" and "Does it come back on its own?" Answers of just certain things and yes, always are churn's fingerprints. They justify going straight to the routing signals below. You can skip the afternoon spent exonerating the wireless, the server, and — inevitably — the printer.
The four usual causes
A route flap is almost never the disease. It is a symptom, and the disease is usually one layer below or one router over. Four suspects account for nearly all of it:
- An unstable circuit. The most common by far. A physical link flaps — failing optics, damaged cable, a marginal provider circuit. Each drop kills the neighbor session riding it, which withdraws the routes. They return when the link blinks back. Link flap becomes route flap, one layer up. The physical-layer half of this story lives in flapping links. When you find route churn, that guide is usually your next stop.
- An overloaded router. A router whose CPU is pinned can be too busy to send or process hello messages on schedule. Its neighbors then declare it dead. When the CPU breathes, the sessions re-form. There is a vicious little cycle here: recalculating routes costs CPU, so the flapping itself feeds the overload that causes the flapping.
- Misconfiguration. Two devices claiming the same address or router identity will fight over it, each installing and overwriting the other's version of reality. Slightly mismatched session settings can let a neighbor relationship form and then collapse, endlessly. These flaps are metronome-regular, which is itself a clue — hardware fails messily, configs fail rhythmically.
- Somebody else's problem. If your routing extends over provider WAN links or site-to-site tunnels, a far site's unstable circuit exports its churn into your map. The flapping road is not even in your city, but every map in your building gets redrawn on its schedule.
Your monitoring's job is not to pick the suspect from an armchair. It is to make the pattern visible and hand you timestamps, because — as we will see below — timestamps are what convict.
Three signals that expose churn
None of these require deep protocol knowledge, special licenses, or new tools. All three come from things this library already uses. One source is SNMP polling (the standard way monitoring systems question network gear). The other is the log messages your devices send to a collector.
Signal 1: route table size over time
Poll each router's total route count on a schedule and graph it. On a healthy network the line is flat for weeks at a stretch — boring is the entire goal. Churn draws a sawtooth: the count dips when routes are withdrawn and recovers when they return, over and over. You may not know which route is flapping yet, but the wiggle itself is the alarm. And the depth of each dip tells you how many routes travel together (one route, or one whole neighbor's worth). The broader craft of watching table size and contents has its own guide.
Signal 2: recalculation counters
Routers count their own map redraws. Interior protocols in particular keep a counter of topology recalculations, readable over SNMP or in a status command. It ticks up each time the router reruns its path math. The absolute number means nothing — it has been counting since boot — but the rate means everything. A quiet network recalculates a handful of times a day, usually zero. Tens of recalculations per hour means the map is being redrawn as a hobby, and something, somewhere, is flapping.
Signal 3: log-based flap detection
Every withdrawal and every neighbor transition produces a log line, and each line looks innocent alone. Counting them is what exposes the pattern. Have your log system count route and adjacency messages per device per hour and alert on a spike. The mechanics are in alerting on logs. This is the signal that names the culprit, because the lines carry specifics:
09:14:22 core-rtr: ROUTE: 10.44.0.0/16 withdrawn (neighbor 10.10.5.2 down)
09:16:05 core-rtr: ROUTE: 10.44.0.0/16 installed (neighbor 10.10.5.2 up)
09:21:48 core-rtr: ROUTE: 10.44.0.0/16 withdrawn (neighbor 10.10.5.2 down)
09:23:31 core-rtr: ROUTE: 10.44.0.0/16 installed (neighbor 10.10.5.2 up)
...the same pair, forty more times before lunch...
Where should these signals come from? You do not need to instrument every device. Churn is network-wide by nature — every router redraws. So a single good vantage point, usually your core router or layer 3 core switch, sees the whole show. Adding a second vantage point at the WAN edge buys you direction. Churn visible at the edge but calm in the core points outward at a provider or a far site. The reverse points inward at your own gear. Two well-chosen watchers beat twenty noisy ones.
| Signal | Where it comes from | Quiet network looks like | Worth investigating |
|---|---|---|---|
| Route table size | SNMP poll, graphed | A flat line for weeks | Any repeated dip-and-recover sawtooth |
| Recalculation counter | SNMP poll or status command | A few ticks per day, often zero | A steady climb; tens per hour |
| Route/adjacency log lines | Your log collector | Near silence outside changes | The same route or neighbor appearing hourly |
Damping: when routers hold a grudge
Routers have their own opinion about flapping roads, and it is worth one gentle paragraph because someday it will explain a mystery. The mechanism is called damping (or dampening): a router can be told to hold a grudge against an unstable route. Each flap adds to a penalty score. Past a threshold, the router suppresses the route — refuses to use or share it. That holds even when it comes back and swears it is fine. The penalty fades with good behavior, and eventually the road is readmitted to the map. It is the routing equivalent of a teacher making the class troublemaker sit in the hall until they can settle down. It exists mostly at network borders. Your provider very likely applies some form of it toward the wider internet.
What you need as a monitoring person is not to configure damping, but to recognize its fingerprints. It explains a spooky case: the circuit is confirmed fixed, the link is clean, and the neighbor is up. Yet one destination stays unreachable for a while longer anyway. The road is open; the map is still refusing to trust it. That is not a new failure. It is the punishment phase of the old one, and it ends on its own. The honest tradeoff: damping protects the whole network's stability at the temporary expense of the flapping destination. That is usually — not always — the right trade.
Correlating churn with circuit events
Since churn is almost always an echo of a lower layer, the single most valuable diagnostic skill is lining up timestamps across layers. Here is what a solved case looks like, assembled from a switch log, a router log, and one user complaint:
09:14:02 access-sw: port 24 (uplink to branch) DOWN
09:14:22 core-rtr: neighbor 10.10.5.2 down; 10.44.0.0/16 withdrawn
09:14:30 core-rtr: route recalculation complete
09:16:01 access-sw: port 24 UP
09:16:20 core-rtr: neighbor up; route installed
...cycle repeats every few minutes...
09:47:00 ticket: "branch office keeps dropping"
Read it bottom-up and the story tells itself: the port event precedes the routing event by seconds, every single cycle. The route churn is an effect; the flapping uplink is the cause. The fix is a transceiver, a cable, or a stern conversation with a circuit provider — not anything in the routing configuration. Without the timeline, this same case reads as "the branch app is flaky," and people start restarting servers. Two practical habits make these timelines assemble themselves. Send switch logs and router logs to the same collector. And keep every device's clock synced to a common time source. Correlation dies the moment your devices disagree about what time it is.
And when the timeline refuses to line up — route churn with no port event underneath it — that absence is evidence too. No lower-layer cause shifts suspicion to the other suspects. Pull up the CPU graph on the routers involved. Look for load spikes that match the flap times (the overload pattern), or study the rhythm itself. Failures of hardware are messy and irregular. Failures of configuration are eerily periodic, the same interval every time. A timer somewhere is doing exactly what it was told. A flap you can set your watch by was configured by somebody.
Tip: test your churn detection before it matters. In a maintenance window, bounce a lab port or a non-critical link a few times. Confirm the sawtooth appears on your graph, the counter rate climbs, and the log alert fires. A detector you have watched catch a fake crime is one you will trust during a real one.
Baselines and starting thresholds
Here is the pleasant surprise: on a small network, healthy routing is so quiet that alerting is easy. The map should change when you change the network, when something fails, and basically never otherwise. That makes your baseline effectively zero. Your first alert rules are simple. Outside your maintenance windows, warn on any route-count movement that repeats and on any sustained rise in recalculation rate. If your routing extends over provider services or many sites, some background churn will be imported from the wider world. In that case, watch the signals for a couple of weeks. Note the normal hum, and set thresholds a comfortable margin above it.
Here are reasonable starting points, to be tuned rather than worshipped. Warn when the route count deviates from its baseline for more than ten minutes or dips more than a few times in an hour. Warn when recalculations exceed a handful per hour. Alert when the same route or neighbor generates more than a few transitions in an hour. That last one is a flap by definition. And suppress all of it during planned maintenance. A change window is churn on purpose, and alerts you expect are noise you scheduled.
Where to go next
Route stability in one breath: flaps are roads blinking in and out of the map. Users feel every blink while ping shrugs. The cause is usually a layer down. Three cheap signals — route counts, recalculation rates, and counted log lines — turn the ghost into a timestamped suspect. Boring, flat graphs are the victory condition.
From here, read the convergence guide next. Every flap you just learned to catch is a convergence event. That guide covers what happens during each one and how to measure it. The rest of Routing Protocol Health picks up the neighbor sessions and the routing table itself from there.
Put this into practice with SysPeep Monitor
Route counts and recalculation counters are just numbers your routers will hand over via SNMP. The trick is polling them faithfully and noticing when they move. SysPeep Monitor polls your gear on a schedule and graphs the values so a sawtooth has nowhere to hide. It alerts when a threshold is crossed. The free trial sets up in minutes on a Windows machine.
Download Free TrialLearn About SysPeep