Home › Topics › Routing Protocol Health › Neighbor State

Monitoring Neighbor State: The Heartbeat Between Routers


Of all the things this library will ask you to monitor, a neighbor session may be the most invisible. Users never see it. When it is healthy, it produces nothing but a trickle of tiny messages between two routers. And when it dies, there is often no immediate symptom at all, because a redundant path quietly absorbs the traffic and the office carries on. That silence is exactly the problem — and exactly why watching neighbor state is the single most valuable routing check you can set up.


The good news is that it is also one of the easiest. Routers are downright eager to talk about their relationships. They keep tables of every neighbor and its current state. They announce every change in a log message the moment it happens. The hard part is not collecting the data. The hard part is knowing which neighbors should exist, and taking their loss seriously at the moment when nothing appears to be broken.


This guide covers both. You will learn how the heartbeat between routers works and the three-state ladder every session climbs. You will learn how to watch sessions by polling and by listening for events. You will learn how to build the expected-neighbor inventory that makes real alerting possible. And you will learn how to think about severity. It closes with a worked example: one neighbor-down alert, followed from first notification to closed ticket.


In this guide

  • The heartbeat between routers
  • The state ladder in plain language
  • Watching by polling: the neighbor table
  • Watching by events: traps and log messages
  • The expected-neighbor inventory
  • Alerting: severity when nothing seems broken
  • Flapping neighbors
  • A worked example: the quiet Tuesday drop
  • Where to go next

The heartbeat between routers


A quick refresher from the plain-language routing guide: when two routers run a dynamic routing protocol, they introduce themselves and form a relationship. It is called a neighbor, an adjacency, or a peer depending on the protocol. They exchange routes over that relationship. To keep that relationship alive, each router sends a small hello message on a fixed rhythm, every few seconds. Each side also runs a countdown called the hold time. Every hello it hears resets the clock. If the clock ever runs out, it declares the neighbor dead and throws away every route it learned from it.


Why such a blunt mechanism? Because from a router's point of view, silence is ambiguous. A neighbor that has nothing new to say and a neighbor that burned down look identical: both send nothing. The heartbeat removes the ambiguity by making silence itself meaningful — a healthy neighbor is never quiet for long, so a quiet neighbor is a dead one. There is a tradeoff buried in the timer values. Short timers detect failure fast but panic over brief hiccups. Long timers stay calm but leave the network aiming traffic at a corpse for longer. Your gear ships with middle-of-the-road defaults, and you do not need to touch them to monitor them.


Here is what that design gives you for free: the routers are already running a continuous, mutual health check on every routing relationship in your network. Your monitoring system does not need to invent a heartbeat — it only needs to observe the results. Two observations, specifically: the current state of each session, and the moments when a state changes. Everything in this guide is one of those two.


The state ladder in plain language


Every routing protocol defines a ladder of internal states a session climbs as it forms. Protocol nerds can name each rung the way sommeliers name grapes. You do not need the rungs. For monitoring, every session in every protocol is in one of three places. It can be down (no conversation at all). It can be establishing (the routers are talking, but not yet exchanging routes). Or it can be up (often labeled Established or Full — the maps are exchanged and the heartbeat is steady). Here is the ladder as your monitoring will see it:


What you seeWhat it meansWhat to do
DownNo hellos are being heard; the far router is dead, unreachable, or not speaking the protocolInvestigate now — this is a broken relationship, whatever the cause
Establishing (any in-between state)Introductions in progress; completely normal for a few secondsNothing, briefly — but a session stuck here for minutes is failing to form and deserves its own alert
Up / Established / FullRoutes exchanged, heartbeat steadyNothing. This is the good place. Enjoy it

The middle rung deserves one extra beat of attention, because it is the sneaky one. A session passing through the establishing states for a moment is a session being born. A session living there is a session that cannot finish its introductions. One classic cause is mismatched settings between the two ends — the routers are speaking slightly different dialects and politely failing to connect. Another is a firewall rule quietly eating the protocol's packets. Configure your monitoring to treat "not up for more than a few minutes" as an alert in its own right, not merely as the absence of "down."


Watching by polling: the neighbor table


The first way to watch sessions is to ask on a schedule. Routers expose their neighbor lists through SNMP (Simple Network Management Protocol, the standard question-and-answer language between monitoring systems and network gear). They use standard catalogs of questions that include each neighbor's address and current state. Your monitoring system polls the table every one to five minutes, compares what it sees against what you expect, and raises a flag on any mismatch. The same information is what the router prints for humans with its summary commands, which are worth knowing for spot checks:


edge-rtr# show bgp summary
Neighbor        State         Up for     Routes received
203.0.113.1     Established   6w2d       812
203.0.113.9     Established   00:03:11   809    <- recently reset!

Notice what is worth recording beyond the bare state. The session age ("up for") tells you how long the relationship has been stable. And a young age on a session you did not restart is a confession. The routes received count catches a subtler failure: a session can be up and useless, with the far end connected but sending nothing. If a peer normally sends hundreds of routes and today sends zero, the heartbeat is fine but the map exchange is broken — up, but mute. Poll all three where your gear offers them: state, age, and route count.


How often? Every one to five minutes, the same rhythm as the rest of your monitoring — faster mostly loads the router for nothing, slower leaves long blind gaps. And one twist is worth knowing before you build anything. On many devices, a fully dead session does not appear in the table marked "down" — it simply vanishes from the table. Your check must treat "missing from the list" as down, not as "nothing to report." This is a classic gap in home-grown scripts, which loop over the neighbors that exist and can never miss the one that does not. Checking against your expected list, rather than against the router's current list, closes the hole.


Polling's virtue is stubbornness: it asks the same questions forever and eventually notices everything. Its weakness is the gap between polls. A session that drops and recovers inside your five-minute window can look continuously healthy in every sample. Meanwhile, behind the scenes the network redrew its maps twice and users felt both redraws. For that, you need the second method.


Watching by events: traps and log messages


The second way is to let the routers tell you the moment something changes. They do this in two overlapping voices. SNMP traps are unsolicited messages fired at your monitoring system when a session changes state. Syslog messages are log lines sent to your log collector describing the change in words. Either way, the notification is immediate — no waiting for the next poll — and it usually carries something polling cannot give you: the reason.


Mar 14 14:07:31 core-rtr: ADJACENCY: neighbor 10.10.5.2
    changed state from Full to Down (hold timer expired)
Mar 14 14:09:02 edge-rtr: PEER: 203.0.113.9 session
    closed by remote end

Read those reasons the way a detective reads footprints. "Hold timer expired" means the link stayed up but the far router went silent — look at the far router, not the cable. "Interface down" as the reason means the physical link itself dropped — look at the cable, the optics, the circuit. "Closed by remote end" means the far side deliberately hung up, which usually means a human or a reboot on the other end. One log line, and your investigation already has a direction. The catch: event delivery is fire-and-forget, and a trap lost in transit is simply gone. So run both methods — events for speed and diagnosis, polling for ground truth — and treat them as one check with two inputs. The mechanics of turning log lines into alerts are covered in alerting on logs.


Two bits of plumbing make the event side work, and both are famously forgotten. First, each router has to be told where to send. Point its syslog output at your log collector and its traps at your monitoring system. These two one-minute settings are skipped on new devices constantly, so put them on your build checklist. Second, somebody has to be listening: a trap that arrives at a system with no matching rule vanishes without a ripple. Send a test event when you set this up, and confirm it actually becomes an alert, not just a line in a file nobody reads.


The expected-neighbor inventory


Here is the step most people skip, and it is the one that turns data into monitoring. Your tools can see every neighbor that currently exists — but only you know which neighbors are supposed to exist. Without that list, a vanished session is invisible. The router does not mourn a missing neighbor for long. The table simply gets shorter, and nothing looks wrong. The expected-neighbor inventory is a small, boring table that fixes this, and it takes an hour to build:


  • Step 1: list every device that runs a routing protocol — routers, firewalls, and layer 3 switches (switches that also route).
  • Step 2: for each one, record every neighbor it should have and the link that relationship rides over.
  • Step 3: note what each adjacency carries — "the default route" or "building two's subnets" — so future-you knows what is at stake.
  • Step 4: enter each pair into monitoring as an explicit check: "session core-rtr to wan-rtr must be up."
  • Step 5: add "update the neighbor inventory" to your change checklist, so the list stays honest as the network grows.

RouterExpected neighborOverCarries
core-rtrwan-rtrFiber uplinkDefault route toward the internet
core-rtrbldg2-swCampus fiberBuilding two's subnets
wan-rtrProvider edge 203.0.113.1Primary circuitInternet (BGP)
wan-rtrProvider edge 203.0.113.9Backup circuitInternet, standby (BGP)

Notice what you just made as a side effect: documentation. At three in the morning, mid-incident, this table answers "what is that session for and how bad is losing it?" faster than any diagram you own.


If your network is small, the exercise might produce four rows, and you may feel a little silly building a table for it. Build it anyway. Four sessions means each one carries a quarter of your redundancy story. The smaller the shop, the better the odds that nobody else on earth knows these relationships exist. The hour you spend here is future-incident time, prepaid at a steep discount.


Alerting: severity when nothing seems broken


Now the judgment call this guide has been building toward. When an expected neighbor goes down and traffic keeps flowing over a surviving path, every instinct says "low priority — nothing is broken." Set that instinct aside. A redundant network that loses one path has not avoided a problem; it has spent its spare. You are now standing on one leg, and the next failure — the one that was survivable yesterday — is a full outage. Neighbor down on an expected session should be high severity precisely because users feel nothing. This is the rare alert that lets you fix a future outage while everyone is calm.


High severity does not have to mean a phone call at three in the morning for every session. Reserve immediate paging for sessions whose loss removes your last path or your internet, and let the redundancy-lost cases be loud, unmissable, next-morning work. Alert on recovery too, at informational level — "session re-established" timestamps the end of the incident and brackets your timeline for free.


One more rule that costs nothing and catches ghosts: alert on session age resets. If a session shows "up" at every poll but its age is suddenly three minutes, it fell down and got back up while you were not looking. The outage hid between polls; the evidence did not. Any expected session whose age is younger than your polling interval deserves a warning, because whatever knocked it over is probably not finished.


Tip: when a router with many neighbors dies, you will get one alert per vanished session plus the device-down alert itself. That is a small storm describing one event. If your monitoring supports dependencies or grouping, make the neighbor alerts children of the device alert. Then one dead router reads as one incident instead of nine.


Flapping neighbors


A flapping neighbor is one that cycles — down, up, down, up — and it is genuinely worse than one that stays down. A dead session fails once and then holds still; the network reroutes and stabilizes. A flapping session forces every router to redraw its map on every transition, over and over. Users feel each redraw as a stall or a moment of weirdness. Worse, each recovery resets the naive alert: down (alert!), up (resolved!), down (alert!). Eventually the notifications train you to ignore that neighbor entirely, which is how flaps quietly become permanent residents.


The fix is to count transitions, not just observe states. Track state changes per session per hour, and fire a distinct "neighbor flapping" alert when the count passes a small threshold. Three or more transitions in an hour is a reasonable start. Let that alert suppress the individual down/up notifications while it is active. But keep the counter visible, because "42 transitions since last night" tells you the magnitude at a glance.


As for causes: a flapping session is almost always a symptom of something beneath it. That could be a marginal circuit, a physical link that is itself flapping, or a router whose CPU is too overloaded to send hellos on schedule. The layer-below story is covered in flapping links. And the route-level consequences — churn, and what it does to your network — get their own guide in route stability.


A worked example: the quiet Tuesday drop


Let us run one investigation end to end. Tuesday, 14:07. Your monitoring raises: "core-rtr: neighbor 10.10.5.2 (wan-rtr) down — redundancy lost." The ticket queue is silent. Nobody has called. Here is the walk:


  • 1. Confirm it is current. Log into core-rtr and check the neighbor table. Still down. Not a stale alert, not a blip — a real, ongoing loss.
  • 2. Check your own end. The interface toward wan-rtr shows up, no errors climbing. The physical link at this end is fine, which already narrows things.
  • 3. Read the reason. The log says hold timer expired — the link stayed up but wan-rtr went silent. Footprints point at the far router, not the cable.
  • 4. Look at the far router. wan-rtr does not answer pings at all. It is not a routing problem anymore; the whole device is down. A call to the facilities team finds an electrician doing unannounced work on that closet's power circuit. Of course.
  • 5. Check the traffic. The routing table on core-rtr shows internet-bound routes now pointing via the backup path. Users are working. Redundancy did its job — and, for once, you knew within a minute instead of never.
  • 6. Watch the recovery. 14:23: power restored, wan-rtr boots, the session climbs the ladder and re-establishes. Route counts return to normal. You note both timestamps and close the loop with facilities about, ahem, announcements.

Total user impact: none. Total time to a correct diagnosis: about ten minutes, most of it walking. Now subtract the monitoring and replay the same event. The session dies, traffic fails over, and nothing alerts. The network runs without a spare until the day the backup path fails too. At that point you get a hard outage plus a mystery, because the trail went cold weeks ago. Same event, two completely different stories, and the only difference is that one neighbor check existed.


Where to go next


Neighbor monitoring in one breath. Know which sessions should exist. Poll their state, age, and route counts. Listen for state-change events with reasons attached. Alert loudly when an expected session drops even if nobody screams. And count transitions to catch flapping. It is a modest amount of setup for what may be the earliest early warning your network can give you.


From here, the natural next read is route stability, which follows what those flapping sessions do to the map itself. The rest of Routing Protocol Health continues from there, through convergence and the routing table.


Put this into practice with SysPeep Monitor

Neighbor tables, session states, and router logs are exactly the kind of data SysPeep Monitor collects all day. It discovers your routers automatically and polls them over SNMP on a schedule you choose. Its alert rules and dependencies keep a dead router from turning into a storm of tickets. Try the free trial on any Windows machine.

Download Free TrialLearn About SysPeep