Early-Warning Signs of Loops and Storms
A full broadcast storm feels instantaneous: one moment the network works, the next moment nothing does. So it is easy to conclude that storms give no warning and that monitoring is useless against them. In that view, you find out when everything dies, same as everyone else. That conclusion is wrong, and this guide is about why.
Many loops do not detonate on contact. They smolder. A loop that runs through a slow link floods only as fast as that link allows, producing a network that is mysteriously miserable rather than dead. This can last for hours or days. And even the loops that do detonate are often preceded by minutes of measurable weirdness. Broadcast rates creep up, a switch CPU climbs, and the first MAC-flap messages appear in a log nobody reads. The signals exist. The question is whether anything is watching for them.
This guide covers the five signals that show up before (or during the opening seconds of) a meltdown. It explains where each one is visible, what it implies, and how to turn each into a concrete alert with sensible starting thresholds. If you already read Anatomy of a Network Loop, you will recognize each signal as a symptom of the machinery described there. Flooding, amplification, and MAC-table chaos each cast their own shadow into your monitoring data.
In this guide
- Why storms give you a warning window at all
- Signal 1: broadcast and multicast rates above baseline
- Signal 2: switch CPU spiking for no reason
- Signal 3: MAC-flapping notifications
- Signal 4: ports pegged — and symmetric
- Signal 5: topology-change counters ticking up
- Setting your baselines and alerts
- The early-warning checklist
Why storms give you a warning window at all
The textbook storm saturates every link in under a second, which leaves no time for alerts, coffee, or dignity. Real networks, though, produce a surprising number of partial loops — and partial loops are slow-motion storms.
The amplification in a loop is limited by the slowest segment in the circle. If the loop runs through a wireless bridge, an ancient desk switch, or a congested uplink, the flood grows only until that bottleneck saturates, then holds there. The rest of the network receives a steady drizzle of duplicate broadcasts instead of a firehose. It is enough to raise broadcast rates well above normal, burn switch CPU, and make everything feel sticky. It is not enough to kill the VLAN outright. Users report "the network is slow today." Nobody suspects a loop, because the network is not down. These smoldering loops can persist across days and reboots until someone finally measures the right thing.
There is a second kind of warning window: the loop that exists but is not yet carrying disaster. A looped cable behind a healthy spanning-tree blocked port generates topology-change events and small anomalies long before any failure unblocks it. And even a full-speed meltdown has opening seconds during which counters lurch upward. If your monitoring polls frequently and alerts aggressively, "the building is down" can arrive as "broadcast rate alarm on closet 2" instead of as thirty simultaneous phone calls. Every signal below works in both modes: early warning for the smolder, first-and-loudest diagnosis for the detonation.
A quick worked example, because the smolder is easier to picture than to define. Monday morning, users on one floor say the network "feels heavy." Utilization graphs look fine, so the complaint gets parked. Nobody checks the following facts. Broadcast pps on that VLAN has been running at many times its usual level since Friday evening. One access port shows equal transmit and receive around the clock. The closet switch's CPU has been sitting at a plateau it has never touched before. The cause turns out to be a wireless bridge looping a conference-room jack back into the network at radio speed. It is a circle too slow to kill anything, fast enough to tax everything. Three of the five signals below had been raising their hands for two days. This guide exists so that next time, something is listening.
Signal 1: broadcast and multicast rates above baseline
The most direct signal is the one storms are named for: the rate of broadcast (and multicast) traffic, measured in packets per second (pps). Every switch counts these per port, and monitoring systems can collect the counters over SNMP (Simple Network Management Protocol, the standard question-and-answer language for network gear). A healthy VLAN hums along at a low, fairly steady broadcast rate driven by ARP, DHCP, and discovery chatter. A loop multiplies exactly this traffic, so the broadcast pps counter is the storm's most honest witness.
One trap: watch packets per second, not just bandwidth. Broadcast frames are mostly tiny — an ARP request fits in the minimum Ethernet frame. So a smoldering loop can multiply the packet rate dramatically while the utilization graph, measured in bits per second, still looks respectable. A link showing modest bandwidth but a wildly elevated packet count, most of it broadcast, is a loop until proven otherwise.
"Elevated" only means something relative to your normal, which is why this signal depends on having a baseline — a record of what each counter looks like on an ordinary day. If baselining is new to you, What Is a Baseline? covers it properly. The short version is that a few weeks of collected data turns "is this number bad?" from a guess into a comparison. Broadcast rates are one of the most rewarding metrics to baseline, because normal is so boringly flat that trouble stands out like a flare.
Signal 2: switch CPU spiking for no reason
A switch forwards traffic in dedicated hardware, so its management CPU normally idles — single-digit load, hour after hour, forever. That is what makes the CPU such a clean signal. It has almost no legitimate reason to spike outside of a config change, a firmware task, or you poking at it. When a loop forms, the CPU gets dragged into the storm from several directions at once. MAC-table churn generates constant updates and notifications. Broadcast traffic that the switch itself must examine — ARP handling on a routing switch, discovery protocols, management traffic — arrives at thousands of times its normal rate. Logging goes berserk.
So a switch CPU that jumps from its usual idle to a sustained high plateau, without a maintenance explanation, deserves an immediate look. That is especially true if two or three switches on the same VLAN jump together. That pattern of correlated CPU spikes across neighboring switches is close to a storm signature, and it frequently shows up a beat before users notice anything. It is also self-protective monitoring: the same CPU exhaustion is what will soon make those switches unreachable. So the spike is your last easy chance to look around while management access still works.
Polling frequency matters more here than for most metrics. A check that samples CPU every fifteen minutes will happily average a violent five-minute spike into a gentle bump, or miss it entirely. Poll switch health every minute or two if your platform allows it, and graph the result rather than only alerting on it. The shape of a CPU plateau (sudden vertical edge, dead-flat top) is itself a clue that you are looking at a flood rather than a busy day.
Signal 3: MAC-flapping notifications
As the anatomy guide explains, a loop makes the same source MAC address arrive at a switch on two different ports. So the switch's MAC table entry for that address flips back and forth — MAC flapping. Most managed switches announce this in their logs, and the message names the two ports involved. That makes it the single most diagnostic early warning you can collect. A typical entry looks something like this:
MACFLAP: host 00:1b:44:11:3a:b7 flapping between port 12 and port 24 MACFLAP: host 00:1b:44:11:3a:b7 flapping between port 24 and port 12 MACFLAP: host 64:d1:54:9e:22:c0 flapping between port 12 and port 24
Read the pattern, not just the message. Many different hosts all flapping between the same two ports means those two ports both lead to the same place. That is a loop, with the two ports naming the circle for you. That is a gift: the log has essentially drawn an arrow at the problem. One caveat keeps this signal honest: occasional single-host flaps are normal life. A laptop roaming between two wireless access points, a virtual machine migrating between hosts, or a device moved to a new jack will each produce a flap or two. The loop signature is different — rapid, sustained, repeating, multi-host. Alert on the pattern (several flap messages within a minute), not on every lone message, or you will teach yourself to ignore the alert.
The practical requirement is that switch logs must actually go somewhere. Configure your switches to send syslog to a collector, or have your monitoring system receive their SNMP traps. A perfect warning written to the internal log buffer of a switch you can no longer reach is a diary entry, not an alert.
Signal 4: ports pegged — and symmetric
Normal traffic is lopsided. Users download far more than they upload; a server port sends more than it receives; an internet uplink is busy inbound. Look at any healthy utilization graph and the transmit and receive lines wander independently, like two people having different days. Loop traffic breaks that pattern in a distinctive way: the flood circulates, so an interface caught in the circle transmits the flood and receives it back. The graph shows both directions climbing together into a flat ceiling — pegged at line rate, perfectly symmetric, with none of the bursty texture of human traffic. It looks less like a busy road and more like a pipe that has become part of a fountain.
Symmetric-and-pegged on an access port is doubly damning, because a single desk has no business saturating anything in both directions for minutes on end. On uplinks it pairs beautifully with signal 1: saturation plus a broadcast-heavy packet mix identifies not just that there is a storm but which links form the ring. During response, this is the map you will use to decide what to unplug — a job covered step by step in Storm Response. The practical prerequisite is unglamorous. Your monitoring has to be graphing both directions of every uplink and access port, not just the handful of links someone once decided were interesting. Storms have a talent for forming on exactly the port nobody graphed.
Signal 5: topology-change counters ticking up
Spanning tree — the protocol that blocks redundant paths before they can loop — keeps a running count of topology changes. These are events where the tree of active paths had to be recalculated because a link appeared, disappeared, or changed roles. Every switch exposes this count, along with the time since the last change, in its spanning-tree status output and over SNMP. (If spanning tree is new territory, Spanning Tree Without Tears explains the whole machine in plain language.)
On a stable network, this counter is nearly frozen — days or weeks since the last change. A counter that starts incrementing steadily tells you the topology is churning. In that case, a link is flapping up and down or a new switch keeps appearing and vanishing. Or someone is plugging and unplugging something that participates in spanning tree. None of those are storms yet. All of them are the conditions storms grow from. And a burst of topology changes is often the last recorded event before a meltdown. That is the sound of the protective machinery straining right before something slips past it. Trend it, alert on unusual movement, and treat a sudden flurry as a reason to go look at what changed. As a bonus, each topology change briefly makes switches flush MAC-table entries and flood more than usual. So heavy churn is itself a small performance problem with the same fingerprints as a mini-storm.
Tip: no single signal is proof. Broadcast rates rise during legitimate events (a lab full of machines booting), CPUs spike during firmware tasks, and lone MAC flaps are everyday noise. The storm pattern is convergence: two or more signals moving together, on the same VLAN, at the same time. When broadcast pps and switch CPU rise in step, stop wondering and start looking.
Setting your baselines and alerts
Knowing the signals is half the job; the other half is wiring them into alerts before you need them. Here is a practical sequence for doing that, in order of payoff:
- 1. Collect the right counters. Make sure your monitoring polls, per port on every managed switch: broadcast and multicast packet counters, total packet and byte counters, and interface status. Add switch CPU and, where exposed, the spanning-tree topology-change count. All of these are standard SNMP objects on business-grade gear.
- 2. Baseline for a couple of weeks. Let the data accumulate through at least two normal weekly cycles, then note the busy-hour broadcast pps for your key VLANs and uplinks. Write down what normal looks like; it will be reassuringly boring.
- 3. Alert on broadcast rate multiples. A sensible start: warn when a port's broadcast rate holds several times its busy-hour baseline for a few minutes. Alert loudly when it reaches tens of multiples or a hard absolute ceiling. Exact numbers vary by network — tune until quiet weeks are quiet.
- 4. Alert on the MAC-flap pattern. Forward switch syslog to a collector and match flap messages, with a threshold like "more than a handful from one switch within a minute." Repeated flaps between one pair of ports should page someone.
- 5. Alert on switch CPU. Warn at a sustained moderate plateau, alert high; more importantly, alert on correlated spikes across switches in one VLAN.
- 6. Watch topology changes. Alert when the change count moves more than a few times in an hour, or at all on segments that never change.
- 7. Test the pipeline. Trigger a harmless port flap in a maintenance window and confirm the alert reaches a human. An untested alert path fails exactly once, at the worst time.
For step 1, the per-port packet counters live in the same standard SNMP interfaces table as the byte counters you already collect. You are just asking for a few more columns:
ifInBroadcastPkts broadcasts received on the port ifInMulticastPkts multicasts received on the port ifHCInBroadcastPkts the same, as a 64-bit counter for fast links ifInUcastPkts ordinary one-to-one packets, for comparison
Two honest warnings about tuning. First, thresholds you copy from anyone — including this page — are only starting points. Your network's normal is the real reference, which is the entire argument for baselining first. Second, resist the temptation to alert on every wobble. The goal is a small set of alarms you trust completely, because the day one fires, you want your reflex to be motion, not skepticism.
The early-warning checklist
Here is the whole guide as one table — the five signals, where to see each, and what each is trying to tell you:
| Signal | Where you see it | What it implies | Starting alert idea |
|---|---|---|---|
| Broadcast/multicast pps above baseline | Per-port SNMP counters, trended | Flood forming or smoldering; partial loop if sustained but below line rate | Warn at several times busy-hour baseline for a few minutes |
| Switch CPU spike | Switch health polling | Control plane drowning in churn; management access about to fail | Sustained high plateau, or several switches spiking together |
| MAC flapping | Syslog / SNMP traps from switches | Same MAC on two ports: a circle exists; the two ports locate it | Several flaps in one minute, same port pair |
| Ports pegged and symmetric | Interface utilization graphs | Interface is inside the circulating flood; identifies ring links | Both directions near line rate on an access port |
| Topology-change counter climbing | Spanning-tree status / SNMP | Topology churning: flapping link or appearing/vanishing switch | More than a few changes per hour, or any on frozen segments |
One deliberate omission from the table: user complaints. They are real data, but they arrive late, vaguely worded, and all at once. The point of everything above is to hear the storm before the humans do.
When a signal fires
Treat a converging set of these alarms as a pre-incident, not a curiosity for later. Confirm quickly (is broadcast pps still climbing? are two switches' CPUs up together?). Then move with the structured calm described in The First 15 Minutes. Note the time and tell someone you are investigating. Start from the evidence you already have — the flapping port pair, the pegged links. If the network is already melting, switch documents: Storm Response: Containing a Meltdown in Progress is the playbook for exactly that hour. It leans heavily on the counters and logs this guide had you collecting. Early-warning data does not just predict storms; it is also the forensic record that later tells you which port, which closet, and which helpful cable.
Where to go next
Storms only look unannounced. Broadcast rates drift up, CPUs climb, MAC tables flap, ports peg symmetrically, and topology counters stir — all measurable, all alertable, often before the first complaint. Baseline the quiet, alert on the divergence, and make sure switch logs reach a collector. From here, the rest of Loops, Broadcast Storms & Spanning Tree continues the story. You can learn how the protective protocol works in Spanning Tree Without Tears. And you can learn how to design loops out entirely in Designing Loops Out.
Put this into practice with SysPeep Monitor
Every signal in this guide is ordinary monitoring data: broadcast counters, CPU load, port status, syslog. SysPeep Monitor collects all of it over SNMP and companion protocols. It baselines what your network normally does and alerts when a counter leaves its lane. So a smoldering loop shows up as a tidy warning instead of a mystery. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep