Storm Response: Containing a Meltdown in Progress
This is the guide you read on a calm afternoon so that it is already in your head on the loud one. The scenario: the entire network has stopped — file shares, printing, phones, internet, everything — and your monitoring dashboard is either screaming or, worse, unreachable. You suspect a broadcast storm: somewhere a loop has formed, and a self-sustaining flood is saturating every link in the VLAN. (If the phrase "self-sustaining flood" is new, Anatomy of a Network Loop explains the mechanics; this guide assumes the fire is already burning.)
Storm response is different from ordinary troubleshooting in one important way: your usual tools are among the casualties. You cannot SSH into switches that will not answer. You cannot browse a dashboard across a saturated network or look anything up on an intranet that lives behind the flood. So the playbook leans on things that work when the network does not: the console port, out-of-band paths, and a printed diagram. It also uses a methodical process of elimination that converges on the loop in a handful of steps.
By the end you will have a complete sequence. You will confirm it is really a storm, get eyes inside a switch, and bisect the network to contain the flood. You will identify the exact port and cable that caused it, avoid the classic response mistakes, and keep humans informed. You will verify recovery thoroughly enough to declare victory honestly.
In this guide
- Is it actually a storm? The fingerprint
- The first three minutes
- Getting in when the network path is dead
- Bisection: halving your way to the loop
- Pinpointing the culprit port
- The do-not list
- Communicating during the chaos
- The post-storm checklist
Is it actually a storm? The fingerprint
"Everything is down" has several possible causes, and the response for each is different — so the first minute goes to reading the fingerprint. A broadcast storm looks like this. Every device is powered but nothing communicates. The switches hum away with every port LED blinking furiously, rapidly, and eerily in unison, because every port is carrying the same flood. Switch fans often spin up as CPUs max out. And the failure hit everywhere on the segment at essentially the same moment. That LED synchronization is the classic tell. A healthy switch twinkles randomly, each port to its own rhythm. A storming switch strobes like a wall of hazard lights.
| Fingerprint | Likely cause | Quick confirmation |
|---|---|---|
| All port LEDs blinking hard, in unison; switches powered but unreachable; everything died at once | Broadcast storm / loop | Console into a switch: CPU pegged, MAC-flap logs scrolling |
| No lights anywhere, silence | Power failure | Check the UPS and the room lights |
| Local shares and printers fine, internet dead | Upstream / ISP or firewall issue | Ping the firewall, then its far side |
| Some users fine, others failing; new connections fail while old ones work | DHCP or DNS trouble | Check lease and lookup behavior on one client |
| One area dead, rest fine, LEDs normal elsewhere | Failed switch or uplink | Look at the dead area's closet switch |
If your monitoring survived (it often does, on a different VLAN or site), it shortens this step to seconds. You see broadcast packet rates pinned at absurd values and several switch CPUs at maximum simultaneously. Then devices go unreachable in one big wave. Those are the signals described in Early-Warning Signs of Loops and Storms, now at full volume.
While confirming, also note the boundary of the damage. A storm floods exactly one broadcast domain, so anything still working is telling you which VLAN or segment holds the loop. If the server VLAN answers pings while the user VLAN is dead, you have already narrowed the search before touching a single cable. The edge of the blast radius is the first bisection, performed for free.
The first three minutes
Resist the reflex to start pulling cables immediately. It feels productive, but random unplugging is bisection without the bookkeeping. You change several variables, learn nothing from the result, and risk making the topology stranger than it already is. The storm will still be there in three minutes — that is the one dependable property of a storm. Three minutes of setup makes the next thirty dramatically more effective. Note the time — the incident clock matters for communication and for the postmortem. Tell one person what you believe and what you are doing ("suspected loop, I am going to the main closet with a console cable"). That gives the organization a thread to pull while you vanish into a wiring closet. Then collect the storm kit. Bring a laptop that does not depend on the broken network, a console cable and adapters, and the network diagram. Bring a marker with tape or labels for marking every cable you touch. If your diagrams are current, this is the moment they pay for themselves. Network Diagrams That People Actually Use exists precisely for the day the diagram in someone's head is unavailable. The broader discipline of those opening minutes — timekeeping, roles, first actions — is covered in The First 15 Minutes. Here we stay with the storm-specific moves.
Getting in when the network path is dead
You need eyes inside at least one switch, and the network path is gone. Two doors remain. The first is the console port: the serial connector on every managed switch, made for exactly this moment. A console session does not travel over the network at all. It is a direct cable between your laptop and the switch's management brain. So it works, if sometimes sluggishly, while the data plane is a hurricane. The second door is out-of-band management: any access path that bypasses the production network. Options range from a dedicated management network to a small console server reachable over a separate connection. If you have this, storm response starts at your desk instead of in a closet. If you do not, put it on the list for later. Out-of-Band Access makes the case and shows the options.
Once in, confirm the diagnosis in seconds. Look for CPU at or near maximum, a log buffer scrolling with MAC-flap messages, and broadcast counters on several ports incrementing at ferocious rates. Note which ports the flap messages name — that pair is your first clue to where the circle is. If the switch is responsive enough, you can even begin containment from the console by shutting suspect ports individually. Often, though, the storming switch is too busy to be cooperative, which is why the next tool is physical.
No console cable, or unmanaged gear with no console to offer? You still have instruments: the LEDs and the topology. Watch a closet switch while a colleague unplugs its uplink. If the frantic strobing collapses to calm the moment the uplink is out, the storm is arriving from upstream, not born locally. Working that logic around the building is slower than reading logs. But it is the same bisection you are about to do anyway, navigated by blinkenlights instead of counters.
Bisection: halving your way to the loop
The loop is somewhere; the network is unusable everywhere. The fastest honest way to localize it is bisection. It is the same halving strategy you would use to find a bad bulb in a string of lights. Disconnect half the network, see whether the storm stops, and you have cut the search space in two. Repeat. Even a large network converges in a few rounds, because halving is exponential in your favor. The same math that makes storms explode makes the search collapse.
- 1. Stand at the center. Go to the core or distribution switch where uplinks to the wiring closets converge. The diagram tells you which ports are uplinks; so do the cable labels you hopefully have.
- 2. Label before touching. Photograph the patching, and tag every cable you are about to move with its port. Under adrenaline, memory is a liar. This step is not optional.
- 3. Disconnect half the uplinks. Pull (or shut from console, if responsive) the uplinks to half the closets. Wait a full thirty to sixty seconds. The flood drains almost instantly once a circle breaks, but you want unambiguous readings. Spanning tree may need a moment to settle.
- 4. Read the result. Storm stopped (LEDs calm, CPU falling, pings returning)? The loop is in the disconnected half. Storm still raging? It is in the half still connected — or in the core itself.
- 5. Halve again. Reconnect the innocent half (one uplink at a time, watching for relapse after each), then split the guilty group and repeat. Each round cuts the suspects in half.
- 6. Converge on one closet, then one port. Inside the guilty closet, repeat the game with its ports, or use the closet switch's console — now responsive — to shut ports in batches. You end with a single port that, when connected, storms the network. Leave it disconnected or shut.
Two rules make bisection trustworthy. Change one thing at a time, and wait and observe after every change. The storm's on/off response is your only instrument, and it only measures the last thing you did. Restoring service usually happens well before you find the exact port. The moment the guilty closet is isolated, the rest of the building recovers. You can then hunt the final port with the pressure reduced from "everything is down" to "one closet is down."
A variant worth knowing: the partial storm, where the network is miserable rather than dead because the loop runs through a slow link. Bisection works identically, but the on/off signal is subtler. Use broadcast pps on a monitored port as your instrument instead of eyeballing LEDs, and give each step a little longer. And if the loop is intermittent, the storm may stop mid-hunt on its own. Examples are a Wi-Fi bridge that only sometimes associates and a cable that only loops when a dock is in use. Do not stand down; note which half of the network you had isolated when it stopped, because that is still real evidence about where it lives.
Tip: when the storm stops, MAC tables all over the network still hold garbage learned during the chaos. So a few seconds of residual confusion is normal. Give the network a minute to relearn before judging the result of each bisection step — patience here prevents false conclusions.
Pinpointing the culprit port
With stability restored, switch from surgeon to detective. The switches recorded the evidence for you. The MAC-flap log entries name the two ports the flood circulated between. Many hosts flapping between the same pair is the loop's signature. The pair usually reads as "the real uplink" plus "the port the loop came in on." The second one is your culprit. Port counters corroborate. Compare broadcast counters across ports, and the loop ports stand out by orders of magnitude. Transmit and receive are both enormous and nearly equal:
switch# show interface counters port broadcasts-in broadcasts-out 1 884 412,655 2 1,102 412,391 12 412,644,908 412,598,377 <— the whirlpool 24 412,601,554 412,633,046 <— its partner
Then walk to the jack. At the end of that cable you will find one of the classics. It might be a spare cable joining two wall jacks or a little desk switch cabled to the wall twice. Or it might be a device bridging Wi-Fi to its Ethernet port. That is the full rogues' gallery from the anatomy guide. Photograph the cause in place before touching anything. The picture anchors the postmortem, settles disputes kindly, and makes the case for the prevention budget better than any memo. Remove or correct the cause, but keep the port administratively shut until you have decided what protections it gets before returning to service.
While you are in the area, ask the gentlest question in IT: "did anything change around here recently?" You are not asking to assign blame. Storms are caused by helpful people, and you want those people to keep talking to you. You ask because the answer often completes the timeline. "Facilities rearranged the meeting room yesterday" or "we got new docks on Tuesday" turns a mystery into a documented cause. The answer also tells you where else the same change might be waiting to loop.
The do-not list
Storms generate pressure, and pressure generates famous mistakes. These are the ones to rehearse not making:
- Do not reboot everything at once. The building-wide power-cycle feels decisive and proves nothing. The loop is physical: when the switches come back, the storm resumes within seconds. And you have erased the log buffers and counters that would have located it. You have also added a surge of boot-time traffic to the confusion. Reboots treat symptoms that cables cause.
- Do not remove two suspects at once. Unplug two things and watch recovery, and you have learned that one of them was guilty — which is to say, almost nothing. Every change you make in parallel divides the meaning of the result.
- Do not disable spanning tree to "simplify" things. Mid-storm is the worst possible moment to remove loop protection. If the protocol seems to be misbehaving, the real cause is usually a lying link or a rogue root. See Spanning Tree Without Tears. Turning spanning tree off converts one loop into a standing invitation for more.
- Do not accept a storm that "fixed itself." If the flood stopped without an explanation, someone unplugged something, a flaky link dropped, or a protection finally tripped. The loop's cause is still out there. Find the flap logs and finish the investigation while the trail is fresh.
- Do not restore the guilty port casually. "It seems fine now" is how sequels get made. The port stays shut until the physical cause is removed and, ideally, the port has protections that make a repeat automatic to contain.
- Do not rewire from memory. Every cable you moved goes back according to your labels and photos, not your recollection. Post-incident miswiring is how one outage becomes two.
Communicating during the chaos
A building-wide outage is loud, and silence from IT makes it louder. The complication: your normal channels — email, chat, ticketing — may be riding the same broken network. Decide now, on the calm day, what the fallback is. Options are SMS, a phone tree, a status page hosted outside, or simply a person at the helpdesk relaying by phone. During the incident, send short structured updates on a promised cadence, even when the update is "still working on it." Two audiences matter most. One is whoever runs the business (they need impact and a next-update time, not port numbers). The other is the frontline colleagues fielding questions (they need one sentence they can repeat). A format that survives adrenaline:
NETWORK STATUS — 10:42 What: building-wide outage, suspected switching loop Doing: isolating wiring closets one by one; floors 1 and 3 restored Next update: by 11:15
Three habits raise trust. Promise the time of the next update, and keep it — a kept promise about communication compensates for uncertainty about the fix. Do not guess at causes in public before you have evidence. "Suspected loop" is honest. "Someone plugged in a switch" before you have found it is a rumor with your name on it. And when service returns, say so explicitly — users who never hear the all-clear keep not trusting the network long after it deserves trust again. If two of you are responding, split roles: one pair of hands in the closet, one voice on communication. The closet worker should never be the one fielding calls.
The post-storm checklist
Recovery claims deserve verification. Before declaring victory, walk this list:
- 1. Broadcast rates back to baseline on every monitored port — not just "lower," but back inside the normal band your baseline defines.
- 2. Switch CPUs at their usual idle across the segment, with no stragglers.
- 3. MAC-flap logs silent for at least fifteen minutes of real user traffic.
- 4. Topology-change counters quiet and the spanning-tree root confirmed as the switch you intended.
- 5. Physical cause found, photographed, and removed — not merely "storm stopped." A storm without a found cause is an intermission.
- 6. Every moved cable restored per labels, and the guilty port's status decided deliberately.
- 7. Timeline written while it is fresh: when it started, what was tried, what worked, what the cause was. Twenty minutes now saves the postmortem.
- 8. Prevention follow-ups scheduled — the guards, storm control, and port hygiene in Designing Loops Out are exactly the homework a storm assigns.
Item 5 is the one teams skip under fatigue. It is the one that determines whether this was the last such storm or merely the most recent.
Where to go next
The playbook in one breath: read the fingerprint, take three minutes to set up, get console or out-of-band eyes inside a switch, bisect uplinks one change at a time until the storm collapses, then let flap logs and counters name the exact port, communicate on a cadence throughout, and verify recovery against your baseline before you celebrate. Print it, keep a console cable where you can find it in the dark, and then make the whole document less necessary. The rest of Loops, Broadcast Storms & Spanning Tree covers the early warnings that shrink storms into alerts, and the layered protections that stop the next loop from becoming a storm at all.
Put this into practice with SysPeep Monitor
Half this playbook depends on data your network was recording before the storm: broadcast baselines, CPU graphs, port status, collected logs. SysPeep Monitor gathers those continuously over SNMP and friends, maps your devices, and alerts the moment counters go abnormal. So the next storm starts with evidence instead of guesswork. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep