Home › Topics › Out-of-Band & Management-Plane Monitoring › Crisis-Path Drills

Crisis-Path Drills: Proving You Can Get In When It's All Down


On paper, your worst-case access story is airtight. The servers have BMCs. The switches have consoles wired to a console server. There is an out-of-band link for the day the whole site goes dark. The passwords are in the vault. Every piece was tested when it was installed, and every piece has sat there since, radiating quiet confidence. Here is the uncomfortable question this guide exists to ask: when did a human being last actually use any of it? Not ping it — use it. Log in through it, under something like real conditions, all the way to a device's prompt?


If the honest answer is "installation day," then what you have is not a crisis plan. It is a collection of parts that once resembled one. Emergency equipment decays in the dark: credentials rotate, certificates expire, and cables get borrowed. Firmware updates change menus, and documentation drifts out of date. None of it announces itself, because the back door has no users to complain. Buildings figured out the answer long ago and named it honestly: the fire drill. Nobody waits for smoke to find out whether the stairwell doors open.


A crisis-path drill is the IT version: a scheduled, low-stakes rehearsal. You pretend the front doors are gone and get into your equipment using only the emergency ones. This guide gives you the full practice. It includes a drill script you can run this month and a menu of scenarios to rotate through. It covers the special problem of credentials during outages and what to do with the embarrassing findings. It also explains how the whole habit ties into your runbooks. By the end, "we could probably get in" becomes "we got in last month, in eleven minutes, and here is the list of what we fixed afterward."


In this guide

  • What a drill is (and is not)
  • The drill script, step by step
  • A scenario menu to rotate through
  • The credential problem: break-glass access
  • Findings: documenting and fixing what failed
  • Cadence, rotation, and keeping it lightweight
  • The drill is a runbook rehearsal
  • The payoff, and where to go next

What a drill is (and is not)


Scope matters, so draw the box first. A crisis-path drill tests access: can the on-call human, starting from realistic conditions, reach the management surfaces of critical equipment without using the production network? It is not a disaster-recovery exercise — you are not failing over databases or restoring backups. It is not a chaos experiment — you break nothing on purpose. And it is not an audit of the whole monitoring stack. It answers one narrow, priceless question: do the emergency doors still open, and does the person on call know the way through them? Narrow scope is what keeps drills cheap enough to actually happen.


Three ground rules keep them safe. First, read-mostly: during a drill you log in, look around, and log out. You do not reboot, reconfigure, or "fix a little thing while we are here." The one exception is a pre-agreed harmless action — checking a status page, reading a log — that proves the access is real. Second, timeboxed: set a hard stop (an hour is generous) so a drill can never eat an afternoon. Anything unfinished becomes a finding, which is itself useful data. Third, announced: tell the team a drill is happening so nobody mistakes your console logins for an intruder. Your security monitoring noticing you is a finding of the good kind, but only if nobody escalates it into an incident.


And a word to the shop that is "too small for this". The smaller the team, the more a drill matters, because a one-person IT department has no colleague to phone for the forgotten detail. The solo version is entirely legitimate. Pick a quiet Friday hour, pretend the core switch is dead, and see whether you can reach a console and a BMC. Use only crisis paths and printed notes. It feels faintly silly for about ten minutes, right up until the moment you discover the envelope is stale or the modem is dead. At that point, it feels like the best hour of the quarter.


The drill script, step by step


Here is a script that works for a first drill and keeps working after twenty. Adapt freely; keep the timer and the notes.


  • Step 1 — pick the scenario. Choose one failure from the menu below — say, "the core switch is dead." Write down what that removes: the production network, everything reachable only through it, and (be honest) probably the office wireless too.
  • Step 2 — set the rules of the game. The only tools allowed are the ones the scenario leaves alive: crisis paths (BMC, console server, OOB link), offline documentation, and break-glass credentials. If you catch yourself using the production network "just to look something up," note it — that reflex is exactly what the drill exists to expose.
  • Step 3 — assign two roles. One person drives; one person scribes with a timer, noting each attempt and timestamp. Solo shop? Drive and scribe yourself — slower, still priceless.
  • Step 4 — attempt the access, for real. Reach the OOB gateway or console server through the secondary path. Retrieve the break-glass credentials from wherever they actually live. Get to the console of the affected device and to the BMC of one critical server. Prove each hop with the agreed harmless action.
  • Step 5 — record every snag, however small. The wrong-guess at an address, the adapter that took ten minutes to find, the prompt that looked different from the doc. Small snags at 2 PM are big snags at 2 AM; the notes are the product.
  • Step 6 — stop the clock and debrief for ten minutes. Total time to working access, snag list, and one question answered while memories are fresh: "if this had been real, what would have hurt most?"

The scribe's notes want a consistent shape so drills compare across quarters. A template:


CRISIS-PATH DRILL LOG
Scenario:        core switch dead (production network unavailable)
Participants:    driver: J.  scribe: R.
Started:         14:02

14:04  reached OOB gateway via cellular path        OK
14:09  break-glass envelope retrieved               OK (safe key was in the OTHER safe: snag)
14:13  console server login                         OK
14:15  core switch console reachable, prompt shown  OK
14:19  BMC of file server: login page               FAIL (certificate expired, browser refused)
14:24  BMC reached via fallback method              OK
Stopped:         14:31   time to full access: 29 min

Findings: (1) safe key location wrong in doc  (2) BMC cert expired
          (3) console escape sequence missing from runbook

Notice what the log measures: not whether the team is smart, but whether the path is healthy. Twenty-nine minutes with three findings is not a failure — it is a successful drill that just bought you a faster real outage. The only failing drill is the one that never runs.


A scenario menu to rotate through


ScenarioCrisis paths exercisedWhat it proves
Core switch deadConsole server, OOB link, offline docsYou can work with no production network at all
WAN down at a branch siteSecondary path into the site, site console serverRemote sites are reachable without the tunnel
Hypervisor host frozenServer BMC: remote console, power controlYou can see and reset a wedged host from outside its OS
Firewall lockout after bad changeFirewall console via console serverThe recovery path for your own mistakes works
Alert storm in progressConsole access while dashboards are floodedYou can operate from the quiet side during chaos
Monitoring server darkIts own OOB access pathThe watcher itself can be reached and revived
Power event in the server roomUPS/PDU management cards, sealed proceduresYou can assess and control power remotely

Rotate through the menu rather than repeating a favorite: each row decays independently, so each needs its own occasional daylight. Two rows point at their own deeper guides. Operating through an alert storm is covered in the Storm Response Playbook. The monitoring-server scenario has a dedicated guide in Out-of-Band Access to Your Monitoring. The power scenario leans on the gear described in the Power & Environment Monitoring pillar.


For your very first drill, resist the temptation to start with the apocalypse. The hypervisor-host scenario is the gentlest opener: it exercises one BMC, one set of credentials, and one page of documentation. And it can succeed in fifteen minutes. That matters, because a first drill that ends in total defeat teaches people to dread drills rather than trust them. Work up the table as confidence grows. Save "the whole site is dark and we are coming in from outside" for the drill where the easier rows already run clean. Rehearsals, like weightlifting, are supposed to get heavier gradually.


The credential problem: break-glass access


Every drill eventually collides with the same wall, so let us face it directly: credentials. On a normal day your passwords live in a password manager — which is a service, on a server, behind the network. It may be behind single sign-on. It may be protected by a second factor that needs the wireless or a phone signal. Trace the dependency chain of "I need the console server password" during a core-switch outage. You may find it quietly loops through the very network that is down. The vault is a fine front door for credentials; during a crisis it can be one more locked building.


The answer is break-glass credentials: a small set of emergency accounts whose secrets are stored in a way that depends on nothing. Classically, they go in a sealed, tamper-evident envelope in a safe. Or they go in an encrypted file on an offline device whose passphrase the on-call staff hold in their heads. The rules that keep break-glass from becoming a liability:


  • Scope them tightly. Emergency accounts exist for the crisis-path devices — console server, OOB gateway, BMCs, core network gear — not for every system you own.
  • Make use loud. Wherever the gear supports it, log and alert on any break-glass login. Opening the envelope should always leave a mark, drill or not.
  • Rotate after every opening. Once unsealed — in a drill or a real incident — the secrets are changed and a fresh envelope is sealed. The drill conveniently rehearses this too.
  • Check the second factor's dependencies. If emergency logins require a code from a phone, ask what happens when the outage takes out the signal or the authentication service. Hardware tokens or pre-agreed offline methods close that loop; the drill is where you find out whether yours does.
  • Store the map with the keys. The envelope should contain not just passwords but the one-page route: addresses, port numbers, escape sequences, and where the console kit lives. Credentials without directions rescue nobody at 2 AM.

Tip: put a "last verified" date on the outside of the envelope, updated at each drill. A sealed envelope nobody has opened in ages is Schrödinger's crisis plan: simultaneously fine and useless until observed. The date turns that superposition into a checkable fact — and its staleness into an agenda item.


Findings: documenting and fixing what failed


The drill's real output is the findings list, and the findings are wonderfully predictable across organizations. The BMC that stopped answering some unknown number of months ago — which is precisely the dead-BMC problem, caught by rehearsal instead of catastrophe. The expired certificate on the console server that makes browsers refuse the login page. The documentation that still lists the old address scheme. The console cable that walked away. The safe key in the wrong drawer. The second factor that needed the network. Each finding is small, cheap, and utterly capable of stretching a one-hour outage into a five-hour one.


Treat findings with the same respect as production bugs. Write each one as a ticket with an owner and a date. Fix them within the month. Then comes the step that separates ritual from engineering: re-test the fix at the next drill or sooner. Then close the loop with your monitoring: many findings are really monitoring gaps wearing costumes. A dead BMC found in a drill means the BMC reachability alert is missing or broken. An expired certificate means expiry tracking skipped a device. Every finding should trigger the question "what standing check would have caught this before the drill did?" And the answer usually belongs in the management-plane dashboard described in A Management Network You Can Trust. Drills and monitoring are partners on different timescales. Monitoring watches continuously for the failures you know about. Drills sweep occasionally for the ones you have not imagined yet.


The findings log also earns its keep upward. You may need budget for a console server, a second circuit, or an afternoon of BMC cabling. Bring a dated log reading "drill found the emergency path broken; here is what it would have cost during a real outage." It is the most persuasive document you can bring — concrete, local, and impossible to dismiss as vendor scaremongering.


And keep one simple number across drills: time to working access. Watching it fall — from a fumbling first hour to a crisp handful of minutes — is the most honest progress bar in this whole pillar. It is a persuasive one when someone asks what the drills are for.


Cadence, rotation, and keeping it lightweight


Quarterly is the sweet spot for most teams: frequent enough that decay gets caught within months, rare enough that nobody resents it. Add an extra, targeted mini-drill after any big change to the management network — new console server, re-addressed BMCs, a different OOB provider. Change is where paths silently break. Rotate the scenario each time, and rotate the driver too. Have the senior admin watch silently while the newest team member navigates from the docs alone. That drill tests the documentation instead of the veteran's memory. If a step only works when one particular person does it, that step is not done. It is memorized, which is a single point of failure with a salary.


Mechanically, a cadence survives only if it lives on the calendar with an owner. Put the next drill's date and driver in the schedule before the current one's debrief ends. Hang it on a rhythm that already exists — the same week as the quarterly maintenance window, say. That way, it inherits that habit's momentum. When a quarter slips (one will), do not let the miss quietly become the new normal. Run a shortened fifteen-minute version rather than skipping outright, because the streak itself is part of what you are maintaining.


Guard the lightweight feel fiercely. One scenario, one hour, two people, one log, a handful of tickets — that is the whole ceremony. The moment drills grow steering committees and slide decks, they die of importance and stop happening. The fire-drill spirit is brief, boring, and relentlessly regular.


The drill is a runbook rehearsal


If your team keeps runbooks — written step-by-step procedures for operational situations — you have already noticed what a drill is. It is a runbook rehearsal wearing a scenario. "Emergency access when the production network is down" should exist as a written runbook (offline and printed, for obvious reasons). The drill is simply that runbook, executed while someone takes notes on where reality has drifted from the text. Every finding that says "the doc lied" becomes an edit; every hesitation becomes a clarified step. The eleven-minute drill becomes the proof that the pages still describe the building. If runbooks are new to you, Your First Runbooks shows how to write ones worth rehearsing — and emergency access belongs in your first batch.


The payoff, and where to go next


Here is the story shape this practice buys, and it is worth wanting. A real outage arrives one night — the core switch, of course. The on-call admin does not improvise; she has literally done this before. Envelope from the safe, OOB path to the console server, switch console in minutes, cause identified, service restored. Here is the part nobody sees: the cellular modem she relied on had died quietly some months back. It was found dead by a drill, ticketed, and replaced on a calm Tuesday. The disaster went fine because its rehearsal had already absorbed the surprises. That is the whole trade: drills convert 2 AM discoveries into afternoon chores, one quarter at a time.


Everything the drill exercises is built in this pillar's other guides. Find the inventory and mindset in The Management Plane: Your Network's Service Entrance. Find the server-side doors in Monitoring BMCs. Find the consoles and secondary paths in Console Servers and OOB Paths. Find the network that carries it all in A Management Network You Can Trust. Build the doors, watch the doors, and every quarter, walk through them on purpose. The map of the whole topic lives at Out-of-Band & Management-Plane Monitoring.


Put this into practice with SysPeep Monitor

Drills prove the crisis paths work on the day you test them; monitoring vouches for all the days in between. SysPeep Monitor keeps standing reachability and health checks on the BMCs, console servers, and gateways your drills depend on. It alerts the moment one goes quiet, so the next rehearsal starts from working doors. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep