Home › Topics › A General Troubleshooting Methodology › Under Pressure

Troubleshooting Under Pressure Without Making Things Worse


Quiet-afternoon troubleshooting and outage troubleshooting are the same puzzle played under different physics. On a quiet afternoon you have time, privacy, and the luxury of curiosity. During an outage you have an audience and a clock made of other people's lost work. You have adrenaline where your judgment used to be. And a growing crowd offers the helpful observation that it is still broken. Same network, same tools — but every mistake now costs more, and pressure is a machine for manufacturing mistakes.


Here is the uncomfortable truth this guide is built around: a large share of the worst outage stories are not about the original fault at all. They are about the response: the hasty config change that locked everyone out, or the reboot of the one box holding things together. Or they are about the failover onto a standby that turned out to be dead. The original problem was an outage; the response made it a catastrophe. Your first job under pressure is not to fix things faster. It is to not become the second, bigger problem.


This guide is the discipline for that. Learn how to tell safe experiments from risky ones and why one-change-at-a-time is non-negotiable when it matters most. Learn how to think in rollbacks, and learn the classic self-inflicted disasters and their guardrails. And — hardest of all — learn how to recognize the moment to stop, escalate, or call the vendor.


In this guide

  • What pressure does to your brain
  • First: look, don't touch
  • Safe tests vs. risky experiments
  • One change at a time
  • Rollback thinking
  • The second outage: self-inflicted classics
  • Communicating while you work
  • Knowing when to stop
  • After the storm
  • Where to go next

What pressure does to your brain


Under stress, thinking narrows. You fixate on one theory and stop generating alternatives. You favor action over observation, because typing feels like progress and reading logs feels like standing still. You skip the boring verification steps you would never skip on a Tuesday afternoon. And you become suggestible — when someone important says "could it be the firewall?", the firewall suddenly enjoys a prosecution it has done nothing to deserve. None of this is a character flaw; it is standard-issue human wiring. It happens to everyone, including the calm-looking senior admin (they have just built better habits around it).


The counterweight is a phrase borrowed from people whose jobs involve actual emergencies: slow is smooth, smooth is fast. The methodical responder finishes first, not despite going slower, but because they never spend an hour undoing their own panic. Every practice in this guide is a way of bolting that calm onto the outside of your skull, so it is available even when the inside is busy screaming.


First: look, don't touch


The opening minutes of an incident have their own rhythm: confirm it is real, size the blast radius, tell people you are on it. That rhythm is walked minute by minute in The First Fifteen Minutes. So we will not duplicate it here. What matters for this guide is the posture those minutes establish: observation before intervention. You gather the dashboard view, the scope, and the evidence before your hands touch a single config. Everything below assumes that posture and answers the next question: once you do start touching things, how do you keep each touch from making the hole deeper?


Safe tests vs. risky experiments


Every diagnostic act sits somewhere on a spectrum from "purely looking" to "definitely changing things." Under pressure you must know, consciously, where each act sits. The risky end needs a justification, a rollback plan, and an announcement. The safe end needs none of that. The honest complication is that a few "safe-looking" acts carry hidden teeth:


ActionFeels likeActually isRule of thumb
Reading dashboards, graphs, status pagesSafeSafeDo freely, and start here
Ping, traceroute, DNS lookupsSafeSafeDo freely
Reading logs and counters on a healthy boxSafeSafeDo freely
Logging into a device already at its limitsSafeMildly risky — your session adds loadPrefer the monitoring view of it first
Verbose debug output on struggling gearSafeRisky — debug floods can finish a wounded boxNarrow filters, shortest bursts, know how to stop it
Packet capture on the affected serverSafeMildly risky — adds load and fills disksCapture with filters and size caps, or from another box
Clearing counters, caches, ARP tablesHarmless resetRisky — erases evidence, can disrupt trafficRecord values first; ask why you are clearing
Restarting a service or deviceThe classic moveRisky — destroys state and evidence; may not come backEvidence first, rollback known, announced
Config changes, failovers, upgradesFixingMost risky — you are now changing the crime sceneOne at a time, reversible, announced, logged

The table's quiet message: you can go a surprisingly long way down the diagnosis road using only the top rows. Layer walks, path halving, and hypothesis tests are mostly built from reads — pings, lookups, counters, comparisons. Exhaust the reads before you spend a write. And borrow curiosity's rule from the emergency room: first, do no harm. When two tests would teach you the same thing, take the one that cannot possibly make the patient worse.


One change at a time


When you do start changing things, the iron rule is one change, then observation, then the next change. The reasoning is cold logic, not tidiness. Make two changes together and the outcome is uninterpretable. If things recover, you do not know which change worked. So you cannot document the fix, and you are left running a mystery config. If things get worse, you do not know which change to undo. And undoing both may not return you to your starting point. You have converted one unknown into three.


Three practices make the rule stick when adrenaline argues otherwise. Announce each change before making it — out loud to the room, or one line in the incident chat. For example: "disabling port security on port 12, 14:32, watching for two minutes." The announcement forces a beat of reflection and timestamps your trail. Write it down as you go, in the same diagnosis log this pillar recommends for hypotheses — the format is in Guess, Test, Learn. And wait out the feedback delay: many fixes take real time to show — spanning-tree reconvergence, DNS caches expiring, leases renewing, sessions rebuilding. Impatience here is the classic trap: the fix needed ninety seconds, but you gave it thirty and declared failure. Then you stacked a second change on top of a first one that was quietly working. Know roughly how long your change needs, and give it that long before you judge it.


A thirty-second story shows the trap in motion. An admin fixes a wrong DNS record — correctly — but the old answer lives on in caches for several more minutes. Seeing no improvement, she "also" restarts the DNS service, then "also" flushes the firewall's sessions. The caches expire on schedule, service returns, and now three changes stand between her and knowing what fixed it. So the wrong lesson ("you have to restart DNS and flush the firewall") enters office folklore, ready to prolong the next incident. The fix was right the first time. Only the waiting was missing.


The rule has a social clause, too: one keyboard at a time. When three people each "try things" on the same outage, the network experiences a distributed attack by its own defenders, and nobody can reconstruct what happened. If several of you are working, split the reads freely — but route every write through one person.


Rollback thinking


Before every risky action, ask one question: "how do I get back to where I am right now, and how long does that take?" If you have a crisp answer, proceed. If you do not, stop. That action just moved itself to the maintenance window, where mistakes are cheap and scheduled. The art of running those windows is covered in the maintenance windows guides. Rollback thinking, made concrete:


  • 1. Snapshot before you touch. Save the device config, screenshot the settings page, copy the file. Ten seconds now buys you a guaranteed road home.
  • 2. Prefer changes with built-in reverse gear. Many network devices offer an apply-with-automatic-revert mode — the change undoes itself after a timer unless you confirm it. For anything that could cut off your own access, that mode is worth learning before the day you need it.
  • 3. Keep a path back that does not depend on the thing you are changing. Reconfiguring remote access over that same remote access is sawing the branch you sit on. Know your out-of-band option — a console cable, a management network, or at minimum a human on site with hands.
  • 4. Try it on the least critical instance first. If the fix will go on six switches, it goes on the least important one first, then waits for proof.
  • 5. Name the point of no return. Some actions cannot be rolled back — firmware upgrades that migrate configs, deletions, re-addressing. Those need a deliberate, stated decision, not a reflex. Saying "this one is irreversible — agreed?" out loud has stopped many a 2 AM mistake mid-keystroke.

There is one honest exception to all this caution. When everything is already down and provably cannot get worse, bolder moves are rational. The branch office with no connectivity at all cannot lose more connectivity. Even then, say what you are doing, and keep the log running. "Nothing left to lose" has a way of being wrong about the backup systems.


The second outage: self-inflicted classics


Every experienced admin can recite these from memory, usually while staring into the middle distance. Learn them from this list instead:


  • Rebooting the last survivor. The overloaded box limping along is the only reason anything still works; rebooting it to "clear things up" completes the outage. Guardrail: before any restart, ask "what is this device currently carrying, and what happens if it does not come back?"
  • Failing over to a corpse. The standby link, spare unit, or secondary server that nobody has tested in living memory turns out to have quietly died. Guardrail: check the standby's health before failing over, not after — and outside of outages, test your failovers on purpose, on your schedule.
  • The while-I'm-in-here fix. Mid-outage, you spot some unrelated untidiness and tidy it. Now two things changed, one on purpose, and the timeline is mud. Guardrail: outage time is minimum-change time; write the untidiness down for later.
  • Locking yourself out. The firewall rule, VLAN change, or address edit that severs your own management session — sometimes for every admin at once. Guardrail: before applying, ask "does my own session ride on what I am changing?"; keep a second session open, and know your out-of-band path.
  • The mass action. Restarting all the access points, bouncing every uplink, pushing a change to every switch at once — turning one suspect into a fleet-wide experiment. Guardrail: one instance, verify, then the rest.
  • Trusting memory with syntax at 2 AM. Adrenaline plus a from-memory command in a production console is how typos become incidents. Guardrail: paste prepared commands from your notes or a runbook, and for the genuinely scary ones, have a second person read the line before you press enter.

Warning: beware the moment the room starts chanting for a big dramatic action — "just reboot the core switch!" Group pressure peaks exactly when evidence is thinnest. The bigger and more audience-pleasing the proposed action, the more it needs the boring questions. What hypothesis does this test, and how do we get back if it goes wrong?


Communicating while you work


People can forgive an outage; they struggle to forgive silence. The working minimum is small: say what is affected, that you are on it, and when the next update will come. Then actually deliver that update at the promised time, even if it only says "still digging." If you can borrow a second person, let them do the talking while you do the typing. The context-switch between console and audience is more expensive than it looks. Incident communication is a craft with its own patterns and its own pillar. The incident response guides cover updates, roles, and escalation rhythms properly. So here we leave it at the one rule that protects your troubleshooting. Promised update times are the only schedule you owe the audience, and they buy you the quiet between them.


Knowing when to stop


Somewhere past a certain point, continuing alone stops being persistence and starts being ego. The professional move is to see that point coming. Set a time box when you start — a pre-decided limit like "if I have made no measurable progress in thirty minutes, I change approach or add people." Deciding the limit in advance matters. The you of thirty minutes from now is not the person best positioned to notice you are stuck. You will be tired, invested, and sure the answer is five minutes away. It always feels five minutes away.


Alongside the timer, watch for the tripwires that mean stop now. You are re-running tests you already ran, hoping for different results. Your hypotheses have stopped being ranked guesses and become vibes. Each new idea is riskier than the last. The blast radius is growing. Or you are about to do something irreversible mainly because it is the only idea left. Any one of those is the cue to escalate: to a colleague, a manager, or the on-call chain if you have one. The escalation and on-call guides cover building that ladder. Escalating with a clean log is not an admission of defeat; it is a handoff in a relay you are still running. And remember the priority the MTTR guides hammer on: during an outage, restoring service beats explaining it. A rollback or failover you can defend now outranks a root cause you might find eventually.


When you do hand off — to a colleague or up the chain — the quality of the handoff decides whether your thirty minutes of work transfers or evaporates. A good briefing takes four sentences. Give the situation: what is broken, for whom, since when. Give the background: what changed recently, what this system does. Give the actions so far: what you tested, what died, what you changed and rolled back. Then give the ask: what you want from them, whether a second brain, a decision, or an authority you lack. If you kept the log, the briefing writes itself. If you did not, the next person starts from zero and re-runs your first half hour, on the clock, in front of the same audience.


Calling the vendor is its own small art. Before you dial, have your support contract or entitlement details ready. Have the device model and serial number, plus the exact error text, verbatim. Prepare a tight timeline of when it broke and what changed around then. Have your diagnosis log of what you have tried, and the evidence files — logs, captures, configs — already exported. Have a way to give their engineer eyes on the system. A prepared caller skips the first forty minutes of every support script, and support engineers audibly perk up when the caller has timestamps.


After the storm


You fixed it, it is late, and every cell of you wants to go home. Spend ten more minutes anyway. Confirm the fix is actually holding (watch the graphs, not just the absence of complaints). Roll back any temporary bypasses you left open. Write down what happened while it is still true in your memory rather than legend. Those notes feed the post-incident review, which is where an outage pays its tuition — and where "we should really monitor that" becomes a checklist item instead of a wish. If your fix was a mitigation rather than a confirmed cure, add a tripwire before you leave. Aim a monitoring check or alert straight at tonight's symptom, so a recurrence announces itself instead of ambushing you. The pressure will fade by morning. The notes and the tripwire are the only parts of tonight that appreciate in value.


Where to go next


Nothing in this guide replaces the core method — the layer walks, the halving, the ranked hypotheses, the what-changed question all still run during an outage. Pressure just adds the guardrails: reads before writes, one change at a time, a rollback for every risk, a time box on your own stubbornness. Practice the guardrails on small fires and they will hold in big ones. The rest of the method lives at A General Troubleshooting Methodology, and the calm, boring habit of watching your network before it burns starts with the monitoring fundamentals.


Put this into practice with SysPeep Monitor

Half of outage panic is not knowing what is actually down — and exploratory poking is how responders cause second outages. SysPeep Monitor gives you the read-only view first. Live dashboards and network maps show what is unreachable, since when, and what depends on it. Alert rules surface the cause instead of burying it in noise. Calmer inputs, calmer hands. The trial is free on any Windows machine.

Download Free TrialLearn About SysPeep