Home › Topics › Catching Intermittent Problems › Reproducing Rare Failures

Reproducing the Unreproducible: Forcing Rare Failures into the Open


Everything else in this topic is patient: trend for weeks, build timelines, arm tripwires, wait. This guide is the one that stops waiting. At some point in a hunt you have a decent theory, a fault that visits once a week, and no appetite for seven more days per experiment. The move that changes the tempo is reproduction. Instead of waiting for the fault to choose its moment, you recreate its trigger conditions on purpose. You make it perform on your schedule.


A fault you can summon at will is mostly beaten. You can watch it happen with every instrument running. You can try a fix and know within an hour whether it worked. And you can hand a vendor a recipe instead of an anecdote. I have handed vendors both. Only the recipe ever came back with a fix attached. The trick is doing this without lying to yourself, one variable at a time with honest observation windows. You also need to avoid breaking the production network you are paid to protect. That is why this guide spends real time on safety and consent.


You will learn the three families of forcing technique: load and soak, methodical toggling, and condition amplification. You will also learn the etiquette of running them against production. And you will learn the final discipline that closes every hunt: proving the fix stuck. That uses the same trap that caught the fault in the first place.


In this guide

  • From ambush to appointment: why force it
  • Start from a hypothesis, not a hunch
  • Load generation and soak tests
  • Toggling suspects, one at a time
  • Amplifying the conditions
  • The ethics and safety of testing on production
  • Proving the fix stuck
  • A worked example: the uplink that hated warm afternoons
  • Where to go next

From ambush to appointment: why force it


Waiting for a rare fault means every observation costs you days and every experiment costs you a recurrence. Reproduction flips the economics. Once you can trigger the fault deliberately, four things get cheap at once:

  • Observation. The fault now happens while every instrument is running and you are watching. This is the one time the observer effect works in your favor, because you chose the moment. If the diagnosis needs packets, a deliberately triggered fault is also the easiest thing in the world to capture cleanly. The guide to capturing intermittent problems covers that rig.
  • Experimentation. "Does disabling this feature help?" becomes an afternoon's question instead of a month's vigil.
  • Escalation. A vendor who hears "it fails sometimes" opens a ticket; a vendor who receives steps-to-reproduce opens an engineering case.
  • Verification. After the fix, you re-run the recipe. Silence from a summoned fault is worth ten times the silence of a fault that merely has not visited lately.

There is also a quieter benefit: reproduction converts dread into procedure. A ghost that ambushes you owns you; a fault with an appointment is just work. Steps-to-reproduce do for a manager's patience exactly what they do for a vendor's.


One boundary is not negotiable: some faults should not be summoned. The symptom might corrupt data, drop a safety-relevant system, or take down something with no tolerance for another outage. In any of those cases, deliberately triggering it is off the table no matter how curious you are. Those hunts stay in the patient lane, closing in through traps and evidence until the fix can be made on inference alone. Reproduction is a power tool. Part of owning power tools is knowing which jobs they are wrong for.


Start from a hypothesis, not a hunch


Reproduction is aimed, and the aim comes from the earlier phases of the hunt. A usable hypothesis has the shape: when conditions X (and maybe Y) hold, symptom Z appears within some window. "When the uplink runs saturated for an hour on a warm afternoon, the error counters climb." "When the offsite copy is still running at the 05:10 rekey, the tunnel drops." Concrete conditions, concrete symptom, rough latency between them.


If you cannot state your theory in that shape yet, you are not ready to force anything. You would just be shaking the network to see what falls out. That mostly produces new mysteries and new outages. Go back one step: long-term trending finds the conditions that cluster around occurrences, and timeline correlation finds what always precedes them. The one honest exception is a structured stress test used as exploration. For example: "this branch feels fragile under load; let us load it and watch everything". That is legitimate as long as you admit it is a fishing trip. You must also rig it with the same safety rules as everything below.


Northgate Retail had tills in one store that dropped off the network for a minute, "a couple of times a week, afternoons". The vendor sent an engineer for two days. Nothing happened, and the engineer left with a note saying the store network was healthy, which, for those two days, it was. The next afternoon the admin reread the ghost log. They noticed that the store manager had paused the hourly stock sync "so as not to get in the engineer's way". The admin ran the sync by hand at two o'clock with every trap armed. The tills dropped six minutes later, on the first attempt. The four-line recipe went to the vendor that evening, and the engineering case was open by morning.


Write the hypothesis down, with its prediction, before you test. It keeps you from quietly redrawing the target around wherever the arrow lands.


Load generation and soak tests


Load generation means creating artificial work. The system experiences, on demand, the stress it normally only meets at its worst moments. That could be traffic on a link, operations against a server, sessions through a firewall, or writes against a disk. The tooling ranges from purpose-built throughput testers to a shameless loop copying a big file. Fancy is optional; what matters is matching the shape of the load to the hypothesis. Saturating bandwidth is not the same stress as many small packets per second. Many small packets per second are not the same as thousands of concurrent sessions. Thousands of concurrent sessions are not the same as disk pressure. Recreate the stress your evidence points at, not just "some load."


Stress the hypothesis namesLoad shape to generateSimple generator
Link saturationSustained high throughputThroughput tester, or a big-file copy loop
Device forwarding limitsMany small packets per secondTraffic generator in small-packet mode
Session/state exhaustionThousands of concurrent connectionsScripted parallel client sessions
Storage pressureHeavy read/write mix on the volumeParallel file operations against a scratch area

A soak test is load's patient sibling: moderate, realistic stress sustained for hours or days. Soaks exist because some faults need accumulation time — leaks that grow, tables that fill, queues that creep, temperatures that plateau. A ten-minute blast will never mature those faults. A spike answers "does it break under pressure?"; a soak answers "does it break under pressure, eventually?" Rare faults are disproportionately the second kind, which is why the soak is the reproducer's workhorse.


For pushing a path to saturation with zero special tooling, a loop like this from a machine on the suspect segment does the job. Aim it only at a scratch share. See the safety section before running anything like it in business hours:


while ($true) {
  Copy-Item -Path \\fs1\scratch\big-test-file.bin -Destination C:\scratch\ -Force
  Remove-Item -Path C:\scratch\big-test-file.bin
}

Whatever generates the load, the judges of the test are the instruments you already built. The tripwires stay armed. The monitoring keeps polling. And "did the fault occur?" is answered by the symptom check's log, not by anyone's impression. Run the load and watch the graphs live. And let the same traps that caught the wild fault certify the captive one — same log signature, same symptom shape. A fault that looks similar but signs differently is a different fault, and finding one of those is a detour, not a victory.


Toggling suspects, one at a time


The second family is subtraction: change one suspect element and see whether the fault still happens. Swap the patch cable. Move the device to a different switch port. Replace the optic (the small pluggable module that turns the fibre's light into a signal the switch can use). Disable the power-saving feature, the offload setting, the inspection rule. Each toggle is an experiment, and the entire value of the method lives in one word: one. Change two things and a result tells you almost nothing. Change five and you have converted a mystery into a bigger mystery with fresher cabling.


The protocol that keeps you honest:


1. Predict. Write what this toggle should do if the hypothesis is right ("with the optic swapped, the soak should run clean").


2. Timestamp the change. Every toggle goes into the timeline with date, time, and exact detail — future-you must be able to reconstruct the sequence.


3. Fix the observation window in advance. Decide how long silence must last to count as a pass. A fair rule of thumb is several times the fault's longest usual gap. If it strikes every day or two, a quiet week is convincing. A quiet afternoon is not. Reproduction shortcuts this beautifully: if you can summon the fault in an hour, each toggle needs an hour, not a week.


4. Watch with the same traps. Same tripwires, same symptom check, same definitions of "occurred."


5. Conclude and revert. Fault still occurs: that suspect is cleared — put it back and move on (reverting keeps your config from drifting into an unexplainable pile of "temporary" changes). Fault gone: the suspect is implicated. Where the stakes allow, the conviction move is the reverse toggle. In that case, restore the old part, watch the fault return, then swap in the good one for keeps. Appears-on-demand, disappears-on-demand is the strongest causal evidence troubleshooting ever gets.


6. Bisect when the suspect list is long. With many candidates — eight patch panels, a stack of features — toggle half at once, learn which half holds the fault, and split again. You give up per-item information for speed, then recover precision inside the guilty half. If the symptom in question is packet loss along a path, the intermittent packet loss guide walks this exact physical-layer bisection in more detail.


Amplifying the conditions


Some triggers are rare because their ingredients rarely line up. Amplification makes the rare frequent by turning the relevant dials, so probability stops being the bottleneck:

  • Shorten the cycles. The suspect might be a periodic mechanism — a lease renewal, a rekey, a scheduled job. In that case, shrink its interval in a test scope so it fires every few minutes instead of daily. A collision that odds served up weekly now happens before lunch.
  • Force the overlap. If two jobs only hurt when they coincide, schedule them to coincide, deliberately, inside your window. You are not creating a new fault; you are fast-forwarding to the alignment.
  • Raise the concurrency. Script fifty simultaneous logins or file opens where the morning normally produces a ragged thirty. Thresholds that took a bad Monday to reach are now reachable on demand.
  • Ride the environment, do not fight it. For heat-suspected faults, schedule the soak for the naturally warmest window — the afternoon, the post-weekend closet — instead of improvising ways to heat equipment. Deliberately overheating hardware is how a diagnostic session becomes an incident; use the warmth the building already provides, and pair the test with temperature readings.

One caveat: amplification changes the system, and occasionally the amplified setup breaks in a new way instead. Always check the summoned fault's signature — log lines, symptom shape, affected scope — against the wild occurrences from your timeline before declaring a match. Fast-forwarding the film is fine; splicing in a different film is not.


The ethics and safety of testing on production


The uncomfortable truth is that many intermittent faults can only be reproduced on the production network. That is because the fault lives in the one marginal cable, the real traffic mix, or a scale no lab matches. Testing on production is therefore sometimes right — and it is only right inside a frame of consent and control. You are proposing to deliberately invite a failure into a system people depend on. That is a decision your organization gets to make with you, not one you make alone at your desk. "Quick harmless test" is the phrase every post-mortem quotes.


The frame, as a checklist:

  • Permission, explicitly, from whoever owns the affected service — and in writing, even if the writing is a two-line email.
  • A scheduled, announced window, sized so that the worst plausible outcome fits inside it with room to recover. If the worst case is not acceptable even in a window, the test does not run — full stop.
  • A defined blast radius. Know beforehand which users, services, and paths could feel the test. Confine the load with caps and scoping so a scratch-share soak cannot starve the payroll run.
  • An abort switch, tested first. One action — one command, one unplug — stops everything instantly, and you have rehearsed it before starting. Give the load generator a built-in stop timer too. That way, a meeting that runs long cannot leave a forgotten loop hammering a file server all night.
  • Eyes on, the whole time. A production test is the one hunt phase where you do watch live: dashboards open, tripwires armed, finger near the abort.
  • A revert path for every toggle, and a check afterward that the system is genuinely back to its baseline — not just "seems fine."
  • Timestamps into the timeline, and a heads-up to the helpdesk, so test-induced blips get labeled as such instead of spawning a fresh ghost hunt.

Warning: the pressure to skip this ceremony is real — it is Tuesday, the window is paperwork, the test is "just a big file copy." Resist it. The engineer who breaks production by accident gets sympathy; the one who breaks it with an unauthorized experiment spends a year rebuilding trust. Consent and control are what make deliberate failure a professional technique instead of a gamble.


Proving the fix stuck


The hunt does not end when the fix ships. Intermittent faults are infamous for "going quiet" for reasons that have nothing to do with you — the trigger's schedule drifted, the load pattern changed with the season. Mistaking that vacation for a cure is how the same ghost comes back in three months under a new ticket number. Closure needs evidence, and you already own the perfect instrument: the same trap that caught the fault becomes the judge of the fix.


Two tests, in order of strength. First, if you achieved reproduction, re-run the exact recipe against the fixed system: same load shape, same forced overlap, same amplified cycles. The fault used to appear within the hour; now it must not. A recipe that reliably summoned the fault before the fix and reliably fails to summon it after is about as close to proof as this trade offers. Second, leave the tripwires armed through a pre-declared confirmation window. Again, that means several times the old longest gap between occurrences. Let the symptom check's unbroken log of successes accumulate into a verdict. Announce the window when the fix goes in ("if the trap stays silent for three weeks, we close it"), so closure is a criterion, not a mood.


Then write the ending down: hypothesis, recipe, fix, confirmation, in the ticket, next to the timeline. And stand the rig down deliberately. Relax the hot polling and disarm the hunt traps per their expiry labels. But consider leaving one relaxed symptom check running as a permanent sentinel. It costs almost nothing, and if the ghost ever does find a new way in, you will hear about it from a trap instead of a user.


Tip: keep the reproduction recipe even after closure. It is the regression test for this corner of your network. After the next firmware update or config change in that area, twenty minutes of recipe tells you whether the old weakness is still fixed.


A worked example: the uplink that hated warm afternoons


The history: an office uplink drops into brief error bursts every week or two, afternoons only, caught originally by a delta trap on the error counters. Trend graphs add two correlations: bursts cluster on the warmest days, and always during heavy transfer periods. Hypothesis, written down: under sustained saturation while the closet is warm, the uplink's optical module starts corrupting frames within the hour.


Reproduction, by the book: a window is agreed for late afternoon, the helpdesk is warned, the abort is rehearsed. A file-copy soak saturates the uplink while dashboards and traps watch. On a mild morning as a control: ninety minutes clean. On a warm afternoon: error counters begin climbing at minute forty — same signature as the wild occurrences. The fault now performs on demand.


Toggle: the optical module is swapped for a spare; the warm-afternoon soak runs clean twice. Reverse toggle for conviction: the old module goes back in, and the errors return at minute fifty-two. That is appears-on-demand, disappears-on-demand — case closed on the module. The spare goes in permanently. The recipe is filed. And the delta trap stays armed for a month of quiet afternoons before the ticket is closed with a verdict instead of a shrug.


Where to go next


Reproduction completes the set. Trending finds the pattern, timelines find the precursor, tripwires catch the moment, and forcing plus proof ends the story. Everything before this guide waited for the fault to choose its moment. This guide takes the choice away from it. And a fault that keeps appointments is one you can finish. If you arrived here mid-hunt, the foundations live in why intermittent problems are so hard, and the full set of guides is collected in Catching Intermittent Problems.


Put this into practice with SysPeep Monitor

A reproduction session is only as good as the instruments watching it. SysPeep Monitor gives you live dashboards to watch while the soak runs. It provides threshold and alert rules that flag the first twitch of a summoned fault. And it supplies the before-and-after history that proves your fix held. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep