Home › Topics › Router & WAN Link Monitoring › Failover Readiness

WAN Failover Readiness: Testing the Backup Before You Need It


Somewhere in your network there is a failover configuration that was tested exactly once — on the day it was installed. It was probably tested by the person who installed it, possibly with a thumbs-up and the words "yep, that works." Since then, the network has changed a hundred times and the failover has been tested zero times. It now exists in a quantum state: simultaneously working and broken. Nobody will know which until the primary circuit dies and the whole company finds out together.


Here is the uncomfortable truth about failover of any kind. It is a machine that must work perfectly on its worst possible day, after months or years of neglect, in front of an audience. Untested failover fails at an impressive rate. This is not because the technology is bad, but because "configured once" and "ready now" are different claims. Only one of them can be verified from a chair. The good news is that readiness can be verified, most of it continuously and automatically. You can verify the rest with a calm scheduled drill that takes an evening.


This guide covers both halves. First, it covers turning failover readiness into a monitored state your dashboards report on all year. Then it covers running deliberate failover drills: planning them, executing them without chaos, deciding when to abort, and writing down what you learned. By the end, "will it fail over?" stops being a hope and becomes a measurement with a date on it.


In this guide

  • Why failover fails exactly when you need it
  • Readiness as a monitored state
  • Planning a failover drill
  • Running the drill: what to do and what to watch
  • Documenting the results
  • The confidence payoff
  • Where to go next

Why failover fails exactly when you need it


Failover setups do not break randomly; they break in a handful of predictable ways, and every one of them is invisible until tested. Knowing the list is half the defense, because each failure mode maps to a specific readiness check later in this guide.


Config drift

Drift is the slow divergence between the primary path and the backup path as the network evolves. Every change made to the primary is a change that someone must remember to mirror onto the backup path. That includes a new subnet, a new firewall rule, a new VPN tunnel, or a DNS forwarder pointed somewhere new. Nobody remembers every time. Each forgotten mirror is harmless on its own and invisible in daily operation, because the backup carries no traffic to reveal it. Months later, failover technically succeeds — packets flow. But the guest network is offline, one branch VPN will not build, and the new phone subnet cannot reach anything. Drift is why a failover that worked at installation quietly rots into one that "works, except."


A backup that died in its sleep

The simplest failure: the backup circuit itself has been down for weeks, and failover dutifully switches traffic onto a corpse. An idle circuit produces no complaints when it dies. Without a probe running through it continuously, its funeral is held the same day as the primary's. The mechanics of keeping an idle line honestly monitored are covered in multi-WAN visibility.


Missing routes

A router needs to know where to send traffic on the backup path, not just that the backup exists. It is common for the backup to carry a plain default route to the internet. Meanwhile, the primary also carried routes to the data center, the other branches, or the management network — routes nobody duplicated. Failover then works beautifully for web browsing and fails completely for the business applications, which is roughly the opposite of the priority order anyone wanted.


The forgotten dependency

The backup path usually presents a different public address to the world. An ecosystem of things quietly depends on the primary's address. There are VPN tunnels pinned to it and a partner's firewall that allowlists it. There is a cloud service restricting logins by source address, or an email relay that only accepts your mail from it. None of these belong to the router, so no amount of router configuration fixes them — and none of them announce themselves until the address changes. This is the failure mode that turns "failover worked" into a day of mystery tickets that all share one invisible cause.


The monitoring blind spot

One dependency deserves its own headline because it is the one that hides all the others. Your monitoring and alerting may themselves ride the primary path. Remote probes may report home across the WAN. Alert emails may leave through a relay reachable only via the primary. The on-call notification service may allowlist your primary address. With any of these, the system that should tell you how it went is mute or blind at the exact moment failover engages. The failure is wonderfully circular: everything broke, and the tool that says so is among the everything. Test, specifically and on purpose, that an alert generated during failover actually reaches a human phone via the backup path.


The failback trap

Failing over is only half the machine; failing back is the other half, and it has its own bugs. Some setups fail over cleanly and then flap — bouncing between circuits when the primary's recovery is unstable. That breaks every session twice an hour, which users experience as worse than the outage. Others never fail back at all and nobody notices the site running on the expensive metered backup for a month. Failback deserves its own test and its own alert.


Readiness as a monitored state


Now flip each failure mode into a check. The mental shift that makes this work: "ready to fail over" is not a property you configured. It is a state you can measure. Most of it can be measured continuously by the same monitoring system that watches everything else. Aim for a small dashboard group — call it failover readiness. Every light being green should mean the machine would probably work if the primary died right now.


Readiness checkQuestion it answersHow often
Backup circuit probe (out the backup path)Is the backup line alive and clean, even while idle?Continuous
Backup interface state and errorsIs the physical layer of the backup healthy?Continuous
Failover mechanism statusDoes the router itself believe the backup is available?Continuous
Route presence on the backup pathDo the important destinations exist beyond the default route?Each drill + after changes
Dependency list review (addresses, allowlists, tunnels)What breaks when our public address changes?Each drill + after changes
Config parity reviewHas drift crept in since the last look?Each change + quarterly
Days since last successful drillHow stale is our confidence?Continuous counter

The first three rows are ordinary monitoring checks. Use probes and SNMP against the backup path, exactly as you would monitor any circuit. Add whatever status the router exposes about its own failover logic. Most platforms will report which path is currently active and whether the standby is considered usable. Collect that if you can reach it.


While you are looking at that failover logic, review what it actually watches. The router's opinion of "primary dead" comes from somewhere — usually its own little health check: link state on the interface, or a tracked ping to some target. That choice ages badly. If the tracked target was a server someone has since decommissioned, the router now believes the primary is permanently dead. Or it never notices when the primary truly dies. Which happens depends on the logic's direction. If the check is link-state only, the router will cheerfully keep using a primary whose interface is up while the line beyond the handoff is a brick. Two readiness questions, then, are worth asking once per drill. Does the router's detection test through the circuit rather than just the port? And does its target still exist and mean what it meant? A failover machine with a rotten trigger is broken in the most embarrassing possible way: every component works except the part that decides.


The last row deserves a special word, because it is the sneaky-good idea of this whole approach. Treat the age of your last successful drill the way you treat certificate expiry. Confidence decays on a schedule — every network change since the last drill adds a little rot. So give the decay a number and an alert. Warn when the last successful drill is older than, say, six months; alert at a year. It sounds almost silly, and it is the difference between organizations that drill and organizations that mean to. A date on a dashboard nags better than a resolution ever has.


Tip: put the middle rows — routes, dependencies, parity — into your change process, not just your calendar. The cheapest moment to prevent drift is inside the change that causes it. Put one line on the change checklist: "does this need mirroring to the backup path, and does it pin anything to our primary address?" That catches drift at the source, while the knowledge is still in someone's head.


Planning a failover drill


Continuous checks catch rot, but only a real switchover proves the machine end to end. That means deliberately failing over on a day you chose, with everyone informed and an undo plan in your pocket. Admins avoid this because it feels like courting disaster. Planned properly, it is the opposite. A drill is an outage with all the scary parts removed — you know the cause, you know the fix, and you picked the hour.


Plan four things in writing before touching anything. Scope and success criteria: what you will test and what "pass" means, concretely. For example: traffic shifts to backup within a minute, VPN tunnels re-establish, and voice calls survive or reconnect. The branch subnets stay reachable, and failback is clean. Vague drills produce vague confidence. Method: a soft failover (administratively shifting traffic — disabling the primary route or interface from the console) is polite and easily reversed. A hard failover (physically unplugging the primary's cable) is ruder and more honest, because it tests detection as well as switching. A sensible progression is soft first, hard once soft passes. The window: pick a time that is quiet but real — enough traffic to prove something, few enough users to bother. Announce it like any maintenance. Abort criteria: decide in advance what sends you back. That could be a critical service unreachable on the backup, the drill exceeding its time box, or anything safety-related. Decide the maximum time you will stay failed over. Mid-incident is a terrible moment to negotiate with yourself about whether to push on; pre-written criteria make the abort decision boring.


Staffing is part of the plan too, and the magic number is two. One person drives the change while the other watches the dashboards and the clock. The driver's attention will be inside a console session. The observer owns the graphs, the timer, the abort criteria, and the notes. The observer has explicit permission to call the abort. Solo drills are possible. But a lone admin alt-tabbing between the router console and five dashboards at 9 PM is exactly how a small drill finding becomes a large unplanned outage. If the site is remote with nobody in the building, arrange in advance for hands that can reach the hardware. This may be just someone who can move a cable back. Arrange this help before choosing the hard-failover method.


One more planning item: tell your monitoring what is about to happen. Suppress the pages and notifications for the affected site during the window. That way, the drill does not light up phones and burn everyone's trust in alerts. But keep collecting everything, because the drill's entire value is in the data recorded while traffic shifts. Suppression is about silencing notifications, never about pausing measurement. The craft of doing that cleanly is its own topic, covered in maintenance windows and alert suppression.


Running the drill: what to do and what to watch


On the night, the drill itself is a short script. Resist improvisation; the boring version is the good version.


  • Step 1 — Baseline: confirm every readiness light is green. Screenshot or note the healthy graphs: per-circuit probes, utilization on both circuits, VPN tunnel states. You cannot judge "back to normal" later without a picture of normal.
  • Step 2 — Open the window: start the suppression window, post the "starting now" note to whoever was promised one, and start a timer.
  • Step 3 — Fail over: execute the chosen method — shift the routing or pull the cable — and immediately start watching rather than touching.
  • Step 4 — Watch the switch: on your dashboards, the primary's probe should die while the backup's probe stays healthy. The backup's utilization should leap as the primary's collapses. Note the time from failure to traffic flowing again — that number is your real-world failover time, and management will ask for it someday.
  • Step 5 — Test like a user: work through your success criteria by hand. Place a voice call, open the business application, confirm each remote-dependent service, check that inbound things (VPN peers, mail) still arrive. This is where forgotten dependencies surface; write each one down without fixing anything yet.
  • Step 6 — Soak: stay failed over long enough for real traffic to exercise the path — half an hour is a reasonable floor. Watch the backup's utilization, latency, and drops to learn whether it truly carries the load or merely survives it.
  • Step 7 — Fail back: restore the primary and watch the reverse migration with the same attention: traffic should return, tunnels should rebuild, and nothing should flap. Failback is a first-class test, not cleanup.
  • Step 8 — Close out: confirm the graphs match your Step 1 baseline, end the suppression window, and send the two-line "drill complete" note while details are fresh.

If an abort criterion trips at any step, execute the abort without debate, note exactly what failed, and count the drill a success anyway. An ugly discovery during a planned window with rollback ready is the entire reason drills exist. The only failed drill is one that teaches nothing.


Documenting the results


A drill that lives only in memory expires in a week. Keep a plain log — one entry per drill, short enough that you will actually write it:


FAILOVER DRILL LOG - Site: HQ
Date/window:      (planned start-end, actual start-end)
Method:           soft (route shift) / hard (cable pull)
Failover time:    time from cut to traffic flowing on backup
Success criteria: pass/fail, one line each
Found broken:     - guest VLAN unreachable on backup (missing route)
                  - partner SFTP refused our backup address (allowlist)
Fixed during:     nothing (fixes go through change process)
Tickets opened:   (numbers)
Failback:         clean / issues (describe)
Verdict:          PASS WITH FINDINGS
Next drill due:   (date - feeds the readiness dashboard)

Two habits make the log pay compound interest. First, every "found broken" line becomes a ticket before the log is closed — drills find the same drift twice when findings evaporate. Second, the "next drill due" date goes straight into the readiness dashboard from earlier, restarting the confidence clock. Over a few cycles the log also becomes an oddly persuasive document: a dated record of failover working, timed and witnessed. That is exactly what an auditor, an insurer, or a nervous executive means when they ask "are we resilient?"


The confidence payoff


Here is what changes once readiness is monitored and drills are routine. The 2 AM primary-circuit failure stops being an event. Failover engages. The redundancy-degraded alert opens a ticket instead of a war room. And the provider gets called in the morning by someone who slept. Provider maintenance notices stop being threats, because your network does the thing it has rehearsed. On-call gets calmer: the runbook step "confirm failover engaged" refers to a procedure someone has actually watched work, with a known timing, rather than folklore. And when the business asks what the second circuit's monthly bill buys, you have dates, timings, and a log — not a shrug and a theory.


The deeper payoff is cultural. Networks that never test failover treat every failure as a crisis, because every failure is also a first performance. Networks that drill treat failure as a rehearsed scene. That calm shows up everywhere: in cleaner incident notes, shorter outages, and admins who reach for evidence instead of adrenaline.


Where to go next


Failover readiness sits on top of everything else in this pillar. It assumes each circuit is measured like infrastructure (WAN circuit health) and both paths are watched independently (multi-WAN visibility). If your backup paths protect branch offices with no staff, monitoring WAN links to remote sites continues the story. The rest of the pillar lives at Router & WAN Link Monitoring.


Put this into practice with SysPeep Monitor

A failover drill is only as good as what you can see during it. SysPeep Monitor keeps independent checks running on both circuits and graphs the switchover as it happens. Its maintenance windows silence notifications during the drill while every metric keeps recording. Set up the readiness dashboard from this guide with the free trial on any Windows machine.

Download Free TrialLearn About SysPeep