Monitoring Firewall HA Pairs: Ready, or Just Pretending?
Somebody in your organization once made a very sensible argument. The firewall touches everything, so if it dies, everything dies. Therefore we should buy two. The invoice was signed, the second unit racked, high availability configured, and everyone has slept better ever since. Here is the uncomfortable question this guide exists to ask: how do you know the second one still works?
Because here is the trap built into every high-availability pair. The standby unit does nothing all day. No traffic exercises it, and no users complain about it. And — on far too many networks — no monitoring watches it. That is because the monitoring points at the shared address that always answers from the active unit. A standby can sit there dead for months, dashboard green, everyone confident. And the truth only comes out on the day the active unit fails and the "failover" becomes a plain, old-fashioned outage. You did not have an HA pair. You had one firewall and a rumor.
This guide turns the rumor back into a pair you can trust. It covers how HA actually works in plain language, and why standbys die silently. It explains how to monitor both members independently, and what sync status and configuration drift mean. It covers the alerts every failover event deserves, plus split-brain and its warning signs. It also covers the failover drill that proves the whole arrangement — on your schedule, not fate's.
In this guide
- HA pairs in plain language
- The quiet standby problem
- Monitor both members, independently
- Sync status and configuration drift
- Failover events deserve alerts
- Split-brain, explained calmly
- The failover drill
- What "ready" looks like: a checklist
HA pairs in plain language
In the most common arrangement, active/passive, two identical firewalls stand side by side. One (the active) handles all traffic. The other (the standby or passive) handles none — it just stays configured, powered, and ready. The pair shares a floating address (often called a virtual address). The network sends traffic to that address, and whichever member is currently active answers for it. When a failover happens, the standby takes over the floating address and traffic follows it, with the rest of the network mostly none the wiser. Active/active designs, where both members carry traffic at once, exist too. But active/passive is the sane default for most networks. Everything in this guide applies to both.
Two private conversations run constantly between the members. The first is the heartbeat: a steady "are you alive?" exchange over one or more dedicated links. That is how each member knows the other still exists. Missed heartbeats are what trigger a failover. Other conditions the pair watches also trigger a failover, like the active member losing a critical interface or failing an internal health check. The second conversation is state synchronization. The active member continuously copies its session table over to the standby. The session table is the note the active member keeps about every conversation in flight (covered in depth in Session Tables and Connection Limits). That copying is what makes a good failover feel like nothing. The standby already knows about your download, your video call, your database connection, so they continue instead of resetting. Without state sync, a failover technically succeeds while every user's session dies — which users experience as an outage with extra steps.
It is worth knowing what, exactly, convinces a pair to fail over, because those conditions are configurable and each is monitorable. Beyond missed heartbeats, most products let the active member watch its own critical interfaces. For example, lose the outside link and hand over to the peer, whose outside link may be fine. Most products also let it run internal health checks on its own processes and accept an administrator's manual command. That is the graceful version you will use for drills. Read your product's failover conditions once, carefully. They are the pair's constitution. Every drill and incident afterward makes more sense when you know the rules it is following.
Alongside session state, the pair also synchronizes configuration. A rule added on the active member is pushed to the standby so both hold identical policy. Keep those two syncs separate in your head. State sync is about surviving this minute's traffic. Config sync is about the standby enforcing the same rules once it takes over. Both can quietly break, and they break differently.
The quiet standby problem
Every failure needs a witness before it becomes a fact. The active firewall has thousands of witnesses — every user, every application, every graph. The standby has none. Nothing depends on it, so nothing notices when it degrades. That asymmetry is the entire reason HA pairs disappoint. It is worth respecting how many different ways a standby can rot while the pair reports "healthy" to a casual glance:
- Hardware death. A power supply fails, a disk fills or dies, or a fan stops and the box cooks. All of that is invisible if nobody polls the standby's own sensors.
- The half-finished upgrade. Someone upgraded the active member, planned to do the standby "next window," and got interrupted. The pair now runs mismatched software versions — a state in which sync and failover behavior is at best degraded and at worst untested.
- The unplugged sync cable. After rack work, a heartbeat or sync link did not come back. The pair may keep running on a surviving link (if there is one) with nobody aware the redundancy of the redundancy is gone.
- Expired licensing. On products where inspection features are licensed per device, a lapsed standby license means failover technically works — and security features quietly do not.
- Stale configuration. Config sync broke months ago; the standby holds an antique policy and nobody has compared them since.
None of these announce themselves. Every one of them is trivially detectable by monitoring — if the monitoring actually looks at the standby, which brings us to the fix.
Monitor both members, independently
The single most important sentence in this guide: monitor each member through its own dedicated management address, not just the floating address. The floating address always leads to whoever is active. Polling it tells you "some firewall is working," which is exactly the rumor we are trying to retire. Each member needs its own management address (every HA-capable product supports this, and most require it). And your monitoring system should treat the two members as two first-class devices. That means two entries, two sets of checks, two alert streams.
Then give the standby the same attention the active gets. Per member, that means reachability of its own management address, CPU and memory, its own disk or storage, and temperature, fans, and power supplies. The hardware sensor story from Firewall Health: The Metrics Behind the Little Green Light applies to each chassis separately. A dead PSU in the standby is precisely the kind of failure that waits for the worst day. Watch uptime too, because an unexpectedly low uptime means the member silently rebooted. A member that reboots on quiet Tuesdays will pick failover day to do it again. One product-specific note to check in your documentation: some platforms restrict what a standby will report or answer while passive. So verify your polling actually works against the standby now — not during the incident review.
Monitoring two members does create one piece of homework: alert hygiene. When a real failover happens, a naive setup fires a burst of alarms — member down, role changed, floating address moved. During planned maintenance on the standby you do not want "standby offline" paging the on-call at full urgency. Use your monitoring system's dependency and maintenance-window features to shape this. Failover events should page loudly. A standby's health problems should alert at working-hours urgency. And nothing about the standby should ever be suppressed permanently. Permanent suppression is how quiet standbys are made.
Finally, poll each member's view of the HA relationship itself. That means its role (active or standby), its opinion of its peer, and the health of each heartbeat link. Which is the next section.
Sync status and configuration drift
Every HA-capable firewall can report the pair's relationship status. It can do so over SNMP, an API, or at minimum a CLI command your monitoring can run on a schedule. A generic status readout looks like this, and every line of it is monitorable:
fw01a> show ha status Local state: active Peer state: standby (healthy, reachable) Heartbeat links: 2 of 2 up State sync: enabled, current Config sync: IN SYNC (last change pushed successfully) Software versions: local and peer match Last failover: none since counters cleared
Three of those lines deserve standing alerts. Peer state: a peer may be down, unreachable, or in a failed or degraded state. Anything other than a healthy standby means you are running without a spare right now. That is an incident even though users feel nothing. Heartbeat links: alert when any single link goes down, even if others survive. Heartbeat links are usually redundant precisely because losing all of them causes the split-brain mess described below. Running on the last surviving link is the firewall equivalent of one engine left. Config sync: alert when the pair reports out-of-sync for more than a brief settling period after a change.
Config drift deserves its own paragraph, because it is sneakier than a broken sync flag. Drift happens when the standby's configuration stops matching the active's. Possible causes are a sync failure nobody noticed, a hurried change made directly on the standby during some past incident, or a push that half-completed. The pair may still report basic health while holding different rules. A failover then silently changes network policy. That is the kind of failure that takes days to diagnose because nobody suspects the firewall that "just took over successfully." Watch the sync status continuously. Compare configurations (a scheduled diff of exported configs is a perfectly respectable tool). Treat version mismatch between members as a temporary state that belongs inside upgrade windows only. Alert if it persists beyond one.
Failover events deserve alerts
A failover is the pair doing its job — and it is also, always, news. Every HA state transition should page somebody, for two reasons. First, from the moment of failover you are running on the spare with no further safety net. Whatever killed the first member may be warming up for the second. Second, a failover is a symptom: something made the active member fail or falter. That something needs diagnosis while the trail is fresh. The worst version of this story is the silent one. The pair failed over during a long weekend. Nobody was alerted, and the "redundant" network ran on one engine for months. When the survivor eventually hiccupped, the outage was blamed on bad luck. It was actually a calendar.
Wire the alert from both directions. One is from the firewall (SNMP traps or log messages announcing state changes — part of the shortlist in Firewall Logs: Alerting on What Matters). The other is from your poller noticing a member's role changed between polls. Alert on flapping with special urgency. A pair failing back and forth repeatedly is two units taking turns being unhealthy. That usually means a shared cause — a flaky monitored interface, an overload condition both hit in turn. Flapping also usually means a network experiencing a rapid heartbeat of micro-outages. And know your product's failback behavior. Some pairs automatically return to the original member when it recovers (an extra transition worth expecting and alerting on). Others stay put until told. Neither is wrong; being surprised by either is.
Split-brain, explained calmly
Split-brain is what happens when the two members lose sight of each other while both remain alive. Each one, hearing no heartbeat, concludes the same reasonable thing: "my partner is dead — I must take over." Now both are active. Both claim the floating address. The surrounding network receives two contradictory answers about where that address lives. Traffic starts flowing to whichever member answered most recently. Some conversations land on one firewall, some on the other, and sessions break as paths flip. The symptoms from the outside are deeply confusing. They include intermittent connectivity that comes and goes by the minute, address-conflict warnings, and users in one office fine while the next office drowns. It looks like gremlins. It is actually two well-behaved machines each dutifully following the rulebook for a world where the other one no longer exists.
Here is where monitoring both members independently pays a bonus. Split-brain is nearly invisible from the floating address (which answers, chaotically, from both) but obvious from the dedicated ones. Your poller asks each member for its role. The moment both answer "active," you have a detection no single vantage point could give you. Some monitoring systems let you alert on exactly that comparison. If yours does not, alerting on each member's own "peer unreachable" status gets you to the same place moments later.
The cause is almost always the same: every heartbeat path failed at once while both members survived. A shared switch that carried all heartbeat traffic went down, a cable bundle got unplugged together, or heartbeats were routed through infrastructure that itself failed. That gives you the prevention list directly. Use at least two heartbeat links on physically separate paths (one direct cable between the boxes is excellent precisely because nothing else can break it). Never let all heartbeat paths share one switch or conduit. And — the monitoring angle — alert on the first heartbeat link failure. That is because split-brain requires losing the last one. And the last one always used to be the second-to-last. Many products also support a tiebreaker mechanism — a third reference point a confused member can consult before seizing control. If yours does, enabling it is cheap insurance. If you ever do land in split-brain, the practical first aid is decisive: pick a survivor, and pull power or network from the other. Two half-active firewalls are worse than one.
The failover drill
Everything so far tells you the pair should fail over. Only one thing proves it: doing it, on purpose, in a maintenance window, while watching. Pairs that have never been deliberately failed over have a way of revealing their surprises at the least deliberate moment possible. A drill converts those surprises into calm Tuesday-evening findings. A sane drill runs like this:
- 1. Schedule and announce. A real window, with the courtesy warning that brief blips are possible. A drill that would be catastrophic if it failed is not a drill you can learn from — pick the window accordingly.
- 2. Verify readiness first. Both members healthy, sync current, configs matched, versions matched — the checklist below. Drilling a pair you already suspect is broken is just scheduling your outage.
- 3. Start the canaries. A large file transfer, a continuous ping through the firewall, and a live video call make excellent witnesses. One shows whether sessions survive, one measures the blip in seconds, and one tells you what a user would actually feel.
- 4. Fail over gracefully. Use the product's administrative failover command. This is a fire drill, not arson. Save cable-pulling for a more ambitious exercise once graceful failovers are boring.
- 5. Watch and record. Did the floating address move within a few seconds? Did the canaries survive? Did the alerts you built actually fire? (A drill is also a test of the monitoring.)
- 6. Stay on the ex-standby for a while. Stay for minutes at minimum, longer if policy allows. Real traffic finds problems that status pages cannot. This is the only moment the ex-standby ever carries real traffic with the whole team watching.
- 7. Fail back per your product's preference, then write it down. Duration of the blip, anything odd, anything to fix. Repeat after major upgrades and on a regular cadence, so "the pair works" stays a recent fact instead of an old memory.
Tip: the first drill on a long-undisturbed pair fails more often than anyone expects — which is precisely the argument for doing it. Whatever it uncovers was already true; the drill just moved the discovery into a window with coffee, colleagues, and a rollback plan.
What "ready" looks like: a checklist
Condensed to one table you can review in five minutes a month — every row is something this guide showed you how to watch:
| Item | Ready looks like |
|---|---|
| Member reachability | Both management addresses polled independently, both green |
| Standby health | CPU, memory, disk, PSU, fans, temperature monitored on the standby itself |
| State sync | Reported current; alerts on any sync failure |
| Config sync / drift | In sync; periodic config comparison finds no differences |
| Software versions | Matched, except inside a planned upgrade window |
| Licenses | Matched on both members, none lapsed |
| Heartbeat links | Redundant, on separate paths, each link individually monitored |
| Failover alerting | Any state transition pages someone — proven during the last drill |
| Last drill | Recent enough to remember, documented, findings fixed |
Where to go next
An HA pair is two of everything — which means everything in this pillar applies twice. The per-member vitals come from Firewall Health: The Metrics Behind the Little Green Light. The sync that failover depends on protects the session table explained in Session Tables and Connection Limits. And the state-change events worth waking up for are wired in Firewall Logs: Alerting on What Matters. The full series lives at Firewall Monitoring Beyond Up/Down.
Put this into practice with SysPeep Monitor
A standby that nobody watches is a rumor, not redundancy. SysPeep Monitor treats both members of your pair as first-class devices. It polls each one's health and hardware sensors over SNMP and alerts on state changes the moment they happen. And since the monitor should not be a single point of failure either, it supports high-availability clustering of its own. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep