Home › Topics › East-West Visibility & Lateral Movement › Verifying Segmentation

Verifying Segmentation Actually Works


A firewall rule is a promise. It says traffic like this will be blocked, or allowed, at this boundary. The trouble with promises is that they are made once and then quietly outlive the reason they were made. Somebody adds a rule "just for the afternoon" to unblock a vendor. A VLAN gets trunked to one more switch during a cabling job. A test server is left with an address in the wrong subnet. None of these are dramatic; each is a Tuesday. Together, over a year, they turn the segmentation you designed into segmentation you merely hope you have.


The gap between the two has a name worth remembering: drift. Your zone-pair matrix describes the intended state — what should cross between zones. Drift is everything reality has done to that plan while you were not looking. This guide is about closing the gap: proving that the boundaries hold where you think they do. You use tests you run on your own network and evidence you already collect. You find the leaks before someone else does.


A boundary line applies throughout. Everything here is for networks you are authorized to manage. You are testing your own segmentation, from your own hosts, against your own target list. That is ordinary administration. The same actions against a network you do not run are not, so keep the scope to what is yours.


In this guide

  • Why segmentation drifts (the four usual culprits)
  • Two ways to check: active tests and passive evidence
  • Setting up authorized reachability self-tests
  • Reading the results matrix
  • Flow evidence: crossings that should not exist
  • Intended versus observed: putting them side by side
  • A worked example: finding a real leak
  • The periodic verification checklist
  • Fixing leaks and documenting exceptions
  • Where to go next

Why segmentation drifts (the four usual culprits)


Drift is not carelessness; it is the ordinary accumulation of small, reasonable changes. Recognizing the four common shapes helps you know where to look.


  • The temporary rule that stayed. A permit rule is added to solve a problem in the moment, with every honest intention of removing it later. Perhaps a vendor needs in, an application is broken, or a deadline looms. Later never has a calendar entry. This is the single most common leak. And it usually sits near the top of the rulebase where it overrides the careful blocks below it.
  • The over-trunked VLAN. A trunk is a switch link that carries many VLANs at once. When a switch is added or a link is rebuilt, it is easy to trunk "all VLANs" everywhere because that always works and never throws an error. The result is that a zone you believed lived only in the server room is now present on the access switch by the reception desk. The zone is one misconfigured port away from the user zone.
  • The bypass. This is a host with a foot in two zones. Examples include a server with a second network card on the management VLAN and the server VLAN. Another is a virtualization host whose management interface shares a switch with guest VMs. Or it might be a dual-homed workstation. Traffic that should have crossed a monitored boundary instead walks straight through the host, and no boundary device ever sees it.
  • The address in the wrong place. A device configured by hand with an address from the wrong subnet, or a DHCP scope that overlaps a zone it should not. The device behaves as if it belongs to a zone it was never meant to join, and inherits that zone's access.

Every one of these is invisible from the rulebase alone, because the rulebase still reads correctly. You have to test the network as it actually behaves, not as its configuration claims to behave.


Two ways to check: active tests and passive evidence


There are exactly two ways to know whether a boundary holds, and you want both, because each catches what the other misses.


Active testing means trying the crossing on purpose. From a host in one zone, attempt to reach a target in another. Record whether the connection succeeds or is refused. Active testing is decisive: a connection either completes or it does not. Active testing can probe a boundary that currently carries no traffic at all, which passive evidence can never do. Its limit is that it only tests what you thought to try; a boundary you forgot to include goes unchecked.


Passive evidence means reading the flow records and firewall logs you already collect (from the inter-zone monitoring guide). You look for crossings that should not exist. Passive evidence catches leaks you never thought to test, because it reports what actually happened rather than what you asked about. Its limit is the mirror image of active testing's: it can only see a leak that someone has already used. A wide-open boundary that nobody has crossed yet leaves no trace in the logs.


Active testing proves a boundary can be crossed; passive evidence proves one has been. Run active tests on a schedule to find latent holes, and watch passive evidence continuously to catch holes that are being used right now. The rest of this guide covers each in turn.


Setting up authorized reachability self-tests


The tidy way to run active tests is a small test host in each zone. Each runs a scripted set of connection attempts to a fixed target list. A test host is nothing special: a spare small computer, a VM, or a Raspberry-Pi-class box. It is given an address in the zone and left to do one job: try to reach places and report what happened. One per zone is the goal. If you cannot manage that, a laptop you physically move from zone to zone works for a periodic check. That comes at the cost of not being able to schedule it.


The test itself is a connection attempt to a specific address and port, with a short timeout, recording one of three outcomes. The first is open: the connection completed. The second is refused: something actively said no, with a reset or an administratively-prohibited message. The third is filtered: nothing answered before the timeout, the typical signature of a silent drop by a firewall. A single reachability check looks like this:


# Try one TCP port, wait at most 2 seconds, say what happened.
# Run from the test host in the USER zone.
target=10.20.0.11      # a file server in the SERVER zone
port=445               # file sharing
timeout 2 bash -c "echo > /dev/tcp/$target/$port" 2>/dev/null \
  && echo "$target:$port OPEN" \
  || echo "$target:$port BLOCKED-or-DOWN"

To turn one check into a matrix, keep a plain target list. Use one line per destination and port you care about, drawn straight from your zone-pair matrix. Loop over it from each zone's test host:


# targets.txt, one "address port label" per line:
#   10.20.0.11 445 file-share
#   10.20.0.11 3389 remote-desktop
#   10.99.0.5  443 switch-admin
#   10.30.0.20 80  camera-web
while read addr port label; do
  timeout 2 bash -c "echo > /dev/tcp/$addr/$port" 2>/dev/null \
    && result=OPEN || result=BLOCKED
  echo "$(date +%F) $(hostname) -> $addr:$port ($label) $result"
done < targets.txt >> reachability-log.txt

Tip: keep the target list in one file per source zone, checked into the same place as your zone-pair matrix. Make the labels match the matrix exactly. When intended and observed use the same words for the same things, comparing them is a glance instead of a translation exercise. Schedule the run from each test host at a quiet hour and have it append to a central log.


Two cautions. Keep the tests to the destinations and ports on your list. A connection attempt is a light touch, but there is no reason to sweep entire ranges when your matrix already names what matters. And treat a BLOCKED result honestly: it means either the boundary blocked you or the target host was down. Point at least one target per zone at something you know is always up (a server's known-open service). That way, a row of all-blocked results tells you the test host lost its network rather than that every boundary suddenly hardened.


Reading the results matrix


Fold the log into a grid: source zone down the side, destination-and-port across the top, each cell OPEN or BLOCKED. Now lay your intended matrix over it. Every cell falls into one of four states, and only two of them are trouble.


IntendedObservedVerdictMeaning
AllowOPENCorrectThe boundary permits what it should
BlockBLOCKEDCorrectThe boundary denies what it should
BlockOPENLeakA crossing you forbid is possible — the important finding
AllowBLOCKEDBroken dependencySomething users need is refused — a real problem, just not a security one

The Block / OPEN cells are the whole reason you ran the test. Each is a boundary that does not hold: a place where your design says "no" and the network says "sure." Rank them by how much trust they cross. Anything OPEN into the server or management zone from users, IoT, or guest goes to the top of the list. The Allow / BLOCKED cells are worth fixing too; they are usually a rule that is too tight or a service that moved. And they tend to generate help desk tickets, so clearing them buys goodwill as well as accuracy.


Flow evidence: crossings that should not exist


Active tests ask "can this be crossed?" Flow evidence answers "has it been?" Point a query at the flow store for any completed conversation whose zone pair is marked Block in your matrix. A completed session on a forbidden pair is proof of a leak that is not merely possible but in use. A completed session means bytes moved in both directions, not just an attempt.


-- Completed conversations on pairs that should be blocked, last 30 days.
-- "blocked_pairs" is your matrix's Block cells as (src_zone, dst_zone) rows.
SELECT f.src_zone, f.dst_zone, f.src_ip, f.dst_ip, f.dst_port,
       COUNT(*) AS sessions, SUM(f.bytes) AS total_bytes,
       MIN(f.start_time) AS first_seen, MAX(f.start_time) AS last_seen
FROM   flows f
JOIN   blocked_pairs b
  ON   f.src_zone = b.src_zone AND f.dst_zone = b.dst_zone
WHERE  f.start_time >= NOW() - INTERVAL '30 days'
  AND  f.bytes > 0
GROUP  BY f.src_zone, f.dst_zone, f.src_ip, f.dst_ip, f.dst_port
ORDER  BY total_bytes DESC;

Anything this returns is a leak with a timestamp and a byte count — better evidence than any test, because it is not hypothetical. There is one honest gap: the bypass leaks from earlier. A conversation that walks through a dual-homed host never becomes a flow at your core, so the flow store cannot see it. That is exactly why you also run active tests, which would notice that the target is reachable when your design says it should not be. The two methods together close each other's blind spots.


Intended versus observed: putting them side by side


The deliverable from a verification round is one comparison: your intended matrix and your observed matrix, cell for cell, with the disagreements highlighted. It need not be fancy: two grids and a column of exceptions is enough. But it should be explicit, because the disagreements are the entire point. They must not hide among the many cells that agree.


For each disagreement, record five things: the source zone, the destination and port, what you intended, what you observed, and a one-line disposition. The disposition is leak to fix, dependency to restore, or exception to document. That short list becomes your work queue. Re-running the round later shows the queue shrinking. That is the most persuasive evidence you can hand a manager that the effort is working.


A worked example: finding a real leak


A verification round runs over the weekend. Monday morning, one cell in the results glares. The test host in the guest zone reported OPEN to 10.20.0.30:443, an internal application server, on a pair your matrix marks Block. Guest to servers is supposed to be impossible.


You start with the flow store to see whether anyone has used it. The query for forbidden pairs returns nothing from the guest zone. Nobody has actually crossed. So this is a latent hole, found by the active test before anyone found it the other way. That is the best time to find one. Now you trace the path. Three checks, in order:


  • The rulebase. You read the guest-to-inside rules on the firewall. They look correct: guest is permitted to the internet and denied everything internal. So the leak is not a rule; the traffic is not passing the firewall at all.
  • The VLANs on the path. You check which VLANs are trunked to the switch serving the guest wireless access point. There it is: when that access point was replaced, the switch port was rebuilt as a trunk carrying all VLANs, including the server VLAN. The guest access point is bridging the server VLAN onto the guest air.
  • Confirm and close. You prune the trunk to carry only the guest VLAN on that port, re-run the guest-zone test, and watch the cell flip to BLOCKED. The leak is closed, and you have a before-and-after pair of results that documents itself.

Notice what made this tractable. The intended matrix told you the crossing was forbidden. The active test found it while it was still only possible. The flow store confirmed it was not yet abused. And the fix was verified by the same test that found it. No single tool would have given you all of that. And had the flow store instead shown guest devices already reaching that server, the finding would have escalated from "fix a trunk" to the response process in the next guides.


The periodic verification checklist


Verification is not a project you finish; it is a habit that keeps drift small. Two triggers should start a round.


After every change that touches a boundary. This includes any firewall rule added, edited, or reordered, or any VLAN trunked to a new switch or port. It also includes any new dual-homed host, or any DHCP scope change near a zone edge. The test to re-run is narrow — just the boundary you touched — and it takes minutes. This is where drift is born, so this is where it is cheapest to catch.


On a calendar, at least quarterly. Run a full round across every zone pair, whether or not you remember changing anything. The changes you do not remember are exactly the ones that leak.


WhenScopeWhat to run
After any firewall rule changeThe pairs the changed rule touchesActive test from the affected source zones; confirm intended matches observed
After a switch or trunk changeEvery zone present on that switchActive test plus a check of which VLANs the new trunk actually carries
After adding a host with two interfacesThe two zones it bridgesConfirm it is not forwarding between them; active test both directions
Quarterly (minimum)The whole matrixFull active test from every zone, plus the flow query for forbidden pairs
ContinuouslyAll Block pairsThe flow-store alert for any completed session on a forbidden pair

Fixing leaks and documenting exceptions


A leak has two possible fixes, and choosing between them is the real work. Either the crossing should not happen, in which case you close it and re-test to confirm. That means removing the rule, pruning the trunk, splitting the dual-homed host, or correcting the address. Or the crossing is genuinely needed, in which case it was never really a leak. In that case, it was an exception that had not been written down.


The discipline that keeps the whole system honest is this. An exception is not the absence of a rule; it is a rule you have chosen deliberately and recorded. For each one, capture the source, destination, port, the owner who needs it, the reason, and a review date. Add it to the matrix as an explicit Allow-with-note so the next verification round expects it and does not flag it again. The difference between a mature network and a leaky one is not the number of exceptions. Both have plenty. The difference is whether the exceptions are written down. A documented exception is a decision; an undocumented one is a hole with a good story.


Tip: give every documented exception a review date and actually revisit it. The vendor whose support access you opened may have finished the project a season ago. An exception nobody re-examines is just a temporary rule wearing a nicer coat — the very thing that started your drift in the first place.


Where to go next


You can now prove, rather than hope, that your boundaries hold. Active tests from a host in each zone find crossings that are merely possible. Flow evidence finds crossings that are actually happening. And the intended-versus-observed comparison turns both into a short, shrinking work queue. Run a narrow check after every boundary change and a full round each quarter, and drift stays small enough to manage.


A verified boundary is the ground you stand on for the next question. When something does cross that should not, would you recognize it as an attack spreading? Signals of Lateral Movement takes that up, and the whole topic lives on the East-West Visibility & Lateral Movement pillar page.


Put this into practice with SysPeep Monitor

Verification depends on knowing what is on each segment and noticing the moment it changes. SysPeep Monitor auto-discovers devices and interfaces and watches the switch ports and firewall interfaces at your zone boundaries. It alerts when a new device appears on a segment or a boundary link changes. That is the kind of change that quietly introduces drift. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep