Pod-to-Pod Traffic: East-West Visibility Inside the Cluster
Network people sort traffic by compass. North-south traffic crosses the boundary of a system — users coming in, responses going out. It passes your firewalls, your load balancers, your monitoring taps. East-west traffic moves laterally, between the parts inside. On a traditional network the east-west share was modest. Clients talked to servers, servers talked to a database, and most conversations crossed a switch you could see. A Kubernetes cluster inverts the ratio. Applications are now built as swarms of small services calling each other constantly. So for every request that enters from the north, a burst of internal chatter fans out east and west. Nearly all of it happens somewhere your switches cannot follow.
That might sound like an accusation waiting to happen — "the network is slow" about traffic you cannot even see. If so, you have correctly identified the problem this guide exists to right-size. The honest news comes in two halves. You have genuinely lost visibility you used to have, and it is worth understanding exactly what and why. But most of what you lost, a generalist admin running a small or mid-size shop does not operationally need. The parts you do need are still available from vantage points you already control.
By the end of this guide you will know, in plain words, how overlay networking hides pod conversations. You will know why this is the hypervisor visibility problem squared and what your existing tools still show you (more than you might fear). You will know when service-level checks are honestly sufficient and when deeper visibility earns its very real complexity. You will also know how network policies and the platform's own tooling fit into the picture.
In this guide
- Overlay networks in plain words
- The vSwitch problem, squared
- What you can still see
- A five-minute tour of the blind spot
- When service-level checks are enough
- When deep visibility earns its complexity
- Network policies: the cluster's internal firewall
- Asking the platform for its view
Overlay networks in plain words
Pods on the same node talk through a bridge inside that node, which your physical network never sees at all. That is familiar territory if you have read about container basics. The interesting case is pods on different nodes, and many clusters handle it with an overlay network. A pod on node 1 sends a packet to a pod on node 3. The sending node wraps that pod packet inside a second, node-to-node packet and ships it across your ordinary network. Then node 3 unwraps it and delivers the original. An envelope inside an envelope. The pod-level conversation — who is talking to whom, on which ports, about what — rides inside. Your network only carries and only sees the outer wrapper.
Think of interoffice mail between two buildings. Memos go into a sealed courier pouch. The courier company sees pouches moving between building addresses. So its manifest reads "one pouch, building A to building B, again" all day long. It cannot see the memos, the senders, or the subjects — and your switches are the courier here. On the wire, an overlay shows up as a steady stream of node-IP-to-node-IP traffic on a single UDP port, tunnel-style. It is not necessarily encrypted — some setups do encrypt the tunnels, many do not. But it is opaque to your gear either way, because switches, flow collectors, and firewall logs classify by the outer header. The outer header says only "nodes, chatting."
One honest complication before it surprises you: not every cluster tunnels. The wiring is done by the cluster's CNI plugin (its pluggable network layer). Some plugins skip the envelopes, instead routing pod subnets across your network like any other subnet. In those clusters your gear does see pod IPs on the wire. That sounds like a win until you remember those addresses churn hourly and appear in none of your records. So a capture full of pod IPs is only marginally more legible than a capture full of pouches. The one-line question to ask your dev team is "does our CNI use an overlay, or does it route pod subnets?" It tells you which flavor of limited visibility you own. It belongs in your network documentation next to the VLAN table.
The vSwitch problem, squared
If this is giving you déjà vu, it should. Virtualization did the same thing to you years ago. Two VMs on the same hypervisor host talk through a virtual switch in that host's memory. The conversation never touches a physical port you can mirror or count. That blind spot, and the honest workarounds for it, are covered in the guide on VM-to-VM visibility. If you have internalized that lesson, you already own half of this one.
Kubernetes squares it for three compounding reasons. First, both directions of hiding happen at once. Same-node pod traffic is invisible exactly like same-host VM traffic. And cross-node traffic — which a hypervisor shop would at least see plainly on the wire — is wrapped in tunnels. Second, churn: a VM's IP was stable enough that yesterday's capture still meant something today. A pod IP ages out of relevance in hours, so even perfect captures decay into trivia while you analyze them. Third, sheer conversational volume: a microservice application generates internal calls out of all proportion to its user traffic. So the hidden share is not a corner of your traffic — on cluster-heavy networks it can be most of it. Same disease as the vSwitch era, higher dose, faster mutation.
What you can still see
Now the reassuring inventory, because your instruments did not go dark — they just report at a coarser granularity than before. Everything below works today, with tools you already run, no cluster credentials required:
| Vantage point | What it shows you | What it cannot tell you |
|---|---|---|
| Node NIC counters | Total traffic per node, errors, discards, saturation | Which pods or services generated it |
| Flow data at node granularity | Which nodes talk to which, volumes, trends over time | The pod conversations inside the tunnel |
| Switch port stats on node uplinks | Physical-layer health, congestion, microbursts filling buffers | Anything about the traffic's meaning |
| Firewall and edge logs | Everything entering or leaving the cluster (north-south), fully attributed | Anything that stays inside |
| Packet capture on a node itself | Pod traffic before wrapping — the actual conversations | Only that node's slice; needs host access |
Three of those rows quietly matter a great deal. Node-level flow data still answers real questions. Examples are a node pair suddenly exchanging triple its usual volume and a slow ramp that predicts a saturated uplink. Another is a node talking to the internet when it never used to. North-south visibility at your edge remains completely intact. Everything crossing the cluster boundary is as attributable as it ever was. Security questions about what the cluster talks to externally are answered there, not inside. And the last row is your secret staircase: on the node itself, before the envelope goes on, the pod traffic exists in plain, capturable form. A tcpdump on the right node interface sees actual pod conversations. That is why "capture on the node, not the switch" is the single most useful packet-level trick in cluster land. It is one the troubleshooting playbook in this series uses.
A five-minute tour of the blind spot
Abstract claims about visibility are worth checking with your own eyes, and this tour costs five minutes with the dev team's blessing. Start where your instincts start: mirror or capture the switch port feeding a node, or simply watch the node's physical interface. On an overlay cluster, what you get looks like this — an endless, uniform procession of node-to-node UDP on one port:
# On the node's physical NIC: only envelopes $ tcpdump -n -i eth0 udp port 4789 14:02:11.48 IP 10.0.40.7.43217 > 10.0.40.9.4789: UDP, length 1406 14:02:11.48 IP 10.0.40.9.51330 > 10.0.40.7.4789: UDP, length 322 14:02:11.49 IP 10.0.40.7.43217 > 10.0.40.9.4789: UDP, length 1406
Two node addresses, one tunnel port (4789 is a common one), lengths and timestamps, nothing else. Every application conversation on the cluster is in there somewhere, indistinguishable pouch after indistinguishable pouch. This is the entire east-west story as your network gear tells it. Now climb the secret staircase: on the node itself, capture on the bridge interface where pods attach, before the envelopes go on:
# On the same node's bridge: the actual conversations $ tcpdump -n -i cni0 host 10.244.2.9 14:02:11.48 IP 10.244.1.17.51844 > 10.244.2.9.8080: Flags [P.], length 1354 14:02:11.48 IP 10.244.2.9.8080 > 10.244.1.17.51844: Flags [P.], length 270
Same instant, same traffic — and suddenly it is a legible request and response between two pod addresses on a real application port. One caveat keeps this honest: those pod addresses mean something only while the pods live. So the last step is asking the platform (or a dev) to translate. A quick "which pods are these?" against the current pod list turns 10.244.1.17 into "the webshop frontend" while the information is still warm. Wire vantage, node vantage, platform directory: the three rungs of east-west visibility, and now you have personally stood on each.
When service-level checks are enough
Here is the honestly scoped question: what do you operationally need to see? For a small team running a handful of applications, the answer most days is: whether the services work. And that is answered not by east-west traffic analysis but by the layer above it. That layer includes synthetic checks against service front doors, node health, and the dev team's application metrics. If the front doors answer promptly, the nodes are healthy, and the NICs are clean, then the east-west chatter is doing its job by definition. In that case, inspecting it would tell you nothing actionable. The cluster health guide builds exactly that outside-in setup. For most small and mid-size shops that setup is genuinely sufficient. It is not a compromise while you save up for better, but the appropriate amount of visibility for the questions you actually have.
It helps to notice how the platform changed the economics. On a traditional network, deep traffic visibility was how you found the failing conversation so you could fix it. Inside a cluster, the platform already restarts failing pods, reroutes around dead nodes, and updates service directories in seconds. Many failures repair themselves before any human could have acted on a flow record. Visibility that exists to trigger a human response is worth less when the response is automated. What retains full value is knowing the outcomes: services up, latency normal, nodes healthy. Spend there first.
Tip: before deploying anything new for east-west visibility, write down the specific question it must answer. Examples are "which service pair causes the nightly latency spike," and "prove billing pods cannot reach the test database." If nobody can name the question, the tool will be racks of recorded silence within a quarter. "All of it, just in case" is a storage strategy, not a question.
When deep visibility earns its complexity
Sometimes, though, the question is real, and service-level checks cannot answer it. Honest triggers that justify deeper east-west visibility:
- Compliance and audit: a framework demands you demonstrate which workloads can talk to which, with evidence. "Probably fine" does not survive an auditor.
- Chronic mystery latency: users feel slowness, every dashboard is green, and the cause hides somewhere in a chain of internal calls. You need per-conversation timing to find the guilty hop.
- Security investigation: a compromised workload raises the lateral-movement question — what else did it reach internally? — and only recorded east-west flows can answer it after the fact.
- Multi-tenant clusters: separate teams or customers share one cluster, and isolation between them must be verified rather than assumed.
When a trigger is real, the tooling lives mostly inside the platform rather than on your switches, and it comes in three generic families. Some CNI plugins can export pod-level flow logs — the courier finally itemizing the memos. A service mesh puts a small proxy beside every pod, buying complete per-conversation metrics at the price of running one more distributed system. And kernel-level observers watch connections from inside each node's operating system with modest overhead. Every family costs real resources and real expertise, which is precisely why the trigger list matters. Adopt when a named problem demands it, and size the deployment to that problem. Let the dev team drive the in-cluster parts while you consume the output. Completeness is not a trigger.
Network policies: the cluster's internal firewall
One gentle detour from seeing traffic to controlling it, because you are the natural person to ask the question. By default, any pod can reach any pod, cluster-wide — the internal network launches as flat and open as a hub in a closet. Network policies are the cluster's internal firewall rules: allow-lists, attached to groups of pods by label, saying who may talk to whom. They are enforced by the CNI plugin. That detail has one sharp edge: a few plugins quietly do not enforce them at all. So a policy can exist and do nothing — worth one explicit question to the dev team.
$ kubectl get networkpolicies --all-namespaces No resources found
That output — and it is a common sight — means the flat, open default is fully in effect. Every workload can reach every other workload, including whatever test oddities share the cluster. You do not need to learn to write policies to add value here. Simply asking "do we have network policies, and does our CNI enforce them?" is a legitimate first contribution. Help the dev team think in zones: what should reach the database pods? What should the payment services accept? That is segmentation thinking you have done for years, translated from VLANs and firewall rules into labels. Verifying the rules exist is a start; the platform can verify what they block.
If the conversation lands well, the pragmatic opening move mirrors how you would segment any flat network: protect the crown jewels first. In that case, use one policy restricting who may reach the database pods and one around anything handling payments or personal data. Set a default that new namespaces start closed rather than open. That is three rules, enormous risk reduction, and no grand redesign. Every rule doubles as documentation of what the traffic is supposed to look like. That quietly makes any future visibility work easier to interpret.
Asking the platform for its view
Finally, remember that the best observer of the cluster's insides is the cluster. The platform knows, authoritatively and in real time, which pods exist, where each one runs, and which pod addresses currently back each service. That is a directory that turns an anonymous flow record into a name, if you ask within the hour. The built-in command-line tooling exposes all of it. The dev team's metrics stack graphs much of it, and the CNI usually carries its own diagnostics. When you need the inside view, the efficient move is rarely to reconstruct it from the wire. It is to ask the platform, or to ask the person who speaks its language fluently.
In practice this works as a two-vantage partnership. Your flow data says node 2 and node 5 started exchanging five times their usual traffic at 14:00; the dev team's tooling says a batch job scaled out across those nodes at 13:58; together that is a complete, boring explanation reached in minutes. Neither side could have written that sentence alone. The east-west blind spot stops being threatening the moment you stop treating it as territory you lost and start treating it as territory with a different, cooperative surveillance arrangement.
Where to go next
Here is the scorecard, honestly filled in. Your switches see wrappers, not conversations. Your node counters, flow data, and edge logs still answer the questions that are operationally yours. Service-level checks cover the daily need for most small shops. The deep tools are real but earn their keep only against named problems. When an east-west mystery does strike, When the Cluster Can't Talk gives you the test sequence. And Kubernetes Networking in Plain Language supplies the pods-services-ingress model this guide leaned on. The rest of the series lives at Container & Kubernetes Networking for Generalists.
Put this into practice with SysPeep Monitor
The vantage points this guide called yours are the ones SysPeep Monitor is built for. They include node NICs and switch uplinks watched over SNMP, plus node health over WMI or SSH. They include thresholds that flag a node pair trending toward saturation and dashboards that keep the east-west volume story on one screen. SysPeep Monitor will not read the memos inside the tunnels — nothing on the wire can — but it keeps honest watch on the couriers. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep