When the Cluster Can't Talk: A Networking Playbook
The ticket will say something like "the cluster networking is broken." It will arrive with the special urgency of a dev team whose deployment just went sideways. Here is what experience with clusters teaches, and what this playbook is built on: that sentence is almost never literally true. The network — your network, the cables and switches and routes — is usually fine. What has actually broken is one specific link in a chain of four, and each link fails in a recognizably different way. The whole game is finding which link, fast, without anyone rebooting a switch out of superstition.
You do not need to be a Kubernetes expert to run this play. You need the three-layer mental model from the plain-language guide: pods behind services behind an ingress front door, with cluster DNS underneath. You need a dev with cluster access sitting in the same call. That is because some tests run from your side of the fence and some from theirs. That pairing is not a limitation; it is the design. The admin brings the discipline of isolation testing, the dev brings the credentials, and together the pair is faster than any expert working alone.
This guide gives you the four classic failure classes and a test for each. It gives you an outside-in isolation ladder that orders the tests so every step cuts the search space. It gives you a who-runs-what table for working smoothly with the dev team. And it gives you a worked incident from first ticket to fix, so you can see the whole method breathe.
In this guide
- The four failure classes
- Test 1 — is it cluster DNS?
- Test 2 — is it the service wiring?
- Test 3 — is it ingress?
- Test 4 — is it the node's network?
- The outside-in isolation ladder
- Working with the dev team
- A worked incident
The four failure classes
Nearly every "cluster can't talk" incident lands in one of four buckets, and the symptoms point at the bucket before you run a single command. Cluster DNS failures break names but not addresses: things stop finding each other while raw connectivity is perfect. Service wiring failures break the link between a service and its pods. The name resolves, the pods run happily, and connections to the service go nowhere anyway. Ingress failures break the boundary: everything works from inside the cluster, and the outside world gets errors or silence. Node network failures break geography: problems cluster around one machine, or around one traffic pattern — classically, small packets fine and big ones dying.
| Symptom pattern | First suspect | The test |
|---|---|---|
| Names fail, direct IPs work | Cluster DNS | Look up a service name from inside a pod |
| Name resolves, service dead, pods individually fine | Service wiring | Check the service's endpoints list |
| Works from inside the cluster, fails from outside | Ingress | Test the service directly, then through the front door |
| One node's pods misbehave, or big transfers hang | Node network | Node health, node-to-node tests, MTU probe |
Memorize the left column and you will sound psychic in incident calls. The next four sections give each test in detail — what to run, what the outcomes mean, and who fixes what they reveal.
Test 1 — is it cluster DNS?
Start here, for the same reason you start at DNS on the regular network: name resolution sits under everything, fails confusingly, and is tested in seconds. The reasoning is spelled out in why it's always DNS, and every word of it applies indoors. The test needs a vantage point inside the cluster, so a dev runs it from within a pod while you watch. Look up a service name, then contact a known pod IP directly.
# Dev runs, from inside any pod:
$ nslookup webshop
;; connection timed out; no servers could be reached
# Same pod, straight to a pod IP that backs the service:
$ curl -s -o /dev/null -w "%{http_code}\n" http://10.244.2.9:8080/health
200
Read the split: names dead, direct connectivity alive. That combination convicts cluster DNS and fully exonerates the network underneath — packets are flowing beautifully; the phone book is missing. The fix lives with the dev team, and it usually starts with a soothing observation you can offer. The cluster's DNS is itself just pods (the phone book is printed by workers inside the building). So the dev checks whether those DNS pods are running and healthy like anything else. If instead the name resolves and the direct IP works, DNS is innocent; move to Test 2. If the direct pod IP also fails, skip ahead to Test 4, because something under the floorboards is wrong.
Test 2 — is it the service wiring?
Recall the receptionist model: a service is a stable front number, and its endpoints are the live directory of pods currently answering it. The service finds those pods by matching labels. The service's selector says "I front every pod labeled app=webshop," and pods wearing that label are enrolled automatically. The classic failure is a mismatch: a deployment renames or restructures its labels, the selector no longer matches anything, and the directory silently empties. The receptionist sits at her desk, phone ringing, directory blank. Nothing is down. Nothing errors in advance. Connections just die at a healthy-looking address.
# Dev runs: $ kubectl get endpoints webshop NAME ENDPOINTS webshop <none> # The pods are fine — they're just not enrolled: $ kubectl get pods -l app=webshop No resources found
An empty endpoints list against running pods is the single most common self-inflicted cluster networking wound. Diagnosing it takes ten seconds once someone thinks to look. The subtler sibling is a port mismatch: endpoints exist. But the service forwards to a port the container stopped listening on after a refactor. It is the same silent symptom, found by comparing the service's target port against what the application actually opens. Both fixes are one-line changes on the dev side. Your contribution was asking the receptionist question instead of tracing cables, and it will be remembered kindly.
Test 3 — is it ingress?
When insiders are happy and outsiders are not, the boundary is the suspect. The test is a comparison: reach the service directly from inside (dev), then through the front door from outside (you). The inside leg you have already seen. Your outside leg starts with the basics you would run against any web service — resolve the public name, check where it points, connect, request:
# You run, from outside the cluster: $ nslookup shop.example.com Address: 203.0.113.40 $ curl -sv https://shop.example.com/ -o /dev/null * Trying 203.0.113.40:443... * connect to 203.0.113.40 port 443 failed: Connection timed out
From here the boundary failures split into a short list you can walk in order. The public DNS record may point at a stale address. That is yours to fix, and worth checking first because it is embarrassing in direct proportion to how long it goes unnoticed. The ingress controller — the actual doorman software — may be down or unreachable, which the dev checks like any other workload. The TLS certificate may have expired, announcing itself with browser warnings rather than timeouts. Or the ingress rules may simply not include the hostname being asked for. Controllers route by the name in the request, and a rule list that never heard of shop.example.com returns a default nothing. That last one has a delightful test trick. Aim at the ingress IP while supplying the hostname explicitly. Then you can distinguish "wrong DNS" from "wrong rules" in one command: curl -H "Host: shop.example.com" http://203.0.113.40/. If that works while the normal URL fails, DNS is lying; if both fail identically, the rules or the controller are the problem.
Test 4 — is it the node's network?
The final class is your home turf wearing a thin disguise. Symptoms may cluster around one node: every misbehaving pod happens to live there, or things fail only when traffic crosses between two specific machines. In that case, investigate the node as the sick server it is. Check reachability, NIC errors, link speed surprises, and a switch port with climbing discards. The one genuinely cluster-flavored twist deserves its own paragraph, because it is a classic that bites almost every overlay cluster eventually: MTU mismatch.
Overlay networking wraps pod packets inside node-to-node packets, and the envelope costs bytes — commonly around fifty. The cluster therefore expects pods to send slightly smaller packets so the wrapped result still fits through a standard 1500-byte Ethernet path. That arithmetic can go wrong when a pod interface is configured to assume more room than the tunnel leaves. It can also go wrong when a network path quietly drops the "packet too big" messages that would let endpoints adapt. Then you get the signature every network admin should tattoo somewhere: small traffic works, large traffic hangs. Pings succeed, DNS resolves, health checks pass, and file uploads or big query results stall at the same spot every time. The probe is one command with a large payload and fragmentation forbidden:
# From one node toward another, forcing full-size packets: $ ping -M do -s 1472 10.0.40.9 PING 10.0.40.9: 1472 data bytes ping: local error: message too long, mtu=1450
That reply is the confession: the effective path carries less than standard Ethernet assumes, and everything sized for 1500 is dying in the tunnel. The repair is a coordinated setting: the CNI's MTU configuration must match the room the overlay actually leaves. The dev team applies that setting while you verify the physical path honestly carries what it claims. When this one surfaces, it is usually days after "random big-transfer weirdness" started. Being the person who names it in one ping is a reputation event.
The outside-in isolation ladder
The four tests earn their keep when you run them in an order that always shrinks the search space. Work from where the user stands toward the middle of the cluster, and record what works as diligently as what fails. The diagnosis is the boundary between the two:
- Rung 1 — from the user's side: resolve the public name, connect, make a real request through the front door. If all of this works, the "outage" may be one user's machine — a familiar plot twist.
- Rung 2 — from your network, outside the cluster: the same tests from your monitoring host, plus the API server pulse check. Distinguishes "the world can't reach us" from "the cluster is sick."
- Rung 3 — the nodes: all nodes up, NICs clean, node-to-node traffic healthy, MTU probe if large transfers are implicated. Your territory, your tools.
- Rung 4 — inside the cluster (dev at the keyboard): service name lookup (Test 1), endpoints (Test 2), direct-versus-ingress comparison (Test 3).
- Rung 5 — read the boundary: the highest rung that works and the lowest that fails bracket the culprit. Say the bracket out loud; the room usually names the fix.
The order is doing real work, so resist the urge to jump around. Outside-in starts with the tests you can run alone, without credentials, before the dev even joins the call. So the pairing session begins with half the map already colored in. This order also front-loads the checks that establish user impact, which is the first thing anyone senior will ask about. And this order means every "works" you record is load-bearing. A clean rung 2 spares the whole team from re-litigating the firewall at minute forty, because you can point at the timestamped output and keep climbing.
Tip: paste this ladder into your runbook wiki today, with the actual hostnames, node addresses, and commands for your cluster filled in. During an incident, nobody improvises well. The difference between "the playbook exists" and "the playbook exists with our names in it" is about twenty minutes of fumbling. You will not enjoy that at 3 AM.
Working with the dev team
Every rung above assumed two people, so make the pairing explicit before the bad day. The split is natural. You own the vantage points outside the cluster and underneath it, and they own the ones inside. Each side shares raw output rather than conclusions. "The network is broken" is a conclusion. The command nslookup webshop timing out from inside a pod while node-to-node pings run clean is evidence. Evidence is what lets two specialists converge instead of collide. The same courtesy runs in reverse: when your tests exonerate a layer, show the output, not just the verdict. The wider craft of staying allies mid-incident is the subject of working with app teams. That includes blameless language, shared dashboards, and agreeing on what would change your mind. This craft pays double inside clusters, where neither side can see the whole board alone.
| Who | Runs | Answers |
|---|---|---|
| You | Public DNS lookups, external curl, certificate checks | Can the world reach the front door? |
| You | Node reachability, NIC counters, switch port stats, MTU probes | Is the floor solid under the cluster? |
| You | API server pulse check from outside | Is the cluster's brain reachable? |
| Dev | In-pod name lookups and curls to services and pod IPs | Do things resolve and connect from inside? |
| Dev | Endpoints, selectors, ingress rules, controller and DNS pod health | Is the cluster's own wiring intact? |
| Both | The ladder, in order, in one shared call | Where exactly is the boundary between works and fails? |
One more translation habit accelerates everything: listen to the exact error, because the error usually names its own layer. Devs tend to report all failures as "can't reach it," but the network does not fail generically. Thirty seconds spent extracting the precise message often replaces an entire rung of testing:
| The exact error | What it already told you |
|---|---|
| "Could not resolve host" | DNS, full stop — go straight to Test 1 |
| "Connection refused" | Something answered and said no: empty endpoints, wrong port — the path itself is fine |
| "Connection timed out" | Nothing answered at all: path, node, or a filter in between |
| Gateway error from the front door | Ingress is up and reachable; the leg behind it is unhappy |
| Certificate warning | The boundary's TLS, not connectivity — check expiry and names |
| Hangs only on large transfers | The MTU signature — go straight to Test 4 |
A worked incident
Tuesday, 09:40. The dev lead pings you: "Orders are failing — the frontend can't reach the payments service. Pretty sure the network is broken between the nodes." Users are affected; a call starts. Here is the play, annotated.
09:42, rungs 1 and 2 — you, from outside. The public site resolves, loads, and browses normally; only checkout fails. The API server answers its health check. Every node is green in your monitoring, NIC error counters flat, inter-node traffic normal against yesterday's baseline. You say the exact sentence this playbook exists for: "The floor looks solid — nodes, links, and front door all pass. Let's climb inside and split it. Can you test the payments service by name from a frontend pod?"
09:46, rung 4, Test 1 — dev, inside. nslookup payments resolves instantly to a service IP. Cluster DNS exonerated. A curl to the service name gets connection refused, immediately. And "refused" is itself a clue, the crisp no of something that answered, not the silence of a broken path. You ask for Test 2.
09:49, Test 2 — dev. kubectl get endpoints payments returns <none>. The receptionist's directory is empty. The payments pods themselves are running — kubectl get pods shows all three healthy. Someone checks recent changes: this morning's deployment tidied up labels, renaming app: payments-svc to app: payments. The service's selector still hunts for the old label. Directory empty, phones ringing, nobody enrolled.
09:55 — fix and confirm. The dev updates the selector, and endpoints repopulate in seconds. Checkout works from inside, then from outside, then in your synthetic check history. That history also quietly documents the outage's exact start and end for the postmortem. Fifteen minutes ticket-to-fix. No switch was rebooted, and no cable rescinded. And the phrase "the network between the nodes" was retired from the incident by evidence, not by argument — which is the only retirement that sticks.
Where to go next
The pattern to keep: four failure classes, one test each, run outside-in with the dev team on the line, boundary read aloud, evidence shared. Run it twice and it becomes reflex; write it down and it becomes your team's reflex. The background that makes the playbook feel obvious lives in this pillar's other guides. There is the mental model in Kubernetes Networking in Plain Language. There is the everyday watchlist in Monitoring a Kubernetes Cluster Without Becoming a K8s Expert. And there are the visibility boundaries in Pod-to-Pod Traffic. The whole series lives at Container & Kubernetes Networking for Generalists.
Put this into practice with SysPeep Monitor
Half of this playbook's speed comes from evidence you gathered before the incident: node health, NIC counters, front-door checks, and response-time history. SysPeep Monitor keeps exactly that record. It discovers your hosts and watches nodes and interfaces over SNMP, WMI, and SSH. It checks the addresses users depend on, and alerts you the moment one goes quiet. Rung one and two of the ladder, already run. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep