Kubernetes Network Troubleshooting: A Layer-by-Layer Decision Tree and Five Real Incidents
It Started With a 3 A.M. Page
The page said payment-service was failing health checks. I had the usual suspects in my head: database down, OOM-killed pod, bad deploy. Then kubectl get pods showed every replica Ready, zero restarts, healthy logs. The pod was alive and the process was fine. The problem was the road between the pod and the thing it needed to talk to. That is the moment Kubernetes networking stops being theory and becomes a 2 a.m. debugging session.
I have worked enough of these to know the failure is rarely in one magical place. It is usually a chain: DNS, then Service, then the proxy rules, then the overlay, then policy, then the cloud firewall. This article walks the chain the way I actually do it, one layer at a time, with the five incidents that burned me the hardest.
The Decision Tree: Six Layers, One Direction
When a pod cannot reach a backend, I never guess. I start at the top and peel down. Each layer has one cheap command that tells me whether to keep going or stop and fix.
| Layer | What it answers | One command |
|---|---|---|
| DNS | Does the name resolve to the right IP? | nslookup inside the pod |
| Service/Endpoints | Are there healthy backends behind the ClusterIP? | kubectl get endpoints |
| kube-proxy | Are the NAT rules programmed on the node? | iptables -t nat -L |
| CNI | Does the overlay tunnel carry cross-node traffic? | ip route, ss -lunp |
| NetworkPolicy | Is policy silently dropping packets? | kubectl describe networkpolicy |
| Node/cloud firewall | Is the host or security group blocking the port? | firewall-cmd, cloud console |
The order matters: DNS is the cheapest to test, the cloud firewall is the most expensive to touch. Every incident below follows this exact order, and each could have been found in under five minutes if I had trusted the tree instead of my instincts.
Layer 1 — DNS: The Name That Resolves to the Wrong Place
Incident #1: CoreDNS resolving a stale IP. A worker pod kept trying to reach redis.internal.svc.cluster.local and getting connection refused. DNS resolved fine — to an IP that was no longer the Redis service.
The address was plausible, so I wasted twenty minutes on the Service. The real bug was upstream: a hand-edited /etc/hosts entry in a CoreDNS ConfigMap had pinned that name to an old ClusterIP. The fix was deleting the stale hosts block, not touching any network path.
kubectl get configmap -n kube-system coredns -o yaml
kubectl get configmap -n kube-system coredns -o yaml | grep -A3 hosts
The signal is a name that resolves to an IP nothing listens on. The command runs inside the pod's own network namespace, so it tests exactly the path the application uses.
Layer 2 — Service and Endpoints: The Selector Mismatch
Incident #2: an empty Endpoints object. This is my number one cause of "service is down" pages. The Service exists, the pods look fine, but the traffic goes nowhere.
NAME TYPE CLUSTER-IP PORT(S) AGE
backend ClusterIP 10.96.7.88 8080/TCP 21d
NAME ENDPOINTS AGE
backend <none> 21d
<none> is the smoking gun. The Service and the pods have a mismatch. I compare the selector with the pod labels:
The selector was app: backend-api, but every pod carried app: backend. One typo, no endpoints, total outage. Fixing the selector brought the endpoints back instantly. This layer is also where a pod that is Ready but on the wrong port shows up — check targetPort if the endpoint count looks right but connections still fail.
Layer 3 — kube-proxy: The Rules in the Middle
When endpoints exist but traffic still fails from another node, I look at whether the load-balancing rules exist on the node. kube-proxy programs iptables (or IPVS) on every node to redirect ClusterIP traffic to the pod IPs.
kubectl logs -n kube-system daemonset/kube-proxy --tail=20
iptables -t nat -L KUBE-SERVICES -n | grep backend
If the rule is missing, kube-proxy is not syncing. Check its logs and confirm the kube-proxy pods are not stuck in CrashLoopBackOff. A less obvious variant: the rule exists under IPVS but the real-servers list is empty. The layer is healthy when the rule maps ClusterIP to the right backend list.
Layer 4 — CNI: Flannel, Calico, Cilium
Same-node traffic working while cross-node traffic hangs is the CNI signature. The overlay tunnel is the culprit. This is where the plugin choice determines what you check.
| CNI | Data plane | Check |
|---|---|---|
| flannel | VXLAN over UDP 8472 | ss -lunp \| grep 8472 |
| calico | BGP over TCP 179 or IPIP proto 4 | calicoctl node status, ip route |
| cilium | eBPF datapath | cilium status, hubble status |
Incident #3: VXLAN port 8472 blocked. Cross-node traffic was black-holing on a flannel cluster. The tunnel was up, both nodes had a flannel.1 interface, but packets never crossed.
The port was listening, so I checked the host firewall and found UDP 8472 denied in the security group on the target node. VXLAN needs 8472 open between every node pair. The fix was opening the port in the security group, not touching Kubernetes at all.
For calico, the same failure mode appears when BGP neighbor sessions are down:
For cilium, the eBPF datapath makes drops visible. cilium status shows datapath health, and hubble observe --verdict DROPPED frequently names the policy that did it before I ever open a YAML file.
Layer 5 — NetworkPolicy: The Silent Packet Killer
Incident #4: default-deny blocking DNS. A freshly deployed namespace came up and every pod could not resolve anything. DNS itself was fine from the host. Inside the namespace, everything timed out.
The policy had a podSelector: {} matching everything and an empty ingress list. That is the default-deny pattern, and it silently killed UDP 53 toward CoreDNS. The application never saw an error; packets just vanished. The tell is that DNS works from the host but not from the pod, and the policy description shows no allowed ingress.
The fix is an explicit allow rule to the DNS service. NetworkPolicy is the layer to suspect whenever a service starts failing right after a new namespace is created or a policy is applied.
Layer 6 — Node and Cloud Firewalls
The final layer is the one everyone resists checking because it is outside the cluster. On bare metal, the node firewall:
On a cloud, the security group is the same thing with a different console. I lost a morning to a security group that allowed 443 inbound but not the node-to-node pod CIDR. The symptom: everything inside the cluster is healthy, but external traffic to a NodePort or LoadBalancer times out.
Pod-Level Capture: netshoot and tcpdump
When every layer looks clean but the connection still fails, I stop theorizing and capture the packets. Modern Kubernetes lets me do this without a debug pod image of my own.
Inside the ephemeral container I run tcpdump on the pod's veth. The question is always the same: do packets leave, and do replies come back?
If I see SYN packets leaving but no SYN-ACK, the backend is unreachable — that points to the overlay or a firewall. If I see nothing, the application is not even sending, which is a code problem. This capture has saved me from two days of blaming the wrong layer.
Incident #5 — The Ingress Object With No Controller
One last story that does not fit neatly into the tree. A user swore the Ingress was broken. The object existed and the hostname looked right, but no address was assigned.
The <pending> address was the clue. The Ingress controller namespace had no Deployment running at all — someone had scaled it to zero and forgotten. The Ingress resource is just declarative state; without a controller watching it, it does nothing. Once I scaled the controller back up, the address was assigned within seconds.
FAQ
What is the single most common cause of Kubernetes service failures I should check first?
An empty Endpoints object. Run kubectl get endpoints <svc> first. If the list is empty, the pod selector is wrong, the pod is not Ready, or the port name does not match targetPort.
When should I suspect the CNI plugin instead of kube-proxy?
When same-node pod-to-pod traffic works but cross-node hangs. Cross-node reachability is the CNI overlay tunnel. Check the tunnel port: UDP 8472 for flannel VXLAN, BGP 179 for calico.
How do I tell a NetworkPolicy from a firewall drop?
Run kubectl describe networkpolicy and inspect the podSelector and allowed ingress rules. NetworkPolicy is namespace-scoped and enforced by the CNI. A security group drop shows as timeouts with no reply packets in tcpdump.
Can an Ingress object exist while the controller is not running?
Yes. The Ingress resource is just declarative state. If no controller watches it, it gets no address and no endpoints. Check kubectl get deploy -n ingress-nginx before blaming the Service.
What is the fastest way to capture packets from inside a pod?
Run kubectl debug with nicolaka/netshoot, then tcpdump inside that container. This captures traffic on the pod's veth and tells you whether packets leave and replies return.