How kube-proxy Works (Part 2): Packet Flows, Proxy Modes, and Debugging
In Part 1, we traced the control chain behind a Kubernetes Service: the Service object declares an entry point, the EndpointSlice controller tracks ready backends, kube-proxy syncs forwarding rules onto each node, and the Linux kernel delivers the packets.
NOTE
Further reading: How kube-proxy Works (Part 1): A Rule Synchronizer, Not a Proxy Server
This part moves into the data plane. Using a concrete IP topology, we will trace how packet addresses change across DNAT (Destination Network Address Translation), SNAT (Source Network Address Translation), and conntrack (connection tracking); examine where iptables, IPVS, and nftables stand in 2026; and walk through a four-stage debugging toolkit along with common failure scenarios.
Proxy Modes in 2026: From IPVS Deprecation to nftables
As of Kubernetes v1.37, the official kube-proxy implementation on Linux nodes supports three modes: iptables, IPVS, and nftables. (Windows nodes use a separate kernelspace mode; this article focuses on Linux networking.)
+---------------------------------------------------------+
| kube-proxy Modes |
+---------------------------------------------------------+
| iptables : Default, battle-tested, linear rule cost |
| IPVS : Deprecated since v1.35, slated for removal |
| nftables : Modern alternative, set-based, fast updates |
+---------------------------------------------------------+
Here is how the three modes compare in underlying mechanism and lifecycle status:
| Mode | Kernel Target | Processing Layer | Status (v1.37) | Characteristics and Limitations |
|---|---|---|---|---|
| iptables | netfilter rule chains | Linux kernel | Current default | Battle-tested and stable; long chains at massive scale make full rule refreshes expensive |
| IPVS | Virtual / Real Servers | Linux kernel | Deprecated | Cannot natively express full Service semantics; still relies on auxiliary iptables rules |
| nftables | Tables, Sets, and Maps | Linux kernel | Recommended modern path | Requires kernel 5.13+; uses set lookups and incremental updates whose cost scales with the delta |
Why Is nftables Replacing IPVS?
The community originally introduced IPVS to bypass the linear rule-traversal bottleneck of iptables when clusters scaled to tens of thousands of rules. As Kubernetes Service semantics grew richer—adding externalTrafficPolicy: Local, source IP range filtering, topology-aware routing, and more—the IPVS API could no longer express every condition natively, forcing kube-proxy to fall back on iptables chains alongside IPVS.
nftables natively supports Set and Map data structures (see KEP-3866), performing efficient lookups within a single rule and supporting atomic incremental updates: kube-proxy only pushes the endpoints that changed in a given sync cycle instead of rewriting the entire ruleset, and atomic replacement prevents partially applied rules from ever taking effect.
NOTE
Under the official IPVS deprecation plan, IPVS mode was deprecated in Kubernetes v1.35, is slated to be disabled by default in v1.40, and is scheduled for removal in v1.43. New clusters should no longer adopt IPVS based on older guides; stay on iptables or evaluate migrating to nftables.
NOTE
Further reading: Why Do Service Requests Slow Down in Large Clusters? iptables vs. IPVS in the Kernel
Tracing Core Packet Flows: How Addresses Change Across Scenarios
Reference Topology
To make each address translation step concrete, all packet traces below share the following network topology:
| Role | IP / Port | Notes |
|---|---|---|
| External Client | 198.51.100.30 | Test client outside the cluster |
| Node A | 192.0.2.10 | Host IP of Node A |
| Node B | 192.0.2.11 | Host IP of Node B |
| Client Pod | 10.244.1.10 | The originating Pod, running on Node A |
| API Pod A | 10.244.1.20:8080 | Backend Pod running on Node A |
| API Pod B | 10.244.2.20:8080 | Backend Pod running on Node B |
| Service ClusterIP | 10.96.0.50:80 | Virtual IP for the api Service |
| Service NodePort | 30080 | NodePort for the api Service |
Here is the corresponding Service manifest:
apiVersion: v1
kind: Service
metadata:
name: api
spec:
type: NodePort
selector:
app: api
ports:
- port: 80
targetPort: 8080
nodePort: 30080
Path 1: ClusterIP to a Cross-Node Pod
When the Client Pod initiates a connection to the Service, the packet enters the network stack on its local node, Node A:
- Client Pod: 10.244.1.10
- dst: 10.96.0.50:80Node A Netfilter / Data Plane1. Match ClusterIP 10.96.0.50:80; 2. Select backend Pod B (10.244.2.20:8080); 3. Execute DNAT (dst -> 10.244.2.20:8080); 4. Record connection tuple into conntrack table
- src: 10.244.1.10, dst: 10.244.2.20:8080 (CNI Routing / Overlay)Node B / API Pod B: 10.244.2.20:8080
- Outbound DNAT: Node A’s kernel network stack matches the ClusterIP rule, selects API Pod B as the backend, and rewrites the destination to
10.244.2.20:8080. The source IP stays10.244.1.10(no SNAT). - CNI forwarding: The underlying CNI plugin routes the packet across nodes—via an overlay network or direct routing—to Pod B on Node B.
- Return reverse translation: Pod B sends its reply back to
10.244.1.10. When the packet arrives at Node A, Linux conntrack matches the tracked connection and automatically translates the source address back from10.244.2.20:8080to10.96.0.50:80. From the Client Pod’s perspective, it has only ever communicated with the Service IP.
Path 2: Default NodePort Routing (Cluster Policy)
When an external client hits NodePort 30080 on Node A, and the data plane picks Pod B on Node B:
- External Client: 198.51.100.30
- dst: 192.0.2.10:30080Node A Netfilter / Data Plane1. DNAT: dst 192.0.2.10:30080 -> 10.244.2.20:8080; 2. SNAT (MASQUERADE): src -> 192.0.2.10
- src: 192.0.2.10, dst: 10.244.2.20:8080Node B / API Pod B
Under externalTrafficPolicy: Cluster (the default), as detailed in the official Using Source IP guide, Node A must perform SNAT—replacing the source IP with its own node IP—so that reply packets from Node B return through Node A and can have their pre-NAT addresses restored before heading back to the client.
- Pros: External traffic is distributed evenly across all ready Pods in the cluster.
- Trade-offs: Backend Pod B cannot see the real client IP (all requests appear to come from Node A), and cross-node forwarding adds an extra network hop.
Path 3: Preserving the Client IP (Local Policy)
Setting externalTrafficPolicy: Local changes this behavior:
spec:
externalTrafficPolicy: Local
Forwarding is now restricted to endpoints on the receiving node:
+---------------------------------------------------------+
| externalTrafficPolicy: Local Routing |
+---------------------------------------------------------+
| Traffic to Node A (with local Pod A) |
| -> DNAT only (dst -> 10.244.1.20:8080) |
| -> Preserves Client IP (src remains 198.51.100.30) |
+---------------------------------------------------------+
| Traffic to Node without local Pod |
| -> Packet is DROPPED (No cross-node forwarding) |
+---------------------------------------------------------+
Node A forwards traffic exclusively to its local API Pod A and skips SNAT entirely, so API Pod A sees the true client IP (198.51.100.30).
WARNING
If external traffic lands on a node that has no local Pod for that Service, kube-proxy drops the packet instead of forwarding it to another node. When paired with a cloud load balancer (type: LoadBalancer), Kubernetes automatically allocates a healthCheckNodePort for the Service; kube-proxy returns HTTP 200 on that port only when at least one local Pod is ready, allowing the load balancer to take nodes without local Pods out of rotation.
Path 4: Hairpin Traffic (A Pod Calling Itself via a Service)
When API Pod A (10.244.1.20) sends a request to its own Service ClusterIP (10.96.0.50:80) and load balancing selects API Pod A itself as the backend, the resulting U-turn is known as hairpin traffic.
[API Pod A: 10.244.1.20]
|
| (dst: 10.96.0.50:80)
v
[veth / Bridge / Netfilter]
|
| DNAT dst -> 10.244.1.20:8080
| SNAT src -> Bridge / Masq IP
|
v
[API Pod A: 10.244.1.20] (src: Masq, dst: 8080)
After the packet leaves Pod A’s network namespace and hits the host bridge or veth interface, DNAT rewrites the destination to Pod A’s own IP. Without SNAT on the source address, Pod A would receive a packet whose source and destination are both 10.244.1.20—which the Linux kernel typically drops during routing or TCP state validation.
For hairpin traffic to succeed, both kubelet networking flags and the CNI plugin must enable hairpin mode (such as hairpin-veth or promiscuous-bridge) so packets can loop back through the same virtual interface and the connection is tracked correctly.
NOTE
Further reading: What Is a Pod Sandbox? Pause Containers, CRI, and the Node Network Chain
Traffic Control Matrix: Policy, Preference, and Affinity
Kubernetes exposes several fields for steering Service traffic across different failure and latency trade-offs:
+---------------------------------------------------------+
| Traffic Control Mechanisms |
+---------------------------------------------------------+
| Local Policy : Hard filter. Local endpoints only, |
| or drop |
| Distribution : Soft preference. Prefer same node/zone |
| Affinity : L4 state stickiness. Same client |
| IP to pod |
+---------------------------------------------------------+
| Mechanism | Core Behavior | When No Local / Same-Zone Pod Exists | Typical Use Case |
|---|---|---|---|
internalTrafficPolicy: Local | Restricts in-cluster traffic to node-local Pods | Connection fails; never forwards to remote nodes | Lower latency and eliminate cross-node traffic |
externalTrafficPolicy: Local | Restricts external traffic to node-local Pods | Packet is dropped; never forwards to remote nodes | Preserve client IP and avoid an extra cross-node hop |
trafficDistribution | Prefers Pods on the same node or availability zone | Falls back to same-zone or cluster-wide Pods | Balance availability with cross-AZ data transfer costs |
sessionAffinity: ClientIP | Pins requests from the same client IP to the same Pod within a time window | Selects another ready Pod | Legacy stateful workloads relying on L4 stickiness |
How conntrack Shapes Long-Lived Connections
Kubernetes Service virtual IPs and NAT rely heavily on Linux Netfilter connection tracking (conntrack) to maintain connection state—a design that is also a common cause of uneven application load and connection anomalies.
TCP Handshakes and Cached State
For TCP traffic, only the initial SYN packet traverses the full Service rule chain and triggers load-balancing selection. Once a backend is chosen, the kernel records a 5-tuple entry (Protocol, Src IP, Src Port, Dst IP, Dst Port) in the conntrack table.
Every subsequent packet belonging to that TCP connection matches the existing conntrack entry and undergoes NAT directly, bypassing load-balancing rules altogether.
Connection Timeline:
1. [Client] --- SYN ---> [Node Netfilter]
(Triggers LB selection -> Pod A)
2. [Client] --- ACK / Data ---> [Node Netfilter]
(Follows conntrack -> Pod A)
3. [Client] --- Subsequent Requests
on same TCP connection ---> [Pod A only]
This behavior produces two classic operational pitfalls:
- HTTP/2 and gRPC persistent connections: Because many application requests multiplex over a single long-lived TCP connection, every request on that connection lands on the same Pod, which can overload individual instances under heavy traffic.
- Lingering traffic after rolling updates: Even after a Pod is removed from its EndpointSlice, existing TCP connections stay pinned in conntrack. New requests sent over an already-open connection continue going to the terminating Pod until the connection times out or the application closes it.
Special Handling for UDP
Because UDP is connectionless, the kernel has no SYN/FIN lifecycle to track. Instead, conntrack uses a timeout window to treat matching UDP packets as a pseudo-connection and preserve their NAT mapping. Until that timer expires, UDP packets arriving from the same source IP and port are still routed to the same backend Pod.
An Outside-In Debugging Workflow
When troubleshooting a broken Service, work from the outside in, following the official Debug Services workflow, testing your assumptions across seven checkpoints:
+---------------------------------------------------------+
| Service Diagnostic Workflow |
+---------------------------------------------------------+
| 1. DNS Check: Can CoreDNS resolve the service name? |
| 2. Spec Check: Are ClusterIP, Port, and targetPort ok? |
| 3. Endpoint Check: Does EndpointSlice contain |
| ready pods? |
| 4. Direct Pod Ping: Can client reach Pod IP:targetPort? |
| 5. Node Scope: Does failure happen on specific Node? |
| 6. Plane Inspection: Check kube-proxy logs |
| & kernel tables |
| 7. Packet Trace: Use tcpdump to locate the missing hop |
+---------------------------------------------------------+
Structured Debugging Toolkit: Four Verification Stages
The seven checkpoints above map cleanly into four hands-on verification stages, moving from the API control plane down to kernel packet captures:
Stage 1: Verify Control Plane and Endpoint State (Checkpoints 1–3)
Before blaming the network data plane, confirm that the API objects and backend endpoints are actually in place:
-
Inspect the Service spec and port mappings:
kubectl -n <namespace> get service <service-name> -o yamlVerify
clusterIP,ports(portvs.targetPort), and traffic policy fields. -
Check the EndpointSlice objects:
kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service-name> -o yamlUsing the standard EndpointSlice label, confirm that the slice lists expected Pod IPs and ports with
ready: true. -
Verify which Pods match the selector:
kubectl -n <namespace> get pods -l '<selector-key>=<selector-value>' -o wideCheck the count, node placement, and readiness status of matching Pods.
Stage 2: Isolate Layers from a Debug Pod (Checkpoints 4–5)
Run targeted probes from a debug Pod inside the cluster to narrow down the failing layer:
# Test 1: DNS resolution layer
kubectl -n <namespace> exec <debug-pod> -- nslookup <service-name>.<namespace>.svc.cluster.local
# Test 2: Service virtual IP layer
kubectl -n <namespace> exec <debug-pod> -- wget -S -T 3 -O- http://<cluster-ip>:<port>/healthz
# Test 3: Direct backend Pod IP layer (bypassing the Service)
kubectl -n <namespace> exec <debug-pod> -- wget -S -T 3 -O- http://<pod-ip>:<target-port>/healthz
- Test 1 fails, Test 2 succeeds: The issue is in CoreDNS or the Pod’s
/etc/resolv.conf. - Test 2 fails, Test 3 succeeds: The issue is in the Service data plane (kube-proxy forwarding rules).
- Test 3 also fails: The issue is in the backend application listener, CNI routing, or a NetworkPolicy.
Stage 3: Inspect kube-proxy Instances and Node Rules (Checkpoint 6)
Once you confirm the fault lies in the data plane, inspect sync status and kernel rules on the affected node:
-
Check the kube-proxy DaemonSet and Pod status:
kubectl -n kube-system get daemonset,pods -l k8s-app=kube-proxy -o wide -
Inspect mode configuration and sync logs:
# Check proxy mode and configuration parameters kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' # Tail recent sync logs on the affected node kubectl -n kube-system logs <kube-proxy-pod-name> --since=30m --prefix -
Inspect kernel forwarding rules on the Linux node:
# iptables mode (Note: KUBE-* chains are internal implementation details, not a stable API—do not couple automation to them) sudo iptables-save -t nat | rg 'KUBE-SVC-|KUBE-SEP-' # nftables mode sudo nft list ruleset | rg -A 10 'kube-proxy' # IPVS mode (legacy reference only) sudo ipvsadm -Ln
Stage 4: Check Kernel conntrack and Capture Packets (Checkpoint 7)
When rules look healthy on paper but traffic still fails, drop down to the kernel connection tracking table and packet captures:
-
Check conntrack table utilization and drop counters:
sudo conntrack -S sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_maxWatch whether
droporinsert_failedcounters are climbing—both mean the kernel failed to confirm new connections in the tracking table: risingdropusually indicates a full conntrack table (the kernel logstable full, dropping packet), whileinsert_failedis more often caused by tuple clashes or race conditions, not necessarily a full table. -
Verify outbound DNAT and return traffic with
tcpdump:sudo tcpdump -ni any 'host <cluster-ip> or host <pod-ip>' -nn -vConfirm whether outbound packets undergo DNAT as expected and whether reply packets make it back to the node and have their source addresses restored.
Common Failure Scenarios
Scenario 1: The Service Exists, but Its EndpointSlice Is Empty
- Symptom: DNS resolves the Service name, but requests to the ClusterIP time out; inspecting the EndpointSlice shows zero endpoints.
- Root cause: A subtle mismatch between
spec.selectoron the Service and the labels on the Pods (such as a typo or mismatched environment label), or failing Readiness Probes on the Pods. - Common misstep: Restarting kube-proxy does nothing here. Because the control plane has no ready endpoints to publish, kube-proxy is working as designed by not programming forwarding targets.
Scenario 2: Local Pods Work, Cross-Node Pods Time Out
- Symptom: Requests succeed whenever load balancing picks a Pod on the same node as the client, but time out whenever it picks a Pod on another node. Direct requests to the remote Pod IP also fail.
- Root cause: A broken CNI overlay tunnel, cloud security groups blocking VXLAN/Geneve ports between nodes, or an MTU mismatch across network interfaces causing large packets to drop silently.
Scenario 3: Intermittent External Drops After Enabling Local Policy
- Symptom: After setting
externalTrafficPolicy: Local, hitting the NodePort works on some nodes but times out on others. - Root cause: This is expected behavior—nodes without a local Pod instance drop external traffic by design. Verify that your external load balancer is configured to probe the Service’s
healthCheckNodePort.
Scenario 4: Terminating Pods Still Receive Traffic During Rolling Updates
- Symptom: A Deployment rollout finishes and old Pods enter
Terminating, yet logs show requests still hitting old Pods or failing with connection resets. - Root cause:
- Connection reuse: Clients using HTTP/2 or gRPC keep sending new requests over existing TCP connections that conntrack continues routing to the old Pod.
- Propagation delay: Removing an endpoint from the EndpointSlice and propagating updated rules to every node takes a brief moment, creating a gap after Pod termination begins.
- Missing graceful shutdown: The Pod is terminated before finishing in-flight requests and draining active connections.
Summary: Quick Diagnostic Reference
Use this cheat sheet to map common symptoms to the first layer you should investigate:
| Symptom | Primary Suspect Layer | Next Verification Step |
|---|---|---|
| Service DNS resolution fails | CoreDNS or Pod DNS config | Run nslookup and inspect /etc/resolv.conf in the Pod |
| EndpointSlice has no endpoints | Label selector or Readiness Probe | Compare Service selector with Pod labels and probe status |
| Direct Pod IP access also fails | App listener, CNI routing, or NetworkPolicy | Check the container’s listening port and inter-node firewall rules |
| Direct Pod IP works, ClusterIP fails | Node data plane rules | Check kube-proxy sync logs and node NAT rules (iptables / nft) |
| NodePort works only on some nodes | externalTrafficPolicy: Local | Check which nodes run matching Pods and verify LB health checks |
| Old Pods still get traffic on rollout | conntrack state and connection reuse | Audit client connection pooling and Pod graceful shutdown handling |