How kube-proxy Works (Part 2): Packet Flows, Proxy Modes, and Debugging

In Part 1, we traced the control chain behind a Kubernetes Service: the Service object declares an entry point, the EndpointSlice controller tracks ready backends, kube-proxy syncs forwarding rules onto each node, and the Linux kernel delivers the packets.

This part moves into the data plane. Using a concrete IP topology, we will trace how packet addresses change across DNAT (Destination Network Address Translation), SNAT (Source Network Address Translation), and conntrack (connection tracking); examine where iptables, IPVS, and nftables stand in 2026; and walk through a four-stage debugging toolkit along with common failure scenarios.


Proxy Modes in 2026: From IPVS Deprecation to nftables

As of Kubernetes v1.37, the official kube-proxy implementation on Linux nodes supports three modes: iptables, IPVS, and nftables. (Windows nodes use a separate kernelspace mode; this article focuses on Linux networking.)

 +---------------------------------------------------------+
 | kube-proxy Modes                                        |
 +---------------------------------------------------------+
 | iptables : Default, battle-tested, linear rule cost     |
 | IPVS     : Deprecated since v1.35, slated for removal   |
 | nftables : Modern alternative, set-based, fast updates  |
 +---------------------------------------------------------+

Here is how the three modes compare in underlying mechanism and lifecycle status:

ModeKernel TargetProcessing LayerStatus (v1.37)Characteristics and Limitations
iptablesnetfilter rule chainsLinux kernelCurrent defaultBattle-tested and stable; long chains at massive scale make full rule refreshes expensive
IPVSVirtual / Real ServersLinux kernelDeprecatedCannot natively express full Service semantics; still relies on auxiliary iptables rules
nftablesTables, Sets, and MapsLinux kernelRecommended modern pathRequires kernel 5.13+; uses set lookups and incremental updates whose cost scales with the delta

Why Is nftables Replacing IPVS?

The community originally introduced IPVS to bypass the linear rule-traversal bottleneck of iptables when clusters scaled to tens of thousands of rules. As Kubernetes Service semantics grew richer—adding externalTrafficPolicy: Local, source IP range filtering, topology-aware routing, and more—the IPVS API could no longer express every condition natively, forcing kube-proxy to fall back on iptables chains alongside IPVS.

nftables natively supports Set and Map data structures (see KEP-3866), performing efficient lookups within a single rule and supporting atomic incremental updates: kube-proxy only pushes the endpoints that changed in a given sync cycle instead of rewriting the entire ruleset, and atomic replacement prevents partially applied rules from ever taking effect.

NOTE

Under the official IPVS deprecation plan, IPVS mode was deprecated in Kubernetes v1.35, is slated to be disabled by default in v1.40, and is scheduled for removal in v1.43. New clusters should no longer adopt IPVS based on older guides; stay on iptables or evaluate migrating to nftables.


Tracing Core Packet Flows: How Addresses Change Across Scenarios

Reference Topology

To make each address translation step concrete, all packet traces below share the following network topology:

RoleIP / PortNotes
External Client198.51.100.30Test client outside the cluster
Node A192.0.2.10Host IP of Node A
Node B192.0.2.11Host IP of Node B
Client Pod10.244.1.10The originating Pod, running on Node A
API Pod A10.244.1.20:8080Backend Pod running on Node A
API Pod B10.244.2.20:8080Backend Pod running on Node B
Service ClusterIP10.96.0.50:80Virtual IP for the api Service
Service NodePort30080NodePort for the api Service

Here is the corresponding Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  type: NodePort
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080

Path 1: ClusterIP to a Cross-Node Pod

When the Client Pod initiates a connection to the Service, the packet enters the network stack on its local node, Node A:

  1. Client Pod: 10.244.1.10
  2. dst: 10.96.0.50:80Node A Netfilter / Data Plane1. Match ClusterIP 10.96.0.50:80; 2. Select backend Pod B (10.244.2.20:8080); 3. Execute DNAT (dst -> 10.244.2.20:8080); 4. Record connection tuple into conntrack table
  3. src: 10.244.1.10, dst: 10.244.2.20:8080 (CNI Routing / Overlay)Node B / API Pod B: 10.244.2.20:8080
  1. Outbound DNAT: Node A’s kernel network stack matches the ClusterIP rule, selects API Pod B as the backend, and rewrites the destination to 10.244.2.20:8080. The source IP stays 10.244.1.10 (no SNAT).
  2. CNI forwarding: The underlying CNI plugin routes the packet across nodes—via an overlay network or direct routing—to Pod B on Node B.
  3. Return reverse translation: Pod B sends its reply back to 10.244.1.10. When the packet arrives at Node A, Linux conntrack matches the tracked connection and automatically translates the source address back from 10.244.2.20:8080 to 10.96.0.50:80. From the Client Pod’s perspective, it has only ever communicated with the Service IP.

Path 2: Default NodePort Routing (Cluster Policy)

When an external client hits NodePort 30080 on Node A, and the data plane picks Pod B on Node B:

  1. External Client: 198.51.100.30
  2. dst: 192.0.2.10:30080Node A Netfilter / Data Plane1. DNAT: dst 192.0.2.10:30080 -> 10.244.2.20:8080; 2. SNAT (MASQUERADE): src -> 192.0.2.10
  3. src: 192.0.2.10, dst: 10.244.2.20:8080Node B / API Pod B

Under externalTrafficPolicy: Cluster (the default), as detailed in the official Using Source IP guide, Node A must perform SNAT—replacing the source IP with its own node IP—so that reply packets from Node B return through Node A and can have their pre-NAT addresses restored before heading back to the client.

Path 3: Preserving the Client IP (Local Policy)

Setting externalTrafficPolicy: Local changes this behavior:

spec:
  externalTrafficPolicy: Local

Forwarding is now restricted to endpoints on the receiving node:

 +---------------------------------------------------------+
 | externalTrafficPolicy: Local Routing                    |
 +---------------------------------------------------------+
 | Traffic to Node A (with local Pod A)                    |
 |  -> DNAT only (dst -> 10.244.1.20:8080)                 |
 |  -> Preserves Client IP (src remains 198.51.100.30)     |
 +---------------------------------------------------------+
 | Traffic to Node without local Pod                       |
 |  -> Packet is DROPPED (No cross-node forwarding)        |
 +---------------------------------------------------------+

Node A forwards traffic exclusively to its local API Pod A and skips SNAT entirely, so API Pod A sees the true client IP (198.51.100.30).

WARNING

If external traffic lands on a node that has no local Pod for that Service, kube-proxy drops the packet instead of forwarding it to another node. When paired with a cloud load balancer (type: LoadBalancer), Kubernetes automatically allocates a healthCheckNodePort for the Service; kube-proxy returns HTTP 200 on that port only when at least one local Pod is ready, allowing the load balancer to take nodes without local Pods out of rotation.

Path 4: Hairpin Traffic (A Pod Calling Itself via a Service)

When API Pod A (10.244.1.20) sends a request to its own Service ClusterIP (10.96.0.50:80) and load balancing selects API Pod A itself as the backend, the resulting U-turn is known as hairpin traffic.

 [API Pod A: 10.244.1.20]
        |
        | (dst: 10.96.0.50:80)
        v
 [veth / Bridge / Netfilter]
        |
        | DNAT dst -> 10.244.1.20:8080
        | SNAT src -> Bridge / Masq IP
        |
        v
 [API Pod A: 10.244.1.20]  (src: Masq, dst: 8080)

After the packet leaves Pod A’s network namespace and hits the host bridge or veth interface, DNAT rewrites the destination to Pod A’s own IP. Without SNAT on the source address, Pod A would receive a packet whose source and destination are both 10.244.1.20—which the Linux kernel typically drops during routing or TCP state validation.

For hairpin traffic to succeed, both kubelet networking flags and the CNI plugin must enable hairpin mode (such as hairpin-veth or promiscuous-bridge) so packets can loop back through the same virtual interface and the connection is tracked correctly.


Traffic Control Matrix: Policy, Preference, and Affinity

Kubernetes exposes several fields for steering Service traffic across different failure and latency trade-offs:

 +---------------------------------------------------------+
 | Traffic Control Mechanisms                              |
 +---------------------------------------------------------+
 | Local Policy  : Hard filter. Local endpoints only,      |
 |                 or drop                                 |
 | Distribution  : Soft preference. Prefer same node/zone  |
 | Affinity      : L4 state stickiness. Same client        |
 |                 IP to pod                               |
 +---------------------------------------------------------+
MechanismCore BehaviorWhen No Local / Same-Zone Pod ExistsTypical Use Case
internalTrafficPolicy: LocalRestricts in-cluster traffic to node-local PodsConnection fails; never forwards to remote nodesLower latency and eliminate cross-node traffic
externalTrafficPolicy: LocalRestricts external traffic to node-local PodsPacket is dropped; never forwards to remote nodesPreserve client IP and avoid an extra cross-node hop
trafficDistributionPrefers Pods on the same node or availability zoneFalls back to same-zone or cluster-wide PodsBalance availability with cross-AZ data transfer costs
sessionAffinity: ClientIPPins requests from the same client IP to the same Pod within a time windowSelects another ready PodLegacy stateful workloads relying on L4 stickiness

How conntrack Shapes Long-Lived Connections

Kubernetes Service virtual IPs and NAT rely heavily on Linux Netfilter connection tracking (conntrack) to maintain connection state—a design that is also a common cause of uneven application load and connection anomalies.

TCP Handshakes and Cached State

For TCP traffic, only the initial SYN packet traverses the full Service rule chain and triggers load-balancing selection. Once a backend is chosen, the kernel records a 5-tuple entry (Protocol, Src IP, Src Port, Dst IP, Dst Port) in the conntrack table.

Every subsequent packet belonging to that TCP connection matches the existing conntrack entry and undergoes NAT directly, bypassing load-balancing rules altogether.

 Connection Timeline:
 1. [Client] --- SYN ---> [Node Netfilter]
    (Triggers LB selection -> Pod A)
 2. [Client] --- ACK / Data ---> [Node Netfilter]
    (Follows conntrack -> Pod A)
 3. [Client] --- Subsequent Requests
    on same TCP connection ---> [Pod A only]

This behavior produces two classic operational pitfalls:

  1. HTTP/2 and gRPC persistent connections: Because many application requests multiplex over a single long-lived TCP connection, every request on that connection lands on the same Pod, which can overload individual instances under heavy traffic.
  2. Lingering traffic after rolling updates: Even after a Pod is removed from its EndpointSlice, existing TCP connections stay pinned in conntrack. New requests sent over an already-open connection continue going to the terminating Pod until the connection times out or the application closes it.

Special Handling for UDP

Because UDP is connectionless, the kernel has no SYN/FIN lifecycle to track. Instead, conntrack uses a timeout window to treat matching UDP packets as a pseudo-connection and preserve their NAT mapping. Until that timer expires, UDP packets arriving from the same source IP and port are still routed to the same backend Pod.


An Outside-In Debugging Workflow

When troubleshooting a broken Service, work from the outside in, following the official Debug Services workflow, testing your assumptions across seven checkpoints:

 +---------------------------------------------------------+
 | Service Diagnostic Workflow                             |
 +---------------------------------------------------------+
 | 1. DNS Check: Can CoreDNS resolve the service name?     |
 | 2. Spec Check: Are ClusterIP, Port, and targetPort ok?  |
 | 3. Endpoint Check: Does EndpointSlice contain           |
 |    ready pods?                                          |
 | 4. Direct Pod Ping: Can client reach Pod IP:targetPort? |
 | 5. Node Scope: Does failure happen on specific Node?    |
 | 6. Plane Inspection: Check kube-proxy logs              |
 |    & kernel tables                                      |
 | 7. Packet Trace: Use tcpdump to locate the missing hop  |
 +---------------------------------------------------------+

Structured Debugging Toolkit: Four Verification Stages

The seven checkpoints above map cleanly into four hands-on verification stages, moving from the API control plane down to kernel packet captures:

Stage 1: Verify Control Plane and Endpoint State (Checkpoints 1–3)

Before blaming the network data plane, confirm that the API objects and backend endpoints are actually in place:

  1. Inspect the Service spec and port mappings:

    kubectl -n <namespace> get service <service-name> -o yaml

    Verify clusterIP, ports (port vs. targetPort), and traffic policy fields.

  2. Check the EndpointSlice objects:

    kubectl -n <namespace> get endpointslice -l kubernetes.io/service-name=<service-name> -o yaml

    Using the standard EndpointSlice label, confirm that the slice lists expected Pod IPs and ports with ready: true.

  3. Verify which Pods match the selector:

    kubectl -n <namespace> get pods -l '<selector-key>=<selector-value>' -o wide

    Check the count, node placement, and readiness status of matching Pods.

Stage 2: Isolate Layers from a Debug Pod (Checkpoints 4–5)

Run targeted probes from a debug Pod inside the cluster to narrow down the failing layer:

# Test 1: DNS resolution layer
kubectl -n <namespace> exec <debug-pod> -- nslookup <service-name>.<namespace>.svc.cluster.local

# Test 2: Service virtual IP layer
kubectl -n <namespace> exec <debug-pod> -- wget -S -T 3 -O- http://<cluster-ip>:<port>/healthz

# Test 3: Direct backend Pod IP layer (bypassing the Service)
kubectl -n <namespace> exec <debug-pod> -- wget -S -T 3 -O- http://<pod-ip>:<target-port>/healthz

Stage 3: Inspect kube-proxy Instances and Node Rules (Checkpoint 6)

Once you confirm the fault lies in the data plane, inspect sync status and kernel rules on the affected node:

  1. Check the kube-proxy DaemonSet and Pod status:

    kubectl -n kube-system get daemonset,pods -l k8s-app=kube-proxy -o wide
  2. Inspect mode configuration and sync logs:

    # Check proxy mode and configuration parameters
    kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}'
    
    # Tail recent sync logs on the affected node
    kubectl -n kube-system logs <kube-proxy-pod-name> --since=30m --prefix
  3. Inspect kernel forwarding rules on the Linux node:

    # iptables mode (Note: KUBE-* chains are internal implementation details, not a stable API—do not couple automation to them)
    sudo iptables-save -t nat | rg 'KUBE-SVC-|KUBE-SEP-'
    
    # nftables mode
    sudo nft list ruleset | rg -A 10 'kube-proxy'
    
    # IPVS mode (legacy reference only)
    sudo ipvsadm -Ln

Stage 4: Check Kernel conntrack and Capture Packets (Checkpoint 7)

When rules look healthy on paper but traffic still fails, drop down to the kernel connection tracking table and packet captures:

  1. Check conntrack table utilization and drop counters:

    sudo conntrack -S
    sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

    Watch whether drop or insert_failed counters are climbing—both mean the kernel failed to confirm new connections in the tracking table: rising drop usually indicates a full conntrack table (the kernel logs table full, dropping packet), while insert_failed is more often caused by tuple clashes or race conditions, not necessarily a full table.

  2. Verify outbound DNAT and return traffic with tcpdump:

    sudo tcpdump -ni any 'host <cluster-ip> or host <pod-ip>' -nn -v

    Confirm whether outbound packets undergo DNAT as expected and whether reply packets make it back to the node and have their source addresses restored.


Common Failure Scenarios

Scenario 1: The Service Exists, but Its EndpointSlice Is Empty

Scenario 2: Local Pods Work, Cross-Node Pods Time Out

Scenario 3: Intermittent External Drops After Enabling Local Policy

Scenario 4: Terminating Pods Still Receive Traffic During Rolling Updates


Summary: Quick Diagnostic Reference

Use this cheat sheet to map common symptoms to the first layer you should investigate:

SymptomPrimary Suspect LayerNext Verification Step
Service DNS resolution failsCoreDNS or Pod DNS configRun nslookup and inspect /etc/resolv.conf in the Pod
EndpointSlice has no endpointsLabel selector or Readiness ProbeCompare Service selector with Pod labels and probe status
Direct Pod IP access also failsApp listener, CNI routing, or NetworkPolicyCheck the container’s listening port and inter-node firewall rules
Direct Pod IP works, ClusterIP failsNode data plane rulesCheck kube-proxy sync logs and node NAT rules (iptables / nft)
NodePort works only on some nodesexternalTrafficPolicy: LocalCheck which nodes run matching Pods and verify LB health checks
Old Pods still get traffic on rolloutconntrack state and connection reuseAudit client connection pooling and Pod graceful shutdown handling