How kube-proxy Works (Part 1): A Rule Synchronizer, Not a Proxy Server
The name “kube-proxy” is misleading. Many engineers picture a daemon sitting on each node like Nginx or Envoy, accepting packets in user space and forwarding them to backends.
In reality, kube-proxy itself never forwards application packets.
At its core, it is a node-local controller for Service forwarding rules. Running in the background, it watches the Kubernetes API and translates declarative Services and EndpointSlices into Linux kernel forwarding rules (iptables, nftables, or IPVS). The actual per-packet work—filtering, load-balancing selection, and destination network address translation (DNAT)—is mostly performed by the Linux kernel’s network stack.
Core Role: Synchronizing Layer 4 Forwarding Rules
To define kube-proxy in a single sentence:
kube-proxy turns the stable virtual entry point declared by a Kubernetes Service into Layer 4 forwarding rules enforced on every node.
This role comes down to four properties:
- A Service is purely declarative: A Service object specifies a virtual IP, ports, and backend selection criteria. It is not a running process that forwards packets.
- A ClusterIP is a purely virtual IP (VIP): As documented in Virtual IPs and Service Proxies, a ClusterIP is not bound to any physical NIC or virtual interface, and no process listens on a socket at that address. It exists solely in Linux kernel forwarding rules—which is why running
pingagainst a ClusterIP from a node usually gets no response. - Every node handles ClusterIP traffic locally: Regardless of which node a Pod runs on, that node’s data plane must recognize traffic destined for a ClusterIP and forward it accordingly.
- It operates at Layer 4 (transport layer): kube-proxy mainly handles IP and port translation for TCP, UDP, and SCTP traffic. It cannot interpret Layer 7 application details such as HTTP headers, URL paths, or cookies.
Why Do We Need Services and Virtual IPs?
In Kubernetes, Pods are ephemeral. Rolling updates, node failovers, and horizontal autoscaling constantly replace Pods, and every replacement gets a new Pod IP.
If client Pods cached backend Pod IPs directly, every client would have to implement health checks, cache invalidation, load-balancing algorithms, and race-condition handling whenever endpoints changed. The Service mechanism separates these two responsibilities:
- Client Pod
- Send to Service ClusterIP: stable virtual IPService ClusterIP
- Kernel DNATSelected Backend PodDynamic Pod IP
- Clients talk to a stable virtual entry point: They communicate through a fixed Service DNS name and ClusterIP.
- The control plane dynamically maintains the backend set: The EndpointSlice controller in the control plane (covered next) continuously tracks which Pods are healthy and ready.
Note: When you need to bypass virtual IP proxying—for example, with a Headless Service (clusterIP: None)—CoreDNS returns all backend Pod IPs directly, leaving load balancing to the client.
How the Five Networking Components Work Together
Internal service access in Kubernetes relies on a tight division of labor across control plane controllers, a node daemon, and the Linux kernel:
- ServiceVIP definition
- EndpointSlice Controller reconciles selectorEndpointSliceReady endpoint IPs & ports
- Watch APIkube-proxyNode Daemon
- Write rulesLinux Kernel Datapathiptables / IPVS / nftables
- Client Pod
- Send to VIP:PortLinux Kernel Datapath
- Kernel DNAT to Pod IP & route via CNIBackend Pod
1. Service: Declaring the Entry Point and Intent
When a developer creates a Service, the Kubernetes API server allocates a ClusterIP (for example, 10.96.0.50). This object simply records an intent in etcd: “Connections sent to 10.96.0.50:80 should route to backend containers carrying the label app: api.”
2. EndpointSlice: Tracking Available Backends
The EndpointSlice controller in the control plane watches Pods matching the Service selector and compiles their IPs, ports, node placement, and readiness conditions (Ready, Serving, Terminating) into EndpointSlice objects (solving the performance and update-broadcast bottlenecks that the legacy monolithic Endpoints object caused in large clusters).
EndpointSlices serve as kube-proxy’s source of truth when choosing forwarding targets. If a Pod has not passed its Readiness Probe, its address is excluded from the active forwarding endpoints.
3. kube-proxy: The Reconciling Rule Controller
On every node, kube-proxy watches the API server for changes to Services and EndpointSlices. Internally, it runs a control loop: whenever it detects updates, it batches the changes and triggers a local sync, updating the kernel forwarding rules to match the latest desired state.
NOTE
Further reading: An Introduction to etcd: The Memory of a Kubernetes Cluster
4. Linux Kernel Data Plane: Per-Packet Forwarding
Once kube-proxy writes the rules, packets entering the Linux kernel’s network stack are intercepted on the fly. Following those rules, the kernel picks a suitable backend endpoint and performs DNAT to rewrite the destination address from the ClusterIP to the Pod IP. The entire process happens in kernel space without switching back to a user-space application.
5. CNI and the Pod Network: Underlying Packet Delivery
After the kernel completes DNAT, the packet’s destination IP is now an actual Pod IP. At this point, the underlying network topology built by the CNI (Container Network Interface) plugin takes over, delivering the packet to the target Pod on the same node or across nodes.
NOTE
Further reading: What Is a Pod Sandbox? Pause Containers, CRI, and the Node Network Chain
Why Decoupling the Control and Data Planes Matters
Separating the control plane (declaring state) from the data plane (forwarding packets) brings three main advantages:
- No intermediary lookups on the packet path: Forwarding a packet never requires querying the API server or kube-proxy; the kernel matches and rewrites packets directly.
- High fault tolerance: Even if the API server lags or the kube-proxy process restarts briefly, the iptables, nftables, or IPVS rules already loaded into the Linux kernel keep running, leaving existing connections undisturbed.
- Clear separation of responsibilities: CoreDNS handles domain name resolution, controllers compute active endpoints, kube-proxy syncs the forwarding rules, and the CNI plugin plus the Linux kernel handle underlying packet routing.
Common Misconceptions Clarified
When learning or troubleshooting Kubernetes service networking, three misconceptions come up repeatedly:
Misconception 1: Does restarting kube-proxy immediately drop connections to Services?
No. Kernel forwarding rules live in the Linux kernel’s memory, independent of the kube-proxy process lifecycle.
When kube-proxy restarts or goes temporarily offline, the configured forwarding rules normally remain in effect, and connections to existing Services continue to flow. The main impact during that window is that the rules stop tracking current state: if Pods are added or deleted, or their readiness state changes, the node cannot immediately sync the latest endpoint list, and Services created during the outage do not get forwarding rules yet.
Misconception 2: Are HTTP requests distributed evenly across backend Pods?
Not necessarily. kube-proxy is a Layer 4 (transport layer) mechanism. The Linux kernel’s conntrack (connection tracking) module selects a backend endpoint and applies DNAT only when it receives the initial TCP SYN packet. Once the connection state is established, every subsequent packet reuses that exact same NAT mapping.
As a result, when applications use HTTP/2, gRPC, keep-alive connections, or connection pools, a large volume of HTTP requests shares a single long-lived TCP connection, which lands all of those requests on the same Pod.
NOTE
Further reading: How kube-proxy Works (Part 2): Packet Flows, Proxy Modes, and Debugging
If you need per-request load balancing based on HTTP headers or paths, use a Layer 7 Ingress controller or the Gateway API.
NOTE
Further reading: A Practical Guide to Kubernetes Gateway API
Misconception 3: Does a cluster need kube-proxy in order to use Service virtual IPs?
Not necessarily. kube-proxy is Kubernetes’ default component for synchronizing Service forwarding rules, but it is not the only option.
If a cluster runs a modern eBPF-capable CNI—such as Cilium’s kube-proxy-replacement—the CNI attaches eBPF programs directly in the Linux kernel: a cgroup hook performs ClusterIP address translation at the socket layer, the TC (traffic control) layer handles DNAT, and XDP serves as an optional accelerator. In this architecture, Services and EndpointSlices normally continue to work even with kube-proxy completely disabled.
In Part 2, we will step into the Linux kernel data plane, trace concrete IP topologies across iptables, nftables, and IPVS, and follow the SNAT and cross-node packet flows that NodePort and LoadBalancer introduce.
NOTE
Further reading: How kube-proxy Works (Part 2): Packet Flows, Proxy Modes, and Debugging