Why Do Service Requests Slow Down in Large Clusters? iptables vs. IPVS in the Kernel
When calling a Kubernetes Service’s ClusterIP, you quickly run into a counterintuitive phenomenon: in the default iptables mode, this virtual IP is not bound to any network interface on the node and never responds to ping (ICMP), yet TCP and UDP requests sent to the designated port reliably reach backend Pods.
In a cluster with a few hundred Services, the default iptables mode is barely noticeable. Once the cluster scales to thousands or tens of thousands of Services, however, node CPU usage spikes, newly started Pods take noticeably longer to start receiving traffic, and sporadic connection timeouts begin to surface.
Ever since Kubernetes 1.2 replaced the early userspace proxy mode with iptables as the default, kube-proxy has stopped forwarding packets itself (staying out of the data plane). Instead, it operates purely in the control plane—watching the API Server for Service and EndpointSlice changes and programming forwarding rules into the Linux kernel.
At scale, forwarding performance comes down to three aspects of the Linux kernel networking stack: which Netfilter hooks a packet traverses, whether the kernel matches rules against an linear array or an hash table, and whether adding a single Pod forces the kernel to lock and rewrite the entire table or update a single entry in place.
1. Where Packets Go Inside the Kernel: The Five Netfilter Hooks and conntrack
In the Linux kernel architecture, the iptables firewall we interact with every day is split into two layers:
- Netfilter: The packet-processing framework inside Linux kernel space. It places five hook points along critical paths in the networking stack, allowing kernel modules to register callbacks that inspect, modify, or drop packets.
- iptables: The command-line configuration utility in userspace. Using
getsockopt/setsockoptsystem calls or the Netlink interface, it compiles rules and injects them into the kernel’s Netfilter tables.
Netfilter’s Five Hook Points and Routing Decisions
When a network packet enters the Linux kernel, it passes through a subset of the five hook points defined by Netfilter, depending on the routing decision:
Netfilter Five Hooks and Packet Flow
NIC Receives Packet
|
v
+-----------------------------------------
| 1. NF_INET_PRE_ROUTING
| Pre-routing: DNAT rewrites dst IP
+-----------------------------------------
|
v
Routing Decision (Kernel FIB Lookup)
|
+--> [Branch A: Dst IP is a local address]
| |
| v
| +---------------------------------
| | 2. NF_INET_LOCAL_IN
| | Pre-local: IPVS intercepts here
| +---------------------------------
| |
| v
| Local Process (Outbound packet)
| |
| v
| +---------------------------------
| | 4. NF_INET_LOCAL_OUT
| | Outbound: route lookup & filter
| +---------------------------------
| |
| +---> Joins NF_INET_POST_ROUTING
|
+--> [Branch B: Dst IP is non-local]
|
v
+---------------------------------
| 3. NF_INET_FORWARD
| Cross-NIC forward (ip_forward=1)
+---------------------------------
|
v
+-----------------------------------------
| 5. NF_INET_POST_ROUTING
| Pre-egress: SNAT / Masquerade
+-----------------------------------------
|
v
NIC Transmits Packet
Each of the five hook points serves a distinct purpose:
NF_INET_PRE_ROUTING(PREROUTING): The first stop after a packet is received by the NIC driver and passes basic IP header checksum validation. Because the routing table has not been consulted yet, this is the ideal place for Destination Network Address Translation (DNAT)—such as rewriting a targetClusterIPto an actual Pod IP.NF_INET_LOCAL_IN(INPUT): Triggered after the kernel routing table (FIB, Forwarding Information Base) determines that the destination IP is a local address on the host, before the packet is handed off to a local user process (such as Nginx or a Python socket). As we will see below, IPVS mode binds everyClusterIPto a local dummy interface specifically to steer traffic into this hook for interception.NF_INET_FORWARD(FORWARD): Triggered when the routing decision finds that the destination IP does not belong to the local host and kernel IP forwarding (net.ipv4.ip_forward = 1) is enabled, forwarding the packet to another interface or next-hop node.NF_INET_LOCAL_OUT(OUTPUT): Triggered when a local process generates an outbound packet and performs its initial route lookup.NF_INET_POST_ROUTING(POSTROUTING): Triggered after the final routing decision is complete and right before the packet leaves a physical or virtual NIC. This is where Source Network Address Translation (SNAT / Masquerade) takes place.
Rules attached to the same hook point are organized into four tables by function. Different hooks attach different subsets of tables; taking OUTPUT (which traverses all four) as an example, the execution order from highest to lowest priority is: raw (marks packets as NOTRACK to bypass connection tracking) → mangle (modifies TTL or TOS, or attaches a kernel MARK) → nat (address translation) → filter (the default table, enforcing ACCEPT, DROP, and REJECT access control).
conntrack: Why Only the First Packet Traverses the nat Table
Under iptables, only the first packet of each connection (such as a TCP SYN) walks the nat table in full. Subsequent packets skip the table entirely thanks to Netfilter’s conntrack (connection tracking) module.
When the initial packet (such as a TCP SYN) enters the kernel, conntrack extracts the 5-tuple (protocol, source/destination IP, source/destination port) from the L3/L4 headers and marks the state as NEW. Once the packet traverses the nat table, selects a backend Pod, and completes DNAT, the kernel records the pre- and post-translation mapping in an in-memory bidirectional hash table. Each entry holds two directional tuples:
- ORIGINAL Tuple: Records the outbound direction before translation (e.g.,
Client IP:Port -> Service VIP:Port). - REPLY Tuple: Records the expected return direction from the backend Pod after DNAT (e.g.,
Pod IP:Port -> Client IP:Port).
When the backend Pod sends back its acknowledgment packet (such as SYN-ACK), the kernel matches the REPLY Tuple, transitions the connection state to ESTABLISHED, and automatically rewrites the source address from Pod IP back to Service VIP before delivering it to the client. Once a connection is ESTABLISHED, all subsequent packets in both directions look up the conntrack hash table directly to translate addresses without ever re-entering the nat table.
This explains why the rule-scanning overhead of iptables is concentrated at the moment a new connection is established (the initial SYN packet). Long-lived connections breeze through via conntrack once established, whereas high-concurrency short-lived connections force every new connection to walk the entire rule chain from scratch.
WARNING
Short-Lived Connection Storms and Packet Drops from Full conntrack Tables: conntrack must track every active connection in kernel memory (whether the cluster runs iptables or IPVS mode). If a sudden burst of short-lived connections pushes active entries past the system limit (net.netfilter.nf_conntrack_max), the kernel drops incoming packets that would need new tracking entries and logs nf_conntrack: table full, dropping packet in the kernel ring buffer.
2. Why iptables Slows Down in Large Clusters: O(N) Linear Scans and Full-Table Locks
In large Kubernetes clusters, iptables hits two distinct bottlenecks: linear searches in the data plane and full-table rule replacement in the control plane.
Data Plane: O(N) Matching Across Contiguous Arrays and Probabilistic Chains
Inside the Linux kernel, an iptables chain is stored as a contiguous array of rules. When a packet enters a chain, the kernel matching engine must inspect IP addresses, ports, protocols, and network interfaces sequentially starting from rule #1.
If a rule does not match, evaluation advances to the next rule until it hits one and executes the target action (such as ACCEPT, DROP, MARK, or JUMP to a subchain). Consequently, iptables rule matching has a time complexity of , where is the total number of rules in the chain.
In Kubernetes iptables mode, whether traffic arrives from an external NIC (entering PREROUTING) or originates from a local process or Pod calling a Service (entering OUTPUT), kube-proxy steers it into a custom KUBE-SERVICES chain and routes it through a three-tier chain hierarchy:
- PREROUTING / OUTPUT
- KUBE-SERVICES (Entry chain)Match target ClusterIP:Port (O(N) scan)
- KUBE-SVC-XXXXX (Per-Service subchain)
-
- Rule 1: Random prob 1/3 (0.33333)KUBE-SEP-AAAADNAT to Pod A
- Rule 2: 1/2 of remaining (0.50000)KUBE-SEP-BBBBDNAT to Pod B
- Rule 3: 1/1 of remaining (1.00000)KUBE-SEP-CCCCDNAT to Pod C
KUBE-SERVICES: The top-level entry point for all traffic destined for aClusterIP, where packets are compared rule by rule to identify which Service they belong to.KUBE-SVC-*: A subchain dedicated to a single Service, responsible for distributing traffic across its backend Pods.KUBE-SEP-*(Service Endpoint): Represents a specific backend Pod IP and performs the actual DNAT rewrite.
Because iptables has no built-in load-balancing algorithms, kube-proxy simulates randomized traffic splitting inside KUBE-SVC-* chains using -m statistic --mode random --probability P.
With three backend Pods, Rule 1 jumps to Pod A with probability ; if not selected, Rule 2 jumps to Pod B with a probability across the remaining traffic; the final rule unconditionally jumps to Pod C with probability .
In a small cluster with few rules, this works fine. But when a cluster runs thousands of Services—each backed by dozens or hundreds of Pods—KUBE-SERVICES and KUBE-SVC-* balloon into chains tens of thousands of rules long, subjecting the SYN packet of every new connection to heavy linear scanning overhead.
Control Plane: Full-Table Copy, Write-Back, and xtables_lock
The second bottleneck lies in how iptables updates rules. iptables does not support atomic insertion or deletion of a single rule. Whenever kube-proxy needs to modify a single rule after a Pod restart or scaling event, it must execute a five-step cycle:
- Acquire the global userspace mutex lock (
xtables_lock, located at/run/xtables.lockby default). - Dump and copy the entire rule table from kernel space into userspace.
- Modify the rule array in userspace memory.
- Call
iptables-restoreto write the entire updated rule array back to the kernel and replace the old table in a single system call (while the kernel holds an exclusive table mutex for the protocol family). - Release the global userspace mutex lock.
When a cluster undergoes frequent Horizontal Pod Autoscaling (HPA) or rolling updates, this “full-table copy + global lock + full-table replacement” cycle keeps xtables_lock under constant contention, stalling rule synchronization and network responsiveness across the node.
3. How IPVS Breaks the Bottleneck: O(1) Hash Tables, LOCAL_IN Interception, and Incremental Updates
Unlike iptables, which is a general-purpose packet filter, IPVS (IP Virtual Server) (originating from the LVS project and merged into the mainline Linux kernel in 2.4.23/2.6.0) is a kernel subsystem purpose-built for Layer 4 (transport layer) load balancing. It schedules TCP and UDP traffic destined for virtual service addresses (VIP:Port) across backend Real Servers (RIP:Port).
O(1) Hash Table Lookups and Native Scheduling Algorithms
Inside the kernel, IPVS replaces linear arrays with large hash tables to store state:
- Service Hash Table: Keyed by
(VIP, Protocol, Port). Whether a node hosts 10 or 100,000 Services, IPVS needs only a single hash computation to check if a target VIP exists, achieving lookup time complexity. - IPVS Connection Table: For established connections, IPVS maintains its own connection state keyed by the client 5-tuple, which also resolves in time.
For rule updates, IPVS uses the Netlink API to add or remove individual virtual services or backend Real Servers in place, with reads protected by RCU and updates guarded by fine-grained kernel locks—eliminating the need to copy and replace the entire rule table.
Furthermore, IPVS natively provides more than eight L4 scheduling algorithms in kernel space—including round-robin (rr), weighted round-robin (wrr), least-connection (lc), the default weighted least-connection (wlc), and source/destination IP hashing (sh for session affinity, and dh)—removing the need to approximate load balancing with random probability rules.
Why Does IPVS Hook into LOCAL_IN? How kube-ipvs0 and ipset Work Together
Even though IPVS is an independent load-balancing module, it still relies on Netfilter hooks to intercept packets—except it registers at NF_INET_LOCAL_IN rather than PREROUTING, where DNAT normally happens.
Why LOCAL_IN? The answer goes back to the historical architecture of IPVS, which grew out of the LVS (Linux Virtual Server) project: it was originally designed for dedicated standalone load-balancer appliances. On a traditional bare-metal load balancer, the virtual IP (VIP) is bound directly to a physical NIC on the host, so the kernel naturally classifies incoming VIP traffic as destined for the local machine and delivers it to LOCAL_IN for IPVS to schedule.
In a Kubernetes cluster, however, IPVS is distributed across every worker node, and a Service ClusterIP is not a physical IP of any node.
To reuse IPVS’s LOCAL_IN interception model, kube-proxy has to “convince” the Linux kernel that every ClusterIP is a local address owned by the host; otherwise, after passing PREROUTING and consulting the routing table, the packet would be routed to the FORWARD path or dropped outright.
At the same time, IPVS focuses strictly on L4 VIP-to-Real-Server distribution and lacks general firewall marking and complex filtering capabilities. When traffic involves cross-node NodePort routing or a Pod connecting back to itself through a Service VIP (Hairpin NAT), failing to rewrite the source IP via SNAT causes the backend Pod to reply directly to the sender, bypassing the return translation path and causing the client to reset the connection due to a source IP mismatch.
Because IPVS cannot handle these edge cases on its own, it still depends on iptables to mark packets in PREROUTING and perform masquerading in POST_ROUTING.
To bridge both requirements, kube-proxy in IPVS mode combines a dummy virtual network interface (kube-ipvs0) with ipset:
- Packet Arrives at Node (Dst: ClusterIP 10.96.0.10:80)
- 1. NF_INET_PRE_ROUTINGiptables matches port in O(1) via ipset; Marks packet with KUBE-MARK-MASQ if needed
- 2. Kernel Routing Decision (FIB Lookup)Sees 10.96.0.10 bound to kube-ipvs0; Treats as local IP, routes to LOCAL_IN
- 3. NF_INET_LOCAL_IN (IPVS hook, before local delivery)Hijacks packet before local user process; O(1) lookup in Service & connection tables; Picks Pod via scheduler (e.g., wlc / rr); Rewrites dst IP:Port, sends to egress path
- 4. NF_INET_POST_ROUTINGApplies SNAT (Masquerade) if marked; Forwards to backend Pod via CNI network
1. Dummy Interface kube-ipvs0 Triggers Local Routing: Unlike iptables mode, which only writes rules without binding IP addresses, kube-proxy in IPVS mode creates a virtual dummy interface named kube-ipvs0 on each node and binds every Service ClusterIP in the cluster to it as a /32 address:
$ ip addr show kube-ipvs0
7: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
inet 10.96.0.1/32 scope global kube-ipvs0
inet 10.96.0.10/32 scope global kube-ipvs0
inet 10.108.45.123/32 scope global kube-ipvs0
NOTE
Why Does kube-ipvs0 Work Even in the DOWN State?
A dummy interface serves purely as a “virtual anchor” to register IP addresses in the kernel’s local FIB routing table; it never transmits or receives physical signals. As long as an IP is assigned to the interface, the kernel route lookup recognizes it as a host-local address and steers matching packets into LOCAL_IN. Seeing state DOWN on kube-ipvs0 is completely normal.
When the kernel routing module consults the FIB table and sees that the ClusterIP belongs to a local interface, it directs the packet into LOCAL_IN. In standard Linux networking, packets entering LOCAL_IN are destined for a local user process (or trigger a TCP RST if no socket is listening).
However, IPVS registers a priority-99 kernel callback on LOCAL_IN—after filter processing and before delivery to a local process—that intercepts the packet mid-flight before it reaches the application layer, completes the hash table lookup and header rewrite, bypasses local delivery, and pushes the packet straight out to NF_INET_POST_ROUTING.
Although IPVS also supports Direct Routing (DR, which rewrites only MAC addresses) and IP Tunneling (TUN), kube-proxy defaults to NAT mode—where both request and reply traffic pass through the node—because Pods are frequently distributed across nodes and subnets.
2. ipset Keeps Rule Filtering at : For the SNAT marking and NodePort filtering that still rely on iptables, kube-proxy brings in Netfilter’s companion extension, ipset, packing thousands of IP addresses, CIDRs, or ports into indexed kernel hash sets. iptables then needs only a single rule to match against the entire set:
iptables -t nat -A PREROUTING -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
Matching against a hash set via --match-set also runs in time, preventing the auxiliary iptables rules from growing linearly with the number of Services.
4. Core Design and Performance Benchmark Comparison: iptables vs. IPVS
Putting the two architectures side by side shows why their performance diverges sharply once a cluster crosses a thousand Services:
| Dimension | iptables Mode | IPVS Mode (with ipset) |
|---|---|---|
| Core Role | General-purpose packet filter and firewall | Dedicated Layer 4 transport-layer load balancer |
| Interception Point | Matches and rewrites directly in PREROUTING / OUTPUT | Triggers local routing via kube-ipvs0, intercepts at LOCAL_IN |
| Lookup Structure & Complexity | Linear scan over contiguous arrays; degrades to | Kernel hash tables and ipset sets; stays flat at |
| Rule Updates & Locking | Full-table copy and write-back; contends on global xtables_lock | Incremental per-entry updates via Netlink API; RCU reads with fine-grained locks |
| Scheduling Algorithms | Simulates randomized traffic splitting via statistic random probabilities | Natively supports 8+ algorithms (rr, wrr, lc, wlc, sh, etc.) |
| Large-Cluster Scaling Threshold | Noticeable latency begins around 1,000 to 5,000 Services | Huawei’s 2016 test maintained ~9 ms response times at 50,000 Services |
According to benchmarks conducted by engineering teams at Huawei and Tigera across varying Service counts, the performance curves fall into three distinct stages:
- Fewer than 1,000 Services: Packet forwarding latency for both modes sits in the microsecond () range, and the gap in CPU overhead is equally negligible; with very few rules,
iptablescan even edge out IPVS slightly. - 10,000 Services (Data Plane Latency): In Kubernetes, each Service generates not only its basic forwarding chain but also auxiliary rules for entry matching,
NodePorthandling, and Masquerade marking—Huawei’s measurements put the average at about 8 rules per Service, or roughly 40,000 rules at 5,000 Services. Forwarding latency climbs from about 575 microseconds with 1 Service to about 1 millisecond at 10,000 Services, approaching 7 milliseconds at 50,000 Services in Huawei’s 2017 tests, while node CPU usage rises noticeably (Tigera measured CPU usage climbing by roughly 35% of a core foriptables, versus about 8% for IPVS). IPVS, by contrast, resolves targets via direct hash hits, keeping its latency curve nearly flat. - 10,000+ Services (Control Plane Sync Time): In Huawei’s 2016 large-scale tests, syncing a new rule on a 20,000-Service cluster took up to 5 hours with
iptables, while adding a single Real Server in IPVS mode via Netlink completed within 100 microseconds; a 2017 follow-up test still needed 11 minutes to add one rule at 5,000 Services (about 40,000 rules). Since Kubernetes 1.28,iptablesmode syncs only the rules that actually changed to ease this cost, but the official documentation still warns that rule updates can take a long time in clusters with tens of thousands of Pods and Services.
Verifying Kernel State and Troubleshooting on the Node
When diagnosing forwarding latency or connection anomalies in a large production cluster, you can inspect the kernel state directly along the paths described above:
# 1. iptables mode: inspect KUBE-SERVICES rules and check if packet hit counters are incrementing
sudo iptables -t nat -nvL KUBE-SERVICES --line-numbers
# 2. IPVS mode: inspect Virtual Server / Real Server mappings and connection stats (-n skips reverse DNS lookup)
sudo ipvsadm -ln --stats
# 3. ipset sets: verify whether NodePort or ClusterIP members have been written into kernel hash sets
sudo ipset list KUBE-NODE-PORT-TCP
# 4. conntrack watermark: check current tracked connections against the system limit to diagnose "table full" drops
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
5. Why IPVS Is Being Deprecated: Passing the Torch to nftables and eBPF
If IPVS solved the linear scan and full-table locking bottlenecks in large clusters, why did the Kubernetes community officially deprecate IPVS mode in v1.35 (targeting default-off in v1.40 and full removal in v1.43)?
The core issue is a semantic mismatch. IPVS is a kernel module designed for traditional L4 load balancing, and its API cannot natively express Kubernetes’s increasingly rich Service semantics (such as externalTrafficPolicy: Local, source IP filtering, and topology-aware routing).
As a result, kube-proxy in IPVS mode was forced to bolt iptables and ipset onto IPVS to handle edge cases. Stitching two subsystems together created heavy maintenance overhead and subtle connection-state bugs. Today, the community is converging on two successor paths:
nftablesmode: Requires Linux kernel 5.13+ and natively provides set and map (verdict map) data structures within a single unified framework. Without mixing two separate subsystems, it achieves near- set lookups and atomic incremental updates while supporting the full range of Kubernetes Service semantics.- eBPF and Cilium: All the approaches above still live inside the Netfilter framework, meaning every packet must traverse the full TCP/IP stack and Netfilter hook points. eBPF-based solutions like Cilium attach directly at the kernel TC (Traffic Control) queueing layer or the NIC driver’s XDP (eXpress Data Path) layer—intercepting packets before they ever reach Netfilter’s
PREROUTINGhook, performing VIP translation through BPF maps, and bypassing the Netfilter stack altogether to deliver near-line-rate throughput.
| Data Plane Approach | Kernel Interception Point | Lookup & Update Mechanism | Architectural Trade-offs & Current Status |
|---|---|---|---|
iptables | Netfilter PREROUTING / OUTPUT | Contiguous array ; full-table rewrite via xtables_lock | Linear latency and slow updates in large clusters; traditional default for small-to-medium clusters |
IPVS + ipset | Netfilter LOCAL_IN + kube-ipvs0 | Kernel hash table ; incremental Netlink updates | Cannot express full Service semantics alone, requiring iptables alongside; deprecated since v1.35 |
nftables | Native Netfilter tables and maps | Verdict map / set near-; atomic incremental updates | Full Service semantics in a single framework; official built-in successor to iptables and IPVS |
eBPF (e.g., Cilium) | TC / NIC driver XDP | BPF map ; direct socket / packet rewrite | Bypasses the entire Netfilter stack, delivering near-line-rate throughput |
From iptables to IPVS, and now to nftables and eBPF, the underlying trajectory has remained the same: drive lookup complexity down from to , replace full-table rewrites with incremental updates, and shorten the path every packet takes through the kernel.