Kubernetes Controllers Explained: How Does the Reconcile Loop Work?
In day-to-day backend architecture and microservice operations, you are probably used to writing replicas: 3 in a Kubernetes manifest and watching the system automatically recreate Pods whenever one is manually deleted. Where does this “self-healing” capability actually come from?
Deep inside the control plane, kube-controller-manager (KCM) quietly bundles dozens of core controllers into a single process. This article is written for engineers with backend development experience who are getting started with DevOps and Kubernetes, and assumes no deep familiarity with Go.
We will break down four key questions in order: What problem do controllers solve? Where does KCM sit in the control plane? Which five internal stages does an event pass through from creation to state convergence? And finally—what is right and what is wrong about the common claim that “a controller is essentially a goroutine”?
1. What Problem Controllers Solve: Declarative APIs and the Reconcile Loop
To understand controllers, we first need to look at the fundamental divide between imperative thinking and declarative architecture.
Traditional operations scripts are mostly imperative: you write step-by-step instructions telling the system to “create three VMs, install the application, and attach a load balancer.” Once the script finishes, it exits. In production, however, state drift happens constantly—nodes crash, networks flap, someone tweaks a configuration by hand, or a disk fills up. A script that has already exited cannot detect these changes, let alone recover from them automatically.
Kubernetes takes a declarative approach: instead of describing execution steps, you declare the final state you want. When you submit a YAML manifest to the API Server, that desired state is persisted in etcd (the distributed key-value store Kubernetes uses to hold all cluster state) and recorded in the API object’s spec field.
The job of continuously pulling the real world back toward that desired state belongs to controllers. Conceptually, a controller is a never-ending loop:
for {
desired := readDesiredState()
// spec: user declares "3 replicas needed"
actual := observeActualState()
// current running state in the cluster
diff := calculateDiff(desired, actual)
if diff is not empty {
take action to move actual toward desired
// this step is called Reconcile
}
wait for next round (watch event or periodic resync)
}
This operating model is called the reconcile loop (or control loop). The name Kubernetes comes from the Greek word for “helmsman” (κυβερνήτης), sharing the same root as Norbert Wiener’s classic work Cybernetics, and the underlying idea is the same: set a target value, and let the system eliminate deviation through continuous feedback and adjustment.
+---------------------------------------------------------+
| Reconcile Loop |
| |
| +-------------+ Read Spec +---------------+ |
| | Desired |<--------------------| API Server | |
| | State | | (etcd) | |
| +-------------+ +---------------+ |
| | ^ |
| v Compare | Update |
| +-------------+ Drive Action | Status |
| | Calculate |-----------------------------+ |
| | Difference | |
| +-------------+ |
| ^ |
| | Observe Actual |
| +-------------+ |
| | Actual | |
| | State | |
| +-------------+ |
+---------------------------------------------------------+
Level-Triggered Logic: Current State as the Source of Truth
For engineers with networking or systems programming backgrounds, there is an important distinction here: level-triggered vs. edge-triggered logic.
In event-driven architectures:
- Edge-triggered systems focus on deltas (changes): the system reacts to each individual event notification. If it misses an event, it may permanently lose the chance to resynchronize.
- Level-triggered systems focus on current state: every decision is based on the full state as it exists right now, while event notifications serve only as wake-up signals to run reconciliation.
Kubernetes controllers follow level-triggered design principles. In a distributed environment, network connections drop, controllers restart, and operators intervene manually. Tying reconciliation logic to individual events would make the system brittle.
Instead, controllers achieve reliability by combining two mechanisms:
- Real-time change notifications (Watch): Long-lived HTTP connections stream resource changes in real time, acting as an alarm bell to trigger reconciliation.
- Periodic resynchronization (Resync): At fixed intervals, the controller walks through all objects cached in local memory and actively triggers re-evaluation.
Even if an event notification is lost due to a network blip, the next reconciliation pass still compares the latest state and catches up. For example, when you manually delete a managed Pod, the underlying controller does not recreate it because it “received a delete event”—it recreates the Pod because, during reconciliation, it sees that 3 replicas are desired while only 2 exist. The state discrepancy itself drives the action.
Idempotency and Eventual Consistency
A reconcile loop relies on two core engineering properties:
- Idempotency: Running the same reconciliation logic once or ten times produces the exact same end state. Even if a controller crashes and restarts mid-flight, or duplicate evaluations run concurrently, cluster state is never corrupted.
- Eventual consistency: Going from a submitted command to a Pod that is scheduled, has pulled its image, and passes health checks takes time. Controllers do not guarantee instant readiness; they guarantee that the system continuously converges toward the target state.
This architecture is reflected directly in the separation of spec and status. Across most Kubernetes API resource objects:
specrecords the desired state declared by the user (for example,spec.replicas: 3).statusrecords the actual state observed and written back by the controller (for example,status.readyReplicas: 3).
Because both live centrally in the API Server, the gap between desired and actual state is visible right on the object for anyone to inspect, and the controller itself does not need to maintain any persistent state of its own.
NOTE
Further reading: An Introduction to etcd: The Memory of a Kubernetes Cluster
2. Where kube-controller-manager Fits: Who Runs Which Controllers
Inside the Kubernetes control plane, each core component has a clear division of responsibilities:
| Component | Runtime Form | Core Responsibility |
|---|---|---|
| kube-apiserver | Daemon service | The sole entry point that reads/writes etcd directly; the communication hub for all components |
| etcd | Distributed store | The single source of truth storing all cluster state |
| kube-scheduler | Standalone binary | Dedicated to placement decisions: selecting the best node for unscheduled Pods |
| kube-controller-manager | Standalone binary | Bundles dozens of built-in core controllers into a single OS process |
| cloud-controller-manager | Standalone binary | Runs controllers that interact with cloud provider APIs |
+--------------------------------------------------------+
| Kubernetes Control Plane |
| |
| +---------------------+ +---------------------+ |
| | kube-scheduler | | KCM / CCM | |
| | (Placement) | | (Reconcile Loops) | |
| +---------------------+ +---------------------+ |
| | | |
| | Watch / Update | |
| v v |
| +------------------------------------------------+ |
| | kube-apiserver | |
| +------------------------------------------------+ |
| | |
| v |
| +------------------------------------------------+ |
| | etcd | |
| +------------------------------------------------+ |
+--------------------------------------------------------+
Dozens of Controllers in One Process: The Engineering Trade-Off
KCM compiles and packages dozens of controllers—including Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, Node Lifecycle, Pod GC, Namespace, ServiceAccount, and ResourceQuota—into a single binary and a single OS process.
At the source code level, KCM manages the full catalog of controllers through a central registry (NewControllerDescriptors). On startup, KCM iterates over this registry, instantiates each controller, and launches its Run reconciliation method.
Packing dozens of controllers into one process is a deliberate architectural trade-off:
- Pros: deployment and operations are vastly simpler, and multiple controllers can share Informer local caches and long-lived Watch connections (an Informer is a shared component that caches resource state in local memory, covered in detail in Section 3), drastically reducing connection and query load on the API Server.
- Cons: Fault isolation is weaker. If a fatal bug in one controller crashes the process, all other controllers inside KCM go down with it, relying on an external process supervisor and Leader Election to recover and reconverge.
To prevent over-privileged access, KCM supports the --use-service-account-credentials flag, which runs each internal controller with its own dedicated ServiceAccount token when calling the API Server, enforcing the principle of least privilege.
NOTE
Further reading: How K3s Packs the Kubernetes Control Plane into a Single Process
Why Aren’t Other Components Inside KCM?
- kube-scheduler runs separately: The scheduler only makes placement decisions (scoring nodes and binding Pods to them); it does not create objects, scale replica counts, or heal workloads. Separating the placement engine from reconciliation loops keeps responsibility boundaries clean.
- cloud-controller-manager (CCM) is split out: Logic for load balancers (
LoadBalancer), cloud block storage (Volume), and node IP management depends heavily on vendor-specific cloud APIs. Extracting CCM into a standalone binary lets cloud providers release and maintain their integrations independently, keeping the core Kubernetes repository lean and decoupling cloud credentials from the core codebase.
High Availability (HA) and Leader Election
When a control plane is deployed across multiple nodes for high availability (HA), having KCM instances on multiple nodes reconcile the exact same resources simultaneously would cause severe race conditions and conflicting state updates.
Multi-instance control planes therefore run KCM in active-standby mode, using leader election (--leader-elect=true) to ensure that only a single active instance runs reconciliation loops at any given time.
Node 1 (Leader) Node 2, 3 (Standby)
+------------------+ +------------------+
| KCM Process | | KCM Process |
| [Active Workers] | | [Standby Mode] |
+------------------+ +------------------+
| |
| Heartbeat Renew | Watch Lease
v v
+-------------------------------------------+
| Lease: kube-controller-manager |
| Namespace: kube-system |
+-------------------------------------------+
All KCM instances compete for a Lease resource object under the coordination.k8s.io API group in the kube-system namespace:
- Leader: The instance that acquires and holds the Lease becomes the leader, starts all controller reconciliation loops, and periodically renews its lease via heartbeats.
- Standby: Instances that do not hold the Lease stay in hot-standby mode—the process stays up and watches the Lease, but does not start any reconciliation workers. If the leader’s lease expires, a standby instance takes over.
3. Inside a Single Controller: Tracing the Deployment Controller Pipeline
With the control-plane picture in place, let’s zoom in on the internal mechanics of a single controller, using the most common Deployment Controller as our walkthrough example.
Mental Model First: Layered Convergence
The key to understanding Kubernetes workload controllers is layered convergence. When you run kubectl get deploy,rs,pod in your terminal, you are looking at three distinct layers working in concert:
- Deployment ControllerManages ReplicaSet revisions and counts
- ReplicaSet ControllerEnsures Pod replica count matches desired value
- Kubelet (Node Agent)Starts and maintains containers on local node
Note one crucial detail: the Deployment Controller never creates or deletes Pods directly. It only manages underlying ReplicaSets; the ReplicaSet Controller maintains the desired number of Pod replicas; and finally, the kubelet on each node runs the containers on that host.
Each controller layer is responsible only for the state of the objects directly beneath it. There are no direct RPC calls between layers—everything is decoupled through object declarations on the API Server. This layered design is what enables Kubernetes to perform zero-downtime rolling updates smoothly.
The Five-Stage Internal Pipeline
Inside a controller, the processing pipeline consists of five stages:
Deployment Controller Pipeline:
[ API Server ]
|
| LIST / WATCH
v
[ Reflector ]
|
| handleDeltas
v
[ Delta Queue (RealFIFO) ]
|
v
[ SharedInformer ]
|
+--> Update Memory Cache --> [ Local Indexer Cache ]
|
+--> OnAdd / Update --> [ ResourceEventHandler ]
|
| Extract Key (ns/name)
v
[ WorkQueue (RateLimit) ]
|
| Pop Key
v
[ Worker Pool (1..N) ]
|
| syncDeployment
| Read Cache (Lister)
v
[ Local Indexer Cache ]
Stage 1: Reflector (LIST + WATCH)
A controller needs an accurate view of the cluster’s current state. Living in the official Kubernetes Go client library (client-go), Reflector runs on its own dedicated goroutine (Go’s lightweight execution unit, which we will examine in Section 4):
- It fetches a full initial snapshot of the target resource. Recent client-go versions (v1.35+) fetch it by default through a streaming
WATCHwith initial events (watch-list), withLISTkept as the fallback when that is not supported. - It then keeps a long-lived HTTP
WATCHconnection open to stream subsequent additions, updates, and deletions. - If the connection drops, Reflector automatically reconnects with exponential backoff, fetching a fresh full snapshot when necessary so no updates are missed.
Every incoming change event is wrapped in a Delta record tagged with its action type (Added, Updated, or Deleted) and pushed into a Delta queue; client-go (since v1.33) defaults to RealFIFO, which delivers every event without per-key folding, while DeltaFIFO—the earlier implementation that merges events for the same key—predates v1.33.
Stage 2: SharedInformer (Consuming Deltas, Updating Cache, and Dispatching Events)
Driven by another dedicated goroutine, SharedInformer continuously pops Delta records from the Delta queue:
- Updates the local cache (Indexer): Writes the latest object state into an in-memory cache. All subsequent reconciliation reads query this local cache directly instead of making network calls to the API Server.
- Dispatches events: Forwards the event to every ResourceEventHandler registered with that Informer.
- Runs periodic resyncs: At a configured interval, notifies handlers to rescan objects in the cache—acting as a safety net for level-triggered reconciliation so that even if an event was dropped, state is re-verified on schedule.
As the name suggests, the “Shared” in SharedInformer means that a single Watch connection and in-memory cache for a given resource type (such as Pods) are shared across all controllers inside KCM, preventing an explosion of duplicate watches.
Stage 3: Event Handler (Owner Lookup and Key Extraction)
During initialization (NewDeploymentController), the Deployment Controller registers event handlers with the Informers for Deployments, ReplicaSets, and Pods.
These event handlers do not run reconciliation business logic. Their only job is extracting keys and resolving ownership:
- When a Deployment itself changes, the handler extracts its key in
namespace/nameformat (for example,default/my-nginx). - When a child resource (such as a ReplicaSet or Pod) changes, the handler inspects
metadata.ownerReferencesto trace ownership back up to the parent Deployment, then pushes the parent Deployment’s key into the WorkQueue.
Effectively, the handler leaves a note saying: “Something under this Deployment changed—recompute it when a worker is free.”
Stage 4: WorkQueue (Deduplication, Rate Limiting, and Avalanche Protection)
The client-go WorkQueue is far more than a plain FIFO queue; it has built-in safeguards for production reliability:
- Deduplication: Internally, the queue maintains a
dirtyset. If a key is already waiting in the queue, repeated additions for the same key are coalesced into one entry. Even if a Deployment is updated ten times in a single second, the queue holds only one pending task for it. - Rate limiting and backoff: Failed keys are re-queued with exponential backoff to stretch out retry intervals dynamically, paired with an overall token-bucket rate limiter to cap burst throughput—preventing a handful of broken objects from triggering a retry storm that overwhelms the control plane.
NOTE
Further reading: A Guided Tour of Google’s Site Reliability Engineering (Part 2): Overload Spirals, Case Studies, and Ten Years Later
Stage 5: Worker Pool (Executing Reconcile)
When the controller starts, its Run method first calls WaitForNamedCacheSync to block until all required Informers have completed their initial cache sync. Once the caches are warm, it spawns a pool of worker goroutines (5 by default for the Deployment Controller).
Each worker runs processNextWorkItem in a loop:
- Pops a key from the WorkQueue.
- Calls
syncHandler(syncDeployment) to execute the core reconciliation logic. - Reads the Deployment’s desired state through the local cache interface (Lister) and locates all ReplicaSets owned by that Deployment via Owner References.
- Compares replica counts across old and new ReplicaSets, scales ReplicaSets up or down according to the rolling-update strategy, and writes the updated status back to the API Server.
- If reconciliation fails, puts the key back into the WorkQueue for a rate-limited retry (up to the retry limit) to be processed again on a later pass.
4. “A Controller = A Goroutine”? Correcting the Mental Model
A common shorthand in the community goes: “KCM manages all controllers, and a controller is basically just a Go goroutine.” How accurate is that statement?
The Half That Is True: In-Process Startup
The statement gets one thing right: KCM compiles its built-in controllers into a single binary, and during KCM startup, each controller’s Run method is indeed launched on its own goroutine (go func() { ... controller.Run(ctx) }()).
You can think of a Go goroutine as a lightweight user-space thread: it starts with a tiny stack of roughly 2 KB, and the Go runtime schedules thousands of goroutines onto a small pool of OS kernel threads, with all scheduling handled in user space. That makes it cheap for a single process to sustain tens of thousands of concurrent loops—an ideal execution environment for controllers.
The Half That Misses the Mark: Cardinality and Essence
Still, equating a controller with a single goroutine is misleading on two levels:
1. Underestimating the Goroutine Count
A single controller is never just one goroutine:
- Across KCM, dozens of SharedInformers run continuously, and each Informer spins up its own goroutines for Reflector watch loops and Delta-queue consumption.
- Inside each controller,
Runspawns a dedicated worker pool (for instance, both the Deployment and ReplicaSet controllers launch 5 worker goroutines by default).
When you add up the dozens of controllers inside KCM along with their backing Informers and worker pools, an active KCM leader process typically runs hundreds of cooperating goroutines at the same time.
2. Confusing the Execution Vehicle with the Architectural Pattern
More importantly, a controller is a software architectural pattern, not a language-specific concurrency primitive.
Looked at from outside Go, a controller consists of four building blocks: Controller = Informer (observe the world) + Local Cache (in-memory store) + WorkQueue (deduplication & rate limiting) + Reconcile Function (idempotent convergence)
This pattern is completely language-agnostic:
- In the Rust ecosystem, kube-rs implements the exact same reconciliation pipeline.
- In the Java ecosystem,
java-operator-sdkfollows the same design. - In the Go operator ecosystem, controller-runtime (the foundation of Kubebuilder) abstracts this entire pipeline behind a clean
Reconcilerinterface.
Goroutines are simply the execution vehicle Go uses to implement this pattern; the reconcile loop and state convergence are the controller’s architectural soul.
+--------------------------------------------------+
| Architecture Hierarchy |
| |
| 1. OS Process : kube-controller-manager |
| (Monolithic Binary) |
| | |
| 2. Architectural Pattern : Controllers |
| (Informer + WorkQueue + Reconcile) |
| | |
| 3. Execution Vehicle : goroutines |
| (Reflectors, Informers, Workers) |
+--------------------------------------------------+
Conclusion
We can sum up the core takeaways of this article in three points:
- At the heart of every controller is the reconcile loop: Using level-triggered logic and idempotent reconciliation, controllers compare
specagainststatusand eliminate state drift through eventual consistency. - KCM is the consolidated home for built-in controllers: It packages dozens of controllers into a single OS process to share caches and Watch connections, using a Lease lock to elect a single active leader.
- Goroutines are the vehicle; the reconcile loop is the essence: A single controller is powered by multiple cooperating goroutines (Reflector, Informer, WorkQueue, and Worker Pool). A controller is an architectural pattern, not just a thread.
If you want to inspect these mechanisms firsthand on a live cluster, try running the following commands:
# Check which instance holds the KCM leader Lease
kubectl -n kube-system get lease kube-controller-manager -o yaml
# Observe the three-tier object hierarchy (Deployment -> RS -> Pod)
kubectl get deploy,rs,pod -l app=myapp
# Query KCM's exported WorkQueue metrics (queue depth, latency)
kubectl get --raw /metrics | grep workqueue
To go deeper into implementation, check out the official sample-controller repository or build a Custom Resource Definition (CRD) controller with Kubebuilder to design your own Reconcile() function and get a feel for where its boundaries lie. You can also read pkg/controller/deployment/deployment_controller.go directly in the Kubernetes source tree to see how each stage collaborates in production code.
NOTE
Further reading: From a Vanished Blog Post to a CNCF Standard: The Origins and Reality of OpenGitOps
References
- kubernetes/kubernetes - cmd/kube-controller-manager/app/: Source code for controller descriptor registration, goroutine startup, and leader election.
- kubernetes/kubernetes - pkg/controller/deployment/: Deployment Controller implementation covering
Run, the worker pool, andsyncDeployment. - k8s.io/client-go: Core client library housing Reflector (
tools/cache/reflector.go), SharedInformer (tools/cache/shared_informer.go), and WorkQueue (util/workqueue/). - kube-controller-manager CLI Reference: Official command-line flag reference and default values.
- Kubernetes Leases Concept: Documentation on Lease objects and control-plane HA leader election.
- Effective Go - Goroutines and Go Runtime Package: Concurrency semantics and M:N scheduler documentation in Go.
- sample-controller: The official minimal Kubernetes controller reference implementation.