Kubernetes Controllers Explained: How Does the Reconcile Loop Work?

In day-to-day backend architecture and microservice operations, you are probably used to writing replicas: 3 in a Kubernetes manifest and watching the system automatically recreate Pods whenever one is manually deleted. Where does this “self-healing” capability actually come from?

Deep inside the control plane, kube-controller-manager (KCM) quietly bundles dozens of core controllers into a single process. This article is written for engineers with backend development experience who are getting started with DevOps and Kubernetes, and assumes no deep familiarity with Go.

We will break down four key questions in order: What problem do controllers solve? Where does KCM sit in the control plane? Which five internal stages does an event pass through from creation to state convergence? And finally—what is right and what is wrong about the common claim that “a controller is essentially a goroutine”?


1. What Problem Controllers Solve: Declarative APIs and the Reconcile Loop

To understand controllers, we first need to look at the fundamental divide between imperative thinking and declarative architecture.

Traditional operations scripts are mostly imperative: you write step-by-step instructions telling the system to “create three VMs, install the application, and attach a load balancer.” Once the script finishes, it exits. In production, however, state drift happens constantly—nodes crash, networks flap, someone tweaks a configuration by hand, or a disk fills up. A script that has already exited cannot detect these changes, let alone recover from them automatically.

Kubernetes takes a declarative approach: instead of describing execution steps, you declare the final state you want. When you submit a YAML manifest to the API Server, that desired state is persisted in etcd (the distributed key-value store Kubernetes uses to hold all cluster state) and recorded in the API object’s spec field.

The job of continuously pulling the real world back toward that desired state belongs to controllers. Conceptually, a controller is a never-ending loop:

 for {
   desired := readDesiredState()
   // spec: user declares "3 replicas needed"
   actual  := observeActualState()
   // current running state in the cluster
   diff    := calculateDiff(desired, actual)
   if diff is not empty {
     take action to move actual toward desired
     // this step is called Reconcile
   }
   wait for next round (watch event or periodic resync)
 }

This operating model is called the reconcile loop (or control loop). The name Kubernetes comes from the Greek word for “helmsman” (κυβερνήτης), sharing the same root as Norbert Wiener’s classic work Cybernetics, and the underlying idea is the same: set a target value, and let the system eliminate deviation through continuous feedback and adjustment.

 +---------------------------------------------------------+
 | Reconcile Loop                                          |
 |                                                         |
 | +-------------+      Read Spec      +---------------+   |
 | | Desired     |<--------------------| API Server    |   |
 | | State       |                     | (etcd)        |   |
 | +-------------+                     +---------------+   |
 |        |                                    ^           |
 |        v Compare                            | Update    |
 | +-------------+     Drive Action            | Status    |
 | | Calculate   |-----------------------------+           |
 | | Difference  |                                         |
 | +-------------+                                         |
 |        ^                                                |
 |        | Observe Actual                                 |
 | +-------------+                                         |
 | | Actual      |                                         |
 | | State       |                                         |
 | +-------------+                                         |
 +---------------------------------------------------------+

Level-Triggered Logic: Current State as the Source of Truth

For engineers with networking or systems programming backgrounds, there is an important distinction here: level-triggered vs. edge-triggered logic.

In event-driven architectures:

Kubernetes controllers follow level-triggered design principles. In a distributed environment, network connections drop, controllers restart, and operators intervene manually. Tying reconciliation logic to individual events would make the system brittle.

Instead, controllers achieve reliability by combining two mechanisms:

  1. Real-time change notifications (Watch): Long-lived HTTP connections stream resource changes in real time, acting as an alarm bell to trigger reconciliation.
  2. Periodic resynchronization (Resync): At fixed intervals, the controller walks through all objects cached in local memory and actively triggers re-evaluation.

Even if an event notification is lost due to a network blip, the next reconciliation pass still compares the latest state and catches up. For example, when you manually delete a managed Pod, the underlying controller does not recreate it because it “received a delete event”—it recreates the Pod because, during reconciliation, it sees that 3 replicas are desired while only 2 exist. The state discrepancy itself drives the action.

Idempotency and Eventual Consistency

A reconcile loop relies on two core engineering properties:

This architecture is reflected directly in the separation of spec and status. Across most Kubernetes API resource objects:

Because both live centrally in the API Server, the gap between desired and actual state is visible right on the object for anyone to inspect, and the controller itself does not need to maintain any persistent state of its own.


2. Where kube-controller-manager Fits: Who Runs Which Controllers

Inside the Kubernetes control plane, each core component has a clear division of responsibilities:

ComponentRuntime FormCore Responsibility
kube-apiserverDaemon serviceThe sole entry point that reads/writes etcd directly; the communication hub for all components
etcdDistributed storeThe single source of truth storing all cluster state
kube-schedulerStandalone binaryDedicated to placement decisions: selecting the best node for unscheduled Pods
kube-controller-managerStandalone binaryBundles dozens of built-in core controllers into a single OS process
cloud-controller-managerStandalone binaryRuns controllers that interact with cloud provider APIs
 +--------------------------------------------------------+
 | Kubernetes Control Plane                               |
 |                                                        |
 |  +---------------------+    +---------------------+    |
 |  | kube-scheduler      |    | KCM / CCM           |    |
 |  | (Placement)         |    | (Reconcile Loops)   |    |
 |  +---------------------+    +---------------------+    |
 |             |                          |               |
 |             |      Watch / Update      |               |
 |             v                          v               |
 |  +------------------------------------------------+    |
 |  |                 kube-apiserver                 |    |
 |  +------------------------------------------------+    |
 |                          |                             |
 |                          v                             |
 |  +------------------------------------------------+    |
 |  |                      etcd                      |    |
 |  +------------------------------------------------+    |
 +--------------------------------------------------------+

Dozens of Controllers in One Process: The Engineering Trade-Off

KCM compiles and packages dozens of controllers—including Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, Node Lifecycle, Pod GC, Namespace, ServiceAccount, and ResourceQuota—into a single binary and a single OS process.

At the source code level, KCM manages the full catalog of controllers through a central registry (NewControllerDescriptors). On startup, KCM iterates over this registry, instantiates each controller, and launches its Run reconciliation method.

Packing dozens of controllers into one process is a deliberate architectural trade-off:

To prevent over-privileged access, KCM supports the --use-service-account-credentials flag, which runs each internal controller with its own dedicated ServiceAccount token when calling the API Server, enforcing the principle of least privilege.

Why Aren’t Other Components Inside KCM?

High Availability (HA) and Leader Election

When a control plane is deployed across multiple nodes for high availability (HA), having KCM instances on multiple nodes reconcile the exact same resources simultaneously would cause severe race conditions and conflicting state updates.

Multi-instance control planes therefore run KCM in active-standby mode, using leader election (--leader-elect=true) to ensure that only a single active instance runs reconciliation loops at any given time.

 Node 1 (Leader)          Node 2, 3 (Standby)
 +------------------+     +------------------+
 | KCM Process      |     | KCM Process      |
 | [Active Workers] |     | [Standby Mode]   |
 +------------------+     +------------------+
          |                        |
          | Heartbeat Renew        | Watch Lease
          v                        v
 +-------------------------------------------+
 | Lease: kube-controller-manager            |
 | Namespace: kube-system                    |
 +-------------------------------------------+

All KCM instances compete for a Lease resource object under the coordination.k8s.io API group in the kube-system namespace:


3. Inside a Single Controller: Tracing the Deployment Controller Pipeline

With the control-plane picture in place, let’s zoom in on the internal mechanics of a single controller, using the most common Deployment Controller as our walkthrough example.

Mental Model First: Layered Convergence

The key to understanding Kubernetes workload controllers is layered convergence. When you run kubectl get deploy,rs,pod in your terminal, you are looking at three distinct layers working in concert:

  1. Deployment ControllerManages ReplicaSet revisions and counts
  2. ReplicaSet ControllerEnsures Pod replica count matches desired value
  3. Kubelet (Node Agent)Starts and maintains containers on local node

Note one crucial detail: the Deployment Controller never creates or deletes Pods directly. It only manages underlying ReplicaSets; the ReplicaSet Controller maintains the desired number of Pod replicas; and finally, the kubelet on each node runs the containers on that host.

Each controller layer is responsible only for the state of the objects directly beneath it. There are no direct RPC calls between layers—everything is decoupled through object declarations on the API Server. This layered design is what enables Kubernetes to perform zero-downtime rolling updates smoothly.

The Five-Stage Internal Pipeline

Inside a controller, the processing pipeline consists of five stages:

 Deployment Controller Pipeline:
 [ API Server ]
   |
   | LIST / WATCH
   v
 [ Reflector ]
   |
   | handleDeltas
   v
 [ Delta Queue (RealFIFO) ]
   |
   v
 [ SharedInformer ]
   |
   +--> Update Memory Cache --> [ Local Indexer Cache ]
   |
   +--> OnAdd / Update --> [ ResourceEventHandler ]
                             |
                             | Extract Key (ns/name)
                             v
                           [ WorkQueue (RateLimit) ]
                             |
                             | Pop Key
                             v
                           [ Worker Pool (1..N) ]
                             |
                             | syncDeployment
                             | Read Cache (Lister)
                             v
                           [ Local Indexer Cache ]

Stage 1: Reflector (LIST + WATCH)

A controller needs an accurate view of the cluster’s current state. Living in the official Kubernetes Go client library (client-go), Reflector runs on its own dedicated goroutine (Go’s lightweight execution unit, which we will examine in Section 4):

  1. It fetches a full initial snapshot of the target resource. Recent client-go versions (v1.35+) fetch it by default through a streaming WATCH with initial events (watch-list), with LIST kept as the fallback when that is not supported.
  2. It then keeps a long-lived HTTP WATCH connection open to stream subsequent additions, updates, and deletions.
  3. If the connection drops, Reflector automatically reconnects with exponential backoff, fetching a fresh full snapshot when necessary so no updates are missed.

Every incoming change event is wrapped in a Delta record tagged with its action type (Added, Updated, or Deleted) and pushed into a Delta queue; client-go (since v1.33) defaults to RealFIFO, which delivers every event without per-key folding, while DeltaFIFO—the earlier implementation that merges events for the same key—predates v1.33.

Stage 2: SharedInformer (Consuming Deltas, Updating Cache, and Dispatching Events)

Driven by another dedicated goroutine, SharedInformer continuously pops Delta records from the Delta queue:

  1. Updates the local cache (Indexer): Writes the latest object state into an in-memory cache. All subsequent reconciliation reads query this local cache directly instead of making network calls to the API Server.
  2. Dispatches events: Forwards the event to every ResourceEventHandler registered with that Informer.
  3. Runs periodic resyncs: At a configured interval, notifies handlers to rescan objects in the cache—acting as a safety net for level-triggered reconciliation so that even if an event was dropped, state is re-verified on schedule.

As the name suggests, the “Shared” in SharedInformer means that a single Watch connection and in-memory cache for a given resource type (such as Pods) are shared across all controllers inside KCM, preventing an explosion of duplicate watches.

Stage 3: Event Handler (Owner Lookup and Key Extraction)

During initialization (NewDeploymentController), the Deployment Controller registers event handlers with the Informers for Deployments, ReplicaSets, and Pods.

These event handlers do not run reconciliation business logic. Their only job is extracting keys and resolving ownership:

Effectively, the handler leaves a note saying: “Something under this Deployment changed—recompute it when a worker is free.”

Stage 4: WorkQueue (Deduplication, Rate Limiting, and Avalanche Protection)

The client-go WorkQueue is far more than a plain FIFO queue; it has built-in safeguards for production reliability:

Stage 5: Worker Pool (Executing Reconcile)

When the controller starts, its Run method first calls WaitForNamedCacheSync to block until all required Informers have completed their initial cache sync. Once the caches are warm, it spawns a pool of worker goroutines (5 by default for the Deployment Controller).

Each worker runs processNextWorkItem in a loop:

  1. Pops a key from the WorkQueue.
  2. Calls syncHandler (syncDeployment) to execute the core reconciliation logic.
  3. Reads the Deployment’s desired state through the local cache interface (Lister) and locates all ReplicaSets owned by that Deployment via Owner References.
  4. Compares replica counts across old and new ReplicaSets, scales ReplicaSets up or down according to the rolling-update strategy, and writes the updated status back to the API Server.
  5. If reconciliation fails, puts the key back into the WorkQueue for a rate-limited retry (up to the retry limit) to be processed again on a later pass.

4. “A Controller = A Goroutine”? Correcting the Mental Model

A common shorthand in the community goes: “KCM manages all controllers, and a controller is basically just a Go goroutine.” How accurate is that statement?

The Half That Is True: In-Process Startup

The statement gets one thing right: KCM compiles its built-in controllers into a single binary, and during KCM startup, each controller’s Run method is indeed launched on its own goroutine (go func() { ... controller.Run(ctx) }()).

You can think of a Go goroutine as a lightweight user-space thread: it starts with a tiny stack of roughly 2 KB, and the Go runtime schedules thousands of goroutines onto a small pool of OS kernel threads, with all scheduling handled in user space. That makes it cheap for a single process to sustain tens of thousands of concurrent loops—an ideal execution environment for controllers.

The Half That Misses the Mark: Cardinality and Essence

Still, equating a controller with a single goroutine is misleading on two levels:

1. Underestimating the Goroutine Count

A single controller is never just one goroutine:

When you add up the dozens of controllers inside KCM along with their backing Informers and worker pools, an active KCM leader process typically runs hundreds of cooperating goroutines at the same time.

2. Confusing the Execution Vehicle with the Architectural Pattern

More importantly, a controller is a software architectural pattern, not a language-specific concurrency primitive.

Looked at from outside Go, a controller consists of four building blocks: Controller = Informer (observe the world) + Local Cache (in-memory store) + WorkQueue (deduplication & rate limiting) + Reconcile Function (idempotent convergence)

This pattern is completely language-agnostic:

Goroutines are simply the execution vehicle Go uses to implement this pattern; the reconcile loop and state convergence are the controller’s architectural soul.

 +--------------------------------------------------+
 | Architecture Hierarchy                           |
 |                                                  |
 | 1. OS Process : kube-controller-manager          |
 |    (Monolithic Binary)                           |
 |         |                                        |
 | 2. Architectural Pattern : Controllers           |
 |    (Informer + WorkQueue + Reconcile)            |
 |         |                                        |
 | 3. Execution Vehicle : goroutines                |
 |    (Reflectors, Informers, Workers)              |
 +--------------------------------------------------+

Conclusion

We can sum up the core takeaways of this article in three points:

  1. At the heart of every controller is the reconcile loop: Using level-triggered logic and idempotent reconciliation, controllers compare spec against status and eliminate state drift through eventual consistency.
  2. KCM is the consolidated home for built-in controllers: It packages dozens of controllers into a single OS process to share caches and Watch connections, using a Lease lock to elect a single active leader.
  3. Goroutines are the vehicle; the reconcile loop is the essence: A single controller is powered by multiple cooperating goroutines (Reflector, Informer, WorkQueue, and Worker Pool). A controller is an architectural pattern, not just a thread.

If you want to inspect these mechanisms firsthand on a live cluster, try running the following commands:

# Check which instance holds the KCM leader Lease
kubectl -n kube-system get lease kube-controller-manager -o yaml

# Observe the three-tier object hierarchy (Deployment -> RS -> Pod)
kubectl get deploy,rs,pod -l app=myapp

# Query KCM's exported WorkQueue metrics (queue depth, latency)
kubectl get --raw /metrics | grep workqueue

To go deeper into implementation, check out the official sample-controller repository or build a Custom Resource Definition (CRD) controller with Kubebuilder to design your own Reconcile() function and get a feel for where its boundaries lie. You can also read pkg/controller/deployment/deployment_controller.go directly in the Kubernetes source tree to see how each stage collaborates in production code.


References