A Beginner's Guide to Container Runtimes (Part 2): Operational Practice
Part 1: Core Mechanics established three foundational concepts: a container is an ordinary Linux process constrained by namespaces and cgroups, the runtime stack is split into high-level (containerd) and low-level (runc) tiers, and Kubernetes communicates with the runtime through the CRI.
If Part 1 answered “What is a container runtime?”, this operational guide focuses on the core day-to-day question: “How do you debug it when things break, and how do you configure it properly in the first place?”
This article walks through six practical scenarios cloud-native engineers encounter most often in production:
- cgroup v2 and cgroup Driver Unification: Why does Kubernetes 1.35 refuse to start on cgroup v1 by default, and why do mismatched cgroup drivers destabilize nodes?
- How K8s Resource Limits Map to the Linux Kernel: How do Pod
limitsbecomememory.maxandcpu.max? What really happens under the hood duringOOMKilledand CPU throttling? - Hands-On Node Troubleshooting: When the
dockerCLI isn’t on the node, how do you pinpointImagePullBackOffand crash loops usingcrictlandjournalctl? - Essential containerd Configuration: Navigating the configuration breaking changes between containerd 1.x and 2.x for
SystemdCgroup, private registries, and thepausesandbox image. - runc Escape Vulnerabilities and RuntimeClass Sandboxing: Security lessons from CVE-2019-5736 to Leaky Vessels, plus how to isolate workloads with gVisor and Kata Containers.
- How Containers Really Work on macOS: Why do containers on a Mac always run inside a Linux VM, and how does that explain slow bind mounts and cross-architecture emulation overhead?
cgroup v2 and cgroup Driver Unification
cgroup v2 Is Now a Hard Requirement in Modern Kubernetes
cgroups (Control Groups) enforce resource quotas on processes. Over the past few years, most mainstream Linux distributions have switched their default from v1 to cgroup v2:
| Aspect | cgroup v1 | cgroup v2 |
|---|---|---|
| Hierarchy | Separate, independent tree for each resource controller (CPU, memory, etc.) | A single unified hierarchy shared across all resource controllers |
| Memory Accounting | Tracked separately per subsystem; misses kernel memory, network buffers, etc. | Unified accounting that properly tracks delayed costs like page cache writeback |
| Pressure Metrics | No native unified metric | Built-in PSI (Pressure Stall Information—the share of time workloads stall waiting for CPU, memory, or I/O) to pinpoint resource bottlenecks |
| Kubernetes Status | Maintenance mode in 1.31; officially deprecated starting in 1.35 | Generally Available (GA) since 1.25 |
The critical milestone: starting with Kubernetes 1.35, kubelet refuses to start on cgroup v1 nodes by default.
While you can temporarily bypass this check by setting failCgroupV1 to false in the kubelet configuration, this is only a stopgap—cgroup v1 support will eventually be removed according to the Kubernetes deprecation policy. To enable cgroup v2 properly, your environment must meet four requirements:
- Linux kernel 5.8 or later
- A Linux distribution with cgroup v2 enabled by default
- containerd 1.4+ (or CRI-O 1.20+)
- Both kubelet and the container runtime set to the
systemdcgroup driver
You can check which cgroup version a node is running with a single command:
stat -fc %T /sys/fs/cgroup/
# Output "cgroup2fs" means cgroup v2
# Output "tmpfs" means cgroup v1
The Core cgroup Driver Conflict: Who Owns the Resource Ledger?
Both kubelet and the container runtime need to create and manage cgroups. The mechanism they use to do so is called the cgroup driver:
cgroupfs: Manipulates cgroups by reading and writing files directly under/sys/fs/cgroup.systemd: Creates and manages cgroups through systemd’s native APIs.
On Linux distributions that use systemd as their init system, systemd is already the top-level cgroup manager, allocating cgroups for every systemd unit on the machine. If kubelet and the container runtime bypass systemd and write directly to /sys/fs/cgroup via cgroupfs, the node ends up with two managers maintaining separate resource ledgers on the same machine.
According to Kubernetes documentation, nodes configured this way become unstable under resource pressure. That is why the official Container Runtimes specification states unequivocally: on systemd-based systems, you must use the systemd driver, and cgroup v2 requires the systemd driver.
The Cost of Mismatched cgroup Drivers
In manual setups or older Kubernetes versions, configuring kubelet to use systemd while leaving containerd on cgroupfs (or vice versa) produces two well-documented failure modes:
- Node Instability: Under rising resource load, kubelet’s eviction logic clashes with the container runtime’s resource constraints.
- Pod Sandbox Errors That a Restart May Not Fix: Switching drivers in place on a live node can break sandbox recreation for existing Pods. The official guidance is to recreate the node with the correct configuration rather than modifying a running node in place.
A common symptom reported across the community is periodic, unexplained restarts of control-plane static Pods (such as etcd and kube-apiserver).
Because kubelet runs as a systemd unit, it expects static Pods to live under the systemd-managed cgroup hierarchy. When containerd places them under a separate cgroupfs path instead, kubelet loses track of their health status, assumes the processes have become unresponsive, and repeatedly restarts them.
More often than not, the root cause traces back to a single line in containerd’s configuration file: SystemdCgroup left at its default value of false.
KubeletCgroupDriverFromCRI: Automatic Driver Alignment
To eliminate manual mismatches, Kubernetes introduced the KubeletCgroupDriverFromCRI feature gate:
- At startup, kubelet queries the container runtime via the CRI
RuntimeConfiggRPC interface to discover which cgroup driver the runtime is using. - kubelet automatically aligns with the runtime’s driver and ignores the
cgroupDriverfield in its own configuration file. - Introduced as alpha in Kubernetes 1.28, this feature reached beta (enabled by default) in 1.31 and graduated to stable in 1.34.
This shifts the troubleshooting focus: on modern nodes running containerd 2.x, the decisive setting is SystemdCgroup in containerd’s configuration file. Even with automatic alignment enabled in kubelet, if containerd leaves SystemdCgroup set to false, the entire node will still fall back to the discouraged cgroupfs mode.
How K8s Resource Limits Map to the Linux Kernel
From Pod YAML to Kernel Enforcement
When you declare resource requests and limits in a Pod manifest:
resources:
requests:
cpu: 250m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
Kubernetes does not enforce these limits itself. Instead, it passes them down the runtime call chain to the Linux kernel:
- Pod YAML
- CRI: CreateContainercontainerd
- OCI: config.jsonrunc
- Writes to virtual filesystem/sys/fs/cgroup/...
- Enforced by Linux Kernel
On a cgroup v2 node, each YAML field maps to the underlying kernel cgroup v2 control files as follows:
| YAML Field | cgroup v2 File | Actual Kernel Behavior |
|---|---|---|
limits.memory: 256Mi | memory.max = 268435456 | Hard ceiling. When memory is exhausted and cannot be reclaimed, the kernel invokes the OOM Killer to kill the process. |
requests.memory: 128Mi | Used by scheduler | Used by Kubernetes for node capacity planning and scheduling; does not map to a single hard kernel limit. |
limits.cpu: 500m | cpu.max = 50000 100000 | Over each 100ms period, the container may use at most 50ms of CPU time; paused once the quota is exceeded. |
requests.cpu: 250m | cpu.weight | Relative CPU share weight when processes contend for CPU cycles, not a hard ceiling. |
Running Out of Memory: OOMKilled and Exit Code 137
Memory is an incompressible resource that cannot be “deferred.” Once a container’s memory usage hits memory.max and cannot be freed by reclaiming page cache, the Linux kernel detects memory pressure and sends SIGKILL to terminate the process.
Running kubectl describe pod reveals the classic signature:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 3
Exit code 137 equals 128 + 9 (where signal 9 is SIGKILL). Two operational details are worth keeping straight here:
- Container OOMKilled ≠ Node Eviction: Container
OOMKilledis triggered immediately by the kernel inside an individual cgroup. Node eviction happens when the node as a whole runs low on memory, prompting kubelet to proactively select and evict lower-priority Pods. - The Hidden Trap of Memory-Backed
emptyDir: If you configure anemptyDirvolume withmedium: Memoryfor fast scratch storage, every file written to that directory counts directly against the container’s memory usage—which can trigger unexpectedOOMKilledcrashes.
Exceeding CPU Limits: The Hidden Latency Cost of Throttling
Unlike memory, CPU is a compressible resource whose allocation can be deferred. When a container’s CPU consumption exceeds cpu.max, the kernel does not terminate the program; instead, it enforces throttling, freezing the process until the next scheduling period begins.
By default, the Linux CFS (Completely Fair Scheduler, the kernel mechanism that allocates CPU time) operates in 100ms (100,000 microseconds) periods. A quota of 500m represents 0.5 CPU cores, which translates to at most 50ms of CPU compute time per 100ms period.
If a multi-threaded application runs flat out across 4 CPU cores, every 1ms of wall-clock time consumes 4ms of its CPU quota. That means the container burns through its entire 50ms quota in just 12.5ms, and the kernel forcibly pauses all of its threads for the remaining 87.5ms of the period.
This creates a notoriously deceptive performance issue: your monitoring dashboard shows average CPU utilization sitting around 50%, yet your API’s p99 latency (the threshold below which 99% of requests respond) suffers from violent spikes of tens of milliseconds.
Hands-On Verification: Inspecting Node cgroup Files Directly
On a cgroup v2 node, you can verify the kernel’s live quota state directly from the command line with crictl (covered in the next section):
# 1. Get the container ID and its host PID on the node
crictl ps --name <CONTAINER_NAME>
crictl inspect <CONTAINER_ID> | rg '"pid"'
# 2. Check the cgroup hierarchy path for that PID
cat /proc/<PID>/cgroup
# Example output: 0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/cri-containerd-<id>.scope
# 3. Read the enforced limits and runtime metrics
CG=/sys/fs/cgroup$(cut -d: -f3 /proc/<PID>/cgroup)
cat $CG/memory.max # Shows 268435456 (i.e., 256MiB)
cat $CG/cpu.max # Shows 50000 100000 (i.e., 500m)
cat $CG/memory.events # oom_kill > 0 indicates past OOM events
cat $CG/cpu.stat # Rising nr_throttled and throttled_usec indicate active throttling
The .slice and .scope suffixes in the path indicate that the node is using the systemd driver, while burstable reflects the Pod’s QoS class (Kubernetes classifies Pods into Guaranteed, Burstable, or BestEffort based on their requests and limits).
Hands-On Node Troubleshooting
Node Diagnostic Tooling: The Central Role of crictl
Since dockershim was removed, modern Kubernetes nodes no longer ship with the docker CLI. Operators must understand how the three main node-level tools fit together:
- Operator / SRE
-
- crictlTalks to containerd/CRI-O over the CRI socket (Shares the exact same view as kubelet)
- ctrTalks directly to containerd's internal gRPC (For low-level debugging only)
- nerdctlProvides a Docker CLI-like experience (For local development and testing)
NOTE
Further reading: Why Kubernetes Removed Docker
crictl (part of the cri-tools project) is the primary tool for node operations, and its version numbers stay synchronized with Kubernetes. Its configuration file lives at /etc/crictl.yaml:
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
Quick Reference: Essential Troubleshooting Commands
| Command | Practical Use |
|---|---|
crictl pods | List all Pod sandboxes on the node |
crictl ps -a | List all containers (always pass -a, or you will miss containers that just exited) |
crictl logs <CONTAINER_ID> | Read container output (pass --previous to read logs from a previous crashed run) |
crictl inspect <CONTAINER_ID> | Query full container state, including host PID, exit code, mounts, and image digest |
crictl inspectp <POD_ID> | Query Pod sandbox network state and IP configuration |
crictl images / crictl pull | Verify whether images can be pulled normally at the node level |
crictl info | Inspect runtime operational status and CNI network readiness |
journalctl -u containerd | Inspect container runtime system logs |
journalctl -u kubelet | Inspect kubelet system logs |
Troubleshooting SOP 1: Pod Stuck in ImagePullBackOff
When a Pod fails to pull its image, follow this debugging sequence:
- 1. kubectl describe podIdentify the error reason (404 Not Found / 401 / connection timeout)
- 2. Log in to the node and run crictl pullRule out K8s scheduling interference; Test direct node-to-registry connectivity
- 3. Node-level pull also failsInspect journalctl -u containerd (Check DNS, TLS certificates, and proxy); Check /etc/containerd/certs.d/ (see next section)
- 4. Manual node-level pull succeedsIssue narrowed down to imagePullSecrets (Pod did not get valid auth credentials)
Troubleshooting SOP 2: Pod in CrashLoopBackOff with Empty Application Logs
When an application crashes before initializing its logging framework, kubectl logs often returns empty output. At that point, drop down to the node level and extract kernel and process details from the runtime’s perspective:
- Locate the exited container: Run
crictl ps -a --name <CONTAINER_NAME>to find the container instance in theExitedstate. - Analyze the exit code: Run
crictl inspect <CONTAINER_ID>and checkexitCodeandreason:137+OOMKilled: Memory limit exceeded; raiselimits.memory.127: Executable not found; usually an incorrect path in the DockerfileENTRYPOINTorCMD.1: Application exited due to an internal error.
- Extract raw output: Run
crictl logs <CONTAINER_ID>to capture any standard error messages printed right before the process terminated. - Check for failures at container creation: If
crictl ps -ashows no container at all, the container was never created in the first place. Inspectjournalctl -u containerdand search forfailed to createorfailed to start. Common causes includeruncfailing to create cgroups, missing volume mount paths on disk, orseccomp/AppArmor blocking a required system call.
Troubleshooting SOP 3: Node Status Shows NotReady
When a node transitions to NotReady, the core diagnostic goal is determining whether the container runtime or the CNI plugin is at fault:
- Run
kubectl describe node <NODE>, and inspect the status messages under theConditionssection. - If
NetworkPluginNotReadyappears, log in to the node and runcrictl infoto checkRuntimeReadyandNetworkReady:NetworkReady: false: CNI packages are not installed, or/etc/cni/net.d/lacks a valid configuration (the runtime itself is not broken; it is reporting the network plugin status on behalf of the node).RuntimeReady: falseor socket connection failure: Runsystemctl status containerdandjournalctl -u containerdto check whether containerd crashed due to a syntax error in its configuration file.
Essential containerd Configuration
The Configuration Gap Between containerd 1.x and 2.x
containerd 2.0 refactored its architecture, upgrading the configuration schema to version = 3 and splitting the CRI plugin into io.containerd.cri.v1.runtime and io.containerd.cri.v1.images.
If you paste a 1.x configuration snippet from the web directly into a 2.x environment, those settings may be ignored—leaving you wondering why your edits had no effect.
containerd --version # Confirm the major version number
containerd config default | head -5 # Check the version declaration in the default config
containerd config dump # Dump the effective merged configuration currently active
Additionally, some Linux distribution packages for containerd disable cri by default under disabled_plugins. The officially recommended workflow is to generate a baseline template with containerd config default > /etc/containerd/config.toml, apply your targeted customizations, and always run sudo systemctl restart containerd afterward.
Configuration 1: Enabling SystemdCgroup
In a systemd environment, you must explicitly switch runc’s cgroup driver to systemd.
containerd 2.x (modern standard, version = 3):
version = 3
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true
(Note: If you are still on containerd 1.x, the plugin path is plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.)
If your node defines multiple custom runc-based runtime execution environments (covered in the RuntimeClass section below), SystemdCgroup must be kept identical across every handler.
Configuration 2: Private Registry Mirrors and Self-Signed Certificates
The legacy registry.mirrors setting has been deprecated. The modern architecture specifies config_path in the main configuration file and creates a dedicated hosts.toml for each registry under its own subdirectory.
config.toml configuration (containerd 2.x):
version = 3
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = "/etc/containerd/certs.d"
(Note: In containerd 1.x, this is set under plugins."io.containerd.grpc.v1.cri".registry.)
Create /etc/containerd/certs.d/docker.io/hosts.toml:
server = "https://registry-1.docker.io"
[host."https://mirror.example.internal"]
capabilities = ["pull", "resolve"]
ca = "/etc/containerd/certs.d/docker.io/company-ca.crt"
Key operational rules:
- The directory structure must match the registry hostname exactly (if a custom port is used, include it in full, such as
certs.d/192.168.1.10:5000/). - Modifying
hosts.tomldoes not require restarting containerd—changes take effect immediately; you only need to restart the daemon when changingconfig_pathitself.
Configuration 3: Sandbox (pause) Image in Air-Gapped Environments
Every Kubernetes Pod relies on a pause container to maintain shared Linux namespaces. By default, containerd attempts to download this image from registry.k8s.io. In an air-gapped environment, failing to override this setting will cause all Pod creations to fail.
containerd 2.x:
version = 3
[plugins.'io.containerd.cri.v1.images'.pinned_images]
sandbox = "registry.internal.example/pause:3.10.2"
(Note: In containerd 1.x, the field is named sandbox_image directly under plugins."io.containerd.grpc.v1.cri".)
Run containerd config default on the node to verify which pause version your containerd build expects, then mirror that exact version into your internal private registry.
NOTE
Further reading: What Is a Pod Sandbox? CRI Sandboxes, the Pause Container, and the Node Networking Call Chain
runc Escape Vulnerabilities and RuntimeClass Sandboxing
Why Low-Level Runtime Bugs Are Critical
Low-level runtimes (such as runc) run with root privileges on the host so they can invoke kernel APIs to place processes inside isolated environments. In default setups without User Namespaces enabled, UID 0 inside the container is identical to UID 0 on the host in the eyes of the kernel.
If a logic bug occurs in runc, an attacker can break out of the namespace boundary and gain host-level root privileges.
- Container Process (UID 0)
- runc Binary Instance
- Exploit: Breaks isolation to host rootHost Linux Kernel (UID 0: root)
Two Landmark Escape CVEs and Their Security Lessons
Two major CVEs highlight the risks of shared-kernel containers:
- CVE-2019-5736 (Overwriting the Host
runcBinary): During execution,runcfailed to properly protect its own/proc/self/exefile descriptor (an integer handle assigned by the OS for a process to access an open file). A root process inside the container could overwrite theruncbinary on the host, causing subsequent host commands (such asdocker exec) to execute malicious code. Later releases patched this by cloning the binary into an anonymous in-memory file (memfd). - CVE-2024-21626 (Leaky Vessels File Descriptor Leak): Internal initialization in
runcleaked a file descriptor (FD) pointing to a host directory. An attacker could pre-configure an image’s metadata to set the working directory to/proc/self/fd/<leaked-fd>. Whenruncstarted the container and changed directories, the process’s working directory landed directly on the host filesystem—meaning merely pulling and running a malicious image could lead to a container breakout.
The community continues to uncover escape vulnerabilities involving mount handling and /proc (such as CVE-2025-31133 in late 2025). Your container runtime is core infrastructure that requires regular upgrades and patching.
Defense in Depth and RuntimeClass Sandboxing
When running untrusted workloads (such as user-submitted code or workloads in multi-tenant environments), relying solely on shared Linux kernel isolation is not enough. You can use RuntimeClass to route specific Pods to sandboxed runtimes with hardware virtualization or kernel syscall interception (such as gVisor or Kata Containers).
- Kubernetes Pod
-
- runtimeClassName: runccontainerd-shim-runc-v2
- Standard Shared-Kernel Container
- runtimeClassName: gvisorcontainerd-shim-runsc-v1
- Sentry Syscall-Intercepting Sandbox
- runtimeClassName: katacontainerd-shim-kata-v2
- Dedicated Lightweight MicroVM
Setting Up RuntimeClass in Three Steps
Step 1: Register the runtime handlers in containerd (using 2.x syntax)
version = 3
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
Step 2: Create the RuntimeClass object in the cluster
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
scheduling:
nodeSelector:
sandbox.example.com/gvisor: "true" # Ensure Pods schedule only onto nodes with gVisor
Step 3: Declare the RuntimeClass in your Pod YAML
apiVersion: v1
kind: Pod
metadata:
name: untrusted-workload
spec:
runtimeClassName: gvisor
containers:
- name: worker
image: registry.internal.example/untrusted-job@sha256:7f83b...
Practical pitfalls to avoid:
- Scheduling Guards: By default,
RuntimeClassassumes every node in the cluster has that runtime installed. If only a subset of nodes supports it, always configurescheduling.nodeSelectorso Pods are not scheduled onto unsupported nodes and left stuck inFailed. - Resource Overhead: Kata Containers spins up a dedicated MicroVM for each Pod. Declare
overheadin theRuntimeClassso the scheduler accounts for the virtual machine’s own CPU and memory footprint.
How Containers Really Work on macOS: The Linux VM Architecture
Why macOS Cannot Run Linux Containers Natively
Many operators and developers write and test code locally on a Mac, only to hit a frustrating disconnect: workloads pass local testing but fail with cgroup or OOM errors once deployed to Kubernetes nodes.
Recall the definition of a Linux container: a container is a Linux process built on Linux namespaces and cgroups.
macOS runs on the XNU kernel, which has neither Linux namespaces nor cgroups, and cannot natively parse ELF-format Linux binaries. Consequently, the only way to run Linux containers on a Mac is to spin up a Linux virtual machine (VM) in the background.
Whether you use Docker Desktop, OrbStack, Colima, or Lima, the underlying architecture is fundamentally the same:
+----------------------------------------------
| macOS Host (XNU Kernel)
|
| [ docker CLI / nerdctl ]
| |
| v (Unix Socket / Port Forwarding)
|
| +----------------------------------+
| | Linux VM (Linux Kernel) |
| | |
| | [ dockerd / containerd ] |
| | | |
| | v |
| | [ runc ] |
| | | |
| | v |
| | [ Linux container process ] |
| +----------------------------------+
+----------------------------------------------
Every docker command you run in a macOS terminal is forwarded over a socket to the daemon inside that VM. Mounted local directories and exposed ports also rely on the virtualization tooling silently syncing files and forwarding network traffic behind the scenes.
Five Practical Implications of the VM Architecture for Developers
Once you understand that a Linux virtual machine sits in the middle, five common issues on macOS become easy to explain:
- Volume Bind Mount I/O Bottlenecks: File I/O crossing the VM virtualization boundary carries heavy overhead. In projects with tens of thousands of small files (such as frontend
node_modulesor PHP projects), disk read/write speeds are noticeably slower than on native Linux. - Dual Memory Ceilings: The memory available to a container on a Mac is capped by the total memory allocated to the VM. A container getting
OOMKilledlocally might simply mean your Docker Desktop resource allocation is set too low. - Cross-Architecture CPU Emulation Cost: Running
amd64images on an Apple Silicon (arm64) Mac requires instruction translation via QEMU or Rosetta, significantly degrading performance for CPU-intensive tasks. - Container Processes Are Invisible on the Host: Running
ps auxin a macOS terminal will never show processes inside your containers because they all run inside that Linux VM. To inspect/procor/sys/fs/cgroup, you must shell into the VM viacolima sshorlimactl shell. - The “Works on My Machine, Breaks in Production” Environment Gap: Locally on a Mac, containers run inside a VM-backed Docker environment; in production, they run on Kubernetes nodes powered by
containerd+runc. Because the two environments differ in Linux kernel versions, cgroup generations, and defaultseccomppolicies, never assume local Mac behavior matches production when troubleshooting live incidents.
NOTE
Further reading: OCI 101: The Common Language of the Container World
Further Learning
To deepen your operational skills, explore these official specifications and core documentation:
- Kubernetes Container Runtimes Guide
- Kubernetes cgroup v2 Architecture
- containerd CRI Plugin Configuration Guide
- Linux Kernel Control Group v2 Documentation