A Beginner's Guide to Container Runtimes (Part 2): Operational Practice

Part 1: Core Mechanics established three foundational concepts: a container is an ordinary Linux process constrained by namespaces and cgroups, the runtime stack is split into high-level (containerd) and low-level (runc) tiers, and Kubernetes communicates with the runtime through the CRI.

If Part 1 answered “What is a container runtime?”, this operational guide focuses on the core day-to-day question: “How do you debug it when things break, and how do you configure it properly in the first place?”

This article walks through six practical scenarios cloud-native engineers encounter most often in production:

  1. cgroup v2 and cgroup Driver Unification: Why does Kubernetes 1.35 refuse to start on cgroup v1 by default, and why do mismatched cgroup drivers destabilize nodes?
  2. How K8s Resource Limits Map to the Linux Kernel: How do Pod limits become memory.max and cpu.max? What really happens under the hood during OOMKilled and CPU throttling?
  3. Hands-On Node Troubleshooting: When the docker CLI isn’t on the node, how do you pinpoint ImagePullBackOff and crash loops using crictl and journalctl?
  4. Essential containerd Configuration: Navigating the configuration breaking changes between containerd 1.x and 2.x for SystemdCgroup, private registries, and the pause sandbox image.
  5. runc Escape Vulnerabilities and RuntimeClass Sandboxing: Security lessons from CVE-2019-5736 to Leaky Vessels, plus how to isolate workloads with gVisor and Kata Containers.
  6. How Containers Really Work on macOS: Why do containers on a Mac always run inside a Linux VM, and how does that explain slow bind mounts and cross-architecture emulation overhead?

cgroup v2 and cgroup Driver Unification

cgroup v2 Is Now a Hard Requirement in Modern Kubernetes

cgroups (Control Groups) enforce resource quotas on processes. Over the past few years, most mainstream Linux distributions have switched their default from v1 to cgroup v2:

Aspectcgroup v1cgroup v2
HierarchySeparate, independent tree for each resource controller (CPU, memory, etc.)A single unified hierarchy shared across all resource controllers
Memory AccountingTracked separately per subsystem; misses kernel memory, network buffers, etc.Unified accounting that properly tracks delayed costs like page cache writeback
Pressure MetricsNo native unified metricBuilt-in PSI (Pressure Stall Information—the share of time workloads stall waiting for CPU, memory, or I/O) to pinpoint resource bottlenecks
Kubernetes StatusMaintenance mode in 1.31; officially deprecated starting in 1.35Generally Available (GA) since 1.25

The critical milestone: starting with Kubernetes 1.35, kubelet refuses to start on cgroup v1 nodes by default.

While you can temporarily bypass this check by setting failCgroupV1 to false in the kubelet configuration, this is only a stopgap—cgroup v1 support will eventually be removed according to the Kubernetes deprecation policy. To enable cgroup v2 properly, your environment must meet four requirements:

You can check which cgroup version a node is running with a single command:

stat -fc %T /sys/fs/cgroup/
# Output "cgroup2fs" means cgroup v2
# Output "tmpfs"     means cgroup v1

The Core cgroup Driver Conflict: Who Owns the Resource Ledger?

Both kubelet and the container runtime need to create and manage cgroups. The mechanism they use to do so is called the cgroup driver:

On Linux distributions that use systemd as their init system, systemd is already the top-level cgroup manager, allocating cgroups for every systemd unit on the machine. If kubelet and the container runtime bypass systemd and write directly to /sys/fs/cgroup via cgroupfs, the node ends up with two managers maintaining separate resource ledgers on the same machine.

According to Kubernetes documentation, nodes configured this way become unstable under resource pressure. That is why the official Container Runtimes specification states unequivocally: on systemd-based systems, you must use the systemd driver, and cgroup v2 requires the systemd driver.

The Cost of Mismatched cgroup Drivers

In manual setups or older Kubernetes versions, configuring kubelet to use systemd while leaving containerd on cgroupfs (or vice versa) produces two well-documented failure modes:

  1. Node Instability: Under rising resource load, kubelet’s eviction logic clashes with the container runtime’s resource constraints.
  2. Pod Sandbox Errors That a Restart May Not Fix: Switching drivers in place on a live node can break sandbox recreation for existing Pods. The official guidance is to recreate the node with the correct configuration rather than modifying a running node in place.

A common symptom reported across the community is periodic, unexplained restarts of control-plane static Pods (such as etcd and kube-apiserver).

Because kubelet runs as a systemd unit, it expects static Pods to live under the systemd-managed cgroup hierarchy. When containerd places them under a separate cgroupfs path instead, kubelet loses track of their health status, assumes the processes have become unresponsive, and repeatedly restarts them.

More often than not, the root cause traces back to a single line in containerd’s configuration file: SystemdCgroup left at its default value of false.

KubeletCgroupDriverFromCRI: Automatic Driver Alignment

To eliminate manual mismatches, Kubernetes introduced the KubeletCgroupDriverFromCRI feature gate:

This shifts the troubleshooting focus: on modern nodes running containerd 2.x, the decisive setting is SystemdCgroup in containerd’s configuration file. Even with automatic alignment enabled in kubelet, if containerd leaves SystemdCgroup set to false, the entire node will still fall back to the discouraged cgroupfs mode.


How K8s Resource Limits Map to the Linux Kernel

From Pod YAML to Kernel Enforcement

When you declare resource requests and limits in a Pod manifest:

resources:
  requests:
    cpu: 250m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

Kubernetes does not enforce these limits itself. Instead, it passes them down the runtime call chain to the Linux kernel:

  1. Pod YAML
  2. CRI: CreateContainercontainerd
  3. OCI: config.jsonrunc
  4. Writes to virtual filesystem/sys/fs/cgroup/...
  5. Enforced by Linux Kernel

On a cgroup v2 node, each YAML field maps to the underlying kernel cgroup v2 control files as follows:

YAML Fieldcgroup v2 FileActual Kernel Behavior
limits.memory: 256Mimemory.max = 268435456Hard ceiling. When memory is exhausted and cannot be reclaimed, the kernel invokes the OOM Killer to kill the process.
requests.memory: 128MiUsed by schedulerUsed by Kubernetes for node capacity planning and scheduling; does not map to a single hard kernel limit.
limits.cpu: 500mcpu.max = 50000 100000Over each 100ms period, the container may use at most 50ms of CPU time; paused once the quota is exceeded.
requests.cpu: 250mcpu.weightRelative CPU share weight when processes contend for CPU cycles, not a hard ceiling.

Running Out of Memory: OOMKilled and Exit Code 137

Memory is an incompressible resource that cannot be “deferred.” Once a container’s memory usage hits memory.max and cannot be freed by reclaiming page cache, the Linux kernel detects memory pressure and sends SIGKILL to terminate the process.

Running kubectl describe pod reveals the classic signature:

 Last State:     Terminated
   Reason:       OOMKilled
   Exit Code:    137
 Restart Count:  3

Exit code 137 equals 128 + 9 (where signal 9 is SIGKILL). Two operational details are worth keeping straight here:

Exceeding CPU Limits: The Hidden Latency Cost of Throttling

Unlike memory, CPU is a compressible resource whose allocation can be deferred. When a container’s CPU consumption exceeds cpu.max, the kernel does not terminate the program; instead, it enforces throttling, freezing the process until the next scheduling period begins.

By default, the Linux CFS (Completely Fair Scheduler, the kernel mechanism that allocates CPU time) operates in 100ms (100,000 microseconds) periods. A quota of 500m represents 0.5 CPU cores, which translates to at most 50ms of CPU compute time per 100ms period.

If a multi-threaded application runs flat out across 4 CPU cores, every 1ms of wall-clock time consumes 4ms of its CPU quota. That means the container burns through its entire 50ms quota in just 12.5ms, and the kernel forcibly pauses all of its threads for the remaining 87.5ms of the period.

This creates a notoriously deceptive performance issue: your monitoring dashboard shows average CPU utilization sitting around 50%, yet your API’s p99 latency (the threshold below which 99% of requests respond) suffers from violent spikes of tens of milliseconds.

Hands-On Verification: Inspecting Node cgroup Files Directly

On a cgroup v2 node, you can verify the kernel’s live quota state directly from the command line with crictl (covered in the next section):

# 1. Get the container ID and its host PID on the node
crictl ps --name <CONTAINER_NAME>
crictl inspect <CONTAINER_ID> | rg '"pid"'

# 2. Check the cgroup hierarchy path for that PID
cat /proc/<PID>/cgroup
# Example output: 0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/cri-containerd-<id>.scope

# 3. Read the enforced limits and runtime metrics
CG=/sys/fs/cgroup$(cut -d: -f3 /proc/<PID>/cgroup)
cat $CG/memory.max        # Shows 268435456 (i.e., 256MiB)
cat $CG/cpu.max           # Shows 50000 100000 (i.e., 500m)
cat $CG/memory.events     # oom_kill > 0 indicates past OOM events
cat $CG/cpu.stat          # Rising nr_throttled and throttled_usec indicate active throttling

The .slice and .scope suffixes in the path indicate that the node is using the systemd driver, while burstable reflects the Pod’s QoS class (Kubernetes classifies Pods into Guaranteed, Burstable, or BestEffort based on their requests and limits).


Hands-On Node Troubleshooting

Node Diagnostic Tooling: The Central Role of crictl

Since dockershim was removed, modern Kubernetes nodes no longer ship with the docker CLI. Operators must understand how the three main node-level tools fit together:

  1. Operator / SRE
    1. crictlTalks to containerd/CRI-O over the CRI socket (Shares the exact same view as kubelet)
    1. ctrTalks directly to containerd's internal gRPC (For low-level debugging only)
    1. nerdctlProvides a Docker CLI-like experience (For local development and testing)

NOTE

Further reading: Why Kubernetes Removed Docker

crictl (part of the cri-tools project) is the primary tool for node operations, and its version numbers stay synchronized with Kubernetes. Its configuration file lives at /etc/crictl.yaml:

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock

Quick Reference: Essential Troubleshooting Commands

CommandPractical Use
crictl podsList all Pod sandboxes on the node
crictl ps -aList all containers (always pass -a, or you will miss containers that just exited)
crictl logs <CONTAINER_ID>Read container output (pass --previous to read logs from a previous crashed run)
crictl inspect <CONTAINER_ID>Query full container state, including host PID, exit code, mounts, and image digest
crictl inspectp <POD_ID>Query Pod sandbox network state and IP configuration
crictl images / crictl pullVerify whether images can be pulled normally at the node level
crictl infoInspect runtime operational status and CNI network readiness
journalctl -u containerdInspect container runtime system logs
journalctl -u kubeletInspect kubelet system logs

Troubleshooting SOP 1: Pod Stuck in ImagePullBackOff

When a Pod fails to pull its image, follow this debugging sequence:

  1. 1. kubectl describe podIdentify the error reason (404 Not Found / 401 / connection timeout)
  2. 2. Log in to the node and run crictl pullRule out K8s scheduling interference; Test direct node-to-registry connectivity
  3. 3. Node-level pull also failsInspect journalctl -u containerd (Check DNS, TLS certificates, and proxy); Check /etc/containerd/certs.d/ (see next section)
  4. 4. Manual node-level pull succeedsIssue narrowed down to imagePullSecrets (Pod did not get valid auth credentials)

Troubleshooting SOP 2: Pod in CrashLoopBackOff with Empty Application Logs

When an application crashes before initializing its logging framework, kubectl logs often returns empty output. At that point, drop down to the node level and extract kernel and process details from the runtime’s perspective:

  1. Locate the exited container: Run crictl ps -a --name <CONTAINER_NAME> to find the container instance in the Exited state.
  2. Analyze the exit code: Run crictl inspect <CONTAINER_ID> and check exitCode and reason:
    • 137 + OOMKilled: Memory limit exceeded; raise limits.memory.
    • 127: Executable not found; usually an incorrect path in the Dockerfile ENTRYPOINT or CMD.
    • 1: Application exited due to an internal error.
  3. Extract raw output: Run crictl logs <CONTAINER_ID> to capture any standard error messages printed right before the process terminated.
  4. Check for failures at container creation: If crictl ps -a shows no container at all, the container was never created in the first place. Inspect journalctl -u containerd and search for failed to create or failed to start. Common causes include runc failing to create cgroups, missing volume mount paths on disk, or seccomp/AppArmor blocking a required system call.

Troubleshooting SOP 3: Node Status Shows NotReady

When a node transitions to NotReady, the core diagnostic goal is determining whether the container runtime or the CNI plugin is at fault:

  1. Run kubectl describe node <NODE>, and inspect the status messages under the Conditions section.
  2. If NetworkPluginNotReady appears, log in to the node and run crictl info to check RuntimeReady and NetworkReady:
    • NetworkReady: false: CNI packages are not installed, or /etc/cni/net.d/ lacks a valid configuration (the runtime itself is not broken; it is reporting the network plugin status on behalf of the node).
    • RuntimeReady: false or socket connection failure: Run systemctl status containerd and journalctl -u containerd to check whether containerd crashed due to a syntax error in its configuration file.

Essential containerd Configuration

The Configuration Gap Between containerd 1.x and 2.x

containerd 2.0 refactored its architecture, upgrading the configuration schema to version = 3 and splitting the CRI plugin into io.containerd.cri.v1.runtime and io.containerd.cri.v1.images.

If you paste a 1.x configuration snippet from the web directly into a 2.x environment, those settings may be ignored—leaving you wondering why your edits had no effect.

containerd --version                   # Confirm the major version number
containerd config default | head -5    # Check the version declaration in the default config
containerd config dump                 # Dump the effective merged configuration currently active

Additionally, some Linux distribution packages for containerd disable cri by default under disabled_plugins. The officially recommended workflow is to generate a baseline template with containerd config default > /etc/containerd/config.toml, apply your targeted customizations, and always run sudo systemctl restart containerd afterward.

Configuration 1: Enabling SystemdCgroup

In a systemd environment, you must explicitly switch runc’s cgroup driver to systemd.

containerd 2.x (modern standard, version = 3):

version = 3

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

(Note: If you are still on containerd 1.x, the plugin path is plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options.)

If your node defines multiple custom runc-based runtime execution environments (covered in the RuntimeClass section below), SystemdCgroup must be kept identical across every handler.

Configuration 2: Private Registry Mirrors and Self-Signed Certificates

The legacy registry.mirrors setting has been deprecated. The modern architecture specifies config_path in the main configuration file and creates a dedicated hosts.toml for each registry under its own subdirectory.

config.toml configuration (containerd 2.x):

version = 3

[plugins.'io.containerd.cri.v1.images'.registry]
  config_path = "/etc/containerd/certs.d"

(Note: In containerd 1.x, this is set under plugins."io.containerd.grpc.v1.cri".registry.)

Create /etc/containerd/certs.d/docker.io/hosts.toml:

server = "https://registry-1.docker.io"

[host."https://mirror.example.internal"]
  capabilities = ["pull", "resolve"]
  ca = "/etc/containerd/certs.d/docker.io/company-ca.crt"

Key operational rules:

Configuration 3: Sandbox (pause) Image in Air-Gapped Environments

Every Kubernetes Pod relies on a pause container to maintain shared Linux namespaces. By default, containerd attempts to download this image from registry.k8s.io. In an air-gapped environment, failing to override this setting will cause all Pod creations to fail.

containerd 2.x:

version = 3

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = "registry.internal.example/pause:3.10.2"

(Note: In containerd 1.x, the field is named sandbox_image directly under plugins."io.containerd.grpc.v1.cri".)

Run containerd config default on the node to verify which pause version your containerd build expects, then mirror that exact version into your internal private registry.


runc Escape Vulnerabilities and RuntimeClass Sandboxing

Why Low-Level Runtime Bugs Are Critical

Low-level runtimes (such as runc) run with root privileges on the host so they can invoke kernel APIs to place processes inside isolated environments. In default setups without User Namespaces enabled, UID 0 inside the container is identical to UID 0 on the host in the eyes of the kernel.

If a logic bug occurs in runc, an attacker can break out of the namespace boundary and gain host-level root privileges.

  1. Container Process (UID 0)
  2. runc Binary Instance
  3. Exploit: Breaks isolation to host rootHost Linux Kernel (UID 0: root)

Two Landmark Escape CVEs and Their Security Lessons

Two major CVEs highlight the risks of shared-kernel containers:

The community continues to uncover escape vulnerabilities involving mount handling and /proc (such as CVE-2025-31133 in late 2025). Your container runtime is core infrastructure that requires regular upgrades and patching.

Defense in Depth and RuntimeClass Sandboxing

When running untrusted workloads (such as user-submitted code or workloads in multi-tenant environments), relying solely on shared Linux kernel isolation is not enough. You can use RuntimeClass to route specific Pods to sandboxed runtimes with hardware virtualization or kernel syscall interception (such as gVisor or Kata Containers).

  1. Kubernetes Pod
    1. runtimeClassName: runccontainerd-shim-runc-v2
    2. Standard Shared-Kernel Container
    1. runtimeClassName: gvisorcontainerd-shim-runsc-v1
    2. Sentry Syscall-Intercepting Sandbox
    1. runtimeClassName: katacontainerd-shim-kata-v2
    2. Dedicated Lightweight MicroVM

Setting Up RuntimeClass in Three Steps

Step 1: Register the runtime handlers in containerd (using 2.x syntax)

version = 3

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"

Step 2: Create the RuntimeClass object in the cluster

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    sandbox.example.com/gvisor: "true"   # Ensure Pods schedule only onto nodes with gVisor

Step 3: Declare the RuntimeClass in your Pod YAML

apiVersion: v1
kind: Pod
metadata:
  name: untrusted-workload
spec:
  runtimeClassName: gvisor
  containers:
    - name: worker
      image: registry.internal.example/untrusted-job@sha256:7f83b...

Practical pitfalls to avoid:


How Containers Really Work on macOS: The Linux VM Architecture

Why macOS Cannot Run Linux Containers Natively

Many operators and developers write and test code locally on a Mac, only to hit a frustrating disconnect: workloads pass local testing but fail with cgroup or OOM errors once deployed to Kubernetes nodes.

Recall the definition of a Linux container: a container is a Linux process built on Linux namespaces and cgroups.

macOS runs on the XNU kernel, which has neither Linux namespaces nor cgroups, and cannot natively parse ELF-format Linux binaries. Consequently, the only way to run Linux containers on a Mac is to spin up a Linux virtual machine (VM) in the background.

Whether you use Docker Desktop, OrbStack, Colima, or Lima, the underlying architecture is fundamentally the same:

 +----------------------------------------------
 | macOS Host (XNU Kernel)
 |
 |   [ docker CLI / nerdctl ]
 |          |
 |          v  (Unix Socket / Port Forwarding)
 |
 |   +----------------------------------+
 |   | Linux VM (Linux Kernel)          |
 |   |                                  |
 |   |   [ dockerd / containerd ]       |
 |   |               |                  |
 |   |               v                  |
 |   |   [ runc ]                       |
 |   |       |                          |
 |   |       v                          |
 |   |   [ Linux container process ]    |
 |   +----------------------------------+
 +----------------------------------------------

Every docker command you run in a macOS terminal is forwarded over a socket to the daemon inside that VM. Mounted local directories and exposed ports also rely on the virtualization tooling silently syncing files and forwarding network traffic behind the scenes.

Five Practical Implications of the VM Architecture for Developers

Once you understand that a Linux virtual machine sits in the middle, five common issues on macOS become easy to explain:

  1. Volume Bind Mount I/O Bottlenecks: File I/O crossing the VM virtualization boundary carries heavy overhead. In projects with tens of thousands of small files (such as frontend node_modules or PHP projects), disk read/write speeds are noticeably slower than on native Linux.
  2. Dual Memory Ceilings: The memory available to a container on a Mac is capped by the total memory allocated to the VM. A container getting OOMKilled locally might simply mean your Docker Desktop resource allocation is set too low.
  3. Cross-Architecture CPU Emulation Cost: Running amd64 images on an Apple Silicon (arm64) Mac requires instruction translation via QEMU or Rosetta, significantly degrading performance for CPU-intensive tasks.
  4. Container Processes Are Invisible on the Host: Running ps aux in a macOS terminal will never show processes inside your containers because they all run inside that Linux VM. To inspect /proc or /sys/fs/cgroup, you must shell into the VM via colima ssh or limactl shell.
  5. The “Works on My Machine, Breaks in Production” Environment Gap: Locally on a Mac, containers run inside a VM-backed Docker environment; in production, they run on Kubernetes nodes powered by containerd + runc. Because the two environments differ in Linux kernel versions, cgroup generations, and default seccomp policies, never assume local Mac behavior matches production when troubleshooting live incidents.

Further Learning

To deepen your operational skills, explore these official specifications and core documentation: