Checkpoint and Restore of a Running Container

A running process is memory and kernel state held on behalf of a task. Saving it to disk means enumerating both, and restoring it means convincing the kernel to accept the enumeration as if it had produced it. The enumeration is not complete, the reconstruction is not always exact, and the gap between the two is what determines whether a workload can be checkpointed at all.

Freezing is a request, not a stop

The first step is to make the process tree stop changing. A cgroup provides this through its freezer: writing 1 to cgroup.freeze on the container’s cgroup tells the kernel to freeze every task in the subtree, and cgroup.events reports frozen 1 when it is done. The same cgroup tree that supplies the memory and CPU limits in Memory and CPU Limits Under the cgroup v2 Unified Hierarchy supplies this control, and freezing does not suspend accounting.

The freezer does not stop a task mid-instruction. It sets a flag that tasks observe at defined points, and a task in an interruptible sleep wakes, notices, and parks itself in the kernel. A task blocked in uninterruptible sleep, in a driver or on page I/O that cannot be interrupted, does not observe the request until the operation returns. The freeze therefore completes with a delay bounded by the longest uninterruptible operation in the tree, and a workload with a device call that never returns will not freeze at all. Implementations that impose a timeout and proceed anyway are checkpointing a tree that is still executing.

The freeze has a second effect that matters for consistency. Files the workload has written are flushed by the kernel’s normal writeback over time, and a checkpoint records file descriptors, not the file contents behind them. A writer frozen with dirty pages in the page cache, and those pages not yet on disk, is restored with descriptors pointing at a file whose content is whatever the disk has. Before dumping, a tool flushes the workload’s open files through their descriptors, which forces the dirty data out. This is not optional: skipping it produces a restored process reading stale bytes through an unchanged descriptor.

Enumerating memory

The memory of each task is a set of virtual memory areas, each with a start, an end, protection flags, an offset into a backing file, and a set of resident pages. The area list is read from /proc/<pid>/maps; the residency and the location of each page are read from /proc/<pid>/pagemap, a file of 64-bit entries, one per page of virtual address space. Each entry carries a present bit and a swap or file offset, which is how a tool distinguishes anonymous pages it must save from file-backed pages it can re-map.

Dumping memory is therefore two mechanisms. Private anonymous pages must be copied, because nothing else on the system holds them; they are read with process_vm_readv or through /proc/<pid>/mem once the process is frozen and stopped. Shared file-backed pages are usually not copied at all: the checkpoint records the file path and offset, and restore maps the file again. That decision is what makes a checkpoint of a container running binaries from the image small relative to the process’s apparent memory, since most of its mapped pages are the image’s own files, already present on disk.

The saving is also a correctness risk. If the file changed between checkpoint and restore, the mapped pages differ from what the process saw. For a read-only image layer the file cannot change; for a writable file the restore silently observes the new content. Copying the pages would be correct and expensive; recording the path is cheap and only correct if the file is immutable. Tools expose this as a choice, and the choice interacts with the overlay semantics described in Copy-Up, Whiteouts and Rename in Overlay Roots: a page mapped from a lower layer is file-backed and re-mappable, while a page copied up to a per-container upper directory is backed by a file that will not exist on the restoring host.

Kernel state is not in memory

Everything the process told the kernel during its life is in kernel structures, and each kind has to be read and replayed. A partial list of what a checkpoint must capture:

File descriptors, with their type, flags, offset, and the described object. A regular file descriptor is recreated by opening the same path and seeking to the recorded offset; the offset is state the file does not contain. A pipe descriptor is recreated as a pipe, and the data currently buffered in the pipe must be saved, because the pipe’s content exists only in the kernel. A socket descriptor requires the socket to be recreated with the same protocol, options, and addresses, and for a connected stream socket, with the peer’s cooperation.

A Unix domain socket is the case that forces cooperation. The peer may be outside the checkpointed tree, in which case the checkpoint cannot capture the peer’s end and the connection is broken on restore. Tools report this and either refuse to checkpoint or note the connection as lost; a workload that assumes a connected stream survives a checkpoint does not survive one.

Process credentials, including the capability sets, the UID and GID, the supplementary groups, and the LSM label. Restoring these in a user namespace requires the mapping described in Capability Sets and UID Mapping in Rootless Containers, and a checkpoint from a namespaced container restored into a differently mapped namespace has IDs that cannot be reproduced.

Signal dispositions and pending signals, the itimers timers including their remaining time, the interval timers, the rlimits, the CPU affinity mask, the scheduling policy and priority, and the cgroup membership with its limits. None of these live in the process’s address space, and none of them are recreated by execve.

Threads. A process with threads has one task per thread, each with its own register state and its own stack, sharing the address space. Restoring requires creating the threads with the same clone flags and then setting each thread’s registers to where they were. The registers are read with ptrace after stopping each task, which is why a checkpointing tool is a debugger as well as a reader: it uses PTRACE_ATTACH or PTRACE_SEIZE, PTRACE_GETREGS, and PTRACE_CONT, and the same mechanism gives it the ability to read memory via PTRACE_PEEKDATA.

Restoring into a new kernel

Restore inverts each step. The tree is rebuilt by creating processes, and because the checker must create tasks that look like the original ones, the ordering is constrained: the namespace setup must happen first, since PID 1 of a new PID namespace must be created before anything that refers to it, and the mapping files must be written before capabilities are applied.

The restored processes re-enter user space at the instruction after the syscall or signal that was interrupted, with registers set from the image. This is where the technique’s limits show.

An interrupted syscall in progress cannot generally be resumed. If a task was blocked in read on a socket, the kernel has no way to re-enter the same blocking state on a different socket with the same semantics; the restore either completes the syscall with the data it has or restarts it. Some calls are marked restartable and are re-executed after the signal return; others are not.

The process ID may differ. A checkpoint records the PIDs, and restore can attempt to reproduce them only if they are free. In a new PID namespace this is easy, since the namespace is empty and PIDs 1 upward are available. Restoring into the host PID namespace requires the original PIDs to be unoccupied, and a collision means either failing or accepting different PIDs, which breaks any state that refers to a process by number: a thread ID in a futex, a PID in a kill argument stored in a variable, a PID in a lock file.

Kernel-internal state that was not enumerated is simply absent. A task sleeping in a filesystem lock, a task waiting on an inotify event, a task inside an epoll wait with a large watch list, a task blocked on a futex whose memory is shared with a process outside the tree: all of these restore into a process that is runnable but not in the state it was in. Whether that is fatal depends on the program. A server that was in accept simply accepts again; a program in the middle of a lock protocol may have a peer that has moved on.

What makes a workload checkpointable

The properties that keep a workload checkpointable are properties of its design rather than of the tooling.

State that exists only in file descriptors, pipes, and sockets is fragile. State written to files with fsync before the checkpoint is durable, because it survives as contents rather than as kernel bookkeeping. A workload that can be asked to flush and close its connections before a freeze, and resume by reopening them, survives a checkpoint across hosts and kernel versions; one that holds a million open connections cannot.

Identity claims must be re-establishable. A process that cached its hostname, its IP address, or a peer’s address and refuses to run when they differ will fail on restore into a new network namespace. Reading those values after restore, rather than caching them, is what permits migration.

Timers must be able to absorb a pause. A checkpoint taken while a task waits on a timerfd with 50 ms remaining will restore with that remaining time, so the wall-clock delay is invisible to the timer. Absolute deadlines held in application memory are not adjusted, and a workload that computes a deadline once and compares it against a clock afterwards will appear to have lost the intervening time.

And the kernel version matters, in a way that cannot be worked around. Kernel structures change between versions, so a checkpoint written against one kernel’s understanding of a socket’s internal state is not restorable on a kernel that changed it. In-practice checkpoint images are treated as valid for a limited window of kernel versions, and migration across a version boundary is a testing exercise rather than a configuration flag. The image format is a serialisation of kernel state, so its compatibility is bounded by the kernel’s, which is the least documented and most restrictive constraint in the whole mechanism.

The practical reading is that checkpoint and restore is an exact operation on a subset of process state and an approximation on the rest. The subset is large enough for computation-bound and file-state workloads; the approximation is why long-lived connection state and deep kernel waits are the parts that break.