Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Checkpoint/restore podman

Prerequisites

  1. Create an account with Cedana, to get access to the podman plugin. See authentication.

  2. Set the Cedana URL & authentication token in the configuration.

  3. Install the podman plugin with sudo cedana plugin install podman.

  4. Install the oci-runtime plugin with sudo cedana plugin install oci-runtime. This is the OCI runtime podman uses for containers run or restored by Cedana (see How it works).

  5. Install the lower-level plugin for podman's OCI runtime (see Supported runtimes):

    • If the runtime is crun (default), install the crun plugin with sudo cedana plugin install crun.

    • If the runtime is runc, install the runc plugin with sudo cedana plugin install runc.

  6. Ensure the daemon is running, see installation.

  7. Do a health check to ensure the plugin is ready, see health checks.

Only rootful podman is supported.

Basic

  1. Run a new podman container, for example:

sudo podman run -d --name <container> docker.io/library/nginx:latest
  1. Checkpoint:

  1. Restore, as a new container (use --id to name it):

The restored container is a regular podman container, so podman ps/logs/exec/stop etc. work on it as usual.

Managed

  1. Run a new managed podman container:

  1. Checkpoint:

  1. Restore:

Existing (or upcoming) podman containers can also be managed with cedana manage podman <container>, see managed process/container.

GPU support

Just like for processes, as explained in checkpoint/restore with GPUs, GPU support is also available for podman containers.

  1. Run a new managed podman container with GPU support:

--gpus adds the given GPUs to the container (e.g. --gpus 0,1), like podman's --gpus.

  1. Checkpoint:

  1. Restore:

The restored container gets the same GPUs as the job was run with.

GPU restore requires podman's overlay storage driver (the default). CDI injects NVIDIA's driver library symlinks into the container's read/write layer when it's created, and they are only restored with it (see read/write layer).

For GPU containers run directly with podman, see Using the Cedana OCI runtime directly.

Using the Cedana OCI runtime directly

Instead of running containers through Cedana (cedana run podman), you can keep using podman as usual and just switch its OCI runtime to the Cedana OCI runtime (cedana-oci-runtime, from the oci-runtime plugin). Such containers can be checkpointed and restored, including GPU containers, and remain regular podman containers in every other way.

Containers run with podman's default runtime (crun/runc) can also be checkpointed as in basic, but only containers run with the Cedana OCI runtime get GPU support, as GPU interception must be set up when the container is created.

Setup

Add the runtime to podman's containers.conf (e.g. /etc/containers/containers.conf):

Then use --runtime cedana with podman run/podman create. Alternatively, pass the full path every time: --runtime /usr/local/bin/cedana-oci-runtime. To use it for all containers, also set runtime = "cedana" under [engine].

Run

The container runs on the real runtime (crun, or runc if crun is not installed). To choose one, add --annotation cedana.ai/runtime=runc (see annotations).

Checkpoint

The container exits after the checkpoint, unless --leave-running is passed.

Restore

Create a container from the same image with the cedana.ai/checkpoint-path annotation:

Instead of starting the image's command, the Cedana OCI runtime restores the checkpointed process and read/write layer into the new container.

GPU containers

GPU support is enabled automatically for containers with NVIDIA GPUs, i.e. with NVIDIA_VISIBLE_DEVICES set, as in CUDA images. For other images, set it with --env NVIDIA_VISIBLE_DEVICES=all, or add --annotation cedana.ai/gpu=true. See checkpoint/restore with GPUs for GPU prerequisites.

  1. Run:

  1. Checkpoint:

  1. Restore:

A checkpoint does not record the container's GPUs, so pass --gpus again on restore. The checkpointed GPU state is restored onto the given GPUs.

Instead of podman's --gpus (podman 5.0+), GPUs can also be added with CDI, e.g. --device nvidia.com/gpu=<N>, for both run and restore.

How it works

Checkpointing is delegated to the plugin of the container's OCI runtime, as a rootful podman container is just a regular crun (or runc) container.

Running and restoring however must go through podman, as podman owns the container's sandbox: its network namespace, rootfs mount, and the conmon process that monitors it. For this, the container is created with the Cedana OCI runtime (cedana-oci-runtime, from the oci-runtime plugin), which podman invokes in place of crun/runc. It routes the container creation to cedana run <runtime>, or cedana restore <runtime> for a restore, and passes every other command through to the real runtime. This is podman's equivalent of the Cedana containerd runtime (shim).

Annotations

The Cedana OCI runtime reads the following container annotations. The podman plugin sets them, and they can also be set directly with podman run --annotation <key>=<value>. JSON values may be base64-encoded, as some podman versions split --annotation values on commas.

Annotation
Description

cedana.ai/runtime

The real runtime (crun or runc). Default: crun if installed, else runc

cedana.ai/checkpoint-path

Restore from this checkpoint, instead of creating

cedana.ai/criu-opts

CRIU options JSON for the restore

cedana.ai/config

Cedana configuration JSON to run Cedana with

cedana.ai/gpu

Enable GPU interception. Automatic if NVIDIA_VISIBLE_DEVICES is set

cedana.ai/gpu-tracing

Enable GPU tracing

cedana.ai/gpu-id

GPU controller to use, if already attached

cedana.ai/executable-path

Cedana binary to use. Set by the podman plugin (disables automatic GPU interception, which the plugin handles itself)

Supported runtimes

Runtime
Required plugins

crun

crun, oci-runtime

runc

runc, oci-runtime

The runtime is detected per-container. A checkpoint records the container's runtime, and a restore recreates the container with the same one.

Read/write layer (rootfs)

Checkpoints automatically include the container's read/write layer (rootfs) in the dump, if podman's storage driver is overlay (the default). Any changes made to the container's rootfs before checkpointing will be preserved and restored. For other storage drivers (e.g. vfs), the read/write layer is skipped gracefully.

For all available CLI options, see CLI reference. Directly interacting with daemon is also possible through gRPC, see API reference.

Last updated

Was this helpful?