Checkpoint/restore with GPUs

Checkpoint/restore with GPUs is currently only supported for NVIDIA GPUs.

Prerequisites

1. Create an account with Cedana, to get access to the GPU plugin. See authentication.

2. Set the Cedana URL & authentication token in the configuration.

3. Install a GPU plugin.

• Option 1: GPU Plugin

  sudo cedana plugin install gpu

  The GPU plugin is Cedana's proprietary plugin for high performance GPU checkpoint/restore. If unavailable to you, check option 2.

  • Minimum NVIDIA driver version: 452 (API 11.8)

  • Maximum NVIDIA driver version: 550 (API 12.4). Newer drivers are unstable and may not work.

  • Minimum CRIU version: 3.0

• Option 2: CRIU CUDA Plugin

  sudo cedana plugin install criu/cuda

  • Minimum NVIDIA driver version: 570 (API 12.8)

  • Minimum CRIU version: 4.0

  Check out Cedana vs. CRIU CUDA for GPU Checkpoint/Restore for a performance comparison between the two plugins.

1. Ensure the daemon is running, see installation.

2. Do a health check to ensure the plugin is ready, see health checks.

Usage (GPU plugin)

NOTE: Cedana GPU checkpoint/restore is only possible for managed processes/containers, i.e., those that are spawned using cedana run --gpu-enabled (see managed process/container).

1. You may clone the cedana-samples repository for some example GPU workloads.

2. Run a process with GPU support:

cedana run process --attach --gpu-enabled --jid <job_id> -- cedana-samples/gpu_smr/vector_add

1. Checkpoint:

cedana dump job <job_id>

1. Restore:

cedana restore job --attach <job_id>

Usage (CRIU CUDA plugin)

You can checkpoint/restore normally as you do for CPU workloads. See checkpoint/restore basics.

For all available CLI options, see CLI reference. Directly interacting with daemon is also possible through gRPC, see API reference.