Installation

Begin checkpoint/migrate/restoring stateful workloads in SLURM in under 5 minutes!

You can install Cedana on a SLURM node in 3 ways:

1. [Option 1] Using web installer (sane defaults)

2. [Option 2] Using Cedana

3. [Option 3] Building from source

These steps must be performed on all SLURM controller and compute nodes in your cluster!

To use Cedana in Kubernetes, you need to be registered with us! Reach out to [email protected] to get set up with an organization.

You can also deploy fully self-hosted, with zero limitations on where you can store your checkpoints! Check out configuration. If you have any questions, please reach out to us at [email protected].

Using web installer

The web installer will automatically install the latest stable version of Cedana and all plugins required for SLURM support with sane defaults.

Install

Check Authentication for more details on how to get an authentication token.

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token

curl -fsSL ${CEDANA_URL}/install/slurm -H "Authorization: Bearer ${CEDANA_AUTH_TOKEN}" | sudo -E bash

• Use ?version=x.y.z query parameter to install a specific version.

• Use ?build=alpha&version=feat/my-branch to install an alpha build from a branch.

Configure

For configuration, follow instructions on Cedana Daemon configuration. Use the environment variable corresponding to a configuration item to override it when calling the installer script.

export CEDANA_CHECKPOINT_DIR=s3://my-checkpoints-bucket
export CEDANA_CHECKPOINT_STREAMS=4
export CEDANA_CHECKPOINT_COMPRESSION=gzip

curl -fsSL ${CEDANA_URL}/install/slurm -H "Authorization: Bearer ${CEDANA_AUTH_TOKEN}" | sudo -E bash

Using Cedana

You can also install SLURM support directly from Cedana.

Install

First, install Cedana by following instructions on Cedana Daemon installation.

Then, install the slurm plugin:

sudo cedana plugin install slurm
sudo cedana slurm setup

Installing the plugins directly

For deployments that require installing the plugin files manually, you can download the files directly.

First, download the Cedana binary.

curl -fsSL https://github.com/cedana/cedana/releases/latest/download/cedana-amd64.tar.gz | tar xzf - && chmod 755 cedana

To download the SLURM plugin binaries, use the Cedana binary. For configuration, follow instructions on Cedana Daemon configuration. Use the environment variable corresponding to a configuration item to override it when calling the installer script.

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token

sudo ./cedana plugin remove slurm slurm/wlm
sudo ./cedana plugin install slurm slurm/wlm

For nightly SLURM plugin binaries, use the alpha builds.

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token
export CEDANA_PLUGINS_BUILDS=alpha

sudo ./cedana plugin remove slurm slurm/wlm
sudo ./cedana plugin install slurm@main slurm/wlm@main

The installer will download the cedana-slurm binary to /usr/local/bin and the SLURM plugin files to /usr/local/lib. To install the files, transfer the files to the required directories.

# install to the worker nodes (slurmd), controller nodes (slurmctld), and the database node (slurmdbd)
sudo install /usr/local/bin/cedana-slurm <binary-directory>/cedana-slurm
sudo install /usr/local/lib/cli_filter_cedana.so <slurm-plugin-directory>/cli_filter_cedana.so
sudo install /usr/local/lib/job_submit_cedana.so <slurm-plugin-directory>/job_submit_cedana.so
sudo install /usr/local/lib/task_cedana.so <slurm-plugin-directory>/task_cedana.so
sudo install /usr/local/lib/spank_cedana.so <slurm-plugin-directory>/spank_cedana.so

Update the /etc/slurm/plugstack.conf to include the spank_cedana.so

+required <slurm-plugin-directory>/spank_cedana.so

Update the /etc/slurm/slurm.conf to include the plugins.

-TaskPlugin=task/affinity,task/cgroup
+TaskPlugin=task/affinity,task/cgroup,task/cedana
+CliFilterPlugins=cli_filter/cedana
+JobSubmitPlugins=job_submit/cedana

Reload the slurmctld and slurmd with

sudo systemctl restart slurmctld
sudo systemctl restart slurmd

On the database node (slurmdbd), start cedana-slurm.

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token

sudo cedana-slurm daemon

Configure

For configuration, follow instructions on Cedana Daemon configuration.

On all nodes, initialize the /etc/cedana/config.json with

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token

cedana version --init-config

On the slurmdbd node, if you are using systemd, create the service file

export LOG_PATH=/var/log/cedana-slurm.log
export SERVICE_FILE=/etc/systemd/system/cedana-slurm.service
export APP_PATH=/usr/local/bin/cedana-slurm

cat <<EOF | tee "$SERVICE_FILE" >/dev/null
[Unit]
Description=Cedana Daemon
[Service]
ExecStart=$APP_PATH daemon start
User=root
Group=root
Restart=no

[Install]
WantedBy=multi-user.target

[Service]
StandardError=append:$LOG_PATH
StandardOutput=append:$LOG_PATH
EOF

Privileged Mode (root)

In privileged mode, the checkpointing and restoring are done as root. Privileged mode requires no additional configuration.

Unprivileged Mode (user)

In unprivileged mode, the checkpointing and restoring are done as the job's user, i.e., the UID of the SLURM job performs the checkpoint and restore. This configuration is useful when the root is demoted for security purposes. For example, a NFS with root_squash requires unprivileged mode.

To enable unprivileged mode, initialize the /etc/cedana/config.json with

export CEDANA_URL=https://myorg.cedana.ai/v1
export CEDANA_AUTH_TOKEN=your_auth_token
export CEDANA_SLURM_UNPRIVILEGED=1

cedana version --init-config

In addition, the cedana-slurm, cedana, and criu binaries must have the required capabilities for users to perform checkpoint and restore.

setcap CAP_SYS_PTRACE,CAP_DAC_READ_SEARCH,CAP_CHECKPOINT_RESTORE+eip /usr/local/bin/criu
setcap CAP_SYS_PTRACE,CAP_DAC_READ_SEARCH,CAP_CHECKPOINT_RESTORE+eip /usr/local/bin/cedana
setcap CAP_SYS_PTRACE,CAP_DAC_READ_SEARCH,CAP_CHECKPOINT_RESTORE+eip /usr/local/bin/cedana-slurm

Building from source

Check make help for available build targets.

Build all binaries:

make all

By default, the binaries will be built using the cedana/cedana-slurm:build docker image.

These binaries are useless on their own. You need to install Cedana to use them.

First, install Cedana by following instructions on Cedana Daemon installation. Then, install the slurm plugin and set up SLURM support:

sudo cedana plugin install slurm
cd build && sudo cedana slurm setup

You need to be in the build directory for the cedana slurm setup command to work, as it needs to find the binaries you just built.

You're all set up! Let's checkpoint some workloads. Continue to Checkpoint/restore to get started.

Uninstall

To remove Cedana SLURM completely, run:

sudo cedana slurm destroy