Automatic Checkpoint/Restore

Automating checkpoint/restore with SLURM

In most cases, the end user will likely not be using the UI to manually checkpoint/restore workloads. Cedana offers seamless automation for SLURM workloads.

Preemption-based

Let's deploy a sample SLURM job and cause a preemption to test out Cedana's preemption-based automation. In short, a job will be automatically checkpointed on preemption and resumed when it starts running again after being re-queued by SLURM (if configured to).

Below example assumes your SLURM cluster/partition is configured with PreemptMode=REQUEUE and PreemptType=preempt/partition_prio. Check out SLURM preemption for more information.

For simplicity, we have two partitions debug and high overlapping on the same compute node. high has higher priority tier than debug. The configuration looks like:

PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
SchedulerParameters=preempt_reorder_count=100,preempt_strict_order

PartitionName=debug Nodes=compute-01 Default=YES MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=1 GraceTime=15
PartitionName=high  Nodes=compute-01 MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=2

GraceTime of 15 seconds on the debug partition allows Cedana plenty of time to perform a checkpoint on the job before it gets completely evicted/killed.

Deploy victim

Below is an example CPU workload that we will use as our preemption victim. By default, Cedana must be enabled to checkpoint this job by adding the CEDANA_ENABLE=1 environment variable. Additionally, you must set CEDANA_CHECKPOINT=<label name> to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

victim.sbatch

#!/bin/bash
#SBATCH --job-name=victim                    # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=victim    # Label for all checkpoints

echo "Starting victim job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done

Submit the victim to the debug (low priority) partition on your node using sbatch.

sbatch -p debug -w compute-01 victim.sbatch

Deploy preemptor

The preemptor job can be any job submitted to the high partition (higher priority than debug) as long as it occupies all the CPUs available on the node. This ensures SLURM preempts any low partition jobs before deploying this job.

Use nproc to find number of CPUs on your node.

preemptor.sbatch

#!/bin/bash
#SBATCH --job-name=preemptor                 # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task

echo "Starting preemptor job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done

Note that this job does not need any CEDANA_ environment variable as we don't need it to be checkpointed.

Submit the preemptor to the high partition on your node using sbatch, ensuring it occupies all the CPUs available on the node. In this example, compute-01 has 4 CPUs.

sbatch -p high -w compute-01 --cpus-per-task=4 preemptor.sbatch

You should be able to see that our victim job has been preempted and re-queued onto the node (in pending PD state), while our preemptor job is freely running. Check the queue using squeue.

JOBID PARTITION        NAME     USER ST       TIME  NODES NODELIST(REASON)
  182     debug      victim     root PD       0:00      1 (BeginTime)
  183      high   preemptor     root  R       0:02      1 compute-01

Restore

Now let's restore our victim job. All we need to do is scancel our preemptor job or wait for it to finish running.

Cancel the higher priority job using scancel.

scancel 183

After some time, the victim job should resume running on the node. Check the output file victim.out to verify that it was resumed from where it was preempted!

If for some reason, a checkpoint could not be taken during preemption, the job should simply start fresh. This can happen if the GraceTime on the partition is not sufficient for the size of the job. Alternatively, any older available checkpoints will be restored. Check out Policy-based automation for taking regular checkpoints.

Policy-based

Let's deploy a sample SLURM job and explore Cedana's policy-based automation for checkpoint/restore.

Deploy

Below is an example CPU workload that counts from 0 to 600, sleeping for 1 second between each count. By default, Cedana must be enabled to checkpoint this job by adding the CEDANA_ENABLE=1 environment variable.

counting.sbatch

#!/bin/bash
#SBATCH --job-name=counting                  # Job name
#SBATCH --output=counting.out                # Standard output log
#SBATCH --error=counting.err                 # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=counting  # Label for all checkpoints

echo "Starting counter job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done

Additionally, you must set CEDANA_CHECKPOINT=<label name> to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

Submit the job using sbatch.

sbatch counting.sbatch

Configure policy

You may now take manual checkpoints of this workload or configure an automation policy.

1. Head over to the SLURM Policies Page and click on "Create Policy".

   
2. Set a unique name for this policy and select a checkpoint interval.

3. Select the job to apply this policy to and click on "Create Policy".

You should start seeing new checkpoints of your job on the SLURM Checkpoints Page.

Restore

As soon as a checkpoint is available, you should be able to cancel your job using scancel.

scancel <jobid>

Cancellation should also trigger an automatic checkpoint if the GraceTime in your SLURM configuration allows sufficient time for a checkpoint to be taken.

Simply submit the job again using sbatch. You will notice that the job will automatically resume from the latest checkpoint instead of starting fresh.

Maintenance Windows

SLURM node reservations integrate natively with Cedana's preemption-based automation. To migrate jobs on a reserved node, create a SLURM node reservation.

scontrol create reservation ReservationName=maint_window \
  starttime=2026-01-01T02:00:00 duration=240 \
  flags=maint,ignore_jobs nodes=node[0-8] user=root \
  MaxStartDelay=5

Cedana's preemption-based automation will automatically handle the migration of the jobs to an available node or requeue to restore once a node becomes available.

Setup

To perform Cedana's preemption-based checkpoint and restore on SLURM maintenance windows, the slurm.conf must have JobRequeue=1. Otherwise, the drained jobs will only be checkpointed but not automatically restored.

slurm.conf

```ini ... JobRequeue=1 ... ```

In addition, all SLURM node reservations must have a MaxStartDelay=<minutes> to give the drained jobs enough time to checkpoint. The node reservation must also have flags=...,ignore_jobs to allow jobs that would exceed the reservation window to get provisioned and migrated after the reservation window is encountered.