> For the complete documentation index, see [llms.txt](https://docs.cedana.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cedana.ai/cedana-slurm/automation.md).

# Automatic Checkpoint/Restore

Automating checkpoint/restore with SLURM

In most cases, the end user will likely *not* be using the UI to manually checkpoint/restore workloads. Cedana offers seamless automation for SLURM workloads.

## Preemption-based

Let's deploy a sample SLURM job and cause a preemption to test out Cedana's preemption-based automation. In short, a job will be automatically checkpointed on preemption and resumed when it starts running again after being re-queued by SLURM (if configured to).

{% hint style="info" %}
Below example assumes your SLURM cluster/partition is configured with `PreemptMode=REQUEUE` and `PreemptType=preempt/partition_prio`. Check out [SLURM preemption](https://slurm.schedmd.com/preempt.html) for more information.
{% endhint %}

For simplicity, we have two partitions `debug` and `high` overlapping on the same compute node. `high` has higher priority tier than `debug`. The configuration looks like:

```
PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
SchedulerParameters=preempt_reorder_count=100,preempt_strict_order

PartitionName=debug Nodes=compute-01 Default=YES MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=1 GraceTime=15
PartitionName=high  Nodes=compute-01 MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=2
```

{% hint style="success" %}
`GraceTime` of 15 seconds on the `debug` partition allows Cedana plenty of time to perform a checkpoint on the job before it gets completely evicted/killed.
{% endhint %}

### Deploy victim

Below is an example CPU workload that we will use as our preemption victim. By default, Cedana must be enabled to checkpoint this job by adding the `CEDANA_ENABLE=1` environment variable. Additionally, you must set `CEDANA_CHECKPOINT=<label name>` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

{% code title="victim.sbatch" lineNumbers="true" %}

```bash
#!/bin/bash
#SBATCH --job-name=victim                    # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=victim    # Label for all checkpoints

echo "Starting victim job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
```

{% endcode %}

Submit the victim to the `debug` (low priority) partition on your node using `sbatch`.

```shellscript
sbatch -p debug -w compute-01 victim.sbatch
```

### Deploy preemptor

The preemptor job can be any job submitted to the `high` partition (higher priority than `debug`) as long as it occupies all the CPUs available on the node. This ensures SLURM preempts any low partition jobs before deploying this job.

{% hint style="info" %}
Use `nproc` to find number of CPUs on your node.
{% endhint %}

{% code title="preemptor.sbatch" lineNumbers="true" %}

```shellscript
#!/bin/bash
#SBATCH --job-name=preemptor                 # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task

echo "Starting preemptor job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
```

{% endcode %}

Note that this job *does not* need any `CEDANA_` environment variable as we don't need it to be checkpointed.

Submit the preemptor to the `high` partition on your node using `sbatch`, ensuring it occupies all the CPUs available on the node. In this example, `compute-01` has 4 CPUs.

```shellscript
sbatch -p high -w compute-01 --cpus-per-task=4 preemptor.sbatch
```

You should be able to see that our victim job has been preempted and re-queued onto the node (in pending `PD` state), while our preemptor job is freely running. Check the queue using `squeue`.

```
JOBID PARTITION        NAME     USER ST       TIME  NODES NODELIST(REASON)
  182     debug      victim     root PD       0:00      1 (BeginTime)
  183      high   preemptor     root  R       0:02      1 compute-01
```

### Restore

Now let's restore our victim job. All we need to do is `scancel` our preemptor job or wait for it to finish running.

Cancel the higher priority job using `scancel`.

```shellscript
scancel 183
```

After some time, the victim job should resume running on the node. Check the output file `victim.out` to verify that it was resumed from where it was preempted!

{% hint style="warning" %}
If for some reason, a checkpoint could not be taken during preemption, the job should simply start fresh. This can happen if the `GraceTime` on the partition is not sufficient for the size of the job. Alternatively, any older available checkpoints will be restored. Check out [#policy-based](#policy-based "mention") automation for taking regular checkpoints.
{% endhint %}

## Policy-based

Let's deploy a sample SLURM job and explore Cedana's policy-based automation for checkpoint/restore.

### Deploy

Below is an example CPU workload that counts from 0 to 600, sleeping for 1 second between each count. By default, Cedana must be enabled to checkpoint this job by adding the `CEDANA_ENABLE=1` environment variable.

<pre class="language-bash" data-title="counting.sbatch" data-line-numbers><code class="lang-bash"><strong>#!/bin/bash
</strong>#SBATCH --job-name=counting                  # Job name
#SBATCH --output=counting.out                # Standard output log
#SBATCH --error=counting.err                 # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=counting  # Label for all checkpoints

echo "Starting counter job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
</code></pre>

Additionally, you must set `CEDANA_CHECKPOINT=<label name>` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

Submit the job using `sbatch`.

```bash
sbatch counting.sbatch
```

### Configure policy

You may now take [manual checkpoints](/cedana-slurm/cr.md) of this workload or configure an automation policy.

1. Head over to the [SLURM Policies Page](https://staging.cedana.com/slurm/policies) and click on "Create Policy".<br>

   <figure><img src="https://3970289663-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2VUqakyWqaX9NCnQNYjD%2Fuploads%2FHkwzNwWqBwmhOJeNDNDz%2Fimage.png?alt=media&amp;token=55f4ba82-d173-40a0-b817-79bd2705d52f" alt=""><figcaption></figcaption></figure>
2. Set a unique name for this policy and select a checkpoint interval.
3. Select the job to apply this policy to and click on "Create Policy".

You should start seeing new checkpoints of your job on the [SLURM Checkpoints Page](https://ui.cedana.com/slurm/checkpoints).

### Restore

As soon as a checkpoint is available, you should be able to cancel your job using `scancel`.

```bash
scancel <jobid>
```

{% hint style="info" %}
Cancellation should also trigger an automatic checkpoint if the `GraceTime` in your SLURM configuration allows sufficient time for a checkpoint to be taken.
{% endhint %}

Simply submit the job again using `sbatch`. You will notice that the job will automatically resume from the latest checkpoint instead of starting fresh.

## Maintenance Windows

SLURM node reservations integrate natively with Cedana's preemption-based automation. To migrate jobs on a reserved node, create a SLURM node reservation.

```bash
scontrol create reservation ReservationName=maint_window \
  starttime=2026-01-01T02:00:00 duration=240 \
  flags=maint,ignore_jobs nodes=node[0-8] user=root \
  MaxStartDelay=5
```

Cedana's preemption-based automation will automatically handle the migration of the jobs to an available node or requeue to restore once a node becomes available.

### Setup

To perform Cedana's preemption-based checkpoint and restore on SLURM maintenance windows, the `slurm.conf` must have `JobRequeue=1`. Otherwise, the drained jobs will only be checkpointed but not automatically restored.

{% code title="slurm.conf" lineNumbers="true" %}

```
```

{% endcode %}

\`\`\`ini ... JobRequeue=1 ... \`\`\`

In addition, all SLURM node reservations must have a `MaxStartDelay=<minutes>` to give the drained jobs enough time to checkpoint. The node reservation must also have `flags=...,ignore_jobs` to allow jobs that would exceed the reservation window to get provisioned and migrated after the reservation window is encountered.