> For the complete documentation index, see [llms.txt](https://docs.cedana.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cedana.ai/cedana-slurm/automation.md).

# Automatic Checkpoint/Restore

In most cases, the end user will likely *not* be using the UI to manually checkpoint/restore workloads. Cedana offers seamless automation for SLURM workloads.

## Preemption-based

Let's deploy a sample SLURM job and cause a preemption to test out Cedana's preemption-based automation. In short, a job will be automatically checkpointed on preemption and resumed when it starts running again after being re-queued by SLURM (if configured to).

{% hint style="info" %}
Below example assumes your SLURM cluster/partition is configured with `PreemptMode=REQUEUE` and `PreemptType=preempt/partition_prio`. Check out [SLURM preemption](https://slurm.schedmd.com/preempt.html) for more information.
{% endhint %}

For simplicity, we have two partitions `debug` and `high` overlapping on the same compute node. `high` has higher priority tier than `debug`. The configuration looks like:

```
PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
SchedulerParameters=preempt_reorder_count=100,preempt_strict_order

PartitionName=debug Nodes=compute-01 Default=YES MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=1 GraceTime=15
PartitionName=high  Nodes=compute-01 MaxTime=INFINITE State=UP PreemptMode=REQUEUE PriorityTier=2
```

{% hint style="success" %}
`GraceTime` of 15 seconds on the `debug` partition allows Cedana plenty of time to perform a checkpoint on the job before it gets completely evicted/killed.
{% endhint %}

### Deploy victim

Below is an example CPU workload that we will use as our preemption victim. By default, Cedana must be enabled to checkpoint this job by adding the `CEDANA_ENABLE=1` environment variable. Additionally, you must set `CEDANA_CHECKPOINT=<label name>` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

{% code title="victim.sbatch" lineNumbers="true" %}

```bash
#!/bin/bash
#SBATCH --job-name=victim                    # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=victim    # Label for all checkpoints

echo "Starting victim job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
```

{% endcode %}

Submit the victim to the `debug` (low priority) partition on your node using `sbatch`.

```shellscript
sbatch -p debug -w compute-01 victim.sbatch
```

### Deploy preemptor

The preemptor job can be any job submitted to the `high` partition (higher priority than `debug`) as long as it occupies all the CPUs available on the node. This ensures SLURM preempts any low partition jobs before deploying this job.

{% hint style="info" %}
Use `nproc` to find number of CPUs on your node.
{% endhint %}

{% code title="preemptor.sbatch" lineNumbers="true" %}

```shellscript
#!/bin/bash
#SBATCH --job-name=preemptor                 # Job name
#SBATCH --output=victim.out                  # Standard output log
#SBATCH --error=victim.err                   # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task

echo "Starting preemptor job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
```

{% endcode %}

Note that this job *does not* need any `CEDANA_` environment variable as we don't need it to be checkpointed.

Submit the preemptor to the `high` partition on your node using `sbatch`, ensuring it occupies all the CPUs available on the node. In this example, `compute-01` has 4 CPUs.

```shellscript
sbatch -p high -w compute-01 --cpus-per-task=4 preemptor.sbatch
```

You should be able to see that our victim job has been preempted and re-queued onto the node (in pending `PD` state), while our preemptor job is freely running. Check the queue using `squeue`.

```
JOBID PARTITION        NAME     USER ST       TIME  NODES NODELIST(REASON)
  182     debug      victim     root PD       0:00      1 (BeginTime)
  183      high   preemptor     root  R       0:02      1 compute-01
```

### Restore

Now let's restore our victim job. All we need to do is `scancel` our preemptor job or wait for it to finish running.

Cancel the higher priority job using `scancel`.

```shellscript
scancel 183
```

After some time, the victim job should resume running on the node. Check the output file `victim.out` to verify that it was resumed from where it was preempted!

{% hint style="warning" %}
If for some reason, a checkpoint could not be taken during preemption, the job should simply start fresh. This can happen if the `GraceTime` on the partition is not sufficient for the size of the job. Alternatively, any older available checkpoints will be restored. Check out [#policy-based](#policy-based "mention") automation for taking regular checkpoints.
{% endhint %}

## Policy-based

Let's deploy a sample SLURM job and explore Cedana's policy-based automation for checkpoint/restore.

### Deploy

Below is an example CPU workload that counts from 0 to 600, sleeping for 1 second between each count. By default, Cedana must be enabled to checkpoint this job by adding the `CEDANA_ENABLE=1` environment variable.

<pre class="language-bash" data-title="counting.sbatch" data-line-numbers><code class="lang-bash"><strong>#!/bin/bash
</strong>#SBATCH --job-name=counting                  # Job name
#SBATCH --output=counting.out                # Standard output log
#SBATCH --error=counting.err                 # Standard error log
#SBATCH --time=00:10:00                      # Time limit (hh:mm:ss)
#SBATCH --nodes=1                            # Run on 1 node
#SBATCH --ntasks=1                           # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1             # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=counting  # Label for all checkpoints

echo "Starting counter job on $(hostname)..."

# Loop from 0 to 600
for i in {0..600}
do
    echo "Counter: $i"
    sleep 1
done
</code></pre>

Additionally, you must set `CEDANA_CHECKPOINT=<label name>` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.

Submit the job using `sbatch`.

```bash
sbatch counting.sbatch
```

### Configure policy

You may now take [manual checkpoints](/cedana-slurm/cr.md) of this workload or configure an automation policy.&#x20;

1. Head over to the [SLURM Policies Page](https://staging.cedana.com/slurm/policies) and click on "Create Policy".<br>

   <figure><img src="/files/NQIv1Dk08QBnXueDSG4c" alt=""><figcaption></figcaption></figure>
2. Set a unique name for this policy and select a checkpoint interval.
3. Select the job to apply this policy to and click on "Create Policy".

You should start seeing new checkpoints of your job on the [SLURM Checkpoints Page](https://ui.cedana.com/slurm/checkpoints).

### Restore

As soon as a checkpoint is available, you should be able to cancel your job using `scancel`.&#x20;

```bash
scancel <jobid>
```

{% hint style="info" %}
Cancellation should also trigger an automatic checkpoint if the `GraceTime` in your SLURM configuration allows sufficient time for a checkpoint to be taken.
{% endhint %}

Simply submit the job again using `sbatch`. You will notice that the job will automatically resume from the latest checkpoint instead of starting fresh.