> For the complete documentation index, see [llms.txt](https://docs.cedana.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cedana.ai/cedana-kubernetes/automation.md).

# Automatic Checkpoint/Restore

Adding automation to checkpoint/restore in Kubernetes

Cedana offers two distinct types of automation to best take advantage of our checkpoint/restore.

### Integration into Kubernetes

The first is simply our deep integration into Kubernetes. As users, you are likely to use tools such as Armada, Kueue, KServe, Dynamo, etc, that are come with their own opinions about container lifecycle, management, preemption and more.

As we work seamlessly with the scheduler, cedana automatically inserts the checkpoint if a pod (that had a checkpoint previously taken) auto-scales. For example, if using a tool like Karpenter, cedana will automatically restore from a checkpoint when the underlying node is preempted and a new node is spun up, *instead of starting from scratch.*

Our goal is to be as pain-free to integrate into your existing tooling as possible, and as such simply work with schedulers like the ones listed above. The best demonstration of this can be found in [Jobs](/cedana-kubernetes/jobs.md), which can more or less be applied universally to kubernetes kinds.

All you have to do is set the CEDANA\_CHECKPOINT variable to a unique ID (that will then be used to key in our backend for the latest checkpoint taken with that ID). A simple example: 

```
apiVersion: v1
kind: Pod
metadata:
  generateName: transformers-stablelm-
  namespace: default
spec:
  runtimeClassName: cedana 
  containers:
    - name: transformers-stablelm
      image: cedana/cedana-samples:cuda12.4-torch2.5
      command:
        - python3
        - -u
        - /app/gpu_smr/pytorch/llm/transformers_inference.py
        - --readiness-port=8888
        - --model=stabilityai/stablelm-2-1_6b
      ports:
        - containerPort: 8888
      env: 
        - name: CEDANA_CHECKPOINT 
          value: transformers-stablelm-1
      readinessProbe:
        tcpSocket:
          port: 8888
        initialDelaySeconds: 1
        periodSeconds: 1
        timeoutSeconds: 1
      resources:
        limits:
          nvidia.com/gpu: 1

```

Note the CEDANA\_CHECKPOINT key and value under env. Now, after a checkpoint has been taken (either through the UI, via a policy or through our [API](https://demo.cedana.ai/v1/scalar)) - you need to simply reapply the yaml, after which the next start triggers a restore instead of a cold start. 

At the node level, Cedana uses the `cedana.ai/not-ready` taint while a node is bootstrapping. That taint is removed only after both the `cedana-helper` and `cedana-health-check` pods are Ready, which prevents workloads from landing on a node before Cedana is ready to restore them.

If you need Cedana to preserve additional pod spec fields during checkpointing, or map an older field name into the saved spec, see [Pod Spec Whitelist and Remapping](/cedana-kubernetes/podspec-whitelist-remapping.md).

See our Managed Dynamo for an example with a more complex Kubernetes deployment.

### Policies

Through the dashboard, you can enable and set policies, that configure bits of automation like heartbeat checkpointing, which will act as a cron job and checkpoint every *n* minutes.

<figure><img src="https://3970289663-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2VUqakyWqaX9NCnQNYjD%2Fuploads%2FlIISCz0AGmgNTPizv945%2Fimage.png?alt=media&amp;token=d6a56f82-db42-436c-8a40-756002c04462" alt=""><figcaption></figcaption></figure>

Policies in cedana operate through a Trigger -> Filter -> Action mechanism, allowing users to build complex automations. For example, you could build:

* a policy that automatically fires a checkpoint if a pod exceeds 80% memory utilization in a specific namespace or deployment,
* a policy that deletes checkpoints past a certain age,
* a policy that checkpoints on some webhook

We aim for the policy engine to be extensible, so value feedback for more features! Policies work best when integrated into Kubernetes however, as the below section illustrates.