# Storage

There are two aspects to storage we cover here - ***where*** we checkpoint to, and ***what*** we checkpoint.

## Where we checkpoint *to* 

Simple, anywhere that's POSIX complaint. Our helm charts come configured to write to local storage at first so there's no issues, but while setting up Cedana you can configure a folder that's mounted in on a NAS or even an S3-compatible bucket. Here's an example: <br>

```
--set config.awsAccessKeyId=$AWS_ACCESS_KEY_ID \ 
--set config.awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY \
--set config.awsRegion=$PROVIDER_REGION \ 
--set config.awsEndpoint="https://storage.eu-north1.nebius.cloud"
```

Note that in the above case, it's not sending to S3 but an S3-compatible bucket hosted on another cloud provider (Nebius).&#x20;

## What *else* we checkpoint

While we can move the state of your process (including both CPU and GPU state), there should be some careful consideration with open files.&#x20;

Given this, we have three ways with which we currently deal with files that are being written to (as a restored or migrated process *expects* the file to have the exact same size so it can pick up where it left off). There are 2 scenarios we recommend for 2 different filesystem-writing regimes:&#x20;

* files <\~ 1GB: *Default Behavior*
* files >> 1GB: *Volume snapshotting*

Each scenario is described below.&#x20;

## Default Behavior&#x20;

If you're writing files into the rootfs of the container, we just take them with us on the snapshot! We perform a diff of the filesystem and add it to our process/system-level checkpoint. This includes JIT compiled files, intermediate files created during install and more.&#x20;

However, when files become very large >> 1GB, and persistence is necessary (if the files themselves are outputs of the run, like in physical simulations for example), we recommend using volumes.

## Volume Snapshotting

{% hint style="info" %}
This is still a very early work in progress! Please reach out to us if you're planning on using this.&#x20;
{% endhint %}

The alternate method requires coordination with your CSI driver. We take advantage of the snapshotting primitives already present (<https://kubernetes.io/docs/concepts/storage/volume-snapshots/>), and take a reference to these with us; so when we restore, we restore from a Kubernetes Volume Snapshot.