# Storage

There are two aspects to storage we cover here - ***where*** we checkpoint to, and ***what*** we checkpoint.

## Where we checkpoint *to*

Simple, anywhere that's POSIX complaint. Our helm charts come configured to write to local storage at first so there's no issues, but while setting up Cedana you can configure a folder that's mounted in on a NAS or even an S3-compatible bucket. Here's an example:<br>

```
--set config.awsAccessKeyId=$AWS_ACCESS_KEY_ID \ 
--set config.awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY \
--set config.awsRegion=$PROVIDER_REGION \ 
--set config.awsEndpoint="https://storage.eu-north1.nebius.cloud"
```

Note that in the above case, it's not sending to S3 but an S3-compatible bucket hosted on another cloud provider (Nebius).

## What *else* we checkpoint

While we can move the state of your process (including both CPU and GPU state), there should be some careful consideration with open files.

Given this, we have three ways with which we currently deal with files that are being written to (as a restored or migrated process *expects* the file to have the exact same size so it can pick up where it left off). There are 2 scenarios we recommend for 2 different filesystem-writing regimes:

* files <\~ 1GB: *Default Behavior*
* files >> 1GB: *Volume snapshotting*

Each scenario is described below.

## Default Behavior

If you're writing files into the rootfs of the container, we just take them with us on the snapshot! We perform a diff of the filesystem and add it to our process/system-level checkpoint. This includes JIT compiled files, intermediate files created during install and more.

However, when files become very large >> 1GB, and persistence is necessary (if the files themselves are outputs of the run, like in physical simulations for example), we recommend using volumes.

## Volume Snapshotting

{% hint style="info" %}
This is still a very early work in progress! Please reach out to us if you're planning on using this.
{% endhint %}

The alternate method requires coordination with your CSI driver. We take advantage of the snapshotting primitives already present (<https://kubernetes.io/docs/concepts/storage/volume-snapshots/>), and take a reference to these with us; so when we restore, we restore from a Kubernetes Volume Snapshot.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cedana.ai/cedana-kubernetes/storage.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.