> For the complete documentation index, see [llms.txt](https://docs.cedana.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cedana.ai/cedana-kubernetes/storage/where-we-checkpoint-to.md).

# Where We Checkpoint To

Simple, anywhere that's POSIX complaint. Our helm charts come configured to write to local storage at first so there's no issues. 

### CSX (Cedana Storage Express)

CSX is Cedana's custom NFS-backed storage solution to maximize checkpoint and restore speeds. It uses a combination of tmpfs and local disk to effectively cache and stage checkpoints. CSX is itself a systemd daemon that communicates directly with the cedana daemon and runs on each node. 

<figure><img src="https://3970289663-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2VUqakyWqaX9NCnQNYjD%2Fuploads%2FI1J8GWOvole10i9F0isr%2Fimage.png?alt=media&amp;token=30e7e2d5-76f5-437e-95f2-9213fff8c073" alt=""><figcaption></figcaption></figure>

Here's the minimal config you can get started with:

```
--set config.checkpointDir=csx:// \
--set csx.nfsPath=$NFS_DIR
```

CSX expects `$NFS_DIR` to be available at the same path on all nodes in the cluster and for it to be the only program that modifies the contents of that directory.

CSX always checkpoints to `tmpfs` to reduce application freeze times and checkpoint time, after which it ensures that these checkpoints are persisted to NFS.

You can customize this behavior by modifying the following:

```
// Path where CSX temporarily stores objects while they are being written or
// have not yet been copied to NFS. CSX creates a tmpfs at this path.
--set csx.tmpDir=$CSX_TMP_DIR \
// Maximum number of checkpoints allowed in tmpDir at once. Once reached, new
// checkpoints are written straight to NFS. Setting to 0 automatically checkpoints
// to NFS. Default is 1.
--set csx.tmpDirObjectLimit=1
```

You can customize the amount of RAM (used by `tmpfs` ) and local disk space used to cache checkpoints and where these caches are created.

<pre><code><strong>// Percentage of system RAM CSX can use to cache checkpoints (1-100).
</strong>// CSX creates a tmpfs of this size at memoryCachePath. default is 25.
--set csx.memoryCachePercent=25 \
<strong>// Amount of local disk space, in GiB, CSX can use to cache checkpoints.
</strong><strong>// default is 1000.
</strong>--set csx.localDiskCacheLimit=1000 \
// Path on the local disk where CSX caches checkpoints.
// CSX creates the path if it does not exist.
--set csx.localDiskCachePath=/cedana/diskstore/ \
// Path where CSX mounts the tmpfs used to cache checkpoints in RAM.
// CSX creates the path if it does not exist.
--set csx.memoryCachePath: /cedana/memstore/
</code></pre>

Take a look at [Configuration](/cedana-kubernetes/configuration.md) for more details.

### Object Storage

You can configure the helm chart to checkpoint to any S3-compatible bucket. Here's an example:

<pre><code><strong>--set config.checkpointDir=s3://$BUCKET_NAME \
</strong><strong>--set config.awsAccessKeyId=$AWS_ACCESS_KEY_ID \ 
</strong>--set config.awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY \
--set config.awsRegion=$PROVIDER_REGION \ 
--set config.awsEndpoint="https://storage.eu-north1.nebius.cloud"
</code></pre>

We also provide an option to checkpoint to a bucket managed by us:

```
--set config.checkpointDir=cedana://$BUCKET_NAME
```

### Local Storage

Modify the local path Cedana writes to by changing `checkpointDir`

```
// default is /tmp
--set config.checkpointDir=/checkpoints/
```