Where We Checkpoint To

Simple, anywhere that's POSIX complaint. Our helm charts come configured to write to local storage at first so there's no issues.

CSX (Cedana Storage Express)

CSX is Cedana's custom NFS-backed storage solution to maximize checkpoint and restore speeds. It uses a combination of tmpfs and local disk to effectively cache and stage checkpoints. CSX is itself a systemd daemon that communicates directly with the cedana daemon and runs on each node.

Here's the minimal config you can get started with:

--set config.checkpointDir=csx:// \
--set csx.nfsPath=$NFS_DIR

CSX expects $NFS_DIR to be available at the same path on all nodes in the cluster and for it to be the only program that modifies the contents of that directory.

CSX always checkpoints to tmpfs to reduce application freeze times and checkpoint time, after which it ensures that these checkpoints are persisted to NFS.

You can customize this behavior by modifying the following:

// Path where CSX temporarily stores objects while they are being written or
// have not yet been copied to NFS. CSX creates a tmpfs at this path.
--set csx.tmpDir=$CSX_TMP_DIR \
// Maximum number of checkpoints allowed in tmpDir at once. Once reached, new
// checkpoints are written straight to NFS. Setting to 0 automatically checkpoints
// to NFS. Default is 1.
--set csx.tmpDirObjectLimit=1

You can customize the amount of RAM (used by tmpfs ) and local disk space used to cache checkpoints and where these caches are created.

// Percentage of system RAM CSX can use to cache checkpoints (1-100).
// CSX creates a tmpfs of this size at memoryCachePath. default is 25.
--set csx.memoryCachePercent=25 \
// Amount of local disk space, in GiB, CSX can use to cache checkpoints.
// default is 1000.
--set csx.localDiskCacheLimit=1000 \
// Path on the local disk where CSX caches checkpoints.
// CSX creates the path if it does not exist.
--set csx.localDiskCachePath=/cedana/diskstore/ \
// Path where CSX mounts the tmpfs used to cache checkpoints in RAM.
// CSX creates the path if it does not exist.
--set csx.memoryCachePath: /cedana/memstore/

Take a look at Configuration for more details.

Object Storage

You can configure the helm chart to checkpoint to any S3-compatible bucket. Here's an example:

--set config.checkpointDir=s3://$BUCKET_NAME \
--set config.awsAccessKeyId=$AWS_ACCESS_KEY_ID \ 
--set config.awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY \
--set config.awsRegion=$PROVIDER_REGION \ 
--set config.awsEndpoint="https://storage.eu-north1.nebius.cloud"

We also provide an option to checkpoint to a bucket managed by us:

--set config.checkpointDir=cedana://$BUCKET_NAME

Local Storage

Modify the local path Cedana writes to by changing checkpointDir

// default is /tmp
--set config.checkpointDir=/checkpoints/