Redis Save Migrate & Resume (SMR) on Kubernetes

Performing a save/migrate/resume of an in-cluster redis db.

Setup

Let's first set up a sample redis database on k8s. Create a namespace called cedana-examples and install redis-example.yaml , which can be found here: https://github.com/cedana/cedana-helm-charts/tree/main/examples.

kubectl create namespace cedana-examples
kubectl apply -f redis-example.yaml

Now let's port-forward the service on 127.0.0.1 and connect to the database:

kubectl port-forward service/redis -n cedana-examples 6379:6379
redis-cli -h 127.0.0.1

Lets store some data in the db:

HSET 'user:001' first_name 'John' last_name 'doe' dob '12-JUN-1970'
HSET 'user:002' first_name 'David' last_name 'Bloom' dob '03-MAR-1981'

Listing all the pods in the cedana-examples namespace using the /list endpoint gives us all the information we need to do a checkpoint and restore.

curl -X GET -H 'Content-Type: application/json' -d '{
  "root": "'$ROOT'"
}' $CONTROLLER_URL:1324/list/cedana-examples

Great! Now its time to checkpoint the container. Lets set the necessary environment variables before we proceed, which were present in the output of the /list endpoint. The following variables should work on most default containerd clusters.

export CHECKPOINT_CONTAINER=redis \
export CHECKPOINT_SANDBOX=redis-6b5bcbb6b6-tdb4p \
export RESTORE_CONTAINER=redis-restore \
export RESTORE_SANDBOX=redis-restore-c6c794b64-h7ccs \
export NAMESPACE=cedana-examples \
export CONTROLLER_URL=localhost \
export ROOT=/run/containerd/runc/k8s.io \
export CHECKPOINT_PATH=/tmp/ckpt-redis

Checkpoint

curl -X POST -H "Content-Type: application/json" -d '{
  "checkpoint_data": {
    "container_name": "'$CHECKPOINT_CONTAINER'",
    "sandbox_name": "'$CHECKPOINT_SANDBOX'",
    "namespace": "'$NAMESPACE'",
    "checkpoint_path": "'$CHECKPOINT_PATH'",
    "root": "'$ROOT'"
  }
}' http://$CONTROLLER_URL:1324/checkpoint

Restore

Once this completes, we need a new pod to restore the redis checkpoint into - effectively acting as a sleeping pod that we morph into a restored redis pod. We do this to avoid colliding with Kubernetes' scheduling and state management, so the users have full control over when pods are being restored.

k apply -f redis-restore-example.yaml

Then, performing a restore is as simple as another curl call:

curl -X POST -H "Content-Type: application/json" -d '{
  "checkpoint_data": {
    "container_name": "'$RESTORE_CONTAINER'",
    "sandbox_name": "'$RESTORE_SANDBOX'",
    "namespace": "'$NAMESPACE'",
    "checkpoint_path": "'$CHECKPOINT_PATH'",
    "root": "'$ROOT'"
  }
}' http://$CONTROLLER_URL:1324/restore

Finally connect to redis-cli once again to check if the new restored container has the previously set data:

HGETALL user:001
HGETALL user:002