Deploying Cedana on your cluster
Using helm chart
We provide a helm chart, to easily install our services on your cluster.
Currently the host images should be debian:bookworm
or ubuntu:22.04
and above.
Installation
Make sure you have the helm
tool installed.
# install cedanacontroller-system
helm install cedana oci://registry-1.docker.io/cedana/cedana-helm \
--create-namespace -n cedanacontroller-system
Using Cedana API
We provide endpoints for attaching and detaching our cluster services to kubernetes cluster.
When running Cedana against an EKS cluster, cgroupsv2 must be used by your container runtime and be installed on your AMI. The minimum Linux kernel version is 6.1.x. On AWS, Amazon Linux 2023 will meet this requirement.
Authentication
Create an account at https://auth.cedana.com/ui/registration
. Using your email and password, you can grab a session token using the following steps:
export LOGIN_URL=$(curl -s -X GET -H "Accept: application/json" 'https://auth.cedana.com/self-service/login/api' | jq -r '.ui.action')
Use this action flow URL to authenticate and grab a token:
export AUTH_TOKEN=$(curl -s -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d '{"identifier": "your-email", "password": "your-password", "method": "password"}' "$LOGIN_URL" | jq -r '.session_token')
This token is valid for 720 hours, and can be used to authenticate all requests to our services.
Bootstrapping
For a quick setup, set the following environment variables:
export CLUSTER_NAME=<cluster-name> \
export SA_NAME=<service-account-name> \
export SA_TOKEN_NAME=<service-account-token-name>
To get started, create a Service Account for Cedana to deploy the Cedana Binary onto your instances.
kubectl -n kube-system create serviceaccount $SA_NAME
Now, create a cluster role binding to make the service account cluster-admin
by applying the following:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: service-account-admin
subjects:
- kind: ServiceAccount
name: $SA_NAME
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
envsubst < cluster-role-binding.yaml | kubectl apply -f -
To obtain an authentication token, start by applying the following secret:
sa-secret.yamlapiVersion: v1
kind: Secret
metadata:
name: $SA_TOKEN_NAME
namespace: kube-system
annotations:
kubernetes.io/service-account.name: $SA_NAME
type: kubernetes.io/service-account-token
envsubst < sa-secret.yaml | kubectl apply -f -
We can now obtain the data needed to hit Cedana's attach to Kubernetes endpoint. Get the service account token:
export SA_TOKEN=$(kubectl get secret $SA_TOKEN_NAME -n kube-system -o jsonpath='{.data.token}' | base64 --decode)
Get the certificate of authority:
export SA_CERT=$(kubectl get secret $SA_TOKEN_NAME -n kube-system -o jsonpath='{.data.ca\.crt}')
Finally, get your cluster endpoint URL:
export CLUSTER_URL=$(aws eks describe-cluster --name $CLUSTER_NAME | jq -r ".cluster.endpoint")
We can now hit the attach to Kubernetes endpoint and deploy Cedana to your Kubernetes cluster:
curl -X POST -H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" -d '{
"server": "'$CLUSTER_URL'",
"token": "'$SA_TOKEN'",
"cert": "'$SA_CERT'",
"versions": { "controller_version": "latest", "binary_version": "latest"}
}' https://sandbox.cedana.ai/kubernetes/attach
Once Cedana is attached, a CustomResourceDefinition
called Cedana
and a Kubernetes Operator called Cedana_Controller
are deployed to your Kubernetes cluster. You can now create an instance of the Cedana
resource and conduct checkpoint/restore of containers in your cluster. The Cedana_Controller
pod also contains a REST service with the following endpoints:
Using the service
Now that the controller is setup you can use the service to perform checkpoint and restore.
Known Limitations
- Currently we don't support io_uring API checkpoint restore, consider checkpointing before creating and setting up urings.
- Currently we don't automatically detect and change behaviour of our checkpointing services, for example, for CRIO and Rootfs use the separately provided api endpoints.
- Restore requires the pod we restore to not be active. This generally means you should put the
restore pod to sleep using custom command and arg,
while true; do sleep infinity; done;
.
Checkpoint/Restore - REST Service
The Cedana REST Service provides a REST API for checkpointing and restoring containers in your Kubernetes cluster. The API runs concurrently with the Cedana Controller. Below are curl
commands illustrating the schema of the API. All curls are using the in-cluster IP of the cedanacontroller pod. In order to do out-of-cluster checkpoint and restore, you can expose the pod and create an external IP address with Kubernetes services:
export CEDANA_CONTROLLER=$(kubectl get pods -n cedanacontroller-system | grep cedanacontroller-controller-manager | awk '{print $1}')
kubectl port-forward $CEDANA_CONTROLLER -n cedanacontroller-system 1324:1324
where CEDANA_CONTROLLER
is the name of the cedana controller pod.
List Containers in Namespace
Choose root depending on container runtime:
- K8s
export ROOT=/run/containerd/runc/k8s.io
- K3s
export ROOT=/host/run/containerd/runc/k8s.io
- default
export ROOT=/run/runc
Set controller cluster URL (ex. localhost
) and namespace (ex. default
):
export CONTROLLER_URL=<controller_cluster_url> \
export NAMESPACE=<namespace>
List containers in a specific namespace by querying Kubernetes pods with specific labels:
curl -X GET -H 'Content-Type: application/json' -d '{
"root": "'$ROOT'"
}' http://$CONTROLLER_URL:1324/list/$NAMESPACE | jq
Response
- Returns JSON array containing a list of containers in the specified namespace.
Save container name and sandbox name for checkpointing and restoring:
export CHECKPOINT_CONTAINER=<checkpoint-container-name> \
export CHECKPOINT_SANDBOX=<checkpoint-sandbox-name> \
export RESTORE_CONTAINER=<restore-container-name> \
export RESTORE_SANDBOX=<restore-sandbox-name>
Checkpoint
Initiate a checkpoint for a container:
export CHECKPOINT_PATH=/tmp/ckpt-$(date +%s%N)
curl -X POST -H "Content-Type: application/json" -d '{
"checkpoint_data": {
"container_name": "'$CHECKPOINT_CONTAINER'",
"sandbox_name": "'$CHECKPOINT_SANDBOX'",
"namespace": "'$NAMESPACE'",
"checkpoint_path": "'$CHECKPOINT_PATH'",
"root": "'$ROOT'"
},
"leave_running": true
}' http://$CONTROLLER_URL:1324/checkpoint
Arguments:
container_name
: Name of the container to checkpoint.sandbox_name
: Name of the sandbox to checkpoint.namespace
: Namespace in which the container resides.checkpoint_path
: Optional directory to dump checkpoint into (empty for remote).root
: Container runtime being used.leave_running
: Optional flag to determine whether container should be terminated on checkpoint (defaultfalse
).
Response:
checkpoint_id
: A uuid that is associated with the checkpoint, used for remote restore.
Restore
Restore a container from a checkpoint:
curl -X POST -H "Content-Type: application/json" -d '{
"checkpoint_data": {
"container_name": "'$RESTORE_CONTAINER'",
"sandbox_name": "'$RESTORE_SANDBOX'",
"namespace": "'$NAMESPACE'",
"checkpoint_path": "'$CHECKPOINT_PATH'",
"root": "'$ROOT'"
}
}' http://$CONTROLLER_URL:1324/restore
Argument:
container_name
: Name of the container to restore into.sandbox_name
: Name of the sandbox to restore into.namespace
: Namespace in which the container resides.checkpoint_path
: Optional directory to restore checkpoint from (empty for remote).checkpoint_id
: Optional identifier to restore checkpoint from (empty for local).root
: Container runtime being used.
Response:
- Status Code: 200 OK
CRI-O Rootfs Checkpoint
The rootfs checkpoint endpoint is specific to snapshotting the read and write layers of a container in Kubernetes. We do an image diff given an original image ref to compare to and then a new image ref to commit the changes to. This checkpoint specifically supports the CRI-O runtime manager, soon this will be deprecated and a combined endpoint that is agnostic of your container runtime manager will be available. The image produced gets pushed to the registry prefixed in the image ref, as long as the credentials live on the node. The image can then be used in a new pod spec.
Here is an example checkpoint hitting our REST service on the cedanacontroller-manager :1324
curl -X POST -H "Content-Type: application/json" -d '{
"container_name": "'$CONTAINER_NAME'",
"sandbox_name": "'$SANDBOX_NAME'",
"namespace": "'$NAMESPACE'",
"image_ref": "'$IMAGE_REF'",
"new_image_ref": "'$NEW_IMAGE_REF'"
}' http://$CONTROLLER_URL:1324/checkpoint/rootfs/crio
Arguments:
container_name
: Name of the container to checkpoint.sandbox_name
: Name of the sandbox to checkpoint.namespace
: Namespace in which the container resides.address
: The containerd sock address located on your worker nodes.image_ref
: The reference ID of your image (ex. cedana/checkpoint:latest).
Response:
- Status Code: 200 OK
- Returns the same image reference used in the CRI-O content store.
Rootfs Checkpoint
The rootfs checkpoint endpoint is specific to snapshotting the read and write layers of a container in Kubernetes. We are able to do an image diff by way of the containerd image service. If you are using either containerd or CRI-O as your higher level runtime, both should be able to support our feature as both use the containerd content store.
Here is an example checkpoint hitting our REST service on the cedanacontroller-manager :1324
curl -X POST -H "Content-Type: application/json" -d '{
"container_name": "'$CONTAINER_NAME'",
"sandbox_name": "'$SANDBOX_NAME'",
"namespace": "'$NAMESPACE'",
"address": "'$ADDRESS'",
"image_ref": "'$IMAGE_REF'"
}' http://$CONTROLLER_URL:1324/checkpoint/rootfs
Arguments:
container_name
: Name of the container to checkpoint.sandbox_name
: Name of the sandbox to checkpoint.namespace
: Namespace in which the container resides.address
: The containerd sock address located on your worker nodes.image_ref
: The reference ID of your image (ex. cedana/checkpoint:latest).
Response:
- Status Code: 200 OK
- Returns the same image reference used in containerd content store.
Rootfs Restore
Restore a container from a checkpointed rootfs image:
curl -X POST -H "Content-Type: application/json" -d '{
"sandbox_name": "'$SANDBOX_NAME'",
"container_name": "'$CONTAINER_NAME'",
"namespace": "'$NAMESPACE'",
"address": "'$ADDRESS'",
"image_ref": "'$IMAGE_REF'"
}' http://$CONTROLLER_URL:1324/restore/rootfs
Arguments:
container_name
: Name of the container to restore into.sandbox_name
: Name of the sandbox to restore into.namespace
: Namespace in which the container resides.address
: The containerd sock address located on your worker nodes.image_ref
: The reference ID of your image (ex. cedana/checkpoint:latest).
Response:
- Status Code: 200 OK
Detach
To remove Cedana from your cluster, hit the DELETE
endpoint:
curl -X DELETE -H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" -d '{
"server": "'$CLUSTER_URL'",
"token": "'$SA_TOKEN'",
"cert": "'$SA_CERT'"
}' https://sandbox.cedana.ai/kubernetes/destroy