Deploying with Dynamo
Deploying Cedana + Dynamo
The good news is that Cedana just works with Dynamo! You can follow this guide on setting up Cedana on your Kubernetes cluster: Installation, and follow this to setup Dynamo on your cluster: https://docs.nvidia.com/dynamo/v1.0.1/kubernetes-deployment/deployment-guide.
Dynamo requires another controller from the cedana side, so in your helm values.yml you'll need to add the following config:
# Only enable on clusters that have NVIDIA Dynamo (DynamoGraphDeployment CRD) installed.
dynamo:
enabled: true
image:
repository: cedana/cedana-dynamo-watcher
tag: main
digest: # ignores tag if set
imagePullPolicy: IfNotPresent
resources: {}
tolerations:
- key: "cedana.ai/not-ready"
operator: "Exists"
effect: "NoSchedule"
affinity: {}
nodeSelector: {}
metricsService:
ports:
- name: https
port: 8443
protocol: TCP
targetPort: https
type: ClusterIP
Once set up, you can manage your dynamo deployment via our UI.
Disaggregated Prefill/Decode worker restores and scaling
Cedana can checkpoint a disaggregated prefill or decode worker once it has loaded and warmed its model, then restore new workers from that checkpoint instead of cold-starting them. Each role restores and scales from its own checkpoint, and one set of checkpoints can bring up workers for a whole new deployment.
Cedana works seamlessly with Dynamo's snapshot control file. Once you turn on the Cedana controller's P/D webhook, it sets snapshot mode up for you after you annotate the worker with cedana.ai/pd-disagg: prefill or cedana.ai/pd-disagg: decode.
For a diagram of checkpointing each role once and scaling it out, and for how this works under the hood, see How P/D Checkpoint/Restore Works.
Turn it on
The webhook is off by default. Turn it on in your Helm values:
or with --set controllerManager.pdDisaggWebhook.enabled=true on helm install or helm upgrade. There is nothing else to install.
The controller creates the webhook's cedana-pd-webhook Service, its self-signed serving cert and the cedana-pd-disagg MutatingWebhookConfiguration itself, and the chart grants it the access it needs. Setting the value back to false removes the webhook on the controller's next start.
Mark the workers
Run the prefill and decode pods under the Cedana runtime and annotate them:
The annotations are:
cedana.ai/pd-disagg
Marks the pod as a P/D worker. The value (prefill or decode) is informational. false turns it off.
unset
cedana.ai/pd-disagg-container
Name of the container to set up. Without it, every container in the pod is set up.
unset
cedana.ai/pd-disagg-framework
The inference engine: vllm or sglang.
vllm
When the pod is created, the Cedana controller adds:
Dynamo snapshot mode:
DYN_SNAPSHOT_CONTROL_DIRand anemptyDirfor it. In snapshot mode the worker loads and warms the model, then pauses without registering with Dynamo.Cedana's patch for the engine, mounted at
/opt/cedana-pd-disaggfrom thecedana-pd-disaggConfigMap, which the controller creates in the pod's namespace, and put first onPYTHONPATH. It closes the engine's NIXL and Mooncake connections for the snapshot and reopens them after a restore.CEDANA_SNAPSHOT_KEEP_GPU_MEMORY=1, so weights, KV cache and NIXL registrations stay on the GPU through the pause.CEDANA_PERSISTENT_MOUNTS=/dev/shm, so/dev/shmis carried through the checkpoint.CEDANA_GPU_UCX_CR=1, so the engine's UCX state (NIXL's KV transfers) is proxied and can be checkpointed. UCX is passed through unproxied in pods without it. This needs cedana-gpu v0.8.9 or later; earlier versions readCEDANA_UCX_CRinstead, so set that yourself in the worker spec if you run one.
Anything you already set in the pod spec is kept.
Checkpoint
Request the checkpoint whenever you like, through the UI or the API, even while the worker is still loading. Cedana waits until the worker has paused and is ready for a snapshot, checkpoints it, and then tells it to resume: the original worker keeps running, registers with Dynamo and starts serving.
Restore and scale
Restores work as for any Cedana workload (see Jobs): give each worker container a checkpoint name with CEDANA_CHECKPOINT, one name per role.
Checkpoints of that container are recorded under the name. A pod with CEDANA_CHECKPOINT restores from the latest checkpoint under its name, and starts normally when there is none yet. Put it on the workload's pod template from the start:
The first workers find no checkpoint, start in snapshot mode and pause.
Checkpoint them.
Every worker created from then on is restored from that checkpoint, so scaling a role up scales it out from the checkpoint.
A restored worker resumes straight into serving: Cedana gives it the new pod's identity (pod name, Dynamo namespace and the rest of its Dynamo environment) and it registers with Dynamo like a new worker. One set of checkpoints can also bring up a new deployment, restored into a different DYN_NAMESPACE.
Restored pods keep the annotations and environment of the checkpointed pod, no need to remove them.
Grove
Grove's PodCliqueSets work the same way: put the annotations on each worker clique, and CEDANA_CHECKPOINT on its worker container. Grove copies both to every pod it creates from the clique. Leave the frontend clique as it is.
Checkpoint, restore and scale
Apply the PodCliqueSet. There is no checkpoint yet, so the worker pods start in snapshot mode: they load the model, warm up and pause.
Checkpoint each worker (prefill and decode) through the UI or the API. When the checkpoint is done, the original worker resumes and starts serving. It keeps running, with no restart and no restore.
Scale a clique, for example
kubectl scale podclique my-model-0-prefill --replicas=2. Every new pod is restored from its role's checkpoint, as is a clique scaled to 0 and back.
There is nothing to change in the PodCliqueSet between these steps, whatever its update strategy.
A second group from the same checkpoints
To bring up another deployment from the same checkpoints, copy the PodCliqueSet with a new name and a new DYN_NAMESPACE on the frontend and the workers, and keep the workers' CEDANA_CHECKPOINT names. The restored workers take their Dynamo namespace from the new pods, not from the checkpoint, so they serve only the new group's frontend.
How the workers find their frontend depends on Dynamo's discovery backend:
etcd (
DYN_DISCOVERY_BACKEND=etcd):DYN_NAMESPACEis all that changes.Kubernetes (DynamoGraphDeployment): Dynamo finds workers through the deployment's prefill and decode Services. Run the frontend from a DynamoGraphDeployment that also declares the prefill and decode components with
replicas: 0, so it has those Services while Grove runs the workers. In the copied cliques, also setDYN_PARENT_DGD_K8S_NAME, thenvidia.com/dynamo-namespacelabel andserviceAccountNameto the new deployment's values.
Limitations
Frontends aren't restored. Start the frontend normally and without the Cedana runtime, it needs no GPU and starts in seconds.
Workers already serving traffic are not checkpointed while connected to their peers. You must checkpoint workers at their snapshot pause as live traffic is not yet supported.
Both sides of a NIXL connection must run under Cedana, and restored workers must be able to reach their peers over TCP. UCX 1.11 to 1.22 is supported.
See Requirements and limits and Troubleshooting for details.
Last updated
Was this helpful?