> For the complete documentation index, see [llms.txt](https://docs.cedana.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cedana.ai/cedana-inference/deploying-with-dynamo.md). # Deploying with Dynamo #### Deploying Cedana + Dynamo The good news is that Cedana just works with Dynamo! You can follow this guide on setting up Cedana on your Kubernetes cluster: [Installation](/cedana-kubernetes/installation.md), and follow this to setup Dynamo on your cluster: . Dynamo requires another controller from the cedana side, so in your helm values.yml you'll need to add the following config:
``` # Only enable on clusters that have NVIDIA Dynamo (DynamoGraphDeployment CRD) installed. dynamo: enabled: true image: repository: cedana/cedana-dynamo-watcher tag: main digest: # ignores tag if set imagePullPolicy: IfNotPresent resources: {} tolerations: - key: "cedana.ai/not-ready" operator: "Exists" effect: "NoSchedule" affinity: {} nodeSelector: {} metricsService: ports: - name: https port: 8443 protocol: TCP targetPort: https type: ClusterIP ``` Once set up, you can manage your dynamo deployment via our UI. ## Disaggregated Prefill/Decode worker restores and scaling Cedana can checkpoint a disaggregated prefill or decode worker once it has loaded and warmed its model, then restore new workers from that checkpoint instead of cold-starting them. Each role restores and scales from its own checkpoint, and one set of checkpoints can bring up workers for a whole new deployment. Cedana works seamlessly with Dynamo's snapshot control file. Once you turn on the Cedana controller's P/D webhook, it sets snapshot mode up for you after you annotate the worker with `cedana.ai/pd-disagg: prefill` or `cedana.ai/pd-disagg: decode`. For a diagram of checkpointing each role once and scaling it out, and for how this works under the hood, see [How P/D Checkpoint/Restore Works](/cedana-inference/deploying-with-dynamo/pd-disagg-checkpoint-restore.md). ### Turn it on The webhook is off by default. Turn it on in your Helm values: ```yaml controllerManager: pdDisaggWebhook: enabled: true ``` or with `--set controllerManager.pdDisaggWebhook.enabled=true` on `helm install` or `helm upgrade`. There is nothing else to install. The controller creates the webhook's `cedana-pd-webhook` Service, its self-signed serving cert and the `cedana-pd-disagg` MutatingWebhookConfiguration itself, and the chart grants it the access it needs. Setting the value back to `false` removes the webhook on the controller's next start. {% hint style="warning" %} If the controller is down when a worker pod is created, the pod starts without the P/D setup rather than failing. It can still run, but its checkpoint can't restore into a working P/D worker. Check that a worker was set up before you checkpoint it: its `PYTHONPATH` starts with `/opt/cedana-pd-disagg`. ```bash kubectl get pod -o jsonpath='{.spec.containers[*].env[?(@.name=="PYTHONPATH")].value}' ``` {% endhint %} ### Mark the workers Run the prefill and decode pods under the Cedana runtime and annotate them: ```yaml apiVersion: v1 kind: Pod metadata: name: my-model-prefill annotations: cedana.ai/pd-disagg: prefill cedana.ai/pd-disagg-framework: vllm spec: runtimeClassName: cedana containers: - name: prefill # ... ``` The annotations are: | Annotation | Description | Default | | ------------------------------- | -------------------------------------------------------------------------------------------------------- | ------- | | `cedana.ai/pd-disagg` | Marks the pod as a P/D worker. The value (`prefill` or `decode`) is informational. `false` turns it off. | unset | | `cedana.ai/pd-disagg-container` | Name of the container to set up. Without it, every container in the pod is set up. | unset | | `cedana.ai/pd-disagg-framework` | The inference engine: `vllm` or `sglang`. | `vllm` | When the pod is created, the Cedana controller adds: * Dynamo snapshot mode: `DYN_SNAPSHOT_CONTROL_DIR` and an `emptyDir` for it. In snapshot mode the worker loads and warms the model, then pauses without registering with Dynamo. * Cedana's patch for the engine, mounted at `/opt/cedana-pd-disagg` from the `cedana-pd-disagg` ConfigMap, which the controller creates in the pod's namespace, and put first on `PYTHONPATH`. It closes the engine's NIXL and Mooncake connections for the snapshot and reopens them after a restore. * `CEDANA_SNAPSHOT_KEEP_GPU_MEMORY=1`, so weights, KV cache and NIXL registrations stay on the GPU through the pause. * `CEDANA_PERSISTENT_MOUNTS=/dev/shm`, so `/dev/shm` is carried through the checkpoint. * `CEDANA_UCX_CR=1`, so the engine's UCX state (NIXL's KV transfers) is proxied and can be checkpointed. UCX is passed through unproxied in pods without it. Anything you already set in the pod spec is kept. ### Checkpoint Request the checkpoint whenever you like, through the UI or the API, even while the worker is still loading. Cedana waits until the worker has paused and is ready for a snapshot, checkpoints it, and then tells it the snapshot is done, at which point the original worker exits and frees its GPUs. {% hint style="info" %} The wait counts toward the checkpoint request's timeout. For very large models, request the checkpoint once the worker is close to ready. {% endhint %} ### Restore and scale Restores work as for any Cedana workload (see [Jobs](/cedana-kubernetes/jobs.md)): give each worker container a checkpoint name with `CEDANA_CHECKPOINT`, one name per role. ```yaml env: - name: CEDANA_CHECKPOINT value: my-model-prefill ``` Checkpoints of that container are recorded under the name. A pod with `CEDANA_CHECKPOINT` restores from the latest checkpoint under its name, and starts normally when there is none yet. Put it on the workload's pod template from the start: 1. The first workers find no checkpoint, start in snapshot mode and pause. 2. Checkpoint them. 3. Every worker created from then on is restored from that checkpoint, so scaling a role up scales it out from the checkpoint. A restored worker resumes straight into serving: Cedana gives it the new pod's identity (pod name, Dynamo namespace and the rest of its Dynamo environment) and it registers with Dynamo like a new worker. One set of checkpoints can also bring up a new deployment, restored into a different `DYN_NAMESPACE`. Restored pods keep the annotations and environment of the checkpointed pod, no need to remove them. {% hint style="info" %} Restored workers must be able to read the checkpoint. With the default local checkpoint directory, it is on the node where it was taken, so keep each role on that node. With [CSX](/cedana-kubernetes/storage/where-we-checkpoint-to.md#csx-cedana-storage-express), every node can restore it. Object storage (S3, GCS) works too, but the whole checkpoint is downloaded before the restore starts. {% endhint %} ### Grove Grove's PodCliqueSets work the same way: put the annotations on each worker clique, and `CEDANA_CHECKPOINT` on its worker container. Grove copies both to every pod it creates from the clique. Leave the frontend clique as it is. ```yaml apiVersion: grove.io/v1alpha1 kind: PodCliqueSet metadata: name: my-model spec: replicas: 1 template: cliqueStartupType: CliqueStartupTypeAnyOrder cliques: - name: frontend # unchanged: no annotations, no Cedana runtime # ... - name: prefill annotations: cedana.ai/pd-disagg: prefill cedana.ai/pd-disagg-framework: vllm cedana.ai/pd-disagg-container: prefill spec: roleName: prefill replicas: 1 podSpec: runtimeClassName: cedana containers: - name: prefill env: - name: CEDANA_CHECKPOINT value: my-model-prefill # ... - name: decode annotations: cedana.ai/pd-disagg: decode cedana.ai/pd-disagg-framework: vllm cedana.ai/pd-disagg-container: decode spec: roleName: decode replicas: 1 podSpec: runtimeClassName: cedana containers: - name: decode env: - name: CEDANA_CHECKPOINT value: my-model-decode # ... ``` #### Checkpoint, restore and scale 1. **Apply the PodCliqueSet.** There is no checkpoint yet, so the worker pods start in snapshot mode: they load the model, warm up and pause. 2. **Checkpoint each worker** (prefill and decode) through the UI or the API. When the checkpoint is done, the original worker process exits and Grove restarts the container, which now restores from the checkpoint. 3. **Scale a clique**, for example `kubectl scale podclique my-model-0-prefill --replicas=2`. Every new pod is restored from its role's checkpoint, as is a clique scaled to 0 and back. There is nothing to change in the PodCliqueSet between these steps, whatever its update strategy. #### A second group from the same checkpoints To bring up another deployment from the same checkpoints, copy the PodCliqueSet with a new name and a new `DYN_NAMESPACE` on the frontend and the workers, and keep the workers' `CEDANA_CHECKPOINT` names. The restored workers take their Dynamo namespace from the new pods, not from the checkpoint, so they serve only the new group's frontend. How the workers find their frontend depends on Dynamo's discovery backend: * **etcd** (`DYN_DISCOVERY_BACKEND=etcd`): `DYN_NAMESPACE` is all that changes. * **Kubernetes** (DynamoGraphDeployment): Dynamo finds workers through the deployment's prefill and decode Services. Run the frontend from a DynamoGraphDeployment that also declares the prefill and decode components with `replicas: 0`, so it has those Services while Grove runs the workers. In the copied cliques, also set `DYN_PARENT_DGD_K8S_NAME`, the `nvidia.com/dynamo-namespace` label and `serviceAccountName` to the new deployment's values. ### Limitations * **Frontends aren't restored.** Start the frontend normally and without the Cedana runtime, it needs no GPU and starts in seconds. * **Workers already serving traffic** are not checkpointed while connected to their peers. You must checkpoint workers at their snapshot pause as live traffic is not yet supported. * **Both sides of a NIXL connection must run under Cedana**, and restored workers must be able to reach their peers over TCP. UCX 1.11 to 1.22 is supported. See [Requirements and limits](/cedana-inference/deploying-with-dynamo/pd-disagg-checkpoint-restore.md#requirements-and-limits) and [Troubleshooting](/cedana-inference/deploying-with-dynamo/pd-disagg-checkpoint-restore.md#troubleshooting) for details.