Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Deploying with Dynamo

Deploying Cedana + Dynamo

The good news is that Cedana just works with Dynamo! You can follow this guide on setting up Cedana on your Kubernetes cluster: Installation, and follow this to setup Dynamo on your cluster: https://docs.nvidia.com/dynamo/v1.0.1/kubernetes-deployment/deployment-guide.

Dynamo requires another controller from the cedana side, so in your helm values.yml you'll need to add the following config:

# Only enable on clusters that have NVIDIA Dynamo (DynamoGraphDeployment CRD) installed.
dynamo:
  enabled: true
  image:
    repository: cedana/cedana-dynamo-watcher
    tag: main
    digest: # ignores tag if set
    imagePullPolicy: IfNotPresent
  resources: {}
  tolerations:
    - key: "cedana.ai/not-ready"
      operator: "Exists"
      effect: "NoSchedule"
  affinity: {}
  nodeSelector: {}
metricsService:
  ports:
    - name: https
      port: 8443
      protocol: TCP
      targetPort: https
  type: ClusterIP

Once set up, you can manage your dynamo deployment via our UI.

Disaggregated Prefill/Decode worker restores and scaling

Cedana can checkpoint a disaggregated prefill or decode worker once it has loaded and warmed its model, then restore new workers from that checkpoint instead of cold-starting them. Each role restores and scales from its own checkpoint, and one set of checkpoints can bring up workers for a whole new deployment.

Cedana works seamlessly with Dynamo's snapshot control file. Once you turn on the Cedana controller's P/D webhook, it sets snapshot mode up for you after you annotate the worker with cedana.ai/pd-disagg: prefill or cedana.ai/pd-disagg: decode.

For a diagram of checkpointing each role once and scaling it out, and for how this works under the hood, see How P/D Checkpoint/Restore Works.

Turn it on

The webhook is off by default. Turn it on in your Helm values:

or with --set controllerManager.pdDisaggWebhook.enabled=true on helm install or helm upgrade. There is nothing else to install.

The controller creates the webhook's cedana-pd-webhook Service, its self-signed serving cert and the cedana-pd-disagg MutatingWebhookConfiguration itself, and the chart grants it the access it needs. Setting the value back to false removes the webhook on the controller's next start.

Mark the workers

Run the prefill and decode pods under the Cedana runtime and annotate them:

The annotations are:

Annotation
Description
Default

cedana.ai/pd-disagg

Marks the pod as a P/D worker. The value (prefill or decode) is informational. false turns it off.

unset

cedana.ai/pd-disagg-container

Name of the container to set up. Without it, every container in the pod is set up.

unset

cedana.ai/pd-disagg-framework

The inference engine: vllm or sglang.

vllm

When the pod is created, the Cedana controller adds:

  • Dynamo snapshot mode: DYN_SNAPSHOT_CONTROL_DIR and an emptyDir for it. In snapshot mode the worker loads and warms the model, then pauses without registering with Dynamo.

  • Cedana's patch for the engine, mounted at /opt/cedana-pd-disagg from the cedana-pd-disagg ConfigMap, which the controller creates in the pod's namespace, and put first on PYTHONPATH. It closes the engine's NIXL and Mooncake connections for the snapshot and reopens them after a restore.

  • CEDANA_SNAPSHOT_KEEP_GPU_MEMORY=1, so weights, KV cache and NIXL registrations stay on the GPU through the pause.

  • CEDANA_PERSISTENT_MOUNTS=/dev/shm, so /dev/shm is carried through the checkpoint.

  • CEDANA_GPU_UCX_CR=1, so the engine's UCX state (NIXL's KV transfers) is proxied and can be checkpointed. UCX is passed through unproxied in pods without it. This needs cedana-gpu v0.8.9 or later; earlier versions read CEDANA_UCX_CR instead, so set that yourself in the worker spec if you run one.

Anything you already set in the pod spec is kept.

Checkpoint

Request the checkpoint whenever you like, through the UI or the API, even while the worker is still loading. Cedana waits until the worker has paused and is ready for a snapshot, checkpoints it, and then tells it to resume: the original worker keeps running, registers with Dynamo and starts serving.

The wait counts toward the checkpoint request's timeout. For very large models, request the checkpoint once the worker is close to ready.

Restore and scale

Restores work as for any Cedana workload (see Jobs): give each worker container a checkpoint name with CEDANA_CHECKPOINT, one name per role.

Checkpoints of that container are recorded under the name. A pod with CEDANA_CHECKPOINT restores from the latest checkpoint under its name, and starts normally when there is none yet. Put it on the workload's pod template from the start:

  1. The first workers find no checkpoint, start in snapshot mode and pause.

  2. Checkpoint them.

  3. Every worker created from then on is restored from that checkpoint, so scaling a role up scales it out from the checkpoint.

A restored worker resumes straight into serving: Cedana gives it the new pod's identity (pod name, Dynamo namespace and the rest of its Dynamo environment) and it registers with Dynamo like a new worker. One set of checkpoints can also bring up a new deployment, restored into a different DYN_NAMESPACE.

Restored pods keep the annotations and environment of the checkpointed pod, no need to remove them.

Restored workers must be able to read the checkpoint. With the default local checkpoint directory, it is on the node where it was taken, so keep each role on that node. With CSX, every node can restore it. Object storage (S3, GCS) works too, but the whole checkpoint is downloaded before the restore starts.

Grove

Grove's PodCliqueSets work the same way: put the annotations on each worker clique, and CEDANA_CHECKPOINT on its worker container. Grove copies both to every pod it creates from the clique. Leave the frontend clique as it is.

Checkpoint, restore and scale

  1. Apply the PodCliqueSet. There is no checkpoint yet, so the worker pods start in snapshot mode: they load the model, warm up and pause.

  2. Checkpoint each worker (prefill and decode) through the UI or the API. When the checkpoint is done, the original worker resumes and starts serving. It keeps running, with no restart and no restore.

  3. Scale a clique, for example kubectl scale podclique my-model-0-prefill --replicas=2. Every new pod is restored from its role's checkpoint, as is a clique scaled to 0 and back.

There is nothing to change in the PodCliqueSet between these steps, whatever its update strategy.

A second group from the same checkpoints

To bring up another deployment from the same checkpoints, copy the PodCliqueSet with a new name and a new DYN_NAMESPACE on the frontend and the workers, and keep the workers' CEDANA_CHECKPOINT names. The restored workers take their Dynamo namespace from the new pods, not from the checkpoint, so they serve only the new group's frontend.

How the workers find their frontend depends on Dynamo's discovery backend:

  • etcd (DYN_DISCOVERY_BACKEND=etcd): DYN_NAMESPACE is all that changes.

  • Kubernetes (DynamoGraphDeployment): Dynamo finds workers through the deployment's prefill and decode Services. Run the frontend from a DynamoGraphDeployment that also declares the prefill and decode components with replicas: 0, so it has those Services while Grove runs the workers. In the copied cliques, also set DYN_PARENT_DGD_K8S_NAME, the nvidia.com/dynamo-namespace label and serviceAccountName to the new deployment's values.

Limitations

  • Frontends aren't restored. Start the frontend normally and without the Cedana runtime, it needs no GPU and starts in seconds.

  • Workers already serving traffic are not checkpointed while connected to their peers. You must checkpoint workers at their snapshot pause as live traffic is not yet supported.

  • Both sides of a NIXL connection must run under Cedana, and restored workers must be able to reach their peers over TCP. UCX 1.11 to 1.22 is supported.

See Requirements and limits and Troubleshooting for details.

Last updated

Was this helpful?