Dynamo
Cedana API for Dynamo deployments
Returns checkpoints in the given cluster, optionally filtered by CEDANA_CHECKPOINT name. Each entry is a reusable warm-state asset that any deployment with a matching CEDANA_CHECKPOINT env restores from.
Cluster to scope checkpoints to.
Optional CEDANA_CHECKPOINT name to filter by
Checkpoint library
One CEDANA_CHECKPOINT "slot" in the library. A name can have many individual checkpoints over time (you can checkpoint the same env value repeatedly); this aggregates them into a single slot row.
Number of (non-deprecated) checkpoints captured under this name
Timestamp of the most recent checkpoint under this name
Deployment whose CEDANA_CHECKPOINT env equals this checkpoint's name, if any.
Model served by that deployment (drives the model logo + "deploy from").
CEDANA_CHECKPOINT name (the stable restore key)
True if at least one checkpoint under this name is ready + restorable
Server error
Client error
Server error
GET /v1/dynamo/checkpoints?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"checkpoint_count": 1,
"created_at": null,
"deployment_name": null,
"model": null,
"name": "text",
"restorable": true
}
]Publishes a DynamoGraphDeployment manifest to the cluster's RabbitMQ queue. The controller applies it via the dynamic Kubernetes client.
Request body for submitting a DynamoGraphDeployment to a cluster.
ID of the target cluster
If true, inject runtimeClassName: cedana and CEDANA_CHECKPOINT into worker pod specs
Full DynamoGraphDeployment manifest (nvidia.com/v1alpha1)
Deployment queued
Bad request
Server error
Client error
Server error
POST /v1/dynamo/deploy HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 65
{
"cluster_id": "text",
"enable_checkpointing": true,
"manifest": null
}textReturns all DynamoGraphDeployments synced from the specified cluster.
UUID of the cluster to filter by
Kubernetes namespace
List of Dynamo deployments
CEDANA_CHECKPOINT name of the first worker service, if checkpointing is enabled. This is the stable key a worker restores from. None if unset.
Resolved frontend endpoint: external_ip:port or cluster_ip:port
Server error
Client error
Server error
GET /v1/dynamo/deployments?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"backend": null,
"cedana_managed": true,
"cluster_id": "123e4567-e89b-12d3-a456-426614174000",
"created_at": "2026-01-01T00:00:00.000Z",
"id": "123e4567-e89b-12d3-a456-426614174000",
"model": null,
"name": "text",
"namespace": "text",
"spec": null,
"status": "text",
"updated_at": "2026-01-01T00:00:00.000Z",
"worker_replicas": null,
"worker_replicas_ready": null,
"checkpoint_name": null,
"frontend_endpoint": null
}
]Returns a single DynamoGraphDeployment by name.
DGD name
UUID of the cluster to filter by
Kubernetes namespace
Dynamo deployment
CEDANA_CHECKPOINT name of the first worker service, if checkpointing is enabled. This is the stable key a worker restores from. None if unset.
Resolved frontend endpoint: external_ip:port or cluster_ip:port
Not found
Server error
Client error
Server error
GET /v1/dynamo/deployments/{name}?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"backend": null,
"cedana_managed": true,
"cluster_id": "123e4567-e89b-12d3-a456-426614174000",
"created_at": "2026-01-01T00:00:00.000Z",
"id": "123e4567-e89b-12d3-a456-426614174000",
"model": null,
"name": "text",
"namespace": "text",
"spec": null,
"status": "text",
"updated_at": "2026-01-01T00:00:00.000Z",
"worker_replicas": null,
"worker_replicas_ready": null,
"checkpoint_name": null,
"frontend_endpoint": null
}Publishes a delete event for the DynamoGraphDeployment and removes the DB record.
DGD name
UUID of the cluster to filter by
Kubernetes namespace
Deletion queued
cluster_id required
Not found
Server error
Client error
Server error
DELETE /v1/dynamo/deployments/{name}?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
textWorker pods resolved by exact Dynamo label match, not pod_name.startsWith — the prefix heuristic could hit a different DGD sharing a name prefix in the same namespace (e.g. vllm-agg vs vllm-agg-cedana).
DGD name
UUID of the cluster to filter by
Kubernetes namespace
Checkpoints issued
Result of checkpointing a DGD: the workers a checkpoint was issued for.
No live worker pods found
Server error
Client error
Server error
POST /v1/dynamo/deployments/{name}/checkpoint?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"workers": [
{
"action_id": "text",
"pod_name": "text"
}
]
}Returns a summary of checkpoints taken for all worker pods belonging to the deployment. Matches pods whose name starts with the deployment name.
DGD name
UUID of the cluster to filter by
Kubernetes namespace
Checkpoint summary
A lightweight summary of checkpoints taken for a Dynamo deployment's workers.
True if at least one checkpoint with status='ready' exists
Timestamp of the most recent checkpoint
Total number of ready checkpoints across all worker pods
Server error
Client error
Server error
GET /v1/dynamo/deployments/{name}/checkpoints?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"has_cache": true,
"latest": null,
"pods": {
"ANY_ADDITIONAL_PROPERTY": 1
},
"total": 1
}DGD name
Relative window in hours (default 1, max 720 = metrics TTL). Ignored when
from_ms/to_ms are set. Anchored to the LATEST datapoint (not now()) so
a deleted deployment still returns its historical data.
Optional absolute range start (Unix ms). If set with to_ms, overrides hours.
Optional absolute range end (Unix ms). If set with from_ms, overrides hours.
Time series per metric
A single metric time-series point returned to the UI.
Unix timestamp in milliseconds
Server error
Client error
Server error
GET /v1/dynamo/deployments/{name}/metrics HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"metric_name": "text",
"timestamp_ms": 1,
"value": 1
}
]DGD name
Batch of metric samples posted by the watcher after scraping /metrics.
Metrics stored
Server error
Client error
Server error
POST /v1/dynamo/deployments/{name}/metrics HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 117
{
"cluster_id": "text",
"model": "text",
"namespace": "text",
"samples": [
{
"metric_name": "text",
"timestamp_ms": 1,
"value": 1
}
]
}textReturns one row per worker pod with the time it took to reach Ready, and whether Cedana restored it from a checkpoint (cedana_restore=true) or it cold-started.
DGD name
UUID of the cluster to filter by
Kubernetes namespace
Per-pod startup times
Startup time for a single worker pod.
Whether this pod was started via a Cedana restore (true) or cold-started (false)
Pod name
Time from PodScheduled → Ready in milliseconds
Server error
Client error
Server error
GET /v1/dynamo/deployments/{name}/pod-startup-times?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"cedana_restore": true,
"pod_name": "text",
"start_time_ms": 1
}
]Called by the dynamo-watcher after querying /v1/models on the deployment's frontend endpoint. Overwrites model and backend with ground-truth values.
DGD name
Body sent by the dynamo-watcher after scraping /v1/models from the live frontend.
Updated
Not found
Server error
Client error
Server error
POST /v1/dynamo/deployments/{name}/scrape HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 100
{
"backend": null,
"cluster_id": "123e4567-e89b-12d3-a456-426614174000",
"model": null,
"namespace": "text"
}textDGD name
UUID of the cluster to filter by
Kubernetes namespace
Cold versus restore latency distribution
Cold-start versus Cedana-restore startup latency for a deployment.
Milliseconds saved by restore at p50.
Cold p50 divided by restore p50. Values above 1 mean restore is faster.
Server error
Client error
Server error
GET /v1/dynamo/deployments/{name}/startup-comparison?cluster_id=123e4567-e89b-12d3-a456-426614174000 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"cold": null,
"p50_saved_ms": null,
"p50_speedup": null,
"restore": null
}Last updated
Was this helpful?