Metrics
Cedana API for metrics
Get checkpoint CPU data for efficiency calculations Returns the CPU usage at checkpoint time for each protected pod, allowing the frontend to calculate actual preservation ratios.
Time window in hours (default: 24, max: 168 = 7 days)
Returns per-pod checkpoint CPU data
Per-pod checkpoint resource data for efficiency calculations
Total number of checkpoints for this pod
Timestamp of the last checkpoint
CPU usage (seconds) at the time of the last checkpoint This represents the work preserved by the checkpoint
GPU memory used (MiB) at checkpoint time, if GPU was used
GPU model name (e.g., "NVIDIA H100 80GB HBM3"), if GPU was used
GPU utilization at checkpoint time (percentage 0-100), if GPU was used Derived from DCGM_FI_DEV_GPU_UTIL metric
Namespace
Pod name
Database error
Client error
Server error
GET /v1/metrics/checkpoint-cpu HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"checkpoint_count": 1,
"checkpoint_timestamp": "text",
"cpu_at_checkpoint": 1,
"gpu_memory_used_mib": null,
"gpu_model": null,
"gpu_utilization_at_checkpoint": null,
"namespace": "text",
"pod_name": "text"
}
]Returns checkpoint savings data per pod
Per-pod checkpoint savings data
Average interval between checkpoints (seconds)
Node capacity type: "spot" or "on-demand"
Estimated savings (if spot instance)
Number of automatic heartbeat checkpoints
Node instance type (e.g., "m5.xlarge", "n1-standard-4")
Number of manual checkpoints
Maximum interval - worst case data loss on failure (seconds)
Minimum interval between checkpoints (seconds)
Node name (for linking to node details page)
Node region
On-demand price per hour (from user config)
Ratio of time protected to checkpoint overhead (higher = more efficient)
Spot price per hour (from user config)
Total time (seconds) between first and last checkpoint - time "protected"
Total time spent checkpointing (nanoseconds)
Total data checkpointed (bytes)
Database error
Client error
Server error
GET /v1/metrics/checkpoint-savings HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"avg_interval_seconds": null,
"capacity_type": null,
"checkpoint_count": null,
"cluster_id": null,
"estimated_savings": null,
"first_checkpoint": null,
"gpu": null,
"heartbeat_checkpoints": null,
"instance_type": null,
"last_checkpoint": null,
"manual_checkpoints": null,
"max_recovery_point_seconds": null,
"min_interval_seconds": null,
"namespace": null,
"node_name": null,
"node_region": null,
"ondemand_price_per_hour": null,
"platform": null,
"pod_name": null,
"protection_efficiency_ratio": null,
"refreshed_at": null,
"spot_price_per_hour": null,
"time_protected_seconds": null,
"total_checkpoint_duration_ns": null,
"total_checkpoint_size_bytes": null
}
]Hourly compute cost rate (e.g., 10.50 for $10.50/hour)
Returns cost savings estimate
Cost savings estimate with user-provided hourly rate
Cost of checkpoint overhead (time spent checkpointing)
User-provided hourly compute cost
Maximum potential data loss on failure (seconds) - worst case recovery point
Estimated maximum cost saved per failure event (time_protected - max_recovery_point) * hourly_rate / 3600
Net potential savings = max_savings - overhead_cost
Number of pods protected
Total time protected by checkpoints (seconds)
Total checkpoints taken
Invalid hourly rate
Database error
Client error
Server error
GET /v1/metrics/checkpoint-savings/estimate?hourly_rate=1 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"checkpoint_overhead_cost": 1,
"hourly_rate": 1,
"max_recovery_point_seconds": 1,
"max_savings_per_failure": 1,
"net_potential_savings": 1,
"pods_protected": 1,
"time_protected_seconds": 1,
"total_checkpoints": 1
}Returns organization-level savings summary
Organization-level checkpoint savings summary
Total estimated savings across all pods (spot vs on-demand)
Database error
Client error
Server error
GET /v1/metrics/checkpoint-savings/summary HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"avg_checkpoint_interval_seconds": null,
"org_id": null,
"pods_protected": null,
"refreshed_at": null,
"total_checkpoint_overhead_ns": null,
"total_checkpoint_storage_bytes": null,
"total_checkpoints": null,
"total_estimated_savings": null,
"total_heartbeat_checkpoints": null,
"total_manual_checkpoints": null,
"total_time_protected_seconds": null,
"worst_case_recovery_point_seconds": null
}Prometheus text exposition (version 0.0.4)
ClickHouse unavailable or query timed out
Client error
Server error
GET /v1/metrics/export HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
textFilter by node name
Time window in hours (default: 1, max: 24)
Returns GPU metrics grouped by node
GPU metrics grouped by node
Failed to fetch GPU metrics
Client error
Server error
GET /v1/metrics/gpu HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"gpus": [
{
"gpu_index": "text",
"gpu_utilization": null,
"gpu_uuid": "text",
"memory_free_mib": null,
"memory_used_mib": null,
"memory_utilization": null,
"model_name": null,
"namespace": null,
"node_name": "text",
"pod_name": null,
"power_watts": null,
"temperature_celsius": null,
"timestamp": "text"
}
],
"node_name": "text"
}
]GPU UUID to get time-series for
Time window in minutes (default: 60, max: 1440)
Returns GPU time-series data
Time-series data for a single GPU
Failed to fetch GPU time-series
Client error
Server error
GET /v1/metrics/gpu/timeseries?gpu_uuid=text HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"data": [
{
"gpu_utilization": null,
"memory_free_mib": null,
"memory_used_mib": null,
"memory_utilization": null,
"namespace": null,
"pod_name": null,
"timestamp": "text"
}
],
"gpu_index": "text",
"gpu_uuid": "text",
"model_name": null,
"node_name": "text"
}Returns k8 metrics
New structured event state
Failed to fetch metrics
Client error
Server error
GET /v1/metrics/k8 HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"details": [
{
"collector": "text",
"cpu_core": "text",
"cpu_mode": "text",
"filesystem": "text",
"metric_name": "text",
"mountpoint": "text",
"resource": null,
"timestamp": "text",
"value": "text"
}
],
"name": "text"
}
]Window in seconds (default 604800, max 90 days)
kubernetes | slurm
checkpoint | restore
Counts, percentiles, histograms, stages, failures and a timeline for the window
Windowed aggregates over the same facts and bucket boundaries as the Prometheus export.
ClickHouse unavailable or query timed out
Client error
Server error
GET /v1/metrics/operations/summary HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"artifact_size_buckets": [
{
"count": 1,
"le": null
}
],
"attempts": 1,
"failed": 1,
"failures": [
{
"count": 1,
"operation": "text",
"reason": "text",
"stage": "text"
}
],
"fallbacks": 1,
"operations": [
{
"attempts": 1,
"bytes_total": 1,
"duration_buckets": [
{
"count": 1,
"le": null
}
],
"failed": 1,
"fallbacks": 1,
"operation": "text",
"p50_ns": 1,
"p95_ns": 1,
"pause_p50_ns": 1,
"pause_p95_ns": 1,
"succeeded": 1,
"to_ready_count": 1,
"to_ready_p50_ns": 1,
"to_ready_p95_ns": 1
}
],
"stages": [
{
"count": 1,
"operation": "text",
"p50_ns": 1,
"p95_ns": 1,
"stage": "text",
"total_ns": 1
}
],
"step_seconds": 1,
"succeeded": 1,
"tiers": [
{
"attempts": 1,
"bytes_total": 1,
"operation": "text",
"p50_ns": 1,
"storage_tier": "text",
"succeeded": 1
}
],
"timeline": [
{
"bucket_ms": 1,
"failed": 1,
"fallbacks": 1,
"operation": "text",
"succeeded": 1
}
],
"window_seconds": 1
}Filter by pod UID
Filter by node name
Limit results (default: 100, max: 1000)
Time window in hours (default: 1, max: 168 = 7 days)
Returns pod utilization metrics
Pod utilization metrics grouped by pod
Owner kind from kube-state-metrics (e.g., "Job", "ReplicaSet", "DaemonSet", "StatefulSet")
Owner name from kube-state-metrics (e.g., "my-job", "my-deployment-abc123")
Failed to fetch pod metrics
Client error
Server error
GET /v1/metrics/pods HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"cpu_usage_seconds": 1,
"memory_bytes": 1,
"namespace": null,
"node_name": "text",
"owner_kind": null,
"owner_name": null,
"pod_name": null,
"pod_uid": "text",
"timestamp": "text"
}
]Number of pods to return (default: 10, max: 100)
Returns top pods by CPU
Top pods by resource usage
Failed to fetch top pods
Client error
Server error
GET /v1/metrics/pods/top/cpu HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"cpu_usage_seconds": 1,
"memory_bytes": 1,
"namespace": null,
"node_name": "text",
"pod_name": null,
"pod_uid": "text"
}
]Number of pods to return (default: 10, max: 100)
Returns top pods by memory
Top pods by resource usage
Failed to fetch top pods
Client error
Server error
GET /v1/metrics/pods/top/memory HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"cpu_usage_seconds": 1,
"memory_bytes": 1,
"namespace": null,
"node_name": "text",
"pod_name": null,
"pod_uid": "text"
}
]Returns SLURM metrics
Failed to fetch SLURM metrics
Client error
Server error
GET /v1/metrics/slurm HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"counter_value": null,
"gauge_value": null,
"id": 1,
"kind": null,
"name": "text",
"tags": null,
"timestamp": 1
}
]Returns SLURM cluster overview
Failed to fetch SLURM cluster overview
Client error
Server error
GET /v1/metrics/slurm/overview HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"cpu_load": null,
"cpus_idle": null,
"cpus_total": null,
"mem_alloc": null,
"mem_free": null,
"mem_real": null,
"timestamp": 1
}Returns state breakdown
Failed to fetch state breakdown
Client error
Server error
GET /v1/metrics/slurm/overview/state HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"cpus_by_state": [
{
"state": "text",
"value": 1
}
],
"nodes_by_state": [
{
"state": "text",
"value": 1
}
],
"timestamp": 1
}Returns partition statistics
Failed to fetch partition statistics
Client error
Server error
GET /v1/metrics/slurm/partitions HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"cpu_load": null,
"free_memory": null,
"idle_cpus": null,
"idle_nodes": null,
"partition_name": "text",
"real_memory": null,
"timestamp": 1,
"total_cpus": null
}
]Returns node counts per partition per state
Node count per state for a partition
Failed to fetch partition node counts
Client error
Server error
GET /v1/metrics/slurm/partitions/nodes HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"count": 1,
"partition_name": "text",
"state": "text"
}
]Partition name
Returns partition time series
Failed to fetch partition time series
Client error
Server error
GET /v1/metrics/slurm/partitions/{name}/timeseries HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"cpu_load": [
{
"timestamp": 1,
"value": 1
}
],
"free_memory": [
{
"timestamp": 1,
"value": 1
}
],
"partition_name": "text"
}Returns CPU load time series
Failed to fetch CPU load time series
Client error
Server error
GET /v1/metrics/slurm/timeseries/cpu-load HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
[
{
"timestamp": 1,
"value": 1
}
]Returns memory time series
Failed to fetch memory time series
Client error
Server error
GET /v1/metrics/slurm/timeseries/memory HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"allocated_memory": [
{
"timestamp": 1,
"value": 1
}
],
"free_memory": [
{
"timestamp": 1,
"value": 1
}
]
}Checkpoint id or restore uuid
Operation fact, its stages and, for a checkpoint, the restores made from it
Restores only: PodScheduled / Ready transitions of the restored pod, once known.
Milliseconds since the Unix epoch.
No fact recorded for this id
ClickHouse unavailable
Client error
Server error
GET /v1/operations/{id} HTTP/1.1
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*
{
"action_id": "text",
"checkpoint_id": "text",
"cluster_id": "text",
"completed_at_ms": null,
"duration_ns": 1,
"failure_reason": "text",
"failure_stage": "text",
"gpu_model": "text",
"kind": "text",
"logical_bytes": 1,
"name": "text",
"namespace": "text",
"node_name": "text",
"op_type": "text",
"operation_id": "text",
"pause_ns": 1,
"physical_bytes": 1,
"platform": "text",
"pod_ready_at_ms": null,
"pod_scheduled_at_ms": null,
"pod_uid": "text",
"reason": "text",
"restore_uuid": "text",
"result": "text",
"source": "text",
"started_at_ms": 1,
"storage_tier": "text",
"workload_type": "text",
"restores": [
{
"action_id": "text",
"checkpoint_id": "text",
"cluster_id": "text",
"completed_at_ms": null,
"duration_ns": 1,
"failure_reason": "text",
"failure_stage": "text",
"gpu_model": "text",
"kind": "text",
"logical_bytes": 1,
"name": "text",
"namespace": "text",
"node_name": "text",
"op_type": "text",
"operation_id": "text",
"pause_ns": 1,
"physical_bytes": 1,
"platform": "text",
"pod_ready_at_ms": null,
"pod_scheduled_at_ms": null,
"pod_uid": "text",
"reason": "text",
"restore_uuid": "text",
"result": "text",
"source": "text",
"started_at_ms": 1,
"storage_tier": "text",
"workload_type": "text"
}
],
"stages": [
{
"bytes_in": 1,
"bytes_out": 1,
"duration_ns": 1,
"reason": "text",
"result": "text",
"seq": 1,
"stage": "text"
}
]
}Last updated
Was this helpful?