` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.
{% code title="victim.sbatch" lineNumbers="true" %}
```bash
#!/bin/bash
#SBATCH --job-name=victim # Job name
#SBATCH --output=victim.out # Standard output log
#SBATCH --error=victim.err # Standard error log
#SBATCH --time=00:10:00 # Time limit (hh:mm:ss)
#SBATCH --nodes=1 # Run on 1 node
#SBATCH --ntasks=1 # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1 # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=victim # Label for all checkpoints
echo "Starting victim job on $(hostname)..."
# Loop from 0 to 600
for i in {0..600}
do
echo "Counter: $i"
sleep 1
done
```
{% endcode %}
Submit the victim to the `debug` (low priority) partition on your node using `sbatch`.
```shellscript
sbatch -p debug -w compute-01 victim.sbatch
```
### Deploy preemptor
The preemptor job can be any job submitted to the `high` partition (higher priority than `debug`) as long as it occupies all the CPUs available on the node. This ensures SLURM preempts any low partition jobs before deploying this job.
{% hint style="info" %}
Use `nproc` to find number of CPUs on your node.
{% endhint %}
{% code title="preemptor.sbatch" lineNumbers="true" %}
```shellscript
#!/bin/bash
#SBATCH --job-name=preemptor # Job name
#SBATCH --output=victim.out # Standard output log
#SBATCH --error=victim.err # Standard error log
#SBATCH --time=00:10:00 # Time limit (hh:mm:ss)
#SBATCH --nodes=1 # Run on 1 node
#SBATCH --ntasks=1 # Run 1 task
echo "Starting preemptor job on $(hostname)..."
# Loop from 0 to 600
for i in {0..600}
do
echo "Counter: $i"
sleep 1
done
```
{% endcode %}
Note that this job *does not* need any `CEDANA_` environment variable as we don't need it to be checkpointed.
Submit the preemptor to the `high` partition on your node using `sbatch`, ensuring it occupies all the CPUs available on the node. In this example, `compute-01` has 4 CPUs.
```shellscript
sbatch -p high -w compute-01 --cpus-per-task=4 preemptor.sbatch
```
You should be able to see that our victim job has been preempted and re-queued onto the node (in pending `PD` state), while our preemptor job is freely running. Check the queue using `squeue`.
```
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
182 debug victim root PD 0:00 1 (BeginTime)
183 high preemptor root R 0:02 1 compute-01
```
### Restore
Now let's restore our victim job. All we need to do is `scancel` our preemptor job or wait for it to finish running.
Cancel the higher priority job using `scancel`.
```shellscript
scancel 183
```
After some time, the victim job should resume running on the node. Check the output file `victim.out` to verify that it was resumed from where it was preempted!
{% hint style="warning" %}
If for some reason, a checkpoint could not be taken during preemption, the job should simply start fresh. This can happen if the `GraceTime` on the partition is not sufficient for the size of the job. Alternatively, any older available checkpoints will be restored. Check out [#policy-based](#policy-based "mention") automation for taking regular checkpoints.
{% endhint %}
## Policy-based
Let's deploy a sample SLURM job and explore Cedana's policy-based automation for checkpoint/restore.
### Deploy
Below is an example CPU workload that counts from 0 to 600, sleeping for 1 second between each count. By default, Cedana must be enabled to checkpoint this job by adding the `CEDANA_ENABLE=1` environment variable.
#!/bin/bash
#SBATCH --job-name=counting # Job name
#SBATCH --output=counting.out # Standard output log
#SBATCH --error=counting.err # Standard error log
#SBATCH --time=00:10:00 # Time limit (hh:mm:ss)
#SBATCH --nodes=1 # Run on 1 node
#SBATCH --ntasks=1 # Run 1 task
#SBATCH --export=CEDANA_ENABLE=1 # Enable Cedana
#SBATCH --export=CEDANA_CHECKPOINT=counting # Label for all checkpoints
echo "Starting counter job on $(hostname)..."
# Loop from 0 to 600
for i in {0..600}
do
echo "Counter: $i"
sleep 1
done
` to label all checkpoints of this workload. So that when this workload is restored, the latest checkpoint from this label is automatically used.
Submit the job using `sbatch`.
```bash
sbatch counting.sbatch
```
### Configure policy
You may now take [manual checkpoints](/cedana-slurm/cr) of this workload or configure an automation policy.
1. Head over to the [SLURM Policies Page](https://staging.cedana.com/slurm/policies) and click on "Create Policy".
```
{% hint style="info" %}
Cancellation should also trigger an automatic checkpoint if the `GraceTime` in your SLURM configuration allows sufficient time for a checkpoint to be taken.
{% endhint %}
Simply submit the job again using `sbatch`. You will notice that the job will automatically resume from the latest checkpoint instead of starting fresh.
# Cedana + Dynamo for Inference at Scale
#### Why Cedana + Dynamo?
NVIDIA [Dynamo](https://github.com/ai-dynamo/dynamo) offers SOTA inference serving at scale, designed to enhance the performance of LLM inference. By leveraging advanced optimization techniques and dynamic scheduling, Dynamo significantly accelerates inference workloads.
Dynamo's architecture is built on the modular disaggregation of the prefill and decode phases, allowing each to be scaled independently on hardware best suited for their specific compute or memory profiles.
Its core components include:
* a Smart Router that uses Radix Tree indexing to track KV-cache locality across the cluster, ensuring requests land on workers that already hold relevant context.
* the KV Block Manager (KVBM) (for large caches) offloads cache data across a storage hierarchy of host DRAM, SSDs, and network storage
* NVIDIA Inference Transfer Library (NIXL), which enables zero-copy, asynchronous KV cache migration at near-memory speeds
And more! See their [Core Capabilities](https://github.com/ai-dynamo/dynamo?tab=readme-ov-file#core-capabilities) section in the docs for all the features Dynamo offers.
What makes Dynamo incredibly compelling is that it's a fast-moving project, keeping track of developments in research and the field to bring frontier-lab level performance to users with access to clusters of GPUs, both in datacenters and the cloud.
However some challenges still remain with using a tool like Dynamo that an integration with Cedana solves!
* **Cold Start Times:** Cold starts won't generally go away, even with infrastructure optimizations like Dynamo presents as the weights still need to be downloaded and loaded into GPU memory for a worker or set of workers. Model initialization for large models like Llama-3-70B can exceed 80 seconds. Cedana solves this by restoring "ready-to-serve" Dynamo workers from checkpoints, enabling new replicas to come online 10x faster than a native cold start.
* **A Recompute Tax**: In long-context agentic sessions, losing a decode worker node results in the total loss of its in-memory KV cache. To resume, a new node must pay a "recompute tax", re-running the expensive prefill phase which can take 10+ seconds and waste significant GPU cycles. Cedana provides system-level Save, Migrate, and Resume (SMR) capabilities that transparently capture the exact state of the GPU and CPU memory, allowing generation to resume from the very next token without re-prefilling the context (or regenerating the KV-Cache!).
* **Resilience:** Dynamo's stateful nature makes it difficult to run reliably on high-volatility, low-cost infrastructure like spot nodes. Cedana integrates with cloud-native schedulers to monitor preemption notices and live-migrate active inference tasks to healthy nodes before they are terminated.
# Deploying with Dynamo
#### Deploying Cedana + Dynamo
The good news is that Cedana just works with Dynamo! You can follow this guide on setting up Cedana on your Kubernetes cluster: [Installation](/cedana-kubernetes/installation), and follow this to setup Dynamo on your cluster: .
Dynamo requires another controller from the cedana side, so in your helm values.yml you'll need to add the following config:Memory throughput (GPU)
Compute throughput (GPU)
Concurrent kernels execution duration
Memory throughput (GPU to host)
Memory throughput (host to GPU)
ML training time per iteration
ML training time
Cold start time (smaller models)
Cold start time (larger models)
Cold start time (time to first token, on GPT2 models)
Cold start time (time to first token, on popular LLMs)
Warm checkpoint time & throughput (GPT2 models)
Warm checkpoint time & throughput (popular LLMs)
Save migrate resume (GPT2 models)
Save migrate resume (popular LLMs)
Throughput Native Cedana Requests/s 11.69 13.85 Total tokens/s 4833.43 5725.69 Output tokens/s 2318.23 2746.18
Runtime throughput when running with Cedana is about **11% faster**. This is surprising, because using GPU interception for enabling GPU checkpoint/restore typically introduces some overhead to runtime performance. However, this overhead has been mitigated by optimizations that are only possible due to the asynchronous design of our GPU virtual machine (internally referred to as the GPU controller). This design allows the GPU controller to make decisions before actually executing CUDA driver API calls intercepted during operation. For example, it can combine multiple driver API calls or cancel redundant ones, effectively working like a[ just-in-time compiler (JIT)](https://en.wikipedia.org/wiki/Just-in-time_compilation).
### Setup
*Cedana version: v0.9.240-35-g9bd7886e*\
*Cedana GPU plugin version: v0.4.7-9-g1062be1*\
*CRIU version: v4.0*
The benchmarks were run on the following machine:
CPU cache (L1 data) 1.9 MiB (30 instances) CPU cache (L2) 15 MiB (30 instances) CPU cache (L3) 480 MiB (30 instances) CPU cores 30 CPU memory (DRAM) 216.26 GiB CPU model AMD EPYC 7J13 64-Core Processor CPU threads/core 1 Disk read speed 7.410 GiB/s Disk write speed .67225325884543761638 GiB/s GPU API 12.8 GPU SM clock speed (max) 1410 MHz GPU compute capability 8.0 GPU driver 570.124.04 GPU memory (total) 40960 MiB GPU memory clock speed (max) 1215 MHz GPU model NVIDIA A100-SXM4-40GB Mock internet speed 500 MiB/s
## Why Cedana?
The above performance results make up a good case for why we continue to advance our interception-based GPU checkpoint/restore solution. As highlighted in this [paper](https://arxiv.org/abs/2502.16631), there are several challenges to testing and maintaining an interception-based solution, but we believe it’s worth it.
1. **Checkpoint/restore performance** – As in [benchmarks](#benchmarks), our checkpoint/restore solution is much faster than CRIU CUDA. GPU interception provides enhanced visibility into program execution, enabling optimization decisions like identifying memory regions for checkpointing and determining which CUDA calls to replay or skip during restore. This confers additional benefits, such as incremental checkpointing.
2. **Runtime performance** – GPU interception enables our GPU controller to function like a JIT compiler, allowing real-time optimizations such as merging multiple CUDA graph API calls, cancelling redundancies, etc. Even with our current minimal set of optimizations, we’ve observed a 10-12% runtime performance improvement on the [vLLM throughput benchmarks](#vllm-throughput). There’s substantial untapped potential that can only be unlocked through this level of control.
3. **Multi-GPU / Multi-node scaling** – For workloads distributed across multiple GPUs—whether on the same host or across different hosts—transparent checkpoint/restore requires synchronization that is only possible through knowledge of the GPU state. CRIU CUDA currently does **not** support NCCL-accelerated workloads. While application-aware C/R is always an option, achieving true transparency without modifying the application requires this approach.
4. **Beyond checkpoint/restore** – GPU interception enables far more than just checkpoint/restore. It lays the groundwork for automatic GPU failover, live GPU scaling, and live GPU migration and more!
## References
1. [CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads](https://arxiv.org/abs/2502.16631)
2. [Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads](https://arxiv.org/pdf/2202.07848)
# Redis Save Migrate & Resume (SMR) on Kubernetes
Performing a save/migrate/resume of an in-cluster redis db.
{% hint style="warning" %}
This example is outdated. Please look at our other examples.
{% endhint %}
## Setup
Let's first set up a sample redis database on k8s. Create a namespace called `cedana-examples` and install `redis-example.yaml` , which can be found here: .
```bash
kubectl create namespace cedana-examples
kubectl apply -f redis-example.yaml
```
Now let's port-forward the service on 127.0.0.1 and connect to the database:
```bash
kubectl port-forward service/redis -n cedana-examples 6379:6379
redis-cli -h 127.0.0.1
```
Lets store some data in the db:
```bash
HSET 'user:001' first_name 'John' last_name 'doe' dob '12-JUN-1970'
HSET 'user:002' first_name 'David' last_name 'Bloom' dob '03-MAR-1981'
```
Listing all the pods in the `cedana-examples` namespace using the `/list` endpoint gives us all the information we need to do a checkpoint and restore.
```bash
curl -X GET -H 'Content-Type: application/json' -d '{
"root": "'$ROOT'"
}' $CONTROLLER_URL:1324/list/cedana-examples
```
Great! Now its time to checkpoint the container. Lets set the necessary environment variables before we proceed, which were present in the output of the `/list` endpoint. The following variables should work on most default containerd clusters.
```bash
export CHECKPOINT_CONTAINER=redis \
export CHECKPOINT_SANDBOX=redis-6b5bcbb6b6-tdb4p \
export RESTORE_CONTAINER=redis-restore \
export RESTORE_SANDBOX=redis-restore-c6c794b64-h7ccs \
export NAMESPACE=cedana-examples \
export CONTROLLER_URL=localhost \
export ROOT=/run/containerd/runc/k8s.io \
export CHECKPOINT_PATH=/tmp/ckpt-redis
```
## Checkpoint
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"checkpoint_data": {
"container_name": "'$CHECKPOINT_CONTAINER'",
"sandbox_name": "'$CHECKPOINT_SANDBOX'",
"namespace": "'$NAMESPACE'",
"checkpoint_path": "'$CHECKPOINT_PATH'",
"root": "'$ROOT'"
}
}' http://$CONTROLLER_URL:1324/checkpoint
```
## Restore
Once this completes, we need a new pod to restore the redis checkpoint into - effectively acting as a sleeping pod that we morph into a restored redis pod. We do this to avoid colliding with Kubernetes' scheduling and state management, so the users have full control over when pods are being restored.
```bash
k apply -f redis-restore-example.yaml
```
Then, performing a restore is as simple as another `curl` call:
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"checkpoint_data": {
"container_name": "'$RESTORE_CONTAINER'",
"sandbox_name": "'$RESTORE_SANDBOX'",
"namespace": "'$NAMESPACE'",
"checkpoint_path": "'$CHECKPOINT_PATH'",
"root": "'$ROOT'"
}
}' http://$CONTROLLER_URL:1324/restore
```
Finally connect to redis-cli once again to check if the new restored container has the previously set data:
```bash
HGETALL user:001
HGETALL user:002
```
# LLaMA Inference GPU Save, Migrate & Resume (SMR)
Performing an SMR of a running LLaMA inference task using HuggingFace weights.
Save, Migrate and Resume (SMR) has enormouse value for running inference workloads, especially when you're sensitive to cold starts. By virtue of what we're collecting from the GPU on a save (pretty much everything sitting in VRAM at the time) you can bypass all the initialization time and model optimizations that PyTorch or other inference engines (like vLLM or transformers-inference) do, and start inference workloads significantly faster.
Here's some benchmarking data to support that, which compares time to first token (TTFT) for a native start vs restoring from our checkpoint:
.cedana.ai/v2/scalar).
# User
Cedana API to get user information
## Get user
> Used to validate authentication token
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"User","description":"Cedana API to get user information"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/user":{"get":{"tags":["User"],"summary":"Get user","description":"Used to validate authentication token","operationId":"get","responses":{"200":{"description":"Returns UUID","content":{"text/plain":{"schema":{"type":"string"}}}},"401":{"description":"Invalid API key","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"User not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Service
Cedana API for service discovery
## GET /v2/discover/{name}
> Service discovery
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Service","description":"Cedana API for service discovery"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/discover/{name}":{"get":{"tags":["Service"],"summary":"Service discovery","operationId":"discovery","parameters":[{"name":"name","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns url for the service named","content":{"text/plain":{"schema":{"type":"string"}}}},"404":{"description":"Service not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to fetch service configuration","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Checkpoints
Cedana API for checkpoints
## List checkpoints
> Use query params to filter checkpoints. Supports filtering by \`ids\` (comma-separated UUIDs for single or multiple checkpoints)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Checkpoints","description":"Cedana API for checkpoints"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"Checkpoint":{"type":"object","description":"Checkpoint is basic unit for cedana service operation\nit stores all the information regarding the snapshot we can use to save and then restore state","required":["id","status","gpu","platform"],"properties":{"checksum":{"type":["string","null"]},"gpu":{"type":"string"},"id":{"type":"string","format":"uuid"},"info":{},"name":{"type":["string","null"]},"platform":{"type":"string"},"status":{"$ref":"#/components/schemas/CheckpointStatus"}}},"CheckpointStatus":{"type":"string","enum":["initializing","updated_info","possibly_uploaded","ready","deprecated"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoints":{"get":{"tags":["Checkpoints"],"summary":"List checkpoints","description":"Use query params to filter checkpoints. Supports filtering by `ids` (comma-separated UUIDs for single or multiple checkpoints)","operationId":"list","parameters":[{"name":"ids","in":"query","description":"Comma-separated list of checkpoint UUIDs to filter by","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"Returns checkpoints","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Checkpoint"}}}}},"400":{"description":"Invalid query parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Create checkpoint
> Builds a new checkpoint without the metadata and information about the checkpoint with status initializing
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Checkpoints","description":"Cedana API for checkpoints"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoints":{"post":{"tags":["Checkpoints"],"summary":"Create checkpoint","description":"Builds a new checkpoint without the metadata and information about the checkpoint with status initializing","operationId":"add","responses":{"200":{"description":"Returns the checkpoint ID for the new checkpoint","content":{"text/plain":{"schema":{"type":"string"}}}},"500":{"description":"Failed to create checkpoint in database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Deprecate checkpoint
> Marks checkpoint as deprecated, so that they can be removed on next internal state cleanup
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Checkpoints","description":"Cedana API for checkpoints"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoints/deprecate/{id}":{"patch":{"tags":["Checkpoints"],"summary":"Deprecate checkpoint","description":"Marks checkpoint as deprecated, so that they can be removed on next internal state cleanup","operationId":"deprecate","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Successfully deprecated checkpoint and returns id","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid UUID provided","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Put checkpoint info
> Add information about the checkpoint
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Checkpoints","description":"Cedana API for checkpoints"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CheckpointInfo":{"type":"object","required":["gpu","platform"],"properties":{"gpu":{"type":"string"},"info":{},"platform":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoints/info/{id}":{"put":{"tags":["Checkpoints"],"summary":"Put checkpoint info","description":"Add information about the checkpoint","operationId":"update_info","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string","format":"uuid"}}],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CheckpointInfo"}}},"required":true},"responses":{"200":{"description":"Successfully updated checkpoint info and returns ID","content":{"text/plain":{"schema":{"type":"string"}}}},"500":{"description":"Failed to execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Uploaded checkpoint
> Marks checkpoint as successfully uploaded and ready for restore
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Checkpoints","description":"Cedana API for checkpoints"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CheckpointSuccessInfo":{"type":"object","properties":{"restore_path":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoints/uploaded/{id}":{"post":{"tags":["Checkpoints"],"summary":"Uploaded checkpoint","description":"Marks checkpoint as successfully uploaded and ready for restore","operationId":"uploaded","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CheckpointSuccessInfo"}}},"required":true},"responses":{"200":{"description":"Marks the checkpoint as successfully uploaded","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid request parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to update checkpoint in database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Restores
Cedana API for restores
## Put restore info
> Add information about a restore
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Restores","description":"Cedana API for restores"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"RestoreInfo":{"type":"object","properties":{"error":{"type":["string","null"]},"profiling":{},"total_duration":{"type":["integer","null"],"format":"int64"},"total_io":{"type":["integer","null"],"format":"int64"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/restores/info/{id}":{"put":{"tags":["Restores"],"summary":"Put restore info","description":"Add information about a restore","operationId":"update_info","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string","format":"uuid"}}],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RestoreInfo"}}},"required":true},"responses":{"200":{"description":"Successfully updated restore info and returns ID","content":{"text/plain":{"schema":{"type":"string"}}}},"500":{"description":"Failed to execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Actions
Cedana API for actions
## GET /v2/actions
> List actions
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"Action":{"type":"object","required":["action_id","checkpoint_id","status","type","details","gpu","platform"],"properties":{"action_id":{"type":"string","format":"uuid"},"action_timestamp":{"type":["string","null"],"format":"date-time"},"checkpoint_completed_timestamp":{"type":["string","null"],"format":"date-time"},"checkpoint_id":{"type":"string","format":"uuid"},"details":{},"gpu":{"type":"string"},"node_name":{"type":["string","null"]},"platform":{"type":"string"},"reason":{"type":["string","null"]},"status":{"type":"string"},"total_duration":{"type":["integer","null"],"format":"int64"},"total_io":{"type":["integer","null"],"format":"int64"},"type":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/actions":{"get":{"tags":["Actions"],"summary":"List actions","operationId":"list","parameters":[{"name":"type","in":"query","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"Returns the actions with status","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Action"}}}}},"400":{"description":"Failed list","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from jobs table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/actions/from\_pod/{id}
> Get latest action belonging to a pod
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/actions/from_pod/{id}":{"get":{"tags":["Actions"],"summary":"Get latest action belonging to a pod","operationId":"latest_pod_action_id","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"The action_id of the given pod_id has been fetched successfully","content":{"text/plain":{"schema":{"type":"string"}}}},"404":{"description":"Failed to get action_id for the given pod_id","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from pods_to_actions table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## POST /v2/actions/upload
> Upload pod spec
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CheckpointStatusWithPodSpec":{"type":"object","required":["status","action_id"],"properties":{"action_id":{"type":"string"},"persistent_volume_claim":{},"pod_id":{"type":["string","null"]},"pod_metadata_spec":{},"pod_spec":{},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/actions/upload":{"post":{"tags":["Actions"],"summary":"Upload pod spec","operationId":"upload_podspec","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CheckpointStatusWithPodSpec"}}},"required":true},"responses":{"200":{"description":"Success","content":{"text/plain":{"schema":{"type":"string"}}}},"404":{"description":"Failed to get action_id within the db","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from pods_to_actions table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Get path of latest checkpoint
> Returns the path of the latest successful checkpoint for a given checkpoint name
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoint/path":{"get":{"tags":["Actions"],"summary":"Get path of latest checkpoint","description":"Returns the path of the latest successful checkpoint for a given checkpoint name","operationId":"get_latest_checkpoint_for_container_in_action","responses":{"200":{"description":"Returns the restore path","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Failed to get action for request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from jobs table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## POST /v2/checkpoint/pod
> Checkpoint pod
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CheckpointPod":{"type":"object","properties":{"action_id":{"type":["string","null"],"readOnly":true},"cluster_id":{"type":["string","null"]},"kind":{"$ref":"#/components/schemas/CheckpointKind"},"namespace":{"type":["string","null"]},"overrides":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/CheckpointOverrides"}]},"pod_id":{"type":["string","null"]},"pod_name":{"type":["string","null"]},"reason":{"$ref":"#/components/schemas/CheckpointReason"}}},"CheckpointKind":{"type":"string","enum":["simple","rootfs","rootfsonly"]},"CheckpointOverrides":{"type":"object","properties":{"asynchronous":{"type":"boolean"},"compression":{"type":["string","null"]},"criu_opts":{"type":["string","null"]},"directory":{"type":["string","null"]},"streams":{"type":"integer","format":"int32"}}},"CheckpointReason":{"type":"string","enum":["heartbeat","manual"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoint/pod":{"post":{"tags":["Actions"],"summary":"Checkpoint pod","operationId":"checkpoint_pod","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CheckpointPod"}}},"required":true},"responses":{"200":{"description":"Returns the ID for the task","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid request parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Failed to find checkpointable resource","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to create checkpoint or connect to eventstream","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Get status of checkpoint action
> Returns the current status of a checkpoint action
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"StatusResponse":{"type":"object","required":["status"],"properties":{"details":{"type":["string","null"]},"status":{"$ref":"#/components/schemas/CheckpointOperationStatus"}}},"CheckpointOperationStatus":{"type":"string","enum":["initialized","processing","checkpoint_created","ready","error","not_found"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/checkpoint/status/{action_id}":{"get":{"tags":["Actions"],"summary":"Get status of checkpoint action","description":"Returns the current status of a checkpoint action","operationId":"get_checkpoint_status","parameters":[{"name":"action_id","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns the current status of the checkpoint operation","content":{"application/json":{"schema":{"$ref":"#/components/schemas/StatusResponse"}}}},"404":{"description":"Action ID not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to retrieve status from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## POST /v2/restore/pod
> Restore pod
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions","description":"Cedana API for actions"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"RestorePod":{"type":"object","required":["action_id","reason"],"properties":{"action_id":{"type":"string"},"cluster_id":{"type":"string"},"overrides":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RestoreOverrides"}]},"reason":{"$ref":"#/components/schemas/RestoreReason"}}},"RestoreOverrides":{"type":"object","properties":{"criu_opts":{"type":["string","null"]},"pod_name":{"type":["string","null"]}}},"RestoreReason":{"type":"string","enum":["nodeTermination","nodeUnschedulable","manual"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/restore/pod":{"post":{"tags":["Actions"],"summary":"Restore pod","operationId":"restore_pod","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RestorePod"}}},"required":true},"responses":{"200":{"description":"Returns the action ID for the new restore","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid action_id or request parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to create restore or connect to eventstream","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Files
Cedana API for files
## GET /v2/files/dir/{path}
> Read directory contents
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Files","description":"Cedana API for files"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/files/dir/{path}":{"get":{"tags":["Files"],"summary":"Read directory contents","operationId":"read_dir","parameters":[{"name":"path","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns the list of files in the directory","content":{"application/json":{"schema":{"type":"array","items":{"type":"string"}}}}},"500":{"description":"Failed to list directory contents","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/files/{path}
> Download file
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Files","description":"Cedana API for files"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/files/{path}":{"get":{"tags":["Files"],"summary":"Download file","operationId":"get_file","parameters":[{"name":"path","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns the URL to download the file from","content":{"text/plain":{"schema":{"type":"string"}}}},"500":{"description":"Failed to generate download URL","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## PUT /v2/files/{path}
> Upload file
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Files","description":"Cedana API for files"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/files/{path}":{"put":{"tags":["Files"],"summary":"Upload file","operationId":"push_files","parameters":[{"name":"path","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns the URL to upload the file to","content":{"text/plain":{"schema":{"type":"string"}}}},"500":{"description":"Failed to generate upload URL","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Resources
Cedana API for resources
## GET /v2/cluster/count
> Get clusters total count
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"TotalCountResponse":{"type":"object","required":["total_count"],"properties":{"total_count":{"type":"integer","format":"int64"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/cluster/count":{"get":{"tags":["Resources"],"summary":"Get clusters total count","operationId":"clusters_count","responses":{"200":{"description":"Total count retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TotalCountResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs
> List jobs
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"JobResponse":{"type":"object","required":["id","name","namespace","status","age","monitored_by_policies","policy_count","pod_ids"],"properties":{"age":{"type":"string"},"completions":{"type":["integer","null"],"format":"int32"},"duration":{"type":["string","null"]},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"pod_ids":{"type":"array","items":{"type":"string"}},"policy_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs":{"get":{"tags":["Resources"],"summary":"List jobs","operationId":"job_list","responses":{"200":{"description":"Returns jobs","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/JobResponse"}}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/count
> Get total count of jobs
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"TotalCountResponse":{"type":"object","required":["total_count"],"properties":{"total_count":{"type":"integer","format":"int64"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs/count":{"get":{"tags":["Resources"],"summary":"Get total count of jobs","operationId":"jobs_count","responses":{"200":{"description":"Total count retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TotalCountResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/filter
>
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"paths":{"/v2/jobs/filter":{"get":{"tags":["Resources"],"operationId":"jobs_filter","parameters":[{"name":"priority","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int32"}},{"name":"namespace","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"status","in":"query","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"List of jobs with applied filters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/FilteredJobResponse"}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"FilteredJobResponse":{"type":"object","required":["jobs","total_count","available_priorities"],"properties":{"available_priorities":{"type":"array","items":{"type":"integer","format":"int32"}},"jobs":{"type":"array","items":{"$ref":"#/components/schemas/JobFilterResponse"}},"total_count":{"type":"integer","format":"int64"}}},"JobFilterResponse":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## GET /v2/jobs/namespaces
> Get job namespaces
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"NamespacesResponse":{"type":"object","required":["namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs/namespaces":{"get":{"tags":["Resources"],"summary":"Get job namespaces","operationId":"job_namespaces","responses":{"200":{"description":"Available namespaces retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/NamespacesResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/paginated
> List jobs (paginated)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PaginatedJobResponse":{"type":"object","required":["jobs","total_count"],"properties":{"available_filters":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AvailableFilters"}]},"jobs":{"type":"array","items":{"$ref":"#/components/schemas/JobResponse"}},"total_count":{"type":"integer","format":"int64"}}},"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}},"JobResponse":{"type":"object","required":["id","name","namespace","status","age","monitored_by_policies","policy_count","pod_ids"],"properties":{"age":{"type":"string"},"completions":{"type":["integer","null"],"format":"int32"},"duration":{"type":["string","null"]},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"pod_ids":{"type":"array","items":{"type":"string"}},"policy_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs/paginated":{"get":{"tags":["Resources"],"summary":"List jobs (paginated)","operationId":"job_list_paginated","parameters":[{"name":"limit","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int64"}},{"name":"offset","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int64"}},{"name":"sort","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"ascending","in":"query","required":false,"schema":{"type":["boolean","null"]}},{"name":"status","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"namespace","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"job_name","in":"query","description":"job name to query against (uses postgres ILIKE pattern search)","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"Returns paginated jobs with total count and available filters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PaginatedJobResponse"}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/priorities
> Get available job priorities with counts
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"api_key":[]}],"components":{"securitySchemes":{},"schemas":{"JobPrioritiesResponse":{"type":"object","required":["priorities","priority_counts"],"properties":{"priorities":{"type":"array","items":{"type":"integer","format":"int32"}},"priority_counts":{"type":"object","additionalProperties":{"type":"integer","format":"int64"},"propertyNames":{"type":"string"}}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs/priorities":{"get":{"tags":["Resources"],"summary":"Get available job priorities with counts","operationId":"job_priorities","responses":{"200":{"description":"Job priorities retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/JobPrioritiesResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/priority/{priority}
>
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"paths":{"/v2/jobs/priority/{priority}":{"get":{"tags":["Resources"],"operationId":"jobs_by_priority","parameters":[{"name":"priority","in":"path","description":"Priority value to filter jobs by","required":true,"schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"description":"List of jobs filtered by priority","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PaginatedJobResponse"}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"PaginatedJobResponse":{"type":"object","required":["jobs","total_count"],"properties":{"available_filters":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AvailableFilters"}]},"jobs":{"type":"array","items":{"$ref":"#/components/schemas/JobResponse"}},"total_count":{"type":"integer","format":"int64"}}},"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}},"JobResponse":{"type":"object","required":["id","name","namespace","status","age","monitored_by_policies","policy_count","pod_ids"],"properties":{"age":{"type":"string"},"completions":{"type":["integer","null"],"format":"int32"},"duration":{"type":["string","null"]},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"pod_ids":{"type":"array","items":{"type":"string"}},"policy_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## GET /v2/jobs/statuses
> Get job statuses
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"StatusesResponse":{"type":"object","required":["statuses"],"properties":{"statuses":{"type":"array","items":{"type":"string"}}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/jobs/statuses":{"get":{"tags":["Resources"],"summary":"Get job statuses","operationId":"job_statuses","responses":{"200":{"description":"Available statuses retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/StatusesResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/jobs/{job\_id}/pods
>
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"paths":{"/v2/jobs/{job_id}/pods":{"get":{"tags":["Resources"],"operationId":"job_pods","parameters":[{"name":"job_id","in":"path","description":"Job ID to get pods for","required":true,"schema":{"type":"string","format":"uuid"}}],"responses":{"200":{"description":"List of pods managed by the job","content":{"application/json":{"schema":{"$ref":"#/components/schemas/JobPodsResponse"}}}},"404":{"description":"Job not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"JobPodsResponse":{"type":"object","required":["job_id","job_name","pods","total_count"],"properties":{"job_id":{"type":"string","format":"uuid"},"job_name":{"type":"string"},"pods":{"type":"array","items":{"$ref":"#/components/schemas/JobPodResponse"}},"total_count":{"type":"integer","format":"int64"}}},"JobPodResponse":{"type":"object","required":["id","name","namespace","status"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"owner_reference":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## List nodes
> Will only return nodes from clusters with status 'active' if no params are provided. Also, nodes\
> with last\_sync older than 5 minutes are not returned.
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"NodeResponse":{"type":"object","required":["id","name","status","cluster_name","compute_type","instance_type","region"],"properties":{"cluster_name":{"type":"string"},"compute_type":{"type":"string"},"id":{"type":"string","format":"uuid"},"instance_type":{"type":"string"},"name":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/nodes":{"get":{"tags":["Resources"],"summary":"List nodes","description":"Will only return nodes from clusters with status 'active' if no params are provided. Also, nodes\nwith last_sync older than 5 minutes are not returned.","operationId":"node_list","parameters":[{"name":"id","in":"query","required":false,"schema":{"type":["string","null"],"format":"uuid"}}],"responses":{"200":{"description":"Returns nodes","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/NodeResponse"}}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/nodes/count
> Get nodes total count
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"TotalCountResponse":{"type":"object","required":["total_count"],"properties":{"total_count":{"type":"integer","format":"int64"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/nodes/count":{"get":{"tags":["Resources"],"summary":"Get nodes total count","operationId":"nodes_count","responses":{"200":{"description":"Total count retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TotalCountResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## List pods
> Will not return pods with status 'deleted', and only from clusters with status 'active' and\
> pods belonging to nodes whose last\_sync is within the last 5 minutes.
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PodResponse":{"type":"object","required":["id","name","namespace","status","age","ready","node","restarts","monitored_by_policies"],"properties":{"age":{"type":"string"},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"node":{"type":"string"},"ready":{"type":"string"},"restarts":{"type":"integer","format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/pods":{"get":{"tags":["Resources"],"summary":"List pods","description":"Will not return pods with status 'deleted', and only from clusters with status 'active' and\npods belonging to nodes whose last_sync is within the last 5 minutes.","operationId":"pod_list","parameters":[{"name":"id","in":"query","required":false,"schema":{"type":["string","null"],"format":"uuid"}}],"responses":{"200":{"description":"Returns pods","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/PodResponse"}}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/pods/count
> Get total count of pods
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"TotalCountResponse":{"type":"object","required":["total_count"],"properties":{"total_count":{"type":"integer","format":"int64"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/pods/count":{"get":{"tags":["Resources"],"summary":"Get total count of pods","operationId":"pods_count","responses":{"200":{"description":"Total count retrieved successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/TotalCountResponse"}}}},"500":{"description":"Database error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/pods/namespaces
> List pod namespaces
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"NamespacesResponse":{"type":"object","required":["namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/pods/namespaces":{"get":{"tags":["Resources"],"summary":"List pod namespaces","operationId":"pod_namespaces","responses":{"200":{"description":"Returns distinct pod namespaces","content":{"application/json":{"schema":{"$ref":"#/components/schemas/NamespacesResponse"}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/pods/paginated
> List pods (paginated)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PaginatedPodResponse":{"type":"object","required":["pods","total_count"],"properties":{"available_filters":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AvailableFilters"}]},"pods":{"type":"array","items":{"$ref":"#/components/schemas/PodResponse"}},"total_count":{"type":"integer","format":"int64"}}},"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}},"PodResponse":{"type":"object","required":["id","name","namespace","status","age","ready","node","restarts","monitored_by_policies"],"properties":{"age":{"type":"string"},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"node":{"type":"string"},"ready":{"type":"string"},"restarts":{"type":"integer","format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/pods/paginated":{"get":{"tags":["Resources"],"summary":"List pods (paginated)","operationId":"pod_list_paginated","parameters":[{"name":"id","in":"query","required":false,"schema":{"type":["string","null"],"format":"uuid"}},{"name":"limit","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int64"}},{"name":"offset","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int64"}},{"name":"sort","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"ascending","in":"query","required":false,"schema":{"type":["boolean","null"]}},{"name":"status","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"namespace","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"pod_name","in":"query","description":"pod name to query against (uses postgres ILIKE pattern search)","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"Returns paginated pods with total count and available filters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PaginatedPodResponse"}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/pods/statuses
> List pod statuses
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Resources","description":"Cedana API for resources"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"StatusesResponse":{"type":"object","required":["statuses"],"properties":{"statuses":{"type":"array","items":{"type":"string"}}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/pods/statuses":{"get":{"tags":["Resources"],"summary":"List pod statuses","operationId":"pod_statuses","responses":{"200":{"description":"Returns distinct pod statuses","content":{"application/json":{"schema":{"$ref":"#/components/schemas/StatusesResponse"}}}},"500":{"description":"Failed to fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Clusters
Cedana API for clusters
## List clusters
> Returns all clusters associated with the authenticated user's organization
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters","description":"Cedana API for clusters"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"Cluster":{"type":"object","required":["id","name","status"],"properties":{"id":{"type":"string","format":"uuid"},"last_sync":{"type":["string","null"],"format":"date-time"},"metadata":{},"name":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/cluster":{"get":{"tags":["Clusters"],"summary":"List clusters","description":"Returns all clusters associated with the authenticated user's organization","operationId":"list","responses":{"200":{"description":"Returns a list of clusters","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Cluster"}}}}},"500":{"description":"Failed to fetch clusters from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Create cluster
> Creates or updates info regarding a Kubernetes cluster
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters","description":"Cedana API for clusters"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CreateClusterRequest":{"type":"object","description":"CreateClusterRequest is the request body for creating a cluster","required":["cluster_name"],"properties":{"cluster_name":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/cluster":{"post":{"tags":["Clusters"],"summary":"Create cluster","description":"Creates or updates info regarding a Kubernetes cluster","operationId":"create","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateClusterRequest"}}},"required":true},"responses":{"200":{"description":"Returns the cluster id for the newly created or updated cluster","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Bad request or missing cluster name","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to create cluster in database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Sync cluster
> Syncs cluster resources such as pods, jobs, and nodes
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters","description":"Cedana API for clusters"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ClusterSyncRequest":{"type":"object","properties":{"cluster_id":{"type":["string","null"],"format":"uuid"},"metadata":{},"resource":{"$ref":"#/components/schemas/K8sResource"}}},"K8sResource":{"oneOf":[{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["None"]}}},{"allOf":[{"$ref":"#/components/schemas/Pod"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Pod"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Node"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Node"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Job"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Job"]}}}]}]},"Pod":{"type":"object","title":"Pod","description":"Entire json for the Kubernetes "},"Node":{"type":"object","title":"Node","description":"Entire json for the Kubernetes Node"},"Job":{"type":"object","title":"Job","description":"Entire json for the Kubernetes Job"},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/cluster/sync":{"post":{"tags":["Clusters"],"summary":"Sync cluster","description":"Syncs cluster resources such as pods, jobs, and nodes","operationId":"sync","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/ClusterSyncRequest"}}},"required":true},"responses":{"200":{"description":"Resource id registered internally","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"400":{"description":"Bad request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Cluster or node not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to sync resource to database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Create workload
> Takes a cluster name and workload specification, then creates a Kubernetes job\
> on the specified cluster
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters","description":"Cedana API for clusters"}],"paths":{"/v2/cluster/workload":{"post":{"tags":["Clusters"],"summary":"Create workload","description":"Takes a cluster name and workload specification, then creates a Kubernetes job\non the specified cluster","operationId":"create_workload","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/WorkloadReq"}}},"required":true},"responses":{"200":{"description":"Added workload to queue","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"400":{"description":"Error parsing body","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"WorkloadReq":{"type":"object","required":["cluster_id"],"properties":{"cluster_id":{"type":"string"},"cluster_name":{"type":["string","null"],"deprecated":true},"deployment":{},"job":{},"pod":{}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## Delete cluster
> This endpoint deletes a cluster and all its dependent entities in a single transaction\
> which is rolled back if any part of the deletion fails
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters","description":"Cedana API for clusters"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/cluster/{id}":{"delete":{"tags":["Clusters"],"summary":"Delete cluster","description":"This endpoint deletes a cluster and all its dependent entities in a single transaction\nwhich is rolled back if any part of the deletion fails","operationId":"delete","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string","format":"uuid"}}],"responses":{"200":{"description":"Delete the given cluster, and return Status OK"},"404":{"description":"Cluster not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to delete cluster from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Policy
Cedana API for policies
## POST /v2/policy/create
> Create policy
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PolicyRequest":{"type":"object","required":["heartbeat_time","resource_ids","cluster_id"],"properties":{"cluster_id":{"type":"string"},"heartbeat_time":{"type":"integer","format":"int64","minimum":0},"id":{"type":["string","null"]},"job_filter":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/JobFilter"}]},"policy_type":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyType"}]},"resource":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyResource"}]},"resource_ids":{"type":"array","items":{"type":"string","format":"uuid"}},"resource_name":{"type":"string"},"runc_root":{"type":["string","null"]}}},"JobFilter":{"type":"object","properties":{"namespace":{"type":["string","null"]},"priority":{"type":["integer","null"],"format":"int32"}}},"PolicyType":{"type":"string","enum":["heartbeat"]},"PolicyResource":{"type":"string","enum":["pods","namespace","jobs"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/create":{"post":{"tags":["Policy"],"summary":"Create policy","operationId":"policy_create","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/PolicyRequest"}}},"required":true},"responses":{"200":{"description":"Policy created successfully","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid request body","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to create policy in policy table","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## DELETE /v2/policy/delete
> Delete policy
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PolicyDeleteRequest":{"type":"object","required":["id"],"properties":{"id":{"type":"string","format":"uuid"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/delete":{"delete":{"tags":["Policy"],"summary":"Delete policy","operationId":"policy_delete","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/PolicyDeleteRequest"}}},"required":true},"responses":{"200":{"description":"Policy deleted successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"400":{"description":"Invalid request body","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to delete policy from policy table","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/policy/list
> List policies
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PolicyResponse":{"type":"object","required":["id","status","policy_type","input","resource","resource_list","pod_count"],"properties":{"id":{"type":"string"},"input":{"type":"integer","format":"int64"},"job_count":{"type":["integer","null"],"format":"int64"},"pod_count":{"type":"integer","format":"int64"},"policy_type":{"type":"string"},"resource":{"type":"string"},"resource_list":{"type":"array","items":{"type":"string"}},"status":{"type":"string"},"timestamp":{"type":["string","null"],"format":"date-time"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/list":{"get":{"tags":["Policy"],"summary":"List policies","operationId":"policy_list","responses":{"200":{"description":"Policies retrieved successfully","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/PolicyResponse"}}}}},"400":{"description":"Invalid request body","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to retriev policies from policy table","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## POST /v2/policy/sync-jobs
> Sync job policies manually
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"api_key":[]}],"components":{"securitySchemes":{},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/sync-jobs":{"post":{"tags":["Policy"],"summary":"Sync job policies manually","operationId":"sync_job_policies","responses":{"200":{"description":"Job policies synced successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to sync job policies"}}}}}}
```
## POST /v2/policy/update
> Update policy
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PolicyRequest":{"type":"object","required":["heartbeat_time","resource_ids","cluster_id"],"properties":{"cluster_id":{"type":"string"},"heartbeat_time":{"type":"integer","format":"int64","minimum":0},"id":{"type":["string","null"]},"job_filter":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/JobFilter"}]},"policy_type":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyType"}]},"resource":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyResource"}]},"resource_ids":{"type":"array","items":{"type":"string","format":"uuid"}},"resource_name":{"type":"string"},"runc_root":{"type":["string","null"]}}},"JobFilter":{"type":"object","properties":{"namespace":{"type":["string","null"]},"priority":{"type":["integer","null"],"format":"int32"}}},"PolicyType":{"type":"string","enum":["heartbeat"]},"PolicyResource":{"type":"string","enum":["pods","namespace","jobs"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/update":{"post":{"tags":["Policy"],"summary":"Update policy","operationId":"policy_update","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/PolicyRequest"}}},"required":true},"responses":{"200":{"description":"Policy updated successfully","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid request body","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to update policy from policy table","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/policy/{policy\_id}/job-status
> Get job policy tracking status
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"api_key":[]}],"components":{"securitySchemes":{},"schemas":{"JobPolicyStatus":{"type":"object","required":["policy_id","job_ids","tracked_pods","pod_count"],"properties":{"job_ids":{"type":"array","items":{"type":"string"}},"job_status":{"type":["string","null"]},"last_updated":{"type":["string","null"]},"pod_count":{"type":"integer","format":"int64"},"policy_id":{"type":"string"},"tracked_pods":{"type":"array","items":{"type":"string"}}}}}},"paths":{"/v2/policy/{policy_id}/job-status":{"get":{"tags":["Policy"],"summary":"Get job policy tracking status","operationId":"get_job_policy_status","parameters":[{"name":"policy_id","in":"path","description":"Policy ID to get status for","required":true,"schema":{"type":"string","format":"uuid"}}],"responses":{"200":{"description":"Job policy status","content":{"application/json":{"schema":{"$ref":"#/components/schemas/JobPolicyStatus"}}}},"404":{"description":"Policy not found"},"500":{"description":"Internal server error"}}}}}}
```
## GET /v2/policy/{policy\_id}/jobs
> Get jobs affected by a policy
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"api_key":[]}],"components":{"securitySchemes":{},"schemas":{"PolicyJobsResponse":{"type":"object","required":["policy_id","affected_jobs","count"],"properties":{"affected_jobs":{"type":"array","items":{"$ref":"#/components/schemas/PolicyJobInfo"}},"count":{"type":"integer","format":"int64"},"policy_id":{"type":"string"}}},"PolicyJobInfo":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"status":{"type":"string"}}}}},"paths":{"/v2/policy/{policy_id}/jobs":{"get":{"tags":["Policy"],"summary":"Get jobs affected by a policy","operationId":"policy_jobs","parameters":[{"name":"policy_id","in":"path","description":"Policy ID to get affected jobs for","required":true,"schema":{"type":"string","format":"uuid"}}],"responses":{"200":{"description":"Jobs affected by the policy","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PolicyJobsResponse"}}}},"404":{"description":"Policy not found or not a job policy"},"500":{"description":"Internal server error"}}}}}}
```
## GET /v2/policy/{policy\_id}/pods
> Get pods monitored by a specific policy
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy","description":"Cedana API for policies"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PolicyPodResponse":{"type":"object","required":["id","name","namespace","status"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/policy/{policy_id}/pods":{"get":{"tags":["Policy"],"summary":"Get pods monitored by a specific policy","operationId":"policy_pods","parameters":[{"name":"policy_id","in":"path","description":"Policy UUID","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns pods currently monitored by the policy","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/PolicyPodResponse"}}}}},"400":{"description":"Invalid policy_id format","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Policy not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Cloud Bursting
Cedana API for cloud bursting
## GET /v2/instances
> List instances
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"paths":{"/v2/instances":{"get":{"tags":["Cloud Bursting"],"summary":"List instances","operationId":"list_instances","responses":{"200":{"description":"List of instances","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/InstanceInfo"}}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"InstanceInfo":{"type":"object","required":["id","name","status","cloud_provider","region","instance_type"],"properties":{"cloud_provider":{"type":"string"},"created_at":{"type":["string","null"]},"id":{"type":"string"},"instance_type":{"type":"string"},"name":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## POST /v2/instances
> Create instance
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CreateInstanceRequest":{"type":"object","required":["cloud","region","shade_instance_type","shade_cloud","name"],"properties":{"alert":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/Alert"}]},"auto_delete":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AutoDelete"}]},"cloud":{"type":"string"},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"launch_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/LaunchConfiguration"}]},"name":{"type":"string"},"os":{"type":["string","null"]},"region":{"type":"string"},"shade_cloud":{"type":"boolean"},"shade_instance_type":{"type":"string"},"ssh_key_id":{"type":["string","null"]},"tags":{"type":["array","null"],"items":{"type":"string"}},"template_id":{"type":["string","null"]},"volume_ids":{"type":["array","null"],"items":{"type":"string"}},"volume_mount":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/VolumeMount"}]}}},"Alert":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}},"AutoDelete":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}},"EnvironmentVariable":{"type":"object","required":["name","value"],"properties":{"name":{"type":"string"},"value":{"type":"string"}}},"LaunchConfiguration":{"type":"object","required":["type"],"properties":{"docker_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/DockerConfiguration"}]},"script_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/ScriptConfiguration"}]},"type":{"type":"string"}}},"DockerConfiguration":{"type":"object","required":["image"],"properties":{"args":{"type":["string","null"]},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"image":{"type":"string"},"port_mappings":{"type":["array","null"],"items":{"$ref":"#/components/schemas/PortMapping"}},"registry_credentials":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RegistryCredentials"}]},"shared_memory_in_gb":{"type":["integer","null"],"format":"int32"},"volume_mounts":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VolumeMapping"}}}},"PortMapping":{"type":"object","required":["host_port","container_port"],"properties":{"container_port":{"type":"integer","format":"int32"},"host_port":{"type":"integer","format":"int32"}}},"RegistryCredentials":{"type":"object","required":["username","password"],"properties":{"password":{"type":"string"},"username":{"type":"string"}}},"VolumeMapping":{"type":"object","required":["host_path","container_path"],"properties":{"container_path":{"type":"string"},"host_path":{"type":"string"}}},"ScriptConfiguration":{"type":"object","required":["base64_script"],"properties":{"base64_script":{"type":"string"}}},"VolumeMount":{"type":"object","properties":{"auto":{"type":["boolean","null"]}}},"CreateInstanceResponse":{"type":"object","required":["instance_id","status","cloud_provider","region","instance_type"],"properties":{"cloud_provider":{"type":"string"},"instance_id":{"type":"string"},"instance_type":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/instances":{"post":{"tags":["Cloud Bursting"],"summary":"Create instance","operationId":"create_instance","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateInstanceRequest"}}},"required":true},"responses":{"200":{"description":"Instance created successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateInstanceResponse"}}}},"400":{"description":"Invalid request","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/instances/bursting-status
> Get configuration status
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"paths":{"/v2/instances/bursting-status":{"get":{"tags":["Cloud Bursting"],"summary":"Get configuration status","operationId":"get_bursting_status","responses":{"200":{"description":"Bursting configuration status","content":{"application/json":{"schema":{"$ref":"#/components/schemas/BurstingStatus"}}}}}}}},"components":{"schemas":{"BurstingStatus":{"type":"object","required":["status","message","can_use_bursting"],"properties":{"can_use_bursting":{"type":"boolean"},"message":{"type":"string"},"status":{"$ref":"#/components/schemas/BurstingConfigStatus"}}},"BurstingConfigStatus":{"type":"string","enum":["not_configured","configured","invalid"]}}}}
```
## GET /v2/instances/types
> Get available instance types
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"paths":{"/v2/instances/types":{"get":{"tags":["Cloud Bursting"],"summary":"Get available instance types","operationId":"get_instance_types","parameters":[{"name":"num_gpus","in":"query","description":"Filter by number of GPUs","required":false,"schema":{"type":"integer","format":"int32"}},{"name":"gpu_type","in":"query","description":"Filter by GPU type","required":false,"schema":{"type":"string"}},{"name":"sort","in":"query","description":"Sort results (e.g., 'price')","required":false,"schema":{"type":"string"}}],"responses":{"200":{"description":"Available instance types","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/InstanceType"}}}}},"500":{"description":"Internal server error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}},"components":{"schemas":{"InstanceType":{"type":"object","required":["cloud","shade_instance_type","cloud_instance_type","configuration","hourly_price","deployment_type","availability","boot_time"],"properties":{"availability":{"type":"array","items":{"$ref":"#/components/schemas/InstanceTypeAvailability"}},"boot_time":{"$ref":"#/components/schemas/InstanceTypeBootTime"},"cloud":{"type":"string"},"cloud_instance_type":{"type":"string"},"configuration":{"$ref":"#/components/schemas/InstanceTypeConfiguration"},"deployment_type":{"type":"string"},"hourly_price":{"type":"integer","format":"int32"},"shade_instance_type":{"type":"string"}}},"InstanceTypeAvailability":{"type":"object","required":["region","available","display_name"],"properties":{"available":{"type":"boolean"},"display_name":{"type":"string"},"region":{"type":"string"}}},"InstanceTypeBootTime":{"type":"object","required":["min_boot_in_sec","max_boot_in_sec"],"properties":{"max_boot_in_sec":{"type":"integer","format":"int32"},"min_boot_in_sec":{"type":"integer","format":"int32"}}},"InstanceTypeConfiguration":{"type":"object","required":["memory_in_gb","storage_in_gb","vcpus","num_gpus","gpu_type","interconnect","nvlink","vram_per_gpu_in_gb","os_options"],"properties":{"gpu_type":{"type":"string"},"interconnect":{"type":"string"},"memory_in_gb":{"type":"integer","format":"int32"},"num_gpus":{"type":"integer","format":"int32"},"nvlink":{"type":"boolean"},"os_options":{"type":"array","items":{"type":"string"}},"storage_in_gb":{"type":"integer","format":"int32"},"vcpus":{"type":"integer","format":"int32"},"vram_per_gpu_in_gb":{"type":"integer","format":"int32"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## DELETE /v2/instances/{instance\_id}
> Delete instance
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/instances/{instance_id}":{"delete":{"tags":["Cloud Bursting"],"summary":"Delete instance","operationId":"delete_instance","parameters":[{"name":"instance_id","in":"path","description":"Instance ID to delete","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Instance deleted successfully","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to delete instance","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/instances/{instance\_id}/status
> Get instance status
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Cloud Bursting","description":"Cedana API for cloud bursting"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"InstanceStatus":{"type":"object","required":["instance_id","status","metadata"],"properties":{"created_at":{"type":["string","null"]},"instance_id":{"type":"string"},"ip_address":{"type":["string","null"]},"metadata":{},"ssh_port":{"type":["integer","null"],"format":"int32"},"ssh_user":{"type":["string","null"]},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/instances/{instance_id}/status":{"get":{"tags":["Cloud Bursting"],"summary":"Get instance status","operationId":"get_instance_status","parameters":[{"name":"instance_id","in":"path","description":"Instance ID","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Instance status","content":{"application/json":{"schema":{"$ref":"#/components/schemas/InstanceStatus"}}}},"404":{"description":"Instance not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to get status","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Jobs (Slurm)
Cedana API for jobs (Slurm)
## List jobs
> Use query params to filter jobs
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Jobs (Slurm)","description":"Cedana API for jobs (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmJob":{"type":"object","required":["priority"],"properties":{"job_id":{"type":["string","null"]},"job_name":{"type":["string","null"]},"priority":{"type":"integer","format":"int32"},"state":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/jobs":{"get":{"tags":["Jobs (Slurm)"],"summary":"List jobs","description":"Use query params to filter jobs","operationId":"list","responses":{"200":{"description":"Returns jobs","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmJob"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## List jobs (paginated)
> Use query params to filter jobs
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Jobs (Slurm)","description":"Cedana API for jobs (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmJob":{"type":"object","required":["priority"],"properties":{"job_id":{"type":["string","null"]},"job_name":{"type":["string","null"]},"priority":{"type":"integer","format":"int32"},"state":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/jobs_paginated":{"get":{"tags":["Jobs (Slurm)"],"summary":"List jobs (paginated)","description":"Use query params to filter jobs","operationId":"list_paginated","parameters":[{"name":"offset","in":"query","description":"Number of records to skip (default: 0)","required":false,"schema":{"type":"integer","format":"int32"}},{"name":"limit","in":"query","description":"Maximum number of records to return (default: 50, max: 100)","required":false,"schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"description":"Returns jobs","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmJob"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Nodes (Slurm)
Cedana API for nodes (Slurm)
## List nodes
> Use query params to filter nodes
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Nodes (Slurm)","description":"Cedana API for nodes (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmNode":{"type":"object","required":["id"],"properties":{"cpus":{"type":["integer","null"],"format":"int32"},"id":{"type":"string","format":"uuid"},"memory":{"type":["integer","null"],"format":"int64"},"name":{"type":["string","null"]},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/nodes":{"get":{"tags":["Nodes (Slurm)"],"summary":"List nodes","description":"Use query params to filter nodes","operationId":"list","responses":{"200":{"description":"Returns nodes","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmNode"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## List nodes (paginated)
> Use query params to filter nodes
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Nodes (Slurm)","description":"Cedana API for nodes (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmNode":{"type":"object","required":["id"],"properties":{"cpus":{"type":["integer","null"],"format":"int32"},"id":{"type":"string","format":"uuid"},"memory":{"type":["integer","null"],"format":"int64"},"name":{"type":["string","null"]},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/nodes_paginated":{"get":{"tags":["Nodes (Slurm)"],"summary":"List nodes (paginated)","description":"Use query params to filter nodes","operationId":"list_paginated","parameters":[{"name":"offset","in":"query","description":"Number of records to skip (default: 0)","required":false,"schema":{"type":"integer","format":"int32"}},{"name":"limit","in":"query","description":"Maximum number of records to return (default: 50, max: 100)","required":false,"schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"description":"Returns nodes","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmNode"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Actions (Slurm)
Cedana API for actions (Slurm)
## POST /v2/slurm/checkpoint/job
> Checkpoint job
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions (Slurm)","description":"Cedana API for actions (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"CheckpointSlurmJob":{"type":"object","properties":{"action_id":{"type":["string","null"],"readOnly":true},"job_id":{"type":["string","null"]},"job_name":{"type":["string","null"]},"kind":{"$ref":"#/components/schemas/CheckpointKind"},"reason":{"$ref":"#/components/schemas/CheckpointReason"}}},"CheckpointKind":{"type":"string","enum":["simple","rootfs","rootfsonly"]},"CheckpointReason":{"type":"string","enum":["heartbeat","manual"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/checkpoint/job":{"post":{"tags":["Actions (Slurm)"],"summary":"Checkpoint job","operationId":"checkpoint_slurm_job","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CheckpointSlurmJob"}}},"required":true},"responses":{"200":{"description":"Returns the id for the task","content":{"text/plain":{"schema":{"type":"string"}}}},"404":{"description":"Failed to find checkpointable resource","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from jobs table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## POST /v2/slurm/restore/job
> Restore job
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Actions (Slurm)","description":"Cedana API for actions (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"RestoreSlurmJob":{"type":"object","required":["action_id","cluster_id","reason"],"properties":{"action_id":{"type":"string"},"cluster_id":{"type":"string"},"reason":{"$ref":"#/components/schemas/RestoreReason"}}},"RestoreReason":{"type":"string","enum":["nodeTermination","nodeUnschedulable","manual"]},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/restore/job":{"post":{"tags":["Actions (Slurm)"],"summary":"Restore job","operationId":"restore_slurm_job","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/RestoreSlurmJob"}}},"required":true},"responses":{"200":{"description":"Returns the checkpoint id for the new checkpoint","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Failed create checkpoint","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Failed create restore","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from jobs table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Clusters (Slurm)
Cedana API for clusters (Slurm)
## List clusters
> Use query params to filter clusters
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters (Slurm)","description":"Cedana API for clusters (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmCluster":{"type":"object","required":["id","name","status"],"properties":{"id":{"type":"string","format":"uuid"},"metadata":{},"name":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/clusters":{"get":{"tags":["Clusters (Slurm)"],"summary":"List clusters","description":"Use query params to filter clusters","operationId":"list","responses":{"200":{"description":"Returns clusters","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmCluster"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## List clusters (paginated)
> Use query params to filter clusters
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Clusters (Slurm)","description":"Cedana API for clusters (Slurm)"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmCluster":{"type":"object","required":["id","name","status"],"properties":{"id":{"type":"string","format":"uuid"},"metadata":{},"name":{"type":"string"},"status":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/clusters_paginated":{"get":{"tags":["Clusters (Slurm)"],"summary":"List clusters (paginated)","description":"Use query params to filter clusters","operationId":"list_paginated","parameters":[{"name":"offset","in":"query","description":"Number of records to skip (default: 0)","required":false,"schema":{"type":"integer","format":"int32"}},{"name":"limit","in":"query","description":"Maximum number of records to return (default: 50, max: 100)","required":false,"schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"description":"Returns clusters","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmCluster"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Policy (Slurm)
Cedana API for policies (Slurm)
## List maintenance windows
> Returns all maintenance windows
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy (Slurm)","description":"Cedana API for policies (Slurm)"}],"security":[{"Auth Token":[]}],"components":{"securitySchemes":{},"schemas":{"SlurmMaintenanceWindow":{"type":"object","properties":{"created_at":{"type":["string","null"],"format":"date-time"},"end_time":{"type":["string","null"],"format":"date-time"},"id":{"type":["string","null"]},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":["string","null"]},"start_time":{"type":["string","null"],"format":"date-time"},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/maintenance_window":{"get":{"tags":["Policy (Slurm)"],"summary":"List maintenance windows","description":"Returns all maintenance windows","operationId":"list_maintenance_windows","responses":{"200":{"description":"Returns all SLURM maintenance windows","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmMaintenanceWindow"}}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Create maintenance window
> Creates a new maintenance window for SLURM clusters
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy (Slurm)","description":"Cedana API for policies (Slurm)"}],"security":[{"Auth Token":[]}],"components":{"securitySchemes":{},"schemas":{"CreateMaintenanceWindow":{"type":"object","required":["start_time","end_time","reason"],"properties":{"end_time":{"type":"string","format":"date-time"},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":"string"},"start_time":{"type":"string","format":"date-time"},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/maintenance_window":{"post":{"tags":["Policy (Slurm)"],"summary":"Create maintenance window","description":"Creates a new maintenance window for SLURM clusters","operationId":"create_maintenance_window","requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/CreateMaintenanceWindow"}}},"required":true},"responses":{"201":{"description":"Returns the maintenance window id for the new maintenance window","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid request parameters","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Get maintenance window
> Returns a specific maintenance window by ID
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy (Slurm)","description":"Cedana API for policies (Slurm)"}],"security":[{"Auth Token":[]}],"components":{"securitySchemes":{},"schemas":{"SlurmMaintenanceWindow":{"type":"object","properties":{"created_at":{"type":["string","null"],"format":"date-time"},"end_time":{"type":["string","null"],"format":"date-time"},"id":{"type":["string","null"]},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":["string","null"]},"start_time":{"type":["string","null"],"format":"date-time"},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/maintenance_window/{id}":{"get":{"tags":["Policy (Slurm)"],"summary":"Get maintenance window","description":"Returns a specific maintenance window by ID","operationId":"get_maintenance_window","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns the maintenance window","content":{"application/json":{"schema":{"$ref":"#/components/schemas/SlurmMaintenanceWindow"}}}},"400":{"description":"Invalid UUID provided","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Maintenance window not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/fetch from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Update maintenance window
> Updates an existing maintenance window
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy (Slurm)","description":"Cedana API for policies (Slurm)"}],"security":[{"Auth Token":[]}],"components":{"securitySchemes":{},"schemas":{"UpdateMaintenanceWindow":{"type":"object","properties":{"end_time":{"type":["string","null"],"format":"date-time"},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":["string","null"]},"start_time":{"type":["string","null"],"format":"date-time"},"status":{"type":["string","null"]}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/maintenance_window/{id}":{"put":{"tags":["Policy (Slurm)"],"summary":"Update maintenance window","description":"Updates an existing maintenance window","operationId":"update_maintenance_window","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"$ref":"#/components/schemas/UpdateMaintenanceWindow"}}},"required":true},"responses":{"200":{"description":"Successfully updated maintenance window and returns id","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid UUID provided","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Maintenance window not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Delete maintenance window
> Deletes a maintenance window by ID
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Policy (Slurm)","description":"Cedana API for policies (Slurm)"}],"security":[{"Auth Token":[]}],"components":{"securitySchemes":{},"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/slurm/maintenance_window/{id}":{"delete":{"tags":["Policy (Slurm)"],"summary":"Delete maintenance window","description":"Deletes a maintenance window by ID","operationId":"delete_maintenance_window","parameters":[{"name":"id","in":"path","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Successfully deleted maintenance window and returns id","content":{"text/plain":{"schema":{"type":"string"}}}},"400":{"description":"Invalid UUID provided","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"404":{"description":"Maintenance window not found","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}},"500":{"description":"Failed to connect/execute request against database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Events
Cedana API for events
## List events
> Returns events within the specified time window, optionally filtered by operation or resource type.\
> The metadata field contains structured information about the event that can be used for filtering and display.
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Events","description":"Cedana API for events"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"Event":{"type":"object","description":"A single Event record with structured metadata","required":["id","status","operation","resource_type","message","metadata","source","timestamp"],"properties":{"id":{"type":"integer","format":"int64"},"message":{"type":"string"},"metadata":{},"operation":{"type":"string"},"resource_type":{"type":"string"},"source":{"type":"string"},"status":{"type":"string"},"timestamp":{"type":"integer","minimum":0}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/events":{"get":{"tags":["Events"],"summary":"List events","description":"Returns events within the specified time window, optionally filtered by operation or resource type.\nThe metadata field contains structured information about the event that can be used for filtering and display.","operationId":"list_events","parameters":[{"name":"time","in":"query","description":"Time window in seconds (default: 3600 = 1 hour)","required":false,"schema":{"type":"integer","format":"int32"}},{"name":"operation","in":"query","description":"Filter by operation (e.g., \"checkpoint\", \"restore\")","required":false,"schema":{"type":["string","null"]}},{"name":"resource_type","in":"query","description":"Filter by resource type (e.g., \"pod\", \"action\", \"checkpoint\")","required":false,"schema":{"type":["string","null"]}}],"responses":{"200":{"description":"Returns list of events","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Event"}}}}},"500":{"description":"Failed to fetch events from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Metrics
Cedana API for metrics
## GET /v2/metrics/k8
> Get metrics (K8s)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"NodeResourceMetricList":{"type":"object","required":["name"],"properties":{"details":{"type":"array","items":{"$ref":"#/components/schemas/NodeMetricItem"}},"name":{"type":"string"}}},"NodeMetricItem":{"type":"object","required":["metric_name","collector","value","timestamp"],"properties":{"collector":{"type":"string"},"cpu_core":{"type":"string"},"cpu_mode":{"type":"string"},"filesystem":{"type":"string"},"metric_name":{"type":"string"},"mountpoint":{"type":"string"},"resource":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0},"value":{"type":"string"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/k8":{"get":{"tags":["Metrics"],"summary":"Get metrics (K8s)","operationId":"get_k8_metrics","responses":{"200":{"description":"Returns k8 metrics from duckdb","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/NodeResourceMetricList"}}}}},"500":{"description":"Failed to connect/fetch from metrics table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm
> Get metrics (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmMetric":{"type":"object","required":["id","name","timestamp"],"properties":{"counter_value":{"type":["number","null"],"format":"double"},"gauge_value":{"type":["number","null"],"format":"double"},"id":{"type":"integer","format":"int64"},"kind":{"type":["string","null"]},"name":{"type":"string"},"tags":{},"timestamp":{"type":"integer","minimum":0}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm":{"get":{"tags":["Metrics"],"summary":"Get metrics (Slurm)","operationId":"get_slurm_metrics","responses":{"200":{"description":"Returns top 10 latest SLURM metrics from duckdb","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/SlurmMetric"}}}}},"500":{"description":"Failed to connect/fetch from SLURM metrics table in database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm/overview
> Get cluster overview (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmClusterOverview":{"type":"object","required":["timestamp"],"properties":{"cpu_load":{"type":["number","null"],"format":"double"},"cpus_idle":{"type":["number","null"],"format":"double"},"cpus_total":{"type":["number","null"],"format":"double"},"mem_alloc":{"type":["number","null"],"format":"double"},"mem_free":{"type":["number","null"],"format":"double"},"mem_real":{"type":["number","null"],"format":"double"},"timestamp":{"type":"integer","minimum":0}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/overview":{"get":{"tags":["Metrics"],"summary":"Get cluster overview (Slurm)","operationId":"get_slurm_cluster_overview","responses":{"200":{"description":"Returns latest SLURM cluster overview statistics","content":{"application/json":{"schema":{"$ref":"#/components/schemas/SlurmClusterOverview"}}}},"500":{"description":"Failed to fetch SLURM cluster overview","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm/overview/state
> Get state breakdown (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"SlurmStateBreakdown":{"type":"object","required":["timestamp","cpus_by_state","nodes_by_state"],"properties":{"cpus_by_state":{"type":"array","items":{"$ref":"#/components/schemas/StateItem"}},"nodes_by_state":{"type":"array","items":{"$ref":"#/components/schemas/StateItem"}},"timestamp":{"type":"integer","minimum":0}}},"StateItem":{"type":"object","required":["state","value"],"properties":{"state":{"type":"string"},"value":{"type":"number","format":"double"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/overview/state":{"get":{"tags":["Metrics"],"summary":"Get state breakdown (Slurm)","operationId":"get_slurm_state_breakdown","responses":{"200":{"description":"Returns state breakdown for CPUs and nodes","content":{"application/json":{"schema":{"$ref":"#/components/schemas/SlurmStateBreakdown"}}}},"500":{"description":"Failed to fetch SLURM state breakdown","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm/partitions
> Get partition statistics (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PartitionStats":{"type":"object","required":["partition_name","timestamp"],"properties":{"cpu_load":{"type":["number","null"],"format":"double"},"free_memory":{"type":["number","null"],"format":"double"},"idle_cpus":{"type":["number","null"],"format":"double"},"idle_nodes":{"type":["number","null"],"format":"double"},"partition_name":{"type":"string"},"real_memory":{"type":["number","null"],"format":"double"},"timestamp":{"type":"integer","minimum":0},"total_cpus":{"type":["number","null"],"format":"double"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/partitions":{"get":{"tags":["Metrics"],"summary":"Get partition statistics (Slurm)","operationId":"get_partition_stats","responses":{"200":{"description":"Returns list of all partitions with their statistics","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/PartitionStats"}}}}},"500":{"description":"Failed to fetch partition statistics","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## Get partition time series (Slurm)
> Returns CPU load and free memory time series for the specified partition over the last 10 data\
> points
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PartitionTimeSeries":{"type":"object","required":["partition_name","cpu_load","free_memory"],"properties":{"cpu_load":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"free_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"partition_name":{"type":"string"}}},"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/partitions/{name}/timeseries":{"get":{"tags":["Metrics"],"summary":"Get partition time series (Slurm)","description":"Returns CPU load and free memory time series for the specified partition over the last 10 data\npoints","operationId":"get_partition_timeseries","parameters":[{"name":"name","in":"path","description":"Partition name to query timeseries for","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Returns time series data for the specified partition","content":{"application/json":{"schema":{"$ref":"#/components/schemas/PartitionTimeSeries"}}}},"500":{"description":"Failed to fetch partition time series","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm/timeseries/cpu-load
> Get CPU load time series (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/timeseries/cpu-load":{"get":{"tags":["Metrics"],"summary":"Get CPU load time series (Slurm)","operationId":"get_cpu_load_timeseries","responses":{"200":{"description":"Returns CPU load time series (last 10 data points)","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}}}}},"500":{"description":"Failed to fetch CPU load time series","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
## GET /v2/metrics/slurm/timeseries/memory
> Get memory time series (Slurm)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Metrics","description":"Cedana API for metrics"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"MemoryTimeSeries":{"type":"object","required":["free_memory","allocated_memory"],"properties":{"allocated_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"free_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}}}},"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/metrics/slurm/timeseries/memory":{"get":{"tags":["Metrics"],"summary":"Get memory time series (Slurm)","operationId":"get_memory_timeseries","responses":{"200":{"description":"Returns memory time series (last 10 data points for free and allocated)","content":{"application/json":{"schema":{"$ref":"#/components/schemas/MemoryTimeSeries"}}}},"500":{"description":"Failed to fetch memory time series","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Logs
Cedana API for logs
## GET /v2/logs/k8
> Get logs (K8s)
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"tags":[{"name":"Logs","description":"Cedana API for logs"}],"security":[{"Token":[]}],"components":{"securitySchemes":{"Token":{"type":"http","scheme":"bearer"}},"schemas":{"PodLogList":{"type":"object","required":["pod_name","namespace"],"properties":{"logs":{"type":"array","items":{"$ref":"#/components/schemas/PodLogEntry"}},"namespace":{"type":"string"},"pod_name":{"type":"string"}}},"PodLogEntry":{"type":"object","required":["container_name","message","timestamp"],"properties":{"container_name":{"type":"string"},"message":{"type":"string"},"stream":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0}}},"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}},"paths":{"/v2/logs/k8":{"get":{"tags":["Logs"],"summary":"Get logs (K8s)","operationId":"get_k8_logs","parameters":[{"name":"pod","in":"query","required":false,"schema":{"type":["string","null"]}},{"name":"limit","in":"query","required":false,"schema":{"type":["integer","null"],"format":"int64"}}],"responses":{"200":{"description":"Returns k8 pod logs from duckdb","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/PodLogList"}}}}},"500":{"description":"Failed to connect/fetch from logs table from database","content":{"application/json":{"schema":{"$ref":"#/components/schemas/ApiError"}}}}}}}}}
```
# Models
## The Action object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Action":{"type":"object","required":["action_id","checkpoint_id","status","type","details","gpu","platform"],"properties":{"action_id":{"type":"string","format":"uuid"},"action_timestamp":{"type":["string","null"],"format":"date-time"},"checkpoint_completed_timestamp":{"type":["string","null"],"format":"date-time"},"checkpoint_id":{"type":"string","format":"uuid"},"details":{},"gpu":{"type":"string"},"node_name":{"type":["string","null"]},"platform":{"type":"string"},"reason":{"type":["string","null"]},"status":{"type":"string"},"total_duration":{"type":["integer","null"],"format":"int64"},"total_io":{"type":["integer","null"],"format":"int64"},"type":{"type":"string"}}}}}}
```
## The Alert object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Alert":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}}}}}
```
## The ApiError object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"ApiError":{"type":"object","description":"Standard error response for API endpoints","required":["message"],"properties":{"code":{"type":["string","null"],"description":"HTTP status code"},"details":{"description":"Optional additional error details"},"message":{"type":"string","description":"Error message describing what went wrong"}}}}}}
```
## The AutoDelete object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"AutoDelete":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}}}}}
```
## The AvailableFilters object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}}}}}
```
## The BurstingConfigStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"BurstingConfigStatus":{"type":"string","enum":["not_configured","configured","invalid"]}}}}
```
## The BurstingStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"BurstingStatus":{"type":"object","required":["status","message","can_use_bursting"],"properties":{"can_use_bursting":{"type":"boolean"},"message":{"type":"string"},"status":{"$ref":"#/components/schemas/BurstingConfigStatus"}}},"BurstingConfigStatus":{"type":"string","enum":["not_configured","configured","invalid"]}}}}
```
## The Checkpoint object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Checkpoint":{"type":"object","description":"Checkpoint is basic unit for cedana service operation\nit stores all the information regarding the snapshot we can use to save and then restore state","required":["id","status","gpu","platform"],"properties":{"checksum":{"type":["string","null"]},"gpu":{"type":"string"},"id":{"type":"string","format":"uuid"},"info":{},"name":{"type":["string","null"]},"platform":{"type":"string"},"status":{"$ref":"#/components/schemas/CheckpointStatus"}}},"CheckpointStatus":{"type":"string","enum":["initializing","updated_info","possibly_uploaded","ready","deprecated"]}}}}
```
## The CheckpointInfo object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointInfo":{"type":"object","required":["gpu","platform"],"properties":{"gpu":{"type":"string"},"info":{},"platform":{"type":"string"}}}}}}
```
## The CheckpointKind object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointKind":{"type":"string","enum":["simple","rootfs","rootfsonly"]}}}}
```
## The CheckpointOperationStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointOperationStatus":{"type":"string","enum":["initialized","processing","checkpoint_created","ready","error","not_found"]}}}}
```
## The CheckpointOverrides object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointOverrides":{"type":"object","properties":{"asynchronous":{"type":"boolean"},"compression":{"type":["string","null"]},"criu_opts":{"type":["string","null"]},"directory":{"type":["string","null"]},"streams":{"type":"integer","format":"int32"}}}}}}
```
## The CheckpointPod object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointPod":{"type":"object","properties":{"action_id":{"type":["string","null"],"readOnly":true},"cluster_id":{"type":["string","null"]},"kind":{"$ref":"#/components/schemas/CheckpointKind"},"namespace":{"type":["string","null"]},"overrides":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/CheckpointOverrides"}]},"pod_id":{"type":["string","null"]},"pod_name":{"type":["string","null"]},"reason":{"$ref":"#/components/schemas/CheckpointReason"}}},"CheckpointKind":{"type":"string","enum":["simple","rootfs","rootfsonly"]},"CheckpointOverrides":{"type":"object","properties":{"asynchronous":{"type":"boolean"},"compression":{"type":["string","null"]},"criu_opts":{"type":["string","null"]},"directory":{"type":["string","null"]},"streams":{"type":"integer","format":"int32"}}},"CheckpointReason":{"type":"string","enum":["heartbeat","manual"]}}}}
```
## The CheckpointReason object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointReason":{"type":"string","enum":["heartbeat","manual"]}}}}
```
## The CheckpointSlurmJob object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointSlurmJob":{"type":"object","properties":{"action_id":{"type":["string","null"],"readOnly":true},"job_id":{"type":["string","null"]},"job_name":{"type":["string","null"]},"kind":{"$ref":"#/components/schemas/CheckpointKind"},"reason":{"$ref":"#/components/schemas/CheckpointReason"}}},"CheckpointKind":{"type":"string","enum":["simple","rootfs","rootfsonly"]},"CheckpointReason":{"type":"string","enum":["heartbeat","manual"]}}}}
```
## The CheckpointStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointStatus":{"type":"string","enum":["initializing","updated_info","possibly_uploaded","ready","deprecated"]}}}}
```
## The CheckpointStatusWithPodSpec object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointStatusWithPodSpec":{"type":"object","required":["status","action_id"],"properties":{"action_id":{"type":"string"},"persistent_volume_claim":{},"pod_id":{"type":["string","null"]},"pod_metadata_spec":{},"pod_spec":{},"status":{"type":"string"}}}}}}
```
## The CheckpointSuccessInfo object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CheckpointSuccessInfo":{"type":"object","properties":{"restore_path":{"type":["string","null"]}}}}}}
```
## The Cluster object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Cluster":{"type":"object","required":["id","name","status"],"properties":{"id":{"type":"string","format":"uuid"},"last_sync":{"type":["string","null"],"format":"date-time"},"metadata":{},"name":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The ClusterSyncRequest object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"ClusterSyncRequest":{"type":"object","properties":{"cluster_id":{"type":["string","null"],"format":"uuid"},"metadata":{},"resource":{"$ref":"#/components/schemas/K8sResource"}}},"K8sResource":{"oneOf":[{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["None"]}}},{"allOf":[{"$ref":"#/components/schemas/Pod"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Pod"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Node"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Node"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Job"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Job"]}}}]}]},"Pod":{"type":"object","title":"Pod","description":"Entire json for the Kubernetes "},"Node":{"type":"object","title":"Node","description":"Entire json for the Kubernetes Node"},"Job":{"type":"object","title":"Job","description":"Entire json for the Kubernetes Job"}}}}
```
## The CreateClusterRequest object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CreateClusterRequest":{"type":"object","description":"CreateClusterRequest is the request body for creating a cluster","required":["cluster_name"],"properties":{"cluster_name":{"type":"string"}}}}}}
```
## The CreateInstanceRequest object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CreateInstanceRequest":{"type":"object","required":["cloud","region","shade_instance_type","shade_cloud","name"],"properties":{"alert":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/Alert"}]},"auto_delete":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AutoDelete"}]},"cloud":{"type":"string"},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"launch_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/LaunchConfiguration"}]},"name":{"type":"string"},"os":{"type":["string","null"]},"region":{"type":"string"},"shade_cloud":{"type":"boolean"},"shade_instance_type":{"type":"string"},"ssh_key_id":{"type":["string","null"]},"tags":{"type":["array","null"],"items":{"type":"string"}},"template_id":{"type":["string","null"]},"volume_ids":{"type":["array","null"],"items":{"type":"string"}},"volume_mount":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/VolumeMount"}]}}},"Alert":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}},"AutoDelete":{"type":"object","properties":{"date_threshold":{"type":["string","null"]},"spend_threshold":{"type":["string","null"]}}},"EnvironmentVariable":{"type":"object","required":["name","value"],"properties":{"name":{"type":"string"},"value":{"type":"string"}}},"LaunchConfiguration":{"type":"object","required":["type"],"properties":{"docker_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/DockerConfiguration"}]},"script_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/ScriptConfiguration"}]},"type":{"type":"string"}}},"DockerConfiguration":{"type":"object","required":["image"],"properties":{"args":{"type":["string","null"]},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"image":{"type":"string"},"port_mappings":{"type":["array","null"],"items":{"$ref":"#/components/schemas/PortMapping"}},"registry_credentials":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RegistryCredentials"}]},"shared_memory_in_gb":{"type":["integer","null"],"format":"int32"},"volume_mounts":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VolumeMapping"}}}},"PortMapping":{"type":"object","required":["host_port","container_port"],"properties":{"container_port":{"type":"integer","format":"int32"},"host_port":{"type":"integer","format":"int32"}}},"RegistryCredentials":{"type":"object","required":["username","password"],"properties":{"password":{"type":"string"},"username":{"type":"string"}}},"VolumeMapping":{"type":"object","required":["host_path","container_path"],"properties":{"container_path":{"type":"string"},"host_path":{"type":"string"}}},"ScriptConfiguration":{"type":"object","required":["base64_script"],"properties":{"base64_script":{"type":"string"}}},"VolumeMount":{"type":"object","properties":{"auto":{"type":["boolean","null"]}}}}}}
```
## The CreateInstanceResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CreateInstanceResponse":{"type":"object","required":["instance_id","status","cloud_provider","region","instance_type"],"properties":{"cloud_provider":{"type":"string"},"instance_id":{"type":"string"},"instance_type":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The CreateMaintenanceWindow object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"CreateMaintenanceWindow":{"type":"object","required":["start_time","end_time","reason"],"properties":{"end_time":{"type":"string","format":"date-time"},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":"string"},"start_time":{"type":"string","format":"date-time"},"status":{"type":["string","null"]}}}}}}
```
## The DockerConfiguration object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"DockerConfiguration":{"type":"object","required":["image"],"properties":{"args":{"type":["string","null"]},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"image":{"type":"string"},"port_mappings":{"type":["array","null"],"items":{"$ref":"#/components/schemas/PortMapping"}},"registry_credentials":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RegistryCredentials"}]},"shared_memory_in_gb":{"type":["integer","null"],"format":"int32"},"volume_mounts":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VolumeMapping"}}}},"EnvironmentVariable":{"type":"object","required":["name","value"],"properties":{"name":{"type":"string"},"value":{"type":"string"}}},"PortMapping":{"type":"object","required":["host_port","container_port"],"properties":{"container_port":{"type":"integer","format":"int32"},"host_port":{"type":"integer","format":"int32"}}},"RegistryCredentials":{"type":"object","required":["username","password"],"properties":{"password":{"type":"string"},"username":{"type":"string"}}},"VolumeMapping":{"type":"object","required":["host_path","container_path"],"properties":{"container_path":{"type":"string"},"host_path":{"type":"string"}}}}}}
```
## The EnvironmentVariable object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"EnvironmentVariable":{"type":"object","required":["name","value"],"properties":{"name":{"type":"string"},"value":{"type":"string"}}}}}}
```
## The Event object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Event":{"type":"object","description":"A single Event record with structured metadata","required":["id","status","operation","resource_type","message","metadata","source","timestamp"],"properties":{"id":{"type":"integer","format":"int64"},"message":{"type":"string"},"metadata":{},"operation":{"type":"string"},"resource_type":{"type":"string"},"source":{"type":"string"},"status":{"type":"string"},"timestamp":{"type":"integer","minimum":0}}}}}}
```
## The FilteredJobResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"FilteredJobResponse":{"type":"object","required":["jobs","total_count","available_priorities"],"properties":{"available_priorities":{"type":"array","items":{"type":"integer","format":"int32"}},"jobs":{"type":"array","items":{"$ref":"#/components/schemas/JobFilterResponse"}},"total_count":{"type":"integer","format":"int64"}}},"JobFilterResponse":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The InstanceInfo object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceInfo":{"type":"object","required":["id","name","status","cloud_provider","region","instance_type"],"properties":{"cloud_provider":{"type":"string"},"created_at":{"type":["string","null"]},"id":{"type":"string"},"instance_type":{"type":"string"},"name":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The InstanceStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceStatus":{"type":"object","required":["instance_id","status","metadata"],"properties":{"created_at":{"type":["string","null"]},"instance_id":{"type":"string"},"ip_address":{"type":["string","null"]},"metadata":{},"ssh_port":{"type":["integer","null"],"format":"int32"},"ssh_user":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The InstanceType object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceType":{"type":"object","required":["cloud","shade_instance_type","cloud_instance_type","configuration","hourly_price","deployment_type","availability","boot_time"],"properties":{"availability":{"type":"array","items":{"$ref":"#/components/schemas/InstanceTypeAvailability"}},"boot_time":{"$ref":"#/components/schemas/InstanceTypeBootTime"},"cloud":{"type":"string"},"cloud_instance_type":{"type":"string"},"configuration":{"$ref":"#/components/schemas/InstanceTypeConfiguration"},"deployment_type":{"type":"string"},"hourly_price":{"type":"integer","format":"int32"},"shade_instance_type":{"type":"string"}}},"InstanceTypeAvailability":{"type":"object","required":["region","available","display_name"],"properties":{"available":{"type":"boolean"},"display_name":{"type":"string"},"region":{"type":"string"}}},"InstanceTypeBootTime":{"type":"object","required":["min_boot_in_sec","max_boot_in_sec"],"properties":{"max_boot_in_sec":{"type":"integer","format":"int32"},"min_boot_in_sec":{"type":"integer","format":"int32"}}},"InstanceTypeConfiguration":{"type":"object","required":["memory_in_gb","storage_in_gb","vcpus","num_gpus","gpu_type","interconnect","nvlink","vram_per_gpu_in_gb","os_options"],"properties":{"gpu_type":{"type":"string"},"interconnect":{"type":"string"},"memory_in_gb":{"type":"integer","format":"int32"},"num_gpus":{"type":"integer","format":"int32"},"nvlink":{"type":"boolean"},"os_options":{"type":"array","items":{"type":"string"}},"storage_in_gb":{"type":"integer","format":"int32"},"vcpus":{"type":"integer","format":"int32"},"vram_per_gpu_in_gb":{"type":"integer","format":"int32"}}}}}}
```
## The InstanceTypeAvailability object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceTypeAvailability":{"type":"object","required":["region","available","display_name"],"properties":{"available":{"type":"boolean"},"display_name":{"type":"string"},"region":{"type":"string"}}}}}}
```
## The InstanceTypeBootTime object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceTypeBootTime":{"type":"object","required":["min_boot_in_sec","max_boot_in_sec"],"properties":{"max_boot_in_sec":{"type":"integer","format":"int32"},"min_boot_in_sec":{"type":"integer","format":"int32"}}}}}}
```
## The InstanceTypeConfiguration object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"InstanceTypeConfiguration":{"type":"object","required":["memory_in_gb","storage_in_gb","vcpus","num_gpus","gpu_type","interconnect","nvlink","vram_per_gpu_in_gb","os_options"],"properties":{"gpu_type":{"type":"string"},"interconnect":{"type":"string"},"memory_in_gb":{"type":"integer","format":"int32"},"num_gpus":{"type":"integer","format":"int32"},"nvlink":{"type":"boolean"},"os_options":{"type":"array","items":{"type":"string"}},"storage_in_gb":{"type":"integer","format":"int32"},"vcpus":{"type":"integer","format":"int32"},"vram_per_gpu_in_gb":{"type":"integer","format":"int32"}}}}}}
```
## The Job object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Job":{"type":"object","title":"Job","description":"Entire json for the Kubernetes Job"}}}}
```
## The JobFilter object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobFilter":{"type":"object","properties":{"namespace":{"type":["string","null"]},"priority":{"type":["integer","null"],"format":"int32"}}}}}}
```
## The JobFilterResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobFilterResponse":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The JobPodResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobPodResponse":{"type":"object","required":["id","name","namespace","status"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"owner_reference":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The JobPodsResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobPodsResponse":{"type":"object","required":["job_id","job_name","pods","total_count"],"properties":{"job_id":{"type":"string","format":"uuid"},"job_name":{"type":"string"},"pods":{"type":"array","items":{"$ref":"#/components/schemas/JobPodResponse"}},"total_count":{"type":"integer","format":"int64"}}},"JobPodResponse":{"type":"object","required":["id","name","namespace","status"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"owner_reference":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The JobPolicyStatus object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobPolicyStatus":{"type":"object","required":["policy_id","job_ids","tracked_pods","pod_count"],"properties":{"job_ids":{"type":"array","items":{"type":"string"}},"job_status":{"type":["string","null"]},"last_updated":{"type":["string","null"]},"pod_count":{"type":"integer","format":"int64"},"policy_id":{"type":"string"},"tracked_pods":{"type":"array","items":{"type":"string"}}}}}}}
```
## The JobPrioritiesResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobPrioritiesResponse":{"type":"object","required":["priorities","priority_counts"],"properties":{"priorities":{"type":"array","items":{"type":"integer","format":"int32"}},"priority_counts":{"type":"object","additionalProperties":{"type":"integer","format":"int64"},"propertyNames":{"type":"string"}}}}}}}
```
## The JobResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"JobResponse":{"type":"object","required":["id","name","namespace","status","age","monitored_by_policies","policy_count","pod_ids"],"properties":{"age":{"type":"string"},"completions":{"type":["integer","null"],"format":"int32"},"duration":{"type":["string","null"]},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"pod_ids":{"type":"array","items":{"type":"string"}},"policy_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The K8sResource object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"K8sResource":{"oneOf":[{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["None"]}}},{"allOf":[{"$ref":"#/components/schemas/Pod"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Pod"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Node"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Node"]}}}]},{"allOf":[{"$ref":"#/components/schemas/Job"},{"type":"object","required":["type"],"properties":{"type":{"type":"string","enum":["Job"]}}}]}]},"Pod":{"type":"object","title":"Pod","description":"Entire json for the Kubernetes "},"Node":{"type":"object","title":"Node","description":"Entire json for the Kubernetes Node"},"Job":{"type":"object","title":"Job","description":"Entire json for the Kubernetes Job"}}}}
```
## The LaunchConfiguration object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"LaunchConfiguration":{"type":"object","required":["type"],"properties":{"docker_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/DockerConfiguration"}]},"script_configuration":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/ScriptConfiguration"}]},"type":{"type":"string"}}},"DockerConfiguration":{"type":"object","required":["image"],"properties":{"args":{"type":["string","null"]},"envs":{"type":["array","null"],"items":{"$ref":"#/components/schemas/EnvironmentVariable"}},"image":{"type":"string"},"port_mappings":{"type":["array","null"],"items":{"$ref":"#/components/schemas/PortMapping"}},"registry_credentials":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RegistryCredentials"}]},"shared_memory_in_gb":{"type":["integer","null"],"format":"int32"},"volume_mounts":{"type":["array","null"],"items":{"$ref":"#/components/schemas/VolumeMapping"}}}},"EnvironmentVariable":{"type":"object","required":["name","value"],"properties":{"name":{"type":"string"},"value":{"type":"string"}}},"PortMapping":{"type":"object","required":["host_port","container_port"],"properties":{"container_port":{"type":"integer","format":"int32"},"host_port":{"type":"integer","format":"int32"}}},"RegistryCredentials":{"type":"object","required":["username","password"],"properties":{"password":{"type":"string"},"username":{"type":"string"}}},"VolumeMapping":{"type":"object","required":["host_path","container_path"],"properties":{"container_path":{"type":"string"},"host_path":{"type":"string"}}},"ScriptConfiguration":{"type":"object","required":["base64_script"],"properties":{"base64_script":{"type":"string"}}}}}}
```
## The MemoryTimeSeries object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"MemoryTimeSeries":{"type":"object","required":["free_memory","allocated_memory"],"properties":{"allocated_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"free_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}}}},"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}}}}}
```
## The NamespacesResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"NamespacesResponse":{"type":"object","required":["namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}}}}}}}
```
## The Node object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Node":{"type":"object","title":"Node","description":"Entire json for the Kubernetes Node"}}}}
```
## The NodeMetricItem object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"NodeMetricItem":{"type":"object","required":["metric_name","collector","value","timestamp"],"properties":{"collector":{"type":"string"},"cpu_core":{"type":"string"},"cpu_mode":{"type":"string"},"filesystem":{"type":"string"},"metric_name":{"type":"string"},"mountpoint":{"type":"string"},"resource":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0},"value":{"type":"string"}}}}}}
```
## The NodeResourceMetricList object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"NodeResourceMetricList":{"type":"object","required":["name"],"properties":{"details":{"type":"array","items":{"$ref":"#/components/schemas/NodeMetricItem"}},"name":{"type":"string"}}},"NodeMetricItem":{"type":"object","required":["metric_name","collector","value","timestamp"],"properties":{"collector":{"type":"string"},"cpu_core":{"type":"string"},"cpu_mode":{"type":"string"},"filesystem":{"type":"string"},"metric_name":{"type":"string"},"mountpoint":{"type":"string"},"resource":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0},"value":{"type":"string"}}}}}}
```
## The NodeResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"NodeResponse":{"type":"object","required":["id","name","status","cluster_name","compute_type","instance_type","region"],"properties":{"cluster_name":{"type":"string"},"compute_type":{"type":"string"},"id":{"type":"string","format":"uuid"},"instance_type":{"type":"string"},"name":{"type":"string"},"region":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The PaginatedJobResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PaginatedJobResponse":{"type":"object","required":["jobs","total_count"],"properties":{"available_filters":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AvailableFilters"}]},"jobs":{"type":"array","items":{"$ref":"#/components/schemas/JobResponse"}},"total_count":{"type":"integer","format":"int64"}}},"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}},"JobResponse":{"type":"object","required":["id","name","namespace","status","age","monitored_by_policies","policy_count","pod_ids"],"properties":{"age":{"type":"string"},"completions":{"type":["integer","null"],"format":"int32"},"duration":{"type":["string","null"]},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"pod_ids":{"type":"array","items":{"type":"string"}},"policy_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The PaginatedPodResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PaginatedPodResponse":{"type":"object","required":["pods","total_count"],"properties":{"available_filters":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/AvailableFilters"}]},"pods":{"type":"array","items":{"$ref":"#/components/schemas/PodResponse"}},"total_count":{"type":"integer","format":"int64"}}},"AvailableFilters":{"type":"object","required":["statuses","namespaces"],"properties":{"namespaces":{"type":"array","items":{"type":"string"}},"statuses":{"type":"array","items":{"type":"string"}}}},"PodResponse":{"type":"object","required":["id","name","namespace","status","age","ready","node","restarts","monitored_by_policies"],"properties":{"age":{"type":"string"},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"node":{"type":"string"},"ready":{"type":"string"},"restarts":{"type":"integer","format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The PartitionStats object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PartitionStats":{"type":"object","required":["partition_name","timestamp"],"properties":{"cpu_load":{"type":["number","null"],"format":"double"},"free_memory":{"type":["number","null"],"format":"double"},"idle_cpus":{"type":["number","null"],"format":"double"},"idle_nodes":{"type":["number","null"],"format":"double"},"partition_name":{"type":"string"},"real_memory":{"type":["number","null"],"format":"double"},"timestamp":{"type":"integer","minimum":0},"total_cpus":{"type":["number","null"],"format":"double"}}}}}}
```
## The PartitionTimeSeries object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PartitionTimeSeries":{"type":"object","required":["partition_name","cpu_load","free_memory"],"properties":{"cpu_load":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"free_memory":{"type":"array","items":{"$ref":"#/components/schemas/TimeSeriesDataPoint"}},"partition_name":{"type":"string"}}},"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}}}}}
```
## The Pod object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"Pod":{"type":"object","title":"Pod","description":"Entire json for the Kubernetes "}}}}
```
## The PodLogEntry object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PodLogEntry":{"type":"object","required":["container_name","message","timestamp"],"properties":{"container_name":{"type":"string"},"message":{"type":"string"},"stream":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0}}}}}}
```
## The PodLogList object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PodLogList":{"type":"object","required":["pod_name","namespace"],"properties":{"logs":{"type":"array","items":{"$ref":"#/components/schemas/PodLogEntry"}},"namespace":{"type":"string"},"pod_name":{"type":"string"}}},"PodLogEntry":{"type":"object","required":["container_name","message","timestamp"],"properties":{"container_name":{"type":"string"},"message":{"type":"string"},"stream":{"type":["string","null"]},"timestamp":{"type":"integer","minimum":0}}}}}}
```
## The PodResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PodResponse":{"type":"object","required":["id","name","namespace","status","age","ready","node","restarts","monitored_by_policies"],"properties":{"age":{"type":"string"},"id":{"type":"string","format":"uuid"},"monitored_by_policies":{"type":"array","items":{"type":"string"}},"name":{"type":"string"},"namespace":{"type":"string"},"node":{"type":"string"},"ready":{"type":"string"},"restarts":{"type":"integer","format":"int32"},"start_time":{"type":["string","null"]},"status":{"type":"string"}}}}}}
```
## The PolicyDeleteRequest object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyDeleteRequest":{"type":"object","required":["id"],"properties":{"id":{"type":"string","format":"uuid"}}}}}}
```
## The PolicyJobInfo object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyJobInfo":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"status":{"type":"string"}}}}}}
```
## The PolicyJobsResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyJobsResponse":{"type":"object","required":["policy_id","affected_jobs","count"],"properties":{"affected_jobs":{"type":"array","items":{"$ref":"#/components/schemas/PolicyJobInfo"}},"count":{"type":"integer","format":"int64"},"policy_id":{"type":"string"}}},"PolicyJobInfo":{"type":"object","required":["id","name","namespace","status","pod_count"],"properties":{"id":{"type":"string"},"name":{"type":"string"},"namespace":{"type":"string"},"pod_count":{"type":"integer","format":"int64"},"priority":{"type":["integer","null"],"format":"int32"},"status":{"type":"string"}}}}}}
```
## The PolicyPodResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyPodResponse":{"type":"object","required":["id","name","namespace","status"],"properties":{"id":{"type":"string","format":"uuid"},"name":{"type":"string"},"namespace":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The PolicyRequest object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyRequest":{"type":"object","required":["heartbeat_time","resource_ids","cluster_id"],"properties":{"cluster_id":{"type":"string"},"heartbeat_time":{"type":"integer","format":"int64","minimum":0},"id":{"type":["string","null"]},"job_filter":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/JobFilter"}]},"policy_type":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyType"}]},"resource":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/PolicyResource"}]},"resource_ids":{"type":"array","items":{"type":"string","format":"uuid"}},"resource_name":{"type":"string"},"runc_root":{"type":["string","null"]}}},"JobFilter":{"type":"object","properties":{"namespace":{"type":["string","null"]},"priority":{"type":["integer","null"],"format":"int32"}}},"PolicyType":{"type":"string","enum":["heartbeat"]},"PolicyResource":{"type":"string","enum":["pods","namespace","jobs"]}}}}
```
## The PolicyResource object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyResource":{"type":"string","enum":["pods","namespace","jobs"]}}}}
```
## The PolicyResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyResponse":{"type":"object","required":["id","status","policy_type","input","resource","resource_list","pod_count"],"properties":{"id":{"type":"string"},"input":{"type":"integer","format":"int64"},"job_count":{"type":["integer","null"],"format":"int64"},"pod_count":{"type":"integer","format":"int64"},"policy_type":{"type":"string"},"resource":{"type":"string"},"resource_list":{"type":"array","items":{"type":"string"}},"status":{"type":"string"},"timestamp":{"type":["string","null"],"format":"date-time"}}}}}}
```
## The PolicyType object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PolicyType":{"type":"string","enum":["heartbeat"]}}}}
```
## The PortMapping object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"PortMapping":{"type":"object","required":["host_port","container_port"],"properties":{"container_port":{"type":"integer","format":"int32"},"host_port":{"type":"integer","format":"int32"}}}}}}
```
## The RegistryCredentials object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RegistryCredentials":{"type":"object","required":["username","password"],"properties":{"password":{"type":"string"},"username":{"type":"string"}}}}}}
```
## The RestoreInfo object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RestoreInfo":{"type":"object","properties":{"error":{"type":["string","null"]},"profiling":{},"total_duration":{"type":["integer","null"],"format":"int64"},"total_io":{"type":["integer","null"],"format":"int64"}}}}}}
```
## The RestoreOverrides object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RestoreOverrides":{"type":"object","properties":{"criu_opts":{"type":["string","null"]},"pod_name":{"type":["string","null"]}}}}}}
```
## The RestorePod object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RestorePod":{"type":"object","required":["action_id","reason"],"properties":{"action_id":{"type":"string"},"cluster_id":{"type":"string"},"overrides":{"oneOf":[{"type":"null"},{"$ref":"#/components/schemas/RestoreOverrides"}]},"reason":{"$ref":"#/components/schemas/RestoreReason"}}},"RestoreOverrides":{"type":"object","properties":{"criu_opts":{"type":["string","null"]},"pod_name":{"type":["string","null"]}}},"RestoreReason":{"type":"string","enum":["nodeTermination","nodeUnschedulable","manual"]}}}}
```
## The RestoreReason object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RestoreReason":{"type":"string","enum":["nodeTermination","nodeUnschedulable","manual"]}}}}
```
## The RestoreSlurmJob object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"RestoreSlurmJob":{"type":"object","required":["action_id","cluster_id","reason"],"properties":{"action_id":{"type":"string"},"cluster_id":{"type":"string"},"reason":{"$ref":"#/components/schemas/RestoreReason"}}},"RestoreReason":{"type":"string","enum":["nodeTermination","nodeUnschedulable","manual"]}}}}
```
## The ScriptConfiguration object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"ScriptConfiguration":{"type":"object","required":["base64_script"],"properties":{"base64_script":{"type":"string"}}}}}}
```
## The SlurmCluster object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmCluster":{"type":"object","required":["id","name","status"],"properties":{"id":{"type":"string","format":"uuid"},"metadata":{},"name":{"type":"string"},"status":{"type":"string"}}}}}}
```
## The SlurmClusterOverview object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmClusterOverview":{"type":"object","required":["timestamp"],"properties":{"cpu_load":{"type":["number","null"],"format":"double"},"cpus_idle":{"type":["number","null"],"format":"double"},"cpus_total":{"type":["number","null"],"format":"double"},"mem_alloc":{"type":["number","null"],"format":"double"},"mem_free":{"type":["number","null"],"format":"double"},"mem_real":{"type":["number","null"],"format":"double"},"timestamp":{"type":"integer","minimum":0}}}}}}
```
## The SlurmJob object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmJob":{"type":"object","required":["priority"],"properties":{"job_id":{"type":["string","null"]},"job_name":{"type":["string","null"]},"priority":{"type":"integer","format":"int32"},"state":{"type":["string","null"]}}}}}}
```
## The SlurmMaintenanceWindow object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmMaintenanceWindow":{"type":"object","properties":{"created_at":{"type":["string","null"],"format":"date-time"},"end_time":{"type":["string","null"],"format":"date-time"},"id":{"type":["string","null"]},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":["string","null"]},"start_time":{"type":["string","null"],"format":"date-time"},"status":{"type":["string","null"]}}}}}}
```
## The SlurmMetric object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmMetric":{"type":"object","required":["id","name","timestamp"],"properties":{"counter_value":{"type":["number","null"],"format":"double"},"gauge_value":{"type":["number","null"],"format":"double"},"id":{"type":"integer","format":"int64"},"kind":{"type":["string","null"]},"name":{"type":"string"},"tags":{},"timestamp":{"type":"integer","minimum":0}}}}}}
```
## The SlurmNode object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmNode":{"type":"object","required":["id"],"properties":{"cpus":{"type":["integer","null"],"format":"int32"},"id":{"type":"string","format":"uuid"},"memory":{"type":["integer","null"],"format":"int64"},"name":{"type":["string","null"]},"status":{"type":["string","null"]}}}}}}
```
## The SlurmStateBreakdown object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"SlurmStateBreakdown":{"type":"object","required":["timestamp","cpus_by_state","nodes_by_state"],"properties":{"cpus_by_state":{"type":"array","items":{"$ref":"#/components/schemas/StateItem"}},"nodes_by_state":{"type":"array","items":{"$ref":"#/components/schemas/StateItem"}},"timestamp":{"type":"integer","minimum":0}}},"StateItem":{"type":"object","required":["state","value"],"properties":{"state":{"type":"string"},"value":{"type":"number","format":"double"}}}}}}
```
## The StateItem object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"StateItem":{"type":"object","required":["state","value"],"properties":{"state":{"type":"string"},"value":{"type":"number","format":"double"}}}}}}
```
## The StatusResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"StatusResponse":{"type":"object","required":["status"],"properties":{"details":{"type":["string","null"]},"status":{"$ref":"#/components/schemas/CheckpointOperationStatus"}}},"CheckpointOperationStatus":{"type":"string","enum":["initialized","processing","checkpoint_created","ready","error","not_found"]}}}}
```
## The StatusesResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"StatusesResponse":{"type":"object","required":["statuses"],"properties":{"statuses":{"type":"array","items":{"type":"string"}}}}}}}
```
## The TimeSeriesDataPoint object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"TimeSeriesDataPoint":{"type":"object","required":["timestamp","value"],"properties":{"timestamp":{"type":"integer","minimum":0},"value":{"type":"number","format":"double"}}}}}}
```
## The TotalCountResponse object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"TotalCountResponse":{"type":"object","required":["total_count"],"properties":{"total_count":{"type":"integer","format":"int64"}}}}}}
```
## The UpdateMaintenanceWindow object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"UpdateMaintenanceWindow":{"type":"object","properties":{"end_time":{"type":["string","null"],"format":"date-time"},"nodes":{"type":["array","null"],"items":{"type":"string"}},"partitions":{"type":["array","null"],"items":{"type":"string"}},"reason":{"type":["string","null"]},"start_time":{"type":["string","null"],"format":"date-time"},"status":{"type":["string","null"]}}}}}}
```
## The VolumeMapping object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"VolumeMapping":{"type":"object","required":["host_path","container_path"],"properties":{"container_path":{"type":"string"},"host_path":{"type":"string"}}}}}}
```
## The VolumeMount object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"VolumeMount":{"type":"object","properties":{"auto":{"type":["boolean","null"]}}}}}}
```
## The WorkloadReq object
```json
{"openapi":"3.1.0","info":{"title":"Cedana API","version":"0.0.0-staging"},"components":{"schemas":{"WorkloadReq":{"type":"object","required":["cluster_id"],"properties":{"cluster_id":{"type":"string"},"cluster_name":{"type":["string","null"],"deprecated":true},"deployment":{},"job":{},"pod":{}}}}}}
```
# Cedana Daemon
" %}
## [CRIU](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L108-L117)
{% @github-files/github-code-block url="" %}
## [Checkpoint](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L69-L85)
{% @github-files/github-code-block url="" %}
## [Client](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L103-L106)
{% @github-files/github-code-block url="" %}
## [Connection](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L60-L67)
{% @github-files/github-code-block url="" %}
## [DB](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L87-L92)
{% @github-files/github-code-block url="" %}
## [GPU](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L119-L132)
{% @github-files/github-code-block url="" %}
## [Plugins](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L134-L143)
{% @github-files/github-code-block url="" %}
## [Profiling](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L94-L101)
{% @github-files/github-code-block url="" %}
## [AWS](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L145-L154)
{% @github-files/github-code-block url="" %}
## [SLURM](https://github.com/cedana/cedana/blob/d7e70263c563d6b24367f105be4fdc4ecd13aeb1/pkg/config/types.go#L45-L58)
{% @github-files/github-code-block url="" %}
# Health checks
Cedana can be health checked to ensure it fully supports the system and is ready to accept requests.
## Basic
To do a simple health check, ensure the daemon is running, and then run:
```sh
cedana daemon check
```
This just checks for basic checkpoint/restore capabilities on the host.
## Complete
To do a more complete health check, ensure the daemon is running, and then run:
```sh
cedana daemon check --full
```
This health checks all the installed plugins, including some optional checks. If you have installed plugins, you should see an output similar to:
```
CRIU
version 40100 ✔
features available ✔
GPU
status available ✔
version ✔
device count 1 ✔
driver API 12040 ✔
STREAMER
version v0.0.6 ✔
criu supported ✔
pipe pages soft limit 16 KiB • For optimal performance, `echo 0 >
/proc/sys/fs/pipe-user-pages-soft`
pipe pages hard limit unlimited ✔
pipe max size 1 MiB • For optimal performance, `echo 4194304 >
/proc/sys/fs/pipe-max-size`
RUNC
version v0.9.239-1-g9acef6f ✔
runc binary available ✔
runc version 1.2.4 ✔
runc spec 1.2.0 ✔
runc libseccomp 2.5.5 ✔
CONTAINERD
version v0.9.239-1-g9acef6f ✔
containerd version v2.0.1 ✔
containerd revision 88aa2f531d6c2922003cc7929e51daf1c14caa0a ✔
containerd runtime io.containerd.runc.v2 ✔
Looks good, with 2 warning(s).
```
## Without daemon
You can also run the health check directly, without the daemon:
```sh
cedana check --full
```
Warnings are shown in yellow and are usually related to system configuration or performance. They are not critical but may affect the performance of Cedana.
# Plugins
Plugins extend the support of Cedana's checkpoint/restore to various container runtimes, GPUs, etc. Plugins can either be installed from the online registry or built locally.
## Online registry
{% hint style="success" %}
To access released plugins, you need to be authenticated. See [authentication](/daemon/get-started/authentication).
{% endhint %}
You can list all available plugins with:
```sh
cedana plugin list
```
```
PLUGIN SIZE STATUS INSTALLED VERSION AVAILABLE VERSION PUBLISHED
criu 6 MiB outdated v3.9 v4.0 one month ago
runc 35 MiB installed v0.9.239 v0.9.239 2 hours ago
containerd 38 MiB installed v0.9.239 v0.9.239 2 hours ago
crio 27 MiB installed v0.9.239 v0.9.239 2 hours ago
k8s 32 MiB installed v0.9.239 v0.9.239 2 hours ago
gpu 32 MiB available v0.4.7 2 minutes ago
streamer 1 MiB available v0.0.6 9 days ago
7 installed, 0 available
```
This will show you all the plugins that are available, installed, or outdated.
You can also list specific plugins by name & version:
```sh
cedana plugin list gpu@v0.5.0 streamer@v0.0.5
```
```sh
PLUGIN SIZE STATUS INSTALLED VERSION AVAILABLE VERSION PUBLISHED
gpu 48 MiB available v0.5.0 one day ago
streamer 6 MiB available v0.0.5 4 months ago
0 installed, 2 available
```
## Locally built plugins
If plugins are built locally (in the current directory), running `cedana plugin list` in the current directory will show the locally built plugins instead:
```
PLUGIN SIZE STATUS INSTALLED VERSION AVAILABLE VERSION PUBLISHED
criu 6 MiB outdated v3.9 v4.0 one month ago
runc 35 MiB installed v0.9.239 local 2 hours ago
containerd 38 MiB installed v0.9.239 local 2 hours ago
crio 27 MiB installed v0.9.239 local 2 hours ago
k8s 32 MiB installed v0.9.239 local 2 hours ago
gpu 32 MiB available v0.4.7 2 minutes ago
streamer 1 MiB available v0.0.6 9 days ago
7 installed, 0 available
```
Notice the `LATEST VERSION` column shows `local` for locally built plugins.
Instead of depending on current directory, you can also specify the paths to search for locally built plugins by setting the `CEDANA_PLUGINS_LOCAL_SEARCH_PATH` (paths are colon-separated just like the `PATH` env var). This is convenient for developers who are working on multiple plugins at once.
{% hint style="info" %}
Check out the guide on [writing plugins](/daemon/developer-guides/writing_plugins) if you're interested in building your own plugins.
{% endhint %}
## Install
Once a plugin appears in the list, you can install it with:
```sh
sudo cedana plugin install @ ...
```
## Uninstall
To uninstall a plugin, use:
```sh
sudo cedana plugin remove ...
```
## Features
To see which features are supported by the installed plugins, use:
```sh
cedana plugin features
```
{% hint style="info" %}
See [features](/daemon/get-started/features) for more information.
{% endhint %}
## Health check
The full health check command will also check the health of all installed plugins. See [health checks](/daemon/get-started/health).
{% hint style="info" %}
Check out the [CLI reference](/daemon/references/cli/cedana_plugin) for all plugin-related subcommands.
{% endhint %}
# Features
Run `cedana features` (shorthand for `cedana plugin features`) to see the features currently supported by each plugin.
```
FEATURE CLOUD-HYPERVISOR RUNC CONTAINERD CRIO KATA STORAGE/CEDANA STORAGE/S3 STORAGE/GCS K8S SLURM
Dump command • ✔ ✔ • • • • • • —
Restore command • ✔ ✔ • • • • • • —
Run command • ✔ ✔ • • • • • • —
Manage command • ✔ ✔ • • • • • • —
Freeze command • ✔ ✔ • • • • • • —
Unfreeze command • ✔ ✔ • • • • • • —
Query command • ✔ ✔ • • • • • ✔ —
Helper command(s) • • • • • • • • ✔ —
Dump middleware • ✔ ✔ ✔ • • • • • —
Dump handler • • • • • • • • • —
Dump VM middleware ✔ • • • ✔ • • • • —
Dump VM handler ✔ • • • • • • • • —
Restore middleware • ✔ ✔ ✔ • • • • • —
Restore middleware (late) • ✔ • • • • • • • —
Restore handler • • ✔ • • • • • • —
Restore VM middleware ✔ • • • ✔ • • • • —
Restore VM handler ✔ • • • • • • • • —
Freeze handler • ✔ ✔ • • • • • • —
Unfreeze handler • ✔ ✔ • • • • • • —
Run handler • ✔ ✔ ✔ • • • • • —
Run handler (serverless) • ✔ • • • • • • • —
Run middleware • ✔ ✔ ✔ • • • • • —
Run middleware (late) • • • • • • • • • —
Manage handler • ✔ ✔ • • • • • • —
Custom kill signal • ✔ ✔ • ✔ • • • • —
Custom cleanup • ✔ ✔ • • • • • • —
Custom reaper • ✔ • • • • • • • —
GPU interception • ✔ ✔ • • • • • • —
GPU interception restore • ✔ ✔ • • • • • • —
GPU tracing • ✔ ✔ • • • • • • —
GPU tracing restore • ✔ ✔ • • • • • • —
Checkpoint storage • • • • • ✔ ✔ • • —
Query handler • ✔ ✔ • • • • • ✔ —
Health checks • ✔ ✔ • ✔ ✔ ✔ • • —
✔ = implemented, • = unimplemented, — = not installed, ✖ = incompatible
Not showing external plugins: criu, criu/cuda, gpu, gpu/tracer, streamer, containerd/runtime-runc
```
When developing a new plugin, or adding a feature to a plugin, it's helpful to see more information on any compatibility issues. Use `cedana features --errors` to see any incompatibility errors.
{% hint style="info" %}
Check out the guide on [writing plugins](/daemon/developer-guides/writing_plugins) for more information.
{% endhint %}
{% hint style="info" %}
Check out the [CLI reference](/daemon/references/cli/cedana_plugin) for all plugin-related subcommands.
{% endhint %}
# Managed process/container
The Cedana daemon is designed to manage the entire lifecycle of a process/container, including checkpoint/restore, in the larger Cedana system.
Managed processes/containers are those that are spawned using `cedana run` ([CLI reference](/daemon/references/cli/cedana_run)). This command creates a managed job, which can be checkpointed and restored using `cedana dump job` and `cedana restore job` subcommands.
...
```
Where `` can be `process`, `containerd`, `runc`, etc. See [features](/daemon/get-started/features) for all plugins that support running managed jobs.
For example, to run a new managed process:
```sh
cedana run process --jid -- echo "Hello, world!"
```
The `--jid` flag is optional, and if not provided, a random job ID will be generated.
## Manage an existing job
It's also possible to start managing an existing process/container:
```sh
cedana manage ...
```
Where `` can be `process`, `containerd`, `runc`, etc. See [features](/daemon/get-started/features) for all plugins that support managing existing jobs.
For example, to manage an existing process:
```sh
cedana manage process [args...]
```
## Manage an upcoming job
It's also possible to start managing an upcoming process/container:
```sh
cedana manage --upcoming ...
```
Where `` can be `process`, `containerd`, `runc`, etc. See [features](/daemon/get-started/features) for all plugins that support managing upcoming jobs. The manage command will block and wait for the process/container to become available.
For example, to manage an existing runc container:
```sh
cedana manage --upcoming runc [args...]
```
## List managed jobs
The subcommand `cedana job` has many subcommands such as `list`, `kill`,m `delete`, etc. Check the [CLI reference](/daemon/references/cli/cedana_job) for all available subcommands.
```sh
cedana job list
```
```
JOB TYPE PID STATUS GPU CHECKPOINT SIZE LOG
used_gould8 process 5336 halted yes /var/log/cedana-output-used_gould8.log
clean_davinci7 runc 3680 halted no
easy_tu9 process 3892 halted yes
```
`cedana ps` is a shorthand for `cedana job list`.
## Attach I/O
To attach to the I/O of a job, use the `--attach` flag:
```sh
cedana run --attach ...
```
This will attach the standard input, output, and error streams to the terminal, including the job's exit status. Press `Ctrl+C` to detach, and the job will continue running in the background.
If you list the jobs, you will see that the job is attachable:
```
JOB TYPE PID STATUS GPU CHECKPOINT SIZE LOG
clean_davinci7 runc 3680 halted no [Attachable]
```
To attach to the job again, use the `cedana job attach` subcommand:
```sh
cedana job attach
```
Or the generic `cedana attach` if you want to attach using the job's PID:
```sh
cedana attach
```
If you want to make a job attachable, but not attach to it immediately, you can use the `--attachable` flag:
```sh
cedana run --attachable ...
```
## View job logs
By default, the jobs stdout/stderr are stored in the `/var/log/` directory. If you do `cedana job list`, you will see the path to the log file.
```
JOB TYPE PID STATUS GPU CHECKPOINT SIZE LOG
used_gould8 process 5336 halted yes /var/log/cedana-output-used_gould8.log
```
You can also pass a custom log path. See [CLI reference](/daemon/references/cli/cedana_run) for `cedana run` subcommand.
If the job has attachable I/O, it will appear as such:
```
JOB TYPE PID STATUS GPU CHECKPOINT SIZE LOG
used_gould8 process 5336 halted yes [Attachable]
```
## Checkpoint/restore
Once the daemon has started managing a job, it can be checkpointed and restored using the `cedana dump job` and `cedana restore job` subcommands. See [managed checkpoint/restore basics](/daemon/checkpoint-restore/cr#managed-checkpoint-restore) for more information.
## Checkpoint/restore with GPUs
GPU C/R support is also available for managed jobs. Check out the [checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1) guide for more information.
# Checkpoint/restore basics
The Cedana daemon is designed to checkpoint/restore processes as well as containers.
## Checkpoint
To checkpoint:
```sh
cedana dump ...
```
Where `` can be `process`, `containerd`, `runc`, `job`, etc. See [features](/daemon/get-started/features) for all plugins that support checkpointing.
For example, to checkpoint a process:
```sh
cedana dump process --dir /tmp
```
A `--dir` flag can be used to specify the *parent* directory where the checkpoint will be stored. If not provided, the checkpoint will be stored in the default checkpoint directory as specified in the [configuration](/daemon/get-started/configuration), or in `/tmp` if not set. You may also specify a `--name` flag to give a custom name to the checkpoint file.
{% hint style="info" %}
See [CLI reference](/daemon/references/cli/cedana_dump_process) for all available options for process checkpoint.
{% endhint %}
## Restore
### Using daemon
```sh
cedana restore ...
```
Where `` can be `process`, `containerd`, `runc`, `job`, etc. See [features](/daemon/get-started/features) for all plugins that support restoring.
For example, to restore a process:
```sh
cedana restore process --path
```
Notice that for restore the flag is called `--path` instead of `--dir` (as in dump), this is because it can be a path to a compressed file, or to a directory if not compressed.
### Without daemon
{% hint style="warning" %}
Not all plugins support restoring without the daemon. Please check [features](/daemon/get-started/features) for details.
{% endhint %}
It's also possible to restore directly as a child of the current shell command without the daemon:
```sh
cedana restore process --path --no-server
```
This is useful for scenarios where you want to restore a process as a child of the current shell, for example, to restore a shell process and interact with it directly.
{% hint style="info" %}
See [CLI reference](/daemon/references/cli/cedana_restore_process) for all available options for process restore.
{% endhint %}
## Managed checkpoint/restore
As explained in [managed process/container](/daemon/checkpoint-restore/managed), a job can be of any type, and thus can be checkpointed and restored using the `cedana dump job` and `cedana restore job` subcommands.
The `cedana dump/restore job` subcommands have the same options as their non-managed counterparts, but with pretty good defaults. For e.g., the `--path` flag is not required for `cedana restore job`, as the checkpoint path is stored in the job metadata.
If you do `cedana job list` after checkpointing a job, you will see the latest checkpoint time and size:
```
JOB TYPE PID STATUS GPU CHECKPOINT SIZE LOG
famous_hopper7 process 32675 halted no 3 seconds ago 610 KiB
```
To view all checkpoints for a job, use `cedana job checkpoints `:
```
ID TIME SIZE PATH
141d52b4-0d1f-4911-a0da-abfab3358d16 2025-02-19 12:32:01 586 KiB /tmp/dump-process-famous_hopper7-1739986321.tar
386dcce4-a29d-4acb-ab03-12d41b7c42ce 2025-02-19 12:30:36 610 KiB /tmp/dump-process-famous_hopper7-1739986236.tar
```
## Compression
The `cedana dump` subcommand supports a `--compression` flag to specify the compression algorithm to use. For example:
```sh
cedana dump process --dir /tmp --name xyz --compression gzip
```
This will create a compressed checkpoint file with the path `/tmp/xyz.tar.gz`. The `--name` flag is optional, and if not provided, the daemon will choose a unique name based on some metadata.
When restoring, the daemon will automatically detect the compression algorithm used and decompress the file. Simply provide the path to the compressed file:
```sh
cedana restore process --path /tmp/xyz.tar.gz
```
Supported values for `--compression` are `none`, `tar`, `gzip`, `lz4`, `zlib`.
{% hint style="info" %}
You may also specify the default compression algorithm in the [configuration](/daemon/get-started/configuration).
{% endhint %}
## Remote storage
Cedana supports checkpointing/restoring to/from remote storage, through storage plugins. Check out the following guides for specific remote storage:
* [Amazon S3](/daemon/storage/s3)
* [Google Cloud Storage](/daemon/storage/gcs)
* [Cedana Storage](/daemon/storage/cedana)
## Advanced
* [Checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1)
* [Checkpoint/restore runc](/daemon/checkpoint-restore/cr-2)
* [Checkpoint/restore containerd](/daemon/checkpoint-restore/cr-2)
* [Checkpoint/restore streamer](/daemon/checkpoint-restore/cr-4)
* [Checkpoint/restore kubernetes](/daemon/checkpoint-restore/cr-5)
# Checkpoint/restore with GPUs
{% hint style="warning" %}
Checkpoint/restore with GPUs is currently only supported for NVIDIA GPUs.
{% endhint %}
## Prerequisites
1. Create an account with Cedana, to get access to the GPU plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install a GPU plugin.
* **Option 1: GPU Plugin**
The GPU plugin is Cedana's proprietary plugin for high performance GPU checkpoint/restore that supports multi-process/node. If unavailable to you, check option 2.
```sh
sudo cedana plugin install gpu
```
* **Option 2: CRIU CUDA Plugin**
The CRIU CUDA plugin (CRIUgpu) is developed by the CRIU community and uses the [NVIDIA CUDA checkpoint utility](https://github.com/NVIDIA/cuda-checkpoint) under the hood.
```sh
sudo cedana plugin install criu/cuda
```
4. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
5. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
Check out [Cedana vs. CRIUgpu for GPU Checkpoint/Restore](/articles/cedana-vs.-criu-cuda-for-gpu-checkpoint-restore) for a performance comparison between the two plugins.
| | Min driver | Max driver | Multi-GPU | Multi-process | Arch |
| -------------- | ---------- | ---------- | --------- | ------------- | ------------ |
| **Cedana GPU** | 452 | 570 | ✅ | ✅ | amd64, arm64 |
| **CRIU CUDA** | 570 | 570 | ✅ | ❌ | amd64 |
## Usage (GPU plugin)
### Single process
{% hint style="info" %}
Cedana GPU checkpoint/restore is only possible for managed processes/containers, i.e., those that are spawned using `cedana run --gpu-enabled` or managed using `cedana manage --gpu-enabled` (see [managed process/container](/daemon/checkpoint-restore/managed)). If you don't want to run as a managed process/container, you can use `ceadna run --no-server` that will spawn the process/container as a child process of the CLI instead of the daemon.
{% endhint %}
1. You may clone the [cedana-samples repository](https://github.com/cedana/cedana-samples) for some example GPU workloads.
2. Run a process with GPU support:
```sh
cedana run process --attach --gpu-enabled --jid -- cedana-samples/gpu_smr/vector_add
```
3. Checkpoint:
```sh
cedana dump job
```
4. Restore:
```sh
cedana restore job --attach
```
### Multi-process/node
For multi-process/node workloads, you just need to specify the `--gpu-freeze-type` option during dump. If the workload is multi-process/multi-node and using [NCCL](https://developer.nvidia.com/nccl), use the `nccl` option.
```sh
cedana dump job --gpu-freeze-type nccl
```
You can then restore as usual. You may also set the default GPU freeze type in the [configuration](/daemon/get-started/configuration).
## Usage (CRIU CUDA plugin)
### Single process
You can checkpoint/restore normally as you do for CPU workloads. See [checkpoint/restore basics](/daemon/checkpoint-restore/cr).
### Multi-process/node
This is currently not supported. You should use the Cedana GPU plugin for multi-process/node workloads.
{% hint style="info" %}
For all available CLI options, see [CLI reference](/daemon/references/cli/cedana). Directly interacting with daemon is also possible through gRPC, see [API reference](/daemon/references/api).
{% endhint %}
# Checkpoint/restore runc
## Prerequisites
1. Create an account with Cedana, to get access to the GPU plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install the **runc** plugin with `sudo cedana plugin install runc`.
4. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
5. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
## Basic
1. Run a new runc container, for example:
```sh
sudo runc run --detach --bundle ./my-bundle
```
2. Checkpoint:
```sh
cedana dump runc --dir
```
3. Restore:
```sh
cedana restore runc --bundle ./my-bundle --path
```
## Managed
1. Run a new managed runc container:
```sh
cedana run runc --attach --jid --bundle ./my-bundle
```
2. Checkpoint:
```sh
cedana dump job
```
3. Restore:
```sh
cedana restore job --attach
```
## GPU support
Just like for processes, as explained in [checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1), GPU support is also available for managed runc containers.
1. Run a new managed runc container with GPU support:
```sh
cedana run runc --attach --gpu-enabled --jid --bundle ./my-bundle
```
2. Checkpoint:
```sh
cedana dump job
```
3. Restore:
```sh
cedana restore job --attach
```
{% hint style="info" %}
For all available CLI options, see [CLI reference](/daemon/references/cli/cedana). Directly interacting with daemon is also possible through gRPC, see [API reference](/daemon/references/api).
{% endhint %}
# Checkpoint/restore containerd
## Prerequisites
1. Create an account with Cedana, to get access to the containerd plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install the **containerd** plugin with `sudo cedana plugin install containerd`.
4. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
5. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
## Basic
1. Run a new containerd container, for example:
```sh
sudo ctr run docker.io/library/nginx:latest
```
2. Checkpoint:
```sh
cedana dump containerd --dir
```
3. Restore is currently WIP. However, you can restore this container as a runc container, see [checkpoint/restore runc](/daemon/checkpoint-restore/cr-2).
## Managed
1. Run a new managed containerd container:
```sh
cedana run containerd --attach --jid --image docker.io/library/nginx:latest
```
2. Checkpoint:
```sh
cedana dump job
```
3. Restore is currently WIP.
## GPU support
Just like for processes, as explained in [checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1), GPU support is also available for managed containerd containers.
1. Run a new managed containerd container with GPU support:
```sh
cedana run containerd --attach --gpu-enabled --jid --image docker.io/library/nginx:latest
```
2. Checkpoint:
```sh
cedana dump job
```
3. Restore is currently WIP.
## Rootfs
To include the rootfs in the checkpoint, additionally set the `--image` flag with a new image name. For example:
```sh
cedana dump containerd --dir --image
```
For checkpoint *only* the rootfs, set the `--rootfs-only` flag. For example:
```sh
cedana dump containerd --dir --image --rootfs-only
```
{% hint style="info" %}
For all available CLI options, see [CLI reference](/daemon/references/cli/cedana). Directly interacting with daemon is also possible through gRPC, see [API reference](/daemon/references/api).
{% endhint %}
# Checkpoint/restore streamer
The Cedana daemon supports checkpoint/restore via high-performance low-overhead streaming. It's powered by the [streamer plugin](https://github.com/cedana/cedana-image-streamer), which is a fork of CRIU's [image streamer](https://github.com/checkpoint-restore/criu-image-streamer).
{% hint style="info" %}
Real benefits of streaming are realized only when checkpointing and restoring to/from a remote location. See [remote storage](#remote-storage).
{% endhint %}
## Prerequisites
1. Create an account with Cedana, to get access to the streamer plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install the streamer plugin with `sudo cedana plugin install streamer`.
4. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
5. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
## Checkpoint
The `cedana dump` subcommand supports a `--streams ` flag, where `n` is the number of parallel streams to use (minimum of 2). For example:
```sh
cedana dump process --streams 4
```
This will directly stream the checkpoint to a directory, using 4 parallel streams. You will notice that the checkpoint directory contains 4 separate image files:
```
-rw-r--r-- 1 root root 145K Feb 19 15:13 img-0
-rw-r--r-- 1 root root 17K Feb 19 15:13 img-1
-rw-r--r-- 1 root root 209K Feb 19 15:13 img-2
-rw-r--r-- 1 root root 188K Feb 19 15:13 img-3
```
## Restore
The `cedana restore` will automatically detect if the checkpoint was taken with streaming, and will use the same number of streams to restore. For example:
```sh
cedana restore process --path
```
## Compression
All compression algorithms supported for basic checkpoint/restore are supported. See [compression](/daemon/checkpoint-restore/cr#compression) for more information.
## Remote storage
Cedana supports streaming to/from remote storage, through storage plugins. Check out the following guides for specific remote storage:
* [Amazon S3](/daemon/storage/s3)
* [Google Cloud Storage](/daemon/storage/gcs)
* [Cedana Storage](/daemon/storage/cedana)
{% hint style="info" %}
The daemon simply reads/writes from the filesystem. This is also the case for streaming, with the additional requirement that the underlying filesystem must be [POSIX-compliant](https://grimoire.carcano.ch/blog/posix-compliant-filesystems/). To checkpoint/restore to/from a remote directory, you can also use a FUSE-based filesystem mount backed by your network storage. For Amazon's S3, check out [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse).
{% endhint %}
## Enable by default
To enable streaming by default, set the `Checkpoint.Stream` field in the [configuration](/daemon/get-started/configuration) to the desired number of parallel streams. Zero means no streaming.
{% hint style="info" %}
For all available CLI options, see [CLI reference](/daemon/references/cli/cedana). Directly interacting with daemon is also possible through gRPC, see [API reference](/daemon/references/api).
{% endhint %}
# Checkpoint/restore kubernetes
Checkpoint/restore of Kubernetes workloads is powered by the Cedana daemon too, but it's not as simple as a single container checkpoint/restore.
For detailed documentation on our managed Kubernetes or the larger Cedana system, please see [here](https://docs.cedana.ai).
# Amazon S3
Checkpoint/restore to/from Amazon S3 is as seamless as it is to/from local storage.
{% hint style="info" %}
Both [general-purpose](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html) and [directory](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html) S3 buckets are supported. Use directory buckets for better performance for the cost of lower availability. See [Amazon S3 Storage Classes](https://aws.amazon.com/s3/storage-classes/#topic-0) for more details.
{% endhint %}
## Prerequisites
1. Create an account with Cedana, to get access to the GPU plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install the **storage/s3** plugin with `sudo cedana plugin install storage/s3`.
4. Set AWS credentials in the [configuration](/daemon/get-started/configuration).
5. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
6. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
## Checkpoint
To checkpoint to an S3 bucket, simply set the `--dir` to a path that starts with `s3://`, for example:
```sh
cedana dump ... --dir s3://my-bucket/path/to/dir
```
For example, as explained in [managed checkpoint/restore](/daemon/checkpoint-restore/cr#managed-checkpoint-restore), to checkpoint a job to S3:
```sh
cedana dump job my-job-1 --dir s3://checkpoints-bucket
```
If you do `cedana job list`, you will see the latest checkpoint:
```
ID TIME SIZE PATH
my-job-1 2025-02-19 12:30:36 - s3://checkpoints-bucket/dump-job.tar
```
## Restore
Similarly, to restore from an S3 bucket, simply set the `--path` to your checkpoint path in S3, for example:
```sh
cedana restore ... --path s3://my-bucket/path/to/dump.tar
```
For example, as explained in [managed checkpoint/restore](/daemon/checkpoint-restore/cr#managed-checkpoint-restore), to restore a job from S3:
```sh
cedana restore job --attach my-job-1
```
This will automatically restore from the latest checkpoint for `my-job-1`, which is stored in S3.
## Compression
All compression algorithms supported for basic checkpoint/restore are supported. See [compression](/daemon/checkpoint-restore/cr#compression) for more information.
{% hint style="info" %}
For better performance when remote checkpointing/restoring large processes/containers, especially when using [checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1), always use compression. The `lz4` compression algorithm is a good compromise between speed and compression ratio.
{% endhint %}
## Streaming
High-performance low-overhead streaming of checkpoints is also supported by the `storage/s3` plugin. Follow instructions on [checkpoint/restore streamer](/daemon/checkpoint-restore/cr#checkpoint-restore-streamer) to use streaming with this plugin.
## Enable by default
To enable streaming by default, set the `Checkpoint.Dir` field in the [configuration](/daemon/get-started/configuration) to a path that starts with `s3://`.
## See also
* [Google Cloud Storage](/daemon/storage/gcs)
* [Cedana Storage](/daemon/storage/cedana)
# Google Cloud Storage
Checkpoint/restore to/from Google Cloud Storage is currently WIP.
## See also
* [Amazon S3](/daemon/storage/s3)
* [Cedana Storage](/daemon/storage/cedana)
# Cedana Storage
Cedana Storage is a global storage for checkpoints that is backed by multiple cloud providers, providing low latency and high availability. This is the fastest way to get started with remote checkpoint/restore, as you only need to be [authenticated](/daemon/get-started/authentication) with Cedana.
{% hint style="warning" %}
If you're using Cedana on an Amazon EKS cluster, you'll likely get higher performance using [Amazon S3](/daemon/storage/s3). Similarly, if you're using Cedana on a GKE cluster, you'll likely get higher performance using [Google Cloud Storage](/daemon/storage/gcs).
{% endhint %}
## Prerequisites
1. Create an account with Cedana, to get access to the GPU plugin. See [authentication](/daemon/get-started/authentication).
2. Set the Cedana URL & authentication token in the [configuration](/daemon/get-started/configuration).
3. Install the **storage/cedana** plugin with `sudo cedana plugin install storage/cedana`.
4. Ensure the daemon is running, see [installation](/daemon/get-started/installation).
5. Do a health check to ensure the plugin is ready, see [health checks](/daemon/get-started/health).
## Checkpoint
To checkpoint to Cedana Storage, simply set the `--dir` to a path that starts with `cedana://`, for example:
```sh
cedana dump ... --dir cedana://path/to/dir
```
For example, as explained in [managed checkpoint/restore](/daemon/checkpoint-restore/cr#managed-checkpoint-restore), to checkpoint a job to Cedana Storage:
```sh
cedana dump job my-job-1 --dir cedana://my-checkpoints
```
If you do `cedana job list`, you will see the latest checkpoint:
```
ID TIME SIZE PATH
my-job-1 2025-02-19 12:30:36 - cedana://my-checkpoints/dump-job.tar
```
## Restore
Similarly, to restore from Cedana Storage, simply set the `--path` to your checkpoint path in Cedana Storage, for example:
```sh
cedana restore ... --path cedana://path/to/dump.tar
```
For example, as explained in [managed checkpoint/restore](/daemon/checkpoint-restore/cr#managed-checkpoint-restore), to restore a job from Cedana Storage:
```sh
cedana restore job --attach my-job-1
```
This will automatically restore from the latest checkpoint for `my-job-1`, which is stored in Cedana Storage.
## Compression
All compression algorithms supported for basic checkpoint/restore are supported. See [compression](/daemon/checkpoint-restore/cr#compression) for more information.
{% hint style="info" %}
For better performance when remote checkpointing/restoring large processes/containers, especially when using [checkpoint/restore with GPUs](/daemon/checkpoint-restore/cr-1), always use compression. The `lz4` compression algorithm is a good compromise between speed and compression ratio.
{% endhint %}
## Streaming
High-performance low-overhead streaming of checkpoints is also supported by the `storage/cedana` plugin. Follow instructions on [checkpoint/restore streamer](/daemon/checkpoint-restore/cr#checkpoint-restore-streamer) to use streaming with this plugin.
## Enable by default
To enable streaming by default, set the `Checkpoint.Dir` field in the [configuration](/daemon/get-started/configuration) to a path that starts with `cedana://`.
## See also
* [Amazon S3](/daemon/storage/s3)
* [Google Cloud Storage](/daemon/storage/gcs)
# Architecture
The design mostly follows what's illustrated below. Below is a simplified runtime view of invoking `cedana dump runc ...`:
Cedana's plugin architecture
" %}
3. This way, the runc plugin only implements the specifics of the runc runtime, while the daemon handles the common logic, and invoking CRIU.
4. The same pattern is followed for `dump`, `restore`, `run`, and `manage`.
## Features
Symbols that can be exported by a plugin are well-typed and are defined in `pkg/features/supported.go`. A `feature` implements a convenient method called `IfAvailable(do func(), filter ...string)`, which is the *only* method you will ever need to access a feature exported by a plugin. An example usage:
{% @github-files/github-code-block url="" %}
A useful helper command is `cedana features` (alias of `cedana plugin features`), which lists all the features supported by all the plugins. This is useful for debugging, when implementing a new feature, or when you want to know what a plugin supports. Use the `--errors` flag to also output all errors encountered while loading the plugins.
{% hint style="info" %}
See [features](/daemon/get-started/features) for more info.
{% endhint %}
# Profiling
The profiling system tries to be contextual and *invisible*. Profiling data is sent as [gRPC Metadata](https://grpc.io/docs/guides/metadata/). To enable, set `Profiling.Enabled=true` in [configuration](/daemon/get-started/configuration).
Since each adapter to a request (see [architecture](/daemon/developer-guides/architecture)) is a well-defined single-responsibility function, it makes sense to profile each of them. When enabled, the daemon will profile each adapter, and send back flattened data as a gRPC trailer. For readability's sake, this profiling is completely invisible and handled by the adapter logic in `pkg/types/adapter.go` (also see `pkg/types/timer.go`). This flattened data is parsed by the `cmd` package and displayed as shown below. An example out of `cedana dump containerd ...` when profiling is enabled:
Above, you can see complete flow of the request before it reaches CRIU's dump function, including *which* plugin (2nd column) the adapter belongs to. For `containerd`, you can see that the request is largely handled by the low-level runtime `runc`'s plugin. The second table above shows a compressed view of the same data, with only the total time spent in each plugin/category. Note that, above there are some components that are executed concurrently, e.g. `rootfs`, so the time you see in the (flattened) data above is only the time spent *waiting* for `rootfs` to finish.
In many cases, we may need to add more context to this data, or add more components to it. Helpers defined in `pkg/profiling/timing.go` can be used. A good example is adding a new component to the `compression` category in `internal/server/filesystem/dump_adapters.go` as seen above:
{% @github-files/github-code-block url="" %}
These helpers use the passed `context` to store profiling data. If the `context` already has parent profiling data, the data is added as a component to the parent.
{% hint style="info" %}
Behind the scenes, if metrics is enabled ([configuration](/daemon/get-started/configuration) `Metrics=true`), this data is also captured as OTel spans.
{% endhint %}
# Testing
Tests are configured to run inside a Docker container, ridding the need for tedious setup scripts. This allows for testing locally, during development, without affecting the environment.
The test directory looks like this:
```
test
├── regression
│ ├── basic.bats
│ ├── cr.bats
│ ├── run.bats
│ ├── manage.bats
│ ├── profiling.bats
│ ├── plugins
│ │ ├── containerd.bats
│ │ ├── crio.bats
│ │ ├── gpu.bats
│ │ ├── gpu_runc.bats
│ │ ├── gpu_streamer.bats
│ │ ├── streamer.bats
│ │ ├── streamer_runc.bats
│ │ └── runc.bats
│ ├── helpers
│ │ └── ...
└── workloads
```
Tests are grouped by functionality, and each plugin has its own test file.
## Running tests
Running `make help` you'll see the following test commands:
```sh
Testing
test Run all tests (PARALLELISM=, GPU=[0|1], TAGS=, RETRIES=, DEBUG=[0|1])
test-unit Run unit tests (with benchmarks)
test-regression Run regression tests (PARALLELISM=, GPU=[0|1], TAGS=, RETRIES=, DEBUG=[0|1])
test-k8s Run kubernetes e2e tests (PARALLELISM=, GPU=[0|1], TAGS=, RETRIES=, DEBUG=[0|1], ...)
test-enter Enter the test environment
test-enter-cuda Enter the test environment (CUDA)
test-k9s Enter k9s in the test environment
```
When running any of the test commands locally, the tests are automatically run inside a Docker container using `cedana/cedana-test:latest` or `cedana/cedana-test:cuda` if `GPU=1`. GPU tests are skipped by default, unless `GPU=1` is set.
## Running GPU tests
Use `GPU=1` to include all GPU tests. If `GPU=0`, any tests that require GPU-support automatically get skipped. You may want to specify a low `PARALLELISM` value when running GPU tests, as each test requires a significant amount of RAM.
## Running specific tests
Use `TAGS` to filter tests by tags. For example, `make test-regression TAGS=runc` will run all tests tagged with `runc`. `make test-regression TAGS=runc,gpu` will run all tests tagged with `runc` and `gpu`. If `gpu` tag is included, you must set `GPU=1` to run the tests, otherwise they will be skipped.
## Test modes
Each test command above runs the test suite **two times**, in different modes:
1. Unique daemon & DB instance for each test.
2. Single persistent daemon & DB instance across a test suite.
This is to allow catching bugs that may arise due to the daemon's state being persisted across tests.
Each test command is also configured to run parallel-y, configured by the `PARALLELISM` variable passed to the test command. E.g. `make test-regression PARALLELISM=4` will run at most 4 tests in parallel at a time.
For mode 1, parallelism offers no benefits apart from fast execution time, as each test is completely isolated. However, for mode 2, parallelism may shed light on bugs in the daemon when it's handling multiple requests concurrently.
# Writing plugins
Plugins are currently part of Cedana's source tree ([here](https://github.com/cedana/cedana/tree/main/plugins)) and are built and released together with the main Cedana binary. This guide will walk you through the process of writing a plugin for Cedana.
Check out [features](/daemon/get-started/features) to see which features are currently supported by each plugin.
# CLI
# cedana
Root command for Cedana
## Synopsis
```
________ _______ ________ ________ ________ ________
|\ ____\|\ ___ \ |\ ___ \|\ __ \|\ ___ \|\ __ \
\ \ \___|\ \ __/|\ \ \_|\ \ \ \|\ \ \ \\ \ \ \ \|\ \
\ \ \ \ \ \_|/_\ \ \ \\ \ \ __ \ \ \\ \ \ \ __ \
\ \ \____\ \ \_|\ \ \ \_\\ \ \ \ \ \ \ \\ \ \ \ \ \ \
\ \_______\ \_______\ \_______\ \__\ \__\ \__\\ \__\ \__\ \__\
\|_______|\|_______|\|_______|\|__|\|__|\|__| \|__|\|__|\|__|
```
Instance Brokerage, Orchestration and Migration System. Property of Cedana, Corp.
v0.9.279
## Options
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
-h, --help help for cedana
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana attach](/daemon/references/cli/cedana_attach) - Attach stdin/out/err to a process/container
* [cedana check](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_check.md) - Health check
* [cedana checkpoint](/daemon/references/cli/cedana_checkpoint) - Manage job checkpoints (alias of `job checkpoint`)
* [cedana checkpoints](/daemon/references/cli/cedana_checkpoints) - List all checkpoints for a job (alias of `checkpoint list `)
* [cedana completion](/daemon/references/cli/cedana_completion) - Generate the autocompletion script for the specified shell
* [cedana daemon](/daemon/references/cli/cedana_daemon) - Manage the daemon
* [cedana delete](/daemon/references/cli/cedana_delete) - Delete a managed process/container (job) (alias of `job delete ...`)
* [cedana dump](/daemon/references/cli/cedana_dump) - Dump a container/process
* [cedana dump-vm](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_dump-vm.md) - Dump a VM
* [cedana exec](/daemon/references/cli/cedana_exec) - Run a managed process (job) (alias of `run process [args...]`)
* [cedana features](/daemon/references/cli/cedana_features) - Show feature matrix of plugins (alias of `plugin features [plugin]...`)
* [cedana freeze](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_freeze.md) - Freeze a container/process
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
* [cedana jobs](/daemon/references/cli/cedana_jobs) - List all managed processes/containers (jobs) (alias of `job list`)
* [cedana k8s](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_k8s.md) - Helper for setting up and running in Kubernetes
* [cedana kill](/daemon/references/cli/cedana_kill) - Kill a managed process/container (job) (alias of `job kill ...`)
* [cedana manage](/daemon/references/cli/cedana_manage) - Manage an existing/upcoming process/container (create a job)
* [cedana plugin](/daemon/references/cli/cedana_plugin) - Manage plugins
* [cedana plugins](/daemon/references/cli/cedana_plugins) - List plugins (specify plugin @ to filter) (alias of `plugin list [plugin]...`)
* [cedana ps](/daemon/references/cli/cedana_ps) - List all managed processes/containers (jobs) (alias of `job list`)
* [cedana query](/daemon/references/cli/cedana_query) - Query containers/processes
* [cedana restore](/daemon/references/cli/cedana_restore) - Restore a container/process
* [cedana run](/daemon/references/cli/cedana_run) - Run a managed process/container (create a job)
* [cedana unfreeze](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_unfreeze.md) - Unfreeze a container/process
# cedana attach
Attach stdin/out/err to a process/container
```
cedana attach [flags]
```
## Options
```
-h, --help help for attach
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana check
Health check
```
cedana check [flags]
```
## Options
```
--full perform a full check (including plugins)
-h, --help help for check
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana checkpoint
Manage job checkpoints (alias of `job checkpoint`)
## Synopsis
(alias of `job checkpoint`)
## Options
```
-h, --help help for checkpoint
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana checkpoint inspect](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_checkpoint_inspect.md) - Inspect a checkpoint
* [cedana checkpoint list](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_checkpoint_list.md) - List all checkpoints for a job
# cedana checkpoints
List all checkpoints for a job (alias of `checkpoint list `)
## Synopsis
(alias of `checkpoint list `)
```
cedana checkpoints [flags]
```
## Options
```
-h, --help help for list
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana completion
Generate the autocompletion script for the specified shell
## Synopsis
Generate the autocompletion script for cedana for the specified shell. See each sub-command's help for details on how to use the generated script.
## Options
```
-h, --help help for completion
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana completion bash](/daemon/references/cli/cedana_completion_bash) - Generate the autocompletion script for bash
* [cedana completion fish](/daemon/references/cli/cedana_completion_fish) - Generate the autocompletion script for fish
* [cedana completion powershell](/daemon/references/cli/cedana_completion_powershell) - Generate the autocompletion script for powershell
* [cedana completion zsh](/daemon/references/cli/cedana_completion_zsh) - Generate the autocompletion script for zsh
# cedana completion bash
Generate the autocompletion script for bash
## Synopsis
Generate the autocompletion script for the bash shell.
This script depends on the 'bash-completion' package. If it is not installed already, you can install it via your OS's package manager.
To load completions in your current shell session:
```
source <(cedana completion bash)
```
To load completions for every new session, execute once:
### Linux:
```
cedana completion bash > /etc/bash_completion.d/cedana
```
### macOS:
```
cedana completion bash > $(brew --prefix)/etc/bash_completion.d/cedana
```
You will need to start a new shell for this setup to take effect.
```
cedana completion bash
```
## Options
```
-h, --help help for bash
--no-descriptions disable completion descriptions
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana completion](/daemon/references/cli/cedana_completion) - Generate the autocompletion script for the specified shell
# cedana completion fish
Generate the autocompletion script for fish
## Synopsis
Generate the autocompletion script for the fish shell.
To load completions in your current shell session:
```
cedana completion fish | source
```
To load completions for every new session, execute once:
```
cedana completion fish > ~/.config/fish/completions/cedana.fish
```
You will need to start a new shell for this setup to take effect.
```
cedana completion fish [flags]
```
## Options
```
-h, --help help for fish
--no-descriptions disable completion descriptions
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana completion](/daemon/references/cli/cedana_completion) - Generate the autocompletion script for the specified shell
# cedana completion powershell
Generate the autocompletion script for powershell
## Synopsis
Generate the autocompletion script for powershell.
To load completions in your current shell session:
```
cedana completion powershell | Out-String | Invoke-Expression
```
To load completions for every new session, add the output of the above command to your powershell profile.
```
cedana completion powershell [flags]
```
## Options
```
-h, --help help for powershell
--no-descriptions disable completion descriptions
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana completion](/daemon/references/cli/cedana_completion) - Generate the autocompletion script for the specified shell
# cedana completion zsh
Generate the autocompletion script for zsh
## Synopsis
Generate the autocompletion script for the zsh shell.
If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:
```
echo "autoload -U compinit; compinit" >> ~/.zshrc
```
To load completions in your current shell session:
```
source <(cedana completion zsh)
```
To load completions for every new session, execute once:
### Linux:
```
cedana completion zsh > "${fpath[1]}/_cedana"
```
### macOS:
```
cedana completion zsh > $(brew --prefix)/share/zsh/site-functions/_cedana
```
You will need to start a new shell for this setup to take effect.
```
cedana completion zsh [flags]
```
## Options
```
-h, --help help for zsh
--no-descriptions disable completion descriptions
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana completion](/daemon/references/cli/cedana_completion) - Generate the autocompletion script for the specified shell
# cedana daemon
Manage the daemon
## Options
```
-h, --help help for daemon
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana daemon check](/daemon/references/cli/cedana_daemon_check) - Health check the daemon
* [cedana daemon start](/daemon/references/cli/cedana_daemon_start) - Start the daemon
# cedana daemon check
Health check the daemon
```
cedana daemon check [flags]
```
## Options
```
--full perform a full check (including plugins)
-h, --help help for check
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana daemon](/daemon/references/cli/cedana_daemon) - Manage the daemon
# cedana daemon start
Start the daemon
```
cedana daemon start [flags]
```
## Options
```
--db string path to local database
-h, --help help for start
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana daemon](/daemon/references/cli/cedana_daemon) - Manage the daemon
# cedana delete
Delete a managed process/container (job) (alias of `job delete ...`)
## Synopsis
(alias of `job delete ...`)
```
cedana delete ... [flags]
```
## Options
```
-a, --all delete all jobs
-h, --help help for delete
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana dump
Dump a container/process
## Options
```
--compression string compression algorithm (none, tar, gzip, lz4, zlib)
--criu-opts string criu options JSON (overriddes individual CRIU flags)
-d, --dir string directory to dump into
--external strings resources from external namespaces (can be multiple)
--file-locks dump file locks
-h, --help help for dump
--leave-running leave the process running after dump
--link-remap remap links to files in the dump
--name string name of the dump
--shell-job process is not session leader (shell job)
--skip-in-flight skip in-flight tcp connections
--streams int32 number of streams to use for dump (0 for no streaming)
--tcp-established dump tcp established connections
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana dump containerd](/daemon/references/cli/cedana_dump_containerd) - Dump a containerd container (w/ rootfs)
* [cedana dump job](/daemon/references/cli/cedana_dump_job) - Dump a managed process/container (job)
* [cedana dump process](/daemon/references/cli/cedana_dump_process) - Dump a process
* [cedana dump runc](/daemon/references/cli/cedana_dump_runc) - Dump a runc container
# cedana dump containerd
Dump a containerd container (w/ rootfs)
## Synopsis
Dump a containerd container (w/ rootfs). If an image ref is provided, rootfs will also be dumped
```
cedana dump containerd [flags]
```
## Options
```
--address string containerd socket address
-h, --help help for containerd
--image string image ref (rootfs). leave empty to skip rootfs
--namespace string containerd namespace
--rootfs dump with rootfs
--rootfs-only dump only the rootfs
```
## Options inherited from parent commands
```
--compression string compression algorithm (none, tar, gzip, lz4, zlib)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--criu-opts string criu options JSON (overriddes individual CRIU flags)
-d, --dir string directory to dump into
--external strings resources from external namespaces (can be multiple)
--file-locks dump file locks
--leave-running leave the process running after dump
--link-remap remap links to files in the dump
--name string name of the dump
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
--shell-job process is not session leader (shell job)
--skip-in-flight skip in-flight tcp connections
--streams int32 number of streams to use for dump (0 for no streaming)
--tcp-established dump tcp established connections
```
## SEE ALSO
* [cedana dump](/daemon/references/cli/cedana_dump) - Dump a container/process
# cedana dump job
Dump a managed process/container (job)
```
cedana dump job [flags]
```
## Options
```
--address string (containerd) containerd socket address
-h, --help help for job
--image string (containerd) image ref (rootfs). leave empty to skip rootfs
--namespace string (containerd) containerd namespace
--root string (runc) root
--rootfs (containerd) dump with rootfs
--rootfs-only (containerd) dump only the rootfs
```
## Options inherited from parent commands
```
--compression string compression algorithm (none, tar, gzip, lz4, zlib)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--criu-opts string criu options JSON (overriddes individual CRIU flags)
-d, --dir string directory to dump into
--external strings resources from external namespaces (can be multiple)
--file-locks dump file locks
--leave-running leave the process running after dump
--link-remap remap links to files in the dump
--name string name of the dump
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
--shell-job process is not session leader (shell job)
--skip-in-flight skip in-flight tcp connections
--streams int32 number of streams to use for dump (0 for no streaming)
--tcp-established dump tcp established connections
```
## SEE ALSO
* [cedana dump](/daemon/references/cli/cedana_dump) - Dump a container/process
# cedana dump process
Dump a process
```
cedana dump process [flags]
```
## Options
```
-h, --help help for process
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--compression string compression algorithm (none, tar, gzip, lz4, zlib)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--criu-opts string criu options JSON (overriddes individual CRIU flags)
-d, --dir string directory to dump into
--external strings resources from external namespaces (can be multiple)
--file-locks dump file locks
--leave-running leave the process running after dump
--link-remap remap links to files in the dump
--name string name of the dump
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
--shell-job process is not session leader (shell job)
--skip-in-flight skip in-flight tcp connections
--streams int32 number of streams to use for dump (0 for no streaming)
--tcp-established dump tcp established connections
```
## SEE ALSO
* [cedana dump](/daemon/references/cli/cedana_dump) - Dump a container/process
# cedana dump runc
Dump a runc container
```
cedana dump runc [flags]
```
## Options
```
-h, --help help for runc
--root string root
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--compression string compression algorithm (none, tar, gzip, lz4, zlib)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--criu-opts string criu options JSON (overriddes individual CRIU flags)
-d, --dir string directory to dump into
--external strings resources from external namespaces (can be multiple)
--file-locks dump file locks
--leave-running leave the process running after dump
--link-remap remap links to files in the dump
--name string name of the dump
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
--shell-job process is not session leader (shell job)
--skip-in-flight skip in-flight tcp connections
--streams int32 number of streams to use for dump (0 for no streaming)
--tcp-established dump tcp established connections
```
## SEE ALSO
* [cedana dump](/daemon/references/cli/cedana_dump) - Dump a container/process
# cedana dump-vm
Dump a VM
## Options
```
-d, --dir string directory to dump into
-h, --help help for dump-vm
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana dump-vm cloud-hypervisor](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_dump-vm_cloud-hypervisor.md) - Dump a clh vm
* [cedana dump-vm kata](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_dump-vm_kata.md) - Dump a kata vm or container (w/o rootfs)
# cedana exec
Run a managed process (job) (alias of `run process [args...]`)
## Synopsis
(alias of `run process [args...]`)
```
cedana exec [args...] [flags]
```
## Options
```
--as-root run as root
-h, --help help for exec
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
-a, --attach attach stdin/out/err
--attachable make it attachable, but don't attach
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
-g, --gpu-enabled enable GPU support
--gpu-id string specify existing GPU controller ID to attach (internal use only)
--gpu-tracing enable GPU tracing
-j, --jid string job id
--no-server run without server
-o, --out string file to forward stdout/err
--pid-file string file to write PID to
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana features
Show feature matrix of plugins (alias of `plugin features [plugin]...`)
## Synopsis
(alias of `plugin features [plugin]...`)
```
cedana features [plugin]... [flags]
```
## Options
```
--errors Show all errors
-h, --help help for features
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana freeze
Freeze a container/process
## Options
```
-h, --help help for freeze
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana freeze containerd](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_freeze_containerd.md) - Freeze a containerd container
* [cedana freeze job](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_freeze_job.md) - Freeze a managed process/container (job)
* [cedana freeze process](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_freeze_process.md) - Freeze a process
* [cedana freeze runc](https://github.com/cedana/cedana/blob/main/docs/references/cli/cedana_freeze_runc.md) - Freeze a runc container
# cedana job
Manage jobs
## Options
```
-h, --help help for job
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana job attach](/daemon/references/cli/cedana_job_attach) - Attach stdin/out/err to a managed process/container (job)
* [cedana job checkpoint](/daemon/references/cli/cedana_job_checkpoint) - Manage job checkpoints
* [cedana job checkpoints](/daemon/references/cli/cedana_job_checkpoints) - List all checkpoints for a job (alias of `checkpoint list `)
* [cedana job delete](/daemon/references/cli/cedana_job_delete) - Delete a managed process/container (job)
* [cedana job inspect](/daemon/references/cli/cedana_job_inspect) - Inspect a managed process/container (job)
* [cedana job kill](/daemon/references/cli/cedana_job_kill) - Kill a managed process/container (job)
* [cedana job list](/daemon/references/cli/cedana_job_list) - List all managed processes/containers (jobs)
# cedana job attach
Attach stdin/out/err to a managed process/container (job)
```
cedana job attach [flags]
```
## Options
```
-h, --help help for attach
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana job checkpoint
Manage job checkpoints
## Options
```
-h, --help help for checkpoint
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
* [cedana job checkpoint inspect](/daemon/references/cli/cedana_job_checkpoint_inspect) - Inspect a checkpoint
* [cedana job checkpoint list](/daemon/references/cli/cedana_job_checkpoint_list) - List all checkpoints for a job
# cedana job checkpoint inspect
Inspect a checkpoint
```
cedana job checkpoint inspect [flags]
```
## Options
```
-h, --help help for inspect
-t, --type string specify image file {ps|fd|mem|rss|sk|gpu}
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job checkpoint](/daemon/references/cli/cedana_job_checkpoint) - Manage job checkpoints
# cedana job checkpoint list
List all checkpoints for a job
```
cedana job checkpoint list [flags]
```
## Options
```
-h, --help help for list
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job checkpoint](/daemon/references/cli/cedana_job_checkpoint) - Manage job checkpoints
# cedana job checkpoints
List all checkpoints for a job (alias of `checkpoint list `)
## Synopsis
(alias of `checkpoint list `)
```
cedana job checkpoints [flags]
```
## Options
```
-h, --help help for checkpoints
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana job delete
Delete a managed process/container (job)
```
cedana job delete ... [flags]
```
## Options
```
-a, --all delete all jobs
-h, --help help for delete
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana job inspect
Inspect a managed process/container (job)
```
cedana job inspect [flags]
```
## Options
```
-h, --help help for inspect
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana job kill
Kill a managed process/container (job)
```
cedana job kill ... [flags]
```
## Options
```
-a, --all kill all jobs
-h, --help help for kill
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana job list
List all managed processes/containers (jobs)
```
cedana job list [flags]
```
## Options
```
-a, --all include jobs from remote hosts
-h, --help help for list
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana job](/daemon/references/cli/cedana_job) - Manage jobs
# cedana jobs
List all managed processes/containers (jobs) (alias of `job list`)
## Synopsis
(alias of `job list`)
```
cedana jobs [flags]
```
## Options
```
-a, --all include jobs from remote hosts
-h, --help help for list
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana k8s
# cedana kill
Kill a managed process/container (job) (alias of `job kill ...`)
## Synopsis
(alias of `job kill ...`)
```
cedana kill ... [flags]
```
## Options
```
-a, --all kill all jobs
-h, --help help for kill
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
# cedana manage
Manage an existing/upcoming process/container (create a job)
## Options
```
-g, --gpu-enabled enable GPU support
--gpu-id string specify existing GPU controller ID to attach (internal use only)
--gpu-tracing enable GPU tracing
-h, --help help for manage
-j, --jid string job id
--pid-file string file to write PID to
--upcoming wait for upcoming process/container
```
## Options inherited from parent commands
```
--address string address to use (host:port for TCP, path for UNIX, cid:port for VSOCK)
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
```
## SEE ALSO
* [cedana](/daemon/references/cli/cedana) - Root command for Cedana
* [cedana manage containerd](/daemon/references/cli/cedana_manage_containerd) - Manage a containerd container
* [cedana manage process](/daemon/references/cli/cedana_manage_process) - Managed existing process (job)
* [cedana manage runc](/daemon/references/cli/cedana_manage_runc) - manage an existing runc container
# cedana manage containerd
Manage a containerd container
```
cedana manage containerd [flags]
```
## Options
```
--address string containerd socket address
-h, --help help for containerd
--image string image to use
--namespace string containerd namespace
```
## Options inherited from parent commands
```
--config string one-time config JSON string (merge with existing config)
--config-dir string custom config directory
-g, --gpu-enabled enable GPU support
--gpu-id string specify existing GPU controller ID to attach (internal use only)
--gpu-tracing enable GPU tracing
-j, --jid string job id
--pid-file string file to write PID to
--profiling enable profiling/show profiling data
--protocol string protocol to use (TCP, UNIX, VSOCK)
--upcoming wait for upcoming process/container
```
## SEE ALSO
* [cedana manage](/daemon/references/cli/cedana_manage) - Manage an existing/upcoming process/container (create a job)
---
[Next Page](/llms-full.txt/1)