Profiling

The profiling system tries to be contextual and invisible. Profiling data is sent as gRPC Metadata. To enable, set Profiling.Enabled=true in configuration.

Since each adapter to a request (see architecture) is a well-defined single-responsibility function, it makes sense to profile each of them. When enabled, the daemon will profile each adapter, and send back flattened data as a gRPC trailer. For readability's sake, this profiling is completely invisible and handled by the adapter logic in pkg/types/adapter.go (also see pkg/types/timer.go). This flattened data is parsed by the cmd package and displayed as shown below. An example out of cedana dump containerd ... when profiling is enabled:

image

Above, you can see complete flow of the request before it reaches CRIU's dump function, including which plugin (2nd column) the adapter belongs to. For containerd, you can see that the request is largely handled by the low-level runtime runc's plugin. The second table above shows a compressed view of the same data, with only the total time spent in each plugin/category. Note that, above there are some components that are executed concurrently, e.g. rootfs, so the time you see in the (flattened) data above is only the time spent waiting for rootfs to finish.

In many cases, we may need to add more context to this data, or add more components to it. Helpers defined in pkg/profiling/timing.go can be used. A good example is adding a new component to the compression category in internal/server/filesystem/dump_adapters.go as seen above:

		_, end := profiling.StartTimingCategory(ctx, "compression", utils.Tar)
		tarball, err := utils.Tar(imagesDirectory, imagesDirectory, compression)
		end()

These helpers use the passed context to store profiling data. If the context already has parent profiling data, the data is added as a component to the parent.

Behind the scenes, if OTel is enabled (configuration Metrics.Otel=true), this data is also captured in OTel spans.