Enable NVIDIA CDI support for HAMi

This page explains what CDI does and which Helm chart values enable HAMi's NVIDIA CDI integration.

What is CDI?

CDI (Container Device Interface) is a specification for describing the OCI configuration that a container needs to use a device. A CDI specification can define device nodes, mounts, environment variables, and OCI hooks in JSON or YAML.

CDI is vendor-neutral. HAMi currently supports CDI device injection only for NVIDIA GPUs. The HAMi configuration and troubleshooting guidance on this page do not apply to devices from other vendors.

CDI identifies each device by a fully qualified name:

vendor.com/class=device-name

When a container runtime receives this name, it finds the matching CDI specification in a directory such as /etc/cdi or /var/run/cdi and applies the specified changes to the container's OCI runtime specification.

CDI does not schedule devices, allocate resources, or enforce quotas. In HAMi's NVIDIA CDI integration, the scheduler and NVIDIA Device Plugin continue to select and allocate GPUs; CDI injects the allocated GPUs and their runtime requirements into containers.

What problems does CDI solve?

Injecting a complex device often requires more than mounting a single /dev node. A GPU container may also need driver libraries, several device nodes, environment variables, and lifecycle hooks. Without a common interface, vendors must adapt this logic to individual container runtimes, and runtimes may need vendor-specific implementations.

CDI moves these OCI changes into a common specification. It provides:

• A standard format for device nodes, mounts, environment variables, and hooks.
• Stable device names for passing allocation results from Device Plugins to container runtimes.
• Less dependency on runtime-specific device injection implementations.
• An inspectable record of the device configuration applied to a container.

How HAMi uses CDI for NVIDIA GPUs

With cdi-annotations enabled for the NVIDIA Device Plugin, HAMi uses the following injection flow:

1. The HAMi scheduler and NVIDIA Device Plugin select and allocate a GPU.
2. The Device Plugin generates a CDI specification and writes it to /var/run/cdi on the node.
3. The Device Plugin returns the allocated device name through a CDI annotation.
4. A CDI-aware container runtime resolves the annotation and applies the matching CDI specification to the container's OCI runtime specification.

HAMi uses the following CDI kind for NVIDIA GPUs:

k8s.device-plugin.nvidia.com/gpu

Prerequisites

Before enabling NVIDIA CDI support in HAMi, confirm that:

• The NVIDIA driver and NVIDIA Container Toolkit are installed on each GPU node.
• The container runtime supports CDI and has CDI enabled.
• The container runtime can read CDI specifications from /var/run/cdi.
• The actual NVIDIA driver root and nvidia-ctk path on each node are known.

For runtime-specific instructions, see the upstream CDI configuration guide.

Configure the Helm chart

Save the following values as values-cdi.yaml:

devicePlugin:
  deviceListStrategy: "cdi-annotations"
  nvidiaDriverRoot: "<driver-root>"
  nvidiaHookPath: "<nvidia-ctk-path>"

• devicePlugin.deviceListStrategy: Set this to cdi-annotations so that the HAMi Device Plugin passes device names through CDI annotations.
• devicePlugin.nvidiaDriverRoot: The NVIDIA driver root on the node. HAMi uses this directory to discover driver files and device nodes.
• devicePlugin.nvidiaHookPath: The nvidia-ctk executable used on the node. HAMi writes this path into the generated CDI specification.

Replace the two path placeholders according to the deployment:

• Host-installed NVIDIA driver and Container Toolkit: use / for <driver-root> and /usr/bin/nvidia-ctk for <nvidia-ctk-path>.
• GPU Operator-managed driver and Container Toolkit: use /run/nvidia/driver for <driver-root> and /usr/local/nvidia/toolkit/nvidia-ctk for <nvidia-ctk-path>.

warning

The two path values must match the actual node filesystem. Do not copy values from a different deployment model. nvidiaHookPath must point to an executable that is available to the container runtime on the node.

Install or upgrade HAMi:

helm upgrade --install hami hami-charts/hami \
  --version 2.9.0 \
  --namespace kube-system \
  --create-namespace \
  --values values-cdi.yaml

Verify the configuration

First, confirm that Helm applied the expected values:

helm get values hami -n kube-system

On a GPU node, check the selected nvidia-ctk executable and the CDI specification generated by HAMi. The following example uses the host-installed path; replace NVIDIA_CTK_PATH in GPU Operator environments:

NVIDIA_CTK_PATH=/usr/bin/nvidia-ctk
sudo test -x "$NVIDIA_CTK_PATH"
"$NVIDIA_CTK_PATH" --version
sudo ls -l /var/run/cdi
sudo jq . /var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json

Confirm that:

• The CDI kind is k8s.device-plugin.nvidia.com/gpu.
• The allocated GPU UUID is present in the device list.
• Every device node and hook path in the specification exists on the node.

Finally, check the Device Plugin logs:

kubectl logs -n kube-system \
  -l app.kubernetes.io/component=hami-device-plugin \
  --tail=200 | grep -i cdi

The logs should not contain errors about CDI specification generation, unresolved devices, or missing hook paths.

Troubleshooting

failed to stat CDI host device "/dev/nvidia-modeset": no such file or directory

devicePlugin.nvidiaDriverRoot does not match the driver layout on the node. Correct the value using the path rules above, upgrade HAMi, and inspect the regenerated CDI specification.

stderr: No help topic for 'disable-device-node-modification'

The NVIDIA Container Toolkit executable referenced by the CDI specification does not support the disable-device-node-modification command. This command requires NVIDIA Container Toolkit 1.18.0 or later.

Upgrade NVIDIA Container Toolkit and verify both the hook path recorded in the CDI specification and the version of the executable at that path. Host-installed and GPU Operator-managed Toolkit installations are independent; upgrading one does not update the other.

GPU Operator v25.10.1 uses NVIDIA Container Toolkit v1.18.1. For related details, see Toolkit version parser broken for casual kind demo user.

error running createContainer hook #0: fork/exec /usr/bin/nvidia-ctk: no such file or directory

The hook path recorded in the CDI specification does not exist on the node. Locate the actual nvidia-ctk executable, correct devicePlugin.nvidiaHookPath, and upgrade HAMi to regenerate the CDI specification.