Skip to main content
Version: main 🚧

Host devices and GPU passthrough

A KubeVirt VM can take a whole physical GPU through PCI passthrough. Two pieces work together:

  • The vCluster Device Operator prepares the KubeVirt cluster's GPU nodes. It enables IOMMU, binds the GPUs to vfio-pci, and makes KubeVirt advertise them under a resource name. You describe the nodes and GPUs to prepare in a HostDevice resource.
  • A node type requests that resource in its VM template, so every VM of that type gets a GPU.

The KubeVirt path passes through whole GPUs only. For vGPU or other GPU sharing, hand the nodes to the NVIDIA GPU Operator instead. See Use the NVIDIA GPU Operator.

Before you begin​

  • A KubeVirt node provider whose cluster has nodes with GPUs.
  • CPUs and firmware that support IOMMU, Intel VT-d or AMD-Vi. The operator can turn IOMMU on in the kernel, but the hardware has to support it.
  • Platform administrator access, and a license with the Machine Management and VM Management with KubeVirt features. The Host Devices tab is shown only to platform administrators.

Node preparation can reboot nodes or restart their kubelets. The operator cordons nodes but doesn't drain workloads, and uncordons them when it's done. Plan a maintenance window before configuring host devices. See What preparing a node costs.

Deploy the device operator​

The KubeVirt node provider deploys the operator with Helm into its clusterRef.namespace. Turn it on when you create the provider:

spec:
kubeVirt:
deploy:
vClusterDeviceOperator:
enabled: true

In the platform UI, the vCluster Device Operator toggle can only be set when the provider is created. The platform installs chart vcluster-device-operator version 0.2.2 from oci://ghcr.io/loft-sh/charts by default. Override it with chartRepo, chart, version, and helmValues under vClusterDeviceOperator. The platform uninstalls the operator when you delete the provider, as long as enabled is still true at that point. The HostDevice CRD and any existing host devices stay on the cluster.

vCluster Platform v4.12 installs chart 0.0.7, so upgrading the platform also upgrades an operator whose version isn't pinned. The platform takes ownership of the HostDevice CRD that the older chart installed, so the upgrade doesn't fail on it.

Configure host devices​

A HostDevice selects the GPU nodes, says how to prepare them, and names the resource KubeVirt advertises. It's a cluster-scoped resource on the KubeVirt cluster, with API version devices.vcluster.com/v1alpha1.

  1. Open Bare Metal > VMs, select the KubeVirt cluster, and click the Host Devices tab.

  2. Click and enter a Display name and a Host Device ID.

  3. In Node selector, add at least one label that matches the GPU nodes, such as nvidia.com/gpu.present: "true". An empty selector selects no nodes.

  4. Turn on Manage IOMMU only if IOMMU isn't already active on the nodes. The operator then edits the kernel command line and reboots each node.

  5. Under Device selector, narrow the GPUs to manage with GPU vendor ID, such as 10de for NVIDIA, and optionally GPU device IDs. Without them, the operator manages every display controller on the selected nodes.

  6. Under GPU passthrough, select KubeVirt and set KubeVirt resource name, such as nvidia.com/A100. VMs request the GPU by this name. Use a different name in each host device.

  7. Optionally turn on Configure hugepages and Configure NUMA for NUMA-pinned GPU VMs. See Align a VM with its GPU.

  8. Click to save.

The same configuration as YAML, applied to the KubeVirt cluster:

hostdevice.yaml
apiVersion: devices.vcluster.com/v1alpha1
kind: HostDevice
metadata:
name: a100
spec:
nodeSelector:
nvidia.com/gpu.present: "true"
# Omit iommu if IOMMU is already active on the nodes.
iommu:
enabled: true
deviceSelector:
vendor: "10de"
kubevirt:
resourceName: nvidia.com/A100

What preparing a node costs​

Preparing nodes disrupts them, so plan a window:

  • Turning on IOMMU or hugepages reboots the node. One reboot applies both.
  • NUMA settings restart the kubelet instead.
  • The operator cordons each node first but doesn't drain it. It uncordons the node once the node is ready again.
  • It works through one node at a time within each host device.

If a kubelet reconfiguration doesn't come back healthy, the operator restores the previous configuration and marks the node Failed. It retries only after the desired NUMA configuration changes.

Watch progress​

The Host Devices tab shows each host device's status and its ready nodes against the nodes it selects. Click Show Devices to see each node's phase and its IOMMU, hugepage, and NUMA state. It also lists every GPU with its PCI address, vendor ID, device ID, driver, IOMMU group, and whether it's selected and bound.

A node moves through Pending, RebootPending, Rebooting, Configuring, and Binding to Ready. Once a node is Ready, KubeVirt advertises its GPUs under the resource name. To confirm, check the nodes' allocatable count for that name:

kubectl get nodes -l nvidia.com/gpu.present=true \
-o custom-columns='NAME:.metadata.name,GPUS:.status.allocatable.nvidia\.com/A100'

Give each VM a GPU​

Request the resource name in the node type's VM template, under domain.devices.gpus. Each entry passes through one whole GPU:

spec:
kubeVirt:
nodeTypes:
- name: vm-gpu
resources:
cpu: "4"
memory: 8Gi
nvidia.com/gpu: "1"
properties:
# The template replaces the base template, so name the OS image to boot.
vcluster.com/os-image: ubuntu2404-kubevirt
virtualMachineTemplate:
spec:
template:
spec:
domain:
devices:
gpus:
- name: gpu1
# Must match the host device's KubeVirt resource name.
deviceName: nvidia.com/A100
cpu:
cores: 4
memory:
guest: 8Gi
resources:
requests:
cpu: "4"
memory: 8Gi

The gpus list is what attaches GPUs. The nvidia.com/gpu entry in resources, the GPU count field in the platform UI, doesn't attach one. The platform uses it to choose node types. Unless the node type sets its own cost, the platform ranks a node type with a GPU count as more expensive than one without. Dynamic Auto Nodes pools pick such a node type only for requests that ask for GPUs, unless the node type carries the annotation nodetype.loft.sh/allow-unneeded-special-resources: "true". Keep the count equal to the number of entries in gpus.

The node type editor in the platform UI offers GPU passthrough and GPU with NUMA pinning examples to start from.

Align a VM with its GPU​

For the best GPU performance, keep a VM's CPUs and memory on the GPU's NUMA node. Configure hugepages and NUMA in the host device first:

spec:
hugePages:
pageSize: "1Gi"
count: 16
numa:
reservedCPUs: "0"
memoryManager: true

Then back the VM with dedicated CPUs and hugepages in the node type's template:

domain:
devices:
gpus:
- name: gpu1
deviceName: nvidia.com/A100
cpu:
cores: 4
dedicatedCpuPlacement: true
numa:
guestMappingPassthrough: {}
memory:
guest: 8Gi
hugepages:
pageSize: "1Gi"

NUMA settings edit the kubelet configuration at /var/lib/kubelet/config.yaml, so they apply to kubeadm-style nodes.

Use the NVIDIA GPU Operator​

To let the NVIDIA GPU Operator bind and advertise the GPUs, select NVIDIA GPU Operator under GPU passthrough instead of KubeVirt. The device operator then only prepares the nodes, with IOMMU, hugepages, and NUMA. It sets the nvidia.com/gpu.workload.config label from Workload configuration:

Workload configurationLabel valueUse
VM passthroughvm-passthroughFull-GPU passthrough to VMs
vGPUvm-vgpuNVIDIA vGPU
ContainercontainerGPUs for containers

The device operator doesn't install or configure the NVIDIA GPU Operator. A host device can use the KubeVirt path or the NVIDIA GPU Operator path, not both.

Limitations​

  • Each node belongs to one host device. If two node selectors overlap, only the host device whose name sorts first prepares nodes. The other reports a Conflict condition and prepares none of its nodes. Keep node selectors from overlapping.
  • Settings only apply forward. Clearing IOMMU, hugepages, or NUMA settings doesn't revert a node, and allocated hugepages aren't reclaimed.
  • The KubeVirt path binds each selected GPU to vfio-pci as a whole device, so a VM always gets whole GPUs. MIG, time slicing, and vGPU aren't available on it.