Skip to main content
Version: v5.0 Stable

Node providers and node types

Node providers let you configure sources for private nodes for clusters automatically. Platform supports the following node providers:

  • Terraform (powered by OpenTofu under the hood)
  • KubeVirt for VM-backed private nodes. To inspect and operate KubeVirt VMs directly, see Virtual Machines.
  • Metal3 for bare metal provisioning using Metal3 and Ironic.
  • External platform for private nodes from a Metal3 provider on another vCluster Platform.
  • NVIDIA Infra Controller (NICo) for bare metal provisioning using NVIDIA's infrastructure management platform.
  • Cluster API, for infrastructure provisioned through a Cluster API provider's own contract. Available as a driver mechanism rather than a specific provider.

These node providers implement the common vMetal NodeClaim API and lifecycle. Clusters request capacity through the same model without depending on the underlying infrastructure API. See vMetal overview for how they fit together.

Node types inside a tenant

A tenant is granted node types, not providers. Granting one makes the provider it belongs to reachable as well, since the grant would be unusable otherwise. That shows up as a derived grant in the tenant's status boundary.

Each node provider defines provider-specific configuration and node types that clusters select from. When a cluster requests a new node, it creates a node claim, telling the platform to provision a node with the given requirements. The platform then matches these requirements to the cheapest available node type and creates a node from that type.

vCluster PlatformvClusterNode ProviderNode ClaimNode Typecreatesdefinescreatesreferences

An example node provider with two distinct node types:

apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: terraform-example
spec:
terraform:
nodeTemplate:
git:
repository: https://github.com/my-org/my-repo.git
nodeTypes:
- name: medium
resources:
cpu: "2"
memory: 4Gi
properties:
instance-type: medium
- name: large
resources:
cpu: "4"
memory: 8Gi
properties:
instance-type: large

This creates a node provider with a Terraform node template. For each node claim, the platform executes the Terraform script. Reference this node provider in your cluster using:

# Use this in your vcluster.yaml
privateNodes:
enabled: true
autoNodes:
provider: terraform-example
dynamic:
# This node pool automatically chooses the node type
# to right size the amount of pods.
- name: my-node-pool
Single node provider, many clusters

Don't create a node provider per cluster. Create one provider, then make its node types available only within the applicable Tenant and Project resource boundaries.

Node types​

Node types are machine types that vCluster selects based on properties, resources, and cost. Create a node type for each instance type you want vCluster to choose from, such as different VM sizes or cloud instance types.

vCluster automatically selects the best fitting node type based on requirements. If there are no requirements or multiple node types match, vCluster selects the cheapest available node type.

Properties​

Properties on a node type are similar to labels and let users specify traits to include or exclude certain node types when clusters are created. Users can define custom properties in addition to the always-available built-in properties.

Use properties​

Use properties through the requirements field under autoNodes node pools to include or exclude node types:

# Inside your vcluster.yaml
privateNodes:
enabled: true
autoNodes:
provider: my-provider
properties:
# Exact match
my-propert: my-value

# One of
my-property/value-1: true
my-property/value-2: true
my-property/value-3: true

# Not in
my-property/value-1: false
my-property/value-2: false
my-property/value-3: false

# Exists
my-property: true

# Not exists
my-property: false
static:
- name: my-static-pool

The following operators are available (and similar to Kubernetes set-based requirements):

  • In (default): Matches one or multiple values on the node type
  • NotIn: Matches if the given values aren't part of the properties
  • Exists: Matches if the property is defined on the node type
  • NotExists: Matches if the property is not defined on the node type

Custom properties​

Custom properties are defined using the properties field and map a key to one or multiple values. For example:

...
kind: NodeProvider
...
spec:
terraform:
nodeTypes:
- name: medium
[...]
properties:
# Single value
my-property: my-value
# Multiple values
my-property-multi: my-value-1,my-value-2
# All values
my-property-all: "*"

Property values must adhere to the syntax and character set of Kubernetes labels, as they are set as labels on created nodes.

In-built properties​

The platform automatically adds these properties:

  • vcluster.com/node-type: The node type's fully qualified name, which the platform generates as <provider>.<node type>. Maps vCluster node pools to a specific node type. When the qualified and bare names differ, the platform sets both as a comma-separated pair so a selector written against either one still matches. Prefer the qualified form. It's what the API and the platform UI show, and the platform is migrating to it.
  • node.kubernetes.io/instance-type: The official Kubernetes label, carrying the fully qualified node type name. Unlike vcluster.com/node-type, it only ever holds that single value, because Karpenter rejects a list here.
  • kubernetes.io/os: Always linux.
  • topology.kubernetes.io/zone: Maps to the spec.zone field of the node type. Defaults to global if unspecified.
  • karpenter.sh/capacity-type: on-demand unless the node type sets it in its properties.

Resources​

The resources field is required on each node type and specifies the computing resources nodes of that type provide. Certain node providers, such as KubeVirt, auto-discover resources, while others require you to specify at least cpu and memory for each node type. Resources are specified according to the Kubernetes resource units.

You can also specify custom resources such as nvidia.com/gpu, used for scheduling inside the cluster. The platform does not schedule a node claim to a node type with custom resources unless those resources are explicitly requested. A cluster that does not request a GPU is never assigned a GPU node type, even if it is the only available type.

Cost​

Each node type has an associated cost, either set manually using spec.cost or calculated from resources. The cost unit is intentionally unspecified. It is not a currency but a relative value vCluster and the platform use to select the cheapest matching node type.

Default costs when not specified manually:

  • Each cpu costs 10
  • Each GB of memory costs 2
  • Each other resource costs 10000, discouraging scheduling on specialized resources.

Karpenter always tries to find the cheapest node and automatically provisions or deprovisions nodes as costs change.

Capacity​

Capacity is the number of available nodes for a given node type. The appropriate limit depends on the scenario. For bare metal, for example, you may want to cap the number of usable machines.

Depending on the provider, capacity is set manually or discovered automatically:

  • Terraform: set using maxCapacity, otherwise unlimited
  • KubeVirt: set using maxCapacity, otherwise unlimited
  • Metal3: determined by the available servers matching the node type's machines.selector and resources
  • External platform: the remote node type's free capacity. For Assigned types with finite capacity, this is capped at the tenant's free machines across the remote provider, excluding machines shared with all tenants. OnDemand and unlimited types aren't capped. See Node types and capacity.
  • NICo: counted from the provider's Machines that match the node type's machine label selector and are available or already claimed by that node type

For Metal3 and NICo, free capacity follows the machine allocation policy. OnDemand types count only unassigned machines as free capacity. When choosing an Assigned type for a claim, the platform checks the requesting tenant's free reserved capacity.

If specified, the platform tracks how many nodes of this type have been created and marks the node type as unavailable once the limit is reached. vCluster stops using that node type until nodes are deprovisioned.

Property merging​

Provisioning properties, such as OS image, SSH keys, and network configuration, can be set at the node provider level, the node type level, or the individual node claim level. When the same property is set at more than one level, the most specific level wins: node claim properties override node type properties, which override node provider properties.

Machine allocation​

For providers that draw from a machine inventory, a node type's machine allocation policy decides which machines a tenant's node claims can use. Metal3 and NICo place servers by it. KubeVirt ignores it and runs every VM on the KubeVirt cluster's own nodes.

An operator assigns a machine to a Tenant by setting the tenant.vcluster.com/exclusive-to label on the Machine, or with the Assign to tenant action on the admin Machines page. A Machine can also belong to a tenant through the tenant.vcluster.com/owner label. A machine is unassigned only when neither label is present.

PolicyMetal3 UI optionA tenant's node claims use
AssignedReserved hosts onlyOnly machines owned by or exclusively assigned to the tenant. A claim waits when none of them is free.
OnDemandUnassigned hosts onlyOnly machines that no tenant owns or holds. Reserved machines are never used. A claim waits when no unassigned machine is free.

A node claim made outside any tenant, such as one by a platform administrator, draws from unassigned machines under either policy.

Set the policy on the node type with machineAllocationPolicy, or on the node provider with spec.machineAllocationPolicy as the default for node types that leave theirs empty. When neither is set, the node type uses Assigned.

apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: metal3-provider
spec:
# Default for this provider's node types.
machineAllocationPolicy: Assigned
metal3:
nodeTypes:
- name: compute-node
# This node type draws only from unassigned servers.
machineAllocationPolicy: OnDemand
machines:
selector:
matchLabels:
role: compute

Under Assigned, free unassigned machines are withheld from tenants. A tenant's claim waits even when an unassigned server of the right type is idle. Switch the node type to OnDemand to let it take one.

What users without edit access see​

A user who can't update a node provider or node type sees a restricted copy of it, without its infrastructure configuration. They see the provider's name and the node type's spec, phase, cost, and capacity, but not labels, deploy settings, or the Metal3 clusterRef. When no provider the user can read exposes clusterRef, the platform UI hides Add Server.

Users in a tenant see this restricted copy of any provider or node type their tenant doesn't own. If they can't tell which node type to use, or a node type needs changing, an operator with edit access has to identify or edit it.

Change or delete a node type in use​

The platform refuses a change that would take a node type away from node claims that still use it. Each refusal names those node claims. The following changes are refused while node claims use the node type:

  • Deleting the node type.
  • Removing the node type from its node provider's nodeTypes.
  • Relabeling the node type, directly or through its node provider, so that a tenant using it is no longer granted it.
  • Editing a tenant's spec.nodeTypes so it no longer grants the node type to that tenant.

Delete the named node claims first. Deleting or removing the node type goes through once their deletion has started. A change that hides the node type waits until they're fully gone. See A node type can't be hidden or deleted while node claims use it.

Node claims​

When a cluster requests a new node, either dynamically using Karpenter or through static configuration, vCluster creates a node claim in the platform with requirements. An administrator can also create a node claim directly, with no cluster attached, to provision a server that runs no Kubernetes. See Independently provisioned instances. The platform then selects the node type that best fulfills those requirements. If multiple node types match, the platform chooses the cheapest available one.

Tenant resource boundaries define the maximum set of node types available to an organization. A Project can only narrow that set. Use spec.allowedNodeTypes to restrict which admitted node types the Project's NodeClaims may use.

vCluster PlatformvClusterNode ClaimNode TypeNode ProviderNodecreatesschedulesreferencescreatesjoins
Graceful drain on deprovisioning

When a NodeClaim for a joined worker is deleted, the platform first deletes the corresponding cluster Node so its workloads evict gracefully. It then waits until the Node is gone before destroying the underlying server. If the drain doesn't complete within 30 minutes, the platform stops waiting and proceeds with teardown. This is the same backstop that applies when the cluster stays unreachable for that long. To skip the drain and destroy the server immediately, set the vcluster.com/skip-node-drain annotation on the NodeClaim. Or click Force delete in the platform to do the same.

The drain doesn't apply to control plane instances or to instances that never joined a cluster, such as independently provisioned instances. Those are destroyed directly. See Change how many nodes you have.

Network environments​

A network environment is a cluster-scoped NetworkEnvironment resource that describes the network a provider's machines join. A platform administrator creates it explicitly and either assigns it to a Tenant or leaves it unassigned for the platform. The platform doesn't create environments automatically. Projects, clusters, and standalone machines share environments rather than owning them, and a project's allowed network environments narrow which ones it can use.

Some providers need infrastructure before they can create nodes. For example, a Terraform provider with a network environment template can create a VPC, routing tables, and subnets. The platform provisions that infrastructure when the environment is created, and node creation against it waits until the environment is ready. Other providers only merge the environment's properties into each node's configuration.

A NodeClaim selects an environment through the vcluster.com/network-environment property, or falls back to the provider's default environment. See Network environments for assignment, immutability, and selection rules.