Skip to main content
Version: main 🚧

Node pools

Enterprise
Available in these plansFreeDevProdScale
Private Nodes Auto Nodes

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

A NodePool gives a cluster a set of private worker nodes. It names the node provider, the node types, the network environment, and how many nodes to run. The cluster itself only needs private nodes turned on.

Keeping these choices off the cluster is what lets one cluster template serve many tenants. Each tenant may draw on different node types and network environments. A template can't express "use whatever this tenant is allowed," but a pool created for each cluster can. The template sets privateNodes.enabled: true, and the pools carry the infrastructure.

A pool belongs to one cluster, and a cluster can have many pools. For example, a cluster might have one pool of GPU nodes and another of general compute nodes from a different provider.

Private nodes only

Node pools apply to clusters that use private nodes. Clusters that run workloads on shared nodes of the control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both.Related: Cluster, Control plane cluster, Cluster don't use them and don't change.

A cluster can use node pools and privateNodes.autoNodes in its vcluster.yaml at the same time. The cluster runs the nodes from both.

privateNodes.autoNodes is deprecated

privateNodes.autoNodes is deprecated in vCluster v0.38. It still works, and no removal date is set. Use node pools for new clusters. Control plane auto nodes (controlPlane.standalone.autoNodes) aren't deprecated.

Modify the following with your specific values to replace on the whole page and generate copyable commands:

Static and dynamic pools​

Every pool is either static or dynamic. You choose when you create it, and you can't switch a pool between the two later.

Static poolDynamic pool
Node countA fixed number you set with static.replicasFollows demand, within limits you set
Who creates and removes nodesThe platformKarpenter, running inside the cluster
Good fitReserved capacity, predictable workloads, training jobsBursty workloads, inference that scales with traffic

Before you begin​

  • A NodeProvider with at least one node type, and capacity behind it. For bare metal, see Cluster nodes for how to define a node type.
  • The node types, network environment, and node profile you plan to use must be allowed for the project. See Manage allowed node types and Manage allowed node profiles.
  • A cluster in the project that runs vCluster v0.38 or later, with privateNodes.enabled: true. This applies to static and dynamic pools. The platform refuses to create a pool for a cluster that already reports an unsupported version. The cluster can also be created at the same time as its pools. See Create a pool before its cluster.
  • The platform admin, tenant admin, or project admin role. These roles can create, change, and delete pools within their scope. Project users and project viewers can't manage pools by default.

Run the kubectl commands on this page against the control plane cluster where the platform runs, in the project's namespace. New installs name it p-<project>, and older installs use loft-p-<project>.

Pool names must not repeat names in vcluster.yaml

If the cluster's vcluster.yaml already defines an auto nodes pool with the same name, the platform rejects the pool. Choose a different name. The platform also rejects a pool for a cluster that doesn't have privateNodes.enabled: true.

Create a pool in the UI​

Add pools to an existing cluster from the cluster's Nodes page.

  1. Select the project from the project selector (top left), then click Clusters.

  2. Click the cluster, then click Nodes in the cluster's sidebar.

  3. Click the Node Pools tab, then click .

  4. Under Basics, enter a name for the pool and choose the Infra provider. Set the Network environment if the pool's nodes need a specific one.

  5. Under Nodes, click Static Pool or Dynamic Pool. For a static pool, set the Requested quantity.

    If your license includes node profiles, you can also choose a Node Profile that the project allows.

  6. Choose the node type. Instance type picks one node type from the provider. Properties (advanced) matches node types on their properties, the same as requirements in YAML.

  7. For a dynamic pool, optionally set Node count, CPU limit, Memory limit, and GPU limit. Leave a field empty for no limit.

  8. Click .

The pool appears in the Node Pools tab, and its nodes appear in the Nodes tab as they join.

To add pools while you create a cluster, choose Private Nodes under Worker Nodes in the create cluster form. Then click under Node Pools. The platform creates the pools together with the cluster.

To change a pool, click on its row in the Node Pools tab. To delete it, open the row's actions menu, click , and confirm with the pool's name. You can't switch a pool between static and dynamic after you create it.

The rest of this page uses kubectl and YAML. Those sections also describe what the platform does when you create, change, or delete a pool from the UI.

Create a static pool​

A static pool keeps a fixed number of nodes. Use one when you know how much capacity the cluster needs, for example three compute nodes reserved for a team.

static-pool.yaml
apiVersion: management.loft.sh/v1
kind: NodePool
metadata:
name: compute
namespace: p-my-project
spec:
# The cluster the nodes join. Must be in the same project. Can't be changed later.
vClusterRef: my-cluster
static:
# The platform keeps exactly this many nodes.
replicas: 3
template:
# The NodeProvider the nodes come from.
providerRef: metal3-provider
# Which node types the nodes may use, matched on node type properties.
requirements:
- key: vcluster.com/node-type
operator: In
values:
- metal3-provider.compute-node
# Labels the nodes register with.
nodeLabels:
workload.example.com/tier: compute
kubectl apply -f static-pool.yaml

The platform creates one instance (NodeClaim) for each replica, named after the pool. Each instance carries the vcluster.com/node-pool=compute label so you can find it later. The instance claims a machine of a matching node type and joins the cluster as a worker node.

The pool reports Ready once every replica has joined. To watch its progress, see Check a pool's status.

requirements use the same matching as nodeTypeSelector in vcluster.yaml. The vcluster.com/node-type property holds each node type's fully qualified name, <provider>.<node type>. List the names with kubectl get nodetypes.

Create a dynamic pool​

A dynamic pool hands scaling to Karpenter inside the cluster. Karpenter adds nodes when pods can't be scheduled, removes nodes it can consolidate, and replaces nodes that drifted from the template.

dynamic-pool.yaml
apiVersion: management.loft.sh/v1
kind: NodePool
metadata:
name: burst
namespace: p-my-project
spec:
vClusterRef: my-cluster
dynamic:
# Upper bound on the total resources of the pool's nodes.
limits:
cpu: "128"
memory: 512Gi
disruption:
# Only remove nodes that have no workload pods left.
consolidationPolicy: WhenEmpty
consolidateAfter: 5m
# Disrupt at most one node at a time.
budgets:
- nodes: "1"
# Replace each node after 30 days.
expireAfter: 720h
template:
providerRef: metal3-provider
requirements:
- key: vcluster.com/node-type
operator: In
values:
- metal3-provider.compute-node

Every field under dynamic is optional. When you leave them out, Karpenter consolidates nodes that are empty or underutilized, acts right away, and disrupts up to 10% of the pool's nodes at once. Nodes don't expire, and a node gets 30 seconds to drain before it's removed.

The cluster picks up the pool, creates a matching Karpenter NodePool (karpenter.sh) inside the cluster, and reports progress back to the platform. The pool's AutoscalerReady condition turns true once Karpenter reports the pool ready.

A new dynamic pool has no nodes until pods need them. Zero nodes on a ready pool is expected. Karpenter adds the first node when a pod can't be scheduled on the cluster's existing nodes.

When a cluster has more than one dynamic pool, Karpenter tries pools with a higher weight first. Weights range from 1 to 100.

The platform checks a pool's name against the cluster's vcluster.yaml only when you create the pool. A clash can still happen if you create the pool before its cluster, or add a pool with the same name to vcluster.yaml later. When a dynamic pool's name clashes, the cluster ignores the pool. Its AutoscalerReady condition stays false with the reason NameConflict. Delete the pool and create it again with a different name.

Create a pool before its cluster​

A pool doesn't need its cluster to exist yet. GitOps tools such as Argo CD often apply a cluster and its pools together, in no fixed order.

Until the cluster appears, the pool reports Failed with the reason VClusterNotFound. That's expected. When the cluster is created, the pool picks it up and starts creating nodes without any change on your side.

Choose a network environment​

To place a pool's nodes in a specific network environment, set the vcluster.com/network-environment property in the template:

template:
properties:
vcluster.com/network-environment: tenant-a-network

The project must allow that network environment, and the environment must belong to the pool's node provider. Otherwise, the platform rejects the pool. The environment must also belong to the project's tenant. If it doesn't, or doesn't exist, the pool is accepted but reports NetworkEnvironmentNotFound and creates no nodes.

If you don't name one, the pool inherits any network environment set in its provider's properties. Otherwise, it uses the provider's default network environment. The pool records the environment it chose in status.networkEnvironment and keeps using it, even if the provider's default changes later. All of a pool's nodes stay on one network.

For how network environments work, see Networking.

Change a pool​

Edit the pool and apply it again. What happens depends on what you change.

  • Replica count of a static pool. The platform adds or removes instances to match. When it scales down, it removes instances that haven't joined yet and instances on an older template before healthy current ones. Removing a joined node drains it first, the same as scaling down cluster nodes.
  • Template of a static pool. The platform replaces the nodes one at a time, so the pool runs one node short during the rollout and never needs a spare machine. status.updatedReplicas counts the nodes already on the new template.
  • Dynamic settings or template of a dynamic pool. The cluster updates its Karpenter NodePool. Karpenter replaces drifted nodes within the pool's disruption budgets.

You can't change vClusterRef, and you can't switch a pool between static and dynamic. To move capacity to another cluster or change a pool's kind, create a new pool and delete the old one.

Delete a pool​

kubectl delete nodepool.management.loft.sh compute -n p-my-project

The platform removes the pool's instances before it removes the pool, draining each joined node first.

For a dynamic pool, the platform first gives Karpenter in the cluster up to 10 minutes to drain and remove the nodes. After that, it deletes any instances that are left. If the cluster is missing, being deleted, or doesn't support the pool, the platform deletes the instances without this wait.

When you delete a cluster, the platform deletes its pools and their instances with it. You don't need to delete the pools first.

Check a pool's status​

List the pools in a project:

kubectl get nodepools.management.loft.sh -n p-my-project

For each pool, the output shows its status, cluster, and mode (Static or Dynamic). It also shows the replica count, the ready count, and the network environment. A pool's status is Pending while it scales, rolls out, or waits for its cluster's version. It's Ready once it has converged, and Failed when it can't run. A missing cluster gives Failed with VClusterNotFound, even when you intentionally create the pool first.

To see the reason behind a status, read the pool's conditions:

kubectl get nodepool.management.loft.sh compute -n p-my-project \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
ConditionTrue when
SupportedThe license allows the pool, the cluster exists and isn't being deleted, and the cluster runs a supported vCluster version.
NetworkReadyThe network environment the pool places its nodes in is available. Absent when the pool uses no network environment.
ScaledA static pool has every replica built from the current template and available.
AutoscalerReadyThe cluster has applied the current version of a dynamic pool, and Karpenter reports it ready.

Status reasons​

ReasonWhat it meansWhat to do
FeatureNotAllowedThe license doesn't include auto nodes. A pool whose provider uses a machine inventory also needs machine management. Existing nodes stay, but the pool can't be changed.Check the platform license.
VClusterNotFoundNo cluster with the vClusterRef name exists in the project.Expected while the cluster is being created. Otherwise, check the name.
VClusterDeletingThe cluster is being deleted.None. The pool is deleted with the cluster.
VClusterVersionUnknownThe cluster hasn't reported its vCluster version yet.Wait for the cluster to come up.
VClusterVersionUnsupportedThe cluster runs a vCluster version older than v0.38. This happens when the cluster reports its version after the pool is created.Upgrade the cluster.
NodeProviderNotFoundThe NodeProvider in providerRef doesn't exist.Fix providerRef.
NetworkEnvironmentNotAllowedThe project no longer allows the pool's network environment.Allow it for the project, or move the nodes to a new pool in an allowed environment.
NetworkEnvironmentNotFoundThe network environment doesn't exist, or doesn't belong to the project's tenant.Fix the vcluster.com/network-environment property.
NetworkEnvironmentNotAvailableThe network environment is still provisioning, has failed, or is being deleted.Instances wait for it. Check the network environment's status.
QuotaExceededAnother node for a static pool would go over the project's quotas.Raise the quota or lower static.replicas. The pool resumes on its own once the nodes fit.
NameConflictA dynamic pool has the same name as a pool in the cluster's vcluster.yaml.Delete the pool and create it again with a different name.

When the pool itself looks healthy but a node never appears, check its instances. They carry the pool's name as a label:

kubectl get nodeclaims -n p-my-project -l vcluster.com/node-pool=compute

An instance that waits for capacity, quota, or a network environment shows the reason on the instance itself.

Template fields​

The template describes every node in the pool. It applies to static and dynamic pools alike.

FieldRequiredDescription
providerRefYesThe NodeProvider the nodes come from.
requirementsNoWhich node types the nodes may use, as Kubernetes node selector requirements keyed on node type properties.
propertiesNoExtra properties for each instance. Common ones are the vcluster.com/network-environment and vcluster.com/os-image properties.
taintsNoTaints applied to the nodes.
nodeLabelsNoLabels the nodes register with. Labels under kubernetes.io, k8s.io, karpenter.sh, and vcluster.loft.sh are rejected, except under node.kubernetes.io.
profileRefNoA node profile to apply to the nodes. The project must allow it.

The pool name becomes a label value and the prefix of each instance name, so it must be a valid DNS label of at most 63 characters.

For every field of the resource, including dynamic and status, see the NodePool API reference.

If you want toGo to
Define node types and attach bare metal capacityCluster nodes
Apply reusable labels and taints to nodesNode profiles
Configure auto nodes directly in vcluster.yamlAuto nodes
Understand how a node joins a clusterCluster attachment
Look up every NodePool field and API operationNodePool API reference