Node pools
| Enterprise | ||||
|---|---|---|---|---|
| Available in these plans | Free | Dev | Prod | Scale |
| Private Nodes Auto Nodes | ||||
Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.
A NodePool gives a cluster a set of private worker nodes. It names the node provider, the node types, the network environment, and how many nodes to run. The cluster itself only needs private nodes turned on.
Keeping these choices off the cluster is what lets one cluster template serve many tenants. Each tenant may draw on different node types and network environments. A template can't express "use whatever this tenant is allowed," but a pool created for each cluster can. The template sets privateNodes.enabled: true, and the pools carry the infrastructure.
A pool belongs to one cluster, and a cluster can have many pools. For example, a cluster might have one pool of GPU nodes and another of general compute nodes from a different provider.
Node pools apply to clusters that use private nodes. Clusters that run workloads on shared nodes of the control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both. don't use them and don't change.
A cluster can use node pools and privateNodes.autoNodes in its vcluster.yaml at the same time. The cluster runs the nodes from both.
privateNodes.autoNodes is deprecatedprivateNodes.autoNodes is deprecated in vCluster v0.38. It still works, and no removal date is set. Use node pools for new clusters. Control plane auto nodes (controlPlane.standalone.autoNodes) aren't deprecated.
Static and dynamic pools​
Every pool is either static or dynamic. You choose when you create it, and you can't switch a pool between the two later.
| Static pool | Dynamic pool | |
|---|---|---|
| Node count | A fixed number you set with static.replicas | Follows demand, within limits you set |
| Who creates and removes nodes | The platform | Karpenter, running inside the cluster |
| Good fit | Reserved capacity, predictable workloads, training jobs | Bursty workloads, inference that scales with traffic |
Before you begin​
- A NodeProvider with at least one node type, and capacity behind it. For bare metal, see Cluster nodes for how to define a node type.
- The node types, network environment, and node profile you plan to use must be allowed for the project. See Manage allowed node types and Manage allowed node profiles.
- A cluster in the project that runs vCluster v0.38 or later, with
privateNodes.enabled: true. This applies to static and dynamic pools. The platform refuses to create a pool for a cluster that already reports an unsupported version. The cluster can also be created at the same time as its pools. See Create a pool before its cluster. - The platform admin, tenant admin, or project admin role. These roles can create, change, and delete pools within their scope. Project users and project viewers can't manage pools by default.
Run the kubectl commands on this page against the control plane cluster where the platform runs, in the project's namespace. New installs name it p-<project>, and older installs use loft-p-<project>.
If the cluster's vcluster.yaml already defines an auto nodes pool with the same name, the platform rejects the pool. Choose a different name. The platform also rejects a pool for a cluster that doesn't have privateNodes.enabled: true.
Create a pool in the UI​
Add pools to an existing cluster from the cluster's Nodes page.
Select the project from the project selector (top left), then click Clusters.
Click the cluster, then click Nodes in the cluster's sidebar.
Click the Node Pools tab, then click .
Under Basics, enter a name for the pool and choose the Infra provider. Set the Network environment if the pool's nodes need a specific one.
Under Nodes, click Static Pool or Dynamic Pool. For a static pool, set the Requested quantity.
If your license includes node profiles, you can also choose a Node Profile that the project allows.
Choose the node type. Instance type picks one node type from the provider. Properties (advanced) matches node types on their properties, the same as
requirementsin YAML.For a dynamic pool, optionally set Node count, CPU limit, Memory limit, and GPU limit. Leave a field empty for no limit.
Click .
The pool appears in the Node Pools tab, and its nodes appear in the Nodes tab as they join.
To add pools while you create a cluster, choose Private Nodes under Worker Nodes in the create cluster form. Then click under Node Pools. The platform creates the pools together with the cluster.
To change a pool, click on its row in the Node Pools tab. To delete it, open the row's actions menu, click , and confirm with the pool's name. You can't switch a pool between static and dynamic after you create it.
The rest of this page uses kubectl and YAML. Those sections also describe what the platform does when you create, change, or delete a pool from the UI.
Create a static pool​
A static pool keeps a fixed number of nodes. Use one when you know how much capacity the cluster needs, for example three compute nodes reserved for a team.
apiVersion: management.loft.sh/v1
kind: NodePool
metadata:
name: compute
namespace: p-my-project
spec:
# The cluster the nodes join. Must be in the same project. Can't be changed later.
vClusterRef: my-cluster
static:
# The platform keeps exactly this many nodes.
replicas: 3
template:
# The NodeProvider the nodes come from.
providerRef: metal3-provider
# Which node types the nodes may use, matched on node type properties.
requirements:
- key: vcluster.com/node-type
operator: In
values:
- metal3-provider.compute-node
# Labels the nodes register with.
nodeLabels:
workload.example.com/tier: compute
kubectl apply -f static-pool.yaml
The platform creates one instance (NodeClaim) for each replica, named after the pool. Each instance carries the vcluster.com/node-pool=compute label so you can find it later. The instance claims a machine of a matching node type and joins the cluster as a worker node.
The pool reports Ready once every replica has joined. To watch its progress, see Check a pool's status.
requirements use the same matching as nodeTypeSelector in vcluster.yaml. The vcluster.com/node-type property holds each node type's fully qualified name, <provider>.<node type>. List the names with kubectl get nodetypes.
Create a dynamic pool​
A dynamic pool hands scaling to Karpenter inside the cluster. Karpenter adds nodes when pods can't be scheduled, removes nodes it can consolidate, and replaces nodes that drifted from the template.
apiVersion: management.loft.sh/v1
kind: NodePool
metadata:
name: burst
namespace: p-my-project
spec:
vClusterRef: my-cluster
dynamic:
# Upper bound on the total resources of the pool's nodes.
limits:
cpu: "128"
memory: 512Gi
disruption:
# Only remove nodes that have no workload pods left.
consolidationPolicy: WhenEmpty
consolidateAfter: 5m
# Disrupt at most one node at a time.
budgets:
- nodes: "1"
# Replace each node after 30 days.
expireAfter: 720h
template:
providerRef: metal3-provider
requirements:
- key: vcluster.com/node-type
operator: In
values:
- metal3-provider.compute-node
Every field under dynamic is optional. When you leave them out, Karpenter consolidates nodes that are empty or underutilized, acts right away, and disrupts up to 10% of the pool's nodes at once. Nodes don't expire, and a node gets 30 seconds to drain before it's removed.
The cluster picks up the pool, creates a matching Karpenter NodePool (karpenter.sh) inside the cluster, and reports progress back to the platform. The pool's AutoscalerReady condition turns true once Karpenter reports the pool ready.
A new dynamic pool has no nodes until pods need them. Zero nodes on a ready pool is expected. Karpenter adds the first node when a pod can't be scheduled on the cluster's existing nodes.
When a cluster has more than one dynamic pool, Karpenter tries pools with a higher weight first. Weights range from 1 to 100.
The platform checks a pool's name against the cluster's vcluster.yaml only when you create the pool. A clash can still happen if you create the pool before its cluster, or add a pool with the same name to vcluster.yaml later. When a dynamic pool's name clashes, the cluster ignores the pool. Its AutoscalerReady condition stays false with the reason NameConflict. Delete the pool and create it again with a different name.
Create a pool before its cluster​
A pool doesn't need its cluster to exist yet. GitOps tools such as Argo CD often apply a cluster and its pools together, in no fixed order.
Until the cluster appears, the pool reports Failed with the reason VClusterNotFound. That's expected. When the cluster is created, the pool picks it up and starts creating nodes without any change on your side.
Choose a network environment​
To place a pool's nodes in a specific network environment, set the vcluster.com/network-environment property in the template:
template:
properties:
vcluster.com/network-environment: tenant-a-network
The project must allow that network environment, and the environment must belong to the pool's node provider. Otherwise, the platform rejects the pool. The environment must also belong to the project's tenant. If it doesn't, or doesn't exist, the pool is accepted but reports NetworkEnvironmentNotFound and creates no nodes.
If you don't name one, the pool inherits any network environment set in its provider's properties. Otherwise, it uses the provider's default network environment. The pool records the environment it chose in status.networkEnvironment and keeps using it, even if the provider's default changes later. All of a pool's nodes stay on one network.
For how network environments work, see Networking.
Change a pool​
Edit the pool and apply it again. What happens depends on what you change.
- Replica count of a static pool. The platform adds or removes instances to match. When it scales down, it removes instances that haven't joined yet and instances on an older template before healthy current ones. Removing a joined node drains it first, the same as scaling down cluster nodes.
- Template of a static pool. The platform replaces the nodes one at a time, so the pool runs one node short during the rollout and never needs a spare machine.
status.updatedReplicascounts the nodes already on the new template. - Dynamic settings or template of a dynamic pool. The cluster updates its Karpenter
NodePool. Karpenter replaces drifted nodes within the pool's disruption budgets.
You can't change vClusterRef, and you can't switch a pool between static and dynamic. To move capacity to another cluster or change a pool's kind, create a new pool and delete the old one.
Delete a pool​
kubectl delete nodepool.management.loft.sh compute -n p-my-project
The platform removes the pool's instances before it removes the pool, draining each joined node first.
For a dynamic pool, the platform first gives Karpenter in the cluster up to 10 minutes to drain and remove the nodes. After that, it deletes any instances that are left. If the cluster is missing, being deleted, or doesn't support the pool, the platform deletes the instances without this wait.
When you delete a cluster, the platform deletes its pools and their instances with it. You don't need to delete the pools first.
Check a pool's status​
List the pools in a project:
kubectl get nodepools.management.loft.sh -n p-my-project
For each pool, the output shows its status, cluster, and mode (Static or Dynamic). It also shows the replica count, the ready count, and the network environment. A pool's status is Pending while it scales, rolls out, or waits for its cluster's version. It's Ready once it has converged, and Failed when it can't run. A missing cluster gives Failed with VClusterNotFound, even when you intentionally create the pool first.
To see the reason behind a status, read the pool's conditions:
kubectl get nodepool.management.loft.sh compute -n p-my-project \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
| Condition | True when |
|---|---|
Supported | The license allows the pool, the cluster exists and isn't being deleted, and the cluster runs a supported vCluster version. |
NetworkReady | The network environment the pool places its nodes in is available. Absent when the pool uses no network environment. |
Scaled | A static pool has every replica built from the current template and available. |
AutoscalerReady | The cluster has applied the current version of a dynamic pool, and Karpenter reports it ready. |
Status reasons​
| Reason | What it means | What to do |
|---|---|---|
FeatureNotAllowed | The license doesn't include auto nodes. A pool whose provider uses a machine inventory also needs machine management. Existing nodes stay, but the pool can't be changed. | Check the platform license. |
VClusterNotFound | No cluster with the vClusterRef name exists in the project. | Expected while the cluster is being created. Otherwise, check the name. |
VClusterDeleting | The cluster is being deleted. | None. The pool is deleted with the cluster. |
VClusterVersionUnknown | The cluster hasn't reported its vCluster version yet. | Wait for the cluster to come up. |
VClusterVersionUnsupported | The cluster runs a vCluster version older than v0.38. This happens when the cluster reports its version after the pool is created. | Upgrade the cluster. |
NodeProviderNotFound | The NodeProvider in providerRef doesn't exist. | Fix providerRef. |
NetworkEnvironmentNotAllowed | The project no longer allows the pool's network environment. | Allow it for the project, or move the nodes to a new pool in an allowed environment. |
NetworkEnvironmentNotFound | The network environment doesn't exist, or doesn't belong to the project's tenant. | Fix the vcluster.com/network-environment property. |
NetworkEnvironmentNotAvailable | The network environment is still provisioning, has failed, or is being deleted. | Instances wait for it. Check the network environment's status. |
QuotaExceeded | Another node for a static pool would go over the project's quotas. | Raise the quota or lower static.replicas. The pool resumes on its own once the nodes fit. |
NameConflict | A dynamic pool has the same name as a pool in the cluster's vcluster.yaml. | Delete the pool and create it again with a different name. |
When the pool itself looks healthy but a node never appears, check its instances. They carry the pool's name as a label:
kubectl get nodeclaims -n p-my-project -l vcluster.com/node-pool=compute
An instance that waits for capacity, quota, or a network environment shows the reason on the instance itself.
Template fields​
The template describes every node in the pool. It applies to static and dynamic pools alike.
| Field | Required | Description |
|---|---|---|
providerRef | Yes | The NodeProvider the nodes come from. |
requirements | No | Which node types the nodes may use, as Kubernetes node selector requirements keyed on node type properties. |
properties | No | Extra properties for each instance. Common ones are the vcluster.com/network-environment and vcluster.com/os-image properties. |
taints | No | Taints applied to the nodes. |
nodeLabels | No | Labels the nodes register with. Labels under kubernetes.io, k8s.io, karpenter.sh, and vcluster.loft.sh are rejected, except under node.kubernetes.io. |
profileRef | No | A node profile to apply to the nodes. The project must allow it. |
The pool name becomes a label value and the prefix of each instance name, so it must be a valid DNS label of at most 63 characters.
For every field of the resource, including dynamic and status, see the NodePool API reference.
Related tasks​
| If you want to | Go to |
|---|---|
| Define node types and attach bare metal capacity | Cluster nodes |
| Apply reusable labels and taints to nodes | Node profiles |
Configure auto nodes directly in vcluster.yaml | Auto nodes |
| Understand how a node joins a cluster | Cluster attachment |
| Look up every NodePool field and API operation | NodePool API reference |