External platform
| Enterprise | ||||
|---|---|---|---|---|
| Available in these plans | Free | Dev | Prod | Scale |
| External Platform Node Provider | ||||
| Machine Management | ||||
Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.
The external platform provider lets one vCluster Platform supply private nodes from bare metal servers that another platform manages. On this page, the local platform runs the clustersClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster., and the remote platform owns the servers.
The remote platform keeps full control of its hardware. It provisions the servers with its own Metal3 provider, and it decides which node types and servers the local platform can use. The local platform uses that inventory like its own. Clusters select the node types through Auto Nodes, and the servers appear as Machines on the local platform.
This provider requires a working Metal3 provider on the remote platform. See Before you begin.
Use this provider when one platform operates a bare metal fleet and another platform consumes part of it. For example, an AI cloud provider can give a customer's own platform a share of its GPU servers. The customer's platform gets an access key for one Tenant, not access to the Metal3 infrastructure.
How the provider works​
The provider connects to the remote platform's API with an access key. It mirrors what the access key can read from the remote Metal3 provider, and it creates a remote NodeClaim for every local one.
| Local platform | Remote platform |
|---|---|
NodeProvider with spec.externalPlatform | Metal3 NodeProvider named in nodeProvider |
NodeType <local provider>.<name> | NodeType <remote provider>.<name> |
Machine <local provider>.<machine ID> | Machine <remote provider>.<machine ID> |
| NodeClaim in a local project | NodeClaim in the project named in project |
Node types and capacity​
The provider mirrors every node type of the remote provider that the access key can read, and refreshes them periodically.
A mirrored node type copies the remote resources, cost, display name, properties, and machine allocation policy.
It records the remote node type in the platform.vcluster.com/remote-node-type property.
The local platform resolves some properties itself and drops them from the mirrored node type, such as vcluster.com/os-image. See Configuration properties.
A remote node type counts the servers of every tenant on the remote platform.
The local capacity depends on the remote node type's spec.machineAllocationPolicy:
Assigned: the remote node type's free capacity is capped at the tenant's free Machines across that remote provider. Machines shared with all tenants are excluded from this cap. Claims use Machines the tenant owns or has been assigned exclusively. The cap doesn't check which node type's pool a Machine belongs to, so it can overstate capacity for a specific node type.OnDemand: the remote capacity is used without that cap. Claims use only unassigned Machines. The tenant's reserved Machines aren't used.
A remote node type with unlimited capacity stays unlimited regardless of its allocation policy.
Metal3 node types default to Assigned unless the remote provider sets another policy.
Node provisioning​
For every local NodeClaim, the provider creates a NodeClaim in the remote project. It resolves every reference to a local object first, so the remote platform never needs those objects:
- OS image: the provider reads the OSImage named in
vcluster.com/os-imageon the local platform. It copies the OSImage's Metal3 image properties,metal3.vcluster.com/image-url,metal3.vcluster.com/image-checksum, andmetal3.vcluster.com/image-checksum-type, onto the remote claim. Every claim needs an OSImage. - User data: the local platform renders cloud-init, including SSH keys, user data templates, and the command that joins the node to the local cluster. The remote claim receives the finished document, so the server joins the local cluster when it boots. See Keep the remote operator's cloud-init.
- Network: the remote platform allocates the node's network from one of its own network environments. A local network environment names that remote environment in its
platform.vcluster.com/remote-network-environmentproperty, and the remote claim references it. The local environment's other network settings aren't sent. A network data template stays unrendered, because the remote platform renders it with the IP address it allocates.
The remote claim also carries the other properties of the local node provider and the local claim, and the remote Metal3 provider applies them. The local node type's properties aren't sent. The remote claim names its remote node type, and the remote platform applies that node type's own properties.
The provider checks the properties it forwards against the claim rules of the remote Metal3 provider. Suppose the remote provider would refuse one of them on a claim, such as a network setting. Then the local claim fails with the reason InvalidConfiguration, and the message names the property.
The local NodeClaim is provisioned once the remote one is.
The provider copies the remote machine ID and the node IP address onto the local claim, in the machines.vcluster.com/machine-id and nodeclaim.vcluster.com/ip-address annotations.
A local NodeClaim that references one of the mirrored Machines is provisioned on the same machine on the remote platform.
Node removal​
Deleting a local NodeClaim deletes its remote NodeClaim. The local claim is removed once the remote one is gone. The remote Metal3 provider then returns the server to its pool, the same as for any other claim.
Power control​
Power a server from its local NodeClaim, as with any other instance. The provider passes the request to the remote NodeClaim, and the remote platform powers the server. The remote platform's UI shows the remote NodeClaim's power actions as disabled, and a change made there is reverted. The remote platform must report power state, which platforms older than v5.0 don't. See Power control.
Before you begin​
Make sure you have the following before you set up the provider:
- A remote vCluster Platform with a working Metal3 provider. Only Metal3 providers are supported on the remote side. The remote hardware, BMC, and network must meet the Metal3 requirements.
- Network access:
- The local platform reaches the remote platform's URL.
- The remote servers reach the cluster they join, as with any private node.
- The remote site can download the OS image, because the remote Metal3 provider installs the server from its URL.
- An OSImage on the local platform that sets the Metal3 image properties, including the image URL and checksum.
Leave
spec.connectorRefunset. The external platform provider rejects images backed by an OS image store. The examples on this page use an image namedubuntu-noble. - If your clusters select a network environment, a network environment on the remote platform for each local one. Set the remote environment's name in the local environment's
platform.vcluster.com/remote-network-environmentproperty.
Prepare the remote platform​
Complete these steps on the remote platform. They give the local platform an access key scoped to one tenant. The granted node types and their machine allocation policies determine which servers it can use.
If someone else operates the remote platform, send them these steps and ask for the following:
- The URL of the remote platform.
- The access key.
- The name of the Metal3 node provider.
- The project to create NodeClaims in. For a Tenant this is
<tenant>--default. - The node types the access key can use.
Create a Tenant and grant it node types.
Grant the node types of the Metal3 provider by name. The local platform mirrors only the node types granted here.
tenant.yamlapiVersion: management.loft.sh/v1kind: Tenantmetadata:name: acmespec:nodeTypes:enabled: trueallow:byName:- metal3-provider.gpu-nodeCreating the Tenant also creates the tenant admin user
acme--adminand the projectacme--default.Assign servers for
Assignednode types.For
Assignednode types, NodeClaims only use Machines the tenant owns or that you assign to it exclusively. Assign each Machine with thetenant.vcluster.com/exclusive-tolabel.Assign a Machine to the Tenantkubectl label machines.management.loft.sh metal3-provider.gpu-server-1 tenant.vcluster.com/exclusive-to=acmeFor
OnDemandnode types, leave Machines unassigned. Assigning a Machine to the tenant makes it unavailable toOnDemandrequests.Create an access key for the tenant admin.
Create an access key for the
acme--adminuser and copy it. You need it on the local platform in the next section.
Changing or resetting the password of the tenant admin deletes its access keys. The local platform then can't reach the remote platform until you replace the access key.
Use a remote user instead of a tenant admin​
The access key can also belong to a remote user outside any Tenant. The user must be able to manage NodeClaims in the project and read the remote node provider, its node types, and its Machines. The local platform then sees every node type and Machine that user can read.
Connect the local platform​
Complete these steps on the local platform.
Store the access key in a Secret.
Create the Secret in the platform namespace, for example
vcluster-platform. The access key goes under theaccessKeykey.Modify the following with your specific values to generate a copyable command:access-key-secret.yamlapiVersion: v1kind: Secretmetadata:name: remote-platform-access-keynamespace: vcluster-platformstringData:accessKey: ""Create the node provider.
node-provider.yamlapiVersion: management.loft.sh/v1kind: NodeProvidermetadata:name: remote-metal3spec:displayName: Remote bare metalproperties:# OSImage on the local platform to install on every nodevcluster.com/os-image: ubuntu-nobleexternalPlatform:# URL of the remote platformhost: https://platform.example.com# Secret that holds the access key under the key "accessKey"secretRef:name: remote-platform-access-keynamespace: vcluster-platform# Metal3 NodeProvider on the remote platformnodeProvider: metal3-provider# Project on the remote platform to create NodeClaims inproject: acme--defaultIn the UI, create a node provider and choose External Platform as its type. The Remote Platform Configuration section holds the same fields.
If the remote platform uses a certificate from a private CA, also set
caData. See Configure TLS.Check that the provider is available.
Check the provider and its node typeskubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.phase}'kubectl get nodetypes.management.loft.shThe provider reports the
Availablephase. The node types of the remote provider appear with the local provider as prefix, for exampleremote-metal3.gpu-node. If the provider doesn't become available, see Troubleshoot.Check that the remote servers are mirrored.
List the mirrored Machineskubectl get machines.management.loft.shThe Machines the access key can read appear as
<local provider>.<machine ID>, for exampleremote-metal3.gpu-server-1. A tenant access key can't list unassigned Machines, even when anOnDemandnode type has capacity. When a cluster claims a node, the server joins that cluster and its NodeClaim reports as provisioned.
Configure TLS​
By default, the provider verifies the remote platform's serving certificate against the system root certificates.
If the remote platform uses a certificate from a private CA, set caData to the base64-encoded PEM CA bundle.
Generate the value from your CA certificate file:
base64 < ca.crt | tr -d '\n'
Then set the value in the node provider:
spec:
externalPlatform:
host: https://platform.internal.example.com
caData: ""
caData must contain a valid PEM certificate, or the provider is rejected.
For development only, insecureSkipTLSVerify: true turns verification off.
You can't combine it with caData.
Configuration properties​
Properties merge in this order, with later ones overriding earlier ones: the node provider, the node type, the network environment, and the node claim. The provider then splits the merged result into two groups before it creates the remote claim.
Resolved on the local platform​
These properties name local objects or feed the local cloud-init rendering, so the remote platform never receives them as set. The provider reads the objects they reference and sends the result instead.
| Property | What the provider sends to the remote claim |
|---|---|
vcluster.com/os-image | The Metal3 image URL, checksum, and checksum type of the referenced OSImage. Image properties set anywhere else aren't sent. |
vcluster.com/ssh-keys | The keys rendered into the cloud-init user data. |
vcluster.com/user-data, vcluster.com/user-data-template-config, and the other user data template properties | One rendered user data document that includes the cluster join command. |
vcluster.com/network-data-template-config and the other network data template properties | The network data template as an unrendered string. The remote platform renders it with the IP address it allocates. |
vcluster.com/network-environment | The name of the remote network environment, from the local environment's platform.vcluster.com/remote-network-environment property. A local environment without that property fails the claim with the reason InvalidConfiguration. |
vcluster.com/user-data-claim-policy | Not sent. It applies to the local rendering. Set the remote operator's policy on the remote platform. |
See OSImages and Machine config templates for how to define these objects.
The local platform renders SSH keys into the cloud-init when it creates the remote claim.
A key you add later doesn't reach a server that is already provisioned, so you would have to release and reprovision it.
Add vcluster.com/ssh-keys to the node provider, node type, or node pool if you want shell access to these servers.
Passed through to the remote provider​
Every other property of the local node provider and the local claim goes to the remote claim unchanged, and the remote Metal3 provider applies it.
This includes the metal3.vcluster.com/* properties that the remote provider allows on a claim, such as metal3.vcluster.com/server-name to pin a specific server.
Network settings, such as metal3.vcluster.com/network-cidr, come from the remote network environment instead. Sending one fails the claim.
See the Metal3 configuration properties for the full list.
Keep the remote operator's cloud-init​
By default, the user data on the remote claim replaces any user data the remote node provider or node type sets. To keep the remote operator's cloud-init, the remote platform admin sets vcluster.com/user-data-claim-policy: Merge on the remote node provider or node type. The remote platform then adds the local document to its own, and the remote operator's commands run first. See vcluster.com/user-data-claim-policy.
The OSImage still decides whether cloud-init runs at all. A custom image that skips cloud-init doesn't run either document.
Use the node types in a cluster​
Reference the provider in the vcluster.yaml of a cluster, the same as any other node provider.
The vcluster.com/node-type property matches the mirrored node type with or without the provider prefix.
privateNodes:
enabled: true
autoNodes:
- provider: remote-metal3
static:
- name: gpu-pool
quantity: 2
nodeTypeSelector:
- property: vcluster.com/node-type
value: gpu-node
Node claim properties set in vcluster.yaml override the ones on the NodeProvider.
For example, to install a different OSImage for one cluster, set vcluster.com/os-image under properties in its node pool.
Check that a node provisions​
After the cluster requests nodes, list the NodeClaims on the local platform:
kubectl get nodeclaims.management.loft.sh --all-namespaces
A NodeClaim starts in the Pending phase.
The Provisioned condition confirms that the remote server is provisioned.
The Joined condition confirms that the node has joined the cluster.
A healthy NodeClaim reports the Available phase after both succeed.
A Failed phase means provisioning failed. See Troubleshoot.
Rotate or revoke access​
To stop using the remote inventory, delete the clusters that claim its nodes, then delete the node provider on the local platform. Deleting a NodeClaim deletes its remote NodeClaim, which returns the server to the remote pool. The remote platform can also revoke access on its side by deleting the access key. The local platform then can't reach it until you replace the key.
Replace an access key​
The provider reads the Secret once, when the platform starts the provider, and keeps using that key. Updating the Secret alone leaves the old key in use, so a revoked key keeps failing.
Update the
accessKeyvalue in the Secret on the local platform with the new access key.Restart the local platform deployment so the provider reads the Secret again.
Modify the following with your specific values to generate a copyable command:kubectl --namespace vcluster-platform rollout restart deployment/loftkubectl --namespace vcluster-platform rollout status deployment/loftCheck that the provider is available again.
Modify the following with your specific values to generate a copyable command:Check the providerkubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.phase}'
Limitations​
- The remote provider must be a Metal3 provider.
- OSImages backed by an OS image store (
spec.connectorRef) aren't supported. Use image URL and checksum properties instead. - You can't change
nodeProviderorprojectafter you create the provider, because the provider finds its remote NodeClaims through them. Create another provider to target a different remote provider or project. - One provider uses one access key. To consume the inventory of another remote Tenant, create another provider.
Troubleshoot​
Problems on the remote side, such as a server that fails to provision, show up as a failure on the remote NodeClaim. Troubleshoot them with the Metal3 troubleshooting guide on the remote platform.
The provider reports problems as the reason of a condition on the NodeProvider or the NodeClaim:
kubectl get nodeproviders.management.loft.sh remote-metal3 -o jsonpath='{.status.conditions}'
| Reason | Meaning |
|---|---|
RemoteNodeProviderNotFound | The remote node provider doesn't exist, or the access key can't read it. |
UnsupportedRemoteNodeProvider | The remote node provider isn't a Metal3 provider. |
InvalidConfiguration | The Secret has no accessKey key. Or a NodeClaim has no OSImage, references a missing OSImage or one backed by an OS image store, uses a local network environment without platform.vcluster.com/remote-network-environment, or carries a property the remote Metal3 provider refuses on a claim. |
RemoteNodeClaimConflict | A NodeClaim with the same name in the remote project belongs to another claim, for example of another platform that shares the project. |
RemoteNodeClaimDeleted | The remote NodeClaim is being deleted on the remote platform, for example because someone deleted it there. |
A failure on the remote NodeClaim fails the local NodeClaim with the reason and message of the remote claim.