Skip to main content
Version: v5.0 Stable

NVIDIA Infra Controller

Enterprise
Available in these plansFreeDevProdScale
NICo Node Provider

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

The NVIDIA Infra Controller (NICo) driver connects vCluster Platform to an existing NICo installation. It mirrors the site's machine inventory and offers node types that you define from those machines. It provisions bare metal instances and supplies private nodes to clusters through Auto Nodes. NICo manages the hardware, operating system installation, and site networking.

You can provision workloads for the platform itself, for platform Tenants, or both. Platform workloads don't need a Tenant object. A NodeProvider can serve the platform and multiple platform Tenants. You don't need a separate provider for each Tenant.

This guide is for the platform administrator who connects and operates NICo. Complete the setup once, from connecting the provider through preparing an image, then hand off to project users. Project users then follow Provision instances and cluster nodes with NICo to launch instances or supply cluster nodes with Auto Nodes.

The NICo driver is one of the platform's NVIDIA DSX OS integrations. To see how it works with the GPU health templates and the NVIDIA Certified Stacks, see NVIDIA DSX OS with vCluster Platform.

How the integration works​

vCluster Platform resourceNICo resource or operation
NodeProviderConnection to one infrastructure provider organization and site.
NodeTypeA pool of the provider's Machines, selected by label. You define it in the NodeProvider's spec.nico.nodeTypes.
MachineA mirror of a physical machine in the configured NICo site, including its NICo labels and instance type.
TenantA NICo tenant organization with the same name as the Tenant, its provider account relationship, and managed allocations.
Platform-assigned network or instanceUses the NodeProvider's organization as a NICo tenant, in addition to its provider role.
NetworkEnvironmentOne tenant-owned VPC, plus one VPC prefix in FNN mode.
OSImageA reusable image definition, materialized as an operating system in the consuming NICo tenant.
NodeClaimA NICo instance on a Machine that the platform picks from the NodeType's pool, or on a pinned Machine.

NICo uses an organization as an identity boundary. An organization can have both an infrastructure provider and a tenant record. Provider operations manage inventory and allocations. Tenant operations manage VPCs, operating systems, and instances. The platform authenticates as PROVIDER_ADMIN for inventory and allocations, and as TENANT_ADMIN for tenant resources. See NVIDIA's authentication model and tenant management workflow.

Adding a NodeProvider doesn't create a default network. Create a NetworkEnvironment with an explicit assignment before provisioning instances. NICo requires a tenant-owned VPC in both supported network modes.

Prerequisites​

  • A vCluster Platform license that includes the NICo node provider and the features you use, such as instance management or private nodes.
  • Administrator access to vCluster Platform and the NICo REST API configuration.
  • An operational NICo 2.2.0 or later installation with a registered site and discovered machines.
  • The NICo REST API base URL, provider organization, and site UUID.
  • A provider-owned parent IP block for platform networks, tenant address space, and tenant site access, unless you adopt the required resources.
  • A NICo-compatible OS image on a reachable image server or uploaded to the platform, or an existing NICo operating system usable by the target tenant.
  • A project, for platform or Tenant workloads, in which project users create instances or clusters.

Install NICo and configure its hardware discovery, BMC credentials, provisioning services, and site network before connecting the driver. The driver doesn't install NICo or register physical machines. Use the NVIDIA documentation for that setup.

Verify these network paths:

SourceDestinationPurpose
vCluster PlatformNICo REST APIInventory, tenant onboarding, and resource lifecycle operations.
NICo REST APIPlatform /oidc/keys endpointRetrieve the keys that verify platform-issued tokens.
NICo site provisioning servicesOS image server, or the platform for an uploaded imageDownload the installation image.
Provisioned hostsNICo phone-home serviceReport OS initialization when phone home is enabled.
Provisioned hostsCluster control plane or configured VPN endpointJoin and operate as private nodes.

Use trusted TLS certificates and stable endpoint names. For private nodes, also satisfy the node requirements and connectivity requirements.

Choose the network mode​

The NodeProvider's spec.nico.networkMode selects the network model for all its environments and instances. It can't change after creation.

Settingfnnflat
Hardware and siteDPU-backed native networking. The NICo site must have native networking enabled.Zero-DPU hosts or hosts operating in NIC mode, with a configured HostInband network.
NICo VPC typeFNNFLAT
VPC prefixCreated from the tenant's allocated IP block, or adopted.Not created.
Instance networkingOne physical interface attached to the VPC prefix.NICo automatic networking with autoNetwork: true.
Address assignmentVPC prefix configuration.Site HostInband configuration.

fnn is the default. Select flat explicitly for a FLAT site. The driver doesn't support ETHERNET_VIRTUALIZER mode.

FLAT isn't DPU-enforced network isolation

A FLAT VPC provides NICo resource ownership, but NICo doesn't program its data plane. Separate Tenant or VPC records don't create isolated networks on a shared HostInband segment. Configure any required isolation in the physical network. See NVIDIA's FLAT VPC documentation.

Connect the provider​

Complete provider, Tenant, network, and image setup as a platform administrator through the management API. Set these values to match your installation. UUIDs must identify existing resources in NICo.

Modify the following with your specific values to replace on the whole page and generate copyable commands:

Connect kubectl to the management API before running the commands in this guide:

vcluster platform login https://platform.example.com
vcluster platform connect management

Save each manifest to the filename shown above it before running its kubectl apply command.

  1. Configure NICo to trust platform-issued tokens​

    NICo must trust platform-issued JWTs. Add an issuer with these settings:

    NICo issuer settings
    name: vcluster-platform
    origin: custom
    issuer: "https://platform.example.com/oidc"
    jwks: "https://platform.example.com/oidc/keys"
    audiences:
    - "https://nico.example.com"
    claimMappings:
    - orgAttribute: organization
    orgDisplayAttribute: organization
    rolesAttribute: roles

    Follow NVIDIA's authentication documentation to configure the issuer in your NICo installation.

    Use the platform's configured hostname for issuer and the NodeProvider's REST endpoint for audiences, without a trailing slash. Use the dynamic organization and role mappings shown above to read these values from platform-issued tokens. NICo permits only one dynamic organization mapping across its issuers. Check for an existing mapping before adding this issuer. Don't add required token scopes that the platform doesn't issue.

    The platform issues and refreshes short-lived tokens automatically. The NodeProvider doesn't accept a static bearer token or a credentials Secret for this integration. Trusting this issuer allows the platform to act as provider and tenant administrator, so restrict access to its configuration and signing keys.

  2. Create the NodeProvider​

    In Nodes & Providers, create a node provider and select NVIDIA NICo. Alternatively, apply this manifest through the management API:

    Modify the following with your specific values to generate a copyable command:
    nico-provider.yaml
    apiVersion: management.loft.sh/v1
    kind: NodeProvider
    metadata:
    name: nico
    spec:
    nico:
    endpoint: "https://nico.example.com"
    org: vcluster-autonodes
    siteId: 11111111-1111-4111-8111-111111111111
    siteIPBlockID: 22222222-2222-4222-8222-222222222222
    networkMode: fnn
    identity:
    platformIssued:
    enabled: true
    nodeTypes:
    - name: gpu-h100
    machines:
    selector:
    matchLabels:
    nico.vcluster.com/instance-type-id: 33333333-3333-4333-8333-333333333333

    Set networkMode to flat if your hosts use the FLAT model. The endpoint is the REST API's scheme, hostname, and optional port, not a Core gRPC endpoint or a URL containing an organization path. The driver appends /v2/org/{org}/nico.

    siteIPBlockID identifies the provider's parent block, not a tenant IP block or a VPC prefix. The platform uses it to allocate tenant address space and to establish site access for platform and Tenant networks. It doesn't create this parent block.

    Each entry in nodeTypes defines one NodeType. Its machines.selector matches labels on the provider's Machines. The example selects every Machine of one NICo instance type. Replace the instance type UUID with one from your site, or select on any other Machine label. An empty selector matches every Machine of the provider. Without nodeTypes, the provider offers no NodeTypes, and users can only provision by pinning a machine.

    Apply the manifest:

    kubectl apply -f nico-provider.yaml
  3. Verify discovery​

    kubectl get nodeprovider nico -o yaml
    kubectl get machines --show-labels
    kubectl get nodetypes

    Discovery worked when kubectl get machines lists your site's machines. Machine resources mirror NICo's physical inventory. Their names combine the provider name and NICo REST machine UUID. Don't create Machine resources manually to register NICo hardware.

    Each Machine carries the labels of its NICo machine. Keys under vcluster.com and loft.sh aren't copied. A Machine with a NICo instance type also carries it as the nico.vcluster.com/instance-type-id label. You can add labels to a Machine directly, and node type selectors match them too.

    The platform creates one NodeType for each nodeTypes entry. Its name combines the provider name and the entry name, such as nico.gpu-h100. A NodeType's status.capacity counts the matching Machines that are available. The platform doesn't copy CPU, memory, or GPU values from the instance type onto the NodeType. To describe a node's size for scheduling, set resources on the entry. The provisioning task uses this NodeType.

Provider configuration reference​

Field under spec.nicoBehavior
endpointNICo REST API base URL. Required.
orgProvider organization. Defaults to vcluster-autonodes. Immutable.
siteIdSite UUID used for provisioning and discovery. Supply it when creating the provider. Once set, it is immutable.
siteIPBlockIDExisting provider-owned parent IP block for the site. Required when the platform must allocate address space or site access for a network or Tenant.
networkModefnn or flat. Defaults to fnn and is immutable.
nodeTypesThe NodeTypes this provider offers. Each entry has a name, a machines.selector that matches Machine labels, and optional NodeType fields such as properties and resources.
identity.platformIssued.enabledMust be true. Platform-issued identity is the supported authentication method.
insecureSkipTLSVerifyDefaults to false. Use only for isolated development environments with untrusted certificates.

spec.machineAllocationPolicy, outside spec.nico, sets which machines a Tenant's claims can use. A NodeType entry can override it. See Give the Tenant machines.

Choose the workload owner​

Choose one of the following paths before creating a network. A NetworkEnvironment's assignment determines which NICo tenant owns its VPC. An instance belongs to the NICo tenant of its project's Tenant, even when a platform administrator launches it. Platform projects use the NodeProvider's own organization. Networks don't belong to a project. You manage them and lend each one exclusively to a Tenant. Tenants can't create or manage their own networks.

Platform-owned workloads​

For a platform project, create a platform-assigned NetworkEnvironment. The platform then initializes a NICo tenant in the NodeProvider's own organization and establishes its provider account relationship. It does this when infrastructure is first needed, not when the NodeProvider is created.

No Tenant object or organization setting is required. Platform claims use Machines that no Tenant holds. They never use a Machine assigned to a Tenant. Skip Tenant onboarding below and continue with network creation.

Platform-assigned networks aren't available to tenant projects.

Tenant-owned workloads​

Configure an existing platform Tenant as a platform administrator. Each Tenant onboards into the NICo organization with the same name as the Tenant. There's nothing to configure for the mapping, and it can't change, because Tenant names are immutable.

The Tenant name can't match a provider organization. The API refuses a Tenant named after a provider's org, and a NodeProvider whose org matches an existing Tenant name.

Modify the following with your specific values to replace on the whole page and generate copyable commands:
  1. Grant NodeTypes to the Tenant​

    Merge this configuration into the Tenant, preserving its existing users, permissions, and other settings:

    Tenant configuration
    apiVersion: management.loft.sh/v1
    kind: Tenant
    metadata:
    name: research
    spec:
    nodeTypes:
    enabled: true
    allow:
    byName:
    - nico.gpu-h100
    quota:
    byType:
    "nico.gpu-h100": "2"
    osImages:
    enabled: true
    allow:
    byLabels:
    example.com/image-catalog: nico

    Onboarding starts automatically once the Tenant is granted a NICo NodeType or assigned a NICo network. If a NICo tenant with the Tenant's name already exists, the platform links it. Otherwise, NICo creates it. Either way, NICo must authorize the platform's tokens for that organization. Linking doesn't transfer the tenant's existing identity-provider credentials. If a static issuer mapping in NICo reserves the same organization name, NICo rejects the platform's dynamic mapping. Resolve that conflict in NICo's issuer configuration first.

    The platform finds the Tenant's NodeProvider from its granted NICo NodeTypes and assigned NICo networks. Onboarding fails if those resolve to more than one NICo NodeProvider. This is an integration restriction, not a requirement to dedicate a NodeProvider to each Tenant.

    The platform enforces quota.byType itself. It doesn't create reservations in NICo. The quota caps only on-demand placements. Claims on Machines assigned to the Tenant, and claims pinned to a Machine, don't count. A claim over quota waits with the reason TenantNodeTypeQuotaExceeded.

    The osImages permission makes images with the example catalog label visible to the Tenant. Additional capabilities, such as control plane clusters, templates, and custom SSH keys, must be granted for the workflows the Tenant will use.

  2. Give the tenant machines​

    A NodeType's machineAllocationPolicy decides which Machines a Tenant's claims can use:

    PolicyTenant claims use
    AssignedOnly Machines owned by or exclusively assigned to the Tenant. This is the default for NICo.
    OnDemandOnly unassigned Machines, which no Tenant owns or holds. Reserved Machines are never used.

    Set the policy on the NodeProvider's spec.machineAllocationPolicy, or on a nodeTypes entry to override it.

    With Assigned, assign Machines to the Tenant before it provisions. In Bare Metal > Machines, open a Machine's actions and click Assign to tenant. Alternatively, label the Machine with the Tenant's name:

    Modify the following with your specific values to generate a copyable command:
    kubectl label machine nico.machine-uuid tenant.vcluster.com/exclusive-to=research

    Without an assigned Machine, a Tenant claim on an Assigned NodeType stays pending, even when unassigned Machines are free.

  3. Allocate address space for the Tenant​

    FNN and FLAT need different allocations, so follow the paragraphs for your provider's network mode.

    For FNN, request a tenant IP block from the provider's parent block:

    Modify the following with your specific values to generate a copyable command:
    kubectl annotate tenant research \
    nico.vcluster.com/ip-block-prefix-length="20"

    The parent block must have enough free space for the requested prefix. The tenant block must, in turn, fit all VPC prefixes you plan to create for that Tenant. For example, a /20 tenant block can contain multiple /24 VPC prefixes.

    To use an existing NICo tenant IP block instead, set nico.vcluster.com/ip-block-id to its UUID. The block must belong to the linked NICo tenant and be ready. Use either the ID annotation or the prefix-length annotation, not both.

    FLAT doesn't need a tenant IP block for instance addresses. Creating a FLAT VPC still requires an allocation at its site. The platform creates that site access automatically. It allocates a /32 block from the provider's siteIPBlockID unless the Tenant already holds another allocation at the site. Without siteIPBlockID, onboarding reports SiteAccessUnavailable.

    Changing nico.vcluster.com/ip-block-prefix-length later resizes the platform's block, and removing the annotation releases it.

  4. Wait for onboarding​

    kubectl get tenant research -o yaml
    kubectl wait tenant/research --for=condition=NICoOnboarded --timeout=10m

    Check status.nico for the organization, provider, tenant ID, account ID, IP block ID, and site access allocation. Three conditions report progress:

    ConditionReports
    NICoNodeProviderFoundWhether the platform resolved exactly one NICo NodeProvider for the Tenant. False when a referenced provider is missing or isn't a NICo provider, or when there's more than one.
    NICoOnboardedTrue once the Tenant's NICo organization exists. False with a reason while authentication, account linking, site access, or IP block setup is pending or failing.
    NICoSiteCredentialsAvailableWhether the platform could load the Tenant's credentials for the NICo endpoint.

    The platform creates the tenant account relationship when needed. Existing account relationships and externally managed allocations stay external resources.

    The platform places every instance on a specific machine, whether a claim names a NodeType or a Machine. So every claim needs NICo's targeted-instance-creation capability for the tenant account. The platform requests that capability during onboarding, on new and existing accounts. The request is best effort. Successful onboarding alone doesn't confirm that targeted provisioning is enabled for the site, so verify it in NICo.

Create a network​

Create one NetworkEnvironment for the workload owner you chose. Select the corresponding tab and save its manifest as nico-network.yaml. Use a different name for each environment if you need both platform and Tenant networks.

Modify the following with your specific values to replace on the whole page and generate copyable commands:
Choose the assignment and network settings before creating the environment

The assignment, provider, and network properties can't change after creation. You can still edit the display name. To change the others, create a replacement environment and reprovision its workloads. If you plan to adopt an existing VPC, set the network properties before applying the manifest.

nico-network.yaml
apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: nico-network
spec:
providerRef: nico
properties: {}

A network without an assignment label belongs to the platform.

Both manifests work with either provider mode:

  • FNN: The driver creates a VPC and a VPC prefix. The prefix length defaults to /24. Tenant networks use their Tenant's IP block. Platform networks allocate a block from siteIPBlockID for that environment.
  • FLAT: The driver creates a FLAT VPC without a VPC prefix. For a new platform network, it reserves a /32 block from siteIPBlockID to establish site access. This reservation isn't the host's address pool. Host addresses come from NICo's HostInband network.

Apply the manifest for your chosen assignment and wait for the environment's Ready condition.

kubectl apply -f nico-network.yaml
kubectl get networkenvironments.management.loft.sh nico-network -o yaml
kubectl wait networkenvironments.management.loft.sh/nico-network --for=condition=Ready --timeout=10m

Give this network name to project users who provision on it. Platform-assigned networks are for platform projects. Tenant-assigned networks are for that Tenant's workloads. Set providerRef to the NICo NodeProvider. Only that provider's node claims can use the network, and the API refuses a claim from another provider. See Network environments for the assignment labels, immutability, and how a machine selects a network.

Network properties​

Set these in NetworkEnvironment.spec.properties before creating the environment:

PropertyBehavior
nico.vcluster.com/vpc-prefix-lengthFNN prefix length, from 8 through 31. Defaults to 24. The source block must have sufficient free space.
nico.vcluster.com/vpc-cidrsOptional comma-separated CIDRs for a new FNN VPC, with no spaces after the commas. Invalid for FLAT and incompatible with vpc-id. Doesn't replace tenant IP allocation.
nico.vcluster.com/vpc-idAdopt an existing VPC in the assigned NICo tenant. Its network mode must match the provider.
nico.vcluster.com/vpc-prefix-idAdopt an existing FNN VPC prefix. It must belong to the selected VPC.

To adopt an existing FNN network, replace spec.properties: {} with:

properties:
nico.vcluster.com/vpc-id: <vpc-uuid>
nico.vcluster.com/vpc-prefix-id: <vpc-prefix-uuid>

For FLAT, supply only vpc-id. For FNN, adopting only the VPC still requires address space to create its prefix. Tenant-owned FNN networks require the Tenant's IP block configuration even when adopting a prefix.

Use properties to adopt resources

Don't prepopulate nico.vcluster.com/vpc-id or nico.vcluster.com/vpc-prefix-id in metadata annotations. The controller writes those annotations itself to track what it manages. Supplying IDs as properties records adoption, so the platform doesn't delete the adopted resources during teardown.

Prepare an operating system image​

An OSImage lets multiple claims reference one image definition. Use an image that meets NICo's boot requirements and, for cluster nodes, the private node requirements. The image must support the instance user data used by the platform.

Modify the following with your specific values to replace on the whole page and generate copyable commands:
Imaging overwrites the selected disk

Choose the target disk and root filesystem identifier for your actual hardware and image. Don't copy an example device name without verifying it. See NVIDIA's image-based operating system guidance.

Modify the following with your specific values to generate a copyable command:
nico-os-image.yaml
apiVersion: management.loft.sh/v1
kind: OSImage
metadata:
name: ubuntu-nico
labels:
example.com/image-catalog: nico
spec:
displayName: Ubuntu for NICo
properties:
machines.vcluster.com/os-image-type: nico
nico.vcluster.com/os-image-url: "https://images.example.com/ubuntu-nico.img"
nico.vcluster.com/os-image-sha: "REPLACE_WITH_IMAGE_SHA"
nico.vcluster.com/os-image-disk: "/dev/nvme0n1"
nico.vcluster.com/os-image-rootfs-label: "cloudimg-rootfs"

Replace the image URL, digest, disk, and filesystem label with values for your image. The catalog label matches the Tenant image permission in the earlier example. Platform users also need permission to use the OSImage.

kubectl apply -f nico-os-image.yaml
OSImage propertyPurpose
machines.vcluster.com/os-image-typeSet to nico for image filtering in the UI.
nico.vcluster.com/os-image-urlDownload URL for an image-based NICo operating system. Required unless the image was uploaded to the platform, which supplies the URL and checksum itself.
nico.vcluster.com/os-image-shaImage checksum, as a hexadecimal digest with no prefix or separators. Required unless the image was uploaded to the platform. The driver passes it to NICo unchanged.
nico.vcluster.com/os-image-diskWhole-disk target for installation. If omitted, NICo selects the disk.
nico.vcluster.com/os-image-rootfs-idRoot filesystem UUID. Use an identifier that matches the image.
nico.vcluster.com/os-image-rootfs-labelRoot filesystem label, as an alternative to its UUID.
nico.vcluster.com/os-image-idReference an existing NICo operating system instead of creating one from an image URL.

NICo image-based operating systems require exactly one of the root filesystem UUID and label. The platform checks these settings when a claim that boots from the OSImage is created or changes its image. It rejects the claim if the image URL is missing, the checksum isn't hexadecimal, or the root filesystem UUID and label aren't set exactly once. An uploaded image supplies its own URL and checksum, so it needs only the root filesystem setting. Don't set os-image-url, os-image-sha, or os-image-id on an uploaded image. The platform rejects an uploaded OSImage that carries them. The check is skipped when nico.vcluster.com/os-image-id is set on the image. Refer to the NICo operating system API for image field requirements.

For an image-based definition, the driver creates or finds vcluster-<OSImage name> in the consuming NICo tenant at first use. It waits for that operating system to become ready. It enables user data overrides so each instance can receive its bootstrap configuration. Use a new OSImage name for a new image version. Editing the image URL doesn't update an existing NICo operating system.

Use an existing NICo operating system​

Set nico.vcluster.com/os-image-id in the OSImage properties, and reference that OSImage with vcluster.com/os-image. The driver reads the ID only from the OSImage. It ignores the property on a node provider, node type, or network environment, and a NodeClaim that sets it is rejected. An instance without an OSImage reference fails with the reason InvalidConfiguration. The existing operating system must be usable by the target NICo tenant at the configured site. Verify its readiness, image settings, and allowOverride: true in NICo. The direct ID path doesn't configure those settings for you.

Phone home, user data, and SSH​

The driver enables NICo phone home by default for newly created operating systems and instances. The host must reach NICo's phone-home service for the NICo provisioning workflow to finish. Set the workload property nico.vcluster.com/phone-home-enabled: "false" only if your site intentionally doesn't use phone home. This setting doesn't remove the requirement for a private node to join its cluster.

The platform combines its bootstrap configuration with the supported user data templates and SSH key settings. For cluster nodes, the bootstrap configuration includes the join command. For standalone instances, users reference an SSHKey with the vcluster.com/ssh-keys workload property. Selecting an image alone doesn't grant SSH access. Tenant users need permission to use the key or create their own SSH keys.

Hand off to project users​

Project users provision with Provision instances and cluster nodes with NICo. Before they start, give them these names and confirm their access:

  • The NodeProvider name, such as nico, and the NodeType name, such as nico.gpu-h100.
  • A ready NetworkEnvironment lent to their Tenant, or a platform network for a platform project.
  • An OSImage they have permission to use.
  • For standalone instances, an SSHKey they can use, or permission to create their own.
  • A project in that Tenant whose allowedNodeTypes permits the NodeType and whose allowedNetworkEnvironments allowlist doesn't exclude the network.
  • For a Tenant on an Assigned NodeType, at least one Machine assigned to the Tenant.

Cleanup and resource ownership​

Delete workloads before deleting their networks, and delete networks before removing a Tenant or NodeProvider. Keep the provider connection and issuer trust available until cleanup completes. Deleting a Kubernetes resource can start a destructive, asynchronous NICo operation.

ResourceCleanup behavior
NodeClaimDeletes its NICo instance and waits for deletion. The Machine shows the Deprovisioning phase until NICo reports it ready again. NICo controls hardware cleanup and return to service.
Platform-created VPC and prefixDeleted with the NetworkEnvironment, prefix first. A platform network's managed IP block allocation is released afterward.
VPC or prefix supplied through adoption propertiesRetained in NICo. Any additional resources created by the platform are still cleaned up.
Tenant site access allocation and carved IP blockReleased during Tenant cleanup. Dependencies or active use can block release.
Account relationship created for a platform TenantDeleted during Tenant cleanup only if no allocations remain. Retained if external allocations remain.
Existing tenant IP block, account relationship, or external allocationNot deleted as a platform-owned resource.
NICo tenant organizationNot deleted by the integration.
NICo operating system created from an OSImageRetained after instance deletion. Deleting the OSImage or the NodeProvider deletes the vcluster-<OSImage name> operating system once no NodeClaim references it.
NICo operating system referenced by os-image-idNot deleted by the integration.

For Auto Nodes, remove the pool or cluster first so it stops maintaining its NodeClaims. Then wait for those instances to deprovision before removing the environment. Tenant cleanup waits while associated NetworkEnvironments remain. Don't bypass cleanup by removing finalizers or manually deleting stored NICo IDs.

To take a Tenant off NICo without deleting it, remove its NICo NodeType grants and NICo networks. The platform then releases the Tenant's platform-owned NICo resources and clears status.nico. Moving a Tenant to a different NICo NodeProvider directly isn't supported. Onboarding reports NodeProviderChanged. Deleting a NodeProvider waits while any Tenant still uses it.

Troubleshoot provisioning​

Start with the management resources:

Modify the following with your specific values to replace on the whole page and generate copyable commands:
kubectl get nodeprovider nico -o yaml
kubectl get tenant research -o yaml
kubectl get networkenvironments.management.loft.sh nico-network -o yaml
kubectl get nodeclaims -n p-default -o yaml

Run these checks as a platform administrator. Skip the Tenant command for platform-owned workloads. Set the project's management namespace, such as p-default, for the NodeClaim command. For server logs, use the Kubernetes context hosting the platform and your actual deployment namespace. Correlate controller messages with NICo REST and site logs using the recorded instance, VPC, or prefix ID.

SymptomChecks and next action
Provider rejected for licensingVerify that the platform license includes the NICo node provider and machine management.
NICo returns 401 or 403Check issuer, audience, role and organization mappings, JWKS reachability, TLS, and clock synchronization. Check NICo's REST API logs for the rejected claim.
No NodeTypesCheck spec.nico.nodeTypes on the NodeProvider. The platform creates NodeTypes only from those entries.
NodeTypes exist but there is no capacityCheck the NodeType's status.capacity, its machines.selector, and the Machines' labels and phases. For a Tenant, check Machine assignments and machineAllocationPolicy, and for OnDemand types, quota.byType.
Tenant doesn't onboardCheck that the Tenant has a granted NICo NodeType or an assigned NICo network. Then check NICoNodeProviderFound and NICoOnboarded. The Tenant name can't equal a provider org.
Onboarding reports multiple providersGrant NodeTypes and assign networks from only one NICo NodeProvider to that Tenant.
Network stays pendingCheck its assignment and conditions, Tenant onboarding, site access, and NICo VPC status. A Tenant network waits with the reason WaitingForTenantOnboarding until the Tenant is onboarded. For FNN, also check its IP block and prefix.
FLAT network can't provisionConfigure a valid siteIPBlockID with space for the site access allocation. FLAT still needs a VPC and allocation-based site access.
Visible network missing from instance selectorCheck that one of the project's allowed node types, or a Machine assigned to the project's Tenant, uses the network's provider. Then check the project's Tenant, assignment, and Project.spec.allowedNetworkEnvironments. Network readiness is checked during provisioning, not used to filter this selector.
Claim rejected with os image "<name>" must set ...The OSImage is missing a setting NICo needs to boot it. Set the property the message names on the OSImage. See Prepare an operating system image.
Instance remains pending after bootCheck OS readiness, image reachability, that the root filesystem UUID or label matches the image, user data override permission, and the NICo phone-home path.
Existing OS rejects user dataVerify allowOverride: true. This is especially important when selecting a NICo operating system ID directly.
NICo instance is ready but the node has not joinedCheck bootstrap logs on the host, private node prerequisites, control plane or VPN connectivity, DNS, and the cluster's node status.
Launch fails at NICoVerify that the tenant account has the targeted-instance-creation capability. For a pinned claim, also verify the Machine name and provider, and that it's available and unclaimed. A Machine in the Deprovisioning phase becomes available once NICo finishes cleaning it.
Claim fails with InsufficientCapacityNo matching Machine became available within the provisioning wait. Check the capacity row above.
A delete remains pendingCheck remaining instances, prefixes, allocations, and NICo resource status. Restore provider connectivity rather than removing finalizers.

Limitations​

  • Tenant onboarding supports one NICo NodeProvider per Tenant, counting both granted NodeTypes and assigned networks. Multiple Tenants can share that provider.
  • A Tenant's NICo organization is always the Tenant name. To link an existing NICo tenant, give the platform Tenant the same name.
  • Network assignment and configuration are immutable. The integration doesn't move VPCs or running instances between tenants.
  • One VPC per NetworkEnvironment. FNN provisioning attaches one physical interface to one VPC prefix.
  • No configuration for additional VPC interfaces, InfiniBand partitions, Spectrum-X partitions, NVLink partitions, or VPC peering through this driver.
  • No ETHERNET_VIRTUALIZER support. FLAT isolation depends on the site's physical network.
  • Platform-issued identity only. No static-token or external credentials configuration on the NodeProvider.
  • No automatic parent IP block creation. Supply an existing block with siteIPBlockID.
  • Machine power state comes from NICo host BMC health alerts and can lag by a few minutes. The platform can't power machines on or off through NICo. Use NICo for hardware power management.
  • Changes to an OSImage don't reimage running instances. Use a new image definition and replace the affected instances.
  • NICo operating systems referenced directly by os-image-id require separate cleanup in NICo.