Skip to main content
Version: v5.0 Stable

Metal3 network integration

The Metal3 provider provisions bare metal servers, but on its own it doesn't isolate the network those servers run on. Network integration determines how strongly each tenant's traffic is separated on the metal fabric and how the platform manages addresses for provisioned servers. The right choice depends on how much you trust the tenants that share your hardware. A single trusted team can run on a flat provisioning network, while untrusted or external tenants need isolation enforced in the physical network.

The metal3.neutronEnabled NodeProvider field enables Neutron integration. Within Neutron mode, the metal3.vcluster.com/network-provider property selects an optional fabric provider on a network environment (a cluster-scoped NetworkEnvironment resource). Set the property to netris for Netris or aviz for Aviz, or leave it empty for a flat network. Netris and Aviz are network integrations for the Metal3 provider, not separate node providers.

Requires a network environment

Neutron, Netris, and Aviz require a network environment. They support instances provisioned independently or used as private nodes for a clusterClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Cluster. For untrusted tenants, use private nodes and give each network environment its own Netris server cluster or Aviz tenant.

Choose a network integration mode​

Compare the four network integration modes before configuring the provider.

ModeIsolation levelIPAMWhen to use
No network integration (default)None. The provisioning network is the tenant network.Optional. Platform IPAM allocates addresses when a network CIDR is configured.A single trusted tenant, or a lab where the provisioning network is the only network.
Neutron (flat network)None at the fabric. The network stays flat and shared. Ironic owns the port lifecycle.Platform IPAM, per tenant network.You want Ironic to drive ports through its neutron interface, but don't need hardware isolation.
NetrisHardware. Environment-level L2 isolation through Netris.Platform IPAM. The subnet is derived from the Netris server cluster unless a CIDR is pinned.Untrusted, external, or regulated tenants that must be isolated at the physical network layer.
AvizDepends on the ONES fabric configuration. The platform allocates servers to an Aviz tenant.Platform IPAM. Requires an explicit network CIDR.Integrating Metal3 provisioning with an existing Aviz ONES fabric.

The two flat modes keep each cluster's control plane and API separate and manage their addresses independently. Every server still shares the same layer 2 network, so these modes don't provide network isolation. Netris provides hardware network isolation. Aviz connects server provisioning to an existing ONES fabric. Its segmentation depends on that fabric's configuration.

No network integration (default)​

With metal3.neutronEnabled set to false, the provisioning network is also the tenant network. When metal3.vcluster.com/network-cidr is configured, the platform's built-in IPAM allocates an address on that shared network. It writes the address, gateway, and DNS servers directly onto the BareMetalHost as DHCP annotations.

Without a network CIDR, the platform doesn't allocate an address or write managed DHCP annotations. The bundled DHCP server can't serve a PXE client unless another system supplies the required IP address annotation on its BareMetalHost. Alternatively, use external DHCP. You can provide custom network data to configure the installed operating system, but it doesn't replace provisioning DHCP.

Addresses come from the existing network properties documented on the Metal3 provider page:

  • metal3.vcluster.com/network-cidr sets the gateway and subnet to allocate from.
  • metal3.vcluster.com/network-ip-range restricts allocation to explicit ranges.
  • metal3.vcluster.com/network-data supplies a complete custom network-data document.

Because all servers share one network, use this mode only where the tenants sharing the hardware are trusted.

Neutron (flat network)​

In this mode Ironic drives its neutron network interface instead of writing addresses straight onto the BareMetalHost. The platform allocates the address for each port and Ironic manages the port lifecycle across provisioning, cleaning, and inspection.

Neutron doesn't isolate the network

Neutron changes how ports are managed, not how the network is segmented. The underlying network is still flat and shared across tenants. For isolation enforced in hardware, use Netris. Aviz segmentation depends on your ONES fabric configuration.

Enable Neutron and supply the externally provisioned network names that Ironic requests. The platform derives the chart's neutron.enabled value from metal3.neutronEnabled, but it doesn't set the network names. The chart refuses to install with Neutron enabled until neutron.networks.provisioning is set in helmValues.

Flat Neutron also requires metal3.vcluster.com/network-cidr in the effective properties of every NodeClaim. The platform can't allocate the tenant port without this CIDR. The following example defines the CIDR on a node type and binds its NodeClaims to a network environment assigned to their Tenant:

apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: metal3-provider
spec:
metal3:
clusterRef:
cluster: bare-metal-cluster
namespace: metal3-system
# Enables the platform network controller and the chart's Neutron support.
neutronEnabled: true
deploy:
metal3:
enabled: true
helmValues: |
neutron:
networks:
provisioning: provisioning
cleaning: cleaning
inspection: inspection
nodeTypes:
- name: compute
properties:
# Flat Neutron can't derive a subnet, so configure one explicitly.
metal3.vcluster.com/network-cidr: 192.168.100.1/24
# Select this NetworkEnvironment by name. A Tenant's claim needs one lent to its Tenant.
vcluster.com/network-environment: tenant-a-network
---
apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: tenant-a-network
labels:
# Lend the environment to a Tenant. Omit the label to keep it for the platform.
tenant.vcluster.com/exclusive-to: tenant-a
spec:
providerRef: metal3-provider

When cleaning or inspection are left empty, they fall back to the provisioning network. Servers provisioned in this mode select their network through a network environment, so a network environment is required.

Netris (hardware network isolation)​

Netris provides network isolation in hardware. On provisioning, the platform attaches each server in a network environment to that environment's Netris server cluster. Give each tenant network environment its own server cluster to isolate tenant traffic at the physical network layer.

Requires Neutron mode. Netris builds on Neutron and adds environment-level L2 isolation, so it also requires a network environment.

Provide Netris API credentials​

Create a Secret on the management cluster that contains the Netris API url, username, and password. Reference its management-plane namespace on the NodeProvider. This namespace is independent of the namespace in metal3.clusterRef, which identifies where Metal3 resources run on the connected control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both.Related: Cluster, Control plane cluster, Cluster.

Add the following Netris settings to the provider configuration from the preceding Neutron section. The fragment configures the credentials and binds the node type to the network environment shown later on this page:

apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: metal3-provider
spec:
metal3:
neutronEnabled: true
netris:
secretRef:
name: netris-credentials
namespace: vcluster-platform
nodeTypes:
- name: compute
properties:
# Select the NetworkEnvironment shown later on this page by name.
vcluster.com/network-environment: tenant-a-network

Select and configure the Netris provider​

The Netris provider is selected and configured through properties on the network environment. Set them under spec.properties.

PropertyRequiredDescription
metal3.vcluster.com/network-providerYesSet to netris to select the Netris network provider.
netris.vcluster.com/server-clusterYesName of the Netris server cluster to attach each server to. Also the subnet source when no CIDR is pinned.
netris.vcluster.com/server-cluster-templateConditionalName of a Netris server-cluster template. The platform creates the server cluster from it when the cluster doesn't already exist.
netris.vcluster.com/server-cluster-template-jsonConditionalJSON array used as the template's vnets array when a template of that name doesn't already exist.
netris.vcluster.com/adminConditionalNetris tenant that owns the created server cluster. Required when the platform creates the server cluster.
netris.vcluster.com/siteConditionalNetris site the created server cluster belongs to. Required when the platform creates the server cluster.
netris.vcluster.com/vpcNoName of an existing VPC for the created server cluster. When empty, Netris creates a VPC implicitly as part of server-cluster creation.

When the named server cluster already exists at the initial lookup, the platform leaves it unmanaged. In that case, server-cluster-template, admin, and site are unneeded.

When the platform creates a server cluster or template, it records what it created on the network environment so it can remove exactly those resources on teardown:

  • netris.vcluster.com/managed-server-cluster
  • netris.vcluster.com/managed-server-cluster-template

Only the platform writes these annotations. The management API refuses any create or update that sets, changes, or removes one, for every caller, platform admins included. It returns Forbidden with annotation <key> is set by the platform and cannot be changed.

To re-create a network environment from exported YAML, or with Clone in the platform UI, delete both annotations, and any other platform annotation, from the YAML first. If the server cluster or template still exists in Netris, the re-created environment adopts it without marking it managed. Deleting that environment later leaves the adopted resource in place, so remove it in Netris yourself when you no longer need it.

Don't share Netris resources between network environments

Give each network environment its own server cluster and template. Don't reference the same server cluster or template from multiple network environments.

The platform deletes only the server cluster and template recorded as managed by a network environment. It doesn't delete a VPC specified by netris.vcluster.com/vpc. If other network environments reference a managed server cluster or template, deleting its creator can remove the shared resource or fail while the resource is still in use.

The subnet for each server is derived from the Netris server cluster's network unless metal3.vcluster.com/network-cidr pins it, in which case the pinned CIDR always wins.

Override the Netris server name​

By default a server is registered in Netris under its BareMetalHost name. To register it under a different name, set the following annotation on the BareMetalHost:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: server-01
annotations:
netris.vcluster.com/server-name: rack1-server-01

Example​

This network environment selects the Netris provider and points at a server cluster that the platform creates from a template if it doesn't already exist:

apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: tenant-a-network
labels:
# The assignment, providerRef, and properties can't be changed after creation.
tenant.vcluster.com/exclusive-to: tenant-a
spec:
providerRef: metal3-provider
properties:
metal3.vcluster.com/network-provider: netris
netris.vcluster.com/server-cluster: tenant-a
netris.vcluster.com/server-cluster-template: default-template
netris.vcluster.com/admin: tenant-a-admin
netris.vcluster.com/site: datacenter-1

Servers select this network environment through the vcluster.com/network-environment property, described on the Metal3 provider page. Any server provisioned into it is attached to the tenant-a Netris server cluster. The server cluster forms the hardware-backed network boundary for this environment.

Aviz fabric integration​

Aviz ONES manages the physical network fabric. The Metal3 integration allocates each server to the Aviz tenant selected by its network environment.

Network segmentation depends on your ONES fabric configuration. Successful tenant creation doesn't verify traffic isolation. Validate the required separation across server-to-server, front-end, and storage networks before deploying tenant workloads.

A server joins the Aviz tenant the same way whether it runs as a standalone bare metal instance or as a private node in a cluster. The network environment decides the Aviz tenant, not the workload on the server.

An Aviz tenant is an object in Aviz ONES that groups the servers attached to it. It's unrelated to the platform's Tenant resource. A platform Tenant can use an Aviz-backed network environment when that environment is lent to it, but the platform doesn't map the two by name.

Aviz requires Neutron mode and a network environment, the same as Netris.

Prerequisites​

As a platform admin, configure the Metal3 provider after your network administrator prepares the ONES fabric. Follow the Aviz fabric deployment guides for the ONES-side setup.

  • An existing Aviz ONES fabric. The platform looks up the fabric by name and doesn't create it.
  • Aviz API credentials that permit tenant management and server allocation.
  • Each server known to Aviz ONES under the name the platform registers it with. By default this is its BareMetalHost name. See Override the Aviz server name.
  • A platform license that includes the aviz feature. Without it, provisioning the network environment fails with the reason FeatureNotLicensed.
  • Neutron enabled on the NodeProvider, with neutron.networks.provisioning set in helmValues. See Neutron (flat network).

Provide Aviz API credentials​

Create a Secret on the management cluster that contains the Aviz ONES API url, username, and password. The url is the root of the ONES instance, for example https://ones.example.com:8089.

apiVersion: v1
kind: Secret
metadata:
name: aviz-credentials
namespace: vcluster-platform
type: Opaque
stringData:
url: https://ones.example.com:8089
username: admin
password: <password>

Reference the Secret through spec.metal3.aviz.secretRef on the NodeProvider. As with Netris, this namespace is independent of the namespace in metal3.clusterRef. Add the following settings to the provider configuration from the Neutron section:

apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: metal3-provider
spec:
metal3:
neutronEnabled: true
aviz:
secretRef:
name: aviz-credentials
namespace: vcluster-platform
nodeTypes:
- name: compute
properties:
# Select the NetworkEnvironment shown later on this page by name.
vcluster.com/network-environment: tenant-a-network

The platform reads the credentials from the NodeProvider that the network environment's providerRef names. A network environment without a providerRef can't use Aviz.

Select and configure the Aviz provider​

Set the following properties under spec.properties on the network environment.

PropertyRequiredDescription
metal3.vcluster.com/network-providerYesSet to aviz to select the Aviz network provider.
aviz.vcluster.com/fabricYesName of the existing Aviz ONES fabric.
aviz.vcluster.com/tenantYesName of the Aviz tenant to attach each server to. 1 to 12 characters. It starts with a letter, followed by letters, digits, underscores, or hyphens.
metal3.vcluster.com/network-cidrYesGateway and subnet that platform IPAM allocates from. Aviz doesn't supply subnet configuration, so you must set it. You can also set it on the node type or NodeProvider.

Configuration errors appear in two places:

  • On the network environment, as the reason on its Provisioned condition. A missing or invalid Aviz property reports InvalidConfiguration, a fabric that doesn't exist reports FabricNotFound, and a license without the aviz feature reports FeatureNotLicensed.
  • On each NodeClaim, which fails with InvalidConfiguration when no network CIDR is set.

Aviz tenant lifecycle​

When the network environment is provisioned, the platform checks the named Aviz tenant:

  • If the tenant doesn't exist, the platform creates it as an unshared tenant. It records the tenant in the aviz.vcluster.com/managed-tenant annotation on the network environment. Don't set this annotation by hand.
  • If the tenant already exists at the initial lookup, the platform leaves it unmanaged.

Before deleting a network environment, release its instances or remove the cluster nodes that use it. The platform blocks deletion while any NodeClaims remain bound to the environment, even if their servers are already detached. A blocked network environment reports the reason NodeClaimsBound.

When you delete the network environment, the platform deletes only a tenant recorded as managed. Aviz ONES refuses to delete a tenant while any server is still allocated to it. Any remaining allocations must be released before deletion can finish. Deletion is asynchronous and can take several minutes. The platform retries until the tenant is gone.

Don't share Aviz tenants between network environments

Give each network environment its own Aviz tenant. Don't reference the same Aviz tenant from multiple network environments.

The platform deletes tenants recorded as managed when their network environment is deleted. Other network environments referencing the same tenant don't prevent its deletion once all servers are detached. Once the tenant is gone, servers in those network environments can't attach.

Server attachment​

During provisioning and cleaning, Ironic uses the networks specified in the Neutron configuration. The platform attaches the server to the Aviz tenant for its network environment. When the NodeClaim is released, the platform detaches the server so it can be provisioned for another tenant.

The platform requests whole-server, unshared allocations. It doesn't configure per-GPU allocations between Aviz tenants.

Attachment and detachment are asynchronous and can take several minutes while Aviz ONES applies the fabric configuration. The platform waits for Aviz ONES to report the tenant's configuration as applied before treating a server as attached. Detachment is complete once the server is no longer allocated to the tenant.

Override the Aviz server name​

By default a server is attached in Aviz ONES under its BareMetalHost name. If Aviz ONES knows the server by a different name, set the following annotation on the BareMetalHost:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: server-01
annotations:
aviz.vcluster.com/server-name: rack1-server-01

The platform reads the annotation on every attach and detach. Don't change it while the server is attached, because detachment then targets the new name and leaves the server attached under the old one.

Example​

This network environment selects the Aviz provider. The platform creates the tenanta Aviz tenant on the datacenter-1 fabric if it doesn't already exist:

apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: tenant-a-network
labels:
# The assignment, providerRef, and properties can't be changed after creation.
tenant.vcluster.com/exclusive-to: tenant-a
spec:
providerRef: metal3-provider
properties:
metal3.vcluster.com/network-provider: aviz
aviz.vcluster.com/fabric: datacenter-1
aviz.vcluster.com/tenant: tenanta
# Aviz doesn't supply a subnet, so the CIDR is required.
metal3.vcluster.com/network-cidr: 10.10.0.1/24

Servers select this network environment through the vcluster.com/network-environment property. Any server provisioned into it is attached to the tenanta Aviz tenant.

Verify the integration​

  1. Check that the network environment's Provisioned condition is True:

    kubectl get networkenvironment tenant-a-network -o jsonpath='{.status.conditions[?(@.type=="Provisioned")]}'

    A False status carries a reason and the error message. The configuration reasons are listed under Select and configure the Aviz provider. Other failures, such as wrong Aviz credentials, report FailedToProvision and retry.

  2. Provision a server using the compute node type configured above. Follow Independently provisioned instances for a standalone instance, or Cluster nodes for a private node. Use metal3-provider as the provider and compute as the node type in those guides.

  3. After the NodeClaim is ready, check in Aviz ONES that the server is allocated to the tenanta tenant. Check that the tenant's configuration has been applied. Allow several minutes for the configuration push.

  4. Before deploying tenant workloads, verify that server-to-server, front-end, and storage traffic follow your required isolation policy. Successful allocation alone doesn't verify traffic isolation.