Skip to main content
Version: main ๐Ÿšง

Troubleshooting

This guide covers how to diagnose and resolve common Metal3 provisioning failures. Start by identifying the state your server is in using Bare metal server lifecycle, then jump to the matching section below.

Set the project, instance, BareMetalHost namespace, and BareMetalHost name once. Every interpolated command on this page updates automatically.

Modify the following with your specific values to replace on the whole page and generate copyable commands:

This page uses up to three Kubernetes contexts.

ContextUse it for
<management-context>vCluster Platform resources, including NodeClaim, SSHKey, and the ipam-default ConfigMap.
<control-plane-context>Metal3 resources, including BareMetalHost, BMC Secrets, webhooks, and component logs.
<cluster-context>Kubernetes nodes and workloads in a cluster.

The control plane context is the cluster referenced by NodeProvider.spec.metal3.clusterRef.cluster. Replace each placeholder with a name from kubectl config get-contexts.

server stuck atregistering1 Check BMO logskubectl describe bmh ยท look for last error event2 BMC address reachable?ping ยท curl from a node in the control plane cluster3 Check firewall rulesport 623 (IPMI) ยท port 443 (Redfish) must reach BMC4 Verify credentials SecretsecretName and namespace in BareMetalHost specFix and return BareMetalHostset status back to availableserver stuck atinspecting1 Check BMC consoleserver powering on? network boot visible?stalls at PXE?2 Check DHCP server logsno entries for MAC?โ†’ verify Multus / hostNetwork config3 Check IPA ramdisk logsno logs? server never contacted Ironic ยทcheck bootMACAddress4 Check Ironic logslook for MAC address or IPMI errorsOther: ramdisk-downloader failedcheck init container logs ยท ramdisk image pull issueserver stuck atprovisioning1 Check Ironic logskubectl logs -n vmetal deploy/ironic2 Image URL unreachable?curl image URL from a debug pod in the cluster3 Checksum mismatch?recheck spec.image.checksum in BareMetalHost4 Wrong rootDeviceHints?verify spec.rootDeviceHints matches target diskForce re-provisiondelete Machine in UI or NodeClaim via API

Common failuresโ€‹

Stuck in registeringโ€‹

The BareMetalHost stays in registering when the Bare Metal Operator (BMO) can't reach the BMC.

Check BMO logs for connection errors:

kubectl --context <control-plane-context> logs deployment/bare-metal-operator -n my-metal3-namespace -c bare-metal-operator | grep my-baremetalhost-name

Verify:

  • The BMC address in spec.bmc.address is reachable from the control plane cluster.
  • Firewall rules allow traffic from the Bare Metal Operator and Ironic workloads to the BMC port (623 for IPMI, 443 for Redfish).
  • The Secret referenced by spec.bmc.credentialsName contains valid username and password keys.

Stuck in inspectingโ€‹

The BareMetalHost stays in inspecting when the server can't boot into the IPA ramdisk and phone home to Ironic. Work through these checks in order. Each one narrows the failure to a specific layer.

Check the BMC consoleโ€‹

The BMC console shows the server's boot sequence directly. Because it's out-of-band, it works even if the Metal3 setup is misconfigured.

Access the console through the BMC web interface or using ipmitool:

ipmitool -I lanplus -H <bmc-address> -U <user> -P <pass> sol activate

You should see this sequence:

  1. The server powers on or reboots (triggered by Metal3 over BMC).
  2. The firmware attempts a network boot on the provisioning NIC.
  3. The NIC gets a DHCP lease.
  4. The server downloads a kernel and initramfs over TFTP or HTTP.

If the server doesn't power on or reboot, the problem is BMC access. Check the BMC address, network reachability from the Bare Metal Operator and Ironic workloads, and the credentials in the Secret referenced by spec.bmc.credentialsName.

If the server powers on but stalls during network boot, check the DHCP server logs next.

Check DHCP server logsโ€‹

The DHCP server logs every request and response it sees. It's the most targeted tool for diagnosing a stuck network boot.

kubectl --context <control-plane-context> logs -n my-metal3-namespace -l app=dhcp-proxy

The logs include DHCP requests from the server, including requests from MACs that don't match any registered BareMetalHost (a sign of a wrong spec.bootMACAddress), DHCP responses, TFTP requests for the bootloader, and HTTP requests for the kernel and initramfs images.

If no entries appear for the server, the DHCP broadcast isn't reaching the pod. Check that the DHCP server is attached to the provisioning network using a Multus NetworkAttachmentDefinition, or that it's running with hostNetwork: true for flat networks. See Network configuration.

Check ramdisk and Ironic logsโ€‹

The ramdisk-logs container tails IPA log files as they arrive. If no logs appear, the server never contacted Ironic.

kubectl --context <control-plane-context> logs statefulset/ironic -n my-metal3-namespace -c ramdisk-logs
kubectl --context <control-plane-context> logs statefulset/ironic -n my-metal3-namespace -c ironic

Look for errors related to the server's MAC address or IPMI/Redfish operations.

Other causes include a wrong boot MAC (spec.bootMACAddress must match the NIC used for PXE boot; the DHCP server logs show requests from unmatched MACs), or a failed IPA image download. The ipa-images Deployment downloads and serves the kernel and initramfs independently from Ironic. Inspect it with kubectl --context <control-plane-context> describe deployment/ipa-images -n <namespace> and kubectl --context <control-plane-context> logs deployment/ipa-images -n <namespace>.

Stuck in provisioningโ€‹

The BareMetalHost stays in provisioning when Ironic can't complete the OS installation. Check Ironic logs:

kubectl --context <control-plane-context> logs statefulset/ironic -n my-metal3-namespace -c ironic

Common causes:

  • Image URL unreachable. The server's IPA agent downloads the OS image directly. It must be reachable from the bare metal server's provisioning or tenant network, not just from the control plane cluster. Test the URL from a host on the same routed network as the server.
  • Checksum mismatch. Ironic verifies the downloaded image against spec.image.checksum. A wrong checksum fails provisioning with a checksum error in the Ironic logs.
  • Wrong rootDeviceHint. If a server has multiple disks, Ironic may target the wrong one. Set spec.rootDeviceHints on the BareMetalHost to target the correct disk by size, WWN, or device name.

To force a re-provision, delete the instance from the vCluster Platform UI, or delete its NodeClaim through the API. The platform deprovisions the BareMetalHost and returns it to available. A new instance request then triggers a fresh attempt. Don't manually edit the BareMetalHost's spec.image field while a NodeClaim holds the server. The platform manages that field and reconciles it on the next cycle.

Stuck in deprovisioningโ€‹

When an instance's NodeClaim is deleted, the platform removes the image reference from the BareMetalHost and sets it offline. The BMO then triggers automated cleaning and returns the server to available. With the default metadata cleaning mode, this removes disk metadata such as partition tables. It isn't a full-disk secure erase, and the disabled mode skips automated cleaning altogether. See Reuse and reprovisioning before choosing a mode for sensitive workloads. The BMC must remain reachable from Metal3 throughout, and the cleaning ramdisk must be able to reach Ironic and the DHCP proxy when booting. Cleaning can take several minutes on servers with many disks or large disk capacities.

kubectl --context <control-plane-context> logs deployment/bare-metal-operator -n my-metal3-namespace -c bare-metal-operator | grep -i "clean\|deprovi"
kubectl --context <control-plane-context> logs statefulset/ironic -n my-metal3-namespace -c ironic | grep -i "clean"

If a server is physically removed or permanently unavailable, first delete the instance that claims it through the vCluster Platform UI, or delete its NodeClaim through the API. After the instance teardown completes, delete the BareMetalHost resource from inventory. If normal teardown can't complete because the server is unreachable, resolve the stuck NodeClaim before removing the inventory record.

Error stateโ€‹

A BareMetalHost whose operational status is error remains in its current provisioning state while BMO retries reconciliation. Read the error details and events first:

kubectl --context <control-plane-context> get baremetalhost my-baremetalhost-name -n my-metal3-namespace -o jsonpath='{.status.errorMessage}'
kubectl --context <control-plane-context> get baremetalhost my-baremetalhost-name -n my-metal3-namespace -o jsonpath='{.status.errorType}'
kubectl --context <control-plane-context> describe baremetalhost my-baremetalhost-name -n my-metal3-namespace
  • Monitor a transient error such as a network blip or temporary BMC unresponsiveness. BMO retries without a manual annotation. If the error remains, verify connectivity and inspect the BMO and Ironic logs before changing the host.
  • Fix credentials when the error reports wrong BMC credentials. Update the referenced Secret with the correct values. BMO reads the changed Secret during reconciliation and retries without reprovisioning the server.
  • Stop and create a new instance when the error is a checksum mismatch or image URL failure. The image URL and checksum are written to Ironic's database at provisioning time, so updating the OSImage resource doesn't affect an in-flight request. Delete the instance in the UI, or delete its NodeClaim through the API, then create a new one with corrected image configuration.
  • Correct the BareMetalHost configuration when the BMC address, credentials reference, or boot MAC is wrong. Edit the resource in place when the field permits it. If you must recreate the host, release its instance first and wait for the NodeClaim to finish deleting before recreating it. Deleting a claimed BareMetalHost doesn't by itself clean up the claim.

Instance stays pending without a serverโ€‹

An instance that can't find an eligible BareMetalHost waits, and its NodeClaim conditions say why:

Condition and reasonMessage containsCause
Provisioned, InsufficientCapacityno available BareMetalHost found for NodeType <provider>.<node type>No free host passes the node type's selector and resource check.
Provisioned, InsufficientCapacityMachine pool is "Unassigned hosts only"The node type uses OnDemand. No eligible unassigned host is free, but the tenant has free reserved hosts. Use a node type whose Machine pool is Reserved hosts only to use them.
Scheduled, InsufficientCapacitynode type <name> has no free machine this tenant may useNo matching Assigned type has free reserved capacity for the tenant.
Scheduled, InsufficientCapacitynode type <name> has no free unassigned machineA matching OnDemand type has no free unassigned capacity.
Scheduled, FailedToScheduleno available and matching node type found for node claim requirementsThe instance didn't name a node type, and no node type matches its requirements.

A Scheduled failure retries automatically with no time limit. A Provisioned wait is limited instead. If no host becomes available within the provisioning wait timeout, 30 minutes by default, the instance moves to Failed. To change the timeout, see When provisioning fails.

A selector that matches nothing doesn't raise a condition on the node type. Its HasCapacity condition stays true, and only the capacity total shows the problem:

Modify the following with your specific values to generate a copyable command:
kubectl --context <management-context> get nodetype metal3-provider.compute-node \
-o jsonpath='{.status.capacity}'

A total of 0 means no usable host matches the node type. Check that the hosts carry the labels the node type's machines.selector names. Also check that they're in available state. The provider doesn't compare their inspected CPU and memory with the node type's resources. Keys under vcluster.com and loft.sh aren't copied from a host to its Machine. See How host labels reach Machines.

For the Assigned case, assign another host to the tenant, release one of its instances, or switch the node type to OnDemand. See Machine allocation.

If the instance has already timed out, correct the capacity problem. Increasing the timeout or making capacity available doesn't restart a failed claim. See When provisioning fails for manual replacement and Auto Nodes recovery.

SSH key is refused or not foundโ€‹

An instance can only use SSH keys that its tenant can see. Two symptoms point to a key the tenant can't see.

  • The platform refuses the NodeClaim at creation with ssh key "<name>" not found. The key doesn't exist, or it isn't visible to the instance's tenant. The claim can reference the key directly or inherit it from its node type or node provider.
  • An existing instance stops provisioning and reports an error containing failed to get ssh key <name>. A required key doesn't exist or isn't visible to the instance's tenant. This can happen if the key was deleted or stopped being shared after the instance was created.

To fix either one:

  1. Confirm the key exists. SSH key names are generated, so check the exact name, including its random suffix.

    kubectl --context <management-context> get sshkeys
  2. For an instance in a tenant's project, share the key with that tenant, or reference a key the tenant owns. See Share SSH keys with tenants.

A Metal3 instance in this state keeps reporting the error and retries provisioning. It provisions once the key is visible to its tenant again.

Outcome-specific failuresโ€‹

Use these checks after Metal3 reports the BareMetalHost as provisioned. First, map the instance to its underlying host.

The NodeClaim records the BareMetalHost name in the metal3.vcluster.com/claimed-node annotation:

kubectl --context <management-context> get nodeclaim my-machine \
-n loft-p-my-project \
-o jsonpath='{.metadata.annotations.metal3\.vcluster\.com/claimed-node}'

To look up the mapping in the other direction, select the BareMetalHost by the metal3.vcluster.com/node-claim label:

kubectl --context <control-plane-context> get baremetalhost \
-n my-metal3-namespace \
-l 'metal3.vcluster.com/node-claim=my-machine,vcluster.com/project=my-project'

If the first command returns a host other than this page's BareMetalHost name, update that page variable before running the remaining interpolated commands.

Instance is available but SSH failsโ€‹

An independently provisioned instance can be Available even when SSH access fails. Available means the operating system was provisioned and the platform completed the claim. It doesn't test an SSH login.

  1. Inspect the NodeClaim on the management cluster. Confirm that the IP annotation is present and that the claim references the expected node type.

    kubectl --context <management-context> get nodeclaim my-machine \
    -n loft-p-my-project -o yaml
  2. Confirm that every cluster-scoped SSHKey named by the node type exists and contains the intended public key. A key that the instance's tenant can't see isn't installed. See SSH key is refused or not found.

    kubectl --context <management-context> get sshkey <ssh-key-name> -o yaml
  3. Inspect the cloud-init payload generated for the host. The Secret is on the control plane cluster. Treat the decoded payload as sensitive, because it contains bootstrap configuration.

    kubectl --context <control-plane-context> get secret vcluster-userdata-my-baremetalhost-name \
    -n my-metal3-namespace -o jsonpath='{.data.userData}' | base64 --decode
  4. Use the BMC console to inspect cloud-init when SSH is unavailable.

    sudo cloud-init status --long
    sudo tail -n 200 /var/log/cloud-init-output.log
    sudo journalctl -u cloud-final --no-pager
  5. Verify that you're using the image's correct login user, that TCP port 22 is reachable, and that the reported IP belongs to the server. A successful provision doesn't validate routing or firewall policy between the operator and that IP.

Worker instance is provisioned but not joinedโ€‹

For a cluster worker, Provisioned=True with Joined=False means the operating system installation finished but the bootstrap process didn't register the node with the cluster.

  1. Read the NodeClaim conditions and events on the management cluster.

    kubectl --context <management-context> describe nodeclaim my-machine \
    -n loft-p-my-project
  2. Use SSH or the BMC console to check cloud-init.

    sudo cloud-init status --long
    sudo tail -n 200 /var/log/cloud-init-output.log
    sudo journalctl -u cloud-final --no-pager
  3. From the server, verify DNS and network access to the vCluster Platform bootstrap endpoint and the tenant Kubernetes API endpoint. Bootstrap can't finish if either endpoint is unreachable or its certificate name can't be resolved.

  4. Check whether the node registered in the cluster.

    kubectl --context <cluster-context> get nodes -o wide

Control plane instance is available but the cluster is offlineโ€‹

The platform can mark a control plane instance Available after provisioning completes even if the vCluster service on the server fails to start. The cluster controller's VirtualClusterReady condition isn't sufficient evidence either. For a standalone control plane, it can report that the requested instance objects were created before the servers finish provisioning. Verify the service on the server and connect to the cluster API.

  1. Use SSH or the BMC console to inspect cloud-init and the vCluster systemd service.

    sudo cloud-init status --long
    sudo tail -n 200 /var/log/cloud-init-output.log
    sudo systemctl status vcluster --no-pager
    sudo journalctl -u vcluster --no-pager -n 200
  2. Verify that /etc/vcluster exists and that /etc/systemd/system/vcluster.service was created. If either is missing, use the cloud-init output to find the failed bootstrap step.

  3. Confirm the host meets the control plane prerequisites. Those are root access, systemd, curl, iptables, and the raised inotify limits listed in the standalone node requirements. Also verify DNS and network access to vCluster Platform and, for a multi-node control plane, the other control plane instances.

  4. Check the cluster's status and events from the management cluster. A ready instance alone doesn't prove that the cluster API is online.

IP allocation issuesโ€‹

The platform stores IP allocations in a ConfigMap named ipam-default in the loft management namespace (typically loft). If NodeClaim provisioning fails with a failed to allocate IP error, check here.

The error appears in the platform logs and in NodeClaim events, as failed to allocate IP: CIDR exhausted; no free addresses left (CIDR mode) or failed to allocate IP: all ranges exhausted; no free addresses left (explicit range mode).

Inspect current allocations:

kubectl --context <management-context> get configmap ipam-default -n loft -o yaml

The ConfigMap data is a flat map of <ip>: NodeClaim/<namespace>/<name> entries, one per allocated address:

data:
10.10.1.2: NodeClaim/loft-p-myproject/machine-abc
10.10.1.3: NodeClaim/loft-p-myproject/machine-def

Leaked IPs from a failed deprovision. If the platform deleted an instance but didn't release its IP (for example, a crash during cleanup), the entry remains in the ConfigMap even though the NodeClaim no longer exists. To clean up: list active NodeClaims (kubectl --context <management-context> get nodeclaims -A), compare against the ConfigMap entries, and edit the ConfigMap (kubectl --context <management-context> edit configmap ipam-default -n loft) to remove entries whose NodeClaim no longer exists.

CIDR or range too small. If the number of instances regularly exceeds the available address space, expand the range using the metal3.vcluster.com/network-cidr or metal3.vcluster.com/network-ip-range property. See Network configuration.

Host operationsโ€‹

Redfish tab is missingโ€‹

The Metal3 UI requires a platform admin outside tenant scope, with the machine management and Metal3 features enabled. The Redfish entry additionally requires create permission on metal3bmcs/redfish in cluster.loft.sh for the host's cluster and namespace.

Check those permissions before changing the BMC. Reading a Machine or its BareMetalHost doesn't grant permission to send BMC requests. See Inspect the BMC.

Redfish request failsโ€‹

First distinguish a platform API error from an HTTP response returned by the BMC:

SymptomCheck
Permission denied before a BMC responseThe caller's create permission on the Redfish subresource.
Host or credential configuration errorThe BareMetalHost's spec.bmc.address, credentialsName, and Secret in the same namespace.
BMC type doesn't speak RedfishUse a Redfish-compatible BMC scheme. An ipmi:// address can't serve Redfish resources.
Certificate or connection errorConnectivity and certificate trust from the platform agent to the BMC. Ironic's CA mount doesn't configure the agent.
BMC returns HTTP 401 or 403The referenced credentials and the vendor permissions needed for the requested resource or action.
BMC returns HTTP 404Whether that path exists on this firmware. Browse from /redfish/v1 instead of assuming another vendor's resource IDs.
Redirect responseThe proxy returns redirects without following them. Check the BMC address and use the intended Redfish endpoint.

A successful platform API request can contain an unsuccessful BMC HTTP status. Read that status and the BMC's error message before retrying. Requests have a 30-second timeout. If an action times out, inspect the hardware state before sending it again.

Don't copy raw responses containing passwords or session tokens into support tickets. Keep audit capture at Metadata level.

Host has no usable DHCP addressโ€‹

The platform DHCP server needs the host's metal3.vcluster.com/ip-address annotation as an IPv4 address with a prefix. A claim with no allocated address fails with InvalidConfiguration when the provider-managed DHCP server is enabled and the host's annotations aren't usable.

Inspect the host on the control plane cluster:

kubectl --context <control-plane-context> get baremetalhost my-baremetalhost-name -n my-metal3-namespace -o yaml

Check the boot MAC, address prefix, gateway, and comma-separated DNS addresses. Gateway and DNS entries must be IPv4 addresses. Configure a host address before initial inspection. For claim-allocated addresses, check the node type's metal3.vcluster.com/network-cidr property and IPAM configuration.

After fixing a claim's terminal InvalidConfiguration failure, release the failed instance and create a new one. Follow When provisioning fails for recovery.

During provisioning, if Ironic has waited more than 15 minutes for the deploy ramdisk's callback without the agent starting, the Machine reports ramdisk not booted. Check DHCP replies, boot files, and the IPA API path. This is separate from the instance's capacity-wait timeout. A host that never boots the ramdisk during initial inspection doesn't get this reason. It stays in inspecting instead. See Stuck in inspecting.

Unexpected or missing IP addressโ€‹

Compare the host's metal3.vcluster.com/ip-address annotation with the Metal3 list and the Machine's network details.

  • Empty IP column: the Metal3 list uses the configured annotation. It doesn't substitute an address from hardware inspection.
  • Inspection address in Machine details: check the NIC inventory, annotation, and spec.bootMACAddress. The configured address replaces inspected addresses only on the matching boot NIC.
  • Different address in the running OS: inspect any custom network data. The Machine address overlay doesn't parse a custom document to discover its configured address.
  • Address changes during claim or release: a claim can temporarily replace the host reservation. Release restores the operator's original DHCP annotations. See Reuse and reprovisioning.

Test routing and access from the intended client separately. A displayed IP isn't a connectivity check.

Unexpected power statusโ€‹

Compare Ironic's current and target power states with the BareMetalHost's spec.online and status.poweredOn. The Metal3 badge prefers Ironic's power reading, but falls back to the host's status when no usable Ironic reading is available.

During inspection, provisioning, or deprovisioning, BMO's own power reading can be stale. Powering On, Powering Off, and Rebooting indicate a transition, rather than proof that it completed.

Check BMC connectivity and Ironic logs if a transition doesn't finish. Don't change spec.online or send repeated resets to fix an instance allocation problem. An On server can still have a provisioning or bootstrap failure. See Common failures.

Webhook readinessโ€‹

The Metal3 ValidatingWebhook validates every BareMetalHost create and update. It runs in the bare-metal-operator container on port 9443, reached through the bare-metal-operator-webhook Service, and defaults to failurePolicy: Fail. If the webhook endpoint is unreachable when you create a BareMetalHost, the API server rejects the request and the BareMetalHost is never created.

Check whether the BMO container is ready:

kubectl --context <control-plane-context> get deployment bare-metal-operator -n my-metal3-namespace
kubectl --context <control-plane-context> get pods -n my-metal3-namespace -l app=bare-metal-operator

If a pod isn't ready, inspect its events and the configured readiness probe:

kubectl --context <control-plane-context> describe pods -n my-metal3-namespace -l app=bare-metal-operator

Check the ValidatingWebhookConfiguration for the current CA bundle and service endpoint:

kubectl --context <control-plane-context> get validatingwebhookconfiguration bare-metal-operator -o yaml

Log locationsโ€‹

Metal3 chart 0.6.0 and later deploy separate workloads on the control plane cluster, in the namespace configured in the NodeProvider's clusterRef.

ComponentContainerLog command
Ironicironic StatefulSet, ironic containerkubectl --context <control-plane-context> logs statefulset/ironic -n <namespace> -c ironic
Bare Metal Operatorbare-metal-operator Deploymentkubectl --context <control-plane-context> logs deployment/bare-metal-operator -n <namespace> -c bare-metal-operator
IPA ramdisk outputironic StatefulSet, ramdisk-logs containerkubectl --context <control-plane-context> logs statefulset/ironic -n <namespace> -c ramdisk-logs
IPA kernel and initramfs serveripa-images Deploymentkubectl --context <control-plane-context> logs deployment/ipa-images -n <namespace>
Managed databasemariadb StatefulSetkubectl --context <control-plane-context> logs statefulset/mariadb -n <namespace>

First inspect BareMetalHost events on the control plane cluster:

kubectl --context <control-plane-context> describe baremetalhost my-baremetalhost-name -n my-metal3-namespace

Then, on the management cluster, inspect NodeClaim events for platform-level failures such as no available hosts, an IPAM failure, or an image resolution error:

kubectl --context <management-context> get nodeclaims -A
kubectl --context <management-context> describe nodeclaim <nodeclaim-name> -n <project-namespace>