Restore snapshots
There are multiple ways to back up and restore a cluster. vClustervClusterAn open-source software product that creates and manages clusters within Kubernetes infrastructure. vCluster provides tenant isolation capabilities while reducing infrastructure costs. provides a built-in method to create and restore snapshots using its CLI.
A vCluster snapshot includes:
- Backing store data (such as etcdetcdA distributed key-value store that provides reliable storage for Kubernetes cluster data. In vCluster, etcd can be deployed externally or embedded within the vCluster pod. or SQLite)
- vCluster Helm release information
- vCluster configuration (for example,
vcluster.yaml)
If the cluster uses private nodes and its nodes changed since the snapshot, follow the additional steps for private nodes after the restore.
Restore existing cluster from a snapshot​
Restoring from a snapshot pauses the vCluster, scales down all workload pods to 0, and launches a temporary restore pod. Once the restore completes, vCluster resumes and scales all workload pods back up. This process results in temporary downtime while the restore is in progress.
vCluster pauses the cluster and scales workload pods to 0 before the restore pod attempts to download the snapshot. An invalid snapshot URL or expired credential is therefore discovered after downtime begins. Before restoring, run vcluster snapshot get with the same URL and credential options. Continue only if it reports the snapshot as saved. This check doesn't pause the cluster.
If the restore fails for this or any other reason, the process stops. You must retry the restore to avoid leaving the clusterClusterA fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster. in an inconsistent or broken state.
Restore a cluster using vcluster restore with the following commands. You can use any snapshot option and set the snapshot URL with different credentials. Whichever credentials you provide, vCluster never writes them into the restore pod's spec. It mounts them from a Kubernetes Secret through secretKeyRef, and deletes that Secret along with the pod once the restore finishes.
Restore clusters from OCI​
- Local credentials
- URL credentials
vcluster restore my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag"
export OCI_USERNAME=my-username
export OCI_PASSWORD=$(echo -n "my-password" | base64)
vcluster restore my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag?username=$OCI_USERNAME&password=$OCI_PASSWORD&skip-client-credentials=true"
Restore clusters from S3​
vcluster restore my-vcluster "s3://my-s3-bucket/my-snapshot-key"
Restore clusters from Azure Blob​
- Azure CLI
- SAS token
- Storage account key
vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz" \
--azure-subscription-id my-subscription-id \
--azure-resource-group my-resource-group
vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz?sv=2022-11-02&ss=b&srt=co&sp=rwlacuptfx&se=2027-01-01T00:00:00Z&sig=YOUR_SAS_SIGNATURE"
export AZURE_STORAGE_KEY=my-storage-account-key
vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz"
Restore clusters from container​
vcluster restore my-vcluster "container:///data/my-snapshot.tar.gz"
Restore a standalone vCluster​
For standalone vCluster running as a systemd service, use the --standalone flag instead of a cluster name. This flag can't be used with --driver docker. The CLI automatically stops the vCluster service, restores the backing store data from the snapshot, and restarts the service.
If the standalone instance runs vCluster Platform, the restore leaves node provider resources behind that you must clean up. Before you restore, read Clean up node provider resources after an etcd restore.
vcluster restore --standalone "container:///var/lib/vcluster/my-snapshot.tar.gz"
You can also restore from any external storage backend:
vcluster restore --standalone "oci://ghcr.io/my-user/my-repo:my-tag"
Restoring a standalone vCluster that uses the embedded etcd backing store can invalidate service account tokens for running system pods, such as CoreDNS, kube-proxy, and the konnectivity agent. The restored etcd data predates any tokens issued after the snapshot was taken, so the API serverAPI ServerThe core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects. rejects them on the UID check. kubectl get pods still reports these pods as Running, but in-pod DNS resolution stops working.
To restore full capability, manually restart the vCluster control planeControl PlaneThe container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. In vCluster, each cluster has its own control plane components. again after the automatic restart from vcluster restore --standalone completes:
systemctl restart vcluster
Restoring to a different machine leaves stale node details in etcd that require manual cleanup. To make the snapshot available on another node, use an external storage backend (OCI, S3, or Azure Blob Storage). Plan to manually remove stale node resources after restore.
Restore a standalone vCluster in HA mode​
Restoring a standalone vCluster configured with multiple etcd peers (HA mode) requires manual intervention. Shut down all nodes except the restore node, then run the restore on that node.
The following steps assume a three-node setup where you run the restore on node A:
-
Stop the vCluster service on all nodes except the restore node:
# Run on nodes B and Csystemctl stop vcluster -
Run the restore on node A:
Modify the following with your specific values to generate a copyable command:vcluster restore --standalone "snapshot-url" -
Delete the stale node objects for the stopped nodes from the restored cluster:
kubectl delete node <node-B> <node-C> -
On nodes B and C, move the old vCluster state aside so they rejoin cleanly:
# Run on nodes B and Cmv /var/lib/vcluster /var/lib/vcluster-bcp -
On node A, create a join token with
vcluster token create:vcluster token create --control-plane -
Use the join script output to rejoin nodes B and C one at a time. Wait for each node to reach
Readybefore joining the next to rebuild etcd quorum sequentially.
Clean up node provider resources after an etcd restore​
An etcd restore of a control plane cluster that runs vCluster Platform doesn't roll back node provider state. The platform keeps node state, such as NodeClaim objects, in that etcd. External systems keep their own state, so after the restore it can differ from what etcd describes. Examples include Metal3 and Ironic, NICo, and cloud provider resources.
This applies whether or not the control plane cluster is a standalone instance. For example:
vcluster restore --standalone, in single-node or HA mode, on a standalone instance that runs vCluster Platform.- A Velero or etcd snapshot restore of a control plane cluster that runs vCluster Platform.
Restoring with vcluster platform backup and kubectl apply doesn't cause this, because that backup doesn't include NodeClaim objects or provider objects.
The restored etcd data stays the source of truth. Controllers reconcile external infrastructure toward it. They reuse existing provider resources they can match and recreate missing resources where it's safe. Controllers never delete a provider resource that has no matching Kubernetes object, because they can't tell whether deleting it destroys workload data.
You must find these orphaned resources and clean them up with each provider's own tooling. To find them, compare each provider's own inventory with the instances the platform shows after the restore. Instances created after the snapshot was taken are the most likely to be left behind:
- Metal3: The machine shows
Available, but Ironic still has the old deployment on it. The next instance on that machine starts from that deployment, including the previous instance's SSH key and data. Remove the deployment in Ironic before another instance uses the machine. - NICo: NICo keeps the instance running. The machine stays
Provisioned, and you can't provision it again or delete it from the UI.
Clone a cluster to a new cluster​
You can use snapshots to clone an existing cluster and create a new cluster from that snapshot. When creating a new cluster from a snapshot, it also restores all workloads in the cluster.
When you run vcluster create --restore without -f, --values, or --set, vCluster reads the Helm configuration from the snapshot before installing the cluster. If it can't access the snapshot, the command exits before the new cluster starts. Providing configuration through one of those flags skips this check because vCluster uses the supplied configuration instead.
If the restore fails while using vcluster create, vCluster automatically deletes the new cluster.
# Create a new cluster from an OCI snapshot (uses local credentials).
vcluster create my-vcluster --restore oci://ghcr.io/my-user/my-repo:my-tag
vCluster certificates change when you create a cluster with a new name or namespace. This is expected, as clusters shouldn't share certificates.
Migrate and override vCluster configuration options to create a new cluster​
Some configuration options can't change on an existing cluster. For example, you can't change the backing store type in place. See Backing store for details. To change these options, migrate the cluster by creating a new one from a snapshot with updated configuration.
The one exception is restoring an embedded etcd snapshot onto a different backing store type, which doubles as a migration path. See Migrate from embedded etcd to a different backing store.
When creating a new cluster from a snapshot, it also restores all workloads in the cluster.
If the restore fails while using vcluster create, vCluster automatically deletes the new cluster.
# Upgrade an existing vCluster by restoring from a snapshot and applying a new vcluster.yaml.
# Configuration options in the vcluster.yaml override the options from the snapshot.
vcluster create my-vcluster --upgrade -f vcluster.yaml --restore oci://ghcr.io/my-user/my-repo:my-tag
vCluster certificates change when you create a cluster with a new name or namespace. This is expected, as clusters shouldn't share certificates.
Migrate from embedded etcd to a different backing store​
Restoring an embedded etcd snapshot onto a cluster configured with a different backing store type converts the snapshot automatically. This lets you move a cluster off embedded etcd onto an embedded database, external database, or deployed etcd, using the same snapshot and restore workflow as any other migration.
This migration path is separate from deployed etcd's own snapshot and restore support. Deployed etcd is a supported target for a converted embedded-etcd snapshot, but taking a snapshot of a cluster that already runs on deployed etcd, to back up or restore that same cluster, isn't validated. See Back up and restore.
vCluster can't change a cluster's backing store type in place. Restore this into a new cluster (a different name, namespace, or both) rather than the original one:
# Take a snapshot of the source cluster, which uses embedded etcd.
vcluster snapshot create my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag"
# Restore it into a new cluster whose vcluster.yaml configures a different backing store.
# vCluster detects the etcd snapshot and converts it before restoring.
vcluster create my-vcluster-external-db --namespace vcluster-my-vcluster-external-db -f vcluster.yaml --restore oci://ghcr.io/my-user/my-repo:my-tag
This guide covers migrating from embedded etcd only. Migrating in the other direction, such as from a database backing store to etcd, isn't documented or tested.
Limitations​
When taking snapshots and restoring clusters, there are limitations:
Sleeping clusters
- Snapshots require a running vCluster control plane and don't work with sleeping clusters.
Clusters using an external database
- Restoring onto an external database, whether configured with
dataSourceorconnector, deletes the cluster's existing key/value records (through kine, key by key) before replaying the snapshot. vCluster doesn't back up or roll back the external database itself. Before runningvcluster restore, the database owner or administrator must take a native database backup so that a native external database restore can be triggered if the restore fails.
Clusters using the embedded database
-
Restore refuses to write into an embedded database (SQLite) that's still serving requests, so a running control plane can't corrupt the restored data. The restore fails with this error:
embedded database backing store is still serving requests at <endpoint>, refusing to restore into a running control plane: stop the vCluster control plane and any of its leftover processes, then retryStop the control plane and any leftover processes, then retry the restore. On a standalone instance, the restore stops leftover control plane and database processes before it starts.
Clusters using deployed etcd
- Restoring a cluster that already runs on deployed etcd (
controlPlane.backingStore.etcd.deploy) isn't supported. Restore etcd with its own tooling instead. See Back up and restore. Deployed etcd is a supported target when migrating an embedded etcd snapshot onto a new cluster. See Migrate from embedded etcd to a different backing store.
vcluster.yaml configuration
- Although the vcluster.yaml configuration is backed up, it isn't automatically restored. After restoring a cluster, you must manually reapply your vcluster.yaml configuration.
Standalone: service account tokens invalidated after restore
- Restoring a standalone vCluster that uses the embedded etcd backing store can invalidate service account tokens for system pods, breaking DNS resolution even though the affected pods report as
Running. Manually restart the vCluster control plane after the restore completes. See Restore a standalone vCluster.
Standalone: restore to a new node
- If you restore a snapshot on a different node, the standalone instance's etcd contains node details from the original machine. You must manually clean up stale node resources after the restore. See Use snapshots with private nodes.
Standalone: cluster workloads aren't restored
- A standalone snapshot only restores the standalone instance's own etcd data. Each cluster running on the standalone instance has its own etcd, so workloads deployed inside clusters aren't restored by a standalone snapshot.
Node provider state isn't rolled back
- Any etcd-level restore of a control plane cluster that runs vCluster Platform, standalone or not, doesn't roll back external systems that keep their own state. Examples include Metal3 and Ironic, NICo, and cloud provider resources. Provider resources with no matching Kubernetes object aren't deleted automatically, so you must find and clean them up. See Clean up node provider resources after an etcd restore.
Clusters managed through the Argo CD integration
- The
ArgoCDApplicationobject itself is unaffected by a restore, since it lives outside the cluster's snapshot and restore boundary. The resources Argo CD deployed into the cluster are reverted along with everything else. If the application hasselfHeal: true, Argo CD reapplies the Git revision after it detects the drift, undoing the restore for those resources.prune: trueis also required to remove any restored resources no longer defined in Git. See Restoring a cluster doesn't roll back what Argo CD manages for details.
Use snapshots with private nodes​
The snapshot also includes node resources. Restoring to a different set of nodes requires manual steps for clusters using private nodes. With host nodes, vCluster automatically updates node information when nodes change between snapshot and restore.
Nodes removed between snapshot and restore​
When nodes exist in the snapshot but not in the current cluster, they still show up in kubectl get nodes after the restore. Delete each removed node from the restored cluster:
export NODE_NAME=my-node
kubectl delete node $NODE_NAME
Nodes added between snapshot and restore​
When nodes joined the cluster after the snapshot, they don't exist in the restored cluster and aren't listed in kubectl get nodes. Rejoin each of these private nodes with the --force-join flag.
Create a join token:
export VCLUSTER_NAME=my-vcluster
# Connect to your vcluster
vcluster connect $VCLUSTER_NAME
# Create a token
vcluster token create --expires=1h
Append the --force-join flag to the join command from the output, then run it on the worker node:
curl -sfLk https://vcluster-endpoint/node/join?token=token | sh - -- --force-join