Skip to main content
Version: main 🚧

Migrate etcd backing store

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

This guide explains how to migrate your vClustervClusterAn open-source software product that creates and manages virtual Kubernetes clusters inside a host Kubernetes cluster. vCluster improves isolation and multi-tenancy capabilities while reducing infrastructure costs.Related: Virtual Cluster, Host Cluster from deployed (external) etcd to embedded etcd without data loss.

Why migrate?

If you're currently using deployed etcdetcdA distributed key-value store that provides reliable storage for Kubernetes cluster data. In vCluster, etcd can be deployed externally or embedded within the vCluster pod.Related: Control Plane and want to simplify your deployment, reduce resource consumption, or streamline operations, migrating to embedded etcd is a straightforward process that preserves all your data.

Overview​

The migration from deployed to embedded etcd is a seamless process that:

  • Preserves all your vCluster data and workloads
  • Requires minimal downtime (only during the upgrade process)
  • Automatically handles the data transfer between etcd instances
  • Cleans up the deployed etcd resources after successful migration
Critical requirement

The migrateFromDeployedEtcd flag is essential for preserving data. Without this flag, your vCluster state will be wiped during the transition.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster CLI - Required for migration commands 
    brew install loft-sh/tap/vcluster

    The binaries in the tap are signed using the Sigstore framework for enhanced security.

    Confirm that you've installed the correct version of the vCluster CLI.

    vcluster --version
  • vCluster version 0.24.2 or later - Earlier versions don't support this migration
  • Existing vCluster with deployed etcd - This guide assumes you have a vCluster currently using deployed (external) etcd
  • Backup your vCluster data - Create a snapshot before migration (see backup guide)
  • Storage requirements:
    • Storage class available for the embedded etcd PVC (default: 5Gi)
    • Sufficient storage capacity for your etcd data
  • RBAC permissions - You need permissions to:
    • Modify StatefulSets and Deployments in the vCluster namespace
    • Create and delete PVCs
    • View pod logs for verification

Migration process​

  1. Verify current backing store

    Check that you're using deployed etcd:

    Modify the following with your specific values to generate a copyable command:
    kubectl get statefulset -n vcluster-my-team my-vcluster-etcd

    You should see the deployed etcd StatefulSet running.

  2. Create migration configuration

    Create a new values file that enables embedded etcd with the migration flag:

    migrate-to-embedded.yaml
    controlPlane:
      backingStore:
        etcd:
          embedded:
            enabled: true
            # CRITICAL: This flag preserves your data during migration
            migrateFromDeployedEtcd: true
            # Optional: Configure embedded etcd settings
            extraArgs:
              - '--auto-compaction-mode=periodic'
              - '--auto-compaction-retention=30m'
              - '--quota-backend-bytes=8589934592'  # 8GB
      # Optional: Configure storage for embedded etcd
      persistence:
        volumeClaim:
          size: 10Gi  # Adjust based on your deployed etcd size
          # storageClass: "your-storage-class"  # Optional: specify if not using default
    Important notes
    • The migrateFromDeployedEtcd: true flag is crucial for data preservation
    • During migration, both etcd instances will temporarily coexist
    • The deployed etcd remains available as a fallback during migration

  3. Apply the migration configuration

    Upgrade your vCluster with the migration configuration:

    Modify the following with your specific values to generate a copyable command:
    helm upgrade my-vcluster vcluster \
    --repo https://charts.loft.sh \
    --namespace vcluster-my-team \
    --version 0.27.0 \
    --reuse-values \
    -f migrate-to-embedded.yaml \
    --wait
    Migration duration

    The migration typically completes within 1-5 minutes, depending on:

    • The size of your etcd database
    • Network latency between pods
    • Storage performance

  4. Verify successful migration

    Check the vCluster logs for the success message:

    Modify the following with your specific values to generate a copyable command:
    kubectl logs -n vcluster-my-team my-vcluster-0 | grep -i "migrated"

    You should see: Successfully migrated etcd database to embedded etcd

  5. Remove deployed etcd resources

    After confirming successful migration, remove the migration flag to clean up deployed etcd:

    final-embedded.yaml
    controlPlane:
      backingStore:
        etcd:
          embedded:
            enabled: true
            # Migration flag removed - deployed etcd will be deleted
            extraArgs:
              - '--auto-compaction-mode=periodic'
              - '--auto-compaction-retention=30m'
              - '--quota-backend-bytes=8589934592'  # 8GB
      persistence:
        volumeClaim:
          size: 10Gi  # Keep the same size as during migration
          # storageClass: "your-storage-class"  # Optional: if you specified one during migration

    Apply the final configuration:

    Modify the following with your specific values to generate a copyable command:
    helm upgrade my-vcluster vcluster \
    --repo https://charts.loft.sh \
    --namespace vcluster-my-team \
    --version 0.27.0 \
    --reuse-values \
    -f final-embedded.yaml \
    --wait

    This removes:

    • The deployed etcd StatefulSet
    • Associated etcd Services
    • The deployed etcd pods (they are terminated gracefully)
    About PVCs

    The deployed etcd PVCs are NOT automatically deleted to prevent accidental data loss. You can manually delete them after verifying the migration:

    Modify the following with your specific values to generate a copyable command:
    kubectl delete pvc -n vcluster-my-team data-my-vcluster-etcd-0

Pod deletion behavior​

During the migration process, pod management follows this sequence:

  1. During initial upgrade (with migrateFromDeployedEtcd: true):

    • vCluster deployment is replaced with a StatefulSet
    • Embedded etcd starts within the vCluster pod
    • Deployed etcd pods remain running for data migration
    • Data is copied from deployed to embedded etcd
    • No vCluster workload pods are affected

  2. After migration completion:

    • The embedded etcd becomes the primary backing store
    • Deployed etcd pods continue running but are no longer used
    • vCluster workloads continue uninterrupted

  3. After removing migration flag:

    • Deployed etcd StatefulSet is deleted
    • Deployed etcd pods are terminated gracefully
    • PVCs remain for manual cleanup
    • vCluster workloads are not affected

Storage considerations​

Storage sizing​

Embedded etcd creates a new PVC for storage. The default size is 5Gi, but you can customize it:

controlPlane:
  persistence:
    volumeClaim:
      size: 10Gi  # Adjust based on your needs

Storage migration​

The data migration process:

  • Creates a new PVC for embedded etcd
  • Copies all data from deployed etcd to embedded etcd
  • Verifies data integrity before switching over
  • Maintains the deployed etcd as a fallback until explicitly removed

Certificate and path differences​

When migrating, be aware of these differences between deployed and embedded etcd:

AspectDeployed etcdEmbedded etcd
Certificate location/run/config/pki//data/pki/etcd/
Data directorySeparate PVC/data in vCluster PVC
Service endpoint<vcluster>-etcd:2379127.0.0.1:2379 (localhost)
ManagementSeparate StatefulSetPart of vCluster pod

Resolve common issues​

Migration doesn't start​

If you don't see the migration message in logs:

  1. Verify the deployed etcd is accessible:

    Modify the following with your specific values to generate a copyable command:
    kubectl exec -n vcluster-my-team my-vcluster-0 -- curl -k https://my-vcluster-etcd:2379/health

  2. Check for Helm upgrade issues:

    Modify the following with your specific values to generate a copyable command:
    helm status my-vcluster -n vcluster-my-team

  3. Ensure the configuration is correct:

    Modify the following with your specific values to generate a copyable command:
    helm get values my-vcluster -n vcluster-my-team | grep -A5 etcd

Data appears to be missing​

If resources seem missing after migration:

  1. Verify you used migrateFromDeployedEtcd: true
  2. Check the migration completed successfully in logs
  3. Ensure you're connected to the correct vCluster context
  4. Review the vCluster pod logs for any errors

StatefulSet modification errors​

If you encounter errors like "StatefulSet.apps is invalid: spec: forbidden":

  1. Delete the StatefulSet with --cascade=orphan to keep pods running:

    Modify the following with your specific values to generate a copyable command:
    kubectl delete statefulset my-vcluster-etcd -n vcluster-my-team --cascade=orphan

  2. Then retry the Helm upgrade

Next steps​

After successfully migrating to embedded etcd, you can further customize your configuration: