High Availability
By default, vCluster runs one instance of each of its components. That’s fine for many use cases, like ones that are very ephemeral (dev environments, CI/CD, etc.). But suppose your situation requires virtual clusters with more redundancy. In that case, you can use vCluster’s High Availability feature to run multiple copies of the vCluster components so that the cluster is more resistant to partial failures.
Prerequisites
For this tutorial, you will need a multi-node Kubernetes cluster. This can be a local minikube or kind cluster, or any full Kubernetes cluster. You just need more than one node in the cluster.
You also need your kube config pointing to that cluster (you can connect and run kubectl commands against it).
1. Install the vCluster client
If you’re on a Mac and using Homebrew, you can install the vCluster CLI with this command:
brew install loft-sh/tap/vcluster
For other platforms, see the installation instructions.
2. Understanding vCluster distros
A distro in vcluster is the Kubernetes distribution that runs inside the virtual cluster. These are the supported distributions:
- k3s (the default distro)
- k0s
- k8s (a Kubernetes with etcd)
- eks (the Kubernetes that is installed in AWS EKS clusters)
Though the EKS distribution is supported in v0.19, it is deprecated. Reminder: The Kubernetes distribution of the virtual cluster doesn't need to match the the distribution of your host cluster. You can use any distribution of vCluster to deploy vCluster on EKS clusters.
For this tutorial, we will use the k8s distro. If you're interested in enabling HA in rootless mode, or using the k3s distro, see some examples at the bottom of this page.
3. Create a values.yaml file
The values.yaml file is used to specify configuration options for the virtual cluster. In the case of the HA feature, we will specify the number of replicas we want to run for each vCluster component.
Create the file called values.yaml on the computer that the vcluster client is installed on with these contents:
# Scale up syncer replicas
syncer:
replicas: 3
# Scale up etcd
etcd:
replicas: 3
# Scale up DNS server
coredns:
replicas: 3
Set the number of replicas for each to equal the number of compute nodes your cluster has. In this example I’m using a four node Minikube cluster (one control plane node and three compute nodes), so I set the number of replicas to three in values.yaml.
Set replicas to the desired number to deploy in high availability. In this example I’m using a four node Minikube cluster (one control plane node and three compute nodes), so I set the number of replicas to three in values.yaml.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 2m5s v1.26.3
minikube-m02 Ready <none> 105s v1.26.3
minikube-m03 Ready <none> 93s v1.26.3
minikube-m04 Ready <none> 83s v1.26.3
4. Create the HA virtual cluster
To create a virtual cluster using the vcluster CLI, we run the vcluster create command. To enable HA, we’ll need to specify the distro and the values.yaml file to use.
vcluster create ha-tutorial --connect=false --distro k8s -f values.yaml
We’ve named the virtual cluster ha-tutorial. By default, the vcluster create command connects to the virtual cluster, but for the purposes of this tutorial, we’ve disabled that with the --connect=false flag. And we’ve specified the distro and the values.yaml file to use when creating the virtual cluster.
You should see output like this:
info Creating namespace vcluster-ha-tutorial
info failed to find IPv6 service CIDR: couldn't find host cluster Service CIDR ("Service "test-service-tm4c9" is invalid: spec.clusterIPs[0]: Invalid value: []string{"2001:DB8::1"}: IPv6 is not configured on this cluster")
info Detected local kubernetes cluster minikube. Will deploy vcluster with a NodePort & sync real nodes
info Create vcluster ha-tutorial...
info execute command: helm upgrade ha-tutorial /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/vcluster-k8s-0.15.2.tgz-1797632188 --kubeconfig /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/3126958598 --namespace vcluster-ha-tutorial --install --repository-config='' --values /var/folders/gy/d3_c4t1x731_hl8qtrfkhr_h0000gn/T/2770602786 --values values.yaml
done √ Successfully created virtual cluster ha-tutorial in namespace vcluster-ha-tutorial.
- Use 'vcluster connect ha-tutorial --namespace vcluster-ha-tutorial' to access the virtual cluster
Some of your output may differ depending on whether you use a local or remote cluster.
As you can see, vcluster has created a namespace called vcluster-ha-tutorial. The virtual cluster lives inside that namespace on the host cluster. Next, let’s see what pods are running in that namespace.
kubectl get pods -n vcluster-ha-tutorial
NAME READY STATUS RESTARTS AGE
ha-tutorial-7c5c5844c5-27j2v 0/1 Running 0 20s
ha-tutorial-7c5c5844c5-gb2sm 0/1 Running 0 20s
ha-tutorial-7c5c5844c5-pwn7k 0/1 Running 0 20s
ha-tutorial-etcd-0 0/1 Running 0 20s
ha-tutorial-etcd-1 0/1 Running 0 20s
ha-tutorial-etcd-2 0/1 Running 0 20s
There are now three replicas of each component of the virtual cluster running. If one API server pod were down, the virtual cluster would continue functioning.
If you’d like more information about how the vcluster pods were scheduled, add the -o wide flag to that previous command.
kubectl get pods -n vcluster-ha-tutorial -o wide
The hostnames of the nodes will be listed in the NODES column.
5. Connect to the virtual cluster
We can connect to the vcluster using the vcluster connect command.
vcluster connect ha-tutorial
info Starting proxy container...
done √ Switched active kube context to vcluster_ha-tutorial_vcluster-ha-tutorial_minikube
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
vcluster connect automatically switches our kube context for kubectl to the virtual cluster. Now we can see the namespaces inside of the virtual cluster by running this command:
kubectl get namespaces
Copy
NAME STATUS AGE
default Active 31s
kube-node-lease Active 33s
kube-public Active 33s
kube-system Active 33s
Our virtual cluster only contains the default namespaces that are created by Kubernetes.
Now let’s disconnect from the virtual cluster.
vcluster disconnect
This will switch your kube context back to the host cluster.
6. Cleanup
One of the great things about vcluster is that it’s very fast and easy to clean up the virtual clusters when you’re done using them.
vcluster delete ha-tutorial
That will delete the vcluster and the namespace it was in.
Other Examples
Enabling High Availability with k3s
In order to run vCluster with k3s as Kubernetes distribution in high availability mode, the following steps are required:
- Create and use an external datastore (as opposed to the embedded SQLite datastore used in single-server setups)
- Run two or more k3s pods that will serve the Kubernetes API and run other control plane services
First create a values.yaml in the following form and make sure to change the connection string in K3S_DATASTORE_ENDPOINT:
# Scale up k3s replicas
syncer:
replicas: 2
# Set external datastore endpoint
vcluster:
env:
- name: K3S_DATASTORE_ENDPOINT
value: mysql://username:password@tcp(hostname:3306)/database-name
# Disable persistent storage as all data (including bootstrap data) is stored in external datastore
syncer:
storage:
persistence: false
# Scale up CoreDNS replicas
coredns:
replicas: 2
Then create the vCluster with the following command:
vcluster create ... --connect=false -f values.yaml
Check that vCluster including the control plane is running correctly:
kubectl get pods -n vcluster
NAME READY STATUS RESTARTS AGE
coredns-66ffcc6b58-bhk4s-x-kube-system-x-vcluster 1/1 Running 0 21s
coredns-66ffcc6b58-n7npd-x-kube-system-x-vcluster 1/1 Running 0 21s
vcluster-54fb5dd76-92szq 2/2 Running 0 3m1s
vcluster-54fb5dd76-ntbrh 2/2 Running 0 3m1s
Now connect to the vCluster:
vcluster connect vcluster -n vcluster
# Then execute in a new terminal
export KUBECONFIG=kubeconfig.yaml
kubectl get ns
...
Check the GitHub repository for all available chart options.
Enabling High Availability with Vanilla k8s
In order to run vCluster in high availability mode, create a values.yaml in the following form:
# Scale up syncer replicas
syncer:
replicas: 3
# Scale up etcd
etcd:
replicas: 3
# Scale up DNS server
coredns:
replicas: 3
Then create the vCluster with the following command:
vcluster create ... --connect=false --distro k8s -f values.yaml
Check that vCluster including the control plane are running correctly:
kubectl get po -n vcluster
NAME READY STATUS RESTARTS AGE
coredns-6ff7df994d-m5pcd-x-kube-system-x-vcluster 1/1 Running 0 23m
coredns-6ff7df994d-dfgjb-x-kube-system-x-vcluster 1/1 Running 0 23m
coredns-6ff7df994d-weuir-x-kube-system-x-vcluster 1/1 Running 0 23m
vcluster-9d88f577-m55xf 1/1 Running 0 30m
vcluster-9d88f577-drsxz 1/1 Running 0 30m
vcluster-9d88f577-maslo 1/1 Running 0 30m
vcluster-etcd-0 1/1 Running 0 30m
vcluster-etcd-1 1/1 Running 0 29m
vcluster-etcd-2 1/1 Running 0 29m
Now connect to the vCluster:
vcluster connect vcluster-1 -n host-namespace-1
# Then execute in a new terminal
export KUBECONFIG=kubeconfig.yaml
kubectl get ns
...
Enable HA in rootless mode
Rootless mode means running vCluster without root user privileges in container, making host k8s cluster more secure. You can find more about rootless mode here.
Below is HA configuration for running rootless vCluster with vanilla Kubernetes distribution.
# Scale up syncer replicas
syncer:
replicas: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 12345
runAsNonRoot: true
runAsUser: 12345
seccompProfile:
type: RuntimeDefault
# Scale up etcd
etcd:
replicas: 3
fsGroup: 12345
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 12345
runAsNonRoot: true
runAsUser: 12345
seccompProfile:
type: RuntimeDefault
# Scale up DNS server
coredns:
replicas: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 12345
runAsNonRoot: true
runAsUser: 12345
seccompProfile:
type: RuntimeDefault
Check the github repository for all available chart options.