Deploy with Isolated vCluster Control Plane
This feature is available in the vCluster Pro tier. Contact us for more details and to start a trial.
vCluster creates isolated Kubernetes environments by deploying a virtual cluster that contains its own dedicated virtual control plane. This control plane orchestrates operations within the virtual cluster and facilitates interaction with the underlying host cluster. The virtual control plane is deployed as a single pod managed as a StatefulSet (default) or Deployment. By default, any workloads from a virtual cluster are deployed in the same namespace of a host cluster.
If you don't want to have your workloads running in the same namespace of the same host cluster, you can enable our isolated control plane feature, which allows you to run a vCluster control plane pod in one host cluster while a separate, headless vCluster runs workloads in another cluster. A headless vcluster is a vCluster without a control plane pod.
Separation of management and workload execution is often critical for security, management, or operational reasons.
The release names in both clusters must be the same to ensure that the vCluster can properly sync across them. In this example, they are both called my-vcluster
.
Part 1: Set up the Headless vCluster​
Create a Headless vCluster​
On the host cluster that will run the workloads, create a Headless vCluster with the following vcluster.yaml
.
experimental:
isolatedControlPlane:
headless: true
- vCluster CLI
- Helm
- Terraform
- Argo CD
- Cluster API
Install the vCluster CLI.
- Homebrew
- Mac (Intel/AMD)
- Mac (Silicon/ARM)
- Linux (AMD)
- Linux (ARM)
- Download Binary
- Windows Powershell
brew install loft-sh/tap/vcluster-experimental
If you installed the CLI using
brew install vcluster
, you shouldbrew uninstall vcluster
and then install the experimental version. The binaries in the tap are signed using the Sigstore framework for enhanced security.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.Download the binary for your platform from the GitHub Releases page and add this binary to your $PATH.
md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
$env:Path += ";" + $Env:APPDATA + "\vcluster";
[Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);Replace
<VCLUSTER_VERSION>
with the version you want to download.Reboot RequiredYou may need to reboot your computer to use the CLI due to changes to the PATH variable (see below).
Check Environment Variable $PATHLine 4 of this install script adds the install directory
%APPDATA%\vcluster
to the$PATH
environment variable. This is only effective for the current Powershell session, i.e. when opening a new terminal window,vcluster
may not be found.Make sure to add the folder
%APPDATA%\vcluster
to thePATH
environment variable after installing vcluster CLI via Powershell. Afterward, a reboot might be necessary.Confirm that you've installed the correct version of the vCluster CLI.
vcluster --version
Deploy vCluster.
vcluster create my-vcluster --namespace team-x --values vcluster.yaml
When the installation finishes, you are automatically connected to the virtual cluster. You Kubernetes context is updated to point to your new virtual cluster. You can run local
kubectl
commands for the new virtual cluster.
Deploy vCluster using
helm upgrade
command.helm upgrade --install my-vcluster vcluster \
--values vcluster.yaml \
--repo https://charts.loft.sh \
--namespace team-x \
--repository-config='' \
--create-namespace
Create a
main.tf
file.provider "helm" {
kubernetes {
config_path = "~/.kube/config"
}
}
resource "helm_release" "my_vcluster" {
name = "my-vcluster"
namespace = "team-x"
create_namespace = true
repository = "https://charts.loft.sh"
chart = "vcluster"
# If you didn't create a vcluster.yaml, remove the values section.
values = [
file("${path.module}/vcluster.yaml")
]
}Install the required Helm provider.
terraform init
Generate a plan.
terraform plan
Verify that the provider can access your cluster and that the proposed changes are correct.
Deploy vCluster.
terraform apply
To deploy vCluster using ArgoCD, you need the following files:
vcluster.yaml
for your configuration options of your vCluster.my-vcluster-app.yaml
for your ArgoCDApplication
definition.
Create the ArgoCD
Application
filemy-vcluster-app.yaml
, which refers to the vCluster Helm chart.apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-vcluster
namespace: argocd
spec:
project: default
source:
chart: vcluster
repoURL: https://charts.loft.sh
helm:
releaseName: my-vcluster
valueFiles:
- vcluster.yaml
destination:
server: https://kubernetes.default.svc
namespace: team-xCommit and push these files to your configured ArgoCD repository.
Synchronize your ArgoCD repository with your configured cluster.
Learn more about Cluster API Provider for vCluster.
Install the
clusterctl
CLI.Install the vCluster provider.
clusterctl init --infrastructure vcluster:v0.2.0
Export environment variables to be used by the cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which will deploy a vCluster.
export CLUSTER_NAME=my-vcluster
export CLUSTER_NAMESPACE=team-x
export VCLUSTER_YAML=$(awk '{printf "%s\\n", $0}' vcluster.yaml)Create the namespace for the vCluster.
kubectl create namespace ${CLUSTER_NAMESPACE}
Generate the required manifests and apply them.
clusterctl generate cluster ${CLUSTER_NAME} \
--infrastructure vcluster \
--target-namespace ${CLUSTER_NAMESPACE} \
| kubectl apply -f -Kubernetes VersionThe Kubernetes version for the vCluster is not set at the CAPI provider command. It is configured the
vcluster.yaml
file based on your Kubernetes distribution.Wait for vCluster to come up by watching for the vCluster custom resource to report a
ready
status.kubectl wait --for=condition=ready vcluster -n $CLUSTER_NAMESPACE $CLUSTER_NAME --timeout=300s
Obtain the kubeconfig of the Headless vCluster​
On the host cluster, you must obtain the kubeconfig of the Headless vCluster. This will be used to allow the control plane vCluster to communicate with it.
export WORKLOAD_KUBECONFIG=$(kubectl config view --raw -o json | TOKEN=$(kubectl create token vc-my-vcluster \
-n team-x \
--duration=48h \
) jq '."current-context" as $current | (.contexts[] | select(.name == $current)) as $context | {
"current-context": ."current-context",
"kind": ."kind",
"apiVersion": ."apiVersion",
"contexts": [$context],
"clusters": [.clusters[] | select(.name == $context.context.cluster) | del(.cluster."certificate-authority-data") | .cluster."insecure-skip-tls-verify" = true],
"users": [.users[] | select(.name == $context.context.user) | del(.user."client-certificate-data") | del(.user."client-key-data") | .user.token = $ENV.TOKEN]
}' | yq -P)
Confirm that your kube context is set to the host cluster and not the newly created Headless vCluster. If you used the vCluster CLI, use vcluster disconnect
to be
set back to your previous kube context.
Part 2: Set Up the Control Plane vCluster​
Switch to the kube context of the other Host​
Now, switch to the kube context of the host where the control plane will be deployed:
kubectl config use-context CONTEXT_NAME
Create the Namespace for the Control Plane​
Create a namespace in your cluster that will host the control plane vCluster:
kubectl create namespace vc-control-plane
Configure Kubeconfig Secret​
Using the kubeconfig obtained from the headless vCluster setup, create a Kubernetes secret in the control plane namespace.
kubectl create secret generic vc-remote-workloads-kubeconfig \
--namespace vc-control-plane \
--from-literal=kubeconfig=$WORKLOAD_KUBECONFIG
Optional: Configure Platform Kubeconfig Secret​
If you won't activate your vCluster through other means, you can manually activate it to ensure it is fully operational. This step is essential if you're setting up the vCluster for the first time or have yet to be automatically activated during deployment (by e.g. the CLI or platform section). Perform this step in the Kubernetes context of your control plane cluster:
kubectl create secret generic vcluster-platform-api-key \
--namespace vc-control-plane \
--from-literal=accessKey=$ACCESS_KEY \
--from-literal=host=$PLATFORM_HOST
You can also specify the project via project
key, if the platform is insecure via insecure
and the virtual cluster name in the platform via name
.
This command activates the my-vcluster
vCluster in the vc-control-plane
namespace.
Create the Control Plane vCluster​
Set up the control plane vCluster using the following vcluster.yaml
.
experimental:
isolatedControlPlane:
enabled: true
namespace: "team-x"
service: "my-vcluster"
kubeConfig: "/worker-cluster/kubeconfig.yaml"
controlPlane:
statefulSet:
persistence:
addVolumes:
- name: workload-kubeconfig
secret:
secretName: vc-remote-workloads-kubeconfig
items:
- key: kubeconfig
path: kubeconfig.yaml
addVolumeMounts:
- mountPath: /worker-cluster
name: workload-kubeconfig
readOnly: true
service:
spec:
type: LoadBalancer
networking:
advanced:
proxyKubelets:
byHostname: false
byIP: false
Setting up the control plane without Load Balancer support
By default vCluster starts a LoadBalancer
type Service to serve API. In some environments like local development clusters or bare metal clusters you might want to use NodePort
type instead. In order to do this, include this configuration in your vcluster.yaml
file.
controlPlane:
service:
spec:
type: NodePort
httpsNodePort: 30999
On the host cluster that will run the control, create a control plane vCluster.
- vCluster CLI
- Helm
- Terraform
- Argo CD
- Cluster API
Install the vCluster CLI.
- Homebrew
- Mac (Intel/AMD)
- Mac (Silicon/ARM)
- Linux (AMD)
- Linux (ARM)
- Download Binary
- Windows Powershell
brew install loft-sh/tap/vcluster-experimental
If you installed the CLI using
brew install vcluster
, you shouldbrew uninstall vcluster
and then install the experimental version. The binaries in the tap are signed using the Sigstore framework for enhanced security.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
Replace
<VCLUSTER_VERSION>
with the version you want to download.Download the binary for your platform from the GitHub Releases page and add this binary to your $PATH.
md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12';
Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/download/<VCLUSTER_VERSION>/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe;
$env:Path += ";" + $Env:APPDATA + "\vcluster";
[Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User);Replace
<VCLUSTER_VERSION>
with the version you want to download.Reboot RequiredYou may need to reboot your computer to use the CLI due to changes to the PATH variable (see below).
Check Environment Variable $PATHLine 4 of this install script adds the install directory
%APPDATA%\vcluster
to the$PATH
environment variable. This is only effective for the current Powershell session, i.e. when opening a new terminal window,vcluster
may not be found.Make sure to add the folder
%APPDATA%\vcluster
to thePATH
environment variable after installing vcluster CLI via Powershell. Afterward, a reboot might be necessary.Confirm that you've installed the correct version of the vCluster CLI.
vcluster --version
Deploy vCluster.
vcluster create my-vcluster --namespace team-x --values vcluster.yaml
When the installation finishes, you are automatically connected to the virtual cluster. You Kubernetes context is updated to point to your new virtual cluster. You can run local
kubectl
commands for the new virtual cluster.
Deploy vCluster using
helm upgrade
command.helm upgrade --install my-vcluster vcluster \
--values vcluster.yaml \
--repo https://charts.loft.sh \
--namespace team-x \
--repository-config='' \
--create-namespace
Create a
main.tf
file.provider "helm" {
kubernetes {
config_path = "~/.kube/config"
}
}
resource "helm_release" "my_vcluster" {
name = "my-vcluster"
namespace = "team-x"
create_namespace = true
repository = "https://charts.loft.sh"
chart = "vcluster"
# If you didn't create a vcluster.yaml, remove the values section.
values = [
file("${path.module}/vcluster.yaml")
]
}Install the required Helm provider.
terraform init
Generate a plan.
terraform plan
Verify that the provider can access your cluster and that the proposed changes are correct.
Deploy vCluster.
terraform apply
To deploy vCluster using ArgoCD, you need the following files:
vcluster.yaml
for your configuration options of your vCluster.my-vcluster-app.yaml
for your ArgoCDApplication
definition.
Create the ArgoCD
Application
filemy-vcluster-app.yaml
, which refers to the vCluster Helm chart.apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-vcluster
namespace: argocd
spec:
project: default
source:
chart: vcluster
repoURL: https://charts.loft.sh
helm:
releaseName: my-vcluster
valueFiles:
- vcluster.yaml
destination:
server: https://kubernetes.default.svc
namespace: team-xCommit and push these files to your configured ArgoCD repository.
Synchronize your ArgoCD repository with your configured cluster.
Learn more about Cluster API Provider for vCluster.
Install the
clusterctl
CLI.Install the vCluster provider.
clusterctl init --infrastructure vcluster:v0.2.0
Export environment variables to be used by the cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which will deploy a vCluster.
export CLUSTER_NAME=my-vcluster
export CLUSTER_NAMESPACE=team-x
export VCLUSTER_YAML=$(awk '{printf "%s\\n", $0}' vcluster.yaml)Create the namespace for the vCluster.
kubectl create namespace ${CLUSTER_NAMESPACE}
Generate the required manifests and apply them.
clusterctl generate cluster ${CLUSTER_NAME} \
--infrastructure vcluster \
--target-namespace ${CLUSTER_NAMESPACE} \
| kubectl apply -f -Kubernetes VersionThe Kubernetes version for the vCluster is not set at the CAPI provider command. It is configured the
vcluster.yaml
file based on your Kubernetes distribution.Wait for vCluster to come up by watching for the vCluster custom resource to report a
ready
status.kubectl wait --for=condition=ready vcluster -n $CLUSTER_NAMESPACE $CLUSTER_NAME --timeout=300s
Testing out the Isolated Control Plane​
In the kube context of the control plane vCluster, try deploying an application and see that the pods only run in the host cluster of the headless vCluster and not in the control plane vCluster.