Kubernetes distro

info

vCluster uses Kubernetes as the default distribution for the virtual cluster.

By default, vClustervClusterAn open-source software product that creates and manages virtual Kubernetes clusters inside a host Kubernetes cluster. vCluster improves isolation and multi-tenancy capabilities while reducing infrastructure costs.Related: Virtual Cluster, Host Cluster uses an embedded SQLite as the backing store when you use the K8sK8sThe standard Kubernetes distribution that can be used in virtual clusters, offering full compatibility with upstream Kubernetes features.Related: K3s, K0s distribution.

warning

After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying.

Compatibility matrix

Host cluster compatibility

The Kubernetes distro of vCluster is completely independent of the Kubernetes distro of the host cluster. This means you can run vCluster using k8s distribution regardless of the distribution type of the host cluster.

vCluster supports specific Kubernetes versions on host clusters. For more information, see the Kubernetes compatibility matrix.

Configure Kubernetes version and image options

When using the K8s distribution, you can configure the container image for the Kubernetes components. The K8s distribution deploys the API serverAPI ServerThe core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects.Related: Control Plane, rate-limiting, controller manager, and scheduler as a single container from the loft-sh/kubernetes repository.

You can customize the container image by specifying the registry, repository, and tag in your vCluster configuration:

vcluster.yaml

controlPlane:
  distro:
    k8s:
      image:
        registry: ghcr.io           # Default: ghcr.io (can be overridden globally with controlPlane.advanced.defaultImageRegistry)
        repository: loft-sh/kubernetes  # Default: loft-sh/kubernetes
        tag: v1.32.1                # Default: v1.32.1 (or matches host Kubernetes version)

The tag field specifies the Kubernetes version to use. By default, it uses v1.32.1 or matches the host clusterHost ClusterThe physical Kubernetes cluster where virtual clusters are deployed and run. The host cluster provides the infrastructure resources (CPU, memory, storage, networking) that virtual clusters leverage, while maintaining isolation between different virtual environments.Related: Virtual Cluster's Kubernetes version.

note

The controlPlane.distro.k8s.version field is deprecated. Use controlPlane.distro.k8s.image.tag instead to specify the Kubernetes version.

Configure API server rate limiting

You can configure rate limiting for the vCluster API server to control the maximum number of requests it handles. This helps prevent the virtual clusterVirtual ClusterA certified Kubernetes distribution that runs as an isolated, virtual environment nested inside a physical host cluster. Virtual clusters run inside host cluster namespaces but operate as independent Kubernetes environments, each with its own API server, control plane, syncer, and set of resources.Related: vCluster, Host Cluster from being overwhelmed and ensures stable performance under high load.

Use the extraArgs field to pass rate limiting flags to the API server:

vcluster.yaml

controlPlane:
  distro:
    k8s:
      apiServer:
        extraArgs:
          - --max-requests-inflight=300
          - --max-mutating-requests-inflight=100

Rate limiting options

The Kubernetes API server supports these rate limiting flags:

• --max-requests-inflight: Maximum number of non-mutating requests (GET, LIST) that can be served at the same time. Default is 400.
• --max-mutating-requests-inflight: Maximum number of mutating requests (POST, PUT, PATCH, DELETE) that can be served at the same time. Default is 200.

Impact on host cluster requests

The --max-requests-inflight and --max-mutating-requests-inflight flags only indirectly affect the number of requests sent to the host cluster. With the exception of Service creation, vCluster handles all resource creation asynchronously through the syncer, which self-throttles at a default rate of 40 QPS.

API Priority and Fairness

Starting with Kubernetes 1.20, the API Priority and Fairness feature changes how these limits work. When enabled, the total concurrency limit becomes the sum of both flags, and requests are distributed across configurable priority levels.

Choose appropriate values

When setting rate limits, consider:

• Workload patterns: Higher limits for read-heavy workloads, balanced limits for mixed workloads
• Available resources: Ensure the host cluster has sufficient CPU and memory to handle the configured limits
• Client behavior: Applications with many concurrent API calls may need higher limits
• Platform limits: Some managed Kubernetes platforms (like EKS) may scale these values automatically

tip

Start with conservative values and monitor the API server's performance. Increase limits gradually based on actual usage patterns and available resources. Consider implementing resource quotas and CPU/memory limits alongside rate limiting for comprehensive resource management. You can also configure admission control policies to further control resource usage.

Config reference

k8s required object

K8S holds K8s relevant configuration.

enabled required boolean false

Enabled specifies if the K8s distro should be enabled. Only one distro can be enabled at the same time.

version required string

[Deprecated] Version field is deprecated. Use controlPlane.distro.k8s.image.tag to specify the Kubernetes version instead.

apiServer required object

APIServer holds configuration specific to starting the api server.

enabled required boolean true

Enabled signals this container should be enabled.

command required string[] []

Command is the command to start the distro binary. This will override the existing command.

extraArgs required string[] []

ExtraArgs are additional arguments to pass to the distro binary.

controllerManager required object

ControllerManager holds configuration specific to starting the controller manager.

enabled required boolean true

Enabled signals this container should be enabled.

command required string[] []

Command is the command to start the distro binary. This will override the existing command.

extraArgs required string[] []

ExtraArgs are additional arguments to pass to the distro binary.

scheduler required object

Scheduler holds configuration specific to starting the scheduler. Enable this via controlPlane.advanced.virtualScheduler.enabled

enabled required boolean false

Enabled signals this container should be enabled.

command required string[] []

Command is the command to start the distro binary. This will override the existing command.

extraArgs required string[] []

ExtraArgs are additional arguments to pass to the distro binary.

image required object

Image is the distro image

registry required string ghcr.io

Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub.

repository required string loft-sh/kubernetes

Repository is the repository of the container image, e.g. my-repo/my-image

tag required string v1.32.1

Tag is the tag of the container image, e.g. latest. If set to the default, it will use the host Kubernetes version.

imagePullPolicy required string

ImagePullPolicy is the pull policy for the distro image

env required object[] []

Env are extra environment variables to use for the main container and NOT the init container.

resources required object map[limits:map[cpu:100m memory:256Mi] requests:map[cpu:40m memory:64Mi]]

Resources for the distro init container

securityContext required object {}

Security options can be used for the distro init container