Skip to main content
Version: v0.34 Stable

Hybrid scheduling

Enterprise
Available in these plansFreeDevProdScale
Hybrid Scheduling
Supported Configurations
Running the control plane as a container with:
Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Hybrid scheduling enables tenant clusters to use multiple schedulers simultaneously - both from the control plane cluster and tenant cluster. Pods within the tenant cluster can use different schedulers based on their individual configuration.

You can enable hybrid scheduling using the following:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
  fromHost:
    nodes:
      enabled: true

You must also enable syncing of real nodes from the control plane cluster to the tenant cluster. If you don’t, the tenant cluster displays an error.

By default, all nodes from the control plane cluster are synced to the tenant cluster. To sync only specific nodes, you can define a label selector as shown in the following example:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
  fromHost:
    nodes:
      enabled: true
      labels:
        environment: production
        team: backend
Example

Checkout this guide that shows how to deploy custom host and virtual schedulers and use them with Hybrid scheduling.

Use host schedulers​

sync.toHost.pods.hybridScheduling.hostSchedulers specifies a list of schedulers available on the control plane cluster that tenant cluster pods can use.

The default Kubernetes scheduler on the control plane cluster is always implicitly included in the sync.toHost.pods.hybridScheduling.hostSchedulers list.

A default scheduler can be used in two ways: by omitting .spec.schedulerName or by explicitly setting it to default-scheduler. In both cases, the default Kubernetes scheduler running on the control plane cluster schedules the pod.

Use custom host schedulers​

The following hybrid scheduling configuration allows tenant cluster pods to use the control plane cluster's custom schedulers my-custom-scheduler and my-gpu-scheduler.

note

This example omits the full vCluster configuration to highlight the relevant hybrid scheduling settings.

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true
        hostSchedulers:
        - my-custom-scheduler
        - my-gpu-scheduler

You can now deploy pods in the tenant cluster that use the control plane cluster’s custom schedulers my-custom-scheduler and my-gpu-scheduler. For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-example
spec:
  schedulerName: my-custom-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
---
apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-example
spec:
  schedulerName: my-gpu-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10

Use custom virtual schedulers​

Tenant cluster pods can use a custom scheduler that runs inside the tenant cluster. To ensure these pods are scheduled correctly, do not include virtual schedulers in the sync.toHost.pods.hybridScheduling.hostSchedulers list in your vCluster configuration.

The following example shows the hybrid scheduling configuration and a pod that uses the my-virtual-ai-scheduler running in the tenant cluster.

Hybrid scheduling configuration:

sync:
  toHost:
    pods:
      hybridScheduling:
        enabled: true

Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-ai-example
spec:
  schedulerName: my-virtual-ai-scheduler
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10

When hybrid scheduling is enabled and PersistentVolumeClaims are synced from the tenant cluster to the control plane cluster, additional resources must also be present in the tenant cluster for scheduling to function correctly.

By default, vCluster automatically syncs the following resources from the control plane cluster to the tenant cluster, unless this behavior is explicitly disabled in the configuration:

  • CSINodes,
  • CSIStorageCapacities
  • CSIDrivers

When PersistentVolumeClaims are synced to the control plane cluster, vCluster also enables syncing of StorageClasses from the control plane cluster to the tenant cluster. However, this only happens if syncing StorageClasses from the tenant cluster to the control plane cluster is disabled in the configuration.

warning

Manually disabling CSINodes, CSIStorageCapacities, CSIDrivers and StorageClasses sync in the above cases can cause pod scheduling to fail in your tenant cluster.

vCluster automatically adds the required ClusterRole rules to ensure syncing works correctly for automatically enabled CSINodes, CSIStorageCapacities, CSIDrivers, and StorageClasses.