Custom resources

Limited vCluster Tenancy Configuration Support

This feature is only available when using the following worker node types:

Host Nodes

Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

vCluster allows you to sync custom resources from the virtual cluster to the host cluster. This is useful for syncing resources that are not included in the default sync behavior.

This feature only works for resources that have a corresponding CustomResourceDefinition (CRD) installed in the host cluster.

If a synced custom resource creates additional resources in the host cluster, vCluster attempts to detect and sync those resources back to the virtual cluster. For example, a custom resource that creates a Secret has that Secret automatically synced into the virtual cluster.

vCluster also adds the necessary cluster and namespace-level RBAC permissions to retrieve the CRD and sync the corresponding resources.

Only Namespace-Scoped Resource

This feature currently only works for namespace-scoped resources only.

Namespace syncing

If you want to sync many custom resources, consider using namespace syncing.

Enable custom resource syncing

To enable custom resource syncing from the virtual cluster to the host cluster, first identify which CustomResourceDefinitions (CRDs) you want to sync by running kubectl get crds. Then, add the name of each CRD to the customResources section under sync in your vcluster.yaml configuration file.

Although the custom resources themselves are synced from the virtual cluster to the host cluster, vCluster also copies the corresponding CRDs from the host cluster into the virtual cluster. This ensures the virtual cluster can understand and manage those custom resources correctly.

By default, you do not need to specify an API version when syncing a custom resource. If the CRD defines multiple versions and no version is explicitly set, vCluster uses the storage version. To sync a specific version, such as when the storage version is not suitable, you can define that version directly in the vcluster.yaml.

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true

Versioned example

To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a /, using the format <resource>/<version> in your configuration.

The specified version must exist in the host cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the host cluster's storage version, vCluster automatically converts custom resources between the two versions.

For example, to use version v1 of the certificaterequests.cert-manager.io CRD, specify:

Configure CR sync to host

sync:
  toHost:
    customResources:
      certificaterequests.cert-manager.io/v1:
        enabled: true
        

Unique CRD versions

vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the virtual cluster fails to start.

Upgrade custom resources with explicit CRD API versions

vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the virtual cluster fails to start.

vCluster modifies the CRD inside the virtual cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the virtual cluster or change the sync configuration.

Upgrade from no version to specified version

When upgrading from an unversioned CRD to a versioned one (certificaterequests.cert-manager.io → certificaterequests.cert-manager.io/v1), vCluster checks the CRD's storage version. If the storage version is not v1, vCluster updates the CRD to include v1 as a new version in addition to the current storage version.

Upgrade from one specified version to another specified version

When upgrading between versioned CRDs (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io/v2), the CRD is updated to include the new version v2, and the old version v1 is kept as well.

Upgrade from a specified version to no version specified

When upgrading from a versioned CRD to an unversioned one (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io), the behavior depends on the host's storage version:

• If the CRD's storage version in the host is v1, nothing happens and the CRD in the virtual cluster remains as certificaterequests.cert-manager.io/v1.
• If the storage version is not v1, vCluster updates the CRD to include the storage version as a new version in addition to v1.

Patches

You can modify the sync behavior with patches that target specific paths. Currently, there are two different kinds of patches supported.

Wildcard patches

You can use * in paths to select all entries of an array or object, for example spec.containers[*].name or spec.containers[*].volumeMounts[*]. vCluster calls the patch multiple times.

Reference patches

You can use a reference patch to make a specific field in one resource point to another resource that vCluster should rewrite. If the referenced resource exists in the host cluster, vCluster automatically imports it into the virtual cluster.

This is useful when working with resources that reference other resources like Secrets:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.secretName
          reference:
            apiVersion: v1
            kind: Secret

vCluster translates the path spec.secretName as it points to a Secret. If the Secret is created in the host cluster, vCluster automatically imports it into the virtual cluster.

JavaScript expression patches

These are JavaScript ES6 compatible expression patches that can be used to change a field while syncing. You define how it changes when syncing from the virtual cluster into the host cluster or when syncing from the host cluster into the virtual cluster. To add a prefix to field values you can:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.names[*]
          # specifies the sync direction here again, because you can also react on change for fromHost with an expression
          expression: '"prefix-"+value'
          # optional reverseExpression, if omitted patches from host are discarded for that path
          # reverseExpression: 'value.slice("prefix-".length)'

There is also a variable called context besides value that can be used to access vCluster specific data:

• context.vcluster.name: Name of the virtual cluster
• context.vcluster.namespace: Namespace of the virtual cluster
• context.vcluster.config: Config of the virtual cluster, basically vcluster.yaml merged with the defaults
• context.hostObject: Host object (can be null if not available)
• context.virtualObject: Virtual object (can be null if not available)
• context.path: The matched path on the object, useful when using wildcard path selectors (*)

For example, to add a prefix to values in a custom resource field, you can use the following patch:

sync:
  toHost:
    customResources:
      customobjects.example.io:
        enabled: true
        patches:
        - path: spec.hosts[*]
          # specifies the sync direction here again, because you can also react on change for fromHost
          expression: "value.startsWith('prod-') ? value : `prod-${value}`"
          # specifies how to sync back changes to the virtual cluster. If omitted does not sync back changes.
          reverseExpression: "value.startsWith('prod-') ? value.slice(5) : value"

The patch creates a new custom object within the vCluster:

apiVersion: example.io/v1
kind: CustomObject
metadata:
  name: name-within-vcluster
spec:
  hosts:
    - server.example.com

vCluster syncs the host cluster and applies your patch, creating this modified resource:

apiVersion: example.io/v1
kind: CustomObject
metadata:
  name: synced-name
spec:
  hosts:
    - prod-server.example.com # the patch added prod- prefix to this field

If you directly edit the resource in the host cluster and change the value:

apiVersion: example.io/v1
kind: CustomObject
metadata:
  name: synced-name
spec:
  hosts:
    - prod-other-server.example.com # changed from prod-server.example.com

vCluster detects the change, applies the reverse patch, and updates the resource in your virtual cluster:

apiVersion: example.io/v1
kind: CustomObject
metadata:
  name: name-within-vcluster
spec:
  hosts:
    - other-server.example.com # the patch removed the prod- prefix

Configure Kubernetes Gateway API sync

To use the Kubernetes Gateway API with custom resources, install the Gateway API CRDs in the host cluster and then create the waypoint gateway.

Install Gateway CRD in the host

To create gateway resources, install the Gateway API CustomResourceDefinition in the host cluster if it's not already present:

Install Gateway CRD

kubectl --context="${HOST_CTX}" get crd gateways.gateway.networking.k8s.io &> /dev/null || \
kubectl --context="${HOST_CTX}" apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

Create waypoint gateway

Create a waypoint gateway configuration:

waypoint-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Apply it to your host cluster:

Create Waypoint Gateway

kubectl --context="${HOST_CTX}" create -f waypoint-gateway.yaml --namespace="${VCLUSTER_HOST_NAMESPACE}"

After it is configured, you can configure your custom resources to sync Gateway API resources between the virtual and host clusters.

Config reference

customResources required {key: object}

CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work.

enabled required boolean

Enabled defines if this option should be enabled.

scope required string

Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported.

patches required object[]

Patches patch the resource according to the provided specification.

path required string

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression required string

Expression transforms the value according to the given JavaScript expression.

reverseExpression required string

ReverseExpression transforms the value according to the given JavaScript expression.

reference required object

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string

APIVersion is the apiVersion of the referenced object.

apiVersionPath required string

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string

Kind is the kind of the referenced object.

kindPath required string

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath required string

NamePath is the optional relative path to the reference name within the object.

namespacePath required string

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels required object

Labels treats the path value as a labels selector.