Custom resources

		Enterprise
Available in these plans	Free	Dev	Prod	Scale
Custom Resource Syncing

Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.

Supported Configurations

Running the control plane as a container with:

• Shared Nodes

vClustervClusterAn open-source software product that creates and manages tenant clusters within Kubernetes infrastructure. vCluster provides tenant isolation capabilities while reducing infrastructure costs.Related: Tenant cluster, Control plane cluster allows you to sync custom resources from the control plane cluster to the tenant clusterTenant clusterA fully isolated Kubernetes environment provisioned for a single tenant. Each tenant cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Tenant cluster. This feature creates read-only copies of custom resources inside your tenant cluster, making them available to applications and users without giving them direct access to the control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for tenant clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both.Related: Tenant cluster, Control plane cluster, Tenant cluster.

When you enable custom resource syncing, vCluster first makes the resource type available in the tenant cluster. It copies the control plane cluster's CustomResourceDefinition (CRD) for a CRD-backed resource. For a resource served by an extension API serverAPI ServerThe core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects.Related: Control Plane, rate-limiting, vCluster generates a schemaless CRD instead. It then begins syncing the resource instances. This is particularly useful for sharing cluster-wide resources or ClusterStores from external-secrets with tenant workloads. For details, see Aggregated API resources.

The synced resources are read-only in the tenant cluster. Changes in the control plane cluster automatically update the tenant copies. Tenant changes never propagate to the control plane cluster. vCluster removes tenant-created objects and restores tenant-deleted objects. Tenant field updates remain local until the same fields change in the control plane cluster.

If you need to sync resources from the tenant cluster to the control plane cluster instead, see syncing custom resources to the control plane cluster. For help choosing a direction, and for lifecycle and troubleshooting guidance, see Manage custom resources.

For a full worked example, see Sync a namespaced custom resource from the control plane cluster.

Automatic RBAC for configured resources

vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the configured resources. Extension API servers can require additional permissions for related resources. See Authorization on the control plane cluster.

Cluster-scoped example

Cluster-scoped custom resources exist at the cluster level and are not tied to any specific namespace. These are the simplest to sync because they don't require namespace mapping.

To sync cluster-scoped custom resources from the control plane cluster, configure the resource in your vcluster.yaml file:

configure cluster-scoped custom resource sync from host

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Cluster

By default, vCluster does not require you to specify an API version when syncing custom resources. When a CRD has multiple versions and you don't specify a version, vCluster uses the storage version.

If you want to sync a specific version of a CRD that is not the storage version, you can explicitly specify it in the sync configuration.

Aggregated API resources

Not every extension type in a Kubernetes cluster is backed by a CustomResourceDefinition. Kubernetes can also serve types through API aggregation, where an extension API server registered as an APIService owns the type instead of a CRD.

vCluster syncs these resources through the same customResources configuration. You don't declare that a resource comes from an extension API server. vCluster resolves every configured resource against the control plane cluster on startup:

• If the control plane cluster has a CRD for the resource, vCluster copies that CRD into the tenant cluster.
• If it doesn't, vCluster looks the resource up in the control plane cluster's discovery data. When an extension API server serves the resource, vCluster generates a schemaless CRD for it in the tenant cluster.
• If neither a CRD nor an extension API server serves the resource, the tenant cluster fails to start.

This resolution happens only on startup. If the extension API server isn't serving the resource then, vCluster fails to start and tries again when the control plane pod restarts. An extension API server that appears or recovers later is picked up on that next restart.

Because kubectl get crds doesn't list resources that an extension API server serves, use kubectl api-resources to find them.

Configure aggregated API resource sync from host

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Cluster

Generated CRDs for aggregated API resources

vCluster builds the generated CRD from the control plane cluster's discovery data. The kind, plural name, scope, and presence of a status subresource come from the control plane cluster. vCluster derives the singular name by lowercasing the kind and derives the list kind by appending List. It marks the CRD with the vcluster.loft.sh/generated-from: aggregated-apiserver annotation and creates or updates it when the tenant cluster starts.

Objects of a generated CRD live in the tenant cluster's own data store, like any other custom resource, so snapshots include them.

The generated CRD has no resource-specific OpenAPI schema and preserves unknown fields. The tenant API server doesn't validate custom fields against the extension API server's schema. Keep the following behaviors in mind:

• kubectl explain returns no field documentation for the resource inside the tenant cluster.
• The tenant cluster doesn't prune unknown fields.
• When the resource key includes no version, vCluster uses the API group's preferred version on the control plane cluster. For CRD-backed resources, it uses the storage version instead.
• When you change the version in the resource key, vCluster serves the new version and retains previously stored versions without serving them.

vCluster uses the vcluster.loft.sh/generated-from: aggregated-apiserver annotation as the adoption marker. If an existing CRD has this marker, vCluster adopts it and updates its specification. If a CRD of the same name doesn't have this marker, the tenant cluster fails to start. Delete that CRD or remove the resource from customResources.

Removing an entry from customResources, or setting enabled: false, stops the syncer. vCluster doesn't delete the generated CRD, its objects, or the objects it created on the control plane cluster. The tenant cluster keeps serving the resource, and writes to it stay tenant-local because nothing validates or syncs them.

Authorization on the control plane cluster

vCluster grants itself the permissions it needs for every resource listed in customResources. Some extension API servers authorize a request against a related resource that you never configure for sync. For example, an API server can check access to a grouping resource before it serves or accepts an object. Grant those permissions explicitly. This example grants access to a related resource in the vCluster control plane namespace:

Grant extra permissions for an extension API server

rbac:
  role:
    extraRules:
      - apiGroups: ["example.io"]
        resources: ["customobjectgroups"]
        verbs: ["get", "list", "watch"]

Use rbac.role.extraRules for resources in the vCluster control plane namespace. Use rbac.clusterRole.extraRules for cluster-scoped resources or namespaced resources in other control plane cluster namespaces. For details, see Add custom RBAC rules.

Read-only sync of aggregated API resources

vCluster generates a CRD in the tenant cluster for cluster-scoped and namespace-scoped aggregated API resources alike. Namespace-scoped resources still require mappings.byName, the same as CRD-backed ones.

The synced copies are read-only in the tenant cluster. To let tenant workloads create and modify a resource that an extension API server serves, sync it in the other direction with syncing custom resources to the control plane cluster.

Limitations

The Kubernetes API accepts standard CRD operations for an aggregated API resource, but from-host reconciliation remains read-only. vCluster removes tenant-created objects and restores tenant-deleted objects. Tenant field updates remain local until the same fields change in the control plane cluster.

The generated CRD serves the resource itself, plus the status subresource when the control plane cluster serves it. Extension API servers can expose additional subresources, such as scale, logs, or exec. Those aren't available inside the tenant cluster, because a CRD can't serve them.

Versioned example

To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a /, using the format <resource>/<version> in your configuration.

The specified version must exist in the control plane cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the control plane cluster's storage version, vCluster automatically converts custom resources between the two versions.

For example, to use version v1 of the certificaterequests.cert-manager.io CRD, specify:

Configure cluster-scoped CR sync from host

sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io/v1:
        enabled: true
        scope: Cluster

Unique CRD versions

vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the tenant cluster fails to start.

Upgrade custom resources with explicit CRD API versions

vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the tenant cluster fails to start.

vCluster modifies the CRD inside the tenant cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the tenant cluster or change the sync configuration.

Upgrade from no version to specified version

When upgrading from an unversioned CRD to a versioned one (certificaterequests.cert-manager.io → certificaterequests.cert-manager.io/v1), vCluster checks the CRD's storage version. If the storage version is not v1, vCluster updates the CRD to include v1 as a new version in addition to the current storage version.

Upgrade from one specified version to another specified version

When upgrading between versioned CRDs (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io/v2), the CRD is updated to include the new version v2, and the old version v1 is kept as well.

Upgrade from a specified version to no version specified

When upgrading from a versioned CRD to an unversioned one (certificaterequests.cert-manager.io/v1 → certificaterequests.cert-manager.io), the behavior depends on the host's storage version:

• If the CRD's storage version in the control plane cluster is v1, nothing happens and the CRD in the tenant cluster remains as certificaterequests.cert-manager.io/v1.
• If the storage version is not v1, vCluster updates the CRD to include the storage version as a new version in addition to v1.

Namespace-scoped custom resources

By default, namespace-scoped custom resource syncing is disabled.

Enabling this feature allows you to sync namespaced custom resources from specific namespaces in the control plane cluster to corresponding namespaces in the tenant cluster.

Configure namespace-scoped custom resource sync from host

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged.
            "foo/*": "bar/*"

Key behaviors for namespace scoped syncing:

• You can modify the name of the synced resource in the tenant cluster.
• You cannot sync from all namespaces in the control plane cluster.
• Sync is one-directional, from the control plane cluster to the tenant cluster. If you modify an object in the control plane cluster, vCluster syncs the change to the tenant object.
• When you delete a tenant object, vCluster re-creates it if the control plane cluster object still exists.
• When you delete a control plane cluster object, vCluster deletes the tenant object.

vCluster creates namespaces in the tenant cluster automatically during the sync if they don't already exist.

Prerequisites

Ensure you have the following:

• All specified namespaces must exist in the control plane cluster when vCluster starts.
• The control plane cluster must serve the resource when vCluster starts, either through a CustomResourceDefinition or through an extension API server.

Aggregated API resources

Namespace-scoped resources served by an extension API server sync the same way, including the mappings.byName requirement. See Aggregated API resources for the generated CRD and its limitations.

Example

For namespace scoped custom resources, you must specify mappings.byName in the config. This tells vCluster which control plane cluster resources should be synced and where to place them in the tenant cluster.

vCluster can't sync custom resources that were already synced from the tenant cluster to the control plane cluster. It skips these resources.

The following is an example with a generic custom resource.

To sync all custom resources from a given namespace in the control plane cluster to a given namespace in the tenant cluster, use the "namespace/*" wildcard:

Configure custom resource sync from host namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged.
            "foo/*": "bar/*"

To sync only specific custom resources from namespaces, provide namespace/name as the key and value:

Configure custom resource sync from host for one object

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CustomObject named "my-object" from "foo" host namespace
            # to the "bar" namespace in virtual.
            "foo/my-object": "bar/my-object"

There is also a syntax to sync all custom resources from the tenant cluster's own control plane cluster namespace to the tenant namespace. Since the tenant cluster's namespace is not always known upfront (for example, when vCluster is created by the platform), "" (empty string) is treated as "tenant cluster's own control plane cluster namespace":

configure custom resource sync from host for tenant cluster's namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CustomObjects from virtual cluster's host namespace
            # to "my-virtual" namespace in a virtual cluster.
            "": "my-virtual"

You can also specify only a few custom resources from the tenant cluster's own control plane cluster namespace:

configure custom resource sync from host for objects in tenant cluster's namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CustomObject named "my-object" from virtual cluster's host namespace
            # to "my-virtual-namespace" in a virtual cluster.
            "/my-object": "my-virtual/my-object"

It's also possible to modify the custom resource name during the sync:

configure custom resource sync from host and modify name and namespace

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs "foo" CustomObject from "system" namespace in the host
            # as "my-foo" in "my-virtual" namespace in a virtual cluster.
            "system/foo": "my-virtual/my-foo"

Patches

Use sync.fromHost.customResources.<resource>.patches to transform custom resource fields while syncing from the control plane cluster to the tenant cluster. For from-host sync, use reverseExpression for values that should appear in the tenant cluster. See Patching synced resources for syntax, directionality, and custom resource labels patches.

configure custom resource sync from host with patches

sync:
  fromHost:
    customResources:
      customobjects.example.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            "default/my-object": "barfoo2/object-my"
        patches:
          - path: metadata.annotations[*]
            # optional reverseExpression to reverse the change from the host cluster
            reverseExpression: "value.startsWith('www.') ? value.slice(4) : value"

Config reference

customResources {key: object}

CustomResources defines what custom resources should get synced read-only to the virtual cluster from the host cluster. vCluster will automatically add any required RBAC to the vCluster cluster role.

enabled required boolean

Enabled defines if this option should be enabled.

scope required string

Scope defines the scope of the resource

patches object[]

Patches patch the resource according to the provided specification.

path required string

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression string

Expression transforms the value according to the given JavaScript expression.

reverseExpression string

ReverseExpression transforms the value according to the given JavaScript expression.

reference object

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string

APIVersion is the apiVersion of the referenced object.

apiVersionPath string

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string

Kind is the kind of the referenced object.

kindPath string

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath string

NamePath is the optional relative path to the reference name within the object.

namespacePath string

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels object

Labels treats the path value as a labels selector.

mappings object

Mappings for Namespace and Object

byName object

ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:

1. To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/": "foo-in-virtual/"
2. To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object"
3. To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object"
4. To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/*"
5. To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object"