Skip to main content
Version: main 🚧

Custom Resources


Pro Feature

This feature is available in the vCluster Pro tier. Contact us for more details and to start a trial.

vCluster allows you to sync CustomResources from the host cluster to the virtual cluster. Resources are only be read by vCluster and then synced in read-only mode into the vCluster. vCluster copies the CRD itself in the beginning from the host to the virtual cluster and then start syncing the resources into the vCluster. This is especially helpful if you want to show certain resources inside the vCluster, such as ClusterIssuers (for cert-manager) or ClusterStores (for external-secrets). If you are looking to sync resources from the vCluster to the host cluster, see syncing custom resources to the host cluster

No need to configure RBAC

vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the resources from the host to the virtual cluster.

Cluster scoped example​

To configure vCluster to sync ClusterIssuers from the host cluster (from cert-manager):

configure Cluster scoped CRD sync from host
sync:
  fromHost:
    customResources:
      clusterissuers.cert-manager.io:
        enabled: true
        scope: Cluster

Namespace scoped CustomResourceDefinitions​

By default, this is turned off.

Enabling this allow you to sync namespaced CustomResources from the specified namespaces in the host to the specified namespaces in the vCluster.

configure Namespace scoped CRD sync from host
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CertificateRequests from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CertificateRequests names are unchanged.
            "foo/*": "bar/*"
  • It is also possible to modify the name of the synced resource in the virtual cluster.
  • There is no option to sync from all namespaces in the host.
  • Sync is one-directional, from host to virtual. If you modify an object in the host, vCluster syncs the change to virtual object.
  • When you delete virtual object, vCluster re-creates it if the host object still exist.
  • When you delete host object, vCluster deletes virtual object.

Namespaces in the virtual cluster are created automatically during the sync (if they do not exist already).

Prerequisites​

  • All the specified namespaces have to exist in the host at the vCluster startup.
  • CustomResourceDefinition needs to exist in the host at the vCluster startup.

Example​

No need to configure RBAC

vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the resources from the host to the virtual cluster.

For Namespace scoped CustomResources, you need to specify mappings.byName in the config. This tells vCluster which host resources should be synced and where (in the virtual cluster).

Namespaces in the virtual cluster are created automatically during the sync (if they do not exist already).

It is not possible to sync CustomResources that were already synced from virtual to host, they are skipped by vCluster.

Here is an example with cert-manager’s CertificateRequest CustomResource.

To sync all CustomResources from a given namespace in the host to the given namespace in the virtual cluster, “namespace/*” wildcard can be used, e.g.:

configure CustomResource sync from host namespace
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CertificateRequests from "foo" namespace
            # to the "bar" namespace in a virtual cluster. CertificateRequests names are unchanged.
            "foo/*": "bar/*"

To sync only specific CustomResources from namespaces, you need to provide namespace/name as the key and value:

configure CustomResource sync from host for one object
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CertificateRequest named "cm-name" from "foo" host namespace
            # to the "bar" namespace in virtual.
            "foo/cm-name": "bar/cm-name"

There is also a handy syntax to sync all CustomResources from virtual cluster’s own host namespace to the virtual namespace. As vCluster’s namespace is not always known upfront (e.g. when vCluster is created by the platform), "" (empty string) is treated as “vCluster’s own host namespace”.

configure CustomResource sync from host for virtual cluster's namespace
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs all CertificateRequests from virtual cluster's host namespace
            # to "my-virtual" namespace in a virtual cluster.
            "": "my-virtual"

you can also specify only a few CustomResources from virtual cluster’s own host namespace this way:

configure CustomResource sync from host for objects in virtual cluster's namespace
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs CertificateRequest named "my-cm" from virtual cluster's host namespace
            # to "my-virtual-namespace" in a virtual cluster.
            "/my-cm": "my-virtual/my-cm"

It’s also possible to modify CustomResource name during the sync:

configure CustomResource sync from host and modify name and namespace
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            # syncs "foo" CertificateRequest from "cert-manager" namespace in the host
            # as "my-foo" in "my-virtual" namespace in a virtual cluster.
            "cert-manager/foo": "my-virtual/my-foo"

Patches​

You can specify reverseExpression in the sync.fromHost.customResources[*].patches .

They are applied on the host object and appear in the virtual object.

expressions have no effect.

So, for the following vcluster.yaml :

configure CustomResource sync from host with patches
sync:
  fromHost:
    customResources:
      certificaterequests.cert-manager.io:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            "default/my-cm": "barfoo2/cm-my"
        patches:
          - path: metadata.annotations[*]
            # optional reverseExpression to reverse the change from the host cluster
            reverseExpression: "value.startsWith('www.') ? value.slice(4) : value"
  1. Your CustomResource in the host namespace default named my-cm is synced to the namespace barfoo2 in virtual and named cm-my.
  2. If default/my-cm host object has annotation which value starts with www. , e.g.: my-address: www.loft.sh then synced object in the virtual cluster barfoo2/cm-my has annotation my-address: loft.sh .

From host namespaced CustomResource example​

This guide shows how to sync k8s namespaced CustomResources from host cluster. Example CRD is used in this guide.

Set up cluster contexts​

Setting up the host and virtual cluster contexts makes it easier to switch between them.

set up kubectl contexts
export HOST_CTX="your-host-context"
export VCLUSTER_CTX="vcluster-ctx"

then, create a namespace in your host cluster, use foobar2 as an example:

create namespace
kubectl --context="${HOST_CTX}" create namespace foobar2
tip

You can find your contexts by running kubectl config get-contexts

Create CustomResourceDefinition in the host​

Saved following Custom Resource Definition:

example-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: examples.demo.loft.sh
spec:
  group: demo.loft.sh
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
      additionalPrinterColumns:
      - name: Image
        type: string
        description: The image of an example
        jsonPath: .spec.image
      - name: Replicas
        type: integer
        description: The number of replicas in example
        jsonPath: .spec.replicas
  scope: Namespaced
  names:
    plural: examples
    singular: example
    kind: Example

save this file locally and then apply it in the host cluster:

Create Example CRD in the host
kubectl --context="${HOST_CTX}" create -f example-crd.yaml

Enable from host syncing for your CustomResource​

Enable the from host syncing for Example CustomResources in your virtual cluster configuration:

Enable from host syncing for CustomResource
sync:
  fromHost:
    customResources:
      examples.demo.loft.sh:
        enabled: true
        scope: Namespaced
        mappings:
          byName:
            "": "default"

This configuration:

  • Enables from host syncing of the examples.demo.loft.sh from the vCluster's host namespace
  • Automatically configures RBAC permissions for vCluster, so it can get, watch and list Examples in vCluster's host namespace.
  • Syncs all Examples from vCluster host namespace to the default namespace in your vCluster.
create virtual cluster

Create or update a virtual Cluster following the vCluster quick start guide.

Sync namespaced CustomResource to virtual cluster​

  1. First, you create an example that you want to sync in the host cluster:

    Copy this file and save it locally as example-cr.yaml

    apiVersion: demo.loft.sh/v1
    kind: Example
    metadata:
      name: my-example
      namespace: vcluster
    spec:
      image: "my-image:latest"
      replicas: 2

    then, create an example in the host cluster:

    Create example in the host
    kubectl --context="${HOST_CTX}" create -f example-cr.yaml

    2. Ensure that CustomResource got synced to the virtual cluster​

    Your CustomResource should be now accessible in the virtual cluster. Keep in mind, that any edit made in the virtual object is overwritten by the host object data.

  2. Check CustomResource in the virtual cluster:

    Get synced CustomResource
    kubectl --context="${VCLUSTER_CTX}" et examples.demo.loft.sh --namespace default

    you should see similar output:

    examples in vCluster
    NAME         IMAGE             REPLICAS
    my-example   my-image:latest   2

    3. Edit CustomResource in the host​

  3. Now, you can edit your example in the host and see that the change is synced to the vCluster. To set replicas to 4, run:

    Patch Example CR
    kubectl --context="${HOST_CTX}" patch examples.demo.loft.sh my-example --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 4}]'  --namespace vcluster

    4. Verify that Example is updated in vCluster​

  4. Virtual Cluster Check number of replicas

    Check example
    kubectl --context="${VCLUSTER_CTX}" get --namespace default examples.demo.loft.sh

    you should see number of replicas updated from host object:

    Example updated in vCluster
    NAME         IMAGE             REPLICAS
    my-example   my-image:latest   4

Config reference​

customResources required {key: object} pro​

CustomResources defines what custom resources should get synced read-only to the virtual cluster from the host cluster. vCluster will automatically add any required RBAC to the vCluster cluster role.

enabled required boolean pro​

Enabled defines if this option should be enabled.

scope required string pro​

Scope defines the scope of the resource

patches required object[] pro​

Patches patch the resource according to the provided specification.

path required string pro​

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression required string pro​

Expression transforms the value according to the given JavaScript expression.

reverseExpression required string pro​

ReverseExpression transforms the value according to the given JavaScript expression.

reference required object pro​

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string pro​

APIVersion is the apiVersion of the referenced object.

apiVersionPath required string pro​

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string pro​

Kind is the kind of the referenced object.

kindPath required string pro​

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath required string pro​

NamePath is the optional relative path to the reference name within the object.

namespacePath required string pro​

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels required object pro​

Labels treats the path value as a labels selector.

mappings required object pro​

Mappings for Namespace and Object

byName required object pro​

ByName is a map of host-object-namespace/host-object-name: virtual-object-namespace/virtual-object-name. There are several wildcards supported:

  1. To match all objects in host namespace and sync them to different namespace in vCluster: byName: "foo/": "foo-in-virtual/"
  2. To match specific object in the host namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object"
  3. To match specific object in the host namespace and sync it to the same namespace with different name: byName: "foo/my-object": "foo/my-virtual-object"
  4. To match all objects in the vCluster host namespace and sync them to a different namespace in vCluster: byName: "": "my-virtual-namespace/*"
  5. To match specific objects in the vCluster host namespace and sync them to a different namespace in vCluster: byName: "/my-object": "my-virtual-namespace/my-object"