Skip to main content

Custom Resources


Pro Feature

This feature is available in the vCluster Pro tier. Contact us for more details and to start a trial.

vCluster allows you to sync custom resources from the virtual cluster to the host cluster. This allows you to sync arbitrary resources that are by default not synced by vCluster. This only works for resources that have a custom resource definition in the host cluster.

If those custom resources will create other resources inside the host cluster, vCluster will try to find them and sync them back to the host cluster as well. E.g. a cert-manager certificate creates a secret which will be synced back automatically into the virtual cluster.

vCluster will automatically add the required cluster and namespace RBAC permissions for retrieving the custom resource definition and syncing the resources from the virtual cluster to the host cluster.


Only Namespace-Scoped Resource

This feature currently only works for namespace-scoped resources only.

Multi-Namespace-Mode

If you want to sync many custom resources, consider using multi-namespace-mode.

Enable Custom Resource Syncing​

To enable custom resource syncing from the virtual cluster to the host cluster, figure out what CRDs you want to sync via kubectl get crds. Add the name into the customResources section in the sync section. Even though vCluster syncs custom resources from the virtual cluster to the host cluster, the CRDs are also copied from the host cluster to the virtual cluster.

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true

Patches​

You can modify the sync behaviour with patches that target specific paths. Currently there is 2 different kinds of patches supported.

Wildcard patches

You can use * in paths to select all entries of an array or object, e.g. spec.containers[*].name or spec.containers[*].volumeMounts[*]. vCluster calls the patch multiple times.

Reference patches​

A reference patch can be used to have a specific field of one resource point to a different resource that should get rewritten. vCluster automatically imports the referenced resource to the virtual cluster if it can find it in the host cluster. For example:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.secretName
          reference:
            apiVersion: v1
            kind: Secret

vCluster translates the path spec.secretName as it points to a secret. If the secret is created in the host cluster, vCluster automatically imports it into the virtual cluster.

Multi-Namespace-Mode

With multi-namespace-mode you only need to rewrite references that include a namespace. You can use the namespacePath option to specify the path of the namespace of the reference.

JavaScript Expression Patches​

These are powerful JavaScript ES6 compatible expression patches that can be used to change a field while syncing. You define how it changes when syncing from the virtual cluster into the host cluster or when syncing from the host cluster into the virtual cluster. To add a suffix to certificate DNS names you can do:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.dnsNames[*]
          # specifies the sync direction here again, because you can also react on change for fromHost with an expression
          expression: '"www."+value'
          # optional reverseExpression, if omitted patches from host will be discarded for that path
          # reverseExpression: 'value.slice("my-prefix".length)'

There is also a variable called context besides value that can be used to access vCluster specific data:

  • context.vcluster.name: Name of the virtual cluster
  • context.vcluster.namespace: Namespace of the virtual cluster
  • context.vcluster.config: Config of the virtual cluster, basically vcluster.yaml merged with the defaults
  • context.hostObject: Host object (can be null if not available)
  • context.virtualObject: Virtual object (can be null if not available)
  • context.path: The matched path on the object, useful when using wildcard path selectors (*)

For example, let's assume you want to add www. to every DNS name specified in a cert-manager certificate in the path spec.dnsNames, you could use the following patch:

sync:
  toHost:
    customResources:
      certificates.cert-manager.io:
        enabled: true
        patches:
        - path: spec.dnsNames[*]
          # specifies the sync direction here again, because you can also react on change for fromHost
          expression: "value.startsWith('www.') ? value : `www.${value}`"
          # specifies how to sync back changes to the virtual cluster. If omitted will not sync back changes.
          reverseExpression: "value.startsWith('www.') ? value.slice(4) : value"

With that patch, creating a new certificate within the vCluster like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: name-within-vcluster
spec:
  dnsNames:
    - example.com

Would create the following certificate in the host cluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: synced-name
spec:
  dnsNames:
    - www.example.com # the patch added www. to this field

When you change now the certificate in the host cluster like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: synced-name
spec:
  dnsNames:
    - www.other-domain.com # changed from www.example.com

vCluster would sync back the certificate like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: name-within-vcluster
spec:
  dnsNames:
    - other-domain.com # the patch removed the www. from www.other-domain.com

Full Example: Using cert-manager custom resource in Single-Namespace-Mode (default)​

1. Install cert-manager on your host cluster​

Ensure that you have cert-manager installed and running on your host cluster. Use your preferred method of installing cert-manager.

2. Install vCluster​

Use following vcluster.yaml to create virtual cluster on your host. Save this file as vcluster.yaml

sync:
  toHost:
    # we want to rewrite ingress cert-manager.io/issuer annotations
    ingresses:
      enabled: true
      patches:
        - path: metadata.annotations["cert-manager.io/issuer"]
          reference:
            apiVersion: cert-manager.io/v1
            kind: Issuer

    customResources:
      # sync certificates
      certificates.cert-manager.io:
        enabled: true
        patches:
          - path: spec.secretName
            reference:
              apiVersion: v1
              kind: Secret
          - path: spec.issuerRef
            reference:
              apiVersion: cert-manager.io/v1
              kind: Issuer   # defaults to Issuer
              kindPath: kind # to also allow ClusterIssuer
              namePath: name
          - path: status.nextPrivateKeySecretName
            reference:
              apiVersion: v1
              kind: Secret
      # sync issuers
      issuers.cert-manager.io:
        enabled: true
        patches:
          - path: spec.acme.privateKeySecretRef.name
            reference:
              apiVersion: v1
              kind: Secret

  fromHost:
    customResources:
      # sync cluster issuers in read-only mode
      clusterissuers.cert-manager.io:
        enabled: true
        scope: Cluster

And run:

vcluster create my-vcluster -f vcluster.yaml

After you started the vCluster with the above configuration you should see that the custom resource definitions have synced:

$ kubectl get customresourcedefinitions
NAME                             CREATED AT
certificates.cert-manager.io     2024-08-21T14:36:07Z
clusterissuers.cert-manager.io   2024-08-21T14:36:09Z
issuers.cert-manager.io          2024-08-21T14:36:08Z

3. Create Issuer and Certificate inside your virtual cluster​

We'll use a simple self signed certificate just to demonstrate vCluster capabilities. First, you'll need to create an Issuer resource:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
spec:
  selfSigned: {}
kubectl apply -f issuer.yaml

And then Certificate that uses our test-selfsigned Issuer:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
spec:
  secretName: test-cert-tls
  subject:
    organizations:
    - example.com
  commonName: example.com
  issuerRef:
    name: test-selfsigned
    kind: Issuer
kubectl apply -f cert.yaml

4. Validate that Secret was created inside your virtual cluster​

Thats it! You should have now test-cert-tls Secret available inside your virtual cluster! Just run:

kubectl get secret test-cert-tls

And you should see

NAME            TYPE                DATA   AGE
test-cert-tls   kubernetes.io/tls   3      2s

Full Example: Using cert-manager custom resource in Multi-Namespace-Mode​

1. Install cert-manager on your host cluster​

Ensure that you have cert-manager installed and running on your host cluster. Use your preferred method of installing cert-manager.

2. Install vCluster​

Use following vcluster.yaml to create virtual cluster on your host. Save this file as vcluster.yaml

experimental:
  multiNamespaceMode:
    enabled: true

sync:
  toHost:
    # sync all secrets
    secrets:
      all: true

    # sync ingresses and allow "cert-manager.io/issuer" annotation
    ingresses:
      enabled: true

    customResources:
      # sync cert-manager certificates
      certificates.cert-manager.io:
        enabled: true
      # sync cert-manager issuers
      issuers.cert-manager.io:
        enabled: true

  fromHost:
    customResources:
      # sync cert-manager cluster issuers
      clusterissuers.cert-manager.io:
        enabled: true
        scope: Cluster

And run:

vcluster create my-vcluster -f vcluster.yaml

After you started the vCluster with the above configuration you should see that the custom resource definitions have synced:

$ kubectl get customresourcedefinitions
NAME                             CREATED AT
certificates.cert-manager.io     2024-08-21T14:36:07Z
clusterissuers.cert-manager.io   2024-08-21T14:36:09Z
issuers.cert-manager.io          2024-08-21T14:36:08Z

3. Create Issuer and Certificate inside your virtual cluster​

We'll use a simple self signed certificate just to demonstrate vCluster capabilities. First, you'll need to create an Issuer resource:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
spec:
  selfSigned: {}
kubectl apply -f issuer.yaml

And then Certificate that uses our test-selfsigned Issuer:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
spec:
  secretName: test-cert-tls
  subject:
    organizations:
    - example.com
  commonName: example.com
  issuerRef:
    name: test-selfsigned
    kind: Issuer
kubectl apply -f cert.yaml

4. Validate that Secret was created inside your virtual cluster​

Thats it! You should have now test-cert-tls Secret available inside your virtual cluster! Just run:

kubectl get secret test-cert-tls

And you should see

NAME            TYPE                DATA   AGE
test-cert-tls   kubernetes.io/tls   3      2s

Config reference​

customResources required {key: object} pro​

CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work.

enabled required boolean pro​

Enabled defines if this option should be enabled.

scope required string pro​

Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported.

patches required object[] pro​

Patches patch the resource according to the provided specification.

path required string pro​

Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied.

expression required string pro​

Expression transforms the value according to the given JavaScript expression.

reverseExpression required string pro​

ReverseExpression transforms the value according to the given JavaScript expression.

reference required object pro​

Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name.

apiVersion required string pro​

APIVersion is the apiVersion of the referenced object.

apiVersionPath required string pro​

APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion.

kind required string pro​

Kind is the kind of the referenced object.

kindPath required string pro​

KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind.

namePath required string pro​

NamePath is the optional relative path to the reference name within the object.

namespacePath required string pro​

NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object.

labels required object pro​

Labels treats the path value as a labels selector.