Skip to main content
Version: main 🚧

Using Karpenter with vCluster Platform

vCluster is compatible with Karpenter and vCluster's features can be used in conjuntion with Karpenter to further reduce infrastructure costs.

This article explains how to create a designated node pool for a vCluster's workload pods, create a designated node pool for a project, and target node pool based on vCluster pod labels. All three of which can be powerful cost saving tools.

This article assumes you are using a control plane cluster provisioned on one of the following managed Kubernetes services: EKS (Elastic Kubernetes Service), AKS (Azure Kubernetes Service), or GKE (Google Kubernetes Engine).

The pattern <value> repeatedly occurs in this documentation and is meant to imply that the user should replace the value, angular brackets included, with their own.

Much of this article borrows and builds upon the Getting Started with Karpenter documentation, the Karpenter Provider Azure README.md, and the Karpenter Provider GCP documentation. Most if not all scripts and yaml manifests were written by the authors of those articles or articles of other AKS, Azure, and GCP documentation. Some of those works were then modified while others remain the same.

note

The Karpenter core API version and NodeClass kind differ by cloud provider. The EKS and AKS examples in this article use karpenter.sh/v1beta1. The GCP provider (cloudpilot-ai/karpenter-provider-gcp) requires the current karpenter.sh/v1 API and its own GCENodeClass kind. Follow only the tab that matches your control plane cluster's cloud provider, and don't mix fields between tabs.

Pre-requisites​

  1. Either an EKS Kubernetes cluster, an AKS Kubernetes cluster, or a GKE Standard cluster.
  2. vCluster Platform installed on control plane cluster. Read Deploy the vCluster Platform docs.
  3. EKS specific requirements for vCluster Platform compatibility:
    1. EBS driver installed. Get the Amazon EBS CSI driver add-on.
    2. Delete the gp2 StorageClass with kubectl delete storageclass gp2 and replace with gp3 by applying following yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
allowVolumeExpansion: true

  1. GKE specific requirements: enable the Compute Engine, GKE, and IAM Service Account Credentials APIs used by the Karpenter GCP provider:
gcloud services enable compute.googleapis.com container.googleapis.com iamcredentials.googleapis.com

Install Karpenter​

  1. Run helm install for karpenter:

Set KARPENTER_VERSION per the Karpenter getting started guide before running this command.

Modify the following with your specific values to generate a copyable command:
export KARPENTER_NAMESPACE=kube-system

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set "settings.clusterName=my-cluster" \
  --set "settings.interruptionQueue=my-cluster" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Configure control plane cluster Kubernetes RBAC permissions​

To deploy the manifests presented in the remainder of this article you will need certain permissions in the Kubernetes control plane cluster. The following Kubernetes permissions will be required in the control plane cluster for the user:

  • Create NodePools.karpenter.sh/v1beta1 in control plane cluster.
  • Create EC2NodeClass.karpenter.sh/v1beta1 in control plane cluster.

Create node pools​

Create designated node pool for a vCluster's workload pods​

Required permissions

The following permissions will be required in vCluster Platform:

  • Create clusters

Create node pool

This section utilizes taints and tolerations to guide cluster pods to the desired node. This works best if every node pool or node utilizes a taint. Otherwise, the pods may be scheduled on non-tainted nodes.

  1. Select "Create Cluster"
  2. Select "Configure" and add a toleration to cluster pods using the yaml editor. If the below yaml is out of date then see the applying tolerations docs.
sync:
  toHost:
    pods:
      enforceTolerations:
      - vclusterID=<vcluster-name>:NoSchedule
  1. Create a node pool with a taint matching the toleration and create node class for it to use.
cat <<EOF | kubectl apply -f -
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: <name-of-your-choosing>
spec:
  template:
    spec:
      taints:
      - key: vclusterID
        value:  <vcluster-id>
        effect: NoSchedule
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: <node-class-name>
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: Never
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: <node-class-name>
spec:
  amiFamily: AL2 # Amazon Linux 2
  role: "KarpenterNodeRole-<cluster-name>"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "<cluster-name>"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "<cluster-name>"
  amiSelectorTerms:
    - id: "<arm-ami-id>"
    - id: "<amd-ami-id>"
EOF
  1. Now we will confirm the node pool is working. Run kubectl get nodes --watch. Open a separate terminal tab. Connect to your vCluster by running vcluster connect <vcluster-id>. Then run:
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
EOF

kubectl scale deployment inflate --replicas 5

You should see nodes provision for the deployment. If you do the above on a vCluster without the tolerations then the nodes should not be provisioned on the node pool.

  1. Tear down the deployment used above.
kubectl delete deployment inflate

Create designated node pool for a project​

We will designate a node pool for a project by creating templates and enforcing the template for that project.

Required permissions

The following permissions will be required in vCluster Platform:

  • Create vCluster Templates
  • Create Projects

Create node pool

  1. Using the vCluster Platform add a template: Templates -> Clusters -> Add Cluster Template.
  2. In the yaml editor box, which contains the vcluster.yaml, add the following:
sync:
  toHost:
    pods:
      enforceTolerations:
      - vclusterID=<project-name>:NoSchedule
  1. Create a node pool with the taint corresponding to the toleration in step 2, e.g.:
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: <name-of-your-choosing>
spec:
  template:
    spec:
      taints:
      - key: <project-name>
        value: "true"
        effect: NoSchedule
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: <node-class-name>
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: Never
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: <node-class-name>
spec:
  amiFamily: AL2 # Amazon Linux 2
  role: "<karpenter-node-role>" # replace with your cluster name
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "<cluster-name>"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "<cluster-name>"
  amiSelectorTerms:
    - id: "<arm-ami-id>"
    - id: "<amd-ami-id>"
EOF
  1. Create a new project that only allows templates with set tolerations. Go to project dropdown and create a new project. In the Allowed Templates section, remove the All Templates option. Select the templates that include tolerations for the node pool created in step 3. Now, all pods created in clusters belonging to this project will deploy to the designated node pools.

Target node pool based on vCluster pod labels​

Required permissions

The following permissions will be required in vCluster Platform:

  • Create clusters

Create node pool

  1. Create a node pool with custom labels, for example my.org/environ.
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
        - key: my.org/environ
          operator: Exists
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: <node-class-name>
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: Never
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: <node-class-name>
spec:
amiFamily: AL2 # Amazon Linux 2
role: "<karpenter-node-role>" # replace with your cluster name
subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: "<cluster-name>"
securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: "<cluster-name>"
amiSelectorTerms:
  - id: "<arm-ami-id>"
  - id: "<amd-ami-id>"
EOF

  1. Create a cluster.

  2. Add nodeSelector to deployments in your cluster using a nodeSelector that targets the label from step 1 with the desired value.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      nodeSelector:
        my.org/environ: <any key, "prod", "dev", "test" make sense here>
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: 1
EOF

kubectl scale deployment inflate --replicas 5

If you make a deployment with a key/value pair that matches an existing node it can be scheduled to that node. Otherwise, a new node will be provisioned. Since you are adding the key to the node pool but not the value, using different values necessitates different nodes. You can read more about user-defined labels in the Karpenter's User-Defined Labels docs.