Skip to main content

Pro Feature

This feature is available in the vCluster Pro tier. Contact us for more details and to start a trial.

Native sleep mode

warning

Native sleep mode is intended for pre-production use cases only, and comes with some limitations and caveats and should not be enabled on virtual clusters that are using the current sleep mode configuration found in external.platform.

Not all workloads need to run all the time, and scaling them down saves time and money. With vCluster-native sleep mode you can scale down workloads based on a set schedule, or activity from users and ingress.

Enable native sleep mode​

To enable vCluster-native sleep mode, use the following configuration inside your vcluster.yaml:

native sleep mode configuration
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 1h

How it works​

Sleep mode operations​

Sleep mode performs two main operations:

Sleeping Deletes bare pods and scales down the following resources:

  • Deployments
  • ReplicaSets
  • ReplicationControllers
  • DaemonSets

Waking

  • Scales resources back to their original state
  • Cannot restore previously deleted bare pods

Resource exemption​

Resources can be exempted from sleep mode using either:

  • Adding the annotation sleepmode.loft.sh/exclude: true
  • Configuring sleepMode with specific labels
  • Adding configured labels to workloads that should keep running

Detecting activity​

To wake a sleeping cluster or to update the last active time, native sleep mode captures the following:

  • Access of cluster resources through API calls (e.g., kubectl get <resource>)
  • Attempts to contact Ingress endpoints (NGINX only)
note

Ingress activity detection is only available for NGINX ingress controllers, making use of the mirror-target annotation. This has the effect of overwriting any previously set mirror-target annotation.

Ingress configuration​

Sync to host​

If you install your ingress controllers in the vCluster you'll need to exempt the controller from sleeping so it remains active for requests that would wake the vCluster. If you install the Ingress controller in the host cluster, you'll need to enable syncing ingresses to the host.

enable ingress syncing
sync:
  toHost:
    ingresses:
      enabled: true

Reachability​

For proper ingress activity detection:

  • The vCluster pod must be discoverable by the ingress controller
  • DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local must resolve
  • Proper dnsPolicy configuration in the ingress controller Helm installation

Examples​

The below examples demonstrate how to configure sleep mode in a virtual cluster. Please make sure to install the necessary prerequisites before proceeding.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
    brew install loft-sh/tap/vcluster

    The binaries in the tap are signed using the Sigstore framework for enhanced security.

    Confirm that you've installed the correct version of the vCluster CLI.

    vcluster --version
  • docker: Container runtime installation guide.
  • kind: Kubernetes in Docker installation guide.
  • curl: Command-line tool for transferring data (pre-installed on most systems, see installation guide if needed)

Deployment​

Sleep mode with deployment resource.

Steps​

  1. Create the kind cluster

    create kind cluster
    kind create cluster --name sleep-mode-demo

  2. Deploy a virtual cluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    experimental:
      sleepMode:
        enabled: true
        autoSleep:
          afterInactivity: 30s
          exclude:
            selector:
              labels:
                sleep: no-thanks

    And run:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

    Note that under the exclude section, workloads with the label sleep: no-thanks won't be put to sleep after the 30 seconds. So lets put that to the test.

  3. Create demo deployments in your virtual cluster.

    Use the following deployment yaml to create two deployments

    example deployments
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleepy-deployment
      labels:
        app: sleepy-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-1
      template:
        metadata:
          labels:
            app: demo-dep-1
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: sleepy-demo

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: no-sleep-deployment
      labels:
        sleep: no-thanks
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-2
      template:
        metadata:
          labels:
            app: demo-dep-2
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: not-sleepy-demo

    The first deployment has nothing special about it related to sleep mode. Feel free to use another in its place if you'd prefer. The second has the special label on the Deployment. As a result the Deployment won't be scaled down after the 30 seconds.

    You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

  4. Verify Deployments sleep status.

    deployment sleep check
    > sleep 30; kubectl get deployments
    NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    default       no-sleep-deployment   2/2     2            2           1m
    default       sleepy-deployment     0/2     0            0           1m

    The sleepy-deployment reports 0/2 replicas after the 30 seconds. The act of running kubectl counts as cluster activity, which is why its reporting 0/2 not 0/0. kubectl has triggered vCluster to update the replicas count back to the original 2, they just haven't become ready in the time it took for kubectl get ... to return.

Try the following​

Experiment with the sleep mode feature by trying the following:

  • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
  • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Ingress controller​

Sleep mode with ingress resource.

Steps​

  1. Create the kind cluster.

    create kind cluster
    kind create cluster --name ingress-demo --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      apiServerAddress: "0.0.0.0"
    nodes:
      - role: control-plane
        extraPortMappings:
          - containerPort: 80
            hostPort: 80
            protocol: TCP
          - containerPort: 443
            hostPort: 443
            protocol: TCP
    EOF

  2. Install the NGINX IngressController.

    install ingress controller
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.dnsPolicy=ClusterFirstWithHostNet \
      --set controller.hostNetwork=true \
      --set controller.service.type=ClusterIP

  3. Create the vCluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sync:
      toHost:
        ingresses:
          enabled: true
    experimental:
      sleepMode:
        enabled: true
        autoSleep:
          afterInactivity: 30s

    And run:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

  4. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

  5. Create resources.

    Create resources for the Ingress such as a Deployment and Service

    Use the following manifest to create

    • A new Namespace called bar
    • A Deployment for the pods backing the Service
    • A Service to back the Ingress
    • An Ingress
    example deployments
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bar

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bar-deployment
      namespace: bar
      labels:
        app: bar-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: bar
      template:
        metadata:
          labels:
            app: bar
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: bar-app

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: bar-service
      namespace: bar
    spec:
      selector:
        app: bar
      ports:
        # Default port used by the image
        - port: 8080

    ---

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: example-ingress
      namespace: bar
    spec:
      ingressClassName: nginx
      rules:
        - http:
            paths:
              - pathType: Prefix
                path: /bar
                backend:
                  service:
                    name: bar-service
                    port:
                      number: 8080
          host: backend.local

  6. Verify. Verify the ingress is working properly with curl.

    Keep trying the Ingress endpoint within the 30 second activity window with curl --silent backend.local/bar You should see the name of whichever pod in the Deployment responds.

  7. Put cluster into sleep mode. Allow the virtual cluster to go to sleep.

    Wait the 30 seconds for the cluster to sleep and try the curl command again. For convenience with this test you can run watch -d curl --silent backend.local/bar to continually try the endpoint. This time, because an HTTP request was sent to the HTTPS wake endpoint on the virtual cluster, you should see Client sent an HTTP request to an HTTPS server. on the first attempt, and new pod names on subsequent requests.

Additional examples​

Sleep mode label selectors and schedule.
Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 3h  # Uses Go's Duration with a max unit of hour
      exclude:
        selector:
          labels:
            dont: sleep
Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
experimental:
  sleepMode:
    enabled: true
    timezone: America/Denver
    autoSleep:
      schedule: 30 17 * * 5
      wakeup:
        schedule: 0 7 * * 1

Config reference​

sleepMode required object pro​

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean pro​

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string pro​

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object pro​

AutoSleep holds autoSleep details

afterInactivity required string pro​

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string pro​

Schedule represents a cron schedule for when to sleep workloads

wakeup required object pro​

Wakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string pro​

exclude required object pro​

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object pro​
labels required object pro​

Labels defines what labels should be looked for