Skip to main content
Version: v0.24 Stable
Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Sleep mode

Not all workloads need to run all the time, and scaling them down saves time and money. With sleep mode you can scale down workloads based on a set schedule, or activity from users and ingress.

warning

Sleep mode is intended for pre-production use cases only, and comes with some limitations and caveats when used on a stand alone vCluster that is not connected to the Platform.

Enable sleep mode​

To enable sleep mode, use the following configuration inside your vcluster.yaml:

sleep mode configuration
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 1h

How it works​

Sleep mode operations​

Sleep mode performs two main operations:

Sleeping Deletes bare pods and scales down the following resources:

  • Deployments
  • ReplicaSets
  • ReplicationControllers
  • DaemonSets

Waking

  • Scales resources back to their original state
  • Cannot restore previously deleted bare pods

Resource exemption​

Resources can be exempted from sleep mode using by:

  • Adding the annotation sleepmode.loft.sh/exclude: true
  • Configuring sleepMode with specific labels
  • Adding configured labels to workloads that should keep running

Detecting activity​

To wake a sleeping cluster or to update the last active time, sleep mode captures the following:

  • Access of cluster resources through API calls (e.g., kubectl get <resource>)
  • Attempts to contact Ingress endpoints (NGINX only)
note

Ingress activity detection is only available for NGINX ingress controllers, making use of the mirror-target annotation. This has the effect of overwriting any previously set mirror-target annotation.

Ignoring other types of activity​

While resource exemption allows specifically configured resources to remain active when the vCluster goes to sleep, the following describes how to ignore requests that would otherwise denote the cluster should stay awake, or wake up if it is sleeping.

AnnotationBehaviorFormat
"sleepmode.loft.sh/ignore-all"ignore all activity"true" OR "false"
"sleepmode.loft.sh/ignore-ingresses"ignore requests to associated ingresses"true" OR "false"
"sleepmode.loft.sh/ignore-groups"ignore specified groups the requesting user belongs to"group2,group3"
"sleepmode.loft.sh/ignore-resources"ignore requests to specific resource types"pods,resource2,resource3"
"sleepmode.loft.sh/ignore-verbs"ignore requests with specific verbs"create,verb1,verb2"
"sleepmode.loft.sh/ignore-resource-verbs"ignore requests with specific verbs to specific resources"myresource.mygroup=create update delete, myresource2.mygroup=create update"
"sleepmode.loft.sh/ignore-resource-names"ignore requests to specific resources with specific names"myresource.mygroup=name1 name2,myotherresource.mygroup=name1"
"sleepmode.loft.sh/ignore-user-agents"ignore specific useragents with trailling wildcard support"kubectl*,argo,useragent3"
"sleepmode.loft.sh/disable-ingress-wakeup"ignore ingress access"true"
note

These annotations are to be added to the vCluster workload, e.g. the StatefulSet or Deployment running the virtual cluster, and any request can be ignored dynamically by adding the header X-Sleep-Mode-Ignore.

Differences and compatibility with the platform​

The documentation on this page describes two primary methods for sleep mode. The first is without an agent, where workloads are scaled down, but the control plane remains active to monitor for activity that should wake it up. When a vCluster is connected to the platform via an agent, the agent can fully shut down the control plane, reducing resource usage further. Additionally, auto-deletion is only available when an agent is present.

Connecting to the platform​

When a vCluster, configured for sleep mode, is connected to the platform, it ceases trying to manage its own sleep mode actions, allowing the agent to take over. This requires compatible versions between vCluster and platform. Below is a table of compatibility and additional steps needed to be performed when installing or uninstalling an agent.


vCluster VersionPlatform VersionCompatibilityInstalling an agentUninstalling an agent
0.22.x4.2.x✅Manually update vCluster config, moving it from experimental to external.platform. Translate any durations like 90m into equivalent seconds e.g. 5400. Schema validation prevents configuring both at the same time.Configuration needs to be manual reverted.
0.23.x4.2.x✅Manually update vCluster config, moving it from experimental to external.platform. Translate any durations like 90m into equivalent seconds e.g. 5400. Schema validation prevents configuring both at the same time.Configuration needs to be manual reverted.
0.24.0<4.3.0❌These versions are not compatible, as the vCluster version is ahead of the platform, causing the vcluster creation to be rejected.N/A
0.24.0≥4.3.0✅No action required. Platform reads the unified vCluster config and take over as though it had been configured for the platform all along.The annotation vcluster.loft.sh/agent-installed needs to be removed from the vCluster config secret in the host cluster to inform the vCluster it needs to take over sleep mode again. That secret's name has the form vc-config-<vcluster-name>.
≥0.24.0 (future release)≥4.3.0✅No action required. Platform reads the unified config and take over as though it had been configured for the platform all along.No action required, and the vCluster resumes sleep mode on only the workloads, not the control plane.
note
  • In 0.23.x, the platform won't know how to wake a vCluster that put itself to sleep, so the vCluster should be resumed, before installing the agent to take over.
  • In 0.24.x, the autoDelete configuration is still under external.platform to be moved under sleepMode in a future release.

Ingress configuration​

Sync to host​

If you install your ingress controllers in the vCluster you'll need to exempt the controller from sleeping so it remains active for requests that would wake the vCluster. If you install the Ingress controller in the host cluster, you'll need to enable syncing ingresses to the host.

enable ingress syncing
sync:
  toHost:
    ingresses:
      enabled: true

Reachability​

For proper ingress activity detection:

  • The vCluster pod must be discoverable by the ingress controller
  • DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local must resolve
  • Proper dnsPolicy configuration in the ingress controller Helm installation

Examples​

The below examples demonstrate how to configure sleep mode in a virtual cluster. Please make sure to install the necessary prerequisites before proceeding.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
    brew install loft-sh/tap/vcluster

    The binaries in the tap are signed using the Sigstore framework for enhanced security.

    Confirm that you've installed the correct version of the vCluster CLI.

    vcluster --version
  • docker: Container runtime installation guide.
  • kind: Kubernetes in Docker installation guide.
  • curl: Command-line tool for transferring data (pre-installed on most systems, see installation guide if needed)

Deployment​

Sleep mode with deployment resource.

Steps​

  1. Create the kind cluster

    create kind cluster
    kind create cluster --name sleep-mode-demo

  2. Deploy a virtual cluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s
        exclude:
          selector:
            labels:
              sleep: no-thanks

    And run:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

    Note that under the exclude section, workloads with the label sleep: no-thanks won't be put to sleep after the 30 seconds. So lets put that to the test.

  3. Create demo deployments in your virtual cluster.

    Use the following deployment yaml to create two deployments

    example deployments
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleepy-deployment
      labels:
        app: sleepy-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-1
      template:
        metadata:
          labels:
            app: demo-dep-1
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: sleepy-demo

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: no-sleep-deployment
      labels:
        sleep: no-thanks
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-2
      template:
        metadata:
          labels:
            app: demo-dep-2
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: not-sleepy-demo

    The first deployment has nothing special about it related to sleep mode. Feel free to use another in its place if you'd prefer. The second has the special label on the Deployment. As a result the Deployment won't be scaled down after the 30 seconds.

    You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

  4. Verify Deployments sleep status.

    deployment sleep check
    > sleep 30; kubectl get deployments
    NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    default       no-sleep-deployment   2/2     2            2           1m
    default       sleepy-deployment     0/2     0            0           1m

    The sleepy-deployment reports 0/2 replicas after the 30 seconds. The act of running kubectl counts as cluster activity, which is why its reporting 0/2 not 0/0. kubectl has triggered vCluster to update the replicas count back to the original 2, they just haven't become ready in the time it took for kubectl get ... to return.

Try the following​

Experiment with the sleep mode feature by trying the following:

  • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
  • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Ingress controller​

Sleep mode with ingress resource.

Steps​

  1. Create the kind cluster.

    create kind cluster
    kind create cluster --name ingress-demo --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      apiServerAddress: "0.0.0.0"
    nodes:
      - role: control-plane
        extraPortMappings:
          - containerPort: 80
            hostPort: 80
            protocol: TCP
          - containerPort: 443
            hostPort: 443
            protocol: TCP
    EOF

  2. Install the NGINX IngressController.

    install ingress controller
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.dnsPolicy=ClusterFirstWithHostNet \
      --set controller.hostNetwork=true \
      --set controller.service.type=ClusterIP

  3. Create the vCluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sync:
      toHost:
        ingresses:
          enabled: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s

    And run:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

  4. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

  5. Create resources.

    Create resources for the Ingress such as a Deployment and Service

    Use the following manifest to create

    • A new Namespace called bar
    • A Deployment for the pods backing the Service
    • A Service to back the Ingress
    • An Ingress
    example deployments
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bar

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bar-deployment
      namespace: bar
      labels:
        app: bar-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: bar
      template:
        metadata:
          labels:
            app: bar
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: bar-app

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: bar-service
      namespace: bar
    spec:
      selector:
        app: bar
      ports:
        # Default port used by the image
        - port: 8080

    ---

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: example-ingress
      namespace: bar
    spec:
      ingressClassName: nginx
      rules:
        - http:
            paths:
              - pathType: Prefix
                path: /bar
                backend:
                  service:
                    name: bar-service
                    port:
                      number: 8080
          host: backend.local

  6. Verify. Verify the ingress is working properly with curl.

    Keep trying the Ingress endpoint within the 30 second activity window with curl --silent backend.local/bar You should see the name of whichever pod in the Deployment responds.

  7. Put cluster into sleep mode. Allow the virtual cluster to go to sleep.

    Wait the 30 seconds for the cluster to sleep and try the curl command again. For convenience with this test you can run watch -d curl --silent backend.local/bar to continually try the endpoint. This time, because an HTTP request was sent to the HTTPS wake endpoint on the virtual cluster, you should see Client sent an HTTP request to an HTTPS server. on the first attempt, and new pod names on subsequent requests.

Additional examples​

Sleep mode label selectors and schedule.
Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 3h  # Uses Go's Duration with a max unit of hour
    exclude:
      selector:
        labels:
          dont: sleep
Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
sleepMode:
  enabled: true
  timeZone: America/Denver
  autoSleep:
    schedule: 30 17 * * 5
  autoWakeup:
    schedule: 0 7 * * 1

Config reference​

sleepMode required object pro​

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean pro​

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string pro​

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object pro​

AutoSleep holds autoSleep details

afterInactivity required string pro​

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string pro​

Schedule represents a cron schedule for when to sleep workloads

exclude required object pro​

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object pro​
labels required object pro​

Labels defines what labels should be looked for

autoWakeup required object pro​

AutoWakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string pro​