Skip to main content
Version: main 🚧
Enterprise-Only Feature

This feature is an Enterprise feature. See our pricing plans or contact our sales team for more information.

Sleep mode

Not all workloads need to run continuously. Scaling them down saves time and reduces costs. With sleep mode, you can scale workloads based on a set schedule or user activity and ingress.

warning

Sleep mode is intended for pre-production use cases only, and has limitations when used on a standalone vCluster not connected to the platform.

Enable sleep mode​

To enable sleep mode, add the following configuration to your vcluster.yaml:

Sleep mode configuration
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 1h

Sleep mode operations​

Sleep mode involves two main actions: sleeping and waking. These actions help save resources when the cluster is not in use.

Sleeping reduces resource consumption and costs during inactive periods.

  • Delete bare pods: Pods not managed by controllers are removed permanently.
  • Scale down resources: The following resources are scaled down to reduce resource usage:
    • Deployments
    • ReplicaSets
    • ReplicationControllers
    • DaemonSets

Waking restore scaled-down resources.

  • Restore scaled-down resources: Deployments, ReplicaSets, ReplicationControllers, and DaemonSets are returned to their original state.
  • Bare pods are not restored: Bare pods deleted during the Sleeping step cannot be recovered.

Resource exemption in sleep mode​

In sleep mode, you can prevent specific resources from entering sleep mode by configuring them with the following options:

  • Add the annotation sleepmode.loft.sh/exclude: true to the resource.
  • Configure sleepMode with specific labels to define which resources should remain active. This allows targeting specific resources based on their labels.
  • Add labels to workloads that you want to remain active. By using labels, it is possible to control which resources are exempt from sleep mode at a more granular level.

Detect activity in sleep mode​

Sleep mode tracks certain actions to detect activity and wake the cluster when necessary. To wake a sleeping cluster or update the last active time, sleep mode captures the following actions:

  • Accessing cluster resources through API calls (for example kubectl get <resource>).
  • Attempting to contact Ingress endpoints (NGINX only).
note

Ingress activity detection works only with NGINX ingress controllers, using the mirror-target annotation. This overwrites any previously set mirror-target annotation.

Ignore other types of activity in sleep mode​

The resource exemption feature keeps specifically configured resources active when the vCluster enters sleep mode. The Configure request handling for sleep mode table describes how to ignore requests that keep the cluster awake or wake it up if it is asleep.

Configure request handling for sleep mode table
AnnotationBehaviorFormat
sleepmode.loft.sh/ignore-allIgnore all activity.true or false
sleepmode.loft.sh/ignore-ingressesIgnore requests to associated ingresses.true or false
sleepmode.loft.sh/ignore-groupsIgnore specified groups the requesting user belongs to.group2,group3
sleepmode.loft.sh/ignore-resourcesIgnore requests to specific resource types.pods,resource2,resource3
sleepmode.loft.sh/ignore-verbsIgnore requests with specific verbs.create,verb1,verb2
sleepmode.loft.sh/ignore-resource-verbsIgnore requests with specific verbs to specific resources.myresource.mygroup=create update delete, myresource2.mygroup=create update
sleepmode.loft.sh/ignore-resource-namesIgnore requests to specific resources with specific names.myresource.mygroup=name1 name2,myotherresource.mygroup=name1
sleepmode.loft.sh/ignore-user-agentsIgnore specific user agents with trailing wildcard support.kubectl*,argo,useragent3
sleepmode.loft.sh/disable-ingress-wakeupIgnore ingress access.true
note

Add the annotations to the vCluster workload, such as the StatefulSet or Deployment running the virtual cluster. You can dynamically ignore any request by adding the X-Sleep-Mode-Ignore header.

vCluster sleep mode compatibility with the platform​

You can configure sleep mode in a virtual cluster in two ways:

  • Without an agent: This scales down workloads (like pods) while keeping the control plane active. The cluster remains in a minimal state and can wake up when activity is detected.

  • With an agent: When connected to the platform using an agent, the agent fully manages sleep mode by shutting down the control plane. This reduces resource usage more and enables features like auto-deletion, which aren’t available without the agent.

When a vCluster connects to the platform using an agent, the agent manages sleep mode actions by shutting down the control plane completely. This reduces resource usage and enables features like auto-deletion, which is not available when the vCluster is not connected to the platform using an agent.

Connect to the platform​

When you connect a vCluster configured for sleep mode to the platform, it stops managing its own sleep mode actions and allows the agent to take control. This requires compatible versions of vCluster and the platform. The vCluster and platform version compatibility with agent installation table provides the steps needed for installing or uninstalling an agent.


vCluster and platform version compatibility with agent installation table
vCluster versionPlatform versionCompatibilityInstalling an agentUninstalling an agent
0.22.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
0.23.x4.2.x✅Update the vCluster config manually, moving it from experimental to external.platform. Convert durations, such as "90m", into seconds, for example, "5400". Schema validation prevents both configurations from being applied at the same time.Manually revert the configuration.
0.24.0<4.3.0❌These versions are not compatible, as the vCluster version is ahead of the platform, which causes the vCluster creation to be rejected.Not applicable.
0.24.0≥4.3.0✅No action is required. The platform reads the unified vCluster config and takes over as if it had been configured for the platform all along.Remove the annotation vcluster.loft.sh/agent-installed from the vCluster config secret in the host cluster to notify the vCluster that it needs to resume sleep mode. The secret’s name is vc-config-<vcluster-name>.
≥0.24.0 (future release)≥4.3.0✅No action is required. The platform reads the unified config and takes over as if it had been configured for the platform all along.No action is required, and the vCluster resumes sleep mode for workloads only, not the control plane.
vCluster sleep mode considerations
  • In version 0.23.x, the platform does not know how to wake a vCluster that has put itself to sleep. Ensure the vCluster is resumed before installing the agent to take over.
  • In version 0.24.x, the autoDelete configuration is still under external.platform and is set to move under sleepMode in a later release. external.platform to be moved under sleepMode in a future release.

Ingress configuration​

Sync to host​

If you install ingress controllers in the vCluster, prevent them from sleeping. This keeps the controllers active and allows them to handle requests that would wake the vCluster. If you install the Ingress controller in the host cluster, enable ingress syncing to the host.

Enable ingress syncing
sync:
  toHost:
    ingresses:
      enabled: true

Reachability for ingress activity detection​

To enable ingress activity detection in your vCluster, ensure the following conditions are met:

  • The ingress controller needs to be able to discover the vCluster pod to route traffic correctly.
  • Ensure that the DNS lookup for <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local resolves properly. This allows the ingress controller to reach the vCluster's service address.
  • Ensure that the correct dnsPolicy is set in the ingress controller's Helm chart. This ensures DNS resolution works as expected and allows communication between the ingress controller and the vCluster.

Configure sleep mode in a virtual cluster​

The following examples demonstrate how to configure sleep mode in a virtual cluster.

Ensure you install the necessary prerequisites.

Prerequisites​

  • Administrator access to a Kubernetes cluster: See Accessing Clusters with kubectl for more information. Run the command kubectl auth can-i create clusterrole -A to verify that your current kube-context has administrative privileges.

    info

    To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using kubectl config commands or authenticating through your cloud provider's CLI tools.

  • helm: Helm v3.10 is required for deploying the platform. Refer to the Helm Installation Guide if you need to install it.

  • kubectl: Kubernetes command-line tool for interacting with the cluster. See Install and Set Up kubectl for installation instructions.

  • vCluster: vCluster command-line tool to provision and manage virtual clusters. 
    brew install loft-sh/tap/vcluster

    The binaries in the tap are signed using the Sigstore framework for enhanced security.

    Confirm that you've installed the correct version of the vCluster CLI.

    vcluster --version
  • docker: Platform for building and running containerized applications.
  • kind: Tool for running local Kubernetes clusters in Docker.
  • curl: Command-line tool for transferring data over the network.

Deployment​

A deployment resource in Kubernetes manages a set of identical pods. Configuring sleep mode for a deployment scales down the pods while keeping the control plane active. This setup allows the virtual cluster to reduce resource usage while still being able to monitor activity and trigger wake-up actions when needed.

Configure sleep mode with a deployment resource

Configure sleep mode for a deployment resource​

  1. Create the kind cluster.

    create kind cluster
    kind create cluster --name sleep-mode-demo

  2. Deploy a virtual cluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s
        exclude:
          selector:
            labels:
              sleep: no-thanks

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

    Workloads with the label sleep: no-thanks don't enter sleep mode after 30 seconds.

  3. Create demo deployments in your virtual cluster.

    Use the following deployment YAML to create two deployments.

    Example deployments
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleepy-deployment
      labels:
        app: sleepy-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-1
      template:
        metadata:
          labels:
            app: demo-dep-1
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: sleepy-demo

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: no-sleep-deployment
      labels:
        sleep: no-thanks
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: demo-dep-2
      template:
        metadata:
          labels:
            app: demo-dep-2
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: not-sleepy-demo

    The first deployment does not have any special configurations for sleep mode. You can replace it with another deployment if needed. The second deployment includes a special label on the Deployment, preventing it from scaling down after 30 seconds.

    You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

  4. Verify Deployments sleep status.

    deployment sleep check
    > sleep 30; kubectl get deployments
    NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
    default       no-sleep-deployment   2/2     2            2           1m
    default       sleepy-deployment     0/2     0            0           1m

    The sleepy-deployment reports 0/2 replicas after 30 seconds. Running kubectl counts as cluster activity, which is why it reports 0/2 instead of 0/0. The kubectl command triggers vCluster to update the replica count back to the original value of 2, but the replicas haven't become ready by the time kubectl get ... returns.

Next steps​

Experiment with the sleep mode feature by trying the following:

  • Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
  • Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Ingress controller​

An ingress controller manages external HTTP/S traffic to services in a Kubernetes cluster. To configure sleep mode for the ingress controller, make sure it stays responsive to incoming traffic and can wake up the cluster if needed. This setup keeps the ingress controller active, even when the vCluster is in sleep mode and allows the controller to handle requests that trigger the virtual cluster to wake up.

Configure sleep mode with an ingress resource

Configure sleep mode for an ingress controller​

  1. Create the kind cluster.

    Create a kind cluster
    kind create cluster --name ingress-demo --config - <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    networking:
      apiServerAddress: "0.0.0.0"
    nodes:
      - role: control-plane
        extraPortMappings:
          - containerPort: 80
            hostPort: 80
            protocol: TCP
    EOF

  2. Install the NGINX IngressController.

    install ingress controller
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.dnsPolicy=ClusterFirstWithHostNet \
      --set controller.hostNetwork=true \
      --set controller.service.type=ClusterIP

  3. Create the vCluster.

    Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

    vCluster config for auto sleep
    pro: true
    sync:
      toHost:
        ingresses:
          enabled: true
    sleepMode:
      enabled: true
      autoSleep:
        afterInactivity: 30s

    And run the following command:

    Create vCluster with autoSleep config
    vcluster create my-vcluster -f vcluster.yaml

  4. Enable local DNS resolution for the virtual cluster.

    Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

  5. Create resources for the Ingress such as a Deployment and Service.

    Use the following manifest to create:

    • Namespace named bar
    • Deployment for the pods backing the Service
    • Service for the Ingress
    • Ingress resource
    Example deployments
    apiVersion: v1
    kind: Namespace
    metadata:
      name: bar

    ---

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bar-deployment
      namespace: bar
      labels:
        app: bar-dep
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: bar
      template:
        metadata:
          labels:
            app: bar
        spec:
          containers:
            - command:
                - /agnhost
                - serve-hostname
                - --http=true
                - --port=8080
              image: registry.k8s.io/e2e-test-images/agnhost:2.39
              name: bar-app

    ---

    kind: Service
    apiVersion: v1
    metadata:
      name: bar-service
      namespace: bar
    spec:
      selector:
        app: bar
      ports:
        # Default port used by the image
        - port: 8080

    ---

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: example-ingress
      namespace: bar
    spec:
      ingressClassName: nginx
      rules:
        - http:
            paths:
              - pathType: Prefix
                path: /bar
                backend:
                  service:
                    name: bar-service
                    port:
                      number: 8080
          host: backend.local

  6. Verify the ingress is working properly with curl.

    Test the Ingress endpoint within the 30-second activity window by running curl --silent backend.local/bar. The pod name from the Deployment that responds is displayed.

  7. Allow the virtual cluster to go to sleep.

    Wait 30 seconds for the cluster to sleep, then run the curl command again. For convenience, run watch -d curl --silent backend.local/bar to repeatedly test the endpoint. On the first attempt, you’ll see the message Client sent an HTTP request to an HTTPS server. because an HTTP request was sent to the HTTPS wake endpoint. Subsequent requests display new pod names.

Enable sleep mode with label selectors and schedules​

Use label selectors and schedules to configure sleep mode based on inactivity or specific timing:

Sleep mode label selectors and schedule
Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep
sleepMode:
  enabled: true
  autoSleep:
    afterInactivity: 3h  # Uses Go duration with a max unit of hour
    exclude:
      selector:
        labels:
          dont: sleep
Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone
sleepMode:
  enabled: true
  timeZone: America/Denver
  autoSleep:
    schedule: 30 17 * * 5
  autoWakeup:
    schedule: 0 7 * * 1

Config reference​

sleepMode required object pro​

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean pro​

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string pro​

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object pro​

AutoSleep holds autoSleep details

afterInactivity required string pro​

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string pro​

Schedule represents a cron schedule for when to sleep workloads

exclude required object pro​

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object pro​
labels required object pro​

Labels defines what labels should be looked for

autoWakeup required object pro​

AutoWakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string pro​