Native Sleep Mode

Pro Feature

This feature is available in the vCluster Pro tier. Contact us for more details and to start a trial.

warning

Native sleep mode is intended for pre-production use cases only, and comes with some limitations and caveats.

How it works

Sleeping deletes bare pods and scales down the following resources:

• Deployments
• ReplicaSets
• ReplicationControllers
• DaemonSets

Waking scales sleeping resources back up, however bare pods that were killed cannot be restored. To exempt resources from being put to sleep, either add the annotation sleepmode.loft.sh/exclude: true or configure sleepMode with a label, or set of labels to exclude and add those labels to all the workloads you'd like to keep running.

Detecting activity

To wake a sleeping cluster or to update the last active time, native sleep mode captures the following:

• Access of cluster resources. Think kubectl get <resource>
• Attempts to contact Ingress endpoints (NGINX only).

note

Ingress activity detection is only available for NGINX ingress controllers, making use of the mirror-target annotation. This has the effect of overwriting any previously set mirror-target annotation.

Ingress considerations

Sync to host

If you install your ingress controllers in the vCluster you'll need to exempt the controller from sleeping so it remains active for requests that would wake the vCluster. If you install the Ingress controller in the host cluster, you'll need to enable syncing ingresses to the host.

enable ingress syncing

sync:
  toHost:
    ingresses:
      enabled: true

Reachability

In order to detect ingress activity, the vCluster pod must be discoverable by the ingress controller. i.e. an nslookup for the <vcluster-namespace>.<vcluster-svc-name>.svc.cluster.local should be able to find the vCluster service. If you're installing the ingress controller with Helm be sure to set the dnsPolicy correctly, as is covered in the ingress example.

Full example with Deployments

Tools used in this example

• Docker
• Kind

Steps in this example

• Create a kind cluster
• Create a virtual cluster with sleepMode enabled, and a label exclusion configured.
• Install two Deployments with 2 replicas to verify sleep/wake, one with labels to exclude it from sleeping.
• Show one Deployment is scaled down after the 30 second inactivity timeout, while the other remains active.
• Show that the slept Deployment wakes when the cluster is accessed with kubectl.

1. Create the kind cluster

create kind cluster

kind create cluster --name sleep-mode-demo

2. Create the vCluster

Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

vCluster config for auto sleep

pro: true
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 30s
      exclude:
        selector:
          labels:
            sleep: no-thanks

And run:

Create vCluster with autoSleep config

vcluster create my-vcluster -f vcluster.yaml

Note that under the exclude section, workloads with the label sleep: no-thanks won't be put to sleep after the 30 seconds. So lets put that to the test.

3. Create a couple demo deployments in your virtual cluster

Use the following deployment yaml to create two deployments

example deployments

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleepy-deployment
  labels:
    app: sleepy-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-dep-1
  template:
    metadata:
      labels:
        app: demo-dep-1
    spec:
      containers:
        - command:
            - /agnhost
            - serve-hostname
            - --http=true
            - --port=8080
          image: registry.k8s.io/e2e-test-images/agnhost:2.39
          name: sleepy-demo

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-sleep-deployment
  labels:
    sleep: no-thanks
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo-dep-2
  template:
    metadata:
      labels:
        app: demo-dep-2
    spec:
      containers:
        - command:
            - /agnhost
            - serve-hostname
            - --http=true
            - --port=8080
          image: registry.k8s.io/e2e-test-images/agnhost:2.39
          name: not-sleepy-demo

The first deployment has nothing special about it related to sleep mode. Feel free to use another in its place if you'd prefer. The second has the special label on the Deployment. As a result the Deployment won't be scaled down after the 30 seconds.

You can verify this by waiting 30 seconds and then getting information about the Deployments. For example

4. Verify Deployments sleep status

deployment sleep check

> sleep 30; kubectl get deployments
NAMESPACE     NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
default       no-sleep-deployment   2/2     2            2           1m
default       sleepy-deployment     0/2     0            0           1m

The sleepy-deployment reports 0/2 replicas after the 30 seconds. The act of running kubectl counts as cluster activity, which is why its reporting 0/2 not 0/0. kubectl has triggered vCluster to update the replicas count back to the original 2, they just haven't become ready in the time it took for kubectl get ... to return.

Things to try on your own with this setup

• Add the sleep: no-thanks label to the first deployment and verify neither sleeps.
• Remove the sleep: no-thanks label from both the deployments and verify that both go to sleep.

Full example with host Ingress Controller

Tools used in this example

• Docker
• Kind
• Helm
• curl

Steps in this example

• Create a kind cluster with ports necessary for demonstrating Ingress capabilities.
• Install an NGINX ingress controller on the host cluster, with syncing to host enabled.
• Create a virtual cluster with sleepMode enabled
• Install several components in the virtual cluster to test the ingress
• Test the ingress before it goes to sleep
• Test the ingress after it has gone to sleep to show that it wakes

1. Create the kind cluster

create kind cluster

kind create cluster --name ingress-demo --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: "0.0.0.0"
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP
EOF

2. Install the NGINX IngressController

install ingress controller

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.dnsPolicy=ClusterFirstWithHostNet \
  --set controller.hostNetwork=true \
  --set controller.service.type=ClusterIP

3. Create the vCluster

Use the following vcluster.yaml to create a virtual cluster on your host. Save this file as vcluster.yaml

vCluster config for auto sleep

pro: true
sync:
  toHost:
    ingresses:
      enabled: true
experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 30s

And run:

Create vCluster with autoSleep config

vcluster create my-vcluster -f vcluster.yaml

4. Edit your /etc/hosts for the Ingress domain

Add 127.0.0.1 backend.local to your /etc/hosts file to match the host configured in the Ingress rules of the next step.

5. Create resources for the Ingress such as a Deployment and Service

Use the following manifest to create

• A new Namespace called bar
• A Deployment for the pods backing the Service
• A Service to back the Ingress
• An Ingress

example deployments

apiVersion: v1
kind: Namespace
metadata:
  name: bar

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar-deployment
  namespace: bar
  labels:
    app: bar-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bar
  template:
    metadata:
      labels:
        app: bar
    spec:
      containers:
        - command:
            - /agnhost
            - serve-hostname
            - --http=true
            - --port=8080
          image: registry.k8s.io/e2e-test-images/agnhost:2.39
          name: bar-app

---

kind: Service
apiVersion: v1
metadata:
  name: bar-service
  namespace: bar
spec:
  selector:
    app: bar
  ports:
    # Default port used by the image
    - port: 8080

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: bar
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /bar
            backend:
              service:
                name: bar-service
                port:
                  number: 8080
      host: backend.local

6. Verify the ingress is working properly with curl

Keep trying the Ingress endpoint within the 30 second activity window with curl --silent backend.local/bar You should see the name of whichever pod in the Deployment responds.

7. Allow the virtual cluster to go to sleep

Wait the 30 seconds for the cluster to sleep and try the curl command again. For convenience with this test you can run watch -d curl --silent backend.local/bar to continually try the endpoint. This time, because an HTTP request was sent to the HTTPS wake endpoint on the virtual cluster, you should see Client sent an HTTP request to an HTTPS server. on the first attempt, and new pod names on subsequent requests.

Additional examples

Sleep after 3 hours of inactivity, anything that does not have the label dont=sleep

experimental:
  sleepMode:
    enabled: true
    autoSleep:
      afterInactivity: 3h  # Uses Go's Duration with a max unit of hour
      exclude:
        selector:
          labels:
            dont: sleep

Sleep every Friday at 17:30 and wake every Monday at 7:00 in Mountain timezone

experimental:
  sleepMode:
    enabled: true
    timezone: America/Denver
    autoSleep:
      schedule: 30 17 * * 5
      wakeup:
        schedule: 0 7 * * 1

Config reference

sleepMode required object pro

SleepMode holds the native sleep mode configuration for Pro clusters

enabled required boolean pro

Enabled toggles the sleep mode functionality, allowing for disabling sleep mode without removing other config

timeZone required string pro

Timezone represents the timezone a sleep schedule should run against, defaulting to UTC if unset

autoSleep required object pro

AutoSleep holds autoSleep details

afterInactivity required string pro

AfterInactivity represents how long a vCluster can be idle before workloads are automaticaly put to sleep

schedule required string pro

Schedule represents a cron schedule for when to sleep workloads

wakeup required object pro

Wakeup holds configuration for waking the vCluster on a schedule rather than waiting for some activity.

schedule required string pro

exclude required object pro

Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep

selector required object pro

labels required object pro

Labels defines what labels should be looked for