Audit Logging

		Enterprise
Available in these plans	Free	Dev	Prod	Scale
Audit Logging

Overview

Audit logging in the platform provides a security-relevant, chronological set of records documenting the sequence of actions. It audits activities generated by users and applications that use the platform API.

The platform can log activities related to:

• Management instance changes, such as creation of new tenant clusters, spaces, and so on.
• Changes within a tenant clusterTenant clusterA fully isolated Kubernetes environment provisioned for a single tenant. Each tenant cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Tenant cluster or space
• Changes within a connected cluster

Auditing in the platform is similar to auditing Kubernetes clusters in general.

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

• RequestReceived: The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
• ResponseComplete: The response body has been completed and no more bytes are sent.
• Panic: Events generated when a panic occurred.

The audit logging feature increases the memory consumption of the platform because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Enable auditing

Two separate audit fields

values.yaml has two distinct audit fields with different purposes:

• audit (top-level Helm value) — configures the audit sidecar and log persistence (PVC) for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart.
• config.audit (under the config section) — controls audit recording and policy, and enables the Audit Log UI. This can be set in Platform > Platform Config > Config in the UI or under config in values.yaml. config.audit.enabled defaults to true, so audit events are recorded without any additional configuration.

Changes to config.audit made through the UI take effect automatically within about 10-15 seconds, no restart required. Changing config.audit through a Helm upgrade also restarts the platform pod, since the chart stamps the pod template with a config hash. The setting itself still takes effect through the same hot-reload mechanism either way.

Configure auditing through the platform config in the platform UI (Platform > Platform Config > Config). Content added in the UI goes automatically under the config section of values.yaml.

An example configuration could look like:

Example audit configuration in platform UI

audit:
  enabled: true
  level: 1

See the complete list of fields under the config.audit section of values.yaml.

Audit levels

The platform provides audit levels, which are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most:

• Level 1: Logs modifying requests such as creation, modification, or deletion of any objects. Creation requests also log the request payload.
• Level 2: Like Level 1 but also logs the metadata of reading requests, such as listing pods inside a tenant cluster or space. Reading requests won't log the response or request payload and instead only the metadata, such as request origin and target.
• Level 3: Like Level 2 but instead of only logging the request metadata, also logs the complete request payload sent to the platform.
• Level 4: Like Level 3 but instead of only logging metadata and request payload, also logs the response the platform has sent to the requester.

Captured payloads are normalized to JSON and size-limited before storage. Unconvertible or oversized payloads are omitted with an annotation. See Request and response payload storage.

For all of these levels, certain internal read-only APIs aren't logged since those might pollute the log and drastically increase log size. To log these, create a custom audit policy as described below.

Configure the audit level through the platform config, which can be modified either through the platform UI or Helm:

UI

1. Go to the Platform > Platform Config > Config view using the menu on the left.

2. In the input field that appears, enter the following config:

   audit:
     enabled: true
     level: 1

3. Click on the button. The change takes effect automatically within about 10-15 seconds, with no platform restart required.

Helm

Create a new file values.yaml:

values.yaml

config:
  audit:
    enabled: true
    level: 1

Then apply these values with Helm (make sure to specify the correct platform version):

Apply helm values

helm upgrade loft vcluster-platform --namespace vcluster-platform \
                                         --repo https://charts.loft.sh \
                                         --version $LOFT_VERSION \
                                         --reuse-values \
                                         --values values.yaml

Request and response payload storage

Audit levels and custom policies both determine payload storage through one of four granularity settings. Request stores the request payload. RequestResponse also stores the response payload. See Audit policy for the full list.

The platform stores payloads for resource requests only. Long-running requests, such as log streaming, never store payloads. Creates of sensitive resources, such as shared secrets and access keys, keep metadata-only events on every level.

The platform always stores payloads as JSON:

• Payloads that arrive as plain JSON are stored unchanged.
• Gzip-compressed request and response bodies are stored in their decompressed form.
• Payloads sent in the Kubernetes Protobuf format are converted to their JSON equivalent for known Kubernetes API types. The stored payload matches what a JSON client would have sent.

Payload size limit

The platform captures at most 2 MiB per request or response payload. It doesn't store payloads above this limit.

Omitted payloads

When the platform can't store a payload, it keeps the audit event with all of its metadata but leaves out the payload. The event then carries an annotation that names the omission reason:

• audit.platform.vcluster.com/request-object-omitted for the request payload
• audit.platform.vcluster.com/response-object-omitted for the response payload

The following reasons can appear as annotation values:

Reason	Description
truncated	The payload exceeded the capture limit, so the platform captured only a partial copy.
too-large	The payload exceeded the capture limit after decompression or conversion to JSON.
decode-failed	The platform couldn't decompress or decode the payload.
unsupported-encoding	The payload used a content encoding or media type that the platform can't convert to JSON.
invalid-json	The payload wasn't valid JSON and not in any recognized format.

Audit policy

Audit policy is a more complex alternative to audit levels, letting you define exact rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit granularity of the event. The defined audit granularity options are:

• None: Don't log events that match this rule.
• Metadata: Log request metadata (requesting user, timestamp, resource, verb, and so on) but not request or response body.
• Request: Log event metadata and request body but not response body. This doesn't apply for non-resource requests.
• RequestResponse: Log event metadata, request and response bodies. This doesn't apply for non-resource requests.

An example policy that catches all requests:

Example audit policy in values.yaml

config:
  audit:
    enabled: true
    policy:
      rules:
        - level: Metadata
          omitStages:
            - RequestReceived

See the complete policy reference.

Persist audit logs

Persist platform audit logs in one of two ways: deploy the platform with a persistent volume claim (PVC), or connect the platform to a persistent database.

HA only supported with persistent database

If you are running the platform in high availability (HA mode), the only way to persist audit logs is to use a persistent database. The PVC approach doesn't work.

Deploy the platform with a PVC to save audit logs

Create a new values.yaml with the following values:

values.yaml for PVC

audit:
  persistence:
    enabled: true
    # size: 30Gi

This is the top-level audit field in values.yaml, not config.audit (see Enable auditing for the distinction).

Apply the values using Helm:

Apply Helm values

helm upgrade vcluster-platform vcluster-platform \
  --namespace vcluster-platform \
  --repo https://charts.loft.sh \
  --reuse-values \
  -f values.yaml

Use a persistent database as platform audit backend

Instead of the internal sqlite storage (see View and export audit logs), the platform can write audit events to a persistent MySQL database. This is a config.audit setting, so you can configure it through the platform UI or Helm.

Through the UI, go to Platform > Platform Config > Config and specify the following platform config setting:

Platform config for persistent database

audit:
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Through Helm, add the following to values.yaml:

values.yaml for persistent database

config:
  audit:
    enabled: true
    dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Apply the changes. Through the UI, they take effect automatically within about 10-15 seconds, with no platform restart required. Through Helm, helm upgrade restarts the platform pod as a side effect of the upgrade, though the setting itself takes effect through the same hot-reload mechanism either way.

View and export audit logs

By default, the platform logs audit events to the following locations:

• To a log file in JSON format located at /var/lib/loft/audit.log inside the platform container. Each line inside the log represents a single audit event.
• To an internal sqlite storage located at /var/lib/loft/audit.db inside the platform container. This sqlite database is used to display audit log events in the platform UI. By default audit events in the sqlite are not persisted, so restarting the platform clears the database. See Use a persistent database as platform audit backend to configure a persistent MySQL backend instead.

Enable audit sidecar

To export audit events to third-party systems, enable the audit log sidecar. It prints all audit events to stdout in a separate container, so you can watch and export them from there. You can only enable the sidecar through Helm values.

Create a values.yaml with the following contents:

values.yaml for audit sidecar

audit:
  enableSideCar: true

This is the top-level audit field in values.yaml, not config.audit, so it cannot be set in Platform > Platform Config > Config (see Enable auditing for the distinction).

Update the Helm release:

Update Helm release

helm upgrade vcluster-platform vcluster-platform \
  --namespace vcluster-platform \
  --repo https://charts.loft.sh \
  --reuse-values \
  --values values.yaml

Wait until the platform has restarted, then view the audit logs:

View audit logs

kubectl logs -n vcluster-platform -l app=loft -c audit -f

Audit logging for Regional Cluster Endpoints

Regional Cluster Endpoints allow direct communication between users and clusters, bypassing the central platform instance. When this feature is enabled, the platform audit configuration is synced to each agent. Each platform agent propagates audit events it receives back to the central platform instance, which then logs it as a regular audit event. These "propagated" events can be identified by the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.

To disable this behavior, set the audit config option disableAgentSyncBack.