Skip to main content

Enabling Audit Logging

Overview​

Audit logging in the platform provides a security-relevant, chronological set of records documenting the sequence of actions. It audits activities generated by users and applications that use the platform API.

The platform can log activities related to:

  • Management instance changes, such as creation of new virtual clusters, spaces, etc.
  • Changes within a virtual cluster or space
  • Changes within a connected cluster

Auditing in the platform is similar to auditing Kubernetes clusters in general.

Enable auditing​

Configure auditing through the platform config in the platform UI (Admin -> Config).

An example configuration could look like:

Example audit configuration
audit:
  enabled: true
  level: 1
Platform restart required

Changing the platform auditing configuration requires a restart to take effect. Restart the platform either through the platform UI or via kubectl: kubectl rollout restart deploy/loft -n loft

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

  • RequestReceived: The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
  • ResponseComplete: The response body has been completed and no more bytes are sent.
  • Panic: Events generated when a panic occurred.
info

The audit logging feature increases the memory consumption of the platform because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Audit levels​

The platform provides audit levels, which are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most:

  • Level 1: Logs modifying requests such as creation / modification or deletion of any objects
  • Level 2: Like Level 1 but also logs the metadata of reading requests, such as listing pods inside a virtual cluster or space. It won't log the response or request payload and instead only the metadata such as request origin, target, etc.
  • Level 3: Like Level 2 but instead of only logging the request metadata also logs the complete request payload sent to the platform
  • Level 4: Like Level 3 but instead of only logging metadata and request payload, also logs the response the platform has sent to the requester
info

For all of these levels certain internal read-only APIs are not logged since those might pollute the log and drastically increase log size. To log these, create a custom audit policy as described below.

Configure the audit level through the platform config, which can be modified either through the platform UI or helm:

  1. Go to the Admin > Config view using the menu on the left.

  2. In the input field that appears, enter the following config:

    audit:
      enabled: true
      level: 1

  3. Click on the button and wait until vCluster Platform has restarted.

Optional: Audit policy​

warning

It is recommended to use audit levels instead of audit policy directly, because a policy is much more complex to define.

As an alternative to audit levels, policy allows you to define exact rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit granularity of the event. The defined audit granularity options are:

  • None: Don't log events that match this rule.
  • Metadata: Log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
  • Request: Log event metadata and request body but not response body. This does not apply for non-resource requests.
  • RequestResponse: Log event metadata, request and response bodies. This does not apply for non-resource requests.

An example policy that catches all requests:

Example audit policy
audit:
  enabled: true
  policy:
    rules:
      - level: Metadata
        omitStages:
          - RequestReceived

Below you can find a complete policy reference:

rules required object[] pro​

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string pro​

The Level that requests matching this rule are recorded at.

users required string[] pro​

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[] pro​

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[] pro​

The verbs that match this rule. An empty list implies every verb.

resources required object[] pro​

Resources that this rule matches. An empty list implies all kinds in all API groups.

group required string pro​

Group is the name of the API group that contains the resources. The empty string represents the core API group.

resources required string[] pro​

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

resourceNames required string[] pro​

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[] pro​

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[] pro​

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[] pro​

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[] pro​

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[] pro​

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[] pro​

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

Persisting audit logs​

There are two ways to persist platform audit logs: deploying the platform with a persistent volume claim (PVC) or connecting the platform to a persistent database. The PVC approach does not work for HA mode for the platform.

Deploy the platform with a PVC to save audit logs​

Create a new values.yaml with the following values:

values.yaml for PVC
audit:
  persistence:
    enabled: true
    # size: 30Gi

Apply the values via helm:

Apply helm values
helm upgrade vcluster-platform vcluster-platform -n vcluster-platform --version 4.0.0-alpha.12 \
--repo https://charts.loft.sh \
--reuse-values \
-f values.yaml

Use a persistent database as platform audit backend​

Go to Admin > Config and specify the following platform config setting:

Platform config for persistent database
audit:
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Apply the changes and wait until the platform is restarted.

Viewing and exporting audit logs​

By default, the platform logs audit events to the following locations:

  • To a log file in JSON format located at /var/log/loft/audit.log inside the platform container. Each line inside the log represents a single audit event.
  • To an internal sqlite storage located at /var/log/loft/audit.db inside the platform container. This sqlite database is used to display audit log events in the platform UI. By default audit events in the sqlite are not persisted, so restarting the platform clears the database. Instead of using a sqlite database, the platform can write those events to a persistent mysql database configured through the platform config:
Platform config for MySQL database
audit:
  enabled: true
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Enable audit sidecar​

To easily export the audit events to third party systems, enable the audit log sidecar that prints all the audit events onto stdout in a separate container which can then be easily watched and exported. Enabling the sidecar is only possible through helm values.

Create a values.yaml with the following contents:

values.yaml for audit sidecar
audit:
  enableSideCar: true
warning

You cannot configure this under Admin > Config, since this requires a change in the platform deployment itself, which is why this is a helm option only

Update the helm release:

Update helm release
helm upgrade vcluster-platform vcluster-platform --namespace vcluster-platform \
                                         --repo https://charts.loft.sh \
                                         --version 4.0.0-alpha.12 \
                                         --reuse-values \
                                         --values values.yaml

Wait until the platform has restarted, then view the audit logs:

View audit logs
kubectl logs -n vcluster-platform -l app=loft -c audit -f

Audit config reference​

enabled required boolean pro​

If audit is enabled and incoming api requests will be logged based on the supplied policy.

disableAgentSyncBack required boolean pro​

If true, the agent will not send back any audit logs to Loft itself.

level required integer pro​

Level is an optional log level for audit logs. Cannot be used together with policy

policy required object pro​

The audit policy to use and log requests. By default loft will not log anything

rules required object[] pro​

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string pro​

The Level that requests matching this rule are recorded at.

users required string[] pro​

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[] pro​

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[] pro​

The verbs that match this rule. An empty list implies every verb.

resources required object[] pro​

Resources that this rule matches. An empty list implies all kinds in all API groups.

group required string pro​

Group is the name of the API group that contains the resources. The empty string represents the core API group.

resources required string[] pro​

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

resourceNames required string[] pro​

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[] pro​

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[] pro​

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[] pro​

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[] pro​

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[] pro​

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[] pro​

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

dataStoreEndpoint required string pro​

DataStoreEndpoint is an endpoint to store events in.

dataStoreTTL required integer pro​

DataStoreMaxAge is the maximum number of hours to retain old log events in the datastore

path required string pro​

The path where to save the audit log files. This is required if audit is enabled. Backup log files will be retained in the same directory.

maxAge required integer pro​

MaxAge is the maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is not to remove old log files based on age.

maxBackups required integer pro​

MaxBackups is the maximum number of old log files to retain. The default is to retain all old log files (though MaxAge may still cause them to get deleted.)

maxSize required integer pro​

MaxSize is the maximum size in megabytes of the log file before it gets rotated. It defaults to 100 megabytes.

compress required boolean pro​

Compress determines if the rotated log files should be compressed using gzip. The default is not to perform compression.

Audit logging for direct cluster endpoints​

Direct cluster endpoints allow direct communication between users and clusters, bypassing the central platform instance. When this feature is enabled, the platform audit configuration is synced to each agent. Each platform agent propagates audit events it receives back to the central platform instance, which then logs it as a regular audit event. These "propagated" events can be identified by the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.

Disable agent sync back

Disable event sync back from the agent to the central platform instance via the audit config option disableAgentSyncBack.