Skip to main content

Audit Logging in vCluster.Pro

vCluster.Pro auditing provides a security-relevant, chronological set of records documenting the sequence of actions in vCluster.Pro. vCluster.Pro audits the activities generated by users and by applications that use the vCluster.Pro API.

vCluster.Pro is able to log activities related to:

  • Management Instance changes, such as creation of new virtual clusters, spaces etc.
  • Changes within a virtual cluster or space
  • Changes within a connected cluster

Auditing in vCluster.Pro is very similar to auditing Kubernetes clusters in general.

Enable Auditing​

vCluster.Pro auditing is configured through the vCluster.Pro config in the vCluster.Pro UI (Admin -> Config).

An example configuration could look like:

audit:
  enabled: true
  level: 1
vCluster.Pro Restart Required

Changing the vCluster.Pro auditing configuration requires a restart to take effect. You can restart vCluster.Pro either through the vCluster.Pro UI or via kubectl: kubectl rollout restart deploy/loft -n loft

Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records.

Each request can be recorded with an associated stage. The defined stages are:

  • RequestReceived - The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain.
  • ResponseComplete - The response body has been completed and no more bytes will be sent.
  • Panic - Events generated when a panic occurred.
info

The audit logging feature increases the memory consumption of vCluster.Pro because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration.

Audit Levels​

For easier configuration, vCluster.Pro provides audit levels, that are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most. A more detailed description can be found below:

  • Level 1: will log modifying requests such as creation / modification or deletion of any objects
  • Level 2: like Level 1 but will also log the metadata of reading requests, such as listing pods inside a virtual cluster or space. It won't log the response or request payload and instead only the metadata such as request origin, target etc.
  • Level 3: like Level 2 but instead of only logging the request metadata will also log the complete request payload sent to vCluster.Pro
  • Level 4: like Level 3 but instead of only logging metadata and request payload, will also log the response vCluster.Pro has sent to the requester
info

For all of these levels certain internal read-only apis are not logged since those might pollute the log and drastically increase log size. If you also want to log these, please create a custom audit policy as described below.

You can configure the audit level through the vCluster.Pro config, that can be modified either through the vCluster.Pro UI or helm:

  1. Go to the Admin > Config view using the menu on the left.
  2. In the input field that appears, enter the following config:
    audit:
      enabled: true
      level: 1
  3. Click on the button and wait until vCluster.Pro has restarted.

Optional: Audit Policy​

danger

It is recommended to use audit levels instead of audit policy directly, because a policy is much more complex to define.

As an alternative to Audit levels, policy allows you to define exact rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit granularity of the event. The defined audit granularity options are:

  • None - don't log events that match this rule.
  • Metadata - log request metadata (requesting user, timestamp, resource, verb, etc.) but not request or response body.
  • Request - log event metadata and request body but not response body. This does not apply for non-resource requests.
  • RequestResponse - log event metadata, request and response bodies. This does not apply for non-resource requests.

An example policy that catches all requests would look like this:

audit:
  enabled: true
  policy:
    rules:
      - level: Metadata
        omitStages:
          - RequestReceived

Below you can find a complete policy reference:

policy required object ​

The audit policy to use and log requests. By default vCluster.Pro will not log anything

rules required object[] ​

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string ​

The Level that requests matching this rule are recorded at.

users required string[] ​

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[] ​

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[] ​

The verbs that match this rule. An empty list implies every verb.

resources required object[] ​

Resources that this rule matches. An empty list implies all kinds in all API groups.

group required string ​

Group is the name of the API group that contains the resources. The empty string represents the core API group.

resources required string[] ​

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

resourceNames required string[] ​

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[] ​

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[] ​

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[] ​

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[] ​

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[] ​

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[] ​

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

Persisting Audit Logs​

There are 2 ways how to persist vCluster.Pro audit logs. Either you can deploy vCluster.Pro with a persistent volume claim or let vCluster.Pro connect to a persistent database. The PVC approach does not work for HA mode for vCluster.Pro.

Deploy vCluster.Pro with a PVC to save Audit Logs​

Create a new values.yaml with the following values:

audit:
  persistence:
    enabled: true
    # size: 30Gi

Then apply the values via helm:

helm upgrade loft -n vcluster-pro --version $VERSION \
--repo https://charts.loft.sh \
--reuse-values \
-f values.yaml

Use a persistent database as vCluster.Pro audit backend​

Go to Admin > Config and specify the following vCluster.Pro config setting:

audit:
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Then press Apply and wait until vCluster.Pro is restarted.

Viewing and Exporting Audit Logs​

By default, vCluster.Pro will log audit events to the following locations:

  • To a log file in json format located at /var/log/loft/audit.log inside the vCluster.Pro container. Each line inside the log represents a single audit event.
  • To an internal sqlite storage located at /var/log/loft/audit.db inside the vCluster.Pro container. This sqlite database is used to display audit log events in the vCluster.Pro UI. By default audit events in the sqlite are not persisted, so restarting vCluster.Pro will clear the database. Instead of using a sqlite database, vCluster.Pro is also able to write those events to a persistent mysql database that can be configured through the vCluster.Pro config. E.g.:
audit:
  enabled: true
  dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name

Enable Audit SideCar​

To easily export the audit events to third party systems, we recommend to enable the audit log sidecar that will print all the audit events onto stdout in a separate container which then can be easily watched and exported. Enabling the sidecar is only possible through helm values.

Create a values.yaml with the following contents:

audit:
  enableSideCar: true
danger

You cannot configure this under Admin > Config, since this requires a change in the vCluster.Pro deployment itself, which is why this is a helm option only

Then update the helm release via:

helm upgrade loft vcluster-control-plane --namespace vcluster-pro \
                                         --repo https://charts.loft.sh \
                                         --version $LOFT_VERSION \
                                         --reuse-values \
                                         --values values.yaml

Wait until vCluster.Pro has restarted, then you can view the audit logs via:

kubectl logs -n vcluster-pro -l app=loft -c audit -f

Audit Config Reference​

audit required object ​

Audit holds audit configuration

enabled required boolean false ​

If audit is enabled and incoming api requests will be logged based on the supplied policy.

disableAgentSyncBack required boolean false ​

If true, the agent will not send back any audit logs to Loft itself.

level required integer ​

Level is an optional log level for audit logs. Cannot be used together with policy

policy required object ​

The audit policy to use and log requests. By default loft will not log anything

rules required object[] ​

Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered.

level required string ​

The Level that requests matching this rule are recorded at.

users required string[] ​

The users (by authenticated user name) this rule applies to. An empty list implies every user.

userGroups required string[] ​

The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group.

verbs required string[] ​

The verbs that match this rule. An empty list implies every verb.

resources required object[] ​

Resources that this rule matches. An empty list implies all kinds in all API groups.

group required string ​

Group is the name of the API group that contains the resources. The empty string represents the core API group.

resources required string[] ​

Resources is a list of resources this rule applies to.

For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '' matches all resources and their subresources. 'pods/' matches all subresources of pods. '*/scale' matches all scale subresources.

If wildcard is present, the validation rule will ensure resources do not overlap with each other.

An empty list implies all resources and subresources in this API groups apply.

resourceNames required string[] ​

ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched.

namespaces required string[] ​

Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace.

nonResourceURLs required string[] ​

NonResourceURLs is a set of URL paths that should be audited. s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz" - Log all health checks

omitStages required string[] ​

OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply.

requestTargets required string[] ​

RequestTargets is a list of request targets for which events are created. An empty list implies every request.

clusters required string[] ​

Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply.

omitStages required string[] ​

OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted.

dataStoreEndpoint required string ​

DataStoreEndpoint is an endpoint to store events in.

dataStoreTTL required integer ​

DataStoreMaxAge is the maximum number of hours to retain old log events in the datastore

path required string ​

The path where to save the audit log files. This is required if audit is enabled. Backup log files will be retained in the same directory.

maxAge required integer ​

MaxAge is the maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is not to remove old log files based on age.

maxBackups required integer ​

MaxBackups is the maximum number of old log files to retain. The default is to retain all old log files (though MaxAge may still cause them to get deleted.)

maxSize required integer ​

MaxSize is the maximum size in megabytes of the log file before it gets rotated. It defaults to 100 megabytes.

compress required boolean false ​

Compress determines if the rotated log files should be compressed using gzip. The default is not to perform compression.

How does Audit Logging work for Direct Cluster Endpoints?​

If the direct cluster endpoint feature is enabled, vCluster.Pro audit configuration is synced to each agent and each vCluster.Pro agent will propagate audit events that it receives back to the central vCluster.Pro instance, which then logs it as a regular audit event. Such "propagated" events can be identified through the annotations.audit.loft.sh/sent-by-agent identifier in an audit event.

Disable Agent Sync Back

You can disable event sync back from the agent to the central vCluster.Pro instance via the audit config option disableAgentSyncBack.

Last updated on