Skip to main content

Metrics And Monitoring

Logging​

By default, vCluster.Pro will print out structured logs to the console with a log level of info.

These logs will include constant log messages, a human-readable timestamp, log levels, the file and line number of the source caller, and with variables in the body:

2023-07-11 09:20:56     INFO    controller-runtime.metrics      metrics/listener.go:44  Metrics server is starting to listen    {"component": "loft", "addr": "127.0.0.1:12000"}
2023-07-11 09:20:56     INFO    initialize/context.go:68        Initialize...   {"component": "loft"}
2023-07-11 09:20:56     INFO    initialize/context.go:72        Ensure certificates...  {"component": "loft"}
2023-07-11 09:21:02     INFO    initialize/context.go:80        Ensure crds...  {"component": "loft"}

You can change the log level by setting the helm value logging.level to one of the following values: debug, info, error, and upgrading your vCluster.Pro release with the following command:

helm upgrade loft vcluster-control-plane --repo https://charts.loft.sh/ \
  --namespace vcluster-pro \
  --reuse-values \
  --set logging.level=error # or debug or info

JSON Output Encoding​

In addition to the console output, vCluster.Pro also supports printing logs encoded in JSON format.

This output format is helpful for structured logging, which allows you to search and filter logs easily.

You can change the log output by setting the helm value logging.encoding to one of the following values: json, and upgrading your vCluster.Pro release with the following command:

helm upgrade loft vcluster-control-plane --repo https://charts.loft.sh/ \
  --namespace vcluster-pro \
  --reuse-values \
  --set logging.encoding=json

These JSON-encoded logs will include constant log messages with different variables in the body, like the default console output:

{"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"}
{"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"}
{"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"}
{"level":"info","ts":1689067389.678155,"caller":"initialize/context.go:80","msg":"Ensure crds...","component":"loft"}

The JSON encoded logs will contain at least the following fields:

  • level: The log level of the message
  • ts: The timestamp of the message (floating-point number of seconds since the Unix epoch)
  • caller: The file and line number of the message
  • msg: The message itself
  • component: The vCluster.Pro component that emitted the message

Metrics​

vCluster.Pro exposes several prometheus conformant metrics that can be scraped.

Create the Prometheus ServiceMonitor​

The metrics can be scraped with the included ServiceMonitor in the vCluster.Pro chart, which can be deployed with helm. In order for this to work, make sure you have installed a prometheus operator into your cluster. You'll also need to upgrade your vCluster.Pro release with the following command to create the ServiceMonitor:

# Make sure you change serviceMonitor.namespace if your prometheus installation requires this.
helm upgrade loft vcluster-control-plane --repo https://charts.loft.sh/ \
  --namespace vcluster-pro \
  --reuse-values \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=loft

Access the metrics endpoint directly without a ServiceMonitor​

vCluster.Pro will expose its internal metrics at https://my-loft-subdomain.my-url.com/metrics. By default vCluster.Pro will require an Authorization http header field with a kubernetes bearer token that has access to the non resource url /metrics in the kubernetes cluster where vCluster.Pro is installed. If the header is not provided, vCluster.Pro will deny the request.

Disable metrics authentication

If you wish to scrape metrics without authentication, you can disable it via the environment variable INSECURE_METRICS=true in the vCluster.Pro helm chart.

If you have kubernetes service account token that has the appropriate rights, you can access the metrics via curl:

# Add an optional --insecure, if your vCluster.Pro instance is using an untrusted certificate
$ curl https://my-vcluster-pro-subdomain.my-url.com/metrics -H "Authorization: Bearer eyJhbGci..."

# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
...

Which metrics are exposed?​

vCluster.Pro collects metrics from its internal components and merges them together into a single metrics endpoint at /metrics. The following component metrics are merged together:

  • API gateway metrics: vCluster.Pro exposes metrics for its internal api gateway such as the total amount of requests (apigateway_ENDPOINT_request_total), request latency (apigateway_ENDPOINT_request_duration_seconds) and request sizes (apigateway_ENDPOINT_response_sizes) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available:
    • ui: all requests that target vCluster.Pro UI assets (metrics are apigateway_ui_request_total, apigateway_ui_request_duration_seconds, apigateway_ui_response_sizes)
    • auth: all requests that target vCluster.Pro authentication endpoints (such as login, token refresh etc.)
    • kubernetes: all forwarded requests that target a kubernetes cluster. The targeted backend is provided by a label and can be either management (the integrated vCluster.Pro kubernetes api server), cluster (a request forwarded to a connected cluster) or vcluster (a virtual kubernetes cluster deployed by vCluster.Pro). Further labels are provided as well such as kubernetes resource, api version, api group, subresource etc.
    • oidc: all requests that target the internal OIDC server
  • Integrated Kubernetes API server metrics: all internal kubernetes api server metrics are exposed. An incomplete list can be found in the kubernetes repository
  • Integrated Controller manager metrics: all vCluster.Pro management cluster controller metrics are exposed.
  • Go runtime metrics: some general metrics about the golang runtime and resource usage are exposed.
Last updated on