Configure metrics access

Fleet Observability uses scoped access keys to separate write access from read access. The gateway never trusts the metrics payload to define scope. Each request is authenticated with Platform and authorized against the requested cluster or tenant cluster.

Access roles

Role	Used by	Allows
metrics-writer	Edge collectors	Write metrics for the access key's scoped cluster or tenant cluster.
metrics-reader	Grafana, PromQL clients, automation	Read metrics for the access key's scoped cluster or tenant cluster.

The roles are intentionally separate. A metrics-writer key can't read metrics, and a metrics-reader key can't write metrics. A key can carry both roles when an integration needs both paths.

Scope choices

Scope metrics keys as tightly as possible:

• Use spec.scope.clusters for control plane cluster metrics.
• Use spec.scope.virtualClusters for tenant cluster metrics.
• Use * only for trusted fleet-wide automation.

The Platform authorization check still applies the permissions of the user or subject that owns the key. Scope narrows permission. It doesn't expand it.

Examples are blueprints, not prescriptions

The snippets below are one example of how to create access keys. In particular, the line that generates the KEY value shows just one way to produce a random secret. Use whatever key generation, secret storage, and provisioning workflow fits your environment. Treat these commands as general blueprints rather than the required procedure.

Create a writer key for a tenant cluster

Create one metrics-writer key for each tenant cluster collector, or for each automation unit that writes on behalf of one tenant cluster.

Set PROJECT and TENANT_CLUSTER to the target tenant cluster's project and instance name. The KEY value is a freshly generated secret, and the command prints it as token: <value> at the end.

Modify the following with your specific values to generate a copyable command:

PROJECT

TENANT_CLUSTER

PROJECT=default
TENANT_CLUSTER=my-vcluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
  name: "loft-metrics-writer-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
  type: Other
  key: "${KEY}"
  subject: "loft:metrics-writer:p-${PROJECT}:${TENANT_CLUSTER}"
  groups:
    - loft:system:metrics-writers
  scope:
    roles:
      - role: metrics-writer
    virtualClusters:
      - project: ${PROJECT}
        virtualCluster: "${TENANT_CLUSTER}"
EOF
echo "token: ${KEY}"

Use the printed token as the collector bearer token. Store it in a Kubernetes Secret in the cluster where the collector runs.

Create a writer key for a control plane cluster

Use cluster scope when a collector writes metrics for a control plane cluster.

Set CLUSTER to the control plane cluster name.

Modify the following with your specific values to generate a copyable command:

CLUSTER

CLUSTER=my-cluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
  name: "loft-metrics-writer-${CLUSTER}"
spec:
  type: Other
  key: "${KEY}"
  subject: "loft:metrics-writer:${CLUSTER}"
  groups:
    - loft:system:metrics-writers
  scope:
    roles:
      - role: metrics-writer
    clusters:
      - cluster: "${CLUSTER}"
EOF
echo "token: ${KEY}"

Create a reader key

Use a metrics-reader key for automation that queries metrics directly through the Query Proxy.

Set PROJECT and TENANT_CLUSTER to the target tenant cluster's project and instance name.

Modify the following with your specific values to generate a copyable command:

PROJECT

TENANT_CLUSTER

PROJECT=default
TENANT_CLUSTER=my-vcluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
  name: "loft-metrics-reader-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
  type: Other
  key: "${KEY}"
  subject: "loft:metrics-reader:p-${PROJECT}:${TENANT_CLUSTER}"
  groups:
    - loft:system:metrics-readers
  scope:
    roles:
      - role: metrics-reader
    virtualClusters:
      - project: ${PROJECT}
        virtualCluster: "${TENANT_CLUSTER}"
EOF
echo "token: ${KEY}"

Grafana datasources generated by Platform can also use the user's Platform session cookie. In that case the Query Proxy authorizes reads as the user instead of a static reader key. For details, see Authorization with your access key in Query fleet metrics.

Request scope headers

Writers and readers declare scope with headers.

For a control plane cluster:

X-Vcluster-Platform-Cluster: local-cluster

For a tenant cluster:

X-Vcluster-Platform-Project: my-project
X-Vcluster-Platform-Instance: my-vcluster

The Write Gateway can derive tenant scope from a writer key only when the key contains exactly one concrete tenant cluster scope. Explicit headers are easier to audit and are recommended for collector configurations.

Argo CD collectors set these headers for you

When you deploy collectors with the cluster-collector template as described in Deploy collectors to each cluster, the template sets these scope headers automatically from the deployment destination. You only set them by hand when you configure a collector manually.

Rotation

Restart collectors after a key or config change

Edge collectors don't roll their pods automatically when their token or config changes. After you rotate a metrics-writer key or change collector config, you must restart the collector so it loads the new value. Until you restart it, the collector keeps using the old token, and its writes fail once you disable the old key.

To rotate a metrics key:

1. Create a new key with the same role and scope.
2. Update the collector or query client Secret with the new key value.
3. Restart the collector so it picks up the new token. Reload query clients that hold the reader token.
4. Delete or disable the old key.

Restart the collector workloads in the collector namespace. For collectors deployed with the cluster-collector template, restart the OpenTelemetry Collector Deployment and DaemonSet:

kubectl rollout restart daemonset,deployment -n observability \
  -l app.kubernetes.io/name=opentelemetry-collector

For a manually deployed collector, restart its own Deployment or DaemonSet the same way.

Authorization decisions are cached briefly. After you delete, disable, or narrow a key, allow up to the cache TTL for all gateway and query-proxy decisions to expire.

Next step

Configure collectors to send metrics with these keys in Configure edge collectors.