Configure metrics access
Fleet Observability uses scoped access keys to separate write access from read access. The gateway never trusts the metrics payload to define scope. Each request is authenticated with Platform and authorized against the requested control plane cluster or cluster.
Access roles​
| Role | Used by | Allows |
|---|---|---|
metrics-writer | Edge collectors | Write metrics for the control plane cluster or cluster the access key is scoped to. |
metrics-reader | Grafana, PromQL clients, automation | Read metrics for the control plane cluster or cluster the access key is scoped to. |
The roles are intentionally separate. A metrics-writer key can't read metrics, and
a metrics-reader key can't write metrics. A key can carry both roles when an
integration needs both paths.
Scope choices​
Scope metrics keys as tightly as possible:
- Use
spec.scope.clustersfor control plane cluster metrics. - Use
spec.scope.virtualClustersfor cluster metrics. - Use
*only for trusted fleet-wide automation.
The Platform authorization check still applies the permissions of the user or subject that owns the key. Scope narrows permission. It doesn't expand it.
The snippets below are one example of how to create access keys. In particular, the line
that generates the KEY value shows just one way to produce a random secret. Use whatever
key generation, secret storage, and provisioning workflow fits your environment. Treat these
commands as general blueprints rather than the required procedure.
Create a writer key for a cluster​
A cluster with the observability integration turned on gets its metrics-writer
key from Platform, delivered into the cluster, so skip this step for it. See
Configure edge collectors.
Create one metrics-writer key for each collector you configure manually, or for each
automation unit that writes on behalf of one cluster.
Set PROJECT and TENANT_CLUSTER to the target cluster's project and instance name. The
KEY value is a freshly generated secret, and the command prints it as token: <value>
at the end.
PROJECT=default
TENANT_CLUSTER=my-vcluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
name: "loft-metrics-writer-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
type: Other
key: "${KEY}"
subject: "loft:metrics-writer:p-${PROJECT}:${TENANT_CLUSTER}"
groups:
- loft:system:metrics-writers
scope:
roles:
- role: metrics-writer
virtualClusters:
- project: ${PROJECT}
virtualCluster: "${TENANT_CLUSTER}"
EOF
echo "token: ${KEY}"
Use the printed token as the collector bearer token. Store it in a Kubernetes Secret in the cluster where the collector runs.
Create a writer key for a control plane cluster​
Use cluster scope when a collector writes metrics for a control plane cluster.
Set CLUSTER to the control plane cluster name.
CLUSTER=my-cluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
name: "loft-metrics-writer-${CLUSTER}"
spec:
type: Other
key: "${KEY}"
subject: "loft:metrics-writer:${CLUSTER}"
groups:
- loft:system:metrics-writers
scope:
roles:
- role: metrics-writer
clusters:
- cluster: "${CLUSTER}"
EOF
echo "token: ${KEY}"
Create a reader key​
Use a metrics-reader key for automation that queries metrics directly through the
Query Proxy.
Set PROJECT and TENANT_CLUSTER to the target cluster's project and instance name.
PROJECT=default
TENANT_CLUSTER=my-vcluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
name: "loft-metrics-reader-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
type: Other
key: "${KEY}"
subject: "loft:metrics-reader:p-${PROJECT}:${TENANT_CLUSTER}"
groups:
- loft:system:metrics-readers
scope:
roles:
- role: metrics-reader
virtualClusters:
- project: ${PROJECT}
virtualCluster: "${TENANT_CLUSTER}"
EOF
echo "token: ${KEY}"
Grafana datasources generated by Platform can also use the user's Platform session cookie. In that case the Query Proxy authorizes reads as the user instead of a static reader key. For details, see Authorization with your access key in Query fleet metrics.
Request scope headers​
Writers and readers declare scope with headers.
For a control plane cluster:
X-Vcluster-Platform-Cluster: local-cluster
For a cluster:
X-Vcluster-Platform-Project: my-project
X-Vcluster-Platform-Instance: my-vcluster
The Write Gateway can derive tenant scope from a writer key only when the key contains exactly one concrete cluster scope. Explicit headers are easier to audit and are recommended for collector configurations.
When you deploy collectors with the cluster-collector template as described in
Deploy collectors with Argo CD, the template sets
these scope headers automatically from the deployment destination. You only set them by hand
when you configure a collector manually.
Rotation​
The bundled cluster collectors mount the metrics-writer Secret as a file and reload the
token when the file changes. A rotated token takes effect without a restart, once the kubelet
refreshes the mounted Secret.
Changes to the endpoint or CA in the mounted Secret require a manual collector restart. With the bundled chart defaults, collector configuration changes trigger a rollout when Argo CD syncs them.
A collector you configure manually keeps the token it loaded at startup unless it reloads
the token file when it changes. Mounting the Secret with subPath prevents that, because
the kubelet doesn't update subPath mounts. Restart it after a rotation, or its writes fail once you disable the
old key.
For a key Platform issued to a cluster, delete the AccessKey. The cluster
fetches a new key on its next refresh and updates its metrics-writer Secret.
With default settings, refreshes are 10 to 12 minutes apart. The kubelet then needs
time to refresh the mounted Secret before collectors pick up the new token without a restart.
Deleting the AccessKey invalidates its token immediately. Until collectors load the replacement token, the gateway rejects their writes and those metrics are lost. The gap can exceed 12 minutes because Secret propagation adds time after the next refresh. Fetch or delivery failures can extend it further. Rotate during a window where a gap in metrics is acceptable.
To rotate any other metrics key:
- Create a new key with the same role and scope.
- Update the collector or query client Secret with the new key value.
- Restart the collector so it picks up the new token. Reload query clients that hold the reader token.
- Delete or disable the old key.
Restart the collector workloads in the collector namespace. For collectors deployed with
the cluster-collector template, restart the OpenTelemetry Collector Deployment and
DaemonSet:
kubectl rollout restart daemonset,deployment -n observability \
-l app.kubernetes.io/name=opentelemetry-collector
For a manually deployed collector, restart its own Deployment or DaemonSet the same way.
Authorization decisions are cached briefly. After you delete, disable, or narrow a key, allow up to the cache TTL for all gateway and query-proxy decisions to expire.
Next step​
Configure collectors to send metrics with these keys in Configure edge collectors.