Troubleshooting Fleet Observability

Use this page to diagnose the Fleet Observability path from edge collectors through the observability gateway and into your metrics backend.

Gateway resources aren't created

Check the connector Secret:

kubectl get secret -n vcluster-platform fleet-observability -o yaml

Verify:

• The Secret is in the Platform namespace.
• metadata.labels["loft.sh/connector-type"] is observability.
• The fleet connector has the platform.vcluster.com/fleet-observability-connector label set to "true".
• metricsBackendOtlpEndpoint and metricsBackendPromQLEndpoint are set.

The gateway isn't reconciled until both backend endpoints are present.

Multiple fleet connectors

Only one connector can carry the fleet label:

platform.vcluster.com/fleet-observability-connector: "true"

If multiple connector Secrets have this label, Platform still creates per-connector gateway resources, but it doesn't publish the fleet-wide platform admin Grafana datasource. Remove the fleet label from all but one connector.

Collector writes are rejected

Check the collector exporter configuration:

• The Authorization header uses a valid bearer token.
• The token is a metrics-writer access key.
• The request includes either X-Vcluster-Platform-Cluster or the X-Vcluster-Platform-Project and X-Vcluster-Platform-Instance pair.
• The header values match the key scope.

Check gateway logs:

kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c otel-gateway

Common causes:

Symptom	Likely cause
Missing bearer token	Collector didn't set the Authorization header.
Scope denied	Key scope doesn't include the requested cluster or tenant cluster.
Ambiguous scope	Request included both cluster and tenant scope, or multiple values for one scope header.
Headerless write rejected	The key has zero or multiple concrete tenant scopes, so scope can't be derived.

Queries are rejected

Check query-proxy logs:

kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c query-proxy

Common causes:

Symptom	Likely cause
HTTP 400	Missing scope for a non-admin user, invalid scope matcher, or malformed PromQL.
HTTP 403	The user or key is authenticated but not authorized for the requested metrics scope.
Query succeeds for admins only	The datasource is using the unscoped fleet-admin path. Add tenant or cluster scope headers for non-admin users.
Regex query rejected	Scope-controlling labels only allow equality or literal alternation regular expression matchers.

Platform labels are missing

If metrics arrive without vcluster_platform_* labels:

• Confirm the metrics went through the Write Gateway. Direct writes to the backend don't get platform labels.
• Confirm the collector used a valid metrics writer key and scope headers.
• Confirm the backend stores labels from OTLP datapoint attributes.
• Check whether the metric is expected to have tenant labels or only cluster labels.

The Write Gateway strips caller-supplied platform labels and then stamps authenticated values. Empty authenticated values are dropped.

Backend TLS failures

If gateway pods start but can't write or query the backend:

• Verify metricsBackendOtlpEndpoint and metricsBackendPromQLEndpoint.
• Set metricsBackendServerName when the backend certificate's DNS name differs from the endpoint host.
• Use metricsBackendCertSecretName when the backend requires mTLS. This Secret is also the source of trust for a private backend CA. It must contain ca.crt, tls.crt, and tls.key.
• Use metricsBackendInsecureSkipVerify: "true" only as a temporary test setting.

remote error: tls: certificate required

This error means the backend rejected the connection because the gateway didn't present a client certificate, not because the gateway distrusts the backend's certificate. metricsBackendInsecureSkipVerify only skips the gateway's verification of the backend's server certificate, so it can't fix this. Set metricsBackendCertSecretName instead. The bundled Prometheus backend enforces mTLS and expects this client certificate. When deployed with its default parameters, it's the auto-generated metrics-backend-mtls Secret. See Deploy the metrics backend with Argo CD.

Gateway TLS failures

The gateway always serves OTLP and query traffic over TLS. If collectors or query clients can't connect:

• For the Platform-managed certificate, read ca.crt from the gateway-<connector-id>-tls Secret and configure the client to trust it.
• For a custom private certificate, include ca.crt in the Secret referenced by observabilityGatewayCertSecretName.
• Verify that the client uses https for OTLP HTTP and query traffic or enables TLS for OTLP gRPC.
• If the gateway uses a LoadBalancer Service and a Platform-managed certificate, wait for the LoadBalancer address before expecting the gateway Deployment to become available.

Observability pods are OOMKilled or evicted

The bundled templates deploy Prometheus, Grafana, and the collectors without an explicit resources block, so those pods likely end up in the BestEffort QoS class, which the kubelet evicts first under node pressure.

Check the QoS class, last container state, and status message:

kubectl describe pod -n vcluster-platform <pod>

Look for QoS Class, Last State, and Message in the output.

The fix depends on which failure you see:

• OOMKilled: raise the memory limit on the affected template's resources parameter.
• Evicted: set memory requests so the kubelet ranks the pod above pods that exceed their own requests. Only memory, disk, and PID pressure trigger eviction, not CPU.

Set requests and limits on the affected template:

• Prometheus: the resources parameter. See Deploy the metrics backend with Argo CD.
• Grafana: the resources parameter. See Deploy the bundled Grafana with Argo CD.
• Collectors: the otelDaemonResources and otelDeployResources parameters. See Deploy collectors with Argo CD.

Prometheus is the most likely to be OOMKilled, because its memory grows with active series count.

Platform API outage

The gateway and query proxy fail closed when Platform authorization is unavailable. Previously allowed decisions can continue only until their short positive cache TTL expires.

Expected behavior:

• Collectors may queue and retry writes locally, depending on their exporter settings.
• Queries fail visibly after cached authorization decisions expire.
• Writes and reads recover when the Platform API is reachable again.

Useful checks

Gateway pods:

CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"

Gateway service:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"

Write Gateway metrics:

CONNECTOR_ID=fleet-observability
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889
curl http://127.0.0.1:8889/metrics

Query Proxy metrics:

CONNECTOR_ID=fleet-observability
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8080:8080
curl http://127.0.0.1:8080/metrics