Fleet Observability quickstart
This walks through the default, out-of-the-box path. It deploys the bundled Prometheus
backend and edge collectors with Argo CD to one cluster, using YAML and kubectl
throughout. Each step links to a reference page with the full parameter list and the
Platform UI walkthrough. See those pages for alternatives, such as bringing your own
backend or targeting a control plane cluster.
Prerequisites​
- The Argo CD integration is enabled on both the cluster where Platform is installed and the cluster you collect from, each referencing an Argo CD connector. See Connect to Argo CD.
kubectlaccess to a project namespace. Platform creates a namespace namedp-<project-name>for each project, so the default project's namespace isp-default. Installations with a customized project namespace prefix, including installations upgraded from older versions, can use a different namespace.- A cluster to collect metrics from.
1. Deploy the metrics backend​
Deploy the bundled Prometheus backend to the local cluster, the cluster where Platform itself is installed:
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-metrics-backend
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet metrics backend"
destination:
cluster:
name: loft-cluster # the reserved name for Platform's local cluster; don't change this
templateRef:
name: fleet-observability-prometheus
parameters:
destinationNamespace: vcluster-platform
prometheusStorageSize: 20Gi
project: default
kubectl apply -f fleet-metrics-backend.yaml
Confirm the application landed in your project's namespace:
kubectl get argocdapplications -n p-default
Substitute your project's namespace if you're not using the default project.
For other backends, the Platform UI, and the full parameter list, see Deploy the metrics backend with Argo CD.
2. Create the connector and deploy the gateway​
Create the observability connector Secret. These endpoints match the backend's default namespace from step 1. The Secret below also configures the connector for the bundled Grafana deployed in step 6, using that template's default certificate Secret name. The same connector is also available through the Connector API. See Configure an observability connector.
apiVersion: v1
kind: Secret
metadata:
name: fleet-observability
namespace: vcluster-platform
labels:
loft.sh/connector-type: observability
platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
metricsBackendCertSecretName: metrics-backend-mtls
grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local
grafanaCertSecretName: grafana-server-tls
fleetObservabilityDashboardUid: vcluster-fleet-observability
kubectl apply -f fleet-observability-connector.yaml
Platform deploys the gateway from this Secret. Check that the gateway pod comes up:
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability
For connector keys, TLS options, and the Platform UI, see Install the observability gateway.
3. Turn on the observability integration​
Turn on the observability integration in the cluster's vcluster.yaml, referencing
the connector from step 2:
integrations:
observability:
enabled: true
connector: fleet-observability
Platform then issues a metrics-writer access key scoped to the cluster, and the
cluster writes it into a metrics-writer Secret in its observability namespace,
together with the gateway endpoint and the CA that issued the gateway's serving
certificate. You don't create the key or the Secret yourself. The cluster refreshes
the Secret automatically, replacing a deleted key. Collectors reload a rotated token after the
kubelet refreshes the mounted Secret, without a restart. See Rotation
for timing and the gap in metrics after deleting a key.
The endpoint Platform hands out depends on where the cluster runs:
- Same cluster as Platform
- A different cluster
If the cluster uses shared nodes on the control plane cluster that runs Platform,
Platform hands out the gateway's in-cluster Service name. Map that Service into the tenant
cluster's DNS so the collectors can resolve it. Add this to the same vcluster.yaml:
controlPlane:
coredns:
enabled: true
embedded: true
networking:
resolveDNS:
- service: vcluster-platform/gateway-fleet-observability
target:
hostService: vcluster-platform/gateway-fleet-observability
See Resolve DNS for how
to apply vcluster.yaml changes and the full option reference.
If the cluster runs on a different cluster than Platform, or runs its workloads on
private nodes, it can't reach the gateway's in-cluster Service. Expose the gateway with a
LoadBalancer Service instead. Add serviceType: LoadBalancer to the connector Secret from
step 2 and reapply it. Platform then hands out the load balancer address once the cloud
provider assigns it, and adds it to the gateway's serving certificate.
Until you do, Platform refuses to issue a key for the cluster, because the collectors couldn't reach the endpoint it would hand out.
See Expose the gateway with a LoadBalancer for certificate and Service annotation details.
For control plane cluster keys, reader keys, and rotation, see Configure metrics access.
4. Deploy edge collectors​
Deploy the collector to your cluster. The collectors read the endpoint, token, and
CA from the metrics-writer Secret from step 3, so the application only names the
destination:
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-edge-collectors-my-vcluster # include the cluster name for uniqueness
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet edge collectors"
destination:
virtualCluster:
name: my-vcluster # replace with your cluster name
target: vCluster # deploy into the cluster
templateRef:
name: cluster-collector
parameters:
destinationNamespace: observability
project: default
kubectl apply -f fleet-edge-collectors.yaml
The collectors verify the gateway against the delivered CA, so the gateway's self-signed certificate works without extra configuration.
For control plane cluster targets, manual collector configuration, and the full parameter list, see Configure edge collectors.
5. Verify​
Check that metrics are reaching the gateway. Run the port-forward in the background and
capture its PID with $!. --retry-connrefused makes curl wait for the tunnel instead
of racing it:
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 &
PF_PID=$!
curl --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:8889/metrics
kill $PF_PID
Gateway metrics only confirm a write request arrived, not that it reached the backend.
Create a metrics-reader key scoped to the same cluster and query the Query Proxy
for your first metric. Set PROJECT and TENANT_CLUSTER to the cluster's project and
instance name. See Create a reader key for the
full parameter list:
PROJECT=default
TENANT_CLUSTER=my-vcluster
METRICS_READER_TOKEN=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
name: "loft-metrics-reader-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
type: Other
key: "${METRICS_READER_TOKEN}"
subject: "loft:metrics-reader:p-${PROJECT}:${TENANT_CLUSTER}"
groups:
- loft:system:metrics-readers
scope:
roles:
- role: metrics-reader
virtualClusters:
- project: ${PROJECT}
virtualCluster: "${TENANT_CLUSTER}"
EOF
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8081:8081 &
PF_PID=$!
curl -k -G --retry 5 --retry-connrefused --retry-delay 1 \
-H "Authorization: Bearer $METRICS_READER_TOKEN" \
-H "X-Vcluster-Platform-Project: $PROJECT" \
-H "X-Vcluster-Platform-Instance: $TENANT_CLUSTER" \
--data-urlencode 'query=up' \
https://127.0.0.1:8081/api/v1/query
kill $PF_PID
A non-empty result array confirms the write reached the backend and is queryable.
If writes aren't showing up, see Troubleshooting.
6. Deploy the bundled Grafana (optional)​
Deploy Grafana for pre-provisioned dashboards:
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-grafana
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet Grafana"
destination:
cluster:
name: loft-cluster # the reserved name for Platform's local cluster; don't change this
templateRef:
name: fleet-observability-grafana
parameters:
destinationNamespace: vcluster-platform
platformHost: platform.example.com
project: default
kubectl apply -f fleet-grafana.yaml
Once the Grafana pod is running, sign in at https://<platform-host>/grafana/ with
Platform SSO to see the pre-provisioned Fleet Observability, Cluster Metrics, and GPU
Overview dashboards. For the connector settings Grafana needs and the full dashboard
list, see
Deploy the bundled Grafana with Argo CD.
Next steps​
- Query fleet metrics for dashboards and PromQL access.
- GPU observability templates to add the GPU stack and NVSentinel.
- Troubleshooting if something isn't working.