Skip to main content
Version: main 🚧

Fleet Observability quickstart

This walks through the default, out-of-the-box path. It deploys the bundled Prometheus backend and edge collectors with Argo CD to one cluster, using YAML and kubectl throughout. Each step links to a reference page with the full parameter list and the Platform UI walkthrough. See those pages for alternatives, such as bringing your own backend or targeting a control plane cluster.

Prerequisites​

  • The Argo CD integration is enabled on both the cluster where Platform is installed and the cluster you collect from, each referencing an Argo CD connector. See Connect to Argo CD.
  • kubectl access to a project namespace. Platform creates a namespace named p-<project-name> for each project, so the default project's namespace is p-default. Installations with a customized project namespace prefix, including installations upgraded from older versions, can use a different namespace.
  • A cluster to collect metrics from.

1. Deploy the metrics backend​

Deploy the bundled Prometheus backend to the local cluster, the cluster where Platform itself is installed:

fleet-metrics-backend.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-metrics-backend
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet metrics backend"
destination:
cluster:
name: loft-cluster # the reserved name for Platform's local cluster; don't change this
templateRef:
name: fleet-observability-prometheus
parameters:
destinationNamespace: vcluster-platform
prometheusStorageSize: 20Gi
project: default
kubectl apply -f fleet-metrics-backend.yaml

Confirm the application landed in your project's namespace:

kubectl get argocdapplications -n p-default

Substitute your project's namespace if you're not using the default project.

For other backends, the Platform UI, and the full parameter list, see Deploy the metrics backend with Argo CD.

2. Create the connector and deploy the gateway​

Create the observability connector Secret. These endpoints match the backend's default namespace from step 1. The Secret below also configures the connector for the bundled Grafana deployed in step 6, using that template's default certificate Secret name. The same connector is also available through the Connector API. See Configure an observability connector.

fleet-observability-connector.yaml
apiVersion: v1
kind: Secret
metadata:
name: fleet-observability
namespace: vcluster-platform
labels:
loft.sh/connector-type: observability
platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
metricsBackendCertSecretName: metrics-backend-mtls
grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local
grafanaCertSecretName: grafana-server-tls
fleetObservabilityDashboardUid: vcluster-fleet-observability
kubectl apply -f fleet-observability-connector.yaml

Platform deploys the gateway from this Secret. Check that the gateway pod comes up:

kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability

For connector keys, TLS options, and the Platform UI, see Install the observability gateway.

3. Turn on the observability integration​

Turn on the observability integration in the cluster's vcluster.yaml, referencing the connector from step 2:

vcluster.yaml
integrations:
observability:
enabled: true
connector: fleet-observability

Platform then issues a metrics-writer access key scoped to the cluster, and the cluster writes it into a metrics-writer Secret in its observability namespace, together with the gateway endpoint and the CA that issued the gateway's serving certificate. You don't create the key or the Secret yourself. The cluster refreshes the Secret automatically, replacing a deleted key. Collectors reload a rotated token after the kubelet refreshes the mounted Secret, without a restart. See Rotation for timing and the gap in metrics after deleting a key.

The endpoint Platform hands out depends on where the cluster runs:

If the cluster uses shared nodes on the control plane cluster that runs Platform, Platform hands out the gateway's in-cluster Service name. Map that Service into the tenant cluster's DNS so the collectors can resolve it. Add this to the same vcluster.yaml:

vcluster.yaml
controlPlane:
coredns:
enabled: true
embedded: true
networking:
resolveDNS:
- service: vcluster-platform/gateway-fleet-observability
target:
hostService: vcluster-platform/gateway-fleet-observability

See Resolve DNS for how to apply vcluster.yaml changes and the full option reference.

For control plane cluster keys, reader keys, and rotation, see Configure metrics access.

4. Deploy edge collectors​

Deploy the collector to your cluster. The collectors read the endpoint, token, and CA from the metrics-writer Secret from step 3, so the application only names the destination:

fleet-edge-collectors.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-edge-collectors-my-vcluster # include the cluster name for uniqueness
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet edge collectors"
destination:
virtualCluster:
name: my-vcluster # replace with your cluster name
target: vCluster # deploy into the cluster
templateRef:
name: cluster-collector
parameters:
destinationNamespace: observability
project: default
kubectl apply -f fleet-edge-collectors.yaml

The collectors verify the gateway against the delivered CA, so the gateway's self-signed certificate works without extra configuration.

For control plane cluster targets, manual collector configuration, and the full parameter list, see Configure edge collectors.

5. Verify​

Check that metrics are reaching the gateway. Run the port-forward in the background and capture its PID with $!. --retry-connrefused makes curl wait for the tunnel instead of racing it:

GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 &
PF_PID=$!
curl --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:8889/metrics
kill $PF_PID

Gateway metrics only confirm a write request arrived, not that it reached the backend. Create a metrics-reader key scoped to the same cluster and query the Query Proxy for your first metric. Set PROJECT and TENANT_CLUSTER to the cluster's project and instance name. See Create a reader key for the full parameter list:

PROJECT=default
TENANT_CLUSTER=my-vcluster
METRICS_READER_TOKEN=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
name: "loft-metrics-reader-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
type: Other
key: "${METRICS_READER_TOKEN}"
subject: "loft:metrics-reader:p-${PROJECT}:${TENANT_CLUSTER}"
groups:
- loft:system:metrics-readers
scope:
roles:
- role: metrics-reader
virtualClusters:
- project: ${PROJECT}
virtualCluster: "${TENANT_CLUSTER}"
EOF

GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8081:8081 &
PF_PID=$!
curl -k -G --retry 5 --retry-connrefused --retry-delay 1 \
-H "Authorization: Bearer $METRICS_READER_TOKEN" \
-H "X-Vcluster-Platform-Project: $PROJECT" \
-H "X-Vcluster-Platform-Instance: $TENANT_CLUSTER" \
--data-urlencode 'query=up' \
https://127.0.0.1:8081/api/v1/query
kill $PF_PID

A non-empty result array confirms the write reached the backend and is queryable.

If writes aren't showing up, see Troubleshooting.

6. Deploy the bundled Grafana (optional)​

Deploy Grafana for pre-provisioned dashboards:

fleet-grafana.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-grafana
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet Grafana"
destination:
cluster:
name: loft-cluster # the reserved name for Platform's local cluster; don't change this
templateRef:
name: fleet-observability-grafana
parameters:
destinationNamespace: vcluster-platform
platformHost: platform.example.com
project: default
kubectl apply -f fleet-grafana.yaml

Once the Grafana pod is running, sign in at https://<platform-host>/grafana/ with Platform SSO to see the pre-provisioned Fleet Observability, Cluster Metrics, and GPU Overview dashboards. For the connector settings Grafana needs and the full dashboard list, see Deploy the bundled Grafana with Argo CD.

Next steps​