Skip to main content
Version: v4.11 Stable

Fleet Observability quickstart

This walks through the default, out-of-the-box path. It deploys the bundled Prometheus backend and edge collectors with Argo CD to one tenant cluster, using YAML and kubectl throughout. Each step links to a reference page with the full parameter list and the Platform UI walkthrough. See those pages for alternatives, such as bringing your own backend or targeting a control plane cluster.

Prerequisites​

  • The Argo CD integration is enabled on both the cluster where Platform is installed and the tenant cluster you collect from, each referencing an Argo CD connector. See Connect to Argo CD.
  • A vCluster Platform license that includes Fleet Observability.
  • kubectl access to a project namespace. Platform creates a namespace named p-<project-name> for each project, so the default project's namespace is p-default. Installations with a customized project namespace prefix, including installations upgraded from older versions, can use a different namespace.
  • A tenant cluster to collect metrics from.

1. Deploy the metrics backend​

Deploy the bundled Prometheus backend to the local cluster, the cluster where Platform itself is installed:

fleet-metrics-backend.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
  name: fleet-metrics-backend
  namespace: p-my-project          # replace with your project namespace
spec:
  displayName: "Fleet metrics backend"
  destination:
    cluster:
      name: loft-cluster           # the reserved name for Platform's local cluster; don't change this
  templateRef:
    name: fleet-observability-prometheus
  parameters:
    destinationNamespace: vcluster-platform
    prometheusStorageSize: 20Gi
    project: default
kubectl apply -f fleet-metrics-backend.yaml

Confirm the application landed in your project's namespace:

kubectl get argocdapplications -n p-default

Substitute your project's namespace if you're not using the default project.

For other backends, the Platform UI, and the full parameter list, see Deploy the metrics backend with Argo CD.

2. Create the connector and deploy the gateway​

Create the observability connector Secret. These endpoints match the backend's default namespace from step 1. The Secret below also configures the connector for the bundled Grafana deployed in step 6, using that template's default certificate Secret name:

fleet-observability-connector.yaml
apiVersion: v1
kind: Secret
metadata:
  name: fleet-observability
  namespace: vcluster-platform
  labels:
    loft.sh/connector-type: observability
    platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
  metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
  metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
  metricsBackendCertSecretName: metrics-backend-mtls
  grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local
  grafanaCertSecretName: grafana-server-tls
  fleetObservabilityDashboardUid: vcluster-fleet-observability
kubectl apply -f fleet-observability-connector.yaml

Platform deploys the gateway from this Secret. Check that the gateway pod comes up:

kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability

For connector keys, TLS options, and the Platform UI, see Install the observability gateway.

3. Create a metrics-writer key​

Create a metrics-writer access key scoped to your tenant cluster. Set PROJECT and TENANT_CLUSTER to your tenant cluster's project and instance name. The command prints the generated token as token: <value> at the end.

Modify the following with your specific values to generate a copyable command:
PROJECT=default
TENANT_CLUSTER=my-vcluster
KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
  name: "loft-metrics-writer-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
  type: Other
  key: "${KEY}"
  subject: "loft:metrics-writer:p-${PROJECT}:${TENANT_CLUSTER}"
  groups:
    - loft:system:metrics-writers
  scope:
    roles:
      - role: metrics-writer
    virtualClusters:
      - project: ${PROJECT}
        virtualCluster: "${TENANT_CLUSTER}"
EOF
echo "token: ${KEY}"

Keep the printed token for the next step.

For control plane cluster keys, reader keys, and rotation, see Configure metrics access.

4. Deploy edge collectors​

The token from step 3 goes into a Secret inside the tenant cluster, in the namespace the collectors run in. Switch your kubectl context to the tenant cluster, then create the namespace and the Secret:

kubectl create namespace observability
otel-otlp-auth.yaml
apiVersion: v1
kind: Secret
metadata:
  name: otel-otlp-auth
  namespace: observability
type: Opaque
stringData:
  authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY"
kubectl apply -f otel-otlp-auth.yaml

Switch your kubectl context back to the cluster where Platform runs before continuing. The ArgoCDApplication you apply next, and its project namespace, are Platform custom resources that only exist on that cluster, not on the tenant cluster.

Then deploy the collector to your tenant cluster. Edge collectors run inside the tenant cluster, which resolves DNS in its own zone, so otlpEndpoint needs to point somewhere the tenant cluster can actually reach. Which value to use depends on where the tenant cluster runs:

If the tenant cluster runs on the same control plane cluster as Platform, map the gateway Service into the tenant cluster's DNS instead of exposing it externally. This requires shared nodes tenancy. Add this to the tenant cluster's vcluster.yaml:

controlPlane:
  coredns:
    enabled: true
    embedded: true
networking:
  resolveDNS:
    - service: vcluster-platform/gateway-fleet-observability
      target:
        hostService: vcluster-platform/gateway-fleet-observability

See Resolve DNS for how to apply vcluster.yaml changes and the full option reference. otlpEndpoint keeps using the gateway's in-cluster Service name:

fleet-edge-collectors.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
  name: fleet-edge-collectors-my-vcluster   # include the tenant cluster name for uniqueness
  namespace: p-my-project          # replace with your project namespace
spec:
  displayName: "Fleet edge collectors"
  destination:
    virtualCluster:
      name: my-vcluster            # replace with your tenant cluster name
      target: vCluster             # deploy into the tenant cluster
  templateRef:
    name: cluster-collector
  parameters:
    destinationNamespace: observability
    otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318
    otlpInsecureSkipVerify: "true"
    project: default

Both branches keep otlpInsecureSkipVerify: "true" to accept the gateway's self-signed certificate for this quickstart. For a production setup, distribute the gateway's ca.crt instead, as described in Customize the gateway serving certificate.

kubectl apply -f fleet-edge-collectors.yaml

For control plane cluster targets, manual collector configuration, and the full parameter list, see Configure edge collectors.

5. Verify​

Check that metrics are reaching the gateway. Run the port-forward in the background and capture its PID with $!. --retry-connrefused makes curl wait for the tunnel instead of racing it:

GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 &
PF_PID=$!
curl --retry 5 --retry-connrefused --retry-delay 1 http://127.0.0.1:8889/metrics
kill $PF_PID

Gateway metrics only confirm a write request arrived, not that it reached the backend. Create a metrics-reader key scoped to the same tenant cluster and query the Query Proxy for your first metric. Match PROJECT and TENANT_CLUSTER to the values you used in step 3. See Create a reader key for the full parameter list:

PROJECT=default
TENANT_CLUSTER=my-vcluster
METRICS_READER_TOKEN=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n')
kubectl apply -f - <<EOF
apiVersion: storage.loft.sh/v1
kind: AccessKey
metadata:
  name: "loft-metrics-reader-p-${PROJECT}-${TENANT_CLUSTER}"
spec:
  type: Other
  key: "${METRICS_READER_TOKEN}"
  subject: "loft:metrics-reader:p-${PROJECT}:${TENANT_CLUSTER}"
  groups:
    - loft:system:metrics-readers
  scope:
    roles:
      - role: metrics-reader
    virtualClusters:
      - project: ${PROJECT}
        virtualCluster: "${TENANT_CLUSTER}"
EOF

GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8081:8081 &
PF_PID=$!
curl -k -G --retry 5 --retry-connrefused --retry-delay 1 \
  -H "Authorization: Bearer $METRICS_READER_TOKEN" \
  -H "X-Vcluster-Platform-Project: $PROJECT" \
  -H "X-Vcluster-Platform-Instance: $TENANT_CLUSTER" \
  --data-urlencode 'query=up' \
  https://127.0.0.1:8081/api/v1/query
kill $PF_PID

A non-empty result array confirms the write reached the backend and is queryable.

If writes aren't showing up, see Troubleshooting.

6. Deploy the bundled Grafana (optional)​

Deploy Grafana for pre-provisioned dashboards:

fleet-grafana.yaml
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
  name: fleet-grafana
  namespace: p-my-project          # replace with your project namespace
spec:
  displayName: "Fleet Grafana"
  destination:
    cluster:
      name: loft-cluster           # the reserved name for Platform's local cluster; don't change this
  templateRef:
    name: fleet-observability-grafana
  parameters:
    destinationNamespace: vcluster-platform
    platformHost: platform.example.com
    project: default
kubectl apply -f fleet-grafana.yaml

Once the Grafana pod is running, sign in at https://<platform-host>/grafana/ with Platform SSO to see the pre-provisioned Fleet Observability, Cluster Metrics, and GPU Overview dashboards. For the connector settings Grafana needs and the full dashboard list, see Deploy the bundled Grafana with Argo CD.

Next steps​