Skip to main content
Version: v5.0 Stable

Install the observability gateway

The observability gateway runs in the vCluster Platform namespace. It contains two containers:

  • otel-gateway, the Write Gateway that receives OTLP.
  • query-proxy, the read proxy that serves a Prometheus-compatible query API for PromQL reads.

Platform deploys the gateway from an observability connector Secret. The connector stores the metrics backend endpoints and optional TLS settings.

Prerequisites​

Before you create the connector, prepare:

  • A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for queries. It doesn't need to run on the same cluster as Platform. It can run anywhere the gateway can reach over the network. Deploy the bundled Prometheus out of the box with the Argo CD integration, or bring your own.
  • A backend that accepts TLS or mTLS as the sole authentication method. TLS verifies the backend's identity, and mTLS additionally gives the gateway a client identity the backend can authenticate. Without an mTLS Secret, the gateway verifies the backend against the system trust store only. There's no setting for a CA alone, so a backend with a private CA needs the mTLS Secret, or metricsBackendInsecureSkipVerify for testing. There's no way to attach a bearer token, basic auth, or a static header, and the Query Proxy strips the caller's own Authorization header before forwarding reads, so backends that mandate bearer, basic, or API-key authentication aren't supported.
  • Network access from the Platform namespace to both backend endpoints, wherever the backend runs.
  • Optional custom gateway serving certificate or backend mTLS Secrets.
note

The observability gateway doesn't install or size the metrics backend for you. Bring your own, or deploy the bundled Prometheus backend with Argo CD as described below.

Deploy the metrics backend with Argo CD​

If you use the Argo CD integration, deploy the bundled Prometheus backend with the fleet-observability-prometheus ArgoCDApplicationTemplate instead of installing and sizing a backend by hand. The backend runs on the local cluster, the cluster where Platform itself is installed. Deploy it once, before you create the connector below.

Deploy an ArgoCDApplication that references fleet-observability-prometheus. This deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform namespace on the local cluster.

  1. Go to Infra Management > Control Plane Clusters, then click the local cluster (named loft-cluster by default) to open its detail page.

  2. Select the Argo CD Apps tab and click .

  3. In the Basics section, enter a Display name such as Fleet metrics backend.

  4. From the Project dropdown, select the project that owns the application, such as default.

  5. Under Source, select From template.

  6. From the Argo CD Application Template dropdown, select the Fleet Observability: Prometheus template.

  7. Fill the Template Parameters. Set Destination Namespace to the namespace Platform is installed in, which defaults to vcluster-platform. Adjust Prometheus storage size or Version if needed.

  8. Click .

ParameterDefaultDescription
destinationNamespacevcluster-platformNamespace for the Prometheus backend.
projectdefaultArgo CD project that owns the application.
repoURLPrometheus community Helm repoHelm repository for the Prometheus chart.
version29.39.0Prometheus Helm chart version.
prometheusStorageSize20GiPersistent volume size for Prometheus storage.
metricsBackendTLSSecret(empty)Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named metrics-backend-mtls.
resources(empty)Resource requests and limits for the Prometheus server container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits.

resources is a multiline parameter that takes the contents of a Kubernetes resources field.

fleet-metrics-backend.yaml
spec:
parameters:
resources: |
requests:
cpu: 500m
memory: 2Gi
limits:
memory: 2Gi

The connector you create below must use these endpoints when the template is deployed in its default namespace:

stringData:
metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
metricsBackendCertSecretName: metrics-backend-mtls

The /api/v1/otlp path is required for OTLP ingestion. If you change destinationNamespace, replace vcluster-platform in both endpoints. If you supply metricsBackendTLSSecret, use that Secret name instead of metrics-backend-mtls.

If destinationNamespace differs from the namespace where Platform is installed, Prometheus and the gateway no longer share a namespace, so the auto-generated mTLS Secret can't serve both sides on its own. See Configure backend mTLS across namespaces or clusters.

Create the connector Secret​

Create the Fleet Observability connector in the Platform namespace. Use no more than 51 lowercase letters, numbers, or hyphens for the connector ID, like the fleet-observability example. That keeps every name Platform derives from it predictable: the gateway resources as gateway-<connector-id>, the managed certificate Secret as gateway-<connector-id>-tls (see Customize the gateway serving certificate), and the app.kubernetes.io/instance label as the connector ID itself.

A longer or dotted connector ID is a technically valid Secret name, but Platform normalizes the derived names inconsistently, so they stop matching each other. Stick to a short, dot-free ID instead of working around this.

The loft.sh/connector-type: observability label is required. Platform only recognizes and reconciles a Secret as an observability connector when this label is present.

Mark the connector as the fleet default with the platform.vcluster.com/fleet-observability-connector label set to "true". Only one connector can be the fleet default.

  1. Click Secrets & Credentials > Connectors and select the Observability tab.

  2. Click .

  3. In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default.

  4. In the Metrics backend OTLP endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway.

  5. In the Metrics backend PromQL endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy. Both endpoint fields start with the bundled Prometheus addresses in the vcluster-platform namespace. Keep them if you deployed the bundled Prometheus there.

  6. Choose the gateway's Service type. Choose LoadBalancer when clusters whose collectors run outside the Platform cluster deliver metrics or logs through this connector.

  7. If you deployed the bundled Prometheus, select metrics-backend-mtls in Metrics backend client certificate secret (optional). The bundled backend rejects clients without a certificate. For other backends, configure TLS as needed. See Connector keys for the available settings.

  8. Grafana URL and Fleet observability dashboard UID start with the bundled Grafana values. Keep them if you'll deploy the bundled Grafana. After it's running, edit the connector and select grafana-server-tls in Grafana certificate secret (optional).

  9. Click .

Platform reconciles the following resources:

  • Deployment/gateway-fleet-observability
  • Service/gateway-fleet-observability
  • Grafana datasource ConfigMaps for fleet, control plane cluster, and cluster views when Grafana datasource reconciliation is enabled.

Connector keys​

KeyRequiredDescription
metricsBackendOtlpEndpointYesOTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends /v1/metrics (for example, https://prom-prometheus-server.vcluster-platform/api/v1/otlp receives writes at /api/v1/otlp/v1/metrics).
metricsBackendPromQLEndpointYesPrometheus HTTP API endpoint queried by the Query Proxy.
metricsBackendInsecureSkipVerifyNoSet to "true" only for test backends with unverifiable TLS.
metricsBackendServerNameNoTLS server name override for the Query Proxy's PromQL reads. It doesn't affect OTLP writes, so the OTLP endpoint's certificate must still match the endpoint hostname.
observabilityGatewayCertSecretNameNoCustom Secret with tls.crt and tls.key for gateway serving TLS. Include ca.crt when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted.
metricsBackendCertSecretNameNoSecret with ca.crt, tls.crt, and tls.key for mTLS to the backend. The gateway trusts the backend with ca.crt and presents the certificate and key as its client identity. ca.crt is optional when metricsBackendInsecureSkipVerify is "true".
gatewayResourcesNoYAML resources block for the Write Gateway container.
queryProxyResourcesNoYAML resources block for the Query Proxy container.
serviceTypeNoGateway Service type. Valid values are ClusterIP, the default, and LoadBalancer. Must be LoadBalancer when a cluster delivers metrics or logs through this connector and its collector runs outside the Platform cluster. See Which clusters need a LoadBalancer gateway.
serviceAnnotationsNoYAML map of annotations applied to a LoadBalancer gateway Service.
grafanaUrlYesGrafana that the Platform UI embeds on the Fleet Observability page and in each cluster's observability view. A Connector API create request that leaves it empty gets the Grafana that the Fleet Observability: Grafana Argo CD template deploys in the platform namespace.
grafanaInsecureSkipVerifyNoSet to "true" only to test a Grafana endpoint with an unverifiable certificate.
grafanaCertSecretNameNoGrafana serving certificate Secret. Platform uses its ca.crt to trust the Grafana upstream.
fleetObservabilityDashboardUidNoUID of the fleet dashboard embedded by the Fleet Observability page.
destinationsNoYAML list of extra OTLP HTTP endpoints that receive metrics or logs. Required for log collection. The Platform UI connector form has no field for it. See Forward telemetry to additional destinations.
Bring-your-own Grafana requirements

When you point grafanaUrl at a Grafana you manage instead of the bundled Grafana, that Grafana must:

  • Serve from the /grafana subpath. Set server.root_url and server.serve_from_sub_path so Grafana works behind the Platform /grafana/ reverse proxy. See serve_from_sub_path.
  • Allow embedding. Set security.allow_embedding = true so the Fleet Observability page and each cluster's Observability tab can embed the dashboards in an iframe. See allow_embedding.

Verify the gateway​

Check that the gateway pod is running:

CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"

Check the service ports:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"

Expected service ports:

PortNamePurpose
4317otlpOTLP gRPC writes.
4318otlp-httpOTLP HTTP writes.
8081queryPrometheus HTTP API through the Query Proxy.

Next, turn on the observability integration for each cluster and deploy the collectors, as described in Configure edge collectors. Create keys in Configure metrics access only for control plane clusters, collectors you configure manually, and readers.