Install the observability gateway

The observability gateway runs in the vCluster Platform namespace. It contains two containers:

• otel-gateway, the Write Gateway that receives OTLP.
• query-proxy, the read proxy that serves a Prometheus-compatible query API for PromQL reads.

Platform deploys the gateway from an observability connector Secret. The connector stores the metrics backend endpoints and optional TLS settings.

Prerequisites

Before you create the connector, prepare:

• A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for queries. It doesn't need to run on the same cluster as Platform. It can run anywhere the gateway can reach over the network. Deploy the bundled Prometheus out of the box with the Argo CD integration, or bring your own.
• A backend that accepts TLS or mTLS as the sole authentication method. TLS verifies the backend's identity, and mTLS additionally gives the gateway a client identity the backend can authenticate. There's no way to attach a bearer token, basic auth, or a static header, and the Query Proxy strips the caller's own Authorization header before forwarding reads, so backends that mandate bearer, basic, or API-key authentication aren't supported.
• Network access from the Platform namespace to both backend endpoints, wherever the backend runs.
• A vCluster Platform license that includes Fleet Observability.
• Optional custom gateway serving certificate or backend mTLS Secrets.

note

The observability gateway doesn't install or size the metrics backend for you. Bring your own, or deploy the bundled Prometheus backend with Argo CD as described below.

Deploy the metrics backend with Argo CD

If you use the Argo CD integration, deploy the bundled Prometheus backend with the fleet-observability-prometheus ArgoCDApplicationTemplate instead of installing and sizing a backend by hand. The backend runs on the local cluster, the cluster where Platform itself is installed. Deploy it once, before you create the connector below.

Deploy an ArgoCDApplication that references fleet-observability-prometheus. This deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform namespace on the local cluster.

Platform UI

Platform creates a namespace named p-<project-name> for each project, so the default project's namespace is p-default.

Create the ArgoCDApplication object in your project namespace:

fleet-metrics-backend.yaml

apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
  name: fleet-metrics-backend
  namespace: p-my-project          # replace with your project namespace
spec:
  displayName: "Fleet metrics backend"
  destination:
    cluster:
      name: loft-cluster           # the reserved name for Platform's local cluster; don't change this
  templateRef:
    name: fleet-observability-prometheus
  parameters:
    destinationNamespace: vcluster-platform
    prometheusStorageSize: 20Gi
    project: default

YAML

1. Go to Infrastructure > Control Plane Clusters, then click the local cluster (named loft-cluster by default) to open its detail page.

2. Select the ArgoCD Apps tab and click .

3. In the Basics section, enter a Display Name such as Fleet metrics backend.

4. From the Argo CD Application Template dropdown, select the Fleet Observability: Prometheus template.

5. Fill the Template Parameters. Set destinationNamespace to the namespace Platform is installed in, which defaults to vcluster-platform, and adjust the storage size or chart version if needed.

6. Click .

Parameter	Default	Description
destinationNamespace	vcluster-platform	Namespace for the Prometheus backend.
project	default	Argo CD project that owns the application.
repoURL	Prometheus community Helm repo	Helm repository for the Prometheus chart.
version	29.14.0	Prometheus Helm chart version.
prometheusStorageSize	20Gi	Persistent volume size for Prometheus storage.
metricsBackendTLSSecret	(empty)	Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named metrics-backend-mtls.
resources	(empty)	Resource requests and limits for the Prometheus server container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits.

resources is a multiline parameter that takes the contents of a Kubernetes resources field.

fleet-metrics-backend.yaml

spec:
  parameters:
    resources: |
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        memory: 2Gi

The connector you create below must use these endpoints when the template is deployed in its default namespace:

stringData:
  metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
  metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
  metricsBackendCertSecretName: metrics-backend-mtls

The /api/v1/otlp path is required for OTLP ingestion. If you change destinationNamespace, replace vcluster-platform in both endpoints. If you supply metricsBackendTLSSecret, use that Secret name instead of metrics-backend-mtls.

If destinationNamespace differs from the namespace where Platform is installed, Prometheus and the gateway no longer share a namespace, so the auto-generated mTLS Secret can't serve both sides on its own. See Configure backend mTLS across namespaces or clusters.

Create the connector Secret

Create the Fleet Observability connector in the Platform namespace. Use no more than 51 lowercase letters, numbers, or hyphens for the connector ID, like the fleet-observability example. That keeps every name Platform derives from it predictable: the gateway resources as gateway-<connector-id>, the managed certificate Secret as gateway-<connector-id>-tls (see Customize the gateway serving certificate), and the app.kubernetes.io/instance label as the connector ID itself.

A longer or dotted connector ID is a technically valid Secret name, but Platform normalizes the derived names inconsistently, so they stop matching each other. Stick to a short, dot-free ID instead of working around this.

The loft.sh/connector-type: observability label is required. Platform only recognizes and reconciles a Secret as an observability connector when this label is present.

Mark the connector as the fleet default with the platform.vcluster.com/fleet-observability-connector label set to "true". Only one connector can be the fleet default.

Platform UI

fleet-observability-connector.yaml

apiVersion: v1
kind: Secret
metadata:
  name: fleet-observability
  namespace: vcluster-platform
  labels:
    loft.sh/connector-type: observability
    platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
  metricsBackendOtlpEndpoint: https://metrics.example.com/otlp
  metricsBackendPromQLEndpoint: https://metrics.example.com

Apply the Secret:

kubectl apply -f fleet-observability-connector.yaml

YAML

1. Click Connectors and select the Observability tab.

2. Click .

3. In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default.

4. In the Metrics Backend OTLP Endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway.

5. In the Metrics Backend PromQL Endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy.

6. Optionally configure TLS for the metrics backend and the gateway. See Connector keys for the available settings.

7. Click .

Platform reconciles the following resources:

• Deployment/gateway-fleet-observability
• Service/gateway-fleet-observability
• Grafana datasource ConfigMaps for fleet, control plane cluster, and tenant cluster views when Grafana datasource reconciliation is enabled.

Connector keys

Key	Required	Description
metricsBackendOtlpEndpoint	Yes	OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends /v1/metrics (for example, https://prom-prometheus-server.vcluster-platform/api/v1/otlp receives writes at /api/v1/otlp/v1/metrics).
metricsBackendPromQLEndpoint	Yes	Prometheus HTTP API endpoint queried by the Query Proxy.
metricsBackendInsecureSkipVerify	No	Set to "true" only for test backends with unverifiable TLS.
metricsBackendServerName	No	TLS server name override for the Query Proxy's PromQL reads. It doesn't affect OTLP writes; the OTLP endpoint's certificate must still match the endpoint hostname.
observabilityGatewayCertSecretName	No	Custom Secret with tls.crt and tls.key for gateway serving TLS. Include ca.crt when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted.
metricsBackendCertSecretName	No	Secret with ca.crt, tls.crt, and tls.key for mTLS to the backend. The gateway trusts the backend with ca.crt and presents the certificate and key as its client identity.
gatewayResources	No	YAML resources block for the Write Gateway container.
queryProxyResources	No	YAML resources block for the Query Proxy container.
serviceType	No	Gateway Service type. Valid values are ClusterIP, the default, and LoadBalancer.
serviceAnnotations	No	YAML map of annotations applied to a LoadBalancer gateway Service.
grafanaUrl	No	Grafana upstream URL used by the Platform /grafana/ reverse proxy.
grafanaInsecureSkipVerify	No	Set to "true" only to test a Grafana endpoint with an unverifiable certificate.
grafanaCertSecretName	No	Grafana serving certificate Secret. Platform uses its ca.crt to trust the Grafana upstream.
fleetObservabilityDashboardUid	No	UID of the fleet dashboard embedded by the Fleet Observability page.

Bring-your-own Grafana requirements

When you point grafanaUrl at a Grafana you manage instead of the bundled Grafana, that Grafana must:

• Serve from the /grafana subpath. Set server.root_url and server.serve_from_sub_path so Grafana works behind the Platform /grafana/ reverse proxy. See serve_from_sub_path.
• Allow embedding. Set security.allow_embedding = true so the Fleet Observability page and each tenant cluster's Observability tab can embed the dashboards in an iframe. See allow_embedding.

Customize the gateway serving certificate

The gateway always serves OTLP and query traffic over TLS. When you don't specify a certificate Secret, Platform creates and manages a self-signed Secret named gateway-<connector-id>-tls. This only works for connector IDs of 51 characters or fewer; see Create the connector Secret for why longer IDs need a custom certificate Secret instead.

To use your own certificate, create a Secret in the Platform namespace. Platform never validates or modifies a custom certificate, so it must already cover every hostname a client connects to. At minimum, include the gateway's in-cluster Service hostname, gateway-<connector-id>.<platform-namespace>.svc.cluster.local, since Platform-generated Grafana datasources and any in-cluster collector always connect through it. If you expose the gateway with a LoadBalancer (see Expose the gateway with a LoadBalancer below), also include its external hostname or IP as a SAN.

gateway-tls.yaml

apiVersion: v1
kind: Secret
metadata:
  name: fleet-gateway-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Reference it from the connector:

stringData:
  observabilityGatewayCertSecretName: fleet-gateway-tls

Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom certificate cases. Distribute ca.crt to collectors that don't already trust the issuing CA. Grafana datasources generated by Platform trust the ca.crt from the gateway serving certificate Secret. A certificate signed by a public CA still needs to cover the hostnames above; once it does, ca.crt can be omitted, since clients already trust the public CA through their system trust stores.

Expose the gateway with a LoadBalancer

The gateway Service defaults to ClusterIP. To make the gateway reachable from collectors and query clients outside the Platform cluster, set serviceType to LoadBalancer. You can optionally apply provider-specific Service annotations:

stringData:
  serviceType: LoadBalancer
  serviceAnnotations: |
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal

For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. A custom certificate needs that same SAN added by hand, since Platform never modifies it. See Customize the gateway serving certificate for how clients trust the resulting certificate either way.

Retrieve the load balancer address

The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \
  -o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}'

The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address:

Endpoint	Port	Purpose
https://<loadbalancer-address>:4318	4318	OTLP HTTP writes. Use this for collectors.
https://<loadbalancer-address>:4317	4317	OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use 4318 for collectors.
https://<loadbalancer-address>:8081	8081	Prometheus HTTP API through the Query Proxy. Use this for external query clients.

Use this address wherever a collector or query client connects from outside the Platform cluster.

Use backend mTLS

To authenticate the gateway to your backend with mTLS, create a Secret:

backend-client-tls.yaml

apiVersion: v1
kind: Secret
metadata:
  name: metrics-backend-client-tls
  namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN PRIVATE KEY-----
    ...
    -----END PRIVATE KEY-----

Reference it from the connector:

stringData:
  metricsBackendCertSecretName: metrics-backend-client-tls

Configure backend mTLS across namespaces or clusters

The bundled Prometheus template's default mTLS setup works because Prometheus and the gateway both mount the same Secret object. By default they share the namespace Platform itself is installed in (vcluster-platform, unless you renamed it), so metricsBackendTLSSecret (the Prometheus side) and metricsBackendCertSecretName (the connector side) both point at the same auto-generated metrics-backend-mtls Secret.

That single-Secret shortcut only works when Prometheus and the gateway share a namespace and a cluster. What to do instead depends on where Prometheus runs.

Different namespace, same cluster

If you set destinationNamespace to a namespace other than the one Platform is installed in, Prometheus stays on the local cluster, so its bootstrap certificate still covers the right in-cluster hostname (prom-prometheus-server.<destinationNamespace>.svc.cluster.local). A single Secret object just can't be mounted in two namespaces at once, so copy it instead of sharing one object:

• Leave metricsBackendTLSSecret empty so the Argo CD template bootstraps the shared certificate for you, named metrics-backend-mtls in Prometheus's destinationNamespace. See Deploy the metrics backend with Argo CD.
• Once Prometheus reports ready, read that generated Secret and create an identical copy in the Platform namespace, then reference it from the connector's metricsBackendCertSecretName.

Setting metricsBackendTLSSecret to a name of your own disables this bootstrap entirely, so only use it if you're supplying your own certificate material for both Secrets instead of copying the generated one.

A different cluster, or a backend other than the bundled Prometheus

The bundled workflow above deploys Prometheus to the local cluster, and its bootstrap certificate only covers in-cluster Service hostnames. Copying that Secret doesn't work once the gateway reaches the backend through an external hostname, because the certificate has no SAN for it. Create the gateway-side Secret as described in Use backend mTLS above, but with two differences:

• The backend's server certificate must cover the hostname used in metricsBackendOtlpEndpoint and metricsBackendPromQLEndpoint, not an in-cluster Service name.
• Configure the backend to trust the CA that issues the gateway's client certificate, the same way the bundled Prometheus config trusts it through client_ca_file. Without this, the backend rejects the gateway's mTLS handshake even if the gateway trusts the backend.

Verify the gateway

Check that the gateway pod is running:

CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"

Check the service ports:

CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"

Expected service ports:

Port	Name	Purpose
4317	otlp	OTLP gRPC writes.
4318	otlp-http	OTLP HTTP writes.
8081	query	Prometheus HTTP API through the Query Proxy.

Next, create metrics access keys in Configure metrics access.