Install the observability gateway
The observability gateway runs in the vCluster Platform namespace. It contains two containers:
otel-gateway, the Write Gateway that receives OTLP.query-proxy, the read proxy that serves a Prometheus-compatible query API for PromQL reads.
Platform deploys the gateway from an observability connector Secret. The connector stores the metrics backend endpoints and optional TLS settings.
Prerequisites​
Before you create the connector, prepare:
- A metrics backend that ingests OTLP metrics and serves the Prometheus HTTP API for queries. It doesn't need to run on the same cluster as Platform. It can run anywhere the gateway can reach over the network. Deploy the bundled Prometheus out of the box with the Argo CD integration, or bring your own.
- A backend that accepts TLS or mTLS as the sole authentication method. TLS verifies the
backend's identity, and mTLS additionally gives the gateway a client identity the backend
can authenticate. Without an mTLS Secret, the gateway verifies the backend against the
system trust store only. There's no setting for a CA alone, so a backend with a private CA
needs the mTLS Secret, or
metricsBackendInsecureSkipVerifyfor testing. There's no way to attach a bearer token, basic auth, or a static header, and the Query Proxy strips the caller's ownAuthorizationheader before forwarding reads, so backends that mandate bearer, basic, or API-key authentication aren't supported. - Network access from the Platform namespace to both backend endpoints, wherever the backend runs.
- Optional custom gateway serving certificate or backend mTLS Secrets.
The observability gateway doesn't install or size the metrics backend for you. Bring your own, or deploy the bundled Prometheus backend with Argo CD as described below.
Deploy the metrics backend with Argo CD​
If you use the Argo CD integration, deploy the
bundled Prometheus backend with the fleet-observability-prometheus
ArgoCDApplicationTemplate instead of installing and sizing a backend by hand. The backend
runs on the local cluster,
the cluster where Platform itself is installed. Deploy it once, before you create the
connector below.
Deploy an ArgoCDApplication that references fleet-observability-prometheus. This
deploys Prometheus as a pure OTLP receiver, sized for fleet metrics, into the Platform
namespace on the local cluster.
- Platform UI
- YAML
Platform creates a namespace named p-<project-name> for each project, so the default project's namespace is p-default.
Create the ArgoCDApplication object in your project namespace:
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-metrics-backend
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet metrics backend"
destination:
cluster:
name: loft-cluster # the reserved name for Platform's local cluster; don't change this
templateRef:
name: fleet-observability-prometheus
parameters:
destinationNamespace: vcluster-platform
prometheusStorageSize: 20Gi
project: default
Go to Infra Management > Control Plane Clusters, then click the local cluster (named
loft-clusterby default) to open its detail page.Select the Argo CD Apps tab and click .
In the Basics section, enter a Display name such as
Fleet metrics backend.From the Project dropdown, select the project that owns the application, such as
default.Under Source, select From template.
From the Argo CD Application Template dropdown, select the Fleet Observability: Prometheus template.
Fill the Template Parameters. Set Destination Namespace to the namespace Platform is installed in, which defaults to
vcluster-platform. Adjust Prometheus storage size or Version if needed.Click .
| Parameter | Default | Description |
|---|---|---|
destinationNamespace | vcluster-platform | Namespace for the Prometheus backend. |
project | default | Argo CD project that owns the application. |
repoURL | Prometheus community Helm repo | Helm repository for the Prometheus chart. |
version | 29.39.0 | Prometheus Helm chart version. |
prometheusStorageSize | 20Gi | Persistent volume size for Prometheus storage. |
metricsBackendTLSSecret | (empty) | Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named metrics-backend-mtls. |
resources | (empty) | Resource requests and limits for the Prometheus server container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits. |
resources is a multiline parameter that takes the contents of a Kubernetes resources
field.
spec:
parameters:
resources: |
requests:
cpu: 500m
memory: 2Gi
limits:
memory: 2Gi
The connector you create below must use these endpoints when the template is deployed in its default namespace:
stringData:
metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp
metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local
metricsBackendCertSecretName: metrics-backend-mtls
The /api/v1/otlp path is required for OTLP ingestion. If you change
destinationNamespace, replace vcluster-platform in both endpoints. If you supply
metricsBackendTLSSecret, use that Secret name instead of metrics-backend-mtls.
If destinationNamespace differs from the namespace where Platform is installed,
Prometheus and the gateway no longer share a namespace, so the auto-generated mTLS Secret
can't serve both sides on its own. See
Configure backend mTLS across namespaces or clusters.
Create the connector Secret​
Create the Fleet Observability connector in the Platform namespace. Use no more than 51
lowercase letters, numbers, or hyphens for the connector ID, like the
fleet-observability example. That keeps every name Platform derives from it predictable:
the gateway resources as gateway-<connector-id>, the managed certificate Secret as
gateway-<connector-id>-tls (see Customize the gateway serving
certificate), and the
app.kubernetes.io/instance label as the connector ID itself.
A longer or dotted connector ID is a technically valid Secret name, but Platform normalizes the derived names inconsistently, so they stop matching each other. Stick to a short, dot-free ID instead of working around this.
The loft.sh/connector-type: observability label is required. Platform only recognizes and
reconciles a Secret as an observability connector when this label is present.
Mark the connector as the fleet default with the
platform.vcluster.com/fleet-observability-connector label set to "true". Only one
connector can be the fleet default.
- Platform UI
- YAML
apiVersion: v1
kind: Secret
metadata:
name: fleet-observability
namespace: vcluster-platform
labels:
loft.sh/connector-type: observability
platform.vcluster.com/fleet-observability-connector: "true"
type: Opaque
stringData:
metricsBackendOtlpEndpoint: https://metrics.example.com/otlp
metricsBackendPromQLEndpoint: https://metrics.example.com
grafanaUrl: https://grafana.example.com
Apply the Secret:
kubectl apply -f fleet-observability-connector.yaml
Writing the Secret directly skips the checks the Connector API runs, such as allowing only one fleet default. To manage the connector through the API instead, see Configure an observability connector and the Connector API reference.
Click Secrets & Credentials > Connectors and select the Observability tab.
Click .
In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default.
In the Metrics backend OTLP endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway.
In the Metrics backend PromQL endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy. Both endpoint fields start with the bundled Prometheus addresses in the
vcluster-platformnamespace. Keep them if you deployed the bundled Prometheus there.Choose the gateway's Service type. Choose LoadBalancer when clusters whose collectors run outside the Platform cluster deliver metrics or logs through this connector.
If you deployed the bundled Prometheus, select
metrics-backend-mtlsin Metrics backend client certificate secret (optional). The bundled backend rejects clients without a certificate. For other backends, configure TLS as needed. See Connector keys for the available settings.Grafana URL and Fleet observability dashboard UID start with the bundled Grafana values. Keep them if you'll deploy the bundled Grafana. After it's running, edit the connector and select
grafana-server-tlsin Grafana certificate secret (optional).Click .
Platform reconciles the following resources:
Deployment/gateway-fleet-observabilityService/gateway-fleet-observability- Grafana datasource ConfigMaps for fleet, control plane cluster, and cluster views when Grafana datasource reconciliation is enabled.
Connector keys​
| Key | Required | Description |
|---|---|---|
metricsBackendOtlpEndpoint | Yes | OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends /v1/metrics (for example, https://prom-prometheus-server.vcluster-platform/api/v1/otlp receives writes at /api/v1/otlp/v1/metrics). |
metricsBackendPromQLEndpoint | Yes | Prometheus HTTP API endpoint queried by the Query Proxy. |
metricsBackendInsecureSkipVerify | No | Set to "true" only for test backends with unverifiable TLS. |
metricsBackendServerName | No | TLS server name override for the Query Proxy's PromQL reads. It doesn't affect OTLP writes, so the OTLP endpoint's certificate must still match the endpoint hostname. |
observabilityGatewayCertSecretName | No | Custom Secret with tls.crt and tls.key for gateway serving TLS. Include ca.crt when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted. |
metricsBackendCertSecretName | No | Secret with ca.crt, tls.crt, and tls.key for mTLS to the backend. The gateway trusts the backend with ca.crt and presents the certificate and key as its client identity. ca.crt is optional when metricsBackendInsecureSkipVerify is "true". |
gatewayResources | No | YAML resources block for the Write Gateway container. |
queryProxyResources | No | YAML resources block for the Query Proxy container. |
serviceType | No | Gateway Service type. Valid values are ClusterIP, the default, and LoadBalancer. Must be LoadBalancer when a cluster delivers metrics or logs through this connector and its collector runs outside the Platform cluster. See Which clusters need a LoadBalancer gateway. |
serviceAnnotations | No | YAML map of annotations applied to a LoadBalancer gateway Service. |
grafanaUrl | Yes | Grafana that the Platform UI embeds on the Fleet Observability page and in each cluster's observability view. A Connector API create request that leaves it empty gets the Grafana that the Fleet Observability: Grafana Argo CD template deploys in the platform namespace. |
grafanaInsecureSkipVerify | No | Set to "true" only to test a Grafana endpoint with an unverifiable certificate. |
grafanaCertSecretName | No | Grafana serving certificate Secret. Platform uses its ca.crt to trust the Grafana upstream. |
fleetObservabilityDashboardUid | No | UID of the fleet dashboard embedded by the Fleet Observability page. |
destinations | No | YAML list of extra OTLP HTTP endpoints that receive metrics or logs. Required for log collection. The Platform UI connector form has no field for it. See Forward telemetry to additional destinations. |
When you point grafanaUrl at a Grafana you manage instead of the bundled Grafana, that
Grafana must:
- Serve from the
/grafanasubpath. Setserver.root_urlandserver.serve_from_sub_pathso Grafana works behind the Platform/grafana/reverse proxy. Seeserve_from_sub_path. - Allow embedding. Set
security.allow_embedding = trueso the Fleet Observability page and each cluster's Observability tab can embed the dashboards in an iframe. Seeallow_embedding.
Verify the gateway​
Check that the gateway pod is running:
CONNECTOR_ID=fleet-observability
kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID"
Check the service ports:
CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID"
Expected service ports:
| Port | Name | Purpose |
|---|---|---|
4317 | otlp | OTLP gRPC writes. |
4318 | otlp-http | OTLP HTTP writes. |
8081 | query | Prometheus HTTP API through the Query Proxy. |
Next, turn on the observability integration for each cluster and deploy the collectors, as described in Configure edge collectors. Create keys in Configure metrics access only for control plane clusters, collectors you configure manually, and readers.