Configure gateway networking and TLS
The observability gateway serves OTLP writes and PromQL reads over TLS, and connects to the metrics backend over TLS or mTLS. After you install the gateway, use this page to bring your own serving certificate, expose the gateway to collectors outside the Platform cluster, or authenticate the gateway to the backend with mTLS.
Changing the contents of the gateway serving certificate Secret or the metrics backend certificate Secret restarts the gateway pod. Telemetry the gateway accepted but hasn't delivered yet is lost on restart, because its queues are held in memory only. Rotate certificates while the metrics backend and every destination are reachable. See When the gateway restarts.
Customize the gateway serving certificate​
The gateway always serves OTLP and query traffic over TLS. When you don't specify a
certificate Secret, Platform creates and manages a self-signed Secret named
gateway-<connector-id>-tls. This only works for connector IDs of 51 characters or fewer;
see Create the connector Secret for why longer IDs need a
custom certificate Secret instead.
To use your own certificate, create a Secret in the Platform namespace. Platform never
validates or modifies a custom certificate, so it must already cover every hostname a
client connects to. At minimum, include the gateway's in-cluster Service hostname,
gateway-<connector-id>.<platform-namespace>.svc.cluster.local, since Platform-generated
Grafana datasources and any in-cluster collector always connect through it. If you expose
the gateway with a LoadBalancer (see Expose the gateway with a
LoadBalancer below), also include its external
hostname or IP as a SAN.
apiVersion: v1
kind: Secret
metadata:
name: fleet-gateway-tls
namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Reference it from the connector:
stringData:
observabilityGatewayCertSecretName: fleet-gateway-tls
Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom
certificate cases. Distribute ca.crt to collectors that don't already trust the issuing
CA. Grafana datasources generated by Platform trust the ca.crt from the gateway serving
certificate Secret. A certificate signed by a public CA still needs to cover the hostnames
above; once it does, ca.crt can be omitted, since clients already trust the public CA
through their system trust stores.
Expose the gateway with a LoadBalancer​
The gateway Service defaults to ClusterIP. To make the gateway reachable from collectors
and query clients outside the Platform cluster, set serviceType to LoadBalancer. You can
optionally apply provider-specific Service annotations:
stringData:
serviceType: LoadBalancer
serviceAnnotations: |
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. A custom certificate needs that same SAN added by hand, since Platform never modifies it. See Customize the gateway serving certificate for how clients trust the resulting certificate either way.
Which clusters need a LoadBalancer gateway​
A ClusterIP gateway is only reachable from inside the Platform cluster. It can serve a
cluster whose control plane runs on the Platform cluster and that doesn't use private
nodes, such as a cluster with shared nodes, because that cluster's collector runs as a pod
on a Platform cluster node. Such a cluster still needs one extra step to resolve the
gateway; see
Use a ClusterIP gateway with clusters on the Platform cluster.
Every other cluster runs its collector where a ClusterIP gateway is unreachable:
- A cluster hosted on a connected control plane cluster runs its collector on that connected cluster's nodes.
- A cluster with private nodes runs its collector on its own nodes, even when its control plane runs on the Platform cluster.
- A vCluster Standalone cluster runs on its own machine.
Rather than hand out an address those clusters can't reach, Platform refuses them the
metrics-writer credential for a ClusterIP connector, so no credential Secret is
delivered. Platform also refuses a cluster whose status it can't read yet. Set
serviceType: LoadBalancer on the connector to serve these clusters.
Retrieve the load balancer address​
The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service:
CONNECTOR_ID=fleet-observability
kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \
-o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}'
The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address:
| Endpoint | Port | Purpose |
|---|---|---|
https://<loadbalancer-address>:4318 | 4318 | OTLP HTTP writes. Use this for collectors. |
https://<loadbalancer-address>:4317 | 4317 | OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use 4318 for collectors. |
https://<loadbalancer-address>:8081 | 8081 | Prometheus HTTP API through the Query Proxy. Use this for external query clients. |
Use this address wherever a collector or query client connects from outside the Platform cluster.
Use a ClusterIP gateway with clusters on the Platform cluster​
A ClusterIP gateway suits evaluation and single-cluster installs where every cluster
runs on the Platform cluster without private nodes. Those clusters' collectors can reach
the gateway Service over the network, but can't resolve its name by default. A collector
resolves names through its cluster's own CoreDNS, which only answers for the cluster's own
Services, so the delivered endpoint
gateway-<connector-id>.<platform-namespace>.svc.cluster.local doesn't resolve.
Replicate the gateway Service into each such cluster under the same namespace and name. The delivered hostname then resolves and still matches the gateway serving certificate:
networking:
replicateServices:
fromHost:
- from: vcluster-platform/gateway-fleet-observability
to: vcluster-platform/gateway-fleet-observability
vCluster creates the target namespace in the cluster if it's missing. See Replicate services for how replication works.
An install where the Platform cluster only hosts clusters configured this way needs no load balancer.
Use backend mTLS​
To authenticate the gateway to your backend with mTLS, create a Secret:
apiVersion: v1
kind: Secret
metadata:
name: metrics-backend-client-tls
namespace: vcluster-platform
type: kubernetes.io/tls
stringData:
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
tls.key: |
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
Reference it from the connector:
stringData:
metricsBackendCertSecretName: metrics-backend-client-tls
Configure backend mTLS across namespaces or clusters​
The bundled Prometheus template's default mTLS setup works because Prometheus and the
gateway both mount the same Secret object. By default they share the namespace Platform
itself is installed in (vcluster-platform, unless you renamed it), so
metricsBackendTLSSecret (the Prometheus side) and metricsBackendCertSecretName (the
connector side) both point at the same auto-generated metrics-backend-mtls Secret.
That single-Secret shortcut only works when Prometheus and the gateway share a namespace and a cluster. What to do instead depends on where Prometheus runs.
Different namespace, same cluster​
If you set destinationNamespace to a namespace other than the one Platform is installed
in, Prometheus stays on the local cluster, so its bootstrap certificate still covers the
right in-cluster hostname (prom-prometheus-server.<destinationNamespace>.svc.cluster.local).
A single Secret object just can't be mounted in two namespaces at once, so copy it instead
of sharing one object:
- Leave
metricsBackendTLSSecretempty so the Argo CD template bootstraps the shared certificate for you, namedmetrics-backend-mtlsin Prometheus'sdestinationNamespace. See Deploy the metrics backend with Argo CD. - Once Prometheus reports ready, read that generated Secret and create an identical copy
in the Platform namespace, then reference it from the connector's
metricsBackendCertSecretName.
Setting metricsBackendTLSSecret to a name of your own disables this bootstrap entirely,
so only use it if you're supplying your own certificate material for both Secrets instead
of copying the generated one.
A different cluster, or a backend other than the bundled Prometheus​
The bundled Prometheus workflow deploys Prometheus to the local cluster, and its bootstrap certificate only covers in-cluster Service hostnames. Copying that Secret doesn't work once the gateway reaches the backend through an external hostname, because the certificate has no SAN for it. Create the gateway-side Secret as described in Use backend mTLS above, but with two differences:
- The backend's server certificate must cover the hostname used in
metricsBackendOtlpEndpointandmetricsBackendPromQLEndpoint, not an in-cluster Service name. - Configure the backend to trust the CA that issues the gateway's client certificate, the
same way the bundled Prometheus config trusts it through
client_ca_file. Without this, the backend rejects the gateway's mTLS handshake even if the gateway trusts the backend.