Configure edge collectors
Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the
workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer
token and scope headers that match a metrics-writer access key.
Deploy collectors once the gateway is running, as described in Install the observability gateway.
Deploy collectors with Argo CD​
Collectors run on every cluster and control plane cluster you want metrics from.
If you use the Argo CD integration, deploy one
cluster-collector ArgoCDApplication to each cluster instead of configuring the
OpenTelemetry Collector by hand.
The out-of-the-box collector path relies on the Argo CD integration, which vCluster Standalone clusters can't use. There is no bundled collector for them. Configure collectors manually instead.
The collectors read the gateway endpoint, their metrics-writer token, and the gateway CA
from a metrics-writer Secret in the collector namespace. The Secret carries three keys:
| Key | Content |
|---|---|
endpoint | OTLP HTTP URL of the Write Gateway, including the https:// scheme. |
token | The metrics-writer access key, with no Bearer prefix. |
ca.crt | PEM CA that issued the gateway serving certificate. Must be present, but can be empty to verify against the system CA trust store. |
The collectors reload the token after the kubelet refreshes the mounted Secret, so token rotation doesn't require a restart. A changed endpoint or CA in that Secret requires a manual restart. See Rotation for timing and restart guidance.
How the Secret gets there depends on the destination:
-
Cluster: turn on the observability integration in the cluster's
vcluster.yaml. The integration requires vCluster v0.38 or later. Platform issues themetrics-writerkey, and the cluster writes the Secret into itsobservabilitynamespace and keeps it up to date:vcluster.yamlintegrations:observability:enabled: trueconnector: fleet-observability # the connector Secret's nameIf the cluster's project belongs to a tenant, grant the connector to that tenant.
To deliver the Secret elsewhere, set
integrations.observability.gatewaySecret. Set bothnamespaceandname, because the cluster rejects agatewaySecretthat sets only one:vcluster.yamlintegrations:observability:enabled: trueconnector: fleet-observabilitygatewaySecret:namespace: monitoring # namespace the collectors run inname: fleet-metrics-writerThen set the template's
destinationNamespaceandgatewaySecretNameparameters to match:fleet-edge-collectors.yamlspec:parameters:destinationNamespace: monitoringgatewaySecretName: fleet-metrics-writer -
Control plane cluster: nothing delivers the Secret, so create it yourself, as described in Create the Secret for a control plane cluster.
The walkthrough below deploys to a cluster. For a control plane cluster, swap
destination.cluster in for destination.virtualCluster, as described at the end of this
section.
- Platform UI
- YAML
Platform creates a namespace named p-<project-name> for each project, so the default project's namespace is p-default.
Create the ArgoCDApplication object in your project namespace:
apiVersion: management.loft.sh/v1
kind: ArgoCDApplication
metadata:
name: fleet-edge-collectors
namespace: p-my-project # replace with your project namespace
spec:
displayName: "Fleet edge collectors"
destination:
virtualCluster:
name: my-vcluster # replace with your cluster name
target: vCluster # deploy into the cluster
templateRef:
name: cluster-collector
parameters:
destinationNamespace: observability
project: default
Select your project from the projects dropdown at the top of the left navigation bar.
Navigate to Clusters and open the cluster you want to collect from.
Select the Argo CD Apps tab and click .
In the Basics section, enter a Display name such as
Fleet edge collectors.Select Deploy to cluster to deploy into the cluster itself.
Under Source, select From template.
From the Argo CD Application Template dropdown, select the Fleet Observability: Cluster Collector template.
Fill the Template Parameters. Set Destination Namespace to the namespace of the
metrics-writerSecret, which isobservabilityby default.Click .
| Parameter | Default | Description |
|---|---|---|
destinationNamespace | observability | Namespace for the collectors. It must contain the metrics-writer Secret. |
project | default | Argo CD project that owns the application. |
gatewaySecretName | metrics-writer | Name of the Secret the collectors read the endpoint, token, and CA from. |
otlpCaData | (empty) | PEM CA data used to verify the gateway serving certificate in place of the Secret's ca.crt. |
collectionInterval | 30s | Scrape interval for the jobs the collectors scrape themselves. |
collectLogs | false | Collects pod logs from every node the otel-daemon collector runs on and sends them to the gateway. Use it only on clusters with private nodes. See Collect pod logs. |
otelDaemonResources | (empty) | Resource requests and limits for the otel-daemon collector container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits. |
otelDeployResources | (empty) | Resource requests and limits for the otel-deploy collector container, in the same format as otelDaemonResources. When empty, the chart default applies, which sets no requests or limits. |
bearerToken | (empty) | Deprecated. Leave empty so the collectors read the metrics-writer Secret. See Upgrade collectors that use the token parameters. |
Both resource parameters are multiline parameters that take the contents of a Kubernetes
resources field.
spec:
parameters:
otelDaemonResources: |
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 256Mi
otelDeployResources: |
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 512Mi
The collectors stamp the request scope from the destination, so target a cluster
for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set
destination.cluster instead of destination.virtualCluster to target a control plane
cluster. Don't set vcluster_platform_* labels in collector configuration; the Write
Gateway owns that namespace. See Protect platform labels for
the label rules, and Gateway endpoints for the scope headers
the template sets on your behalf.
On a cluster with the GPU stack deployed, this template also scrapes GPU metrics without extra configuration. See The cluster-collector template scrapes GPU metrics automatically.
On a cluster, this template also scrapes the cluster control plane's backing store. See Cluster backing store metrics below.
Create the Secret for a control plane cluster​
Nothing delivers the metrics-writer Secret to a control plane cluster, so create it
before you deploy the collectors. Create the writer key first, as described in
Create a writer key for a control plane cluster.
On the cluster where Platform runs, export the CA of the default Platform-managed gateway certificate:
CONNECTOR_ID=fleet-observability
kubectl get secret -n vcluster-platform "gateway-${CONNECTOR_ID}-tls" \
-o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt
For a gateway certificate issued by a public CA, create an empty ca.crt file instead.
On the control plane cluster, create the Secret. Use the in-cluster Service endpoint for
the cluster where Platform runs, or the gateway LoadBalancer address
(https://<loadbalancer-address>:4318) for any other cluster. See
Gateway endpoints.
kubectl create namespace observability
kubectl create secret generic metrics-writer -n observability \
--from-literal=endpoint="https://gateway-${CONNECTOR_ID}.vcluster-platform.svc.cluster.local:4318" \
--from-literal=token=REPLACE_WITH_METRICS_WRITER_KEY \
--from-file=ca.crt=ca.crt
Upgrade collectors that use the token parameters​
Earlier versions of the cluster-collector template took the gateway endpoint and token as
the otlpEndpoint and bearerToken parameters, plus otlpInsecureSkipVerify to skip
certificate verification. These parameters are deprecated and will be removed in a later
release. Until then, the template still reads them from existing applications:
-
Both
otlpEndpointandbearerTokenset: the collectors keep working. They read theotel-daemon-otlp-authandotel-deploy-otlp-authSecrets the template renders from both parameters, not themetrics-writerSecret. -
Only one of them set, such as
otlpEndpointwith a hand-madeotel-otlp-authSecret: Platform can't render the application. The collectors keep running with the configuration they had before the upgrade, but the application applies no further changes. ItsSyncedcondition reportsBuildApplicationFailedwith this message:cluster-collector: the deprecated bearerToken and otlpEndpoint must be set together, or both cleared to read the Metrics-Writer SecretThe collectors keep running only while that application exists. If you delete and re-create it, for example by reinstalling or rebuilding the cluster, Argo CD removes the collectors and metrics stop until you migrate.
To move an application to the delivered metrics-writer Secret:
-
Upgrade the cluster to vCluster v0.38 or later, then turn on the observability integration in its
vcluster.yaml, as described in Deploy collectors with Argo CD. Earlier vCluster versions don't haveintegrations.observability. The collectors must reach the endpoint Platform hands out, which needs either a DNS mapping or a gateway LoadBalancer depending on where the cluster runs. See step 3 of the quickstart. If the application'sdestinationNamespaceisn'tobservability, also setintegrations.observability.gatewaySecretand the template'sgatewaySecretNameto match. For a control plane cluster, create the Secret yourself instead. -
In the cluster, confirm the cluster delivered the Secret. A delivered Secret shows
vclusterin theMANAGED-BYcolumn:kubectl get secret metrics-writer -n observability -L observability.platform.vcluster.com/managed-byIf the column is empty, you created the Secret yourself, and the cluster doesn't overwrite it. Delete it so the cluster can deliver its own.
-
In one edit, remove
otlpEndpoint,otlpInsecureSkipVerify, andbearerTokenfrom theArgoCDApplication'sspec.parameters. Removing only one ofotlpEndpointandbearerTokenfails rendering. For example:fleet-edge-collectors.yamlparameters:destinationNamespace: observability- otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318- otlpInsecureSkipVerify: "true"- bearerToken: REPLACE_WITH_METRICS_WRITER_KEYproject: default -
When Argo CD syncs the edit, the collectors roll on their own, because their pod spec now mounts the
metrics-writerSecret. If the collector pods didn't restart, restart them so they read the new Secret:kubectl rollout restart daemonset,deployment -n observability -l app.kubernetes.io/name=opentelemetry-collector -
Argo CD prunes the Secrets the template rendered from
bearerToken, but not a hand-madeotel-otlp-authSecret. Delete it if it exists:kubectl delete secret otel-otlp-auth -n observability
Cluster backing store metrics​
The otel-deploy collector scrapes the cluster control plane's backing store with no extra
configuration. vCluster serves these metrics on the cluster control plane's HTTPS proxy. That's
the same target the collector already scrapes for the cluster's API server metrics, so the
collector reaches them with its own ServiceAccount token and the cluster's CA.
| Scrape job | Path | Backing store |
|---|---|---|
vcluster-etcd | /metrics/etcd | Embedded etcd. Requires vCluster v0.34.0 or later. |
vcluster-kine | /metrics/kine | Embedded or external database. Requires vCluster v0.35.0 or later. |
The cluster's own RBAC authorizes these scrapes. The template grants the collector's
ServiceAccount get on both paths, so no action is needed unless you replace the bundled
ClusterRole.
Deployed etcd and external etcd aren't covered. vCluster serves no metrics route for either, so no scrape job collects them.
To query the collected series, see Query fleet metrics.
One backing store job is always down​
A cluster has one backing store, so only one of the two jobs ever returns data. The
path for the other backing store doesn't exist on that cluster, so its job reports
up == 0 and logs a scrape failure on every scrape interval, for the lifetime of the
cluster. This is expected, and a failing job doesn't affect the others on the collector.
Prometheus generates up itself rather than reading it from the target, so
metric_relabel_configs can't drop it at the collector. Exclude both backing store jobs
from any general up == 0 alert:
up{job!~"vcluster-etcd|vcluster-kine"} == 0
Then alert on the job that matches the cluster's backing store. For a cluster on embedded etcd:
up{job="vcluster-etcd"} == 0
Verify writes​
Check the Write Gateway metrics endpoint:
CONNECTOR_ID=fleet-observability
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889
curl http://127.0.0.1:8889/metrics
Then query through the Query Proxy as described in Query fleet metrics.