Skip to main content
Version: main 🚧

Configure edge collectors

Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer token and scope headers that match a metrics-writer access key.

Deploy collectors once the gateway is running, as described in Install the observability gateway.

Deploy collectors with Argo CD​

Collectors run on every cluster and control plane cluster you want metrics from. If you use the Argo CD integration, deploy one cluster-collector ArgoCDApplication to each cluster instead of configuring the OpenTelemetry Collector by hand.

vCluster Standalone clusters

The out-of-the-box collector path relies on the Argo CD integration, which vCluster Standalone clusters can't use. There is no bundled collector for them. Configure collectors manually instead.

The collectors read the gateway endpoint, their metrics-writer token, and the gateway CA from a metrics-writer Secret in the collector namespace. The Secret carries three keys:

KeyContent
endpointOTLP HTTP URL of the Write Gateway, including the https:// scheme.
tokenThe metrics-writer access key, with no Bearer prefix.
ca.crtPEM CA that issued the gateway serving certificate. Must be present, but can be empty to verify against the system CA trust store.

The collectors reload the token after the kubelet refreshes the mounted Secret, so token rotation doesn't require a restart. A changed endpoint or CA in that Secret requires a manual restart. See Rotation for timing and restart guidance.

How the Secret gets there depends on the destination:

  • Cluster: turn on the observability integration in the cluster's vcluster.yaml. The integration requires vCluster v0.38 or later. Platform issues the metrics-writer key, and the cluster writes the Secret into its observability namespace and keeps it up to date:

    vcluster.yaml
    integrations:
    observability:
    enabled: true
    connector: fleet-observability # the connector Secret's name

    If the cluster's project belongs to a tenant, grant the connector to that tenant.

    To deliver the Secret elsewhere, set integrations.observability.gatewaySecret. Set both namespace and name, because the cluster rejects a gatewaySecret that sets only one:

    vcluster.yaml
    integrations:
    observability:
    enabled: true
    connector: fleet-observability
    gatewaySecret:
    namespace: monitoring # namespace the collectors run in
    name: fleet-metrics-writer

    Then set the template's destinationNamespace and gatewaySecretName parameters to match:

    fleet-edge-collectors.yaml
    spec:
    parameters:
    destinationNamespace: monitoring
    gatewaySecretName: fleet-metrics-writer
  • Control plane cluster: nothing delivers the Secret, so create it yourself, as described in Create the Secret for a control plane cluster.

The walkthrough below deploys to a cluster. For a control plane cluster, swap destination.cluster in for destination.virtualCluster, as described at the end of this section.

  1. Select your project from the projects dropdown at the top of the left navigation bar.

  2. Navigate to Clusters and open the cluster you want to collect from.

  3. Select the Argo CD Apps tab and click .

  4. In the Basics section, enter a Display name such as Fleet edge collectors.

  5. Select Deploy to cluster to deploy into the cluster itself.

  6. Under Source, select From template.

  7. From the Argo CD Application Template dropdown, select the Fleet Observability: Cluster Collector template.

  8. Fill the Template Parameters. Set Destination Namespace to the namespace of the metrics-writer Secret, which is observability by default.

  9. Click .

ParameterDefaultDescription
destinationNamespaceobservabilityNamespace for the collectors. It must contain the metrics-writer Secret.
projectdefaultArgo CD project that owns the application.
gatewaySecretNamemetrics-writerName of the Secret the collectors read the endpoint, token, and CA from.
otlpCaData(empty)PEM CA data used to verify the gateway serving certificate in place of the Secret's ca.crt.
collectionInterval30sScrape interval for the jobs the collectors scrape themselves.
collectLogsfalseCollects pod logs from every node the otel-daemon collector runs on and sends them to the gateway. Use it only on clusters with private nodes. See Collect pod logs.
otelDaemonResources(empty)Resource requests and limits for the otel-daemon collector container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits.
otelDeployResources(empty)Resource requests and limits for the otel-deploy collector container, in the same format as otelDaemonResources. When empty, the chart default applies, which sets no requests or limits.
bearerToken(empty)Deprecated. Leave empty so the collectors read the metrics-writer Secret. See Upgrade collectors that use the token parameters.

Both resource parameters are multiline parameters that take the contents of a Kubernetes resources field.

fleet-edge-collectors.yaml
spec:
parameters:
otelDaemonResources: |
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 256Mi
otelDeployResources: |
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 512Mi

The collectors stamp the request scope from the destination, so target a cluster for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set destination.cluster instead of destination.virtualCluster to target a control plane cluster. Don't set vcluster_platform_* labels in collector configuration; the Write Gateway owns that namespace. See Protect platform labels for the label rules, and Gateway endpoints for the scope headers the template sets on your behalf.

On a cluster with the GPU stack deployed, this template also scrapes GPU metrics without extra configuration. See The cluster-collector template scrapes GPU metrics automatically.

On a cluster, this template also scrapes the cluster control plane's backing store. See Cluster backing store metrics below.

Create the Secret for a control plane cluster​

Nothing delivers the metrics-writer Secret to a control plane cluster, so create it before you deploy the collectors. Create the writer key first, as described in Create a writer key for a control plane cluster.

On the cluster where Platform runs, export the CA of the default Platform-managed gateway certificate:

CONNECTOR_ID=fleet-observability
kubectl get secret -n vcluster-platform "gateway-${CONNECTOR_ID}-tls" \
-o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

For a gateway certificate issued by a public CA, create an empty ca.crt file instead.

On the control plane cluster, create the Secret. Use the in-cluster Service endpoint for the cluster where Platform runs, or the gateway LoadBalancer address (https://<loadbalancer-address>:4318) for any other cluster. See Gateway endpoints.

kubectl create namespace observability
kubectl create secret generic metrics-writer -n observability \
--from-literal=endpoint="https://gateway-${CONNECTOR_ID}.vcluster-platform.svc.cluster.local:4318" \
--from-literal=token=REPLACE_WITH_METRICS_WRITER_KEY \
--from-file=ca.crt=ca.crt

Upgrade collectors that use the token parameters​

Earlier versions of the cluster-collector template took the gateway endpoint and token as the otlpEndpoint and bearerToken parameters, plus otlpInsecureSkipVerify to skip certificate verification. These parameters are deprecated and will be removed in a later release. Until then, the template still reads them from existing applications:

  • Both otlpEndpoint and bearerToken set: the collectors keep working. They read the otel-daemon-otlp-auth and otel-deploy-otlp-auth Secrets the template renders from both parameters, not the metrics-writer Secret.

  • Only one of them set, such as otlpEndpoint with a hand-made otel-otlp-auth Secret: Platform can't render the application. The collectors keep running with the configuration they had before the upgrade, but the application applies no further changes. Its Synced condition reports BuildApplicationFailed with this message:

    cluster-collector: the deprecated bearerToken and otlpEndpoint must be set together, or both cleared to read the Metrics-Writer Secret

    The collectors keep running only while that application exists. If you delete and re-create it, for example by reinstalling or rebuilding the cluster, Argo CD removes the collectors and metrics stop until you migrate.

To move an application to the delivered metrics-writer Secret:

  1. Upgrade the cluster to vCluster v0.38 or later, then turn on the observability integration in its vcluster.yaml, as described in Deploy collectors with Argo CD. Earlier vCluster versions don't have integrations.observability. The collectors must reach the endpoint Platform hands out, which needs either a DNS mapping or a gateway LoadBalancer depending on where the cluster runs. See step 3 of the quickstart. If the application's destinationNamespace isn't observability, also set integrations.observability.gatewaySecret and the template's gatewaySecretName to match. For a control plane cluster, create the Secret yourself instead.

  2. In the cluster, confirm the cluster delivered the Secret. A delivered Secret shows vcluster in the MANAGED-BY column:

    kubectl get secret metrics-writer -n observability -L observability.platform.vcluster.com/managed-by

    If the column is empty, you created the Secret yourself, and the cluster doesn't overwrite it. Delete it so the cluster can deliver its own.

  3. In one edit, remove otlpEndpoint, otlpInsecureSkipVerify, and bearerToken from the ArgoCDApplication's spec.parameters. Removing only one of otlpEndpoint and bearerToken fails rendering. For example:

    fleet-edge-collectors.yaml
    parameters:
    destinationNamespace: observability
    - otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318
    - otlpInsecureSkipVerify: "true"
    - bearerToken: REPLACE_WITH_METRICS_WRITER_KEY
    project: default
  4. When Argo CD syncs the edit, the collectors roll on their own, because their pod spec now mounts the metrics-writer Secret. If the collector pods didn't restart, restart them so they read the new Secret:

    kubectl rollout restart daemonset,deployment -n observability -l app.kubernetes.io/name=opentelemetry-collector
  5. Argo CD prunes the Secrets the template rendered from bearerToken, but not a hand-made otel-otlp-auth Secret. Delete it if it exists:

    kubectl delete secret otel-otlp-auth -n observability

Cluster backing store metrics​

The otel-deploy collector scrapes the cluster control plane's backing store with no extra configuration. vCluster serves these metrics on the cluster control plane's HTTPS proxy. That's the same target the collector already scrapes for the cluster's API server metrics, so the collector reaches them with its own ServiceAccount token and the cluster's CA.

Scrape jobPathBacking store
vcluster-etcd/metrics/etcdEmbedded etcd. Requires vCluster v0.34.0 or later.
vcluster-kine/metrics/kineEmbedded or external database. Requires vCluster v0.35.0 or later.

The cluster's own RBAC authorizes these scrapes. The template grants the collector's ServiceAccount get on both paths, so no action is needed unless you replace the bundled ClusterRole.

Deployed etcd and external etcd aren't covered. vCluster serves no metrics route for either, so no scrape job collects them.

To query the collected series, see Query fleet metrics.

One backing store job is always down​

A cluster has one backing store, so only one of the two jobs ever returns data. The path for the other backing store doesn't exist on that cluster, so its job reports up == 0 and logs a scrape failure on every scrape interval, for the lifetime of the cluster. This is expected, and a failing job doesn't affect the others on the collector.

Prometheus generates up itself rather than reading it from the target, so metric_relabel_configs can't drop it at the collector. Exclude both backing store jobs from any general up == 0 alert:

up{job!~"vcluster-etcd|vcluster-kine"} == 0

Then alert on the job that matches the cluster's backing store. For a cluster on embedded etcd:

up{job="vcluster-etcd"} == 0

Verify writes​

Check the Write Gateway metrics endpoint:

CONNECTOR_ID=fleet-observability
GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889
curl http://127.0.0.1:8889/metrics

Then query through the Query Proxy as described in Query fleet metrics.