fix(collector): Don't split metric exports across batches

[Amund211]

Problem

The collector sidecar on flashlight-cr was logging Exporting failed. Dropping data. from the googlemanagedprometheus exporter at a dead-steady ~10/min (1,748 of 1,760 export failures in a 3h window). GMP rejected the two exporter-generated info gauges — target_info and otel_scope_info — with:

code = InvalidArgument desc = One or more TimeSeries could not be written: ...
One or more points were written more frequently than the maximum sampling
period configured for the metric.
{Metric: prometheus.googleapis.com/otel_scope_info/gauge,
 Timestamps: {Youngest Existing: '...:49:51.913', New: '...:49:51.915'}}

i.e. the same info series written ~2ms apart.

Root cause

The metrics pipeline's batch processor used send_batch_max_size: 200 (copied from GCP's Cloud Run OTel sidecar tutorial). That splits each ~60s export into ceil(series / 200) separate batches. The googlemanagedprometheus exporter emits target_info / otel_scope_info once per PushMetrics call, so every extra batch re-writes those series with near-identical timestamps → GMP rejects the duplicates.

The cap is redundant: the exporter already chunks CreateTimeSeries to GCM's hard limit of 200 timeseries/request internally (sendBatchSize = 200 in opentelemetry-operations-go), and target_info is written once per PushMetrics call regardless of series count. So send_batch_max_size: 200 bought nothing except this bug.

⚠️ Likely trigger — verify before merging

We were almost certainly seeing this because we were pushing a lot of high-cardinality labels (user_agent, ip_hash) that blew ports_request_count_total past the OTel SDK's 2000-series cap. At ~2000 series each export split into ~10 batches → the observed steady ~10 rejections/min.

#306 removed those labels. With series back to dozens, each export now fits in a single 200-point batch and target_info is written once per cycle — so the error should stop on its own without this change.

This PR is therefore defense-in-depth, not the primary fix: it makes the failure structurally impossible regardless of future cardinality. We should confirm the error stays gone now that #306 is deployed, and only merge this once we've seen it calm down — that way we're merging a durability guardrail, not papering over an unverified root cause.

(Early signal: revision flashlight-cr-00275-2g2 with #306 served 26,496 requests in 90 min with zero export failures of any kind; the last failure came from the prior revision seconds before cutover. Looks resolved — worth watching for longer before merging.)

Change

Give the metrics pipeline its own batch/metrics processor without send_batch_max_size, so a single export is never split into multiple PushMetrics calls. The traces pipeline keeps the original batch processor unchanged.

   batch:
-    # batch metrics before sending to reduce API usage
+    # Traces pipeline: cap request size. Metrics use batch/metrics below.
     send_batch_max_size: 200
     send_batch_size: 200
     timeout: 5s
+
+  batch/metrics:            # no send_batch_max_size -> export never split
+    send_batch_size: 200
+    timeout: 5s

Debugging methodology

• Cadence — gcloud logging read '... "maximum sampling period"' showed a dead-steady 10/min, not bursty and not correlated with cold starts, with no retry / sending_queue / Permanent error log lines → not a retry-amplification loop.
• Arithmetic — 2000 live series / send_batch_size 200 = 10 batches per export, at 1 export/min = the observed 10 errors/min. Exact match.
• Post-fix: Drop high-cardinality attributes from request metrics #306 check — new revision flashlight-cr-00275-2g2 served 26,496 requests in 90 min with zero export failures; last failure predates cutover.
• Source read — opentelemetry-operations-go collector/metrics.go PushMetrics: target_info is appended once per resource via the ExtraMetrics hook, then the flat timeseries list is chunked into sendBatchSize = 200 slices → target_info lands in exactly one CreateTimeSeries request per PushMetrics call. Splitting upstream is the only way to duplicate it.

Verification

• collector/config.yaml parses as valid YAML; pipeline wiring confirmed (metrics → batch/metrics, traces → batch).
• Could not run otelcol validate locally — the otelcol-google:0.154.0 image registry is outside the sandbox network allowlist. The change is standard OTel config structure.

Sources

• opentelemetry-operations-go collector/metrics.go — PushMetrics, sendBatchSize = 200, ExtraMetrics / target_info
• googlemanagedprometheusexporter README — enable_target_info / enable_scope_info
• collector-contrib issue #31507 — same GMP "maximum sampling period" failure mode
• batch processor README — send_batch_max_size splitting semantics
• GCP: Deploy Google-Built OpenTelemetry Collector on Cloud Run — the source of the original send_batch_max_size: 200 config

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adjusts the OpenTelemetry Collector sidecar configuration to prevent Google Managed Prometheus export rejections caused by the exporter re-emitting info series (target_info, otel_scope_info) when upstream batching splits a single metrics export into multiple PushMetrics calls.

Changes:

• Keeps the existing batch processor (with send_batch_max_size) for the traces pipeline.
• Introduces a dedicated batch/metrics processor without send_batch_max_size for the metrics pipeline to avoid upstream split-batching.
• Rewires the metrics pipeline to use batch/metrics while leaving the rest of the pipeline unchanged.