From 3fc23ad849c6efa0c56f8cba182f7bbfef98c7d4 Mon Sep 17 00:00:00 2001 From: Amund Eggen Svandal Date: Sun, 12 Jul 2026 01:22:10 +0200 Subject: [PATCH] fix(collector): Don't split metric exports across batches The metrics pipeline's batch processor used send_batch_max_size: 200 (copied from GCP's Cloud Run OTel sidecar tutorial). That split each ~60s export into ceil(series / 200) separate batches. The googlemanagedprometheus exporter emits the target_info / otel_scope_info info series once per PushMetrics call, so every extra batch re-wrote those series with near-identical timestamps, and Google Managed Prometheus rejected the duplicates with: code = InvalidArgument desc = One or more TimeSeries could not be written: ... One or more points were written more frequently than the maximum sampling period configured for the metric. The cap is redundant: the exporter already chunks CreateTimeSeries to GCM's hard limit of 200 timeseries/request internally (sendBatchSize = 200 in opentelemetry-operations-go), and it writes target_info once per PushMetrics call regardless of series count. Moving metrics to a dedicated batch/metrics processor with no send_batch_max_size keeps each export in a single PushMetrics call, so the info series are written exactly once no matter how high cardinality goes. The traces pipeline keeps the original batch processor unchanged. Likely trigger: we were attaching high-cardinality labels (user_agent, ip_hash) that pushed ports_request_count_total past the OTel SDK's 2000 series cap, so each export split into ~10 batches -> a dead-steady ~10 rejections/min. PR #306 dropped those labels; with series back to dozens each export fits in one 200-point batch and the error should stop on its own. This change is defense-in-depth so the failure cannot silently return if cardinality later creeps back over 200. We should confirm the error stays gone after #306 has been deployed for a while before merging this (see methodology below). Debugging methodology: - Cadence: gcloud logging read '... "maximum sampling period"' showed a dead-steady 10/min, not bursty and not correlated with cold starts, with no retry / sending_queue log lines -> not a retry-amplification loop. - Arithmetic: 2000 live series / send_batch_size 200 = 10 batches per export at 1 export/min = the observed 10 errors/min. - Post-#306 check: revision flashlight-cr-00275-2g2 served 26,496 requests in 90 min with zero export failures of any kind; the last failure came from the prior revision seconds before cutover. - Source read: opentelemetry-operations-go collector/metrics.go PushMetrics adds target_info once per resource via the ExtraMetrics hook, then chunks the flat timeseries list into sendBatchSize=200 slices -> target_info lands in exactly one CreateTimeSeries request per PushMetrics call. Sources: - https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/exporter/collector/metrics.go - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/googlemanagedprometheusexporter/README.md - https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31507 - https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md - https://docs.cloud.google.com/stackdriver/docs/instrumentation/opentelemetry-collector-cloud-run Co-Authored-By: Claude Opus 4.8 (1M context) --- collector/config.yaml | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/collector/config.yaml b/collector/config.yaml index 893fa8c5..5703e797 100644 --- a/collector/config.yaml +++ b/collector/config.yaml @@ -8,11 +8,27 @@ receivers: processors: batch: - # batch metrics before sending to reduce API usage + # Traces pipeline: cap request size. Metrics use batch/metrics below. send_batch_max_size: 200 send_batch_size: 200 timeout: 5s + # Metrics pipeline: batch WITHOUT send_batch_max_size (0 = no upper limit). + # + # The googlemanagedprometheus exporter already chunks CreateTimeSeries into + # GCM's hard limit of 200 timeseries/request internally, and it emits the + # target_info / otel_scope_info series once per PushMetrics call. If the + # batch processor splits a single ~60s export into multiple batches (as + # send_batch_max_size: 200 does once total series > 200), the exporter runs + # once per batch and re-emits those info series with near-identical + # timestamps, which GMP rejects with "One or more points were written more + # frequently than the maximum sampling period configured for the metric". + # Leaving send_batch_max_size unset keeps each export in a single PushMetrics + # call, so the info series are written exactly once regardless of cardinality. + batch/metrics: + send_batch_size: 200 + timeout: 5s + memory_limiter: # drop metrics if memory usage gets too high check_interval: 1s @@ -53,5 +69,5 @@ service: exporters: [googlecloud] metrics: receivers: [otlp] - processors: [batch, memory_limiter, resource_detection, resource] + processors: [batch/metrics, memory_limiter, resource_detection, resource] exporters: [googlemanagedprometheus]