You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
🐛 benchmarks publish wedges on hub 413: batch capped by count not bytes, identical batch retried forever #18880
On the operator's hosted spoke, /api/statusbenchmarks shows the publisher's last error as hub answered 413 Request Entity Too Large (the spoke only records resp.Status, not the response body).
Spoke src/pkg/dashboard/api_benchmarks.gopushBenchmarks: batches are capped by record COUNT only (benchmarksPushMaxRecords = 2000), not bytes. On any non-2xx it calls benchmarksFail and returns without advancing the watermark, so the next hourly tick rebuilds the same batch and fails the same way. A deterministic 413 therefore wedges that hive's publishing permanently, with no backoff or shrink.
Hub src/pkg/hub/benchmarks.go: benchmarksIngestMaxBodyBytes = 8 << 20 returns 413 with body payload too large; the comment assumes 2000 records is about 2 MB.
Measured record size from published data (hivecommons/benchmarks data/fleet/d782f73067f8d8d4/2026-11.json): about 885 bytes per record, so a 2000-record batch is about 1.8 MB. That is well under the hub's 8 MB cap, so the hub handler's own limit is probably NOT what fires for typical batches; more likely a body-size limit in front of the hub (an ingress or proxy, for example nginx's 1 MB default client_max_body_size) or unusually large records (many verdicts / human_actions) on that hive. I could not distinguish these from outside the cluster; the spoke discards the body that would say which layer answered.
Other hives publish fine: 12 distinct hive hashes have hub-App commits on benchmarks main since 2026-11-16, so this is specific to a hive or to batch size.
Suggested direction (not done here)
Build batches by serialized bytes (for example target under 512 KB, below common 1 MB proxy defaults) as well as count, and on a 413 halve the batch and retry rather than retrying the identical payload.
Log the response body (truncated) in benchmarksFail so hub answered 413 says whether the handler or a proxy produced it.
Check the hub ingress annotations (proxy-body-size) for the /api/benchmarks/ingest path.
Add a test: oversized batch yields progress (smaller batch) instead of a stuck watermark.
Delivered by #18887 (merged to v5): byte-budgeted batching (384 KB, 2000 records), halve-on-413 with a single-record skip, per-batch watermark advance, and the hub response body (truncated, token redacted) in benchmarks.last_error. The fix reaches a spoke with its next image update. Separately, the hub ingress proxy-body-size for /api/benchmarks/ingest should be checked. #18886 was closed in favour of this.
Activity
hivecommons-hive commentedon Nov 18, 2026
🔓 Claim by @Karibhou released (covering PR merged or closed).
Claim history
Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; covering PR merged or closed)clubanderson commentedon Nov 18, 2026
Claimed by the operator's dispatcher; fix in progress (byte-based batching and halve-on-413 on the spoke publisher).
clubanderson commentedon Nov 18, 2026
Delivered by #18887 (merged to v5): byte-budgeted batching (384 KB, 2000 records), halve-on-413 with a single-record skip, per-batch watermark advance, and the hub response body (truncated, token redacted) in benchmarks.last_error. The fix reaches a spoke with its next image update. Separately, the hub ingress proxy-body-size for /api/benchmarks/ingest should be checked. #18886 was closed in favour of this.
🐛 Batch benchmark pushes by bytes, halve on 413, surface hub response…