Skip to content

🐛 benchmarks publish wedges on hub 413: batch capped by count not bytes, identical batch retried forever #18880

Closed

Description

@clubanderson

Refs hivecommons/benchmarks#1, #17694, #17716.

Symptom

On the operator's hosted spoke, /api/status benchmarks shows the publisher's last error as hub answered 413 Request Entity Too Large (the spoke only records resp.Status, not the response body).

Evidence (v5 @ 8fe262e)

  • Spoke src/pkg/dashboard/api_benchmarks.go pushBenchmarks: batches are capped by record COUNT only (benchmarksPushMaxRecords = 2000), not bytes. On any non-2xx it calls benchmarksFail and returns without advancing the watermark, so the next hourly tick rebuilds the same batch and fails the same way. A deterministic 413 therefore wedges that hive's publishing permanently, with no backoff or shrink.
  • Hub src/pkg/hub/benchmarks.go: benchmarksIngestMaxBodyBytes = 8 << 20 returns 413 with body payload too large; the comment assumes 2000 records is about 2 MB.
  • Measured record size from published data (hivecommons/benchmarks data/fleet/d782f73067f8d8d4/2026-11.json): about 885 bytes per record, so a 2000-record batch is about 1.8 MB. That is well under the hub's 8 MB cap, so the hub handler's own limit is probably NOT what fires for typical batches; more likely a body-size limit in front of the hub (an ingress or proxy, for example nginx's 1 MB default client_max_body_size) or unusually large records (many verdicts / human_actions) on that hive. I could not distinguish these from outside the cluster; the spoke discards the body that would say which layer answered.
  • Other hives publish fine: 12 distinct hive hashes have hub-App commits on benchmarks main since 2026-11-16, so this is specific to a hive or to batch size.

Suggested direction (not done here)

  1. Build batches by serialized bytes (for example target under 512 KB, below common 1 MB proxy defaults) as well as count, and on a 413 halve the batch and retry rather than retrying the identical payload.
  2. Log the response body (truncated) in benchmarksFail so hub answered 413 says whether the handler or a proxy produced it.
  3. Check the hub ingress annotations (proxy-body-size) for the /api/benchmarks/ingest path.
  4. Add a test: oversized batch yields progress (smaller batch) instead of a stuck watermark.

Activity

  1. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @Karibhou released (covering PR merged or closed).

    Claim history
    • 2026-11-18 04:07 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-18 04:27 UTC — released by Karibhou (contributor; covering PR merged or closed)
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 18, 2026
  3. self-assigned this
    on Nov 18, 2026
  4. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    Claimed by the operator's dispatcher; fix in progress (byte-based batching and halve-on-413 on the spoke publisher).

  5. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    Delivered by #18887 (merged to v5): byte-budgeted batching (384 KB, 2000 records), halve-on-413 with a single-record skip, per-batch watermark advance, and the hub response body (truncated, token redacted) in benchmarks.last_error. The fix reaches a spoke with its next image update. Separately, the hub ingress proxy-body-size for /api/benchmarks/ingest should be checked. #18886 was closed in favour of this.

  6. added a commit that references this issue on Nov 18, 2026
    41029a4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions