Skip to content

✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/benchmarks - #17716

Merged
hivecommons-hive[bot] merged 6 commits into
v5from
feat/benchmarks-export
Nov 11, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
v5from
feat/benchmarks-export

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Implements #17694 (RFC) — the hive side of the public leaderboard at https://hivecommons.dev/benchmarks/ (data repo: https://github.com/hivecommons/benchmarks, already live with a fleet-wide backfill of ~15k records from 13 hives).

What

  • pkg/benchmarks — Record types matching schema/result.schema.json, HiveHash() (salted sha256 of hive id, first 16 hex), Convert() folding evidence bundles → one author record + one reviewer record per PR. Golden test walks the vendored schema (required/enum/additionalProperties).
  • Spoke — GET /api/benchmarks/export?since= (owner auth), benchmarks{} block in /api/status (hash, toggles, last push, watermark), hourly pusher to the hub, Settings → Features → Benchmarks (two toggles + read-only hash).
  • Hub — POST /api/benchmarks/ingest (per-hive heartbeat bearer like NPS; hash recomputed server-side, foreign-hash records rejected), month-file store under /data/benchmarks/<hash>/, hourly publisher committing data/fleet/<hash>/<YYYY-MM>.json to hivecommons/benchmarks via the hub App identity (Contents API, direct to main), GET /api/hub/benchmarks status.
  • Evidence — new bundles stamp author.runtime (effective crux sandbox) and author.task_kind (lane hint) — the two gaps the backfill surfaced.
  • Config — benchmarks.publish (default true, opt-out), benchmarks.repo_consent (default true: name repos only when GitHub reports them public; private never named).
  • Docs: src/docs/benchmarks.md (privacy + cadence), operator-reference rows, api-reference rows (spoke + hub), documentation-map, changelog fragment at repo root.

Privacy

Default-on anonymized: hashed hive id, model/harness/runtime/task kind/outcome/ACMM/review quality. Never: hive id, PR numbers, SHAs, prompts, diffs, logins, costs. Repo/org only for public repos (gh api repos/{o}/{r} → private:false, cached 24h, unknown ⇒ private).

Cadence (as implemented)

spoke → hub push hourly (3-min initial delay, ≤2000 records/batch, persisted watermark) → hub → GitHub hourly (10-min initial delay, ≤60 files/run, skip identical) → Pages rebuild on push. Site lag ≤ ~2h. Push chosen over hub-pull because the hub holds no spoke credential and most spoke dashboards are cluster-private.

Operator follow-up

  • The hub's GitHub App installation needs contents: write on hivecommons/benchmarks (or set HIVE_BENCHMARKS_PUBLISH_DISABLED=1 to store-only).
  • Until rolled out, hivecommons/benchmarks scripts/backfill-from-cluster.sh + its nightly workflow keep the site current.

Tests

go test ./pkg/benchmarks ./pkg/config ./pkg/evidence ./pkg/github ./pkg/hub and ./pkg/dashboard -run 'TestBenchmarks|TestSettingsDialog|TestOpenAPI|TestStatus' pass; static -check clean; api-reference citations re-pinned; docs links resolve. agent_runtime_golden.json gains "benchmarks": {} (new config block).

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 11, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

/lgtm

@clubanderson

Copy link
Copy Markdown
Member Author

/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow kubestellar-prow Bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 11, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 11, 2026
…benchmarks

Every hive now contributes anonymized per-PR benchmark records (salted hive
hash; agent/harness/model/sandbox runtime/task kind/outcome/review quality)
to https://hivecommons.dev/benchmarks/, built from the review evidence
bundles it already seals. Publication is on by default; benchmarks.publish:
false opts out and benchmarks.repo_consent (default on) names only
GitHub-public repos.

Cadence is hub-driven and hourly: spoke -> POST /api/benchmarks/ingest on
the hub (heartbeat bearer, hash recomputed server-side) -> hub commits
data/fleet/<hash>/<YYYY-MM>.json to hivecommons/benchmarks via its App
identity -> Pages rebuild.

- pkg/benchmarks: record types, HiveHash, Convert, schema golden test
- spoke: GET /api/benchmarks/export?since=, benchmarks block in /api/status,
  hourly pusher, Settings -> Features -> Benchmarks toggles + hash
- hub: ingest, month-file store, hourly publisher, GET /api/hub/benchmarks
- evidence: Author.runtime / Author.task_kind stamped at bundle write
- docs/benchmarks.md, operator-reference, api-reference, changelog fragment

Refs #17694

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/benchmarks-export branch from 7a6bb8a to c80ea41 Compare November 11, 2026 17:34
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17714 ✨ Workflow models: signed bundles, hub marketplace catalogue with nightly refresh, spoke import/publish same files dashboard/openapi.json
src/docs/api-reference/hub-server.md
src/pkg/config/config.go
src/pkg/config/testdata/agent_runtime_golden.json
src/pkg/dashboard/static/index.html
src/pkg/hub/background_pollers.go
src/pkg/hub/server.go
#17706 ✨ Workflow models: visual editor with live compile, YAML, diff, simulate and export same files dashboard/openapi.json
src/pkg/dashboard/static/index.html

@hivecommons-hive

hivecommons-hive Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • coverage

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Evidence bundle `#17716cd3b768` sha256:9f3b051b2850b446d0b17230dc7694da7dd0826bbf8a938d4135c5ea00f21f22 — download from the dashboard or `hivectl review evidence #17716`.

Move the converter to pkg/evidence/benchmarks (it is a view over evidence
bundles) so pkg/dashboard gains no new top-level dependency, and give the
read-only hive-hash field its ? help mark (#7256).

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

/lgmt

@clubanderson

Copy link
Copy Markdown
Member Author

/lgtm

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing

⏱️ The 900s mutation budget ran out, so 3 package(s) were not fully mutated and the results below do not cover them:

  • pkg/evidence: skipped (not started)
  • pkg/github: skipped (not started)
  • pkg/hub: skipped (not started)

❌ gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):

  • pkg/dashboard: gremlins exited 137

Verdict: error (#17349).

…oke pushers

Also make the since-filter exclusive at the watermark so the hourly push
does not resend the newest PR every run.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/benchmarks-export branch from 92610ae to d7b9396 Compare November 11, 2026 18:07
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 11, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 8fe262e into v5 Nov 11, 2026
85 of 86 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/benchmarks-export branch November 11, 2026 18:44
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at cd3b768.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant