Repository navigation
✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/benchmarks - #17716
Conversation
|
/lgtm |
|
/approve |
|
@clubanderson: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…benchmarks Every hive now contributes anonymized per-PR benchmark records (salted hive hash; agent/harness/model/sandbox runtime/task kind/outcome/review quality) to https://hivecommons.dev/benchmarks/, built from the review evidence bundles it already seals. Publication is on by default; benchmarks.publish: false opts out and benchmarks.repo_consent (default on) names only GitHub-public repos. Cadence is hub-driven and hourly: spoke -> POST /api/benchmarks/ingest on the hub (heartbeat bearer, hash recomputed server-side) -> hub commits data/fleet/<hash>/<YYYY-MM>.json to hivecommons/benchmarks via its App identity -> Pages rebuild. - pkg/benchmarks: record types, HiveHash, Convert, schema golden test - spoke: GET /api/benchmarks/export?since=, benchmarks block in /api/status, hourly pusher, Settings -> Features -> Benchmarks toggles + hash - hub: ingest, month-file store, hourly publisher, GET /api/hub/benchmarks - evidence: Author.runtime / Author.task_kind stamped at bundle write - docs/benchmarks.md, operator-reference, api-reference, changelog fragment Refs #17694 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
7a6bb8a to
c80ea41
Compare
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. Evidence bundle `#17716cd3b768` sha256:9f3b051b2850b446d0b17230dc7694da7dd0826bbf8a938d4135c5ea00f21f22 — download from the dashboard or `hivectl review evidence #17716`. |
Move the converter to pkg/evidence/benchmarks (it is a view over evidence bundles) so pkg/dashboard gains no new top-level dependency, and give the read-only hive-hash field its ? help mark (#7256). Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
/lgmt |
|
/lgtm |
|
@clubanderson: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Mutation testing⏱️ The 900s mutation budget ran out, so 3 package(s) were not fully mutated and the results below do not cover them:
❌ gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):
Verdict: |
…oke pushers Also make the since-filter exclusive at the watermark so the hourly push does not resend the newest PR every run. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
92610ae to
d7b9396
Compare
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at cd3b768. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Implements #17694 (RFC) — the hive side of the public leaderboard at https://hivecommons.dev/benchmarks/ (data repo: https://github.com/hivecommons/benchmarks, already live with a fleet-wide backfill of ~15k records from 13 hives).
What
pkg/benchmarks—Recordtypes matchingschema/result.schema.json,HiveHash()(salted sha256 of hive id, first 16 hex),Convert()folding evidence bundles → one author record + one reviewer record per PR. Golden test walks the vendored schema (required/enum/additionalProperties).GET /api/benchmarks/export?since=(owner auth),benchmarks{}block in/api/status(hash, toggles, last push, watermark), hourly pusher to the hub, Settings → Features → Benchmarks (two toggles + read-only hash).POST /api/benchmarks/ingest(per-hive heartbeat bearer like NPS; hash recomputed server-side, foreign-hash records rejected), month-file store under/data/benchmarks/<hash>/, hourly publisher committingdata/fleet/<hash>/<YYYY-MM>.jsontohivecommons/benchmarksvia the hub App identity (Contents API, direct to main),GET /api/hub/benchmarksstatus.author.runtime(effective crux sandbox) andauthor.task_kind(lane hint) — the two gaps the backfill surfaced.benchmarks.publish(default true, opt-out),benchmarks.repo_consent(default true: name repos only when GitHub reports them public; private never named).src/docs/benchmarks.md(privacy + cadence), operator-reference rows, api-reference rows (spoke + hub), documentation-map, changelog fragment at repo root.Privacy
Default-on anonymized: hashed hive id, model/harness/runtime/task kind/outcome/ACMM/review quality. Never: hive id, PR numbers, SHAs, prompts, diffs, logins, costs. Repo/org only for public repos (
gh api repos/{o}/{r}→private:false, cached 24h, unknown ⇒ private).Cadence (as implemented)
spoke → hub push hourly (3-min initial delay, ≤2000 records/batch, persisted watermark) → hub → GitHub hourly (10-min initial delay, ≤60 files/run, skip identical) → Pages rebuild on push. Site lag ≤ ~2h. Push chosen over hub-pull because the hub holds no spoke credential and most spoke dashboards are cluster-private.
Operator follow-up
contents: writeonhivecommons/benchmarks(or setHIVE_BENCHMARKS_PUBLISH_DISABLED=1to store-only).hivecommons/benchmarksscripts/backfill-from-cluster.sh+ its nightly workflow keep the site current.Tests
go test ./pkg/benchmarks ./pkg/config ./pkg/evidence ./pkg/github ./pkg/huband./pkg/dashboard -run 'TestBenchmarks|TestSettingsDialog|TestOpenAPI|TestStatus'pass; static-checkclean; api-reference citations re-pinned; docs links resolve.agent_runtime_golden.jsongains"benchmarks": {}(new config block).