You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
✨ Hive exporter: GET /api/benchmarks/export + hub aggregation emitting schema records #1
Hive spokes should be able to emit records that validate against schema/result.schema.json with no manual work, and the hub should aggregate them across hosted hives for owners who opted in.
ci.status on the bundle with empty previous_bundle_id
outcome.review_verdict, reviewer_models
verdicts[]
outcome.human_interventions
len(human_actions)
outcome.retries
bundles chained by previous_bundle_id
quality.reverted_within_14d …
reverted_by, followed_up_by, issue_reopens
task.acmm_level
policy.acmm_level
pkg/outcomebench already folds these for the monthly report; the exporter is a different projection of the same fold.
Not in the bundle today — exporter has to add
sandbox_runtime — stamp from the agent's resolved SandboxRuntime() at kick time (store it in the bundle or a sidecar keyed by PR).
task.kind — derive from agent role/lane (review agents → pr-review, fix lane → issue-fix, CI-repair lane → ci-repair).
cost.* — GET /api/cost is per agent/model/session, not per PR. First version: apportion a session's tokens to the PRs it touched (basis: apportioned), or omit usd and emit per-model unit cost separately.
Scope
Spoke: GET /api/benchmarks/export?since=&until= → JSON array of schema records, redacted per GOVERNANCE.md (no logins, finding text, paths; task.repo only when repo_consent). Gated by benchmarks.publish: true in hive.yaml (default false).
Hub: collect from opted-in spokes, hash hive_id with a per-hub salt, write data/<org>/<yyyy-mm>/<hive_hash>.json and open a PR here (or publish an artifact a maintainer commits).
hivectl benchmarks export offline variant over /data/evidence so owners can publish by hand.
Design discussion lives in the hive RFC (linked below once opened). Implementation PRs go to hivecommons/hive; this issue tracks the contract from the benchmarks side.
Live data is up:https://hivecommons.dev/benchmarks/ — 15,140 records from 13 hives (15,114 attributed to public repos, 23 anonymized; 3 synthetic samples remain under data/sample/ and are hidden by default). 9,556 agent-authored PRs, 87% merged; top model by volume claude-fable-5.1 (94% merged), best merge rate claude-opus-4.7 (98%).
How it got here (stopgap):scripts/backfill-from-cluster.sh pulls /data/evidence straight from every hive-hosted-* spoke pod on the clusters Andy can reach (hive-oke, lke648397), converts with scripts/convert-evidence.py, writes data/fleet/<hive-hash>/<YYYY-MM>.json. The Pages workflow also runs nightly (03:17 UTC) to rebuild the index.
spoke: GET /api/benchmarks/export?since=, benchmarks{} in /api/status, hourly push to hub
hub: POST /api/benchmarks/ingest (heartbeat bearer, hash recomputed server-side), hourly commit to data/fleet/<hash>/<month>.json via the hub App, GET /api/hub/benchmarks
config: benchmarks.publish (default on, anonymized), benchmarks.repo_consent (default on; names only GitHub-public repos)
new bundles stamp sandbox_runtime + task.kind — the two fields that are none/unknown across today's backfill
Remaining after merge: grant the hub App contents: write on this repo; retire the backfill cron once /api/hub/benchmarks shows publishes landing. Per-PR cost/tokens are still not in evidence bundles — tracked in #3's methodology discussion.
The spoke exporter and hub publisher were implemented in hivecommons/hive#17716 (merged into v5). The requested hivectl benchmarks export offline variant is not included in that PR and must be implemented in hivecommons/hive, where hivectl and the evidence conversion package live; this repository cannot implement that command. The hub publisher also requires the hub GitHub App to receive contents:write on this repository, as noted in the merged PR.
Hub aggregation and publish: delivered in the same PR and working in production. hive.hivecommons.dev/api/hub/benchmarks reports 22 hives and 10,310 records held, last ingest 14:00 UTC; this repo's history shows hourly fleet: hourly benchmark records for <hash> (<month>) commits from the hive App bot straight to main, the latest at 13:49 UTC, and the site index was rebuilt at 13:50 UTC (24 hives, 16,748 records). So the App already has contents: write here and nothing is owed on permissions. Layout is data/fleet/<hash>/<YYYY-MM>.json rather than data/<org>/<yyyy-mm>/<hash>.json; it is a commit to main, not a PR.
Offline hivectl benchmarks export: was the missing piece. Delivered in ✨ hivectl benchmarks export: offline schema records from local evidence hive#18995 (issue ✨ hivectl benchmarks export: offline schema records from local evidence hive#18990), merged as 51a0b4c0976765732099da3d3ec6b55c4b334ac0. It reads a local evidence tree with the same converter and privacy rules as the endpoint, names the hive by hash only (--hive-id or --hive-hash), names repos only when passed with --public-repo, and --by-month DIR writes the data/fleet/<hash>/<YYYY-MM>.json files ready to commit here. Documented in src/docs/hivectl.md and src/docs/benchmarks.md in hivecommons/hive.
Not in this issue's scope and still open elsewhere: per-PR cost and tokens are not in evidence bundles (tracked in #3), and scripts/backfill-from-cluster.sh can be retired by the repo owner now that publishes are landing. Everything this issue asked for is delivered, so closing.
Activity
clubanderson commentedon Nov 11, 2026
Design discussion: hivecommons/hive#17694 (RFC).
clubanderson commentedon Nov 11, 2026
Status (2026-11-11)
Live data is up: https://hivecommons.dev/benchmarks/ — 15,140 records from 13 hives (15,114 attributed to public repos, 23 anonymized; 3 synthetic samples remain under
data/sample/and are hidden by default). 9,556 agent-authored PRs, 87% merged; top model by volume claude-fable-5.1 (94% merged), best merge rate claude-opus-4.7 (98%).How it got here (stopgap):
scripts/backfill-from-cluster.shpulls/data/evidencestraight from everyhive-hosted-*spoke pod on the clusters Andy can reach (hive-oke, lke648397), converts withscripts/convert-evidence.py, writesdata/fleet/<hive-hash>/<YYYY-MM>.json. The Pages workflow also runs nightly (03:17 UTC) to rebuild the index.Exporter + hourly publisher (this issue): hivecommons/hive#17716 → v5
GET /api/benchmarks/export?since=,benchmarks{}in/api/status, hourly push to hubPOST /api/benchmarks/ingest(heartbeat bearer, hash recomputed server-side), hourly commit todata/fleet/<hash>/<month>.jsonvia the hub App,GET /api/hub/benchmarksbenchmarks.publish(default on, anonymized),benchmarks.repo_consent(default on; names only GitHub-public repos)sandbox_runtime+task.kind— the two fields that arenone/unknownacross today's backfillRemaining after merge: grant the hub App
contents: writeon this repo; retire the backfill cron once/api/hub/benchmarksshows publishes landing. Per-PR cost/tokens are still not in evidence bundles — tracked in #3's methodology discussion.hivecommons-hive commentedon Nov 18, 2026
🔓 Claim by @Karibhou released (lease revoked).
Claim history
Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; lease revoked)Karibhou commentedon Nov 18, 2026
The spoke exporter and hub publisher were implemented in hivecommons/hive#17716 (merged into v5). The requested
hivectl benchmarks exportoffline variant is not included in that PR and must be implemented in hivecommons/hive, wherehivectland the evidence conversion package live; this repository cannot implement that command. The hub publisher also requires the hub GitHub App to receive contents:write on this repository, as noted in the merged PR.— hive: backend=omp model=openai-codex/gpt-6.1-terra
hivecommons-hive commentedon Nov 18, 2026
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.
Unblocked: the
blockedlabel was applied by the hive without a blocked-on marker, so nothing could lift it. This issue is actionable again.✨ hivectl benchmarks export: offline schema records from local eviden…
clubanderson commentedon Nov 18, 2026
Status against the three scope items, checked on 2026-11-18:
GET /api/benchmarks/export): delivered in ✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/benchmarks hive#17716 (merged 2026-11-11,8fe262e). One deliberate difference from this issue's text:benchmarks.publishships default on (anonymized, opt-out) per RFC 📖 RFC: publish agent/model/runtime benchmark results to hivecommons/benchmarks hive#17694, not default false. Repo and org are named only for repos GitHub reports public.hive.hivecommons.dev/api/hub/benchmarksreports 22 hives and 10,310 records held, last ingest 14:00 UTC; this repo's history shows hourlyfleet: hourly benchmark records for <hash> (<month>)commits from the hive App bot straight to main, the latest at 13:49 UTC, and the site index was rebuilt at 13:50 UTC (24 hives, 16,748 records). So the App already hascontents: writehere and nothing is owed on permissions. Layout isdata/fleet/<hash>/<YYYY-MM>.jsonrather thandata/<org>/<yyyy-mm>/<hash>.json; it is a commit to main, not a PR.hivectl benchmarks export: was the missing piece. Delivered in ✨ hivectl benchmarks export: offline schema records from local evidence hive#18995 (issue ✨ hivectl benchmarks export: offline schema records from local evidence hive#18990), merged as51a0b4c0976765732099da3d3ec6b55c4b334ac0. It reads a local evidence tree with the same converter and privacy rules as the endpoint, names the hive by hash only (--hive-idor--hive-hash), names repos only when passed with--public-repo, and--by-month DIRwrites thedata/fleet/<hash>/<YYYY-MM>.jsonfiles ready to commit here. Documented insrc/docs/hivectl.mdandsrc/docs/benchmarks.mdin hivecommons/hive.Not in this issue's scope and still open elsewhere: per-PR cost and tokens are not in evidence bundles (tracked in #3), and
scripts/backfill-from-cluster.shcan be retired by the repo owner now that publishes are landing. Everything this issue asked for is delivered, so closing.🌱 chore: top up v6 from v5 (2026-11-18) (#19100)