Repository navigation
Description
Summary
Publish per-model / per-harness / per-sandbox-runtime benchmark results from opted-in hives to a shared public site, https://github.com/hivecommons/benchmarks (GitHub Pages, JSON in git, zero infra). Project Bluefin (clanker agents) wants to publish alongside us.
This RFC proposes the hive side: an exporter endpoint on the spoke, hub aggregation across hosted hives with per-owner opt-in, redaction rules and cadence. It does not propose implementing anything yet — design review first.
Motivation
Hive already answers "does the review swarm help?" (src/docs/benchmarks/outcomes-2026-10.md), but nobody can answer "which model / harness / runtime is worth running for issue-fix at ACMM L3, and what does a merged PR cost?" — across hives, in public, with intervals. Every hive has the data in /data/evidence; none of it leaves the spoke except as a monthly hand-supplied aggregate.
What we already collect (inventory)
Full write-up: benchmarks/docs/INVENTORY.md. Short version:
| Need | Have today | Gap |
|---|---|---|
| agent, harness, exact model | evidence.Bundle.author{agent,backend,model} |
— |
| merged / CI at head / verdicts / reverts / follow-ups / time-to-merge / human actions / retries | Bundle.merge, ci, verdicts[], reverted_by, human_actions[], previous_bundle_id chain |
— (pkg/outcomebench already folds these) |
| ACMM level | Bundle.policy.acmm_level; GET /api/acmm/evaluation (repo maturity, context only) |
— |
| cost | GET /api/cost per agent / model / session (pkg/tokens); /api/repo-cost interval join |
not per PR |
| sandbox runtime | agent config SandboxRuntime() (podman / job / openshell) |
not stamped on the bundle |
| task kind | agent role / lane | not recorded |
| cross-hive | hub heartbeat evidence (/api/hub/evidence: calibration, merged/reverted 90d, model families, TTM medians; admin-only) |
families only, no exact model, not public |
| opt-in | — | none |
Proposal
1. Record schema
Emit result.schema.json records. Field names were chosen to match the bundle and /api/cost so the mapping is mostly a projection.
2. Spoke exporter
GET /api/benchmarks/export?since=&until=&format=json (dashboard auth) and hivectl benchmarks export --root /data/evidence. Both are a fold over evidence bundles (reuse pkg/outcomebench/pkg/evidence fold) plus:
sandbox_runtimestamped into the bundle at kick time (new optional fieldruntimeonBundle.Author, or sidecar) — schema v1 → v1.1, additive.task.kindfrom the agent lane/role.cost: v1 apportions a session's tokens across the PRs it touched (basis: apportioned); exact per-PR attribution is follow-up work.
3. Opt-in and redaction
benchmarks:
publish: false # default; nothing leaves the spoke
publish_name: false # hive_id is a salted hash unless true
cost: true # include estimated usd (token counts always included)
repos: # explicit consent list for naming repositories
- hivecommons/hiveRedaction is in the exporter, not the hub: never emit logins, finding text/paths, prompts, session ids, Nous questions; task.repo only for listed repos that are public. The hub re-validates against the schema and rejects anything with a repo not on the spoke's declared list.
4. Hub aggregation
Spokes with publish: true include an export_url/token in heartbeat; hub pulls monthly (or spokes POST), salts hive_id per hub, writes data/<org>/<yyyy-mm>/<hive_hash>.json and opens a PR against hivecommons/benchmarks via a GitHub App. Owners can also run hivectl benchmarks export and open the PR themselves — the hub path is a convenience, not a requirement.
5. Cadence
Monthly, aligned with monthly-outcomes.yml (15th, 06:17 UTC). The benchmarks repo rebuilds data/index.json and redeploys Pages on every merged PR.
6. Non-goals
- No new telemetry beyond
sandbox_runtimeandtask.kindon the bundle. - No ranking inside hive's dashboard; the public site owns presentation.
- No change to the mediated/unmediated monthly report.
Open questions
- Stamp
runtime/task_kindon the bundle (schema bump) or keep a sidecar keyed by(repo, number, head_sha)? - Per-PR cost: apportion by wall-clock overlap, by tool-call count, or publish per-model unit cost only?
- Should the hub salt be stable (trends) or rotate monthly (unlinkability)? See 📝 Methodology review: canonical task set and consent rules for repo names benchmarks#3.
- Consent granularity for repo names: hive owner vs repo owner.
Links
- Repo: https://github.com/hivecommons/benchmarks · site: https://hivecommons.dev/benchmarks/
- Help wanted: ✨ Hive exporter: GET /api/benchmarks/export + hub aggregation emitting schema records benchmarks#1 (exporter), docs: add EXECUTOR MODE pattern and multi-agent topology #2 (site), ✨ Add macOS/launchd support, hybrid scanner pattern, and KubeStellar case study #3 (methodology)
Activity
✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/…
✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/…
hivecommons-hive commentedon Nov 11, 2026
🔓 Claim by @Karibhou released (lease revoked).
Claim history
Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; lease revoked)Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; lease revoked)Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; lease revoked)Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; lease revoked)8 remaining items
Karibhou commentedon Nov 11, 2026
The RFC explicitly requests design review before implementation. Maintainer decisions are needed on the public-data privacy and consent policy: whether the hive identifier salt is stable or rotates, and whether repository-name consent belongs to the hive owner or repository owner. The RFC also leaves bundle versus sidecar storage and per-PR cost attribution open. Implementing publication before those decisions could expose linkable hive or repository information in permanent public Git history.
— hive: backend=omp model=openai-codex/gpt-6.1-terra
🐝 Hive Agent:
contributor| SHA:unknownhivecommons-hive commentedon Nov 11, 2026
What to reply
/hive approve: go with the recommendation/hive decision <your instructions>: go ahead, following your instructions/hive help: show these againIf what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply
/fixedto close it as completed; Hive clears its park labels.Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.
✨ benchmarks: per-PR estimated cost in /api/benchmarks/export (#17791)
clubanderson commentedon Nov 11, 2026
Accepted and implemented.
GET /api/benchmarks/export, hub ingest + hourly publisher tohivecommons/benchmarks(data/fleet/<hash>/<YYYY-MM>.json),benchmarks.publishopt-out,repo_consent(public repos only), Settings → Benchmarks hash display.cost_usd_est+cost_basis, per-model $/merged PR, pinnedschema/pricing.json); hive side in ✨ benchmarks: per-PR estimated cost in /api/benchmarks/export (launch attribution × price table) #17791 (issue ✨ benchmarks export: per-PR estimated cost from launch attribution #17790).Closing as accepted.
🌱 Top up v6 with v5 latest (v5.237.2) (#18030)
✨ benchmarks: per-PR estimated cost in /api/benchmarks/export (#17791)