Skip to content

📖 RFC: publish agent/model/runtime benchmark results to hivecommons/benchmarks #17694

Closed

Description

@clubanderson

Summary

Publish per-model / per-harness / per-sandbox-runtime benchmark results from opted-in hives to a shared public site, https://github.com/hivecommons/benchmarks (GitHub Pages, JSON in git, zero infra). Project Bluefin (clanker agents) wants to publish alongside us.

This RFC proposes the hive side: an exporter endpoint on the spoke, hub aggregation across hosted hives with per-owner opt-in, redaction rules and cadence. It does not propose implementing anything yet — design review first.

Motivation

Hive already answers "does the review swarm help?" (src/docs/benchmarks/outcomes-2026-10.md), but nobody can answer "which model / harness / runtime is worth running for issue-fix at ACMM L3, and what does a merged PR cost?" — across hives, in public, with intervals. Every hive has the data in /data/evidence; none of it leaves the spoke except as a monthly hand-supplied aggregate.

What we already collect (inventory)

Full write-up: benchmarks/docs/INVENTORY.md. Short version:

Need Have today Gap
agent, harness, exact model evidence.Bundle.author{agent,backend,model} —
merged / CI at head / verdicts / reverts / follow-ups / time-to-merge / human actions / retries Bundle.merge, ci, verdicts[], reverted_by, human_actions[], previous_bundle_id chain — (pkg/outcomebench already folds these)
ACMM level Bundle.policy.acmm_level; GET /api/acmm/evaluation (repo maturity, context only) —
cost GET /api/cost per agent / model / session (pkg/tokens); /api/repo-cost interval join not per PR
sandbox runtime agent config SandboxRuntime() (podman / job / openshell) not stamped on the bundle
task kind agent role / lane not recorded
cross-hive hub heartbeat evidence (/api/hub/evidence: calibration, merged/reverted 90d, model families, TTM medians; admin-only) families only, no exact model, not public
opt-in — none

Proposal

1. Record schema

Emit result.schema.json records. Field names were chosen to match the bundle and /api/cost so the mapping is mostly a projection.

2. Spoke exporter

GET /api/benchmarks/export?since=&until=&format=json (dashboard auth) and hivectl benchmarks export --root /data/evidence. Both are a fold over evidence bundles (reuse pkg/outcomebench/pkg/evidence fold) plus:

  • sandbox_runtime stamped into the bundle at kick time (new optional field runtime on Bundle.Author, or sidecar) — schema v1 → v1.1, additive.
  • task.kind from the agent lane/role.
  • cost: v1 apportions a session's tokens across the PRs it touched (basis: apportioned); exact per-PR attribution is follow-up work.

3. Opt-in and redaction

benchmarks:
  publish: false        # default; nothing leaves the spoke
  publish_name: false   # hive_id is a salted hash unless true
  cost: true            # include estimated usd (token counts always included)
  repos:                # explicit consent list for naming repositories
    - hivecommons/hive

Redaction is in the exporter, not the hub: never emit logins, finding text/paths, prompts, session ids, Nous questions; task.repo only for listed repos that are public. The hub re-validates against the schema and rejects anything with a repo not on the spoke's declared list.

4. Hub aggregation

Spokes with publish: true include an export_url/token in heartbeat; hub pulls monthly (or spokes POST), salts hive_id per hub, writes data/<org>/<yyyy-mm>/<hive_hash>.json and opens a PR against hivecommons/benchmarks via a GitHub App. Owners can also run hivectl benchmarks export and open the PR themselves — the hub path is a convenience, not a requirement.

5. Cadence

Monthly, aligned with monthly-outcomes.yml (15th, 06:17 UTC). The benchmarks repo rebuilds data/index.json and redeploys Pages on every merged PR.

6. Non-goals

  • No new telemetry beyond sandbox_runtime and task.kind on the bundle.
  • No ranking inside hive's dashboard; the public site owns presentation.
  • No change to the mediated/unmediated monthly report.

Open questions

  1. Stamp runtime/task_kind on the bundle (schema bump) or keep a sidecar keyed by (repo, number, head_sha)?
  2. Per-PR cost: apportion by wall-clock overlap, by tool-call count, or publish per-model unit cost only?
  3. Should the hub salt be stable (trends) or rotate monthly (unlinkability)? See 📝 Methodology review: canonical task set and consent rules for repo names benchmarks#3.
  4. Consent granularity for repo names: hive owner vs repo owner.

Links

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    kind/designCreated by Hive for agent-filed issue provenance
    on Nov 11, 2026
  2. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @Karibhou released (lease revoked).

    Claim history
    • 2026-11-11 19:32 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-11 19:33 UTC — released by Karibhou (contributor; lease revoked)
    • 2026-11-11 19:34 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-11 19:34 UTC — released by Karibhou (contributor; lease revoked)
    • 2026-11-11 19:34 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-11 19:35 UTC — released by Karibhou (contributor; lease revoked)
    • 2026-11-11 19:34 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-11 19:38 UTC — released by Karibhou (contributor; lease revoked)
  3. added
    stage/020-claimedClaimed by the scanner or a relay contributor
    stage/018-open-for-discussionOpen for discussion: proposal / design issue with no live claim; claimable, no agent work queued
    and removed
    stage/018-open-for-discussionOpen for discussion: proposal / design issue with no live claim; claimable, no agent work queued
    stage/020-claimedClaimed by the scanner or a relay contributor
    on Nov 11, 2026
  4. 8 remaining items

  5. added
    stage/018-open-for-discussionOpen for discussion: proposal / design issue with no live claim; claimable, no agent work queued
    and removed
    stage/018-open-for-discussionOpen for discussion: proposal / design issue with no live claim; claimable, no agent work queued
    on Nov 11, 2026
  6. Karibhou commented on Nov 11, 2026

    @Karibhou
    Contributor

    The RFC explicitly requests design review before implementation. Maintainer decisions are needed on the public-data privacy and consent policy: whether the hive identifier salt is stable or rotates, and whether repository-name consent belongs to the hive owner or repository owner. The RFC also leaves bundle versus sidecar storage and per-PR cost attribution open. Implementing publication before those decisions could expose linkable hive or repository information in permanent public Git history.

    — hive: backend=omp model=openai-codex/gpt-6.1-terra


    🐝 Hive Agent: contributor | SHA: unknown

  7. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    Contributor

    What to reply

    • /hive approve: go with the recommendation
    • /hive decision <your instructions>: go ahead, following your instructions
    • /hive help: show these again

    If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

    Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

  8. clubanderson commented on Nov 11, 2026

    @clubanderson
    MemberAuthor

    Accepted and implemented.

    Closing as accepted.

  9. removed
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/designCreated by Hive for agent-filed issue provenance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions