Skip to content

✨ Hive exporter: GET /api/benchmarks/export + hub aggregation emitting schema records #1

Closed

Description

@clubanderson

Goal

Hive spokes should be able to emit records that validate against schema/result.schema.json with no manual work, and the hub should aggregate them across hosted hives for owners who opted in.

What already exists (see docs/INVENTORY.md)

Nearly every field is in hive's signed per-PR evidence bundle (pkg/evidence.Bundle, /data/evidence/**):

schema field evidence source
agent.name / agent.harness author.agent / author.backend
model.name (+ provider via pkg/modelfamily) author.model
outcome.pr_merged, merge_method, opened_at, merged_at merge, outcome
outcome.ci_green_first_try ci.status on the bundle with empty previous_bundle_id
outcome.review_verdict, reviewer_models verdicts[]
outcome.human_interventions len(human_actions)
outcome.retries bundles chained by previous_bundle_id
quality.reverted_within_14d … reverted_by, followed_up_by, issue_reopens
task.acmm_level policy.acmm_level

pkg/outcomebench already folds these for the monthly report; the exporter is a different projection of the same fold.

Not in the bundle today — exporter has to add

  • sandbox_runtime — stamp from the agent's resolved SandboxRuntime() at kick time (store it in the bundle or a sidecar keyed by PR).
  • task.kind — derive from agent role/lane (review agents → pr-review, fix lane → issue-fix, CI-repair lane → ci-repair).
  • cost.* — GET /api/cost is per agent/model/session, not per PR. First version: apportion a session's tokens to the PRs it touched (basis: apportioned), or omit usd and emit per-model unit cost separately.

Scope

  1. Spoke: GET /api/benchmarks/export?since=&until= → JSON array of schema records, redacted per GOVERNANCE.md (no logins, finding text, paths; task.repo only when repo_consent). Gated by benchmarks.publish: true in hive.yaml (default false).
  2. Hub: collect from opted-in spokes, hash hive_id with a per-hub salt, write data/<org>/<yyyy-mm>/<hive_hash>.json and open a PR here (or publish an artifact a maintainer commits).
  3. hivectl benchmarks export offline variant over /data/evidence so owners can publish by hand.

Design discussion lives in the hive RFC (linked below once opened). Implementation PRs go to hivecommons/hive; this issue tracks the contract from the benchmarks side.

Activity

  1. clubanderson commented on Nov 11, 2026

    @clubanderson
    MemberAuthor

    Design discussion: hivecommons/hive#17694 (RFC).

  2. clubanderson commented on Nov 11, 2026

    @clubanderson
    MemberAuthor

    Status (2026-11-11)

    Live data is up: https://hivecommons.dev/benchmarks/ — 15,140 records from 13 hives (15,114 attributed to public repos, 23 anonymized; 3 synthetic samples remain under data/sample/ and are hidden by default). 9,556 agent-authored PRs, 87% merged; top model by volume claude-fable-5.1 (94% merged), best merge rate claude-opus-4.7 (98%).

    How it got here (stopgap): scripts/backfill-from-cluster.sh pulls /data/evidence straight from every hive-hosted-* spoke pod on the clusters Andy can reach (hive-oke, lke648397), converts with scripts/convert-evidence.py, writes data/fleet/<hive-hash>/<YYYY-MM>.json. The Pages workflow also runs nightly (03:17 UTC) to rebuild the index.

    Exporter + hourly publisher (this issue): hivecommons/hive#17716 → v5

    • spoke: GET /api/benchmarks/export?since=, benchmarks{} in /api/status, hourly push to hub
    • hub: POST /api/benchmarks/ingest (heartbeat bearer, hash recomputed server-side), hourly commit to data/fleet/<hash>/<month>.json via the hub App, GET /api/hub/benchmarks
    • config: benchmarks.publish (default on, anonymized), benchmarks.repo_consent (default on; names only GitHub-public repos)
    • new bundles stamp sandbox_runtime + task.kind — the two fields that are none/unknown across today's backfill

    Remaining after merge: grant the hub App contents: write on this repo; retire the backfill cron once /api/hub/benchmarks shows publishes landing. Per-PR cost/tokens are still not in evidence bundles — tracked in #3's methodology discussion.

  3. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @Karibhou released (lease revoked).

    Claim history
    • 2026-11-18 07:22 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-18 07:25 UTC — released by Karibhou (contributor; lease revoked)
  4. Karibhou commented on Nov 18, 2026

    @Karibhou

    The spoke exporter and hub publisher were implemented in hivecommons/hive#17716 (merged into v5). The requested hivectl benchmarks export offline variant is not included in that PR and must be implemented in hivecommons/hive, where hivectl and the evidence conversion package live; this repository cannot implement that command. The hub publisher also requires the hub GitHub App to receive contents:write on this repository, as noted in the merged PR.

    — hive: backend=omp model=openai-codex/gpt-6.1-terra

  5. added
    blockedWaiting on something outside this repository; not contributor work until a human clears the label
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    blockedWaiting on something outside this repository; not contributor work until a human clears the label
    on Nov 18, 2026
  6. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

    Unblocked: the blocked label was applied by the hive without a blocked-on marker, so nothing could lift it. This issue is actionable again.

  7. clubanderson commented on Nov 18, 2026

    @clubanderson
    MemberAuthor

    Status against the three scope items, checked on 2026-11-18:

    1. Spoke exporter (GET /api/benchmarks/export): delivered in ✨ benchmarks: per-PR evidence export + opt-in publish to hivecommons/benchmarks hive#17716 (merged 2026-11-11, 8fe262e). One deliberate difference from this issue's text: benchmarks.publish ships default on (anonymized, opt-out) per RFC 📖 RFC: publish agent/model/runtime benchmark results to hivecommons/benchmarks hive#17694, not default false. Repo and org are named only for repos GitHub reports public.
    2. Hub aggregation and publish: delivered in the same PR and working in production. hive.hivecommons.dev/api/hub/benchmarks reports 22 hives and 10,310 records held, last ingest 14:00 UTC; this repo's history shows hourly fleet: hourly benchmark records for <hash> (<month>) commits from the hive App bot straight to main, the latest at 13:49 UTC, and the site index was rebuilt at 13:50 UTC (24 hives, 16,748 records). So the App already has contents: write here and nothing is owed on permissions. Layout is data/fleet/<hash>/<YYYY-MM>.json rather than data/<org>/<yyyy-mm>/<hash>.json; it is a commit to main, not a PR.
    3. Offline hivectl benchmarks export: was the missing piece. Delivered in ✨ hivectl benchmarks export: offline schema records from local evidence hive#18995 (issue ✨ hivectl benchmarks export: offline schema records from local evidence hive#18990), merged as 51a0b4c0976765732099da3d3ec6b55c4b334ac0. It reads a local evidence tree with the same converter and privacy rules as the endpoint, names the hive by hash only (--hive-id or --hive-hash), names repos only when passed with --public-repo, and --by-month DIR writes the data/fleet/<hash>/<YYYY-MM>.json files ready to commit here. Documented in src/docs/hivectl.md and src/docs/benchmarks.md in hivecommons/hive.

    Not in this issue's scope and still open elsewhere: per-PR cost and tokens are not in evidence bundles (tracked in #3), and scripts/backfill-from-cluster.sh can be retired by the repo owner now that publishes are landing. Everything this issue asked for is delivered, so closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions