Repository navigation
✨ Reproducible mediated-vs-unmediated outcome benchmark with README outcomes block - #14588
Conversation
… with README outcomes block Adds pkg/outcomebench (fold evidence bundles per PR, split by whether the review swarm touched any head, revert/change-failure/time-to-merge/CI/ changes-requested/cost with Wilson intervals, n<10 not reported, generated SVG charts), hivectl evidence benchmark, src/scripts/outcome-benchmark.sh, the first report from hive's own spoke (2026-10) and the README Outcomes block it regenerates; a test fails when README, report or charts drift from the committed JSON. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 2fbd8ec. |
|
@clubanderson — headline numbers and chart paths for an eyeball before merge (all from hive's own spoke evidence snapshot, 2026-10-27; hub
Scale fact: 11,571 PRs / 10 repos / 15,054 verdicts. First mediated merge is 2026-10-09, so the matured-mediated bucket is only n=43 — the revert comparison is honest but not yet discriminating; the README says so. What's in the README block (5 bullets, each with n + date): changes-requested share, revert rate, median TTM, cost/PR, scale fact. Deliberately not in README: CI-failure comparison (unmediated side not captured). Charts (SVG committed, regenerated by the drift test):
PNG renders are in the PR body above (light); dark renders live locally at |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #14577
A committed, re-runnable comparison of pull requests that went through Hive's review swarm (mediated: ≥1 verdict or posted review on any head) with pull requests merged without it, computed only from the evidence bundles a spoke already writes. No new telemetry.
What's in it
pkg/outcomebench— fold/data/evidenceper PR, split cohorts, compute revert rate, change-failure rate (14-day maturation), median time-to-merge, CI failure at merged head (with a comparability guard), changes-requested-before-merge, cost per merged PR (from a saved/api/cost); 95% Wilson intervals; n < 10 →not reported. Renders markdown, two SVG charts (Go only, hive amber palette, dark/light safe) and the README block. 96.6% coverage.hivectl evidence benchmark(+src/scripts/outcome-benchmark.sh) with--month --now --spend --hub --source --out-dir --readme.src/docs/benchmarks/outcomes-2026-10.md+.json+assets/outcomes-2026-10-{bars,trend}.svg,src/docs/benchmarks/README.md(methodology, caveats, how to regenerate).## Outcomessection (+20 lines) with a generated<!-- outcomes:start/end -->block and one real, checkable example (hive#13523 review that caught a bad changelog fragment pre-merge).TestCommittedReportMatchesReadmere-renders the newest committed JSON and fails if the README block, markdown or SVGs differ.hivectl.md,documentation-map.md; changelog fragment.Headline numbers (spoke snapshot 2026-10-27, merges in 2026-10)
changes_requestedon an earlier head before mergeScale: 11,571 PRs across 10 repos, 15,054 verdicts.
Honest limits (stated in the report)
GET /api/hub/evidencerequires an admin session;--hubaccepts a saved response but this report is one spoke's data and says so./api/cost's estimate (some models unpriced).Verification
go test -cover ./pkg/outcomebench ./pkg/hivectl/commands→ 96.6% / 90.1%;go vetclean; gofmt clean.check-docs-links.py . --no-recurseandsrc/docsclean for touched files;check-docs-citations.pyclean.Evidence (screenshots)
Rendered
docs/benchmarks/outcomes-2026-10.md(GitHub-flavoured markdown):Generated charts (committed under
src/docs/benchmarks/assets/, SVG; PNG renders shown):