Skip to content

✨ Reproducible mediated-vs-unmediated outcome benchmark with README outcomes block - #14588

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
feat/outcome-benchmark
Oct 27, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
feat/outcome-benchmark

Conversation

@clubanderson

@clubanderson clubanderson commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Fixes #14577

A committed, re-runnable comparison of pull requests that went through Hive's review swarm (mediated: ≥1 verdict or posted review on any head) with pull requests merged without it, computed only from the evidence bundles a spoke already writes. No new telemetry.

What's in it

  • pkg/outcomebench — fold /data/evidence per PR, split cohorts, compute revert rate, change-failure rate (14-day maturation), median time-to-merge, CI failure at merged head (with a comparability guard), changes-requested-before-merge, cost per merged PR (from a saved /api/cost); 95% Wilson intervals; n < 10 → not reported. Renders markdown, two SVG charts (Go only, hive amber palette, dark/light safe) and the README block. 96.6% coverage.
  • hivectl evidence benchmark (+ src/scripts/outcome-benchmark.sh) with --month --now --spend --hub --source --out-dir --readme.
  • First report from hive's own spoke: src/docs/benchmarks/outcomes-2026-10.md + .json + assets/outcomes-2026-10-{bars,trend}.svg, src/docs/benchmarks/README.md (methodology, caveats, how to regenerate).
  • README ## Outcomes section (+20 lines) with a generated <!-- outcomes:start/end --> block and one real, checkable example (hive#13523 review that caught a bad changelog fragment pre-merge).
  • Drift guard: TestCommittedReportMatchesReadme re-renders the newest committed JSON and fails if the README block, markdown or SVGs differ.
  • Docs: hivectl.md, documentation-map.md; changelog fragment.

Headline numbers (spoke snapshot 2026-10-27, merges in 2026-10)

Metric Mediated Unmediated
Merged PRs 1455 2224
Revert rate (matured 14d) 0.0% [0.0–8.2] (n=43) 0.1% [0.0–0.4] (n=1513) — indistinguishable
Change-failure 2.3% [0.4–12.1] (n=43) 1.1% [0.7–1.7] (n=1513)
Median time-to-merge 0.44 h (n=1455) 0.19 h (n=2224) — review adds ~15 min
changes_requested on an earlier head before merge 33.7% [31.3–36.1] (n=1455) —
CI failure at merged head 6.9% [5.3–8.9] (n=798) not comparable (captured on 62/2224)
Cost per merged PR ≈$2.13 est. (n=2869, Aug 8 → Oct 27) —

Scale: 11,571 PRs across 10 repos, 15,054 verdicts.

Honest limits (stated in the report)

  • Mediated merges only began 2026-10-09, so the only clean window is October; the mediated matured-revert denominator is 43 — the revert comparison is not a win claim, the intervals overlap.
  • Hub GET /api/hub/evidence requires an admin session; --hub accepts a saved response but this report is one spoke's data and says so.
  • CI at merged head is only captured for reviewed heads; the unmediated cell is hatched/not reported.
  • Cost is /api/cost's estimate (some models unpriced).

Verification

  • go test -cover ./pkg/outcomebench ./pkg/hivectl/commands → 96.6% / 90.1%; go vet clean; gofmt clean.
  • check-docs-links.py . --no-recurse and src/docs clean for touched files; check-docs-citations.py clean.
  • Charts rendered on dark and light backgrounds (PNGs in the evidence bundle).

Evidence (screenshots)

Rendered docs/benchmarks/outcomes-2026-10.md (GitHub-flavoured markdown):

rendered outcome report

Generated charts (committed under src/docs/benchmarks/assets/, SVG; PNG renders shown):

mediated vs unmediated bars with Wilson whiskers
monthly trend of revert rate and time-to-merge

… with README outcomes block

Adds pkg/outcomebench (fold evidence bundles per PR, split by whether the
review swarm touched any head, revert/change-failure/time-to-merge/CI/
changes-requested/cost with Wilson intervals, n<10 not reported, generated
SVG charts), hivectl evidence benchmark, src/scripts/outcome-benchmark.sh,
the first report from hive's own spoke (2026-10) and the README Outcomes
block it regenerates; a test fails when README, report or charts drift
from the committed JSON.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 422e29e into v5 Oct 27, 2026
81 of 85 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/outcome-benchmark branch October 27, 2026 15:53
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#145882fbd8ec` sha256:9eb9c9b1e8c8c3da3dd344b488d8723df366819b078a2b2f88ae20e0a316497e — download from the dashboard or `hivectl review evidence #14588`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 2fbd8ec.

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

@clubanderson — headline numbers and chart paths for an eyeball before merge (all from hive's own spoke evidence snapshot, 2026-10-27; hub /api/hub/evidence is admin-only from where I ran, so the report states it was not consulted):

Metric Mediated Unmediated
Merged PRs (Oct window) 1455 2224
Revert rate (matured 14 d) 0.0 % [0.0–8.2] n=43 0.1 % [0.0–0.4] n=1513
Change-failure (revert or follow-up fix) 2.3 % [0.4–12.1] n=43 1.1 % [0.7–1.7] n=1513
Median time-to-merge 0.44 h 0.19 h
changes_requested on an earlier head 33.7 % [31.3–36.1] n=1455 —
CI failure at merged head 6.9 % [5.3–8.9] n=798 not comparable (captured on 62/2224)
Cost per merged PR ≈ $2.13 (n=2869, 2026-08-08 → 10-27) —

Scale fact: 11,571 PRs / 10 repos / 15,054 verdicts. First mediated merge is 2026-10-09, so the matured-mediated bucket is only n=43 — the revert comparison is honest but not yet discriminating; the README says so.

What's in the README block (5 bullets, each with n + date): changes-requested share, revert rate, median TTM, cost/PR, scale fact. Deliberately not in README: CI-failure comparison (unmediated side not captured).

Charts (SVG committed, regenerated by the drift test):

  • src/docs/benchmarks/assets/outcomes-2026-10-bars.svg
  • src/docs/benchmarks/assets/outcomes-2026-10-trend.svg

PNG renders are in the PR body above (light); dark renders live locally at ~/hive-work/evidence-ponytail/i3-{bars,trend}-dark.png.

@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14590 ✨ feat(evidence): mediated-vs-unmediated outcome benchmark report same files changelog.d/added-outcome-benchmark.md
src/docs/benchmarks/README.md
src/docs/documentation-map.md
src/docs/hivectl.md
src/pkg/hivectl/commands/evidence.go
src/pkg/hivectl/commands/evidence_benchmark.go
src/pkg/hivectl/commands/evidence_benchmark_test.go
src/pkg/outcomebench/outcomebench_test.go
src/pkg/outcomebench/render.go
src/scripts/outcome-benchmark.sh
#14582 ✨ House-style section every kick opens with: repo HIVE_STYLE.md > hive default > built-in same files src/docs/documentation-map.md

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Reproducible mediated-vs-unmediated outcome benchmark report

1 participant