Repository navigation
✨ feat(evidence): mediated-vs-unmediated outcome benchmark report - #14590
hivecommons-hive[bot] wants to merge 1 commit into
Conversation
Add pkg/outcomebench (pure cohorting, Wilson intervals, n<10 suppression, median time to merge, Markdown rendering with methodology), the `hivectl evidence benchmark` subcommand, src/scripts/outcome-benchmark.sh and docs under src/docs/benchmarks/. Refs #14577 Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Supersession sweep found that this PR's claimed issue is already closed by another merged PR.
This PR touches files the merged PR did not, so the hive is resolving it itself, but GitHub refused to update head d0d4fd2 onto This comment is edited in place by the supersession sweep; it is not duplicated. |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #14590 is closed in favour of the already-merged #14588, which implements the same outcome benchmark for #14577 at the same paths.
This PR appears to duplicate #14588, which merged into v5 at 422e29e (feat(evidence): reproducible mediated-vs-unmediated outcome benchmark with README outcomes block, Fixes #14577). Both PRs are for #14577 and both create the same artefacts: src/pkg/outcomebench/{outcomebench.go,render.go,outcomebench_test.go}, src/pkg/hivectl/commands/evidence_benchmark.go (+test), src/scripts/outcome-benchmark.sh, src/docs/benchmarks/README.md, and even the same changelog fragment name changelog.d/added-outcome-benchmark.md. That is why this PR is CONFLICTING — it is not a rebase problem, the files already exist on v5 with different content.
What I compared
- Cohorting logic is the same in both (mediated = any record carries a verdict or posted review;
control_windowas a secondary cut; Wilson 95%;MinN = 10→ "not reported"). Thepkg/outcomebenchcode in this PR reads correctly againstpkg/evidenceon v5 —PR.Bundleis always set byFoldPRs(fold.go:149,153) andBundle.CIis a value type (evidence.go:67), somergedHeadBundle(...).CI.Statuscannot nil-deref. No defect found in the code itself. - The CLI surfaces differ: v5's merged command takes
--month --now --spend --hub --source --out-dir --readme(evidence_benchmark.go:25,37 on v5); this PR's takes--since --until --out. The two cannot coexist under oneevidence benchmarksubcommand, so this is a conflict, not a mergeable addition. - #14588 is the broader of the two: it also ships the SVG charts, the README
<!-- outcomes:start/end -->block with a drift test, and the committedsrc/docs/benchmarks/outcomes-2026-10.{md,json}— which this PR's body lists as the remaining work underRefs #14577. With #14588 merged, that remainder is already done.
Recommendation: close this PR in favour of #14588. If any piece here is wanted (e.g. the --since/--until window form instead of --month), it would be a small follow-up against the merged command rather than a rebase of this branch. The stage/070-fixing label suggests something is trying to repair the conflict; that effort will not converge.
I have not closed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Open PR overlap checkBase branch:
Conflicts are checked with every PR merged onto the current tip of
|
|
Closing: the feature already landed on v5 via #14588 (merged 2026-10-27T15:53Z, 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
Refs #14577 (remaining: commit
src/docs/benchmarks/outcomes-2026-10.mdgenerated from the spoke's/data/evidenceviasrc/scripts/outcome-benchmark.sh 2026-10— needs an operator run with evidence access, since agents cannot build binaries in-pod)What
src/pkg/outcomebench(new, pure functions, no clock/fs/network):evidence.FoldPRs: mediated = any record of the PR carries a Hive reviewer verdict or posted review; unmediated = none.control_windowon/off secondary cut when any PR carries it.evidence.PR.ChangeFailure: reverts + issue reopens withinLinkWindow), CI failure at merged head, Hivechanges_requestedverdict rate (mediated only), median time to merge (evidence.MedianTimeToMerge), optional cost per merged mediated PR.pkg/hub/evidence.go); any cell with n < 10 rendersnot reported (n=…).changes_requestedreviews, cost when no spend given) and a Methodology section. Output is deterministic.hivectl evidence benchmark --root /data/evidence [--since …] [--until …] [--repo …] [--fleet-spend-usd …] [--out path], sibling ofevidence export; bundle loading factored into a sharedloadEvidenceBundles(export behaviour unchanged).src/scripts/outcome-benchmark.sh YYYY-MMpins absolute UTC month bounds so re-runs reproduce the same bytes.src/docs/hivectl.md,src/docs/review-evidence.md#outcome-benchmark,src/docs/benchmarks/README.md(regeneration + reproducibility guarantee), documentation map entry.changelog.d/added-outcome-benchmark.md.Tests
src/pkg/outcomebench/outcomebench_test.go: cohorting (multi-head, posted-review-only, repo/window filters, exclusive--until, merged-head CI selection and fallback), Wilson, n<10 suppression, median TTM (odd/even, skipped unmeasurable), Markdown rendering and determinism, cost branches.src/pkg/hivectl/commands/evidence_benchmark_test.go: stdout/--out/--repo, byte-identical re-run, missing root, usage errors.Decisions
src/docs/benchmarks/(where the docs tree andsrc/docs/research/live) rather than the rootdocs/, which holds only legacy pages.--until(exclusive) so a monthly report has fixed bounds and is reproducible;--sincealone with a duration would depend on the wall clock.head_shamatchesmerge.sha, else the PR's newest record (squash merges record the merge commit).— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson