Skip to content

✨ feat(evidence): mediated-vs-unmediated outcome benchmark report - #14590

Closed
hivecommons-hive[bot] wants to merge 1 commit into
v5from
scanner/fix-14577
Closed

hivecommons-hive[bot] wants to merge 1 commit into
v5from
scanner/fix-14577

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Refs #14577 (remaining: commit src/docs/benchmarks/outcomes-2026-10.md generated from the spoke's /data/evidence via src/scripts/outcome-benchmark.sh 2026-10 — needs an operator run with evidence access, since agents cannot build binaries in-pod)

What

  • src/pkg/outcomebench (new, pure functions, no clock/fs/network):
    • Cohorting from evidence.FoldPRs: mediated = any record of the PR carries a Hive reviewer verdict or posted review; unmediated = none. control_window on/off secondary cut when any PR carries it.
    • Metrics per cohort: revert rate, change-failure rate (hive's existing evidence.PR.ChangeFailure: reverts + issue reopens within LinkWindow), CI failure at merged head, Hive changes_requested verdict rate (mediated only), median time to merge (evidence.MedianTimeToMerge), optional cost per merged mediated PR.
    • 95% Wilson intervals (same formula as pkg/hub/evidence.go); any cell with n < 10 renders not reported (n=…).
    • Markdown renderer with header, headline table, control-window cut, "Not available in evidence schema v1" (post-merge CI re-runs, human changes_requested reviews, cost when no spend given) and a Methodology section. Output is deterministic.
  • hivectl evidence benchmark --root /data/evidence [--since …] [--until …] [--repo …] [--fleet-spend-usd …] [--out path], sibling of evidence export; bundle loading factored into a shared loadEvidenceBundles (export behaviour unchanged).
  • src/scripts/outcome-benchmark.sh YYYY-MM pins absolute UTC month bounds so re-runs reproduce the same bytes.
  • Docs: src/docs/hivectl.md, src/docs/review-evidence.md#outcome-benchmark, src/docs/benchmarks/README.md (regeneration + reproducibility guarantee), documentation map entry.
  • Changelog: changelog.d/added-outcome-benchmark.md.

Tests

  • src/pkg/outcomebench/outcomebench_test.go: cohorting (multi-head, posted-review-only, repo/window filters, exclusive --until, merged-head CI selection and fallback), Wilson, n<10 suppression, median TTM (odd/even, skipped unmeasurable), Markdown rendering and determinism, cost branches.
  • src/pkg/hivectl/commands/evidence_benchmark_test.go: stdout/--out/--repo, byte-identical re-run, missing root, usage errors.
  • Not run locally per AGENTS.md; CI is the verdict. New package targets the default 90% coverage floor, so no floor entry was added.

Decisions

  • Decision taken by hive: reports live in src/docs/benchmarks/ (where the docs tree and src/docs/research/ live) rather than the root docs/, which holds only legacy pages.
  • Decision taken by hive: added --until (exclusive) so a monthly report has fixed bounds and is reproducible; --since alone with a duration would depend on the wall clock.
  • Decision taken by hive: CI at merged head is read from the bundle whose head_sha matches merge.sha, else the PR's newest record (squash merges record the merge commit).

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

Add pkg/outcomebench (pure cohorting, Wilson intervals, n<10 suppression,
median time to merge, Markdown rendering with methodology), the
`hivectl evidence benchmark` subcommand, src/scripts/outcome-benchmark.sh
and docs under src/docs/benchmarks/.

Refs #14577

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

  • Claimed issue: hive#14577
  • Closing PR: #14588

This PR touches files the merged PR did not, so the hive is resolving it itself, but GitHub refused to update head d0d4fd2 onto v5 (merge conflict between base and head). The fix lane owns the rebase; the sweep re-measures the diff once a new head lands.

This comment is edited in place by the supersession sweep; it is not duplicated.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #14590 is closed in favour of the already-merged #14588, which implements the same outcome benchmark for #14577 at the same paths.

This PR appears to duplicate #14588, which merged into v5 at 422e29e (feat(evidence): reproducible mediated-vs-unmediated outcome benchmark with README outcomes block, Fixes #14577). Both PRs are for #14577 and both create the same artefacts: src/pkg/outcomebench/{outcomebench.go,render.go,outcomebench_test.go}, src/pkg/hivectl/commands/evidence_benchmark.go (+test), src/scripts/outcome-benchmark.sh, src/docs/benchmarks/README.md, and even the same changelog fragment name changelog.d/added-outcome-benchmark.md. That is why this PR is CONFLICTING — it is not a rebase problem, the files already exist on v5 with different content.

What I compared

  • Cohorting logic is the same in both (mediated = any record carries a verdict or posted review; control_window as a secondary cut; Wilson 95%; MinN = 10 → "not reported"). The pkg/outcomebench code in this PR reads correctly against pkg/evidence on v5 — PR.Bundle is always set by FoldPRs (fold.go:149,153) and Bundle.CI is a value type (evidence.go:67), so mergedHeadBundle(...).CI.Status cannot nil-deref. No defect found in the code itself.
  • The CLI surfaces differ: v5's merged command takes --month --now --spend --hub --source --out-dir --readme (evidence_benchmark.go:25,37 on v5); this PR's takes --since --until --out. The two cannot coexist under one evidence benchmark subcommand, so this is a conflict, not a mergeable addition.
  • #14588 is the broader of the two: it also ships the SVG charts, the README <!-- outcomes:start/end --> block with a drift test, and the committed src/docs/benchmarks/outcomes-2026-10.{md,json} — which this PR's body lists as the remaining work under Refs #14577. With #14588 merged, that remainder is already done.

Recommendation: close this PR in favour of #14588. If any piece here is wanted (e.g. the --since/--until window form instead of --month), it would be a small follow-up against the merged command rather than a rebase of this branch. The stage/070-fixing label suggests something is trying to repair the conflict; that effort will not converge.

I have not closed anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14603 📖 docs: rewrite README around plain-language evidence cards same files src/docs/benchmarks/README.md
src/docs/documentation-map.md
src/docs/hivectl.md
src/pkg/hivectl/commands/evidence_benchmark.go
src/pkg/outcomebench/outcomebench_test.go
src/pkg/outcomebench/render.go

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot closed this Oct 27, 2026
@hivecommons-hive
hivecommons-hive Bot deleted the scanner/fix-14577 branch October 27, 2026 16:58
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Closing: the feature already landed on v5 via #14588 (merged 2026-10-27T15:53Z, Fixes #14577, issue closed). Every conflicted file here is an add/add against that commit, so this branch has nothing left to contribute.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants