diff --git a/changelog.d/added-outcome-benchmark.md b/changelog.d/added-outcome-benchmark.md new file mode 100644 index 0000000000..4c17511c49 --- /dev/null +++ b/changelog.d/added-outcome-benchmark.md @@ -0,0 +1 @@ +- **Outcome benchmark report:** `hivectl evidence benchmark` (and `src/scripts/outcome-benchmark.sh YYYY-MM`) renders a reproducible Markdown report from the evidence bundles comparing Hive-mediated PRs with unmediated ones — revert rate, change-failure rate, CI failure at the merged head, Hive `changes_requested` verdict rate, median time to merge and optional cost per merged PR, with 95% Wilson intervals, `not reported` below n = 10, a `control_window` on/off cut when present, and a methodology section ([#14577](https://github.com/hivecommons/hive/issues/14577)). diff --git a/src/docs/benchmarks/README.md b/src/docs/benchmarks/README.md new file mode 100644 index 0000000000..889ba90f61 --- /dev/null +++ b/src/docs/benchmarks/README.md @@ -0,0 +1,38 @@ +# Outcome benchmarks + +Monthly reports comparing the outcomes of pull requests Hive mediated with those merged without it +([#14577](https://github.com/hivecommons/hive/issues/14577)). Each `outcomes-YYYY-MM.md` here is generated, never +hand-edited: it is the output of `hivectl evidence benchmark` over a spoke's evidence root +(see [Review evidence bundle](../review-evidence.md#outcome-benchmark) and [hivectl](../hivectl.md)). + +## Regenerating a report + +On a spoke (or any host holding a copy of its `/data/evidence`), with `hivectl` on `PATH`: + +```bash +src/scripts/outcome-benchmark.sh 2026-10 # writes src/docs/benchmarks/outcomes-2026-10.md +src/scripts/outcome-benchmark.sh --fleet-spend-usd 412.50 2026-10 # also report cost per merged mediated PR +src/scripts/outcome-benchmark.sh --root ./evidence-copy --out - 2026-10 # another root, Markdown on stdout +``` + +The script pins the window to the UTC calendar month (`--since YYYY-MM-01T00:00:00Z --until -01T00:00:00Z`) +and calls `hivectl evidence benchmark`. `HIVECTL` picks the binary, `EVIDENCE_ROOT` the root and `FLEET_SPEND_USD` the +optional spend input. + +## Reproducibility + +The report is a pure function of the bundles on disk and the flags: no wall clock appears in it, PRs are folded with +the same `evidence.FoldPRs` the summary API and `hivectl evidence export` use, and every aggregate is order +independent. Re-running the script for the same month on the same evidence root produces the same file byte for +byte; a diff between two runs means the evidence changed (for example a revert or an issue reopen was linked after +the first run, or a backfill added history). + +## Reading a report + +- **Mediated** PRs carry at least one Hive reviewer verdict or a review Hive posted; **unmediated** PRs carry + neither. The split is observational, not randomised; the `control_window` on/off cut, when bundles carry it, is + the prospectively assigned arm. +- Rates carry 95% Wilson intervals. Any cell with fewer than 10 PRs reads `not reported` with its n. +- Metrics evidence schema v1 does not capture (post-merge CI re-runs, human `changes_requested` reviews) are listed + as not available rather than approximated. +- The report's own Methodology section defines every metric. diff --git a/src/docs/documentation-map.md b/src/docs/documentation-map.md index f730072c79..aa7a5725d9 100644 --- a/src/docs/documentation-map.md +++ b/src/docs/documentation-map.md @@ -155,6 +155,7 @@ Documentation for the current Hive line (branch `v5`; the code and docs live und - [Review swarm](review-swarm.md) — the five review perspectives, the verdict collector and its report contract, and the opt-in merge-gate integration and bounded auto-fix cycle for review findings. - [Swarm mode](swarm.md) — focusing the hive on one repository for a bounded window: Discord announcements, theming (`swarm:` config, `HIVE_SWARM_EVENT_NAME` / `HIVE_SWARM_CALL_TO_ARMS`), the `/api/leaderboard/swarm` endpoint and scoring. - [Review evidence bundle](review-evidence.md) — the per-PR evidence bundle schema (v1), canonical hash, Ed25519 signing, offline verification, and retention design. +- [Outcome benchmarks](benchmarks/README.md) — monthly mediated-vs-unmediated outcome reports generated by `hivectl evidence benchmark`, and how to reproduce them. - [Evidence reporting](evidence.md) — the guarded control-arm (`control_window: on|off`) evidence report: per-repository outcome comparison, the minimum-denominator rule, and why it is reporting groundwork rather than an enabled experiment. - [SOC 2 control mapping (not a compliance claim)](soc2-control-mapping.md) — maps common auditor change-management questions to bundle fields, with sample policy language and the gaps Hive does not cover. - [Duplicate PR sweep](duplicate-sweep.md) — the opt-in (`duplicate_sweep.enabled`) cross-PR pass that clusters open PRs by changed-file set and suggests which one to keep: the two confidence tiers, how a survivor is chosen, why a bot regeneration series gets one summary instead of a comment per PR, and why the output is always a suggestion a human acts on and never an automated close. diff --git a/src/docs/hivectl.md b/src/docs/hivectl.md index 2fd4f38380..49f3d50a25 100644 --- a/src/docs/hivectl.md +++ b/src/docs/hivectl.md @@ -288,6 +288,23 @@ Columns: `repo`, `pr`, `head_sha`, `bundle_id`, `author_login`, `author_kind`, ` Reporting only: nothing here reads into or alters merge eligibility. See [Review evidence bundle](review-evidence.md#exporting). +### evidence — outcome benchmark report + +```bash +hivectl evidence benchmark --since 2026-10-01T00:00:00Z --until 2026-11-01T00:00:00Z --out outcomes.md +hivectl evidence benchmark --since 720h --repo hivecommons/hive --fleet-spend-usd 412.50 # Markdown on stdout +src/scripts/outcome-benchmark.sh 2026-10 # one UTC month into src/docs/benchmarks/outcomes-2026-10.md +``` + +`evidence benchmark` folds the same bundles as `evidence export` and renders a Markdown report comparing PRs Hive +mediated (at least one Hive reviewer verdict or posted review) with unmediated ones: revert rate, change-failure rate +(reverts + issue reopens), CI failure at the merged head, Hive `changes_requested` verdict rate (mediated only), median +time to merge and, with `--fleet-spend-usd`, cost per merged mediated PR. `control_window` on/off is a secondary cut +when bundles carry it. Rates carry 95% Wilson intervals and any cell with n < 10 reads `not reported`. `--since` and +`--until` (RFC3339 or a Go duration back from now) window PRs on open time, `since <= opened_at < until`; with +absolute bounds the output depends only on the bundles, so a re-run reproduces it byte for byte. Local filesystem only, +reporting only. See [Outcome benchmark](review-evidence.md#outcome-benchmark) and [benchmarks/](benchmarks/README.md). + ### evidence — backfill bundles from GitHub history ```bash diff --git a/src/docs/review-evidence.md b/src/docs/review-evidence.md index 2a4b804bc4..666967532b 100644 --- a/src/docs/review-evidence.md +++ b/src/docs/review-evidence.md @@ -208,7 +208,7 @@ A PR's outcome (`pkg/evidence.PR.OutcomeKnown`) has three sources, folded per PR - **Work balance (panel 6):** answers "is the queue keeping up?" from GitHub truth rather than from the hive's own audit log alone. `work_balance` in the summary has one row per ISO week of the window with `opened` (issues, split `human` / `agent` / `bot`), `closed`, `prs_opened` (same split), `prs_merged`, `prs_closed` (closed without merge), `queue_issues` / `queue_prs` (open at week end), `issue_source` and `throttled`; `work_balance_sources` names where each series came from. **Sources:** issue opens and closes come from GitHub's issues API (`state=all`, `since`), every author included, pulled by the metrics collector hourly into the durable rollup `/data/hive-metrics/issue-activity.json` (42-day initial lookback, 180-day retention; `issues_github_since` is the first date it covers, `issues_fetched_at` the last refresh); author kind is `agent` when the body carries an agent attribution trailer, `bot` for GitHub `Bot` accounts, else `human`, and an audited agent-created issue overrides the kind to `agent`. Weeks whose Monday predates `issues_github_since` read the backfilled issue series at `///issues/.json` when its coverage contains the requested portion of that week. Event timestamps keep partial date ranges exact; zero-event files establish observed zero counts. The collector stays authoritative for weeks whose Monday is at or after `issues_github_since`, including exact equality. Missing, malformed, or insufficiently covered series files retain the audit fallback per repository (`issue_source: "audit"`, counted in `audit_only_weeks`); mixed repository rows also disclose audit coverage. Backfilled rows use `issue_source: "github"`; `issue_series_weeks` counts the repository-weeks supplied by the series. Series events carry no issue number, so the audited agent-created override above cannot be applied to series weeks: an agent-opened issue without the attribution trailer or 🐝 footer keeps the kind the writer recorded (`bot` for a `Bot` account) in series weeks while the collector reclassifies the same issue as `agent` in rollup weeks. The audit log sees only hive-mediated writes, so the human column is a floor for audit weeks. `hivectl evidence backfill-issues` writes the series (see [CLI reference](hivectl.md#evidence--backfill-bundles-from-github-history)). When the collector starts midweek, backfill through the following Monday (exclusive `--until`) so the entire boundary week has series coverage; stopping at the collector timestamp leaves that week on audit fallback. PR opens, merges and closes come from the folded evidence bundles (live and backfilled) for PRs opened in the window, so PRs this hive never saw are missing. Week-end queue depth is the governor's last state snapshot of the week (`/data/nous/snapshots`, actionable open issues / PRs — the same numbers as the dashboard queue section, held items excluded; `queue_samples`, `queue_retention_days` 30). **Denominator policy:** issue ratio = `closed` ÷ `opened` and PR ratio = `prs_merged` ÷ `prs_opened` over the whole window; nothing is dropped, PRs closed without merge are drawn as a tick but never counted as merged, and an unmatched close (an issue opened before the window) still counts as closed because the question is throughput, not per-issue lifecycle. **Grading** (`EV_THROUGHPUT_*`, `EV_QUEUE_GROWTH_MAX`): issues and PRs are graded separately and the card takes the lower — ratio ≥ 0.95 Excellent, ≥ 0.85 Good, ≥ 0.70 Fair, below Weak, and any grade is Weak when the week-end queue grew more than 10% from the previous week. Raw volume is never graded: a hive that opens 400 issues a week and closes 400 is Excellent. The Evidence sub-panel draws two charts (issues, PRs): opened stacked by kind, a closed / merged line, a closed-unmerged tick and a dashed week-end depth line on the right axis, under a headline "Queue at end of : N open issues · M open PRs". `throttled` stays false because this hive reports no governor issue-creation throttle windows (`missing.throttle_windows_unavailable`). Reporting only ([#12366](https://github.com/hivecommons/hive/issues/12366)). - **Dashboard:** owners get an **Evidence** button on Review Queue rows and Pipeline cards. It opens a modal with the bundle id, head, hash, whether it is signed, the verdict count and merge status, and **Download JSON** / **Download ZIP** buttons. - **CLI:** `hivectl review evidence [--head ] [-o FILE] [--zip]` prints or saves the bundle, and `hivectl review evidence verify FILE [--pubkey KEY]` checks it offline (see [hivectl](hivectl.md#review--review-evidence)). -- **Export:** `hivectl evidence export [--since ] [--format csv|jsonl] [--repo owner/name] [--out FILE] [--root DIR]` dumps one row per PR straight from the bundles on disk; see [Exporting](#exporting). +- **Export:** `hivectl evidence export [--since ] [--format csv|jsonl] [--repo owner/name] [--out FILE] [--root DIR]` dumps one row per PR straight from the bundles on disk; see [Exporting](#exporting). `hivectl evidence benchmark` aggregates the same rows into the mediated-vs-unmediated report; see [Outcome benchmark](#outcome-benchmark). - **On the PR:** when a PR merges through any Hive merge path, the merge event is added to the head's bundle (re-signed) and Hive posts one comment: ``` @@ -232,6 +232,16 @@ hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive # - Rows are ordered by repo (case-insensitive) then PR number, so two runs over the same files are byte-identical. Files that do not parse or are not schema v1 are skipped and counted on stderr; nothing is fabricated. - Reporting only: the export reads bundles and never influences a merge. +## Outcome benchmark + +`hivectl evidence benchmark` turns the per-PR receipts into a standing aggregate: a Markdown report comparing PRs Hive mediated with PRs merged without it ([#14577](https://github.com/hivecommons/hive/issues/14577)). It reads the same bundles and the same `evidence.FoldPRs` fold as [Exporting](#exporting), windows on PR open time (`--since` <= `opened_at` < `--until`), and is built by `pkg/outcomebench` (pure functions, no clock). + +- **Cohorts:** *mediated* = at least one record of the PR carries a Hive reviewer verdict or a review Hive posted; *unmediated* = none. `control_window` (`on`/`off`) is a secondary cut when present. +- **Metrics per cohort:** revert rate, change-failure rate (reverts + issue reopens within the link window, `evidence.PR.ChangeFailure`), CI failure at the merged head (`ci.status` of the bundle whose head is `merge.sha`, else the newest), Hive `changes_requested` verdict rate (mediated only), median time to merge (`merge.at − merge.opened_at`), and cost per merged mediated PR when `--fleet-spend-usd` is given. +- **Intervals:** 95% Wilson, the formula the hub evidence view uses; any cell with n < 10 reads `not reported`. +- **Not available in schema v1:** post-merge CI re-runs and human `changes_requested` reviews are listed, not approximated. +- `src/scripts/outcome-benchmark.sh YYYY-MM` renders one UTC month into [`src/docs/benchmarks/`](benchmarks/README.md); re-running it on the same evidence root reproduces the same file. + ## Signing key Set `evidence.signing_key_file` in `hive.yaml` (see the [operator reference](operator-reference.md#notable-fields)). The file holds an Ed25519 private key as a 32-byte seed or 64-byte key, hex or base64 encoded. If the path is empty, bundles are written **unsigned** with `"signed": false`; they still carry a `hash`, which detects accidental change but not tampering by someone who can recompute it. A missing or unusable configured key also makes writers fall back to unsigned bundles, but dashboard readers fail closed until that configured key is restored, so an unavailable key cannot silently downgrade signed evidence. Keep the private key off shared hosts and publish only the public key. diff --git a/src/pkg/hivectl/commands/evidence.go b/src/pkg/hivectl/commands/evidence.go index 97366657c8..a08ab652b3 100644 --- a/src/pkg/hivectl/commands/evidence.go +++ b/src/pkg/hivectl/commands/evidence.go @@ -100,11 +100,12 @@ var evidenceExportColumns = []string{ func newEvidenceCommand(env *commandEnv) *cobra.Command { cmd := &cobra.Command{ Use: "evidence", - Short: "Per-PR review evidence: export stored bundles or backfill them from forge history", - Long: "Work with the review evidence bundles Hive stores under the evidence root (default /data/evidence).\nexport reads the local filesystem only; backfill talks to the GitHub API. Neither contacts a hive and nothing here\ninfluences a merge.", - Example: " hivectl evidence export --since 168h --format csv --out evidence.csv\n hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive\n hivectl evidence backfill --repo ublue-os/bluefin --since 2025-01-01 --dry-run", + Short: "Per-PR review evidence: export or benchmark stored bundles, or backfill them from forge history", + Long: "Work with the review evidence bundles Hive stores under the evidence root (default /data/evidence).\nexport and benchmark read the local filesystem only; backfill talks to the GitHub API. None contacts a hive and nothing here\ninfluences a merge.", + Example: " hivectl evidence export --since 168h --format csv --out evidence.csv\n hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive\n hivectl evidence benchmark --since 2026-10-01T00:00:00Z --until 2026-11-01T00:00:00Z --out outcomes.md\n hivectl evidence backfill --repo ublue-os/bluefin --since 2025-01-01 --dry-run", } cmd.AddCommand(newEvidenceExportCommand(env)) + cmd.AddCommand(newEvidenceBenchmarkCommand(env)) cmd.AddCommand(newEvidenceBackfillCommand(env)) cmd.AddCommand(newEvidenceBackfillIssuesCommand(env)) return cmd @@ -213,6 +214,32 @@ func parseEvidenceSince(s string, now time.Time) (time.Time, error) { // time), and returns rows ordered by repo then PR number plus the unreadable // count. func exportEvidenceRows(root string, cutoff time.Time, repo string) ([]evidenceExportRow, int, error) { + bundles, unreadable, err := loadEvidenceBundles(root) + if err != nil { + return nil, unreadable, err + } + prs := evidence.FoldPRs(bundles, repo, "") + rows := make([]evidenceExportRow, 0, len(prs)) + for _, p := range prs { + if !cutoff.IsZero() && p.OpenedAt.Before(cutoff) { + continue + } + rows = append(rows, evidenceRowFromPR(p)) + } + sort.Slice(rows, func(i, j int) bool { + ri, rj := strings.ToLower(rows[i].Repo), strings.ToLower(rows[j].Repo) + if ri != rj { + return ri < rj + } + return rows[i].PR < rows[j].PR + }) + return rows, unreadable, nil +} + +// loadEvidenceBundles walks root and returns every readable schema v1 bundle +// plus the count of .json files that did not parse or are not v1. A missing +// root is empty, not an error. +func loadEvidenceBundles(root string) ([]*evidence.Bundle, int, error) { var bundles []*evidence.Bundle unreadable := 0 err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { @@ -236,22 +263,7 @@ func exportEvidenceRows(root string, cutoff time.Time, repo string) ([]evidenceE if err != nil { return nil, unreadable, fmt.Errorf("reading evidence under %s: %w", root, err) } - prs := evidence.FoldPRs(bundles, repo, "") - rows := make([]evidenceExportRow, 0, len(prs)) - for _, p := range prs { - if !cutoff.IsZero() && p.OpenedAt.Before(cutoff) { - continue - } - rows = append(rows, evidenceRowFromPR(p)) - } - sort.Slice(rows, func(i, j int) bool { - ri, rj := strings.ToLower(rows[i].Repo), strings.ToLower(rows[j].Repo) - if ri != rj { - return ri < rj - } - return rows[i].PR < rows[j].PR - }) - return rows, unreadable, nil + return bundles, unreadable, nil } // evidenceRowFromPR flattens one folded PR: the newest bundle's verdicts, diff --git a/src/pkg/hivectl/commands/evidence_benchmark.go b/src/pkg/hivectl/commands/evidence_benchmark.go new file mode 100644 index 0000000000..7ca609c578 --- /dev/null +++ b/src/pkg/hivectl/commands/evidence_benchmark.go @@ -0,0 +1,92 @@ +package commands + +import ( + "fmt" + "math" + "os" + "strings" + + ghpkg "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/outcomebench" + "github.com/spf13/cobra" +) + +// `hivectl evidence benchmark` (hivecommons/hive#14577): a Markdown report +// comparing outcomes of Hive-mediated PRs with unmediated ones, built by +// pkg/outcomebench from the same folded bundles `evidence export` reads. +// Local filesystem only, and reporting only. + +func newEvidenceBenchmarkCommand(_ *commandEnv) *cobra.Command { + var root, since, until, repo, out string + var spend float64 + cmd := &cobra.Command{ + Use: "benchmark", + Short: "Render the mediated-vs-unmediated outcome benchmark report as Markdown", + Long: "Fold the schema v1 evidence bundles under --root per PR (as `evidence export` does) and compare PRs Hive\n" + + "mediated (at least one Hive reviewer verdict or posted review) with unmediated ones: revert rate, change-failure\n" + + "rate (reverts + issue reopens), CI failure at the merged head, Hive changes_requested verdict rate (mediated only),\n" + + "median time to merge and, with --fleet-spend-usd, cost per merged mediated PR. control_window on/off is a\n" + + "secondary cut when bundles carry it. Rates carry 95% Wilson intervals; any cell with n < 10 reads \"not reported\".\n" + + "The report ends with a methodology section and the metrics schema v1 cannot answer.\n\n" + + "--since and --until take an RFC3339 timestamp or a Go duration back from now and window PRs on open time\n" + + "(since <= opened_at < until). Use absolute timestamps for a committed report: the output depends only on the\n" + + "bundles and the flags, so the same root and window reproduce the same bytes.\n" + + "Reads the local filesystem only; no network, no hive session, and nothing here affects merge eligibility.", + Example: " hivectl evidence benchmark --since 2026-10-01T00:00:00Z --until 2026-11-01T00:00:00Z \\\n" + + " --out src/docs/benchmarks/outcomes-2026-10.md\n" + + " hivectl evidence benchmark --since 720h --repo hivecommons/hive --fleet-spend-usd 412.50\n" + + " src/scripts/outcome-benchmark.sh 2026-10 # the same, for one calendar month", + Args: argsNone(), + RunE: func(cmd *cobra.Command, _ []string) error { + now := evidenceExportNow() + from, err := parseEvidenceSince(since, now) + if err != nil { + return &usageError{message: err.Error()} + } + to, err := parseEvidenceSince(until, now) + if err != nil { + return &usageError{message: strings.Replace(err.Error(), "--since", "--until", 1)} + } + if !from.IsZero() && !to.IsZero() && !to.After(from) { + return &usageError{message: "--until must be after --since"} + } + repo = strings.TrimSpace(repo) + if repo != "" && strings.Count(repo, "/") != 1 { + return &usageError{message: fmt.Sprintf("invalid --repo %q; expected owner/name", repo)} + } + opts := outcomebench.Options{Root: root, Repo: repo, Since: from, Until: to} + if cmd.Flags().Changed("fleet-spend-usd") { + if spend < 0 || math.IsNaN(spend) || math.IsInf(spend, 0) { + return &usageError{message: fmt.Sprintf("invalid --fleet-spend-usd %v; expected a non-negative amount", spend)} + } + opts.FleetSpendUSD = &spend + } + + bundles, unreadable, err := loadEvidenceBundles(root) + if err != nil { + return err + } + if unreadable > 0 { + fmt.Fprintf(cmd.ErrOrStderr(), "skipped %d unreadable or non-v1 bundle(s) under %s\n", unreadable, root) + } + report := outcomebench.Build(bundles, unreadable, opts) + md := outcomebench.Render(report) + if out == "" || out == "-" { + _, err = fmt.Fprint(cmd.OutOrStdout(), md) + return err + } + if err := os.WriteFile(out, []byte(md), 0o644); err != nil { + return fmt.Errorf("writing %s: %w", out, err) + } + fmt.Fprintf(cmd.ErrOrStderr(), "wrote benchmark over %d PR(s) to %s\n", report.PRs, out) + return nil + }, + } + cmd.Flags().StringVar(&root, "root", ghpkg.ReviewEvidenceRoot, "evidence root holding ///.json bundles") + cmd.Flags().StringVar(&since, "since", "", "only PRs opened at or after this RFC3339 time or within this Go duration (e.g. 720h)") + cmd.Flags().StringVar(&until, "until", "", "only PRs opened before this RFC3339 time or Go duration back from now") + cmd.Flags().StringVar(&repo, "repo", "", "only this owner/name repository (case-insensitive)") + cmd.Flags().StringVar(&out, "out", "", "write the Markdown report to this file instead of stdout (\"-\" is stdout)") + cmd.Flags().Float64Var(&spend, "fleet-spend-usd", 0, "fleet estimated spend over the window in USD; enables cost per merged mediated PR") + return cmd +} diff --git a/src/pkg/hivectl/commands/evidence_benchmark_test.go b/src/pkg/hivectl/commands/evidence_benchmark_test.go new file mode 100644 index 0000000000..234c68d60c --- /dev/null +++ b/src/pkg/hivectl/commands/evidence_benchmark_test.go @@ -0,0 +1,124 @@ +package commands + +import ( + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestEvidenceBenchmarkStdout(t *testing.T) { + root := t.TempDir() + writeExportFixtures(t, root) + server := httptest.NewServer(nil) + defer server.Close() + + stdout, stderr, err := execute(t, server, "", "evidence", "benchmark", "--root", root) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(stderr, "skipped 2 unreadable or non-v1 bundle(s)") { + t.Fatalf("stderr = %q", stderr) + } + for _, want := range []string{ + "# Outcome benchmark: Hive-mediated vs unmediated pull requests", + "- **PRs in window:** 3 (1 mediated, 2 unmediated;", + "- **Files skipped as unreadable or not schema v1:** 2", + "| Revert rate (merged PRs) | not reported (n=1) | not reported (n=0) |", + "| Cost per merged PR | not available (no fleet spend supplied) |", + "## Methodology", + } { + if !strings.Contains(stdout, want) { + t.Errorf("stdout missing %q\n%s", want, stdout) + } + } + dash, _, err := execute(t, server, "", "evidence", "benchmark", "--root", root, "--out", "-", "--repo", "o/r") + if err != nil || !strings.Contains(dash, "- **Repository:** o/r") || !strings.Contains(dash, "- **PRs in window:** 2 ") { + t.Fatalf("--out - / --repo: err %v\n%s", err, dash) + } +} + +func TestEvidenceBenchmarkOutReproducible(t *testing.T) { + root := t.TempDir() + writeExportFixtures(t, root) + server := httptest.NewServer(nil) + defer server.Close() + dir := t.TempDir() + args := func(out string) []string { + return []string{"evidence", "benchmark", "--root", root, + "--since", "2026-10-01T00:00:00Z", "--until", "2026-11-01T00:00:00Z", + "--fleet-spend-usd", "42.5", "--out", out} + } + a, b := filepath.Join(dir, "a.md"), filepath.Join(dir, "b.md") + _, stderr, err := execute(t, server, "", args(a)...) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(stderr, "wrote benchmark over 3 PR(s) to "+a) { + t.Fatalf("stderr = %q", stderr) + } + if _, _, err := execute(t, server, "", args(b)...); err != nil { + t.Fatal(err) + } + first, err := os.ReadFile(a) + if err != nil { + t.Fatal(err) + } + second, err := os.ReadFile(b) + if err != nil { + t.Fatal(err) + } + if string(first) != string(second) { + t.Fatal("re-running on the same evidence root must reproduce the same report") + } + md := string(first) + for _, want := range []string{ + "- **Window:** PRs opened from 2026-10-01T00:00:00Z to 2026-11-01T00:00:00Z (exclusive)", + "| Cost per merged PR | not reported (n=1) |", + } { + if !strings.Contains(md, want) { + t.Errorf("report missing %q\n%s", want, md) + } + } +} + +func TestEvidenceBenchmarkMissingRoot(t *testing.T) { + server := httptest.NewServer(nil) + defer server.Close() + stdout, _, err := execute(t, server, "", "evidence", "benchmark", "--root", filepath.Join(t.TempDir(), "absent")) + if err != nil || !strings.Contains(stdout, "- **PRs in window:** 0 ") { + t.Fatalf("err %v\n%s", err, stdout) + } +} + +func TestEvidenceBenchmarkUsageErrors(t *testing.T) { + root := t.TempDir() + server := httptest.NewServer(nil) + defer server.Close() + for _, tc := range []struct { + name string + args []string + exit int + want string + }{ + {"bad since", []string{"--since", "yesterday"}, ExitUsage, "invalid --since"}, + {"bad until", []string{"--until", "tomorrow"}, ExitUsage, "invalid --until"}, + {"until before since", []string{"--since", "2026-10-02T00:00:00Z", "--until", "2026-10-01T00:00:00Z"}, ExitUsage, "--until must be after --since"}, + {"bad repo", []string{"--repo", "hive"}, ExitUsage, "invalid --repo"}, + {"negative spend", []string{"--fleet-spend-usd", "-1"}, ExitUsage, "invalid --fleet-spend-usd"}, + {"nan spend", []string{"--fleet-spend-usd", "NaN"}, ExitUsage, "invalid --fleet-spend-usd"}, + {"positional", []string{"extra"}, ExitUsage, ""}, + {"unwritable out", []string{"--out", filepath.Join(root, "missing", "x.md")}, ExitFailure, "writing"}, + } { + t.Run(tc.name, func(t *testing.T) { + _, _, err := execute(t, server, "", append([]string{"evidence", "benchmark", "--root", root}, tc.args...)...) + if ExitCode(err) != tc.exit { + t.Fatalf("exit %d (err %v), want %d", ExitCode(err), err, tc.exit) + } + if tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)) { + t.Fatalf("err = %v, want %q", err, tc.want) + } + }) + } +} diff --git a/src/pkg/outcomebench/outcomebench.go b/src/pkg/outcomebench/outcomebench.go new file mode 100644 index 0000000000..e13198de57 --- /dev/null +++ b/src/pkg/outcomebench/outcomebench.go @@ -0,0 +1,320 @@ +// Package outcomebench compares the outcomes of pull requests Hive mediated +// with those merged without it, from the per-PR review evidence bundles +// (hivecommons/hive#14577). Everything here is a pure function of the bundles +// and the options: no clock, no filesystem, no network, so re-running it over +// the same evidence root reproduces the same table byte for byte. +// +// A PR is mediated when at least one record of it carries a Hive reviewer +// verdict or a review Hive posted; it is unmediated otherwise. The +// prospectively assigned control_window arm (on/off), when bundles carry one, +// is a secondary cut. Rates come with 95% Wilson intervals and any cell whose +// sample is smaller than MinN reads "not reported": no claim without n. +// +// It is reporting only; nothing here feeds a merge decision. +package outcomebench + +import ( + "math" + "sort" + "strings" + "time" + + "github.com/hivecommons/hive/pkg/evidence" +) + +// MinN is the smallest sample a cell is reported for. Below it the cell +// reads "not reported" together with its n. +const MinN = 10 + +// VerdictChangesRequested is the Hive reviewer verdict counted by the +// changes-requested rate (review.VerdictChangesRequested). +const VerdictChangesRequested = "changes_requested" + +// CI statuses read at the merged head. Pending or empty is "not captured". +const ( + CIStatusSuccess = "success" + CIStatusFailure = "failure" +) + +// Cohort labels. +const ( + CohortMediated = "mediated" + CohortUnmediated = "unmediated" +) + +// Options selects the population and carries the optional cost input. +type Options struct { + // Root is the evidence root the bundles came from; a label only. + Root string + // Repo narrows to one owner/name (case-insensitive); empty = all. + Repo string + // Since and Until window PRs on open time: Since <= opened_at < Until. + // A zero bound is open. + Since time.Time + Until time.Time + // FleetSpendUSD is the fleet's estimated spend over the window. Nil + // means not supplied, and cost per merged PR is listed as not available. + FleetSpendUSD *float64 +} + +// PRRow is one folded PR classified for the benchmark. +type PRRow struct { + Key string + Mediated bool + ControlWindow string + Backfilled bool + OutcomeKnown bool + Closed bool + // Merge is the folded merge event; nil when the PR did not merge. + Merge *evidence.MergeEvent + // Reverted and ChangeFailure follow evidence.PR.ChangeFailure: a revert + // link, or (for ChangeFailure) also an issue reopen within LinkWindow. + Reverted bool + ChangeFailure bool + // CIAtMerge is the CI status of the merged head's bundle: success, + // failure, or empty when not captured. + CIAtMerge string + // ChangesRequested is true when any record carries a Hive + // changes_requested verdict. + ChangesRequested bool +} + +// Merged reports whether the PR merged. +func (r PRRow) Merged() bool { return r.Merge != nil } + +// Classify folds bundles per PR with evidence.FoldPRs (the same fold the +// summary API and `hivectl evidence export` use), keeps the PRs opened inside +// the window, and classifies each. Rows are ordered by PR key. +func Classify(bundles []*evidence.Bundle, opts Options) []PRRow { + prs := evidence.FoldPRs(bundles, opts.Repo, "") + byKey := map[string][]*evidence.Bundle{} + for _, b := range bundles { + if b == nil || (opts.Repo != "" && !strings.EqualFold(b.Repo, opts.Repo)) { + continue + } + k := evidence.PRKey(b.Repo, b.Number) + byKey[k] = append(byKey[k], b) + } + rows := make([]PRRow, 0, len(prs)) + for key, p := range prs { + if !opts.Since.IsZero() && p.OpenedAt.Before(opts.Since) { + continue + } + if !opts.Until.IsZero() && !p.OpenedAt.Before(opts.Until) { + continue + } + rows = append(rows, classifyPR(key, p, byKey[key])) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].Key < rows[j].Key }) + return rows +} + +func classifyPR(key string, p *evidence.PR, bundles []*evidence.Bundle) PRRow { + r := PRRow{ + Key: key, ControlWindow: p.ControlWindow, Backfilled: p.Backfilled, + OutcomeKnown: p.OutcomeKnown(), Closed: p.Closed, Merge: p.Merge, + } + for _, b := range bundles { + if len(b.Verdicts) > 0 || len(b.PostedReviews) > 0 { + r.Mediated = true + } + for _, v := range b.Verdicts { + if strings.EqualFold(strings.TrimSpace(v.Verdict), VerdictChangesRequested) { + r.ChangesRequested = true + } + } + } + if p.Merge == nil { + return r + } + reverted, reopened := p.ChangeFailure() + r.Reverted, r.ChangeFailure = reverted, reverted || reopened + switch s := mergedHeadBundle(p, bundles).CI.Status; s { + case CIStatusSuccess, CIStatusFailure: + r.CIAtMerge = s + } + return r +} + +// mergedHeadBundle is the newest record whose head is the merged SHA, or the +// PR's newest record when none matches (a squash merge records the merge +// commit, and the newest reviewed head is the one that merged). +func mergedHeadBundle(p *evidence.PR, bundles []*evidence.Bundle) *evidence.Bundle { + var best *evidence.Bundle + for _, b := range bundles { + if !evidence.SHAMatches(p.Merge.SHA, b.HeadSHA) { + continue + } + if best == nil || b.UpdatedAt.After(best.UpdatedAt) { + best = b + } + } + if best == nil { + return p.Bundle + } + return best +} + +// Wilson returns the 95% Wilson score interval for k successes in n trials, +// the same formula pkg/hub/evidence.go uses for the fleet evidence view. +func Wilson(k, n int) (lo, hi float64) { + if n <= 0 { + return 0, 0 + } + const z = 1.96 + nf, p := float64(n), float64(k)/float64(n) + d := 1 + z*z/nf + centre := (p + z*z/(2*nf)) / d + half := z * math.Sqrt(p*(1-p)/nf+z*z/(4*nf*nf)) / d + return math.Max(0, centre-half), math.Min(1, centre+half) +} + +// Rate is K events out of N eligible PRs. +type Rate struct { + K, N int +} + +// Reported reports whether N reaches MinN. +func (r Rate) Reported() bool { return r.N >= MinN } + +// Value is K/N, or 0 when N is 0. +func (r Rate) Value() float64 { + if r.N <= 0 { + return 0 + } + return float64(r.K) / float64(r.N) +} + +// Interval is the 95% Wilson interval of the rate. +func (r Rate) Interval() (lo, hi float64) { return Wilson(r.K, r.N) } + +// Median is a median duration over N measurable PRs. +type Median struct { + D time.Duration + N int +} + +// Reported reports whether N reaches MinN. +func (m Median) Reported() bool { return m.N >= MinN } + +// CohortStats is one column of the report. +type CohortStats struct { + Label string + // Population: PRs in the cohort by outcome. Unknown counts PRs whose + // records do not say what happened (evidence.PR.OutcomeKnown). + PRs, Merged, Closed, Open, Unknown, Backfilled int + // Revert, ChangeFailure and CIFailure are over merged PRs (CIFailure + // over merged PRs whose merged-head CI status was captured). + Revert, ChangeFailure, CIFailure Rate + // ChangesRequested is over the cohort's mediated PRs only. + ChangesRequested Rate + // MergedMediated counts merged mediated PRs, the cost denominator. + MergedMediated int + TimeToMerge Median +} + +// Aggregate computes one cohort's statistics over rows. +func Aggregate(label string, rows []PRRow) CohortStats { + s := CohortStats{Label: label, PRs: len(rows)} + var merged []*evidence.Bundle + for _, r := range rows { + if r.Backfilled { + s.Backfilled++ + } + if r.Mediated { + s.ChangesRequested.N++ + if r.ChangesRequested { + s.ChangesRequested.K++ + } + } + switch { + case r.Merged(): + case !r.OutcomeKnown: + s.Unknown++ + continue + case r.Closed: + s.Closed++ + continue + default: + s.Open++ + continue + } + s.Merged++ + if r.Mediated { + s.MergedMediated++ + } + merged = append(merged, &evidence.Bundle{Merge: r.Merge}) + s.Revert.N++ + s.ChangeFailure.N++ + if r.Reverted { + s.Revert.K++ + } + if r.ChangeFailure { + s.ChangeFailure.K++ + } + if r.CIAtMerge != "" { + s.CIFailure.N++ + if r.CIAtMerge == CIStatusFailure { + s.CIFailure.K++ + } + } + } + s.TimeToMerge.D, s.TimeToMerge.N = evidence.MedianTimeToMerge(merged) + return s +} + +// Report is the full benchmark: the two cohorts, the optional control-window +// cut and the inputs that produced them. +type Report struct { + Options Options + Unreadable int + PRs int + Mediated CohortStats + Unmediated CohortStats + // HasControlWindow is true when any PR in the window carries an on or + // off assignment; ControlOn and ControlOff are then the secondary cut. + HasControlWindow bool + ControlOn CohortStats + ControlOff CohortStats +} + +// Build classifies bundles and aggregates the report. unreadable is the +// count of files the caller skipped, echoed in the report header. +func Build(bundles []*evidence.Bundle, unreadable int, opts Options) Report { + rows := Classify(bundles, opts) + var med, unmed, on, off []PRRow + for _, r := range rows { + if r.Mediated { + med = append(med, r) + } else { + unmed = append(unmed, r) + } + switch r.ControlWindow { + case evidence.ControlWindowOn: + on = append(on, r) + case evidence.ControlWindowOff: + off = append(off, r) + } + } + return Report{ + Options: opts, Unreadable: unreadable, PRs: len(rows), + Mediated: Aggregate(CohortMediated, med), Unmediated: Aggregate(CohortUnmediated, unmed), + HasControlWindow: len(on)+len(off) > 0, + ControlOn: Aggregate("control_window=on", on), + ControlOff: Aggregate("control_window=off", off), + } +} + +// CostPerMergedPR is the fleet spend divided by merged mediated PRs. ok is +// false when no spend was supplied; reported is false when the denominator +// is below MinN. +func (r Report) CostPerMergedPR() (usd float64, ok, reported bool) { + if r.Options.FleetSpendUSD == nil { + return 0, false, false + } + n := r.Mediated.MergedMediated + if n < MinN { + return 0, true, false + } + return *r.Options.FleetSpendUSD / float64(n), true, true +} diff --git a/src/pkg/outcomebench/outcomebench_test.go b/src/pkg/outcomebench/outcomebench_test.go new file mode 100644 index 0000000000..6854201b35 --- /dev/null +++ b/src/pkg/outcomebench/outcomebench_test.go @@ -0,0 +1,256 @@ +package outcomebench + +import ( + "math" + "strconv" + "strings" + "testing" + "time" + + "github.com/hivecommons/hive/pkg/evidence" +) + +var t0 = time.Date(2026, 10, 1, 12, 0, 0, 0, time.UTC) + +// bundle returns a minimal live bundle for repo o/r opened at t0+offset. +func bundle(number int, head string, offset time.Duration) *evidence.Bundle { + at := t0.Add(offset) + return &evidence.Bundle{ + SchemaVersion: evidence.SchemaVersion, ID: evidence.BundleID("o/r", number, head), + Repo: "o/r", Number: number, Author: evidence.Author{Login: "a", Kind: evidence.AuthorAgent}, + BaseSHA: "base", HeadSHA: head, CreatedAt: at, UpdatedAt: at, + } +} + +func verdict(v string) evidence.Verdict { + return evidence.Verdict{Perspective: "p", Model: "m", Backend: "b", Verdict: v, Confidence: 0.9, RecordedAt: t0} +} + +func merged(b *evidence.Bundle, ttm time.Duration) *evidence.Bundle { + b.Merge = &evidence.MergeEvent{Actor: "hive", Method: "squash", At: b.CreatedAt.Add(ttm), SHA: b.HeadSHA, OpenedAt: b.CreatedAt} + return b +} + +func TestClassifyCohorts(t *testing.T) { + // #1: two heads, verdict only on the older one — still mediated. + old := bundle(1, "aaaaaaa1", 0) + old.Verdicts = []evidence.Verdict{verdict("Changes_Requested")} + newer := merged(bundle(1, "bbbbbbb1", time.Hour), 2*time.Hour) + newer.CI.Status = CIStatusFailure + // #2: posted review only — mediated, no changes requested. + b2 := bundle(2, "ccccccc2", 0) + b2.PostedReviews = []evidence.PostedReview{{URL: "u", Head: "ccccccc2", At: t0}} + // #3: nothing — unmediated and reverted; merge.sha is a merge commit no + // head matches, so CI is read from the newest record (pending). + b3 := merged(bundle(3, "ddddddd3", 0), time.Hour) + b3.Merge.SHA = "9999999abc" + b3.RevertedBy = []string{"o/r#9@x"} + b3.CI.Status = "pending" + // #4: unmediated, merged, reopened issue inside the link window, CI + // captured on a different (older) head that matches merge.sha. + b4old := bundle(4, "eeeeeee4", 0) + b4old.CI.Status = CIStatusSuccess + b4 := bundle(4, "fffffff4", time.Hour) + b4.CI.Status = CIStatusFailure + b4.Merge = &evidence.MergeEvent{Actor: "x", Method: "merge", At: t0.Add(3 * time.Hour), SHA: "eeeeeee4"} + b4.IssueReopens = []evidence.IssueReopen{{Issue: 5, At: t0.Add(4 * time.Hour), Source: "s"}} + // #5: other repo, filtered out by Repo. + b5 := bundle(5, "1111111", 0) + b5.Repo, b5.ID = "x/y", evidence.BundleID("x/y", 5, "1111111") + // #6: opened before Since. + b6 := bundle(6, "2222222", -48*time.Hour) + + rows := Classify([]*evidence.Bundle{newer, old, b2, nil, b3, b4old, b4, b5, b6}, Options{Repo: "O/R", Since: t0.Add(-time.Hour), Until: t0.Add(24 * time.Hour)}) + if len(rows) != 4 { + t.Fatalf("rows = %d, want 4: %+v", len(rows), rows) + } + want := []struct { + key string + mediated, cr, rev, cfail bool + ci string + }{ + {"o/r#1", true, true, false, false, CIStatusFailure}, + {"o/r#2", true, false, false, false, ""}, + {"o/r#3", false, false, true, true, ""}, + {"o/r#4", false, false, false, true, CIStatusSuccess}, + } + for i, w := range want { + r := rows[i] + if r.Key != w.key || r.Mediated != w.mediated || r.ChangesRequested != w.cr || r.Reverted != w.rev || r.ChangeFailure != w.cfail || r.CIAtMerge != w.ci { + t.Errorf("row %d = %+v, want %+v", i, r, w) + } + } + if rows[1].Merged() || !rows[0].Merged() { + t.Fatalf("merged flags wrong: %+v", rows) + } + + // Until is exclusive. + if got := Classify([]*evidence.Bundle{b2}, Options{Until: t0}); len(got) != 0 { + t.Fatalf("until must be exclusive, got %+v", got) + } +} + +func TestWilson(t *testing.T) { + if lo, hi := Wilson(0, 0); lo != 0 || hi != 0 { + t.Fatalf("n=0: %v %v", lo, hi) + } + lo, hi := Wilson(5, 10) + if math.Abs(lo-0.2366) > 1e-3 || math.Abs(hi-0.7634) > 1e-3 { + t.Fatalf("5/10: %v %v", lo, hi) + } + if lo, hi := Wilson(0, 20); lo > 1e-9 || math.Abs(hi-0.1611) > 1e-3 { + t.Fatalf("0/20: %v %v", lo, hi) + } + if lo, hi := Wilson(20, 20); hi < 1-1e-9 || hi > 1 || math.Abs(lo-0.8389) > 1e-3 { + t.Fatalf("20/20: %v %v", lo, hi) + } + if (Rate{}).Value() != 0 || (Rate{K: 1, N: 4}).Value() != 0.25 { + t.Fatal("Rate.Value") + } +} + +func TestSuppressionBelowMinN(t *testing.T) { + if got := FormatRate(Rate{K: 1, N: MinN - 1}); got != "not reported (n=9)" { + t.Fatalf("got %q", got) + } + if got := FormatRate(Rate{K: 5, N: MinN}); got != "50.0% (95% CI 23.7–76.3%; 5/10)" { + t.Fatalf("got %q", got) + } + if got := FormatMedian(Median{D: time.Hour, N: 3}); got != "not reported (n=3)" { + t.Fatalf("got %q", got) + } + if got := FormatMedian(Median{D: 90 * time.Minute, N: MinN}); got != "1.5h (n=10)" { + t.Fatalf("got %q", got) + } +} + +func TestAggregateMedianTTMAndOutcomes(t *testing.T) { + var rows []PRRow + // 11 merged: TTMs 1h..11h → median 6h. One lacks opened_at and is skipped. + for i := 1; i <= 11; i++ { + m := &evidence.MergeEvent{At: t0.Add(time.Duration(i) * time.Hour), OpenedAt: t0} + rows = append(rows, PRRow{Mediated: true, Merge: m, OutcomeKnown: true, CIAtMerge: CIStatusSuccess}) + } + rows = append(rows, + PRRow{Mediated: true, Merge: &evidence.MergeEvent{At: t0}, OutcomeKnown: true, Reverted: true, ChangeFailure: true, CIAtMerge: CIStatusFailure, ChangesRequested: true, Backfilled: true}, + PRRow{Closed: true, OutcomeKnown: true}, + PRRow{OutcomeKnown: true}, + PRRow{}, + ) + s := Aggregate("x", rows) + if s.PRs != 15 || s.Merged != 12 || s.Closed != 1 || s.Open != 1 || s.Unknown != 1 || s.Backfilled != 1 || s.MergedMediated != 12 { + t.Fatalf("population = %+v", s) + } + if s.TimeToMerge.N != 11 || s.TimeToMerge.D != 6*time.Hour { + t.Fatalf("median = %+v", s.TimeToMerge) + } + if s.Revert != (Rate{1, 12}) || s.ChangeFailure != (Rate{1, 12}) || s.CIFailure != (Rate{1, 12}) || s.ChangesRequested != (Rate{1, 12}) { + t.Fatalf("rates = %+v", s) + } + // Even count: mean of the middle two. + even := Aggregate("e", rows[:10]) + if even.TimeToMerge.D != 5*time.Hour+30*time.Minute { + t.Fatalf("even median = %v", even.TimeToMerge.D) + } +} + +// fleet builds n merged PRs (mediated when asked) from number base, every third +// one requesting changes and reverted, plus the given control window. +func fleet(base, n int, mediated bool, arm string) []*evidence.Bundle { + var out []*evidence.Bundle + for i := range n { + num := base + i + b := merged(bundle(num, "head000"+strconv.Itoa(num), time.Duration(i)*time.Minute), time.Duration(i+1)*time.Hour) + b.CI.Status = CIStatusSuccess + b.ControlWindow = arm + if mediated { + b.Verdicts = []evidence.Verdict{verdict("approve")} + } + if i%3 == 0 { + b.RevertedBy = []string{"o/r#1@x"} + if mediated { + b.Verdicts = append(b.Verdicts, verdict(VerdictChangesRequested)) + } + } + out = append(out, b) + } + return out +} + +func TestBuildAndRender(t *testing.T) { + bundles := append(fleet(100, 12, true, evidence.ControlWindowOn), fleet(200, 12, false, evidence.ControlWindowOff)...) + bundles = append(bundles, fleet(300, 3, false, "")...) + spend := 120.0 + opts := Options{Root: "/data/evidence", Since: t0.Add(-time.Hour), Until: t0.Add(30 * 24 * time.Hour), FleetSpendUSD: &spend} + r := Build(bundles, 2, opts) + if r.PRs != 27 || r.Mediated.PRs != 12 || r.Unmediated.PRs != 15 || !r.HasControlWindow || r.ControlOn.PRs != 12 || r.ControlOff.PRs != 12 { + t.Fatalf("report = %+v", r) + } + if usd, ok, rep := r.CostPerMergedPR(); !ok || !rep || usd != 10 { + t.Fatalf("cost = %v %v %v", usd, ok, rep) + } + md := Render(r) + for _, want := range []string{ + "# Outcome benchmark: Hive-mediated vs unmediated pull requests", + "- **Evidence root:** `/data/evidence`", + "- **Repository:** all repositories", + "- **Window:** PRs opened from 2026-10-01T11:00:00Z to 2026-10-31T12:00:00Z (exclusive)", + "- **PRs in window:** 27 (12 mediated, 15 unmediated; 0 backfilled from forge history)", + "- **Files skipped as unreadable or not schema v1:** 2", + "| Metric | mediated | unmediated |", + "| Revert rate (merged PRs) | 33.3% (95% CI ", + "; 4/12) | 33.3% (95% CI ", + "; 5/15) |", + "| Hive `changes_requested` verdict rate (mediated PRs) | 33.3% (95% CI ", + "n/a (no Hive verdicts by definition) |", + "| Median time to merge | 6.5h (n=12) | ", + "| Cost per merged PR | $10.00 ($120.00 ÷ 12 merged mediated PRs) | n/a (no Hive spend by definition) |", + "| Metric | control_window=on | control_window=off |", + "## Not available in evidence schema v1", + "Post-merge CI re-runs", + "Human `changes_requested` reviews", + "## Methodology", + "n < 10 reads `not reported`", + } { + if !strings.Contains(md, want) { + t.Errorf("render missing %q\n%s", want, md) + } + } + if strings.Contains(md, "--fleet-spend-usd") { + t.Error("spend supplied, yet listed as not available") + } + if again := Render(Build(bundles, 2, opts)); again != md { + t.Fatal("render is not deterministic") + } +} + +func TestRenderSmallAndEmpty(t *testing.T) { + // Too few PRs: every rate cell is suppressed; no control window; no spend. + r := Build(fleet(1, 3, true, ""), 0, Options{Repo: "o/r"}) + md := Render(r) + for _, want := range []string{ + "`(not recorded)`", + "- **Repository:** o/r", + "from the first record to the last record", + "| Revert rate (merged PRs) | not reported (n=3) | not reported (n=0) |", + "| Cost per merged PR | not available (no fleet spend supplied) |", + "No PR in the window carries a `control_window` assignment", + "`--fleet-spend-usd`", + } { + if !strings.Contains(md, want) { + t.Errorf("render missing %q\n%s", want, md) + } + } + if strings.Contains(md, "95% CI") { + t.Errorf("suppressed report must not carry intervals:\n%s", md) + } + + spend := 5.0 + r = Build(fleet(1, 3, true, ""), 0, Options{FleetSpendUSD: &spend}) + if _, ok, rep := r.CostPerMergedPR(); !ok || rep { + t.Fatal("cost with n<10 must be supplied but not reported") + } + if md := Render(r); !strings.Contains(md, "| Cost per merged PR | not reported (n=3) |") { + t.Errorf("cost cell:\n%s", md) + } +} diff --git a/src/pkg/outcomebench/render.go b/src/pkg/outcomebench/render.go new file mode 100644 index 0000000000..aabf4b4c85 --- /dev/null +++ b/src/pkg/outcomebench/render.go @@ -0,0 +1,144 @@ +package outcomebench + +import ( + "fmt" + "strings" + "time" +) + +// NotReported is the text of a cell whose sample is below MinN. +const NotReported = "not reported" + +// FormatRate renders a rate as "12.3% (95% CI 8.1–17.9%; 12/97)", or +// "not reported (n=4)" when N is below MinN. +func FormatRate(r Rate) string { + if !r.Reported() { + return fmt.Sprintf("%s (n=%d)", NotReported, r.N) + } + lo, hi := r.Interval() + return fmt.Sprintf("%.1f%% (95%% CI %.1f–%.1f%%; %d/%d)", 100*r.Value(), 100*lo, 100*hi, r.K, r.N) +} + +// FormatMedian renders a median time to merge in hours, or "not reported" +// when N is below MinN. +func FormatMedian(m Median) string { + if !m.Reported() { + return fmt.Sprintf("%s (n=%d)", NotReported, m.N) + } + return fmt.Sprintf("%.1fh (n=%d)", m.D.Hours(), m.N) +} + +func formatBound(t time.Time, open string) string { + if t.IsZero() { + return open + } + return t.UTC().Format(time.RFC3339) +} + +func population(s CohortStats) string { + return fmt.Sprintf("%d (%d merged · %d closed · %d open · %d unknown)", s.PRs, s.Merged, s.Closed, s.Open, s.Unknown) +} + +// table renders the metric rows for the given columns. mediatedOnly marks +// columns whose changes-requested cell is n/a by definition. +func table(b *strings.Builder, cols []CohortStats, mediatedOnly []bool, cost []string) { + b.WriteString("| Metric |") + for _, c := range cols { + fmt.Fprintf(b, " %s |", c.Label) + } + b.WriteString("\n|---|") + for range cols { + b.WriteString("---|") + } + b.WriteString("\n") + row := func(name string, cell func(i int, c CohortStats) string) { + fmt.Fprintf(b, "| %s |", name) + for i, c := range cols { + fmt.Fprintf(b, " %s |", cell(i, c)) + } + b.WriteString("\n") + } + row("PRs in window", func(_ int, c CohortStats) string { return population(c) }) + row("Revert rate (merged PRs)", func(_ int, c CohortStats) string { return FormatRate(c.Revert) }) + row("Change-failure rate (reverts + issue reopens, merged PRs)", func(_ int, c CohortStats) string { return FormatRate(c.ChangeFailure) }) + row("CI failure at merged head", func(_ int, c CohortStats) string { return FormatRate(c.CIFailure) }) + row("Hive `changes_requested` verdict rate (mediated PRs)", func(i int, c CohortStats) string { + if mediatedOnly[i] { + return "n/a (no Hive verdicts by definition)" + } + return FormatRate(c.ChangesRequested) + }) + row("Median time to merge", func(_ int, c CohortStats) string { return FormatMedian(c.TimeToMerge) }) + if cost != nil { + row("Cost per merged PR", func(i int, _ CohortStats) string { return cost[i] }) + } +} + +// Render returns the report as Markdown: header, headline table, the +// control-window cut when present, what schema v1 cannot answer, and the +// methodology. The output depends only on the report, so the same evidence +// and options always render the same bytes. +func Render(r Report) string { + var b strings.Builder + o := r.Options + b.WriteString("# Outcome benchmark: Hive-mediated vs unmediated pull requests\n\n") + b.WriteString("\n\n") + root := o.Root + if root == "" { + root = "(not recorded)" + } + repo := o.Repo + if repo == "" { + repo = "all repositories" + } + fmt.Fprintf(&b, "- **Evidence root:** `%s` (schema %s bundles)\n", root, "v1") + fmt.Fprintf(&b, "- **Repository:** %s\n", repo) + fmt.Fprintf(&b, "- **Window:** PRs opened from %s to %s (exclusive)\n", + formatBound(o.Since, "the first record"), formatBound(o.Until, "the last record")) + fmt.Fprintf(&b, "- **PRs in window:** %d (%d mediated, %d unmediated; %d backfilled from forge history)\n", + r.PRs, r.Mediated.PRs, r.Unmediated.PRs, r.Mediated.Backfilled+r.Unmediated.Backfilled) + fmt.Fprintf(&b, "- **Files skipped as unreadable or not schema v1:** %d\n", r.Unreadable) + fmt.Fprintf(&b, "- **Minimum n per cell:** %d (smaller cells read `%s`)\n\n", MinN, NotReported) + + b.WriteString("## Headline\n\n") + cost := []string{"", "n/a (no Hive spend by definition)"} + switch usd, ok, reported := r.CostPerMergedPR(); { + case !ok: + cost[0] = "not available (no fleet spend supplied)" + case !reported: + cost[0] = fmt.Sprintf("%s (n=%d)", NotReported, r.Mediated.MergedMediated) + default: + cost[0] = fmt.Sprintf("$%.2f ($%.2f ÷ %d merged mediated PRs)", usd, *o.FleetSpendUSD, r.Mediated.MergedMediated) + } + table(&b, []CohortStats{r.Mediated, r.Unmediated}, []bool{false, true}, cost) + + b.WriteString("\n## Secondary cut: control window\n\n") + if r.HasControlWindow { + b.WriteString("PRs carrying a prospective `control_window` assignment, mediated and unmediated together. PRs without an assignment (or with conflicting ones across heads) are in neither column.\n\n") + table(&b, []CohortStats{r.ControlOn, r.ControlOff}, []bool{false, false}, nil) + } else { + b.WriteString("No PR in the window carries a `control_window` assignment, so this cut is omitted.\n") + } + + b.WriteString("\n## Not available in evidence schema v1\n\n") + b.WriteString("These are listed rather than approximated; adding them needs new telemetry, which this report does not introduce.\n\n") + b.WriteString("- **Post-merge CI re-runs:** a bundle captures CI at the reviewed head only; later runs on the base branch are not recorded.\n") + b.WriteString("- **Human `changes_requested` reviews:** `human_actions` records approvals, labels and hold lifts, not human review verdicts, so a human changes-requested rate cannot be computed for either cohort.\n") + if o.FleetSpendUSD == nil { + b.WriteString("- **Cost per merged PR:** spend is not part of a bundle; supply the fleet's estimated spend for the window (`--fleet-spend-usd`) to report it.\n") + } + + b.WriteString("\n## Methodology\n\n") + b.WriteString("- **Population.** Every schema v1 bundle under the evidence root is folded per PR with `evidence.FoldPRs`, the same fold `GET /api/evidence/summary` and `hivectl evidence export` use, and a PR is placed in the window by its open time (earliest `created_at` or `merge.opened_at`), never by when a bundle was written. Nothing is excluded silently: backfilled PRs are counted (and flagged above), files that do not parse are counted as skipped.\n") + b.WriteString("- **Cohorts.** A PR is *mediated* when at least one of its records carries a Hive reviewer verdict (`verdicts[]`) or a review Hive posted (`posted_reviews[]`); otherwise it is *unmediated*. Backfilled bundles never carry verdicts, so PRs known only from forge history are unmediated. This is an observational split, not a randomised one: the two cohorts can differ in author, repository and size, and differences between them are not causal estimates. The `control_window` cut, when present, is the prospectively assigned arm.\n") + b.WriteString("- **Outcome.** Merged when any record holds a merge event; closed when a backfilled or forge-observed outcome says closed without merge; open when an outcome is known and neither; unknown when no record says (a live bundle for a PR this hive did not merge). Rates over merged PRs leave open, closed and unknown PRs out of the denominator.\n") + b.WriteString("- **Revert rate.** Merged PRs with a `reverted_by` link (a revert merged within the link window) ÷ merged PRs.\n") + b.WriteString("- **Change-failure rate.** Hive's existing definition (`evidence.PR.ChangeFailure`): merged PRs reverted, or whose closed issue a person reopened within the link window of the merge, ÷ merged PRs.\n") + b.WriteString("- **CI failure at merged head.** Among merged PRs whose merged-head bundle captured a CI verdict (`ci.status` success or failure; pending and missing are left out), the share that was failure. The merged-head bundle is the record whose `head_sha` matches `merge.sha`, else the PR's newest record.\n") + b.WriteString("- **Hive `changes_requested` verdict rate.** Mediated PRs with at least one `changes_requested` reviewer verdict on any head ÷ mediated PRs. Unmediated PRs have no Hive verdicts by definition.\n") + b.WriteString("- **Median time to merge.** `merge.at − merge.opened_at` over merged PRs that captured `opened_at` (`evidence.MedianTimeToMerge`; an even count takes the mean of the middle two). PRs without it are skipped, never counted as zero; n is the measurable count.\n") + b.WriteString("- **Cost per merged PR.** Optional input: the fleet's estimated spend over the window ÷ merged mediated PRs.\n") + fmt.Fprintf(&b, "- **Intervals and suppression.** Rates carry 95%% Wilson score intervals (z = 1.96, the formula in `pkg/hub/evidence.go`). Any cell with n < %d reads `%s` with its n; no claim is made without n.\n", MinN, NotReported) + b.WriteString("- **Reproducibility.** The report is a pure function of the bundles and the flags (no wall clock, ordered output), so re-running `src/scripts/outcome-benchmark.sh` with the same month on the same evidence root reproduces the same table byte for byte.\n") + return b.String() +} diff --git a/src/scripts/outcome-benchmark.sh b/src/scripts/outcome-benchmark.sh new file mode 100755 index 0000000000..c18e54f653 --- /dev/null +++ b/src/scripts/outcome-benchmark.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# outcome-benchmark.sh — render the mediated-vs-unmediated outcome benchmark +# (hivecommons/hive#14577) for one calendar month from a spoke's evidence root. +# A thin wrapper over `hivectl evidence benchmark` that pins the window to +# absolute UTC month bounds, so re-running it on the same evidence root +# reproduces the same report byte for byte. +set -euo pipefail + +script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +src_root=$(cd -- "${script_dir}/.." && pwd) + +hivectl_bin=${HIVECTL:-hivectl} +root=${EVIDENCE_ROOT:-/data/evidence} +spend=${FLEET_SPEND_USD:-} +repo="" +out="" +month="" + +usage() { + cat <<'USAGE' +Usage: src/scripts/outcome-benchmark.sh [options] YYYY-MM + +Renders src/docs/benchmarks/outcomes-YYYY-MM.md from the evidence bundles of +PRs opened in that UTC calendar month. + +Options: + --root Evidence root (default: $EVIDENCE_ROOT or /data/evidence) + --repo Only this repository + --fleet-spend-usd Fleet estimated spend over the month (default: $FLEET_SPEND_USD) + --out Output file (default: src/docs/benchmarks/outcomes-YYYY-MM.md, "-" for stdout) + -h, --help Show this help + +Environment: + HIVECTL hivectl binary to run (default: hivectl on PATH) +USAGE +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --root) + root=${2:?--root requires a value}; shift 2 ;; + --repo) + repo=${2:?--repo requires a value}; shift 2 ;; + --fleet-spend-usd) + spend=${2:?--fleet-spend-usd requires a value}; shift 2 ;; + --out) + out=${2:?--out requires a value}; shift 2 ;; + -h|--help) + usage; exit 0 ;; + -*) + echo "outcome-benchmark: unknown option $1" >&2; usage >&2; exit 2 ;; + *) + [ -z "$month" ] || { echo "outcome-benchmark: one month only" >&2; exit 2; } + month=$1; shift ;; + esac +done + +if ! [[ "$month" =~ ^([0-9]{4})-(0[1-9]|1[0-2])$ ]]; then + echo "outcome-benchmark: expected a month as YYYY-MM, got '${month}'" >&2 + usage >&2 + exit 2 +fi +year=$((10#${BASH_REMATCH[1]})) +mon=$((10#${BASH_REMATCH[2]})) +if [ "$mon" -eq 12 ]; then + next=$(printf '%04d-01' $((year + 1))) +else + next=$(printf '%04d-%02d' "$year" $((mon + 1))) +fi + +[ -n "$out" ] || out="${src_root}/docs/benchmarks/outcomes-${month}.md" + +args=(evidence benchmark --root "$root" --since "${month}-01T00:00:00Z" --until "${next}-01T00:00:00Z" --out "$out") +[ -z "$repo" ] || args+=(--repo "$repo") +[ -z "$spend" ] || args+=(--fleet-spend-usd "$spend") + +exec "$hivectl_bin" "${args[@]}"