Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-outcome-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **Outcome benchmark report:** `hivectl evidence benchmark` (and `src/scripts/outcome-benchmark.sh YYYY-MM`) renders a reproducible Markdown report from the evidence bundles comparing Hive-mediated PRs with unmediated ones — revert rate, change-failure rate, CI failure at the merged head, Hive `changes_requested` verdict rate, median time to merge and optional cost per merged PR, with 95% Wilson intervals, `not reported` below n = 10, a `control_window` on/off cut when present, and a methodology section ([#14577](https://github.com/hivecommons/hive/issues/14577)).
38 changes: 38 additions & 0 deletions src/docs/benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Outcome benchmarks

Monthly reports comparing the outcomes of pull requests Hive mediated with those merged without it
([#14577](https://github.com/hivecommons/hive/issues/14577)). Each `outcomes-YYYY-MM.md` here is generated, never
hand-edited: it is the output of `hivectl evidence benchmark` over a spoke's evidence root
(see [Review evidence bundle](../review-evidence.md#outcome-benchmark) and [hivectl](../hivectl.md)).

## Regenerating a report

On a spoke (or any host holding a copy of its `/data/evidence`), with `hivectl` on `PATH`:

```bash
src/scripts/outcome-benchmark.sh 2026-10 # writes src/docs/benchmarks/outcomes-2026-10.md
src/scripts/outcome-benchmark.sh --fleet-spend-usd 412.50 2026-10 # also report cost per merged mediated PR
src/scripts/outcome-benchmark.sh --root ./evidence-copy --out - 2026-10 # another root, Markdown on stdout
```

The script pins the window to the UTC calendar month (`--since YYYY-MM-01T00:00:00Z --until <next month>-01T00:00:00Z`)
and calls `hivectl evidence benchmark`. `HIVECTL` picks the binary, `EVIDENCE_ROOT` the root and `FLEET_SPEND_USD` the
optional spend input.

## Reproducibility

The report is a pure function of the bundles on disk and the flags: no wall clock appears in it, PRs are folded with
the same `evidence.FoldPRs` the summary API and `hivectl evidence export` use, and every aggregate is order
independent. Re-running the script for the same month on the same evidence root produces the same file byte for
byte; a diff between two runs means the evidence changed (for example a revert or an issue reopen was linked after
the first run, or a backfill added history).

## Reading a report

- **Mediated** PRs carry at least one Hive reviewer verdict or a review Hive posted; **unmediated** PRs carry
neither. The split is observational, not randomised; the `control_window` on/off cut, when bundles carry it, is
the prospectively assigned arm.
- Rates carry 95% Wilson intervals. Any cell with fewer than 10 PRs reads `not reported` with its n.
- Metrics evidence schema v1 does not capture (post-merge CI re-runs, human `changes_requested` reviews) are listed
as not available rather than approximated.
- The report's own Methodology section defines every metric.
1 change: 1 addition & 0 deletions src/docs/documentation-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ Documentation for the current Hive line (branch `v5`; the code and docs live und
- [Review swarm](review-swarm.md) — the five review perspectives, the verdict collector and its report contract, and the opt-in merge-gate integration and bounded auto-fix cycle for review findings.
- [Swarm mode](swarm.md) — focusing the hive on one repository for a bounded window: Discord announcements, theming (`swarm:` config, `HIVE_SWARM_EVENT_NAME` / `HIVE_SWARM_CALL_TO_ARMS`), the `/api/leaderboard/swarm` endpoint and scoring.
- [Review evidence bundle](review-evidence.md) — the per-PR evidence bundle schema (v1), canonical hash, Ed25519 signing, offline verification, and retention design.
- [Outcome benchmarks](benchmarks/README.md) — monthly mediated-vs-unmediated outcome reports generated by `hivectl evidence benchmark`, and how to reproduce them.
- [Evidence reporting](evidence.md) — the guarded control-arm (`control_window: on|off`) evidence report: per-repository outcome comparison, the minimum-denominator rule, and why it is reporting groundwork rather than an enabled experiment.
- [SOC 2 control mapping (not a compliance claim)](soc2-control-mapping.md) — maps common auditor change-management questions to bundle fields, with sample policy language and the gaps Hive does not cover.
- [Duplicate PR sweep](duplicate-sweep.md) — the opt-in (`duplicate_sweep.enabled`) cross-PR pass that clusters open PRs by changed-file set and suggests which one to keep: the two confidence tiers, how a survivor is chosen, why a bot regeneration series gets one summary instead of a comment per PR, and why the output is always a suggestion a human acts on and never an automated close.
Expand Down
17 changes: 17 additions & 0 deletions src/docs/hivectl.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,23 @@ Columns: `repo`, `pr`, `head_sha`, `bundle_id`, `author_login`, `author_kind`, `
Reporting only: nothing here reads into or alters merge eligibility. See
[Review evidence bundle](review-evidence.md#exporting).

### evidence — outcome benchmark report

```bash
hivectl evidence benchmark --since 2026-10-01T00:00:00Z --until 2026-11-01T00:00:00Z --out outcomes.md
hivectl evidence benchmark --since 720h --repo hivecommons/hive --fleet-spend-usd 412.50 # Markdown on stdout
src/scripts/outcome-benchmark.sh 2026-10 # one UTC month into src/docs/benchmarks/outcomes-2026-10.md
```

`evidence benchmark` folds the same bundles as `evidence export` and renders a Markdown report comparing PRs Hive
mediated (at least one Hive reviewer verdict or posted review) with unmediated ones: revert rate, change-failure rate
(reverts + issue reopens), CI failure at the merged head, Hive `changes_requested` verdict rate (mediated only), median
time to merge and, with `--fleet-spend-usd`, cost per merged mediated PR. `control_window` on/off is a secondary cut
when bundles carry it. Rates carry 95% Wilson intervals and any cell with n < 10 reads `not reported`. `--since` and
`--until` (RFC3339 or a Go duration back from now) window PRs on open time, `since <= opened_at < until`; with
absolute bounds the output depends only on the bundles, so a re-run reproduces it byte for byte. Local filesystem only,
reporting only. See [Outcome benchmark](review-evidence.md#outcome-benchmark) and [benchmarks/](benchmarks/README.md).

### evidence — backfill bundles from GitHub history

```bash
Expand Down
12 changes: 11 additions & 1 deletion src/docs/review-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ A PR's outcome (`pkg/evidence.PR.OutcomeKnown`) has three sources, folded per PR
- **Work balance (panel 6):** answers "is the queue keeping up?" from GitHub truth rather than from the hive's own audit log alone. `work_balance` in the summary has one row per ISO week of the window with `opened` (issues, split `human` / `agent` / `bot`), `closed`, `prs_opened` (same split), `prs_merged`, `prs_closed` (closed without merge), `queue_issues` / `queue_prs` (open at week end), `issue_source` and `throttled`; `work_balance_sources` names where each series came from. **Sources:** issue opens and closes come from GitHub's issues API (`state=all`, `since`), every author included, pulled by the metrics collector hourly into the durable rollup `/data/hive-metrics/issue-activity.json` (42-day initial lookback, 180-day retention; `issues_github_since` is the first date it covers, `issues_fetched_at` the last refresh); author kind is `agent` when the body carries an agent attribution trailer, `bot` for GitHub `Bot` accounts, else `human`, and an audited agent-created issue overrides the kind to `agent`. Weeks whose Monday predates `issues_github_since` read the backfilled issue series at `<evidence-root>/<owner>/<repo>/issues/<YYYY-Www>.json` when its coverage contains the requested portion of that week. Event timestamps keep partial date ranges exact; zero-event files establish observed zero counts. The collector stays authoritative for weeks whose Monday is at or after `issues_github_since`, including exact equality. Missing, malformed, or insufficiently covered series files retain the audit fallback per repository (`issue_source: "audit"`, counted in `audit_only_weeks`); mixed repository rows also disclose audit coverage. Backfilled rows use `issue_source: "github"`; `issue_series_weeks` counts the repository-weeks supplied by the series. Series events carry no issue number, so the audited agent-created override above cannot be applied to series weeks: an agent-opened issue without the attribution trailer or 🐝 footer keeps the kind the writer recorded (`bot` for a `Bot` account) in series weeks while the collector reclassifies the same issue as `agent` in rollup weeks. The audit log sees only hive-mediated writes, so the human column is a floor for audit weeks. `hivectl evidence backfill-issues` writes the series (see [CLI reference](hivectl.md#evidence--backfill-bundles-from-github-history)). When the collector starts midweek, backfill through the following Monday (exclusive `--until`) so the entire boundary week has series coverage; stopping at the collector timestamp leaves that week on audit fallback. PR opens, merges and closes come from the folded evidence bundles (live and backfilled) for PRs opened in the window, so PRs this hive never saw are missing. Week-end queue depth is the governor's last state snapshot of the week (`/data/nous/snapshots`, actionable open issues / PRs — the same numbers as the dashboard queue section, held items excluded; `queue_samples`, `queue_retention_days` 30). **Denominator policy:** issue ratio = `closed` ÷ `opened` and PR ratio = `prs_merged` ÷ `prs_opened` over the whole window; nothing is dropped, PRs closed without merge are drawn as a tick but never counted as merged, and an unmatched close (an issue opened before the window) still counts as closed because the question is throughput, not per-issue lifecycle. **Grading** (`EV_THROUGHPUT_*`, `EV_QUEUE_GROWTH_MAX`): issues and PRs are graded separately and the card takes the lower — ratio ≥ 0.95 Excellent, ≥ 0.85 Good, ≥ 0.70 Fair, below Weak, and any grade is Weak when the week-end queue grew more than 10% from the previous week. Raw volume is never graded: a hive that opens 400 issues a week and closes 400 is Excellent. The Evidence sub-panel draws two charts (issues, PRs): opened stacked by kind, a closed / merged line, a closed-unmerged tick and a dashed week-end depth line on the right axis, under a headline "Queue at end of <week>: N open issues · M open PRs". `throttled` stays false because this hive reports no governor issue-creation throttle windows (`missing.throttle_windows_unavailable`). Reporting only ([#12366](https://github.com/hivecommons/hive/issues/12366)).
- **Dashboard:** owners get an **Evidence** button on Review Queue rows and Pipeline cards. It opens a modal with the bundle id, head, hash, whether it is signed, the verdict count and merge status, and **Download JSON** / **Download ZIP** buttons.
- **CLI:** `hivectl review evidence <owner/repo#n> [--head <sha>] [-o FILE] [--zip]` prints or saves the bundle, and `hivectl review evidence verify FILE [--pubkey KEY]` checks it offline (see [hivectl](hivectl.md#review--review-evidence)).
- **Export:** `hivectl evidence export [--since <RFC3339|duration>] [--format csv|jsonl] [--repo owner/name] [--out FILE] [--root DIR]` dumps one row per PR straight from the bundles on disk; see [Exporting](#exporting).
- **Export:** `hivectl evidence export [--since <RFC3339|duration>] [--format csv|jsonl] [--repo owner/name] [--out FILE] [--root DIR]` dumps one row per PR straight from the bundles on disk; see [Exporting](#exporting). `hivectl evidence benchmark` aggregates the same rows into the mediated-vs-unmediated report; see [Outcome benchmark](#outcome-benchmark).
- **On the PR:** when a PR merges through any Hive merge path, the merge event is added to the head's bundle (re-signed) and Hive posts one comment:

```
Expand All @@ -232,6 +232,16 @@ hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive #
- Rows are ordered by repo (case-insensitive) then PR number, so two runs over the same files are byte-identical. Files that do not parse or are not schema v1 are skipped and counted on stderr; nothing is fabricated.
- Reporting only: the export reads bundles and never influences a merge.

## Outcome benchmark

`hivectl evidence benchmark` turns the per-PR receipts into a standing aggregate: a Markdown report comparing PRs Hive mediated with PRs merged without it ([#14577](https://github.com/hivecommons/hive/issues/14577)). It reads the same bundles and the same `evidence.FoldPRs` fold as [Exporting](#exporting), windows on PR open time (`--since` <= `opened_at` < `--until`), and is built by `pkg/outcomebench` (pure functions, no clock).

- **Cohorts:** *mediated* = at least one record of the PR carries a Hive reviewer verdict or a review Hive posted; *unmediated* = none. `control_window` (`on`/`off`) is a secondary cut when present.
- **Metrics per cohort:** revert rate, change-failure rate (reverts + issue reopens within the link window, `evidence.PR.ChangeFailure`), CI failure at the merged head (`ci.status` of the bundle whose head is `merge.sha`, else the newest), Hive `changes_requested` verdict rate (mediated only), median time to merge (`merge.at − merge.opened_at`), and cost per merged mediated PR when `--fleet-spend-usd` is given.
- **Intervals:** 95% Wilson, the formula the hub evidence view uses; any cell with n < 10 reads `not reported`.
- **Not available in schema v1:** post-merge CI re-runs and human `changes_requested` reviews are listed, not approximated.
- `src/scripts/outcome-benchmark.sh YYYY-MM` renders one UTC month into [`src/docs/benchmarks/`](benchmarks/README.md); re-running it on the same evidence root reproduces the same file.

## Signing key

Set `evidence.signing_key_file` in `hive.yaml` (see the [operator reference](operator-reference.md#notable-fields)). The file holds an Ed25519 private key as a 32-byte seed or 64-byte key, hex or base64 encoded. If the path is empty, bundles are written **unsigned** with `"signed": false`; they still carry a `hash`, which detects accidental change but not tampering by someone who can recompute it. A missing or unusable configured key also makes writers fall back to unsigned bundles, but dashboard readers fail closed until that configured key is restored, so an unavailable key cannot silently downgrade signed evidence. Keep the private key off shared hosts and publish only the public key.
Expand Down
50 changes: 31 additions & 19 deletions src/pkg/hivectl/commands/evidence.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
package commands

import (
Expand Down Expand Up @@ -100,11 +100,12 @@
func newEvidenceCommand(env *commandEnv) *cobra.Command {
cmd := &cobra.Command{
Use: "evidence",
Short: "Per-PR review evidence: export stored bundles or backfill them from forge history",
Long: "Work with the review evidence bundles Hive stores under the evidence root (default /data/evidence).\nexport reads the local filesystem only; backfill talks to the GitHub API. Neither contacts a hive and nothing here\ninfluences a merge.",
Example: " hivectl evidence export --since 168h --format csv --out evidence.csv\n hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive\n hivectl evidence backfill --repo ublue-os/bluefin --since 2025-01-01 --dry-run",
Short: "Per-PR review evidence: export or benchmark stored bundles, or backfill them from forge history",
Long: "Work with the review evidence bundles Hive stores under the evidence root (default /data/evidence).\nexport and benchmark read the local filesystem only; backfill talks to the GitHub API. None contacts a hive and nothing here\ninfluences a merge.",
Example: " hivectl evidence export --since 168h --format csv --out evidence.csv\n hivectl evidence export --since 2026-10-01T00:00:00Z --repo hivecommons/hive\n hivectl evidence benchmark --since 2026-10-01T00:00:00Z --until 2026-11-01T00:00:00Z --out outcomes.md\n hivectl evidence backfill --repo ublue-os/bluefin --since 2025-01-01 --dry-run",
}
cmd.AddCommand(newEvidenceExportCommand(env))
cmd.AddCommand(newEvidenceBenchmarkCommand(env))
cmd.AddCommand(newEvidenceBackfillCommand(env))
cmd.AddCommand(newEvidenceBackfillIssuesCommand(env))
return cmd
Expand Down Expand Up @@ -213,6 +214,32 @@
// time), and returns rows ordered by repo then PR number plus the unreadable
// count.
func exportEvidenceRows(root string, cutoff time.Time, repo string) ([]evidenceExportRow, int, error) {
bundles, unreadable, err := loadEvidenceBundles(root)
if err != nil {
return nil, unreadable, err
}
prs := evidence.FoldPRs(bundles, repo, "")
rows := make([]evidenceExportRow, 0, len(prs))
for _, p := range prs {
if !cutoff.IsZero() && p.OpenedAt.Before(cutoff) {
continue
}
rows = append(rows, evidenceRowFromPR(p))
}
sort.Slice(rows, func(i, j int) bool {
ri, rj := strings.ToLower(rows[i].Repo), strings.ToLower(rows[j].Repo)
if ri != rj {
return ri < rj
}
return rows[i].PR < rows[j].PR
})
return rows, unreadable, nil
}

// loadEvidenceBundles walks root and returns every readable schema v1 bundle
// plus the count of .json files that did not parse or are not v1. A missing
// root is empty, not an error.
func loadEvidenceBundles(root string) ([]*evidence.Bundle, int, error) {
var bundles []*evidence.Bundle
unreadable := 0
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
Expand All @@ -236,22 +263,7 @@
if err != nil {
return nil, unreadable, fmt.Errorf("reading evidence under %s: %w", root, err)
}
prs := evidence.FoldPRs(bundles, repo, "")
rows := make([]evidenceExportRow, 0, len(prs))
for _, p := range prs {
if !cutoff.IsZero() && p.OpenedAt.Before(cutoff) {
continue
}
rows = append(rows, evidenceRowFromPR(p))
}
sort.Slice(rows, func(i, j int) bool {
ri, rj := strings.ToLower(rows[i].Repo), strings.ToLower(rows[j].Repo)
if ri != rj {
return ri < rj
}
return rows[i].PR < rows[j].PR
})
return rows, unreadable, nil
return bundles, unreadable, nil
}

// evidenceRowFromPR flattens one folded PR: the newest bundle's verdicts,
Expand Down
Loading
Loading