Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,26 @@ A terminal agent is a power tool: you hold it, aim it, and it stops when you sto

Around that, Hive picks work from your real queue, sets limits in code with six trust levels from the **AI Codebase Maturity Model (ACMM)** — a framework for how much autonomy a codebase is ready to give agents ([levels](#acmm-levels) · [paper](https://arxiv.org/abs/2604.09388)), checks before and after with deterministic filters and gates, writes every action to an audit log owned by a named human, and runs around the clock. Same agents you use today — governed, scheduled, scoped, and accountable.

## Outcomes

Measured from the review evidence Hive already records, not from a demo. Every figure carries its n and a 95% Wilson interval; anything with n < 10 is not reported. Regenerate with `src/scripts/outcome-benchmark.sh --readme`; a test fails if this block drifts from the committed report.

<!-- outcomes:start -->
<!-- generated by src/scripts/outcome-benchmark.sh --readme; do not edit by hand. Report: src/docs/benchmarks/outcomes-2026-10.md -->
- **34% of reviewed PRs were flagged for changes before they merged** — a `changes_requested` verdict on an earlier head of 490 of 1455 mediated merges [31.3–36.1], 2026-10-27.
- **Revert rate 0.0% mediated vs 0.1% unmediated** — 0/43 vs 1/1513 merges matured 14 days, Wilson [0.0–8.2] vs [0.0–0.4], 2026-10-27.
- **Median time-to-merge 27 min mediated vs 12 min unmediated** — review costs about 15 minutes on the median PR (n=1455 vs 2224), 2026-10-27.
- **≈$2.13 estimated agent spend per merged PR** — `/api/cost` estimate over 2869 hive-attributed merges, 2026-08-08 → 2026-10-27.
- **11571 pull requests across 10 repositories, 15054 review verdicts recorded** — the evidence this is measured from, 2026-10-27.

How measured: per-PR evidence bundles the spoke already writes, folded per PR; 95% Wilson intervals; n < 10 is never reported. Full tables, trend and methodology in [src/docs/benchmarks/outcomes-2026-10.md](src/docs/benchmarks/outcomes-2026-10.md).

![mediated vs unmediated](src/docs/benchmarks/assets/outcomes-2026-10-bars.svg)
![trend by month](src/docs/benchmarks/assets/outcomes-2026-10-trend.svg)
<!-- outcomes:end -->

**One real example.** On [hivecommons/hive#13523](https://github.com/hivecommons/hive/pull/13523) the swarm's [posted review](https://github.com/hivecommons/hive/pull/13523#pullrequestreview-5613922737) flagged a changelog fragment that started with prose (the release compile would have rejected it) and a duplicate fragment; the author fixed both four minutes later and the next head was approved. Methodology and the full tables: [src/docs/benchmarks/](src/docs/benchmarks/README.md).

## Hub and spoke

- **Spoke** — one running Hive, serving your repo(s). It is where agents actually work. Runs on your Kubernetes cluster, Podman, or Docker Compose.
Expand Down
1 change: 1 addition & 0 deletions changelog.d/added-outcome-benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- `hivectl evidence benchmark` and `src/scripts/outcome-benchmark.sh` compute a reproducible mediated-vs-unmediated outcome report (revert rate, change-failure, median time-to-merge, CI at merged head, changes-requested-before-merge, cost per merged PR; Wilson intervals, n < 10 not reported) with generated SVG charts under `src/docs/benchmarks/` and a `--readme` flag that rewrites the README's `<!-- outcomes:start/end -->` block, guarded by a drift test
44 changes: 44 additions & 0 deletions src/docs/benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Outcome benchmarks

Reproducible comparison of pull requests that went through Hive's review swarm (**mediated**) with pull requests merged without it (**unmediated**), computed from the per-PR evidence bundles a spoke already writes under `/data/evidence`. No new telemetry; every rate carries its n and a 95% Wilson interval, and n < 10 is printed as `not reported`.

## Reports

| Month | Report | Charts |
|---|---|---|
| 2026-10 | [outcomes-2026-10.md](outcomes-2026-10.md) | [bars](assets/outcomes-2026-10-bars.svg) · [trend](assets/outcomes-2026-10-trend.svg) |

Each report ships with `outcomes-YYYY-MM.json`, the computed `Report` the markdown, charts and README block are rendered from. `pkg/outcomebench` `TestCommittedReportMatchesReadme` re-renders the newest JSON and fails when the committed markdown, SVGs or the repository README's `<!-- outcomes:start -->…<!-- outcomes:end -->` block differ, so none of them can drift.

## Regenerate

```sh
# on a spoke, or anywhere with a copy of its evidence tree
src/scripts/outcome-benchmark.sh --root /data/evidence --month 2026-10 \
--spend cost.json # saved GET /api/cost, for cost per merged PR
--hub hub.json # saved GET /api/hub/evidence (admin session), optional
--source "hive-oke spoke snapshot 2026-10-27" \
--readme # also rewrite README.md's outcomes block
```

The script wraps `hivectl evidence benchmark`; see [hivectl.md](../hivectl.md#evidence--benchmark) for every flag. To take a read-only copy of a spoke's evidence: `kubectl exec deploy/hive -c hive -- tar czf - -C /data evidence | tar xzf -`.

## Metrics

| Metric | Definition |
|---|---|
| Revert rate | merged PRs the evidence linker tagged `reverted_by` ÷ merged PRs observed ≥ 14 days after merge |
| Change-failure rate | `reverted_by` or `followed_up_by` ÷ the same matured denominator |
| Median time-to-merge | lower median of merge − open time over merged PRs whose bundle recorded an open time |
| CI failure at merged head | captured `failure` ÷ captured success/failure on merged heads; not comparable when a cohort has a captured status on < 50 % of its merges |
| Changes requested pre-merge | mediated merged PRs with ≥ 1 `changes_requested` verdict on any head before the merge |
| Cost per merged PR | `/api/cost` estimated spend ÷ its hive-attributed merged-PR count, same window |

A PR is mediated when any of its heads carries a Hive verdict or a posted Hive review. Backfilled history is unmediated by construction, so the window must start no earlier than the first mediated merge or the comparison is confounded by time; the report states that date.

## Caveats

- Revert and follow-up links depend on the evidence linker having seen the later PR; a spoke that was down misses them.
- CI status is captured only for heads Hive reviewed, so the unmediated CI cell is a small sample until the sweeper records it for every merged head.
- Cost is an estimate: model prices are approximate and some models are unpriced.
- The hub's `GET /api/hub/evidence` needs an admin session; when it is not supplied the report says so and the numbers are one spoke's.
36 changes: 36 additions & 0 deletions src/docs/benchmarks/assets/outcomes-2026-10-bars.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
43 changes: 43 additions & 0 deletions src/docs/benchmarks/assets/outcomes-2026-10-trend.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading