Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,35 +17,35 @@ Numbers below: 11,571 pull requests, 10 repositories, 15,054 AI reviews (oct '26

## Accuracy

<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg"><img alt="2 in 3 pull requests the review agent sent back came back fixed and were approved; 24 minutes typical from sent back to approved" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.png"><img alt="2 in 3 pull requests the review agent sent back came back fixed and were approved; 24 minutes typical from sent back to approved" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.png"></picture>

Humans still merge; what reaches them has passed review and CI. Start at "open PRs only"; once you trust it, turn the [setting](src/docs/acmm-policy-matrix.md) up to merge on green, like a merge queue.

<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.svg"><img alt="Before and after: hivecommons/hive #13523 was sent back once and merged 16 minutes after opening" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.png"><img alt="Before and after: hivecommons/hive #13523 was sent back once and merged 16 minutes after opening" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.png"></picture>

## Security

Hidden instructions in issues or files are screened out before an agent reads them; agents run as a GitHub App, not your PAT.

<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.svg"><img alt="97% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with 1 false alarm in 12 harmless texts" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.png"><img alt="97% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with 1 false alarm in 12 harmless texts" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.png"></picture>

## Cost

Tracked like CI minutes; each job's cost is on the dashboard.

<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.svg"><img alt="about $2 of agent spend per merged pull request" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.png"><img alt="about $2 of agent spend per merged pull request" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.png"></picture>

## Also

<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.svg"><img alt="Also: consistent, readable, efficient, fast" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.png"><img alt="Also: consistent, readable, efficient, fast" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.png"></picture>

Production data, not a lab run; counts regenerate from live records. Reverts are too new to report. [How we count →](src/docs/benchmarks/outcomes-2026-10.md)
<!-- outcomes:end -->

## How it works

<p align="center">
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.svg"><img alt="How Hive works: scan, claim, work, verify, merge, credit" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.svg"></picture>
<picture><source media="(prefers-color-scheme: dark)" srcset="src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.png"><img alt="How Hive works: scan, claim, work, verify, merge, credit" width="880" src="src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.png"></picture>
</p>

## Run it
Expand Down
1 change: 1 addition & 0 deletions changelog.d/fixed-outcome-card-fonts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Render README outcome cards as PNGs with embedded IBM Plex Mono, flat amber figures and consistent spacing; redraw saved reports without the evidence tree using `hivectl evidence benchmark --from-json` or `src/scripts/outcome-benchmark.sh --rerender`.
18 changes: 14 additions & 4 deletions src/docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,18 @@ Reproducible comparison of pull requests that went through Hive's review swarm (

| Month | Report | Charts |
|---|---|---|
| 2026-10 | [outcomes-2026-10.md](outcomes-2026-10.md) | [bars](assets/outcomes-2026-10-bars.svg) · [trend](assets/outcomes-2026-10-trend.svg) · README cards `assets/outcomes-2026-10-card-*.svg` |
| 2026-10 | [outcomes-2026-10.md](outcomes-2026-10.md) | [bars](assets/outcomes-2026-10-bars.svg) · [trend](assets/outcomes-2026-10-trend.svg) · README cards `assets/outcomes-2026-10-card-*.png` (SVG sources alongside) |

Each report ships with `outcomes-YYYY-MM.json`, the computed `Report` the markdown, charts, README cards and README block are rendered from, plus the inputs that are not derivable from the evidence tree: `cost-YYYY-MM.json` (saved `GET /api/cost`), `stories-YYYY-MM.json` (curated caught → fixed → merged examples, timestamps taken from the evidence bundles) and `red-team-YYYY-MM.json` (prompt-injection evaluation counts from [ioscan-red-team.md](../ioscan-red-team.md)). `pkg/outcomebench` `TestCommittedReportMatchesReadme` re-renders the newest JSON and fails when the committed markdown, SVGs, cards or the repository README's `<!-- outcomes:start -->…<!-- outcomes:end -->` block differ, so none of them can drift.
Each report ships with `outcomes-YYYY-MM.json`, the computed `Report` the markdown, charts, README cards and README block are rendered from, plus the inputs that are not derivable from the evidence tree: `cost-YYYY-MM.json` (saved `GET /api/cost`), `stories-YYYY-MM.json` (curated caught → fixed → merged examples, timestamps taken from the evidence bundles) and `red-team-YYYY-MM.json` (prompt-injection evaluation counts from [ioscan-red-team.md](../ioscan-red-team.md)). `pkg/outcomebench` `TestCommittedReportMatchesReadme` re-renders the newest JSON and fails when the committed markdown, SVGs, PNG source hashes and dimensions or the repository README's `<!-- outcomes:start -->…<!-- outcomes:end -->` block differ, so none of them can drift.

## README cards

The repository README shows this data as plain-language cards (one number, one sentence, a footer naming the base and month), with the detailed charts and intervals kept in the monthly report. Rules the generator enforces for the README:

- A production-count card is drawn only when its base is at least `--readme-min-base` (default 200), separate from the n ≥ 10 report threshold. The 43-merge revert bucket, for example, stays in the report.
- The checks-passing card is drawn only when a check result is recorded on at least 80 % of reviewed merges; below that the pass rate would describe a biased subset.
- No statistical vocabulary (intervals, medians, cohorts, n=) appears in the README or on a card; `TestReadmeVocabulary` enforces the list and `TestReadmeProseBudget` keeps README prose under 600 words and 10 sections.
- Every card is a real-text SVG in dark and light (`<picture>` + `prefers-color-scheme`) using the dashboard's own palette.
- No statistical vocabulary (intervals, medians, cohorts, n=) appears in the README or on a card; `TestReadmeVocabulary` enforces the list and `TestReadmeProseBudget` keeps README prose under 300 words and 10 sections.
- Every card has a real-text SVG source and a 2× PNG in dark and light (`<picture>` + `prefers-color-scheme`) using the dashboard's own palette. Both formats consume one drawing operation list. The README embeds the PNG, rasterized in Go with embedded regular and bold IBM Plex Mono faces (SIL OFL, bundled under `pkg/outcomebench/fonts/`), so system fonts and browser effects cannot change its layout. Primary stat cards have one large figure, fixed height, and flat amber; secondary figures stay in the copy. PNG drift checks read the embedded source SHA-256 and dimensions instead of comparing pixels.
- The prompt-injection card comes from Hive's own evaluation set, says so on the card, and is not subject to the minimum base because it is not a production count.

Each report's "What the README shows and leaves out" section records, per card, whether it was drawn and on what base.
Expand All @@ -35,6 +35,16 @@ src/scripts/outcome-benchmark.sh --root /data/evidence --month 2026-10 \
--readme # also rewrite README.md's outcomes block
```

To redraw the committed report without its evidence tree or changing its counts/timestamps:

```sh
src/scripts/outcome-benchmark.sh --rerender --month 2026-10 --readme
# equivalent from src/:
go run ./cmd/hivectl evidence benchmark --from-json docs/benchmarks/outcomes-2026-10.json --readme ../README.md
```

`--from-json` cannot be combined with evidence/computation flags such as `--root`, `--month` or `--readme-min-base`. `--out-dir` and `--readme` still select destinations; `--readme` updates the ladder outside the outcomes block too.

The script wraps `hivectl evidence benchmark`; see [hivectl.md](../hivectl.md#evidence--benchmark) for every flag. To take a read-only copy of a spoke's evidence: `kubectl exec deploy/hive -c hive -- tar czf - -C /data evidence | tar xzf -`.

## Metrics
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 12 additions & 18 deletions src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
28 changes: 12 additions & 16 deletions src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading