diff --git a/NOTICE b/NOTICE index dd6097af1a..f2b3c4b9e0 100644 --- a/NOTICE +++ b/NOTICE @@ -4819,6 +4819,41 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ----------------------------------------------------------------------------- +Package: golang.org/x/image +Version: v0.47.0 +License: BSD-3-Clause +Source: https://cs.opensource.google/go/x/image/+/v0.47.0:LICENSE + +Copyright 2009 The Go Authors. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google LLC nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +----------------------------------------------------------------------------- + Package: golang.org/x/net Version: v0.62.0 License: BSD-3-Clause @@ -4854,7 +4889,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ----------------------------------------------------------------------------- -Package: golang.org/x/sys/unix +Package: golang.org/x/sys Version: v0.49.0 License: BSD-3-Clause Source: https://cs.opensource.google/go/x/sys/+/v0.49.0:LICENSE diff --git a/README.md b/README.md index 8dc316f24f..ec46f19073 100644 --- a/README.md +++ b/README.md @@ -7,45 +7,48 @@ [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/14261/badge)](https://www.bestpractices.dev/projects/14261) [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE) -Every team has one: the maintainer who reads every pull request, never lets red merge, and sends work back in one line. Hive puts that person between your coding agents and your main branch. +You know the one. Reads every pull request. Never lets red merge. Sends it back with one line, and the line is right. Hive puts that person between your coding agents and main. -Hive is like a CI system, except the job is an AI coding agent and the gate is an AI review plus your existing CI. +If you run CI you know the shape. The job is an AI coding agent. The gate is an AI review plus your own CI. - -Numbers below: 11,571 pull requests, 10 repositories, 15,054 AI reviews (oct '26). + +Everything below comes from 11,571 live pull requests in 10 repositories and 15,054 reviews (oct '26). ## Accuracy -2 in 3 pull requests the review agent sent back came back fixed and were approved; 24 minutes typical from sent back to approved +The review agent sent 563 pull requests back. Two out of three came back fixed and approved, usually within 24 minutes. -Humans still merge; what reaches them has passed review and CI. Start at "open PRs only"; once you trust it, turn the [setting](src/docs/acmm-policy-matrix.md) up to merge on green, like a merge queue. - -Before and after: hivecommons/hive #13523 was sent back once and merged 16 minutes after opening +Two out of three pull requests the review agent sent back came back fixed and approved, usually within 24 minutes. +hivecommons/hive #13523 was sent back once and merged 16 minutes after opening. ## Security -Hidden instructions in issues or files are screened out before an agent reads them; agents run as a GitHub App, not your PAT. +We planted 63 texts that try to redirect an agent in issues and files. Hive caught 61 before an agent read a word. One harmless text in 12 got flagged by mistake. Agents run as a GitHub App, not on your token. -97% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with 1 false alarm in 12 harmless texts +Hive caught 61 of 63 planted texts that try to redirect an agent before an agent read a word. One harmless text in 12 got flagged by mistake. ## Cost -Tracked like CI minutes; each job's cost is on the dashboard. +Agents spent about $6,125 to merge 2,869 pull requests. Call it $2 a merge, our estimate (aug–oct '26). + +Agents spent about $6,125 to merge 2,869 pull requests, call it $2 a merge. + +## And the rest -about $2 of agent spend per merged pull request +Every pull request gets the same one-line review, in your style guide's voice. It adds about 15 minutes to a merge. -## Also +Also consistent, readable, efficient and fast. -Also: consistent, readable, efficient, fast +Humans still merge. Start with "open PRs only". When you trust it, [turn it up](src/docs/acmm-policy-matrix.md) and it merges on green, like a merge queue that reads the code. -Production data, not a lab run; counts regenerate from live records. Reverts are too new to report. [How we count →](src/docs/benchmarks/outcomes-2026-10.md) +Reverts are too new to report. [How we count →](src/docs/benchmarks/outcomes-2026-10.md) ## How it works

-How Hive works: scan, claim, work, verify, merge, credit +How Hive works, from scan to credit.

## Run it @@ -57,14 +60,14 @@ printf 'HIVE_GITHUB_TOKEN=ghp_...\nHIVE_DASHBOARD_TOKEN=%s\n' "$(openssl rand -h docker compose -f src/docker-compose.yaml up -d ``` -Each line is explained in the [quick start](src/docs/quickstart-docker-compose.md); [Podman and Kubernetes](src/docs/what-is-hive.md#run-it). +The [quick start](src/docs/quickstart-docker-compose.md) explains each line. [Podman and Kubernetes](src/docs/what-is-hive.md#run-it) too. ## Learn more [Overview](src/docs/what-is-hive.md) · [Architecture](src/docs/architecture.md) · [Security model](src/docs/security-model.md) · [Terminal UI](src/docs/hivectl.md#tui--live-terminal-dashboard) · [Integrations](src/docs/integrations.md) · [Contribute compute](src/docs/contributor-relay.md) · [Contributing](CONTRIBUTING.md) · [Code of Conduct](CODE_OF_CONDUCT.md) · [Adopters](ADOPTERS.md) · [Repositories](docs/repositories.md) · [Changelog](CHANGELOG.md) · [Vulnerabilities](SECURITY.md) · [Telemetry](src/docs/telemetry.md) · [Hosted service](https://hive.hivecommons.dev) -That maintainer now reads every repo. Only run images from `ghcr.io/hivecommons/hive`. +That maintainer now reads every repo. Only run images from `ghcr.io/hivecommons/hive`. Anything else is not ours. --- -Apache-2.0. Hive is a [Cloud Native Computing Foundation](https://www.cncf.io/) sandbox project ([onboarding status](src/docs/cncf-sandbox-onboarding.md)) and follows the [CNCF Code of Conduct](CODE_OF_CONDUCT.md). Copyright The Linux Foundation and Hive contributors; trademarks are used per the [Linux Foundation Trademark Usage page](https://www.linuxfoundation.org/legal/trademark-usage). +Apache-2.0. Hive is a [CNCF](https://www.cncf.io/) sandbox project ([onboarding status](src/docs/cncf-sandbox-onboarding.md)) and follows the [CNCF Code of Conduct](CODE_OF_CONDUCT.md). Copyright The Linux Foundation and Hive contributors. Trademarks follow [Linux Foundation policy](https://www.linuxfoundation.org/legal/trademark-usage). diff --git a/changelog.d/changed-readme-card-png.md b/changelog.d/changed-readme-card-png.md new file mode 100644 index 0000000000..3153f091b2 --- /dev/null +++ b/changelog.d/changed-readme-card-png.md @@ -0,0 +1,2 @@ +- `hivectl evidence benchmark` now also draws the README outcome cards as 2× PNGs with an embedded font (flat colours, no glow), gains `--from-json` to re-render committed assets without the evidence tree, and the README embeds the PNGs so the cards look the same in every browser. +- The README and its outcome cards are written as short full sentences in plain speech, with the numbers inside the sentences. A test now rejects semicolons and label lines in the README prose and card text. diff --git a/src/docs/benchmarks/README.md b/src/docs/benchmarks/README.md index c5245e3650..50e8417459 100644 --- a/src/docs/benchmarks/README.md +++ b/src/docs/benchmarks/README.md @@ -6,7 +6,7 @@ Reproducible comparison of pull requests that went through Hive's review swarm ( | Month | Report | Charts | |---|---|---| -| 2026-10 | [outcomes-2026-10.md](outcomes-2026-10.md) | [bars](assets/outcomes-2026-10-bars.svg) · [trend](assets/outcomes-2026-10-trend.svg) · README cards `assets/outcomes-2026-10-card-*.svg` | +| 2026-10 | [outcomes-2026-10.md](outcomes-2026-10.md) | [bars](assets/outcomes-2026-10-bars.svg) · [trend](assets/outcomes-2026-10-trend.svg) · README cards `assets/outcomes-2026-10-card-*.{svg,png}` | Each report ships with `outcomes-YYYY-MM.json`, the computed `Report` the markdown, charts, README cards and README block are rendered from, plus the inputs that are not derivable from the evidence tree: `cost-YYYY-MM.json` (saved `GET /api/cost`), `stories-YYYY-MM.json` (curated caught → fixed → merged examples, timestamps taken from the evidence bundles) and `red-team-YYYY-MM.json` (prompt-injection evaluation counts from [ioscan-red-team.md](../ioscan-red-team.md)). `pkg/outcomebench` `TestCommittedReportMatchesReadme` re-renders the newest JSON and fails when the committed markdown, SVGs, cards or the repository README's `…` block differ, so none of them can drift. @@ -17,7 +17,8 @@ The repository README shows this data as plain-language cards (one number, one s - A production-count card is drawn only when its base is at least `--readme-min-base` (default 200), separate from the n ≥ 10 report threshold. The 43-merge revert bucket, for example, stays in the report. - The checks-passing card is drawn only when a check result is recorded on at least 80 % of reviewed merges; below that the pass rate would describe a biased subset. - No statistical vocabulary (intervals, medians, cohorts, n=) appears in the README or on a card; `TestReadmeVocabulary` enforces the list and `TestReadmeProseBudget` keeps README prose under 600 words and 10 sections. -- Every card is a real-text SVG in dark and light (`` + `prefers-color-scheme`) using the dashboard's own palette. +- Every card is drawn twice from the same instructions: a real-text SVG (the vector source) and a 2× PNG in dark and light. The README embeds the PNG (`` + `prefers-color-scheme`), because GitHub renders SVG text in whatever monospace the reader has and filters differ by browser; the PNG is rasterised in Go with an embedded IBM Plex Mono (SIL OFL 1.1, `pkg/outcomebench/fonts/`), no external renderer. Flat colours only: no gradient, glow or filter. Each PNG carries the SHA-256 of its SVG in a `tEXt` chunk; the drift test checks that hash and the dimensions rather than pixels, which differ slightly by platform. +- The three stat cards (accuracy, security, cost) are the same height, one figure each, so the README reads as a set. - The prompt-injection card comes from Hive's own evaluation set, says so on the card, and is not subject to the minimum base because it is not a production count. Each report's "What the README shows and leaves out" section records, per card, whether it was drawn and on what base. @@ -35,7 +36,7 @@ src/scripts/outcome-benchmark.sh --root /data/evidence --month 2026-10 \ --readme # also rewrite README.md's outcomes block ``` -The script wraps `hivectl evidence benchmark`; see [hivectl.md](../hivectl.md#evidence--benchmark) for every flag. To take a read-only copy of a spoke's evidence: `kubectl exec deploy/hive -c hive -- tar czf - -C /data evidence | tar xzf -`. +The script wraps `hivectl evidence benchmark`; see [hivectl.md](../hivectl.md#evidence--benchmark) for every flag. After changing the card layout or copy, redraw the committed assets and README block from the newest committed JSON without the evidence tree: `src/scripts/outcome-benchmark.sh --rerender --readme` (`hivectl evidence benchmark --from-json`). To take a read-only copy of a spoke's evidence: `kubectl exec deploy/hive -c hive -- tar czf - -C /data evidence | tar xzf -`. ## Metrics diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.png new file mode 100644 index 0000000000..9e51237e65 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg index 2086db4830..7af5320b45 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-dark.svg @@ -1,20 +1,17 @@ - -2 in 3 pull requests the review agent sent back came back fixed and were approved; 24 minutes typical from sent back to approved - - - - - -ACCURACY · 563 PULL REQUESTS SENT BACK · (oct '26) -The review agent sent 563 pull requests -back. Two in three came back fixed, -typically in 24 minutes. -2 in 3 -2 in 3 -came back fixed -24 min -24 min -typical time to fix - -563 sent back (oct '26); 140 merged as-is after a second review. + +Two out of three pull requests the review agent sent back came back fixed and approved, usually within 24 minutes. + +ACCURACY · 563 PULL REQUESTS SENT BACK · (oct '26) +The review agent sent 563 pull requests back. +Two out of three came back fixed and approved, +usually within 24 minutes. +2 in 3 +came back fixed and approved +369 of 563 came back fixed and +approved, usually within 24 minutes. +Another 140 were fine on a second +look and merged as they were. + +563 pull requests sent back (oct '26), counted from Hive's own review records. Live repos, not a +lab. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.png new file mode 100644 index 0000000000..13e736ecf5 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg index 664cc66e3a..e9bf971f36 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-accuracy-light.svg @@ -1,18 +1,17 @@ - -2 in 3 pull requests the review agent sent back came back fixed and were approved; 24 minutes typical from sent back to approved - - - - - -ACCURACY · 563 PULL REQUESTS SENT BACK · (oct '26) -The review agent sent 563 pull requests -back. Two in three came back fixed, -typically in 24 minutes. -2 in 3 -came back fixed -24 min -typical time to fix - -563 sent back (oct '26); 140 merged as-is after a second review. + +Two out of three pull requests the review agent sent back came back fixed and approved, usually within 24 minutes. + +ACCURACY · 563 PULL REQUESTS SENT BACK · (oct '26) +The review agent sent 563 pull requests back. +Two out of three came back fixed and approved, +usually within 24 minutes. +2 in 3 +came back fixed and approved +369 of 563 came back fixed and +approved, usually within 24 minutes. +Another 140 were fine on a second +look and merged as they were. + +563 pull requests sent back (oct '26), counted from Hive's own review records. Live repos, not a +lab. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.png new file mode 100644 index 0000000000..1a5522fe07 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.svg index 1f8ddbd16f..7b309c5d61 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-dark.svg @@ -1,37 +1,28 @@ - -Also: consistent, readable, efficient, fast - - - - - -ALSO · (oct '26) -Every pull request gets the same AI -review. It adds about 15 minutes. -CONSISTENT -15,054 -15,054 -reviews, one -checklist, one -shared style guide -READABLE -4 parts -4 parts -to every review: -what, problem, fix, -if skipped -EFFICIENT -On demand -On demand -jobs start on work -and stop when done -FAST -merged without review - -12 min -reviewed by Hive - -27 min -review adds about 15 -minutes + +Also consistent, readable, efficient and fast. + +AND THE REST · (oct '26) +Every pull request gets the same one-line +review, in your style guide's voice. It adds +about 15 minutes to a merge. +CONSISTENT +15,054 +reviews, all +against the same +checklist +READABLE +1 line +per finding, in +your style guide's +voice +EFFICIENT +$2 +a merge, our +estimate. Jobs stop +when done +FAST ++15 min +a typical merge +takes 27 minutes, +12 without review diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.png new file mode 100644 index 0000000000..96038045cb Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.svg index 2bdfcc7654..af8e9770a5 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-also-light.svg @@ -1,34 +1,28 @@ - -Also: consistent, readable, efficient, fast - - - - - -ALSO · (oct '26) -Every pull request gets the same AI -review. It adds about 15 minutes. -CONSISTENT -15,054 -reviews, one -checklist, one -shared style guide -READABLE -4 parts -to every review: -what, problem, fix, -if skipped -EFFICIENT -On demand -jobs start on work -and stop when done -FAST -merged without review - -12 min -reviewed by Hive - -27 min -review adds about 15 -minutes + +Also consistent, readable, efficient and fast. + +AND THE REST · (oct '26) +Every pull request gets the same one-line +review, in your style guide's voice. It adds +about 15 minutes to a merge. +CONSISTENT +15,054 +reviews, all +against the same +checklist +READABLE +1 line +per finding, in +your style guide's +voice +EFFICIENT +$2 +a merge, our +estimate. Jobs stop +when done +FAST ++15 min +a typical merge +takes 27 minutes, +12 without review diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.png new file mode 100644 index 0000000000..9081103621 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.svg index fb718cce0b..e1809470bc 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-dark.svg @@ -1,20 +1,15 @@ - -about $2 of agent spend per merged pull request - - - - - -COST · 2,869 MERGED PULL REQUESTS · (aug–oct '26) -Agents spent about $6,125 to merge 2,869 -pull requests, about $2 each. It is -tracked like CI minutes. -$2 -$2 -per merged pull request -$6,125 -$6,125 -total agent spend - -Estimate from provider list prices (aug–oct '26); some models unpriced. + +Agents spent about $6,125 to merge 2,869 pull requests, call it $2 a merge. + +COST · 2,869 MERGED PULL REQUESTS · (aug–oct '26) +Agents spent about $6,125 to merge 2,869 pull +requests. Call it $2 a merge. +$2 +a merge, our estimate +$6,125 of agent spend across 2,869 +merged pull requests (aug–oct '26). +Each job's cost sits on the +dashboard, like CI minutes. + +Our estimate from provider list prices (aug–oct '26). Some models are unpriced and count as zero. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.png new file mode 100644 index 0000000000..51b8ec606b Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.svg index e06263bc04..b3d9041955 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-cost-light.svg @@ -1,18 +1,15 @@ - -about $2 of agent spend per merged pull request - - - - - -COST · 2,869 MERGED PULL REQUESTS · (aug–oct '26) -Agents spent about $6,125 to merge 2,869 -pull requests, about $2 each. It is -tracked like CI minutes. -$2 -per merged pull request -$6,125 -total agent spend - -Estimate from provider list prices (aug–oct '26); some models unpriced. + +Agents spent about $6,125 to merge 2,869 pull requests, call it $2 a merge. + +COST · 2,869 MERGED PULL REQUESTS · (aug–oct '26) +Agents spent about $6,125 to merge 2,869 pull +requests. Call it $2 a merge. +$2 +a merge, our estimate +$6,125 of agent spend across 2,869 +merged pull requests (aug–oct '26). +Each job's cost sits on the +dashboard, like CI minutes. + +Our estimate from provider list prices (aug–oct '26). Some models are unpriced and count as zero. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.png new file mode 100644 index 0000000000..9c1bf9c4b0 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.svg index 266edf7b40..d2a3bd6277 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-dark.svg @@ -1,47 +1,37 @@ - -How Hive works: scan, claim, work, verify, merge, credit - - - - - -HOW IT WORKS · EVERY TASK -A coding agent does the work. A review -agent and your CI gate the merge. -01 -01 -Scan -Reads open issues, pull requests, failing -checks. -like a webhook trigger - -02 -02 -Claim -One agent takes one task. -one job per task - -03 -03 -Work -A short-lived agent does it, then stops. -ephemeral runner - -04 -04 -Verify -AI review plus your CI must pass. -both are required checks - -05 -05 -Merge -Held for a human, or merged on green. -PRs only → merge on green - -06 -06 -Credit -The issue opener is named on the merge. -co-author trailer + +How Hive works, from scan to credit. + +HOW IT WORKS · EVERY TASK +A coding agent does the work. A review agent +and your CI gate the merge. +01 +Scan +Reads open issues, pull requests, failing +checks. +like a webhook trigger + +02 +Claim +One agent takes one task. +one job per task + +03 +Work +A short-lived agent does it, then stops. +ephemeral runner + +04 +Verify +AI review plus your CI must pass. +both are required checks + +05 +Merge +Held for a human, or merged on green. +PRs only, then on green + +06 +Credit +The issue opener is named on the merge. +co-author trailer diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.png new file mode 100644 index 0000000000..09a85e3665 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.svg index 55fce31690..0e3f830c6b 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-how-it-works-light.svg @@ -1,41 +1,37 @@ - -How Hive works: scan, claim, work, verify, merge, credit - - - - - -HOW IT WORKS · EVERY TASK -A coding agent does the work. A review -agent and your CI gate the merge. -01 -Scan -Reads open issues, pull requests, failing -checks. -like a webhook trigger - -02 -Claim -One agent takes one task. -one job per task - -03 -Work -A short-lived agent does it, then stops. -ephemeral runner - -04 -Verify -AI review plus your CI must pass. -both are required checks - -05 -Merge -Held for a human, or merged on green. -PRs only → merge on green - -06 -Credit -The issue opener is named on the merge. -co-author trailer + +How Hive works, from scan to credit. + +HOW IT WORKS · EVERY TASK +A coding agent does the work. A review agent +and your CI gate the merge. +01 +Scan +Reads open issues, pull requests, failing +checks. +like a webhook trigger + +02 +Claim +One agent takes one task. +one job per task + +03 +Work +A short-lived agent does it, then stops. +ephemeral runner + +04 +Verify +AI review plus your CI must pass. +both are required checks + +05 +Merge +Held for a human, or merged on green. +PRs only, then on green + +06 +Credit +The issue opener is named on the merge. +co-author trailer diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.png new file mode 100644 index 0000000000..e2095b82b6 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.svg index 7717c38fa8..228273b0a0 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-dark.svg @@ -1,21 +1,16 @@ - -97% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with 1 false alarm in 12 harmless texts - - - - - -SECURITY · HIVE'S OWN TEST SET · (oct '26) -We hid 63 texts that try to redirect an -agent (prompt injection) in issues and -files. Hive screened out 61 before an -agent read them. -97% -97% -screened out before an agent read it -1 in 12 -1 in 12 -harmless texts wrongly flagged - -Hive's own test set of 75 texts (oct '26); not production traffic. + +Hive caught 61 of 63 planted texts that try to redirect an agent before an agent read a word. One harmless text in 12 got flagged by mistake. + +SECURITY · HIVE'S OWN TEST SET · (oct '26) +We planted 63 texts that try to redirect an +agent in issues and files. Hive caught 61 +before an agent read a word. +97% +of 63 planted texts caught +61 of 63 were caught before an agent +read a word. One harmless text in 12 +got flagged by mistake. + +Hive's own test set of 75 texts (oct '26), not production traffic. Agents run as a GitHub App, not +on your token. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.png new file mode 100644 index 0000000000..f771ad764a Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.svg index 1c574d1ede..5d4d274f95 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-prompt-injection-light.svg @@ -1,19 +1,16 @@ - -97% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with 1 false alarm in 12 harmless texts - - - - - -SECURITY · HIVE'S OWN TEST SET · (oct '26) -We hid 63 texts that try to redirect an -agent (prompt injection) in issues and -files. Hive screened out 61 before an -agent read them. -97% -screened out before an agent read it -1 in 12 -harmless texts wrongly flagged - -Hive's own test set of 75 texts (oct '26); not production traffic. + +Hive caught 61 of 63 planted texts that try to redirect an agent before an agent read a word. One harmless text in 12 got flagged by mistake. + +SECURITY · HIVE'S OWN TEST SET · (oct '26) +We planted 63 texts that try to redirect an +agent in issues and files. Hive caught 61 +before an agent read a word. +97% +of 63 planted texts caught +61 of 63 were caught before an agent +read a word. One harmless text in 12 +got flagged by mistake. + +Hive's own test set of 75 texts (oct '26), not production traffic. Agents run as a GitHub App, not +on your token. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.png new file mode 100644 index 0000000000..a1e98ab489 Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.svg index fc8a4f4ddd..2b0edc7c3c 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-dark.svg @@ -1,49 +1,46 @@ - -Before and after: hivecommons/hive #13523 was sent back once and merged 16 minutes after opening - - - - - -BEFORE / AFTER · ONE REAL PULL REQUEST · (oct '26) -A coding agent opened a pull request. The -review agent sent it back once. It -merged, green, 16 minutes after opening. -Without Hive - -ISSUE -A failing check or an open issue waits -for someone to pick it up. - -PULL REQUEST -Someone writes the fix when they have -time. - -REVIEW -Someone else reads it when they have -time. - -MERGE -Thursday, maybe. -With Hive: hivecommons/hive #13523 - -PULL REQUEST -A coding agent opened it. - -SENT BACK AT 4 MIN -The changelog note was written as a -sentence, not a list entry, so the -release build would have rejected it; a -second, duplicate note was also added. - -FIXED AT 8 MIN -The coding agent pushed a fix; the -review agent approved it. - -MERGED AT 16 MIN -CI green, a human merged it. - - -Same shape: hivecommons/pluk #328, sent back at 6 min, fixed at 16, merged at 23. Times -from Hive's review records. + +hivecommons/hive #13523 was sent back once and merged 16 minutes after opening. + +BEFORE / AFTER · ONE REAL PULL REQUEST · (oct '26) +A coding agent opened a pull request. The +review agent sent it back once. It merged, +green, 16 minutes after opening. +Without Hive + +ISSUE +A failing check or an open issue +waits for someone to pick it up. + +PULL REQUEST +Someone writes the fix when they +have time. + +REVIEW +Someone else reads it when they have +time. + +MERGE +Thursday, maybe. +With Hive · hivecommons/hive #13523 + +PULL REQUEST +A coding agent opened it. + +SENT BACK AT 4 MIN +The changelog note was written as a +sentence, not a list entry, so the +release build would have rejected +it. A second, duplicate note was +also added. + +FIXED AT 8 MIN +The coding agent pushed a fix and +the review agent approved it. + +MERGED AT 16 MIN +CI was green and a human merged it. + + +hivecommons/pluk #328 went the same way, sent back at 6 min, fixed at 16, merged at 23. Times from +Hive's review records. diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.png b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.png new file mode 100644 index 0000000000..28e0f16d0a Binary files /dev/null and b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.png differ diff --git a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.svg b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.svg index 6304fc0559..4b581d5f7a 100644 --- a/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.svg +++ b/src/docs/benchmarks/assets/outcomes-2026-10-card-stories-light.svg @@ -1,49 +1,46 @@ - -Before and after: hivecommons/hive #13523 was sent back once and merged 16 minutes after opening - - - - - -BEFORE / AFTER · ONE REAL PULL REQUEST · (oct '26) -A coding agent opened a pull request. The -review agent sent it back once. It -merged, green, 16 minutes after opening. -Without Hive - -ISSUE -A failing check or an open issue waits -for someone to pick it up. - -PULL REQUEST -Someone writes the fix when they have -time. - -REVIEW -Someone else reads it when they have -time. - -MERGE -Thursday, maybe. -With Hive: hivecommons/hive #13523 - -PULL REQUEST -A coding agent opened it. - -SENT BACK AT 4 MIN -The changelog note was written as a -sentence, not a list entry, so the -release build would have rejected it; a -second, duplicate note was also added. - -FIXED AT 8 MIN -The coding agent pushed a fix; the -review agent approved it. - -MERGED AT 16 MIN -CI green, a human merged it. - - -Same shape: hivecommons/pluk #328, sent back at 6 min, fixed at 16, merged at 23. Times -from Hive's review records. + +hivecommons/hive #13523 was sent back once and merged 16 minutes after opening. + +BEFORE / AFTER · ONE REAL PULL REQUEST · (oct '26) +A coding agent opened a pull request. The +review agent sent it back once. It merged, +green, 16 minutes after opening. +Without Hive + +ISSUE +A failing check or an open issue +waits for someone to pick it up. + +PULL REQUEST +Someone writes the fix when they +have time. + +REVIEW +Someone else reads it when they have +time. + +MERGE +Thursday, maybe. +With Hive · hivecommons/hive #13523 + +PULL REQUEST +A coding agent opened it. + +SENT BACK AT 4 MIN +The changelog note was written as a +sentence, not a list entry, so the +release build would have rejected +it. A second, duplicate note was +also added. + +FIXED AT 8 MIN +The coding agent pushed a fix and +the review agent approved it. + +MERGED AT 16 MIN +CI was green and a human merged it. + + +hivecommons/pluk #328 went the same way, sent back at 6 min, fixed at 16, merged at 23. Times from +Hive's review records. diff --git a/src/docs/benchmarks/outcomes-2026-10.json b/src/docs/benchmarks/outcomes-2026-10.json index c118b5ed6d..59b5fd2fbb 100644 --- a/src/docs/benchmarks/outcomes-2026-10.json +++ b/src/docs/benchmarks/outcomes-2026-10.json @@ -457,7 +457,7 @@ "repo": "hivecommons/hive", "number": 13523, "review_url": "https://github.com/hivecommons/hive/pull/13523#pullrequestreview-5613922737", - "caught": "The changelog note was written as a sentence, not a list entry, so the release build would have rejected it; a second, duplicate note was also added.", + "caught": "The changelog note was written as a sentence, not a list entry, so the release build would have rejected it. A second, duplicate note was also added.", "opened_at": "2026-10-22T12:07:50Z", "caught_at": "2026-10-22T12:12:08Z", "fix_approved_at": "2026-10-22T12:16:15Z", diff --git a/src/docs/benchmarks/outcomes-2026-10.md b/src/docs/benchmarks/outcomes-2026-10.md index 564de06796..194e62a564 100644 --- a/src/docs/benchmarks/outcomes-2026-10.md +++ b/src/docs/benchmarks/outcomes-2026-10.md @@ -44,7 +44,7 @@ Whole evidence history, not just the window; months with n < 10 are gaps. ## Real examples -- [hivecommons/hive#13523](https://github.com/hivecommons/hive/pull/13523): the [posted review](https://github.com/hivecommons/hive/pull/13523#pullrequestreview-5613922737) caught: The changelog note was written as a sentence, not a list entry, so the release build would have rejected it; a second, duplicate note was also added. The fixed head was approved 4 minutes later; the PR merged 16 minutes after opening (opened 12:07, caught 12:12, approved 12:16, merged 2026-10-22 12:23 UTC). +- [hivecommons/hive#13523](https://github.com/hivecommons/hive/pull/13523): the [posted review](https://github.com/hivecommons/hive/pull/13523#pullrequestreview-5613922737) caught: The changelog note was written as a sentence, not a list entry, so the release build would have rejected it. A second, duplicate note was also added. The fixed head was approved 4 minutes later; the PR merged 16 minutes after opening (opened 12:07, caught 12:12, approved 12:16, merged 2026-10-22 12:23 UTC). - [hivecommons/pluk#328](https://github.com/hivecommons/pluk/pull/328): the [posted review](https://github.com/hivecommons/pluk/pull/328#pullrequestreview-5607075007) caught: An apostrophe inside a quoted message ended the string early, so the project would not compile and no tests could run. The fixed head was approved 10 minutes later; the PR merged 23 minutes after opening (opened 22:30, caught 22:36, approved 22:46, merged 2026-10-21 22:54 UTC). ## Scale diff --git a/src/docs/benchmarks/stories-2026-10.json b/src/docs/benchmarks/stories-2026-10.json index 561b4e6cfa..7d01b5ae14 100644 --- a/src/docs/benchmarks/stories-2026-10.json +++ b/src/docs/benchmarks/stories-2026-10.json @@ -3,7 +3,7 @@ "repo": "hivecommons/hive", "number": 13523, "review_url": "https://github.com/hivecommons/hive/pull/13523#pullrequestreview-5613922737", - "caught": "The changelog note was written as a sentence, not a list entry, so the release build would have rejected it; a second, duplicate note was also added.", + "caught": "The changelog note was written as a sentence, not a list entry, so the release build would have rejected it. A second, duplicate note was also added.", "opened_at": "2026-10-22T12:07:50Z", "caught_at": "2026-10-22T12:12:08Z", "fix_approved_at": "2026-10-22T12:16:15Z", diff --git a/src/docs/hivectl.md b/src/docs/hivectl.md index 4479879f79..eeca19aac3 100644 --- a/src/docs/hivectl.md +++ b/src/docs/hivectl.md @@ -293,6 +293,7 @@ Reporting only: nothing here reads into or alters merge eligibility. See ```bash hivectl evidence benchmark --month 2026-10 --out-dir src/docs/benchmarks --readme README.md hivectl evidence benchmark --root ./evidence --month 2026-10 --spend cost.json --hub hub.json --source "hive-oke snapshot" +hivectl evidence benchmark --from-json src/docs/benchmarks/outcomes-2026-10.json --out-dir src/docs/benchmarks --readme README.md ``` `evidence benchmark` folds the schema v1 bundles under `--root` per PR and compares the pull requests Hive's review @@ -304,12 +305,15 @@ interval; n < 10 is `not reported`. `--month YYYY-MM` bounds the merge time (def observation time for maturation, `--hub` adds a saved `GET /api/hub/evidence` response as fleet context, `--source` names the data in the methodology section. It writes `outcomes-YYYY-MM.{md,json}`, `assets/outcomes-YYYY-MM-{bars,trend}.svg` and the plain-language README cards -`assets/outcomes-YYYY-MM-card--{dark,light}.svg` under `--out-dir`, and `--readme FILE` rewrites that file's +`assets/outcomes-YYYY-MM-card--{dark,light}.{svg,png}` under `--out-dir` (the PNG is drawn in-process with an +embedded font and is what the README embeds), and `--readme FILE` rewrites that file's `…` block from the same report. The README block carries only the cards and flat sentences, no intervals: `--readme-min-base N` (default 200) is the smallest base a production-count card may rest on, separate from the n ≥ 10 report threshold; `--stories FILE` supplies curated caught → fixed → merged examples and `--red-team FILE` the prompt-injection evaluation counts (see [benchmarks/](benchmarks/README.md) for -both formats). Local filesystem only; no hive session, no telemetry. `src/scripts/outcome-benchmark.sh` wraps it; see [benchmarks/](benchmarks/README.md). +both formats). `--from-json FILE` skips the evidence tree and re-renders every output from a saved report, for +redrawing committed assets after a layout change. Local filesystem only; no hive session, no telemetry. +`src/scripts/outcome-benchmark.sh` wraps it (`--rerender` = `--from-json` on the newest committed report); see [benchmarks/](benchmarks/README.md). ### evidence — backfill bundles from GitHub history diff --git a/src/go.mod b/src/go.mod index a787500400..f2a42cb444 100644 --- a/src/go.mod +++ b/src/go.mod @@ -27,6 +27,7 @@ require ( go.opentelemetry.io/otel/sdk v1.47.0 go.opentelemetry.io/otel/sdk/metric v1.47.0 go.opentelemetry.io/otel/trace v1.47.0 + golang.org/x/image v0.47.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 gopkg.in/yaml.v3 v3.0.1 ) diff --git a/src/go.sum b/src/go.sum index 7214b2e5f6..69eb18b4f3 100644 --- a/src/go.sum +++ b/src/go.sum @@ -135,6 +135,8 @@ go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= golang.org/x/exp v0.0.0-20240823005443-9b4947da3948 h1:kx6Ds3MlpiUHKj7syVnbp57++8WpuKPcR5yjLBjvLEA= golang.org/x/exp v0.0.0-20240823005443-9b4947da3948/go.mod h1:akd2r19cwCdwSwWeIdzYQGa/EZZyqcOdwWiwj5L5eKQ= +golang.org/x/image v0.47.0 h1:za/8gVz++PPZqmZJBNSISICN5Rrj5zixP/8K++fgmqo= +golang.org/x/image v0.47.0/go.mod h1:IkN5o4A7vq55vXN8FWjjnhg9Rs2GOtffqsSi9mWAKYE= golang.org/x/net v0.62.0 h1:18ELalfM5+XNRwzM2nYqXsZQ9ByS1uhj2O/U7+5jqRY= golang.org/x/net v0.62.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI= golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA= diff --git a/src/pkg/hivectl/commands/evidence_benchmark.go b/src/pkg/hivectl/commands/evidence_benchmark.go index 6339771f44..9c7be3ac06 100644 --- a/src/pkg/hivectl/commands/evidence_benchmark.go +++ b/src/pkg/hivectl/commands/evidence_benchmark.go @@ -22,7 +22,7 @@ var evidenceBenchmarkNow = func() time.Time { return time.Now().UTC() } const benchmarkReadmeDocDir = "src/docs/benchmarks" func newEvidenceBenchmarkCommand(_ *commandEnv) *cobra.Command { - var root, month, now, spendPath, hubPath, storiesPath, redTeamPath, outDir, readme, source string + var root, month, now, spendPath, hubPath, storiesPath, redTeamPath, outDir, readme, source, fromJSON string var readmeMinBase int cmd := &cobra.Command{ Use: "benchmark", @@ -38,9 +38,20 @@ func newEvidenceBenchmarkCommand(_ *commandEnv) *cobra.Command { "for the story card; --red-team the prompt-injection eval summary. --readme rewrites the … block of that README\n" + "from the same report, so the README never drifts from the committed report (a test asserts they match).", Example: " hivectl evidence benchmark --root ./evidence --month 2026-10 --spend cost.json --out-dir src/docs/benchmarks --readme README.md\n" + - " hivectl evidence benchmark --root /data/evidence --month 2026-10 --out-dir /tmp/bench", + " hivectl evidence benchmark --root /data/evidence --month 2026-10 --out-dir /tmp/bench\n" + + " hivectl evidence benchmark --from-json src/docs/benchmarks/outcomes-2026-10.json --out-dir src/docs/benchmarks --readme README.md", Args: argsNone(), RunE: func(cmd *cobra.Command, _ []string) error { + if fromJSON != "" { + if cmd.Flags().Changed("root") || cmd.Flags().Changed("month") || spendPath != "" || hubPath != "" || storiesPath != "" || redTeamPath != "" { + return &usageError{message: "--from-json re-renders a saved report and takes no evidence inputs"} + } + rep, err := loadBenchmarkReport(fromJSON) + if err != nil { + return err + } + return emitBenchmark(cmd, rep, outDir, readme) + } var opts outcomebench.Options var err error if strings.TrimSpace(month) != "" { @@ -86,32 +97,10 @@ func newEvidenceBenchmarkCommand(_ *commandEnv) *cobra.Command { if unreadable > 0 { fmt.Fprintf(cmd.ErrOrStderr(), "skipped %d unreadable or non-v1 bundle(s) under %s\n", unreadable, root) } - rep := outcomebench.Compute(bundles, unreadable, opts) - paths, err := writeBenchmark(rep, outDir) - if err != nil { - return err - } - for _, p := range paths { - fmt.Fprintf(cmd.ErrOrStderr(), "wrote %s\n", p) - } - if readme != "" { - data, err := os.ReadFile(readme) - if err != nil { - return fmt.Errorf("reading %s: %w", readme, err) - } - updated, err := outcomebench.UpdateReadme(string(data), rep.ReadmeBlock(benchmarkReadmeDocDir)) - if err != nil { - return fmt.Errorf("%s: %w", readme, err) - } - if err := os.WriteFile(readme, []byte(updated), 0o644); err != nil { - return fmt.Errorf("writing %s: %w", readme, err) - } - fmt.Fprintf(cmd.ErrOrStderr(), "updated outcomes block in %s\n", readme) - } - fmt.Fprint(cmd.OutOrStdout(), benchmarkSummary(rep)) - return nil + return emitBenchmark(cmd, outcomebench.Compute(bundles, unreadable, opts), outDir, readme) }, } + cmd.Flags().StringVar(&fromJSON, "from-json", "", "re-render markdown, assets and the README block from a saved outcomes-YYYY-MM.json instead of recomputing from evidence") cmd.Flags().StringVar(&root, "root", ghpkg.ReviewEvidenceRoot, "evidence root holding ///.json bundles") cmd.Flags().StringVar(&month, "month", "", "only PRs merged in this YYYY-MM month (default: all)") cmd.Flags().StringVar(&now, "now", "", "observation time (RFC3339) for the maturation window; default now") @@ -126,8 +115,48 @@ func newEvidenceBenchmarkCommand(_ *commandEnv) *cobra.Command { return cmd } -// writeBenchmark writes the markdown, JSON and SVG outputs and returns their -// paths. +// loadBenchmarkReport reads a committed outcomes-YYYY-MM.json. +func loadBenchmarkReport(path string) (outcomebench.Report, error) { + var rep outcomebench.Report + data, err := os.ReadFile(path) + if err != nil { + return rep, err + } + if err := json.Unmarshal(data, &rep); err != nil { + return rep, fmt.Errorf("parsing %s: %w", path, err) + } + return rep, nil +} + +// emitBenchmark writes the report's outputs and, if asked, the README block. +func emitBenchmark(cmd *cobra.Command, rep outcomebench.Report, outDir, readme string) error { + paths, err := writeBenchmark(rep, outDir) + if err != nil { + return err + } + for _, p := range paths { + fmt.Fprintf(cmd.ErrOrStderr(), "wrote %s\n", p) + } + if readme != "" { + data, err := os.ReadFile(readme) + if err != nil { + return fmt.Errorf("reading %s: %w", readme, err) + } + updated, err := outcomebench.UpdateReadme(string(data), rep.ReadmeBlock(benchmarkReadmeDocDir)) + if err != nil { + return fmt.Errorf("%s: %w", readme, err) + } + if err := os.WriteFile(readme, []byte(updated), 0o644); err != nil { + return fmt.Errorf("writing %s: %w", readme, err) + } + fmt.Fprintf(cmd.ErrOrStderr(), "updated outcomes block in %s\n", readme) + } + fmt.Fprint(cmd.OutOrStdout(), benchmarkSummary(rep)) + return nil +} + +// writeBenchmark writes the markdown, JSON, SVG and PNG outputs and returns +// their paths. func writeBenchmark(rep outcomebench.Report, outDir string) ([]string, error) { if err := os.MkdirAll(filepath.Join(outDir, "assets"), 0o755); err != nil { return nil, err @@ -145,6 +174,10 @@ func writeBenchmark(rep outcomebench.Report, outDir string) ([]string, error) { } order := []string{base + ".md", base + ".json", filepath.Join(outDir, "assets", rep.AssetBase()+"-bars.svg"), filepath.Join(outDir, "assets", rep.AssetBase()+"-trend.svg")} cards := rep.Cards() + pngs, err := rep.CardPNGs() + if err != nil { + return nil, err + } for _, name := range outcomebench.CardNames() { for _, theme := range []string{"dark", "light"} { file := rep.CardFile(name, theme) @@ -153,6 +186,11 @@ func writeBenchmark(rep outcomebench.Report, outDir string) ([]string, error) { files[p] = []byte(svg) order = append(order, p) } + if png, ok := pngs[rep.CardPNGFile(name, theme)]; ok { + p := filepath.Join(outDir, "assets", rep.CardPNGFile(name, theme)) + files[p] = png + order = append(order, p) + } } } paths := make([]string, 0, len(files)) diff --git a/src/pkg/hivectl/commands/evidence_benchmark_test.go b/src/pkg/hivectl/commands/evidence_benchmark_test.go index 9dabea1b63..69e835fb19 100644 --- a/src/pkg/hivectl/commands/evidence_benchmark_test.go +++ b/src/pkg/hivectl/commands/evidence_benchmark_test.go @@ -1,6 +1,7 @@ package commands import ( + "bytes" "net/http/httptest" "os" "path/filepath" @@ -43,7 +44,8 @@ func TestEvidenceBenchmarkWritesReportAndReadme(t *testing.T) { if !strings.HasPrefix(stdout, "outcomes 2026-10:") || !strings.Contains(stdout, "mediated 1 merged, unmediated 0 merged") { t.Fatalf("stdout = %q", stdout) } - for _, f := range []string{"outcomes-2026-10.md", "outcomes-2026-10.json", "assets/outcomes-2026-10-bars.svg", "assets/outcomes-2026-10-trend.svg"} { + for _, f := range []string{"outcomes-2026-10.md", "outcomes-2026-10.json", "assets/outcomes-2026-10-bars.svg", "assets/outcomes-2026-10-trend.svg", + "assets/outcomes-2026-10-card-how-it-works-dark.svg", "assets/outcomes-2026-10-card-how-it-works-dark.png", "assets/outcomes-2026-10-card-how-it-works-light.png"} { if _, err := os.Stat(filepath.Join(out, f)); err != nil { t.Errorf("missing %s: %v", f, err) } @@ -56,6 +58,30 @@ func TestEvidenceBenchmarkWritesReportAndReadme(t *testing.T) { if strings.Contains(string(got), "\nold\n") || !strings.HasPrefix(string(got), "# x\n\n"+outcomebench.ReadmeStart) || !strings.HasSuffix(string(got), "\n\nrest\n") { t.Fatalf("readme:\n%s", got) } + + // --from-json re-renders the same outputs from the saved report, so the + // committed assets can be redrawn without the evidence tree. + out2 := t.TempDir() + readme2 := filepath.Join(out2, "README.md") + if err := os.WriteFile(readme2, []byte("# y\n\n"+outcomebench.ReadmeStart+"\nold\n"+outcomebench.ReadmeEnd+"\n"), 0o644); err != nil { + t.Fatal(err) + } + if _, _, err := execute(t, server, "", "evidence", "benchmark", "--from-json", filepath.Join(out, "outcomes-2026-10.json"), "--out-dir", out2, "--readme", readme2); err != nil { + t.Fatal(err) + } + for _, f := range []string{"outcomes-2026-10.md", "assets/outcomes-2026-10-card-how-it-works-dark.png"} { + a, _ := os.ReadFile(filepath.Join(out, f)) + b, err := os.ReadFile(filepath.Join(out2, f)) + if err != nil || !bytes.Equal(a, b) { + t.Errorf("--from-json re-render of %s differs from the original (%v)", f, err) + } + } + if got2, _ := os.ReadFile(readme2); !strings.HasPrefix(string(got2), "# y\n\n"+outcomebench.ReadmeStart) || strings.Contains(string(got2), "\nold\n") { + t.Fatalf("readme after --from-json:\n%s", got2) + } + if _, _, err := execute(t, server, "", "evidence", "benchmark", "--from-json", filepath.Join(out, "outcomes-2026-10.json"), "--spend", spend); err == nil { + t.Error("--from-json with evidence inputs should be a usage error") + } } func TestEvidenceBenchmarkRejectsBadInput(t *testing.T) { diff --git a/src/pkg/outcomebench/cards.go b/src/pkg/outcomebench/cards.go index 9b3b0ac6fd..4cd3461327 100644 --- a/src/pkg/outcomebench/cards.go +++ b/src/pkg/outcomebench/cards.go @@ -122,50 +122,72 @@ func (r Report) MinBase() int { // describes a sample the footer cannot explain in one line. const minCICardCoverage = 0.8 -// cardTheme is the card palette: a warm near-black surface with hairline -// rules, warm off-white text, the dashboard's amber for every numeral and -// for "with Hive" bars, a flat warm gray for "without", and green only for -// pass/merged dots. One SVG is written per theme and the README picks one -// with . The numeral glow is the only decoration and only the dark -// theme has it. +// cardTheme is the card palette: a warm near-black surface with a hairline +// border, warm off-white text, a flat amber for the one figure on each card, +// a flat warm gray for "without", and green only for pass/merged dots. One +// SVG and one 2× PNG are written per theme and the README picks the PNG +// with . There is no gradient, glow or filter: the PNG is the +// rendering of record and must look the same everywhere. type cardTheme struct { - Name, BG, Border, Rule, Head, Text, Muted, Brand, Brand2, Neutral, OK string - Glow bool + Name, BG, Border, Rule, Head, Text, Muted, Brand, Neutral, OK string } var cardThemes = []cardTheme{ - {Name: "dark", BG: "#0f0d0a", Border: "#2a251f", Rule: "#1c1814", Head: "#fffaf0", Text: "#ece5d8", Muted: "#8b949e", Brand: "#f0b429", Brand2: "#ffd166", Neutral: "#3a342c", OK: "#2da44e", Glow: true}, - {Name: "light", BG: "#fbf8f2", Border: "#ddd6c8", Rule: "#e8e2d6", Head: "#1a1612", Text: "#2b2620", Muted: "#6b6459", Brand: "#9a5f0b", Brand2: "#b8741a", Neutral: "#d9d2c5", OK: "#1a7f37"}, + {Name: "dark", BG: "#0f0d0a", Border: "#28231c", Rule: "#211c16", Head: "#fffaf0", Text: "#d9d1c4", Muted: "#8a8174", Brand: "#f0b429", Neutral: "#3a342c", OK: "#3fb950"}, + {Name: "light", BG: "#fbf8f2", Border: "#e2dbcd", Rule: "#ebe5d9", Head: "#1a1612", Text: "#3b342c", Muted: "#7a7165", Brand: "#a3640c", Neutral: "#d6cfc2", OK: "#1a7f37"}, } // Layout constants. Every card spans the README content column and is -// declared at twice its display size so GitHub downsamples it crisp; the -// coordinates below are display pixels (the viewBox). Character budgets -// assume a monospace glyph about 0.6 em wide. +// rendered at twice its display size so GitHub downsamples it crisp; the +// coordinates below are display pixels (the SVG viewBox). The face is IBM +// Plex Mono, whose advance is exactly 0.6 em, so character budgets are +// exact: chars = width / (0.6 × px). const ( cardFont = "'IBM Plex Mono', 'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, Consolas, monospace" cardW = 880 cardScale = 2 - cardPad = 40 - cardRadius = 16 - colX = cardW - 2*cardPad // usable width - eyebrowPx = 11 - eyebrowGap = 40 // eyebrow baseline to headline baseline - headPx = 32 - headLineH = 42 - headChars = 41 - bigPx = 88 - bigPxTight = 64 // three or more stats on one row - bodyPx = 15 - bodyLineH = 22 - footPx = 12 - footLineH = 18 - wideChars = 88 - halfChars = 40 // two columns - tileChars = 20 // four columns + cardPad = 48 + cardRadius = 20 + colX = cardW - 2*cardPad // usable width: 784 + eyebrowPx = 12 + eyebrowTrk = 2.4 + eyebrowGap = 48 // eyebrow baseline to first headline baseline + headPx = 28 + headLineH = 38 + headLines = 3 // headline block is always this tall, so figures line up across cards + bigPx = 112 + bigColW = 400 // the figure and its label + bigLabelPx = 16 + sideX = cardPad + bigColW + 24 // second column of the figure row + bodyPx = 16 + bodyLineH = 24 + footPx = 13 + footLineH = 20 + halfGap = 48 + statCardH = 500 ) -// Card names; the asset file is -card--.svg. +// chars is how many Plex Mono glyphs of size px fit in width display px. +func chars(width, px int) int { return int(float64(width) / (0.6 * float64(px))) } + +// fitPx shrinks a figure so it stays inside width. +func fitPx(s string, px, width int) int { + n := len([]rune(s)) + if w := int(0.6 * float64(px) * float64(n)); w > width { + return width * 10 / (6 * n) + } + return px +} + +var ( + headChars = chars(colX, headPx) // 46 + sideChars = chars(cardW-cardPad-sideX, bodyPx) // 37 + wideChars = chars(colX, footPx) // 100 + halfChars = chars((colX-halfGap)/2-20, bodyPx) // 36 + tileChars = chars(colX/4-24, bodyPx-1) // 19 +) + +// Card names; the asset file is -card--.{svg,png}. const ( CardAccuracy = "accuracy" CardChecksGreen = "checks-green" @@ -181,27 +203,63 @@ func CardNames() []string { return []string{CardAccuracy, CardChecksGreen, CardStories, CardPromptInjection, CardCost, CardAlso, CardHowItWorks} } -// CardFile is the asset filename for a card in a theme. +// CardFile is the SVG asset filename for a card in a theme. func (r Report) CardFile(name, theme string) string { return fmt.Sprintf("%s-card-%s-%s.svg", r.AssetBase(), name, theme) } -// Cards renders every card in every theme, keyed by filename. Cards whose -// data is missing or below MinBase are omitted, never drawn as placeholders. +// CardPNGFile is the rendered PNG the README embeds. +func (r Report) CardPNGFile(name, theme string) string { + return fmt.Sprintf("%s-card-%s-%s.png", r.AssetBase(), name, theme) +} + +// Cards renders every card in every theme as SVG, keyed by filename. Cards +// whose data is missing or below MinBase are omitted, never drawn as +// placeholders. func (r Report) Cards() map[string]string { out := map[string]string{} for _, th := range cardThemes { for _, name := range CardNames() { - if svg, ok := r.Card(name, th); ok { - out[r.CardFile(name, th.Name)] = svg + if c, ok := r.card(name, th); ok { + out[r.CardFile(name, th.Name)] = c.SVG() } } } return out } -// Card renders one named card; ok is false when its data is not shown. +// CardPNGs rasterises every card in every theme at cardScale, keyed by +// filename. Each PNG carries the SHA-256 of its SVG twin in a tEXt chunk so +// the drift test can tell a stale PNG from a fresh one without comparing +// pixels, which differ by platform. +func (r Report) CardPNGs() (map[string][]byte, error) { + out := map[string][]byte{} + for _, th := range cardThemes { + for _, name := range CardNames() { + c, ok := r.card(name, th) + if !ok { + continue + } + png, err := c.PNG() + if err != nil { + return nil, fmt.Errorf("%s: %w", r.CardPNGFile(name, th.Name), err) + } + out[r.CardPNGFile(name, th.Name)] = png + } + } + return out, nil +} + +// Card renders one named card as SVG; ok is false when its data is not shown. func (r Report) Card(name string, th cardTheme) (string, bool) { + c, ok := r.card(name, th) + if !ok { + return "", false + } + return c.SVG(), true +} + +func (r Report) card(name string, th cardTheme) (*card, bool) { switch name { case CardAccuracy: return r.accuracyCard(th) @@ -218,12 +276,12 @@ func (r Report) Card(name string, th cardTheme) (string, bool) { case CardHowItWorks: return howItWorksCard(th), true } - return "", false + return nil, false } // HasCard reports whether a card has data to show. func (r Report) HasCard(name string) bool { - _, ok := r.Card(name, cardThemes[0]) + _, ok := r.card(name, cardThemes[0]) return ok } @@ -289,14 +347,14 @@ func share(rate float64) string { return fmt.Sprintf("%d in 10", int(math.Round(rate*10))) } -// shareWords is share() in words for a headline: "two in three". +// shareWords is share() in words for a headline: "two out of three". func shareWords(rate float64) string { words := []string{"zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"} var k, d int if _, err := fmt.Sscanf(share(rate), "%d in %d", &k, &d); err != nil || k > 10 || d > 10 { return share(rate) } - return words[k] + " in " + words[d] + return words[k] + " out of " + words[d] } // minutes rounds hours to whole minutes. @@ -325,29 +383,55 @@ func wrap(s string, width int) []string { func esc(s string) string { return html.EscapeString(s) } -// card is the shared builder: it collects the body, then wraps it in the -// surface once the height is known. +// textStyle is what varies between runs of text besides size and colour. +type textStyle struct { + Weight int // 400, 500 or 700 + Track float64 // letter-spacing in display px + End bool // right-aligned at x +} + +var ( + plain = textStyle{Weight: 400} + bold = textStyle{Weight: 700} + label = textStyle{Weight: 500, Track: eyebrowTrk} +) + +// op is one drawing instruction. The card records ops once and renders +// them twice: as SVG elements and as pixels (raster.go), so the two assets +// cannot disagree. +type op struct { + kind byte // 't' text, 'r' rect, 'l' line, 'c' circle, 'a' link rect + x, y, w, h int + rx int + px int + fill, stroke string + style textStyle + s string // text, or link href +} + +// card is the shared builder: it collects ops, then closes at a height. type card struct { th cardTheme title string - b strings.Builder + ops []op + h int } -func (c *card) text(x, y, px int, fill, extra, s string) { - fmt.Fprintf(&c.b, `%s`+"\n", x, y, fill, px, extra, esc(s)) +func (c *card) text(x, y, px int, fill string, st textStyle, s string) { + c.ops = append(c.ops, op{kind: 't', x: x, y: y, px: px, fill: fill, style: st, s: s}) } // lines writes wrapped lines and returns the baseline after the last one. -func (c *card) lines(x, y, px, lineH int, fill string, lines []string) int { +func (c *card) lines(x, y, px, lineH int, fill string, st textStyle, lines []string) int { for i, l := range lines { - c.text(x, y+i*lineH, px, fill, "", l) + c.text(x, y+i*lineH, px, fill, st, l) } return y + len(lines)*lineH } // eyebrow is the tracked uppercase line at the top: section, base, window. func (c *card) eyebrow(y int, s string) { - c.text(cardPad, y, eyebrowPx, c.th.Muted, ` font-weight="500" letter-spacing="2.2"`, upperKeepDates(s)) + c.text(cardPad, y, eyebrowPx, c.th.Muted, label, upperKeepDates(s)) } var reDateLabel = regexp.MustCompile(`\([A-Z]{3}(?: '\d\d)?(?:–[A-Z]{3})? '\d\d\)`) @@ -358,162 +442,180 @@ func upperKeepDates(s string) string { return reDateLabel.ReplaceAllStringFunc(strings.ToUpper(s), strings.ToLower) } -// headline is the finding in one plain sentence; the biggest text after -// the numerals. Returns the baseline after it. +// headline is the finding in one plain sentence, bold, the biggest text on +// the card. It returns the baseline after the fixed headline block so the +// row below starts at the same height on every card. func (c *card) headline(y int, s string) int { - return c.lines(cardPad, y, headPx, headLineH, c.th.Head, wrap(s, headChars)) + c.lines(cardPad, y, headPx, headLineH, c.th.Head, bold, wrap(s, headChars)) + return y + headLines*headLineH } -// numeral draws one big amber figure; in the dark theme a blurred copy -// underneath gives it a soft glow. -func (c *card) numeral(x, y, px int, s string) { - if c.th.Glow { - fmt.Fprintf(&c.b, `%s`+"\n", x, y, c.th.Brand, px, esc(s)) - } - c.text(x, y, px, "url(#amber)", ` font-weight="700"`, s) +// figure draws the one big flat-amber number of a card. +func (c *card) figure(x, y, px int, s string) { + c.text(x, y, px, c.th.Brand, bold, s) } func (c *card) rule(x1, x2, y int) { - fmt.Fprintf(&c.b, ``+"\n", x1, y, x2, y, c.th.Rule) + c.ops = append(c.ops, op{kind: 'l', x: x1, y: y, w: x2 - x1, stroke: c.th.Rule}) } -// foot writes the muted footnote above a hairline; y is the hairline. +func (c *card) dot(cx, cy, r int, fill string) { + c.ops = append(c.ops, op{kind: 'c', x: cx, y: cy, rx: r, fill: fill}) +} + +func (c *card) link(href string, x, y, w, h int) { + c.ops = append(c.ops, op{kind: 'a', x: x, y: y, w: w, h: h, s: href}) +} + +// foot writes the muted footnote under a hairline; y is the hairline. +// Returns the baseline after the last line. func (c *card) foot(y int, s string) int { c.rule(cardPad, cardW-cardPad, y) - return c.lines(cardPad, y+footLineH+4, footPx, footLineH, c.th.Muted, wrap(s, wideChars)) + return c.lines(cardPad, y+footLineH+8, footPx, footLineH, c.th.Muted, plain, wrap(s, wideChars)) } -// String closes the card at height h. -func (c *card) String(h int) string { +// close fixes the card height; the bottom padding is cardPad. +func (c *card) close(lastBaseline int) *card { + c.h = lastBaseline - footLineH + cardPad + return c +} + +// closeAt fixes the card at an explicit height. +func (c *card) closeAt(h int) *card { + c.h = h + return c +} + +// SVG renders the ops. The SVG is the vector twin of the PNG: same ops, +// same coordinates, and a font stack that starts with the embedded face. +func (c *card) SVG() string { var out strings.Builder fmt.Fprintf(&out, ` %s - - - - -`, cardW*cardScale, h*cardScale, cardW, h, esc(c.title), cardFont, esc(c.title), c.th.Brand, c.th.Brand2, cardW-1, h-1, cardRadius, c.th.BG, c.th.Border) - out.WriteString(c.b.String()) +`, cardW*cardScale, c.h*cardScale, cardW, c.h, esc(c.title), cardFont, esc(c.title), cardW-1, c.h-1, cardRadius, c.th.BG, c.th.Border) + for _, o := range c.ops { + switch o.kind { + case 't': + attrs := "" + if o.style.Weight != 400 { + attrs += fmt.Sprintf(` font-weight="%d"`, o.style.Weight) + } + if o.style.Track != 0 { + attrs += fmt.Sprintf(` letter-spacing="%.1f"`, o.style.Track) + } + if o.style.End { + attrs += ` text-anchor="end"` + } + fmt.Fprintf(&out, `%s`+"\n", o.x, o.y, o.fill, o.px, attrs, esc(o.s)) + case 'l': + fmt.Fprintf(&out, ``+"\n", o.x, o.y, o.x+o.w, o.y, o.stroke) + case 'c': + fmt.Fprintf(&out, ``+"\n", o.x, o.y, o.rx, o.fill) + case 'a': + fmt.Fprintf(&out, ``+"\n", esc(o.s), o.x, o.y, o.w, o.h) + } + } out.WriteString("\n") return out.String() } -// stat is one numeral with the words under it. -type stat struct{ Big, Under string } +// hero is the one figure of a stat card: the number, the words under it, +// and the sentence beside it that gives the rest of the picture. +type hero struct{ Big, Under, Side string } -// statCard is the shared anatomy: eyebrow, headline, a row of numerals -// each with its base under it, footnote. -func statCard(th cardTheme, title, eye, head string, stats []stat, foot string) string { +// statCard is the shared anatomy: eyebrow, headline, one figure with its +// base under it and context beside it, footnote. Every stat card is +// statCardH tall so the README reads as a set. +func statCard(th cardTheme, title, eye, head string, h hero, foot string) *card { c := &card{th: th, title: title} y := cardPad + eyebrowPx c.eyebrow(y, eye) y = c.headline(y+eyebrowGap, head) - px := bigPx - if len(stats) > 2 { - px = bigPxTight - } - colW := colX / len(stats) - underChars := (colW - 28) / 9 // 15px mono ≈ 9px per glyph, with a gutter - base := y - headLineH + 36 + px - maxUnder := 0 - for i, st := range stats { - x := cardPad + i*colW - c.numeral(x, base, px, st.Big) - under := wrap(st.Under, underChars) - c.lines(x, base+30, bodyPx, bodyLineH, th.Text, under) - if len(under) > maxUnder { - maxUnder = len(under) - } - } - y = base + 30 + maxUnder*bodyLineH + 12 - if foot != "" { - y = c.foot(y, foot) - footLineH - } - return c.String(y + cardPad - 4) + base := y + bigPx - 8 + c.figure(cardPad, base, fitPx(h.Big, bigPx, bigColW), h.Big) + c.lines(cardPad, base+36, bigLabelPx, bodyLineH, th.Text, plain, wrap(h.Under, chars(bigColW, bigLabelPx))) + c.lines(sideX, base-bigPx/2+8, bodyPx, bodyLineH, th.Muted, plain, wrap(h.Side, sideChars)) + c.foot(statCardH-cardPad-2*footLineH-8, foot) + return c.closeAt(statCardH) } -func (r Report) accuracyCard(th cardTheme) (string, bool) { +func (r Report) accuracyCard(th cardTheme) (*card, bool) { f := r.Followup fa, ta := f.FixedAndApproved, f.TimeToApproval if !fa.Reported || fa.K <= 0 || fa.N < r.MinBase() { - return "", false + return nil, false } words := strings.ToUpper(shareWords(fa.Rate)[:1]) + shareWords(fa.Rate)[1:] - stats := []stat{{share(fa.Rate), "came back fixed"}} - title := fmt.Sprintf("%s pull requests the review agent sent back came back fixed and were approved", share(fa.Rate)) - head := fmt.Sprintf("The review agent sent %s pull requests back. %s came back fixed.", commas(fa.N), words) + usually := "" if ta.Reported && ta.N >= r.MinBase() { - stats = append(stats, stat{fmt.Sprintf("%d min", minutes(ta.Value)), "typical time to fix"}) - title += fmt.Sprintf("; %d minutes typical from sent back to approved", minutes(ta.Value)) - head = fmt.Sprintf("The review agent sent %s pull requests back. %s came back fixed, typically in %d minutes.", commas(fa.N), words, minutes(ta.Value)) + usually = fmt.Sprintf(", usually within %d minutes", minutes(ta.Value)) } - foot := fmt.Sprintf("%s sent back %s", commas(fa.N), r.windowLabel()) + title := fmt.Sprintf("%s pull requests the review agent sent back came back fixed and approved%s.", words, usually) + head := fmt.Sprintf("The review agent sent %s pull requests back. %s came back fixed and approved%s.", commas(fa.N), words, usually) + side := fmt.Sprintf("%s of %s came back fixed and approved%s.", commas(fa.K), commas(fa.N), usually) if f.MergedSameHead > 0 { - foot += fmt.Sprintf("; %s merged as-is after a second review", commas(f.MergedSameHead)) + side += fmt.Sprintf(" Another %s were fine on a second look and merged as they were.", commas(f.MergedSameHead)) } return statCard(th, title, fmt.Sprintf("Accuracy · %s pull requests sent back · %s", commas(fa.N), r.windowLabel()), - head, stats, foot+".", + head, + hero{share(fa.Rate), "came back fixed and approved", side}, + fmt.Sprintf("%s pull requests sent back %s, counted from Hive's own review records. Live repos, not a lab.", commas(fa.N), r.windowLabel()), ), true } -func (r Report) checksGreenCard(th cardTheme) (string, bool) { +func (r Report) checksGreenCard(th cardTheme) (*card, bool) { c := r.Mediated if !r.SplitSupported() || !c.CIFailureAtHead.Reported || c.CIFailureAtHead.N < r.MinBase() || !c.CICoverage.Reported || c.CICoverage.Rate < minCICardCoverage { - return "", false + return nil, false } pass := int(math.Round(100 * (1 - c.CIFailureAtHead.Rate))) return statCard(th, - fmt.Sprintf("%d in 100 reviewed pull requests merged with every check passing", pass), + fmt.Sprintf("%d in 100 reviewed pull requests merged with every check green.", pass), fmt.Sprintf("Accuracy · %s reviewed merges · %s", commas(c.CIFailureAtHead.N), r.windowLabel()), fmt.Sprintf("%d in 100 reviewed pull requests merged with every check green.", pass), - []stat{{fmt.Sprintf("%d in 100", pass), "merged with every check green"}}, + hero{fmt.Sprintf("%d in 100", pass), "merged with every check green", fmt.Sprintf("%s reviewed pull requests merged with a recorded check result.", commas(c.CIFailureAtHead.N))}, fmt.Sprintf("%s reviewed merges with a recorded check result %s.", commas(c.CIFailureAtHead.N), r.windowLabel()), ), true } -func (r Report) promptInjectionCard(th cardTheme) (string, bool) { +func (r Report) promptInjectionCard(th cardTheme) (*card, bool) { rt := r.RedTeam if rt == nil || rt.Attacks <= 0 { - return "", false + return nil, false } pct := int(math.Round(100 * float64(rt.Caught) / float64(rt.Attacks))) - fa := fmt.Sprintf("%d in %d", rt.FalseAlarms, rt.Benign) - if rt.FalseAlarms == 0 { - fa = "0" + fa := fmt.Sprintf("One harmless text in %d got flagged by mistake.", rt.Benign) + if rt.FalseAlarms != 1 { + fa = fmt.Sprintf("%d harmless texts in %d got flagged by mistake.", rt.FalseAlarms, rt.Benign) } return statCard(th, - fmt.Sprintf("%d%% of planted text that tries to redirect an agent (prompt injection) was screened out before an agent read it, with %d false alarm in %d harmless texts", pct, rt.FalseAlarms, rt.Benign), + fmt.Sprintf("Hive caught %d of %d planted texts that try to redirect an agent before an agent read a word. %s", rt.Caught, rt.Attacks, fa), fmt.Sprintf("Security · Hive's own test set · %s", monthLabel(rt.RunAt)), - fmt.Sprintf("We hid %d texts that try to redirect an agent (prompt injection) in issues and files. Hive screened out %d before an agent read them.", rt.Attacks, rt.Caught), - []stat{ - {fmt.Sprintf("%d%%", pct), "screened out before an agent read it"}, - {fa, "harmless texts wrongly flagged"}, - }, - fmt.Sprintf("Hive's own test set of %d texts %s; not production traffic.", rt.Attacks+rt.Benign, monthLabel(rt.RunAt)), + fmt.Sprintf("We planted %d texts that try to redirect an agent in issues and files. Hive caught %d before an agent read a word.", rt.Attacks, rt.Caught), + hero{fmt.Sprintf("%d%%", pct), fmt.Sprintf("of %d planted texts caught", rt.Attacks), fmt.Sprintf("%d of %d were caught before an agent read a word. %s", rt.Caught, rt.Attacks, fa)}, + fmt.Sprintf("Hive's own test set of %d texts %s, not production traffic. Agents run as a GitHub App, not on your token.", rt.Attacks+rt.Benign, monthLabel(rt.RunAt)), ), true } -func (r Report) costCard(th cardTheme) (string, bool) { +func (r Report) costCard(th cardTheme) (*card, bool) { per, ok := r.Spend.PerMergedPR() if !ok || r.Spend.MergedPRs < r.MinBase() { - return "", false + return nil, false } - big := fmt.Sprintf("about $%.0f", per) + fig := fmt.Sprintf("$%.0f", per) if per < 1 { - big = fmt.Sprintf("about %d¢", int(math.Round(per*100))) + fig = fmt.Sprintf("%d¢", int(math.Round(per*100))) } window := rangeLabel(r.Spend.WindowStart, r.Spend.WindowEnd) + total := "$" + commas(int(math.Round(r.Spend.USD))) return statCard(th, - big+" of agent spend per merged pull request", + fmt.Sprintf("Agents spent about %s to merge %s pull requests, call it %s a merge.", total, commas(r.Spend.MergedPRs), fig), fmt.Sprintf("Cost · %s merged pull requests · %s", commas(r.Spend.MergedPRs), window), - fmt.Sprintf("Agents spent about $%s to merge %s pull requests, %s each. It is tracked like CI minutes.", commas(int(math.Round(r.Spend.USD))), commas(r.Spend.MergedPRs), big), - []stat{ - {strings.TrimPrefix(big, "about "), "per merged pull request"}, - {"$" + commas(int(math.Round(r.Spend.USD))), "total agent spend"}, - }, - fmt.Sprintf("Estimate from provider list prices %s; some models unpriced.", window), + fmt.Sprintf("Agents spent about %s to merge %s pull requests. Call it %s a merge.", total, commas(r.Spend.MergedPRs), fig), + hero{fig, "a merge, our estimate", fmt.Sprintf("%s of agent spend across %s merged pull requests %s. Each job's cost sits on the dashboard, like CI minutes.", total, commas(r.Spend.MergedPRs), window)}, + fmt.Sprintf("Our estimate from provider list prices %s. Some models are unpriced and count as zero.", window), ), true } @@ -521,25 +623,25 @@ func (r Report) costCard(th cardTheme) (string, bool) { // path without Hive, in plain words and without invented numbers; the right // column is one real pull request with its recorded timings. A second real // pull request, if present, is summarised in the footnote. -func (r Report) storiesCard(th cardTheme) (string, bool) { +func (r Report) storiesCard(th cardTheme) (*card, bool) { if len(r.Stories) == 0 { - return "", false + return nil, false } s := r.Stories[0] - c := &card{th: th, title: fmt.Sprintf("Before and after: %s #%d was sent back once and merged %d minutes after opening", s.Repo, s.Number, s.MergeMinutes())} + c := &card{th: th, title: fmt.Sprintf("%s #%d was sent back once and merged %d minutes after opening.", s.Repo, s.Number, s.MergeMinutes())} y := cardPad + eyebrowPx c.eyebrow(y, fmt.Sprintf("Before / after · one real pull request · %s", r.windowLabel())) - y = c.headline(y+eyebrowGap, fmt.Sprintf("A coding agent opened a pull request. The review agent sent it back once. It merged, green, %d minutes after opening.", s.MergeMinutes())) + 4 - colW := (colX - cardPad) / 2 - top := y + 8 - col := func(i int, label, labelFill string, steps [][2]string, dot []string) int { - x := cardPad + i*(colW+cardPad) - c.text(x, top, bodyPx+2, labelFill, ` font-weight="700"`, label) - yy := top + 36 + y = c.headline(y+eyebrowGap, fmt.Sprintf("A coding agent opened a pull request. The review agent sent it back once. It merged, green, %d minutes after opening.", s.MergeMinutes())) + colW := (colX - halfGap) / 2 + top := y + 20 + col := func(i int, title, titleFill string, steps [][2]string, dots []string) int { + x := cardPad + i*(colW+halfGap) + c.text(x, top, bodyPx, titleFill, bold, title) + yy := top + 44 for j, st := range steps { - fmt.Fprintf(&c.b, ``+"\n", x+4, yy-4, dot[j]) - c.text(x+18, yy, eyebrowPx, th.Muted, ` font-weight="500" letter-spacing="2.2"`, strings.ToUpper(st[0])) - yy = c.lines(x+18, yy+bodyLineH, bodyPx, bodyLineH, th.Text, wrap(st[1], halfChars)) + 12 + c.dot(x+4, yy-4, 4, dots[j]) + c.text(x+20, yy, eyebrowPx, th.Muted, label, strings.ToUpper(st[0])) + yy = c.lines(x+20, yy+bodyLineH, bodyPx, bodyLineH, th.Text, plain, wrap(st[1], halfChars)) + 16 } return yy } @@ -549,41 +651,59 @@ func (r Report) storiesCard(th cardTheme) (string, bool) { {"Review", "Someone else reads it when they have time."}, {"Merge", "Thursday, maybe."}, }, []string{th.Neutral, th.Neutral, th.Neutral, th.Neutral}) - right := col(1, fmt.Sprintf("With Hive: %s #%d", s.Repo, s.Number), th.Brand, [][2]string{ + right := col(1, fmt.Sprintf("With Hive · %s #%d", s.Repo, s.Number), th.Brand, [][2]string{ {"Pull request", "A coding agent opened it."}, {fmt.Sprintf("Sent back at %d min", s.CaughtMinutes()), s.Caught}, - {fmt.Sprintf("Fixed at %d min", s.CaughtMinutes()+s.FixMinutes()), "The coding agent pushed a fix; the review agent approved it."}, - {fmt.Sprintf("Merged at %d min", s.MergeMinutes()), "CI green, a human merged it."}, + {fmt.Sprintf("Fixed at %d min", s.CaughtMinutes()+s.FixMinutes()), "The coding agent pushed a fix and the review agent approved it."}, + {fmt.Sprintf("Merged at %d min", s.MergeMinutes()), "CI was green and a human merged it."}, }, []string{th.Brand, th.Brand, th.OK, th.OK}) bottom := left if right > bottom { bottom = right } - x := cardPad + colW + cardPad - fmt.Fprintf(&c.b, ``+"\n", esc(s.URL()), x, top-20, colW, right-top+20) - foot := "Times from Hive's review records; the right column links to the pull request." + c.link(s.URL(), cardPad+colW+halfGap, top-20, colW, right-top+20) + foot := "Times from Hive's review records. The right column links to the pull request." if len(r.Stories) > 1 { t := r.Stories[1] - foot = fmt.Sprintf("Same shape: %s #%d, sent back at %d min, fixed at %d, merged at %d. Times from Hive's review records.", t.Repo, t.Number, t.CaughtMinutes(), t.CaughtMinutes()+t.FixMinutes(), t.MergeMinutes()) + foot = fmt.Sprintf("%s #%d went the same way, sent back at %d min, fixed at %d, merged at %d. Times from Hive's review records.", t.Repo, t.Number, t.CaughtMinutes(), t.CaughtMinutes()+t.FixMinutes(), t.MergeMinutes()) } - y = c.foot(bottom, foot) - footLineH - return c.String(y + cardPad - 4), true + return c.close(c.foot(bottom+8, foot)), true } // tile is one column of the "also" strip. type tile struct{ Head, Big, Text string } -// alsoTiles are the four secondary properties. Tiles with a number take it -// from the report; the others are plain statements of how Hive works. +// alsoTiles are the four secondary properties, in the order people ask for +// them after accuracy, security and cost. Tiles with a number take it from +// the report; the others are plain statements of how Hive works. func (r Report) alsoTiles() []tile { - return []tile{ - {"Consistent", commas(r.Verdicts), "reviews, one checklist, one shared style guide"}, - {"Readable", "4 parts", "to every review: what, problem, fix, if skipped"}, - {"Efficient", "On demand", "jobs start on work and stop when done"}, + tiles := []tile{ + {"Consistent", commas(r.Verdicts), "reviews, all against the same checklist"}, + {"Readable", "1 line", "per finding, in your style guide's voice"}, + } + if per, ok := r.Spend.PerMergedPR(); ok && r.Spend.MergedPRs >= r.MinBase() { + fig := fmt.Sprintf("$%.0f", per) + if per < 1 { + fig = fmt.Sprintf("%d¢", int(math.Round(per*100))) + } + tiles = append(tiles, tile{"Efficient", fig, "a merge, our estimate. Jobs stop when done"}) + } else { + tiles = append(tiles, tile{"Efficient", "On demand", "jobs start when there is work and stop when done"}) + } + if with, without, ok := r.speedBars(); ok { + d := with - without + if d > 0 { + tiles = append(tiles, tile{"Fast", fmt.Sprintf("+%d min", d), fmt.Sprintf("a typical merge takes %d minutes, %d without review", with, without)}) + } else { + tiles = append(tiles, tile{"Fast", fmt.Sprintf("%d min", with), "typical open to merge with review"}) + } + } else { + tiles = append(tiles, tile{"Fast", "—", "time to merge shows once enough pull requests are on record"}) } + return tiles } -// speedBars is the one two-bar comparison with a base above MinBase on +// speedBars is the one two-sided comparison with a base above MinBase on // both sides: typical minutes from open to merge, with and without review. // ok is false when either side is missing, and the card says so instead. func (r Report) speedBars() (with, without int, ok bool) { @@ -594,65 +714,34 @@ func (r Report) speedBars() (with, without int, ok bool) { return minutes(m.Value), minutes(u.Value), true } -func (r Report) alsoCard(th cardTheme) (string, bool) { +func (r Report) alsoCard(th cardTheme) (*card, bool) { if r.Verdicts < r.MinBase() { - return "", false + return nil, false } - c := &card{th: th, title: "Also: consistent, readable, efficient, fast"} + c := &card{th: th, title: "Also consistent, readable, efficient and fast."} y := cardPad + eyebrowPx - c.eyebrow(y, "Also · "+r.windowLabel()) - head := "Every pull request gets the same AI review, written for people." + c.eyebrow(y, "And the rest · "+r.windowLabel()) + head := "Every pull request gets the same one-line review, in your style guide's voice." if with, without, ok := r.speedBars(); ok && with > without { - head = fmt.Sprintf("Every pull request gets the same AI review. It adds about %d minutes.", with-without) + head = fmt.Sprintf("Every pull request gets the same one-line review, in your style guide's voice. It adds about %d minutes to a merge.", with-without) } - y = c.headline(y+eyebrowGap, head) + 8 + // The tiles start right under the headline, however many lines it took. + headWrapped := wrap(head, headChars) + c.lines(cardPad, y+eyebrowGap, headPx, headLineH, th.Head, bold, headWrapped) + top := y + eyebrowGap + len(headWrapped)*headLineH + 16 tiles := r.alsoTiles() - cols := len(tiles) + 1 - tileW := colX / cols - top := y + tileW := colX / len(tiles) bottom := top for i, t := range tiles { x := cardPad + i*tileW - c.text(x, top, bodyPx, th.Muted, ` font-weight="500" letter-spacing="2.2"`, strings.ToUpper(t.Head)) - c.numeral(x, top+52, 30, t.Big) - yy := c.lines(x, top+80, bodyPx-1, bodyLineH-2, th.Text, wrap(t.Text, tileChars)) - if yy > bottom { - bottom = yy - } - } - // Fourth column: the speed comparison as two thin bars. - x := cardPad + 3*tileW - c.text(x, top, bodyPx, th.Muted, ` font-weight="500" letter-spacing="2.2"`, "FAST") - if with, without, ok := r.speedBars(); ok { - barW := tileW - 76 // room for the value at the bar end - longest := with - if without > longest { - longest = without - } - bar := func(yy int, label string, val int, fill string) { - w := barW * val / longest - c.text(x, yy, footPx, th.Muted, "", label) - fmt.Fprintf(&c.b, ``+"\n", x, yy+9, w, fill) - c.text(x+w+8, yy+17, footPx+1, th.Head, ` font-weight="700"`, fmt.Sprintf("%d min", val)) - } - bar(top+28, "merged without review", without, th.Neutral) - bar(top+68, "reviewed by Hive", with, "url(#amber)") - d := with - without - note := "typical minutes, open to merge" - if d > 0 { - note = fmt.Sprintf("review adds about %d minutes", d) - } - yy := c.lines(x, top+112, bodyPx-1, bodyLineH-2, th.Text, wrap(note, tileChars)) - if yy > bottom { - bottom = yy - } - } else { - yy := c.lines(x, top+28, bodyPx-1, bodyLineH-2, th.Text, wrap("Time to merge is reported once enough pull requests are on record.", tileChars)) + c.text(x, top, eyebrowPx, th.Muted, label, strings.ToUpper(t.Head)) + c.figure(x, top+56, 40, t.Big) + yy := c.lines(x, top+92, bodyPx-1, bodyLineH-2, th.Text, plain, wrap(t.Text, tileChars)) if yy > bottom { bottom = yy } } - return c.String(bottom + cardPad - 8), true + return c.closeAt(bottom - (bodyLineH - 2) + cardPad), true } // rung is one step of the Hive loop for the how-it-works card. @@ -664,29 +753,29 @@ var hiveLoop = []rung{ {"Claim", "One agent takes one task.", "one job per task"}, {"Work", "A short-lived agent does it, then stops.", "ephemeral runner"}, {"Verify", "AI review plus your CI must pass.", "both are required checks"}, - {"Merge", "Held for a human, or merged on green.", "PRs only → merge on green"}, + {"Merge", "Held for a human, or merged on green.", "PRs only, then on green"}, {"Credit", "The issue opener is named on the merge.", "co-author trailer"}, } -func howItWorksCard(th cardTheme) string { - c := &card{th: th, title: "How Hive works: scan, claim, work, verify, merge, credit"} +func howItWorksCard(th cardTheme) *card { + c := &card{th: th, title: "How Hive works, from scan to credit."} y := cardPad + eyebrowPx c.eyebrow(y, "How it works · every task") - y = c.headline(y+eyebrowGap, "A coding agent does the work. A review agent and your CI gate the merge.") - const stepX, textX = cardPad + 48, cardPad + 48 + 96 - y += 6 + c.lines(cardPad, y+eyebrowGap, headPx, headLineH, th.Head, bold, wrap("A coding agent does the work. A review agent and your CI gate the merge.", headChars)) + y = y + eyebrowGap + 2*headLineH + 8 + const stepX, textX = cardPad + 56, cardPad + 56 + 104 for i, r := range hiveLoop { - what := wrap(r.What, 42) - rowH := 20 + len(what)*bodyLineH + 14 - base := y + 26 - c.numeral(cardPad, base, 24, fmt.Sprintf("%02d", i+1)) - c.text(stepX, base, bodyPx, th.Head, ` font-weight="700"`, r.Step) - c.lines(textX, base, bodyPx, bodyLineH, th.Text, what) - c.text(cardW-cardPad, base, footPx+1, th.Muted, ` text-anchor="end"`, r.Outcome) + what := wrap(r.What, chars(cardW-cardPad-textX-220, bodyPx)) + rowH := 24 + len(what)*bodyLineH + 16 + base := y + 30 + c.figure(cardPad, base, 22, fmt.Sprintf("%02d", i+1)) + c.text(stepX, base, bodyPx, th.Head, bold, r.Step) + c.lines(textX, base, bodyPx, bodyLineH, th.Text, plain, what) + c.text(cardW-cardPad, base, footPx, th.Muted, textStyle{Weight: 400, End: true}, r.Outcome) y += rowH if i < len(hiveLoop)-1 { c.rule(cardPad, cardW-cardPad, y) } } - return c.String(y + cardPad - 10) + return c.closeAt(y + cardPad - 8) } diff --git a/src/pkg/outcomebench/cards_test.go b/src/pkg/outcomebench/cards_test.go index 806de4c45d..8075d57792 100644 --- a/src/pkg/outcomebench/cards_test.go +++ b/src/pkg/outcomebench/cards_test.go @@ -1,6 +1,10 @@ package outcomebench import ( + "bytes" + "fmt" + "image" + "image/png" "os" "path/filepath" "strings" @@ -44,17 +48,97 @@ func TestCardsRespectMinimumBase(t *testing.T) { t.Error("checks-green drawn with CI coverage below the floor") } svg, _ := rep.Card(CardAccuracy, cardThemes[0]) - for _, want := range []string{"", `3 in 4<", ">20 min<", "15 sent back (oct '26); 4 merged as-is", `filter="url(#glow)"`, `width="1760"`, `viewBox="0 0 880 `} { + for _, want := range []string{"", `<text`, "ACCURACY · 15 PULL REQUESTS SENT BACK · (oct '26)", "The review agent sent 15 pull requests", "Three out of four", ">3 in 4<", "usually within 20 minutes", "fine on a second look", "15 pull requests sent back (oct '26)", `width="1760"`, `viewBox="0 0 880 500"`, `font-weight="700"`} { if !strings.Contains(svg, want) { t.Errorf("accuracy card missing %q:\n%s", want, svg) } } - if light, _ := rep.Card(CardAccuracy, cardThemes[1]); strings.Contains(light, `filter="url(#glow)"`) { - t.Error("light theme must not glow") + // Flat colour only: no gradient, glow or filter, which render + // differently per browser and are not reproduced in the PNG. + for _, bad := range []string{"<filter", "<linearGradient", "url(#", "<defs>"} { + if strings.Contains(svg, bad) { + t.Errorf("accuracy card contains %q; cards are flat", bad) + } } if len(rep.Cards()) != 2*5 { // accuracy, checks-green, cost, also, how-it-works × dark/light t.Errorf("Cards() = %d files: %v", len(rep.Cards()), rep.Cards()) } + // Every stat card is the same height so the README reads as a set. + for _, name := range []string{CardAccuracy, CardChecksGreen, CardCost} { + if c, _ := rep.card(name, cardThemes[0]); c.h != statCardH { + t.Errorf("%s card is %d tall, want %d", name, c.h, statCardH) + } + } +} + +// TestCardPNGs renders every card to PNG with the embedded face and checks +// each one is the SVG's size at cardScale, carries its source hash, and +// has opaque card pixels inside a transparent rounded corner. +func TestCardPNGs(t *testing.T) { + since, until, _ := ParseWindow("2026-10") + opts := Options{Since: since, Until: until, Now: now, Spend: &Spend{USD: 26, MergedPRs: 13}, ReadmeMinBase: 10} + rep := Compute(fixture(), 0, opts) + pngs, err := rep.CardPNGs() + if err != nil { + t.Fatal(err) + } + svgs := rep.Cards() + if len(pngs) != len(svgs) { + t.Fatalf("%d PNGs for %d SVGs", len(pngs), len(svgs)) + } + for name, svg := range svgs { + pngName := strings.TrimSuffix(name, ".svg") + ".png" + b, ok := pngs[pngName] + if !ok { + t.Errorf("no PNG for %s", name) + continue + } + if got, want := PNGText(b, PNGSourceKey), SourceHash(svg); got != want { + t.Errorf("%s source hash %q, want %q", pngName, got, want) + } + w, h, err := PNGSize(b) + if err != nil { + t.Fatal(err) + } + var sw, sh int + fmt.Sscanf(svg[strings.Index(svg, ` width="`):], ` width="%d" height="%d"`, &sw, &sh) + if w != sw || h != sh || w != cardW*cardScale { + t.Errorf("%s is %dx%d, SVG declares %dx%d", pngName, w, h, sw, sh) + } + img, err := png.Decode(bytes.NewReader(b)) + if err != nil { + t.Fatal(err) + } + if _, _, _, a := img.At(0, 0).RGBA(); a != 0 { + t.Errorf("%s corner pixel is not transparent", pngName) + } + if _, _, _, a := img.At(w/2, h/2).RGBA(); a == 0 { + t.Errorf("%s centre pixel is transparent", pngName) + } + } + if len(pngs["x"]) != 0 { + t.Error("unexpected key") + } +} + +func TestPNGTextRoundTrip(t *testing.T) { + var buf bytes.Buffer + if err := png.Encode(&buf, image.NewNRGBA(image.Rect(0, 0, 2, 2))); err != nil { + t.Fatal(err) + } + out, err := addPNGText(buf.Bytes(), "k", "v") + if err != nil { + t.Fatal(err) + } + if PNGText(out, "k") != "v" || PNGText(out, "other") != "" { + t.Error("tEXt chunk round trip failed") + } + if _, err := png.Decode(bytes.NewReader(out)); err != nil { + t.Errorf("PNG with tEXt chunk does not decode: %v", err) + } + if _, err := addPNGText([]byte("nope"), "k", "v"); err == nil { + t.Error("expected error for non-PNG input") + } } func TestCardHelpers(t *testing.T) { @@ -111,7 +195,7 @@ func TestLoadStoriesAndRedTeam(t *testing.T) { } rep := Report{RedTeam: rt, GeneratedAt: now} svg, ok := rep.Card(CardPromptInjection, cardThemes[1]) - if !ok || !strings.Contains(svg, ">97%<") || !strings.Contains(svg, ">1 in 12<") || !strings.Contains(svg, "We hid 63 texts that try to redirect an") || !strings.Contains(svg, "TEST SET · (oct '26)") || !strings.Contains(svg, "75 texts (oct '26)") || !strings.Contains(svg, "not production traffic") { + if !ok || !strings.Contains(svg, ">97%<") || !strings.Contains(svg, "harmless text in 12") || !strings.Contains(svg, "We planted 63 texts that try to redirect an") || !strings.Contains(svg, "TEST SET · (oct '26)") || !strings.Contains(svg, "75 texts (oct '26)") || !strings.Contains(svg, "not production traffic") { t.Fatalf("prompt-injection card:\n%s", svg) } } diff --git a/src/pkg/outcomebench/drift_test.go b/src/pkg/outcomebench/drift_test.go index ed90d45ceb..8e465cee27 100644 --- a/src/pkg/outcomebench/drift_test.go +++ b/src/pkg/outcomebench/drift_test.go @@ -2,6 +2,7 @@ package outcomebench import ( "encoding/json" + "fmt" "os" "path/filepath" "sort" @@ -12,8 +13,10 @@ import ( // TestCommittedReportMatchesReadme regenerates the newest committed report // from its JSON and asserts the README outcomes block, the markdown and the // SVG assets are exactly what the generator emits, so none of them can -// drift from the data. Fix with: src/scripts/outcome-benchmark.sh --readme -// (or hivectl evidence benchmark … --readme README.md). +// drift from the data. The PNG cards are checked by size and by the source +// hash they carry (pixels differ slightly by platform, so bytes are not +// compared). Fix with: src/scripts/outcome-benchmark.sh --readme (or +// hivectl evidence benchmark … --readme README.md). func TestCommittedReportMatchesReadme(t *testing.T) { docDir := filepath.Join("..", "..", "docs", "benchmarks") reports, err := filepath.Glob(filepath.Join(docDir, "outcomes-*.json")) @@ -63,12 +66,34 @@ func TestCommittedReportMatchesReadme(t *testing.T) { check(filepath.Join(docDir, "assets", rep.AssetBase()+"-trend.svg"), rep.TrendSVG()) for file, svg := range rep.Cards() { check(filepath.Join(docDir, "assets", file), svg) + pngPath := filepath.Join(docDir, "assets", strings.TrimSuffix(file, ".svg")+".png") + b, err := os.ReadFile(pngPath) + if err != nil { + t.Errorf("%s: %v (regenerate with src/scripts/outcome-benchmark.sh)", pngPath, err) + continue + } + if got, want := PNGText(b, PNGSourceKey), SourceHash(svg); got != want { + t.Errorf("%s was rendered from a different SVG (source %.12s, want %.12s); regenerate with src/scripts/outcome-benchmark.sh", pngPath, got, want) + } + w, h, err := PNGSize(b) + if err != nil { + t.Errorf("%s: %v", pngPath, err) + } else if wantW, wantH := svgSize(svg); w != wantW || h != wantH { + t.Errorf("%s is %dx%d, SVG declares %dx%d", pngPath, w, h, wantW, wantH) + } } // A card the generator no longer emits must not linger as a stale asset. - stale, _ := filepath.Glob(filepath.Join(docDir, "assets", rep.AssetBase()+"-card-*.svg")) + stale, _ := filepath.Glob(filepath.Join(docDir, "assets", rep.AssetBase()+"-card-*.*")) for _, p := range stale { - if _, ok := rep.Cards()[filepath.Base(p)]; !ok { + svgName := strings.TrimSuffix(filepath.Base(p), filepath.Ext(p)) + ".svg" + if _, ok := rep.Cards()[svgName]; !ok { t.Errorf("%s is not emitted by the generator any more; delete it", p) } } } + +// svgSize reads the declared width/height of a generated card SVG. +func svgSize(svg string) (w, h int) { + fmt.Sscanf(svg[strings.Index(svg, ` width="`):], ` width="%d" height="%d"`, &w, &h) + return w, h +} diff --git a/src/pkg/outcomebench/fonts/IBMPlexMono-Bold.ttf b/src/pkg/outcomebench/fonts/IBMPlexMono-Bold.ttf new file mode 100644 index 0000000000..b1ce96ad21 Binary files /dev/null and b/src/pkg/outcomebench/fonts/IBMPlexMono-Bold.ttf differ diff --git a/src/pkg/outcomebench/fonts/IBMPlexMono-Medium.ttf b/src/pkg/outcomebench/fonts/IBMPlexMono-Medium.ttf new file mode 100644 index 0000000000..f24cbab268 Binary files /dev/null and b/src/pkg/outcomebench/fonts/IBMPlexMono-Medium.ttf differ diff --git a/src/pkg/outcomebench/fonts/IBMPlexMono-Regular.ttf b/src/pkg/outcomebench/fonts/IBMPlexMono-Regular.ttf new file mode 100644 index 0000000000..f7300999fe Binary files /dev/null and b/src/pkg/outcomebench/fonts/IBMPlexMono-Regular.ttf differ diff --git a/src/pkg/outcomebench/fonts/OFL.txt b/src/pkg/outcomebench/fonts/OFL.txt new file mode 100644 index 0000000000..c35c4c618f --- /dev/null +++ b/src/pkg/outcomebench/fonts/OFL.txt @@ -0,0 +1,93 @@ +Copyright © 2017 IBM Corp. with Reserved Font Name "Plex" + +This Font Software is licensed under the SIL Open Font License, Version 1.1. + +This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/src/pkg/outcomebench/outcomebench_test.go b/src/pkg/outcomebench/outcomebench_test.go index f21e0ce0d8..61bd39b633 100644 --- a/src/pkg/outcomebench/outcomebench_test.go +++ b/src/pkg/outcomebench/outcomebench_test.go @@ -227,13 +227,13 @@ func TestReadmeBlockRoundTrip(t *testing.T) { Stories: []Story{{Repo: "hivecommons/hive", Number: 1, Caught: "A test was skipped.", OpenedAt: now.Add(-20 * time.Minute), CaughtAt: now.Add(-15 * time.Minute), FixApprovedAt: now.Add(-10 * time.Minute), MergedAt: now}}, RedTeam: &RedTeam{Attacks: 10, Caught: 9, Benign: 5, FalseAlarms: 1, RunAt: now}}) block := rep.ReadmeBlock("src/docs/benchmarks") - for _, want := range []string{ReadmeStart, ReadmeEnd, "## Accuracy", "## Security", "## Cost", "## Also", "46 pull requests, 2 repositories", "(oct '26)", "Humans still merge", "open PRs only", "src/docs/benchmarks/outcomes-2026-10.md", - "card-accuracy-dark.svg", "card-accuracy-light.svg", "card-stories-dark.svg", "card-prompt-injection-light.svg", "card-cost-dark.svg", "card-also-light.svg", "prefers-color-scheme: dark", "How we count"} { + for _, want := range []string{ReadmeStart, ReadmeEnd, "## Accuracy", "## Security", "## Cost", "## And the rest", "46 live pull requests in 2 repositories", "(oct '26)", "Humans still merge", "open PRs only", "The review agent sent 15 pull requests back. Three out of four came back fixed and approved, usually within 20 minutes.", "We planted 10 texts", "Hive caught 9 before an agent read a word.", "One harmless text in 5 got flagged by mistake.", "Call it $2 a merge, our estimate (", "Every pull request gets the same one-line review", "src/docs/benchmarks/outcomes-2026-10.md", + "card-accuracy-dark.png", "card-accuracy-light.png", "card-stories-dark.png", "card-prompt-injection-light.png", "card-cost-dark.png", "card-also-light.png", "prefers-color-scheme: dark", "How we count"} { if !strings.Contains(block, want) { t.Errorf("block missing %q:\n%s", want, block) } } - for _, banned := range []string{"Wilson", "interval", "(n=", " n=", "median", "mediated", "95%", "≈"} { + for _, banned := range []string{"Wilson", "interval", "(n=", " n=", "median", "mediated", "95%", "≈", ".svg", ";", "Numbers below"} { if strings.Contains(block, banned) { t.Errorf("block contains %q; statistical vocabulary belongs in the report", banned) } diff --git a/src/pkg/outcomebench/raster.go b/src/pkg/outcomebench/raster.go new file mode 100644 index 0000000000..7657fc38ce --- /dev/null +++ b/src/pkg/outcomebench/raster.go @@ -0,0 +1,233 @@ +package outcomebench + +import ( + "bytes" + "crypto/sha256" + "embed" + "encoding/binary" + "encoding/hex" + "errors" + "fmt" + "hash/crc32" + "image" + "image/color" + "image/draw" + "image/png" + "sync" + + "golang.org/x/image/font" + "golang.org/x/image/font/opentype" + "golang.org/x/image/font/sfnt" + "golang.org/x/image/math/fixed" + "golang.org/x/image/vector" +) + +// The README embeds PNGs, not SVGs: GitHub serves SVG text in whatever +// monospace the reader's machine has, and filters render differently per +// browser. The PNG is drawn here from the same ops as the SVG with an +// embedded face (IBM Plex Mono, SIL OFL 1.1, fonts/OFL.txt), at cardScale +// so GitHub downsamples it crisp. No external renderer is involved. + +//go:embed fonts/IBMPlexMono-Regular.ttf fonts/IBMPlexMono-Medium.ttf fonts/IBMPlexMono-Bold.ttf +var fontFS embed.FS + +var ( + fontOnce sync.Once + fontErr error + fonts = map[int]*sfnt.Font{} +) + +func loadFonts() error { + fontOnce.Do(func() { + for weight, file := range map[int]string{400: "Regular", 500: "Medium", 700: "Bold"} { + b, err := fontFS.ReadFile("fonts/IBMPlexMono-" + file + ".ttf") + if err != nil { + fontErr = err + return + } + f, err := sfnt.Parse(b) + if err != nil { + fontErr = fmt.Errorf("parsing %s: %w", file, err) + return + } + fonts[weight] = f + } + }) + return fontErr +} + +// PNGSourceKey is the tEXt keyword under which a card PNG records the +// SHA-256 of the SVG it was rendered alongside. +const PNGSourceKey = "hive-card-source" + +// SourceHash is the hex SHA-256 of a card's SVG, the value stored in its +// PNG so a stale PNG can be told from a fresh one. +func SourceHash(svg string) string { + sum := sha256.Sum256([]byte(svg)) + return hex.EncodeToString(sum[:]) +} + +// PNG rasterises the card at cardScale and tags it with SourceHash(SVG()). +func (c *card) PNG() ([]byte, error) { + if err := loadFonts(); err != nil { + return nil, err + } + const s = cardScale + img := image.NewNRGBA(image.Rect(0, 0, cardW*s, c.h*s)) + bg, border := hexColor(c.th.BG), hexColor(c.th.Border) + fillRoundRect(img, 0, 0, float64(cardW*s), float64(c.h*s), cardRadius*s, border) + fillRoundRect(img, s, s, float64(cardW*s-2*s), float64(c.h*s-2*s), cardRadius*s-s, bg) + faces := map[[2]int]font.Face{} + face := func(weight, px int) (font.Face, error) { + k := [2]int{weight, px} + if f, ok := faces[k]; ok { + return f, nil + } + ft, ok := fonts[weight] + if !ok { + return nil, fmt.Errorf("no embedded face for weight %d", weight) + } + f, err := opentype.NewFace(ft, &opentype.FaceOptions{Size: float64(px * s), DPI: 72, Hinting: font.HintingNone}) + if err != nil { + return nil, err + } + faces[k] = f + return f, nil + } + for _, o := range c.ops { + switch o.kind { + case 't': + f, err := face(o.style.Weight, o.px) + if err != nil { + return nil, err + } + drawText(img, f, o, s) + case 'l': + draw.Draw(img, image.Rect(o.x*s, o.y*s, (o.x+o.w)*s, o.y*s+s), image.NewUniform(hexColor(o.stroke)), image.Point{}, draw.Over) + case 'c': + fillCircle(img, float64(o.x*s), float64(o.y*s), float64(o.rx*s), hexColor(o.fill)) + } + } + var buf bytes.Buffer + enc := png.Encoder{CompressionLevel: png.BestCompression} + if err := enc.Encode(&buf, img); err != nil { + return nil, err + } + return addPNGText(buf.Bytes(), PNGSourceKey, SourceHash(c.SVG())) +} + +// drawText draws one text op glyph by glyph so letter-spacing and end +// anchoring work the same as in the SVG. +func drawText(dst draw.Image, f font.Face, o op, s int) { + track := fixed.Int26_6(o.style.Track * float64(s) * 64) + runes := []rune(o.s) + width := fixed.Int26_6(0) + for i, r := range runes { + adv, _ := f.GlyphAdvance(r) + width += adv + if i < len(runes)-1 { + width += track + } + } + x := fixed.I(o.x * s) + if o.style.End { + x -= width + } + d := &font.Drawer{Dst: dst, Src: image.NewUniform(hexColor(o.fill)), Face: f, Dot: fixed.Point26_6{X: x, Y: fixed.I(o.y * s)}} + for _, r := range runes { + d.DrawString(string(r)) + d.Dot.X += track + } +} + +// fillRoundRect fills an axis-aligned rounded rectangle with antialiasing. +func fillRoundRect(dst *image.NRGBA, x, y, w, h float64, r float64, col color.Color) { + const k = 0.5522847498 // cubic Bézier circle constant + z := vector.NewRasterizer(dst.Bounds().Dx(), dst.Bounds().Dy()) + z.MoveTo(float32(x+r), float32(y)) + z.LineTo(float32(x+w-r), float32(y)) + z.CubeTo(float32(x+w-r+r*k), float32(y), float32(x+w), float32(y+r-r*k), float32(x+w), float32(y+r)) + z.LineTo(float32(x+w), float32(y+h-r)) + z.CubeTo(float32(x+w), float32(y+h-r+r*k), float32(x+w-r+r*k), float32(y+h), float32(x+w-r), float32(y+h)) + z.LineTo(float32(x+r), float32(y+h)) + z.CubeTo(float32(x+r-r*k), float32(y+h), float32(x), float32(y+h-r+r*k), float32(x), float32(y+h-r)) + z.LineTo(float32(x), float32(y+r)) + z.CubeTo(float32(x), float32(y+r-r*k), float32(x+r-r*k), float32(y), float32(x+r), float32(y)) + z.ClosePath() + z.Draw(dst, dst.Bounds(), image.NewUniform(col), image.Point{}) +} + +func fillCircle(dst *image.NRGBA, cx, cy, r float64, col color.Color) { + const k = 0.5522847498 + z := vector.NewRasterizer(dst.Bounds().Dx(), dst.Bounds().Dy()) + z.MoveTo(float32(cx+r), float32(cy)) + z.CubeTo(float32(cx+r), float32(cy+r*k), float32(cx+r*k), float32(cy+r), float32(cx), float32(cy+r)) + z.CubeTo(float32(cx-r*k), float32(cy+r), float32(cx-r), float32(cy+r*k), float32(cx-r), float32(cy)) + z.CubeTo(float32(cx-r), float32(cy-r*k), float32(cx-r*k), float32(cy-r), float32(cx), float32(cy-r)) + z.CubeTo(float32(cx+r*k), float32(cy-r), float32(cx+r), float32(cy-r*k), float32(cx+r), float32(cy)) + z.ClosePath() + z.Draw(dst, dst.Bounds(), image.NewUniform(col), image.Point{}) +} + +// hexColor parses "#rrggbb"; anything else is opaque magenta so a typo is +// visible in the output rather than silently black. +func hexColor(s string) color.NRGBA { + var r, g, b uint8 + if len(s) == 7 && s[0] == '#' { + if _, err := fmt.Sscanf(s[1:], "%02x%02x%02x", &r, &g, &b); err == nil { + return color.NRGBA{R: r, G: g, B: b, A: 255} + } + } + return color.NRGBA{R: 255, A: 255, B: 255} +} + +// addPNGText inserts a tEXt chunk before IEND. +func addPNGText(data []byte, key, val string) ([]byte, error) { + const iendLen = 12 + if len(data) < iendLen || string(data[len(data)-8:len(data)-4]) != "IEND" { + return nil, errors.New("png: no IEND chunk") + } + payload := append(append([]byte(key), 0), []byte(val)...) + chunk := make([]byte, 0, 12+len(payload)) + chunk = binary.BigEndian.AppendUint32(chunk, uint32(len(payload))) + chunk = append(chunk, "tEXt"...) + chunk = append(chunk, payload...) + chunk = binary.BigEndian.AppendUint32(chunk, crc32.ChecksumIEEE(chunk[4:])) + out := make([]byte, 0, len(data)+len(chunk)) + out = append(out, data[:len(data)-iendLen]...) + out = append(out, chunk...) + out = append(out, data[len(data)-iendLen:]...) + return out, nil +} + +// PNGText returns the value of a tEXt chunk with the given keyword, or "". +func PNGText(data []byte, key string) string { + const sig = 8 + for i := sig; i+8 <= len(data); { + n := int(binary.BigEndian.Uint32(data[i:])) + typ := string(data[i+4 : i+8]) + if i+12+n > len(data) { + return "" + } + if typ == "tEXt" { + body := data[i+8 : i+8+n] + if j := bytes.IndexByte(body, 0); j >= 0 && string(body[:j]) == key { + return string(body[j+1:]) + } + } + if typ == "IEND" { + return "" + } + i += 12 + n + } + return "" +} + +// PNGSize reads the IHDR dimensions. +func PNGSize(data []byte) (w, h int, err error) { + cfg, err := png.DecodeConfig(bytes.NewReader(data)) + if err != nil { + return 0, 0, err + } + return cfg.Width, cfg.Height, nil +} diff --git a/src/pkg/outcomebench/readme.go b/src/pkg/outcomebench/readme.go index 0b2e27776d..33ae39b393 100644 --- a/src/pkg/outcomebench/readme.go +++ b/src/pkg/outcomebench/readme.go @@ -3,6 +3,7 @@ package outcomebench import ( "errors" "fmt" + "math" "strings" ) @@ -17,7 +18,7 @@ var ErrNoMarkers = errors.New("README has no " + ReadmeStart + " … " + ReadmeE // ReadmeBlock renders the README evidence section for a general reader: // one line of scale, then Accuracy / Security / Cost / Also, each a card -// (a <picture> with a dark and a light SVG) and at most two flat sentences, +// (a <picture> with a dark and a light PNG) and at most two flat sentences, // one of which states the limit of the number. No statistical vocabulary // appears here; it lives in the report. docDir is the path from the README // to the benchmark directory. @@ -38,34 +39,41 @@ func (r Report) ReadmeBlock(docDir string) string { } } add(ReadmeStart) - add(fmt.Sprintf("<!-- generated by src/scripts/outcome-benchmark.sh --readme; do not edit by hand. Report: %s -->", report)) + add(fmt.Sprintf("<!-- generated by src/scripts/outcome-benchmark.sh --readme. Do not edit by hand. Report: %s -->", report)) add(r.HeroLine(), "") add("## Accuracy", "") if r.HasCard(CardAccuracy) || r.HasCard(CardStories) { - cards(cardW, CardAccuracy, CardChecksGreen) - add(fmt.Sprintf("Humans still merge; what reaches them has passed review and CI. Start at \"open PRs only\"; once you trust it, turn the [setting](%s/acmm-policy-matrix.md) up to merge on green, like a merge queue.", strings.TrimSuffix(docDir, "/benchmarks")), "") - cards(cardW, CardStories) + if s, ok := r.accuracySentences(); ok { + add(s, "") + } + cards(cardW, CardAccuracy, CardChecksGreen, CardStories) } else { - add("Not enough reviewed pull requests are on record yet to report accuracy figures.", "") + add("Not enough reviewed pull requests are on record yet to say how accurate the review is.", "") } add("## Security", "") - add("Hidden instructions in issues or files are screened out before an agent reads them; agents run as a GitHub App, not your PAT.", "") + if s, ok := r.securitySentences(); ok { + add(s, "") + } else { + add("Agents run as a GitHub App, not on your token.", "") + } cards(cardW, CardPromptInjection) add("## Cost", "") - if r.HasCard(CardCost) { - add("Tracked like CI minutes; each job's cost is on the dashboard.", "") + if s, ok := r.costSentences(); ok { + add(s, "") cards(cardW, CardCost) } else { - add("Cost per merged pull request is reported once enough Hive-attributed merges are on record.", "") + add("Cost per merged pull request shows once enough Hive merges are on record.", "") } - add("## Also", "") + add("## And the rest", "") + add(r.alsoSentences(), "") cards(cardW, CardAlso) - add(fmt.Sprintf("Production data, not a lab run; counts regenerate from live records. %s [How we count →](%s)", r.followupCaveat(), report)) + add(fmt.Sprintf("Humans still merge. Start with \"open PRs only\". When you trust it, [turn it up](%s/acmm-policy-matrix.md) and it merges on green, like a merge queue that reads the code.", strings.TrimSuffix(docDir, "/benchmarks")), "") + add(fmt.Sprintf("%s [How we count →](%s)", r.followupCaveat(), report)) add(ReadmeEnd) return strings.Join(lines, "\n") } @@ -78,15 +86,67 @@ func (r Report) followupCaveat() string { // HeroLine is the one sentence of scale at the top of the README. func (r Report) HeroLine() string { - s := fmt.Sprintf("Numbers below: %s pull requests, %d repositories", commas(r.PRs), len(r.Repos)) + s := fmt.Sprintf("Everything below comes from %s live pull requests in %d repositories", commas(r.PRs), len(r.Repos)) if r.Verdicts > 0 { - s += fmt.Sprintf(", %s AI reviews", commas(r.Verdicts)) + s += fmt.Sprintf(" and %s reviews", commas(r.Verdicts)) } return s + " " + monthLabel(r.GeneratedAt) + "." } -// picture is the GitHub-flavoured <picture> that swaps the card SVG with +// The section sentences repeat each card's finding as plain text, so the +// numbers are searchable and read aloud the way a person would say them. +// Each is a few short sentences with one idea apiece and no semicolons. + +func (r Report) accuracySentences() (string, bool) { + f := r.Followup + fa, ta := f.FixedAndApproved, f.TimeToApproval + if !fa.Reported || fa.K <= 0 || fa.N < r.MinBase() { + return "", false + } + words := strings.ToUpper(shareWords(fa.Rate)[:1]) + shareWords(fa.Rate)[1:] + s := fmt.Sprintf("The review agent sent %s pull requests back. %s came back fixed and approved", commas(fa.N), words) + if ta.Reported && ta.N >= r.MinBase() { + s += fmt.Sprintf(", usually within %d minutes", minutes(ta.Value)) + } + return s + ".", true +} + +func (r Report) securitySentences() (string, bool) { + rt := r.RedTeam + if rt == nil || rt.Attacks <= 0 { + return "", false + } + fa := fmt.Sprintf("One harmless text in %d got flagged by mistake.", rt.Benign) + if rt.FalseAlarms != 1 { + fa = fmt.Sprintf("%d harmless texts in %d got flagged by mistake.", rt.FalseAlarms, rt.Benign) + } + return fmt.Sprintf("We planted %d texts that try to redirect an agent in issues and files. Hive caught %d before an agent read a word. %s Agents run as a GitHub App, not on your token.", rt.Attacks, rt.Caught, fa), true +} + +func (r Report) costSentences() (string, bool) { + per, ok := r.Spend.PerMergedPR() + if !ok || r.Spend.MergedPRs < r.MinBase() { + return "", false + } + fig := fmt.Sprintf("$%.0f", per) + if per < 1 { + fig = fmt.Sprintf("%d¢", int(math.Round(per*100))) + } + total := "$" + commas(int(math.Round(r.Spend.USD))) + return fmt.Sprintf("Agents spent about %s to merge %s pull requests. Call it %s a merge, our estimate %s.", total, commas(r.Spend.MergedPRs), fig, rangeLabel(r.Spend.WindowStart, r.Spend.WindowEnd)), true +} + +func (r Report) alsoSentences() string { + s := "Every pull request gets the same one-line review, in your style guide's voice." + if with, without, ok := r.speedBars(); ok && with > without { + s += fmt.Sprintf(" It adds about %d minutes to a merge.", with-without) + } + return s +} + +// picture is the GitHub-flavoured <picture> that swaps the card PNG with // the reader's colour scheme; the alt text is the card's title sentence. +// The PNG, not the SVG, is embedded: see raster.go. func (r Report) picture(docDir, card string, width int) string { alt := card if svg, ok := r.Card(card, cardThemes[0]); ok { @@ -95,7 +155,7 @@ func (r Report) picture(docDir, card string, width int) string { } } return fmt.Sprintf(`<picture><source media="(prefers-color-scheme: dark)" srcset="%s/assets/%s"><img alt="%s" width="%d" src="%s/assets/%s"></picture>`, - docDir, r.CardFile(card, "dark"), alt, width, docDir, r.CardFile(card, "light")) + docDir, r.CardPNGFile(card, "dark"), alt, width, docDir, r.CardPNGFile(card, "light")) } // HowItWorksPicture is the ladder card for the README's own section. diff --git a/src/pkg/outcomebench/readme_budget_test.go b/src/pkg/outcomebench/readme_budget_test.go index 6c007dc42c..19937962ec 100644 --- a/src/pkg/outcomebench/readme_budget_test.go +++ b/src/pkg/outcomebench/readme_budget_test.go @@ -1,6 +1,7 @@ package outcomebench import ( + "html" "os" "path/filepath" "regexp" @@ -103,6 +104,11 @@ var readmeForbiddenWords = []string{ // Dates in the README and cards are "(oct '26)" or "(aug–oct '26)"; never // ISO dates or bare four-digit years. +// A prose line that is one or two words followed by a colon is a label, not +// a sentence ("Numbers below:"). Headings and link rows are exempt because +// they have no colon. +var readmeLabelLine = regexp.MustCompile(`^\s*[a-z][a-z' -]{0,24}:\s`) + var readmeForbiddenDates = []*regexp.Regexp{ regexp.MustCompile(`\d{4}-\d{2}-\d{2}`), regexp.MustCompile(`\b20\d\d\b`), @@ -118,7 +124,7 @@ func TestReadmeVocabulary(t *testing.T) { if err != nil { t.Fatal(err) } - texts[filepath.Base(p)] = strings.ToLower(reHTMLTag.ReplaceAllString(string(b), " ")) + texts[filepath.Base(p)] = strings.ToLower(html.UnescapeString(reHTMLTag.ReplaceAllString(string(b), " "))) } for name, text := range texts { for _, w := range readmeForbiddenWords { @@ -132,6 +138,17 @@ func TestReadmeVocabulary(t *testing.T) { t.Errorf("%s contains date %q; write dates as (oct '26)", name, m) } } + // The copy reads like a person talking: short full sentences, one + // idea each. No semicolon-joined fragments and no "Numbers below:" + // style labels introducing a list. + for _, line := range strings.Split(text, "\n") { + if strings.Contains(line, ";") { + t.Errorf("%s has a semicolon; write two sentences: %q", name, strings.TrimSpace(line)) + } + if readmeLabelLine.MatchString(line) { + t.Errorf("%s has a label introducing a list: %q", name, strings.TrimSpace(line)) + } + } if strings.Contains(text, "!") && name == "README prose" { for _, line := range strings.Split(text, "\n") { if strings.Contains(line, "!") { diff --git a/src/scripts/outcome-benchmark.sh b/src/scripts/outcome-benchmark.sh index 7931141c4e..472699b70d 100755 --- a/src/scripts/outcome-benchmark.sh +++ b/src/scripts/outcome-benchmark.sh @@ -3,15 +3,20 @@ # evidence and, with --readme, the README's <!-- outcomes:start/end --> block. # # src/scripts/outcome-benchmark.sh --root ./evidence --month 2026-10 --spend cost.json --readme +# src/scripts/outcome-benchmark.sh --rerender --readme # redraw from the committed JSON # # Flags are passed through to `hivectl evidence benchmark`; --readme (no # argument) rewrites the repository README. Defaults: --root /data/evidence, # --month = current UTC month, --out-dir src/docs/benchmarks. A saved # GET /api/cost response (--spend) adds cost per merged PR; a saved -# GET /api/hub/evidence response (--hub) adds fleet context. Nothing here -# contacts a hive or adds telemetry; a test (pkg/outcomebench -# TestCommittedReportMatchesReadme) fails when the README block, report or -# charts no longer match the committed JSON. +# GET /api/hub/evidence response (--hub) adds fleet context. --rerender +# skips the evidence tree and redraws the markdown, SVG and PNG cards and +# the README block from the newest committed outcomes-YYYY-MM.json (use it +# after changing the card layout). The PNG cards are drawn in Go with an +# embedded font; no external renderer is needed. Nothing here contacts a +# hive or adds telemetry; a test (pkg/outcomebench +# TestCommittedReportMatchesReadme) fails when the README block, report, +# charts or cards no longer match the committed JSON. set -euo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" @@ -20,16 +25,25 @@ cd "$repo_root/src" args=() readme="" month="" +rerender="" out_dir="docs/benchmarks" while [ $# -gt 0 ]; do case "$1" in --readme) readme="$repo_root/README.md"; shift ;; + --rerender) rerender=1; shift ;; --month) month="$2"; shift 2 ;; --out-dir) out_dir="$2"; shift 2 ;; - -h|--help) sed -n '2,15p' "$0"; exit 0 ;; + -h|--help) sed -n '2,21p' "$0"; exit 0 ;; *) args+=("$1"); shift ;; esac done +if [ -n "$rerender" ]; then + latest="$(ls "$out_dir"/outcomes-*.json 2>/dev/null | sort | tail -n 1)" + [ -n "$latest" ] || { echo "no $out_dir/outcomes-*.json to re-render" >&2; exit 1; } + cmd=(go run ./cmd/hivectl evidence benchmark --from-json "$latest" --out-dir "$out_dir") + [ -n "$readme" ] && cmd+=(--readme "$readme") + exec "${cmd[@]}" "${args[@]}" +fi [ -n "$month" ] || month="$(date -u +%Y-%m)" cmd=(go run ./cmd/hivectl evidence benchmark --month "$month" --out-dir "$out_dir")