Skip to content

πŸ“– docs: rewrite README around plain-language evidence cards - #14603

Merged
hivecommons-hive[bot] merged 6 commits into
v5from
docs/readme-evidence-cards
Oct 27, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
v5from
docs/readme-evidence-cards

Conversation

@clubanderson

@clubanderson clubanderson commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Fixes #14602

Builds on #14588 (merged). Opened as a draft on purpose: PRs under this login can auto-merge on green through the trusted-author lane, and the owner asked to review the README copy first. Do not mark ready without that review.

What changes

README β€” 1,727 β†’ 336 prose words, 23 β†’ 7 H2 sections. Order: tagline β†’ scale line β†’ Accuracy β†’ Security β†’ Cost β†’ Also β†’ How it works β†’ Run it β†’ Learn more. Every number is in a generated card; the prose is one or two flat sentences per section, with the limit of each number stated in the same section (not a controlled comparison; revert rates indistinguishable; cost is an estimate; prompt-injection numbers are from Hive's own test set).

Cards (src/pkg/outcomebench/cards.go) β€” real-text SVGs in dark and light, swapped with <picture>/prefers-color-scheme, using the dashboard palette. Emitted by hivectl evidence benchmark next to the existing charts and covered by the drift test, so src/scripts/outcome-benchmark.sh --readme regenerates README and cards together.

Card Number shown Base / source
Accuracy Β· sent back, then fixed 2 in 3 369 of 563 PRs that received a changes_requested verdict pushed a later head that received an approve verdict, Oct 2026 (Followup.FixedAndApproved)
Accuracy Β· time to fix 24 min median, first changes_requested β†’ first approve on a later head, 369 PRs (Followup.TimeToApproval)
Accuracy Β· two real pull requests hive#13523 fixed 4 min / merged 16 min; pluk#328 fixed 10 min / merged 23 min stories-2026-10.json, timestamps from the evidence bundles
Security Β· prompt injection 97% caught, 1 false alarm in 12 61/63 attacks, 1/12 benign, Hive's own eval set (red-team-2026-10.json, from ioscan-red-team.md); labelled "not production traffic"
Cost Β· per merged pull request about $2 $6,124.75 / 2,869 hive-attributed merges, 2026-08-08 β†’ 10-27 (cost-2026-10.json, saved /api/cost)
Also strip 15,054 verdicts; 27 min typical, review adds about 15 Verdicts; TimeToMerge 1,455 reviewed vs 2,224 without review
How it works 6-rung ladder static copy

Accuracy rework (second round) β€” the "1 in 3 sent back" card counted reviewer objections, not outcomes, and was dropped from the README (it stays in the report tables). Candidates computed over the full evidence history, all 10 repos (every reviewed merge is in Oct 2026, so widening the window did not change any base):

Candidate Value Base Kept?
Sent back, then fixed and approved 2 in 3 (369) 563 PRs sent back; 404 pushed a new head, 140 merged on the same head (133 of those because another review perspective approved that head), 19 did not merge kept
Time from change request to approval of the fix 24 min median 369 kept
Escape rate after review (revert or follow-up fix, merges β‰₯ 14 d old) 1 of 43 reviewed vs 157 of 8,652 unreviewed 43 cut: base < 200; first reviewed merge is 2026-10
CI green after a reviewer-driven fix 48 of 140 pairs went failing β†’ passing check captured on both heads for 140 of 404 (35%) cut: coverage < 80%
Issues closed by reviewed PRs that stayed closed β‰₯ 14 d 433 closures, 0 matured; reopen records only since 2026-10-22 0 cut: not matured, detection too new

The README caveat sentence under the cards states the 140 / 133 / 19 split and that revert figures need older merges. Methodology bullets define both new measures; "What the README shows and leaves out" lists each cut with its base.

Also not shown in README, and why: revert rate (base 43 < 200 minimum, and indistinguishable); checks-passing at merge (check result recorded on only 55% of reviewed merges, under the 80% floor); fleet attention list and UI screenshots (over the card cap; no live-cost screenshot showing a priced run exists).

Generator rules β€” --readme-min-base (default 200) for production-count cards, separate from the report's n β‰₯ 10; --stories and --red-team inputs; README block has no statistical vocabulary. Detailed charts, intervals and methodology remain in src/docs/benchmarks/.

Tests β€” TestReadmeProseBudget (≀ 600 words, ≀ 10 H2), TestReadmeVocabulary (statistical and marketing word list across README prose and card text), TestCardsRespectMinimumBase, TestLoadStoriesAndRedTeam; drift test extended to every card and the ladder picture.

Relocated, not deleted β€” the former README overview (what Hive is, three rules, hub and spoke, contributor relay, project vs hosted, does-it-fit, run-it table, ACMM table, architecture, TUI, works-with, related, repositories/adopters/contributing/security pointers) is now src/docs/what-is-hive.md, listed in documentation-map.md. External anchors into the removed README sections (bin/README.md, security-insights.yml, CONTRIBUTING.md, examples/kubestellar/README.md) were redirected.

Rendered

Latest push: written for readers who know CI/CD but not Hive. Opens with the maintainer who reads every pull request, then the CI analogy, then numbers. Only two actor names (coding agent, review agent); CI vocabulary in the ladder; dates as (oct '26); auto-merge framed as the earned end state ("PRs only β†’ merge on green"). Prose: 245 words, 7 sections (ratchet 300 / 10). Vocabulary ratchet now also bans insider nouns (reviewer, scanner, verdict, ACMM, spoke, kick, …) and ISO dates.

Dark:

README rendered, dark

Light:

README rendered, light

Before/after scenario card (hive#13523, pluk#328 in the footnote) and the how-it-works ladder:

Before / after

How it works

For the owner's eye before merge

  • "Came back fixed and were approved" measures that the author acted on the reviewer and the fix passed re-review. It is not proof the shipped code was correct; the outcome measures (reverts, reopened issues) have too little matured reviewed history yet and are listed as not shown.

  • The 140 same-head merges are framed as "another review pass approved the same version" because 133 of those heads carry an approve from a different perspective alongside the changes_requested. Only 5 human actions are recorded on those heads, so this is gate consensus, not human override.

  • The prompt-injection card is exempt from the 200 minimum base because it is a labelled self-run eval, not a production count. Say the word and it comes out.

  • The "Caught" sentences on the story card are my plain-English paraphrase of the posted reviews (linked from the card titles and the report).

  • Side-by-side layout of the small cards depends on GitHub's rendering width; each is 280 px.

  • The security sentence ("every merge is signed off and credits the issue author") describes the DCO sign-off and Co-authored-by attribution path; it is not measured by the benchmark.

Generate dark/light SVG cards from the committed outcome snapshot (sent
back before merge, two real pull requests, prompt injection, cost, an
also strip and a how-it-works ladder); the README block lists Accuracy,
Security, Cost and Also with no statistical vocabulary and a minimum
base of 200 for production counts. Detailed charts and intervals stay in
src/docs/benchmarks; the former overview moves to src/docs/what-is-hive.md.
Adds prose-budget, vocabulary and card tests.

Fixes #14602

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the DCO. labels Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

The Compose quick start contract requires every install recipe to write
src/.env and set HIVE_DASHBOARD_TOKEN.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
The Compose quick start contract requires every install recipe to write
src/.env and set HIVE_DASHBOARD_TOKEN.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive hivecommons-hive Bot added the author/human A person pushed this PR themselves label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
43beb18

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 43beb1876dde) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • security_policy_edit β€” edits security policy, scanner config, rulesets or hooks (Edits SECURITY.md, dependabot/CodeQL config, rulesets or git hooks)
    • security-insights.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

Replace the sent-back-before-merge card, which counted reviewer
objections rather than outcomes, with two follow-up measures computed
per head from the same evidence bundles: the share of sent-back pull
requests that came back fixed and were approved (369 of 563), and the
typical time from change request to approval of the fixed version
(24 minutes, 369 PRs). The report states why reverts, post-fix CI and
reopened issues are not drawn yet, and the methodology defines the new
measures.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor
8a7a01d

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 8a7a01d2f81e) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • security_policy_edit β€” edits security policy, scanner config, rulesets or hooks (Edits SECURITY.md, dependabot/CodeQL config, rulesets or git hooks)
    • security-insights.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

- One card builder: eyebrow, story headline, amber numerals with glow,
  footnote; 880px content width declared at 2x (1760) for crisp rendering
- Warm near-black surface and off-white text; light theme via <picture>
- Accuracy tiles merged into one card with two stats; two-bar time
  comparison in the Also strip; numbered how-it-works ladder
- Drift check and tests updated; old came-back-fixed/time-to-approval
  assets removed

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor
48a8777

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 48a877761c88) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • security_policy_edit β€” edits security policy, scanner config, rulesets or hooks (Edits SECURITY.md, dependabot/CodeQL config, rulesets or git hooks)
    • security-insights.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

- Open with the maintainer who reads every pull request, then the CI
  analogy; numbers follow the picture
- Two actor names only (coding agent, review agent); CI vocabulary in the
  how-it-works ladder (webhook trigger, ephemeral runner, required checks,
  PRs only -> merge on green, co-author trailer)
- Before/after scenario card from hive#13523 with pluk#328 in the footnote
- Dates as (oct '26); vocabulary ratchet bans insider nouns and ISO dates;
  prose ratchet lowered to 300 words

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson marked this pull request as ready for review October 27, 2026 19:02
@kubestellar-prow kubestellar-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) dco All commits are DCO signed-off stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
c3fb016

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head c3fb0162245e) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • security_policy_edit β€” edits security policy, scanner config, rulesets or hooks (Edits SECURITY.md, dependabot/CodeQL config, rulesets or git hooks)
    • security-insights.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive
hivecommons-hive Bot merged commit 4e20313 into v5 Oct 27, 2026
84 of 86 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the docs/readme-evidence-cards branch October 27, 2026 20:45
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#14603c3fb016` sha256:b99c0b396c54a4e8a6502d89c277fa7831e5fc7240d66b034d67a44663e20a1c β€” download from the dashboard or `hivectl review evidence #14603`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at c3fb016.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

README: short, plain-language evidence cards generated from the outcome benchmark

1 participant