diff --git a/changelog.d/added-18720-evidence-change-failures-per-model.md b/changelog.d/added-18720-evidence-change-failures-per-model.md new file mode 100644 index 0000000000..82648fda89 --- /dev/null +++ b/changelog.d/added-18720-evidence-change-failures-per-model.md @@ -0,0 +1 @@ +- **Evidence: change failures per model (#18720).** `GET /api/evidence/summary` adds `outcomes_by_author_model`, the same outcome and change-failure counts as `outcomes_by_author_kind` keyed by the PR author's canonical model (the ids the PRs-by-model card uses; `human` for a person's PR, `unknown` for an agent PR with no recorded model). The Evidence card's **Outcome by author kind** panel shows it as a table: model, `n`, merged, reverted and re-opened within 14 days, and the change-failure rate, so a model whose merged work keeps getting undone stands out from one in the same family whose work sticks. It reuses the existing revert and re-open detection; existing fields are unchanged. diff --git a/dashboard/openapi.json b/dashboard/openapi.json index b9a9112c93..4a685adee4 100644 --- a/dashboard/openapi.json +++ b/dashboard/openapi.json @@ -31826,6 +31826,41 @@ } } }, + "outcomes_by_author_model": { + "type": "object", + "description": "outcomes_by_author_kind keyed by the author's canonical model (the ids /api/governor/pr-models uses); human for a person's PR, unknown for an agent PR with no recorded model (#18720).", + "additionalProperties": { + "type": "object", + "properties": { + "n": { + "type": "integer" + }, + "merged": { + "type": "integer" + }, + "closed": { + "type": "integer" + }, + "open": { + "type": "integer" + }, + "reverted_14d": { + "type": "integer", + "nullable": true + }, + "issue_reopened_14d": { + "type": "integer" + }, + "change_failures": { + "type": "integer" + }, + "change_failure_rate": { + "type": "number", + "nullable": true + } + } + } + }, "reviewer_calibration": { "type": "array", "items": { diff --git a/src/docs/api-reference/agents-and-controls.md b/src/docs/api-reference/agents-and-controls.md index ffb786b9e8..3f4e1b42e0 100644 --- a/src/docs/api-reference/agents-and-controls.md +++ b/src/docs/api-reference/agents-and-controls.md @@ -31,7 +31,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `GET` | `/api/review/evidence` | Merger or owner | [Review evidence bundle](../review-evidence.md) for one PR head. Query `repo` (`owner/name`), `number`, optional `head` (full SHA or unique prefix of at least 7 characters; `409` when ambiguous; default is the newest head), `format=json\|zip` (default `json`) and `download=1` for an attachment. `json` returns the bundle bytes exactly as sealed (verdicts, posted reviews, policy, CI check runs, sentinel findings, human actions, merge event, `hash`, optional `signature`); `zip` adds `verdicts.json`, `review-links.json` and a `manifest.json` of SHA-256s plus the signing public key when readable. `400` for a bad `repo`/`number`; `404` with `status: "expired"`/`expired: true` when retention removed a bundle Hive can prove existed, otherwise `status: "not_found"` | `pkg/dashboard/api.go:514` | | `GET` | `/api/review/evidence/list` | Merger or owner | Every retained evidence bundle id and head for one PR (`repo`, `number`), oldest first, with the newest marked latest; `400` for a bad `repo`/`number` | `pkg/dashboard/api.go:515` | | `GET` | `/api/evidence/write-counters` | Merger or owner | Durable attempted vs blocked agent GitHub write counters ([#12069](https://github.com/hivecommons/hive/issues/12069)): `totals` by intent tier (ACMM mode) with `blocked_by_reason`, and `weeks` keyed by the UTC Monday of each ISO week with per-tier and per-reason counts. Persisted across restarts; observation only, never gates a write or merge. Empty `totals`/`weeks` before anything is recorded | `pkg/dashboard/api.go:517` | -| `GET` | `/api/evidence/summary` | Dashboard auth/session | Read-only aggregate over `/data/evidence` bundles (schema v1) plus reviewer accuracy. Query `hours` (default 24, capped like `/api/pr-throughput`; `0` = all time) or a custom `from`/`to` range (RFC3339 or `YYYY-MM-DD`, date-only `to` inclusive, `to` defaults to now; `400` when `to` precedes `from`, the span exceeds 3 × 366 days, or `to` is given without `from`; the response echoes `from`/`to`), `repo` (`owner/name`) and `author_kind=human\|agent\|bot`; `400` for a bad value. A PR falls in the window by its open time (earliest bundle `created_at` or `merge.opened_at`), never by when a bundle was written; all bundles of a PR are folded so the newest supplies verdicts and the merge / `reverted_by` / backfilled close come from whichever record knows them. Returns `available_windows` (`{"24","168","720","all"}` → PR count under the same filters), `outcomes_by_author_kind` (merged/closed/open; `closed` comes from backfilled records, `reverted_14d` is null until bundles record it), `reviewer_calibration` buckets per perspective, `reviewer_accuracy_by_model` split by author kind (blocks split into `enforced_blocks` / `advisory_blocks` by `policy.require_approval`; advisory blocks that preceded a merge are `dissent_before_merge`, not false blocks), `sentinel_hits_per_week`, `shadow_gate` (`n`, `would_have_held`, `would_have_blocked`, and per gate `review_gate` / `intent` the evaluated `n` with `held` / `passed` outcomes `{n, merged, closed, open, reverted}`, plus `review_gate_by_lane` `{merge_eligible, self_merge}`), `write_counters` (per intent tier `{attempted, blocked, blocked_by_reason}` from the proxy write counter file; omitted when absent or unreadable), `write_activity` (blocked ÷ every agent write in the window: `total_writes`, `blocked`, `direct` `{recorded, since, partial_coverage, attempted, blocked, by_tier, blocked_by_reason}` from the proxy's weekly counter buckets under `/data/hive-metrics/write-counters.json`, `mediated` `{n, refused, by_action}` from the audit log), `claim_vs_verdict` (per ISO week of agent-authored PRs: `n`, `claimed`, `ci_captured`, `ci_not_green`, `verdict_not_approve`, `both`; records with empty CI are excluded from the CI column and disclosed under `missing.records_empty_ci`), `ci_recovery` (`policy` `{self_authored_automerge, required_checks}`, `n`, `at_claim` `{green, red, pending, none}`, `red` `{n, recovered_merged, green_open, abandoned, pending, outcome_unknown, enriched, not_enriched, median_pushes_to_green, median_hours_to_green, same_head_recovered}`, `gate` `{merged_n, with_checks, green, green_from_bundles, non_required_red, required_unknown, breaches, breach_prs[], suspected, suspected_prs[], pending, unknown, green_to_merge_n, median_green_to_merge_hours}` over every merged PR by GitHub-confirmed state of the *required* checks on the merged head, as the merge gate judges (`breach_prs[]` carry `red`, `merged_by`; a red non-required check is disclosed, not a breach; a red review-time snapshot, a required check with no run, or a red read without a resolvable required set is only suspected), and `convergence` `{n, passes {"1","2","3","4+","never"}, settled, in_flight, unknown, median_passes, from_github, from_bundles, by_week[]}` over agent PRs by CI passes to first green, from the durable `/data/hive-metrics/ci-recovery.json` GitHub check-run trail rollup), `work_balance` (per ISO week: `n`, `opened` by actor kind `{human, agent, bot}`, `closed`, `prs_opened`, `prs_merged`, `prs_closed`, `queue_issues`, `queue_prs`, `issue_source` `github|audit`, `throttled`) with `work_balance_sources` (`issues_github_since`, `issues_fetched_at`, `issues_known`, `audit_only_weeks`, `issue_series_weeks`, `queue_samples`, `queue_retention_days`; issues from the durable GitHub rollup `/data/hive-metrics/issue-activity.json` for weeks at or after `issues_github_since` and, for earlier weeks, from the backfilled issue series `///issues/.json` written by the issue backfill ([#12527](https://github.com/hivecommons/hive/pull/12527)) when its coverage spans the requested portion of the week (`issue_series_weeks` counts the repo-weeks it supplied), else the audit log (`audit_only_weeks`), PRs from folded bundles, queue depth from governor snapshots; `missing.issues_human_opens_unobserved` is true while no GitHub rollup exists unless the issue series covers every row, and whenever any row fell back to the audit log, `missing.throttle_windows_unavailable` discloses that no governor throttle windows are reported), `intent_counters` (per intent tier `{attempted, blocked}` from `/data/hive-metrics/intent-counters.json` — evaluations and the subset the intent gate would have blocked with enforcement on, [#12362](https://github.com/hivecommons/hive/issues/12362); omitted when absent or unreadable), `complexity_by_repo` (per managed repo the newest 12 monthly complexity snapshots, oldest first, each `{month, ref, captured_at, files, functions, mean_cyclomatic, median_cyclomatic, p90_cyclomatic, max_cyclomatic, functions_over_threshold, threshold, static_warnings, static_warnings_by_tool}`; omitted when no repo has a snapshot), the `/api/reviewer/accuracy` summary, and `n` plus `missing` counts (unknown-model verdicts, empty CI, unreadable records). Never consulted by merge decisions. | `pkg/dashboard/api.go:520` | +| `GET` | `/api/evidence/summary` | Dashboard auth/session | Read-only aggregate over `/data/evidence` bundles (schema v1) plus reviewer accuracy. Query `hours` (default 24, capped like `/api/pr-throughput`; `0` = all time) or a custom `from`/`to` range (RFC3339 or `YYYY-MM-DD`, date-only `to` inclusive, `to` defaults to now; `400` when `to` precedes `from`, the span exceeds 3 × 366 days, or `to` is given without `from`; the response echoes `from`/`to`), `repo` (`owner/name`) and `author_kind=human\|agent\|bot`; `400` for a bad value. A PR falls in the window by its open time (earliest bundle `created_at` or `merge.opened_at`), never by when a bundle was written; all bundles of a PR are folded so the newest supplies verdicts and the merge / `reverted_by` / backfilled close come from whichever record knows them. Returns `available_windows` (`{"24","168","720","all"}` → PR count under the same filters), `outcomes_by_author_kind` (merged/closed/open; `closed` comes from backfilled records, `reverted_14d` is null until bundles record it), `outcomes_by_author_family` and `outcomes_by_author_model` (the same outcome fields keyed by the author's model family and by canonical author model, `human` / `unknown` when no model was recorded; [#18720](https://github.com/hivecommons/hive/issues/18720)), `reviewer_calibration` buckets per perspective, `reviewer_accuracy_by_model` split by author kind (blocks split into `enforced_blocks` / `advisory_blocks` by `policy.require_approval`; advisory blocks that preceded a merge are `dissent_before_merge`, not false blocks), `sentinel_hits_per_week`, `shadow_gate` (`n`, `would_have_held`, `would_have_blocked`, and per gate `review_gate` / `intent` the evaluated `n` with `held` / `passed` outcomes `{n, merged, closed, open, reverted}`, plus `review_gate_by_lane` `{merge_eligible, self_merge}`), `write_counters` (per intent tier `{attempted, blocked, blocked_by_reason}` from the proxy write counter file; omitted when absent or unreadable), `write_activity` (blocked ÷ every agent write in the window: `total_writes`, `blocked`, `direct` `{recorded, since, partial_coverage, attempted, blocked, by_tier, blocked_by_reason}` from the proxy's weekly counter buckets under `/data/hive-metrics/write-counters.json`, `mediated` `{n, refused, by_action}` from the audit log), `claim_vs_verdict` (per ISO week of agent-authored PRs: `n`, `claimed`, `ci_captured`, `ci_not_green`, `verdict_not_approve`, `both`; records with empty CI are excluded from the CI column and disclosed under `missing.records_empty_ci`), `ci_recovery` (`policy` `{self_authored_automerge, required_checks}`, `n`, `at_claim` `{green, red, pending, none}`, `red` `{n, recovered_merged, green_open, abandoned, pending, outcome_unknown, enriched, not_enriched, median_pushes_to_green, median_hours_to_green, same_head_recovered}`, `gate` `{merged_n, with_checks, green, green_from_bundles, non_required_red, required_unknown, breaches, breach_prs[], suspected, suspected_prs[], pending, unknown, green_to_merge_n, median_green_to_merge_hours}` over every merged PR by GitHub-confirmed state of the *required* checks on the merged head, as the merge gate judges (`breach_prs[]` carry `red`, `merged_by`; a red non-required check is disclosed, not a breach; a red review-time snapshot, a required check with no run, or a red read without a resolvable required set is only suspected), and `convergence` `{n, passes {"1","2","3","4+","never"}, settled, in_flight, unknown, median_passes, from_github, from_bundles, by_week[]}` over agent PRs by CI passes to first green, from the durable `/data/hive-metrics/ci-recovery.json` GitHub check-run trail rollup), `work_balance` (per ISO week: `n`, `opened` by actor kind `{human, agent, bot}`, `closed`, `prs_opened`, `prs_merged`, `prs_closed`, `queue_issues`, `queue_prs`, `issue_source` `github|audit`, `throttled`) with `work_balance_sources` (`issues_github_since`, `issues_fetched_at`, `issues_known`, `audit_only_weeks`, `issue_series_weeks`, `queue_samples`, `queue_retention_days`; issues from the durable GitHub rollup `/data/hive-metrics/issue-activity.json` for weeks at or after `issues_github_since` and, for earlier weeks, from the backfilled issue series `///issues/.json` written by the issue backfill ([#12527](https://github.com/hivecommons/hive/pull/12527)) when its coverage spans the requested portion of the week (`issue_series_weeks` counts the repo-weeks it supplied), else the audit log (`audit_only_weeks`), PRs from folded bundles, queue depth from governor snapshots; `missing.issues_human_opens_unobserved` is true while no GitHub rollup exists unless the issue series covers every row, and whenever any row fell back to the audit log, `missing.throttle_windows_unavailable` discloses that no governor throttle windows are reported), `intent_counters` (per intent tier `{attempted, blocked}` from `/data/hive-metrics/intent-counters.json` — evaluations and the subset the intent gate would have blocked with enforcement on, [#12362](https://github.com/hivecommons/hive/issues/12362); omitted when absent or unreadable), `complexity_by_repo` (per managed repo the newest 12 monthly complexity snapshots, oldest first, each `{month, ref, captured_at, files, functions, mean_cyclomatic, median_cyclomatic, p90_cyclomatic, max_cyclomatic, functions_over_threshold, threshold, static_warnings, static_warnings_by_tool}`; omitted when no repo has a snapshot), the `/api/reviewer/accuracy` summary, and `n` plus `missing` counts (unknown-model verdicts, empty CI, unreadable records). Never consulted by merge decisions. | `pkg/dashboard/api.go:520` | | `GET` | `/api/benchmarks/export` | Owner only | Anonymized per-PR [benchmark records](../benchmarks.md) (schema `hivecommons/benchmarks` `result.schema.json`) built from the evidence bundles updated since `since` (RFC3339 or `YYYY-MM-DD`; default 90 days; `400` when malformed) plus `hive_hash`, `publish`, `generated_at` and `count`. `records` is empty when `benchmarks.publish: false`; repo/org names appear only for GitHub-public repos when `benchmarks.repo_consent` is on | `pkg/dashboard/api.go:521` | | `GET` | `/api/agents` | Dashboard auth/session | Agents List | `pkg/dashboard/api.go:305` | | `POST` | `/api/agents` | Owner only | Agent Create | `pkg/dashboard/api.go:306` | diff --git a/src/docs/review-evidence.md b/src/docs/review-evidence.md index 9324adc98e..221ec8204a 100644 --- a/src/docs/review-evidence.md +++ b/src/docs/review-evidence.md @@ -198,6 +198,7 @@ A PR's outcome (`pkg/evidence.PR.OutcomeKnown`) has three sources, folded per PR - **Window:** the summary takes `hours=` (default 24, `0` = all time, capped at 90 days) or a custom range `from=`/`to=` as RFC3339 or `YYYY-MM-DD` (a date-only `to` is inclusive; `to` defaults to now). `to` before `from`, a span over 3 × 366 days, or `to` without `from` is a `400`. A custom range echoes `from`/`to` in the response and keeps PRs opened in `[from, to)`. **A PR is placed in a window by its open time, never by when a bundle was written:** every bundle of the same PR is folded into one row — the newest record supplies verdicts, CI, sentinel and shadow observations, the earliest `created_at` (or `merge.opened_at`) is the open time, and the merge event, the change-failure links (`revert_of`, `followup_of`, `reverted_by`, `followed_up_by`) and a backfilled `backfill.state: closed` are taken from whichever record knows them. A live bundle is created when review starts and gains a merge event when the hive itself merged, or — when the bundle is started after the PR has already merged (automerge ran before the first review wrote one, or someone else merged) — carries the merge GitHub reports at creation (`method: unrecorded`, since the API says who and when, not how); a merge by another actor after the bundle exists is still unobserved until a backfill; a backfilled bundle ([#12371](https://github.com/hivecommons/hive/issues/12371)) is stamped with the PR's open time and the forge's outcome. Without the fold, a hive that reviews but does not merge (Bluefin) showed every reviewed PR as open and every approval as a false approve, and a review sweep over old PRs pulled them into the 24h window. The fold is `evidence.FoldPRs` in `pkg/evidence`; the [hub heartbeat evidence block](fleet-health.md#evidence-summary-heartbeat-12114) and [`hivectl evidence export`](#exporting) use the same one, so the three agree per PR ([#12516](https://github.com/hivecommons/hive/issues/12516)). The response also carries `available_windows` (`{"24", "168", "720", "all"}` → PR count under the same `repo`/`author_kind` filter) so the card can say "window too short — switch to 7d or 30d" instead of a bare "Not enough data". The Evidence card offers **24h · 7d · 30d · custom** pills (custom accepts backfilled history, up to 3 × 366 days) and remembers the choice per browser (`hive.evidence.window`); its header reads "Evidence · last 30d" or "Evidence · Oct 1 – Oct 15". - **Hub revert rates by author kind:** heartbeat evidence includes `by_author_kind`, a fixed `human` / `agent` / `bot` map of merged PR counts (`n`) and linked revert counts (`bad`) over the newest bundle per PR in the 90-day creation window. Follow-up links do not count as reverts. The owner-only hub evidence drill-down pools these counts into `revert_rate_by_author_kind`, displaying the sample size and 95% Wilson interval (null rate below 10 merges). Older spokes without the map contribute no author-kind samples. Revert links are best-effort, within 14 days of merge; recent merges may not yet have a full observation window. These rates are descriptive, not merge gates. Each hive row's `counts.by_author_kind` carries the same buckets so a hive or ACMM-level selection re-pools them in the browser. - **Outcome by author kind (panel 1):** `outcomes_by_author_kind` in the summary counts, over the folded record per PR in the window, `n`, `merged`, `closed`, `open`, `unknown`, `reverted_14d` (merged and then linked by `reverted_by` within 14 days), `issue_reopened_14d` (merged and then an issue it closed was reopened within 14 days, from `issue_reopens`), `change_failures` (the two added; a PR both reverted and re-opened counts twice) and `change_failure_rate` (change failures ÷ merged, `null` under 1 merge) for `human`, `agent` and `bot` authors ([#13241](https://github.com/hivecommons/hive/issues/13241)). The Evidence sub-panel draws it as the **Outcome by author kind** panel: one grouped bar per kind for merged / closed / undone (reverted or re-opened) plus a hollow bar for outcome unknown, with `n=`, the merged rate and the split of undone in the tooltip. **Denominator policy:** merged rate = merged ÷ PRs with a known outcome; change-failure rate = (reverted + re-opened) ÷ merged. This is the change-failure proxy for "a PR being accepted is not the same as the project being better off": it asks whether agent-written work gets undone more often than human work. Both signals are detected from data already fetched (see [Change-failure links](#change-failure-links)); reverts are only linked since 2026-10-15 and re-opens only since 2026-10-20. `merged_elsewhere` is the subset of `merged` that someone other than this hive performed, learned from the review-outcome ledger; `closed` and `open` come from a backfilled record of the PR or from the ledger (see [Where outcomes come from](#where-outcomes-come-from)). A PR with no merge event, no backfilled record and no ledger row has **no recorded outcome** — it is `unknown` (`pkg/evidence.PR.OutcomeKnown`), counted and shown, and excluded from every rate rather than counted as open or as not merged. The summary's `missing.prs_outcome_unknown` gives the window total and `missing.prs_outcome_from_ledger` how many the ledger resolved. A revert only counts once the reverting PR is linked. Reporting only ([#12364](https://github.com/hivecommons/hive/issues/12364)). +- **Change failures by author model:** `outcomes_by_author_model` has the same fields as `outcomes_by_author_kind`, keyed by the PR author's canonical model (`modelfamily.Canonical` of the recorded author model, the ids `GET /api/governor/pr-models` uses, so `claude-opus-5-5` and `anthropic/claude-opus-5-5` both count under `claude-opus-5.5`). A person's PR is under `human` and an agent PR with no recorded model under `unknown`, as in `outcomes_by_author_family`, so the per-model and per-family maps add up to the same totals. It reuses the same revert and re-open detection; nothing new is detected. The **Outcome by author kind** panel shows it as a table under the bars (model, `n`, merged, reverted, re-opened, change-failure rate), most PRs first. `n` counts only PRs with an evidence record, so it can differ from the PRs-by-model card; read a rate on a small `n` as weak evidence ([#18720](https://github.com/hivecommons/hive/issues/18720)). - **Maintainability trend (panel 7):** the Evidence sub-panel draws `complexity_by_repo` as three per-repo lanes on separate scales — median Go cyclomatic complexity, functions over the threshold, and warnings by tool (dashed, one dash pattern per tool) — with the first and last month on the axis. Snapshots whose month falls in the selected window are kept, boundary months included, and `n=` is the number of repos with at least two snapshots in that window; a repo with one snapshot is drawn as points. Missing months and missing tool measurements stay gaps, never zero. Vertical dotted **ACMM markers** come from `acmm_changes`: for each repo and level, the month of the earliest stored bundle whose `policy.acmm_level` is that level (1 … `acmmadvisor.MaxLevel`). Unlike the other aggregates they are computed over **all** stored bundles for the repo — not the latest bundle per PR, not the `author_kind` filter, not the `hours`/`from`/`to` window — because policy maturity is a repo property, and the chart clips them to the months it shows. A marker is the first observation, not the moment the setting changed, and a return to an earlier level is not shown. The panel is deliberately **not graded**: its explainer ends in a **Not graded** badge (grade key `ungraded`, "report-only; no threshold to grade against") rather than a Strong/Fair/Weak/Not enough data rating, because complexity rising or falling is an observation with no causal claim about agents, so there is no threshold to hold it to. It shows an empty state while no repo has a monthly snapshot. Reporting only ([#12369](https://github.com/hivecommons/hive/issues/12369)). - **Confidence series:** the self-reported / derived toggle selects `reviewer_calibration` or `reviewer_calibration_derived`, with a separate n for each. Old bundles without `confidence_derived` are excluded from the derived series; `reviewer_calibration_legacy` discloses self-series verdicts still using the historical fallback. The plain summary compares the pooled endpoint rise in merged share per bin, only across bins with at least five known outcomes and at least two such bins per series. A falling intermediate bucket disqualifies monotonicity; sparse data produces no winner. The comparison is descriptive, uses each series’ available cohort, and does not establish causation or correctness. - **Reviewer calibration and accuracy (panels 2–3):** `reviewer_calibration` buckets each verdict by its stated confidence per perspective with `n`, `merged`, `blocks` and `unknown` (verdicts whose PR has no recorded outcome). **Denominator policy:** merged rate = merged ÷ (`n` − `unknown`); block rate = blocks ÷ `n` (a block is known at verdict time). The card draws a bucket with fewer than `EV_CALIB_MIN_BUCKET_N` (5) known-outcome verdicts hollow and leaves it out of the joined line, because one or two verdicts invert a line without meaning anything. `reviewer_accuracy_by_model` carries `approvals`, `approved_not_merged`, `approved_unknown`, `blocks`, `merged_after_block` and `blocked_unknown` per model family × author kind, with blocks split by the bundle's `policy.require_approval` into `enforced_blocks` (`merged_after_enforced_block`, `enforced_blocked_unknown`) and `advisory_blocks` (`dissent_before_merge`, `advisory_blocked_unknown`) — no row is dropped and the unsplit totals stay ([#12595](https://github.com/hivecommons/hive/issues/12595)): false-approve rate = approvals on PRs **known** not to have merged (closed without merge, or a backfilled record still open) ÷ approvals on PRs with a known outcome; false-block rate = **enforced** blocks on PRs that merged ÷ enforced blocks on PRs with a known outcome; an advisory block on a PR that merged is dissent before merge, shown as a count and a share of advisory blocks and never as an error rate (when every block in the window was advisory the card reads "Perspective verdicts were advisory on this hive (review.require_approval off); n blocks preceded a merge."); the unknown counts are shown beside each rate. Nothing is dropped silently and nothing unknown is counted as a failure. One bias is stated on the card because it cannot be removed by policy: on a live-only hive the only way an outcome becomes known is a merge this hive performed (or, since the merge-at-start fix, a merge GitHub already reported when the bundle began), so until a backfill covers the window "merged ÷ known" is an upper bound and "false-approve ÷ known" a lower bound; the unknown count is the honest measure of what is unresolved. diff --git a/src/pkg/dashboard/api_evidence_family_test.go b/src/pkg/dashboard/api_evidence_family_test.go index 0a147e06e0..e2daf3d3f4 100644 --- a/src/pkg/dashboard/api_evidence_family_test.go +++ b/src/pkg/dashboard/api_evidence_family_test.go @@ -1,6 +1,7 @@ package dashboard import ( + "encoding/json" "strings" "testing" "time" @@ -116,3 +117,155 @@ func TestEvidenceAccuracyCardFamilyBreakdown(t *testing.T) { t.Errorf("family breakdown must not change the grade: %+v", acc.Grade) } } + +// TestBuildEvidenceSummaryAuthorModelBuckets pins outcomes_by_author_model +// (hivecommons/hive#18720): change failures per canonical author model, +// with raw spellings folded and "human" / "unknown" bucketed as the family +// map does, so the per-model and per-family maps sum to the same totals. +func TestBuildEvidenceSummaryAuthorModelBuckets(t *testing.T) { + now := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC) + recent := now.Add(-2 * time.Hour) + mergedAt := now.Add(-time.Hour) + agentPR := func(n int, model string, merged bool) *evidence.Bundle { + b := evSummaryBundle("a/b", n, evidence.AuthorAgent, recent, merged, nil, 0, true) + b.Author.Model = model + if merged { + b.Merge.At = mergedAt + } + return b + } + // Two models of the same family, each spelled more than one way. + opusReverted := agentPR(1, "claude-opus-5-5", true) + opusReverted.RevertedBy = []string{"a/b#90@merged"} + opusClean := agentPR(2, "anthropic/claude-opus-5-5", true) + opusOpen := agentPR(3, "claude-opus-5.5", false) + sonnetReopened := agentPR(4, "claude-sonnet-5.5", true) + sonnetReopened.IssueReopens = []evidence.IssueReopen{{Issue: 7, At: mergedAt.Add(time.Hour), Source: evidence.IssueReopenSourceClaimLedger}} + gpt := agentPR(5, "openai-codex/gpt-6.1-terra:high", true) + human := evSummaryBundle("a/b", 6, evidence.AuthorHuman, recent, true, nil, 0, true) + human.Merge.At = mergedAt + human.RevertedBy = []string{"a/b#91@merged"} + noModel := agentPR(7, "", true) + + win := evidenceWindow{Hours: 24, From: now.Add(-24 * time.Hour), To: now} + got := buildEvidenceSummaryWindow([]*evidence.Bundle{opusReverted, opusClean, opusOpen, sonnetReopened, gpt, human, noModel}, win, "", "") + + type want struct{ n, merged, reverted, reopened int } + cases := map[string]want{ + "claude-opus-5.5": {n: 3, merged: 2, reverted: 1}, + "claude-sonnet-5.5": {n: 1, merged: 1, reopened: 1}, + "gpt-6.1-terra": {n: 1, merged: 1}, + "human": {n: 1, merged: 1, reverted: 1}, + "unknown": {n: 1, merged: 1}, + } + if len(got.OutcomesByAuthorModel) != len(cases) { + t.Errorf("model keys = %v, want %d keys", got.OutcomesByAuthorModel, len(cases)) + } + for key, w := range cases { + o, ok := got.OutcomesByAuthorModel[key] + if !ok { + t.Errorf("missing model key %q in %v", key, got.OutcomesByAuthorModel) + continue + } + if o.RevertedWithin14d == nil { + t.Errorf("%s: reverted_14d is nil after finishing", key) + continue + } + if o.N != w.n || o.Merged != w.merged || *o.RevertedWithin14d != w.reverted || o.IssueReopenedWithin14d != w.reopened || o.ChangeFailures != w.reverted+w.reopened { + t.Errorf("%s = %+v (reverted %d), want %+v", key, o, *o.RevertedWithin14d, w) + } + } + if r := got.OutcomesByAuthorModel["claude-opus-5.5"].ChangeFailureRate; r == nil || *r != 0.5 { + t.Errorf("claude-opus-5.5 change_failure_rate = %v, want 0.5", r) + } + if r := got.OutcomesByAuthorModel["gpt-6.1-terra"].ChangeFailureRate; r == nil || *r != 0 { + t.Errorf("gpt-6.1-terra change_failure_rate = %v, want 0", r) + } + + // Summing over model keys equals summing over family keys, field by field. + sum := func(m map[string]EvidenceOutcomes) EvidenceOutcomes { + var tot EvidenceOutcomes + reverted := 0 + for _, o := range m { + tot.N += o.N + tot.Merged += o.Merged + tot.MergedElsewhere += o.MergedElsewhere + tot.Closed += o.Closed + tot.Open += o.Open + tot.Unknown += o.Unknown + tot.IssueReopenedWithin14d += o.IssueReopenedWithin14d + tot.ChangeFailures += o.ChangeFailures + if o.RevertedWithin14d != nil { + reverted += *o.RevertedWithin14d + } + } + tot.RevertedWithin14d = &reverted + return tot + } + byModel, byFamily := sum(got.OutcomesByAuthorModel), sum(got.OutcomesByAuthorFamily) + if *byModel.RevertedWithin14d != *byFamily.RevertedWithin14d { + t.Errorf("reverted_14d: models %d, families %d", *byModel.RevertedWithin14d, *byFamily.RevertedWithin14d) + } + byModel.RevertedWithin14d, byFamily.RevertedWithin14d = nil, nil + if byModel != byFamily { + t.Errorf("model sums %+v != family sums %+v", byModel, byFamily) + } + if byModel.N != got.N { + t.Errorf("model sums n = %d, want %d", byModel.N, got.N) + } + + // Existing maps are unchanged by the new grouping. modelfamily.Family + // derives the family from the provider-stripped id (#18803), so + // "anthropic/claude-opus-5-5" joins the other Claude PRs and + // "openai-codex/gpt-6.1-terra:high" is "gpt"; nothing lands in "other". + if o := got.OutcomesByAuthorFamily["claude"]; o.N != 4 || o.Merged != 3 || o.Unknown != 1 || o.ChangeFailures != 2 { + t.Errorf("claude family = %+v", o) + } + if o := got.OutcomesByAuthorFamily["gpt"]; o.N != 1 || o.Merged != 1 || o.ChangeFailures != 0 { + t.Errorf("gpt family = %+v", o) + } + if o, ok := got.OutcomesByAuthorFamily["other"]; ok && o.N != 0 { + t.Errorf("other family = %+v, want empty", o) + } + if o := got.Outcomes["agent"]; o.N != 6 || o.Merged != 5 || o.ChangeFailures != 2 { + t.Errorf("agent kind = %+v", o) + } + + raw, err := json.Marshal(got) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(raw), `"outcomes_by_author_model":{`) { + t.Errorf("JSON lacks outcomes_by_author_model: %s", raw) + } + // An empty window still serialises an empty object, never null. + empty := buildEvidenceSummaryWindow(nil, win, "", "") + if empty.OutcomesByAuthorModel == nil || len(empty.OutcomesByAuthorModel) != 0 { + t.Errorf("empty window OutcomesByAuthorModel = %v", empty.OutcomesByAuthorModel) + } +} + +func TestEvidenceAuthorModel(t *testing.T) { + for _, tc := range []struct { + author evidence.Author + want string + }{ + {evidence.Author{Kind: evidence.AuthorAgent, Model: "claude-opus-5-5"}, "claude-opus-5.5"}, + {evidence.Author{Kind: evidence.AuthorAgent, Model: "anthropic/claude-opus-5-5"}, "claude-opus-5.5"}, + {evidence.Author{Kind: evidence.AuthorAgent, Model: " Claude-Opus-5.5 "}, "claude-opus-5.5"}, + {evidence.Author{Kind: evidence.AuthorAgent, Model: "gpt-6-astra"}, "gpt-6-astra"}, + {evidence.Author{Kind: evidence.AuthorAgent}, "unknown"}, + {evidence.Author{Kind: evidence.AuthorBot}, "unknown"}, + {evidence.Author{Kind: evidence.AuthorHuman}, "human"}, + {evidence.Author{}, "unknown"}, + } { + if got := evidenceAuthorModel(tc.author); got != tc.want { + t.Errorf("evidenceAuthorModel(%+v) = %q, want %q", tc.author, got, tc.want) + } + // Every author lands in one model key and one family key; a + // model-less author shares the family's human/unknown key. + if strings.TrimSpace(tc.author.Model) == "" && evidenceAuthorFamily(tc.author) != tc.want { + t.Errorf("model-less %+v: model key %q, family key %q", tc.author, tc.want, evidenceAuthorFamily(tc.author)) + } + } +} diff --git a/src/pkg/dashboard/api_evidence_summary.go b/src/pkg/dashboard/api_evidence_summary.go index 4683768a54..6a145dbb8c 100644 --- a/src/pkg/dashboard/api_evidence_summary.go +++ b/src/pkg/dashboard/api_evidence_summary.go @@ -15,6 +15,7 @@ import ( "github.com/hivecommons/hive/pkg/evidence" "github.com/hivecommons/hive/pkg/evidence/complexity" ghpkg "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/modelfamily" "github.com/hivecommons/hive/pkg/review" ) @@ -60,6 +61,12 @@ type EvidenceSummary struct { SentinelPerWeek EvidenceSentinelSeries `json:"sentinel_hits_per_week"` ShadowGate EvidenceShadowGate `json:"shadow_gate"` ReviewerAccuracy *review.ReviewerAccuracySummary `json:"reviewer_accuracy,omitempty"` + // OutcomesByAuthorModel is Outcomes keyed by the author's canonical model + // (modelfamily.Canonical, the ids /api/governor/pr-models uses), so + // change failures can be compared per model rather than per family. + // "human" and "unknown" are bucketed as in OutcomesByAuthorFamily — + // hivecommons/hive#18720. + OutcomesByAuthorModel map[string]EvidenceOutcomes `json:"outcomes_by_author_model"` // WriteCounters is the proxy's attempted-vs-blocked agent write totals per // intent tier (#12069); omitted when the counter file is absent or unreadable. WriteCounters map[string]EvidenceWriteCount `json:"write_counters,omitempty"` @@ -428,6 +435,17 @@ func evidenceAuthorFamily(a evidence.Author) string { return review.AuthorFamilyBucket(a.Model, a.Kind != "", hiveAuthored) } +// evidenceAuthorModel is the per-model bucket for a bundle's author: the +// canonical model id when one was recorded, otherwise the same "human" / +// "unknown" bucket evidenceAuthorFamily uses, so every PR lands in exactly +// one key of each map. +func evidenceAuthorModel(a evidence.Author) string { + if strings.TrimSpace(a.Model) != "" { + return modelfamily.Canonical(a.Model) + } + return evidenceAuthorFamily(a) +} + // evidenceVerdictFamilyMatch prefers the verdict's stored same_family (set // when the bundle was written) and falls back to deriving it, so bundles // written before the field existed are still classified. @@ -907,6 +925,7 @@ func buildEvidenceSummaryWindowHints(bundles []*evidence.Bundle, hints map[strin CalibrationDerived: []EvidencePerspectiveCalib{}, OutcomesByAuthorFamily: map[string]EvidenceOutcomes{}, + OutcomesByAuthorModel: map[string]EvidenceOutcomes{}, ReviewerModels: []EvidenceModelAccuracy{}, SentinelPerWeek: EvidenceSentinelSeries{Weeks: []EvidenceSentinelWeek{}}, ClaimVsVerdict: []EvidenceClaimWeek{}, @@ -950,6 +969,10 @@ func buildEvidenceSummaryWindowHints(bundles []*evidence.Bundle, hints map[strin fo := resp.OutcomesByAuthorFamily[family] fo.add(b) resp.OutcomesByAuthorFamily[family] = fo + authorModel := evidenceAuthorModel(b.Author) + mo := resp.OutcomesByAuthorModel[authorModel] + mo.add(b) + resp.OutcomesByAuthorModel[authorModel] = mo if !merged && !b.Closed && !known { resp.Missing.PRsOutcomeUnknown++ } @@ -1139,5 +1162,9 @@ func buildEvidenceSummaryWindowHints(bundles []*evidence.Bundle, hints map[strin o.finishChangeFailures() resp.OutcomesByAuthorFamily[k] = o } + for k, o := range resp.OutcomesByAuthorModel { + o.finishChangeFailures() + resp.OutcomesByAuthorModel[k] = o + } return resp } diff --git a/src/pkg/dashboard/evidence_card_ui_test.go b/src/pkg/dashboard/evidence_card_ui_test.go index c3b6a173cf..3028ed943b 100644 --- a/src/pkg/dashboard/evidence_card_ui_test.go +++ b/src/pkg/dashboard/evidence_card_ui_test.go @@ -67,7 +67,9 @@ func TestEvidenceCardInThroughputSection(t *testing.T) { ".ev-svg .ev-bar.sr", // Panel 1 (#12364): outcome by author kind. "dto.outcomes_by_author_kind || {}", - "evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evControlArmHtml(dto.control_arm),", + "evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evOutcomesByModelHtml(dto.outcomes_by_author_model) + evControlArmHtml(dto.control_arm),", + // #18720: change failures per canonical author model, same panel. + "function evOutcomesByModelHtml(byModel) {", // Issue reopen rate shares the panel chrome and compact table styling. "return evPanel('issue-reopen', 'Issue reopen rate · rolling 30 days', closedN, body,", `
`, @@ -254,7 +256,7 @@ func TestEvidenceMaintainabilityRendering(t *testing.T) { source.WriteString(jsFunc(t, html, "formatNumber")) source.WriteByte('\n') source.WriteString("const EV_PALETTE_SIZE = 6, EV_SVG_W = 320, EV_TOP_REASONS = 3;\nconst EV_WRITE_TIERS = ['ADVISORY', 'ISSUES_ONLY', 'ISSUES_AND_PRS', 'ISSUES_PRS_MERGE'];\nlet evLast = null;\nlet evConfidenceSeries = 'self-reported';\nconst evWindowLabel = () => 'all time';\nconst evWindowControls = () => '';\nconst evExplainerHtml = () => ({ button: '', body: '' });\nconst EV_CALIB_MIN_BUCKET_N = 5, EV_GRADE_MIN_N = 20;\nconst evDetails = {};\nconst EV_OM_DRAIN_MAX = 1, EV_OM_SCOUT_MIN = 1, EV_OM_SCOUT_MAX = 2, EV_OM_PARITY_TOLERANCE = 0.25, EV_OM_ROLLING_DAYS = 30, EV_OM_SPARK_TRAIL_DAYS = 7, EV_OM_BAR_MAX_RATIO = 3, EV_OM_SPARK_MAX_RATIO = 3;\nconst EV_OM_STATUS = { in_band: 'in band', above: 'backlog growing', below: 'scouting below cadence', no_data: 'no data' };\n") - for _, name := range []string{"escapeHtml", "evRate", "evPanel", "evServerCard", "evCalibrationSeries", "evCalibrationControls", "evSelectConfidence", "evOutcomesSvg", "evControlArmHtml", "evCalibrationSvg", "evAccuracySvg", "evShadowGateSvg", "evClaimVsVerdictSvg", "evWriteActivitySvg", "evCIGateHtml", "evCIRecoveryHtml", "evGateGrade", "evConvergenceGrade", "evConvergenceSvg", "evPct", "evWindowHint", "evSentinelSparkline", "evWorkBalanceSvg", "evMaintainabilityRows", "evMaintainabilitySvg", "evGradeLine", "evIssueReopenRatesHtml", "evOmStatusLabel", "evOmRatioText", "evOmVerdict", "evOmSparkline", "evOpensMergesHtml", "renderEvidenceSummary"} { + for _, name := range []string{"escapeHtml", "evRate", "evPanel", "evServerCard", "evCalibrationSeries", "evCalibrationControls", "evSelectConfidence", "evOutcomesSvg", "evOutcomesByModelHtml", "evControlArmHtml", "evCalibrationSvg", "evAccuracySvg", "evShadowGateSvg", "evClaimVsVerdictSvg", "evWriteActivitySvg", "evCIGateHtml", "evCIRecoveryHtml", "evGateGrade", "evConvergenceGrade", "evConvergenceSvg", "evPct", "evWindowHint", "evSentinelSparkline", "evWorkBalanceSvg", "evMaintainabilityRows", "evMaintainabilitySvg", "evGradeLine", "evIssueReopenRatesHtml", "evOmStatusLabel", "evOmRatioText", "evOmVerdict", "evOmSparkline", "evOpensMergesHtml", "renderEvidenceSummary"} { source.WriteString(jsFunc(t, html, name)) source.WriteByte('\n') } @@ -421,6 +423,7 @@ func TestEvidenceCardUIHasNoUndefinedCallees(t *testing.T) { "evPanel", "evRate", "evOutcomesSvg", + "evOutcomesByModelHtml", "evControlArmHtml", "evCalibrationSvg", "evAccuracySvg", diff --git a/src/pkg/dashboard/evidence_model_outcomes.test.js b/src/pkg/dashboard/evidence_model_outcomes.test.js new file mode 100644 index 0000000000..dffea197e2 --- /dev/null +++ b/src/pkg/dashboard/evidence_model_outcomes.test.js @@ -0,0 +1,71 @@ +// Evidence card: change failures per author model (hivecommons/hive#18720). +// Runs evOutcomesByModelHtml from the built index.html against a fixture +// shaped like GET /api/evidence/summary's outcomes_by_author_model. +const test = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs'); +const path = require('node:path'); + +const html = fs.readFileSync(path.join(__dirname, 'static', 'index.html'), 'utf8'); + +function fn(name) { + const re = new RegExp('\\n (?:async )?function ' + name + '\\([^)]*\\) \\{\\n[\\s\\S]*?\\n \\}\\n'); + const m = html.match(re); + assert.ok(m, name + ' not found'); + return m[0]; +} + +function oneLine(name) { + const m = html.match(new RegExp('\\n function ' + name + '\\([^)]*\\) \\{[^\\n]*\\}\\n')); + assert.ok(m, name + ' not found'); + return m[0]; +} + +function load() { + const src = [ + fn('escapeHtml'), + fn('evRate'), + oneLine('evPct'), + fn('evOutcomesByModelHtml'), + 'return evOutcomesByModelHtml;', + ].join('\n'); + return new Function(src)(); +} + +const row = (n, merged, reverted, reopened, rate) => ({ n, merged, reverted_14d: reverted, issue_reopened_14d: reopened, change_failures: reverted + reopened, change_failure_rate: rate }); + +test('renders one row per model with n, merged, reverted, re-opened and rate', () => { + const render = load(); + const out = render({ + 'claude-opus-5.5': row(3, 2, 1, 0, 0.5), + 'claude-sonnet-5.5': row(1, 1, 0, 1, 1), + human: row(1, 1, 1, 0, 1), + unknown: row(1, 0, 0, 0, null), + 'gpt-6.1-terra': row(2, 2, 0, 0, 0), + }); + assert.match(out, /
/); + assert.match(out, /`; + }).join(''); + return `
Change failures by author model: reverted or issue re-opened within 14 d of merge, ÷ merged. n counts only PRs with an evidence record, so it can differ from PRs by model; a rate on a small n is weak evidence.
Author model<\/th>n<\/th>Merged<\/th>Reverted<\/th>Re-opened<\/th>Change failure<\/th>/); + assert.match(out, /
claude-opus-5\.5<\/td>3<\/td>2<\/td>1<\/td>0<\/td>50%<\/td><\/tr>/); + assert.match(out, /
claude-sonnet-5\.5<\/td>1<\/td>1<\/td>0<\/td>1<\/td>100%<\/td><\/tr>/); + assert.match(out, /
gpt-6\.1-terra<\/td>2<\/td>2<\/td>0<\/td>0<\/td>0%<\/td><\/tr>/); + // No merges: no rate, a dash rather than 0%. + assert.match(out, /
unknown \(model not recorded\)<\/td>1<\/td>0<\/td>0<\/td>0<\/td>—<\/td><\/tr>/); + assert.match(out, /
human \(no model\)<\/td>/); + // Biggest n first; human and unknown trail. + const order = [...out.matchAll(/
([^<]+)<\/td>/g)].map((m) => m[1]); + assert.deepStrictEqual(order, ['claude-opus-5.5', 'gpt-6.1-terra', 'claude-sonnet-5.5', 'human (no model)', 'unknown (model not recorded)']); + // Coverage caveat is visible next to the table. + assert.match(out, /n counts only PRs with an evidence record/); + assert.doesNotMatch(out, /NaN|undefined/); +}); + +test('escapes model ids and renders nothing for an absent or empty map', () => { + const render = load(); + const out = render({ '': row(1, 1, 0, 0, 0) }); + assert.doesNotMatch(out, / Number(byModel[k] && byModel[k].n) > 0); + if (!keys.length) return ''; + keys.sort((a, b) => (tail.indexOf(a) - tail.indexOf(b)) || ((Number(byModel[b].n) || 0) - (Number(byModel[a].n) || 0)) || a.localeCompare(b)); + const rows = keys.map((k) => { + const o = byModel[k] || {}; + const merged = Number(o.merged) || 0, rv = Number(o.reverted_14d) || 0, ro = Number(o.issue_reopened_14d) || 0; + const rate = o.change_failure_rate == null ? null : Number(o.change_failure_rate); + const label = k === 'human' ? 'human (no model)' : k === 'unknown' ? 'unknown (model not recorded)' : k; + return `
${escapeHtml(label)}${Number(o.n) || 0}${merged}${rv}${ro}${evPct(rate)}
${rows}
Author modelnMergedRevertedRe-openedChange failure
`; + } + function evOutcomesSvg(outcomes) { outcomes = outcomes || {}; const kinds = EV_AUTHOR_KINDS.concat(Object.keys(outcomes).filter((k) => !EV_AUTHOR_KINDS.includes(k)).sort()); @@ -1158,7 +1179,7 @@ const windowHint = Number(dto.n) > 0 ? '' : evWindowHint(dto); card.innerHTML = `
🔬 Evidence · ${escapeHtml(scope)}n=${Number(dto.n) || 0} PRs
${evWindowControls()}
` + (windowHint ? `
No PRs opened in this window${escapeHtml(windowHint)}.
` : '') - + evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evControlArmHtml(dto.control_arm), + + evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evOutcomesByModelHtml(dto.outcomes_by_author_model) + evControlArmHtml(dto.control_arm), `PRs are placed in the window by their open time, never by when a record was written. Undone counts two change-failure signals, each within 14 d of merge: reverted means a reverted_by link from a later merged PR whose title or body named this PR or its commit (Reverts #N, Revert "…", This reverts commit ) — a revert that names nothing is missed; re-opened means an issue this PR closed was re-opened by a person after the merge, as seen by the issues webhook or the issue listing — an issue closed by hand rather than by a closing keyword is not tracked. A live bundle records a merge only when this hive merged, and never a close-without-merge; PRs reviewed here and then merged by someone else, closed by hand or left open are resolved from the governor's review-outcome ledger instead${outcomeLedger ? ` (${outcomeLedger} PR(s) in this window)` : ''}: ${outcomeElsewhere} merged elsewhere · ${outcomeClosed} closed without merge · ${outcomeOpen} open. A PR the ledger never saw${outcomeUnknown ? ` (${outcomeUnknown} PR(s) in this window)` : ''} is shown as unknown and excluded from every rate, not counted as open or as not merged. Merged is not the same as correct.`) + evIssueReopenRatesHtml(dto) + evPanel('calibration', 'Reviewer calibration', calibN, evCalibrationControls() + evCalibrationSvg(calib), diff --git a/src/pkg/dashboard/static/index.html b/src/pkg/dashboard/static/index.html index 0777b6aece..05da3dccac 100644 --- a/src/pkg/dashboard/static/index.html +++ b/src/pkg/dashboard/static/index.html @@ -43255,6 +43255,27 @@

All labels

}).join(''); } + // Change failures per canonical author model (#18720), read from + // outcomes_by_author_model. Most PRs first; the person and no-model + // buckets trail. n is every PR with an evidence record for that model, + // so a small sample is visible next to its rate. Absent on an older + // server: nothing is rendered. + function evOutcomesByModelHtml(byModel) { + byModel = byModel && typeof byModel === 'object' ? byModel : {}; + const tail = ['human', 'unknown']; + const keys = Object.keys(byModel).filter((k) => Number(byModel[k] && byModel[k].n) > 0); + if (!keys.length) return ''; + keys.sort((a, b) => (tail.indexOf(a) - tail.indexOf(b)) || ((Number(byModel[b].n) || 0) - (Number(byModel[a].n) || 0)) || a.localeCompare(b)); + const rows = keys.map((k) => { + const o = byModel[k] || {}; + const merged = Number(o.merged) || 0, rv = Number(o.reverted_14d) || 0, ro = Number(o.issue_reopened_14d) || 0; + const rate = o.change_failure_rate == null ? null : Number(o.change_failure_rate); + const label = k === 'human' ? 'human (no model)' : k === 'unknown' ? 'unknown (model not recorded)' : k; + return `${escapeHtml(label)}${Number(o.n) || 0}${merged}${rv}${ro}${evPct(rate)}`; + }).join(''); + return `
Change failures by author model: reverted or issue re-opened within 14 d of merge, ÷ merged. n counts only PRs with an evidence record, so it can differ from PRs by model; a rate on a small n is weak evidence.
${rows}
Author modelnMergedRevertedRe-openedChange failure
`; + } + function evOutcomesSvg(outcomes) { outcomes = outcomes || {}; const kinds = EV_AUTHOR_KINDS.concat(Object.keys(outcomes).filter((k) => !EV_AUTHOR_KINDS.includes(k)).sort()); @@ -43864,7 +43885,7 @@

All labels

const windowHint = Number(dto.n) > 0 ? '' : evWindowHint(dto); card.innerHTML = `
🔬 Evidence · ${escapeHtml(scope)}n=${Number(dto.n) || 0} PRs
${evWindowControls()}
` + (windowHint ? `
No PRs opened in this window${escapeHtml(windowHint)}.
` : '') - + evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evControlArmHtml(dto.control_arm), + + evPanel('outcome-by-author', 'Outcome by author kind', outcomeN, evOutcomesSvg(outcomes) + evOutcomesByModelHtml(dto.outcomes_by_author_model) + evControlArmHtml(dto.control_arm), `PRs are placed in the window by their open time, never by when a record was written. Undone counts two change-failure signals, each within 14 d of merge: reverted means a reverted_by link from a later merged PR whose title or body named this PR or its commit (Reverts #N, Revert "…", This reverts commit ) — a revert that names nothing is missed; re-opened means an issue this PR closed was re-opened by a person after the merge, as seen by the issues webhook or the issue listing — an issue closed by hand rather than by a closing keyword is not tracked. A live bundle records a merge only when this hive merged, and never a close-without-merge; PRs reviewed here and then merged by someone else, closed by hand or left open are resolved from the governor's review-outcome ledger instead${outcomeLedger ? ` (${outcomeLedger} PR(s) in this window)` : ''}: ${outcomeElsewhere} merged elsewhere · ${outcomeClosed} closed without merge · ${outcomeOpen} open. A PR the ledger never saw${outcomeUnknown ? ` (${outcomeUnknown} PR(s) in this window)` : ''} is shown as unknown and excluded from every rate, not counted as open or as not merged. Merged is not the same as correct.`) + evIssueReopenRatesHtml(dto) + evPanel('calibration', 'Reviewer calibration', calibN, evCalibrationControls() + evCalibrationSvg(calib),