Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-boundary-enforcement-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Report boundary enforcement as demonstrated when observed writes were blocked, not a refusal-rate grade; keep zero, Sentinel detections, and coverage limitations informational.
6 changes: 4 additions & 2 deletions src/docs/review-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,8 @@ The ⓘ button opens a popover over the chart (never inline) with the plain-lang

### Grades and what they mean

Boundary enforcement is deliberately **not graded**, including in the legacy client fallback. Its status is **Enforcement demonstrated** when at least one actual refusal is observed, even below 20 writes, and **Enforcement not observed** otherwise. This is positive evidence of enforcement on those attempts, not proof that bypasses or false positives are absent. Refusal frequency describes attempted-agent compliance, not enforcement effectiveness; increasing prohibited attempts is not a goal. Sentinel detections are informational, never counted as proven blocked writes. Counts and denominator coverage remain visible; missing direct counters and partial coverage remain explicit.

A grade is a letter from **A+** to **C-** (A band green, B band amber, C band red) or **—** (grey, dashed) when the panel does not have enough data to be graded. Hovering the pill shows the reason ("86% of 22 PRs with a known outcome merged; 1 of 19 merged was reverted"); the popover adds the rubric. Where a rubric produces a 0–1 score the letter is: A+ ≥ 0.97, A ≥ 0.93, A- ≥ 0.90, B+ ≥ 0.87, B ≥ 0.83, B- ≥ 0.80, C+ ≥ 0.77, C ≥ 0.73, otherwise C-. Every rubric needs at least `GradeMinN` (20) samples of its denominator, else **—** with a reason that names what is missing. Rubrics are pure functions in `pkg/evidence/grade.go`, each table-tested; change the table and the code together.

| Panel | Needs | Rubric |
Expand All @@ -279,7 +281,7 @@ A grade is a letter from **A+** to **C-** (A band green, B band amber, C band re
| Reviewer calibration | 20 verdicts with a known outcome in the selected series’ 0.95 bucket | score = merged share in the top bucket, minus 0.10 when the merged share is not non-decreasing across buckets that hold ≥ 5 known outcomes |
| Reviewer accuracy × model family | the rework-based outcome ledger (`pkg/review/accuracy.go`) with ≥ 20 samples | score = 1 − max(false-approve rate, false-block rate) summed over reviewer models; **—** ("needs the reviewer outcome ledger") until the ledger exists — merge-vs-close is never used as ground truth |
| Maintainability trend | one repo with ≥ 2 monthly snapshots in the window | first→last change of median cyclomatic and of functions over threshold, summed across repos: neither worse → A- / A / A+ by how many improved (0 / 1 / 2); one worse → B (B+ if the other improved); both worse → C |
| Boundary enforcement | 20 agent writes (hive-mediated + direct through the proxy) | blocked ÷ writes: 0 blocked → A- ("boundary not yet exercised"); ≤ 1% A+, ≤ 2% A, ≤ 5% B, ≤ 10% C+, above C- |
| Boundary enforcement | No minimum sample; observational, not graded | Any observed block → **Enforcement demonstrated** on those attempts; zero → **Enforcement not observed**, never failure or A-. Counts and blocked ÷ observed writes remain context, not a score |
| CI gate — integrity + convergence | 20 merged PRs with a known merge-time check state | confirmed breaches (required check red or missing on the merged head, as the merge gate judges): 0 → A+; ≤ 1% C+, ≤ 3% C, above C-. Convergence and CI red at claim are disclosed in the card and popover, not graded |
| Work balance | 20 issues opened in the window | issues opened ÷ closed: ≤ 0.8 A+, ≤ 0.9 A, ≤ 1.0 A-, ≤ 1.15 B+, ≤ 1.3 B, ≤ 1.5 B-, ≤ 1.75 C+, ≤ 2.0 C, above (or nothing closed) C- |
| Issue opens : PR merges | 20 hive issue opens + hive PR merges in the window | score = 1 − relative distance of opens ÷ merges outside the current governor mode's band (surge/busy ≤ 1; quiet/idle 1–2; unknown 0.75–1.25); inside the band A+, 10% out A-, 20% B-, ≥ 30% or no merges C-. The rolling 30 d is judged against parity and disclosed, not graded |
Expand All @@ -298,7 +300,7 @@ Grades are **Strong**, **Fair**, **Weak** or **Not enough data** — Work balanc
| Outcome by author kind | (reverts + issue re-opens) within 14 d ÷ merged (`EV_REVERT_*`) | ≤ 5% | ≤ 15% | above |
| Reviewer calibration | merged rate in the selected series’ 0.95 bucket (`EV_CALIB_*`) | ≥ 80% | ≥ 60% | below |
| Reviewer accuracy × model family | worse of false-approve and false-block rate (`EV_ERR_*`); the false-block rate counts enforced blocks only, so advisory dissent before merge never paints the panel red | ≤ 10% | ≤ 25% | above |
| Boundary enforcement | blocked ÷ **all** agent writes in the window (hive-mediated from the audit log + direct through the proxy; `EV_BLOCKED_*`). A handful of refusals in thousands of writes is the boundary working quietly | ≤ 1% | ≤ 5% | above |
| Boundary enforcement | Not graded: observed blocked writes demonstrate exercised enforcement, independent of refusal rate or sample size | — | — | — |
| Shadow gates | would-have-stopped PRs that merged without a revert ÷ stopped PRs that merged (`EV_SHADOW_FALSE_ALARM_*`) | ≤ 25% | ≤ 50% | above |
| CI gate — A, integrity | breaches (merged with a *required* check red or missing, as the merge gate judges; red non-required checks are disclosed, not breaches) ÷ merged PRs with a known merge-time state (`EV_GATE_BREACH_*`); unknown, suspected and pending are counts outside the rate | 0 breaches Excellent; ≤ 1% Good | ≤ 3% | above |
| CI gate — B, convergence | median CI passes to first green over agent PRs that went green AND abandoned (never green, closed unmerged) ÷ settled (`EV_CONV_PASSES_*`, `EV_CONV_ABANDON_*`); panel grade = lower of A and B. CI red at claim is disclosed, not graded: this hive hands off to CI by policy | ≤ 1.5 passes & < 5% Excellent; ≤ 2.5 & < 10% Good | ≤ 4 & < 20% | beyond |
Expand Down
22 changes: 17 additions & 5 deletions src/pkg/dashboard/evidence_card_ui_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -208,14 +208,14 @@ func TestEvidenceCardInThroughputSection(t *testing.T) {
"reasons.slice(0, EV_TOP_REASONS)",
"no agent writes recorded in this window",
"no sentinel hits recorded in this window",
"agent writes blocked</b>",
"prohibited write(s) blocked</b>",
"${bar(row.kind === 'direct' ? 'wa' : 'wm', row.writes, 0, formatNumber(row.writes))}${bar('wb', row.blocked, 1,",
`aria-label="Sentinel hits per ISO week"`,
"direct-write counters recorded since",
"would-block counts from intent verification are shown separately in the explainer and never counted as blocked",
"Writes that bypass both the hive and the proxy are invisible here",
"const EV_BLOCKED_STRONG = 0.01;",
"const EV_BLOCKED_FAIR = 0.05;",
"Enforcement demonstrated",
"Enforcement not observed",
".ev-legend i.wm",
".ev-svg .ev-bar.wb",
".ev-legend i.wa",
Expand Down Expand Up @@ -251,6 +251,8 @@ func TestEvidenceMaintainabilityRendering(t *testing.T) {
}
html := indexHTML(t)
var source strings.Builder
source.WriteString(jsFunc(t, html, "formatNumber"))
source.WriteByte('\n')
source.WriteString("const EV_PALETTE_SIZE = 6, EV_SVG_W = 320, EV_TOP_REASONS = 3;\nconst EV_WRITE_TIERS = ['ADVISORY', 'ISSUES_ONLY', 'ISSUES_AND_PRS', 'ISSUES_PRS_MERGE'];\nlet evLast = null;\nlet evConfidenceSeries = 'self-reported';\nconst evWindowLabel = () => 'all time';\nconst evWindowControls = () => '';\nconst evExplainerHtml = () => ({ button: '', body: '' });\nconst EV_CALIB_MIN_BUCKET_N = 5, EV_GRADE_MIN_N = 20;\nconst evDetails = {};\nconst EV_OM_DRAIN_MAX = 1, EV_OM_SCOUT_MIN = 1, EV_OM_SCOUT_MAX = 2, EV_OM_PARITY_TOLERANCE = 0.25, EV_OM_ROLLING_DAYS = 30, EV_OM_SPARK_TRAIL_DAYS = 7, EV_OM_BAR_MAX_RATIO = 3, EV_OM_SPARK_MAX_RATIO = 3;\nconst EV_OM_STATUS = { in_band: 'in band', above: 'backlog growing', below: 'scouting below cadence', no_data: 'no data' };\n")
for _, name := range []string{"escapeHtml", "evRate", "evPanel", "evServerCard", "evCalibrationSeries", "evCalibrationControls", "evSelectConfidence", "evOutcomesSvg", "evControlArmHtml", "evCalibrationSvg", "evAccuracySvg", "evShadowGateSvg", "evClaimVsVerdictSvg", "evWriteActivitySvg", "evCIGateHtml", "evCIRecoveryHtml", "evGateGrade", "evConvergenceGrade", "evConvergenceSvg", "evPct", "evWindowHint", "evSentinelSparkline", "evWorkBalanceSvg", "evMaintainabilityRows", "evMaintainabilitySvg", "evGradeLine", "evIssueReopenRatesHtml", "evOmStatusLabel", "evOmRatioText", "evOmVerdict", "evOmSparkline", "evOpensMergesHtml", "renderEvidenceSummary"} {
source.WriteString(jsFunc(t, html, name))
Expand Down Expand Up @@ -390,6 +392,16 @@ renderEvidenceSummary({reviewer_calibration: selfRows});
assert.match(card.innerHTML, /<summary>Reviewer calibration<span[^>]*>n=0/);
assert.match(card.innerHTML, /no reviewer verdicts with a confidence/);
evSelectConfidence('self-reported');
// Blocking is an observation, never a penalty, even for one attempt or all attempts.
for (const [total, blocked] of [[652, 77], [1, 1], [19, 1], [30, 30], [1000, 0], [0, 0]]) {
const rendered = evWriteActivitySvg({total_writes: total, blocked, direct: {}});
assert.match(rendered, new RegExp(blocked > 0 ? 'Enforcement demonstrated' : 'Enforcement not observed'));
assert.ok(rendered.includes(blocked + ' prohibited write(s) blocked'));
assert.ok(rendered.includes(formatNumber(total) + ' observed agent writes'));
}
renderEvidenceSummary({write_activity: {total_writes: 652, blocked: 77}, cards: {'boundary-enforcement': {grade: {letter: 'C-', band: 'c', enough_data: true}}}});
assert.match(card.innerHTML, /aria-label="Boundary enforcement is not graded">Not graded/);
assert.doesNotMatch(card.innerHTML, /Grade C-/);
// A band the client does not know falls back to the neutral pill.
renderEvidenceSummary({cards: {'work-balance': {grade: {letter: 'Z', band: 'z', reason: 'x', enough_data: true}}}});
assert.match(card.innerHTML, /<span class="ev-letter ev-band-none" title="x" aria-label="Grade Z: x">Z<\/span>/);
Expand Down Expand Up @@ -492,8 +504,8 @@ func TestEvidenceCardExplainers(t *testing.T) {
"const EV_REVERT_FAIR = 0.15;",
"const EV_SHADOW_FALSE_ALARM_STRONG = 0.25;",
"const EV_SHADOW_FALSE_ALARM_FAIR = 0.5;",
"const EV_BLOCKED_STRONG = 0.01;",
"const EV_BLOCKED_FAIR = 0.05;",
"grade: 'ungraded', reason: blocked > 0",
"Sentinel detections are informational observations, not proof that a write was blocked",
"const EV_DATA_SINCE = '2026-10-08';",
"const EV_REVERTS_SINCE = '2026-10-15';",
"const EV_GRADES = { excellent: 'Excellent', good: 'Good', strong: 'Strong', fair: 'Fair', weak: 'Weak', none: 'Not enough data', ungraded: 'Not graded' };",
Expand Down
33 changes: 31 additions & 2 deletions src/pkg/dashboard/evidence_cards_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,35 @@ func TestEvidenceCardsEmptySummary(t *testing.T) {
}
}

func TestEvidenceBoundaryObservations(t *testing.T) {
now := time.Date(2026, 11, 2, 14, 0, 0, 0, time.UTC)
for _, tc := range []struct {
name string
activity EvidenceWriteActivity
hits int
status string
}{
{"high refusal rate", EvidenceWriteActivity{TotalWrites: 652, Blocked: 77}, 0, "Enforcement demonstrated"},
{"one attempt", EvidenceWriteActivity{TotalWrites: 1, Blocked: 1}, 0, "Enforcement demonstrated"},
{"zero blocks", EvidenceWriteActivity{TotalWrites: 1000}, 0, "Enforcement not observed"},
{"Sentinel only", EvidenceWriteActivity{}, 4, "Enforcement not observed"},
{"missing direct counters", EvidenceWriteActivity{TotalWrites: 5, Mediated: EvidenceMediatedWrites{N: 5}}, 0, "Enforcement not observed"},
{"partial direct coverage", EvidenceWriteActivity{TotalWrites: 10, Blocked: 1, Direct: EvidenceDirectWrites{Recorded: true, PartialCoverage: true, Since: "2026-11-02", Attempted: 10, Blocked: 1}}, 2, "Enforcement demonstrated"},
} {
t.Run(tc.name, func(t *testing.T) {
resp := EvidenceSummary{
WriteActivity: tc.activity,
SentinelPerWeek: EvidenceSentinelSeries{Weeks: []EvidenceSentinelWeek{{Hits: tc.hits}}},
}
card := evidenceCards(&resp, evidenceWindow{Hours: 24}, now)[evidenceCardBoundary]
want := evidence.BoundaryStats{TotalWrites: tc.activity.TotalWrites, Blocked: tc.activity.Blocked, SentinelHits: tc.hits}
if card.Grade != evidence.GradeBoundary(want) || card.PlainSummary != evidence.SummarizeBoundary(want) || !strings.HasPrefix(card.PlainSummary, tc.status) {
t.Fatalf("unexpected boundary card: %+v", card)
}
})
}
}

// TestEvidenceCardsFromSummaryFields pins the mapping from summary fields to
// the grade inputs: reverts, calibration buckets summed across perspectives,
// the rework-based accuracy ledger, write activity, the CI gate, work balance
Expand Down Expand Up @@ -87,7 +116,7 @@ func TestEvidenceCardsFromSummaryFields(t *testing.T) {
evidenceCardCalibration: {"A", "Not enough data to compare slopes: each series needs two bins with at least five known outcomes."},
evidenceCardAccuracy: {"A", "Reviewers wrongly approved 3% and wrongly blocked 0% of 40 PRs with a known result."},
evidenceCardMaintainability: {"A", "Code complexity across 1 repo: median flat, 12 complex functions (was 14)."},
evidenceCardBoundary: {"A+", "4210 agent writes to GitHub; 3 were refused by the proxy (0.07%). 1 sentinel hit."},
evidenceCardBoundary: {evidence.NoGradeLetter, "Enforcement demonstrated — 3 prohibited writes blocked; 4210 observed writes (0.07%). Sentinel detections: 1 (informational)."},
evidenceCardCIGate: {"A+", "All 58 merged PRs with a known state had green required checks. Agent PRs needed a median 1 CI pass."},
evidenceCardWorkBalance: {"A-", "120 issues opened, 130 closed. 90 PRs opened, 80 merged. The issue queue is shrinking."},
evidenceCardShadow: {"A", "The shadow gates would have held 4 of 60 PRs; nothing was actually blocked."},
Expand All @@ -100,7 +129,7 @@ func TestEvidenceCardsFromSummaryFields(t *testing.T) {
if c.PlainSummary != w.summary {
t.Errorf("%s: summary\n got %q\nwant %q", key, c.PlainSummary, w.summary)
}
if !c.Grade.EnoughData || c.Grade.Band != evidence.BandOf(w.letter) {
if c.Grade.EnoughData != (w.letter != evidence.NoGradeLetter) || c.Grade.Band != evidence.BandOf(w.letter) {
t.Errorf("%s: grade %+v", key, c.Grade)
}
}
Expand Down
Loading
Loading