diff --git a/changelog.d/fixed-boundary-enforcement-evidence.md b/changelog.d/fixed-boundary-enforcement-evidence.md new file mode 100644 index 0000000000..6708b5561f --- /dev/null +++ b/changelog.d/fixed-boundary-enforcement-evidence.md @@ -0,0 +1 @@ +- Report boundary enforcement as demonstrated when observed writes were blocked, not a refusal-rate grade; keep zero, Sentinel detections, and coverage limitations informational. diff --git a/src/docs/review-evidence.md b/src/docs/review-evidence.md index 2a4b804bc4..9324adc98e 100644 --- a/src/docs/review-evidence.md +++ b/src/docs/review-evidence.md @@ -271,6 +271,8 @@ The ⓘ button opens a popover over the chart (never inline) with the plain-lang ### Grades and what they mean +Boundary enforcement is deliberately **not graded**, including in the legacy client fallback. Its status is **Enforcement demonstrated** when at least one actual refusal is observed, even below 20 writes, and **Enforcement not observed** otherwise. This is positive evidence of enforcement on those attempts, not proof that bypasses or false positives are absent. Refusal frequency describes attempted-agent compliance, not enforcement effectiveness; increasing prohibited attempts is not a goal. Sentinel detections are informational, never counted as proven blocked writes. Counts and denominator coverage remain visible; missing direct counters and partial coverage remain explicit. + A grade is a letter from **A+** to **C-** (A band green, B band amber, C band red) or **—** (grey, dashed) when the panel does not have enough data to be graded. Hovering the pill shows the reason ("86% of 22 PRs with a known outcome merged; 1 of 19 merged was reverted"); the popover adds the rubric. Where a rubric produces a 0–1 score the letter is: A+ ≥ 0.97, A ≥ 0.93, A- ≥ 0.90, B+ ≥ 0.87, B ≥ 0.83, B- ≥ 0.80, C+ ≥ 0.77, C ≥ 0.73, otherwise C-. Every rubric needs at least `GradeMinN` (20) samples of its denominator, else **—** with a reason that names what is missing. Rubrics are pure functions in `pkg/evidence/grade.go`, each table-tested; change the table and the code together. | Panel | Needs | Rubric | @@ -279,7 +281,7 @@ A grade is a letter from **A+** to **C-** (A band green, B band amber, C band re | Reviewer calibration | 20 verdicts with a known outcome in the selected series’ 0.95 bucket | score = merged share in the top bucket, minus 0.10 when the merged share is not non-decreasing across buckets that hold ≥ 5 known outcomes | | Reviewer accuracy × model family | the rework-based outcome ledger (`pkg/review/accuracy.go`) with ≥ 20 samples | score = 1 − max(false-approve rate, false-block rate) summed over reviewer models; **—** ("needs the reviewer outcome ledger") until the ledger exists — merge-vs-close is never used as ground truth | | Maintainability trend | one repo with ≥ 2 monthly snapshots in the window | first→last change of median cyclomatic and of functions over threshold, summed across repos: neither worse → A- / A / A+ by how many improved (0 / 1 / 2); one worse → B (B+ if the other improved); both worse → C | -| Boundary enforcement | 20 agent writes (hive-mediated + direct through the proxy) | blocked ÷ writes: 0 blocked → A- ("boundary not yet exercised"); ≤ 1% A+, ≤ 2% A, ≤ 5% B, ≤ 10% C+, above C- | +| Boundary enforcement | No minimum sample; observational, not graded | Any observed block → **Enforcement demonstrated** on those attempts; zero → **Enforcement not observed**, never failure or A-. Counts and blocked ÷ observed writes remain context, not a score | | CI gate — integrity + convergence | 20 merged PRs with a known merge-time check state | confirmed breaches (required check red or missing on the merged head, as the merge gate judges): 0 → A+; ≤ 1% C+, ≤ 3% C, above C-. Convergence and CI red at claim are disclosed in the card and popover, not graded | | Work balance | 20 issues opened in the window | issues opened ÷ closed: ≤ 0.8 A+, ≤ 0.9 A, ≤ 1.0 A-, ≤ 1.15 B+, ≤ 1.3 B, ≤ 1.5 B-, ≤ 1.75 C+, ≤ 2.0 C, above (or nothing closed) C- | | Issue opens : PR merges | 20 hive issue opens + hive PR merges in the window | score = 1 − relative distance of opens ÷ merges outside the current governor mode's band (surge/busy ≤ 1; quiet/idle 1–2; unknown 0.75–1.25); inside the band A+, 10% out A-, 20% B-, ≥ 30% or no merges C-. The rolling 30 d is judged against parity and disclosed, not graded | @@ -298,7 +300,7 @@ Grades are **Strong**, **Fair**, **Weak** or **Not enough data** — Work balanc | Outcome by author kind | (reverts + issue re-opens) within 14 d ÷ merged (`EV_REVERT_*`) | ≤ 5% | ≤ 15% | above | | Reviewer calibration | merged rate in the selected series’ 0.95 bucket (`EV_CALIB_*`) | ≥ 80% | ≥ 60% | below | | Reviewer accuracy × model family | worse of false-approve and false-block rate (`EV_ERR_*`); the false-block rate counts enforced blocks only, so advisory dissent before merge never paints the panel red | ≤ 10% | ≤ 25% | above | -| Boundary enforcement | blocked ÷ **all** agent writes in the window (hive-mediated from the audit log + direct through the proxy; `EV_BLOCKED_*`). A handful of refusals in thousands of writes is the boundary working quietly | ≤ 1% | ≤ 5% | above | +| Boundary enforcement | Not graded: observed blocked writes demonstrate exercised enforcement, independent of refusal rate or sample size | — | — | — | | Shadow gates | would-have-stopped PRs that merged without a revert ÷ stopped PRs that merged (`EV_SHADOW_FALSE_ALARM_*`) | ≤ 25% | ≤ 50% | above | | CI gate — A, integrity | breaches (merged with a *required* check red or missing, as the merge gate judges; red non-required checks are disclosed, not breaches) ÷ merged PRs with a known merge-time state (`EV_GATE_BREACH_*`); unknown, suspected and pending are counts outside the rate | 0 breaches Excellent; ≤ 1% Good | ≤ 3% | above | | CI gate — B, convergence | median CI passes to first green over agent PRs that went green AND abandoned (never green, closed unmerged) ÷ settled (`EV_CONV_PASSES_*`, `EV_CONV_ABANDON_*`); panel grade = lower of A and B. CI red at claim is disclosed, not graded: this hive hands off to CI by policy | ≤ 1.5 passes & < 5% Excellent; ≤ 2.5 & < 10% Good | ≤ 4 & < 20% | beyond | diff --git a/src/pkg/dashboard/evidence_card_ui_test.go b/src/pkg/dashboard/evidence_card_ui_test.go index 30c8d3ea66..c3b6a173cf 100644 --- a/src/pkg/dashboard/evidence_card_ui_test.go +++ b/src/pkg/dashboard/evidence_card_ui_test.go @@ -208,14 +208,14 @@ func TestEvidenceCardInThroughputSection(t *testing.T) { "reasons.slice(0, EV_TOP_REASONS)", "no agent writes recorded in this window", "no sentinel hits recorded in this window", - "agent writes blocked", + "prohibited write(s) blocked", "${bar(row.kind === 'direct' ? 'wa' : 'wm', row.writes, 0, formatNumber(row.writes))}${bar('wb', row.blocked, 1,", `aria-label="Sentinel hits per ISO week"`, "direct-write counters recorded since", "would-block counts from intent verification are shown separately in the explainer and never counted as blocked", "Writes that bypass both the hive and the proxy are invisible here", - "const EV_BLOCKED_STRONG = 0.01;", - "const EV_BLOCKED_FAIR = 0.05;", + "Enforcement demonstrated", + "Enforcement not observed", ".ev-legend i.wm", ".ev-svg .ev-bar.wb", ".ev-legend i.wa", @@ -251,6 +251,8 @@ func TestEvidenceMaintainabilityRendering(t *testing.T) { } html := indexHTML(t) var source strings.Builder + source.WriteString(jsFunc(t, html, "formatNumber")) + source.WriteByte('\n') source.WriteString("const EV_PALETTE_SIZE = 6, EV_SVG_W = 320, EV_TOP_REASONS = 3;\nconst EV_WRITE_TIERS = ['ADVISORY', 'ISSUES_ONLY', 'ISSUES_AND_PRS', 'ISSUES_PRS_MERGE'];\nlet evLast = null;\nlet evConfidenceSeries = 'self-reported';\nconst evWindowLabel = () => 'all time';\nconst evWindowControls = () => '';\nconst evExplainerHtml = () => ({ button: '', body: '' });\nconst EV_CALIB_MIN_BUCKET_N = 5, EV_GRADE_MIN_N = 20;\nconst evDetails = {};\nconst EV_OM_DRAIN_MAX = 1, EV_OM_SCOUT_MIN = 1, EV_OM_SCOUT_MAX = 2, EV_OM_PARITY_TOLERANCE = 0.25, EV_OM_ROLLING_DAYS = 30, EV_OM_SPARK_TRAIL_DAYS = 7, EV_OM_BAR_MAX_RATIO = 3, EV_OM_SPARK_MAX_RATIO = 3;\nconst EV_OM_STATUS = { in_band: 'in band', above: 'backlog growing', below: 'scouting below cadence', no_data: 'no data' };\n") for _, name := range []string{"escapeHtml", "evRate", "evPanel", "evServerCard", "evCalibrationSeries", "evCalibrationControls", "evSelectConfidence", "evOutcomesSvg", "evControlArmHtml", "evCalibrationSvg", "evAccuracySvg", "evShadowGateSvg", "evClaimVsVerdictSvg", "evWriteActivitySvg", "evCIGateHtml", "evCIRecoveryHtml", "evGateGrade", "evConvergenceGrade", "evConvergenceSvg", "evPct", "evWindowHint", "evSentinelSparkline", "evWorkBalanceSvg", "evMaintainabilityRows", "evMaintainabilitySvg", "evGradeLine", "evIssueReopenRatesHtml", "evOmStatusLabel", "evOmRatioText", "evOmVerdict", "evOmSparkline", "evOpensMergesHtml", "renderEvidenceSummary"} { source.WriteString(jsFunc(t, html, name)) @@ -390,6 +392,16 @@ renderEvidenceSummary({reviewer_calibration: selfRows}); assert.match(card.innerHTML, /Reviewer calibration]*>n=0/); assert.match(card.innerHTML, /no reviewer verdicts with a confidence/); evSelectConfidence('self-reported'); +// Blocking is an observation, never a penalty, even for one attempt or all attempts. +for (const [total, blocked] of [[652, 77], [1, 1], [19, 1], [30, 30], [1000, 0], [0, 0]]) { + const rendered = evWriteActivitySvg({total_writes: total, blocked, direct: {}}); + assert.match(rendered, new RegExp(blocked > 0 ? 'Enforcement demonstrated' : 'Enforcement not observed')); + assert.ok(rendered.includes(blocked + ' prohibited write(s) blocked')); + assert.ok(rendered.includes(formatNumber(total) + ' observed agent writes')); +} +renderEvidenceSummary({write_activity: {total_writes: 652, blocked: 77}, cards: {'boundary-enforcement': {grade: {letter: 'C-', band: 'c', enough_data: true}}}}); +assert.match(card.innerHTML, /aria-label="Boundary enforcement is not graded">Not graded/); +assert.doesNotMatch(card.innerHTML, /Grade C-/); // A band the client does not know falls back to the neutral pill. renderEvidenceSummary({cards: {'work-balance': {grade: {letter: 'Z', band: 'z', reason: 'x', enough_data: true}}}}); assert.match(card.innerHTML, /Z<\/span>/); @@ -492,8 +504,8 @@ func TestEvidenceCardExplainers(t *testing.T) { "const EV_REVERT_FAIR = 0.15;", "const EV_SHADOW_FALSE_ALARM_STRONG = 0.25;", "const EV_SHADOW_FALSE_ALARM_FAIR = 0.5;", - "const EV_BLOCKED_STRONG = 0.01;", - "const EV_BLOCKED_FAIR = 0.05;", + "grade: 'ungraded', reason: blocked > 0", + "Sentinel detections are informational observations, not proof that a write was blocked", "const EV_DATA_SINCE = '2026-10-08';", "const EV_REVERTS_SINCE = '2026-10-15';", "const EV_GRADES = { excellent: 'Excellent', good: 'Good', strong: 'Strong', fair: 'Fair', weak: 'Weak', none: 'Not enough data', ungraded: 'Not graded' };", diff --git a/src/pkg/dashboard/evidence_cards_test.go b/src/pkg/dashboard/evidence_cards_test.go index c6eaf9496e..4916dd2898 100644 --- a/src/pkg/dashboard/evidence_cards_test.go +++ b/src/pkg/dashboard/evidence_cards_test.go @@ -41,6 +41,35 @@ func TestEvidenceCardsEmptySummary(t *testing.T) { } } +func TestEvidenceBoundaryObservations(t *testing.T) { + now := time.Date(2026, 11, 2, 14, 0, 0, 0, time.UTC) + for _, tc := range []struct { + name string + activity EvidenceWriteActivity + hits int + status string + }{ + {"high refusal rate", EvidenceWriteActivity{TotalWrites: 652, Blocked: 77}, 0, "Enforcement demonstrated"}, + {"one attempt", EvidenceWriteActivity{TotalWrites: 1, Blocked: 1}, 0, "Enforcement demonstrated"}, + {"zero blocks", EvidenceWriteActivity{TotalWrites: 1000}, 0, "Enforcement not observed"}, + {"Sentinel only", EvidenceWriteActivity{}, 4, "Enforcement not observed"}, + {"missing direct counters", EvidenceWriteActivity{TotalWrites: 5, Mediated: EvidenceMediatedWrites{N: 5}}, 0, "Enforcement not observed"}, + {"partial direct coverage", EvidenceWriteActivity{TotalWrites: 10, Blocked: 1, Direct: EvidenceDirectWrites{Recorded: true, PartialCoverage: true, Since: "2026-11-02", Attempted: 10, Blocked: 1}}, 2, "Enforcement demonstrated"}, + } { + t.Run(tc.name, func(t *testing.T) { + resp := EvidenceSummary{ + WriteActivity: tc.activity, + SentinelPerWeek: EvidenceSentinelSeries{Weeks: []EvidenceSentinelWeek{{Hits: tc.hits}}}, + } + card := evidenceCards(&resp, evidenceWindow{Hours: 24}, now)[evidenceCardBoundary] + want := evidence.BoundaryStats{TotalWrites: tc.activity.TotalWrites, Blocked: tc.activity.Blocked, SentinelHits: tc.hits} + if card.Grade != evidence.GradeBoundary(want) || card.PlainSummary != evidence.SummarizeBoundary(want) || !strings.HasPrefix(card.PlainSummary, tc.status) { + t.Fatalf("unexpected boundary card: %+v", card) + } + }) + } +} + // TestEvidenceCardsFromSummaryFields pins the mapping from summary fields to // the grade inputs: reverts, calibration buckets summed across perspectives, // the rework-based accuracy ledger, write activity, the CI gate, work balance @@ -87,7 +116,7 @@ func TestEvidenceCardsFromSummaryFields(t *testing.T) { evidenceCardCalibration: {"A", "Not enough data to compare slopes: each series needs two bins with at least five known outcomes."}, evidenceCardAccuracy: {"A", "Reviewers wrongly approved 3% and wrongly blocked 0% of 40 PRs with a known result."}, evidenceCardMaintainability: {"A", "Code complexity across 1 repo: median flat, 12 complex functions (was 14)."}, - evidenceCardBoundary: {"A+", "4210 agent writes to GitHub; 3 were refused by the proxy (0.07%). 1 sentinel hit."}, + evidenceCardBoundary: {evidence.NoGradeLetter, "Enforcement demonstrated — 3 prohibited writes blocked; 4210 observed writes (0.07%). Sentinel detections: 1 (informational)."}, evidenceCardCIGate: {"A+", "All 58 merged PRs with a known state had green required checks. Agent PRs needed a median 1 CI pass."}, evidenceCardWorkBalance: {"A-", "120 issues opened, 130 closed. 90 PRs opened, 80 merged. The issue queue is shrinking."}, evidenceCardShadow: {"A", "The shadow gates would have held 4 of 60 PRs; nothing was actually blocked."}, @@ -100,7 +129,7 @@ func TestEvidenceCardsFromSummaryFields(t *testing.T) { if c.PlainSummary != w.summary { t.Errorf("%s: summary\n got %q\nwant %q", key, c.PlainSummary, w.summary) } - if !c.Grade.EnoughData || c.Grade.Band != evidence.BandOf(w.letter) { + if c.Grade.EnoughData != (w.letter != evidence.NoGradeLetter) || c.Grade.Band != evidence.BandOf(w.letter) { t.Errorf("%s: grade %+v", key, c.Grade) } } diff --git a/src/pkg/dashboard/static-src/js/evidence.js b/src/pkg/dashboard/static-src/js/evidence.js index 09a6e96215..b31b5c85d9 100644 --- a/src/pkg/dashboard/static-src/js/evidence.js +++ b/src/pkg/dashboard/static-src/js/evidence.js @@ -125,8 +125,6 @@ const EV_THROUGHPUT_GOOD = 0.85; // ≥ → Good const EV_THROUGHPUT_FAIR = 0.70; // ≥ → Fair, below → Weak const EV_QUEUE_GROWTH_MAX = 0.10; // week-end queue depth growing more than this week over week → Weak regardless of ratio - const EV_BLOCKED_STRONG = 0.01; // proxy-blocked share of ALL agent writes (direct + hive-mediated) ≤ → Strong - const EV_BLOCKED_FAIR = 0.05; // ≤ → Fair, above → Weak // CI gate, grade A (integrity): breaches ÷ merged PRs with a known merge-time check state. const EV_GATE_BREACH_GOOD = 0.01; // 0 breaches → Excellent; ≤ this share → Good const EV_GATE_BREACH_FAIR = 0.03; // ≤ this share → Fair, above → Weak @@ -418,15 +416,14 @@ const sentinelWeeks = Array.isArray(sentinel.weeks) ? sentinel.weeks : []; const sentinelHits = sentinelWeeks.reduce((sum, wk) => sum + (Number(wk && wk.hits) || 0), 0); const blockedRate = evRate(blocked, total); - const grade = evGrade(blockedRate, total, EV_BLOCKED_STRONG, EV_BLOCKED_FAIR, false); const coverage = direct.recorded === false ? 'Direct-write counters have not been recorded on this hive yet, so the total only holds hive-mediated writes.' : direct.partial_coverage ? `Direct-write counters have only been recorded since ${direct.since}, so earlier weeks in this window hold hive-mediated writes only.` : ''; return { - matters: 'Agents change GitHub either through the hive (comments, issues, PRs, reviews it files on their behalf, all audited) or directly through a proxy that enforces the trust tiers. A handful of refusals out of thousands of writes means the boundary is doing its job quietly; a refusal rate in the percent range means agents are repeatedly trying things they are not allowed to do.', - measured: 'The headline is blocked ÷ ALL agent GitHub writes in the window: every hive-mediated write from the audit log plus every direct write the proxy saw, per trust tier, and how many the proxy refused and why. Separately: how many evidence records per week carry a "sentinel hit", a canary that fires when an agent touched something it should not have.', + matters: 'A recorded block is positive evidence that enforcement worked on that attempt. Attempted-agent compliance is a separate question: frequent prohibited attempts do not make successful blocking a failure, and generating more prohibited attempts is not a goal.', + measured: 'Observed prohibited writes blocked, with ALL observed agent GitHub writes as denominator context. Separately, Sentinel detections are informational observations, not proof that a write was blocked.', how: `Denominator = hive-mediated writes (audit entries for agent comments, issues, PRs, reviews, labels, closes and pushes) + direct proxied writes (the proxy’s weekly tally per tier, kept under /data so it survives restarts). Numerator = writes the proxy refused (plus audited refusals of mediated writes). Per-tier bars show blocked ÷ direct writes for that tier. Would-block counts from intent verification are shown separately and are never in the blocked count. ${coverage}`, - good: `Blocked at or below ${Math.round(EV_BLOCKED_STRONG * 100)}% of all agent writes is Strong — a boundary that is exercised but rarely needed — ${Math.round(EV_BLOCKED_FAIR * 100)}% or less is Fair, above that Weak. The sentinel line should sit at zero; any hit is worth reading the record for.`, + good: 'One or more observed blocks demonstrates enforcement on those attempts, even in a small window. Zero means enforcement was not observed, not that it failed. There is no target refusal rate or reward for more prohibited attempts. These counts do not establish that bypasses or false positives are absent.', ours: `${formatNumber(total)} agent writes in this window (${formatNumber(Number(mediated.n) || 0)} hive-mediated · ${formatNumber(Number(direct.attempted) || 0)} direct across ${tiers.length} tier(s)), ${blocked} blocked (${blockedRate == null ? '—' : (blockedRate * 100).toFixed(2) + '%'})${topReason ? `, most often for ${topReason[0]} (${topReason[1]})` : ''}.${evaluated ? ` Intent verification evaluated ${evaluated} write(s) and would have blocked ${wouldBlock} (not counted above).` : ''} ${sentinel.n ? `${sentinel.n} record(s) with ${sentinelHits} sentinel hit(s)` : 'No sentinel hits'} in this window. ${coverage} Direct writes that bypass both the hive and the proxy are not seen at all.`, - grade, reason: grade === 'none' ? `needs at least ${EV_GRADE_MIN_N} agent writes` : `${blocked} of ${formatNumber(total)} agent writes blocked (${blockedRate == null ? '—' : (blockedRate * 100).toFixed(2) + '%'})`, + grade: 'ungraded', reason: blocked > 0 ? `Enforcement demonstrated — ${blocked} prohibited write(s) blocked` : 'Enforcement not observed — 0 blocked writes', }; }, 'opens-merges': (dto) => { @@ -466,7 +463,9 @@ const section = (h, body) => body ? `

${h}

${escapeHtml(body)}

` : ''; const sc = evServerCard(dto, key); const g = sc && sc.grade && typeof sc.grade === 'object' ? sc.grade : null; - const gradeHtml = g + const gradeHtml = key === 'boundary-enforcement' + ? `
Not graded: ${escapeHtml(g ? String(g.reason || '') : x.reason)}.${g && g.rubric ? ` ${escapeHtml(String(g.rubric))}.` : ''}
` + : g ? `
${g.enough_data ? `Grade ${escapeHtml(String(g.letter || '—'))}` : 'Not graded yet'}: ${escapeHtml(String(g.reason || ''))}.${g.rubric ? ` Rubric: ${escapeHtml(String(g.rubric))}.` : ''}
` : evGradeLine(x.grade, x.reason); const button = ``; @@ -506,7 +505,9 @@ const band = g && ['a', 'b', 'c'].includes(g.band) ? g.band : 'none'; const letter = g && g.letter ? String(g.letter) : '—'; const why = g ? (g.enough_data ? String(g.reason || '') : `Not enough data yet: ${g.reason || ''}`) : 'Grade not reported by this hive'; - const grade = `${escapeHtml(letter)}`; + const grade = key === 'boundary-enforcement' + ? `Not graded` + : `${escapeHtml(letter)}`; const plain = sc && sc.plain_summary ? `
${escapeHtml(String(sc.plain_summary))}
` : ''; return `
${escapeHtml(title)}${ex.button}n=${Number(n) || 0}${grade}${plain}${ex.body}${bodyHtml}
`; } @@ -782,7 +783,8 @@ const total = Number(act.total_writes) || 0, blocked = Number(act.blocked) || 0; const rate = evRate(blocked, total); evDetails['boundary-enforcement'] = `${formatNumber(Number(mediated.n) || 0)} hive-mediated · ${formatNumber(Number(direct.attempted) || 0)} direct through the proxy${direct.recorded === false ? ' · direct-write counters not recorded yet' : direct.partial_coverage ? ` · direct-write counters recorded since ${String(direct.since || '')}` : ''}.`; - const headline = `
${blocked} of ${formatNumber(total)} agent writes blocked · ${rate == null ? '—' : (rate * 100).toFixed(2) + '%'}
`; + const status = blocked > 0 ? 'Enforcement demonstrated' : 'Enforcement not observed'; + const headline = `
${status} — ${blocked} prohibited write(s) blocked · ${formatNumber(total)} observed agent writes · ${rate == null ? '—' : (rate * 100).toFixed(2) + '%'}
`; const tiers = EV_WRITE_TIERS.concat(Object.keys(byTier).filter((k) => !EV_WRITE_TIERS.includes(k)).sort()); const rows = tiers.map((tier) => { const c = byTier[tier] || {}; @@ -1192,4 +1194,3 @@ renderEvidenceSummary(dto); } catch (e) { if (!evLast) renderEvidenceSummary(null); } } - diff --git a/src/pkg/dashboard/static/index.html b/src/pkg/dashboard/static/index.html index 75ba494323..d252799607 100644 --- a/src/pkg/dashboard/static/index.html +++ b/src/pkg/dashboard/static/index.html @@ -37211,8 +37211,6 @@

All labels

const EV_THROUGHPUT_GOOD = 0.85; // ≥ → Good const EV_THROUGHPUT_FAIR = 0.70; // ≥ → Fair, below → Weak const EV_QUEUE_GROWTH_MAX = 0.10; // week-end queue depth growing more than this week over week → Weak regardless of ratio - const EV_BLOCKED_STRONG = 0.01; // proxy-blocked share of ALL agent writes (direct + hive-mediated) ≤ → Strong - const EV_BLOCKED_FAIR = 0.05; // ≤ → Fair, above → Weak // CI gate, grade A (integrity): breaches ÷ merged PRs with a known merge-time check state. const EV_GATE_BREACH_GOOD = 0.01; // 0 breaches → Excellent; ≤ this share → Good const EV_GATE_BREACH_FAIR = 0.03; // ≤ this share → Fair, above → Weak @@ -37504,15 +37502,14 @@

All labels

const sentinelWeeks = Array.isArray(sentinel.weeks) ? sentinel.weeks : []; const sentinelHits = sentinelWeeks.reduce((sum, wk) => sum + (Number(wk && wk.hits) || 0), 0); const blockedRate = evRate(blocked, total); - const grade = evGrade(blockedRate, total, EV_BLOCKED_STRONG, EV_BLOCKED_FAIR, false); const coverage = direct.recorded === false ? 'Direct-write counters have not been recorded on this hive yet, so the total only holds hive-mediated writes.' : direct.partial_coverage ? `Direct-write counters have only been recorded since ${direct.since}, so earlier weeks in this window hold hive-mediated writes only.` : ''; return { - matters: 'Agents change GitHub either through the hive (comments, issues, PRs, reviews it files on their behalf, all audited) or directly through a proxy that enforces the trust tiers. A handful of refusals out of thousands of writes means the boundary is doing its job quietly; a refusal rate in the percent range means agents are repeatedly trying things they are not allowed to do.', - measured: 'The headline is blocked ÷ ALL agent GitHub writes in the window: every hive-mediated write from the audit log plus every direct write the proxy saw, per trust tier, and how many the proxy refused and why. Separately: how many evidence records per week carry a "sentinel hit", a canary that fires when an agent touched something it should not have.', + matters: 'A recorded block is positive evidence that enforcement worked on that attempt. Attempted-agent compliance is a separate question: frequent prohibited attempts do not make successful blocking a failure, and generating more prohibited attempts is not a goal.', + measured: 'Observed prohibited writes blocked, with ALL observed agent GitHub writes as denominator context. Separately, Sentinel detections are informational observations, not proof that a write was blocked.', how: `Denominator = hive-mediated writes (audit entries for agent comments, issues, PRs, reviews, labels, closes and pushes) + direct proxied writes (the proxy’s weekly tally per tier, kept under /data so it survives restarts). Numerator = writes the proxy refused (plus audited refusals of mediated writes). Per-tier bars show blocked ÷ direct writes for that tier. Would-block counts from intent verification are shown separately and are never in the blocked count. ${coverage}`, - good: `Blocked at or below ${Math.round(EV_BLOCKED_STRONG * 100)}% of all agent writes is Strong — a boundary that is exercised but rarely needed — ${Math.round(EV_BLOCKED_FAIR * 100)}% or less is Fair, above that Weak. The sentinel line should sit at zero; any hit is worth reading the record for.`, + good: 'One or more observed blocks demonstrates enforcement on those attempts, even in a small window. Zero means enforcement was not observed, not that it failed. There is no target refusal rate or reward for more prohibited attempts. These counts do not establish that bypasses or false positives are absent.', ours: `${formatNumber(total)} agent writes in this window (${formatNumber(Number(mediated.n) || 0)} hive-mediated · ${formatNumber(Number(direct.attempted) || 0)} direct across ${tiers.length} tier(s)), ${blocked} blocked (${blockedRate == null ? '—' : (blockedRate * 100).toFixed(2) + '%'})${topReason ? `, most often for ${topReason[0]} (${topReason[1]})` : ''}.${evaluated ? ` Intent verification evaluated ${evaluated} write(s) and would have blocked ${wouldBlock} (not counted above).` : ''} ${sentinel.n ? `${sentinel.n} record(s) with ${sentinelHits} sentinel hit(s)` : 'No sentinel hits'} in this window. ${coverage} Direct writes that bypass both the hive and the proxy are not seen at all.`, - grade, reason: grade === 'none' ? `needs at least ${EV_GRADE_MIN_N} agent writes` : `${blocked} of ${formatNumber(total)} agent writes blocked (${blockedRate == null ? '—' : (blockedRate * 100).toFixed(2) + '%'})`, + grade: 'ungraded', reason: blocked > 0 ? `Enforcement demonstrated — ${blocked} prohibited write(s) blocked` : 'Enforcement not observed — 0 blocked writes', }; }, 'opens-merges': (dto) => { @@ -37552,7 +37549,9 @@

All labels

const section = (h, body) => body ? `

${h}

${escapeHtml(body)}

` : ''; const sc = evServerCard(dto, key); const g = sc && sc.grade && typeof sc.grade === 'object' ? sc.grade : null; - const gradeHtml = g + const gradeHtml = key === 'boundary-enforcement' + ? `
Not graded: ${escapeHtml(g ? String(g.reason || '') : x.reason)}.${g && g.rubric ? ` ${escapeHtml(String(g.rubric))}.` : ''}
` + : g ? `
${g.enough_data ? `Grade ${escapeHtml(String(g.letter || '—'))}` : 'Not graded yet'}: ${escapeHtml(String(g.reason || ''))}.${g.rubric ? ` Rubric: ${escapeHtml(String(g.rubric))}.` : ''}
` : evGradeLine(x.grade, x.reason); const button = ``; @@ -37592,7 +37591,9 @@

All labels

const band = g && ['a', 'b', 'c'].includes(g.band) ? g.band : 'none'; const letter = g && g.letter ? String(g.letter) : '—'; const why = g ? (g.enough_data ? String(g.reason || '') : `Not enough data yet: ${g.reason || ''}`) : 'Grade not reported by this hive'; - const grade = `${escapeHtml(letter)}`; + const grade = key === 'boundary-enforcement' + ? `Not graded` + : `${escapeHtml(letter)}`; const plain = sc && sc.plain_summary ? `
${escapeHtml(String(sc.plain_summary))}
` : ''; return `
${escapeHtml(title)}${ex.button}n=${Number(n) || 0}${grade}${plain}${ex.body}${bodyHtml}
`; } @@ -37868,7 +37869,8 @@

All labels

const total = Number(act.total_writes) || 0, blocked = Number(act.blocked) || 0; const rate = evRate(blocked, total); evDetails['boundary-enforcement'] = `${formatNumber(Number(mediated.n) || 0)} hive-mediated · ${formatNumber(Number(direct.attempted) || 0)} direct through the proxy${direct.recorded === false ? ' · direct-write counters not recorded yet' : direct.partial_coverage ? ` · direct-write counters recorded since ${String(direct.since || '')}` : ''}.`; - const headline = `
${blocked} of ${formatNumber(total)} agent writes blocked · ${rate == null ? '—' : (rate * 100).toFixed(2) + '%'}
`; + const status = blocked > 0 ? 'Enforcement demonstrated' : 'Enforcement not observed'; + const headline = `
${status} — ${blocked} prohibited write(s) blocked · ${formatNumber(total)} observed agent writes · ${rate == null ? '—' : (rate * 100).toFixed(2) + '%'}
`; const tiers = EV_WRITE_TIERS.concat(Object.keys(byTier).filter((k) => !EV_WRITE_TIERS.includes(k)).sort()); const rows = tiers.map((tier) => { const c = byTier[tier] || {}; @@ -38278,7 +38280,6 @@

All labels

renderEvidenceSummary(dto); } catch (e) { if (!evLast) renderEvidenceSummary(null); } } - // ---- Approvals desk (RFC #4000) ------------------------------------- // Pending operator-lane approvals. The panel stays hidden entirely unless // the desk is enabled, so a hive that has not opted in sees no new chrome. diff --git a/src/pkg/evidence/grade.go b/src/pkg/evidence/grade.go index 02a4798c8a..56ef177abe 100644 --- a/src/pkg/evidence/grade.go +++ b/src/pkg/evidence/grade.go @@ -18,7 +18,7 @@ import ( // card reads "—" and says what it is waiting for. const GradeMinN = 20 -// NoGradeLetter is the letter shown when a card has not enough data. +// NoGradeLetter is shown when a card is ungraded or has insufficient data. const NoGradeLetter = "—" // Grade bands colour the badge: a is green, b amber, c red, none grey. @@ -87,7 +87,7 @@ func BandOf(letter string) string { } } -// NoGrade is the "—" grade with the reason the card is waiting for. +// NoGrade is the "—" grade with the reason the card is not graded. func NoGrade(reason, rubric string) Grade { return Grade{Letter: NoGradeLetter, Band: BandNone, Reason: reason, EnoughData: false, Rubric: rubric} } @@ -457,45 +457,24 @@ type BoundaryStats struct { SentinelHits int } -const boundaryRubric = "blocked share of all agent GitHub writes: 0 < share ≤ 1% A+, ≤ 2% A, ≤ 5% B, ≤ 10% C+, more C-; no refusals at all A- (boundary not yet exercised); needs 20 writes" +const boundaryRubric = "not graded: observed blocks demonstrate enforcement on those attempts; zero means not observed, not failure; counts do not prove absence of bypasses or false positives; Sentinel detections are informational, not proven blocks" -// GradeBoundary grades how often the proxy had to refuse an agent write. A -// few refusals in thousands of writes is the boundary working quietly. +// GradeBoundary retains the grade API but reports an ungraded observation: +// refusal frequency describes attempted-agent compliance, not enforcement quality. func GradeBoundary(b BoundaryStats) Grade { - if b.TotalWrites < GradeMinN { - return NoGrade(fmt.Sprintf("needs %d agent writes (have %d)", GradeMinN, b.TotalWrites), boundaryRubric) + if b.Blocked > 0 { + return NoGrade(fmt.Sprintf("Enforcement demonstrated — %d prohibited %s blocked; %d observed agent writes (%s)", b.Blocked, plural(int(b.Blocked), "write", "writes"), b.TotalWrites, pctShare(b.Blocked, b.TotalWrites)), boundaryRubric) } - share := float64(b.Blocked) / float64(b.TotalWrites) - reason := fmt.Sprintf("%d of %d agent writes refused (%.2f%%)", b.Blocked, b.TotalWrites, share*100) - var letter string - switch { - case b.Blocked == 0: - letter = "A-" - reason = fmt.Sprintf("no refusals in %d agent writes; the boundary has not been exercised yet", b.TotalWrites) - case share <= 0.01: - letter = "A+" - case share <= 0.02: - letter = "A" - case share <= 0.05: - letter = "B" - case share <= 0.10: - letter = "C+" - default: - letter = "C-" - } - return graded(letter, reason, boundaryRubric) + return NoGrade(fmt.Sprintf("Enforcement not observed — 0 blocked writes; %d observed agent writes", b.TotalWrites), boundaryRubric) } // SummarizeBoundary is the plain sentence for the Boundary card. func SummarizeBoundary(b BoundaryStats) string { - if b.TotalWrites == 0 { - return "No agent writes to GitHub recorded in this window." - } - sentinel := "No sentinel hits." - if b.SentinelHits > 0 { - sentinel = fmt.Sprintf("%d sentinel %s.", b.SentinelHits, plural(b.SentinelHits, "hit", "hits")) + status := "Enforcement not observed" + if b.Blocked > 0 { + status = "Enforcement demonstrated" } - return fmt.Sprintf("%d agent writes to GitHub; %d %s refused by the proxy (%s). %s", b.TotalWrites, b.Blocked, plural(int(b.Blocked), "was", "were"), pctShare(b.Blocked, b.TotalWrites), sentinel) + return fmt.Sprintf("%s — %d prohibited %s blocked; %d observed writes (%s). Sentinel detections: %d (informational).", status, b.Blocked, plural(int(b.Blocked), "write", "writes"), b.TotalWrites, pctShare(b.Blocked, b.TotalWrites), b.SentinelHits) } func pctShare(num, den int64) string { diff --git a/src/pkg/evidence/grade_test.go b/src/pkg/evidence/grade_test.go index 25915767ce..dc8cca96c4 100644 --- a/src/pkg/evidence/grade_test.go +++ b/src/pkg/evidence/grade_test.go @@ -248,16 +248,23 @@ func TestGradeBoundary(t *testing.T) { in BoundaryStats want string }{ - {"too few", BoundaryStats{TotalWrites: 19, Blocked: 1}, NoGradeLetter}, - {"never exercised", BoundaryStats{TotalWrites: 1000}, "A-"}, - {"quiet boundary", BoundaryStats{TotalWrites: 1000, Blocked: 3}, "A+"}, - {"two percent", BoundaryStats{TotalWrites: 1000, Blocked: 20}, "A"}, - {"five percent", BoundaryStats{TotalWrites: 1000, Blocked: 50}, "B"}, - {"ten percent", BoundaryStats{TotalWrites: 1000, Blocked: 100}, "C+"}, - {"noisy", BoundaryStats{TotalWrites: 100, Blocked: 30}, "C-"}, + {"empty", BoundaryStats{}, "Enforcement not observed"}, + {"one observed block", BoundaryStats{TotalWrites: 1, Blocked: 1}, "Enforcement demonstrated"}, + {"low sample", BoundaryStats{TotalWrites: 19, Blocked: 1}, "Enforcement demonstrated"}, + {"never exercised", BoundaryStats{TotalWrites: 1000}, "Enforcement not observed"}, + {"quiet boundary", BoundaryStats{TotalWrites: 1000, Blocked: 3}, "Enforcement demonstrated"}, + {"high refusal rate", BoundaryStats{TotalWrites: 652, Blocked: 77}, "Enforcement demonstrated"}, + {"all blocked", BoundaryStats{TotalWrites: 30, Blocked: 30}, "Enforcement demonstrated"}, + {"Sentinel only", BoundaryStats{SentinelHits: 4}, "Enforcement not observed"}, + {"Sentinel with unblocked writes", BoundaryStats{TotalWrites: 100, SentinelHits: 4}, "Enforcement not observed"}, + {"block without denominator", BoundaryStats{Blocked: 1}, "Enforcement demonstrated"}, } { - if g := GradeBoundary(tc.in); g.Letter != tc.want { - t.Errorf("%s: letter %q, want %q (reason %q)", tc.name, g.Letter, tc.want, g.Reason) + g := GradeBoundary(tc.in) + if g.Letter != NoGradeLetter || g.Band != BandNone || g.EnoughData || !strings.HasPrefix(g.Reason, tc.want) { + t.Errorf("%s: got %+v, want ungraded observation %q", tc.name, g, tc.want) + } + if !strings.Contains(g.Rubric, "bypasses or false positives") || !strings.Contains(g.Rubric, "informational") { + t.Errorf("%s: missing observation limits: %q", tc.name, g.Rubric) } } } @@ -267,9 +274,11 @@ func TestSummarizeBoundary(t *testing.T) { in BoundaryStats want string }{ - {BoundaryStats{}, "No agent writes to GitHub recorded in this window."}, - {BoundaryStats{TotalWrites: 4210, Blocked: 3}, "4210 agent writes to GitHub; 3 were refused by the proxy (0.07%). No sentinel hits."}, - {BoundaryStats{TotalWrites: 10, Blocked: 1, SentinelHits: 1}, "10 agent writes to GitHub; 1 was refused by the proxy (10.00%). 1 sentinel hit."}, + {BoundaryStats{}, "Enforcement not observed — 0 prohibited writes blocked; 0 observed writes (—). Sentinel detections: 0 (informational)."}, + {BoundaryStats{TotalWrites: 4210, Blocked: 3}, "Enforcement demonstrated — 3 prohibited writes blocked; 4210 observed writes (0.07%). Sentinel detections: 0 (informational)."}, + {BoundaryStats{TotalWrites: 10, Blocked: 1, SentinelHits: 1}, "Enforcement demonstrated — 1 prohibited write blocked; 10 observed writes (10.00%). Sentinel detections: 1 (informational)."}, + {BoundaryStats{TotalWrites: 652, Blocked: 77}, "Enforcement demonstrated — 77 prohibited writes blocked; 652 observed writes (11.81%). Sentinel detections: 0 (informational)."}, + {BoundaryStats{SentinelHits: 4}, "Enforcement not observed — 0 prohibited writes blocked; 0 observed writes (—). Sentinel detections: 4 (informational)."}, } { if got := SummarizeBoundary(tc.in); got != tc.want || words(got) > PlainSummaryMaxWords { t.Errorf("got %q (%d words), want %q", got, words(got), tc.want) diff --git a/src/pkg/hub/assets/dashboard.html b/src/pkg/hub/assets/dashboard.html index 47d4e2340f..af2e39022e 100644 --- a/src/pkg/hub/assets/dashboard.html +++ b/src/pkg/hub/assets/dashboard.html @@ -1711,14 +1711,14 @@

0 ? 'Enforcement demonstrated' : 'Enforcement not observed') + ' \u2014 ' + blk + ' prohibited write(s) blocked; ' + att + ' observed agent write attempts through the guardrail proxy (' + (att ? evidencePct(blk / att) : EVIDENCE_NULL) + ' blocked). Counts do not rule out bypasses or false positives.'; } else out.writes = ''; var fam = v.family_false_approve_rate || {}, fparts = []; diff --git a/src/pkg/hub/evidence_drilldown_ui_test.go b/src/pkg/hub/evidence_drilldown_ui_test.go index 9cf766f77d..e402d849e1 100644 --- a/src/pkg/hub/evidence_drilldown_ui_test.go +++ b/src/pkg/hub/evidence_drilldown_ui_test.go @@ -622,7 +622,7 @@ func TestEvidenceFleetPanelsRender(t *testing.T) { `Review gatewould have passed151230.0% of 12`, // Findings generated from the fleet fixture, one per panel with data. "
  • AI agents wrote 32 PRs: 25 merged (78.1%) and 3 of those were reverted within 14 days. Humans wrote 12 PRs: 10 merged (83.3%) and 1 of those was reverted within 14 days.
  • ", - `

    Agents attempted 53 writes to GitHub through the guardrail proxy and 5 (9.4%) were blocked; a blocked write never reached GitHub.

    `, + `

    Enforcement demonstrated — 5 prohibited write(s) blocked; 53 observed agent write attempts through the guardrail proxy (9.4% blocked). Counts do not rule out bypasses or false positives.

    `, `class="evidence-bar merged"`, `class="evidence-bar reverted"`, `class="evidence-bar open"`, "no merge was held or blocked anywhere in the fleet", } { diff --git a/src/pkg/hub/evidence_findings.go b/src/pkg/hub/evidence_findings.go index 82f05a5cf6..a6ffb642af 100644 --- a/src/pkg/hub/evidence_findings.go +++ b/src/pkg/hub/evidence_findings.go @@ -265,13 +265,15 @@ func evidenceFindingsByKey(v EvidenceView) map[string]string { att += t.Attempted blk += t.Blocked } - if att == 0 { - out["writes"] = "No agent write attempts have been recorded by the guardrail proxy yet." - } else { + status, rateText := "Enforcement not observed", evidenceDash + if blk > 0 { + status = "Enforcement demonstrated" + } + if att > 0 { rate := float64(blk) / float64(att) - out["writes"] = fmt.Sprintf("Agents attempted %d writes to GitHub through the guardrail proxy and %d (%s) %s blocked; a blocked write never reached GitHub.", - att, blk, evidencePctText(&rate), evidencePlural(blk, "was", "were")) + rateText = evidencePctText(&rate) } + out["writes"] = fmt.Sprintf("%s — %d prohibited write(s) blocked; %d observed agent write attempts through the guardrail proxy (%s blocked). Counts do not rule out bypasses or false positives.", status, blk, att, rateText) } else { out["writes"] = "" } diff --git a/src/pkg/hub/evidence_report.go b/src/pkg/hub/evidence_report.go index ae70fc60c0..9817f64c84 100644 --- a/src/pkg/hub/evidence_report.go +++ b/src/pkg/hub/evidence_report.go @@ -129,7 +129,7 @@ func evidenceGrades(v EvidenceView) []evidenceGrade { } out = append(out, evidenceGrade{"Autonomy level × outcome", acmmGrade, lowBasis("Revert rate at the highest autonomy level with enough merged PRs", evidenceRevertStrong, evidenceRevertFair)}) - out = append(out, evidenceGrade{"Boundary enforcement", "Not graded", "There is no \"good\" block rate: zero blocks with many attempts means agents stayed inside their lane, many blocks means the guardrail did its job."}) + out = append(out, evidenceGrade{"Boundary enforcement", "Not graded", "Observed blocks demonstrate enforcement on those attempts; zero means not observed, not failure. Refusal frequency describes attempted-agent compliance, not enforcement quality. Counts do not rule out bypasses or false positives."}) fam := v.FamilyFalseApproveRate biggest := fam.Same diff --git a/src/pkg/hub/testdata/evidence_report_fleet.golden b/src/pkg/hub/testdata/evidence_report_fleet.golden index c0e60ccce4..13adf48826 100644 --- a/src/pkg/hub/testdata/evidence_report_fleet.golden +++ b/src/pkg/hub/testdata/evidence_report_fleet.golden @@ -79,7 +79,7 @@ At autonomy level L4, 1 hive merged 30 PRs and 3 (10.0%) were reverted or needed ### Boundary enforcement -Agents attempted 53 writes to GitHub through the guardrail proxy and 5 (9.4%) were blocked; a blocked write never reached GitHub. +Enforcement demonstrated — 5 prohibited write(s) blocked; 53 observed agent write attempts through the guardrail proxy (9.4% blocked). Counts do not rule out bypasses or false positives. Totals run since each hive's proxy started, not over the window. @@ -145,7 +145,7 @@ The review gate would have held 10 PRs; 6 of those merged and stuck (false alarm | Reviewer confidence | Strong | Share of verdicts that went wrong in the most-confident range with enough verdicts (needs two measurable ranges); Strong at or below 10%, Fair at or below 25%, otherwise Weak. | | Outcome by author kind | Fair | Revert rate of PRs written by AI agents (humans if no agent PRs); Strong at or below 5%, Fair at or below 15%, otherwise Weak. | | Autonomy level × outcome | Fair | Revert rate at the highest autonomy level with enough merged PRs; Strong at or below 5%, Fair at or below 15%, otherwise Weak. | -| Boundary enforcement | Not graded | There is no "good" block rate: zero blocks with many attempts means agents stayed inside their lane, many blocks means the guardrail did its job. | +| Boundary enforcement | Not graded | Observed blocks demonstrate enforcement on those attempts; zero means not observed, not failure. Refusal frequency describes attempted-agent compliance, not enforcement quality. Counts do not rule out bypasses or false positives. | | Same- vs cross-family review | Fair | False-approve rate of the family mix with the most verdicts; Strong at or below 10%, Fair at or below 25%, otherwise Weak. | | Per-hive evidence | Fair | Share of selected hives reporting for the window (no minimum sample); Strong at or above 80%, Fair at or above 40%, otherwise Weak. | | Shadow gates | Weak | False-alarm rate (PRs a gate would have held that merged and stuck) of the gate with the most held PRs; Strong at or below 25%, Fair at or below 50%, otherwise Weak. | diff --git a/src/pkg/hub/testdata/evidence_report_hive.golden b/src/pkg/hub/testdata/evidence_report_hive.golden index 36429e29ea..91f158e777 100644 --- a/src/pkg/hub/testdata/evidence_report_hive.golden +++ b/src/pkg/hub/testdata/evidence_report_hive.golden @@ -73,7 +73,7 @@ No autonomy level has 10 merged PRs yet, so revert rates cannot be reported. ### Boundary enforcement -Agents attempted 10 writes to GitHub through the guardrail proxy and 1 (10.0%) was blocked; a blocked write never reached GitHub. +Enforcement demonstrated — 1 prohibited write(s) blocked; 10 observed agent write attempts through the guardrail proxy (10.0% blocked). Counts do not rule out bypasses or false positives. Totals run since each hive's proxy started, not over the window. @@ -125,7 +125,7 @@ beta reports evidence for the last 90d. | Reviewer confidence | Not enough data | Share of verdicts that went wrong in the most-confident range with enough verdicts (needs two measurable ranges); Strong at or below 10%, Fair at or below 25%, otherwise Weak. | | Outcome by author kind | Not enough data | Revert rate of PRs written by AI agents (humans if no agent PRs); Strong at or below 5%, Fair at or below 15%, otherwise Weak. | | Autonomy level × outcome | Not enough data | Revert rate at the highest autonomy level with enough merged PRs; Strong at or below 5%, Fair at or below 15%, otherwise Weak. | -| Boundary enforcement | Not graded | There is no "good" block rate: zero blocks with many attempts means agents stayed inside their lane, many blocks means the guardrail did its job. | +| Boundary enforcement | Not graded | Observed blocks demonstrate enforcement on those attempts; zero means not observed, not failure. Refusal frequency describes attempted-agent compliance, not enforcement quality. Counts do not rule out bypasses or false positives. | | Same- vs cross-family review | Strong | False-approve rate of the family mix with the most verdicts; Strong at or below 10%, Fair at or below 25%, otherwise Weak. | | Per-hive evidence | Strong | Share of selected hives reporting for the window (no minimum sample); Strong at or above 80%, Fair at or above 40%, otherwise Weak. | | Shadow gates | Not enough data | False-alarm rate (PRs a gate would have held that merged and stuck) of the gate with the most held PRs; Strong at or below 25%, Fair at or below 50%, otherwise Weak. |