Finding
Cheap Verifiers, Large Blind Spots (arXiv:2609.01345, submitted 2026-09-01) reports that verifier blind spots can grow with student capability, a frontier verifier can erase much of the savings by escalating nearly half of hard queries, naive fine-tuning on verifier-rejected data can collapse the student, and metrics computed through the same verifier can remain apparently healthy while true delivered error degrades sharply.
Evidence status: originating-team measured plus theory/synthetic validation, not independently reproduced by RuV.
Missing RuV primitive
Dream Machine and MetaHarness already separate candidate generation from evaluation, but cost-saving cascades and self-improvement loops also need an evaluator that is outside the optimization and routing loop.
Add a VerifierAuditReceipt concept and benchmark contract with three distinct identities:
- student or candidate
- operational verifier used for routing/rejection
- hidden auditor used only for held-out measurement
The hidden auditor and its labels must never feed training, routing, candidate selection, or threshold tuning for the experiment family.
Metrics
- verifier blind-spot fraction
- escalation rate
- reported in-loop error
- independently audited delivered error
- false accept / false reject
- model cost, latency, and token cost
- quality after any corrective update
- divergence between dashboard quality and audited quality
Falsification
A simpler random human/gold audit sample is the mandatory control. If it detects degradation at materially lower cost with comparable confidence, prefer it over a model auditor.
Security and governance
Verifier confidence is evidence, never authority. Independent audit state is write-protected from the optimization loop. Tacet/statistical validity history must include repeated audits and may not be reset by changing run IDs.
Acceptance gate
No self-improving cascade may claim improved reliability from its own verifier metrics alone. Promotion requires independently audited delivered quality within the declared error bound and full cost accounting.
Finding
Cheap Verifiers, Large Blind Spots(arXiv:2609.01345, submitted 2026-09-01) reports that verifier blind spots can grow with student capability, a frontier verifier can erase much of the savings by escalating nearly half of hard queries, naive fine-tuning on verifier-rejected data can collapse the student, and metrics computed through the same verifier can remain apparently healthy while true delivered error degrades sharply.Evidence status: originating-team measured plus theory/synthetic validation, not independently reproduced by RuV.
Missing RuV primitive
Dream Machine and MetaHarness already separate candidate generation from evaluation, but cost-saving cascades and self-improvement loops also need an evaluator that is outside the optimization and routing loop.
Add a
VerifierAuditReceiptconcept and benchmark contract with three distinct identities:The hidden auditor and its labels must never feed training, routing, candidate selection, or threshold tuning for the experiment family.
Metrics
Falsification
A simpler random human/gold audit sample is the mandatory control. If it detects degradation at materially lower cost with comparable confidence, prefer it over a model auditor.
Security and governance
Verifier confidence is evidence, never authority. Independent audit state is write-protected from the optimization loop. Tacet/statistical validity history must include repeated audits and may not be reset by changing run IDs.
Acceptance gate
No self-improving cascade may claim improved reliability from its own verifier metrics alone. Promotion requires independently audited delivered quality within the declared error bound and full cost accounting.