Re-baseline the prediction engine: the audit-verified path to a genuinely more predictive system (fix the measurement → reopen falsely-killed levers → reformulate around real list events → prospective forecasts)

[holden]

Meta / north-star. Succeeds #1113 (→ #1070 → #1081 → #1086/#1087). #1113's Phase 0 did its job — complete substrate, frozen B₀, staged export — and then a full fresh-eyes audit of the entire prediction system (6 dimensions, every critical finding adversarially re-verified against code + DB + git history; full report: docs/scoring/reports/fresh_eyes_audit_2026_06_12.md) found that the program's two governing conclusions — "we're at a metadata ceiling" and "grades are power-bound" — are substantially wrong. The ceiling has never been honestly measured: the board is gated by a mislabeled lift cap, computed through a quantized/clamped eval path, over an objective bucket containing a canon leak, on a baseline missing the season's one validated metadata win, against a task that for 5–7 of 10 lists corresponds to no real future event. This issue is the re-baseline: fix the measurement, reopen the falsely-killed levers, reformulate the task around real list events, redesign the content bet, and pivot the endgame to prospective forecasts. The #1081 honesty protocol carries forward unchanged — strengthened, in fact, by new guardrails the audit showed we need.

---

0. Status — where we actually are (2026-06-12, all claims verified)

The B₀ board (fresh substrate, objective recall@K) re-read with the binding cause attached:

list	obj recall	n_pos	published grade	actual binding cause	honest reading
1001_movies	0.526	114	insufficient	lift-ratio gate (1.429 < 1.5, misses by 0.56 SE)	Wilson LB 0.435 = MODERATE band, today
afi_100	0.476	42	moderate	—	reconstruction, not prediction (list closed 1998)
national_film_registry	0.282	78	insufficient	lift-ratio gate (1.22)	LOW band; eval can't see retro inductions (the real NFR event)
tspdt_1000	0.242→0.273	33	low	score quantization cost it 3pp	LOW band
ebert_great_movies	0.217	60	insufficient	lift-margin gate (0.017)	genuinely ~no lift over popularity — and list is closed
sight_sound_directors	0.184	49	low	—	reconstruction until 2032
letterboxd_top_250	0.151	33	insufficient	lift-margin gate (0.0)	genuinely no lift; 280/250 stale positives
criterion	0.145	69	low	—	LOW band
sight_sound_critics	0.114	35	low	—	reconstruction until 2032
cult_movies_400	0.061	33	low	—	static list, reconstruction

And every number above was measured under five verified defects (§2), so even this corrected reading is provisional until the B1 re-freeze.

1. What we're doing right (keep all of it — grade A−)

The audit's unanimous finding: the honesty machinery is genuinely above industry standard and is the program's best asset. Specifically: pre-registration with falsification thresholds; the sacred-holdout discipline (and the Phase 0 decision to HOLD the spend — verified correct, twice over); the objective-vs-full split with circularity disclosure; full-pool evaluation (the curated-negatives gaming was caught and fixed by us, earlier); the append-only experiment ledger; real abstention (models that fail stay unserved); honest nulls filed as reports; the two-number discipline. The research auditor graded the overall approach B vs. state of the art — the deficit is measurement validity and architecture, not rigor culture. Nothing in this issue weakens a guardrail; several get stronger.

2. The five verified defect clusters (full evidence in the report)

1. Mislabeled scoreboard. Only two caps can produce insufficient: n_pos<10 and the popularity-lift gate (margin≥0.05 AND ratio≥1.5, point estimates, no uncertainty). All four insufficient temporal lists have n_pos≥33 — they are lift-gate failures labeled as evidence failures, and the gate becomes mathematically unsatisfiable when the baseline exceeds 2/3 (every static cell is already ungradeable). The Phase 0 "power-bound" diagnosis was wrong for them.
2. Broken eval path → unsafe kills. Bus.score rounds to 0.1 and clamps at 100 inside the eval; ties break by movie id. tspdt's true number is 0.273. :signed weights saturate the clamp into ~10k-way ties — the recorded ":signed = 0.0" kill measured the clamp (clamp predates :signed, git-verified); with exact scoring signed ties simplex on criterion and beats it on tspdt. The E4 pooled kill ran signed through the same clamp. The :signed and pooling kills are unsafe; hierarchical shrinkage was never tried at all.
3. Baseline integrity. B₀ has zero box-office signal: migration 20260608140000 disabled the raw codes while the band rows it presumed were never seeded (0 band rows in prod's 629 metric_definitions; the fresh pull imported the broken catalog). Separately, person_quality_score is a canon leak in the objective bucket (canonical membership ×10, includes the target film, no time cutoff, not stripped). Oscar wins/noms never enter any model (normalization_type='custom' rejected); Cannes/Berlin codes are permanently dead (abbreviation mismatch); venice_golden_lion fires on all 2,295 Venice wins. 45.7% of 1001 members have no director credit in movie_credits (credits sit unmaterialized in tmdb_data blobs).
4. Task-formulation mismatch (worst grade: D+). The DB stores only current-union membership — no addition events, no edition diffs, no NFR induction years — so "predict the next edition's additions" (the product goal) currently has no labels. 5/10 lists are closed/frozen; their grades are leave-period-out reconstructions presented as predictions. Release-decade holdouts structurally can't see retrospective additions (NFR/criterion's dominant mode). ~34% of the eval pool can never be labeled positive (NULL imdb_id) + ~32% is shorts: a symmetric gate shrinks the 2010s pool 223,670→90,578 losing zero members. Temporal censoring counts future members as misses (1001's 2020s pool: ~20% post-edition impossible negatives).
5. Phase B/C as staged = near-certain false win. The 4,952-film export covers 100% of members and ~0.4% of the eval pool → any LLM feature becomes a membership-presence indicator (coverage circularity). Model-memory leakage has zero mitigation (title+year identifies the film; median synopsis 39 words → scores must come from memorized consensus). The export is 55% director-less, member-asymmetric. The +1pt keep gate is below the noise floor (recall quantum 0.9–3.0pt, SE ±4–8pt). And the $0 test — gating the already-wired TF-IDF text features on recall@K — was never run. Also: the static "sacred" holdout (seed-1337 member split) has been scored by every matrix run all season — it is informationally spent; the planned post-Phase-C promote could not have delivered honest grades for 7/10 lists.

3. The plan — five phases, same one-lever-per-trip discipline

R1 — ONE bundled, pre-registered B1 freeze (measurement integrity; days)

The bundling rule is load-bearing: applied one-by-one, these fixes move the board in opposite directions and we could stop at the most favorable intermediate state. So: log directional predictions per fix first, apply all, mint B1 (B₀ archived and committed, never overwritten; cross-version grade comparisons forbidden).

• Exact (unrounded, unclamped) scoring in the eval path + seeded tie randomization
• CatalogSeed.seed! — restore the 66 band rows (dev + prod, deliberately, knowing it changes served scores)
• Quarantine person_quality_score from objective_only; derive canon_overlap_codes from the catalog (closes the dormant bfi_top_100 leak)
• Admit oscar_wins/oscar_nominations; fix CANNES/BERLINALE/Venice mappings
• Grade taxonomy: NO_LIFT ≠ INSUFFICIENT; uncertainty-aware lift test replacing the 1.5× point-estimate knife-edge; binding cap reason exported on every board row
• Symmetric eligibility charters per list (imdb-observability; feature-length where the list excludes shorts) — derived from external list rules, frozen before any recomputation
• Label hygiene: letterboxd 280→250; seed the undersampling draw; delete the pre-1920 filter
• Commit all baseline artifacts (B₀/B1 json, reports, export task + jsonl) — currently untracked
  Exit gate: B1 published with per-fix directional predictions vs outcomes; no grade published from any partial combination.

R2 — Reopen the falsely-killed model levers on the fixed eval (days; holdout-free)

• :signed / unnormalized extraction re-test (already ties/beats simplex in controlled reruns)
• Hierarchical partial pooling (shrinkage toward cross-list/cluster mean — the textbook lever for n_pos 33–114, never tried)
• Two-stage retrieve-then-rerank: linear bus as retriever → expensive offline reranker on the top-K shortlist (hard negatives = the canon look-alikes from the PU diagnostic) → reranker score stacked as ONE bus feature (stays linear-servable; offline batch means the linear constraint was never real)
• One joint pre-registered family for label machinery: undersampling ratio × hard-negative/full-pool training × censoring-aware masking of recent cohorts

R3 — Reformulate the task around real list events (1–2 weeks)

• list_membership_events: NFR induction years (public record), 1001 edition diffs (published), TSPDT annual archives, Criterion spine/release dates → "predict additions" becomes trainable; NFR alone = ~25 labeled positives/year
• Split the board: forward-prediction lists (1001, NFR, criterion, tspdt, letterboxd) vs reconstruction benchmarks (afi, ebert, cult, both S&S until 2032), labeled as such everywhere incl. the public site
• Next-cohort evaluation with per-list eligibility frontiers; rolling-origin pooled grading (pool binomial counts across strictly-prior-trained origins before the Wilson bound) — on real-event lists only
• Redefine the sacred instrument: fresh-seed member splits never touched by any ledger row (the seed-1337 static split is spent), or prospective events (R5)

R4 — Content channel, redesigned (supersedes #1113 Phases B/C as staged)

• TF-IDF recall@K gate first ($0 — the harness exists)
• Symmetric substrate backfill: materialize credits from tmdb_data blobs + OMDb full plots for members AND eval-decade pools alike, coverage parity reported as a gate (member-targeted backfill = circularity one level down)
• Memorization audit before any extraction spend (~$20 on the existing export): direct membership-recovery probe + blinded-vs-unblinded arms on famous/obscure matched pairs
• LLM features default to the canon_overlap bucket unless the blinded audit clears them; extraction coverage must be label-blind (frozen pre-registered retriever score or full pool — never a member-enriched set)
• Keep gate re-specced above the noise band (pooled across origins; threshold ≥ refit-variance envelope)

R5 — Prospective registered forecasts (the endgame; start now, runs forever)

• Before each real event, freeze and publish (commit hash + ledger row) top-K forecasts: NFR December 2026 (first scoring event, ~6 months out), TSPDT annual re-rank, Criterion slate, Letterboxd rolling, next 1001 edition
• Grade on actual additions. This dissolves LLM pretraining leakage (post-cutoff additions can't be memorized), escapes the spent holdouts and the over-fingered validation tier, makes the closed-list fiction moot, legitimately re-admits cross-list membership as observable-at-forecast-time signal, and converts the public honesty claim into externally verifiable prediction — the product's actual promise. Power accrues every cycle.

4. Success criteria (pre-committed)

• R1: B1 board published with cap-reason labels; the misdiagnosis class ("insufficient" meaning "no lift") is structurally impossible to repeat.
• Issue-level: ≥50% of forward-prediction lists at MOD on the reformulated, honest instrument (1001 plausibly immediate; NFR/tspdt/criterion via R1–R3). HIGH requires point recall ~0.6 at pooled n≥300 — reachable only via R3 pooling + R5 accrual; a documented "this list caps at MOD" remains an acceptable honest outcome.
• First external scorecard: NFR induction, December 2026 — the first grade in the program's history that no amount of leakage can inflate.

5. Guardrails (additions to the #1081 protocol, from the audit's critic)

• Bundling rule: integrity corrections ship as one versioned freeze or not at all.
• Baselines are versioned, archived, and committed — "re-freeze B₀" is retired vocabulary; B1, B2… only.
• The validation tier is an adaptive budget: a season of looks at the same 2010s/seed-1337 slice means fixed-eval reruns are exploratory only; confirmations need fresh splits or future events.
• Eligibility charters from external list rules only, frozen before recomputation — never tuned while watching a gate margin (the 1001 lift ratio can be flipped by pool pruning).
• Backfills coverage-symmetric, parity reported.
• LLM-derived features are guilty until proven blind.
• Serving/public divergence is a tracked workstream: the live /algorithms page currently shows objective chips including PQS, prediction-labeled grades for closed lists, and prod models the trainer can no longer reproduce — the R1 reseed must be deployed deliberately, with the public copy updated to the new taxonomy.

6. Definition of done

• R1 B1 freeze complete + published (board re-read, cap reasons visible, artifacts committed)
• R2 levers each kept-or-killed on the fixed eval with reports (signed, shrinkage, two-stage, label family)
• R3 addition-event data landed; board split forward/reconstruction; rolling-origin grading live
• R4 content channel gated honestly (TF-IDF → memorization audit → blind extraction), kept-or-killed with reports
• R5 first frozen forecast published before the NFR December 2026 announcement; scored when the inductions land
• Honest before/after vs B1 posted here; north-star progress or a documented ceiling stated plainly

Method note: this re-baseline comes from a 31-agent audit (6 fresh-eyes auditors + adversarial verification of every critical finding + completeness critic), ~3.9M tokens. Full evidence with file:line citations: docs/scoring/reports/fresh_eyes_audit_2026_06_12.md.

[holden]

Implementation children filed

• B1 freeze + lever re-tests (#1114 R1+R2 implementation): one bundled measurement-integrity freeze, then re-open the falsely-killed model levers on the fixed eval — two gated sessions #1116 — B1 freeze + lever re-tests (R1+R2). Two gated sessions: the bundled measurement-integrity freeze (directional predictions pre-logged in the issue per the bundling rule), then the falsely-killed levers (signed / hierarchical shrinkage / label family / two-stage reranker) on the fixed eval. Recommended model: Fable 5 for Session 1 and the reranker; Opus 4.8 acceptable for the mechanical lever re-tests.
• Addition-event ground truth spike (#1114 R3 prep): build list_membership_events from public records — NFR induction years, 1001 edition diffs, TSPDT annuals, Criterion spine dates #1115 — Addition-event ground truth spike (R3 prep). list_membership_events from public records (NFR induction years, 1001 edition diffs, TSPDT annuals, Criterion spine dates). One session, pure data, zero model/eval changes, runs in parallel with B1 freeze + lever re-tests (#1114 R1+R2 implementation): one bundled measurement-integrity freeze, then re-open the falsely-killed model levers on the fixed eval — two gated sessions #1116. Recommended model: Opus 4.8.

R4 (content channel) and R5 (prospective forecasts) stay gated here: R4 must not start before its memorization audit, and R5's first deliverable (the frozen-forecast protocol + the NFR December 2026 forecast) becomes actionable once #1116 Session 1 lands an honest B1 board to forecast from.