diff --git a/docs/research/dead-ends.md b/docs/research/dead-ends.md index f4a978a..4de173d 100644 --- a/docs/research/dead-ends.md +++ b/docs/research/dead-ends.md @@ -456,7 +456,7 @@ Artifacts: `scripts/probe_liver_zonation.py`, `tests/integration/test_liver_zona **Why the freeze 5.25 is not a generalization number:** contamination (above) + composition confound — 16 IV drugs (32%; the oral-tuned pipeline under-predicts all ~10×), 8 deliberately-adversarial OATP substrates (subset AAFE 8.45), 1 prodrug outlier (sepiapterin ~4900×). The cleanest extractable slice (hard-clean ∩ oral, N=24 novel oral NMEs) is AAFE ≈4.0 vs the 107-holdout ~2.62 on the same local stack — which **corroborates the documented prospective OOD degradation** (prospective 3.27, novel-chemotype F under-prediction), not a new signal. -**Telltale if it returns:** any citation of "N50 generalization AAFE 5.25" (or a subset of it) as evidence the 2.743 headline is cherry-picking-inflated. The instrument is invalid; the number is quarantined. Fix shipped: `scripts/build_n50_exclusion.py` now keys on InChIKey-14 across all corpora + DrugBank with an `--audit` gate; `holdout_n50.json` + the freeze carry an `invalidated` block; `run_n50_benchmark.py` refuses a freeze on an invalidated file. Full record: [n50_2026q2_invalidation.md](./n50_2026q2_invalidation.md). Re-curation of a clean N50' is deferred. +**Telltale if it returns:** any citation of "N50 generalization AAFE 5.25" (or a subset of it) as evidence the 2.743 headline is cherry-picking-inflated. The instrument is invalid; the number is quarantined. Fix shipped: `scripts/build_n50_exclusion.py` now keys on InChIKey-14 across all corpora + DrugBank with an `--audit` gate; `holdout_n50.json` + the freeze carry an `invalidated` block; `run_n50_benchmark.py` refuses a freeze on an invalidated file. Full record: [n50_2026q2_invalidation.md](./n50_2026q2_invalidation.md). Re-curation of a clean N50' is deferred. **Status 2026-07-07 — a clean N50' cannot be sourced from repo data (feasibility assessment, [n50_prime_feasibility_2026-07-07.md](./n50_prime_feasibility_2026-07-07.md)):** the genuinely-clean candidate pool from `clinical_pk.json` is **0** (331 → 177 with Cmax → 16 after removing holdout+train → 0 valid: the 2 nominal survivors are guanfacine ER = holdout HCl-salt IK14 leak and lanthanum carbonate = inorganic combo), with **0/≥3** OATP1B1 substrates. N=50 is infeasible from repo sources (margin −50) because the clinical reference *is* the training source (167/331 names in holdout+train) and DrugBank covers 14,154 IK14 blocks. A real N50' needs ~50 fresh novel molecules with primary-source Cmax — a **human-led** curation (an agent must not fabricate primary values for a never-touch instrument). The E4 rule could relax to hard-corpora-only *if* the benchmark commits to public-clone (DrugBank hidden doesn't leak), but even so the repo pool is ~2–5. Deferred to a separately-authorized human curation cycle; tooling + spec are ready. --- diff --git a/docs/research/experiment-log.md b/docs/research/experiment-log.md index ea419a1..ad6e9a1 100644 --- a/docs/research/experiment-log.md +++ b/docs/research/experiment-log.md @@ -10,6 +10,33 @@ Reverse-chronological. The project README carries only the **current** headline --- +## 2026-07-07 (cont.) — N50' clean re-curation: infeasible from repo data (pool=0), deferred to human-led curation + +Scoped the clean N50' secondary-holdout curation (the unbiased-generalization instrument the +cherry-picking caveat calls for; the 2026Q2 attempt was invalidated for contamination, DE-53). A +3-agent investigation (spec/requirements, IK14 tooling, candidate-pool feasibility) answered the +make-or-break question first: **is a clean N=50 even sourceable? No.** + +- **Clean pool = 0.** `clinical_pk.json`: 331 drugs → 177 with Cmax → 16 after removing the + 107-holdout + 76-train (name **and** IK14) → 2 nominal clean → **0 valid** (guanfacine ER = holdout + HCl-salt IK14 leak; lanthanum carbonate = inorganic combo). OATP1B1 non-statin substrates in the + clean pool: **0 / ≥3 required.** N=50 infeasible from repo sources by −50; even N=1 not achievable. +- **Why:** the clinical reference *is* the training source (167/331 names in holdout+train); of the 16 + survivors, 11 are in hard corpora and 3 have unparseable SMILES. E4 (DrugBank-absent) is near-impossible + — DrugBank covers 14,154 IK14 blocks (the 2026Q2 set was 47/50 in DrugBank). +- **E4 / public-clone reframe:** the 2.743 headline is public-clone (DrugBank hidden → no enrichment + leak), so E4 could relax to hard-corpora-only *if* the benchmark commits to public-clone — but even + relaxed the repo pool is ~2–5, not 50. +- **Deferred to a human-led cycle.** A real N50' needs ~50 fresh novel molecules with primary-source + Cmax (no back-calc), CID-verified SMILES, ≥3 non-statin OATP1B1 substrates, oral-majority. **Integrity + constraint:** an agent must not generate primary-source Cmax/SMILES/DOI for a never-touch instrument — + a fabricated value is undetectable by the IK14 gate and would be worse than the 2026Q2 contamination. + The tooling (#96) and spec are ready; the blocker is human-verified data sourcing. + +Also flagged a tooling gap: the IK14 gate is defeated by salt forms (guanfacine HCl IK14 ≠ free-base +IK14), a false-negative-only leak a curator would catch but worth a salt-strip fix. Full record: +[n50_prime_feasibility_2026-07-07.md](./n50_prime_feasibility_2026-07-07.md); DE-53 status line. + ## 2026-07-07 — Two DE-44 kill-tests run: adaptive PI walled (DE-55), invivo-F-prior killed via placebo (DE-56) Ran the two cheap pre-registered kill-tests DE-44 flagged as "survivors to test," on the 107-holdout diff --git a/docs/research/n50_prime_feasibility_2026-07-07.md b/docs/research/n50_prime_feasibility_2026-07-07.md new file mode 100644 index 0000000..9e06669 --- /dev/null +++ b/docs/research/n50_prime_feasibility_2026-07-07.md @@ -0,0 +1,99 @@ +# N50' Clean Re-Curation — Feasibility Assessment (2026-07-07) + +**Status: infeasible from repository data → deferred to a human-led curation cycle.** + +Follow-on to the 2026Q2 invalidation ([n50_2026q2_invalidation.md](./n50_2026q2_invalidation.md), +dead-ends.md DE-53). After the exclusion tooling was fixed to key on InChIKey-14 (PR #96), the +next step was to curate a genuinely-clean N50' (the unbiased-generalization instrument the +cherry-picking caveat calls for). This assessment answers the make-or-break question **before** +any curation: *is a clean N=50 even sourceable?* It is not. + +## Verdict + +A clean N50' secondary permanent holdout **cannot be sourced from the existing repository data**. +The genuinely-clean candidate pool is **0**. N=50 is infeasible from repo sources by a margin of +**−50**; even N=1 is not cleanly achievable. Building N50' requires importing ~50 fresh novel +molecules with primary-source clinical Cmax from outside the current reference set — a +**human-verified** curation effort, not an automated/agent execution. + +## The feasibility funnel (`data/reference/clinical_pk.json`) + +| Step | Filter | Count | +|---|---|---| +| 0 | Drugs in `clinical_pk.json` | 331 | +| 1 | Non-null clinical `cmax_mg_L` | **177** | +| 2 | After removing the 107-holdout + 76-train (by name **and** InChIKey-14) | **16** | +| 3 | After removing hard-corpus IK14 hits + unparseable SMILES → nominal clean | 2 | +| **3′** | **After inspection → genuinely valid** | **0** | +| — | OATP1B1 non-statin substrates in the clean pool (spec requires ≥3) | **0** | + +- **Why step 2 is brutal:** 167 of the 331 `clinical_pk` names are literally in holdout+train — + the clinical reference and the holdout corpus are nearly the same file. Of the 16 survivors, + **11 are in hard training corpora** and **3 have unparseable SMILES**. +- **The 2 nominal survivors both fail on inspection:** + - `guanfacine er` — FALSE CLEAN. Its stored SMILES is the **HCl salt** (IK14 `DGFYECXYGUIODH`), + whose connectivity block differs from the free base (`INJOMKTZOLKMBF`), so it dodged the + holdout-side IK14 match — but **guanfacine is explicitly in the holdout list**. This is the exact + salt/synonym leak class the tooling exists to catch; the counterion in the stored SMILES defeated + the IK14 gate (a tooling gap — see Follow-ups). + - `lanthanum carbonate` — inorganic multi-fragment salt (2×La³⁺ + 3×carbonate), not a single-active + small molecule, and present in DrugBank. + +## Why the constraints bind + +- **E1–E3** (107-holdout / MMPK-Cmax / TDC-CLint): the clinical reference **is** the training source. +- **E4** (DrugBank-absent): DrugBank covers **14,154** InChIKey-14 blocks; the 2026Q2 attempt had + **47/50** in DrugBank. "Not in DrugBank" ≈ "not a catalogued drug" → only very-new molecules qualify. +- **E5** (not in any validation file): the prospective N=28 novel-drug set is already consumed (it + lives in `data/validation/`), so those cannot be reused. + +## The E4 / public-clone reframe (a yield lever, not a rescue) + +The 2.743 headline is **public-clone** (DrugBank hidden). In that regime a drug being *in* DrugBank +does **not** leak into its prediction — the fup/pKa/logP enrichment is disabled. So E4's +"DrugBank-absent" rule is over-conservative *for a public-clone benchmark*. **If** the N50 benchmark +commits to running public-clone (DrugBank hidden at freeze — note `run_n50_benchmark.py` does **not** +currently hide it), E4 could relax to **hard-corpora-clean only** (the actual fitted-target leakage). +But even with E4 relaxed, the repo pool is only ~2–5 — still far short of 50. The reframe improves a +*fresh* curation's admissible yield; it does not rescue the repo pool. + +## What a real N50' requires (deferred, human-led) + +1. ~50 fresh novel molecules (likely 2024–2026 NMEs **not** already in the prospective N=28 set), each with: + - Primary-source observed Cmax — **no back-calculation from AUC + t½** (A2), ≥2 independent sources + (the prospective adversarial-verification discipline); + - PubChem-CID-verified canonical SMILES (A3), canonicalized to the free base; + - Source DOI / table reference (A5); + - ≥3 **non-statin** OATP1B1 substrates; + - oral-majority, with any IV / adversarial-transporter drugs quarantined into a **separately-reported + subset** (the 2026Q2 composition-confound lesson). +2. Each candidate gated **pre-curation** through `scripts/build_n50_exclusion.py --audit` (zero + hard-corpus hits) + a DrugBank IK14 review. + +## Integrity constraint — why this is human-led, not agent-automated + +An agent must **not** generate the primary-source Cmax / SMILES / DOI values for a *never-touch* +generalization instrument. A single hallucinated value would invalidate the instrument's entire +purpose — **worse** than the 2026Q2 contamination, because a fabricated "primary source" is +undetectable by the IK14 gate (which checks structure membership, not value provenance). The sourcing +and verification of the actual clinical values must be human-performed. An agent may scaffold (draft +candidate lists, run the IK14 audit, check SMILES parse) but may not be the source of record for the +observed values. + +## Follow-ups (tooling, non-blocking) + +- **Salt canonicalization in the IK14 gate.** The guanfacine case shows a counterion in a stored + SMILES yields a different IK14 than the free base, silently defeating both the exclusion match and + the audit. `build_n50_exclusion.py` should strip salts / take the largest organic fragment before + computing IK14 on both the corpus and the candidate side. (Low priority — it made a *contaminated* + drug look clean here, i.e. it only ever produces false-negatives on exclusion, which a human curator + would catch, but it is a real gap.) + +## Status + +**Deferred to a separately-authorized, human-led curation cycle.** The tooling (IK14 exclusion + +`--audit` gate, PR #96) and the governance spec +(`docs/_internal/specs/2026-04-22-n50-secondary-holdout-design.md`) are ready. The blocker is the +sourcing of fresh, primary-verified clinical data, which is out of scope for automated curation. Until +then, the cherry-picking caveat stands: the 107-holdout AAFE (2.743) is a point estimate whose +true-generalization value is not independently instrumented.