Skip to content

docs(dead-ends): DE-57 — CLint epistemic-uncertainty gate fails (uncertainty-aware meta foreclosed) #107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jam-sudo merged 1 commit into from
Jul 9, 2026
Merged

docs(dead-ends): DE-57 — CLint epistemic-uncertainty gate fails (uncertainty-aware meta foreclosed) #107

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 188 additions & 0 deletions data/validation/clint_uncertainty_gate_2026-07-08.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
{
"seed": 20260422,
"K": 30,
"n": 107,
"boot_resamples": 10000,
"note": "gate: |rho|>=0.30 with CI excluding 0 = PASS",
"corr": {
"U_boot": {
"r_meta": {
"rho": -0.255,
"ci": [
-0.423,
-0.069
]
},
"e_eng": {
"rho": -0.006,
"ci": [
-0.2,
0.184
]
},
"e_ml": {
"rho": -0.122,
"ci": [
-0.303,
0.071
]
},
"diff": {
"rho": 0.119,
"ci": [
-0.073,
0.301
]
},
"obs_mag": {
"rho": 0.002,
"ci": [
-0.182,
0.188
]
}
},
"U_ad": {
"r_meta": {
"rho": 0.032,
"ci": [
-0.159,
0.227
]
},
"e_eng": {
"rho": 0.145,
"ci": [
-0.059,
0.342
]
},
"e_ml": {
"rho": 0.005,
"ci": [
-0.177,
0.186
]
},
"diff": {
"rho": 0.121,
"ci": [
-0.069,
0.311
]
},
"obs_mag": {
"rho": -0.018,
"ci": [
-0.208,
0.176
]
}
},
"clint_mean": {
"r_meta": {
"rho": 0.026,
"ci": [
-0.163,
0.21
]
},
"e_eng": {
"rho": -0.007,
"ci": [
-0.181,
0.171
]
},
"e_ml": {
"rho": 0.012,
"ci": [
-0.184,
0.202
]
},
"diff": {
"rho": -0.008,
"ci": [
-0.199,
0.178
]
},
"obs_mag": {
"rho": 0.222,
"ci": [
0.034,
0.397
]
}
}
},
"trend": {
"U_boot": {
"r_meta": [
0.56,
0.475,
0.368,
0.351
],
"e_eng": [
0.625,
0.673,
0.495,
0.733
],
"diff": [
0.068,
0.18,
0.049,
0.321
]
},
"U_ad": {
"r_meta": [
0.449,
0.347,
0.515,
0.439
],
"e_eng": [
0.584,
0.518,
0.644,
0.775
],
"diff": [
0.108,
0.083,
0.108,
0.316
]
},
"clint_mean": {
"r_meta": [
0.489,
0.357,
0.463,
0.441
],
"e_eng": [
0.649,
0.588,
0.68,
0.607
],
"diff": [
0.149,
0.208,
0.056,
0.206
]
}
},
"u_boot_by_ad": {
"in_ad_mean": 0.156,
"out_ad_mean": 0.168,
"n_out_ad": 26
},
"verdict": "FAIL (null)"
}
14 changes: 14 additions & 0 deletions docs/research/dead-ends.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -494,6 +494,20 @@ Artifacts: `scripts/probe_liver_zonation.py`, `tests/integration/test_liver_zona

---

### DE-57 — Uncertainty-aware meta: no per-drug CLint epistemic-uncertainty signal exists to gate it (2026-07-08)

**Date:** 2026-07-08

**The last §9 CLint-floor candidate able to sidestep error-cancellation, tested and foreclosed.** The idea does not improve the CLint point estimate (floored at R²≈0.24; every prior improvement tripped error-cancellation, §1/§2). Instead it exploits the floor's *heteroscedasticity*: if we knew **per-drug** which CLint predictions are unreliable, the meta could down-weight the engine track there — a *weighting* change that perturbs no track's residuals, so it cannot trip error-cancellation. **Distinct from DE-55**, which killed divergence/AD/magnitude and the *propagated MC-output* uncertainty (ρ=0.039); the shipped CLint `Distribution` CV is the fixed constant `_CLINT_CV=1.0` (`predict/adme.py`), which is *why* DE-55's propagated signal was flat. This test constructs a genuine per-drug epistemic uncertainty two independent ways: **U_boot** = SD of log10(CLint) across a K=30 XGBoost bootstrap ensemble on the 1,213-compound hepatocyte training set (range [0.069, 0.412], non-degenerate); **U_ad** = 1 − max Tanimoto(Morgan) to that set.

**Result (gate = Spearman ρ + bootstrap 95% CI, seed 20260422, N=107).** Core null: **ρ(U_boot, e_eng) = −0.006** [−0.20, +0.18] — CLint epistemic uncertainty does not predict engine error, though the engine *is* the CLint-fed track. Actionable null: **ρ(U, e_eng−e_ml) ≈ +0.12** for **both** proxies (U_boot [−0.07, +0.30], U_ad [−0.07, +0.31]) — CI∋0, no reweighting signal. The one CI-significant term, ρ(U_boot, r_meta) = −0.255 [−0.42, −0.07], is sub-threshold **and wrong-signed** (higher CLint uncertainty ⇒ *lower* meta error — the meta already handles uncertain-CLint drugs well). Confound-clean: ρ(U_boot, obs-magnitude) = +0.002.

**Why it failed — §10.** The binding wall is bioavailability-F blindness, shared across all four tracks and **orthogonal to CLint**; Cmax error is F-driven, so no CLint-derived signal can discriminate it. A perfect CLint-uncertainty router would have nothing to route on. Per §4 the gate precedes integration → the uncertainty-aware meta is **not built**.

**Telltale if it returns:** "route the meta by how uncertain the CLint prediction is." No per-drug CLint uncertainty predicts Cmax error (ρ≈0 on the actionable target for two independent constructions); the discriminating error is F, not CLint. Artifacts: `scripts/clint_uncertainty_gate.py`, `data/validation/clint_uncertainty_gate_2026-07-08.json`.

---

## 3. When to consult this list

- Before writing a design spec for any accuracy improvement.
Expand Down
2 changes: 1 addition & 1 deletion docs/research/diagnosis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ A verified literature review (IMI OrBiTo Part 4, Ahmad 2020/2021, 58 APIs, >200

**Correctness cross-link.** The post-FLUX-1 audit found the canonical well-stirred driving force is blood unbound `fu_b = fup/Rb` (Pang & Han 2019) — independently corroborating that the engine's RBP concentration-basis cluster (clearance sink applies `fup`, not `fup/RBP`) is a deviation from the textbook equation, not merely an internal inconsistency. Spec: `docs/_internal/specs/2026-06-04-rbp-concentration-basis-design.md`. Largely off the `realize_means` headline path (RBP clamped ~[0.5,1.5]); a correctness item, not a benchmark lever.

**Open follow-ups (research in flight 2026-06-04):** whether any *breakthrough* attacks the CLint floor without tripping error-cancellation (per-isoform rCYP direct prediction, kcat×Km×[E] structured decomposition, multi-assay target-denoising, calibrated-CLint-uncertainty→uncertainty-aware meta); and whether conformal/CQR + model-form-discrepancy recalibration fixes the §empirical PI under-coverage (29.9%@90%). Both must clear the end-to-end Cmax-holdout / error-decorrelation gate, not isolated metrics.
**Open follow-ups (research in flight 2026-06-04):** whether any *breakthrough* attacks the CLint floor without tripping error-cancellation (per-isoform rCYP direct prediction, kcat×Km×[E] structured decomposition, multi-assay target-denoising, ~~calibrated-CLint-uncertainty→uncertainty-aware meta~~ — **closed, DE-57**); and whether conformal/CQR + model-form-discrepancy recalibration fixes the §empirical PI under-coverage (29.9%@90%; the PI arm is **closed, DE-55/56**). Both must clear the end-to-end Cmax-holdout / error-decorrelation gate, not isolated metrics. **Update 2026-07-08 (DE-57):** the uncertainty-aware-meta candidate — the one that structurally sidesteps error-cancellation (a weighting signal, not a point-estimate change) — was gate-tested and failed: no per-drug CLint *epistemic* uncertainty (bootstrap-ensemble SD or AD distance) predicts Cmax error (ρ≈0 on `e_eng` and on `e_eng−e_ml`), because the discriminating error is bioavailability-F, orthogonal to CLint (§10). The remaining §9 breakthroughs (rCYP, kcat×Km, multi-assay denoising) all require external enzyme-kinetics data not in the repo.

---

Expand Down
Loading