experiments.md

Experiment registry

One row per training run. Kept in git because it holds no patient data, only configs, commits and aggregate numbers. The lead session maintains it from the operating sessions' reports; report artifacts (JSON, HTML) live in the gitignored research_journal/ and on the VMs.

Recipe shorthand: v6 recipe = bundle-invariant family-restricted loss, family balancing alpha 0.5, visit-scoped concepts, icd3 backoff, medication normalization, time-to-next-event head, per-event hazard heads, randint 0, 64 lanes x 512, 2 epochs, checkpoint_every 2000.

Comparator feature sets: the tuned GBM is the strong-feature bar (609 hand-built columns, odyssey/inference/baseline_features.py). TabICL rows are fit on the basic set (~17 columns) on every dataset: its in-context forward pass re-reads the whole context at O(n^2 + n*m^2) in context rows and feature columns, so 609 columns costs ~70 GB per predict_proba call versus ~8 GB at 17 (measured 2026-08-23; tabicl_baseline.estimate_peak_gb now refuses such a fit). That is a real property of the comparator, not a tuning choice, and it keeps the MIMIC and eICU TabICL rows like-for-like. EBM rows are strong-feature on MIMIC and basic on eICU; SurvivalPFN is basic on both. Every row states its own feature set.

Data versions: MIMIC = mimiciv_3.1_v1 (292/?/37 shards). eICU v1 = eicu_2.0_v1 (spec v1: 36% of medication rows UNK, infusions a bare INFUSION_DRUG token; 134/17/17 shards). eICU v2 = eicu_2.0_v2 (spec v2 from bc7ac4c + 60310b0: HICL segment, named infusions, nurseCharting GCS, intakeOutput urine output; extraction pending the raw tables).

Landmark protocol: LANDMARK_PROTOCOL_VERSION (odyssey/inference/alerts.py) stamps every alerts row dump. v1 = per-chunk landmark-mask reset (spurious rows at chunk boundaries); v2 = state carried across chunks (85dde80); v3 = per-lane visit-bucket state, interleaved-visit duplication fixed (dbfd447); v4 (2026-08-30, 059db53) = ONE information boundary at the landmark instant for every scorer: baseline features (basic + strong + notes) now include events AT the index time t, windows (t-w, t], matching the model-side scorers and MEDS-Tab -- under v1-v3 tabular baselines saw one systematically task-relevant event less than the model at every landmark. The same commit fixes label semantics (sepsis3 first-trigger anchored at evidence completion; shock pools cuff+arterial MAP for the recurrence check; SIRS/qSOFA observed-masks require min_criteria measurable components; GCS trigger stamped at the last paired component), and acbc0e2 versions the clinical bin edges on the binner artifact (old checkpoints re-evaluate under the exact edges they trained with -- clinical_edge_version v1 -- so v4 re-eval deltas isolate the label+boundary changes, never train/eval tokenization skew). 7e4d0a0 fixes note-feature leakage: notes index at max(charttime, storetime) availability (radiology storetime lags charttime by a median 2.25h / 22% >6h; discharge ~20h / 41% >24h, measured on the real 2.2 release), so any pre-fix note_embeddings sidecar must be re-embedded before the next strong_text run (the code now refuses it). v4 baseline columns are NOT comparable with v1-v3 rows; sepsis3/shock/sirs/qsofa label-dependent numbers change at 059db53 the same way AKI numbers changed at 3d7ecbb. One-time data audit (2026-08-30, local mimiciv_3.1_v1): validate_meds_dataset(deep=True) clean, and explicit cross-split subject overlap is 0/0/0 across train 291,702 / tuning 36,463 / held_out 36,462 subjects. Rows below predating 2026-08-23 are v1 unless their Key results cell starts with [protocol vN]; a protocol re-evaluation is a NEW row for the same run name, never an edit of the old one. Wave runbook (archived): docs/archive/reeval_wave_v2.md.

Run	VM	Data	Commit	Purpose	Status	Key results
subset_run_v3	cbm	MIMIC 30 shards	4a1c0a8	visit-scoped concepts, pos_weight, on_vasopressors	done	exact top-1 60.7; same-family set 69.9 (med 35.9, diag 31.0, lab 75.2); concepts 0.62-0.99; lever inert
subset_run_v4	cbm	MIMIC 30 shards	6ac235a+	RandInt 0.25 A/B vs v3	done	exact 51.9 (-8.8); set 65.3; lever correctly signed but +0.1 pt only; retired
subset_run_v5	cbm	MIMIC 30 shards	68ba07c	bundle objective + time head + hazard heads, randint 0	done	set 74.9 (lab 80.6, med 41.9, proc 37.5, diag 23.4 [loss bug], billing 20.8); time calibrated; alerts hazard 8/12 vs GBM (mean 0.831 vs 0.798)
subset_baseline_v5	cbm	MIMIC 30 shards	68ba07c	no-bottleneck baseline for v5	done	set 76.6 (-1.7 vs v5; diag +4.9 for v5); time head equal; alerts hazard 7/12
subset_run_v6	cbm	MIMIC 30 shards	a883441	v5 + family-restricted bundle loss (diagnosis fix)	done	28,775 steps, 3h20m (0.417 s/step); set 75.7 (v5 74.9): diag 41.3 (23.4; v3 31.0), med 43.6 (41.9), proc 38.4 (37.5), lab 81.2 (80.6), billing 26.2 (20.8); concepts 0.61-0.99; time calibration 76.3/92.8/95.8 vs 75.6/92.7/95.5; alerts vs basic GBM: mean AUROC 0.835 vs 0.798, 7/12 (vasopressor and ICU all horizons, death 8h 0.894/0.799; loses AKI by <=0.02, death 24h/72h); banded interventions inert (truth -0.002); report research_journal/12. Strong tuned GBM (609 features, 2327459): hazard mean AUROC 0.835 vs GBM 0.910, hazard wins 0/12; closest vasopressor 8h 0.928/0.939, 24h 0.895/0.920, death 8h 0.894/0.911; largest gaps ICU 24h 0.869/0.959, death 24h 0.849/0.941, AKI 8h 0.777/0.897
subset_baseline_v6	cbm	MIMIC 30 shards	4298374	baseline on the fixed loss	done	28,796 steps, 3h07m; set 77.9 vs v6 75.7 (baseline +2.2): diag 41.25 vs 41.3 (tied; the v5-era 'bottleneck helps diagnoses' was the loss bug), med 47.0/43.6, proc 43.8/38.4, lab 83.2/81.2, billing 31.1/26.2; alerts pending
subset_run_v7	cbm	MIMIC 30 shards	2327459	v6 + value_embeddings (numeric value channel A/B)	done	3h41m; set 76.8 (v6 75.7): lab 82.5 (+1.3), visit +1.7, billing +1.2, proc +0.9, diag 42.0 (+0.7), med 43.5 (-0.1); concepts mean AUROC 0.797 -> 0.807 (tachy/brady/lactate +2); time head unchanged; alerts vs strong GBM 1/12: vasopressor up at all horizons (8h 0.943 beats GBM 0.939), AKI flat (0.764/0.739/0.681), death DOWN 4-6 pts (0.853/0.799/0.768); report journal 14
subset_run_v6_60	cbm	MIMIC 60 shards	2327459	learning-curve point: v6 recipe on 2x data	done, budget-limited	early stop at 31,750 steps (~1.1 epochs on 60 shards vs 2 on 30), 3h38m: set 75.8 (75.7): diag 37.9 (-3.5), med 44.6 (+1.0), proc 39.6 (+1.3), lab 81.2 (flat), billing 27.5 (+1.4); concepts 0.803 (+0.6); alerts vs strong GBM 1/12: death up +2-3 pts (8h 0.914 beats GBM 0.904), vasopressor +1.5-1.8, AKI flat, ICU -0.2 to -2.2. Not a clean scale answer: needs a fixed step budget / larger patience; report journal 17
subset_run_v8	cbm	MIMIC 30 shards	8c0a475	v7 + event_head_hidden 256 + static facts as leading inputs	done	3h28m; set 76.7 (v7 76.8, flat): proc 40.4 (+1.2), visit 35.2 (+1.5), billing 25.8 (-1.5), diag 41.3 (-0.8); concepts mean 0.796 (v7 0.807; vital-threshold concepts -1 to -4, flag); time head unchanged; alerts vs strong GBM 2/12 (vasopressor 8h 0.949/0.941, 24h 0.924/0.923); death +1.4/+2.9/+3.2 vs v7 (MLP head), AKI flat, ICU -0.4 to -1.8; report journal 18
full_run_v8	cbm	MIMIC all 292 train shards	c902eea (stream_shards)	first full-scale MIMIC run, v8 recipe, 1 epoch, early stopping off	done	137,605 steps, 16h41m; learning curve 74.6/76.5/78.5 at 14k/28k/56k; final set top-1 80.4 (subset v8 76.7, +3.7; billing +20.3, proc +10.4, med +7.2); concepts mean AUROC 0.857 (+0.061, every concept up); time NLL 0.256, same-instant 96.1%; banded interventions wrong-signed at full scale (none 36.39, truth 35.86, flip 36.31: truth-flip -0.46 pt; zero_known 5.29, zero_unknown 36.08; all 6 modes); alerts vs strong GBM 2/12, death gap closes +0.09-0.10 (8h flips to a win), AKI +0.02-0.03, ICU +0.03-0.06, vasopressor flat; report research_journal 20
full_run_v8 (N2 time-head probe)	cbm	MIMIC held-out 4 shards (banked 2.89M positions, rate 0.3); fit on 10 train shards (2.0M, rate 0.1), tuning 2 shards (0.98M)	6d7b0a1	Track B item 13: alternative time heads on the frozen flagship features (odyssey/inference/time_head_probe.py)	done	bin NLL / same-instant acc / after-bundle-1h pred vs obs: hazard refit 0.24999 / 0.9627 / 0.7557 vs 0.7577; hazard_mlp 0.24936 / 0.9628 / 0.7518 vs 0.7577; categorical 0.25160 / 0.9624 / 0.7561 vs 0.7577; lognormal3 0.26888 / 0.9618 / 0.7347 vs 0.7577 (continuous NLL 0.1286, median-gap abs error 64.2 h). Decision rule (beat hazard on BOTH NLL and after-1h calibration): none graduates; the binned hazard head stands. 24 min wall; JSON ~/runs/full_run_v8/time_head_probe.json
full_run_v8 (N3 counterfactual forecasts)	cbm	MIMIC held-out 2 shards, 300 subjects at 24h into a visit	6d7b0a1	Track B item 12: input-level what-ifs (odyssey/inference/counterfactual.py; edit the record, re-bin, re-score at the same position)	done	hypotension_6h (SBP set 80 for 6h; 44 edited): vasopressor +0.33 to +0.45 pp, death +0.02 to +0.11 pp, ICU admission +4.7 to +5.4 pp, sign agreement 93-98%; creatinine +1 mg/dL over 24h (246 edited): AKI +0.08 to +0.94 pp, sign agreement 93-94%, concept aki_stage_3 +0.038; lactate x3 (40 edited): death +0.12 to +0.96 pp, vasopressor +0.16 to +0.30 pp, sign agreement 85-90%; normotension (SBP set 120): near-zero deltas, sign agreement ~52% (weak-edit null); remove labs 24h (277 edited): AKI +0.4 to +0.8 pp, tachycardia concept -0.043. Reading: input-level edits move hazards in the clinically expected direction with 85-98% sign agreement, unlike the inert concept-level lever; 8 min wall; JSON ~/runs/full_run_v8/counterfactual.json
full_run_v8 (M2 comparator rescore)	cbm	MIMIC held-out 4 shards (552,756 rows, protocol v3); baselines fit on the 30 train shards	910732d	MIMIC comparator table: hazard vs tuned GBM vs TabICL vs EBM on the same v3 rows	done	[protocol v3] AUROC hazard / GBM / TabICL(basic) / EBM(strong), n at risk (n positive): AKI 8h 0.831/0.893/0.775/0.793 n=95,471(1,505), 24h 0.773/0.845/0.753/0.702 n=84,147(4,019), 72h 0.725/0.783/0.670/0.654 n=57,442(9,580); death 8h 0.962/0.949/0.925/0.921 n=136,850(222), 24h 0.934/0.962/0.902/0.861 n=135,061(682), 72h 0.904/0.936/0.864/0.850 n=130,818(1,949); ICU admission 8h 0.934/0.969/0.858/0.945 n=85,838(1,603), 24h 0.893/0.954/0.801/0.919 n=74,839(2,569), 72h 0.837/0.932/0.722/0.887 n=48,971(3,779); vasopressor 8h 0.938/0.932/0.840/0.834 n=111,450(467), 24h 0.892/0.915/0.793/0.803 n=98,722(974), 72h 0.828/0.886/0.687/0.775 n=67,385(1,801). Hazard beats the tuned GBM on 2/12 (death 8h, vasopressor 8h); the nearest remaining misses are vasopressor 24h (0.023) and death 24h (0.028), and the gap widens with horizon on 3 of 4 events (ICU 0.036 -> 0.095, vasopressor 0.005 win -> 0.058 loss, death 0.013 win -> 0.033 loss) while AKI stays flat at 0.058-0.072; TabICL loses 12/12; EBM loses except ICU admission, where it is close to the GBM and clearly ahead of the hazard heads. SurvivalPFN (capped refit at 2913d15, done 2026-08-24; fit in-context on a train-split subsample, disjoint from the scored rows, same discipline as the other comparators): AUROC, with p_unique = the number of DISTINCT predicted probabilities over the queried rows, which must be read alongside every value here. AKI 0.6074 / 0.6134 / 0.5504 (p_unique 76,134 / 76,291 / 55,611); ICU admission 0.6380 / 0.6236 / 0.6088 (69,865 / 63,636 / 47,308); vasopressor 0.5223 / 0.5373 / 0.5022 (175 / 242 / 11,507); death WITHHELD / WITHHELD / 0.7691 (1 / 1 / -). Loses to hazard and GBM on every cell it can be scored on. TWO GRADES OF DEGENERACY, and only the first was caught by the withholding rule: (a) death 8h/24h have p_unique = 1, every probability identically 0 because the fitted survival curve does not move at those horizons, so 1 - S(h) is zero by construction; these are withheld and must never be reported as 0.5. (b) vasopressor 8h/24h are NOT constant but are resolution-limited: 175 and 242 distinct values across 111,450 and 98,722 rows, at magnitudes ~1e-4, which is the same near-saturation the predict_proba docstring already documented on 2026-08-23. Their 0.5223 and 0.5373 are close to chance and are substantially tie-breaking among a handful of levels, so they are reported here but should NOT be quoted as a measurement of SurvivalPFN's discrimination. The p_unique == 1 withholding criterion is therefore necessary but not sufficient; a graded flag is the right fix and is not yet implemented. By contrast AKI and ICU admission have tens of thousands of distinct values and are genuine, if weak, measurements. Context sparsity is the leading explanation: the death fit held ~53 events in ~4,857 rows (1.1%), while eICU's death cells under the same code path are informative at a 13.3% context event rate (8h 0.7958, 24h 0.7603). Whether a denser context rescues the short-horizon cells is UNRESOLVED -- the direct test produced a degenerate 149-subject all-dying context and was invalid (research journal entry 50). 1h02m, peak RSS 10.2 GB
full_run_v8 (text-modality probe)	cbm	MIMIC held-out 4 shards, 552,756 rows; GBM fit on the 30 train shards	2005a51	Track A item 7 probe gate: do pooled note embeddings move the tuned GBM's alerts? Two encoders (BAAI/bge-base-en-v1.5 and google/medgemma-4b-it), 246,302 MIMIC-IV notes, mean-pooled, PCA to 64 dims, added as note count/staleness/mean/last features (--baseline-features strong_text)	done	AUROC delta vs the strong-only GBM (8h/24h/72h). BGE: vasopressor +0.003/+0.009/+0.011, ICU +0.001/+0.002/+0.001, AKI -0.003/+0.001/-0.002, death -0.015/+0.000/-0.000. MedGemma: vasopressor +0.006/+0.007/+0.009, ICU +0.001/0.000/0.000, AKI -0.003/+0.004/-0.001, death -0.044/+0.006/+0.004. GATE NOT MET: both encoders agree that text does not move the alerts on this slice -- the only consistent movement is vasopressor (+0.3 to +1.1 pp), ICU and AKI are inside noise, and death@8h DROPS in both arms (-1.5 pp BGE, -4.4 pp MedGemma). The two-arm design was chosen to separate 'text carries no signal' from 'the encoder is weak'; a clinical-domain model does no better than a general one, so the weak-encoder explanation is not supported. Hypothesis, untested: death@8h has the fewest positives of any cell (222 of 136,850), so adding ~131 dense columns hurts it by variance -- consistent with the drop being largest for the larger embedding. Per the roadmap's own rule, fusion work does not proceed on this evidence. Caveats: one slice, one seed, no confidence intervals; pooled whole-note embeddings only (no section or finding-level structure); radiology dominates the corpus (213,285 of 246,302 notes) and discharge summaries are only visible to a later visit. BGE 1h46m / MedGemma 2h37m to embed; each pass 1h46m
subset_run_v12_valuehead	cbm	MIMIC 30 shards	47ac590	value-head A/B arm B: v8 recipe + value_head (9-quantile, code-conditioned, pinball) -- does making magnitude a DISTRIBUTIONAL output close the GBM gap?	done	[BASELINE MISMATCH, found 2026-08-24 -- the deltas below are NOT VALID as published] This row's own absolute AUROCs are protocol v3 and stand; but every "vs subset_run_v8" DELTA here was computed against v8's stale v1-protocol dump (1,116,108 rows, no landmark_protocol_version column), because no v3 v8 dump existed on disk until 2026-08-24. Confirmed directly: v8's AKI@8h recomputed from that v1 dump is 0.769988, i.e. exactly the 0.7700 quoted here. Arm-to-arm comparisons among B, C, D and E remain valid (all are v3); only the v8 baseline is mismatched. Fix is a rescore of v8's existing checkpoint, not a retrain. Recompute before quoting any delta from this row. 28,873 steps, 3h44m, best val 2.6662; value loss 0.32 -> 0.10-0.12. Hazard AUROC vs subset_run_v8: AKI 8h 0.7826 (+0.0126), 24h 0.7334 (+0.0016), 72h 0.6859 (+0.0145); vasopressor 8h 0.9058 (-0.0436), 24h 0.8612 (-0.0627), 72h 0.8048 (-0.0762); ICU 8h 0.8986 (-0.0216), 24h 0.8448 (-0.0182), 72h 0.7718 (-0.0085); death 8h 0.9060 (+0.0396), 24h 0.8786 (+0.0511), 72h 0.8607 (+0.0609) -- the largest single move in the arm, larger than AKI's own predicted-direction shift, on an event the value channel has no obvious mechanistic reason to touch. Set top-1 76.91 (+0.17), lab 82.52 (+0.14). Time head: nll 0.2759 (v8 0.2772), same-instant flat, after-bundle 1h
subset_run_v8 (value-tail clip measurement)	cbm	MIMIC 1 train shard, 2,409,997 events / 920,291 valued observations with per-code stats (1,377 codes)	b743cf8	value-head line, arm E motivation: how often does the model's +-5 input clip actually saturate, and where does the threshold land clinically? Read-only pass over subset_run_v8's own fitted quantile_binner.json, no refit	done	Pooled clipped fraction 1.47% (high 1.30%, low 0.17%). Per code, z=+5 in raw units / clipped-high share / p99 / p99.9: creatinine (LAB//RESULT//50912//mg/dL, n=11,973, median 0.900, scale 0.445) 3.124 mg/dL / 7.17% / 9.0 / 15.6; lactate (50813, n=1,883, median 1.700, scale 0.964) 6.518 mmol/L / 3.61% / 9.0 / 13.1; wbc 1.36%; temperature_f 0.45%; spo2 0.55% low-side (ceiling at 100, expected); heart rate 0.012%, SBP ~0-0.03%, resp rate 0.04%. Top clipped-count code is creatinine (859), then SUBJECT_WEIGHT_AT_INFUSION (750) and several undecoded LAB itemids. DECISIVE READING: creatinine's clip point (3.124 mg/dL) sits BELOW the 4.0 mg/dL threshold the aki_stage_3 concept is defined on, so a 3.2 and a 15.6 reach the model as the identical input number -- the feature cannot represent the threshold its own label keys on. Lactate has the same shape (6.5 mmol/L clip vs a septic-shock tail to 13). The vital-sign channels do NOT have this problem (<1.5%, mostly past any plausible value), so this is specific to the right-skewed labs, which are the ones behind the AKI and vasopressor deficits. The comparison is also asymmetric: baselines featurize raw unclipped numeric_value (odyssey/inference/baseline_features.py), so the tuned GBM reads a tail the model discards. Motivates value_tail_transform="symlog" (identity inside +-5, `sign(z)*(5+log1p(
full_run_v8 (M-item: MEDS-Tab comparator)	cbm	MIMIC held-out, shared landmark rows (protocol v3); MEDS-Tab tabularized over the same extraction	358d742	established-baseline check: is the tuned GBM we measure ourselves against actually a hard bar? MEDS-Tab's own xgboost, hyperparameter sweep capped at n_trials=10 with boosting rounds capped at 400	done	JOIN VERIFIED FIRST (the precondition, not an afterthought): every one of our at-risk landmark rows found a MEDS-Tab prediction (frac_our_at_risk_rows_joined = 1.0, all 5 cells); MEDS-Tab side 1.00003-1.0001, i.e. a handful of duplicate (subject_id, prediction_time) fan-outs, the already-documented duplicate-key phenomenon. AUROC recomputed on the joined rows with OUR y@h and at-risk mask matches MEDS-Tab's self-reported test_auc to 5-6 decimal places, so the two are not measuring different populations -- which is what the shared-landmark driver was built to guarantee. AUROC hazard / tuned GBM / MEDS-Tab on the SAME rows (n at risk, n positive): death 8h 0.9623 / 0.9490 / 0.9325 (136,850, 222); death 24h 0.9340 / 0.9617 / 0.9024 (135,061, 682); vasopressor_start 24h 0.8922 / 0.9148 / 0.8284 (98,722, 974); icu_admission 24h 0.8933 / 0.9540 / 0.8261 (74,839, 2,569); acute_kidney_injury 24h 0.7727 / 0.8454 / 0.6996 (84,147, 4,019). RESULT: MEDS-Tab is LAST in all 5 cells -- it does not beat the tuned GBM anywhere, and it loses to our own hazard head everywhere too, including AKI 24h (0.6996 vs hazard 0.7727) where the GBM beats both comfortably. The bar we measure ourselves against is not artificially soft. CAVEAT that must travel with this number: MEDS-Tab was deliberately capped at 10 trials / 400 rounds for parity with our GBM's own 4-config / 400-round search (see the DEFAULT_N_TRIALS reasoning in both driver scripts). This shows MEDS-Tab does not win under a FAIR search budget; it is NOT a claim that the method is categorically weaker at unlimited tuning depth. Scope: 5 cells (4 tasks at 24h plus death at 8h), not the full 12-cell comparator grid. AKI-LABEL CAVEAT RESOLVED 2026-08-31 by B1's v4 rerun: the Aug 27 report flagged that its AKI cells used the pre-KDIGO label and that "the true size of this gap (bigger, smaller, or possibly gone) is not yet established". Measured on post-KDIGO v4 rows, AKI@8h is GBM 0.9153 vs TabICL 0.8734, a gap of 0.042 against the v3 report's 0.128 -- so the completed staging shrank TabICL's one surviving deficit by roughly two thirds. It is still a real gap and AKI is still the only place TabICL trails, so the qualitative reading holds; the magnitude does not. Sweep cost 5.5h for the 4-task 24h run; per-trial time varied ~9x across tasks (icu_admission 94s/trial to death_24h 832s/trial). ARTIFACTS PARTIALLY DELETED 2026-08-31 (Amrit instruction, VM1 was at 85% disk with R8 training): the 230GB meds_tab_out_shared/ tabularization and per-task prediction tree was removed, freeing 229GB (VM1 85% -> 26%). The four result files behind every number in this cell (meds_tab_auroc_comparison.json, meds_tab_sweep_results.json, meds_tab_shared_out.json, config.json) were copied to research_journal/figure_data/vm1/full_run_v8_meds_tab_scoped/ FIRST and verified to contain these exact AUROCs; they had existed only on the VM until then. Retained on VM1: fitted xgboost models (59MB), labels (104MB), the one-shard trust-gate reference (1.4GB), and alerts_rows.parquet (34MB). Reproducing this row from scratch now requires a full re-tabularization plus the 5.5h sweep
subset_run_v12f	cbm	MIMIC 30 shards	47ac590	value-head A/B arm C: v8 + value_fourier (input ENCODING of z as sin/cos at 8 geometric frequencies, replacing [z, z^2, has]) -- isolates "better input encoding" from arm B's "better output objective"	done	[BASELINE MISMATCH, found 2026-08-24 -- any "vs v8" delta here is NOT VALID as published] This row's own absolute AUROCs are protocol v3 and stand; the v8 baseline they were read against was v8's stale v1-protocol dump (1,116,108 rows, no landmark_protocol_version column), because no v3 v8 dump existed until 2026-08-24. Arm-to-arm comparisons among B, C, D and E remain valid (all v3); only the v8 baseline is mismatched. Fix is a rescore of v8's existing checkpoint, not a retrain. 28,873 steps, 4h9m, best val 2.6694 -- effectively tied with arm B's 2.6662, the richer encoding bought nothing on validation loss. CAVEAT that governs the whole row: C ran on the OLD clipped values (tail_transform=clip, unchanged from B/v8), so it tests a richer encoding of a signal that is still tail-saturated at the source (see subset_run_v12_symlog's row for the mechanism). Valid only over the unsaturated ~98.5% of the range; a flat result here does NOT close the encoding question, since encoding may matter once the tail itself is fixed by symlog. Hazard AUROC vs subset_run_v8: AKI 8h 0.7831 (+0.0131), 24h 0.7309 (-0.0009), 72h 0.6862 (+0.0148); vasopressor 8h 0.9052 (-0.0442), 24h 0.8583 (-0.0656), 72h 0.7958 (-0.0852); ICU 8h 0.9008 (-0.0194), 24h 0.8402 (-0.0228), 72h 0.7661 (-0.0143); death 8h 0.9043 (+0.0379), 24h 0.8776 (+0.0501), 72h 0.8460 (+0.0462). C vs B (not vs v8): every cell within +/-0.9pp of B's own value, sign inconsistent cell to cell -- no systematic Fourier-encoding effect, consistent with the val-loss tie. Value metrics: CRPS 0.2162 (B 0.2136, marginally worse), overall median abs err 0.2766 (B 0.2737, marginally worse); creatinine CRPS 0.3901 (B 0.3843, marginally worse), median abs err 0.5133 (B 0.5023, marginally worse) -- richer encoding did not improve creatinine calibration either, consistent with a saturated target no encoding can recover. Set top-1 76.39 (B 76.91, v8 76.74 -- slightly below both); lab 82.10 (B 82.52, v8 82.38 -- slightly below both). Time head: nll 0.2772 (v8 0.2772, B 0.2759, flat); after-bundle 1h
subset_run_v12_symlog	cbm	MIMIC 30 shards	3b0a618	arm E, and the primary test of the value line: v8 + value_tail_transform="symlog", ONE change, input-side only (no value_head, no value_fourier). Motivated by the measured clip defect (row above): the model's input value tail saturated below the threshold its own AKI label is defined on, while the baselines featurize raw unclipped values	done	HYPOTHESIS: part of the AKI/vasopressor deficit against the tuned GBM is a representational handicap we imposed on ourselves, not an architectural limit. PREDICTION, stated before the run: AKI moves most, and the movement concentrates on rows whose visible lookback contained a would-have-been-clipped value. PRE-REGISTERED READOUT (fixed 2026-08-24, before any result): symlog changes only ~1.5% of valued observations and every other position is bit-identical to v8, so an aggregate AUROC is computed mostly over rows where E and v8 are the same model on the same input -- a real effect can read as flat. Because clipping is patient-correlated (a renal-failure patient has ALL creatinines above the clip, not 1 in 14), the affected stratum is small but enriched for AKI positives. So E reports hazard AUROC for E and v8 split by EXPOSED vs UNEXPOSED, per event at 8h/24h, n and n-positive per cell. EXPOSED = the row's lookback contains an observation with `
subset_run_v8_vm2	cbm	MIMIC 30 shards	5fe2d50	cross-VM control, launched directly from arm E's confound (row above). v8's own config, byte-identical to subset_run_v8's (diffed key-for-key against subset_run_v12_symlog's config with only value_tail_transform removed -- confirmed the only difference), trained on VM2 (odyssey-eicu-a100, the machine E trained on) instead of VM1 (odyssey-cbm-a100, where subset_run_v8/B/C were trained). HYPOTHESIS, stated before any result: if subset_run_v8_vm2 differs from subset_run_v8 (VM1) by roughly the magnitude arm E differed from subset_run_v8 on death's UNEXPOSED stratum (+0.065 to +0.096 AUROC, 60-90x the GBM's own cross-arm noise floor), the machine explains arm E's death confound, and every prior VM2-vs-VM1 comparison this program has made needs revisiting, including L1-L4. If the two v8 runs agree to within the GBM's own noise floor (~0.5-1.5pp typical, ~3pp on sparse cells), the machine hypothesis is disconfirmed and the weaker ordinary-training-variance hypothesis (entry 51) becomes the leading explanation instead. DECISION RULE: score both against the same v3-protocol held-out shards, same event/horizon cells as arm E's readout; a hazard AUROC delta on the SAME (unexposed-by-construction, since neither run has ever seen symlog) rows that lands within the GBM noise floor confirms the machine is not the driver; a delta of similar magnitude to E's confirms it is	launched 2026-08-24, training	Prerequisite for interpreting arm D and everything after it -- assigned by the lead directly after arm E's readout surfaced the confound, ahead of the M3/M4/M6/N1 queue (whose own gates are separately under audit, see below)
subset_run_v8 (GBM feature-group ablation)	cbm	MIMIC held-out 4 shards, baselines fit on 30 train shards (v3 protocol)	5fe2d50	Error analysis requested directly by Amrit after arm E's null: what does the GBM have that the hazard head doesn't, on AKI/vasopressor/icu_admission at all 3 horizons. 609 strong features partitioned into 5 groups (recency 64, latest_value 48, summary_stats 384, counts_occurrence 110, static 3), drop-one + keep-only, hyperparameters held fixed at the already-tuned values (caveat: this inflates measured closure -- every % below is an upper bound)	done	AKI: summary_stats leads drop-one at every horizon (20.5%/24.5%/22.2% margin closure) and is strong in both directions (keep-only alone: 0.86/0.83/0.76 vs full 0.89/0.85/0.78), but no group collapses the gap -- 75-80% survives every ablation. Vasopressor/ICU: counts_occurrence explains 70-173% of the margin (>100% cells: ablated GBM falls below the hazard head) and alone recovers 94-99% of the full model at every one of 6 cells -- checked for outcome-code leakage (drug.vasopressor.n_6h/n_24h/ever_visit share vasopressor_start's exact onset regex; corrective re-fit removing just those 3 features showed a noise-level effect, 0.9322->0.9364/0.9118->0.9099/0.8895->0.8911; ICU's family.visit checked by code composition, ICU_ADMISSION//MICU at 227 occurrences vs thousands of routine urine-output/transfer codes, not re-fit since the check was already conclusive). Part B (descriptive disagreement, no new fits): ICU shows 0/0/0 haz_right_gbm_wrong under two binned threshold definitions (top-quartile/below-median and top-half/bottom-half), but 22-25% under an unbinned any-margin test -- reported as "no confident-vs-confident disagreement" not "zero complementary information," the stronger claim is not supported. AKI's GBM wins concentrate on short-history/busy-visit patients; vasopressor's on stale-creatinine rows (17-23h since last vs 8-10h) -- an integrated hypothesis (not confirmed): the hazard head degrades when local recent signal is sparse, while explicit counts/aggregates stay well-defined regardless, which motivates explicit windowed counts as the next arm with a pre-registered stratum (sparse-history/stale-lab rows) rather than an aggregate hope. Full detail research_journal/experiments/52_gbm_error_analysis.html (local-only)
subset_run_v12_mlp	cbm	MIMIC 30 shards	3b0a618	arm D: v8 + symlog + value_head + value_head_hidden=256 + value_head_weight=2.0. A clean SUPERSET of arm E, so E is D's control and the pair separates "fixing the input tail helps" from "fixing the tail AND predicting it distributionally helps"	done	Revised from the original clip-based config after finding that the value head's own regression target is the same clipped numeric_z: under clip, ~7% of creatinine TARGETS are the literal constant 5.0, which is the most likely mechanical cause of arm B's worst-calibrated code (median abs err 0.50 SD) and means a clip-based D would have repeated a known-broken target. Report value metrics against arm B's (CRPS 0.2136, median abs err 0.2737, creatinine CRPS 0.3843 / 0.5023, coverage 0.140/0.169/0.243/0.393/0.516/0.592/0.709/0.818/0.909 at nominal 0.1..0.9) and the head parameter count. If coverage moves toward nominal and creatinine error drops well below 0.50 SD, the saturated-target explanation is confirmed whether or not the alert numbers move -- a result worth having either way. Same EXPOSED/UNEXPOSED stratification as E
eicu_subset_v7	eicu	eICU v2 30 shards	bdb6ef4	v6 recipe on spec-v2 data (HICL medications, named infusions, GCS, urine output)	done	10,483 steps, 1h06m; set 85.3 (flat): lab 87.2, diag 60.4 (58.4), other 62.8 (GCS/urine now 'other'), med 45.2 (45.8; but v1's 31% blank medication events had normalized to a bare MEDICATION//started token, trivially predictable, so v7's flat number with real drug identities is a genuine gain; medication vocab 608 -> 1,260), visit 33.6 (28.6); on_vasopressors AUROC 0.83 -> 0.88, qSOFA 0.85 -> 0.86; time calibration unchanged; alerts vs strong GBM 0/9 (vasopressor 8h 0.870/0.903, AKI 8h 0.711/0.882, death 8h 0.908/0.934); medication audit: novel starts set-hit 16.0 vs repeats 43.1; report journal 15
eicu_subset_v8	eicu	eICU v2 30 shards	3d603cf	v7 + value_embeddings + event_head_hidden 256 + static facts as leading inputs	done	1h08m; set 86.4 (v7 85.3): lab 88.4 (+1.2), proc 46.6 (+1.1), diag 61.1 (+0.7), med 45.6 (flat), visit 29.1 (-4.5, n shifted); concepts mixed (on_vasopressors 0.896, sirs 0.892 up; aki_stage_3 0.923, tachycardia 0.885 down); alerts vs strong GBM 0/9, hazard within +-3 pts of v7 (vasopressor 8h 0.862/0.903, AKI 8h 0.681/0.878, death 8h 0.908/0.943); concept-proxy scorer for vasopressor <0.5 at all horizons (flag); medication audit flat; report journal 16
subset_run_L1	eicu VM	MIMIC 30 shards	a14c386	v8 + concept_global_pairs (leakage test)	done	3h18m; set 75.3 (v8 76.7): med 42.2, diag 37.9, lab 81.0; concepts mean 0.801; banded interventions: none 26.6, truth 24.5, flip 24.2, random 24.3 (truth-flip +0.28 pt, but every intervention below none); alerts 0/12 vs strong GBM
subset_run_L2	eicu VM	MIMIC 30 shards	a14c386	L1 + randint_prob 1.0 (teacher-forced concept inputs)	done	3h27m; set 75.2 (L1 75.3): diag 32.7 (-5.1), proc 30.9 (-7.1), billing 16.5 (-6.0), lab 81.1; interventions none 25.9, truth 25.9, flip 25.8, random 25.9 (separation 0.05 pt); zero_known -1.6 (94% retained), zero_unknown -25.6 (1% retained): teacher forcing made the heads MORE dependent on the residual; alerts 0/12 and worse than L1 (ICU 8h 0.814)
subset_run_L3	eicu VM	MIMIC 30 shards	a14c386	L1 + unknown_dim 8 (residual capped)	done	3h17m; set 75.0 (flat vs L1); exact top-1 none 24.3 (L1 26.6); interventions truth 19.0, flip 19.5, random 19.3 (all -5 vs none; truth-flip -0.5, wrong-signed); zero_known -20.0 (18% retained), zero_unknown -11.4 (53% retained): load crossed onto the named concepts; concepts mean 0.789; alerts 0/12
subset_run_L4	eicu VM	MIMIC 30 shards	e879b97	L1 + unknown_dim 1 (extreme residual cap)	done	3h17m; set 74.2 (L1 75.3): diag 37.9 (flat), med 40.3 (-1.9), lab 80.0 (-1.0); exact top-1 none 22.3; interventions truth 15.6, flip 15.0, random 15.3 (all -7 vs none; truth-flip +0.55 pt, correctly signed but swamped by the collapse under any edit); zero_known -22.2 (0.5% retained, near-total loss), zero_unknown +0.6 (103% retained, residual carries nothing); concepts mean 0.792 (0.628-0.991); alerts 0/12. Capping the residual to 1 dim doesn't fix the lever -- it makes every edit mode (including truth) catastrophic, because the model has nowhere else to route information when the known channel is disturbed
full_run_eicu_v8	eicu	eICU v2 all 134 train shards	e879b97	first full-scale eICU run, v8 recipe, 1 epoch, streamed	done	set top-1 87.5 (subset v8 86.4, +1.1); alerts vs strong GBM 0/12 (death 8h 0.940 vs 0.943); interventions: truth/flip/random within 0.4 pt, zero_known 53.7 -> 17.9, zero_unknown unchanged; report research_journal 21
eicu_subset_indep_a	eicu	eICU v2 30 shards	fa1c90c (branch)	independent training stage A: task_weight 0, concept-only backbone	done	10,488 steps, 69min; feeds entry 25
eicu_subset_indep_b	eicu	eICU v2 30 shards	fa1c90c (branch)	stage B: freeze_bottleneck, init_from stage A, randint 1.0	done	53min; truth-flip +0.78 pt (best of series) but truth ties none; residual still dominant; set top-1 collapsed to 47.1; alerts 0/12; report research_journal 25
subset_run_indep_a	cbm	MIMIC 30 shards	fa1c90c (branch)	independent training stage A on MIMIC	done	28,873 steps, 3h21m; feeds entry 26
subset_run_indep_b	cbm	MIMIC 30 shards	fa1c90c (branch)	stage B on MIMIC	done	banded: none 14.91, truth 14.98, flip 14.46, random 14.74 (truth-flip +0.52 pt, truth beats none, first time); zero_known 18.8% retained (known channel dominant), zero_unknown 94.3%; alerts 0/12, hazard -0.04-0.19 vs v8; report research_journal 26
subset_run_M1	cbm	MIMIC 30 shards	850ed2c	resume subset_run_indep_b (entry 26 stage B) from checkpoint_24000.pt, num_epochs 2->5, early_stopping_patience 15->200	done	71,697 steps, 3h59m, no early stop; best_val_loss 3.6537->3.6233; set top-1 62.9 (indep_b 62.6); truth-flip +0.57pt (best of series), zero_known 12.0% retained (healthiest completeness yet); alerts 0/12 vs strong GBM, flat to slightly worse than indep_b on most rows; report research_journal 32
subset_run_M2	cbm	MIMIC 30 shards	850ed2c	fresh stage B from subset_run_indep_a, unfreeze_top_backbone_layers=2 (of 8), recipe otherwise identical to indep_b	done	17,750 steps, 1h34m, early stop 15/15; best_val_loss 3.3325; set top-1 65.6; truth-flip +0.35pt; zero_known 27.1% retained; alerts 0/12 vs strong GBM, consistent step up over indep_b and M1 on every row; report research_journal 32
subset_run_M3a	cbm	MIMIC 30 shards	850ed2c	fresh stage A, task_weight 0.0->0.1 vs subset_run_indep_a	done	28,873 steps, 3h22m; best_val_loss 2.8883 (task_weight=0.1 scale, not comparable to stage-B numbers); no separate alerts/interventions eval, feeds subset_run_M3b
subset_run_M3b	cbm	MIMIC 30 shards	850ed2c	stage B, init_from subset_run_M3a instead of indep_a, recipe otherwise identical to indep_b	done	11,000 steps, 0h54m, early stop 15/15; best_val_loss 2.8602 (vs indep_b 3.6537); set top-1 73.5, closes ~77% of the set-top-1 gap to fully-joint v8; truth-flip +0.30pt (weakest of the series); zero_known 25.5% retained; alerts 0/12 vs strong GBM but closest of the series to a GBM win (death@8h 0.901, within the GBM refit noise band 0.908-0.930); report research_journal 32
eicu_tabicl_redo	eicu	eICU v2, eicu_subset_v8 checkpoint_best	branch 8a8145f	entry-27 redo at the checkpoint convention after the default-checkpoint fix (2d61902)	done	TabICL (zero-shot, basic features) wins 5/12: all 3 AKI horizons (0.733/0.710/0.664 vs 0.681/0.672/0.636) and ICU 24h/72h (0.860 vs 0.741 at 72h); hazard wins all vasopressor and death pairs and ICU 8h (matches entry 16 exactly); targeted complementarity, not general superiority; report research_journal 27
eicu_subset_v9	eicu	eICU v2 30 shards	c5733f0	v8 + recency_features (per-family hours-since-last, appended after the bottleneck to time/event heads only)	done	1h27m; alerts hazard +0.008 to +0.022 on 9/12 pairs vs v8 (GBM held fixed as control, flat), AKI 8h 0.681 -> 0.697 (~1/3 of the gap to GBM 0.878); death flat (near ceiling); concept AUROC and time calibration untouched (as designed); forecasting NOT free: set 86.7 (+0.3) but diagnosis -2.7, procedure -1.4, category-set -2.1, offset by demographic +3.7, lab, visit; a real family-level tradeoff via shared-backbone gradients. Contamination measured negligible: a clean v1 rerun (2026-08-22) reproduces to ~1e-7 relative on the metrics checked (death@8h, AKI@8h); numbers above stand unchanged, see research journal entry 38. recency_features REMOVED from the codebase (2026-08-25), decision not a retraction: a frozen-feature probe on subset_run_v8 (MIMIC, no recency feed) found the backbone already recovers per-family recency at pooled R^2 0.925 (pre-bottleneck) and 0.916 (post-bottleneck) against ground truth, both from a linear-ish probe with no extra information -- the representation was already substantially there without ever being fed it. Reading: this row's win most likely eased the event-hazard heads' optimization under competing multi-task gradient pressure rather than supplying missing information the model could not otherwise reach. Kept as hand-engineered, GBM-mirroring input plumbing anyway (as was signal_channels, the untested v10 sibling lever, also removed): the project's stated direction is an end-to-end forecaster, not one with per-family staleness handed in. The numbers above stand as history; load_run can no longer reconstruct this run's checkpoint (the head-input-width auto-detection this depended on was removed with the feature), so this row is now read-only.
eicu_ebm_baseline	eicu	eICU v2, v8 checkpoint_best	d332dbf	NAM/GAM comparator (EBM, basic features, speed-tuned: 100k rows subject-grouped, outer_bags 4)	done	12/12 fits, 3h28m; loses to hazard AND GBM on every legitimate event (vasopressor 0.699-0.810, AKI 0.575-0.683, death 0.630-0.863); ICU admission outlier (0.961-0.970) confirmed a shared age-dominated artifact of the eICU-degenerate readmission sub-task (excluded per entries 13/21); no uncapped confirmation run
eicu_survivalpfn	eicu	eICU v2, v8 checkpoint_best	2aaa1e2	Survival-native comparator (SurvivalPFN, basic features, one CPU-only in-context fit per event on (T, delta) targets, scored via a positional re-join against the already-dumped alerts_rows.parquet, zero model forward passes)	done	9/9 fittable pairs scored (icu_admission skipped: single-class survival delta among >135k at-risk rows, consistent with the known-degenerate eICU event); loses to hazard AND GBM everywhere (vasopressor 0.646-0.681 vs hazard 0.767-0.862; AKI 0.582-0.589 vs 0.636-0.681; death 0.704-0.794 vs 0.787-0.908; GBM ahead of both throughout); duplicate-sensitivity check on the stored dumps (alerts_rows.parquet has 9,108 real duplicate landmark keys, up to 5x multiplicity): GBM robust everywhere, max
eicu_subset_v8	eicu	eICU v2 30 shards, checkpoint_best	60d0d45 (dump regen)	[protocol v3] re-evaluation of the v8 alerts row set (E1)	done	[protocol v3] 629,068 rows (stamp confirmed; v9 regen dump matches exactly). Hazard / strong GBM AUROC: AKI 8h 0.699/0.880, 24h 0.681/0.826, 72h 0.642/0.773; vasopressor 8h 0.870/0.908, 24h 0.825/0.886, 72h 0.782/0.853; death 8h 0.895/0.938, 24h 0.857/0.924, 72h 0.786/0.888; ICU admission (degenerate on eICU) 8h 0.780/0.934, 24h 0.756/0.926, 72h 0.722/0.908. Hazard wins 0/12; v1 row above stands as history
eicu_survivalpfn	eicu	eICU v2, v8 checkpoint_best	5d9bd98 (fit script; refit on v3 rows)	[protocol v3] SurvivalPFN refit on the v3 row set (E5)	done	[protocol v3] 629,068 rows, exact key + y@h match to alerts_rows_v3.parquet; 9/9 fittable pairs (icu_admission skipped, same >=50 event+censored floor as v1). SurvivalPFN AUROC: AKI 0.584/0.594/0.591, vasopressor 0.705/0.678/0.670, death 0.796/0.760/0.701 (8h/24h/72h) -- loses to hazard and GBM everywhere, ordering unchanged from v1; RSS 25-28 GB
eicu_subset_v6	eicu	eICU v1 30 shards	4d8a5b2	pipeline replication on eICU with the v6 recipe	done	10,289 steps, 1.02h; set 85.3 (lab 86.8, diag 58.4, med 45.8, proc 46.2, visit 28.6, other 54.2); concepts 0.64-0.96; time calibration within 0.2 pt; alerts hazard vs GBM: vasopressor 8h 0.856/0.838, 24h 0.805/0.800, 72h 0.737/0.763; AKI 0.70/0.74, 0.69/0.71, 0.65/0.68; death 0.878/0.837, 0.825/0.836, 0.756/0.775; ICU admission n/a on eICU (unit stay = visit)
gemini_smoke_1	gemini H200	GEMINI MEDS 5 train shards	gemini mirror, Aug 22	first GEMINI training smoke (14M, v8-family recipe, default lanes)	done	9,266 steps, 26min, best_val 3.63; proves env/feed/checkpointing on the H200
gemini_smoke_2	gemini H200	GEMINI MEDS 30 shards	gemini mirror, Aug 22	6x data at same 14M params (G1 anchor for the data-scaling curve)	done	53,575 steps, 2.2h, best_val 2.73; eval-forecast (held-out): set top-1 56.6, labs 58.3, diagnoses 16.0
gemini_full_14m_v1	gemini H200	GEMINI MEDS 894 train shards	gemini mirror, Aug 22	G2: first full-data GEMINI run (14M, 64x512, 2 epochs)	running	~2.1-3.1 steps/s (feed-bound); val 2.1368 at step 27,200 mid-epoch 0; eval-forecast (G3) after completion
gemini_extract_v1_full	gemini	GEMINI SQL -> MEDS	gemini 6281be6	full 18-table extraction incl. death (derived-flag primary), ICU, ER, transfers, CMG/HIG	done	1.2007B rows (counts rounded to 1k for export), 1118 shards, 1,118,000 subjects; manifest all-complete; recovery from 2 OOM-kill orphan waves (1,590 footerless files deleted, zero row loss verified vs prior summary); finalize gated on train-full + eval
gemini_full_14m_v1 (eval)	gemini H200	GEMINI MEDS held-out	gemini mirror, Aug 23	G3: eval-forecast at full data (the third data-scaling point: 5/30/894 shards at fixed 14M)	done	train: 76,868 steps / 10.2h / 2 epochs, best_val 1.9517 AT FINAL STEP (under-trained, supports G4); held-out: set top-1 76.0 (smoke_2 56.6), top1 21.0, top5 47.7, ppl 557 over 20.9M predictions; by family (set top-1): lab 82.9 (smoke_2 58.3), visit 62.9 (top5 88.9), billing 49.3, medication 43.7, procedure 31.1, diagnosis 27.4 (smoke_2 16.0), other 51.7; time head: nll 0.416, same-instant acc 94.9 at 82.5 rate; time calibration predicted-vs-observed within 0.2pt at every horizon (1h .9215/.9222, 8h .9851/.9848, 24h .9971/.9971; after-bundle 1h .5536/.5551, 8h .9155/.9132, 24h .9836/.9832); every family scales strongly with data at fixed params
subset_run_v8_taskset_v3	cbm	MIMIC 30 shards	9285f68	Found run, unregistered until 2026-08-28: first task_set="v3" training run. v8 recipe otherwise unchanged (value_head/value_fourier off, value_tail_transform="clip", recency_features/signal_channels off); task_set="v3" widens concepts to 26 (v1's 15 + sepsis3 + 11 v3 electrolyte/metabolic/shock concepts) and, via ALERT_TASK_SETS["v3"] == ALERT_EVENTS_V2, widens alert events to add sepsis3 + readmission_30d. Needed to unlock the sepsis3/readmission comparator and to feed the bottleneck-signal probe rows below	done (2026-08-25), found and registered 2026-08-28	28,873 steps, 4h36m (16583.1s), best_val_loss 2.5830, no early stop. [protocol v3] hazard vs tuned GBM AUROC, n at risk (n positive): vasopressor_start 8h 0.896/0.934 (111,450; 467), 24h 0.833/0.914 (98,722; 974), 72h 0.755/0.883 (67,385; 1,801); icu_admission 8h 0.899/0.968 (85,838; 1,603), 24h 0.843/0.954 (74,839; 2,569), 72h 0.773/0.931 (48,971; 3,779); acute_kidney_injury 8h 0.777/0.894 (95,471; 1,505), 24h 0.726/0.845 (84,147; 4,019), 72h 0.682/0.782 (57,442; 9,580); death 8h 0.904/0.953 (136,850; 222), 24h 0.876/0.959 (135,061; 682), 72h 0.850/0.940 (130,818; 1,949); sepsis3, first-ever comparator 8h 0.854/0.937 (89,107; 1,374), 24h 0.834/0.918 (77,651; 2,571), 72h 0.781/0.874 (50,443; 3,662); readmission_30d, first-ever comparator 168h 0.504/0.664 (4,769; 452), 720h 0.527/0.723 (4,578; 1,143). Hazard loses all 17 cells to the GBM at this 30-shard scale, consistent with every other subset-scale comparator in this registry (full_run_v8's 292-shard M2 table is the only one where hazard wins any cells). readmission_30d is the standout: hazard is barely above chance (0.50-0.53) against a GBM that is itself weak (0.66-0.72) -- both near floor, but the hazard-to-GBM gap (16-20pp) is the widest of any task in the table, wider than AKI's own well-studied deficit. Near-chance hazard AUROC against a GBM that clearly beats chance points at the hazard head or task formulation for this event specifically, not just "hard task" -- see the CI-check row below.
subset_run_v8_taskset_v3 (bottleneck signal probe)	cbm	MIMIC held-out 4 shards, fit on 5 train shards	9285f68	Found run, unregistered until 2026-08-28. Direct test of whether the concept bottleneck itself discards task-relevant signal: scripts/probe_bottleneck_signal.py (built 2026-08-25, run same day, never registered) fits frozen StandardScaler+LogisticRegression probes separately on the backbone hidden state (pre-bottleneck) and the bottleneck output (post-bottleneck, what the event/task heads actually read), per (event, horizon). Precedent it extends: the same method applied to recency-recovery found R^2 0.925 pre / 0.916 post (commit cd96842, the finding that got recency_features/signal_channels removed as redundant)	done (2026-08-25), found and registered 2026-08-28	168,875 train landmark rows / 7,301 visit-end rows; 138,189 held-out landmark / 5,869 visit-end. Pre-bottleneck / post-bottleneck AUROC (delta): vasopressor_start 8h 0.9027/0.8689 (-0.034), 24h 0.8551/0.8388 (-0.016), 72h 0.7866/0.7987 (+0.012); icu_admission 8h 0.9011/0.8928 (-0.008), 24h 0.8434/0.8401 (-0.003), 72h 0.7704/0.7800 (+0.010); acute_kidney_injury 8h 0.8055/0.7988 (-0.007), 24h 0.7519/0.7429 (-0.009), 72h 0.6970/0.6890 (-0.008); death 8h 0.9253/0.8295 (-0.096), 24h 0.9149/0.8764 (-0.038), 72h 0.8899/0.8808 (-0.009); sepsis3 8h 0.8732/0.8649 (-0.008), 24h 0.8473/0.8435 (-0.004), 72h 0.7931/0.7904 (-0.003); readmission_30d 168h 0.5964/0.5571 (-0.039), 720h 0.6539/0.5856 (-0.068). READING: for vasopressor/ICU/AKI/sepsis3 the bottleneck's own compression cost is small (0.3-3.4pp on the LINEAR-PROBE ceiling) -- the concept bottleneck is not where most of the GBM gap comes from for these tasks. death@8h and readmission@720h lose far more (6.8-9.6pp), the two largest drops in the table by 2-3x. A 1-train/1-held-out-shard smoke run (probe_smoke.log, same day) reproduces the same sign and rough magnitude on every cell, so this is not a small-sample artifact. CAVEAT stated in the script's own docstring, and load-bearing: this measures what a frozen LINEAR probe can recover, not what the actual (nonlinear, purpose-built) hazard head extracts -- the CI-check row below shows the two diverge sharply on exactly the two cells that lose the most here (death@8h, readmission@720h), in OPPOSITE directions.
subset_run_v8_taskset_v3 (probe vs hazard-head CI check)	cbm	MIMIC held-out 4 shards, joined against alerts_rows_{v3,readmission_v3}.parquet	9285f68	Found run, unregistered until 2026-08-28. Follow-up to the row above: is the post-bottleneck probe's AUROC (a proxy for how much signal is even IN the representation) different from the ACTUAL trained hazard head's own AUROC on the identical held-out rows? scripts/probe_ci_check.py (present only on this VM as an untracked file, not yet committed anywhere as of 2026-08-28) reuses the probe's held-out predictions, joins them against the existing alerts dumps by exact row key, and scores both with odyssey.inference.uncertainty.bootstrap_auroc (subject-clustered, n_boot=1000, alpha=0.05, the module's own defaults) so the comparison sits on the same finite-sample-variance footing	done (2026-08-25), found and registered 2026-08-28	probe / hazard AUROC [95% CI], verdict: vasopressor_start 8h 0.869 [.846,.890] / 0.896 [.870,.920] OVERLAP; 24h 0.839 [.815,.861] / 0.833 [.800,.865] OVERLAP; 72h 0.799 [.771,.824] / 0.755 [.713,.797] OVERLAP; icu_admission 8h 0.893 [.880,.905] / 0.899 [.888,.909] OVERLAP; 24h 0.840 [.825,.857] / 0.843 [.827,.860] OVERLAP; 72h 0.780 [.754,.805] / 0.773 [.745,.801] OVERLAP; acute_kidney_injury 8h 0.799 [.781,.817] / 0.777 [.758,.793] OVERLAP; 24h 0.743 [.721,.764] / 0.726 [.701,.750] OVERLAP; 72h 0.689 [.662,.716] / 0.682 [.654,.710] OVERLAP; death 8h 0.829 [.787,.871] / 0.904 [.875,.931], SEPARATED, hazard wins; 24h 0.876 [.849,.899] / 0.876 [.846,.905] OVERLAP (tied); 72h 0.881 [.859,.900] / 0.850 [.817,.880] OVERLAP; sepsis3 8h 0.865 [.850,.879] / 0.854 [.840,.868] OVERLAP; 24h 0.843 [.827,.859] / 0.834 [.817,.850] OVERLAP; 72h 0.790 [.767,.813] / 0.781 [.756,.805] OVERLAP; readmission_30d 168h 0.557 [.525,.589] / 0.504 [.476,.534] OVERLAP; 720h 0.586 [.565,.608] / 0.527 [.503,.551], SEPARATED, probe wins. READING, the decision-relevant one: on 15/17 cells the trained hazard head is statistically indistinguishable from a frozen linear probe reading the same post-bottleneck vector -- for AKI/vasopressor/ICU/sepsis3 specifically, the head is already extracting essentially everything a linear probe can find there. Closing the GBM gap on those tasks needs MORE SIGNAL reaching the representation in the first place (consistent with the 2026-08-24 GBM feature-group ablation, whose leading finding was that explicit counts/aggregates -- not encoding richness or head capacity -- drove 70-173% of the vasopressor/ICU margin), not a bigger or better head. The two cells that DO separate go opposite directions: death@8h, the hazard head clearly beats the linear-probe ceiling (0.904 vs 0.829) -- real, separately-valuable nonlinear/censoring-aware work, and representation is not what limits death@8h. readmission_30d@720h is the opposite and the more actionable finding: a linear probe on the SAME frozen representation beats the trained hazard head (0.586 vs 0.527, CIs fully separated) -- the representation already contains more readmission signal than the head is using. That is a head/task-formulation problem, fixable without touching the backbone or the bottleneck, not a representation problem.
subset_run_v8_taskset_v3 (EHRSHOT-style probe benchmark, SMOKE SCALE)	cbm	MIMIC 1 train shard, 1 held-out shard	e231fb7	First run of scripts/probe_alerts.py (odyssey/inference/probe_baseline.py, built 2026-08-28 per Amrit's redirect: stop adding trained alert heads per task -- documented multi-task loss competition, e.g. eicu_subset_v9's recency channel cost -2 to -9pp on other families -- and instead measure whether the GENERAL representation supports a task under a FROZEN probe, the same evaluation philosophy EHRSHOT itself uses). 5 lab-severity concept tasks (anemia/hyperkalemia/hypoglycemia/hyponatremia/thrombocytopenia, task_set v3, adapted from EHRSHOT's 5-way severity multiclass to binary "concept triggers within horizon h") plus long_los (EHRSHOT's length-of-stay >7d, scored from a single ~20-28h snapshot per visit, not a landmark sweep, since it is a static per-visit label). SMOKE SCALE ONLY (1/1 shards, ~33k landmark rows): a functional-correctness run for the new library code (also exercised the load_run config-compat fix from the same day), not a decision-grade result -- a 5/4-shard run is the natural next step	done (2026-08-28)	probe_pre / probe_post [95% CI] / gbm [95% CI] AUROC, n at risk (n positive): anemia 8h 0.688/0.634 [.575,.691]/0.867 [.819,.916] (25,969; 151), 24h 0.672/0.643 [.587,.704]/0.828 [.774,.879] (22,708; 357), 72h 0.653/0.644 [.575,.703]/0.823 [.766,.877] (14,787; 790); hyperkalemia 8h 0.718/0.618 [.561,.671]/0.782 [.714,.833] (26,978; 150), 24h 0.669/0.601 [.542,.645]/0.804 [.739,.856] (23,737; 339), 72h 0.580/0.578 [.529,.624]/0.747 [.693,.797] (15,819; 709); hypoglycemia 8h 0.679/0.635 [.587,.681]/0.700 [.631,.768] (27,572; 159), 24h 0.580/0.545 [.497,.590]/0.671 [.607,.732] (24,316; 389), 72h 0.544/0.522 [.468,.562]/0.631 [.567,.692] (16,422; 858); hyponatremia 8h 0.694/0.608 [.539,.668]/0.913 [.881,.941] (28,167; 97), 24h 0.640/0.613 [.547,.672]/0.865 [.806,.910] (24,825; 228), 72h 0.627/0.629 [.582,.679]/0.806 [.742,.863] (16,661; 533); thrombocytopenia 8h 0.768/0.681 [.639,.732]/0.957 [.943,.969] (24,850; 198), 24h 0.772/0.688 [.642,.749]/0.945 [.924,.965] (21,688; 450), 72h 0.704/0.653 [.604,.713]/0.907 [.862,.937] (14,052; 793); long_los (snapshot) n/a/0.647 [.615,.685]/0.705 [.676,.739] (1,168; 243). READING: the GBM wins all 16/16 cells, every probe-vs-GBM CI pair fully separated -- a starker, more uniform gap than the original 4 alert tasks ever showed (subset_run_v8_taskset_v3's own row above has hazard losing 17/17 too, but by smaller margins on several cells, e.g. AKI@8h 11.7pp vs anemia@8h's 23.3pp probe-vs-GBM gap here). SECOND finding, distinct from the GBM comparison and arguably more important for the representation question: post-bottleneck is lower than pre-bottleneck on every single one of the 15 concept-task cells (e.g. anemia@8h 0.688->0.634, thrombocytopenia@8h 0.768->0.681), unlike the original 4 v1/v2 alert tasks where pre/post cost was small (0.3-3.4pp, see the bottleneck-signal-probe row above). These 5 concepts ARE part of this run's own task_set=v3 concept-bottleneck supervision (so the bottleneck was trained to predict them specifically) yet still loses substantially more of their linearly-recoverable signal than the earlier, unrelated v1/v2 concepts did -- consistent with, though not proof of, capacity competition among the 26 concepts task_set v3 trains the bottleneck on simultaneously. CAVEATS that must travel with this row: smoke scale only (1 shard each side, positives as low as 97-198 on the 8h cells) -- CIs are wide and this is a functional-correctness check, not a claim about the architecture at scale; GBM here is the FULL tuned search (tune=True, not the fast tune=False path), so it is a fair, not inflated, comparator. Recommended next step, not yet run: the same benchmark at 5 train / 4 held-out shards, matching every other probe row's convention in this table, before drawing an architecture conclusion from the pre>post pattern specifically.
subset_run_v8_taskset_v3 (EHRSHOT-style probe benchmark, FULL SCALE)	cbm	MIMIC 5 train shards, 4 held-out shards	e231fb7	Same benchmark as the smoke-scale row above, at the scale every other probe row in this table uses (5/4 shards), run to check whether the smoke run's pre>post-bottleneck gap for the 5 v3-only concepts was real or a small-sample artifact	done (2026-08-28)	probe_pre / probe_post [95% CI] / gbm [95% CI] AUROC, n at risk (n positive): anemia 8h 0.779/0.753 [.729,.775]/0.911 [.895,.927] (109,501; 595), 24h 0.753/0.734 [.713,.758]/0.899 [.880,.916] (96,903; 1,526), 72h 0.717/0.707 [.680,.732]/0.877 [.854,.898] (66,044; 3,386); hyperkalemia 8h 0.790/0.735 [.707,.762]/0.817 [.787,.843] (114,502; 587), 24h 0.742/0.706 [.672,.737]/0.814 [.780,.845] (101,896; 1,304), 72h 0.691/0.666 [.634,.698]/0.755 [.719,.790] (70,898; 2,669); hypoglycemia 8h 0.740/0.724 [.702,.746]/0.768 [.743,.792] (115,944; 593), 24h 0.635/0.623 [.602,.648]/0.684 [.652,.715] (103,274; 1,471), 72h 0.615/0.605 [.580,.632]/0.683 [.649,.715] (72,193; 3,258); hyponatremia 8h 0.765/0.716 [.685,.748]/0.914 [.893,.932] (114,595; 494), 24h 0.709/0.686 [.657,.714]/0.887 [.863,.911] (101,806; 1,128), 72h 0.662/0.646 [.615,.676]/0.828 [.801,.855] (70,443; 2,415); thrombocytopenia 8h 0.841/0.825 [.800,.848]/0.968 [.959,.976] (107,552; 785), 24h 0.814/0.804 [.778,.829]/0.952 [.940,.964] (95,239; 1,743), 72h 0.769/0.762 [.733,.789]/0.918 [.899,.936] (65,141; 3,029); long_los (snapshot) n/a/0.681 [.663,.701]/0.735 [.717,.753] (4,620; 1,090). READING: GBM still wins all 16/16 cells at full scale, every CI pair still separated (closest margins: hypoglycemia@8h 0.724 vs 0.768, and long_los 0.681 vs 0.735 -- both real but the smallest gaps in the table; widest: thrombocytopenia@8h 0.825 vs 0.968). The smoke run's headline pre>post finding was partly a small-sample artifact -- the full-scale drops are 2-4x smaller across the board. Pre-bottleneck/post-bottleneck deltas at 8h: anemia -0.026 (smoke: -0.054), hyperkalemia -0.055 (smoke: -0.100), hypoglycemia -0.016 (smoke: -0.044), hyponatremia -0.049 (smoke: -0.086), thrombocytopenia -0.016 (smoke: -0.087) -- every single cell's gap shrank substantially with more data, confirming the smoke row's own stated caveat. REVISED READING: anemia/hypoglycemia/thrombocytopenia now lose about as much through the bottleneck (0.7-2.6pp at 8h, shrinking further by 72h) as the ORIGINAL v1/v2 tasks did (0.3-3.4pp, see the bottleneck-signal-probe row above) -- no longer a distinguishable pattern. hyperkalemia and hyponatremia still lose more (4.9-5.5pp at 8h) than the v1/v2 ceiling, a real but much softer signal than the smoke run suggested, and not uniform across all 5 concepts the way the smoke run implied. Net effect on the capacity-competition hypothesis (smoke row's speculative reading): weakened, not confirmed, and no longer supported as a clean/uniform effect across task_set v3's concepts -- at most 2 of 5 concepts show an elevated bottleneck cost, and even those are a few points, not the 5-10pp the smoke run showed. The dominant, robust finding at both scales is unchanged: GBM wins every cell, by a wide and now tightly-estimated margin.

Retired: context-truncation stratification (2026-08-24). Not deferred, retired: every run in this comparison (v8, E, D, and every prior arm) trains backbone="hybrid", which never invokes PackedContextSampler, so IndexRow.is_tail is always False and no truncated stratum exists on any run we report or will report under the current architecture. Confirmed directly rather than assumed: 0 of 33,340 rows on a sampled real AKI shard. exposure_mask still implements and tests the truncation correction, because it is meant to outlive this arm. Revisit only if a transformer-backbone arm is run. Note this retires the truncation ANALYSIS only; Track A item 9, the model-scale ladder, is untouched and still open pending a costed decision.

Open defect: non-deterministic index-row ordering inflates cross-run comparator variance (found 2026-08-24, fix pending). Symptom: the tuned GBM is a control in the value-tail arms -- it never reads the value channel and scores the same held-out shards in every arm -- yet its own AUROC differs across arms, reading 0.9452 / 0.9518 / 0.9229 on death@8h for C / B / v8, a 2.9pp spread, and 0.5-1.5pp elsewhere.

Diagnosis, empirical and in this order. The GBM classifier is NOT unseeded: random_state=seed is threaded through and three in-process fits on identical data are bit-identical (max abs diff 0.0). Column order does NOT vary: feature_names_hash is identical across separate processes, so feature_names() really is fixed by module constants and the set(codes) at baseline_features.py:418 only builds lookup dicts. What DOES vary is row order: across two invocations of the same script in separate processes, keys_hash and x_full_hash differ while x_nansum is identical to ten decimal places -- the same multiset of values, differently ordered.

Root cause: _index_rows_from_events (odyssey/inference/alerts.py:1934, the default index_mode="landmark" path) calls group_by(...).agg(...) with no maintain_order=True and no subsequent sort, then zips the result positionally. Polars does not guarantee group order without it, and its multi-threaded grouping varies run to run. Siblings at 1861/1906/1915 share the pattern; three other call sites in the same files DO set maintain_order=True, so this reads as an oversight.

Causal link to the AUROC swing: _tune_gbm (alerts.py:1137-1176) subsamples when rows exceed GBM_TUNE_MAX_ROWS = 200_000 using a seeded rng, so it draws the same POSITIONAL indices every run -- applied to a differently-ordered array, fixed positions select a genuinely different subset of subjects, hence a different fitted GBM. vasopressor_start has 990,670 rows at 30 shards, so the path is live in production. This is a real mechanism, not float noise: 1e-15 aggregation differences cannot produce a 2.9pp swing, a different training subsample can.

Blast radius: the hazard head is UNAFFECTED (AUROC is order-invariant and it scores all rows). Every comparator that SUBSAMPLES is exposed to the same mechanism and needs checking: TabICL's row cap, SurvivalPFN's SURVIVALPFN_MAX_ROWS = 5000, EBM's 100k subject-grouped sample.

Framing: published comparisons were internally consistent WITHIN each run. What this cost is cross-run precision, so past numbers are imprecise across arms rather than wrong. Fix queued (explicit sort at the grouping sites, plus keying _tune_gbm's subsample off stable identifiers rather than array positions, which removes the class rather than today's symptom). Acceptance criterion: re-measure the GBM cross-arm spread after the fix; if it does not collapse toward zero the diagnosis is incomplete and nothing is closed. Note this partly REMOVES rather than merely quantifies the noise floor, so it interacts with the bootstrap-uncertainty work (2501436): that measures finite-sample variance, this shrinks refit variance.

Naming collision: there are TWO unrelated "M-series" (found 2026-08-24 by an audit, after the lead issued a "run M3, M4, M6" order that could not be satisfied by either).

1. docs/archive/experiment_plan.md's VM1/MIMIC wave queue: M1 epoch-2 eval chain, M2 wave MIMIC dump, M3 v9-MIMIC case-study regen, M4 L2-L4 intervention reruns, M5 (done), M6 v9-MIMIC seed replicate. That doc is dated 2026-08-23 and is framed entirely around LANDMARK_PROTOCOL_VERSION=2, superseded by the v3 work.
2. This registry's OWN subset_run_M1 / M2 / M3a / M3b rows: the stage-B training-recipe experiments (README roadmap item 10). This series has no M4 and no M6. N1 (docs/archive/track_b_designs.md, "combine the M-series' two partial wins") refers to M2 + M3a from THIS series, not from the plan doc's queue.

Consequences, all verified rather than inferred: M6 as written is gated on eICU E2/E3 and is DEAD under the 2026-08-23 MIMIC-only directive. M3/M4 are gated on "M2 done", meaning a v2-protocol wave dump that has no corresponding row here and has in any case been overtaken by the v3 comparator work landed under different names. N1 itself is real, MIMIC-only, no eICU dependency, ~4-5h A100, but carries the same ambiguous "wave closed" gate.

Authority ordering, per the 2026-08-23 directive: the README Roadmap section is authoritative. docs/archive/experiment_plan.md and docs/archive/track_b_designs.md both need a pass against the README roadmap and against this registry before anyone executes against their gate language again. Do not issue or accept work items by bare M/N identifier without naming which series and checking the gate against this file.

Found work: subset_run_v8_taskset_v3 and both bottleneck-signal probes ran 2026-08-25, sat unregistered on the VM until 2026-08-28. Discovered while looking for prior pre/post-bottleneck probe results: ~/runs/ on odyssey-cbm-a100 had a full v3-task-set training run plus two completed probe log sets (probe_smoke.log, probe_full2.log, probe_ci_check.log) that were never reported to the registry. Recorded as three rows above; headline synthesis for anyone deciding what to try next toward beating the GBM:

1. For vasopressor_start / icu_admission / acute_kidney_injury / sepsis3: the concept bottleneck barely costs signal (0.3-3.4pp on a linear-probe ceiling), and the trained hazard head already extracts essentially all of what a linear probe can find post-bottleneck (15/17 cells statistically overlap in the CI check). The GBM gap on these tasks is upstream of both the bottleneck and the head -- it needs more signal reaching the representation in the first place. This corroborates, from an independent angle, the 2026-08-24 GBM feature-group ablation's finding that explicit counts/aggregates (not encoding or head capacity) drove most of the vasopressor/ICU margin: explicit windowed count/aggregate features feeding the backbone remains the best-supported next architecture change for these tasks, not a bigger bottleneck or head.
2. death@8h is the one cell where the trained hazard head clearly outperforms the linear-probe ceiling (0.904 vs 0.829, CIs separated) -- the nonlinear, censoring-aware head is doing real work there. Not a target for representation changes.
3. readmission_30d@720h is the standout actionable lead: hazard AUROC (0.527) is barely above chance and 20pp behind the GBM (0.723), yet a linear probe on the SAME frozen post-bottleneck representation scores 0.586 (CIs separated from the hazard head) -- the representation already holds more readmission signal than the head is using. This is fixable at the head/task-formulation level (a real architecture or training bug in the readmission hazard head, or a formulation issue specific to the 720h discharge-anchored landmark), independent of any backbone or bottleneck work. Not yet done: scripts/probe_ci_check.py is uncommitted (VM-local only) and should be committed with a test before anyone else relies on it; the KDIGO AKI staging fix (3d7ecbb) and the tabicl_baseline offload work (351d613) landed on main after this run's commit (9285f68), so a rerun on current main would be needed before quoting these AKI numbers as current.

| subset_run_v8_taskset_v3_counting | cbm | MIMIC 30 shards (train), same 4 held-out shards as every other v8 row | 31a86b9 | Counting-hazard widening test (plan: crystalline-watching-phoenix). Mechanism under test: is a time-to-next-occurrence hazard target for a specific drug class, for a rate-stationary process, an estimator of the same rate a window COUNT estimates -- so does training 6 extra low-base-rate hazard targets (vasopressor, inotrope, antiarrhythmic, neuromuscular_blocker, corticosteroid, bicarbonate; new code_regex onset mechanism, regexes copied from baseline_features.DRUG_CLASSES) force a rate-like representation the same way the GBM's own count features are rate estimators? Exact subset_run_v8_taskset_v3 recipe plus auxiliary_event_names widening EventHazardHeads from 6 to 12 event names (trivial width change, +270K params, no architecture change); these 6 are trained but never scored as alerts (alerts.py still iterates alert_events_for, not hazard_events_for) | done (2026-08-29) | 28,873 steps, 15269.4s (4h14m), best_val_loss=2.4268, no early stop, no crashes. Regression guard (existing 6 curated alert tasks, hazard AUROC, baseline subset_run_v8_taskset_v3 -> new, same held-out rows): vasopressor_start 8h 0.896->0.911 (+0.015), 24h 0.833->0.862 (+0.029), 72h 0.755->0.802 (+0.047); icu_admission 8h 0.899->0.899 (0.000), 24h 0.843->0.845 (+0.002), 72h 0.773->0.778 (+0.005); death 8h 0.904->0.901 (-0.003), 24h 0.876->0.876 (0.000), 72h 0.850->0.848 (-0.002); sepsis3 8h 0.854->0.847 (-0.007), 24h 0.834->0.830 (-0.004), 72h 0.781->0.780 (-0.001); readmission_30d 168h 0.504->0.523 (+0.019), 720h 0.527->0.578 (+0.051). acute_kidney_injury moved too (8h 0.777->0.800, 24h 0.726->0.748, 72h 0.682->0.711) but is NOT a clean before/after -- the VM synced to current main between runs, which includes the KDIGO staging fix (3d7ecbb), so this delta is confounded and excluded from the reading below. Net: clean win on vasopressor_start (grows with horizon, plausibly the auxiliary "vasopressor" hazard target sharing signal directly) and readmission_30d (both horizons, the exact cell the CI-check row above flagged as head/formulation-limited); icu_admission/death/sepsis3 flat within noise (<=0.7pp); no regression anywhere. Decision-rule test (scripts/probe_counting_signal.py rerun, 5 train/4 held-out shards, matching probe_counting.log's baseline exactly): pooled mean R^2 pre-bottleneck 0.3059 (baseline 0.3111, -0.0052), post-bottleneck 0.3368 (baseline 0.3393, -0.0025) -- flat, marginally lower. Per-target post-bottleneck R^2, baseline -> new (delta): vasopressor n_6h 0.2035->0.2158 (+0.0122), n_24h 0.3320->0.3343 (+0.0023), ever_visit 0.4185->0.4055 (-0.0130); inotrope n_6h 0.0071->0.0059 (-0.0012), n_24h 0.0148->0.0100 (-0.0049), ever_visit 0.0719->0.0414 (-0.0305); antiarrhythmic n_6h 0.0491->0.0489 (-0.0002), n_24h 0.0745->0.0696 (-0.0049), ever_visit 0.2797->0.2561 (-0.0236); neuromuscular_blocker n_6h 0.0241->0.0339 (+0.0098), n_24h 0.0643->0.0866 (+0.0224), ever_visit 0.2125->0.2056 (-0.0069); corticosteroid n_6h 0.0353->0.0366 (+0.0013), n_24h 0.0445->0.0421 (-0.0024), ever_visit 0.1462->0.1312 (-0.0150); bicarbonate n_6h 0.0552->0.0434 (-0.0118), n_24h 0.0787->0.0639 (-0.0148), ever_visit 0.1296->0.1174 (-0.0121). Of the 18 targeted sub-features, 5 improve (all small, +0.001 to +0.022), 1 is flat, 12 are flat-to-worse (down to -0.031); no target improves on ALL THREE of its own sub-features (n_6h/n_24h/ever_visit) at once -- vasopressor and neuromuscular_blocker each improve on the two count features but lose on ever_visit, the reverse of what the rate-recovery mechanism would predict if it were working (ever_visit, the coarsest/most rate-like target, should benefit most). READING per the plan's pre-registered decision rule: this is a clean null on the mechanism, not a confirming result -- the objective-level fix (widening the hazard head with rate-estimating targets) does not measurably transfer to the counting probe's R^2 ceiling for the 6 targeted low-base-rate drug classes. The architecture-capacity hypothesis (Mamba-2's diagonal-SSM state-tracking weakness limiting exact/near-exact counting regardless of what the training objective asks for) is correspondingly a bit stronger, though not proven by a single null. The vasopressor_start and readmission_30d alert-task gains are real and worth keeping (auxiliary_event_names costs nothing and helps two tasks) but are a DIFFERENT finding from the counting-recovery question this experiment was designed to answer -- do not conflate the two when reporting this row. | | subset_run_v8_taskset_v3_resetprob0 | cbm | MIMIC 30 shards (train), same 4 held-out shards as every other v8 row | 9285f68 (training); eval tooling at 31a86b9 in the same worktree | reset_prob=0.0 ablation. Single-variable change from subset_run_v8_taskset_v3's recipe: TrainingConfig.reset_prob 0.1->0.0, every other field byte-identical (verified via full config diff). Motivated by the counting-probe's worst-recovered targets (row above, _counting): does removing synthetic mid-visit history resets improve the model's ability to track low-base-rate drug-class counts, per the same rate/hazard-estimator logic tested there? Config diff and subject/event counts (29,970 subjects, 72,924,527 events, both arms) confirmed identical before drawing any conclusion. Coordinated live with peer session odyssey-4d, which was concurrently doing label/bug-fix work on the same VM and shared git history, to avoid resource or branch conflicts | done (2026-08-30) | Training anomaly, mechanism jointly traced with peer session odyssey-4d: 5,407 steps in 4,068s (67.8min) vs baseline's 28,873 steps in 16,583s (276min) for the SAME 2 epochs on the SAME data -- a ~4.1x wall-clock speedup and ~5.3x fewer gradient steps, not a smaller run. Mechanism: reset_prob fires at every visit-BOUNDARY token (_LaneBuffer.append_patient), and any event with no hospital-admission id is its own single-event "visit" (sequences.py::_assign_visits), so outpatient-dense stretches offer a reset almost every token. At reset_prob=0.1 this collapses average chunk fill to 15.4% real tokens (946,110,464 token-slots for 145,849,054 real tokens across the run); at reset_prob=0.0, fill is 82.3% (177,176,576 slots, same 145,849,054 real tokens). Same data, ~5.3x fewer/thicker optimizer steps -- "the arm saw less data" was the wrong first read (caught and corrected mid-investigation after the peer challenged it) and "same data, different optimization granularity" is the right one. Every registered run before this one used the 0.1 default, so this reads as a project-wide finding, not just an ablation result: most prior subset/full runs likely trained at ~15% chunk utilization without anyone deciding that deliberately. Decision: TrainingConfig.reset_prob default changed 0.1->0.0 in this commit -- see the alerts and probe evidence below for why, and the one caveat (icu_admission) that should stay on watch at the next full-scale run rather than block this change. Counting probe rerun (scripts/probe_counting_signal.py, 5 train/4 held-out shards, matching probe_counting.log's baseline exactly): pooled mean R^2 pre-bottleneck 0.3585 (baseline 0.3111, +0.0474), post-bottleneck 0.3993 (baseline 0.3393, +0.0600) -- a broad improvement, but NOT concentrated on the 6 targeted low-base-rate drug classes the original hypothesis was about. Per-target post-bottleneck R^2, baseline -> new (delta): vasopressor n_6h 0.2035->0.1873 (-0.0162), n_24h 0.3320->0.3195 (-0.0125), ever_visit 0.4185->0.5241 (+0.1056); inotrope n_6h 0.0071->-0.0019 (-0.0090), n_24h 0.0148->0.0014 (-0.0134), ever_visit 0.0719->0.0500 (-0.0219); antiarrhythmic n_6h 0.0491->0.0335 (-0.0156), n_24h 0.0745->0.0400 (-0.0345), ever_visit 0.2797->0.3232 (+0.0435); neuromuscular_blocker n_6h 0.0241->0.0304 (+0.0063), n_24h 0.0643->0.0711 (+0.0068), ever_visit 0.2125->0.2595 (+0.0470); corticosteroid n_6h 0.0353->0.0281 (-0.0072), n_24h 0.0445->0.0353 (-0.0092), ever_visit 0.1462->0.1430 (-0.0032); bicarbonate n_6h 0.0552->0.0578 (+0.0026), n_24h 0.0787->0.0828 (+0.0041), ever_visit 0.1296->0.1728 (+0.0432). 8 of 18 targeted sub-features improve, 10 get worse; no drug class improves on all three of its own sub-features, and the improvements skew toward the coarsest ever_visit target (4 of the 6 classes' ever_visit improves, sometimes sharply) while the finer n_6h/n_24h counts are mixed-to-worse. READING: the original "reset_prob erodes rare-class counting memory" hypothesis is NOT confirmed -- this pattern (broad pooled gain, concentrated on coarse/high-level targets, inconsistent on fine-grained counts) looks like the general optimization-granularity effect from the training anomaly above (fewer, thicker gradient steps improving overall fit) rather than a targeted rare-event-memory mechanism. Family-level features (lab/visit/billing/diagnosis/procedure/medication counts, not drug-class-specific) improved broadly and consistently (+0.04 to +0.13 across all 18 family features, every single one positive) -- consistent with a general, not counting-specific, benefit. Alerts regression guard, same held-out rows, hazard AUROC baseline -> new (delta), GBM control alongside to confirm stability: landmark leg -- vasopressor_start 8h 0.896->0.887 (-0.009, GBM 0.934->0.933 flat), 24h 0.833->0.839 (+0.006, GBM 0.914->0.914 flat), 72h 0.755->0.775 (+0.020, GBM 0.883->0.886 flat); icu_admission 8h 0.899->0.882 (-0.017, GBM 0.968->0.969 flat), 24h 0.843->0.818 (-0.025, GBM 0.954->0.953 flat), 72h 0.773->0.737 (-0.036, GBM 0.931->0.931 flat) -- the one consistent, horizon-worsening regression in the whole table, and the GBM control's own stability (<=0.001 movement) rules out a comparator-noise explanation; death 8h 0.904->0.917 (+0.013), 24h 0.876->0.891 (+0.015), 72h 0.850->0.863 (+0.013) (GBM flat +/-0.003 throughout); sepsis3 8h 0.854->0.861 (+0.007), 24h 0.834->0.841 (+0.007), 72h 0.781->0.788 (+0.007) (GBM flat +/-0.003 throughout). acute_kidney_injury is CONFOUNDED and excluded from this reading: n at risk changed (95,471->79,389 at 8h, etc.), meaning the eval-tooling worktree picked up the KDIGO staging fix (3d7ecbb) between the two evals, same confound already documented on the _counting row above -- do not quote an AKI delta from this row. Readmission leg (clean, unconfounded: n/pos at risk match baseline exactly) -- readmission_30d 168h 0.504->0.550 (+0.046, GBM 0.664->0.664 flat), 720h 0.527->0.611 (+0.084, GBM 0.723->0.723 flat) -- the largest, cleanest win in the table, on the exact cell the CI-check row (above) flagged as head/formulation-limited rather than representation-limited. Net verdict: 4 of 6 tasks improve or are flat-to-mixed (death and sepsis3 consistently +0.7 to +1.5pp at every horizon; readmission +4.6 to +8.4pp, clean; vasopressor mixed, improving with horizon), 1 task regresses consistently (icu_admission, -1.7 to -3.6pp, worsening with horizon), 1 is confounded (AKI, no reading). Combined with the ~4-5x training-efficiency win, this supports adopting reset_prob=0.0 as the new default while flagging icu_admission for a repeat check at full scale (this is a 30-shard subset; GBM refit noise alone is 0.5-1.5pp typical, up to ~2.9pp on sparse cells, so icu_admission's regression is larger than that floor but has not been checked at scale). Decision (adopt 0.0 as TrainingConfig's default) made explicitly by Amrit on this evidence, 2026-08-30. Caveats for anyone using this row, flagged by peer session odyssey-4d before push: (1) any run comparing against prior-generation (pre-2026-08-30) numbers must pin reset_prob=0.1 explicitly, or the ~5.3x optimizer-step-granularity change confounds with whatever else changed (label/protocol fixes especially) -- subset_run_v10_taskset_v3 (in flight on this VM at push time) pins 0.1 for exactly this reason, as the fixed-code like-for-like validation of the v8 recipe. (2) icu_admission is the natural seed-replicate candidate when the full-scale retrain matrix launches under the new default, since icu_admission cells already showed the largest cross-run variance in this registry's own noise-floor measurements (row 92-98 above). | | full_run_v10 (R2) | cbm | MIMIC all 292 train shards, full held-out eval | cdbd4e7 (train + eval chain) | flagship post-fix MIMIC joint retrain: v10 recipe (task_set v3, 29 concepts, landmark v4, bin-edge v2, reset_prob 0.0, concept_supervision visit) | done (2026-08-31) | Training 40,750 steps, best_val 2.0556, ~7.2h. Eval chain all stages EXIT 0 (report 15:56 UTC). Set top-1 81.60 (prior-gen full_run_v8 80.4), top-1 37.25, top-5 69.94, xent 3.5560; 29-concept readout mean AUROC 0.9307 (0.775-0.999; prior-gen 0.857/15). Banded interventions (0.15, 76.1% coverage): none 37.25, truth -0.19, flip -0.05, random -0.12 (wrong sign SURVIVES on top-1); losses 3.5560/3.5717/3.5730/3.5723 -- prior-gen loss-ordering corroboration did NOT survive (substitutions within 0.0013 nats); displacement truth 0.4904/flip 0.5096. zero_known retains 6.6%, zero_unknown 98.4% and improves loss (3.494). Per-subject dump MISSING from this chain (pre-fix script); regenerated by the supplemental Guide Labs rerun for CIs. alerts.json + alerts_rows.parquet written (B1 gate). SUPPLEMENTAL INTERVENTION RERUN + PAIRED CIs (2026-09-01 06:31-09:30 UTC, 37 shards, 9 modes, --dump-per-subject; the chain's original run had 6 modes and NO per-subject dump, so the paper's headline MIMIC lever claim had carried no interval at all). Modes: none 37.254, truth 37.059, flip 37.204, flip_gated 37.232, truth_calibrated 34.270, flip_calibrated 34.251, random 37.132, zero_known 2.463, zero_unknown 36.645; losses 3.5560 / 3.5717 / 3.5730 / 3.5693 / 3.8584 / 4.0512 / 3.5723 / 8.6736 / 3.4938. truth-flip = -0.1454 [-0.1564, -0.1351] SEPARATED and truth-none = -0.1949 [-0.2010, -0.1892] SEPARATED, so the wrong sign and the "a true override hurts relative to not intervening" claim are both CI-backed on the flagship arm; the paper's "pending its paired CI" caveat is retired. truth_calibrated - flip_calibrated = +0.0193 [-0.0580, +0.1092], NOT separated, so output calibration does not rescue the sign here either, it just leaves it undetermined -- unlike R6 (eICU mixture) where calibration REVERSED it, and unlike R8 (MIMIC global pairs) where calibration came out correct. zero_known -34.79 pt confirms the named channel carries almost everything on this arm (2.46 of 37.25 retained, the 6.6% in tab:decomp) while zero_unknown costs only -0.61. | | eicu_full_v10 (R6) | eicu | eICU v2 all 134 train shards, full held-out eval | cdbd4e7 (train + eval chain) | flagship post-fix eICU joint retrain, same v10 recipe (26 concepts after source drops) | done (2026-08-31) | Training 35,500 steps, best_val 2.0148, ~4.9h. Eval chain all stages EXIT 0 (report 15:07 UTC). Set top-1 88.13 (prior-gen 87.5), top-1 54.01, top-5 87.59, xent 1.9750; 26-concept readout mean AUROC 0.8741 (0.6535-0.9655); time NLL 0.208. Banded interventions (0.15, 99.98% coverage): none 54.01, truth +0.19, flip -0.38, random -0.10; losses 1.9750/1.9985/2.0018/2.0003 (metric disagreement persists: truth helps top-1, hurts loss; but truth no longer worst-on-loss -- ordering now direction-consistent); displacement truth 0.4872/flip 0.5128 (prior-gen sign-flip on this arm did not survive). zero_known retains 48.5% (prior-gen 33.4), zero_unknown 100.7% and improves loss (1.9583). Per-subject dump missing, same recovery path. Paper joint cells swapped for both runs 2026-08-31 (main.tex header log has full provenance). | | eicu_full_v10 (R6, readmission leg) | eicu | eICU v2, all 17 held-out shards, index_mode=visit_end at 168/720h | b12a138 | first readmission_30d scoring attempt on eICU (Amrit: readmission across all three datasets) | DEGENERATE, do not retry (2026-08-31) | 20,122 discharge-anchored index rows written, but only 69 non-null labels at 168h and 5 at 720h, with ZERO positives; alerts_readmission.json is an empty list and no cell is scoreable. Structural, not a bug: eICU's MEDS subject_id is $patienthealthsystemstayid (specs/eICU.yaml), so a subject IS one hospital stay and a next_visit event has nothing after the visit's last event to find -- the same reason icu_admission is degenerate on this source. NOTE a proxy that misleads: 158 of 979 held-out subjects carry 2+ HOSPITAL_ADMISSION// tokens, which looks like readmission is available; those are admission records WITHIN one health-system stay. Check label columns, not token counts. readmission_30d remains deliverable on MIMIC (subject = patient) and on GEMINI (per-source prefix remap, 36e1055), not on eICU short of re-keying the extraction to uniquepid, which would invalidate every existing eICU run. | | full_run_L_v10 (R8) | cbm | MIMIC all 292 train shards, full held-out eval | 882d812 (eval chain; trained at b98ff7d) | L-series full scale on MIMIC: R2's config with concept_global_pairs=True as the only change. The MIMIC counterpart of R9, and the arm that replaces Sec 5.3's [SUBSET-PLACEHOLDER] citation of subset-scale L1. | training done; interventions + paired CIs done (2026-09-01 04:10 UTC); rest of chain in flight | Training: 47,750 steps in 30,204s (8.4h), early_stop=True, best_val_loss=2.1362, zero errors. Against R2's 2.0556 that is a cost of 0.081 nats, versus 0.253 on eICU (R9 2.2677 vs R6 2.0148): the global-pairs capability penalty is roughly 3x smaller on MIMIC than on eICU, so the lever/capability trade is not a fixed property of the architecture and is cheapest on the dataset carrying the paper's wrong-sign headline. Eval chain launched 00:57:59 UTC with ALERT_SHARDS=37 (all MIMIC held-out shards) so it is coverage-matched to R2 and does NOT repeat the R9-vs-R6 mismatch; launched automatically by a detached waiter that refuses to fire unless checkpoint_final.pt exists, so a crash cannot be mistaken for completion. INTERVENTIONS (37 shards, 87,058,459 predictions) + PAIRED CIs, and the headline does NOT replicate R9. Modes at band 0.15: none 34.556, truth 27.868, flip 28.093, flip_gated 32.675, truth_calibrated 33.959, flip_calibrated 33.689, random 27.617, zero_known 19.800, zero_unknown 11.682; losses 4.0238 / 4.9004 / 5.0596 / 4.3650 / 4.0818 / 4.1136 / 5.0507 / 4.3213 / 6.3216. All ten paired deltas SEPARATED. truth-flip = -0.2246 [-0.3003, -0.1505]: WRONG-SIGNED on MIMIC, against R9's +1.279 [+0.863, +1.672] on eICU. So concept_global_pairs=True does NOT correct the banded sign on MIMIC; it stays wrong and gets slightly worse than R2's mixture (-0.146 -> -0.225). The output-calibrated protocol DOES come out correct here: truth_calibrated - flip_calibrated = +0.2698 [+0.2319, +0.3086]. And under the band, LOSS is correctly signed (truth 4.9004 vs flip 5.0596, -0.159 nats) while top-1 is not, so this arm has the metric disagreement R6 showed, with the datasets swapped. Read together with R9: the ablation moves the lever, but the direction it moves depends on dataset AND protocol, which is why the paper's abstract now scopes the correction to eICU instead of claiming an architectural fix. COMPLETENESS REVERSAL REPLICATES: zero_known retains 57.3% of no-intervention top-1 and zero_unknown 33.8%, so the residual is MORE load-bearing than the named channel -- the same inversion R9 showed on eICU, against R2's mixture profile on this very dataset (6.6% / 98.4%). Two arms, two datasets, same direction: global pairs buys sign behaviour partly by shifting reliance off the named concepts. Magnitudes are far larger than the mixture arm's (truth-none -6.69 pt here vs -0.19 for R2), as expected when an override injects a fixed direction instead of re-weighting context-dependent vectors. ALERTS EXIT 0 06:26 UTC, and this is the project's FIRST matched-coverage capability comparison of the global-pairs ablation (R9-vs-R6 was confounded by 4-vs-17 shards; R8 and R2 both scored all 37 MIMIC held-out shards, and n_at_risk agrees cell-for-cell). Hazard AUROC falls in 15 of 15 cells, mean -0.0067, range -0.0013 (death@72h) to -0.0126 (sepsis3@72h): R2 -> R8 AKI 0.8703/0.8205/0.7713 -> 0.8615/0.8137/0.7663, death 0.9616/0.9457/0.9252 -> 0.9597/0.9437/0.9239, ICU 0.9353/0.8916/0.8306 -> 0.9300/0.8864/0.8254, vasopressor 0.9462/0.9156/0.8702 -> 0.9377/0.9067/0.8621, sepsis3 0.9121/0.8913/0.8514 -> 0.9025/0.8803/0.8388. Uniform in sign and small in size, so the ablation's cost shows up at the alert level too and not only in val_loss. Still a cross-run single-seed comparison, hence a labeled hypothesis. | | eicu_full_L_v10 (R9) | eicu | eICU v2 all 134 train shards, full held-out eval | b99be32 (eval chain; trained at 17d28fc) | L-series full scale: R6's exact config with concept_global_pairs=True as the ONLY change (verified by config diff: the two configs differ in that key and output_dir, nothing else). Tests whether the subset-L1 sign correction survives and becomes useful at full scale, per the Guide Labs/Steerling structural-cause argument (Sec 5.3). | training done (2026-08-31 18:04:57 UTC); eval chain in flight | Training: 18,250 steps in 8,995.5s (~2.5h), early_stop=True at 15/15 evals without improvement, best_val_loss=2.2677, zero errors, checkpoint_final.pt written. The capability cost of global pairs is now visible at full scale on eICU and it is LARGER than the subset L1 result suggested: against R6 (the same recipe with context-dependent pairs) this arm converges both EARLIER and WORSE -- 18,250 steps vs 35,500, best_val_loss 2.2677 vs 2.0148 (+0.253 nats). Early stopping fired on genuine plateau, not a crash. Read this as the training-side half of the L-series question: the lever/capability trade-off the paper argues for in Sec 5.3 shows up in the optimisation itself, before any intervention is scored. The intervention half (does truth-flip come out correctly signed, and does any mode beat none) waits on the eval chain. Eval chain ran 18:07:44-20:07:36 UTC (STREAM_BASELINE=1, LANES=64): eval / attribution / alerts / cases all EXIT 0; interventions EXIT 1 on the calibrated-mode device bug (fixed 414cc0d; every calibrated mode had never once completed on a GPU) and report EXIT 1 cascading from its missing output. Re-run of interventions+report+CIs launched 20:09:35 UTC at b12a138; the failed log is preserved as interventions_failed_devicebug.log. NOTE: the chain does NOT run scripts/intervention_cis.py -- that is a separate manual step on interventions_band15_per_subject.json. CAPABILITY COST OF GLOBAL PAIRS, measured R9 vs R6 (SINGLE SEED EACH, so a cross-run comparison and a labeled hypothesis under the stats policy, not a CI-backed claim): forecasting set top-1 88.13 -> 86.67 (-1.46), top-5 87.59 -> 84.60 (-2.99), xent 1.9750 -> 2.1014 (+0.126), but top-1 54.01 -> 55.20 (+1.19, the one metric that improves -- the same metric disagreement this arm shows on interventions); 26-concept readout mean AUROC 0.8741 -> 0.8511 (-0.023), so the readout degrades too, not just the forecaster; hazard alert AUROC worse on 9 of 12 cells, mean -0.016, concentrated in AKI (-0.036 to -0.044) and icu_admission (-0.029 to -0.034), with all three death cells the only improvements (+0.013 to +0.020). SHARD-COUNT CAVEAT (added 2026-08-31 21:35 after the fact): R6's chain scored the FULL held-out split while R9's scored 4 of eICU's 17 shards, so these two arms are compared on DIFFERENT held-out subsets -- R6's alerts carry 4.15x R9's rows on every cell (death 8h 621,125 vs 149,152), and R6's intervention n_predictions equals its own full-eval count (86,897,282) against R9's 20,708,344. AUROC and top-1 estimate the same population quantities either way so the comparison is not invalid, but it is UNPAIRED and noisier than the deltas suggest, and R9's side rests on a quarter of the data. Treat the direction as the finding and the magnitudes as approximate until an arm is rerun at matched coverage. Operational rule this produced: launch an arm with the SAME ALERT_SHARDS as the arm it will be compared against; nothing in eval_run.sh warns about a mismatch. Read alongside the training-side result above (converged earlier AND worse): the global-pairs ablation buys the intervention sign correction at a cost that is broad rather than confined to the forecasting objective. INTERVENTIONS + FIRST PAIRED CIs IN THE PROJECT (re-run 20:09-20:41 UTC at b12a138 after the device-bug crash; all three stages EXIT 0; the four modes that had succeeded before reproduced digit-for-digit, a clean determinism check). Modes at band 0.15: none 55.51, truth 51.13, flip 49.85, random 49.96, flip_gated 54.50, truth_calibrated 53.27, flip_calibrated 52.60; zero_known 40.15, zero_unknown 29.43. Paired subject-clustered CIs (2000 boots, 3,916 subjects, intervention_cis.json), all ten pairs SEPARATED: truth-flip +1.279 [+0.863, +1.672] and truth_calibrated-flip_calibrated +0.677 [+0.293, +1.041], so the sign correction is significant under BOTH the banded protocol and Guide Labs' output-calibrated protocol (no band at all) -- the band was not doing the work, which is what W7 was built to test. Every mode remains significantly BELOW none (truth-none -4.379 [-4.604, -4.172]), so "corrects the sign at a capability cost" holds with intervals. flip_gated-none is only -1.010 vs flip's -5.658, i.e. gating recovers most of flip's damage (Guide Labs Fig 19 artifact check), so much of an override's cost is logit-magnitude disruption rather than meaning. COMPLETENESS REVERSAL vs R6, worth attention: zero_known retains 72.3% here vs R6's 48.5%, while zero_unknown retains 53.0% vs R6's 100.7% -- so in the global-pairs arm the UNNAMED residual is MORE load-bearing than the named channel, the opposite of the flagship pattern. Hypothesis (one seed, cross-run): the ablation fixes the lever partly by making the model rely on named concepts less overall. Needs R8 + replicates before it is more than that. | | full_run_v10 (R2, B1 TabICL leg) | ultra | MIMIC, 8 train / 4 held-out shards, strong 609-feature panel | a02c1b3 (script at d4a82e3 for the matched pass) | full-capability TabICL(strong) on v4 rows and post-KDIGO labels, closing the three provenance problems that blocked the Aug 27 v3 sweep (one dump, one protocol, post-fix labels) | done (2026-09-01 02:14 UTC), ~7h | 15 cells, n_estimators=8, 50,000-row context, offload cpu, ODYSSEY_TABICL_MEMORY_BUDGET_GB=120. TabICL loses 14 of 15 cells to the tuned GBM (its only win is death@8h, 0.9584 vs 0.9546), which RETIRES the old "ties the GBM on 9/12" reading from the v3 sweep. Against our hazard heads it leads in 10 of 15 (four of those by under 0.01, and one hazard lead is a 0.00004 tie at death@72h), decisively on ICU admission: 0.9642/0.9452/0.9153 against 0.9357/0.8937/0.8290 (+0.029/+0.052/+0.086, widening with horizon). COVERAGE, READ THIS BEFORE USING THE NUMBERS: --max-held-out-shards defaults to 4, so every cell here is scored on 4 of the 37 held-out shards while alerts.json covers all 37 (n differs by 8.8x). These numbers are therefore NOT interchangeable with tab:mimic's and must not be spliced into it; the paper gives them their own table (tab:tabicl) whose hazard and GBM columns were rescored on these same rows. See the matched pass below. | | full_run_v10 (R2, matched hazard/GBM pass) | ultra | MIMIC, the same 4 held-out shards B1 scored | d4a82e3 | rescore hazard and GBM on TabICL's own rows so the comparator table's three columns describe one sample | done (2026-09-01 02:17 UTC), ~2 min | tabicl_strong_compare.py --skip-tabicl (new flag: skips the train prep and the fits, which is what makes it minutes rather than hours). The correctness check is built in and it passed: the GBM this pass reports reproduces B1's to 7 decimals on every cell (e.g. vasopressor@8h 0.9386657033975592 in both), which is only possible if the row sets are identical. Matched hazard: AKI 0.873/0.822/0.773, death 0.968/0.946/0.925, ICU 0.936/0.894/0.829, vasopressor 0.938/0.904/0.843, sepsis3 0.908/0.887/0.846. Also wrote tabicl_matched_rows.parquet (690,945 rows) in alerts_cis.py's column layout. make_comparator_tables.py --matched merges the two JSONs and REFUSES if the shared GBM column ever disagrees. | | full_run_v10 (R2, B2 ICU paired leg) | ultra | MIMIC, same 4 held-out shards, icu_admission only | d4a82e3 | B1 did not dump per-row TabICL scores, so the load-bearing "TabICL beats our heads on ICU admission" claim had point estimates but no PAIRED verdict. This reruns that one event with --dump-rows (~1.5h rather than another ~7h for all five). | done (2026-09-01 03:52 UTC), ~1.6h | All three ICU cells separate: hazard-minus-TabICL is -0.0296 [-0.0367, -0.0233] at 8h, -0.0524 [-0.0625, -0.0422] at 24h, -0.0862 [-0.1046, -0.0670] at 72h, so "TabICL beats our hazard heads on ICU admission" now carries a paired subject-clustered verdict at every horizon rather than three point estimates. hazard-minus-GBM separates too (-0.039/-0.067/-0.111). Scorer AUROCs: hazard 0.9357/0.8937/0.8290, GBM 0.9743/0.9606/0.9401, TabICL 0.9653/0.9461/0.9153. Incidental but useful: this is an INDEPENDENT TabICL refit on the identical rows, and it reproduces B1's column to within 0.0011 (0.9642/0.9452/0.9153 there), so TabICL's refit noise is ~0.001, the same order as the tuned GBM's 0.001-0.002 that the paper's noise-floor argument cites. The other 12 cells keep per-column CIs without a paired test; a full paired rerun is ~7h and was judged not worth it against a 0.03-0.09 gap. | | full_run_v10 (R2) / eicu_full_v10 (R6), alert CI legs | cbm / eicu | MIMIC 37 shards, eICU 17 shards | 41c862b | neither flagship had ANY alert CIs, so both comparator tables were bolding on plain arg-max, which their own captions forbid | both done (2026-09-01 ~04:30 UTC) | scripts/alerts_cis.py --scorers hazard gbm, 1000 boots, subject-clustered, plus the paired hazard-vs-GBM delta per cell. CPU-only and niced, so they run beside the GPU jobs on each host. First cells: MIMIC AKI@8h hazard 0.870 [0.867, 0.874] vs GBM 0.913 [0.911, 0.916]; eICU AKI@8h hazard 0.741 [0.733, 0.749] vs GBM 0.891 [0.886, 0.895]. RESULT: every paired hazard-vs-GBM delta separates, 15 of 15 on MIMIC and 12 of 12 on eICU. Both comparator tables were regenerated with --cis, so bold is now gated on separation as the captions promise rather than being plain arg-max. On MIMIC the two hazard wins hold up: death@8h +0.0172 [+0.0097, +0.0253] and vasopressor_start@8h +0.0113 [+0.0066, +0.0165]. The three cells I expected might fail to separate (death@24h -0.0090, vasopressor@24h -0.0084, vasopressor@8h +0.0113) all did separate, because n is 100k-1.2M rows per cell and the subject-clustered interval is still tight at that size. On eICU the losses run -0.032 to -0.185. | | eicu_full_ADD_v10 (additive bottleneck) | eicu | eICU v2 all 134 train shards | (training) | bottleneck_kind="additive": out = h + sum_i p_i v_i, a Steerling-style additive bottleneck where an override is a fixed displacement and the backbone stream survives. The architectural alternative to global pairs, and the arm that decides whether the paper reports a working lever or an operating point. | training done (2026-09-01 06:23 UTC), 45,486 steps in 18,540s (~5.2h), early_stop=False (ran both epochs) | best_val_loss 1.9758, which BEATS the mixture flagship R6 (2.0148) on the same data by 0.039 nats, against R9's global-pairs 2.2677 (+0.253). So the additive bottleneck does not pay the capability cost that global pairs does; on this single seed it is slightly ahead of the mixture. CAVEAT, and it is the paper's own policy: R6-vs-ADD is a CROSS-RUN single-seed comparison, so it is a labeled hypothesis, not a measured effect, and the project's cross-run noise floor is large (Sec 5.5). A zero-concept-loss scare on the final training lines was checked and dismissed: only 35 of 1,819 train lines show concept_loss=0.0000 (sporadic batches with no observable concept, which also switches the log's field order), and the last validation line at step 45,250 has concept_loss 0.3443 / orthogonality 0.0060 / observability 0.1821, all live. Concept supervision ran throughout. orthogonality_loss is live and non-zero throughout (0.0006-0.0017), unlike the mixture penalty which goes silently inert at a narrowed residual. Eval chain armed by a detached waiter (ALERT_SHARDS=17 to match R6; the attribution stage is EXPECTED to fail, since concept_attribution guards its four mixture-only functions on the bottleneck type), followed by intervention_cis.py on the per-subject dump. EVAL CHAIN 06:24-08:28 UTC. eval EXIT 0, alerts EXIT 0, cases EXIT 0; interventions EXIT 1 and attribution EXIT 1, both on the SAME expected cause: the output-calibrated modes call mean_concept_directions, which slices the LM head into per-concept blocks that an additive bottleneck does not have, so _require_mixture_bottleneck raises TypeError. The guard is correct and loud; the chain just should not have lost a two-hour stage over two optional modes, which is fixed in eval_run.sh (595f5bb: reads bottleneck_kind from the run config, drops the calibrated modes and skips attribution for additive runs). Interventions relaunched 09:36 UTC with the seven compatible modes (none/truth/flip/flip_gated/random/zero_known/zero_unknown) plus --dump-per-subject, CIs chained. CAPABILITY, AT MATCHED COVERAGE (both arms all 17 eICU held-out shards, n_at_risk agreeing cell-for-cell): the additive arm BEATS the mixture flagship R6 on hazard AUROC in 9 of 12 cells, mean +0.0053, biggest gains on death (+0.0132/+0.0159/+0.0148), small losses on AKI@72h (-0.0092) and vasopressor@72h (-0.0074). With best_val_loss 1.9758 against 2.0148 that is two independent measurements agreeing that the additive bottleneck costs nothing in capability. Contrast with global pairs, which is the whole point: R8-vs-R2 on MIMIC is worse in 15 of 15 cells (mean -0.0067). So the two architectural routes to a steerable lever are not equivalent -- one pays uniformly and the other does not. Still cross-run single-seed, so a labeled hypothesis; the intervention result is what decides whether the lever actually works. INTERVENTIONS (relaunched 09:36, EXIT 0 11:05 UTC, 7 modes, 17 shards, 86,897,282 predictions) + PAIRED CIs. THE ADDITIVE LEVER DOES NOT WORK, AND THE COMPLETENESS PROBES SAY WHY. Modes: none 56.796, truth 56.775, flip 56.815, flip_gated 56.780, random 56.796, zero_known 55.907, zero_unknown 14.919; losses 1.8297 / 1.8318 / 1.8330 / 1.8323 / 1.8323 / 1.8277 / 8.1219. truth-flip = -0.0392 [-0.0600, -0.0172] SEPARATED, i.e. WRONG-SIGNED, and it is flip that helps: flip-none = +0.0188 [+0.0074, +0.0297] SEPARATED while truth-none = -0.0204 [-0.0309, -0.0098] SEPARATED. random-none is within noise (+0.0006). The mechanism: zeroing the NAMED channel costs only 1.6% of accuracy (retains 98.4%) while zeroing the RESIDUAL costs 73.7% (retains 26.3%). Against the mixture flagship R6 on the same data that is exactly inverted (named 48.5%, residual 100.7%). So the additive arm matches capability BECAUSE it barely uses the concepts: out = h + sum_i p_i v_i leaves the backbone stream intact and the model routes around the bottleneck. Its steerability guarantee is real but vacuous -- an override IS a fixed displacement, and that displacement moves almost nothing. The three eICU architectures line up monotonically in how much the model needs the named channel (mixture 48.5% retained on zeroing, global pairs 72.3%, additive 98.4%), and the lever gets no better along that line. That is the paper's unifying claim now: the more cleanly an override can be injected, the less the model depends on the channel it is injected into. | | full_run_v10 (R2, readmission leg) | cbm | MIMIC, all 37 held-out shards, index_mode=visit_end at 168/720h | 882d812 | readmission_30d on the one source where it is deliverable (subject = patient, unlike eICU where a subject IS one stay) | done (2026-09-01 09:58 UTC) | Both cells scoreable, unlike the eICU leg which had ZERO positives. 168h: n=45,053 (4,581 pos), hazard 0.5865 vs GBM 0.6960. 720h: n=43,349 (10,982 pos), hazard 0.6352 vs GBM 0.7373. The hazard heads lose both by roughly 0.10, the largest per-cell margin anywhere in the paper, and the 168h cell is close to chance. Completes Amrit's "readmission for all three datasets": MIMIC done, eICU structurally degenerate (documented above), GEMINI not attempted. | | full_run_DEC_v10 | cbm | MIMIC all 292 train shards | 1193aa3 (branch fix/steerling-concept-decomposition) | R2's config with bottleneck_kind='decomposed': Steerling's concept module, h = k_hat + u_hat + eps, replacing the additive bottleneck that turned out to be a skip connection with a decorative side-channel | RELAUNCHED 2026-09-01 14:41 UTC at c6eba41; the 12:46 launch was discarded | unknown_ratio 3, unknown_rank null, residual_dropout 0.1, independence_weight 1.0, teacher forcing 1.0->0.5 on both heads over 4,500 steps, orthogonality_weight 0.0 (structurally inert for this bottleneck, and now warned about at startup). reconstruction_weight 1/hidden_size = 0.003906, NOT Steerling's 1.0: Equation (12) sums squared error over the hidden width, so at d=256 the term lands near 215 against a task loss of ~5; at weight 1.0 it would be 97% of the gradient. 1/d is the per-dimension reading of the same weight and leaves the equation untouched. Their 1.0 balances at their width, not ours. BOTH ARMS WERE RESTARTED after ~1.5h, on a performance bug in the new code (c6eba41). _streaming_position_labels iterated chunk.subject_ids element-wise, and that tensor is on the GPU, so each int(s) forced a device sync: 64 lanes x 512 positions = 32,768 syncs per training step. It ran eICU at 43.5 steps/min against 205 after the fix, a 4.7x difference, and would have made the MIMIC arm ~16h rather than ~3.5h. Fixed with one .tolist() transfer plus a stack-once-per-distinct-key gather. The restart double-checks the fix's correctness: val_loss at step 250 is 3.8574 against the discarded run's 3.8575 on eICU, and 5.4980 against 5.4989 on MIMIC. Same trajectory, 4.7x less wall clock. NOTE the earlier ETAs in this file and in the session log were built on R2's throughput and were wrong even before the fix; the discarded runs were tracking 16h, not 9h. Operational note: restarting them, a pkill -f whose pattern also appeared in the relaunch command matched its own shell and killed the SSH session mid-command, leaving TWO trainings on VM1's card for about four minutes. Caught via nvidia-smi --query-compute-apps; kill by PID, or split kill and relaunch into separate invocations. | | eicu_full_DEC_v10 | eicu | eICU v2 all 134 train shards | 1193aa3 (same branch) | R6's config with bottleneck_kind='decomposed'; the eICU counterpart, and the arm directly comparable to the additive run that failed | RELAUNCHED 2026-09-01 14:14 UTC at c6eba41; the 12:45 launch was discarded | Same settings as the MIMIC arm. Reference points on this data: R6 mixture 2.0148, R9 global-pairs 2.2677, additive 1.9758 (which matched capability only by not using the concepts: zeroing them cost 1.6%). The question this arm answers is whether the three interpretability losses buy back concept reliance without giving up that capability. | | eicu_full_v10 (R6, TabICL leg) | ultra | eICU v2, 8 train / 4 held-out shards, strong 609-feature panel | 37892e5 | the last baseline the paper was missing: full-capability TabICL on eICU, matching what B1 gave for MIMIC | launched 2026-09-01 12:55 UTC; done 19:48 UTC (~6.9h), INFO wrote for both the JSON and the 629,068-row dump; ultra STOPPED 19:57 UTC (local SSD preserved). JSON at research_journal/figure_data/vm2/eicu_full_v10/tabicl_strong_v4.json (local, untracked); dump copied to VM2 ~/runs/eicu_full_v10/tabicl_matched_rows.parquet and alerts_cis.py --scorers hazard gbm tabicl launched there 19:51 UTC, ~2.5 min/cell | Matched-rows AUROC, hazard / GBM / TabICL: AKI 0.745/0.890/0.844, 0.728/0.841/0.788, 0.688/0.791/0.722; death 0.917/0.944/0.917, 0.874/0.937/0.897, 0.810/0.906/0.853; ICU 0.831/0.948/0.821, 0.798/0.943/0.775, 0.753/0.926/0.707; vasopressor 0.881/0.913/0.889, 0.840/0.894/0.859, 0.796/0.865/0.818. GBM wins all 12; TabICL beats the hazard heads in 9 of 12 and loses the three ICU-admission cells, the reverse of MIMIC where TabICL beat the heads on ICU admission. n per cell is 19,756 (ICU) to 149,152 (death@8h), a 4-shard subsample, so this is a --matched table (written to paper/ml4h/tables/eicu_tabicl_matched.tex, no CIs yet) and must never be merged into tab:eicu via --tabicl. Paired CIs done (20:18 and 20:59 UTC, VM2, CPU): alerts_cis.py keys pairs by the first scorer, so two runs were needed (--scorers hazard gbm tabicl, then gbm tabicl hazard) and were merged into tabicl_matched_cis_merged.json (marginal CIs identical, 1000 boots, subject-clustered). GBM-minus-runner-up separates in 12 of 12. TabICL-minus-hazard: AKI +0.099/+0.060/+0.034, death -0.000 (tie, not separated)/+0.023/+0.044, ICU -0.010 (not separated)/-0.024/-0.045, vasopressor +0.008 (not separated)/+0.019/+0.022; 9 of 12 separate. Table 5 in the paper regenerated with --cis, bold gated on separation. 4 events x 3 horizons = 12 cells (sepsis3 does not resolve on eICU). One pass, not three: the script now scores hazard off its own join instead of leaving it to be spliced from alerts.json at a different shard count, and --dump-rows feeds alerts_cis.py directly, so paired hazard/GBM/TabICL deltas come out of the same run. The MIMIC side needed B1 + a --skip-tabicl matched pass + B2 to reach the same place. eICU v2 data (322MB slice) copied to the ultra host, which only had v1. | | full_run_DEC_v11 | cbm | MIMIC all 292 train shards | a86616a | DEC_v10 restarted with reconstruction_weight 1.0 (Steerling's lambda_rec, confined to the unknown head) and residual_dropout 0.3 (their Table 26 p_eps) | launched 17:14 UTC; STOPPED 2026-09-01 19:43 UTC at step 13,600, superseded by v12 (checkpoints through 10,000 kept in ~/runs/full_run_DEC_v11) | One-shard coverage probe at step 9,025 (~/coverage_probe_v11.log, JSON saved as ~/coverage_v11_mimic_step9025.json) against the DEC_v10 probe at 9,000: none top-1 29.168 -> 28.368; zero_known costs 1.6% -> 2.2%; zero_unknown costs -0.1% -> 28.5%; zero_residual costs 17.4% -> 18.6%; banded and calibrated truth-flip both +0.322 pt, correct sign. So the unknown head woke up and nothing else moved. DO NOT quote this as "Steerling's remedy correctly implemented": a review of the code against the paper text (Sec 5.3-5.4, 10.2, Table 26) found four deviations, all biasing toward this exact outcome. (1) eps was computed from the model's own parts and held fixed under teacher forcing, so with both heads forced h_bar = h + dropout(eps_own) rather than h; the paper says eps = 0 there. Both alphas start at 1.0, so the LM head trained on a doubled residual direction early and a double-counted one in 3/4 of steady-state steps. (2) u_hat entered the bottleneck attached, letting the LM loss train the unknown head into a second residual; Steerling routes it "detached before the bottleneck". (3) AdamW decayed K and U at 0.01; the expected task gradient into K is zero under inverted dropout, so decay is a one-way shrink toward an inert named channel; Steerling decays "excluding embeddings". (4) Teacher forcing ignored the found-mask, so the 22.6% of MIMIC events with no hadm_id were forced to k_gt = 0 and u_gt = h. All four fixed in 69c8cb0. | | eicu_full_DEC_v11 | eicu | eICU v2 all 134 train shards | a86616a | eICU counterpart of the above, same two changes | training done 19:12:52 UTC, early-stopped at 12,000 steps (15/15 evals without improvement), best_val 2.3337 at step 8,250; eval chain started 19:13:32 on checkpoint_best.pt, eval EXIT 0 at 19:28, interventions stage running | Carries the same four deviations as the MIMIC arm; bank its numbers as the "v11 configuration" row, not as the Steerling arm. Not matched on training length to MIMIC (12,000 vs a planned ~45,000): never report the two v11 arms side by side as if they were. Reference points on this data: R6 mixture 2.0148, additive 1.9758; this arm's 2.3337 is worse than both, on a single seed. INTERVENTIONS EXIT 0 at 22:40 UTC (17 shards, 86,897,282 predictions, the additive arm's exact row count; JSON copied to research_journal/figure_data/vm2/eicu_full_DEC_v11/): none 53.59 / loss 2.0724; truth 53.36 / 2.0930; flip 53.82 / 2.0566; flip_gated 53.87; truth_cal 53.14 / 2.1091; flip_cal 53.56 / 2.0993; random 53.59 / 2.0745; zero_known 50.93 (costs 5.0%), zero_unknown 0.97 (costs 98.2%, loss 9.34), zero_residual 53.12 (costs 0.9%). Reading: on eICU the deviated decomposition puts essentially the whole prediction in the UNKNOWN channel; the residual carries nothing and the named channel 5%. The lever is WRONG-signed under both protocols and on both metrics (flip beats none, truth loses to none), the opposite of what the MIMIC v11 probe showed at 9k steps. Same row set as the additive arm (named 1.6%, residual 73.7%) and the mixture R6 (named 48.5%). Alerts and CI stages running. | | full_run_DEC_v12 | cbm | MIMIC all 292 train shards | 69c8cb0 | v11's config unchanged (~/dec_mimic_exact.json), code fixed for the four deviations above: eps is the remainder of the parts actually used, u_hat is detached on the bottleneck path, K/U are in a weight_decay=0 group, unlabeled positions keep the model's own parts under teacher forcing | launched 2026-09-01 19:44 UTC; chain waiter (ALERT_SHARDS=37) and the one-shard coverage probe at step 9,000 armed, outputs ~/dec_mimic_v12_waiter.log and ~/coverage_probe_v12.log | TRAINING DONE 23:57 UTC: 26,500 steps in 13,182s (~3.7h), early_stop=True after 15 evals, best_val 2.1925 at step 22,750 (v11 was 2.4587 at 10,000 when stopped; R2 mixture flagship trained 40,750 steps). Chain waiter fired 23:58 on checkpoint_best.pt at checkout be11a2f; eval stage running, interventions (37 shards) to follow, expect ~4h. PROBE DONE 21:47 UTC (~/coverage_probe_v12.log, JSON saved as ~/coverage_v12_mimic_step9000.json and in research_journal/figure_data/vm1/), matched to v11 at 9,025 and v10 at 9,000, one shard, 8 modes. none top-1 31.50 (v11 28.37, v10 29.17: the fix buys +3.1 points of capability at matched step); zero_known costs 4.1% (v11 2.2%, v10 1.6%); zero_unknown costs 47.6% (v11 28.5%, v10 -0.1%); zero_residual costs 19.6% (v11 18.6%, v10 17.4%); truth-flip +0.070 pt banded and calibrated (correct sign, v11 +0.322); truth-none -0.160 (v11 -0.139); losses none 4.148 / truth 4.174 / flip 4.154 / zero_known 4.408 / zero_unknown 8.416 / zero_residual 4.184. Reading: the corrected pressures moved the prediction into the UNKNOWN channel (47.6%), doubled the named share but only to 4%, and left the residual where it was (~19%). The lever is unchanged: truth barely beats flip and still loses to none. Still mid-training (9,000 of a planned ~45,000) and one shard; the full chain supersedes it. Hypothesis for the next change if the final chain agrees: Steerling's own mid-training tightening adds a second independence term Lindep(eps, k_hat) precisely because with u_hat detached nothing pressures the transformer residual (their Sec 10.2.2), plus top-k sparsification and annealing teacher forcing to 0; and with 29 named concepts against 87 unknown ones the named share may be capacity-bounded, so the named/unknown ratio is itself a knob. INTERVENTIONS EXIT 0 03:13 UTC (37 shards, 87,058,459 predictions; JSON banked in research_journal/figure_data/vm1/full_run_DEC_v12/): none 34.42 / loss 3.8312; truth 34.35 / 3.8436; flip 34.42 / 3.8270; flip_gated 34.41; truth_cal 34.29 / 3.8576; flip_cal 34.04 / 3.8632; random 34.39; zero_known 33.37 (costs 3.0%), zero_unknown 17.76 (costs 48.4%, loss 7.72), zero_residual 28.15 (costs 18.2%). Reading: on MIMIC the corrected decomposition keeps the prediction in the unknown channel (48%) with the residual still load-bearing (18%) and the named channel at 3% (mixture R2: named 93.4%, residual-slot 1.6%); capability 34.42 vs R2's 37.25 on the same harness, a 2.8-point top-1 cost, the opposite of eICU where v12 beats the mixture by 2.9. Trust test flat: truth-flip -0.06 banded, +0.25 calibrated (both tiny); truth costs 0.012 nats. Matches the step-9,000 probe's shape (4.1% / 47.6% / 19.6%). Paper: tab:decomp rows + Sec 5.2 rewrite landed 03:30 UTC. CHAIN FINISHED 05:23 UTC (alerts, report, intervention CIs all EXIT 0; alerts.json, intervention_cis.json, inference_results.json banked). Hazard AUROC vs the mixture flagship R2 on the same 37 shards, n identical in all 15 cells: v12 better in 2 of 15 (ICU@72h +0.003, vasopressor@72h +0.001), mean delta -0.0028; the largest losses are death@72h -0.007 and AKI@72h -0.006. So on MIMIC the decomposition costs about a quarter of a point of AUROC on average (cf. global pairs -0.007 mean, additive not run on MIMIC). Paired CIs (36,462 subjects, 2,000 boots): truth-flip -0.07 pt [-0.08, -0.06] separated, WRONG sign banded; truth-none -0.06 [-0.07, -0.06]; flip-none 0.00 not separated; calibrated truth-flip +0.25 [+0.23, +0.27] correct sign; zero_known -1.04 [-1.07, -1.02]; zero_residual -6.26 [-6.30, -6.23]. The MIMIC steering mid-training launched 05:24 UTC. | | eicu_full_DEC_v12 | eicu | eICU v2 all 134 train shards | 69c8cb0 | eICU counterpart, ~/dec_eicu_exact.json unchanged | launched 2026-09-01 19:42 UTC, sharing the card with the v11 chain's interventions stage for its first ~1-2h (throughput will read low until that finishes); chain waiter (ALERT_SHARDS=17) armed, ~/dec_eicu_v12_waiter.log | TRAINING DONE 2026-09-02 00:25 UTC: 28,000 steps in 15,208s (~4.2h, sharing the card with the v11 chain), early_stop=True, best_val 2.0963 at step 24,250 (v11: 2.3337 at 8,250, stopped at 12,000; R6 mixture flagship 2.0148; additive 1.9758). Trained 2.3x longer than v11 and to within 0.08 nats of the mixture flagship. Chain waiter fired 00:26 on checkpoint_best.pt at checkout 123970b; eval stage running beside the v11 chain's alerts stage. MIMIC v12 stopped at 26,500, so the two v12 arms are matched on training length to within 6%. INTERVENTIONS EXIT 0 03:01 UTC (17 shards, 86,897,282 predictions; JSON banked in research_journal/figure_data/vm2/eicu_full_DEC_v12/): none 56.92 / loss 1.8889; truth 56.93 / 1.9008; flip 56.83 / 1.8932; flip_gated 56.92; truth_cal 56.82 / 1.9106; flip_cal 57.01 / 1.9444; random 56.89; zero_known 53.22 (costs 6.5%), zero_unknown 0.15 (costs 99.7%, loss 17.4), zero_residual 55.52 (costs 2.5%). Reading: the corrected decomposition on eICU is the most capable arm on this split (56.92 vs additive 56.80, mixture R6 54.01, v11 53.59); the prediction lives in the unknown channel (99.7%), the named channel carries 6.5% (v11 5.0%, additive 1.6%, mixture 48.5%) and the residual 2.5% (v11 0.9%). The trust test is flat as always: truth-flip +0.11 pt banded (correct sign, tiny), -0.19 calibrated (wrong sign); truth costs 0.012 nats. The dial benchmark (registry row above) is where this arm shows its lever. CHAIN FINISHED 05:01 UTC (alerts EXIT 0, report EXIT 0, intervention CIs EXIT 0; alerts.json, intervention_cis.json, inference_results.json banked). Hazard AUROC vs the mixture flagship R6 on the same 17 shards, cell-for-cell identical n: v12 better in 5 of 12, mean delta -0.001; death +0.021/+0.026/+0.026 at 8/24/72h, vasopressor +0.004/+0.002/-0.005, AKI -0.011/-0.014/-0.013, ICU -0.012/-0.015/-0.019. So the corrected decomposition costs nothing overall on eICU alerts and buys mortality. (The chain's own GBM refit differs from R6's by up to 0.03 on ICU@72h, 0.906 vs 0.937: GBM refit noise on the degenerate ICU cells; compare hazard columns, not the GBMs.) Paired CIs (16,635 subjects, 2,000 boots): truth-flip +0.11 pt [+0.08, +0.13] separated, correct sign; truth-none +0.01 [-0.00, +0.02] NOT separated; flip-none -0.10 [-0.11, -0.08]; calibrated truth-flip -0.19 [-0.25, -0.13] wrong sign; zero_known -3.70 [-3.80, -3.61]; zero_residual -1.41 [-1.47, -1.35]. The eICU steering mid-training launched 05:02 UTC from this arm's checkpoint_best. | | gemini_full_v10 (G4 alerts leg) | gemini (Amrit / odyssey-6b lane) | GEMINI MEDS, 895 train shards; scored on GEMINI_ALERT_SHARDS=4 of the ~112 held-out shards (112,000 held-out subjects per finalize_summary.json) | main 52aeb97 (JSON), PR #219 logging | hazard heads vs tuned strong 609-feature GBM on GEMINI, protocol v4; the baseline column is what this run banks (the hazard column is the pre-decomposition mixture architecture) | done; JSON at scripts/gemini/out/evals/gemini_full_v10_alerts.json (42 records) | Hazard beats GBM in 10 of 12 cells (loses AKI@8h 0.868 vs 0.885 and AKI@24h 0.819 vs 0.834), e.g. death@8h 0.974 vs 0.911, ICU@8h 0.960 vs 0.920. GBM starvation ruled out by measurement (odyssey-6b, 2026-09-01 ~21:20 UTC, PR #219 logging re-run): every one of the 12 (event, horizon) fits was BOUND at GBM_FIT_MAX_ROWS = 1,000,000; eligible rows ranged 1,141,809 (ICU@72h, the smallest margin, 14% over the cap) to 2,028,791 (death@8h). The 30-of-895-shard fit therefore gave the GBM its full 1M-row budget everywhere; more shards would not have changed the fit. Re-run spot-check: death@72h GBM 0.895 vs the filed 0.897, ordinary HistGradientBoosting refit noise; the original JSON stays canonical. Not yet paper-grade under the full-data rule: 4 of ~112 held-out shards and no paired CIs. Requested from the GEMINI lane: rerun run.sh alerts gemini_full_v10 with GEMINI_ALERT_SHARDS set to the whole held-out set plus alerts_cis.py --scorers hazard baseline_gbm on the dump; the GBM column then goes into a tab:gemini comparator table, the hazard column waits for a GEMINI retrain on the fixed decomposition (69c8cb0). | | full_run_DEC_v12 (dial controls: random / unnamed direction; hazard heads under zeroing) | cbm | MIMIC, all 37 held-out shards, the paper's ten dials, 16 lanes | PR #231 (merged 2026-09-03; VMs run 688e275 via ~/odyssey_ctrl) | Reviewer C2/C5 (odyssey-2e, Amrit agreed 2026-09-03): does a direction carrying no concept pass the declared pairs as the dials do (severity axis)? and what do the hazard heads retain with the named / unnamed / residual channel zeroed? | queued 14:36 UTC by ~/controls_mimic.sh (waits for tfm fixup done); outputs ~/runs/full_run_DEC_v12/steering_control_{random,unknown}.json, alerts_zero_{known,unknown,residual}.json; marker controls mimic done in ~/controls_mimic.log | Same gamma calibration rule on the control direction (tau/peak). Compare declared-pair counts with steering_full.json; compare zeroed hazard AUROC with alerts.json on identical rows. ~/odyssey_ctrl advanced to c10d8d3 (PR #233 partial writes) before these start, so each dial run leaves a .partial.json. | | eicu_full_DEC_v12 (dial controls + hazard-head zeroing) | eicu | eICU, all 17 held-out shards, the ten salience dials, 32 lanes | PR #231 (merged 2026-09-03; VMs run 688e275 via ~/odyssey_ctrl) | same, on the model behind the paper's eICU dials | queued 14:37 UTC by ~/controls_then_steer2_eicu.sh (waits for post-v13 queue done; replaces the steer2 launcher PID 50386, killed by PID; hands over to ~/steer2_v13_queue.sh afterwards so steer2 and ctrl still run, sequentially) | outputs under ~/runs/eicu_full_DEC_v12/; marker controls eicu done in ~/controls_eicu.log. ~/odyssey_ctrl advanced to c10d8d3 (PR #233 partial writes) before these start, so each dial run leaves a .partial.json. | | full_run_DEC_v12 (dial dose-response, tau 4 and 8) | cbm | MIMIC, all 37 held-out shards, the ten dials, 16 lanes | main acdd02e (= 688e275 tree) via ~/odyssey_ctrl | Reviewer C3 + the Steerling code comparison (their released presets push at 8 logit units; ours at 1; the paper fixes no tau): how do the dials change with strength? | queued 14:43 UTC by ~/dose_mimic.sh (waits for controls mimic done); outputs ~/runs/full_run_DEC_v12/steering_full_tau{4,8}.json; marker dose mimic done | -- ~/odyssey_ctrl advanced to c10d8d3 (PR #233 partial writes) before these start, so each dial run leaves a .partial.json. | | eicu_full_DEC_v12 (dial dose-response, tau 4 and 8) | eicu | eICU, all 17 held-out shards, the ten salience dials, 32 lanes | same | same | queued 14:44 UTC inside ~/controls_dose_then_steer2_eicu.sh (controls -> dose-response -> steer2 -> ctrl, sequential; replaces the previous launcher, PID 64279 killed by PID); outputs ~/runs/eicu_full_DEC_v12/steering_full_tau{4,8}.json; markers dose eicu done, then the steer2 queue's own | -- ~/odyssey_ctrl advanced to c10d8d3 (PR #233 partial writes) before these start, so each dial run leaves a .partial.json. | | full_run_v10 (GBM feature-group ablation, full held-out) | cbm | MIMIC, the paper's own alert rows (~/runs/full_run_v10/alerts_rows.parquet, all 37 held-out shards), GBM refit on 30 train shards | PR #229 (merged 2026-09-03; VMs ran e907f87 via ~/odyssey_ablate) | Amrit 2026-09-03: 'do we discuss why our model falls behind GBMs?' -> no, the August answer was subset-scale and left under the full-data rule; this measures it on the flagship the comparator tables report: strong panel split into static(3)/recency(64)/latest_value(48)/summary_stats(384)/counts_occurrence(110), each dropped and each kept alone, tuned params held per cell, scored on the dump's rows vs the hazard head | launched 2026-09-03 02:00 UTC (nice 10, CPU, alongside the transformer training); log ~/ablate_mimic_v10.log, output ~/runs/full_run_v10/gbm_feature_ablation.json | Feeds a 'why' paragraph in 5.4 and one Discussion sentence; Amrit's framing: the paper is about steering and CBMs, a better forecaster is future work, so this explains the gap rather than closing it. Relaunched 03:35 UTC at e907f87 on ALL 292 train shards, row fraction 0.3 (the 02:00 run used 30 shards; the paper's legs fitted on every shard, and on eICU a 30-shard refit gave AKI@8h 0.763 vs the paper's 0.891, so the first attempt was killed by PID on both VMs; its log kept as ~/ablate_*_v10_30shards.log). 05:02 UTC: matrix 2,905,390 rows x 609 (all 292 shards, 30%), held-out 1,214,849 rows/event (all 37 shards). Reproduction check PASSES: full refit AKI@8h 0.913 = Table 5's GBM 0.913. ~2.3 min/fit, 165 fits -> ~11:00 UTC. RAM 62/83 GB with the transformer training; watching. DONE ~11:05 UTC, banked vm1/full_run_v10/gbm_feature_ablation.json (15 cells x 11 refits, 2.9M train rows from all 292 shards, all 37 held-out shards; hazard AUROC reproduces alerts.json exactly in every cell, full GBM within 0.01). On the flagship the GBM's advantage is the occurrence-count group in every event: unique share of the GBM-minus-hazard gap from dropping counts_occurrence = AKI 0.80/0.66/0.58 (8/24/72h), ICU admission 1.81/1.35/1.02 (dropping counts puts the GBM BELOW the head), vasopressor 6.07/1.71 at 24/72h (8h: head already wins), sepsis3 0.55/0.49/0.35, death 0.93/0.49 at 24/72h (8h: head wins by 0.028). Window summary_stats add 0.19-0.23 uniquely on AKI and nothing elsewhere; static adds 0.32/0.15 on death (age); recency and latest value add <0.15 anywhere. Kept alone, only counts recover a positive share (ICU 0.9, vasopressor 0.6); every other group alone falls below the head. Differs from the August subset-scale reading ('AKI = diffuse window stats'): at full scale AKI is counting too. Paper 11:15 UTC: 5.4 'Why the GBM wins' paragraph, Discussion sentence (counts; forecaster is future work), tab:ablation (tables/gbm_ablation.tex via scripts/make_ablation_table.py, uncommitted, eICU AKI omitted); 18 body lines trimmed elsewhere (dev-run readout sentence, ICU TabICL margin, readout-follows-push sentence, two Discussion sentences, input-level control's last sentence); the two TabICL tables merged into one side-by-side float. 22 pages, pagecheck 0, no overfull. | | eicu_full_v10 (GBM feature-group ablation, full held-out) | eicu | eICU, ~/runs/eicu_full_v10/alerts_rows.parquet (all 17 held-out shards), GBM refit on 30 train shards | PR #229 (merged 2026-09-03; VMs ran e907f87 via ~/odyssey_ablate) | same, on the eICU flagship behind Table 6 | launched 2026-09-03 02:03 UTC; log ~/ablate_eicu_v10.log, output ~/runs/eicu_full_v10/gbm_feature_ablation.json | -- Relaunched 03:36 UTC at e907f87 on ALL 134 train shards, row fraction 0.5. 05:02 UTC: matrix 2,587,275 rows (all 134 shards, 50%), held-out 655,415 rows/event. Reproduction check FAILS on AKI: full refit 0.768 vs Table 6's 0.891 even on every shard. Hypothesis: PR #222 made KDIGO's urine leg resolve on eICU, so the current code labels the TRAIN side with the fuller AKI rule while the dump's held-out AKI rows are the paper's creatinine-only label (the same change that moved v13's AKI row set). Death/vasopressor/ICU rules did not change; if their full refits reproduce (0.942/0.909/0.949), keep those and mark eICU AKI non-comparable, using MIMIC for the AKI story. Decide at ~06:10 UTC when the death cells land. 06:51 UTC: hypothesis CONFIRMED: death@8h full refit 0.942 and @24h 0.935 = Table 6's GBM exactly; only AKI fails to reproduce (train labels now carry the urine leg, the dump's held-out AKI rows do not). Keep vasopressor/death/ICU cells; report eICU AKI as non-comparable and use MIMIC for AKI. DONE ~09:15 UTC, banked vm2/eicu_full_v10/gbm_feature_ablation.json (12 cells x 11 refits, 2.59M train rows, all 17 held-out shards). Comparable cells: vasopressor gap 0.031/0.053/0.063 (8/24/72h): counts_occurrence unique share 0.63/0.58/0.63, keep-alone recovers 0.51/0.65/0.73, no other group above 0.06 -> COUNTING, as on MIMIC in August. Death gap 0.040/0.078/0.105: no group unique above 0.38 (counts 0.38/0.24/0.26), keep-alone summary_stats 0.32/0.60/0.62 and latest_value 0.17/0.47/0.50 -> diffuse window/latest-value signal. ICU admission (degenerate) gap 0.125/0.152/0.190: the STATIC group (age, sex, hours since origin) alone explains 0.58/0.69/0.82 uniquely and 0.60/0.73/0.87 kept alone: the stay-is-the-visit leak, which is why those cells are excluded. AKI: non-comparable (train labels carry the urine leg), full refit 0.768/0.714/0.661 vs table 0.891/0.836/0.784; at 24/72h the hazard head even beats the refit. | | gemini_full_DEC_v12 (decomposed bottleneck, full data) | gemini H200 (Amrit's lane) | GEMINI MEDS, 894/895 train shards, 2 epochs | main at 253e0f2 or later (started after PR #221's lab mapping; includes #217's decomposition; #225's rename is positional and does not affect the checkpoint) | the GEMINI counterpart of full_run_DEC_v12 / eicu_full_DEC_v13: same decomposition recipe, 25 of 29 concepts resolving | training done 2026-09-02 23:51 UTC (Amrit's log): 73,955 steps in 33,298 s (9.2 h), early_stop=False (full 2 epochs), best_val 2.5142; output /mnt/nfs/home/krishnanam/runs/gemini_full_DEC_v12 | Next (Amrit runs, in order): eval-forecast on EVERY held-out shard (GEMINI_EVAL_MAX_SHARDS set to the full count, not the 20-shard default), alerts (GEMINI_ALERT_SHARDS as high as the node's RAM allows; state the count), tabicl on the same shards, steering (all held-out shards by default), atlas; then the exported JSONs under scripts/gemini/out/evals/ come back by git. eval-forecast ran on all 112 held-out shards (Amrit, 2026-09-03 00:45 UTC; export blocked by the whitelist until PR #228, backfilled on the next step). Alerts/TabICL to run with GEMINI_ALERT_SHARDS=1000 (= all 112) and GEMINI_BASELINE_SHARDS=100. 2026-09-03 ~12:00 UTC: alerts leg (all 112 held-out shards, GBM on 100 train shards) finished on the node; mirror synced by odyssey-6b to 8de092b (PR #228 whitelist), the apparent 81-commit divergence was re-filtered identical history; the eval and alerts exports backfill on the next run.sh step (steering, which Amrit runs next). 15:10 UTC: run.sh steering gemini_full_DEC_v12 running on the node since ~08:00 local (Amrit), ~36 min per push, ~50 pushes -> ~30 h; Amrit lets it finish; first pushes: tachycardia up k_c 0.43->0.64 but outcomes x1.00 and death x0.97 (wrong way), i.e. the push reaches the readout but not the hazards. The JSON is written only at the end on that code; PR #233 adds per-dial partial writes for future runs; scratchpad/parse_steering_log.py recovers 24h results from the log if a run is stopped. | | gemini_full_DEC_v12 (alerts leg, full held-out) | gemini H200 (Amrit) | GEMINI MEDS, ALL 112 held-out shards (5.4-7.6M at-risk landmark rows per cell), GBM fit on 100 train shards (strong 609-feature panel) | mirror 8de092b (= main with #228) | the GEMINI hazard-vs-GBM comparison for the paper (item 2) | done 2026-09-03 ~12:00 UTC; export committed by Amrit (0edf46c) and banked research_journal/figure_data/gemini/gemini_full_DEC_v12/alerts.json | The hazard heads beat the tuned GBM in 9 of 12 cells: death 0.976/0.968/0.958 vs 0.924/0.926/0.916, ICU admission 0.961/0.947/0.920 vs 0.928/0.914/0.881, vasopressor 0.948/0.927/0.896 vs 0.901/0.885/0.853 (8/24/72h); AKI loses 0.879/0.827/0.795 vs 0.898/0.847/0.806. Four events resolve (no sepsis3, no readmission cell). Point estimates only: alerts_cis on ~25M rows needs --max-subjects on the node. Paper 13:10 UTC: tab:gemini in App D, 5.5 renamed 'and GEMINI' with the 9-of-12 sentence, Limitations updated; abstract unchanged until CIs. Also: Results subsections now carry (Q1)-(Q4); appendix repacked (TabICL merged float, eICU dial tables side by side, full-dial figures merged). 22 pages, pagecheck 0. 14:00 UTC: odyssey-2e reviewer pass applied (23 mechanical items; see main.tex header log); Amrit: Limitations no longer says 'GEMINI pending' or 'no clinician has read' (GEMINI clinician co-authors will review; now 'no prospective clinician study'). tab:counterfactual added (full_run_v8, 300 subjects). 23 pages, pagecheck 0, no overfull. Fairness check 2026-09-03 15:10 UTC (Amrit): the MIMIC flagship's GBM was fitted on every train shard (the all-shard refit reproduces Table 5's GBM to the third decimal in every AKI/ICU cell; a 30-shard refit gives 0.908/0.868/0.812 vs 0.913/0.878/0.828), eICU's on all 134 (vm_eicu_full.sh BASELINE_SHARDS=134), GEMINI's on 100 of 894 (memory-bound: the streaming fit materializes every landmark row before the 1M cap). GEMINI is the odd one out. PR #232 plumbs --baseline-row-fraction (seeded per-shard thinning from #229) into the alerts fit and defaults the GEMINI step to every shard at 10%; rerun with GEMINI_ALERT_SHARDS=1000 GEMINI_ALERTS_TAG=_allshards scripts/gemini/run.sh alerts gemini_full_DEC_v12 after merge + mirror sync. Until then tab:gemini says 11%. 15:05 UTC: GEMINI mirror synced by a haiku subagent on Amrit's instruction: gemini/main 497cb5d -> 820bbcb, one commit on top of the mirror whose tree = GitHub main 2f3a657 (#229-#232) with scripts/gemini/out/ kept from the mirror; verified by diff on both sides before the push. | | full_run_DEC_v12 (steering benchmark smokes) | cbm | MIMIC held-out 1 shard (986 subjects), lifted sets from 4 train shards | b1366af / 0450505 (odyssey/inference/steering.py) | The lever measured the way Steerling measures it and a clinician reads it: push one concept's unit direction e_c = K_c/‖K_c‖ (their 6.2.1) with gamma = tau/max_y e_c·W_y (Eq. 19, tau = 1) at every block from the middle on (Eq. 18), amplify and suppress (ReLU-gated mask, Eq. 21), and read respond (k_c), express (mass on lifted tokens) and each hazard head's risk at 8/24/72h against CLINICAL_EXPECTATIONS, paired by subject. Second run restricts risk to at-risk positions and names tokens from MEDS codes.parquet. | done 01:34 and ~02:00 UTC (stream site, v1 and corrected v2), bottleneck site running | Every dial responds (k_c: shock 0.21→0.30/0.14, hypotension 0.21→0.27/0.16, lactate 0.48→0.56/0.41; paired CIs tight). Outcome directions differ by dial: hypotension right on all four outcomes in both directions (up: vasopressor ×1.10, ICU ×1.12, AKI ×1.10, death ×1.05; down: ×0.93/0.91/0.92/0.97; 91–98% of patients); aki_stage_3 and sepsis3 100% agreement both ways; shock half right (ICU, AKI right; vasopressor-start and death wrong, unchanged after the at-risk fix: ×0.94 and ×0.83 under amplify); elevated_lactate inverted on every outcome both ways. Bottleneck-site push (k_c unchanged by construction) reproduces the same signs (hypotension up: vasopressor ×1.20, ICU ×1.23; shock up: vasopressor ×0.78), so the sign pattern is how the heads read K_c, not the recurrence. This is Steerling's pre-steering-training picture (a third of their concepts never activated; ours all activate but a subset is mis-signed). Lifted sets: v1 picked rare chart items (pressure-ulcer measurements); v2 with a 0.5% share floor picks readable but generic ICU care-plan tokens shared across concepts; a lift ≥ 2 filter was added in 5328814. JSONs (stream v1, bottleneck) in research_journal/figure_data/vm1/full_run_DEC_v12/. Note for attribution claims: the per-event hazard heads are 2-layer GELU MLPs (event_head_hidden=256), so the exact concept decomposition of a logit holds only for the linear next-event head; hazard-side effects of a dial are measured by the benchmark, not read off weights (an attempt to read K_c·W for the hazard heads failed for exactly this reason). | | full_run_DEC_v12_steer (planned) | cbm | MIMIC all 292 train shards, 1 epoch | 3de2487 (odyssey/training/steering_phase.py, compute_steering_loss) | Steerling's steering training (10.2.4, Table 36) as a mid-training run from v12's checkpoint_best: 4 phases × 250 steps after a 100-step warmup, gamma 1.0 at the middle block, respond + express (weights 1, 1) on top of the forecasting loss with the interpretability losses off during phases; alpha_known 0.5→0 and alpha_unknown 0 per the 10.2.1 prose (Table 27 disagrees); lr 1e-4 constant (their decay-to-zero not implemented); lifted sets over 2,000 train patients. Config ~/dec_mimic_v12_steer.json. | ARMED 01:50 UTC: ~/steer_launcher.sh waits for the v12 chain's "queue done" (the card cannot hold both), launches training, arms the chain waiter (ALERT_SHARDS=37), then reruns the 5-concept, 1-shard stream-site benchmark on the result so it is matched to the pre-phase smoke. | The question: after the phases, do shock and elevated_lactate come right and does hypotension stay right, at no capability cost (val loss vs v12's 2.1925)? | | full_run_DEC_v12 (concept atlas) | cbm | MIMIC held-out 1 shard (2,349,001 positions) | 96af272 (scripts/concept_atlas.py) | Exact alignment of every concept direction with the next-event vocabulary (the LM head is linear, so W·K_c is exactly what one unit of the concept promotes: Steerling's Known Concept Alignment) plus their Concept Contribution shares on the linear head | done 02:45 UTC; JSON in research_journal/figure_data/vm1/full_run_DEC_v12/concept_atlas.json | Contribution shares of each next-event logit: named 5.2%, unknown 78.0%, residual 16.8% (exact, linear head, one shard). The named directions are small (‖K_c‖ 0.55–1.23) against the unknown ones (‖U_j‖ ≈ 7.3–8.1): that is the share gap in weights. Known Concept Alignment is partly sensible (aki_stage_3 → creatinine Q5, dialysis catheter site, lactic acid Q5; anemia → hemoglobin Q1/Q2; hyperkalemia → stop calcium acetate / nephrocaps) and partly nursing-flowsheet noise (shock → arterial-line dressing, restraint location; hypotension → Braden activity). The 87 unknown concepts are dense, not sparse: mean activations 0.38–0.57 for all of them (Steerling mid-trains with top-32/top-128 sparsification, which we do not), and each promotes a mixed but often coherent event set: unknown_59 lipid + immunology panel, unknown_17 arterial blood gas (base excess, TCO2), unknown_53/54 urinalysis, unknown_86 infection care plan + urine tox screen, unknown_0 dialysis + eGFR. Read: the unnamed channel is where the prediction lives and it is organized by lab panel and care pathway rather than by named physiology; a top-k sparsification phase is the obvious next lever for interpretability of the unknowns. | | eicu_full_DEC_v12 (steering benchmark smoke) | eicu | eICU held-out 1 shard (979 subjects), lifted sets from 4 train shards | dc320ea (odyssey/inference/steering.py, stream site, at-risk restricted, tau 1) | Same five-dial clinical what-if as the MIMIC smoke, on the eICU v12 checkpoint: shock, hypotension, aki_stage_3, elevated_lactate, metabolic_acidosis, amplify and suppress | done 2026-09-02 ~02:50 UTC; JSON in research_journal/figure_data/vm2/eicu_full_DEC_v12/steering_smoke.json | Every dial moves every declared outcome the way a clinician expects, in both directions: 10 of 10 dial settings at 100% sign agreement, 84–100% of patients per cell, every paired interval separated. 24h risk ratios pushed up / down: shock vasopressor ×1.10/×0.91, AKI ×1.05/×0.96, death ×1.18/×0.85; hypotension vasopressor ×1.08/×0.93, death ×1.12/×0.90; aki_stage_3 AKI ×1.16/×0.86, death ×1.18/×0.84; elevated_lactate vasopressor ×1.06/×0.94, death ×1.12/×0.90 (its own activation does not move, 0.57→0.57, yet the outcomes do); metabolic_acidosis vasopressor ×1.12/×0.89, AKI ×1.13/×0.89, death ×1.21/×0.83. ICU admission barely moves on eICU (×1.00–1.02), consistent with its degenerate status there. Contrast with MIMIC v12 on the same protocol, where shock and elevated_lactate were mis-signed: which dials the heads read correctly is dataset-dependent before steering training. This is a 1-shard smoke; the paper-grade run on all 17 held-out shards follows. | | eicu_full_DEC_v12_steer | eicu | eICU v2 all 134 train shards, 1 epoch from eicu_full_DEC_v12/checkpoint_best | f8633b3 | Steerling steering mid-training (4 x 250 phase steps after 100 warmup, gamma 1.0 at block 4, respond+express weights 1/1, interpretability losses off in phases, alpha_known 0.5->0, alpha_unknown 0, lr 1e-4) | launched 05:02 UTC; at 06:00 step 1,725, best val 2.0705 (already below v12's 2.0963) | The express loss was inert in this run: with lifted_patients 2,000, min_share 0.5% and min_lift 2, 22 of 26 concepts had EMPTY lifted sets and the rest 1-4 tautological vitals bins (fever -> TEMPERATURE::HIGH), so express_loss logged 0.0000 on every phase step. Respond ran (0.066 -> 0.010 over the phases, n_injected 2.5k-25k per step). Treat this arm as RESPOND-ONLY steering training. A second run (eicu_full_DEC_v12_steer2, lifted_patients 10,000, min_share 0.2%, min_lift 1.5) is armed to start when this one's chain finishes. Its after-run dial benchmark (ten dials, all 17 shards, 32 lanes) is chained by ~/steer_launcher_eicu.sh. TRAINING DONE 09:17 UTC: 20,250 steps in 11,672s, early_stop=True, best_val 1.9786 at step 16,500 (v12 parent 2.0963, mixture R6 2.0148, additive 1.9758): the mid-training from a good checkpoint at lr 1e-4 keeps improving capability. Chain started 09:20 UTC at checkout d7c96ab (eval EXIT 0, interventions running). | | eicu_full_DEC_v12 (alert CI leg) | eicu | eICU v2, every held-out shard (the same rows as its alerts.json) | 789c33f (main) | paired hazard-vs-GBM CIs for the DECOMPOSED eICU model, so the comparator table can bold on a paired delta | done 2026-09-02 12:39 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v12/alerts_cis.json | scripts/alerts_cis.py --scorers hazard gbm, 1000 boots. Every one of the 12 cells is GBM-favoured and separated: hazard-minus-GBM AUROC from -0.020 [-0.027, -0.013] (death@8h) to -0.173 [-0.194, -0.149] (ICU admission@72h); AKI and ICU admission cost 0.11-0.17, death and vasopressor start 0.02-0.06. | | eicu_full_DEC_v12_steer (interventions, full split) | eicu | eICU v2, every held-out shard (86.9M predictions, same rows as eicu_full_DEC_v12) | 789c33f (main) | does one epoch of Steerling steering training (respond-only; the express loss was inert, empty lifted sets) change the truth/flip test or the channel probes? | done 2026-09-02 13:03 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v12_steer/interventions_band15.json | none 55.48 (decomposed 56.92, so the epoch costs 1.4 pt); truth-flip +0.10 pt banded (decomposed +0.10), calibrated truth-flip +0.57 (decomposed -0.19); zero_known retains 96.2% (decomposed 93.5%), zero_unknown 0.5% (0.3%), zero_residual 97.6% (97.5%). Steering training moves nothing on the trust test; its effect, if any, is in the dial benchmark (after-run dials still running). | | full_run_DEC_v12_tfm (vanilla transformer backbone) | cbm | MIMIC all 292 train shards, full held-out eval | main (ca541d8+) via /odyssey_main worktree | the pure-attention control the paper's comparator story needs (ours vs a vanilla transformer vs TabICL, MIMIC only, Amrit 2026-09-02): full_run_DEC_v12's exact config with backbone=transformer, num_lanes=16, max_context=2048 (same 32k tokens per step as the hybrid's 64x512) | queued 2026-09-02 14:17 UTC by ~/tfm_after_steer.sh (waits for queue done in ~/dec_mimic_v12_steer_waiter.log, i.e. after both MIMIC dial runs and the deferred steer chain), then eval chain; logs ~/dec_mimic_v12_tfm_{train,eval}.log, waiter ~/dec_mimic_v12_tfm_waiter.log | runs from a separate main-branch worktree so the running dial processes' checkout is untouched. Alerts leg streams through PackedContextSampler (transformer-aware); the other eval stages use lane chunking, which truncates a stateless model's context to 512 tokens, so quote its forecasting metrics only after checking that path. TRAINING 2026-09-03 01:37 UTC (/odyssey_main 585819b; data prep to 02:11, first step 02:11): 16.9M parameters, ~1.8 steps/s at 16 lanes (400 steps by 02:17), 19 GB GPU. At 16 lanes an epoch is ~4x v12's 24,845 steps, so 2 epochs ~200k steps ~ 30 h unless early stopping fires; expect the eval chain Sep 4 evening UTC. 05:02 UTC: step 11,525, already epoch 1 (an epoch is ~11k steps at 16 lanes, not the 4x v12 I assumed), best val 2.1591 at ~step 11,000 (v12 Mamba parent 2.1925), early-stop counter 2/15; expect training to end ~08:00 UTC, chain after. 06:04 UTC: step 15,475, best val 2.0683 at step 15,250 (already below the Mamba v12 parent's 2.1925 and the MIMIC steer run's 2.0607), early-stop 4/15 before that best. TRAINING DONE 06:29 UTC: 17,165 steps in 15,489 s (4.3 h), 2 full epochs, early_stop=False, best val 2.0548 at step 17,000 (Mamba v12 parent 2.1925; MIMIC steer run 2.0607). Eval chain next (all 37 held-out shards), then CIs and the ten-dial benchmark. Chain problems 06:33 UTC: the 64-lane eval stage OOMed on the A100 (transformer attention at 64x512; training used 16) and the interventions stage hit the tool's explicit backbone='transformer' gate (NotImplementedError). Alerts stage (packed-context path) runs fine. Fix: PR #230 (ed7729b) lifts the gate (the TBTT stream is the transformer's own training view: state ignored, chunk = context); ~/tfm_fixup.sh (armed 06:56 UTC, replaces tfm_post_chain.sh, PID 259745 killed) waits for the launcher's chain, then from ~/odyssey_tfmfix reruns eval and interventions at 16 lanes, intervention CIs, and the ten-dial benchmark; marker tfm fixup done in ~/tfm_fixup.log. The 'cases' stage will also fail on its transformer gate; not needed. Eval16 + alerts banked 09:30 UTC (vm1/full_run_DEC_v12_tfm/{inference_results,alerts}.json). CAVEATS: (a) the transformer's PackedContextSampler (max_context 2048) truncates long histories, so it scored 37,291 patient ends vs the Mamba flagship's 55,328 and its alert row sets differ from the flagship's in every cell (no paired comparison from these files); (b) exact-token top-1 20.6% / CE 9.21 vs Mamba 34.4% / 3.83, while bundle-level set top-1 is 78.7% vs 80.4% and category-set top-1 54.3% vs 48.9%: under the bundle-invariant objective the transformer spreads mass inside a bundle, so exact-token metrics are not the capability comparison for this arm; its interventions 'none' mode (TBTT view, all positions) gives the same 20.7%/8.30. Readout mean 0.952 on its (truncated) ends vs 0.918 (tachycardia/bradycardia/hypoxia +14 to +17 points). Hazard AUROC on its own rows: death 0.986/0.978/0.965, sepsis3 0.930/0.911/0.885, ICU 0.938/0.907/0.868, AKI 0.869/0.813/0.754, vasopressor 0.892/0.846/0.792 (GBM on the same rows 0.875/0.846/0.792). The portability claim will rest on the training-view, all-position stages now running (interventions16 -> CIs, ten-dial benchmark), which pair with the flagship on identical rows. Interventions16 + CIs banked 12:10 UTC (vm1/full_run_DEC_v12_tfm/{interventions_band15,intervention_cis}.json; TBTT view, n_predictions 87,058,459 = the flagship's, so these PAIR). Label-override test has the same shape as the Mamba flagship: banded truth-flip -0.09 [-0.10,-0.09] (Mamba -0.07), truth-none -0.06 [-0.06,-0.06] (Mamba -0.06), calibrated truth-flip -0.12 (Mamba +0.25); truth never beats none. Completeness: zeroing retains named 98.1% / unnamed 68.8% / residual 83.1% of top-1 (Mamba 97.0 / 51.6 / 81.8): the decomposition's channel structure ports (named channel near-inert, unnamed carries the prediction). Oddity: zeroing the residual LOWERS the transformer's exact-token loss (8.30 -> 7.66) while cutting top-1; consistent with its diffuse within-bundle mass. Ten-dial benchmark running (started 11:59). Dials so far (12 of 20 pushes, 15:50 UTC): 47 of 49 declared pairs right vs Mamba's 39 of 49 on the same six dials; elevated lactate correct (Mamba inverted); sustained hypotension misses only vasopressor (Mamba also death); readout responds more (k_c +0.15-0.19 vs +0.06-0.09). Verdict framing agreed with Amrit: no superiority claimed either way; the hybrid is kept for its streaming property, the transformer shows the findings port. | | full_run_DEC_v12 (vasopressor-status stratified dials) | cbm | MIMIC, 8 held-out shards, 16 lanes | 7c8e116 (PR #226, feat/stratified-dials, checked out in ~/odyssey_main) | Amrit 2026-09-02: test the hypothesis that the two mis-signed MIMIC dials (sustained MAP hypotension, elevated lactate) depend on treatment under way: --stratify-by on_vasopressors splits every position by whether the patient is already on vasopressors; hypotension (SBP) runs alongside as the correctly-signed control | queued 2026-09-02 16:02 UTC by ~/strat_after_before_run.sh (waits for steering_full.json); output ~/runs/full_run_DEC_v12/steering_strat_vaso.json, log ~/steering_v12_strat_vaso.log | Prediction if the hypothesis holds: in the not-on-vasopressors stratum the dials move death/ICU the expected way and the wrong sign lives in the on-vasopressors stratum. Note: /odyssey_main on VM1 now sits at 7c8e116 (main + stratify), which the queued transformer arm will also use. | | full_run_DEC_v12 (clinical dial benchmark, FULL split) | cbm | MIMIC, all 37 held-out shards (36,462 subjects), lifted sets from 4 train shards | f8633b3 | the same ten dials as eICU (sepsis3 in place of sirs), stream site, tau=1, 16 lanes | done 2026-09-02 17:18 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12/steering_full.json | 8 of 10 dials move every declared 24h risk the expected way (46 of 46 pairs; 56 of 66 counting the two that miss). Misses: sustained MAP hypotension (still named shock in this JSON) lowers vasopressor initiation (x0.95) and death (x0.85) when pushed up and raises them when pushed down, while ICU admission and AKI move correctly; elevated lactate is inverted on every declared outcome in both directions (x0.94/x0.91/x0.94 up). Magnitudes match eICU (metabolic acidosis up: death x1.23, AKI x1.13; qSOFA up: death x1.32; Sepsis-3 up: death x1.28). At 8h 52/66, at 72h 56/66 pairs as expected. The one-shard smoke (steering_smoke_v2.json) was right about both misses. | | full_run_DEC_v12 (vasopressor-status stratified dials, RESULT) | cbm | MIMIC, 8 held-out shards (8,301 subjects), 16 lanes | 585819b | -- | done 2026-09-02 18:17 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12/steering_strat_vaso.json | Hypothesis NOT supported. In the not-on-vasopressors stratum (7,888 subjects) the sustained-MAP-hypotension dial still lowers vasopressor initiation (x0.95) and death (x0.86) when pushed up and raises them when pushed down, and elevated lactate is inverted on every outcome in both strata; the on-vasopressors stratum is small (413 subjects) and almost never at risk of death (1-3%). The control (SBP hypotension) is right on every outcome in the not-on-pressors stratum. Note the vasopressor outcome could never have been explained this way: the at-risk rule already excludes patients on pressors from it, so stratum 0's number equals the unstratified one by construction. | | eicu_full_DEC_v12_steer (clinical dial benchmark, FULL split, after steering training) | eicu | eICU v2, every held-out shard (16,635 subjects), same ten dials and settings as the untrained model | f8633b3 | does one epoch of respond-only steering training make the dials stronger? | done 2026-09-02 18:31 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v12_steer/steering_full.json | No: the dials get weaker. 36 of 40 declared non-ICU pairs still move the expected way (was 40), but the effects shrink toward 1 on almost every dial (metabolic acidosis up: death x1.23 -> x1.06, vasopressor x1.12 -> x1.00; shock up: death x1.19 -> x1.14), and elevated lactate (vasopressor) and anemia (death) flip to the wrong way. The readout responds MORE (baseline k_c rises, e.g. shock 0.32 -> 0.54, AKI stage 3 0.21 -> 0.31, and the push moves it further), so respond-only steering training taught the bottleneck to answer the push while the hazard heads' dependence on the direction weakened. Together with the chain row above (hazard AUROC up in all 12 cells, truth-none positive): the epoch improved capability and the label-override test, and traded away dial magnitude. | | full_run_DEC_v12_steer (clinical dial benchmark, FULL split, after steering training) | cbm | MIMIC, all 37 held-out shards (36,462 subjects), same ten dials and settings as full_run_DEC_v12 | f8633b3 | does one epoch of respond-only steering training change the MIMIC dials? | done 2026-09-02 20:50 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12_steer/steering_full.json | Same pattern as eICU: the dials get weaker. 53 of 66 declared 24h pairs right (was 56); the two mis-signed dials stay mis-signed (sustained MAP hypotension, elevated lactate) and SBP hypotension now flips on death (x1.04 -> x0.96 up, x0.98 -> x1.04 down). Death effects shrink on every correct dial: acidosis x1.23 -> x1.10, qSOFA x1.32 -> x1.18, Sepsis-3 x1.28 -> x1.17, AKI stage 3 x1.24 -> x1.11, hyperkalemia x1.21 -> x1.10, hypoxia x1.19 -> x1.09. Readout responds more (e.g. hyperkalemia k_c 0.35 -> 0.53 after vs 0.24 -> 0.36 before). Paper text held until steer2 (express-capable) lands, per Amrit. | | eicu_full_DEC_v13 (readout + forecasting, chain eval stage) | eicu | eICU v2, every held-out shard (20,122 patient ends) | ca541d8 (/odyssey_main) | first numbers of the 29-concept eICU retrain | training done 2026-09-02 20:43 UTC (35,500 steps), eval stage done 20:58; banked research_journal/figure_data/vm2/eicu_full_DEC_v13/inference_results.json; interventions/alerts stages running | Readout: 29 concepts, mean AUROC 0.877 (v12: 26 concepts, 0.882); the three new concepts read out at sepsis3 0.958, oliguria 0.829, hypoxemic respiratory failure 0.746; on the 26 shared concepts the mean is 0.881 vs 0.882, with aki_stage_3 -0.12 and hypokalemia -0.10 the largest drops and hypoglycemia +0.07 the largest gain (single seed each). Forecasting: top-1 55.54% (v12 56.92%), top-5 88.23% (87.18%), cross-entropy 1.874 (1.889). | | cross-database unnamed-slot atlas (paper App. B, tab:unknown-cross) | -- | one held-out shard per model | scripts/make_atlas_table.py --cross | Amrit 2026-09-02: the two most active unnamed slots per database side by side, with the 'states versus settings' reading (unnamed slots track the work-up/care setting, named concepts track physiological states) | MIMIC rows in (full_run_DEC_v12 atlas); eICU rows pending eicu_full_DEC_v13's concept_atlas.json (step 3 of ~/post_v13_queue.sh); GEMINI rows pending the runner's atlas step after train-full-dec | regenerate with --cross "MIMIC-IV=..." "eICU-CRD=..." "GEMINI=..." --cross-out paper/ml4h/tables/unknown_cross.tex --cross-top 2 --top-events 4, then resolve the [TBD-ATLAS] note in main.tex. | | eicu_full_DEC_v13 (decomposed, 29-concept registry) | eicu | eICU v2 all 134 train shards, full held-out eval | main (ca541d8) via ~/odyssey_main worktree | Amrit 2026-09-02: retrain eICU on the registry after PR #222 (SOFA support, 26->29 concepts) so every eICU forecaster/concept number in the paper comes from the same 29-concept model; needs the eICU Sepsis-3 sidecars (PR #223, built from raw microLab/medication, copied to ~/data/eicu_2.0_v2/sidecars/) | queued 2026-09-02 14:30 UTC by ~/retrain_v13_after_steer.sh (waits for queue done in ~/dec_eicu_v12_steer_waiter.log), then eval chain; logs ~/dec_eicu_v13_{train,eval}.log, waiter ~/dec_eicu_v13_waiter.log | eicu_full_DEC_v12's exact config. Sidecar pseudotimes verified against the MEDS medication-start events (median gap 0 min, 99.9% exact). After it: steering benchmark, atlas, alerts_cis; then the paper's eICU numbers move to v13. Interventions stage banked 2026-09-02 23:10 UTC (vm2/eicu_full_DEC_v13/interventions_band15.json, every held-out shard, same 86.9M predictions as v12). Modes at band 0.15 (top-1 %, loss): none 55.54/1.874, truth 55.38/1.886, flip 55.64/1.870, flip_gated 55.64/1.870, random 55.51/1.877, truth_calibrated 55.12/1.909, flip_calibrated 55.06/1.933; zero_known 52.06, zero_residual 55.17, zero_unknown 0.23. Banded truth-flip is -0.27 pt on v13 (was +0.11 on v12): the eICU decomposed arm's banded sign flips with the 29-concept retrain; calibrated truth-flip +0.06 (was -0.19). truth-none -0.16 (v12 +0.01). Intervention CIs launched separately (the retrain chain has no CI step): ~/eicu_full_DEC_v13_intervention_cis.log. The paper's 5.3 'truth-flip +0.11 on eICU' and tab:lever eICU decomposed rows must move to these once the CIs are banked. Intervention CIs banked 23:12 UTC (vm2/eicu_full_DEC_v13/intervention_cis.json, 2,000 paired subject-clustered resamples): banded truth-flip -0.27 [-0.30, -0.23] pt SEPARATED (v12: +0.11 [+0.10, +0.13]); truth-none -0.16 [-0.17, -0.15]; flip-none +0.11 [+0.10, +0.12]; flip_gated-none +0.11; random-none 0.00; calibrated truth-flip +0.06 [+0.03, +0.10] (v12 -0.19); calibrated truth-none -0.42, flip-none -0.48. Zeroing: known -3.5 pt (v12 -3.7), residual -0.35 pt (v12 -1.4), unknown -55.3 (v12 -56.8); residual-minus-known +3.1 pt (v12 +2.3). So on the 29-concept model the banded lever is wrong-signed and the calibrated one right-signed, the mirror image of v12; truth never beats none on either protocol. Chain complete 2026-09-03 00:58 UTC; alerts banked (vm2/eicu_full_DEC_v13/alerts.json, 17 held-out shards, 54 cells = 6 events x 3 horizons x 4 scorers; sepsis3 is new on eICU). Hazard AUROC vs v12 (8/24/72h): vasopressor 0.883/0.844/0.801 (v12 0.881/0.843/0.795), death 0.923/0.879/0.817 (0.922/0.883/0.820), ICU admission 0.833/0.802/0.756 (0.813/0.783/0.733), AKI 0.742/0.718/0.692 (0.730/0.703/0.664), sepsis3 0.822/0.818/0.788 (GBM 0.440/0.641/0.729 on 395 positives at 72h, so the tuned GBM loses sepsis3 outright). GBM unchanged on vasopressor/death/ICU (identical row sets). CAVEAT, AKI: the row set changed (at-risk 451,747 -> 308,700; positives 7,733 -> 18,637 at 8h) because PR #222's eICU SOFA config lets the KDIGO urine-output leg of acute_kidney_injury resolve (it was a dropped criterion on the 26-concept eICU), so v12 and v13 AKI cells are different labels, not a model change; GBM AKI 0.888 -> 0.813 at 8h for the same reason. The paper's eICU AKI numbers move to v13 with a note that the eICU AKI label now matches MIMIC's (creatinine + urine legs). Concept readout at 8h: AKI 0.690 (v12 0.567), vasopressor 0.747 (0.623): the 29-concept model's concept scorer is much stronger. Paper moved to v13 2026-09-03 01:25 UTC for the eICU readout (5.1, fig_readout, app:readout caption), completeness (5.2, tab:decomp), and label-override numbers (5.3, tab:lever; steer row kept from the 26-concept parent, said in the caption); Sec 3 now says the eICU model supervises all 29 and that the dial benchmark is from the 26-concept predecessor. Still v12 in the paper: tab:dials/fig:dials (until the v13 dials land) and fig_comparator_deltas + any decomposed-vs-GBM eICU alert sentence (until v13 alerts_cis lands). | | eicu_full_DEC_v13_steer2 (fixed steering recipe, from the v13 checkpoint) | eicu | eICU v2 all 134 train shards, 1 epoch from eicu_full_DEC_v13/checkpoint_best | 2ca65f4 (PR #227 merged; the queue checks out origin/main when it fires) | Amrit 2026-09-02: review against Steerling 10.2.4 found no equation-level bug but a structural one: injecting at running-label positions while scoring the forecast/hazard losses there on unchanged targets teaches invariance to the push. Fix: steering_forecast_at_injected: false (forecast, time, hazard, value losses scored off the injected positions; respond + express there), steering_tau: 1.0 (training pushed at fixed gamma 1 while the benchmark pushes at tau-calibrated 1.06-2.38), express-capable lifted sets (min_share 0.002, min_lift 1.5, 10,000 patients); otherwise the v12 steer recipe (lr 1e-4, 4 x 250 phase steps after 100 warmup, teacher known 0.5 -> 0) | done 2026-09-05 (results in the eval rows below); was queued 2026-09-02 23:01 UTC by ~/steer2_v13_queue.sh (waits for post-v13 queue done; replaces the v12-based steer2 launcher, PID 45684 killed); logs ~/dec_eicu_v13_steer2_{train,eval}.log, ~/steering_dec_eicu_v13_steer2_full.log; marker steer2 queue done in ~/steer2_v13_queue.log | Re-based on v13 (29 concepts, main code) because a v12 (26-concept) init no longer loads on main after PR #222. Paired against eicu_full_DEC_v13's own all-concept dial benchmark (post-v13 step 2) and the control below. Ping Amrit at start and finish. | | eicu_full_DEC_v13_ctrl (control epoch) | eicu | same as above | same checkout | one epoch from the same checkpoint with steering_phases: 0 and everything else identical to steer2: separates 'steering training' from 'one more epoch at lr 1e-4', the confound both earlier steer runs carried | done 2026-09-05 (results in the eval row below); was queued as step B of ~/steer2_v13_queue.sh; logs ~/dec_eicu_v13_ctrl_{train,eval}.log, ~/steering_dec_eicu_v13_ctrl_full.log; marker ctrl queue done | If the control's dials also shrink, the weakening is the epoch, not steering; if only steer2's strengthen, the fix worked. | | eicu_full_DEC_v13 (steering, bottleneck site, ten dials) | eicu | eICU all held-out shards; the ten paper dials x 2 directions; 40 joined non-ICU declared pairs at 24 h against the stream-site steering_full.json | c10d8d3 checkout, odyssey/inference/steering.py --site bottleneck via ~/site_bottleneck_eicu.sh | eICU counterpart of the MIMIC bottleneck-site test: does pushing k_c directly (readout cannot respond) reproduce the stream-site verdicts? | done 2026-09-06 21:10 UTC; banked steering_full_site_bottleneck.json under vm2/eicu_full_DEC_v13; VM2 stopped 21:25Z | Identical verdict on 40 of 40 joined pairs (36 as expected at both sites), larger effects at the bottleneck (median abs(ratio-1) 0.275 vs 0.119), every pair separated, respond_delta exactly 0 on every dial. Same picture as MIMIC (62 of 64, 0.16 vs 0.10). Reframed 2026-09-06 (reviewer pass 3, verified against concept_bottleneck.forward): at inference the decomposition is an identity (residual = h - known - unknown, frozen), so a bottleneck-site push adds step*K_c to the sum exactly as a stream-site push at the last block would; respond_delta = 0 is by construction (concept_probs are computed before the intervention branch), no control exists at this site (direction_override raises), and the step moves a sigmoid coordinate by 0.6-8.7. The result therefore does NOT show that 'the hazard heads read the named channel'; it shows the linear-embedding identity. Removed from the paper (5.3 bridge sentence deleted); both bottleneck JSONs stay banked. Four of the 44 stream-site pairs did not join (one dial's declared set differs at the older checkout) | | eicu_full_DEC_v12 (the 16 remaining dials) | eicu | eICU v2, every held-out shard, same settings as the ten (stream, 32 lanes, checkpoint_best) | ~/odyssey checkout (v12-era) | Amrit 2026-09-02: the paper discloses that only 10 of 26 eligible dials were run; this runs the other 16 (tachycardia, bradycardia, hypertension, fever, hypothermia, sustained_tachypnea, acute_kidney_injury, aki_stage_2, on_vasopressors, hypokalemia, hyponatremia, hypernatremia, hypoglycemia, hyperglycemia, thrombocytopenia, coagulopathy), including hypertension, the only opposite-sign expectation | queued 2026-09-02 15:46 UTC as step 1 of ~/post_v13_queue.sh (waits for v13 queue done); output ~/runs/eicu_full_DEC_v12/steering_full_remaining16.json, log ~/steering_eicu_v12_remaining16.log | Merge with steering_full.json for the paper's dial table/figure (both under the v12 model, old concept names; map via canonical_concept_name). Progress 02:17 UTC: 2 of 16 dials done (tachycardia, bradycardia) in 76 min including load, so 30 min per dial, finish 08:30 UTC; v13 steering/atlas/alerts_cis follow, steer2 after those. DONE 09:54 UTC, banked vm2/eicu_full_DEC_v12/steering_full_remaining16.json (16 dials x 2 directions, all 17 held-out shards, 16,635 subjects). 32 of 34 declared non-ICU 24h pairs move as expected, all separated; the only misses are the hypertension dial, the one opposite-sign expectation: pushing hypertension UP raises vasopressor initiation (x1.07) and pushing it DOWN lowers it (x0.94), i.e. the model treats every push as 'sicker'. With the ten: 72 of 74 declared eICU pairs (ICU admission excluded as degenerate). Paper updated 10:30 UTC: abstract/contribution 3/5.3 say 26 dials, 72 of 74; Sec 4 says every declared concept ran and explains the hypertension miss by the 'sicker' direction; Limitations no longer says the opposite-sign dial was not run; new tab:dials-more (tables/steering_dials_eicu_more.tex, scriptsize) after tab:dials-mimic; fig:dials-all recaptioned as the ten salience-chosen dials and enlarged to 0.7\textwidth on its own page. 22 pages, pagecheck 0. | | eicu_full_DEC_v13 (steering on all declared concepts, atlas, alerts_cis) | eicu | eICU v2, every held-out shard | /odyssey_main (ca541d8) | the 29-concept eICU model's dial benchmark (every declared concept, no --concepts filter), atlas and paired alert CIs | queued as steps 2-4 of ~/post_v13_queue.sh; outputs under ~/runs/eicu_full_DEC_v13/, logs ~/steering_eicu_v13_full.log, ~/atlas_eicu_v13.log, ~/eicu_full_DEC_v13_alerts_cis.log; marker post-v13 queue done in ~/post_v13_queue.log | steer2 is re-queued behind this marker. | | eicu_full_DEC_v12 (clinical dial benchmark, FULL split) | eicu | eICU v2, every held-out shard (16,635 subjects), lifted sets from 4 train shards | f8633b3 | Steerling steering (stream site, block 4 of 8, tau=1 -> gamma 1.06-2.38 per concept) on every concept with a clinical expectation: does pushing a concept's direction move the 24h hazard risks the way a clinician expects? | done 2026-09-02 14:16 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v12/steering_full.json | 58 of 60 declared 24h outcomes move as expected, all 58 paired-separated; the 2 wrong-way are anemia->ICU admission (x0.99 up, x1.01 down). Shock up: vasopressor x1.09, AKI x1.05, death x1.19; down x0.91/0.96/0.84. Metabolic acidosis up: vasopressor x1.12, AKI x1.13, death x1.23; down x0.89/0.89/0.82. AKI stage 3 up: AKI x1.17, death x1.20. Readout responds on every dial (k_c moves 0.02-0.08 except elevated_lactate/sirs, already saturated at 0.57/0.82); lifted mass moves x1.00-1.04 up and x0.91-0.98 down. | | full_run_DEC_v12 (alert CI leg) | cbm | MIMIC, all 37 held-out shards (same rows as its alerts.json) | 789c33f (main) | paired hazard-vs-GBM CIs for the DECOMPOSED MIMIC model | done 2026-09-02 14:35 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12/alerts_cis.json | 15 cells: the decomposed hazard heads beat the GBM in 2 (death@8h +0.022 [+0.013,+0.031], vasopressor@8h +0.015 [+0.009,+0.021]), tie vasopressor@24h (+0.005 [-0.002,+0.012]) and lose 12, all separated; its deltas are smaller in magnitude than the mixture's in 12 of 15 cells (e.g. ICU@72h -0.107 vs -0.117, AKI@72h -0.047 vs -0.057). Decomposition costs nothing against the GBM on MIMIC either. | | eicu_full_DEC_v12_steer (eval chain: alerts, intervention CIs, readout) | eicu | eICU v2, every held-out shard | f8633b3 (train), d7c96ab (chain) | what one epoch of Steerling steering training (respond-only) does to capability, readout and the label-override test | done 2026-09-02 15:07 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v12_steer/{alerts,intervention_cis,inference_results}.json | Hazard AUROC rises in all 12 cells over eicu_full_DEC_v12 (+0.010 to +0.027; AKI +0.023-0.027, death +0.010-0.021), readout mean 0.898 vs 0.882, while top-1 falls 56.92 -> 55.48. Label-override test: truth-none +0.03 [+0.02,+0.04] (separated, the first positive on a full-scale arm), calibrated truth-flip +0.57 [+0.52,+0.62], banded truth-flip +0.10 (unchanged). CONFOUND: the steer run is one further epoch from checkpoint_best, and no control epoch without the steering losses was run, so 'steering training' and 'one more epoch' are not separated; quote as such. | | full_run_DEC_v12_steer | cbm | MIMIC all 292 train shards, 1 epoch from full_run_DEC_v12/checkpoint_best | f8633b3 | Same recipe as the eICU run, with the corrected lifted-set thresholds (lifted_patients 10,000, min_share 0.2%, min_lift 1.5) so the express loss has targets | first attempt (05:24 UTC, old thresholds) killed during data prep at 06:05 before any phase step; relaunched 06:06 UTC via ~/steer_launcher.sh | Phases ran with BOTH objectives live (07:23 UTC, step 2,575): respond 0.203 -> 0.003 and express 3.24 -> 0.32 over steps 100-1,100, n_injected 3.6k-4.3k per step; no OOM beside the 16-lane dial run (22.6 + 8.7 GB). TRAINING DONE ~11:05 UTC: 24,845 steps in 15,003s (~4.2h), early_stop=False (full epoch), best_val 2.0607 at step 21,500 (v12 parent 2.1925, mixture R2 not comparable in loss). Chain and after-run dial benchmark (ten dials, all 37 shards, 16 lanes) started by the launcher. Eval stage banked 2026-09-02 22:20 UTC (research_journal/figure_data/vm1/full_run_DEC_v12_steer/inference_results.json, all 37 held-out shards, 55,328 patient ends, same as v12): top-1 34.42 -> 35.92, CE 3.831 -> 3.731, readout mean AUROC 0.918 -> 0.932 over 29 concepts (largest gains elevated_lactate 0.946 -> 0.984, oliguria 0.840 -> 0.873; no concept drops by more than 0.003). Opposite sign to eICU's steer run on top-1 (56.92 -> 55.48 there). Same confound as eICU: one further epoch from checkpoint_best with no control epoch, so 'steering training' and 'one more epoch' are not separated. Interventions stage running since 21:10 UTC; alerts and intervention CIs follow. Interventions stage banked 2026-09-02 23:48 UTC (vm1/full_run_DEC_v12_steer/interventions_band15.json, all 37 shards): banded truth-flip +0.01 pt (v12 -0.07), truth-none -0.04 (v12 -0.06), calibrated truth-flip +0.49 (v12 +0.25), calibrated truth-none -0.15 (v12 -0.12); zero_known -1.44 pt (v12 -1.04), zero_residual -6.62 (v12 -6.26). Same picture as eICU's steer run: the label-override test does not move (truth never beats none) while every mode's absolute top-1 rises with the epoch. CIs follow after alerts (chain step). Chain complete 2026-09-03 01:32 UTC; alerts + intervention CIs banked (vm1/full_run_DEC_v12_steer/{alerts,intervention_cis}.json, all 37 shards). Hazard AUROC up in all 15 cells over v12 (mean +0.009; sepsis3 +0.015 to +0.021), identical row sets; vs the tuned GBM the steer heads win 3 of 15 (death 8h +0.045, vasopressor 8h +0.018, 24h +0.008). CIs: banded truth-flip +0.01 [+0.00,+0.02], truth-none -0.04 [-0.05,-0.04], calibrated truth-flip +0.49 [+0.47,+0.52]; zero_known -1.44, zero_residual -6.62. Added to tab:lever as 'Decomposed + steering training, MIMIC-IV'. Same confound as eICU (one further epoch, no control; the eICU ctrl run will speak to it). | | full_run_DEC_v12 / eicu_full_DEC_v13 (cohort sensitivity check) | cbm / eicu | MIMIC all 37 held-out shards; eICU v2 all 17 held-out shards (the alerts_rows.parquet dumps) | 708ebcd tree + scratchpad cohort_check.py (CPU, polars) | Amrit's question (2026-09-03): the tasks have no inclusion/exclusion criteria beyond the at-risk and censoring rules; how much does that matter? Flags patients with chronic ESRD/dialysis or palliative/hospice markers anywhere in their record and re-scores the 24 h hazard AUROC without them | done 2026-09-03 16:30 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12/cohort_check.json and vm2/eicu_full_DEC_v13/cohort_check.json; MIMIC: ESRD 430/36,462 patients supply 16% of AKI positives (AUROC 0.816 -> 0.807 without them), palliative 1,320 supply 58% of deaths (0.943 -> 0.934); eICU shifts <= 0.005; under-18: 0 MIMIC rows, 93 eICU patients (0.6%). In the paper as tab:cohort + a Limitations sentence | | full_run_DEC_v12 vs full_run_DEC_v12_tfm (backbone comparison by truncation) | cbm | MIMIC all 37 held-out shards; the two alerts_rows.parquet dumps joined on (subject, visit, time, event): 2,934,950 rows; 7,227 of 22,316 subjects truncated by the transformer's 2048-token tail window (1,534,725 rows) | 442c7f4 (feat/long-history-compare, PR #234) | Does the hybrid's streaming property buy accuracy on long records? Paired subject-clustered hazard AUROC delta per event/horizon on subjects the transformer saw whole vs truncated | done 2026-09-03 16:40 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12_tfm/long_history_compare.json (30 cells). Seen whole: death/ICU/vasopressor tied (CIs cover 0), transformer +0.02 on AKI and Sepsis-3. Truncated: transformer +0.035 on death (0.985 vs 0.950) but every row is within 2048 tokens of the record end and training used the same end-anchored truncation, so the window position leaks time-to-end (window_leak_check: transformer 0.994 in the first quarter of the window); NOT a clean comparison. Streaming advantage not demonstrated; fix = landmark-anchored sliding windows at eval; Amrit (2026-09-03): left for this submission, paper says 'asserted, not demonstrated'. In the paper as tab:backbone | | full_run_DEC_v12_tfm (ten-dial benchmark, FULL split) | cbm | MIMIC, every held-out shard (36,462 subjects), the same ten concepts as the Mamba flagship's dials | ed7729b (/odyssey_tfmfix; /tfm_fixup.sh) | Backbone portability of Steerling steering: same stream-site push (block 4 of 8, tau 1) on the decomposed transformer, paired against full_run_DEC_v12/steering_full.json | done 2026-09-03 17:48 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12_tfm/steering_full.json. On the 66 declared 24 h pairs: transformer 64 right, Mamba 56 (Mamba misses all six elevated-lactate pairs and four sustained-hypotension pairs; the transformer misses only sustained hypotension <-> vasopressor start, both directions, the pair the at-risk rule was introduced for). Both models' other eight dials are perfect. In the paper: 5.3 portability sentence + tab:dials-tfm | | gemini_full_DEC_v12 (dial benchmark, PARTIAL, Amrit's node) | gemini | GEMINI, every held-out shard; declared concepts only (oliguria, sepsis3, sustained_hypotension_map dropped: no code mapping) | run.sh steering on the mirror at 820bbcb-equivalent | Third-dataset dial benchmark; 30 h, JSON only at the end on that code | running; log through hypoxia up (2026-09-03 14:13 UTC) banked as research_journal/figure_data/gemini/gemini_full_DEC_v12/steering_partial_log_20260903T1413Z.txt: 22 of 31 declared 24 h pairs right by 02:30 UTC Sep 4 (tachycardia 4/6, bradycardia 2/2, hypotension 6/8, hypertension 0/2, hypoxia 4/4, fever 0/2, hypothermia up ICU 1/1, elevated lactate down 1/1, sustained tachypnea 2/2, AKI 2/2 at x1.13/x0.88, AKI stage 2 up 0/1; Guacamole session expired 12:00 UTC, no reads since); console capture on the node at /runs/gemini_full_DEC_v12/steering_console_capture.txt; readouts respond (k_c 0.24->0.47) but risk ratios stay within x0.92-1.08. In the paper: one 5.5 sentence with a TBD marker | | eICU patient identity check (data audit, no model) | local Mac + eicu | raw eICU-CRD 2.0 patient.csv.gz (/Volumes/clinical-data) joined to the eicu_2.0_v2 MEDS split subject ids | 7213776 | Amrit (2026-09-03): does eICU link stays to patients? Yes, uniquepid | done 2026-09-03 22:10 UTC: 200,859 unit stays, 166,355 hospital stays, 139,367 patients; 18,881 patients have >1 hospital stay (27.6% of stays; 4,070 of them at >1 hospital; max 24 stays). Our subjects are hospital stays and the split is by stay: 3,814 of 16,233 held-out patients (24%) also have a stay in train, 729 in tuning; 765 held-out stays (4.6%) share a patient with another held-out stay. Paper: Sec 3, Sec 4 bootstrap caveat, Limitations. A uniquepid-keyed re-split would need every eICU model retrained; not before the freeze | | eICU patient-overlap sensitivity (eicu_full_DEC_v13 and eicu_full_v10 alert dumps) | eicu | eICU v2, all 17 held-out shards; held-out stays split by whether the patient (uniquepid) also has a stay in train (4,063 of 16,635 stays) | 63ea766 tree + scratchpad overlap_check.py | Amrit (2026-09-03): is the stay-level split a huge bug? Does patient overlap inflate the hazard heads? | done 2026-09-03 22:40 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v13/patient_overlap_check.json (both arms). No inflation: on overlap stays the decomposed hazard AUROC is equal or LOWER (death 24h 0.854 vs 0.882 clean, 72h 0.775 vs 0.823; AKI/ICU/vasopressor within 0.01), the GBM moves the same way, and hazard-minus-GBM is the same in both strata (within 0.01 except death 72h, where the model is worse on overlap). Same picture on the v10 mixture. Conclusions unchanged; paper: Limitations sentence + tab:eicu-overlap | | full_run_DEC_v12_tfm (alerts, sliding windows) | cbm | MIMIC, all 37 held-out shards, GBM on all train shards | 25eba4f (feat/sliding-window-eval, PR #236); /tfm_slide.sh queued behind /dose_mimic.sh | Amrit (2026-09-03): the clean long-record comparison. Transformer scored with 2048-token windows every 512 tokens, each landmark scored once with >= 1536 tokens of preceding context and no window anchored to the record end; all landmark rows, paired with the hybrid's dump | queued 2026-09-03 23:55 UTC; outputs alerts_slide.json, alerts_rows_slide.parquet, alerts_slide.log; then long_history_compare against the hybrid with the old tfm dump's truncated set as the stratum key | | full_run_DEC_v12_tfm_slide (transformer retrained with sliding windows) | cbm | MIMIC all 292 train shards; same config as full_run_DEC_v12_tfm plus window_stride: 512 (2048-token windows every 512 tokens; every token a target once, every position >= 1536 tokens of context, no end anchor); then the full eval chain (ALERT_SHARDS=37, STREAM_BASELINE=1, LANES=16) | 94d394b (feat/sliding-window-eval, PR #236); /tfm_retrain_slide.sh queued behind /tfm_slide.sh | Amrit (2026-09-03): 'do both'. The airtight backbone comparison: the transformer trained on every position of every record, like the hybrid, and evaluated with landmark-anchored windows | queued 2026-09-04 00:29 UTC; 5 h training (the v12_tfm run took 01:37-06:29 UTC) + eval chain; logs /dec_mimic_v12_tfm_slide_{train,eval}.log | | full_run_DEC_v12 (random-direction control, ten dials) | cbm | MIMIC, every held-out shard; same ten concepts and push strength as the dials, direction replaced by a random unit vector (seeded) | c10d8d3 (/controls_mimic.sh) | Does the sign test discriminate a concept direction from any push? | done 2026-09-04 00:33 UTC; banked research_journal/figure_data/vm1/full_run_DEC_v12/steering_control_random.json. On the same 66 declared 24 h pairs the random direction agrees with the expected sign 52 times (dials 56) but moves risks a third as much (median |ratio-1| 0.032 vs 0.101), is larger than the dial on only 10 of 66 pairs, and leaves the pushed concept's readout unchanged (|delta k_c| <= 0.03 vs 0.1-0.3). Sign agreement alone is not specific; magnitude and readout response are what the dial adds. Unknown-direction control running next | | eicu_full_DEC_v13 (dial benchmark, every declared concept, FULL split) | eicu | eICU v2, all 17 held-out shards | c10d8d3 (/post_v13_queue.sh step 2) | The 29-concept retrain's own dials (the paper's eICU dial tables are the 26-concept model's) | done 2026-09-03 23:39 UTC; banked research_journal/figure_data/vm2/eicu_full_DEC_v13/steering_full.json. Non-ICU declared 24 h pairs: 86 of 94 right over 28 dials (v12: 72 of 74). Misses: hypertension 0/2 (the opposite-sign expectation, as before), aki_stage_3 0/4 and aki_stage_2 death 0/2: the AKI-stage dials INVERT on AKI and death after the label gained KDIGO's urine leg. 23 new declared pairs (oliguria, sepsis3, hypoxemic respiratory failure, the Sepsis-3 outcome) all move as expected. Paper: headline stays v12; v13 reported with the AKI caveat (tab:dials-v13) | | eicu_full_DEC_v13 (atlas + alerts_cis) | eicu | eICU v2, all 17 held-out shards | c10d8d3 (/post_v13_queue.sh steps 3-4) | concept atlas and paired hazard-vs-GBM CIs for the 29-concept model | done 2026-09-04 00:59 UTC; banked concept_atlas.json and alerts_cis.json under research_journal/figure_data/vm2/eicu_full_DEC_v13/; alerts.json rewritten by the CIs step is byte-identical to the banked one. Paper: fig_comparator_deltas eICU arm -> v13; tab:unknown-cross eICU rows -> v13 | | eicu_full_DEC_v13 (controls, zero-channel, tau 4/8), then steer2 + ctrl | eicu | eICU v2, all 17 held-out shards | c10d8d3 (/controls_dose_then_steer2_eicu.sh, relaunched 01:20 UTC) | as MIMIC's controls/dose chain | running: the chain as written targeted eicu_full_DEC_v12 (26 concepts), which no longer loads on main (29-concept heads); every step failed in 7 s at 01:19 UTC (log kept as controls_dose_then_steer2_eicu.failed_v12.log). Re-pointed at v13 with shock -> sustained_hypotension_map and relaunched; random control running from 01:20 UTC | | full_run_DEC_v12 (unknown-direction control, ten dials) | cbm | MIMIC, every held-out shard; same ten concepts and strength, direction = one of the model's unnamed concept embeddings | c10d8d3 (/controls_mimic.sh) | Second negative control for the dial benchmark | done 2026-09-04 07:14 UTC; banked steering_control_unknown.json. Sign agreement 37 of 66 (chance; dial 56, random 52), median |ratio-1| 0.045 vs the dial's 0.101 (dial larger on 50 of 66), readout unmoved (median |dk_c| 0.007 vs 0.075) | | eicu_full_DEC_v13 (random-direction control, ten dials) | eicu | eICU v2, every held-out shard | c10d8d3 (/controls_dose_then_steer2_eicu.sh) | as MIMIC | done 2026-09-04 06:39 UTC; banked vm2/eicu_full_DEC_v13/steering_control_random.json. Non-ICU pairs: sign agreement 22 of 44 (chance; the v13 dials 40), median |ratio-1| 0.025 vs 0.103 (dial larger on 39 of 44), readout 0.014 vs 0.036 | | full_run_DEC_v12 (zero-channel hazard probes: known / unknown / residual) | cbm | MIMIC, all 37 held-out shards, same landmark rows as alerts.json | c10d8d3 (/controls_mimic.sh; alerts --zero-channel) | Hazard-head completeness: which channel of the decomposition carries the alerts | done 2026-09-04 08:07 UTC; banked alerts_zero_{known,unknown,residual}.json under vm1/full_run_DEC_v12. Hazard AUROC vs the full model over 15 cells: zero named -0.005 to -0.023 (mean -0.012), zero unnamed -0.155 to -0.469 (mean -0.306; death 24 h 0.943 -> 0.588), zero residual -0.003 to -0.019 (mean -0.010). Same picture as the top-1 probes in 5.2: the prediction sits in the unnamed half of the concept module. In the paper: one 5.2 sentence | | eicu_full_DEC_v13 (unknown-direction control, ten dials) | eicu | eICU v2, every held-out shard | c10d8d3 (/controls_dose_then_steer2_eicu.sh) | as MIMIC | done 2026-09-04 11:33 UTC; banked vm2/eicu_full_DEC_v13/steering_control_unknown.json. Non-ICU pairs: sign agreement 8 of 38 (the v13 dials 34), median |ratio-1| 0.041 vs 0.111 (dial larger on 30 of 38), readout 0.010 vs 0.036 | | eicu_full_DEC_v13 (zero-channel hazard probes: known / unknown / residual) | eicu | eICU v2, all 17 held-out shards, same rows as alerts.json | c10d8d3 (/controls_dose_then_steer2_eicu.sh) | Hazard-head completeness on eICU | done 2026-09-04 12:16 UTC; banked alerts_zero_{known,unknown,residual}.json under vm2/eicu_full_DEC_v13. Non-ICU hazard AUROC vs full over 12 cells: zero named -0.005 to -0.056 (mean -0.027), zero unnamed -0.190 to -0.413 (mean -0.310; death 24 h 0.879 -> 0.514), zero residual -0.007 to -0.034 (mean -0.015). Same as MIMIC: the alerts live in the unnamed half | | full_run_DEC_v12 (dose response, tau 4, ten dials) | cbm | MIMIC, every held-out shard | c10d8d3 (/dose_mimic.sh) | Does push strength change the verdicts or only the magnitudes? | done 2026-09-04 14:52 UTC; banked steering_full_tau4.json under vm1/full_run_DEC_v12. Same 66 pairs: 55 right (tau 1: 56; the one flip is sustained hypotension suppress -> AKI, x1.00 -> x1.05); median |ratio-1| 0.398 vs 0.101 (4x); readout shift 0.264 vs 0.075; the misses deepen (sustained hypotension up: death x0.85 -> x0.63; lactate still inverted on all six). Sepsis-3 up: death x1.28 -> x2.76. tau 8 running | | eicu_full_DEC_v13 (dose response, tau 4, ten dials) | eicu | eICU v2, every held-out shard | c10d8d3 (/controls_dose_then_steer2_eicu.sh) | as MIMIC | done 2026-09-04 17:10 UTC; banked steering_full_tau4.json under vm2/eicu_full_DEC_v13. Non-ICU pairs: 34 of 38 at both tau 1 and tau 4 (identical verdicts), median |ratio-1| 0.111 -> 0.466, readout shift 0.049 -> 0.176. tau 8 running | | full_run_DEC_v12 (dose response, tau 8, ten dials) | cbm | MIMIC, every held-out shard | c10d8d3 (/dose_mimic.sh) | as tau 4 | done 2026-09-04 21:31 UTC; banked steering_full_tau8.json. Visible 64 pairs: 53 right (tau 1: 54, tau 4: 53), 63 of 64 keep their tau-1 verdict (the same sustained-hypotension suppress -> AKI pair flips, x1.00 -> x1.17), median |ratio-1| 0.770 (tau 1: 0.097, tau 4: 0.391; effects scale 8x), readout shift 0.41; Sepsis-3 up: death x1.28 -> x2.76 -> x7.5. Dose sentence extended to 'four and eight times' | | gemini_full_DEC_v12 (concept readout export + full dial benchmark) | gemini | GEMINI, every held-out shard (223,207 patient-ends for the readout; steering on all held-out patients, at-risk restricted) | node run (Amrit's job 103958); exported by the odyssey-2e session after fixing the run.sh export whitelists (at_risk_restricted/stratify_by for steering, n_positive_gaps for TimeMetrics); provenance copies on the gemini remote at a93ce24 | The two GEMINI results the paper lacked | done 2026-09-04 ~22:45 UTC; banked research_journal/figure_data/gemini/gemini_full_DEC_v12/{inference_results,steering_full}.json. Readout: mean AUROC 0.958 over the 25 resolving concepts (0.878 lactate to 0.997 anemia). Dials: 25 concepts, 108 declared 24 h pairs, 86 right (87 at 8 h, 84 at 72 h; 97 of 108 keep one verdict across horizons); qSOFA 0/6 and AKI stage 2 0/4 fail everywhere, hypertension 0/2 and fever 0/2 as elsewhere, tachycardia 4/6, hypotension 6/8, hyperkalemia 4/6, coagulopathy 2/4; median |ratio-1| 0.025 (the ICU databases' random-control size) while the readout responds (median shift 0.077). Still pending on the node: the all-shard alerts rerun (11% hedge) and the run.sh whitelist fix on main | | eicu_full_DEC_v13 (dose response, tau 8, ten dials) | eicu | eICU v2, every held-out shard | c10d8d3 (/controls_dose_then_steer2_eicu.sh) | as MIMIC | done 2026-09-04 22:04 UTC; banked steering_full_tau8.json. 44 pairs: 39 right (tau 1: 40, tau 4: 40), 43 of 44 keep their tau-1 verdict, median |ratio-1| 0.731 (tau 1: 0.103). Same picture as MIMIC. NOTE: the chain then launched ~/steer2_v13_queue.sh whose checkout guard called TrainingConfig() without its three required args and aborted ('checkout lacks steering_forecast_at_injected' is a false alarm; ~/odyssey_main is at 775d271 = main with PR #236); guard patched to check dataclass_fields and the queue relaunched 23:10 UTC | | full_run_DEC_v12_tfm_slide (retrain, first attempt) | cbm | - | 94d394b via ~/tfm_retrain_slide.sh | - | failed 2026-09-05 00:57 UTC in 2 s: the config JSON carried output_dir/train_shard_dir/tuning_shard_dir and the CLI passed them too (TypeError: multiple values for 'output_dir'). The marker fired and the bottleneck-site dials started on the card at 00:58Z; retrain re-queued as ~/tfm_retrain_slide2.sh (dirs stripped from the config, checkout origin/main) behind 'site bottleneck mimic done' | | full_run_DEC_v12_tfm_slide (retrain, sliding windows in training) | cbm | MIMIC all 292 train shards, window_stride 512, 16 lanes, config otherwise = full_run_DEC_v12_tfm | a5d7807 (PR #239; run from 47d7afe, identical training code) via ~/tfm_retrain_slide2.sh | The train/eval-matched transformer arm: no window anchored to the record end in training or scoring | trained 2026-09-05 11:57-15:00 UTC, eval chain (ALERT_SHARDS=37, sliding windows read from the saved config) started 15:00Z | Attempts 2 and 3 (07:37Z, 11:56Z) died at startup: TrainingConfig had no window_stride on main because the training-side commit was written after PR #236 was merged; cherry-picked as PR #239. Training then early-stopped at step 18750 in epoch 0 (early_stopping_patience 15, eval_every 250, eval_max_chunks 25): best val 3.0745, the 25-chunk validation loss swung 3.08-3.43 between consecutive evals (the original arm's ran 2.05-2.09), so the stop and the 'best' checkpoint were picked on a noisy signal. Steps are comparable to the original arm (18750 vs 17165) because sliding windows put more windows in an epoch. Eval chain done 20:22Z, banked alerts/inference_results/interventions_band15 under vm1/full_run_DEC_v12_tfm_slide. Not usable as the transformer arm: on the same rows as the hybrid it loses every cell by 0.07-0.15 AUROC (ICU@24h whole records 0.806 vs 0.908; AKI@24h 0.640 vs 0.783), far below the eval-only sliding transformer, which ties the hybrid on whole records. The early-stopped checkpoint is under-trained (sliding windows score only the 512 new positions of each long-record window, so a step carries less supervision; the stop fired in epoch 0). Retrain 3 (slide3, early stopping off, 2 epochs) is the replacement; row-matched JSONs banked 21:50Z: long_history_compare.json (hybrid vs slide2) and slide_vs_retrain.json (eval-only sliding transformer vs slide2; whole records the eval-only arm leads by 0.08-0.18 in all 15 cells, e.g. death@24h 0.947 vs 0.812, sepsis3@24h 0.927 vs 0.782) | | full_run_DEC_v12_tfm_slide3 (retrain, early stopping off) | cbm | as above; early_stopping_patience 1000, eval_max_chunks 100, num_epochs 2 | a5d7807 via ~/tfm_retrain_slide3.sh (queued behind slide2's eval chain, marker 'tfm retrain slide2 done') | Same arm trained to the end of its two epochs with a 4x larger validation set, so checkpoint selection and stopping are not decided by a noisy 25-chunk loss | trained 2026-09-05 20:24 - 2026-09-06 18:34 UTC (141,968 steps, two full epochs of ~71k steps, best val 2.2178, no early stop); eval chain done 23:47Z; banked alerts/inference_results/interventions_band15/config under vm1/full_run_DEC_v12_tfm_slide3 | Decision rule: if slide2's alerts are at or above the eval-only sliding transformer on whole records, slide3 is killed by PID and slide2 is the paper's arm; otherwise slide3 replaces it. Result: a sound forecaster (next-token top-1 0.228 vs the end-anchored arm's 0.206, set top-1 0.806 vs 0.787, CE 5.83 vs 9.21) whose hazard heads still sit BELOW the eval-only sliding transformer on every pooled alert cell (death@24h 0.893 vs 0.932; vasopressor@24h 0.823 vs 0.861; ICU@24h 0.848 vs 0.854) and well above slide2 (0.830, 0.762, 0.770). Row-matched comparisons banked 00:50Z (long_history_compare.json = hybrid vs slide3; slide_vs_retrain.json = eval-only sliding transformer vs slide3): on whole records the eval-only arm leads slide3 in all 15 cells (death@24h 0.947 vs 0.885, sepsis3@24h 0.927 vs 0.879, ICU@24h 0.909 vs 0.884, vasopressor@24h 0.846 vs 0.805), the hybrid by similar margins. Decision: tab:backbone keeps the end-anchored arm scored with landmark-anchored windows (the stronger transformer comparator) and its caption reports the retrain in one sentence. VM1 stopped 00:52Z | | full_run_DEC_v12 (two-sided steering: readout shifts + state-transition probes) | cbm | MIMIC all 37 held-out shards; the paper's ten dials; probes fit on 40 train shards' landmark rows, scored on 8 held-out shards | PR #242 (8cdfe4a) odyssey.inference.outcome_probes then steering --outcome-probes via ~/spec_mimic.sh | Amrit Sep 7: 'dials are bound to only sicker, that prevents us from trusting if they are truly capturing state'. (1) every concept's readout shift under each push, opposites declared to move down; (2) frozen probes for ICU discharge, hospital discharge alive, vasopressor stop as outcomes declared to move DOWN for a sicker state (hypertension declared the other way); (3) a random-direction control scored the same way | relaunched 14:19Z (22d090c, after two startup fixes: polars has no look-ahead; MIMIC STOP rows carry no hadm_id and needed infer_visit); probes done 14:40Z (held-out AUROC icu_discharge 0.81, hospital_discharge_alive 0.79, vasopressor_stop 0.76; 3,250 stop positives); dials done 22:00Z, banked steering_twosided.json + outcome_probes.pt under vm1/full_run_DEC_v12; random control running (~8 h) | Ten dials x 2 directions at 24 h: good outcomes (ICU discharge, discharge alive, vasopressor stop) move the declared way on 35 of 52 pairs, bad outcomes on 56 of 66; 6 of 20 directions fully both-sided (AKI stage 3, hypoxia, anemia both ways). Elevated lactate is inverted on EVERY outcome, good and bad (0 of 6 up, 0 of 6 down); sustained hypotension up lowers every hazard together (death and vasopressor start wrong, discharge and stop 'right'): a rate axis, not 'sicker'. Hypotension, metabolic acidosis, Sepsis-3, qSOFA get all bad outcomes and 2 of 3 good ones (discharge alive misses). Readout focus median 0.11 (pushed concept carries ~11% of total readout movement, every other readout shifts); declared opposites move down 6 of 6. Verdict waits for the random-direction control scored the same way | | eicu_full_DEC_v13 (two-sided steering) | eicu | eICU all held-out shards; same ten dials; probes on 30 train shards (no vasopressor_stop: eICU has no STOP tokens); probe held-out AUROC icu_discharge 0.75, hospital_discharge_alive 0.79-0.81 | e1e6159 via ~/spec_eicu.sh | as above | dials done 2026-09-07 19:39 UTC, banked steering_twosided.json + outcome_probes.pt under vm2/eicu_full_DEC_v13; random control running | Ten dials x 2 directions at 24 h: good outcomes (ICU discharge, discharge alive) move the declared way on 32 of 40 pairs, bad outcomes on 56 of 66; 10 of 20 directions fully both-sided (hypotension, sustained hypotension, metabolic acidosis, qSOFA, hypoxia both ways); elevated lactate fails every good outcome (readout barely responds, +0.003), AKI stage 3 fails every bad outcome but passes the good ones, anemia and sepsis3 half. Readout focus median 0.10: the pushed concept carries 10% of total readout movement and every other readout shifts with an interval clear of zero; declared opposites move down 4 of 6 (hyperkalemia's hypokalemia does not). Verdict waits for the random-direction control scored the same way | | full_run_DEC_v12_tfm (alerts, sliding windows) | cbm | MIMIC, all 37 held-out shards, GBM on all train shards | 25eba4f (/tfm_slide.sh) | the clean long-record comparison | done 2026-09-05 00:56 UTC; banked alerts_slide.json under vm1/full_run_DEC_v12_tfm; comparison vs the hybrid dump (strata = the old tail window's truncated subjects) running as ~/slide_compare.py -> long_history_compare_slide.json | | gemini_full_DEC_v12 (concept atlas) | gemini | GEMINI held-out | run.sh atlas on the node, exported to the mirror at 30f3940 (2026-09-05 00:10 UTC) | GEMINI rows for the cross-database atlas | done; banked research_journal/figure_data/gemini/gemini_full_DEC_v12/concept_atlas.json | | full_run_DEC_v12 vs full_run_DEC_v12_tfm, sliding windows (backbone comparison, clean) | cbm | MIMIC all 37 held-out shards; the hybrid's dump vs the transformer's sliding-window dump (5,945,465 joined rows), strata = the 7,227 subjects the old tail window had truncated (4,545,240 rows) | 25eba4f alerts + ~/slide_compare.py | The clean version of the long-record test (no end anchor) | done 2026-09-05 01:40 UTC; banked long_history_compare_slide.json under vm1/full_run_DEC_v12_tfm. Whole records: death/ICU/vasopressor tie, transformer +0.017 to +0.023 on AKI and Sepsis-3. Long records: hybrid leads 12 of 15 cells (AKI +0.025 to +0.036, ICU +0.021 to +0.027, Sepsis-3 +0.008 to +0.013, vasopressor +0.026 to +0.059), death mixed (+0.008 at 8 h, -0.008 at 72 h). Replaces tab:backbone. Caveat: the transformer checkpoint was trained end-anchored; the landmark-anchored retrain is queued behind the bottleneck-site dials | | full_run_DEC_v12 (steering, bottleneck site, full shards) | cbm | MIMIC all 37 held-out shards, ten dials, all 26 concepts as heads; 64 visible-outcome declared pairs at 24 h | main @ 775d271, odyssey/inference/steering.py --site bottleneck via /site_bottleneck_mimic.sh | Does the stream-site verdict survive when the push is applied to k_c directly, where the readout cannot respond? (bridge from the 5.2 zeroing probes to the 5.3 dials) | done 2026-09-05 07:37 UTC; banked steering_full_site_bottleneck.json under vm1/full_run_DEC_v12 | 52 of 64 pairs as expected (stream site: 54); the same verdict on 62 of 64; median abs(ratio-1) 0.156 vs 0.097 at the stream site; every pair separated; respond_delta exactly 0 on every dial (the push sits after the readout). Per dial identical except sustained hypotension (2 of 8 vs 4 of 8). In the paper: one sentence in 5.3's Controls paragraph. eICU counterpart queued behind the steer2+ctrl chain (/site_bottleneck_eicu.sh) | | eicu_full_DEC_v13_steer2 (steering dials, all 29 concepts) | eicu | eICU all held-out shards; 29 dials x 2 directions; declared pairs at 24 h; paired against eicu_full_DEC_v13's own steering_full.json via canonical_concept_name | main a5d7807-era code, odyssey/inference/steering.py (tau 1.0) via ~/steer2_v13_queue.sh | Does steering training with the forecast/hazard losses OFF the injected positions (PR #227 flag false, respond+express, one epoch from v13 checkpoint_best) keep the dial effects that the first steering run (respond-only, losses on injected positions) shrank toward no change? | done 2026-09-05 18:51 UTC; banked steering_full.json under vm2/eicu_full_DEC_v13_steer2 | Effects are kept, not shrunk: median abs(ratio-1) over the 88 non-ICU declared pairs 0.092 (v13 0.103), every pair separated; sign agreement 70 of 88 (v13 80), same verdict on 78. Paper's ten dials, 44 non-ICU pairs: 32 of 44 (v13 40 of 44), same verdict 36; all 8 flips are AKI pairs whose ratios sit within 0.05 of 1 in both runs (hyperkalemia, hypotension, metabolic acidosis, sustained hypotension up/down); death and vasopressor effects unchanged (metabolic acidosis up: death x1.29 -> x1.25; the first steering run took it to x1.06). Readout response unchanged (amplify respond_delta median 0.048 vs 0.056). Interpretation waits for eicu_full_DEC_v13_ctrl (one plain epoch from the same checkpoint), which separates the extra-epoch effect; 5.3's steering-training sentence is rewritten only after both are in | | eicu_full_DEC_v13_ctrl (control epoch: one plain epoch from v13 checkpoint_best, steering_phases 0) | eicu | eICU all 134 train shards, all held-out shards for eval; paired with v13 and steer2 on identical rows | a5d7807-era main via ~/steer2_v13_queue.sh | Separate the extra-epoch effect from the steering-training effect: v13 -> steer2 (respond+express, losses off injected positions) vs v13 -> ctrl (same epoch, no steering) | trained 2026-09-05 18:55-22:04 UTC (22429 steps, best val 1.9016); eval chain done 02:19Z; banked alerts/inference_results/interventions_band15/config under vm2/eicu_full_DEC_v13_ctrl; dials running (29 concepts, 8 h) | The extra epoch, not the steering losses, carries the accuracy gains. Hazard-head AUROC rises in all 15 cells under BOTH runs, ctrl more: mean +0.028 (median +0.016, sepsis3 +0.05 to +0.08) vs steer2 +0.016 (median +0.013). Readout AUROC mean 0.877 (v13) -> 0.885 (steer2) -> 0.894 (ctrl); next-token top-1 0.555 -> 0.558 -> 0.564. Any sentence crediting steering training with better readout or hazard accuracy must go; Label override (band 0.15, top-1 points): truth-none v13 -0.16, steer2 -0.18, ctrl -0.12; truth-flip -0.27 / -0.33 / -0.15: neither epoch makes the override help (the first steering run's +0.03 on the 26-concept parent does not recur on the 29-concept one). Dials done 2026-09-06 16:03 UTC, banked steering_full.json under vm2/eicu_full_DEC_v13_ctrl. Ten dials, 44 non-ICU pairs: ctrl 44 of 44 as expected (v13 40, steer2 32), same verdict as v13 on 40 (the 4 flips are v13's own AKI stage 3 misses, now correct), but the effects SHRINK by a third: median abs(ratio-1) 0.068 (v13 0.101, steer2 0.103); metabolic acidosis up: death x1.37 (v13 x1.29, steer2 x1.25); sepsis3 up: death x1.05 (x1.11, x1.12); readout response 0.067 (0.056, 0.048). All 29 concepts, 88 pairs: 81 of 88, median 0.068 (v13 80, 0.103; steer2 70, 0.092). Reading: one plain epoch keeps or fixes the signs and shrinks the effect; steering training with the losses at the injected positions (first run) shrank it almost to nothing; steering training with the losses moved off (steer2) holds v13's effect size and loses the small AKI effects. No epoch strengthens the lever | | gemini_full_DEC_v12 (alerts, GBM refit on every train shard) | gemini | GEMINI, all 112 held-out shards; GBM on all 894 train shards at run.sh's 10% row thinning | run.sh alerts on the node (GEMINI_ALERTS_TAG _allshards), exported to the mirror at 6501bc9 (2026-09-05 04:23 UTC) | Reviewer B2: the 11%-of-train-shards hedge | done; banked research_journal/figure_data/gemini/gemini_full_DEC_v12/alerts_allshards.json. The all-shard GBM moves by at most 0.006 AUROC from the 100-shard fit; hazard heads still win 9 of 12 (death +0.036 to +0.056, ICU +0.034 to +0.035, vasopressor +0.037 to +0.044; AKI -0.016 to -0.025). tab:gemini regenerated; the hedge leaves 5.5 | | full_run_DEC_v12 (two-sided specificity test, MIMIC-IV) | cbm | MIMIC all 37 held-out shards; 20 dials (ten concepts, up/down), scored against a random-direction control of the same size and strength | main, odyssey/inference/specificity.py + scripts/make_specificity_table.py; source steering_twosided.json (banked) + steering_twosided_control_random.json (pulled via gcloud scp 2026-09-08 07:xx UTC from odyssey-cbm-a100:/runs/full_run_DEC_v12/) | Does the concept push move only the pushed concept and the outcomes it declares, or is it indistinguishable from a generic severity push? (specificity/two-sided steering work, PR #242 branch feat/state-transition-probes) | scored 2026-09-08 | Concept push beats the random control on every metric: median focus 0.11 vs 0.02 (5x); declared-opposite readout deltas 5-6x larger in magnitude though both agree on sign at n=6 (checked directly against raw per-dial shifts, not a pipeline artifact); good-outcome pairs 35/52 vs 29/52; bad-outcome pairs 56/66 vs 52/66; strict both-sided pass (every declared good AND bad pair correct) 6/20 vs 2/20. Real separation, not a clean one -- two thirds of dials fail the both-sided bar even under a genuine push. Does not trigger the archived HANDOFF (docs/archive/HANDOFF_2026-08-31.md) negative-verdict contingency (no rewrite of abstract/contribution 3/5.3 title/Discussion sentence). Written into main.tex: new Section 5.3 paragraph after Controls, new tab:specificity in Appendix E after tab:dials-mimic (probe held-out AUROC 0.81/0.79/0.76 for icu_discharge/hospital_discharge_alive/vasopressor_stop, read directly from steering_twosided.json's outcome_probes block). eICU's control file is already pulled (odyssey-eicu-a100:/runs/eicu_full_DEC_v13/steering_twosided_control_random.json, done 2026-09-08T00:56:58Z) but not yet tabled -- MIMIC only this pass | | eicu_full_DEC_v13 (two-sided specificity test, eICU-CRD) | eicu | eICU all held-out shards; 20 dials (ten concepts, up/down), scored against a random-direction control of the same size and strength | main, odyssey/inference/specificity.py + scripts/make_specificity_table.py --side-by-side (new flag, added this pass); source steering_twosided.json (banked) + steering_twosided_control_random.json (pulled via gcloud scp 2026-09-08 09:15 UTC from odyssey-eicu-a100:/runs/eicu_full_DEC_v13/) | Same specificity/two-sided question as the MIMIC-IV run above, on the other database, for the side-by-side Appendix E table | scored 2026-09-08 | eICU-CRD separates from its control more cleanly than MIMIC-IV: median focus 0.10 vs 0.03 (~3x); good-outcome pairs 32/40 vs 24/40 (only 2 outcome probes on eICU -- no vasopressor_stop probe, held-out AUROC 0.75/0.79 for icu_discharge/hospital_discharge_alive, read from steering_twosided.json's outcome_probes block); bad-outcome pairs 56/66 vs 41/66; strict both-sided pass 10/20 vs 1/20 (a much wider margin than MIMIC-IV's 6/20 vs 2/20). Declared-opposite sign agreement ties at 4/6 for both real and random pushes; spot-checked against raw per-dial deltas (sustained_hypotension_map/hyperkalemia) same as the MIMIC check -- not a pipeline bug, though unlike MIMIC the random control's raw effect on the opposite concept is sometimes larger in magnitude even while getting the sign right (e.g. sust. hypotension -> hypertension: dials -0.005, control -0.069), a genuine cross-database difference worth knowing about even though it doesn't change the aggregate verdict. Both databases now in main.tex's tab:specificity (merged side-by-side table, Appendix E) and the Section 5.3 paragraph after Controls |