[hroskes]

Summary

This PR speeds KoMbine KM/HR profile CIs and the production plot compile. The statistical model, plot n/trials/CLs, and Gurobi success rule are unchanged (OPTIMAL only; no --testing).

• Mode-specialized KM MINLPs — drop unused binomial/patient-wise encodings. binomial_only is a continuous p profile; cox_only locks assignments so presolve can drop the MIP.
• MIP starts — nearby-HR assignment incumbents when cox_only matches and |Δ log H| ≤ 1; within-time KM assignment and continuous profile starts.
• Flow-consistent (r, d) prune — drop binomial indicators that cannot occur on nested risk sets.
• Shared-cache multi-level brentq — one cached walk per side of the MLE for nested CLs, staying on the MLE-connected crossing.

The N=12 unit-test card is a tie. Larger full-NLL cards (N≈100 encoding, lung-style profiles) are where the library changes show up.

GitHub test-kombine vs main (Jul 14)

Step	main	This PR (02059c2)
Compile KM plots	8.3 min	3.4 min
Notebooks	208s	106s
Hazard ratio	9s	5s
KM likelihood	104s	96s
Whole job	~18 min	~12 min

Compile split: km_example 3.9s, greenwood 7.5s, p_value 44s, lung 151s.

Test plan

• python -m test.kombine.test_hazard_ratio
• python -m test.kombine.test_km_likelihood
• CI green on this PR

[hroskes]

Benchmarking (2c204e5 on top of the MIP-start commit)

Have we sped up KoMbine? Yes on larger KM profile solves. No meaningful change on the small cards that dominate current CI.

GitHub test-kombine (production workflow, no --testing)

	main (Jul 14)	This PR, MIP starts only (5a602d7)
Whole job	17.9 min	17.4–17.5 min
KM likelihood	104s	103s
Hazard ratio tests	9s	8s
Notebooks	208s	182s
Compile KM plots	8.3 min	8.1 min

The first commit (nearby-HR Var.Start) does not move CI vs main.

Same-machine A/B of this commit vs committed speed/mip-warmstart / ≈ main KM bands

Warm WLS, .conda Python, 2NLL bands:

Workload	Before	After 2c204e5	Change
Unit-test card (poisson_ratio_km_censoring, N=12, 8 times, 3 reps)	19.8s	19.7s	none
Larger card (fixed_km_censoring_many_patients, N≈100, 4 times, 2 reps)	35.3s	7.5s	~4.7×

Local checks after this commit

• test_km_likelihood --collapse-logic-only: pass (~93s)
• full test_km_likelihood: pass (~218s). This branch is still χ² LRT (no permutation). 218s local vs 103s on GHA is machine difference, not a regression vs main.
• Production compile_km_plots is running locally now; CI will be the official plot timing.

What actually got faster

Mode-specialized MINLPs (no unused binomial/KM subgraphs), pruned choose-(r,d) indicators, assignment MIP starts within a time, and TIME_LIMIT/SUBOPTIMAL retries that still require OPTIMAL (last fallback MIPGap 1e-2, then one OutputFlag=1 debug attempt). Gurobi SUBOPTIMAL is never accepted.

Expect CI KM likelihood to stay ~100s. The win is the larger profile / docs / lung-style cards, not the N=12 unit test.

[hroskes]

Plot-compile speed (6e71198)

Library-only changes (same n, trials, CLs; no --testing). Still require OPTIMAL.

Local production compile_km_plots (same machine)

Stage	2c204e5	6e71198
km_example	(unsplit; in 10.1 min total)	11.7s
greenwood (binomial N=12 + N=100)		31.0s
p_value (500 cox_only, incl. n=1000×100)		152.7s
lung RFS (10 panels × high/low)		372.9s
total	606.7s	568.2s (~6%)

CI Compile KM plots on 2c204e5 was 8.1 min. This push will be the official GHA number.

What landed

• binomial_only: assignments are fixed, so skip the choose-(r,d) binary encoding and keep a continuous p profile (Greenwood N=100, lung A/F).
• cox_only: lock a[i,j] with bounds so presolve can drop the assignment MIP (p-value figure + lung A/F).
• Full LRT: copy null a/beta Starts onto the alternative before changing the null constraint (lung E/J). Reading .X after model.update() is invalid.
• Patient-wise: NumericFocus=0 first; NumericFocus=3 stays in the fallback cascade.
• Within-time assignment + continuous profile Starts (already the right grain).
• compile_km_plots now prints [stage] elapsed …s (PDFs unchanged).

What did not land (caused Gurobi SUBOPTIMAL / status 13)

• Cross-time assignment Starts — failed test_km_likelihood --no-censoring at t=8.
• Flow-consistent (r,d) pruning — same class of risk; left as per-time bounds only.
• Replacing the full-NLL big-M sandwich with binom_penalty == bilinear expr — quadratic equality, status 13.

test_km_likelihood passed locally in ~200s (was ~218s on 2c204e5).

[hroskes]

Flow-consistent (r, d) prune (e34efaf)

This is back. Earlier we listed it with the t=8.0 SUBOPTIMAL failures; that was a bundled revert, not an isolated cause. On collapsed no-censoring t=8 there is only one KM time, so the extra checks never run. Exhaustive 2^12 assignments on both simple cards never lost a realizable pair. Gurobi still must return OPTIMAL.

What it does

After the per-time caps (r ≤ n_at_risk_max, d ≤ n_died_max, d ≤ r), drop choose-(r, d) indicators with s = r − d > n_at_risk_max[i+1] + n_censored_between_times_max[i]. The previous-risk check (r > n_at_risk_max[i−1]) is included but redundant on nested risk sets.

Unit tests

test_km_likelihood now covers:

• per-time bounds, including (r, d) = (0, 0) and d > r
• s-rule boundary and censoring slack (s may exceed the next risk-set max by at most cmax)
• last time / single collapsed group (no next-interval rule)
• injected non-nested previous-risk bound
• every 0/1 assignment on the N=12 cards stays feasible (collapse on/off)

Same-machine interleaved A/B (warm both, then alternate)

Full-NLL bands, CLs [0.68, 0.95], .conda Python. Best KM probabilities matched.

Workload	Extra indicators dropped	Per-time-only	Flow prune	Speedup
N=12 poisson_ratio_km_censoring, collapse on, all plot times (3 pairs)	55 / 91 at last t (n_times=2)	median 8.0s	median 6.1s	~1.3×
N=12 no-collapse, 4 plot times (2 pairs)	18 / 131 at last t	16.6s	16.7s	none
N≈100 fixed_km_censoring_many_patients, first 4 times (3 pairs)	83 / 2939 at 4th t; 0 at 1st	median 14.1s	median 14.1s	none

A first ordered A/B looked like 3× on N=12; that was warmup. The interleaved numbers above are the ones to use.

binomial_only still skips the choose-(r, d) MIP, so Greenwood / cox_only plot stages are unchanged. Expect a small win only on full-NLL walks that keep more than one KM time (censoring that splits groups).

test_km_likelihood passed locally with the prune (~5 min) before this commit.

[hroskes]

Speed summary (whole PR through b9bb2c4)

Does the PR improve time noticeably? Yes on the production plot compile and on larger KM profile solves. The N=12 unit-test card stays about the same. The last four commits (shared multi-level brentq) are a smaller extra win on lung-style full NLL, not on CI’s small cards.

Gurobi still must return OPTIMAL. Same n, trials, and CLs. No --testing.

GitHub test-kombine (official)

Step	main Jul 14	MIP starts 5a602d7	Mode MINLPs 2c204e5	Plots+flow e34efaf	This HEAD b9bb2c4
KM likelihood	104s	103s	(same ballpark)	85s	93s
Hazard ratio	9s	8s		5s	5s
Notebooks	208s	182s		101s	failed (see below)
Compile KM plots	8.3 min	8.1 min	~8.1 min	3.3 min (195s)	skipped (notebooks failed)

e34efaf compile split: km_example 3.5s, greenwood 9.8s, p_value 44s, lung 137s. That 8.3 min → 3.3 min cut is the main CI win (mode-specialized MINLPs, continuous binomial-only, tighter cox-only, flow prune, HR MIP starts).

b9bb2c4 did not get a compile number: notebooks died in 06_compare_thomas_grunkemeier (atol=3e-05, one upper CI off by 3.5e-05). Pyright is also red on the new helper (brentq return type / rtol float vs float64, twoNLL.cache). Local test_km_likelihood still passed.

Local tests this round (same machine, vs e34efaf)

Workload	e34efaf	b9bb2c4
python -m test.kombine.test_km_likelihood	~5 min historically	pass, 198s
N=12 poisson full-NLL bands, 8 times, 2 reps	mean 7.4s (6.4–8.4)	mean 6.5s (6.4–6.5) — tie
Lung panel E high group, 25 pts × 10 times, full NLL	150s	115s (1.31×)

N=12 CL edges match the old independent brentq path to ~1e-5 (test atol 2e-4). At lung t=2 (best p = 1) the 68% lower edge is 0.919 here vs 0.972 on e34efaf (same χ² height, wiggly MINLP profile); t=8 matches to ~1e-6.

How to read it

• Whole PR vs main: noticeable. Plot compile is the one you feel (~2.5× on GHA). KM likelihood 104s → 85s. Notebooks 208s → 101s. Hazard ratio 9s → 5s.
• N=12 unit-test card: not a meaningful speedup at any layer (MIP starts, mode MINLPs, brentq sharing).
• Large / lung full NLL: yes. Earlier ~4.7× on N≈100 binomial encoding; flow prune ~1.3× on N=12 with collapse; this brentq pass ~1.3× on one lung full-NLL group vs e34efaf.
• Last four commits vs e34efaf: worth it for lung, not for CI unit tests. GHA KM likelihood even went 85s → 93s (extra MLE-connected probes). Re-run compile after the notebook atol / pyright fixes for the official b9bb2c4 plot number.

[hroskes]

Timing update (02059c2, CI green)

Notebooks and pyright are fixed. Official GitHub test-kombine on this SHA finished, so compile ran.

Gurobi still must return OPTIMAL. Same n, trials, and CLs. No --testing.

GitHub test-kombine (PR check)

Step	main Jul 14	Plots+flow e34efaf	b9bb2c4 (broken CI)	This HEAD 02059c2
KM likelihood	104s	85s	93s	96s (push 106s)
Hazard ratio	9s	5s	5s	5s
Notebooks	208s	101s	failed	106s (push 104s)
Compile KM plots	8.3 min	3.3 min (195s)	skipped	3.4 min (206s) (push 223s)

Compile split on the PR job: km_example 3.9s, greenwood 7.5s, p_value 44s, lung 151s. Script total_elapsed=206.1s.

test-roc-picker passed in ~5 min on both the push and the PR. The earlier 47 min PR job died during notebook execute after the unit tests had already passed; the matching push of b9bb2c4 had already passed in 5m46s.

Whole test-kombine is ~11.5–12 min vs ~18 min on main.

How to read it vs the last comment

• Filling in the blanks: notebooks are back at ~105s (same as e34efaf). Compile is 3.4 min vs 3.3 min at e34efaf and 8.3 min on main — a tie with e34efaf within GHA noise, still ~2.5× vs main.
• Shared brentq did not move CI plot compile or the N=12 KM unit test. The whole-PR win is still the earlier MINLP / plot work.
• KM likelihood 96–106s is runner scatter around the old ~100s; e34efaf’s 85s is still the best GHA sample.