Implement MC-SIMEX method and enhance comparison notebooks

.github/copilot-instructions.md

@@ -41,7 +41,7 @@ This repository contains two distinct Python packages for biomedical analysis:
   - `discrete_optimization.py`, `utilities.py` - Optimization utilities (used by KM methods)
   - `datacard.py` - Main Datacard class (core data parsing)
   - `command_line_interface.py` - KM-specific CLI functions
-  - `yi_correction.py` - Yi correction method
+  - `comparisons/` - published alternatives (Yi weights, Küchenhoff MC-SIMEX)
 
 ### Test Structure
 - **`test/roc_picker/`**: ROC Picker tests, datacards, and reference data
@@ -61,15 +61,16 @@ This repository contains two distinct Python packages for biomedical analysis:
 - `03_kaplan_meier_example.md` - Jupyter notebook showing Python API usage examples
 - `04_analysis_demo.md` - Jupyter notebook with analysis demonstration
 - `05_compare_lifelines_greenwood.md` - Jupyter notebook comparing to `lifelines` and Greenwood method
-- `06_yi_method_comparison.md` - Jupyter notebook comparing Yi correction method
-- `07_command_line_interface.md` - **Pure Markdown** (no Python cells) documenting all CLI options for `kombine` and `kombine_twogroups` commands (synced with Jupytext)
+- `06_compare_thomas_grunkemeier.md` - Jupyter notebook comparing Thomas–Grunkemeier intervals
+- `07_previous_methods_comparison.md` - Jupyter notebook comparing Yi, MC-SIMEX, and KoMbine
+- `08_command_line_interface.md` - **Pure Markdown** (no Python cells) documenting all CLI options for `kombine` and `kombine_twogroups` commands (synced with Jupytext)
 
 **Documentation style guidelines**:
-- Files `03_*.md` through `06_*.md` are Jupytext notebooks with Python cells for interactive examples
-- Files `01_*.md` and `07_*.md` are pure Markdown with Jupytext headers but no Python cells
+- Files `03_*.md` through `07_*.md` are Jupytext notebooks with Python cells for interactive examples
+- Files `01_*.md` and `08_*.md` are pure Markdown with Jupytext headers but no Python cells
 - All documentation markdown files must have Jupytext headers to allow `jupytext --sync` to process them
-- All CLI options must be documented in `07_command_line_interface.md` and verified by `test/kombine/test_ci_and_documentation.py`
-- When adding new CLI arguments, update `07_command_line_interface.md` and run the documentation test
+- All CLI options must be documented in `08_command_line_interface.md` and verified by `test/kombine/test_ci_and_documentation.py`
+- When adding new CLI arguments, update `08_command_line_interface.md` and run the documentation test
 - The table of contents (`01_table_of_contents.md`) must list all numbered documentation files
 
 **Compiling KoMbine LaTeX documentation (`02_kombine.tex`)**:
@@ -214,6 +215,7 @@ python -m test.kombine.test_km_likelihood           # ~2 minutes, requires unres
 python -m test.kombine.test_hazard_ratio
 python -m test.kombine.test_km_plotting
 python -m test.kombine.test_yi_correction
+python -m test.kombine.test_mc_simex
 python -m test.kombine.test_discrete_class_observable
 python -m test.kombine.test_ci_and_documentation
 ```

.github/workflows/ci-cd.yml

@@ -15,6 +15,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Set up Python 3.x
+        id: setup-python
         uses: actions/setup-python@v5
         with:
           python-version: 3.11
@@ -28,17 +29,23 @@ jobs:
         run: |
           jupytext --sync docs/roc_picker/*.md docs/kombine/*.md
 
+      # success() || failure(): still run later linters if an earlier one fails
+      # (not if the job is cancelled). The job stays red if any step failed.
       - name: pyflakes (all python files)
+        if: success() || failure()
         run: |
           python -m pyflakes kombine roc_picker test docs .github/workflows/checklog.py .github/workflows/__init__.py
 
       - name: pylint (all python files)
+        if: success() || failure()
         run: |
           python -m pylint kombine roc_picker test docs .github/workflows/checklog.py .github/workflows/__init__.py
 
       - uses: jakebailey/pyright-action@v2.3.2
+        if: success() || failure()
         with:
           version: 1.1.400
+          python-path: ${{ steps.setup-python.outputs.python-path }}
 
   test-roc-picker:
     runs-on: ubuntu-latest
@@ -125,16 +132,15 @@ jobs:
           pip install . ipykernel jupytext lifelines nbconvert pyflakes pylint texoutparse &&
           rm -r build
 
-      - name: Install Gurobi
+      # The gurobipy wheel installed by `pip install .` bundles the solver library,
+      # so no separate distribution is needed. The 10.0.3 tarball this replaces was
+      # installed over the wheel and downgraded the solver: Gurobi 10 approximates
+      # every log/exp with a 1000-piece linearization, which on the KM MINLPs is
+      # 2.7x slower than the exact handling in 11+ and shifts 2NLL by ~0.8.
+      # See docs/kombine/compute_budget.md.
+      - name: Report Gurobi version
         run: |
-          wget https://packages.gurobi.com/10.0/gurobi10.0.3_linux64.tar.gz
-          tar -xvzf gurobi10.0.3_linux64.tar.gz
-          echo "export GUROBI_HOME=$PWD/gurobi1003/linux64" >> $GITHUB_ENV
-          echo "$PWD/gurobi1003/linux64/bin" >> $GITHUB_PATH
-          echo "LD_LIBRARY_PATH=$PWD/gurobi1003/linux64/lib" >> $GITHUB_ENV
-          cd gurobi1003/linux64
-          python setup.py install
-          cd ../..
+          python -c "import gurobipy; print(gurobipy.gurobi.version())"
 
       - name: Configure Gurobi WLS license
         run: |
@@ -181,13 +187,21 @@ jobs:
         run: |
           python -m test.kombine.test_yi_correction
 
+      - name: MC-SIMEX unit tests
+        run: |
+          python -m test.kombine.test_mc_simex
+
       - name: convert KoMbine notebooks
         run: |
           jupytext --sync docs/kombine/*.md
 
+      # Notebook 07 defaults to n=20 then n=50. CI sets
+      # KOMBINE_QUICK_COMPARISON=1 to skip n=50 (n=20 only, minutes).
       - name: test KoMbine notebooks
+        env:
+          KOMBINE_QUICK_COMPARISON: "1"
         run: |
-          jupyter nbconvert --execute --to html -- docs/kombine/*.ipynb
+          jupyter nbconvert --execute --to html --ExecutePreprocessor.timeout=3600 -- docs/kombine/*.ipynb
 
       - name: Compile KM plots
         run: |

.gitignore

@@ -26,3 +26,6 @@ test/kombine/datacards/lung/flatfielding_systematic/*.txt
 test/kombine/datacards/lung/poisson_and_flatfielding/*.txt
 test/kombine/datacards/lung/uncorrected_flatfielding_systematic/*.txt
 test/kombine/datacards/lung/poisson_and_uncorrected_flatfielding/*.txt
+
+# On-demand rebinned methods_comparison cards (from rebin_methods_comparison_times.py)
+test/kombine/datacards/simple_examples/rebinned/

docs/__init__.py

@@ -0,0 +1 @@
+"""Docs package (notebooks and helpers; not installed via pip)."""

docs/kombine/01_table_of_contents.md

@@ -8,7 +8,7 @@ jupyter:
       extension: .md
       format_name: markdown
       format_version: '1.3'
-      jupytext_version: 1.18.1
+      jupytext_version: 1.19.5
 ---
 
 ```python
@@ -37,9 +37,9 @@ jupyter:
  - `06_compare_thomas_grunkemeier.html`
    - comparison of KoMbine binomial-only confidence intervals to
      Thomas and Grunkemeier as implemented in R package `km.ci`
- - `07_yi_method_comparison.html`
-   - comprehensive demonstration of Yi's misclassification correction
-     method applied to Kaplan-Meier curves, p-values, and hazard ratios
+ - `07_previous_methods_comparison.html`
+   - comparison of Yi's weights, Küchenhoff MC-SIMEX, and KoMbine
+     on Kaplan-Meier curves, p-values, and hazard ratios
  - `08_command_line_interface.html`
    - documentation for the command line interface (`kombine` and `kombine_twogroups` commands)
 

docs/kombine/03_kaplan_meier_example.md

@@ -6,7 +6,7 @@ jupyter:
       extension: .md
       format_name: markdown
       format_version: '1.3'
-      jupytext_version: 1.18.1
+      jupytext_version: 1.19.5
   kernelspec:
     display_name: rocpicker
     language: python

docs/kombine/04_analysis_demo.md

@@ -6,7 +6,7 @@ jupyter:
       extension: .md
       format_name: markdown
       format_version: '1.3'
-      jupytext_version: 1.18.1
+      jupytext_version: 1.19.5
   kernelspec:
     display_name: rocpicker
     language: python
@@ -69,9 +69,9 @@ print(f"Censored: {sum(1 for p in datacard.patients if p.censored)}")
 
 ### KoMbine p-value calculation
 
-KoMbine provides a likelihood-based p-value that accounts for measurement uncertainty when comparing survival curves across a biomarker threshold. For context, we also compute the standard logrank p-value.
+KoMbine provides a permutation likelihood-ratio p-value that accounts for measurement uncertainty when comparing survival curves across a biomarker threshold. For context, we also compute the standard logrank p-value.
 
-For this dataset and threshold, the likelihood p-value is $2.67\times 10^{-2}$ and the logrank p-value is $1.61\times 10^{-2}$.
+For this dataset and threshold, the KoMbine permutation p-value is $6.00\times 10^{-2}$ and the logrank p-value is $1.61\times 10^{-2}$.
 
 ```python
 # Define biomarker threshold to split patients into two groups
@@ -83,7 +83,11 @@ km_p_value_minlp = datacard.km_p_value(
     parameter_min=-np.inf,
     parameter_max=np.inf,
 )
-kombine_p_value, _, _ = km_p_value_minlp.solve_and_pvalue(cox_only=False)
+kombine_p_value, _, _ = km_p_value_minlp.solve_and_pvalue(
+    cox_only=False,
+    n_permutations=49,
+    rng=0,
+)
 
 # Conventional logrank p-value
 logrank_p_value = km_p_value_minlp.survival_curves_pvalue_logrank()

docs/kombine/05_compare_lifelines_greenwood.md

@@ -6,7 +6,7 @@ jupyter:
       extension: .md
       format_name: markdown
       format_version: '1.3'
-      jupytext_version: 1.18.1
+      jupytext_version: 1.19.5
   kernelspec:
     display_name: rocpicker
     language: python
@@ -75,7 +75,7 @@ We do, in fact, get 1:1 agreement with `lifelines`.
 
 For completeness, KoMbine also provides logrank test p-values and hazard ratio calculations that match `lifelines` for fixed observables. These comparisons demonstrate that our implementation of standard survival analysis methods is correct.
 
-For measurement error corrections using Yi's method, see **notebook 07_yi_method_comparison.ipynb**.
+For measurement error corrections using Yi's method and MC-SIMEX, see **notebook 07_previous_methods_comparison.ipynb**.
 
 
 ### Fixed Observable: P-value and Hazard Ratio vs lifelines

docs/kombine/06_compare_thomas_grunkemeier.md

@@ -6,7 +6,7 @@ jupyter:
       extension: .md
       format_name: markdown
       format_version: '1.3'
-      jupytext_version: 1.18.1
+      jupytext_version: 1.19.5
   kernelspec:
     display_name: rocpicker
     language: python
@@ -17,7 +17,6 @@ jupyter:
 """Compare KoMbine binomial-only KM intervals against Thomas-Grunkemeier via R."""
 
 # pylint: disable=wrong-import-position
-# pyright: reportMissingImports=false
 ```
 
 ```python