Stroke Risk Benchmark

A leakage-aware, reproducible benchmark for tabular stroke-risk prediction on the public Kaggle stroke prediction dataset.

This repository is a methodology and reproducibility project. It does not claim clinical validity, causal discovery, or production readiness.

Why this repository exists

The dataset has 5,110 observations but only 249 positive stroke outcomes (4.87%). On this kind of data, a model can reach roughly 95% accuracy by almost always predicting the majority class. Many public notebooks compare models without controlling for that fact, and some also leak information during imputation, resampling, or model selection.

This project therefore asks:

How well can several standard tabular classifiers rank stroke risk when preprocessing, threshold selection, and model evaluation are kept leakage-free?

What is implemented

• Stratified repeated nested cross-validation: 5 outer folds x 5 random seeds.
• Imputation, one-hot encoding, and scaling fitted on training data only.
• SMOTE fitted inside each training split, never before cross-validation.
• A threshold selected only on an inner validation split.
• A dummy prior baseline to expose the accuracy trap.
• Model comparison using PR-AUC, ROC-AUC, recall, precision, F1, MCC, Brier score, and confusion-matrix counts. Accuracy is reported but not treated as the primary metric.
• Figures and a generated technical report.
• An audit of the earlier exploratory notebook and its leakage risks.

Results

The completed run is available in reports/technical_report.md. Under repeated nested cross-validation, logistic regression obtained the best PR-AUC: 20.20% +/- 3.36%, with recall 48.71% +/- 16.17% and precision 18.97% +/- 2.89% at a validation-selected threshold. The dummy prior baseline obtained 95.13% accuracy but 0% recall, which is the central reason this project does not use accuracy as its headline metric.

These numbers do not establish clinical usefulness. They show that moderate ranking signal exists in this dataset, while precision and split-to-split variability remain serious limitations for a stroke-screening claim.

The benchmark currently evaluates:

• Dummy prior baseline
• Logistic regression
• Logistic regression with balanced class weights
• Random forest
• Random forest with balanced class weights
• Histogram gradient boosting
• SMOTE plus logistic regression

Reproduce

python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
source .venv/bin/activate

python -m pip install -r requirements.txt
python scripts/download_data.py
python scripts/run_benchmark.py

The raw CSV is downloaded from Kaggle and is deliberately excluded from Git. Do not upload it to GitHub.

Repository layout

stroke-risk-benchmark/
├── data/                  # data instructions; raw CSV is ignored
├── reports/               # generated report and old-notebook audit
├── results/               # metrics, predictions, figures
├── scripts/               # command-line entry points
├── src/stroke_risk_benchmark/
│   ├── constants.py
│   ├── data.py
│   ├── evaluation.py
│   ├── experiment.py
│   ├── modeling.py
│   └── reporting.py
└── tests/

Interpretation

A useful result from this dataset is not "we reached 95% accuracy." It is:

• how much ranking performance remains after the majority-class shortcut is removed;
• whether balancing changes recall at an acceptable precision cost;
• how much the result changes across folds and random seeds;
• whether predicted probabilities are calibrated;
• what cannot be concluded without external validation.

The generated report separates those questions instead of collapsing them into one optimistic number.

Data and code licence

The code in this repository is released under the MIT licence. The Kaggle dataset is not covered by that licence and is not redistributed here. Check the original dataset page for its terms of use and attribution requirements.

Limitations

• Only 249 positive outcomes are available.
• The dataset is observational and may not represent a target clinical population.
• There is no external, temporal, or prospective validation cohort.
• Repeated cross-validation measures split sensitivity; it is not a substitute for external validation.
• SMOTE creates synthetic observations and cannot create new clinical knowledge.
• A benchmark result is not a novel clinical finding and should not be presented as one.

Application-ready framing

See docs/resume_and_applications.md for English CV and personal-statement wording, and docs/resume_and_applications_zh.md for the Chinese version. The repository should be presented as a methodology and reproducibility project, not as a clinical breakthrough.