fraud-detection

A controlled experiment: does SMOTE (and its cousins) actually help imbalanced fraud classification, or does a tuned class-weighted baseline match it? Two datasets, six resampling methods, three classifiers — 36 cells, 5-fold stratified CV each.

Takeaway

1. Does SMOTE help?

No — and it actively hurts with strong learners. baseline_weighted ties or beats every SMOTE variant on PR-AUC in 5 of 6 cells; on IEEE-CIS the penalty is statistically reliable (Cohen's d ≈ −2.5 for LightGBM).

The reason: a model that supports per-sample loss weighting already does what SMOTE attempts. SMOTE additionally distorts the input distribution with synthetic linear interpolations, producing a manifold the model has never seen at deployment — which causes over-confident predictions and worse calibration.

The one exception is SMOTEENN on extreme-imbalance + clean features (creditcard_ulb × lgbm, 0.811 vs 0.767). Its cleanup step partially undoes the distortion. This advantage vanishes on messy real features (IEEE-CIS).

2. What actually works?

Priority	Method	When
1	class_weight='balanced' / scale_pos_weight	Always try first — one parameter, no distribution distortion
2	Post-hoc calibration (isotonic / Platt)	When the weighted baseline over-corrects the prior (moderate imbalance, ~3–5% positive rate)
3	SMOTEENN	Only if the learner cannot be weighted and features are clean
✗	Random undersampling	Never — worst PR-AUC and calibration in every cell

---

Hypothesis (written before any results were observed)

With a strong learner (LightGBM) and a tuned decision threshold, SMOTE will not produce a meaningful PR-AUC improvement over a weighted baseline, and will degrade calibration. With a weak learner (logistic regression), SMOTE may help. Cleanup hybrids (SMOTEENN, SMOTETomek) will outperform vanilla SMOTE.

---

Design

Data

Dataset	Source	n	Features	Positive rate
creditcard_ulb	Kaggle mlg-ulb/creditcardfraud (ULB 2013, full)	284,807	30	0.173%
ieee_cis	Kaggle ieee-fraud-detection (stratified subsample)	80,000	218	3.499%

The two datasets bracket two regimes:

• Extreme imbalance, PCA-clean features (creditcard): the textbook case where SMOTE is often recommended.
• Moderate imbalance, messy real features (IEEE-CIS): where calibration matters because the prior matters.

Note on Dataset A: the brief specified "Kaggle Credit Card Fraud 2023." That redistribution (nelgiriyewithana/...) is artificially class-balanced 50/50, which defeats the experiment, so we default to the original ULB version (DATASET_A_SLUG is configurable). Dataset B is stratified-subsampled to 80k rows to keep runtime tractable; full data via FRAUD_IEEE_FULL=1.

6 × 3 method × classifier grid

methods:    baseline_weighted (class_weight='balanced', no resampler)
            random_undersample
            smote               (k_neighbors=5)
            borderline_smote
            smoteenn            (SMOTE + Edited Nearest Neighbours cleanup)
            smotetomek          (SMOTE + Tomek-link cleanup)

classifiers: lgbm    (LGBMClassifier, n_estimators=300, lr=0.05)
             logreg  (LogisticRegression, max_iter=1000)
             rf      (RandomForestClassifier, n_estimators=200)

Pipeline structure for every cell: StandardScaler → (resampler) → classifier, built with imblearn.pipeline.Pipeline so resamplers are correctly skipped at predict time.

Protocol

• 5-fold stratified CV directly on the full dataset.
• Out-of-fold (OOF) probabilities via cross_val_predict. Per fold, the OTHER four folds' OOF probabilities are used as the threshold-tuning validation slice — disjoint from the test slice for that fold, so threshold-dependent metrics (F1@opt, recall@p≥0.80) are not chosen and reported on the same observations.
• Leakage guards live inside the pipeline: SimpleImputer and StandardScaler are fit per-fold, and resamplers (SMOTE et al.) are skipped at predict time by the imblearn.pipeline.Pipeline contract — so no statistic from the test fold ever reaches model fitting.

Metrics

• PR-AUC (average_precision_score) — headline.
• Brier score — calibration squared error.
• ECE (expected calibration error, 10 equal-width bins).
• Recall at precision ≥ 0.80, threshold picked on the validation slice.
• F1 at the threshold that maximises F1 on the validation slice.

Default-threshold (0.5) classification metrics are deliberately omitted — they are essentially meaningless on imbalanced data.

---

Results

PR-AUC (mean over 5 folds, ± 95% CI half-width)

Dataset	Classifier	baseline_weighted	random_undersample	smote	borderline_smote	smoteenn	smotetomek
creditcard_ulb	lgbm	0.854 ± 0.029	0.724 ± 0.033	0.767 ± 0.171	0.848 ± 0.048	0.811 ± 0.055	0.767 ± 0.171
creditcard_ulb	logreg	0.730 ± 0.035	0.701 ± 0.020	0.729 ± 0.041	0.718 ± 0.025	0.739 ± 0.036	0.729 ± 0.041
creditcard_ulb	rf	0.852 ± 0.036	0.737 ± 0.024	0.854 ± 0.036	0.850 ± 0.042	0.843 ± 0.042	0.854 ± 0.036
ieee_cis	lgbm	0.582 ± 0.013	0.516 ± 0.019	0.560 ± 0.006	0.566 ± 0.008	0.551 ± 0.010	0.562 ± 0.007
ieee_cis	logreg	0.302 ± 0.029	0.244 ± 0.022	0.300 ± 0.029	0.293 ± 0.034	0.283 ± 0.033	0.300 ± 0.028
ieee_cis	rf	0.574 ± 0.008	0.488 ± 0.029	0.560 ± 0.015	0.554 ± 0.019	0.536 ± 0.017	0.560 ± 0.012

baseline_weighted is the top cell (or within CI of the top) in 5 / 6 dataset × classifier combinations. The one exception is creditcard_ulb × rf, where vanilla SMOTE and SMOTETomek edge it out by 0.002 PR-AUC — well inside noise.

See results/figures/pr_auc.png for the bar chart with CIs.

SMOTE vs baseline_weighted — paired per-fold difference in PR-AUC

Dataset	Classifier	Δ mean	Cohen's d	95% CI
creditcard_ulb	lgbm	−0.087	−0.62	[−0.260, +0.086]
creditcard_ulb	logreg	−0.001	−0.13	[−0.008, +0.007]
creditcard_ulb	rf	+0.002	+0.35	[−0.006, +0.011]
ieee_cis	lgbm	−0.022	−2.45	[−0.033, −0.011]
ieee_cis	logreg	−0.002	−1.69	[−0.004, −0.001]
ieee_cis	rf	−0.014	−2.04	[−0.022, −0.005]

On IEEE-CIS the difference is small in magnitude but the within-fold variance is even smaller — Cohen's d in the −2 range, with paired 95% CIs that exclude zero. SMOTE doesn't just fail to help; it reliably hurts.

Calibration — Brier score (lower is better)

Dataset	Classifier	baseline	random_under	smote	borderline	smoteenn	smotetomek
creditcard_ulb	lgbm	0.00040	0.02397	0.00097	0.00043	0.00097	0.00097
creditcard_ulb	logreg	0.02277	0.02927	0.02390	0.00847	0.02428	0.02390
creditcard_ulb	rf	0.00042	0.03213	0.00053	0.00042	0.00063	0.00053
ieee_cis	lgbm	0.06073	0.11539	0.02218	0.02180	0.02494	0.02208
ieee_cis	logreg	0.15438	0.16467	0.15600	0.13907	0.18063	0.15610
ieee_cis	rf	0.02140	0.12739	0.02346	0.02277	0.02751	0.02347

The calibration story has two faces, and they tell different mechanistic tales:

1. creditcard_ulb (0.17%): baseline_weighted is the most-calibrated model. SMOTE roughly doubles the Brier of LightGBM here, and the reliability plot (results/figures/calibration_creditcard_ulb_lgbm.png) shows SMOTE-trained LightGBM is wildly over-confident at the high-probability end — it predicts ~0.9 for cases where the empirical positive rate is ~0.15.

2. ieee_cis (3.5%): baseline_weighted is the least-calibrated model. class_weight='balanced' on a 3.5%-positive dataset upweights the rare class by a factor of ~14, which shifts the predicted score distribution sharply right. The model now over-predicts positives, ECE jumps to 0.14, Brier to 0.06. Here SMOTE actually beats the weighted baseline on calibration (Brier 0.022 vs 0.061) even though it loses on PR-AUC. Ranking and calibration come apart.

Random undersampling is bad on both axes

random_undersample is the consistent loser: lowest PR-AUC in every cell except creditcard_ulb × logreg (tied with smote variants), and Brier scores 30–80× worse than baseline_weighted on LightGBM/RF cells with extreme imbalance — predictably, because throwing away 99.8% of the negatives leaves the model thinking positives are common, and probabilities are wildly inflated.

smote ≈ smotetomek exactly

In every cell, vanilla SMOTE and SMOTETomek give identical PR-AUC, Brier, ECE, and recall numbers — within the printed precision. Tomek-link removal found essentially nothing to remove on these feature spaces. The "cleanup" was a no-op.

---

Interpretation

The hypothesis, checked against results

Prediction	Verdict
SMOTE will not improve PR-AUC over a weighted baseline with LightGBM.	Confirmed. SMOTE loses on both datasets; difference is large and reliable on ieee_cis × lgbm (d=−2.45).
SMOTE will degrade calibration with LightGBM.	Confirmed for creditcard_ulb. Contradicted for ieee_cis, but only because class_weight='balanced' over-corrects the prior on a 3.5%-rate dataset. The contradiction is a critique of the baseline, not a vindication of SMOTE.
SMOTE may help logistic regression.	Did not materialise. Logreg with class_weight='balanced' already exploits the imbalance information SMOTE provides. Paired diff is essentially zero.
Cleanup hybrids will outperform vanilla SMOTE.	Mixed. SMOTEENN beats vanilla SMOTE on creditcard_ulb × lgbm (0.811 vs 0.767) and creditcard_ulb × logreg. It loses on the other four cells. SMOTETomek is identical to SMOTE. So: "hybrids dominate" is too strong — they help when the feature space contains genuine label noise to clean.

Why this result is mechanistically sensible

A weighted-loss strong learner (gradient-boosted trees, in particular) is already doing the only thing SMOTE accomplishes for it — telling the model to care more about minority examples. The cost function difference between "duplicate the minority point 100× synthetically" and "multiply its loss contribution by 100" is small in expectation; what differs is the variance and the calibration. SMOTE introduces synthetic samples that are linear interpolations in feature space, which is plausible for PCA-derived features (creditcard) but compresses the input distribution onto a lower-dimensional manifold than the real data lives on. The classifier learns confident decisions on a non-representative input distribution → over-confident at deployment.

Class weighting leaves the input distribution alone and changes only the gradient. For models that natively support it (LightGBM, sklearn logreg, RF), it is strictly less invasive.

When does SMOTE actually help here?

Looking at the cells where SMOTE did outperform: creditcard_ulb × logreg × smoteenn (+0.009), and matched on creditcard_ulb × rf. The pattern is not "weak learner = SMOTE helps." It's "noise-cleaning cousins of SMOTE help under extreme imbalance with PCA-clean features." A small effect, never larger than the spread between classifier choices.

---

Limitations

• Single train/test protocol per cell. 5-fold CV variance estimates are honest but small; a fully nested CV for threshold tuning would be more rigorous (and ~5× the compute).
• Raw features only. No feature engineering on either dataset. IEEE-CIS in particular has known wins from grouping transaction sequences by card / device; we held that constant to keep the SMOTE-vs-not axis isolated.
• Two datasets. Generalising "SMOTE doesn't help" beyond similar tabular fraud-style data needs more.
• IEEE-CIS is subsampled (80k of 590k rows). Full-data results may differ at the margins, especially for the SMOTE variants whose synthetic samples scale with the minority count.
• No ensemble-of-resamplers methods. BalancedBaggingClassifier, BalancedRandomForest, EasyEnsemble are out of scope. Conservatively, those should be compared too before declaring "weight-only baselines win."
• One hyperparameter point per classifier. A tuned LightGBM (scale_pos_weight swept, leaf size, etc.) might widen or shrink the gap. We held this fixed because the comparison axis is the resampling step.

---

Reproducing

uv sync
uv run main.py                # checks/downloads data, trains 36 cells, writes results + figures

main.py is the single end-to-end entrypoint. It detects existing data files and skips the Kaggle downloads if they're already present. Individual stages are also exposed as standalone commands for partial runs:

uv run fraud-download         # just (re)download data
uv run fraud-detection        # just train (assumes data is present)
uv run fraud-analyze          # just regenerate summary CSVs and figures

Kaggle authentication: drop a kaggle.json token at ~/.kaggle/kaggle.json AND copy the raw key string into ~/.kaggle/access_token (the new KGAT_*-style tokens require both for competition endpoints). Accept the IEEE-CIS competition rules at https://www.kaggle.com/c/ieee-fraud-detection/rules once.

Repository layout

main.py          # single entrypoint: ensure data -> train -> analyze
src/fraud_detection/
  data.py        # download + load + stratified subsample
  methods.py     # 6 resampler factories
  pipeline.py    # 3 classifier factories + imblearn pipeline builder
  evaluate.py    # PR-AUC, Brier, ECE, recall@p80, F1@opt
  run.py         # 36-cell orchestrator (fraud-detection)
  analyze.py     # summary CSVs + figures (fraud-analyze)
results/         # results.csv, calibration.csv, summary*, figures/
data/            # gitignored