Classical Statistical Pre-processing vs. AI-based Representation Learning for Business Intelligence

Code and experiment pipeline for the Master's thesis "A Systematic Comparison of Classical Statistical Pre-processing and AI-based Representation Learning for Business Intelligence: Evidence from Bank-Marketing and Retail Transaction Data."

---

Research questions

	Question	Evidence produced by
RQ1 — Predictive performance	How do classical pre-processing pipelines (A–C) and AI-based representation-learning pipelines (D–F) differ in predictive performance on real business datasets?	results_table_<ds>.csv, ablation/
RQ2 — Robustness	Which pre-processing strategy is most robust to missing values, outliers and label noise?	robustness/

RQ2 is answered only in restricted form. See "Robustness for D/E/F is measured with leaking encoders" under Known limitations: the AI pipelines cannot be compared against the classical ones on robustness, and the thesis withdraws that comparison rather than reporting it.

A Wilcoxon signed-rank test with Bonferroni correction was computed (significance_table_<ds>.csv) but is not used to support any conclusion. Its independence assumption is violated by repeated cross-validation, so the p-values are not interpretable; see Known limitations. Every comparison in the thesis is decided on the difference in mean PR-AUC read against the fold standard deviation.

---

Requirements

Python ≥ 3.10. All commands are run from the repository root, which is the code/ directory of the thesis working tree.

pip install -r requirements.txt      # or: conda env create -f environment.yml

Datasets

The datasets are not redistributed here. Download them and place the raw files in data/raw/:

File	Source
bank_marketing.csv (delimiter ;)	https://archive.ics.uci.edu/dataset/222/bank+marketing
online_retail.xlsx	https://archive.ics.uci.edu/dataset/352/online+retail

data/raw/ and data/processed/ are excluded from version control. Everything under data/processed/ is regenerated by the pre-processing step below.

---

Pipelines

ID	Name	Steps
A	Baseline	Imputation → One-Hot Encoding → StandardScaler
B	Statistical	A + IQR winsorising (continuous features only) + Box-Cox + PCA
	b_pca90	PCA retaining 90 % of variance
	b_pca20	PCA retaining 20 % of variance
C	Feature Engineering	A + domain features (Bank) / RFM (Retail) + polynomial interaction terms
D	DAE	Denoising Autoencoder → 16-dim latent representation
E	VAE	Variational Autoencoder → 16-dim latent mean (μ)
F	FT-Transformer	Feature Tokenizer Transformer → 64-dim summary-token embedding

Six pipelines, one of them (B) instantiated in two variants, run on both datasets. Downstream classifiers: lr, rf, xgb, lgbm, mlp — each tuned with Optuna (20 trials, PR-AUC objective). That gives 7 × 5 × 2 = 70 combinations.

The encoders receive categorical features label-encoded and numeric features standardised; Pipeline F embeds categoricals through a lookup table instead.

A note on Pipeline C, since its name is misleading. The ablation attributes its entire advantage to the polynomial interaction terms and none of it to the domain features, which score 0.4121 against 0.4124 for raw input on Bank Marketing and reproduce the raw value exactly on Retail — where all seven pipelines share the same four RFM aggregates anyway, so C differs from A only by the interaction terms. The finding of the thesis is that interaction terms on standardised features pay off, not that domain knowledge does.

Evaluation protocol

• 5 × 3 Repeated Stratified K-Fold (15 splits), primary metric PR-AUC (both datasets are imbalanced: Bank 11.3 % positives, Retail 50.8 %)
• Additional metrics: ROC-AUC, F1, Precision, Recall, Brier score, Lift@Top-10 %
• Held-out test split (20 %) reported alongside the CV estimate
• Seed 42 throughout

The result tables also carry an EMV column from an earlier expected-value analysis. It is not used in the thesis and can be ignored.

Leakage prevention

Three safeguards, each of which materially changed the results:

1. duration is dropped from the Bank dataset. It is only known after the call has ended and is therefore post-hoc information.
2. Retail uses a temporal split. RFM features are aggregated up to a cutoff date; the target is observed strictly afterwards.
3. Encoders D/E/F are re-fitted inside every CV fold in the main experiment. Fitting the encoder once on the full training split and then cross-validating on its output means every validation fold was already seen by the encoder — for the supervised FT-Transformer including its labels. This inflated Bank F from a true ≈ 0.46 to a spurious 0.86 PR-AUC. Fold encodings are cached under data/processed/<ds>/<pipe>/fold_cache/ so all five downstream models reuse the same 15 encodings.

Encoder early stopping is monitored on a held-out validation split, not on the training loss — otherwise the encoder simply trains until it memorises the data.

---

Execution order

# 1) Exploratory data analysis (optional)
python scripts/eda/eda_bank.py
python scripts/eda/eda_retail.py
python scripts/eda/eda_export_csv.py        # CSV export for external plotting

# 2) Pre-processing — writes to data/processed/
python scripts/classical/pipeline_a.py
python scripts/classical/pipeline_b.py
python scripts/classical/pipeline_c.py
python scripts/ai/pipeline_d.py
python scripts/ai/pipeline_e.py
python scripts/ai/pipeline_f.py

# 3) Main experiment (2 datasets x 7 pipeline variants x 5 models = 70 runs)
python scripts/run_experiment.py

# 4) Supplementary analyses
python scripts/ablation.py                     # per-step contribution
python scripts/robustness/run_robustness.py    # MCAR/MAR/MNAR/outliers/label noise
python scripts/learning_curves.py              # PR-AUC vs. training set size

# 5) Tables, tests and figures
python scripts/eval/results_table.py
python scripts/eval/significance_test.py
python scripts/visualize/make_thesis_figures.py   # all figures used in the thesis

scripts/visualize/ also contains the standalone scripts the thesis figures grew out of (ablation_heatmap.py, robustness_plot.py, lift_curves.py, feature_importance.py, representation_viz.py) and scripts/eval/cost_benefit.py. These still run but their output is not part of the submitted thesis.

run_experiment.py options

Flag	Effect
--dataset bank|retail	restrict to one dataset
--pipeline a|b_pca90|b_pca20|c|d|e|f	restrict to one pipeline
--model lr|rf|xgb|lgbm|mlp	restrict to one model
--no-hpo	skip Optuna, use defaults (much faster)
--skip-preprocessing	processed data already exists

Completed runs are skipped based on the presence of results/<dataset>_<pipeline>_<model>_cv.json. Delete the file to force a re-run.

Quick smoke test:

python scripts/run_experiment.py --dataset retail --pipeline a --model lr --no-hpo

---

Repository layout

.                               repository root (= code/ in the thesis tree)
  config.py                     central configuration (seed, folds, HPO budget, paths)
  requirements.txt
  environment.yml

  data/raw/                     input datasets (not versioned, see above)
  data/processed/<ds>/<pipe>/   pipeline output + fold_cache/ (not versioned)

  scripts/
    eda/                        exploratory analysis
    classical/                  pipelines A-C + shared transformers
                                  missing_values.py, encoder.py, scaler.py,
                                  outlier_handler.py, boxcox.py, interaction_terms.py
    ai/                         pipelines D-F (PyTorch)
    eval/                       CV runner, metrics, model factories + Optuna tuners,
                                results table, significance tests
    robustness/                 corruption injection + robustness sweep
    visualize/                  figure generation
    ablation.py                 per-step ablation study
    learning_curves.py

  results/
    <ds>_<pipe>_<model>_cv.json per-run CV + test metrics + runtime/memory
    results_table_<ds>.csv       summary table
    significance_table_<ds>.csv  Wilcoxon + Bonferroni
    ablation/                    per-step ablation results
    robustness/                  robustness sweep results
    learning_curves/
    visualize/                   generated figures and their underlying CSVs
    eda_csv/                     EDA summaries and CSV exports

The results/ tree is versioned deliberately: it contains the numbers behind every table and figure of the thesis, so the reported values can be checked without repeating the full run.

---

Deviations from the research proposal (Disposition)

Two points where the implementation differs from what the proposal described.

• Hyper-parameter optimisation is not nested. The proposal specifies nested CV. Implemented is Optuna HPO with an internal 3-fold CV on the training split, followed by the reported 5 × 3 repeated CV. Fully nested HPO would multiply runtime by the number of outer folds. The consequence — a mild optimistic bias in the CV estimate — is bounded because the same budget (20 trials) applies to every pipeline × model combination, so the comparison remains fair. The held-out test split provides an independent check.
• The Retail target is a retention label, not a "high-value" label. The proposal describes a binary high-value target. Implemented is temporal churn prediction: RFM features are aggregated before a cutoff date, and the target is whether the customer transacted again afterwards. This was chosen because a high-value label derived from Monetary would be a deterministic function of an input feature, i.e. target leakage.

Known limitations

• The significance analysis is not usable. Two independent problems. First, the 15 scores of a 5 × 3 repeated cross-validation are not independent — the repetitions re-use the same data and the folds within a repetition share 80 % of their training material — which violates the assumption of the Wilcoxon signed-rank test, understates the variance and rejects too readily. The corrected resampled t-test of Nadeau & Bengio (2003) would have been the right instrument. Bonferroni does not repair this, since it addresses multiplicity rather than dependence. Second, with 15 paired samples the discrete minimum p-value is 2/2¹⁵ ≈ 6.1 × 10⁻⁵, which a large share of comparisons attains exactly, so the test cannot rank differences by magnitude even where it applies. significance_table_<ds>.csv is retained for completeness; no conclusion rests on it.
• Two datasets do not establish generality, and only one of them discriminates. On Retail all seven pipelines fall within 0.017 PR-AUC of one another, so that dataset works mainly as a negative control; the evidence that separates the pipelines comes essentially from Bank Marketing alone.
• Pipeline F sees the training labels, A–E do not. Even implemented without leakage, the supervised encoder receives a training signal that the design withholds from the other six pipelines. Its defeat under the tuned protocol survives that advantage, which makes the negative result stronger; its win in the ablation is partly explained by it.
• pdays = 999 is a sentinel value ("client not previously contacted", 96.3 % of rows; real values range 0–27). Only Pipeline C decodes it into a binary flag; A/B/D/E/F treat it as a number. The asymmetry is intentional — handling such quirks is what Pipeline C is meant to demonstrate — but note that the ablation finds the four Bank domain features, this flag among them, worth −0.0002 PR-AUC in total. The capability is exercised and measured; it does not pay off.
• Class weighting is not applied uniformly. The MLP weights its loss by the inverse class ratio; LR, RF, XGBoost and LightGBM do not, although all four support it. PR-AUC is threshold-independent and unaffected, and the same model meets every pipeline, so pipeline comparisons hold. Comparisons of F1 or Brier across models should not be made: the MLP reaches F1 0.454 against 0.337–0.358 and Brier 0.152 against 0.077 on Bank.
• Encoder hyper-parameters were not tuned while the downstream models received 20 Optuna trials each. The asymmetry disadvantages D/E/F. It was accepted for compute reasons and because the observed deficits (0.013–0.016 PR-AUC) are of a size that encoder tuning would plausibly narrow but not obviously reverse.
• Learning curves are unavailable for D/E/F. Valid curves would require re-fitting the encoder at each training fraction. Whether the deficit of the learned representations narrows with more data is therefore open.
• Ablation uses Logistic Regression as a fast, deterministic proxy for the downstream model. Step contributions may differ for tree-based learners.
• Robustness for D/E/F is measured with leaking encoders. run_robustness.py loads dae_model.pkl / vae_model.pkl / ftt_model.pkl from data/processed/<ds>/<pipe>/. Those files are written by the standalone scripts scripts/ai/pipeline_[def].py, which fit the encoder once on the whole training split — they are not the per-fold encoders that run_experiment.py caches under fold_cache/. Every validation fold of the sweep is therefore encoded by a model that has already seen it, and for the supervised Pipeline F including its labels. This is exactly the leakage that safeguard 3 above removes from the main experiment. The signature is visible in the results: Bank F enters the sweep at a base PR-AUC of 0.476, against 0.4572 in the main experiment and 0.4607 in the ablation. Consequence: the thesis makes no claim about the robustness of the AI pipelines relative to the classical ones. Valid comparisons are those among D, E and F, which are distorted identically, and those among A, B and C, which involve no encoder. Re-running the sweep with per-fold encoders (24 corruption settings × 15 folds × 3 pipelines × 2 datasets) was outside the compute budget.
• Robustness measures inference-time behaviour. Independently of the point above, the encoders are applied frozen to corrupted data, which corresponds to a deployed model meeting degraded input. It does not measure how robustly they would train on corrupted data. Where corruption pushes a column past the 50 %-missing drop threshold, the column is re-inserted as missing rather than removed — a deployed encoder cannot change its input dimension at inference time. On Retail this is visible at high corruption rates, since the dataset has only four RFM features.
• Per-fold encoder training is capped at 60 epochs (CV_ENCODER_EPOCHS in run_experiment.py) for compute reasons, versus 100 for the one-off pipeline runs.
• A single seed. Seed 42 fixes the classical pipelines exactly, but the three encoders would vary across initialisations. All statements about D, E and F rest on one initialisation.

---

Reproducibility

Seed 42 is fixed for NumPy, PyTorch and all scikit-learn splits. A full run (70 experiments + ablation + robustness + learning curves + figures) takes roughly 8–11 hours on CPU with the 20-trial HPO budget.

License

MIT