diff --git a/Readme.md b/Readme.md index 75ee31e..08f034d 100644 --- a/Readme.md +++ b/Readme.md @@ -2,93 +2,7 @@ > ๐Ÿ”— **Live demo:** https://iambatman07-fraud-detection-mlops.hf.space ยท [HF Space](https://huggingface.co/spaces/IamBatman07/Fraud-Detection-MLOps) -**Fraud-detection MLOps โ€” built around an honest temporal split, with drift detection, auto-retrain, and registry rollback that a notebook can't replicate.** - -Fraud-Detection-MLOps detects fraud in payment transactions (PaySim + Sparkov), but its resume claim is **MLOps discipline at scale**, not raw model quality. A 7-day upgrade sprint found and fixed a data-leakage bug that had been inflating the headline AUC, then layered MLflow registry promotion/rollback, KS+PSI drift detection, auto-retrain-and-promote, Dask-deterministic feature engineering, and Postgres-backed telemetry on top of the existing DVC pipeline. - ---- - -## The headline: the leakage fix is the story - -The original `train.py` used `train_test_split(..., stratify=y)` โ€” a **random** split on time-series fraud data, which leaks future transaction patterns into training. The Day-1 audit found a second leak too: the benchmark fell back to the *training* file instead of the held-out test file. - -| Metric (Sparkov OOT, vs AutoGluon 0.952) | Before fix | After fix | -|---|---:|---:| -| **Honest AUC** | **0.9210** (leaked) | **0.7949** (honest baseline) | -| Apparent gap to AutoGluon | โˆ’0.031 (mirage) | โˆ’0.157 (real) | - -The fix: `temporal_split_per_source()` sorts each source chronologically and takes the final 20% as test. **The ~0.126 AUC drop was the data leakage. The 0.7949 is the number that was always real** โ€” and the rest of the sprint earns it back honestly. - ---- - -## Final champion + the gap closed honestly - -A 30-trial Optuna sweep + source-balanced sample weights closed the entire gap to AutoGluon โ€” **no new features, no ensembling, no SHAP** (that's the joint Fraud-Detection project's territory; Fraud-Detection-MLOps is deliberately MLOps-only). - -| Model | OOT AUC (sparkov_test.csv) | ฮ” vs AutoGluon 0.952 | -|---|---:|---:| -| Day-1 honest baseline (temporal split) | 0.7949 | โˆ’0.157 | -| **Day-5 champion (Optuna + source-balanced)** | **0.9520** | **โˆ’0.00004** (tied) | - -Champion: XGBoost, per-source temporal split + source-balanced weights (paysim 0.60ร—, sparkov 2.95ร—) + Optuna tuning. Stored at `models/fraud_model_tuned_fixed.pkl` (MLflow run `day05_targeted_fix_v1`). - ---- - -## The four MLOps capabilities (what a notebook doesn't have) - -| Capability | Headline metric | -|---|---| -| **Dask distributed features** | Pandas 1.15M rows/s vs Dask 0.26M rows/s on one host, **bit-exact within 5.5e-12** | -| **MLflow registry rollback** | **3.9 ms** median alias flip; 11.9 ms full audited rollback | -| **KS+PSI drift detection** | **0-day** detection lag on a synthetic 2ฯƒ shift; **precision 1.0, recall 1.0** on the 7-day window | -| **Auto-retrain + shadow-promote** | drift โ†’ train โ†’ shadow-eval โ†’ promote in **median 6.85 s**; 3/3 events auto-promoted | - -Dask "loses" the single-host throughput race but is the *scaling primitive* โ€” and it's bit-exact with Pandas, which is the property that lets you swap engines without changing results. - ---- - -## Frontier guard-rail: why LLMs don't do tabular fraud - -Day 6 ran Claude Opus 4.6 as an LLM fraud judge on the same 200-row OOT sample (tool-use schema forcing a structured verdict): - -| Strategy | AUC | AUPRC | Latency/query | Cost @ 1K QPS/day | -|---|---:|---:|---:|---:| -| **Fraud-Detection-MLOps champion (XGBoost)** | **0.9156** | **0.526** | 60 ยตs | **$0.43** | -| Naive notebook XGBoost | 0.626 | 0.415 | 73 ยตs | $0.43 | -| Claude Opus 4.6 LLM-judged | 0.622 | 0.351 | 1.82 s | **$1,250,691** | - -Two findings: (1) the LLM is **30,400ร— slower** and **2.9Mร— more expensive**, and ranks 0.29 AUC worse โ€” it caught only textbook patterns (large online txns at night) and missed the "card skimmed at POS โ†’ grocery abuse" behavioral class. (2) **The naive notebook XGBoost (0.626) ties the LLM (0.622)** โ€” the 0.29-AUC jump to the champion comes from the *discipline layers*, not from "XGBoost vs LLM". The discipline is the model. The LLM judge ran in simulate mode (no API key on host); `--mode api` is the same code path with a real key. - -### MLOps ablation (L0 โ†’ L3, full OOT) -| Layer | OOT AUC | ฮ” vs prev | -|---|---:|---:| -| L0 naive (random split + defaults) | 0.547 | โ€” | -| L1 + temporal split (Day-1 fix) | 0.657 | +0.111 | -| L2 + source-balanced weights | 0.696 | +0.039 | -| L3 + Optuna (champion) | **0.948** | **+0.252** | - -Total L0โ†’L3 gain: **+0.401 OOT AUC**. Every layer is load-bearing; Optuna is the biggest single contributor. - ---- - -## Architecture - -``` - DVC pipeline: load_fdb โ†’ combine โ†’ preprocess โ†’ train โ†’ evaluate โ†’ benchmark_fdb - โ”‚ (temporal split, MLflow tracking) - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ src/features/ Dask + Pandas behavioral features (bit-exact) โ”‚ - โ”‚ src/registry/ MLflow promote / rollback (alias-flip, ~4ms) โ”‚ - โ”‚ src/drift/ KS + PSI detector โ†’ N-consecutive trigger โ†’ retrain โ”‚ - โ”‚ src/serving/ FastAPI /predict + async shadow deployment โ”‚ - โ”‚ src/telemetry/ Postgres 4-table logger (predictions, drift, โ€ฆ) โ”‚ - โ”‚ src/frontier/ LLM-judge negative result + ablations โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - docker compose up โ†’ FastAPI :8000 + Postgres + Redis + MLflow - streamlit run app.py โ†’ "4 Ops" dashboard (drift / retrain / rollback / AUPRC) -``` +Fraud detection on payment transactions (PaySim + Sparkov), built as an MLOps pipeline: a temporal train/test split, MLflow registry promotion and rollback, KS+PSI drift detection, auto-retrain-and-promote, Dask feature engineering, and Postgres-backed telemetry on a DVC pipeline. --- @@ -112,67 +26,25 @@ python -m src.registry.promote --experiment sentinel-day01-temporal-split --ali python -m src.registry.rollback --target-version 1 python -m src.features.benchmark --sizes 100000 500000 1000000 -# 60-second reproducible sprint demo +# Demo script bash scripts/demo.sh -# Tests โ€” 31 tests +# Tests pytest tests/ -q pytest tests/ -q -m "not requires_data" # CI mode (skips full-data replay) ``` --- -## Tests - -31 pytest tests, gated in CI (`.github/workflows/ci.yml` runs `dvc dag` + `pytest -m "not requires_data"` on every push): - -| File | Covers | -|---|---| -| `test_temporal_split.py` | per-source timestamp monotonicity, no-future-leak regression | -| `test_features_determinism.py` | Pandas == Dask bit-exact | -| `test_drift_detector.py` | PSI=0 on identical input, fires on 2ฯƒ shift, KS/PSI paths | -| `test_registry.py` | promote โ†’ rollback end-to-end, alias truly flips | -| `test_retrain_trigger.py` | N-consecutive debounce, single-fire ignored, counter reset | -| `test_api.py`, `test_data_loader.py`, `test_telemetry.py` | serving + DVC loader + telemetry round-trips | - ---- - -## Repo layout - -``` -Fraud-Detection-MLOps/ -โ”œโ”€โ”€ dvc.yaml # 6-stage pipeline -โ”œโ”€โ”€ docker-compose.yml # Postgres + MLflow + Redis + FastAPI -โ”œโ”€โ”€ app.py + pages/ # Streamlit; pages/4_Ops.py = MLOps dashboard -โ”œโ”€โ”€ .github/workflows/ci.yml # DVC DAG + pytest -โ”œโ”€โ”€ src/ -โ”‚ โ”œโ”€โ”€ train.py # temporal split + MLflow tracking (Day 1 fix) -โ”‚ โ”œโ”€โ”€ combine_datasets.py # txn_timestamp + chronological sort -โ”‚ โ”œโ”€โ”€ benchmark_fdb.py # prefer held-out sparkov_test.csv -โ”‚ โ”œโ”€โ”€ features/{engineer,benchmark}.py # Dask/Pandas features + throughput sweep -โ”‚ โ”œโ”€โ”€ registry/{promote,rollback,bench_rollback}.py -โ”‚ โ”œโ”€โ”€ drift/{detector,trigger,bench_retrain}.py -โ”‚ โ”œโ”€โ”€ serving/{api,shadow}.py -โ”‚ โ”œโ”€โ”€ telemetry/logger.py # 4-table SQLAlchemy logger -โ”‚ โ”œโ”€โ”€ tuning/{optuna_sweep,eval_best,targeted_fix,build_leaderboard}.py -โ”‚ โ”œโ”€โ”€ analysis/failure_modes.py -โ”‚ โ””โ”€โ”€ frontier/{llm_judge,compare_models,ablation}.py -โ”œโ”€โ”€ tests/ # 31-test suite -โ”œโ”€โ”€ models/fraud_model_tuned_fixed.pkl # champion -โ””โ”€โ”€ data/raw/{paysim,sparkov,sparkov_test}.csv -``` - ---- - ## Datasets | Dataset | Rows | Window | Use | |---|---:|---|---| -| `paysim.csv` | 6.36M | step-based sim | training (~83% of combined) | +| `paysim.csv` | 6.36M | step-based sim | training | | `sparkov.csv` | 1.30M | 2019-01 โ†’ 2020-06 | training window | -| `sparkov_test.csv` | 555,719 | 2020-06 โ†’ 2020-12 | **held-out OOT** (the honest benchmark) | +| `sparkov_test.csv` | 555,719 | 2020-06 โ†’ 2020-12 | held-out out-of-time | -DVC-tracked; `dvc pull` to materialize. Combined temporal train โ‰ˆ 6.13M rows. +DVC-tracked; `dvc pull` to materialize. The split is chronological per source โ€” the final 20% of each is held out. ---