Multimodal Stress Detection System v3.1

🧠 Real-time, non-invasive stress & drowsiness detection using only a standard webcam. Combines facial action unit analysis with remote photoplethysmography (rPPG) in a privacy-first, fully on-device pipeline.

---

✨ What It Does

Capability	How
💓 Live Heart Rate	Remote PPG via webcam green channel — POS algorithm with automatic CHROM fallback, Kalman-filtered BPM
📊 HRV (RMSSD)	Inter-beat interval analysis with IBI outlier rejection (capped 150 ms)
👁️ Drowsiness Detection	PERCLOS proxy — sustained low Eye Aspect Ratio (per-user adaptive threshold) > 2.5 s → amber flashing banner + 3× alarm beep
🧩 11-D Multimodal Fusion	9 behavioral (EAR, brow, head pose, blinks, lip depression, jaw clenching) + 2 physiological (BPM, HRV) → RF + ExtraTrees ensemble
🎯 Continuous Stress Score	Rule-based 0–100 score (BPM + HRV + AU4 brow + EAR) — always live, independent of signal quality
🔄 State Recovery	Label-buffer vote-decay + score safety valve prevents stuck "Stressed" label
🌐 Zero Cloud	All inference on-device. No API keys. No data leaves your machine.
🔌 Webcam Reconnect	Automatic reconnect with exponential failure cutoff (10 retries)
📄 HTML Reports	Interactive post-session dashboard generated from CSV logs

---

🖥️ Live Demo Screenshot

Run python main.py --no-calib for a quick test without the 15-second calibration phase.

---

🏗️ Architecture

┌────────────────────────────────────────────────────────────────┐
│                     main.py  (v3.1)                            │
│                                                                │
│   Webcam ──► Worker Thread (daemon)                            │
│              │                                                │
│        ┌─────┴─────────────────────┐                          │
│        ▼                           ▼                          │
│   AUExtractor               RPPGExtractor                     │
│   (MediaPipe 468-LM)        (rPPG — Green ch.)                │
│   • EAR / Blink             • POS + CHROM fallback             │
│   • Brow furrow (AU4)       • Kalman-filtered BPM              │
│   • Head pose               • 45–150 BPM physiological gate    │
│   • Lip depression, jaw     • IBI peaks → HRV                  │
│   • PERCLOS drowsiness      • SNR quality gate (tightened)     │
│        │                           │                          │
│        └───────────┬───────────────┘                          │
│                    ▼                                          │
│            StressClassifier                                   │
│            11-D feature vector                                │
│            RF + ExtraTrees VotingClassifier (soft)             │
│            + CalibratedClassifierCV (isotonic)                 │
│            + Rule-based score 0–100                            │
│                    │                                          │
│         ┌──────────┼──────────┐                               │
│         ▼          ▼          ▼                               │
│      HUD (cv2)  Session  HTML Report                          │
│                  CSV log  (post-session)                      │
└────────────────────────────────────────────────────────────────┘

---

🚀 Quick Start

Prerequisites

• Python 3.10+
• Webcam (built-in or USB)
• Windows (for audio alerts via winsound; Linux/Mac require minor modification)

1. Clone & Install

git clone https://github.com/SheeshDarth/Stress-Detection-System.git
cd Stress-Detection-System

python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # Linux / macOS

pip install -r requirements.txt

2. Download MediaPipe Model

python -c "
import urllib.request, os
os.makedirs('models', exist_ok=True)
urllib.request.urlretrieve(
    'https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task',
    'models/face_landmarker.task'
)
print('Downloaded models/face_landmarker.task')
"

3. Run

# Full run with 15-second calibration phase (recommended)
python main.py

# Skip calibration (quick test)
python main.py --no-calib

# Use a specific camera index
python main.py --camera 1

The classifier auto-trains on synthetic data on first run. To retrain manually, or to re-extract features from the UBFC-Phys dataset (download_dataset.py fetches it via kagglehub):

python train.py                  # retrain on synthetic + any cached UBFC features
python train.py --extract        # re-extract UBFC-Phys features first

4. Controls

Key	Action
ESC / Q	Quit
R	Reset all signal buffers
M	Toggle face mesh overlay
S	Save screenshot
L	Start / stop CSV session logging
A	Toggle audio alerts

---

🧪 Testing

# Full test suite (68 tests)
pytest tests/ -v

# Unit tests only (fast, no webcam needed)
pytest tests/test_classifier.py tests/test_rppg.py tests/test_au_extraction.py -v

# Integration test (generates synthetic video, runs full pipeline headlessly)
pytest tests/test_integration.py -v -m integration

# Headless sanity check (no webcam required)
python sanity_check.py

# Ablation study, baselines, significance tests, temporal-stability diagnostic
python run_ablation.py --save results/ablation_results.csv

---

📁 Project Structure

Stress-Detection-System/
├── main.py                      # Real-time webcam application (v3.1)
├── train.py                     # Training pipeline
├── run_ablation.py              # Ablation study, baselines, significance tests
├── sanity_check.py              # Headless pipeline verification
├── download_dataset.py          # UBFC-Phys / WESAD dataset downloader
├── requirements.txt
├── README.md
├── context.md                   # Full technical changelog (v3.0 -> v3.1)
│
├── models/
│   ├── face_landmarker.task     # MediaPipe model (download separately)
│   ├── stress_model.pkl         # Trained classifier (auto-generated)
│   └── stress_model_metrics.json
│
├── results/
│   └── ablation_results_summary.json  # Ablation + baseline + significance results
│
├── src/
│   ├── data_loader.py           # UBFC-Phys dataset loader
│   ├── session_logger.py        # CSV session logging with finalise
│   ├── report_generator.py      # HTML post-session report generator
│   ├── visual/
│   │   └── au_extraction.py     # MediaPipe AU extraction (+ lip/jaw, v3.1)
│   ├── physiological/
│   │   └── rppg_extraction.py   # rPPG pipeline (+ CHROM, Kalman, v3.1)
│   └── fusion/
│       └── classifier.py        # 11-D multimodal fusion classifier (v3.1)
│
├── tests/
│   ├── test_au_extraction.py
│   ├── test_rppg.py
│   ├── test_classifier.py
│   ├── test_session_logger.py
│   └── test_integration.py      # Synthetic video integration test
│
└── logs/                        # Auto-created; session CSVs + app log

---

📊 Model Performance

Full 11-D system (RF + ExtraTrees soft-voting ensemble, CalibratedClassifierCV), 5-fold cross-validation on 2,300 samples:

Metric	Value
Accuracy	98.52% ± 0.46%
F1 Score	98.53% ± 0.45%
ROC AUC	99.95% ± 0.04%

---

🔬 Evaluation & Ablation Study

run_ablation.py runs an ablation + baseline + statistical-significance comparison against the full 11-D system (results in results/ablation_results_summary.json). Every row is 5-fold cross-validated on the same 2,300-sample dataset as the full system.

Ablation — does each component earn its place?

Configuration	Dim	Accuracy	F1	vs. full system
A1 — Full system (11D, RF+ET)	11	98.52%	98.53%	—
A2 — No lip/jaw (9D, RF+ET)	9	97.39%	97.43%	full system better, p = 0.033
A3 — Behavioral only (9D, RF+ET)	9	98.39%	98.40%	no significant difference, p = 0.071
A4 — Physiological only (2D, RF+ET)	2	86.43%	86.30%	full system better, p = 0.0003
A5 — RF only, no ExtraTrees (11D)	11	98.26%	98.28%	full system better, p = 0.032

Baselines — how do other classifiers do on the same 11-D features?

Configuration	Accuracy	F1	vs. full system
B1 — SVM-RBF	99.17%	99.17%	no significant difference, p = 0.157
B2 — Logistic Regression	98.74%	98.74%	no significant difference, p = 0.514
B3 — k-NN (k=5)	98.26%	98.23%	no significant difference, p = 0.444
B4 — AU-only RF (7D behavioral)	96.61%	96.66%	full system better, p = 0.014
B5 — BPM threshold (1D)	78.22%	78.89%	full system better, p < 0.0001

Reading it honestly: the lip/jaw features and multi-classifier ensembling each contribute a statistically significant lift over an ablated version, and dropping to physiological-only or a single naive threshold costs real accuracy. Several strong baselines (SVM-RBF, logistic regression, k-NN) land within noise of the full system — RF + ExtraTrees was kept because it ties the strongest baselines while providing calibrated probabilities and interpretable Gini feature importances, not because it's the single best number in the table.

Temporal stability — does the hysteresis + override logic matter?

Configuration	Label flip rate	Accuracy
With hysteresis + score override	0.0	100%
With hysteresis, no override	0.0	100%
No hysteresis, no override	5.0	95.8%

In the synthetic transition-stability diagnostic, removing the 40-vote hysteresis buffer and rule-based override reintroduces label flapping and measurably drops accuracy — the reason both exist in main.py rather than a raw per-frame prediction.

---

🔬 Signal Processing Details

Behavioral Pipeline (Visual)

1. MediaPipe FaceLandmarker → 468 3-D landmarks at 25+ FPS (CPU-only)
2. Eye Aspect Ratio (EAR) → adaptive blink threshold (calibrated from first 90 frames), reused for per-user drowsiness sensitivity
3. Brow furrow (AU4) → normalized inner-eyebrow distance
4. Head pose → nose-tip displacement variance and mean movement
5. Lip depression & jaw clenching → AU15 proxy and chin-to-nose distance variance (new in v3.1)
6. PERCLOS → consecutive frames below the adaptive EAR threshold → drowsiness flag at 2.5 s
7. 10-second windowed aggregation → 9 statistical features

Physiological Pipeline (rPPG)

1. ROI extraction → forehead + cheek regions via convex hull of landmarks
2. Spatial RGB mean → per-frame in a 10-second rolling buffer
3. POS algorithm → Plane-Orthogonal-to-Skin projection, with automatic CHROM fallback when POS SNR < 1.0
4. Butterworth bandpass → widened to 0.75–2.5 Hz (45–150 BPM), covering athletic and elevated-HR users
5. Welch PSD + Kalman filter → dominant frequency → BPM, smoothed by a 1-D Kalman filter (replaces the earlier rolling median for lower lag)
6. IBI outlier rejection → ±25% of median, RMSSD capped at 150 ms
7. SNR quality gate → "Poor / Fair / Good" (tightened thresholds: Good > 5.0, Fair > 1.0)

Stress Scoring (Dual-Mode)

Mode	When Active	Basis
Rule-based score (0–100)	Always	Physiological thresholds on BPM, HRV, brow, EAR
ML binary label	Signal ≥ "Fair" + buffer ≥ 35%	RF + ExtraTrees ensemble with 75% hysteresis over 40-vote window
Score override	Score < 28	Snaps to "Normal", clears stale ML votes immediately

See context.md for the full file-by-file v3.0 → v3.1 changelog.

---

🔒 Privacy & Security

• 100% on-device — no video, biometric data, or stress readings ever transmitted
• Session logs stored locally in logs/ (auto-created, owner-accessible)
• Model loaded from local .pkl file — no network calls at inference time
• One-time internet access: MediaPipe model download and optional UBFC/WESAD dataset download

---

📚 References

• Wang, W. et al. (2017). "Algorithmic principles of remote-PPG." IEEE TBME, 64(7), 1479–1491
• Soukupová, T. & Čech, J. (2016). "Real-time eye blink detection using facial landmarks." CVWW
• de Haan, G. & Jeanne, V. (2013). "Robust pulse rate from chrominance-based rPPG." IEEE TBME
• Bobbia, S. et al. (2019). "Unsupervised skin tissue segmentation for rPPG." Pattern Recognition Letters
• UBFC-Phys Dataset — Meziati Sabour et al. (2021)

---

📝 License

MIT License — see LICENSE file.

---

Built with ❤️ for IPCV Project — all inference on-device, zero cloud dependencies.