Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors

Authors: Lukáš Prokeš*, Vojtěch Staněk*, Anton Firc, Kamil Malinka — Security@FIT, Brno University of Technology, Brno, Czech Republic; {xproke14, istanek, ifirc, malinka}@fit.vut.cz (* equal contribution)

Abstract: Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, speakers, and recording conditions, with labels unavailable during operation. This makes it difficult to know when a detector operates in familiar conditions and can be relied on. We propose an unsupervised drift monitor that operates on the internal embeddings of a deployed detector. The monitor compares each input against bona fide and spoof references using regularized Mahalanobis distances. These distances define trusted, ambiguous, and novel regions of the detector embedding space, allowing unfamiliar inputs to be routed for review. We evaluate the monitor on four modern detectors on three datasets. The monitor raises alerts for dataset mismatches and substantially reduces the security-critical case of accepting deepfakes at the cost of reduced automatic coverage. The results also show that drift affects both classes: bona fide recordings can move away from the training reference, increasing alert rates even in in-domain settings. Our findings suggest that deployed deepfake speech detectors should be treated not only as a binary classification, but also as a reliability problem.

This repository is a code snapshot accompanying the paper, submitted to IEEE WIFS 2026.

Repository structure

Drift-Monitor
├── analysis/                offline analyses & figures, run on saved .npz profiles
│                            (two-reference categorization, deployment simulator,
│                             covariance diagnostics/sweep, monitor-region overlay, ...)
├── augmentation/            data augmentation (RawBoost, RIR, codec, noise)      [detector codebase]
├── classifiers/             classification heads (single-input FF + pair variants) [detector codebase]
├── datasets/                dataset classes (ASVspoof 2019/2021/5, In-the-Wild, ...) [detector codebase]
├── extractors/              SSL feature extractors (XLS-R, WavLM, HuBERT, Wav2Vec2) [detector codebase]
├── feature_processors/      pooling backends (MHFA, AASIST, SLS, mean)          [detector codebase]
├── trainers/                training/eval loops [detector codebase]
│   └── DriftMonitor.py      drift monitoring layer (metrics, profiles, plots)
├── multidetector/           one-command pipeline for the full 2×2 experiment (PBS)
├── scripts/                 single-run PBS job scripts + eval profile registry
├── docs/                    detailed script reference + original detector codebase README
├── common.py                model/dataset registry, dataloaders
├── config.py                dataset paths & batch sizes per environment
├── drift_config.py          metadata paths/columns per dataset, known attack IDs
├── cov_utils.py             shared covariance regularization + threshold fitting
├── parse_arguments.py       CLI argument parsing
├── train_and_eval.py        detector training entry point
├── eval.py                  detector evaluation + embedding/eval-profile export
├── generate_drift_reference.py          build bona fide / spoof reference profiles
├── extract_asvspoof5_train_bonafide.py  helper: ASVspoof 5 train bona fide embeddings
└── finetune.py              detector fine-tuning entry point

The detector training/evaluation framework (extractors, feature processors, classifiers, datasets, trainers, augmentation) is based on the deepfake speech detection codebase by Vojtěch Staněk, developed for his Master's thesis at FIT BUT; his original README is preserved at docs/ORIGINAL_README.md. This repository adds the drift-monitoring layer on top of that framework — the detectors themselves are treated as fixed components and are not modified.

Requirements

Python 3.10, PyTorch ≥ 2.2 (see pytorch.org for the install command matching your CUDA version):

conda create -n drift_monitoring python=3.10
conda activate drift_monitoring
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt

Usage

The workflow has three stages (GPU stages 1–2, CPU stage 3). All experiments in the paper were run on the Metacentrum grid via the PBS job scripts in scripts/ and multidetector/ — those contain site-specific storage paths; override them via the documented environment variables (PROJECT_DIR, STORAGE_BASE, ...) or run the Python entry points directly.

1. Build reference profiles (GPU)

Runs the detector over its own training set and saves per-class reference profiles (reference_bonafide.npz, reference_spoof.npz: embeddings, mean, precision):

python3 generate_drift_reference.py --metacentrum \
    --extractor XLSR_300M --processor MHFA --classifier FF \
    --checkpoint <detector_checkpoint>.pt

2. Evaluate on a dataset (GPU)

Runs the detector over an evaluation dataset and saves an eval profile (eval_profile_*.npz: embeddings + logits + file names) plus summary metrics:

python3 eval.py --metacentrum \
    --extractor XLSR_300M --processor MHFA --classifier FF \
    --checkpoint <detector_checkpoint>.pt \
    --dataset ASVspoof5Dataset_single

3. Offline analyses (CPU, no model re-run needed)

All monitor logic runs offline on the saved .npz profiles:

• analysis/per_cluster_classification.py — two-reference Mahalanobis categorization (known / borderline / novel), covariance regularization and threshold options, per-sample distance export
• analysis/drift_detector_simulator.py — deployment simulation: trusted/alert zones, correctly-trusted / alerted / wrongly-trusted outcomes
• analysis/detector_baseline_fnr.py — detector-only error rates (no monitor)
• analysis/analyze_covariance_conditioning.py, analysis/cov_regularization_sweep.py — covariance conditioning diagnostics and the regularization ablation

See docs/scripts_reference.md for the full catalog including the exploratory analyses (Wasserstein distances, bootstrap CIs, per-attack EER, threshold studies).

4. Full 2×2 experiment

multidetector/ automates the whole protocol (references → 3 evals → analyses) per detector under identical conditions, driven by the detectors.tsv manifest:

cd multidetector
./drive_detector.sh FF_MHFA_2019   # submits the whole PBS dependency chain

See multidetector/README.md.