Skip to content

benchmark-report.md

Latest commit

 

History

History
223 lines (163 loc) · 12.5 KB

benchmark-report.md

File metadata and controls

223 lines (163 loc) · 12.5 KB
Raw

DeepFilterNet Streaming Engine Benchmark Report

Overview

This report compares four streaming engine implementations for DeepFilterNet real-time speech enhancement on Apple Silicon (M-series Mac).

Historical comparison: the MLX, CPU, Hybrid, and batch Core ML numbers below were recorded before the stateful Core ML graph existed. The current Core ML streaming result is 1.548 seconds for this 52.127646-second file (33.7x real time), 0.288 ms steady compute per hop, and 0.999993 correlation to PyTorch. See live-device-benchmark.md and outputs/benchmark/coreml_streaming_benchmark.json.

For the later wall-clock microphone simulation on both Mac and M1 iPad Pro, see live-device-benchmark.md. That device test supersedes the real-time engine recommendation in this older throughput-focused report.

Engine Description Implementation
MLX GPU Current MLX implementation with Metal kernel fusion GPU dispatch per-hop, ~200+ ops/hop
CPU + Accelerate DSP pipeline using Accelerate/vDSP/BLAS Zero dispatch overhead, direct function calls
Hybrid (compile + CPU GRU) MLX with compile() for conv blocks, GRU gate fusion disabled Fewer GPU dispatches via graph compilation
DeepFilterNet3 Core ML Core ML with all compute units enabled Explicit-state prediction per 10 ms hop

Test Configuration

  • Audio: input_extract_short_48k.wav — 52.1 seconds (2,502,127 samples, 5,212 hops)
  • Model: DeepFilterNet3 (iky1e/DeepFilterNet3-MLX for MLX/CPU/Hybrid, iky1e/DeepFilterNet3-Streaming-CoreML for Core ML)
  • Sample Rate: 48,000 Hz
  • Hop Size: 480 samples (10ms)
  • Precision: FP32 (MLX/CPU/Hybrid), FP16 (CoreML model internals)
  • Warmup Runs: 1
  • Measurement Runs: 3 (median selected)

Metrics Measured

  1. Initialization Time: Time to prepare the engine (extract weights, compile graphs, etc.)
  2. Startup Latency: Average per-hop time for the first 2 output-producing hops
  3. Steady-State Latency: Median per-hop time after warmup
  4. Total Streaming Time: End-to-end time to process the entire audio file
  5. Real-Time Factor: Audio duration / processing time (must be > 1.0 for real-time)
  6. Offline Time: Batch processing time (non-streaming)

Results

Streaming Performance

Engine Init (ms) Startup (ms/hop) Steady (ms/hop) Total (s) RT Factor
MLX GPU 0.0 2.445 1.982 10.626 4.9x
CPU + Accelerate 16 7.774 8.024 41.886 1.2x
Hybrid (compile + CPU GRU) 16 2.425 1.978 10.589 4.9x
DeepFilterNet3 Core ML (latest throughput run) 30 0.312 0.264 1.494 34.9x

The current Core ML row uses the published stateful DeepFilterNet3-Streaming.mlpackage graph.

Output Quality

Comparison RMS Diff Max Diff Correlation
MLX vs PyTorch 0.000161 0.000039 0.9999984
Hybrid vs MLX 0.000000 0.000000 1.0000000
CPU vs MLX 0.012168 0.057427 0.9920989
CPU vs PyTorch 0.012199 0.057612 0.9920596
CoreML streaming vs PyTorch 0.000337 RMSE 0.006281 0.9999933

Offline Performance

Engine Time (s) RT Factor Notes
Hybrid (Accelerate GRU) 0.467 112x Compiled graphs + batch GPU input proj + CPU vDSP hidden proj
MLX GPU (Accelerate GRU) 0.550 95x Metal fused kernels + Accelerate GRU
MLX GPU (GPU-only GRU) 1.555 33.5x Pre-optimization baseline
Hybrid (GPU-only GRU) 1.548 33.7x Pre-optimization baseline
Core ML (historical batch conversion) 29.4 1.8x Superseded; not part of the published runtime

Head-to-Head: DeepFilterNet-mlx vs mlx-audio-swift

Audio Length Mode DeepFilterNet-mlx mlx-audio-swift Winner
10s Offline 0.113s (88x RT) 0.18s (56x RT) Ours 1.6x
52s Offline 0.466s (112x RT) 0.60s (87x RT) Ours 1.3x
1.83hr Offline 89.2s (74x RT) 86.5s (76x RT) ~tied
10s Streaming 2.98s (3.4x RT) 2.86s (3.5x RT) ~tied
52s Streaming 18.4s (2.8x RT) 15.3s (3.4x RT) Theirs 1.2x

Quality: bit-identical outputs (correlation 1.0000000), 0.9999984 vs PyTorch reference.

Analysis

The Dispatch Overhead Problem

DeepFilterNet streaming processes audio in 10ms hops. Each hop runs ~200+ individual tensor operations through the neural network. The actual compute per hop is sub-MFLOP — microseconds of math. However, GPU dispatch overhead (Metal command encoding, queue submission, synchronization) adds ~10-15 microseconds per operation, totaling ~2-3ms per hop.

Accelerate-Optimized GRU (Key Optimization)

The biggest offline optimization was porting the Accelerate-optimized GRU from mlx-audio-swift. The GRU is the main bottleneck for offline processing because it requires sequential per-timestep computation (each step depends on the previous hidden state).

Before: Each GRU timestep dispatched a GPU matmul via MLX.addMM() — for 5,000+ frames across 3 GRU layers, that's ~15,000 tiny GPU dispatches that can't be parallelized.

After: Batch input projection x @ W_ih on GPU (single large matmul), then per-step hidden projection h @ W_hh on CPU via vDSP_mmul with scalar GRU gate computation. This eliminates all sequential GPU dispatches.

Result: 3.3x offline speedup (33.5x → 112x RT for 52s audio).

Note: This optimization only helps offline (batch) processing. For streaming, each hop has only 1 GRU step, and the CPU↔GPU transfer overhead per hop (~2ms from eval/asArray) exceeds the GPU dispatch savings. Streaming keeps the Metal fused GRU gates.

Engine Comparison

Hybrid — 2.0ms/hop streaming, 112x RT offline (fastest)

  • Streaming: compiled graphs + Metal kernel fusion (GRU gate fusion disabled)
  • Offline: compiled conv graphs + Accelerate-optimized GRU = 0.467s for 52s audio
  • Bit-identical to MLX GPU output (correlation = 1.0000000)

MLX GPU — 2.0ms/hop streaming, 95x RT offline

  • Streaming: lazy evaluation + Metal kernel fusion + fused GRU gates
  • Offline: Accelerate-optimized GRU (batch GPU input proj + CPU hidden proj) = 0.550s
  • Near-perfect quality: 0.9999984 correlation with PyTorch reference

CPU + Accelerate (Full Inference) — 8.024ms/hop, 1.2x RT

  • Full neural network reimplementation using Accelerate framework (BLAS, vDSP)
  • Includes complete encoder conv chain (depthwise + pointwise), GRU layers, ERB/DF decoders, and deep filtering
  • Current implementation uses naive scalar loops for conv2d — not yet vectorized with BLAS
  • Output quality: 0.992 correlation with MLX output (correct shape, minor numerical differences)
  • Zero GPU dispatch overhead — all computation on CPU
  • Potential: vectorizing conv loops with CBLAS could bring latency under 1ms/hop

Hybrid (compile + CPU GRU) — 1.978ms/hop, 4.9x RT

  • Uses the MLX streamer with compile() enabled and Metal GRU gate fusion disabled
  • Streaming performance is nearly identical to MLX GPU (~2ms/hop)
  • Offline is the fastest configuration: 0.467s (112x RT) due to compiled conv graphs + Accelerate GRU
  • Output is bit-identical to MLX GPU (correlation = 1.0000000)

DeepFilterNet3 Core ML - 0.264ms/hop, 34.9x RT

  • Uses a converted fixed-shape one-hop graph with explicit recurrent states
  • Emits enhanced audio throughout capture after the fixed lookahead
  • Matches PyTorch at 0.999993 correlation despite FP16 graph execution
  • Keeps model output tensors alive only for the current autorelease scope and copies strided recurrent outputs into persistent state buffers

Offline vs Streaming

The offline path processes all frames in a single batch with Accelerate-optimized GRU:

  • Offline (Hybrid + Accelerate GRU): 0.467s for 52.1s audio = 112x RT (fastest)
  • Offline (MLX + Accelerate GRU): 0.550s for 52.1s audio = 95x RT
  • Streaming (MLX): 18.4s for 52.1s audio = 2.8x RT
  • CoreML historical batch conversion: 29.4s for 52.1s audio = 1.8x RT

The ~56x performance gap between offline and streaming is due to: (1) batch GPU operations vs per-hop dispatch overhead, (2) Accelerate GRU eliminating sequential GPU dispatches in offline mode, (3) compiled graph fusion reducing conv dispatch count.

Recommendations

When to use each engine:

Use Case Recommended Engine Reason
Offline batch processing Hybrid Fastest: 0.467s (112x RT) with compiled graphs + Accelerate GRU
Library/Framework MLX GPU (throughput) Production-ready, well-tested, 0.550s (95x RT) offline
Real-time streaming on current Mac Core ML 0.288ms/hop throughput measurement; 5.622ms live p99
Existing M1 iPad result MLX GPU Core ML must be rerun on that device

Key Takeaways

  1. For streaming, dispatch overhead dominates compute. The model is tiny (~1.5 MFLOP/hop) but launching 200+ GPU operations per hop costs ~2ms vs ~8ms for naive CPU inference. Vectorized CPU inference would be significantly faster.

  2. CPU with full inference works but is slow with naive loops. At roughly 8ms/hop, the current CPU engine is barely real-time. Its scalar Swift conv2d loops leave optimization opportunities, but their benefit must be measured rather than assumed.

  3. The Hybrid approach is the fastest for offline. With Accelerate GRU enabled, the Hybrid configuration (compiled conv graphs + Accelerate GRU) achieves 0.467s (112x RT), beating MLX GPU's 0.550s (95x RT). The compiled graphs reduce conv dispatch overhead that the throughput config's Metal fused GRU gates don't address. Output is bit-identical to MLX GPU.

  4. Core ML now has a valid live path. The converted model exposes all recurrent state and the Swift engine owns the remaining temporal DSP state. Execution-unit assignment and energy use still require an Instruments trace.

  5. FP16 Core ML preserves output quality in the stateful graph. Correlation is 0.999993, RMS ratio is 1.000380, and SER is 48.65 dB against PyTorch.

  6. Offline mode is excellent with Accelerate GRU. At 112x RT (was 33.5x before optimization), the Hybrid configuration with Accelerate-optimized GRU eliminates sequential GPU dispatches and compiles conv graphs. This matches or exceeds mlx-audio-swift performance.

  7. CPU quality is good but not perfect. The 0.992 correlation between CPU and MLX outputs indicates the reimplementation is correct in structure, with minor floating-point differences from accumulation order and potential subtle layer implementation differences.

Implementation Status

Engine Streaming Offline Full Inference Quality (corr.) Notes
MLX GPU Complete Complete Yes 0.9999984 Production-ready baseline
CPU + Accelerate Complete N/A Yes 0.992 Full reimplementation, naive loops
Hybrid Complete Complete Yes 1.0000000 Delegates to MLX streamer with compile()
DeepFilterNet3 Core ML Complete Complete Yes 0.999993 Explicit-state one-hop FP16 graph; separate legacy batch graph

Reproducing Results

# Build the benchmark tool
swift build -c release --product deepfilternet-benchmark

# Download the stateful Core ML model (automatic when omitted by the public API)
hf download iky1e/DeepFilterNet3-Streaming-CoreML \
  --local-dir /path/to/DeepFilterNet3-Streaming-CoreML

# Run full benchmark (all 4 engines)
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines all \
  --warmup 1 \
  --runs 3 \
  --save-output \
  --output-dir benchmark_results

# Generate graphs
python3 Scripts/Visualization/generate_benchmark_graphs.py benchmark_results/benchmark_<timestamp>.json

# Run specific engines only
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --engines mlx,cpu \
  --runs 3

Graphs

(Generated by Scripts/Visualization/generate_benchmark_graphs.py)

  • streaming_total_time.png — Total streaming processing time per engine
  • per_hop_latency.png — Startup vs steady-state per-hop latency
  • hop_timeline.png — Per-hop timing over audio duration
  • realtime_factor.png — Real-time factor comparison
  • init_time.png — Engine initialization time
  • offline_comparison.png — Offline processing comparison