Skip to content

dpdfnet.md

Latest commit

 

History

History
207 lines (163 loc) · 11.1 KB

dpdfnet.md

File metadata and controls

207 lines (163 loc) · 11.1 KB
Raw

DPDFNet Port

Model identity

The model informally described as "DeepFilterNet 4" is CEVA's DPDFNet, not a fourth release of the Rust DeepFilterNet project. DPDFNet adds dual-path RNN blocks to the DeepFilterNet2 encoder. The intra path is a bidirectional GRU over frequency; the inter path is a causal GRU over time.

Authoritative sources:

Published, self-contained Swift model repositories:

Profiles

Profile Rate FFT / hop DPRNN blocks Encoder bins Status
DPDFNet-4 16 kHz 320 / 160 4 ERB 8, DF 48 Offline and streaming implemented
DPDFNet-8_48khz_hr 48 kHz 960 / 480 8 magnitude 40, DF 48 Offline and streaming implemented

The HR model is not a larger drop-in DPDFNet-4. Its dedicated path implements the empirical 481-bin magnitude normalization, 3/2/2 encoder strides, 2/2/3 subpixel decoder strides, extra encoder and decoder embedding projections, direct magnitude masking, and its own streaming state layout.

MLX recurrent implementation

The original pure-MLX port expressed each frequency GRU step as separate MLX operations. These matrices are small enough that GPU dispatch overhead dominated the arithmetic. The production path now uses custom fused DPRNN Metal kernels:

  • reset, update, candidate, and hidden-state operations are fused
  • hidden state remains local during a complete forward or reverse sequence
  • ERB and DF branches run concurrently in one paired dispatch
  • FP32 accumulation is retained for fidelity
  • queued time hops can be processed without changing recurrent ordering

The Accelerate backend remains available. It uses CPU BLAS and vector math for the recurrent core while MLX handles the convolutional and decoder graph.

Core ML streaming implementation

DeepFilterNetCoreMLStreamer runs the complete official DPDFNet neural graph one 10 ms spectrum hop at a time. Swift/Accelerate performs only the streaming STFT and inverse STFT; normalization, encoder, every DPRNN block, decoder, magnitude masking, deep filtering, and recurrent state transitions remain in the converted Core ML graph. Both DPDFNet-4 and the separate DPDFNet-8 48 kHz HR architecture are supported.

The converter produces three graph variants for each profile:

  • Streaming-FP32: explicit FP32 state and activations; this is the production default because it passes the strict full-audio ONNX fidelity gate.
  • Streaming: explicit FP16 state and activations.
  • Streaming-State: FP16 state retained through Core ML MLState on macOS 15 and iOS 18 or newer.

FP16 did not provide a meaningful live-latency advantage on the benchmark Mac and failed the strict spectrogram-correlation gate. It remains selectable for experimentation, but benchmark and quality reports use the FP32 graph.

# Convert both official checkpoints and compile deployable .mlmodelc bundles.
python Scripts/Conversion/convert_dpdfnet_to_coreml.py \
  --model-root /path/to/dpdfnet-assets --profile all --force

# Benchmark the quality-qualified graph. --model selects the converted MLX
# profile whose configuration identifies DPDFNet-4 or DPDFNet-8 HR.
.build/arm64-apple-macosx/release/deepfilternet-benchmark /path/to/test.wav \
  --model /path/to/dpdfnet4 \
  --engines dpdf-coreml-fp32 \
  --coreml-model-dir /path/to/dpdfnet4 \
  --coreml-compute-units all --live --live-duration 10 --runs 3

Core ML compute-unit profiling covered .all, CPU+GPU, CPU+Neural Engine, and CPU-only configurations. .all is retained as the throughput/median-latency default because it produced the lowest median for both profiles. Explicit CPU+Neural Engine or CPU-only placement produced tighter tails in some short runs and remains selectable for applications that prioritize jitter over the lowest median. Core ML decides actual operation placement internally, so these results should be described as Core ML rather than as guaranteed ANE execution.

Streaming semantics

The official stateful graph has two distinct two-hop delays:

  1. The causal encoder predicts a mask for the spectrum buffered by two hops.
  2. The five-frame deep-filter stage delays coefficients by another two hops.

The Swift streamer mirrors that ordering with separate raw-spectrum, masked-spectrum, and coefficient state. Its compensated algorithmic delay is five hops: one STFT overlap hop plus four model hops, or 50 ms. This fixed model/DSP delay is separate from per-hop compute latency and device I/O buffers.

Fidelity gate

Scripts/Benchmarking/benchmark_dpdfnet.py generates the official ONNX output and compares every Swift output using exact output length, finite-value checks, waveform correlation and error, gain and offset, spectrogram error, phase error, RMS envelope error, band-energy error, and clipping count.

The default gate requires exact length, zero best lag, correlation at least 0.999, RMSE no more than 0.002, gain within 0.5%, and spectrogram correlation at least 0.995. A timing is reported as valid only when its output passes this gate.

Latest Extract-short.wav results:

Profile Variant Correlation RMSE Gain Spectrogram correlation Pass
DPDFNet-4 Swift offline MLX 0.99994749 0.00094719 1.00036895 0.99658368 Yes
DPDFNet-4 Swift stream MLX 0.99994761 0.00094606 1.00035863 0.99658128 Yes
DPDFNet-4 Swift stream MLX + Accelerate 0.99994761 0.00094606 1.00035863 0.99658128 Yes
DPDFNet-4 Swift stream Core ML FP32 (60 s) 0.99999999 0.00001004 0.99999747 0.99999980 Yes
DPDFNet-8 HR Swift offline MLX 0.99996559 0.00077253 0.99865285 0.99831167 Yes
DPDFNet-8 HR Swift stream MLX 0.99996590 0.00076840 0.99870450 0.99837320 Yes
DPDFNet-8 HR Swift stream MLX + Accelerate 0.99996590 0.00076840 0.99870451 0.99837320 Yes
DPDFNet-8 HR Swift stream Core ML FP32 (60 s) 0.99999966 0.00007115 1.00002449 0.99996813 Yes

Detailed current Core ML fidelity metrics and plots are in the DPDFNet-4 quality report and DPDFNet-8 HR quality report.

Throughput snapshot

The benchmark measures full Swift process wall time, including model loading. Official ONNX inference excludes session creation. The canonical input is 52.128 seconds long.

Profile Variant Runs Average RTF Speed
DPDFNet-4 Official ONNX inference 1 8.123 s 0.1558 6.42x real time
DPDFNet-4 Swift offline 3 4.186 s 0.0803 12.45x real time
DPDFNet-4 Swift stream MLX 3 28.830 s 0.5531 1.81x real time
DPDFNet-4 Swift stream MLX + Accelerate 1 31.291 s 0.6003 1.67x real time
DPDFNet-8 HR Official ONNX inference 1 21.697 s 0.4162 2.40x real time
DPDFNet-8 HR Swift offline 3 10.065 s 0.1931 5.18x real time
DPDFNet-8 HR Swift stream MLX 1 51.463 s 0.9873 1.01x real time
DPDFNet-8 HR Swift stream MLX + Accelerate 1 49.399 s 0.9477 1.06x real time

The first-run-over-30-seconds rule limits slow variants to one throughput run. Results are machine-specific and retain raw samples in outputs/benchmark/dpdfnet_60s_benchmark.json and outputs/benchmark/dpdfnet8_hr_60s_benchmark.json.

An additional exact-minute run uses the same Extract-short.wav content and silence-pads it after resampling from 52.128 to 60.000 seconds. Fast variants use three measured runs; variants whose first run exceeds 30 seconds use one:

Profile Variant Time RTF Speed Fidelity
DPDFNet-4 Official ONNX inference 8.719 s 0.1453 6.88x Pass
DPDFNet-4 Swift offline 5.170 s 0.0862 11.60x Pass
DPDFNet-4 Swift stream MLX 23.079 s 0.3847 2.60x Pass
DPDFNet-4 Swift stream MLX + Accelerate 23.796 s 0.3966 2.52x Pass
DPDFNet-4 Swift stream Core ML FP32 13.953 s 0.2326 4.30x Pass
DPDFNet-8 HR Official ONNX inference 22.464 s 0.3744 2.67x Pass
DPDFNet-8 HR Swift offline 11.226 s 0.1871 5.34x Pass
DPDFNet-8 HR Swift stream MLX 40.209 s 0.6701 1.49x Pass
DPDFNet-8 HR Swift stream MLX + Accelerate 47.436 s 0.7906 1.26x Pass
DPDFNet-8 HR Swift stream Core ML FP32 38.664 s 0.6444 1.55x Pass

Pure MLX is now 3.0% faster than the hybrid path for DPDFNet-4 aggregate streaming throughput and 15.2% faster for HR. Relative to the previous exact minute results, pure MLX improves by 32.4% for DPDFNet-4 and 34.1% for HR. See outputs/benchmark/dpdfnet_models_60s_throughput.md.

Real-time latency

The live-clock benchmark submits one 10 ms hop every 10 ms. Hop compute latency is the synchronous wait between giving the model a hop and receiving its output; it excludes the fixed 50 ms model/DSP delay. Audio age includes fixed delay, compute time, and accumulated queue delay.

Profile Backend Hop p50 Hop p95 Hop p99 Deadline misses
DPDFNet-4 MLX 8.798 ms 13.426 ms 16.045 ms 1858/3000
DPDFNet-4 MLX + Accelerate 6.038 ms 10.078 ms 11.511 ms 273/3000
DPDFNet-4 Core ML FP32 3.686 ms 4.171 ms 13.478 ms 184/3000
DPDFNet-8 HR MLX 8.145 ms 11.436 ms 17.695 ms 1091/3000
DPDFNet-8 HR MLX + Accelerate 8.999 ms 9.401 ms 10.085 ms 331/3000
DPDFNet-8 HR Core ML FP32 6.326 ms 7.067 ms 7.438 ms 15/3000

Core ML has the lowest synchronous single-hop latency for both DPDFNet profiles. DPDFNet-4 has a 3.686 ms median and 4.171 ms p95, although OS scheduling spikes raised its three-run p99 and deadline-miss count. HR remained tighter at 6.326 ms p50 and 7.438 ms p99. Every backend can still experience scheduler outliers; CoreAudio input/output device buffers are not included.

Queued pure-MLX hops can share one lazy graph evaluation. On a 10-second input, DPDFNet-4 improves from 4.991 seconds at batch one to 4.035 seconds at batch eight; HR improves from 7.915 to 6.837 seconds. A 50 ms injected scheduler stall was recovered by every tested path. These are practical graph-level catch-up batches, not a single monolithic multi-hop DPRNN Metal dispatch. See outputs/benchmark/dpdfnet_multihop_optimized.md and outputs/benchmark/dpdfnet_catchup_optimized.md.

The consolidated charts are outputs/benchmark/realtime_per_hop_latency.png and outputs/benchmark/dpdfnet_models_60s_throughput.png.