Rebuild the benchmark with reproducible PA and DPD results - #19
Merged
Conversation
Full experimental re-run of benchmark/benchmark_report.md. Eight of the ten reported DPD numbers reproduce (QR rows to <=0.05 dB, SGD rows to <=0.5 dB), but the verification exposed three defects: - benchmark_volterra_qr.py never branched on --model, so --model gmp silently ran the Memory Polynomial basis. Present since the script was first added; the report's GMP rows cannot have come from it as committed. Fixed here, and with the fix both GMP rows reproduce. - The APA_200MHz PA NMSE of -43.52 dB in section 4 is unattainable. Re-running the report's own command gives -31.48 dB (converged), a closed-form GMP with 3520 real parameters reaches only -38.83 dB, and the fit residual is near-white with drive-independent power, placing the measurement noise floor at about -38.6 dB. - The APA half of the benchmark therefore rests on a surrogate that is ~7 dB short of that floor. Re-running the QR benchmarks against a converged surrogate moves MP's EVM from -32.70 to -47.41 dB and flips the MP/GMP ranking, which invalidates section 8's "10.6 dB EVM" conclusion. The DPA_160MHz half is unaffected; its surrogate was already at its floor. The dataset itself is clean: CSVs unchanged since 2025-07-09, no clipping or NaN, splits consistent, already time-aligned, and the closed-form ceiling is the same as DPA_160MHz. The -31.5 dB plateau comes from the training recipe -- the same GRU gains 7.4 dB from the OpenDPDv2 hyperparameters alone. Also adds backbones/tres_gru.py (TRes-DeltaGRU with the DeltaGRU cell replaced by a dense nn.GRU), verified numerically equivalent to tres_deltagru at thx=thh=0, and a runner for the fixed-parameter-budget PA backbone sweep. BENCHMARK_FIX_PLAN.md records the evidence and the remaining re-benchmark work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gaochangw
commented
Jul 26, 2026
gaochangw
left a comment
gaochangw
left a comment
Member
Author
There was a problem hiding this comment.
Self-review completed against the final net diff. I reviewed the benchmark runner and provenance validation, MP/GMP solver changes, PA/DPD model paths (including TRes-GRU and TRes-DeltaGRU), CUDA/Triton fast-path fallbacks, public API/default hyperparameters, benchmark-folder cleanup, and the new regression tests. I found no blocking issues. All six required CI checks pass across lint, package build, and Python 3.10-3.13. Squash merge is appropriate because the branch contains iterative benchmark commits whose intermediate artifacts intentionally net out of the final PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The previous report contained an unreproducible
-43.52 dBAPA GRU PA result and used an undertrained PA recipe. APA GMP was also numerically unstable because CUDAgelsassumes a full-rank design matrix, while the 1,350-column GMP basis is ill-conditioned/rank-deficient.The new workflow archives exact checkpoints/logs, records commands and source hashes, validates all evidence before rendering, and keeps only user-facing benchmark artifacts in the repository.
Latest benchmark highlights
-38.7020 / -38.6606 dBvalidation/test NMSE, rank650/1350-38.9088 / -39.4014 dB, rank918/1350-38.8504 / -38.9367 dB20260726T161245Z: 24/24 jobs successful, 22 published resultsUser/developer impact
Users can reproduce the complete report with:
The generated checkpoint, history, commands, job ledger, raw polynomial outputs, source snapshot, report, and figures remain bound to the timestamped run. Current defaults now use batch size 64, 300 epochs, initial learning rate
5e-3, scheduler patience 5, and minimum learning rate5e-5.Addresses #15.
Validation
.venv/bin/pytest -q: 216 passed, 1 PyTorch CUDA-context warning.venv/bin/pytest -q tests/test_benchmark_regressions.py: 26 passedbash -n benchmark/reproduce_benchmark_report.shgit diff --check