Skip to content

Correct figures in the comments and docs added by #543 - #581

Closed
meandmytram wants to merge 1 commit into
mainfrom
fix-543-figures
Closed

meandmytram wants to merge 1 commit into
mainfrom
fix-543-figures

Conversation

@meandmytram

Copy link
Copy Markdown
Member

An audit of every factual claim #543 added (code comments, docstrings, the benchmark README), checked against the experiment result files and fresh measurements, found five wrong figures. This PR corrects them; no code behaviour changes.

where said measured
svd() comment, test_convergence.py docstring wrong verdicts on "24 of 24 shots" at chi_max=400 21 of 22 shots with a non-trivial error (the old count mixed repeated runs of one shot with a second shot); 3 of 24 with the plain SVD on OpenBLAS
svd() comment the pre-reduction "saved 0-7%" at most 7%, nothing measurable on most workloads
_to_numpy docstring the per-call cupy import cost "~18% of a decoding run" about 11,000 failed imports per decode; surface_bitflip 4-13%, dem_d3 about a third (timed on pre-#543 main, with and without the fix)
_contract_cached docstring a shape-keyed cache missed "7-14% of calls on large codes" 1-2% of zip-up calls on dem_d3/dem_d5, 23% on css_optimised
test_convergence.py docstring "30-60 minutes" per chi_max=400 decode the whole test takes about 20 minutes on an idle laptop (21.5 min measured)
benchmarks/README.md classical_ldpc on "random (3,4) LDPC code" three codes (seeds 11, 12, 13)

Checked and correct as written: Apple M5 (10 cores), the ~1700-site DEM chain (1677 mechanisms), the DEM workload parameters in bench_suite.py, the opt_einsum expression overhead on small tensors (42% of a zip step at chi=32, none at chi=64), the Frobenius-versus-maximum bound in the isometry gate comment, and the suite header's agreement of old and new decoders at chi=1e5 (pre- and post-#543 main agree to 1.6e-14 on 22 converged posteriors).

#543's description is corrected in place; the errors in its older comments are listed in a correction comment on #543.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

An audit of the claims #543 added, against the result files and fresh
measurements, found five wrong figures:
- svd() comment and the convergence test's docstring: the chi_max=400
  [[72,12,6]] failure was 21 of 22 shots with a non-trivial error, not
  "24 of 24" (that count mixed repeated runs of one shot with a second).
- svd() comment: the pre-reduction saved at most 7% and nothing measurable
  on most workloads, rather than "0-7%".
- _to_numpy docstring: the per-call cupy import cost about 11,000 failed
  imports per decode, between roughly a tenth and a third of a decode
  (surface_bitflip 4-13%, dem_d3 about a third, timed on pre-#543 main),
  not a flat "~18%".
- _contract_cached docstring: a shape-keyed cache misses 1-2% of zip-up
  calls on the DEM workloads and 23% on the RCM surface workload, not
  "7-14% on large codes".
- test_convergence docstring: the test takes about 20 minutes on an idle
  laptop, not 30-60 minutes per decode.
The benchmark README's classical_ldpc row now says three LDPC codes. Claims
checked and left as they are: Apple M5 10 cores, the ~1700-site DEM chain
(1677 mechanisms), the DEM workload parameters, the opt_einsum overhead at
small chi (42% of a zip step at chi=32), and agreement of pre- and
post-#543 decoders at chi=1e5 (1.6e-14 on 22 posteriors).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud
@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.91%. Comparing base (05c7453) to head (0e73e33).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #581      +/-   ##
==========================================
+ Coverage   96.89%   96.91%   +0.01%     
==========================================
  Files          28       28              
  Lines        5509     5509              
==========================================
+ Hits         5338     5339       +1     
+ Misses        171      170       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The documentation-only corrections are consistent with the benchmark implementation and supplied measurements.

Pull request overview

Corrects inaccurate benchmark and regression figures introduced by #543 without changing runtime behavior.

Changes:

  • Corrects SVD failure rates and test duration.
  • Clarifies measured optimization and cache impacts.
  • Documents that the LDPC benchmark uses three codes.
File summaries
File Description
tests/decoding/test_convergence.py Corrects failure count and runtime.
mdopt/utils/utils.py Corrects performance and SVD measurements.
mdopt/contractor/contractor.py Corrects cache-miss measurements.
benchmarks/README.md Corrects the LDPC workload description.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@meandmytram

Copy link
Copy Markdown
Member Author

Stacked into #582 together with #574; closing in favour of it. The branch is kept.

@meandmytram
meandmytram deleted the fix-543-figures branch September 15, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants