Warn about Accelerate-linked NumPy/SciPy, and correct figures added by #543 by meandmytram · Pull Request #582 · quicophy/mdopt

meandmytram · 2026-09-14T23:41:10Z

Stacks #574 and #581, which touch no files in common, into one PR. Both are merged into a fresh branch off main; the full suite passes on the result, and the backend tests also pass with NumPy and SciPy made to report Accelerate. On the real stock macOS wheels (NumPy 2.4.6 and SciPy 1.18.1, both Accelerate-linked) the warnings fire, the README's commands switch both libraries to OpenBLAS, and the warnings stop.

From #574: warn when NumPy or SciPy links Apple's Accelerate

On Apple silicon running macOS 14 or later, pip installs the macosx_14_0_arm64 NumPy and SciPy wheels, which link Accelerate. On mdopt's rank-deficient, wide-spectrum matrices Accelerate's LAPACK corrupted memory: numpy.linalg.qr died with SIGBUS on a 636x304 matrix and numpy.linalg.svd tripped malloc's heap check inside dgesdd. While mdopt still reduced its SVDs by QR first, a [[72,12,6]] decode at chi_max=400 (SVDs on about 800x1600 matrices) returned wrong verdicts on 21 of 22 shots with a non-trivial error, against 3 of 24 on OpenBLAS, while every unit test passed.

mdopt.backend.array warns at import when NumPy or SciPy is Accelerate-linked, on every backend (the CuPy path still runs NumPy and SciPy LAPACK on the host); MDOPT_ALLOW_ACCELERATE=1 silences it.
The SciPy floor rises from 1.9.2 to 1.15.0, the first release that installs alongside the required NumPy >= 2.3 (SciPy 1.9 to 1.12 require NumPy < 1.29, and 1.13 and 1.14 require NumPy < 2.3). The vendor check's fallback for SciPy 1.9's configuration API could never run with this package, so it is removed; poetry.lock already locks SciPy 1.15.3 and only its hash changes.
The README explains the fault, gives the bond dimensions at which it showed, and gives the commands that install the OpenBLAS wheels of the same versions (macosx_11_0_arm64 NumPy, macosx_12_0_arm64 SciPy).
Tests cover the Accelerate, OpenBLAS and silenced cases, SciPy's capitalised vendor name, an unreadable configuration, and the GPU backend; the no-warnings backend tests allow only the Accelerate warnings, so the macOS runners (which install the Accelerate wheels) pass.

From #581: correct figures in the comments and docs added by #543

An audit of every factual claim #543 added, against the experiment result files and fresh measurements, found five wrong figures; no code behaviour changes.

where	said	measured
`svd()` comment, `test_convergence.py` docstring	wrong verdicts on "24 of 24 shots" at chi_max=400	21 of 22 shots with a non-trivial error
`svd()` comment	the pre-reduction "saved 0-7%"	at most 7%, nothing measurable on most workloads
`_to_numpy` docstring	the per-call cupy import cost "~18% of a decoding run"	about 11,000 failed imports per decode; 4-13% of surface_bitflip, about a third of dem_d3
`_contract_cached` docstring	a shape-keyed cache missed "7-14% of calls on large codes"	1-2% on the DEM workloads, 23% on the RCM surface workload
`test_convergence.py` docstring	"30-60 minutes" per chi_max=400 decode	about 20 minutes for the whole test

The benchmark README's classical_ldpc row now says three LDPC codes. Checked and left as written: the Apple M5 hardware, the ~1700-site DEM chain, the DEM workload parameters, the opt_einsum overhead at small chi, and agreement of pre- and post-#543 decoders at chi=1e5 (1.6e-14 on 22 posteriors).

Supersedes #574 and #581.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

…enBLAS wheel On the macOS arm64 NumPy 2.x wheels, which link Apple's Accelerate framework, the decoders' rank-deficient, wide-spectrum matrices made numpy.linalg.qr die with SIGBUS and numpy.linalg.svd trip malloc's heap-corruption check inside dgesdd (five crash reports from the [[72,12,6]] investigation), and a chi_max=400 decode returned wrong verdicts while every unit test passed. The OpenBLAS build of the same NumPy version (the macosx_11_0_arm64 wheel) runs the same reproducers clean. The backend now warns once at import when Accelerate is found behind NumPy (MDOPT_ALLOW_ACCELERATE=1 silences it) and the README gives the two commands that install the OpenBLAS wheel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

…AS wheels The SVD helpers call SciPy's LAPACK too, and the SciPy macOS arm64 wheels link Accelerate like NumPy's; with only NumPy switched to OpenBLAS the reduced SVD path still returned a wrong factorisation (reconstruction error 0.15) or crashed, and with both switched it passes repeatedly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

…rate SciPy 1.10 and later report their LAPACK through show_config(mode="dicts"), but SciPy 1.9, which the dependency floor allows, has show() without a mode argument, so the SciPy check silently reported an unknown vendor there. The helper now falls back to __config__.get_info("lapack_opt") in that case, and two SciPy-shaped test doubles cover both APIs. The README said the macOS arm64 wheels link Accelerate; only the macosx_14_0_arm64 wheels do (checked NumPy 2.4.6, SciPy 1.15.3 and 1.18.1), which pip picks on macOS 14 and later, while the macosx_11_0_arm64 NumPy and macosx_12_0_arm64 SciPy wheels bundle OpenBLAS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

test_defaults_to_numpy asserted that importing the backend warns about nothing, which fails on exactly the builds the new warning targets: the macOS CI runners install the Accelerate-linked NumPy and SciPy wheels, and all three macOS test jobs failed on it. The test now allows RuntimeWarnings that mention Accelerate and still fails on any other warning. Checked with NumPy and SciPy made to report Accelerate: the backend tests and the full suite pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

The import-time checks were skipped when CuPy was selected, yet the GPU path still runs NumPy's linalg and BLAS on the host (orthogonality-centre moves, host-side contractions) and SciPy's LAPACK in the SVD fallbacks and in qr, so a GPU user on the stock macOS wheels was exposed to the same corruption without a warning. Both checks now run regardless of the backend. The CuPy-selection test allows the Accelerate warnings, and a new test checks that they fire with CuPy selected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

The test faked show_config by delegating to the installed function with mode="dicts", which SciPy 1.9 (still allowed by the dependency floor and handled by _lapack_vendor) does not accept; there the helper fell back to the real OpenBLAS config and the expected SciPy warning never fired. The fake now returns a synthetic modern config and no longer touches the installed API. Checked with real OpenBLAS, with NumPy and SciPy reporting Accelerate, and with SciPy's show_config replaced by the 1.9 signature. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

The README described the failure only as "a bivariate-bicycle decode returned wrong verdicts". It now gives the scale: on the [[72,12,6]] code in the natural qubit order at chi_max=400, where the SVDs act on matrices of about 800x1600, 21 of 22 shots with a non-trivial error decoded wrongly while mdopt still reduced its SVDs by QR first, against 3 of 24 on OpenBLAS; the same test shot decoded correctly at chi_max 64, 128 and 256, which is noted as no guarantee because the fault depends on heap state. The SIGBUS matrix size (636x304) is given as well. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

An audit of the claims #543 added, against the result files and fresh measurements, found five wrong figures: - svd() comment and the convergence test's docstring: the chi_max=400 [[72,12,6]] failure was 21 of 22 shots with a non-trivial error, not "24 of 24" (that count mixed repeated runs of one shot with a second). - svd() comment: the pre-reduction saved at most 7% and nothing measurable on most workloads, rather than "0-7%". - _to_numpy docstring: the per-call cupy import cost about 11,000 failed imports per decode, between roughly a tenth and a third of a decode (surface_bitflip 4-13%, dem_d3 about a third, timed on pre-#543 main), not a flat "~18%". - _contract_cached docstring: a shape-keyed cache misses 1-2% of zip-up calls on the DEM workloads and 23% on the RCM surface workload, not "7-14% on large codes". - test_convergence docstring: the test takes about 20 minutes on an idle laptop, not 30-60 minutes per decode. The benchmark README's classical_ldpc row now says three LDPC codes. Claims checked and left as they are: Apple M5 10 cores, the ~1700-site DEM chain (1677 mechanisms), the DEM workload parameters, the opt_einsum overhead at small chi (42% of a zip step at chi=32), and agreement of pre- and post-#543 decoders at chi=1e5 (1.6e-14 on 22 posteriors). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

The _warn_if_accelerate docstring said the macOS arm64 wheels of NumPy 2.x link Accelerate; only the macOS 14+ wheels (macosx_14_0_arm64) do, while the macosx_11_0_arm64 wheels are OpenBLAS builds. The docstring and the NumPy warning message also attributed the wrong chi_max=400 verdicts to Accelerate alone, but those occurred while mdopt still reduced its SVDs by QR first; with the plain SVD the checked chi_max=400 shots decoded correctly even on Accelerate. The memory corruption itself (SIGBUS in linalg.qr, malloc's heap check in dgesdd) does not depend on that path, so the warning stays; its wording now matches the README. Also reflows an over-long README line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

meandmytram

gooood

Copilot

🟡 Changes recommended

One corrected performance range still contradicts the measurements stated in the PR description.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds import-time warnings for Accelerate-linked NumPy/SciPy and corrects benchmark documentation.

Changes:

Detects Accelerate LAPACK and documents OpenBLAS alternatives.
Adds backend warning tests across supported configurations.
Corrects benchmark and regression figures.

File summaries

File	Description
`mdopt/backend/array.py`	Adds LAPACK detection and warnings.
`tests/backend/test_array.py`	Tests detection, suppression, and GPU behavior.
`README.md`	Documents the Accelerate issue and workaround.
`mdopt/utils/utils.py`	Updates profiling and SVD comments.
`mdopt/contractor/contractor.py`	Corrects cache-miss figures.
`tests/decoding/test_convergence.py`	Corrects regression figures and timing.
`benchmarks/README.md`	Corrects the LDPC workload description.

Review details

Files reviewed: 7/7 changed files
Comments generated: 1
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

codecov · 2026-09-14T23:44:50Z

Codecov Report

❌ Patch coverage is 92.53731% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.84%. Comparing base (05c7453) to head (8e32ce0).

Files with missing lines	Patch %	Lines
mdopt/backend/array.py	86.36%	3 Missing ⚠️
tests/backend/test_array.py	95.55%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #582      +/-   ##
==========================================
- Coverage   96.89%   96.84%   -0.06%     
==========================================
  Files          28       28              
  Lines        5509     5574      +65     
==========================================
+ Hits         5338     5398      +60     
- Misses        171      176       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- The trailing backslash in _warn_if_accelerate's docstring was a line continuation inside a normal string literal, so the two-line pip command rendered as one run-on line; it is escaped now. - The _to_numpy docstring's "between roughly a tenth and a third" left out the measured low end; it now says about 4% on surface_bitflip to about a third on dem_d3, matching the measurements. - test_accelerate_lapack_warns_unless_allowed re-imported warnings, which the module already imports. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

Copilot

🟢 Approval recommended

The detection logic, compatibility handling, tests, and documentation are consistent with the stated purpose.

Review details

Files reviewed: 7/7 changed files
Comments generated: 0 new
Review effort level: Balanced

…llback The dependency table allowed scipy>=1.9.2 next to numpy>=2.3.0, but no SciPy before 1.15 can be installed with NumPy 2.3: SciPy 1.9 to 1.12 require numpy<1.29, and 1.13 and 1.14 require numpy<2.3 (PyPI metadata). SciPy 1.15.0 allows numpy<2.5 and Python >=3.10, matching this package's NumPy and Python floors, so it is the real minimum. poetry.lock already locks SciPy 1.15.3; only its content hash changes. With the floor raised, the vendor check's fallback for SciPy 1.9's configuration API (show() without a mode argument, __config__.get_info) could never run, so it and its test are removed; an unreadable configuration still reports an unknown vendor instead of raising, which a test now checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

Copilot

🟢 Approval recommended

The warning logic, tests, dependency constraint, documentation, and corrected measurements are consistent and complete.

Review details

Files reviewed: 8/9 changed files
Comments generated: 0 new
Review effort level: Balanced

meandmytram

kool

meandmytram and others added 11 commits September 12, 2026 13:56

Stack #574: warn when NumPy or SciPy links Accelerate

3b0a0fe

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

Stack #581: correct figures in the comments and docs added by #543

2beb6d3

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S6mbxc9eJDQMVx7tjvYwud

This was referenced Sep 14, 2026

Warn when NumPy links Accelerate; document the OpenBLAS wheel #574

Closed

Correct figures in the comments and docs added by #543 #581

Closed

meandmytram self-assigned this Sep 14, 2026

meandmytram added documentation Improvements or additions to documentation enhancement New feature or request python Pull requests that update Python code labels Sep 14, 2026

meandmytram requested a balanced review from Copilot September 14, 2026 23:42

Copilot started reviewing on behalf of meandmytram September 14, 2026 23:42 View session

meandmytram commented Sep 14, 2026

View reviewed changes

Copilot AI reviewed Sep 14, 2026

View reviewed changes

Comment thread mdopt/utils/utils.py Outdated

meandmytram requested a balanced review from Copilot September 14, 2026 23:54

Copilot started reviewing on behalf of meandmytram September 14, 2026 23:54 View session

Copilot AI reviewed Sep 14, 2026

View reviewed changes

meandmytram requested a balanced review from Copilot September 14, 2026 23:59

Copilot started reviewing on behalf of meandmytram September 14, 2026 23:59 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

meandmytram commented Sep 15, 2026

View reviewed changes

meandmytram merged commit 7bd950a into main Sep 15, 2026
22 checks passed

meandmytram deleted the stack-574-581 branch September 15, 2026 00:09

Navigation Menu

Uh oh!

Warn about Accelerate-linked NumPy/SciPy, and correct figures added by #543 - #582

Warn about Accelerate-linked NumPy/SciPy, and correct figures added by #543#582
meandmytram merged 13 commits into
mainfrom
stack-574-581

meandmytram commented Sep 14, 2026 •

edited

Loading

Uh oh!

meandmytram left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

codecov Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

meandmytram left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

meandmytram commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

From #574: warn when NumPy or SciPy links Apple's Accelerate

From #581: correct figures in the comments and docs added by #543

Uh oh!

meandmytram left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

codecov Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

meandmytram left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

meandmytram commented Sep 14, 2026 •

edited

Loading

codecov Bot commented Sep 14, 2026 •

edited

Loading