Skip to content

fix: MAX_PRECISION should reflect engine capability, not wrapper bindings#43

Draft
hozblok wants to merge 1 commit into
masterfrom
fix/max-precision-not-tied-to-wrapper-bindings
Draft

fix: MAX_PRECISION should reflect engine capability, not wrapper bindings#43
hozblok wants to merge 1 commit into
masterfrom
fix/max-precision-not-tied-to-wrapper-bindings

Conversation

@hozblok
Copy link
Copy Markdown
Owner

@hozblok hozblok commented Jun 2, 2026

Bug

backend.py's MAX_PRECISION was derived from the bound mp_real_<P> wrapper classes:

MAX_PRECISION = max(int(P) for n in dir(_formula)
                   if n.startswith("mp_real_"))

That made the constant track Python wrapper coverage, not the C++ engine's actual precision ceiling. On a .pyd built before the full mp_real ladder was bound (anything pre-#37 — including any locally-installed editable build people might still be running against), only mp_real_24 is exposed, so MAX_PRECISION came out as 24. Solver.__init__'s bounds check then rejected anything higher:

>>> from formula import Solver
>>> Solver("sqrt(2)", precision=100)()
ValueError: precision must be in [0, 24] (got 100)

This is wrong. Formula evaluation does not depend on mp_real_<P> wrappers; they're independent. Verified against the same stale .pyd:

>>> from formula import Formula
>>> Formula("sqrt(2)", precision=100).get()[:60]
'1.414213562373095048801688724209698078569671875376...'
>>> Formula("sqrt(2)", precision=8192).precision
8192

Fix

Move the constant to where it actually lives — the C++ engine — and re-export.

  • src/cpp/main.cpp — expose csconstants::max_precision as _formula.MAX_PRECISION:
    m.attr("MAX_PRECISION") = static_cast<unsigned>(max_precision);
  • src/formula/backend.py — read _formula.MAX_PRECISION, with a fallback to 8192 (the historic engine ceiling) for older .pyds that predate the attr:
    try:
        MAX_PRECISION = _formula.MAX_PRECISION
    except AttributeError:
        MAX_PRECISION = 8192

The two concepts are now cleanly separated:

concept source
Engine evaluation ceiling csconstants::max_precision (currently p_262144), surfaced as _formula.MAX_PRECISION
Python wrapper coverage which py::class_<mp_real<P>> got registered (currently the full AllowedPrecisionsSeq)

Test

tests/test_max_precision_is_engine_bound.py — 5 cases:

  • MAX_PRECISION >= 8192 (the historic floor; never the 24 the bug produced).
  • MAX_PRECISION exceeds the highest bound mp_real_<P> wrapper P (proves the decoupling on a sparse-wrapper .pyd).
  • Solver("sqrt(2)", precision=100)() succeeds and returns a 100-digit value (the smoking gun — exactly the call that hit the regression).
  • Solver(...) at MAX_PRECISION runs and returns at least MAX_PRECISION decimal digits.
  • Solver(..., precision=MAX_PRECISION + 1) is still rejected — the bound is still enforced, just at the right ceiling.

All 5 pass locally against the stale .pyd (yields MAX_PRECISION = 8192 via the fallback).

Verification context

Local venv has a .pyd built before #37 (only mp_real_24 exposed; _formula.MAX_PRECISION not yet present). My local results:

  • Before fix: MAX_PRECISION = 24, Solver("sqrt(2)", precision=100)()ValueError.
  • After fix: MAX_PRECISION = 8192 (fallback path), Solver("sqrt(2)", precision=100)() → 100-digit result. Solver("sqrt(2)", precision=8192)() returns 8193 chars (1 + "." + 8191 fractional digits).

On a fresh .pyd rebuilt from this branch, _formula.MAX_PRECISION will be 262144 (or whatever max_precision is at the time of the build), and the fallback path won't trigger.

Out of scope

  • Reasoning about whether 262144 is actually a sensible ceiling — Boost 1.79's cpp_dec_float static-asserts on instantiations that large on at least some toolchains (the master Windows MSVC build currently fails for this reason). That's a separate question; this PR just fixes the wrong-constant-derivation issue.

…ings

The pre-fix derivation in src/formula/backend.py was

    MAX_PRECISION = max(int(P) for n in dir(_formula)
                       if n.startswith("mp_real_"))

which made the constant track the highest mp_real_<P> Python wrapper
class on the .pyd. On a .pyd built before the full mp_real ladder was
bound (e.g. one that only exposed mp_real_24), this came out as 24 —
and Solver.__init__'s bounds check rejected anything higher:

    >>> from formula import Solver
    >>> Solver("sqrt(2)", precision=100)()
    ValueError: precision must be in [0, 24] (got 100)

That's wrong: the C++ Formula evaluation engine doesn't depend on the
mp_real_<P> wrappers at all (verified — Formula("sqrt(2)", precision=100)
on the same .pyd runs fine and rounds to 128). The two concepts are
independent: `csconstants::max_precision` is the engine's ceiling;
`py::class_<mp_real<P>>` bindings are a separate Python-side convenience.

Fix:
- main.cpp: expose csconstants::max_precision as `_formula.MAX_PRECISION`.
- backend.py: read `_formula.MAX_PRECISION` with a fallback to 8192 (the
  historic engine ceiling) for older .pyds that predate the attr.

Regression test in tests/test_max_precision_is_engine_bound.py confirms
the decoupling: MAX_PRECISION exceeds the highest bound mp_real_<P>
wrapper; Solver accepts precision higher than any wrapper P; the bound
is still enforced at the right ceiling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant