Skip to content

Python package on the Rust engine: pywhat._pywhat, maturin wheels (#291 phase 5) - #304

Merged
bee-san merged 2 commits into
mainfrom
pywhat-rs-py
Nov 5, 2026
Merged

bee-san merged 2 commits into
mainfrom
pywhat-rs-py

Conversation

@bee-san

@bee-san bee-san commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Phase 5 of #291, first part: the Python package now searches with the Rust crate's regex engine, through the extension module pywhat._pywhat (PyO3, one abi3 wheel per platform for CPython ≥ 3.8), and it is packaged with maturin instead of poetry. The Python API doesn't change. The CLI is still the click one; it moves to pywhat::cli::main in a second PR, once pywhat-rs-cli's CLI has landed.

What changes

The engine (py/, pywhat/regex_identifier.py, src/engine.rs)

  • pywhat::engine::Pattern (new, public): a pattern compiled exactly like Identifier compiles regex.json's (multi-line, the regex crate, the guards, fancy-regex with the prefilter and the 100,000,000 backtracking limit), with find_iter(text, |start, end| ...). It wraps the engine's own Compiled, so no engine code is copied.

  • py/ (pywhat-py, publish = false, [lib] name = "_pywhat", cdylib, pyo3 0.29.3 with abi3-py38; 0.30 dropped Python 3.8) is the module pywhat._pywhat:

    • finditer(pattern, text) -> list[tuple[int, int]]: the character offsets of the matches, as re.finditer(pattern, text, re.MULTILINE) gives them. Compiled patterns are cached (up to 1,024; the cache starts over when it is full, and regex.json needs about 300), and the GIL is released while compiling and matching.
    • Text(text): the text encoded to UTF-8 once. finditer takes a str or a Text. RegexIdentifier makes one Text per string and searches it for every regex: encoding the 4.2 MB benchmark text for each of 154 regexes took 0.65 s of the 1.44 s that identify() took.
    • MatchError (a RuntimeError): fancy-regex gave up. ValueError: the engine can't compile the pattern. UnicodeEncodeError: the text has lone surrogates.
  • RegexIdentifier.iter_check uses string[start:end] and (start, end) where it used group(0) and span(). Python's re still searches when the engine can't:

    • a string that UTF-8 can't encode (lone surrogates, which surrogateescape makes of command-line arguments that aren't UTF-8): every regex for that string;
    • a pattern the engine can't compile (none of regex.json's);
    • a search that fancy-regex gives up on (MatchError).

    tests/test_rust_engine.py counts the searches that fall back (_re_spans) in each case, including a real backtracking-limit error ([ab]*c(?<!bc) on 20,004 characters: fancy-regex gives up, re finds the match).

  • Without the extension: in a checkout where it isn't built (ModuleNotFoundError), the package works on re as before (scripts/golden.py and scripts/rarity_score.py import the checkout), and test_the_rust_engine_is_used fails, so the suite can't pass on re by accident. An extension that is there but can't be loaded raises its ImportError: the first version of this PR fell back to re then too, and so the Docker image below searched on re without anyone noticing (the extension needs libgcc_s, which python:3.10-alpine lacks). test_without_the_extension checks both cases.

  • scripts/golden.py sets regex_identifier._pywhat = None, so the golden test keeps comparing the crate with Python's re, also where the extension is built.

  • __version__ comes from importlib.metadata (6.0.0a1, from Cargo's 6.0.0-alpha.1; it said 5.0.0).

Packaging (pyproject.toml, poetry.lock removed, requirements.in/.txt, noxfile.py, Cargo.toml members, Dockerfile)

  • pyproject.toml builds with maturin (manifest-path = "py/Cargo.toml", python-source = ".", module-name = "pywhat._pywhat", dynamic = ["version"], strip = true). Name, description, readme, authors, scripts pywhat/what, the dependencies (click>=8.0.0,<9.0.0, rich>=11.0.0) and the optimize extra are as before. requires-python = ">=3.8" (was ^3.7).
  • Licence (owner's decision, Data provenance and licences #302): license = "MIT AND CC-BY-SA-4.0" (PEP 639) and license-files = ["LICENSE", "NOTICE"], so the wheel and the sdist carry both files once NOTICE (Data licences: NOTICE for every source, MasterCard bank names removed, data behind features #303) is in the tree; the release workflow's smoke test checks that the installed package has exactly the licence files of the checkout. The License :: OSI Approved :: MIT License classifier goes (PEP 639 doesn't allow both).
  • The sdist also carries benches/*.rs and examples/*.rs: Cargo.toml names them, and cargo can't read Cargo.toml without them (pip install from the sdist failed with "can't find bench_file example").
  • Workspace: members = [".", "py"], default-members = ["."], so cargo at the root (and rust.yml) leaves py/ out.
  • requirements.txt: exact pins of every dev tool and its dependencies for each Python from 3.8, compiled from requirements.in with uv pip compile --universal --python-version 3.8 (the newest release of each package for each Python). Named requirements.txt rather than requirements-dev.txt because SonarCloud's new rule text:S8565 fails the pull request when a pyproject.toml with dependencies has no lock file it knows (uv.lock, poetry.lock, pdm.lock, pylock.toml or "a fully pinned requirements.txt"). Alerts: see below.
  • noxfile.py: installs requirements.txt, then the package with pip install -e ., which builds the extension (release profile) with maturin, and runs pytest with the same plugins and coverage. Editable on purpose: pytest puts the checkout first on sys.path (the root conftest.py and tests/__init__.py), so with pip install . the tests import the checkout's pywhat/, which has no extension. With pip install . and --import-mode=importlib they do run on the installed wheel, but then coverage measures nothing of the checkout (every pywhat/*.py at 0%, I tried it). Editable, maturin builds the extension into pywhat/, and everything else stays as it was. The wheels themselves are installed and tried in the release workflow on Linux, macOS (both) and Windows (all three), and locally.
  • Dockerfile: pip install git+https://… can't build the extension in python:3.10-alpine, which has no Rust. It now builds the wheel in rust:alpine (from GitHub, PYWHAT_REF, default main) and installs it into python:3.10-alpine with libgcc (a wheel that pip builds has no libgcc_s inside; the musllinux wheels of the release workflow carry it, pywhat.libs/libgcc_s-*.so.1).
  • CONTRIBUTING.md: how to run the tests now (Rust, pip install -e .).

Lint tools. Refreshing the pins needed three config changes, because the newest versions that install on Python 3.8 (and 3.12, where the local gate runs) report things in code this PR doesn't touch:

  • black stays at 23.12.1: black 24 would reformat 23 files.
  • mypy 1.14.1 (from 0.942, which doesn't run on 3.12): implicit_optional = true (mypy 0.990 changed the default; identifier.py relies on it) and disable_error_code = ["override"] for pywhat.interactive (typeshed types Cmd.default() as returning None; pyWhat returns the stop flag, as onecmd() does).
  • flake8 7 (pycodestyle 2.11+, which Python 3.12 needs): per-file-ignores of E721 for the two files that compare exact types on purpose (filter.py, test_regex_identifier.py).
  • pytest stays below 9: pytest-black 0.6.0, its last release, doesn't load on pytest 9.

CI

  • main.yaml: the nine jobs keep their names and runners. They get a Rust toolchain (dtolnay/rust-toolchain, pinned like rust.yml) and Swatinem/rust-cache, and virtualenv==20.39.1 next to nox: without poetry's constraint pip picks virtualenv 21.15.5, which can't create Python 3.8 environments any more (the 3.8 jobs would fail).
  • release.yml (new): generated by maturin generate-ci github (maturin 1.16.0): manylinux x86_64, x86, aarch64, armv7, s390x, ppc64le; musllinux x86_64, x86, aarch64, armv7; Windows x64, x86, aarch64; macOS x86_64, aarch64; sdist. Changed: it builds on pull requests that touch the build (Cargo.*, src/, py/, pyproject.toml, the workflow) and on workflow_dispatch, and uploads the artifacts; it tries every wheel the runner can run (Linux x86_64, all of Windows, both macOS) and the sdist (pip builds it, with maturin 1.16.0 and Rust), installing them with pinned versions and --only-binary :all: (SonarCloud's githubactions:S8541 and S8544); actions are pinned to commits; the free-threaded builds are left out (abi3 wheels; free-threaded Python builds from the sdist); and the release job runs only for tags, in the environment pypi, and publishes only if a PYPI_API_TOKEN secret exists or the variable PYPI_TRUSTED_PUBLISHING is true (trusted publishing). I didn't push tags or run it by hand.

Measurements

Before is main at 4030351 (the Python package on re), after is this branch at 6b62f43 (1a867c8 changes the import of the extension and adds two tests that take 0.01 s); the same tool versions (requirements.txt) on both sides, runs interleaved, medians of 3 (1 for the CLI-option runs). The machine is a shared 64-core host with a load average of 48 to 68 during the runs. Raw rows: /local/home/skerraut/herd-synth3/reports/pywhat-rs-py-evidence/measurements.tsv.

The 4.2 MB text of docs/rust.md (python scripts/bench_corpus.py: 4,239,240 bytes, 115,270 lines), read into a str:

Python 3.12 before after Python 3.8 before after
Identifier().identify(text) (API defaults) 5.39 s 0.85 s 5.16 s 0.86 s
same, whole process (start-up, reading, json.dumps) 5.56 s 1.04 s 5.34 s 1.05 s
Identifier(boundaryless=Filter()).identify(text) (the CLI's options) 19.69 s 3.14 s
python -m pywhat --json --no-progress target/bench.txt (the click CLI) 21.81 s 5.28 s
peak RSS, API defaults / CLI options / --json 64 / 652 / 640 MB 83 / 695 / 683 MB

The outputs are identical: json.dumps() of the API result is the same 2,543,638 bytes (540 matches, sha256 ff4ac5b1…) and of the CLI-option result the same 296,955,548 bytes (29,160 matches, 995808bd…), and pywhat --json writes the same 296,955,554 bytes (cmp).

The Python suite (pytest, cold caches as on CI):

Python 3.8 before after Python 3.12 before after
tests only (-o addopts=), the tests main has 13.83 s 12.27 s 11.39 s 9.38 s
tests only, with the 8 new ones 16.12 s 14.10 s
with the plugins (black, isort, flake8, mypy, as nox runs it), the tests main has 38.16 s 36.43 s 35.67 s 35.68 s
with the plugins, with the new tests 41.97 s 38.25 s
peak RSS, tests only 92 MB 157 MB 87 MB 156 MB

Most of the suite is short texts and the lint plugins, so it gains little. The new tests take about 4 s, half of it the backtracking-limit test (fancy-regex gives up after 100,000,000 steps). The compiled Rust patterns cost memory: 65 MB more in the suite, which compiles both forms of every regex.

Wheels and sdist: a local maturin build --release gives pywhat-6.0.0a1-cp38-abi3-manylinux_2_34_x86_64.whl, 1,678,444 bytes; maturin sdist 708,975 bytes. pip install from the sdist into a fresh Python 3.12 venv took 18.1 s (wall; 71 s CPU on 8 jobs, with the shared cargo download cache warm) and the result passes the smoke test. Per-platform sizes from this PR's Release run (41911666621, head 1a867c8, all 16 artifacts):

File Bytes
pywhat-6.0.0a1-cp38-abi3-macosx_10_12_x86_64.whl 1,609,203
pywhat-6.0.0a1-cp38-abi3-macosx_11_0_arm64.whl 1,556,781
pywhat-6.0.0a1-cp38-abi3-manylinux1_i686.manylinux_2_5_i686.whl 1,685,263
pywhat-6.0.0a1-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl 1,623,998
pywhat-6.0.0a1-cp38-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl 1,571,022
pywhat-6.0.0a1-cp38-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl 1,729,577
pywhat-6.0.0a1-cp38-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl 1,729,969
pywhat-6.0.0a1-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl 1,684,819
pywhat-6.0.0a1-cp38-abi3-musllinux_1_2_aarch64.whl 1,805,352
pywhat-6.0.0a1-cp38-abi3-musllinux_1_2_armv7l.whl 1,847,120
pywhat-6.0.0a1-cp38-abi3-musllinux_1_2_i686.whl 1,875,191
pywhat-6.0.0a1-cp38-abi3-musllinux_1_2_x86_64.whl 1,913,893
pywhat-6.0.0a1-cp38-abi3-win32.whl 1,488,259
pywhat-6.0.0a1-cp38-abi3-win_amd64.whl 1,599,925
pywhat-6.0.0a1-cp38-abi3-win_arm64.whl 1,530,513
pywhat-6.0.0a1.tar.gz 720,572

The musllinux wheels carry pywhat.libs/libgcc_s-*.so.1 (auditwheel), and the musllinux x86_64 wheel of the first run loaded in python:3.10-alpine.

Dependabot alerts (31 open on poetry.lock, 34 in all)

poetry.lock goes, and requirements.txt pins the newest release of each tool for each Python (3.8, 3.9, 3.10, 3.11+). Checked by comparing every pinned version with each alert's vulnerable range (alerts.py and alerts.txt in the evidence directory):

Dependabot may keep reporting the 3.8/3.9 pins, as they are in the file.

Commands and results

  • Local CI (/local/home/skerraut/herd-synth3/reports/pywhat-rs-py-evidence/local_ci.sh, run with $CIPHEY_TC/gate.sh from the checkout), on 1a867c8:
    PASS cargo fmt --all -- --check
    PASS cargo clippy --locked --all-targets -- -D warnings
    PASS cargo clippy --locked --all-targets --no-default-features -- -D warnings
    PASS cargo clippy --locked -p pywhat-py --all-targets -- -D warnings
    PASS cargo test --locked
    PASS cargo test --locked --no-default-features
    PASS cargo test --locked -p pywhat-py
    PASS nox -s tests --force-python …/cpython-3.8…/python3.8     2321 passed, 6 skipped in 61.25s
    PASS nox -s tests --force-python …/cpython-3.12…/python3.12   2324 passed, 3 skipped in 52.77s
    PASS maturin build --release, installed into fresh venvs (3.8, 3.12)    (wheel 1,678,444 bytes)
    PASS maturin sdist, pip install from the sdist into a fresh venv (3.12)  (sdist 708,975 bytes)
    LOCAL CI PASSED
    
    (main, the same way: 2305 passed, 6 skipped on 3.8; 2308 passed, 3 skipped on 3.12.) The smoke test of the wheel and the sdist is the release workflow's SMOKE_TEST. An earlier run hit tests/test_progress.py::test_bars_appear_after_the_delay once, a timing test of the progress bars that this PR doesn't touch; it passed when run again (8 of 8 alone).
  • Packaging: maturin build / maturin sdist with and without a NOTICE in the tree: METADATA and PKG-INFO have Metadata-Version: 2.4, License-Expression: MIT AND CC-BY-SA-4.0 and License-File: LICENSE (and License-File: NOTICE with it), the wheel dist-info/licenses/LICENSE (and NOTICE), the sdist LICENSE (and NOTICE). Without NOTICE maturin leaves it out silently, so the smoke test checks the installed licence files against the checkout's.
  • pip install . instead of -e ., with --import-mode=importlib: 2320 passed, and coverage.xml has 0% for every pywhat/*.py (the tests ran the copy in site-packages).
  • uv pip compile --universal --python-version 3.8 requirements.in -o requirements.txt (uv 0.11.28); the file installs on Python 3.8 and 3.12 (local CI) and 3.9/3.10 (CI).
  • Lint tools on main's code, before the config changes: black 24.1.0 would reformat 23 files (22.12.0 and 23.12.1: none); mypy 0.991 to 1.14.1 report the same 2 errors (identifier.py:96 implicit Optional, interactive.py:775 override), mypy 0.942 fails on Python 3.12; flake8 7.x reports E721 six times in filter.py and test_regex_identifier.py, flake8 5.0.4 on 3.12 reports 46 E231 inside f-strings. With the config, all pass on 3.8 and 3.12.
  • actionlint 1.7.12 on main.yaml and release.yml: no findings (it found that the first version of my main.yaml step run: python -m pip install --only-binary :all: … wasn't valid YAML; it is a block now).
  • Docker (8 CPUs, from this branch): DOCKER_BUILDKIT=0 docker build --no-cache --cpuset-cpus 0-7 --build-arg PYWHAT_REF=pywhat-rs-py -f Dockerfile <empty dir>: built in 57 s, 83.4 MB; docker run --rm IMAGE --version prints PyWhat version 6.0.0a1, docker run --rm IMAGE 'thm{a_flag} bee@skerritt.blog' finds the flag, the URL and the address, and regex_identifier._pywhat is the extension. (With the first version of the Dockerfile the image fell back to re: Error loading shared library libgcc_s.so.1.)
  • CI on 1a867c8: the nine Python jobs, the Rust workflow (fmt/clippy, tests on Ubuntu and Windows, MSRV, wasm32, golden) and the 16 Release jobs pass; the Release workflow's release job is skipped (not a tag); SonarCloud passes (0 open issues). On 6b62f43, SonarCloud failed on 11 issues: 8 for the pip install dist/*.whl lines of release.yml (S8541, S8544), 2 in the Dockerfile and text:S8565 for pyproject.toml without a lock file; 1a867c8 fixes them as described above.

Slower: short texts, once per process

Found after the audit, measured on this head (1a867c8), and fixed in the follow-up PR (pywhat-rs-py-b), so not changed here:

  • The first identify() of a process compiles its patterns one after the other. With the Rust engine that takes about 100 ms (the \w{86}-style Unicode classes of the Discord and Guilded webhook regexes take 25 to 30 ms each), against about 40 ms with re. Every later call is as fast as before (0.5 to 0.9 ms on a short text). The follow-up compiles the patterns of a search at once on up to 8 threads that only live while they compile (fork-safe): first identify() 36 to 53 ms.
  • __version__ from importlib.metadata makes import pywhat about 17 ms slower (84 → 101 ms, -X importtime). The follow-up looks it up when it is first used (PEP 562).
  • Together, on a short text (pywhat --no-progress 'thm{a} bee@skerritt.blog', 30 interleaved runs each, medians, load average 66 to 71): main 264 ms, this head 313 ms, the follow-up 244 ms. On fixtures/file: 316, 346 and 288 ms. Large inputs are 4 to 6 times faster, as above.

Not in this PR

  • cli_main and what.main() on the Rust CLI: second PR (pywhat-rs-py-b), after pywhat-rs-cli's PR.
  • Publishing: pywhat-rs files the publish issue.
  • black 24+/26 (two alerts): a formatting-only PR, once the open Python PRs have landed.

… phase 5)

- py/ (pywhat-py): the extension module pywhat._pywhat (PyO3 0.29.3, abi3-py38).
  finditer(pattern, text) returns the character offsets of the matches, caches the
  compiled patterns and releases the GIL; Text is a text encoded once for many patterns.
- pywhat::engine::Pattern: a pattern compiled the way Identifier compiles regex.json's.
- RegexIdentifier searches with it. Python's re still searches strings with lone
  surrogates, patterns the engine can't compile, and searches that fancy-regex gives up on.
- pyproject.toml on maturin: the version from Cargo (6.0.0a1), __version__ from
  importlib.metadata, licence MIT AND CC-BY-SA-4.0 with LICENSE and NOTICE. poetry.lock
  goes; requirements-dev.txt pins the dev tools for Python 3.8 and newer.
- nox installs with pip: pip install -e . builds the extension.
- CI: the Tests jobs build it with Rust; release.yml (from maturin generate-ci) builds
  and tries the wheels and the sdist, and publishes only from tags once PyPI publishing
  is set up.
@bee-san

bee-san commented Nov 4, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto main 74e7314 (#303: NOTICE, the children feature, MasterCard bank names removed) without conflicts. The audited head was 1a867c8 (MERGE_OK in audit-pywhat-rs-py); the new head is 946a217, and the range-diff is empty:

$ git range-diff 4030351..1a867c8 origin/main..946a217
1:  6b62f43 = 1:  c0bd13d Python package on the Rust engine: pywhat._pywhat, maturin wheels (#291 phase 5)
2:  1a867c8 = 2:  946a217 Fix SonarCloud findings, make a broken extension an error, Dockerfile libgcc

Local CI on 946a217 (local_ci.sh under gate.sh): cargo fmt, clippy (3 runs), cargo test (3 runs) pass; nox -s tests on Python 3.8: 2424 passed, 6 skipped; on 3.12: 2427 passed, 3 skipped; the wheel (1,676,698 bytes) installs on 3.8 and 3.12, the sdist (710,820 bytes) on 3.12. With NOTICE now in the tree, the smoke test finds ['LICENSE', 'NOTICE'] in the installed package (wheel and sdist), so license-files ships both.

@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant