Repository navigation
Python package on the Rust engine: pywhat._pywhat, maturin wheels (#291 phase 5) - #304
Conversation
… phase 5) - py/ (pywhat-py): the extension module pywhat._pywhat (PyO3 0.29.3, abi3-py38). finditer(pattern, text) returns the character offsets of the matches, caches the compiled patterns and releases the GIL; Text is a text encoded once for many patterns. - pywhat::engine::Pattern: a pattern compiled the way Identifier compiles regex.json's. - RegexIdentifier searches with it. Python's re still searches strings with lone surrogates, patterns the engine can't compile, and searches that fancy-regex gives up on. - pyproject.toml on maturin: the version from Cargo (6.0.0a1), __version__ from importlib.metadata, licence MIT AND CC-BY-SA-4.0 with LICENSE and NOTICE. poetry.lock goes; requirements-dev.txt pins the dev tools for Python 3.8 and newer. - nox installs with pip: pip install -e . builds the extension. - CI: the Tests jobs build it with Rust; release.yml (from maturin generate-ci) builds and tries the wheels and the sdist, and publishes only from tags once PyPI publishing is set up.
|
Rebased onto main 74e7314 (#303: NOTICE, the Local CI on 946a217 ( |
|



Phase 5 of #291, first part: the Python package now searches with the Rust crate's regex engine, through the extension module
pywhat._pywhat(PyO3, one abi3 wheel per platform for CPython ≥ 3.8), and it is packaged with maturin instead of poetry. The Python API doesn't change. The CLI is still the click one; it moves topywhat::cli::mainin a second PR, once pywhat-rs-cli's CLI has landed.What changes
The engine (
py/,pywhat/regex_identifier.py,src/engine.rs)pywhat::engine::Pattern(new, public): a pattern compiled exactly likeIdentifiercompiles regex.json's (multi-line, theregexcrate, the guards, fancy-regex with the prefilter and the 100,000,000 backtracking limit), withfind_iter(text, |start, end| ...). It wraps the engine's ownCompiled, so no engine code is copied.py/(pywhat-py,publish = false,[lib] name = "_pywhat", cdylib, pyo3 0.29.3 withabi3-py38; 0.30 dropped Python 3.8) is the modulepywhat._pywhat:finditer(pattern, text) -> list[tuple[int, int]]: the character offsets of the matches, asre.finditer(pattern, text, re.MULTILINE)gives them. Compiled patterns are cached (up to 1,024; the cache starts over when it is full, and regex.json needs about 300), and the GIL is released while compiling and matching.Text(text): the text encoded to UTF-8 once.finditertakes astror aText.RegexIdentifiermakes oneTextper string and searches it for every regex: encoding the 4.2 MB benchmark text for each of 154 regexes took 0.65 s of the 1.44 s thatidentify()took.MatchError(aRuntimeError): fancy-regex gave up.ValueError: the engine can't compile the pattern.UnicodeEncodeError: the text has lone surrogates.RegexIdentifier.iter_checkusesstring[start:end]and(start, end)where it usedgroup(0)andspan(). Python'srestill searches when the engine can't:surrogateescapemakes of command-line arguments that aren't UTF-8): every regex for that string;MatchError).tests/test_rust_engine.pycounts the searches that fall back (_re_spans) in each case, including a real backtracking-limit error ([ab]*c(?<!bc)on 20,004 characters: fancy-regex gives up,refinds the match).Without the extension: in a checkout where it isn't built (
ModuleNotFoundError), the package works onreas before (scripts/golden.py and scripts/rarity_score.py import the checkout), andtest_the_rust_engine_is_usedfails, so the suite can't pass onreby accident. An extension that is there but can't be loaded raises itsImportError: the first version of this PR fell back torethen too, and so the Docker image below searched onrewithout anyone noticing (the extension needslibgcc_s, whichpython:3.10-alpinelacks).test_without_the_extensionchecks both cases.scripts/golden.pysetsregex_identifier._pywhat = None, so the golden test keeps comparing the crate with Python'sre, also where the extension is built.__version__comes fromimportlib.metadata(6.0.0a1, from Cargo's6.0.0-alpha.1; it said5.0.0).Packaging (
pyproject.toml,poetry.lockremoved,requirements.in/.txt,noxfile.py,Cargo.tomlmembers,Dockerfile)pyproject.tomlbuilds with maturin (manifest-path = "py/Cargo.toml",python-source = ".",module-name = "pywhat._pywhat",dynamic = ["version"],strip = true). Name, description, readme, authors, scriptspywhat/what, the dependencies (click>=8.0.0,<9.0.0,rich>=11.0.0) and theoptimizeextra are as before.requires-python = ">=3.8"(was^3.7).license = "MIT AND CC-BY-SA-4.0"(PEP 639) andlicense-files = ["LICENSE", "NOTICE"], so the wheel and the sdist carry both files onceNOTICE(Data licences: NOTICE for every source, MasterCard bank names removed, data behind features #303) is in the tree; the release workflow's smoke test checks that the installed package has exactly the licence files of the checkout. TheLicense :: OSI Approved :: MIT Licenseclassifier goes (PEP 639 doesn't allow both).benches/*.rsandexamples/*.rs: Cargo.toml names them, and cargo can't read Cargo.toml without them (pip installfrom the sdist failed with "can't findbench_fileexample").members = [".", "py"],default-members = ["."], so cargo at the root (and rust.yml) leaves py/ out.requirements.txt: exact pins of every dev tool and its dependencies for each Python from 3.8, compiled fromrequirements.inwithuv pip compile --universal --python-version 3.8(the newest release of each package for each Python). Namedrequirements.txtrather thanrequirements-dev.txtbecause SonarCloud's new ruletext:S8565fails the pull request when apyproject.tomlwith dependencies has no lock file it knows (uv.lock,poetry.lock,pdm.lock,pylock.tomlor "a fully pinned requirements.txt"). Alerts: see below.noxfile.py: installsrequirements.txt, then the package withpip install -e ., which builds the extension (release profile) with maturin, and runs pytest with the same plugins and coverage. Editable on purpose: pytest puts the checkout first onsys.path(the rootconftest.pyandtests/__init__.py), so withpip install .the tests import the checkout'spywhat/, which has no extension. Withpip install .and--import-mode=importlibthey do run on the installed wheel, but then coverage measures nothing of the checkout (everypywhat/*.pyat 0%, I tried it). Editable, maturin builds the extension intopywhat/, and everything else stays as it was. The wheels themselves are installed and tried in the release workflow on Linux, macOS (both) and Windows (all three), and locally.Dockerfile:pip install git+https://…can't build the extension inpython:3.10-alpine, which has no Rust. It now builds the wheel inrust:alpine(from GitHub,PYWHAT_REF, defaultmain) and installs it intopython:3.10-alpinewithlibgcc(a wheel that pip builds has nolibgcc_sinside; the musllinux wheels of the release workflow carry it,pywhat.libs/libgcc_s-*.so.1).CONTRIBUTING.md: how to run the tests now (Rust,pip install -e .).Lint tools. Refreshing the pins needed three config changes, because the newest versions that install on Python 3.8 (and 3.12, where the local gate runs) report things in code this PR doesn't touch:
implicit_optional = true(mypy 0.990 changed the default;identifier.pyrelies on it) anddisable_error_code = ["override"]forpywhat.interactive(typeshed typesCmd.default()as returningNone; pyWhat returns the stop flag, asonecmd()does).per-file-ignoresof E721 for the two files that compare exact types on purpose (filter.py,test_regex_identifier.py).CI
main.yaml: the nine jobs keep their names and runners. They get a Rust toolchain (dtolnay/rust-toolchain, pinned like rust.yml) andSwatinem/rust-cache, andvirtualenv==20.39.1next to nox: without poetry's constraint pip picks virtualenv 21.15.5, which can't create Python 3.8 environments any more (the 3.8 jobs would fail).release.yml(new): generated bymaturin generate-ci github(maturin 1.16.0): manylinux x86_64, x86, aarch64, armv7, s390x, ppc64le; musllinux x86_64, x86, aarch64, armv7; Windows x64, x86, aarch64; macOS x86_64, aarch64; sdist. Changed: it builds on pull requests that touch the build (Cargo.*, src/, py/, pyproject.toml, the workflow) and onworkflow_dispatch, and uploads the artifacts; it tries every wheel the runner can run (Linux x86_64, all of Windows, both macOS) and the sdist (pip builds it, with maturin 1.16.0 and Rust), installing them with pinned versions and--only-binary :all:(SonarCloud'sgithubactions:S8541andS8544); actions are pinned to commits; the free-threaded builds are left out (abi3 wheels; free-threaded Python builds from the sdist); and thereleasejob runs only for tags, in the environmentpypi, and publishes only if aPYPI_API_TOKENsecret exists or the variablePYPI_TRUSTED_PUBLISHINGistrue(trusted publishing). I didn't push tags or run it by hand.Measurements
Before is
mainat 4030351 (the Python package onre), after is this branch at 6b62f43 (1a867c8 changes the import of the extension and adds two tests that take 0.01 s); the same tool versions (requirements.txt) on both sides, runs interleaved, medians of 3 (1 for the CLI-option runs). The machine is a shared 64-core host with a load average of 48 to 68 during the runs. Raw rows:/local/home/skerraut/herd-synth3/reports/pywhat-rs-py-evidence/measurements.tsv.The 4.2 MB text of docs/rust.md (
python scripts/bench_corpus.py: 4,239,240 bytes, 115,270 lines), read into astr:Identifier().identify(text)(API defaults)json.dumps)Identifier(boundaryless=Filter()).identify(text)(the CLI's options)python -m pywhat --json --no-progress target/bench.txt(the click CLI)--jsonThe outputs are identical:
json.dumps()of the API result is the same 2,543,638 bytes (540 matches, sha256ff4ac5b1…) and of the CLI-option result the same 296,955,548 bytes (29,160 matches,995808bd…), andpywhat --jsonwrites the same 296,955,554 bytes (cmp).The Python suite (pytest, cold caches as on CI):
-o addopts=), the testsmainhasmainhasMost of the suite is short texts and the lint plugins, so it gains little. The new tests take about 4 s, half of it the backtracking-limit test (fancy-regex gives up after 100,000,000 steps). The compiled Rust patterns cost memory: 65 MB more in the suite, which compiles both forms of every regex.
Wheels and sdist: a local
maturin build --releasegivespywhat-6.0.0a1-cp38-abi3-manylinux_2_34_x86_64.whl, 1,678,444 bytes;maturin sdist708,975 bytes.pip installfrom the sdist into a fresh Python 3.12 venv took 18.1 s (wall; 71 s CPU on 8 jobs, with the shared cargo download cache warm) and the result passes the smoke test. Per-platform sizes from this PR's Release run (41911666621, head 1a867c8, all 16 artifacts):pywhat-6.0.0a1-cp38-abi3-macosx_10_12_x86_64.whlpywhat-6.0.0a1-cp38-abi3-macosx_11_0_arm64.whlpywhat-6.0.0a1-cp38-abi3-manylinux1_i686.manylinux_2_5_i686.whlpywhat-6.0.0a1-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whlpywhat-6.0.0a1-cp38-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whlpywhat-6.0.0a1-cp38-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whlpywhat-6.0.0a1-cp38-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whlpywhat-6.0.0a1-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whlpywhat-6.0.0a1-cp38-abi3-musllinux_1_2_aarch64.whlpywhat-6.0.0a1-cp38-abi3-musllinux_1_2_armv7l.whlpywhat-6.0.0a1-cp38-abi3-musllinux_1_2_i686.whlpywhat-6.0.0a1-cp38-abi3-musllinux_1_2_x86_64.whlpywhat-6.0.0a1-cp38-abi3-win32.whlpywhat-6.0.0a1-cp38-abi3-win_amd64.whlpywhat-6.0.0a1-cp38-abi3-win_arm64.whlpywhat-6.0.0a1.tar.gzThe musllinux wheels carry
pywhat.libs/libgcc_s-*.so.1(auditwheel), and the musllinux x86_64 wheel of the first run loaded inpython:3.10-alpine.Dependabot alerts (31 open on poetry.lock, 34 in all)
poetry.lockgoes, andrequirements.txtpins the newest release of each tool for each Python (3.8, 3.9, 3.10, 3.11+). Checked by comparing every pinned version with each alert's vulnerable range (alerts.pyandalerts.txtin the evidence directory):"plural_name": falseto all current regex #40 (now 2026.10.23), idna [ImgBot] Optimize images #13 API Upgrades and Dependencies! #35 (3.15 / 3.20), requests 2 regex don't work #4 Fix path issues and add YouTube Channel ID regex #14 [ImgBot] Optimize images #19, urllib3 What regex does `What` support? #8 URL Regex doesn't work #10 Fix path issues and fix encoding error #15, Pygments More regex #5, and orjson #11 Add filtration to API and CLI #29 and zipp Fix path issues again and an encoding error #17, which the dev environment doesn't need any more.what#24 Regex for more crypto wallets, IPv6 and improved coordinates #26 Improved detection for emails #41 (Improved detection for emails #41 also on 3.10: its fix needs 3.11). They come from types-requests, codecov and requests (scripts/), from rich, and from pytest-mypy.Dependabot may keep reporting the 3.8/3.9 pins, as they are in the file.
pyproject.toml(poetry's dev-dependencies), the same advisories as Regex is not appropriate for this #12, YouTube Video ID Regex too broad, fails on SSN #33 and Awesome Repo #39. This PR removes those dev-dependencies, so they should close on merge; as black 23.12.1 and pytest 8.x stay pinned, Dependabot will likely report them again on requirements.txt.Commands and results
/local/home/skerraut/herd-synth3/reports/pywhat-rs-py-evidence/local_ci.sh, run with$CIPHEY_TC/gate.shfrom the checkout), on 1a867c8:main, the same way: 2305 passed, 6 skipped on 3.8; 2308 passed, 3 skipped on 3.12.) The smoke test of the wheel and the sdist is the release workflow'sSMOKE_TEST. An earlier run hittests/test_progress.py::test_bars_appear_after_the_delayonce, a timing test of the progress bars that this PR doesn't touch; it passed when run again (8 of 8 alone).maturin build/maturin sdistwith and without aNOTICEin the tree: METADATA and PKG-INFO haveMetadata-Version: 2.4,License-Expression: MIT AND CC-BY-SA-4.0andLicense-File: LICENSE(andLicense-File: NOTICEwith it), the wheeldist-info/licenses/LICENSE(andNOTICE), the sdistLICENSE(andNOTICE). Without NOTICE maturin leaves it out silently, so the smoke test checks the installed licence files against the checkout's.pip install .instead of-e ., with--import-mode=importlib: 2320 passed, andcoverage.xmlhas 0% for everypywhat/*.py(the tests ran the copy in site-packages).uv pip compile --universal --python-version 3.8 requirements.in -o requirements.txt(uv 0.11.28); the file installs on Python 3.8 and 3.12 (local CI) and 3.9/3.10 (CI).main's code, before the config changes: black 24.1.0 would reformat 23 files (22.12.0 and 23.12.1: none); mypy 0.991 to 1.14.1 report the same 2 errors (identifier.py:96implicit Optional,interactive.py:775override), mypy 0.942 fails on Python 3.12; flake8 7.x reports E721 six times infilter.pyandtest_regex_identifier.py, flake8 5.0.4 on 3.12 reports 46 E231 inside f-strings. With the config, all pass on 3.8 and 3.12.actionlint1.7.12 on main.yaml and release.yml: no findings (it found that the first version of my main.yaml steprun: python -m pip install --only-binary :all: …wasn't valid YAML; it is a block now).DOCKER_BUILDKIT=0 docker build --no-cache --cpuset-cpus 0-7 --build-arg PYWHAT_REF=pywhat-rs-py -f Dockerfile <empty dir>: built in 57 s, 83.4 MB;docker run --rm IMAGE --versionprintsPyWhat version 6.0.0a1,docker run --rm IMAGE 'thm{a_flag} bee@skerritt.blog'finds the flag, the URL and the address, andregex_identifier._pywhatis the extension. (With the first version of the Dockerfile the image fell back tore:Error loading shared library libgcc_s.so.1.)releasejob is skipped (not a tag); SonarCloud passes (0 open issues). On 6b62f43, SonarCloud failed on 11 issues: 8 for thepip install dist/*.whllines of release.yml (S8541,S8544), 2 in the Dockerfile andtext:S8565for pyproject.toml without a lock file; 1a867c8 fixes them as described above.Slower: short texts, once per process
Found after the audit, measured on this head (1a867c8), and fixed in the follow-up PR (pywhat-rs-py-b), so not changed here:
identify()of a process compiles its patterns one after the other. With the Rust engine that takes about 100 ms (the\w{86}-style Unicode classes of the Discord and Guilded webhook regexes take 25 to 30 ms each), against about 40 ms withre. Every later call is as fast as before (0.5 to 0.9 ms on a short text). The follow-up compiles the patterns of a search at once on up to 8 threads that only live while they compile (fork-safe): firstidentify()36 to 53 ms.__version__fromimportlib.metadatamakesimport pywhatabout 17 ms slower (84 → 101 ms,-X importtime). The follow-up looks it up when it is first used (PEP 562).pywhat --no-progress 'thm{a} bee@skerritt.blog', 30 interleaved runs each, medians, load average 66 to 71): main 264 ms, this head 313 ms, the follow-up 244 ms. Onfixtures/file: 316, 346 and 288 ms. Large inputs are 4 to 6 times faster, as above.Not in this PR
cli_mainandwhat.main()on the Rust CLI: second PR (pywhat-rs-py-b), after pywhat-rs-cli's PR.