Repository navigation
Conversation
bee-san
force-pushed
the
pywhat-rs-cli
branch
from
November 5, 2026 00:33
cea494f to
5f38d67
Compare
Owner
Author
|
Rebased onto main 0c78b02 (#300, #303, #304). New head 5f38d67. The audited commits,
Local CI on 5f38d67 ( |
Identifier::iter_identify_with_progress() calls a callback with a Progress (the file, how many files and regex searches of the file are done, and a regex that is searched for) when each regex search starts and ends, and once a file is done, like the progress callback of Python's iter_identify(). For texts over 4 KB the regexes run on rayon's threads, and so does the callback, one call at a time. iter_identify() keeps its signature and output and runs without a callback. The CLI's progress bars use it.
pywhat::cli::main(args) -> i32 (feature cli, default) runs the command line of pywhat/what.py: the same options, defaults, help, errors, exit codes and output. The arguments are parsed the way click 8.1 parses them (multi-letter short options, clusters, unknown options as inputs), --help is click's HelpFormatter and Python's textwrap, and raw, custom --format, --json and --stream output are byte for byte Python's; --format pretty has the same cells (comfy-table). Colours only on a terminal (NO_COLOR), progress bars (indicatif) by progress.py's rules. --verify (feature verify, default) asks Google with ureq/rustls, once per key, with GoogleAPIKeyVerifier's messages; the HTTP call is injectable. --interactive needs the Python package. On wasm32 neither feature adds anything. scripts/parity_cli.py compares 272 command lines with python -m pywhat (every case of tests/test_click.py, read from it): 0 differences beyond the 9 known ones it lists. tests/conftest.py runs tests/test_click.py against $PYWHAT_CLI. The Rust workflow runs both. docs/cli.md documents the command.
…s without a [[test]] entry - release.yml: CFLAGS_aarch64_unknown_linux_gnu=-D__ARM_ARCH=8 for maturin's manylinux2014 aarch64 cross build. ring, the TLS of `what --verify`, needs __ARM_ARCH, which that GCC 4.8 doesn't define (briansmith/ring#1728); the job failed with "ARM assembler must define __ARM_ARCH". maturin-action passes CFLAGS* into its container. - Cargo.toml/tests/cli.rs: no [[test]] entry; `#![cfg(feature = "cli")]` in tests/cli.rs instead. The sdist of the Python package has no tests/, and cargo can't read a manifest whose test file is missing, so building the wheel from the sdist failed.
bee-san
force-pushed
the
pywhat-rs-cli
branch
from
November 5, 2026 12:54
5f38d67 to
1705fa6
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Status: audited at cea494f (MERGE_OK,
reports/audit-pywhat-rs-cli.md), then rebased onto main 0c78b02 (#300, #303, #304), with the #303 conflict resolved (childrenkept in the default features;cliimpliesfilesandmac-vendors) and the line of--helpthat #300 changed (src/cli/help.txt). That delta is in re-audit. The audit's non-blocking findings (F1–F10) are fixed in a follow-up PR, as ciphey-lead decided. Until then, two differences from the Python CLI that the audit found are in this PR:what ""with stdin piped searches the empty text (Python reads stdin), and an option without its value (or a flag with one) prints click's usage line before the error, where click prints onlyError: ....The
whatandpywhatcommands in Rust, with the options, help, errors, exit codes and output of pyWhat's click CLI (#291, phase 4).pywhat::cli::main(args) -> i32runs them; both binaries call it, and the Python wheel can too (pywhat-rs-py).What it adds
cli(default):src/cli/,src/bin/what.rs,src/bin/pywhat.rs(required-features = ["cli"]).pywhat/what.py, with the same defaults, help text and exit codes, and the same error messages apart from the usage line above. The arguments are parsed by hand, the way click 8.1 parses them: multi-letter "short" options (-br,-db,-dp,-if,-pt), clusters (-ok name), attached values (-kname,--top=5),--, and unknown options taken as inputs (what -1606586780searches-1606586780). Usage errors exit with 2.--helpis click'sHelpFormatterand Python'stextwrap, ported, so it wraps exactly like Python at every width (50 to 78 columns).--formatstrings (\\and\%, literal substitution),--json(Identified::to_json),--stream(JSON lines without"Fragment", and raw),Nothing found!and the--topnote on stderr print the same bytes as Python. The markup in the data files (some descriptions, file signature links) and in--formatstrings is rendered the way rich renders it.--format pretty: the same cells, in a comfy-table table in rich'sHEAVY_HEADlook. When it fits the terminal, it is the same table.#D7AFFF) only on a terminal, using rich's colour-system detection (COLORTERM,TERM), and no colours withNO_COLOR.progress.py's rules: only on a terminal, only after 0.5 s, on stderr if stdout is a regular file, never into a pipe,--no-progressturns them off. They look like rich's (label, bar, percentage or file count, elapsed time, the regex or file, cut with an ellipsis).--interactivestays in Python: the Rust binary prints a one-line error that it needs the Python package (pip install pywhat) and exits with 2.verify(default):--verifywith ureq 3.5.1 (rustls).GoogleApiKeyVerifierhas the semantics ofGoogleAPIKeyVerifier: one request per key, the same URL, User-Agent and 10 s timeout, the same reasons that mean "valid", the same messages.with_httpinjects the HTTP call; no test touches the network.pywhat::ProgressandIdentifier::iter_identify_with_progress.iter_identifykeeps its signature and output, and runs without a callback. The per-regex callback is called when each regex search starts and ends (under rayon for texts over 4 KB), so the bars show progress while the big regexes run.cliandverifyfeatures add nothing: their dependencies arecfg(not(target_arch = "wasm32")), andcargo build --lib --target wasm32-unknown-unknownworks with default features, as on the base.Parity
scripts/parity_cli.pyruns 272 command lines withCOLUMNS=10000 python -m pywhat …andtarget/release/what …and compares stdout bytes and exit codes. 105 of them are every command line oftests/test_click.py, read from the file so they stay in sync. The others are every option onfixtures/file, the README's and--help's examples, parser corner cases, stdin (UTF-16 with BOM, UTF-16LE strings, invalid UTF-8), files in a scratch directory (signatures,-if, tabs and hidden files), help at several widths, and markup. For--format prettyit compares the text around the table and the cells.Python lists a directory in the order of the file system, Rust in name order (#291, Python bug 5). The script runs Python with glob listing directories in name order (
PYTHON_CLIin the script; only the order changes), so directories compare byte for byte on any file system. Without it, a directory written asb.txt,c/a.txt,aon tmpfs printsTHM{a} THM{c} THM{b}in Python andTHM{a} THM{b} THM{c}in Rust; with it, both print the second.Known differences (listed in
KNOWN, with the reason):pywhat 'thm{[bold]x[/bold]}'printsthm{x}andTHM{:smile:}prints an emoji;pywhat 'thm{[/x]}'crashes with aMarkupError. Rust prints the text as it is (5 command lines, in raw,--formatand pretty output).--formatstring (:smile: %m): rich prints an emoji, Rust the code.-v: the crate's version, 6.0.0-alpha.1 (Python's__version__is 5.0.0).python -m pywhatwithout arguments prints the help (pywhat/__main__.py). Thepywhat/whatcommands read stdin, in Python and in Rust.PermissionErrortraceback, exit code 1). On Windows, output redirected to a file is UTF-8 (Python writes the ANSI code page with\u20bfescapes for what it can't encode). Arguments that aren't valid UTF-8 get U+FFFD (Python keeps surrogate escapes).PYWHAT_CLI=target/release/what pytest tests/test_click.pyruns pyWhat's CLI tests against the binary through a fixture intests/conftest.py:CliRunner.invokeruns$PYWHAT_CLIin a subprocess (stdin, env, merged stdout/stderr like CliRunner's; a pseudo-terminal for the test whose stdin is a terminal) and returns its exit code and output. The 4 test functions that call or patch Python internals are skipped:test_stream_prints_before_the_search_is_complete,test_progress_bars(4 parametrizations),test_no_progressandtest_print_stream_api, 7 tests. 2 more are skipped in Python too ("Key:value turned off"). WithoutPYWHAT_CLInothing changes.Speed
hyperfine 2.0.0,
-N -w 1 -r 5for the big file,-w 2 -r 10for the others, on the shared 64-core host (load average 61 to 83), JSON to/dev/null:what(Rust)pywhat(Python 3.10)--json --no-progress target/bench.txt(4.2 MB,scripts/bench_corpus.py)--json --no-progress fixtures/file--no-progress 'THM{a}'That is 7.5× on the big file (the auditor measured 7.7×). Both print the same 296,955,554 bytes for it (
target/bench.txtof the crate branch:scripts/bench_corpus.pyreadstests/*.py, so with this PR'stests/conftest.pythe file and the output are larger, 298,631,984 bytes, the same in both again) (cmp, sha256c1ef36f2c42e1006…).Commands and results
All on this branch's head unless it says otherwise. Python 3.10.20 with click 8.1.2, rich 12.2.0 and pytest 7.1.2 (poetry.lock's versions), Rust 1.99.1, on a shared 64-core host.
Local CI,
$CIPHEY_TC/gate.sh pywhat-rs-cli-ci.sh(it mirrors the Rust workflow's lint, test, wasm32 and CLI jobs and the Python suite):LOCAL CI PASS.cargo fmt --all -- --checkcargo clippy --locked --all-targets -- -D warningscargo clippy --locked --all-targets --no-default-features -- -D warningscargo clippy --locked --lib --no-default-features --features wasm -- -D warningscargo test --lockedtests/cli.rs23 passed; examples 3, golden 2, whole 2, doctests 4 passedcargo test --locked --no-default-featurescargo build --locked --lib --target wasm32-unknown-unknown(default features)black --check,isort --check-only,flake8,mypy(poetry.lock's versions) ontests/conftest.pyandscripts/parity_cli.pypytest -o addopts=""(the whole Python suite, withoutPYWHAT_CLI)PYWHAT_CLI=target/release/what pytest -o addopts="" tests/test_click.pyPYWHAT_CLI=target/release/what python scripts/parity_cli.pyCI on cea494f: all 17 checks pass: the 9 Python jobs (3.8 to 3.10 on Ubuntu, Windows and macOS), Rust fmt and clippy, Rust tests on Ubuntu and Windows, MSRV 1.88, wasm32, golden, the new parity job, and SonarCloud's quality gate. On bea55e4 the Windows Python jobs failed in mypy on
tests/conftest.py(Module has no attribute "openpty": typeshed has nopty.openptyon Windows). cea494f checkssys.platformfirst, so mypy skips that part on Windows;mypy --platform {linux,win32,darwin} --python-version {3.8,3.9,3.10}passes on both files, and the local CI script now runs those 9 combinations.Other checks:
python scripts/golden.py && PYWHAT_GOLDEN=target/golden.jsonl PYWHAT_GOLDEN_FILES=target/golden-files.jsonl cargo test --locked --release --test golden -- --nocapture(at bd9ae06, before the rebase;src/files.rs,src/identify.rsandsrc/lib.rsare the same at this head)golden files: 16 cases, 0 differences;golden: 40588 texts, 133424 comparisons, 0 differences {}, 30 documented Unicode differencescargo test --locked --test wholecargo package --allow-dirty --lockedPackaged 37 files, 2.1MiB, verified (the bins build from the package;src/cli/help.txtis in it)cargo tree:rust-versionof every dependencypywhat --json --no-progress target/bench.txtvswhat --json --no-progress target/bench.txt,cmpc1ef36f2c42e10062bc8a7f3766722ad67656442002e02ca0c7ea0c6e713f68ahyperfine -N -w 1 -r 5 'target/release/what --json --no-progress target/bench.txt' '.venv/bin/python -m pywhat --json --no-progress target/bench.txt'what --verify --format '%d' AIzaSyA0…0and the same withpython -m pywhat(the README's fake key, one real request each)Verification: invalid, Google rejected the key (API_KEY_INVALID)script(a pseudo-terminal):what --json target/bench.txt > out.jsonout.jsonis byte-identical to--no-progress| wc -c), for a fast search, and with--no-progresswhat --json DIR(two 4.2 MB files) underscript1/2, the file) and a Regexes rowwhat --stream --format '%n' target/bench.txtunderscript, with and without barsPermissionErrortraceback, exit 1; Rust: the readable file's matches,Could not read /b.txt: Permission denied (os error 13)on stderr, exit 1Files outside
src/cli,src/bin,tests/cli.rsandscripts/parity_cli.pyCargo.toml: thecliandverifyfeatures, the bins,[[test]] cli, and the CLI dependencies undercfg(not(target_arch = "wasm32")): comfy-table 8.0.1, indicatif 0.18.6 (no default features), anstyle-query 1.1.5 (Windows consoles), terminal_size 0.4.4, ureq 3.5.1 (rustls, no gzip). The newestrust-versionamong all dependencies is 1.88, the crate's MSRV. ureq's rustls bringsring, which compiles C and assembly; it is only in theverifyfeature and not on wasm32.src/lib.rs:pub mod cli(featurecli, not on wasm32) and theProgressexport.src/files.rs,src/identify.rs: the progress hook above.tests/conftest.py: thePYWHAT_CLIfixture..github/workflows/rust.yml: a job "Rust CLI prints what Python prints (parity)" that builds the binary, runsscripts/parity_cli.pyandpytest tests/test_click.pyagainst it.docs/rust.md: thecliandverifyrows of the features table and the progress hook's row of the API table (3 lines). The CLI's documentation is in a newdocs/cli.md.No change to
pywhat/,pywhat/Data/or the README.