Skip to content

Rust port phase 4: the what/pywhat CLI in Rust, byte for byte like pyWhat's (#291) - #305

Open
bee-san wants to merge 5 commits into
mainfrom
pywhat-rs-cli
Open

bee-san wants to merge 5 commits into
mainfrom
pywhat-rs-cli

Conversation

@bee-san

@bee-san bee-san commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Status: audited at cea494f (MERGE_OK, reports/audit-pywhat-rs-cli.md), then rebased onto main 0c78b02 (#300, #303, #304), with the #303 conflict resolved (children kept in the default features; cli implies files and mac-vendors) and the line of --help that #300 changed (src/cli/help.txt). That delta is in re-audit. The audit's non-blocking findings (F1–F10) are fixed in a follow-up PR, as ciphey-lead decided. Until then, two differences from the Python CLI that the audit found are in this PR: what "" with stdin piped searches the empty text (Python reads stdin), and an option without its value (or a flag with one) prints click's usage line before the error, where click prints only Error: ....

The what and pywhat commands in Rust, with the options, help, errors, exit codes and output of pyWhat's click CLI (#291, phase 4). pywhat::cli::main(args) -> i32 runs them; both binaries call it, and the Python wheel can too (pywhat-rs-py).

What it adds

  • Feature cli (default): src/cli/, src/bin/what.rs, src/bin/pywhat.rs (required-features = ["cli"]).
    • Every option of pywhat/what.py, with the same defaults, help text and exit codes, and the same error messages apart from the usage line above. The arguments are parsed by hand, the way click 8.1 parses them: multi-letter "short" options (-br, -db, -dp, -if, -pt), clusters (-ok name), attached values (-kname, --top=5), --, and unknown options taken as inputs (what -1606586780 searches -1606586780). Usage errors exit with 2.
    • --help is click's HelpFormatter and Python's textwrap, ported, so it wraps exactly like Python at every width (50 to 78 columns).
    • Raw output, custom --format strings (\\ and \%, literal substitution), --json (Identified::to_json), --stream (JSON lines without "Fragment", and raw), Nothing found! and the --top note on stderr print the same bytes as Python. The markup in the data files (some descriptions, file signature links) and in --format strings is rendered the way rich renders it.
    • --format pretty: the same cells, in a comfy-table table in rich's HEAVY_HEAD look. When it fits the terminal, it is the same table.
    • Colours (bold #D7AFFF) only on a terminal, using rich's colour-system detection (COLORTERM, TERM), and no colours with NO_COLOR.
    • Progress bars (indicatif) follow progress.py's rules: only on a terminal, only after 0.5 s, on stderr if stdout is a regular file, never into a pipe, --no-progress turns them off. They look like rich's (label, bar, percentage or file count, elapsed time, the regex or file, cut with an ellipsis).
    • --interactive stays in Python: the Rust binary prints a one-line error that it needs the Python package (pip install pywhat) and exits with 2.
  • Feature verify (default): --verify with ureq 3.5.1 (rustls). GoogleApiKeyVerifier has the semantics of GoogleAPIKeyVerifier: one request per key, the same URL, User-Agent and 10 s timeout, the same reasons that mean "valid", the same messages. with_http injects the HTTP call; no test touches the network.
  • A progress hook in the crate (additive, approved by pywhat-rs): pywhat::Progress and Identifier::iter_identify_with_progress. iter_identify keeps its signature and output, and runs without a callback. The per-regex callback is called when each regex search starts and ends (under rayon for texts over 4 KB), so the bars show progress while the big regexes run.
  • On wasm32 the cli and verify features add nothing: their dependencies are cfg(not(target_arch = "wasm32")), and cargo build --lib --target wasm32-unknown-unknown works with default features, as on the base.

Parity

scripts/parity_cli.py runs 272 command lines with COLUMNS=10000 python -m pywhat … and target/release/what … and compares stdout bytes and exit codes. 105 of them are every command line of tests/test_click.py, read from the file so they stay in sync. The others are every option on fixtures/file, the README's and --help's examples, parser corner cases, stdin (UTF-16 with BOM, UTF-16LE strings, invalid UTF-8), files in a scratch directory (signatures, -if, tabs and hidden files), help at several widths, and markup. For --format pretty it compares the text around the table and the cells.

Python lists a directory in the order of the file system, Rust in name order (#291, Python bug 5). The script runs Python with glob listing directories in name order (PYTHON_CLI in the script; only the order changes), so directories compare byte for byte on any file system. Without it, a directory written as b.txt, c/a.txt, a on tmpfs prints THM{a} THM{c} THM{b} in Python and THM{a} THM{b} THM{c} in Rust; with it, both print the second.

$ PYWHAT_CLI=target/release/what python scripts/parity_cli.py
...
272 command lines (105 from tests/test_click.py): 9 known difference, 263 same   (cea494f)
273 command lines (106 from tests/test_click.py): 9 known difference, 264 same   (5f38d67, on main 0c78b02)

Known differences (listed in KNOWN, with the reason):

  • The matched text and file names as rich markup and emoji codes (Port pyWhat to Rust (building on lemmeknow) #291, Python bug 4): pywhat 'thm{[bold]x[/bold]}' prints thm{x} and THM{:smile:} prints an emoji; pywhat 'thm{[/x]}' crashes with a MarkupError. Rust prints the text as it is (5 command lines, in raw, --format and pretty output).
  • Emoji codes in a --format string (:smile: %m): rich prints an emoji, Rust the code.
  • -v: the crate's version, 6.0.0-alpha.1 (Python's __version__ is 5.0.0).
  • python -m pywhat without arguments prints the help (pywhat/__main__.py). The pywhat/what commands read stdin, in Python and in Rust.
  • Not in the script: a file that can't be read is reported on stderr and skipped, exit code 1 (Python stops with a PermissionError traceback, exit code 1). On Windows, output redirected to a file is UTF-8 (Python writes the ANSI code page with \u20bf escapes for what it can't encode). Arguments that aren't valid UTF-8 get U+FFFD (Python keeps surrogate escapes).

PYWHAT_CLI=target/release/what pytest tests/test_click.py runs pyWhat's CLI tests against the binary through a fixture in tests/conftest.py: CliRunner.invoke runs $PYWHAT_CLI in a subprocess (stdin, env, merged stdout/stderr like CliRunner's; a pseudo-terminal for the test whose stdin is a terminal) and returns its exit code and output. The 4 test functions that call or patch Python internals are skipped: test_stream_prints_before_the_search_is_complete, test_progress_bars (4 parametrizations), test_no_progress and test_print_stream_api, 7 tests. 2 more are skipped in Python too ("Key:value turned off"). Without PYWHAT_CLI nothing changes.

Speed

hyperfine 2.0.0, -N -w 1 -r 5 for the big file, -w 2 -r 10 for the others, on the shared 64-core host (load average 61 to 83), JSON to /dev/null:

what (Rust) pywhat (Python 3.10)
--json --no-progress target/bench.txt (4.2 MB, scripts/bench_corpus.py) 3.05 s ± 0.06 wall, 3.64 s user, 366 MiB 22.87 s ± 0.34 wall, 21.72 s user, 632 MiB
--json --no-progress fixtures/file 0.14 s 0.36 s
--no-progress 'THM{a}' 0.12 s 0.30 s

That is 7.5× on the big file (the auditor measured 7.7×). Both print the same 296,955,554 bytes for it (target/bench.txt of the crate branch: scripts/bench_corpus.py reads tests/*.py, so with this PR's tests/conftest.py the file and the output are larger, 298,631,984 bytes, the same in both again) (cmp, sha256 c1ef36f2c42e1006…).

Commands and results

All on this branch's head unless it says otherwise. Python 3.10.20 with click 8.1.2, rich 12.2.0 and pytest 7.1.2 (poetry.lock's versions), Rust 1.99.1, on a shared 64-core host.

Local CI, $CIPHEY_TC/gate.sh pywhat-rs-cli-ci.sh (it mirrors the Rust workflow's lint, test, wasm32 and CLI jobs and the Python suite): LOCAL CI PASS.

Command Result
cargo fmt --all -- --check ok
cargo clippy --locked --all-targets -- -D warnings ok
cargo clippy --locked --all-targets --no-default-features -- -D warnings ok
cargo clippy --locked --lib --no-default-features --features wasm -- -D warnings ok
cargo test --locked lib 67 passed; tests/cli.rs 23 passed; examples 3, golden 2, whole 2, doctests 4 passed
cargo test --locked --no-default-features ok
cargo build --locked --lib --target wasm32-unknown-unknown (default features) ok
black --check, isort --check-only, flake8, mypy (poetry.lock's versions) on tests/conftest.py and scripts/parity_cli.py ok
pytest -o addopts="" (the whole Python suite, without PYWHAT_CLI) 2116 passed, 6 skipped
PYWHAT_CLI=target/release/what pytest -o addopts="" tests/test_click.py 144 passed, 9 skipped
PYWHAT_CLI=target/release/what python scripts/parity_cli.py 272 command lines (105 from tests/test_click.py): 9 known difference, 263 same

CI on cea494f: all 17 checks pass: the 9 Python jobs (3.8 to 3.10 on Ubuntu, Windows and macOS), Rust fmt and clippy, Rust tests on Ubuntu and Windows, MSRV 1.88, wasm32, golden, the new parity job, and SonarCloud's quality gate. On bea55e4 the Windows Python jobs failed in mypy on tests/conftest.py (Module has no attribute "openpty": typeshed has no pty.openpty on Windows). cea494f checks sys.platform first, so mypy skips that part on Windows; mypy --platform {linux,win32,darwin} --python-version {3.8,3.9,3.10} passes on both files, and the local CI script now runs those 9 combinations.

Other checks:

Command Result
python scripts/golden.py && PYWHAT_GOLDEN=target/golden.jsonl PYWHAT_GOLDEN_FILES=target/golden-files.jsonl cargo test --locked --release --test golden -- --nocapture (at bd9ae06, before the rebase; src/files.rs, src/identify.rs and src/lib.rs are the same at this head) golden files: 16 cases, 0 differences; golden: 40588 texts, 133424 comparisons, 0 differences {}, 30 documented Unicode differences
cargo test --locked --test whole 2 passed
cargo package --allow-dirty --locked Packaged 37 files, 2.1MiB, verified (the bins build from the package; src/cli/help.txt is in it)
cargo tree: rust-version of every dependency at most 1.88 (sha2), the crate's MSRV
pywhat --json --no-progress target/bench.txt vs what --json --no-progress target/bench.txt, cmp the same 296,955,554 bytes, sha256 c1ef36f2c42e10062bc8a7f3766722ad67656442002e02ca0c7ea0c6e713f68a
hyperfine -N -w 1 -r 5 'target/release/what --json --no-progress target/bench.txt' '.venv/bin/python -m pywhat --json --no-progress target/bench.txt' 3.049 s ± 0.058 vs 22.866 s ± 0.338 (7.5×), see Speed
what --verify --format '%d' AIzaSyA0…0 and the same with python -m pywhat (the README's fake key, one real request each) both: Verification: invalid, Google rejected the key (API_KEY_INVALID)
Progress bars, under script (a pseudo-terminal): what --json target/bench.txt > out.json bars on the terminal (stderr) after 0.5 s, then removed; out.json is byte-identical to --no-progress
The same into a pipe (| wc -c), for a fast search, and with --no-progress no bars
what --json DIR (two 4.2 MB files) under script a Files row (1/2, the file) and a Regexes row
what --stream --format '%n' target/bench.txt under script, with and without bars the same 29,160 lines
A directory with an unreadable file, Python and Rust Python: PermissionError traceback, exit 1; Rust: the readable file's matches, Could not read /b.txt: Permission denied (os error 13) on stderr, exit 1

Files outside src/cli, src/bin, tests/cli.rs and scripts/parity_cli.py

  • Cargo.toml: the cli and verify features, the bins, [[test]] cli, and the CLI dependencies under cfg(not(target_arch = "wasm32")): comfy-table 8.0.1, indicatif 0.18.6 (no default features), anstyle-query 1.1.5 (Windows consoles), terminal_size 0.4.4, ureq 3.5.1 (rustls, no gzip). The newest rust-version among all dependencies is 1.88, the crate's MSRV. ureq's rustls brings ring, which compiles C and assembly; it is only in the verify feature and not on wasm32.
  • src/lib.rs: pub mod cli (feature cli, not on wasm32) and the Progress export.
  • src/files.rs, src/identify.rs: the progress hook above.
  • tests/conftest.py: the PYWHAT_CLI fixture.
  • .github/workflows/rust.yml: a job "Rust CLI prints what Python prints (parity)" that builds the binary, runs scripts/parity_cli.py and pytest tests/test_click.py against it.
  • docs/rust.md: the cli and verify rows of the features table and the progress hook's row of the API table (3 lines). The CLI's documentation is in a new docs/cli.md.

No change to pywhat/, pywhat/Data/ or the README.

@bee-san

bee-san commented Nov 5, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto main 0c78b02 (#300, #303, #304). New head 5f38d67. The audited commits, git range-diff 4030351..cea494f origin/main..5f38d67:

1:  2ae2808 ! 1:  e4099e1 files: a progress callback for iter_identify (pywhat::Progress)
    @@ docs/rust.md: assert!(found.iter().any(|m| m.matched == "bee@skerritt.blog" && m
     +| `identifier.iter_identify_with_progress(&inputs, options, found, error, progress)` and `Progress` (feature `files`) | `iter_identify(..., progress=...)` and `Progress`: `progress` gets the file, how many of the files and of the file's regex searches are done, and a regex that is searched for. For texts over 4 KB the regexes are searched for in parallel, so it is called from rayon's threads, one call at a time |
      | `Identified::to_json()` | `json.dumps(identify(...))`, byte for byte |
      | `Filter::new(tags, exclude_tags, min_rarity, max_rarity)`, `Filter::default()`, `Filter::none()`, `.and()`, `.or()` | `Filter({...})`, `&`, `\|` |
    - | `Processor` (trait), `default_processors()`, `processors::OpenStreetMap` | `Processor`, `default_processors()`, `OpenStreetMapProcessor` |
    + | `Processor` (trait), `default_processors()`, `processors::OpenStreetMap`, `processors::Validator` | `Processor`, `default_processors()`, `OpenStreetMapProcessor`, `ValidatorProcessor` |
     
      ## src/files.rs ##
     @@
    @@ src/identify.rs: pub(crate) fn check(text: &str, search: &Search<'_>) -> Result<
     
      ## src/lib.rs ##
     @@ src/lib.rs: pub use bitcoin::is_bitcoin_address;
    - pub use data::{regexes, tags, RegexEntry};
    + pub use data::{regexes, tags, RegexEntry, CHARSET_ONLY};
      pub use engine::{EngineKind, MatchError};
      #[cfg(feature = "files")]
     -pub use files::{Found, Identified, InputOptions};
2:  bea55e4 ! 2:  f1bc150 The what and pywhat commands in Rust, like pyWhat's CLI (#291 phase 4)
    @@ Cargo.lock: dependencies = [
      [[package]]
      name = "hashbrown"
      version = "0.17.1"
    +@@ Cargo.lock: version = "0.5.0"
      source = "registry+https://github.com/rust-lang/crates.io-index"
    - checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
    + checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
      
     +[[package]]
     +name = "http"
    @@ Cargo.lock: version = "1.21.4"
     +version = "2.3.2"
     +source = "registry+https://github.com/rust-lang/crates.io-index"
     +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
    -+
    -+[[package]]
    -+name = "portable-atomic"
    -+version = "1.15.0"
    -+source = "registry+https://github.com/rust-lang/crates.io-index"
    -+checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
     +
      [[package]]
    - name = "proc-macro2"
    - version = "1.0.107"
    + name = "portable-atomic"
    + version = "1.15.0"
     @@ Cargo.lock: dependencies = [
      name = "pywhat"
      version = "6.0.0-alpha.1"
    @@ Cargo.lock: dependencies = [
       "lemmeknow",
       "rayon",
       "regex",
    +@@ Cargo.lock: dependencies = [
       "regex-syntax",
       "serde_json",
       "sha2",
    @@ Cargo.lock: dependencies = [
     +
      [[package]]
      name = "syn"
    - version = "3.0.9"
    -@@ Cargo.lock: dependencies = [
    -  "unicode-ident",
    - ]
    + version = "2.0.119"
    +@@ Cargo.lock: version = "0.13.5"
    + source = "registry+https://github.com/rust-lang/crates.io-index"
    + checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
      
     +[[package]]
     +name = "terminal_size"
    @@ Cargo.lock: dependencies = [
     
      ## Cargo.toml ##
     @@ Cargo.toml: categories = ["command-line-utilities", "text-processing", "wasm"]
    - include = ["/src", "/pywhat/Data/*.json", "/LICENSE", "/README.md"]
    + include = ["/src", "/pywhat/Data/*.json", "/pywhat/Data/regexes/*.json", "/LICENSE", "/NOTICE", "/README.md"]
      
      [features]
    --default = ["parallel", "files", "mac-vendors"]
    -+default = ["parallel", "files", "mac-vendors", "cli", "verify"]
    +-default = ["parallel", "files", "children", "mac-vendors"]
    ++default = ["parallel", "files", "children", "mac-vendors", "cli", "verify"]
     +# The `what` and `pywhat` commands (src/cli/), with the options and the output of the
     +# Python package's CLI: tables, colours and progress bars. pywhat::cli::main is what they
    -+# run, and what the Python package's `pywhat` and `what` commands call.
    ++# run, and what the Python package's `pywhat` and `what` commands call. They need the file
    ++# signatures and every Children table to print what the Python CLI prints.
     +cli = [
     +    "files",
    ++    "mac-vendors",
     +    "dep:comfy-table",
     +    "dep:indicatif",
     +    "dep:anstyle-query",
    @@ Cargo.toml: rayon = { version = "1.12.0", optional = true }
      [dev-dependencies]
      serde_json = { version = "1.0.154", features = ["preserve_order"] }
      # Only for benches/whole.rs, which compares the speed with lemmeknow, the crate that Ciphey
    - # used before.
    - lemmeknow = { version = "=0.8.0", default-features = false }
    +@@ Cargo.toml: lemmeknow = { version = "=0.8.0", default-features = false }
    + # tests/charset_only.rs builds the DFA of every regex
    + regex-automata = { version = "0.4.18", features = ["dfa-build"] }
      
     +[[bin]]
     +name = "what"
    @@ docs/cli.md (new)
     
      ## docs/rust.md ##
     @@ docs/rust.md: Rust only:
    - | `parallel` | yes | rayon: the regexes of a text over 4 KB are searched in parallel, and `warm_up` compiles in parallel |
    - | `files` | yes | `identify_inputs`, `iter_identify`: files, directories (recursive) and file signatures |
    - | `mac-vendors` | yes | the vendors of MAC address prefixes (1.6 MB) in the descriptions of EUI-48 matches |
    -+| `cli` | yes | the `what` and `pywhat` commands and `pywhat::cli::main`, see [cli.md](cli.md) (not on wasm32) |
    + | `files` | yes | `identify_inputs`, `iter_identify`: files, directories (recursive) and file signatures (`file_signatures.json`, under CC BY-SA 4.0) |
    + | `children` | yes | the tables that `Children` read from data files: the locations of phone numbers (`phone_codes.json`) in their descriptions |
    + | `mac-vendors` | yes | `children`, and the vendors of MAC address prefixes (1.6 MB) in the descriptions of EUI-48 matches |
    ++| `cli` | yes | the `what` and `pywhat` commands and `pywhat::cli::main`, see [cli.md](cli.md); it needs `files` and `mac-vendors`, for the output of the Python CLI (not on wasm32) |
     +| `verify` | yes | `what --verify`: asks Google whether Google API keys are valid (not on wasm32) |
      | `wasm` | no | the WebAssembly API in `src/wasm.rs` (`identify(text, options_json)`, `tags()`, `version()`) |
      
3:  cea494f = 3:  896c2fe conftest: the pseudo-terminal part only for mypy on Unix (Windows has no pty.openpty)
-:  ------- > 4:  5f38d67 help.txt: the help of pywhat/what.py after #300 (the checksums that are checked)

Local CI on 5f38d67 (gate.sh pywhat-rs-cli-ci.sh, now with requirements.txt and the extension as nox builds it): LOCAL CI PASS; nox-style suite 2432 passed, 6 skipped; PYWHAT_CLI test_click 144 passed, 9 skipped; parity 273 command lines, 264 the same, 9 known differences. The delta goes to re-audit audit-pywhat-rs-cli-b. The audit's non-blocking findings F1–F10 (and F6's decision: the OS trust store) go into a follow-up PR, as ciphey-lead decided.

Identifier::iter_identify_with_progress() calls a callback with a Progress
(the file, how many files and regex searches of the file are done, and a
regex that is searched for) when each regex search starts and ends, and once
a file is done, like the progress callback of Python's iter_identify(). For
texts over 4 KB the regexes run on rayon's threads, and so does the callback,
one call at a time. iter_identify() keeps its signature and output and runs
without a callback. The CLI's progress bars use it.
pywhat::cli::main(args) -> i32 (feature cli, default) runs the command line of
pywhat/what.py: the same options, defaults, help, errors, exit codes and
output. The arguments are parsed the way click 8.1 parses them (multi-letter
short options, clusters, unknown options as inputs), --help is click's
HelpFormatter and Python's textwrap, and raw, custom --format, --json and
--stream output are byte for byte Python's; --format pretty has the same cells
(comfy-table). Colours only on a terminal (NO_COLOR), progress bars (indicatif)
by progress.py's rules. --verify (feature verify, default) asks Google with
ureq/rustls, once per key, with GoogleAPIKeyVerifier's messages; the HTTP call
is injectable. --interactive needs the Python package. On wasm32 neither
feature adds anything.

scripts/parity_cli.py compares 272 command lines with python -m pywhat (every
case of tests/test_click.py, read from it): 0 differences beyond the 9 known
ones it lists. tests/conftest.py runs tests/test_click.py against $PYWHAT_CLI.
The Rust workflow runs both. docs/cli.md documents the command.
…s without a [[test]] entry

- release.yml: CFLAGS_aarch64_unknown_linux_gnu=-D__ARM_ARCH=8 for maturin's manylinux2014
  aarch64 cross build. ring, the TLS of `what --verify`, needs __ARM_ARCH, which that GCC 4.8
  doesn't define (briansmith/ring#1728); the job failed with "ARM assembler must define
  __ARM_ARCH". maturin-action passes CFLAGS* into its container.
- Cargo.toml/tests/cli.rs: no [[test]] entry; `#![cfg(feature = "cli")]` in tests/cli.rs
  instead. The sdist of the Python package has no tests/, and cargo can't read a manifest
  whose test file is missing, so building the wheel from the sdist failed.
@sonarqubecloud

sonarqubecloud Bot commented Nov 5, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant