Repository navigation
Conversation
Identifier::iter_identify_with_progress() calls a callback with a Progress (the file, how many files and regex searches of the file are done, and a regex that is searched for) when each regex search starts and ends, and once a file is done, like the progress callback of Python's iter_identify(). For texts over 4 KB the regexes run on rayon's threads, and so does the callback, one call at a time. iter_identify() keeps its signature and output and runs without a callback. The CLI's progress bars use it.
pywhat::cli::main(args) -> i32 (feature cli, default) runs the command line of pywhat/what.py: the same options, defaults, help, errors, exit codes and output. The arguments are parsed the way click 8.1 parses them (multi-letter short options, clusters, unknown options as inputs), --help is click's HelpFormatter and Python's textwrap, and raw, custom --format, --json and --stream output are byte for byte Python's; --format pretty has the same cells (comfy-table). Colours only on a terminal (NO_COLOR), progress bars (indicatif) by progress.py's rules. --verify (feature verify, default) asks Google with ureq/rustls, once per key, with GoogleAPIKeyVerifier's messages; the HTTP call is injectable. --interactive needs the Python package. On wasm32 neither feature adds anything. scripts/parity_cli.py compares 272 command lines with python -m pywhat (every case of tests/test_click.py, read from it): 0 differences beyond the 9 known ones it lists. tests/conftest.py runs tests/test_click.py against $PYWHAT_CLI. The Rust workflow runs both. docs/cli.md documents the command.
… for --format; parity compares stderr
… urlopen; stdin when all inputs are empty; docs
…res without matches, null exploits, stdin strip, colours, control codes, meta tags, the 0.5 s delay, progress of every text, 2xx, the HTTP request
…aw and pretty_print split up; docs for the audit's findings
…form-verifier, ring), no bundled CA list (audit F6)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



The fixes for the audit of #305 (
reports/audit-pywhat-rs-cli.md, MERGE_OK with ten non-blocking findings), in their own pull request as ciphey-lead decided, so that #305 didn't wait for them. Stacked on #305: the first four commits of this branch are #305's. Once #305 is on main, this branch is rebased onto it, and the diff below is what is left.What changes
Closer to the Python CLI
get_text()'snot any(value):producer | what ""searches stdin, aspywhat ""does.what --rarity) or a flag with one (--json=1) prints onlyError: Option '--rarity' requires an argument., as click 8.1 does: its parser raises these without a context. An invalid--mapvalue still prints the usage line and the hint first, as click does there.--formatstring with a closing tag that closes nothing (--format '[b]%m[/i]') prints nothing and exits with 1, like Python, where rich raises itsMarkupError. Rust says why on stderr (Error: closing tag '[/i]' at position 9 doesn't match any open tag); Python prints a traceback. Closing tags match the way rich matches them ([b]…[/bold],[bold red]…[/red bold],[/]).--verify: Google's answer is read the wayjson.loads()reads it (a UTF-8 BOM, UTF-16 and UTF-32,NaNandInfinity, integers of any size, the last of duplicate keys, the first detail that isn't an object ends the search). The request works likeurlopen(..., timeout=10): 10 s to connect and for every read and write rather than for the whole request, 301/302/303/307 redirects followed (up to 10), and the common errors described in urllib's words (timed out,<urlopen error [Errno 111] Connection refused>,<urlopen error [Errno -2] Name or service not known>). Against the auditor's fake Google (verify_cmp.py), all 39 scenarios now give Python's answer (32 before).TERMrules: a dumb terminal is checked beforeTERMis stripped, and the colour depth is what follows the last-(TERM=256coloris 8-bit). On a dumb terminal the table is 80 columns wide unlessCOLUMNSandLINESare both set, as in rich. On a terminal,File: <location>is one styled span, as rich prints Python's markup.F6, the root certificates:
--verifytrusts the operating system's root certificates through ureq'splatform-verifierfeature (rustls-platform-verifier 0.7.1, MIT OR Apache-2.0), with rustls on ring, like Python's urllib. webpki-roots (Mozilla's list, CDLA-Permissive-2.0) is no longer a dependency. rustls-platform-verifier depends on webpki-root-certs only for wasm32, whereverifyisn't built:cargo tree -i webpki-root-certsis empty for Linux, macOS and Windows. WithSSL_CERT_FILEandSSL_CERT_DIRpointing nowhere, both CLIs fail the TLS handshake (could not ask Google (...)); with the system store, both get Google's answer.F8, tests: the 20 of the auditor's 41 mutants that survived, rerun on this branch with the auditor's harness: 17 are killed now; 2 can't be applied, because the code they mutate was rewritten (D1 is F1's fix, V2 the old JSON handling); O7 (pretty cells not sanitized) is equivalent, as matches have no control characters left. The new tests: the parser's
=values, empty values, unknown--opt=valueand the order of the callbacks; a file signature without matches; matched text with markup in--formatand the table; null exploits in the table; stdin'sstr.strip(); colour detection and control codes like rich's;[@…]meta tags; the 0.5 s delay of the bars (an in-memory terminal); the progress of every text of a file; any 2xx is valid; the request (User-Agent, redirects, a refused connection) against a local server; Python'sdescribe_error()on 54 bodies.parity_cli.pynow compares stderr too (unless Python prints a traceback), fails a known difference that stops differing, and checks thatsrc/cli/help.txtis what.py's help (--update-helpwrites it).F4, F9, docs: docs/cli.md lists the remaining differences: rich's width effects (only the table and the help depend on the width in Rust), output that is always UTF-8 (with U+FFFD for names that aren't), Ctrl-C, the colours of the table, and the root certificates.
F10, SonarCloud:
main_commandis split into the search,--streamand the complete output (SearchRun,StreamPrinter,complete_search), andprint_raw/pretty_printintoraw_matchanddescription_cell;PoisonError::into_innerfor the 7 closures; no redundant'static.Parity
The known differences are #305's 9 (matched text and file names as markup, Python bug 4; emoji codes in
--format;-v;python -m pywhatwithout arguments) and a tab in a file name in the table: rich counts it as 0 columns wide when it sizes the column and then wraps the cell, comfy-table shows it expanded. 26 command lines are new: the parser's values and errors, markup errors, empty inputs with stdin, stdin's whitespace, a null exploit in the table, a signature without matches, a tab in a file name.A random differential of options and inputs against Python (
random-differential.py) found no other difference: 1,795 command lines on this head (3 seeds), and 2,785 on the earlier revisions.Commands and results
On this branch's head, rebased on main 0c78b02 (with #305's commits below it). Python 3.10.20, Rust 1.99.1, shared 64-core host.
$CIPHEY_TC/gate.sh pywhat-rs-cli-ci.sh(fmt; clippy default, no-default and wasm; cargo test;--no-default-features; the wasm32 lib with default features; requirements.txt, the extension with maturin, the suite as nox runs it; mypy on the CLI's Python files on Python 3.8 and 3.10 for Linux, Windows and macOS; test_click against the binary; parity)LOCAL CI PASScargo test --lockedtests/cli.rs24 passed, examples 3, charset_only 2, golden 2, whole 2, doctests 5pytest --cache-clear(nox's way: black, isort, flake8, mypy, the Rust engine)PYWHAT_CLI=target/release/what pytest -o addopts="" tests/test_click.pyPYWHAT_CLI=target/release/what python scripts/parity_cli.pyverify_cmp.pyagainst a build that reads Google's URL from an environment variable (scratch patch, not committed)mrun.pyon its 20 surviving mutantscargo tree -e normal --target {x86_64-unknown-linux-gnu,x86_64-pc-windows-msvc,aarch64-apple-darwin}, the crates thatcliandverifyaddwhat --verify --format '%d' AIzaSyA0…0(one real request)Verification: invalid, Google rejected the key (API_KEY_INVALID)Evidence:
/local/home/skerraut/herd-synth3/reports/pywhat-rs-cli-evidence/(local-ci-fixes.log,verify-fake-google-next.txt,mutants-next.txt,licences-cli-*.txt,random-differential.*). The answer to each finding:reports/pywhat-rs-cli-audit-response.md.