Skip to content

Follow-ups from the audits of the crate and the false-positive PRs - #306

Merged
bee-san merged 1 commit into
mainfrom
pywhat-rs-followups
Nov 5, 2026
Merged

bee-san merged 1 commit into
mainfrom
pywhat-rs-followups

Conversation

@bee-san

@bee-san bee-san commented Nov 4, 2026

Copy link
Copy Markdown
Owner

The non-blocking findings of the audits of #299 (the crate) and #300 (false positives), fixed. This PR is stacked on #303: only the last commit, a0d89ef, is new here.

Changes

  • Top::of computes percentages the way Python does (crate audit F10).
    • Python's Decimal rounds total × percent to 28 significant digits (half to even), then divides by 100 and takes the ceiling, both exactly. The Rust code now does the same, with integer arithmetic.
    • It no longer overflows: before, 50.000000000000000000% of usize::MAX overflowed u128.
    • It no longer panics on a Top::Percent that parse() wouldn't return.
    • parse() strips whitespace like Python's str.strip() and \s, which include U+001C to U+001F.
    • A differential against Python's Top.of gives 0 differences in 119,016 (percentage, total) pairs, over 6,018 generated and adversarial percentages and 24 totals up to 2^64−1. parse() agrees on all 6,018.
  • validators::facebook_access_token in Rust skips two characters, not two bytes, as Python's token[2:] does (FP audit F15). The two versions differed on non-ASCII input. Both test suites now test the same cases.
  • Two Unicode examples (FP audit F14), so both suites and the golden test cover these paths:
    • a CashAddr with a Kelvin sign K, as an invalid Bitcoin Cash example;
    • 4111 1111 1111 1111 with ideographic spaces, as a valid Visa example.
  • The golden test (crate audit F11):
    • A new option set, and_exclude, catches Filter::and excluding the wrong tags. Replacing its intersection of the excluded tags with a union gives 13 differences {"and_exclude": 13}.
    • scripts/golden.py ends both recordings with their counts. A cut recording, or one without its JSON records, now fails (checked: has no trailer: cut off?).
  • Area pull requests don't conflict in the two shared lists (FP audit F13). A regex that needs fancy-regex goes into its area's block of AREAS_NOT_ON_THE_REGEX_CRATE in tests/examples.rs. A regex whose Charset Only tag is decided by hand goes into its area's block of AREAS_BY_HAND in tests/charset_only.rs. Both lists are compared by name, so an entry's place doesn't depend on its rarity. The contract says so, and also that validators add no crates without the maintainers' agreement, since two areas that each add one would conflict in Cargo.toml and Cargo.lock.
  • Stale text (crate audit F12, FP audit F16):
    • CONTRIBUTING.md starts with the contract and says the wiki predates the area files.
    • The golden test's commands are current.
    • "regex.json" now reads "regex.json or its area's file" where a regex can be in either.
    • The contract gives the Charset Only lengths correctly.
    • The Ciphey migration note lists the 14 files that import lemmeknow (grep -rl 'lemmeknow::' src); detection.rs changes in step 6.
    • The big-file JSON is 296,955,554 bytes.

Checks

  • Local CI mirror: nox on Python 3.10, then cargo fmt, clippy with default features, none, children and wasm, and cargo test with default features, none and children. It passes: pytest 2,410 passed, 6 skipped.
  • Golden: golden: 43454 texts, 143531 comparisons, 0 differences {}, 30 documented Unicode differences, golden files: 16 cases, 0 differences.

Non-blocking findings of audit-pywhat-rs-crate-b and audit-pywhat-rs-fp-b:

* Top::of computes percentages as Python does: Decimal rounds total ×
  percent to 28 significant digits (half to even), then the division by 100
  and the ceiling are exact. It no longer overflows u128 for large totals
  (50.000000000000000000% of usize::MAX), and it no longer panics on a
  Top::Percent that parse() wouldn't return. parse() strips Python's
  whitespace, including U+001C to U+001F. Differential against Python's
  Top.of: 0 differences in 119,016 (percentage, total) pairs.
* validators::facebook_access_token skips the first two characters, not two
  bytes, as Python's token[2:] does (they differed on non-ASCII input).
* Examples that test both languages on Unicode: a CashAddr with a Kelvin
  sign (invalid) and a Visa number with ideographic spaces (valid).
* The golden test: a new option set "and_exclude" catches Filter::and
  excluding the wrong tags (13 differences with that change), and
  scripts/golden.py ends both recordings with their counts, so a cut
  recording or missing JSON records fail.
* Area pull requests don't conflict in the two shared lists: regexes that
  need fancy-regex, or whose Charset Only tag is decided by hand, go into
  their area's block (AREAS_NOT_ON_THE_REGEX_CRATE in tests/examples.rs,
  AREAS_BY_HAND in tests/charset_only.rs), compared by name. The contract
  says so, and that validators add no crates without the maintainers'
  agreement.
* Stale text: CONTRIBUTING.md starts with the contract, the golden test's
  commands, "regex.json" where a regex can be in an area file, the Charset
  Only lengths, the 14 files of Ciphey that import lemmeknow, and the size
  of the big-file JSON (296,955,554 bytes).
@bee-san

bee-san commented Nov 5, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto main (0c78b02, with #303 and #304) without conflicts: the head is now e702a9c. The range-diff against the audited head is empty:

$ git range-diff b5b52b2..a0d89ef origin/main..e702a9c
1:  a0d89ef = 1:  e702a9c Follow-ups from the audits of #299 and #300

MERGE_OK of audit-pywhat-rs-followups (at a0d89ef) carries over. The local CI mirror passes on this head (now on the Rust engine: pytest 2,276 passed, 156 skipped).

@bee-san
bee-san force-pushed the pywhat-rs-followups branch from a0d89ef to e702a9c Compare November 5, 2026 00:41
@sonarqubecloud

sonarqubecloud Bot commented Nov 5, 2026

Copy link
Copy Markdown

@bee-san
bee-san merged commit eeca660 into main Nov 5, 2026
33 checks passed
@bee-san
bee-san deleted the pywhat-rs-followups branch November 5, 2026 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant