From e702a9ca2ba9d129453dd55f5c76b292b5d6bbcf Mon Sep 17 00:00:00 2001 From: Autumn Skerritt Date: Wed, 4 Nov 2026 23:44:06 +0000 Subject: [PATCH] Follow-ups from the audits of #299 and #300 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-blocking findings of audit-pywhat-rs-crate-b and audit-pywhat-rs-fp-b: * Top::of computes percentages as Python does: Decimal rounds total × percent to 28 significant digits (half to even), then the division by 100 and the ceiling are exact. It no longer overflows u128 for large totals (50.000000000000000000% of usize::MAX), and it no longer panics on a Top::Percent that parse() wouldn't return. parse() strips Python's whitespace, including U+001C to U+001F. Differential against Python's Top.of: 0 differences in 119,016 (percentage, total) pairs. * validators::facebook_access_token skips the first two characters, not two bytes, as Python's token[2:] does (they differed on non-ASCII input). * Examples that test both languages on Unicode: a CashAddr with a Kelvin sign (invalid) and a Visa number with ideographic spaces (valid). * The golden test: a new option set "and_exclude" catches Filter::and excluding the wrong tags (13 differences with that change), and scripts/golden.py ends both recordings with their counts, so a cut recording or missing JSON records fail. * Area pull requests don't conflict in the two shared lists: regexes that need fancy-regex, or whose Charset Only tag is decided by hand, go into their area's block (AREAS_NOT_ON_THE_REGEX_CRATE in tests/examples.rs, AREAS_BY_HAND in tests/charset_only.rs), compared by name. The contract says so, and that validators add no crates without the maintainers' agreement. * Stale text: CONTRIBUTING.md starts with the contract, the golden test's commands, "regex.json" where a regex can be in an area file, the Charset Only lengths, the 14 files of Ciphey that import lemmeknow, and the size of the big-file JSON (296,955,554 bytes). --- CONTRIBUTING.md | 6 +- docs/adding-regexes.md | 13 ++-- docs/rust.md | 15 ++-- pywhat/Data/regex.json | 6 +- pywhat/validators/__init__.py | 2 +- scripts/golden.py | 15 ++++ src/ranking.rs | 128 ++++++++++++++++++++++++++-------- src/validators/mod.rs | 15 +++- tests/charset_only.rs | 35 ++++++++-- tests/examples.rs | 32 ++++++++- tests/golden.rs | 45 ++++++++++-- tests/test_validators.py | 6 +- 12 files changed, 254 insertions(+), 64 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e1b87f8..1ed58da 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,11 +1,11 @@ -Please read our wiki here: -https://github.com/bee-san/pyWhat/wiki/Adding-your-own-Regex - New regexes go into the file of their area in `pywhat/Data/regexes/`, and must work the same in Python and in Rust: [docs/adding-regexes.md](docs/adding-regexes.md) is the checklist (test the regex in regex101's Rust flavour, examples, checksums, rarity, the `Charset Only` tag, false positives), and its tests run in CI. +The wiki (https://github.com/bee-san/pyWhat/wiki/Adding-your-own-Regex) predates the area files +and the Rust crate: where it differs, follow docs/adding-regexes.md. + To pick the rarity of a new regex, `python scripts/rarity_score.py 'REGEX'` estimates it from the regex. Every regex needs at least one valid example in its `"Examples"`, as the examples are run as tests. Please add invalid examples too, like values that look similar but must not match. diff --git a/docs/adding-regexes.md b/docs/adding-regexes.md index 604b876..16cac5e 100644 --- a/docs/adding-regexes.md +++ b/docs/adding-regexes.md @@ -69,8 +69,10 @@ every new regex has to pass. The tests check most of it. different characters in the two engines (docs/rust.md, "Differences from Python"). - **No look-arounds or back-references** unless there is no other way. The `regex` crate doesn't have them; such a regex runs on fancy-regex, a backtracking engine that is slower - and has a backtracking limit. If one is needed, add the regex to `NOT_ON_THE_REGEX_CRATE` - in `tests/examples.rs`, with the engine the test reports, and say why in the pull request. + and has a backtracking limit. If one is needed, add the regex to your area's block of + `AREAS_NOT_ON_THE_REGEX_CRATE` in `tests/examples.rs`, with the engine the test reports, and + say why in the pull request. If `tests/charset_only.rs` can't build its DFA, decide the tag + in your area's block of `AREAS_BY_HAND` there, with a reason. - No nested quantifiers that can backtrack a lot, such as `(a+)+`. - **Make it specific.** A regex that only checks which characters a text has and how long it is (`[0-9a-f]{40}`) matches every SHA-1 hash, and Ciphey and other tools run pyWhat on @@ -127,7 +129,9 @@ which no other area's pull request changes: if the regex already fixes part of the checksum, the share is higher). Validators use the Python standard library and pure Rust only: the crate builds for wasm32 and -the wheel has no dependencies for them. (Keccak-256 for EIP-55 isn't in `hashlib`, whose +the wheel has no dependencies for them. Don't add crates to `Cargo.toml` or Python packages for +them without the maintainers' agreement: two areas that each add one conflict in `Cargo.toml` and +`Cargo.lock`. (Keccak-256 for EIP-55 isn't in `hashlib`, whose `sha3_256` is NIST SHA-3; write it out or leave the checksum out.) Add an invalid example with a wrong checksum, and document the validator in its module's docstring rather than in the table above. @@ -148,7 +152,8 @@ regexes with a rarity of 0.1 or more by default. `tests/charset_only.rs` computes, from each regex's DFA, the largest share of random strings of nine alphabets (digits, hex, HEX, lower, upper, alphanumeric, Base64, Base64url, Base58) and of -every length up to 200 that match it, after its checksum. A regex that more than 1 in 1,000 of +every length up to 200 (or 50 past the regex's shortest match, if that is longer) that match it, +after its checksum. A regex that more than 1 in 1,000 of them match has to have the tag `Charset Only`, and only those may have it. When the test fails, it says which regex needs the tag added or removed. Ciphey ignores these regexes when it asks whether a whole text is a known format, and `pywhat --exclude "Charset Only"` leaves them out. diff --git a/docs/rust.md b/docs/rust.md index 6af1b59..58c06b2 100644 --- a/docs/rust.md +++ b/docs/rust.md @@ -152,7 +152,7 @@ it to `target/bench.txt`: 4.2 MB, 115,270 lines), with the CLI's defaults: | | Wall time | CPU time | JSON | |---|---|---|---| -| `pywhat --json --no-progress target/bench.txt` (Python) | 25.0 s | 19.2 s user | 296,955,557 bytes | +| `pywhat --json --no-progress target/bench.txt` (Python) | 25.0 s | 19.2 s user | 296,955,554 bytes | | `cargo run --release --example bench_file -- target/bench.txt --json out.json` (Rust) | 4.8–8.5 s | 3.2–3.4 s user | the same bytes (`cmp`) | `cargo run --release --example bench_file -- FILE` prints the time per regex. Most of the Rust @@ -201,12 +201,13 @@ whole text is, and filters the matches. With pywhat: descriptions don't matter for plaintext detection). Its web build adds `wasm` instead of `parallel`. 2. `use lemmeknow::{Data, Identifier}` becomes `use pywhat::{Identifier, RegexEntry}`, and - `use lemmeknow::Identifier` becomes `use pywhat::Identifier`, in the 15 files that use it: - `checkers/lemmeknow_checker.rs`, `checkers/checker_type.rs`, `checkers/athena.rs`, - `checkers/code_checker.rs`, `checkers/default_checker.rs`, `checkers/english.rs`, - `checkers/json_checker.rs`, `checkers/password.rs`, `checkers/regex_checker.rs`, - `checkers/wait_athena.rs`, `checkers/wordlist.rs`, `decoders/jwt_decoder.rs`, - `config/mod.rs`, `api_library_input_struct.rs` and `detection.rs` (`grep -rln lemmeknow src`). + `use lemmeknow::Identifier` becomes `use pywhat::Identifier`, in the 14 files that use it + (`grep -rl 'lemmeknow::' src`): `checkers/lemmeknow_checker.rs`, `checkers/checker_type.rs`, + `checkers/athena.rs`, `checkers/code_checker.rs`, `checkers/default_checker.rs`, + `checkers/english.rs`, `checkers/json_checker.rs`, `checkers/password.rs`, + `checkers/regex_checker.rs`, `checkers/wait_athena.rs`, `checkers/wordlist.rs`, + `decoders/jwt_decoder.rs`, `config/mod.rs` and `api_library_input_struct.rs`. `detection.rs` + has no `use lemmeknow` line; its change is step 6. `Checker::lemmeknow_config` keeps its name and holds a `pywhat::Identifier`, which is `Clone + Send + Sync`; the checkers that don't identify anything keep `lemmeknow_config: Identifier::default()`. diff --git a/pywhat/Data/regex.json b/pywhat/Data/regex.json index 481184f..1dffd78 100644 --- a/pywhat/Data/regex.json +++ b/pywhat/Data/regex.json @@ -326,7 +326,8 @@ ], "Invalid": [ "bitcoincash:qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr", - "qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr" + "qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr", + "BITCOINCASH:QPM2QSZNH\u212aS23Z7629MMS6S4CWEF74VCWVY22GDX6A" ] } }, @@ -3210,7 +3211,8 @@ "Valid": [ "4111111111111111", "4607 0000 0000 0009", - "4111\u00a01111\u00a01111\u00a01111" + "4111\u00a01111\u00a01111\u00a01111", + "4111\u30001111\u30001111\u30001111" ], "Invalid": [ "4111111111111112" diff --git a/pywhat/validators/__init__.py b/pywhat/validators/__init__.py index 99c429c..9cda9ac 100644 --- a/pywhat/validators/__init__.py +++ b/pywhat/validators/__init__.py @@ -4,7 +4,7 @@ A regex can only check which characters a match has and how long it is, so it also matches random text with the right characters: 1 in 10 random 16-digit numbers that start with 4 are Visa card numbers to a regex. The formats below have a checksum, which a match -must pass. A regex says which one in its "Validator" in regex.json, and +must pass. A regex says which one in its "Validator" (in regex.json or its area's file), and pywhat.processors.ValidatorProcessor filters out the matches whose checksum is wrong. * luhn: the Luhn algorithm (ISO/IEC 7812) of payment card numbers, 1 in 10 random numbers diff --git a/scripts/golden.py b/scripts/golden.py index 7ae8beb..a8d7f7d 100644 --- a/scripts/golden.py +++ b/scripts/golden.py @@ -153,6 +153,15 @@ def options(): "dist": Distribution(FINANCE | BTC), "boundaryless": Filter({"Tags": ["Identifiers"], "MinRarity": 0.2}), }, + # & excludes only what both sides exclude (_intersection): all 9 AWS regexes. + # Excluding what either side excludes would drop the 2 that are Credentials too. + "and_exclude": { + "dist": Distribution( + Filter({"Tags": ["AWS"], "ExcludeTags": ["Credentials"]}) + & Filter({"Tags": ["AWS"]}) + ), + "boundaryless": cli_boundaryless, + }, } @@ -281,6 +290,7 @@ def file_stage(out, files_dir: str, repo_dir: str): ([f"{files_dir}/binary.png", f"{files_dir}/utf16le-bom.txt"], False), ([repo_dir], False), ] + cases_written = jsons = 0 for inputs, include_filenames in cases: for name, identifier in identifiers.items(): identified = identifier.identify_inputs( @@ -304,7 +314,11 @@ def file_stage(out, files_dir: str, repo_dir: str): # Byte for byte, where the order of the files is not the file system's if not any(os.path.isdir(i) for i in inputs): record["json"] = json.dumps(identified) + jsons += 1 out.write(json.dumps(record) + "\n") + cases_written += 1 + # What was written, so that tests/golden.rs notices a cut or partial recording + out.write(json.dumps({"trailer": {"cases": cases_written, "json": jsons}}) + "\n") # Where the recordings go: target/, which cargo and git ignore @@ -365,6 +379,7 @@ def main(): full += 1 out.write(json.dumps(record) + "\n") count += 1 + out.write(json.dumps({"trailer": {"texts": count, "json": full}}) + "\n") print(f"golden: {count} texts in {OUT} ({os.path.getsize(OUT)} bytes)") diff --git a/src/ranking.rs b/src/ranking.rs index 6fcbcdc..44c323a 100644 --- a/src/ranking.rs +++ b/src/ranking.rs @@ -2,6 +2,7 @@ //! `pywhat/ranking.py`. use crate::identify::Match; +use crate::validators::is_python_space; use std::cmp::Ordering; /// What to sort the matches by (`pywhat.Keys`). @@ -77,14 +78,16 @@ pub enum Top { #[default] All, Number(usize), - /// A percentage, as written (digits with an optional decimal point). + /// A percentage, as written (digits with an optional decimal point), as [`Top::parse`] + /// returns it. [`Top::of`] shows all matches for a text that isn't such a number. Percent(String), } impl Top { /// `parse_top()`: "10", "5%", "1.5 %" or "all". `Err` holds Python's message. pub fn parse(text: &str) -> Result { - let text = text.trim(); + // str.strip() and \s: Python's whitespace, which includes U+001C to U+001F + let text = text.trim_matches(is_python_space); let error = || { format!( "Invalid top '{text}', use a number of matches such as 10, a percentage of them \ @@ -107,7 +110,7 @@ impl Top { let Some(number) = text.strip_suffix('%') else { return Err(error()); }; - let number = number.trim_end(); + let number = number.trim_end_matches(is_python_space); let valid = match number.split_once('.') { Some((whole, fraction)) => { whole.bytes().all(|b| b.is_ascii_digit()) @@ -131,39 +134,75 @@ impl Top { match self { Top::All => total, Top::Number(number) => (*number).min(total), - Top::Percent(percent) => { - // ceil(total * percent / 100), exact: with the first 30 significant digits - // of the percentage, the error is below 1 match for any usize total, and - // a nonzero rest still rounds up - let (numerator, scale, rest) = decimal(percent); - let denominator = 100 * 10u128.pow(scale); - let product = (total as u128) * numerator; - let mut shown = product.div_ceil(denominator); - if rest && product.is_multiple_of(denominator) { - shown += 1; - } - usize::try_from(shown).unwrap_or(usize::MAX).min(total) - } + Top::Percent(percent) => percent_of(total, percent), } } } -/// A percentage such as "1.25" as (125, 2, false): its digits up to the 30th significant -/// one, the number of them after the point, and whether nonzero digits were cut off. -/// Percentages are at most 100, so 3 digits before the point and 27 after fit in a u128 -/// whatever `total` is multiplied in. -fn decimal(text: &str) -> (u128, u32, bool) { - let (whole, fraction) = text.split_once('.').unwrap_or((text, "")); - let whole = whole.trim_start_matches('0'); - let kept = fraction.len().min(27); - let digits = format!("{whole}{}", &fraction[..kept]); - let numerator = if digits.is_empty() { - 0 +/// `min(total, math.ceil(total * percent / 100))` as Python computes it: `Decimal` rounds the +/// product to 28 significant digits (its default context, half to even), and the division by +/// 100 and the ceiling are exact. A `percent` that isn't digits with an optional point shows +/// all matches. +fn percent_of(total: usize, percent: &str) -> usize { + let (whole, fraction) = percent.split_once('.').unwrap_or((percent, "")); + let is_digits = |part: &str| part.bytes().all(|b| b.is_ascii_digit()); + if whole.is_empty() && fraction.is_empty() || !is_digits(whole) || !is_digits(fraction) { + return total; + } + // total × percent × 10^scale, exactly, as decimal digits + let product = times(&format!("{whole}{fraction}"), total as u64); + let product = product.trim_start_matches('0'); + if product.is_empty() { + return 0; + } + let (kept, dropped) = product.split_at(product.len().min(28)); + let mut coefficient: u128 = kept.parse().expect("at most 28 digits"); + let half = match dropped.as_bytes().first() { + None => Ordering::Less, + Some(&first) => match first.cmp(&b'5') { + Ordering::Equal if dropped[1..].bytes().any(|b| b != b'0') => Ordering::Greater, + order => order, + }, + }; + if half == Ordering::Greater || half == Ordering::Equal && coefficient % 2 == 1 { + coefficient += 1; + } + // The rounded product is coefficient × 10^(dropped − scale); divided by 100, rounded up + let exponent = dropped.len() as i64 - fraction.len() as i64 - 2; + let shown = if exponent >= 0 { + u32::try_from(exponent) + .ok() + .and_then(|e| 10u128.checked_pow(e)) + .and_then(|power| coefficient.checked_mul(power)) + .unwrap_or(u128::MAX) } else { - digits.parse::().unwrap_or(u128::MAX) + match u32::try_from(-exponent) + .ok() + .and_then(|e| 10u128.checked_pow(e)) + { + Some(power) => coefficient.div_ceil(power), + // 10^39 or more: above the coefficient, which is not 0 + None => 1, + } }; - let rest = fraction[kept..].bytes().any(|b| b != b'0'); - (numerator, kept as u32, rest) + usize::try_from(shown).unwrap_or(usize::MAX).min(total) +} + +/// The decimal digits of `digits × by`. +fn times(digits: &str, by: u64) -> String { + let mut out = Vec::with_capacity(digits.len() + 20); + let mut carry: u128 = 0; + for digit in digits.bytes().rev() { + let value = u128::from(digit - b'0') * u128::from(by) + carry; + out.push(b'0' + (value % 10) as u8); + carry = value / 10; + } + while carry > 0 { + out.push(b'0' + (carry % 10) as u8); + carry /= 10; + } + out.reverse(); + String::from_utf8(out).expect("digits") } /// Whether a decimal number (digits, an optional point and digits) is above 0 and at most @@ -224,7 +263,8 @@ mod tests { #[test] fn long_tops_like_python() { - // Python's Decimal has no limit on the digits + // Python parses a percentage of any length; Top.of() rounds total × percent to 28 + // significant digits (Decimal's default context) let tiny = format!(".{}1%", "0".repeat(38)); assert!(Top::parse(&tiny).is_ok()); assert_eq!(Top::parse(&tiny).unwrap().of(1000), 1); @@ -244,6 +284,32 @@ mod tests { ); } + #[test] + fn percentages_of_like_python() { + let of = |percent: &str, total| Top::parse(percent).unwrap().of(total); + // Exact where 28 digits are enough, also for the largest totals + assert_eq!(of("50.000000000000000000%", usize::MAX), usize::MAX / 2 + 1); + assert_eq!(of("1.5000000000000000000000000001%", 999), 15); + assert_eq!(of("33.333333333333333333334%", 3), 2); + // Where they aren't, Python's rounding: 3 × 33.33…334 = 100.00…002 has 31 digits, + // and rounds to 100 + assert_eq!(of("33.3333333333333333333333333334%", 3), 1); + assert_eq!(of("1.0000000000000000000000000001%", 100), 1); + assert_eq!(of("1.0000000000000000000000000001%", 1_000_000), 10_000); + // Ties round to even: 100 × 1.00…005 keeps 28 digits, 100.0…00 (even), and + // 100 × 1.00…015 becomes 100.0…02 (from the odd …01) + assert_eq!(of("1.0000000000000000000000000005%", 100), 1); + assert_eq!(of("1.0000000000000000000000000015%", 100), 2); + // str.strip() and \s are Python's whitespace, U+001C to U+001F included + assert_eq!(Top::parse("\u{1c}5%"), Ok(Top::Percent("5".into()))); + assert_eq!(Top::parse("5\u{1f}%"), Ok(Top::Percent("5".into()))); + assert_eq!(Top::parse("5\u{3000}%"), Ok(Top::Percent("5".into()))); + // A Percent that parse() doesn't return shows everything, without a panic + for odd in ["\u{e9}5", "", ".", "5..5", "x"] { + assert_eq!(Top::Percent(odd.into()).of(10), 10, "{odd:?}"); + } + } + #[test] fn top_of() { assert_eq!(Top::Percent("1.1".into()).of(1000), 11); diff --git a/src/validators/mod.rs b/src/validators/mod.rs index 7e478b6..7ed5148 100644 --- a/src/validators/mod.rs +++ b/src/validators/mod.rs @@ -78,7 +78,7 @@ pub fn luhn(number: &str) -> bool { } /// Python's `str.isspace()`. -fn is_python_space(c: char) -> bool { +pub(crate) fn is_python_space(c: char) -> bool { matches!( c, '\t' | '\n' @@ -181,7 +181,8 @@ pub fn tc_kimlik(number: &str) -> bool { /// letter, an uppercase letter and a digit, as 190 or more random letters and digits have. /// Decoder output is often all one case. pub fn facebook_access_token(token: &str) -> bool { - let rest = token.get(2..).unwrap_or(""); + // Python's token[2:]: after two characters, not two bytes + let rest = token.char_indices().nth(2).map_or("", |(i, _)| &token[i..]); rest.bytes().any(|b| b.is_ascii_lowercase()) && rest.bytes().any(|b| b.is_ascii_uppercase()) && rest.bytes().any(|b| b.is_ascii_digit()) @@ -191,6 +192,16 @@ pub fn facebook_access_token(token: &str) -> bool { mod tests { use super::*; + #[test] + fn facebook_access_token_skips_two_characters() { + // As Python's token[2:]: the first two characters, whatever their length in bytes + assert!(facebook_access_token("\u{20ac}AAb1C")); + assert!(facebook_access_token("\u{e9}\u{e9}Ab1")); + assert!(!facebook_access_token("Ab1")); + assert!(!facebook_access_token("\u{20ac}")); + assert!(!facebook_access_token("EAab1\u{20ac}")); + } + #[test] fn validators() { assert!( diff --git a/tests/charset_only.rs b/tests/charset_only.rs index 0ef0cd1..7cdceb3 100644 --- a/tests/charset_only.rs +++ b/tests/charset_only.rs @@ -8,8 +8,8 @@ //! the regex's DFA: for every length up to 200 (or 50 past the regex's shortest match), the //! share of the strings of that length that end in a match state. Regexes the DFA can't be built for are in `BY_HAND`, with a reason. //! -//! When this test fails for a new or changed regex, add or remove the tag in regex.json, as the -//! message says. See docs/adding-regexes.md. +//! When this test fails for a new or changed regex, add or remove the tag in the regex's file +//! (regex.json or its area's file), as the message says. See docs/adding-regexes.md. use regex_automata::dfa::{dense, Automaton, StartKind}; use regex_automata::util::primitives::StateID; @@ -66,6 +66,25 @@ const BY_HAND: &[(&str, bool, &str)] = &[ ), ]; +/// The same for the regexes of the area files, each area in its own block, so that pull +/// requests for different areas don't change the same lines. +const AREAS_BY_HAND: &[(&str, bool, &str)] = &[ + // --- crypto --- + // --- end crypto --- + // --- cloud --- + // --- end cloud --- + // --- ctf --- + // --- end ctf --- + // --- network --- + // --- end network --- + // --- ids --- + // --- end ids --- + // --- files --- + // --- end files --- + // --- social --- + // --- end social --- +]; + /// The chance that random strings over `alphabet` of each length up to `max_length` match /// all of the DFA. fn chances(dfa: &dense::DFA>, alphabet: &[u8], max_length: usize) -> Vec { @@ -224,7 +243,11 @@ fn charset_only_tags_are_the_regexes_random_strings_match() { for entry in pywhat::regexes() { let has_tag = entry.tags.iter().any(|tag| tag == TAG); tagged += usize::from(has_tag); - let (charset_only, why) = match BY_HAND.iter().find(|(name, _, _)| *name == entry.name) { + let (charset_only, why) = match BY_HAND + .iter() + .chain(AREAS_BY_HAND) + .find(|(name, _, _)| *name == entry.name) + { Some((_, charset_only, reason)) => { (*charset_only, format!("decided by hand: {reason}")) } @@ -235,7 +258,7 @@ fn charset_only_tags_are_the_regexes_random_strings_match() { ), Err(error) => { wrong.push(format!( - "{}: no DFA ({error}); decide by hand in BY_HAND", + "{}: no DFA ({error}); decide by hand in BY_HAND or your area's block of AREAS_BY_HAND", entry.name )); continue; @@ -263,10 +286,10 @@ fn charset_only_tags_are_the_regexes_random_strings_match() { #[test] fn every_by_hand_decision_is_for_a_regex_that_exists() { - for (name, _, _) in BY_HAND { + for (name, _, _) in BY_HAND.iter().chain(AREAS_BY_HAND) { assert!( pywhat::regexes().iter().any(|entry| entry.name == *name), - "{name} is not in regex.json" + "{name} is not a regex of pywhat/Data" ); } } diff --git a/tests/examples.rs b/tests/examples.rs index a0483cb..579a5a4 100644 --- a/tests/examples.rs +++ b/tests/examples.rs @@ -135,6 +135,26 @@ const NOT_ON_THE_REGEX_CRATE: &[(&str, EngineKind, EngineKind)] = &[ ("YouTube Video ID", EngineKind::Guarded, EngineKind::Guarded), ]; +/// The same for the regexes of the area files (`pywhat/Data/regexes/`), each area in its own +/// block, so that pull requests for different areas don't change the same lines. The order +/// within and between the lists doesn't matter. +const AREAS_NOT_ON_THE_REGEX_CRATE: &[(&str, EngineKind, EngineKind)] = &[ + // --- crypto --- + // --- end crypto --- + // --- cloud --- + // --- end cloud --- + // --- ctf --- + // --- end ctf --- + // --- network --- + // --- end network --- + // --- ids --- + // --- end ids --- + // --- files --- + // --- end files --- + // --- social --- + // --- end social --- +]; + #[test] fn only_the_allowlisted_regexes_need_more_than_the_regex_crate() { let mut actual = Vec::new(); @@ -145,9 +165,17 @@ fn only_the_allowlisted_regexes_need_more_than_the_regex_crate() { actual.push((entry.name.as_str(), anchored, boundaryless)); } } - let expected: Vec<(&str, EngineKind, EngineKind)> = NOT_ON_THE_REGEX_CRATE.to_vec(); + let mut expected: Vec<(&str, EngineKind, EngineKind)> = NOT_ON_THE_REGEX_CRATE + .iter() + .chain(AREAS_NOT_ON_THE_REGEX_CRATE) + .copied() + .collect(); + // Compared by name, so that an entry's place doesn't depend on the regex's rarity + actual.sort_by_key(|(name, _, _)| *name); + expected.sort_by_key(|(name, _, _)| *name); assert_eq!( actual, expected, - "regexes that don't run on the regex crate (update NOT_ON_THE_REGEX_CRATE on purpose)" + "regexes that don't run on the regex crate (update NOT_ON_THE_REGEX_CRATE, or your \ + area's block of AREAS_NOT_ON_THE_REGEX_CRATE, on purpose)" ); } diff --git a/tests/golden.rs b/tests/golden.rs index 3bcac40..35080b7 100644 --- a/tests/golden.rs +++ b/tests/golden.rs @@ -1,9 +1,10 @@ //! The golden parity test: what the Rust crate finds must be what the Python package finds. //! -//! python scripts/golden.py --out target/golden.jsonl -//! PYWHAT_GOLDEN=target/golden.jsonl cargo test --release --test golden -- --nocapture +//! python scripts/golden.py # writes target/golden.jsonl and target/golden-files.jsonl +//! PYWHAT_GOLDEN=target/golden.jsonl PYWHAT_GOLDEN_FILES=target/golden-files.jsonl \ +//! cargo test --release --test golden -- --nocapture //! -//! Without PYWHAT_GOLDEN the test does nothing. See scripts/golden.py for the texts and the +//! Without PYWHAT_GOLDEN (PYWHAT_GOLDEN_FILES) the test (the file test) does nothing. See scripts/golden.py for the texts and the //! option sets. use pywhat::processors::OpenStreetMap; @@ -73,6 +74,16 @@ fn identifiers() -> BTreeMap<&'static str, Identifier> { .with_filter(finance().or(&btc())) .with_boundaryless_filter(with(Some(&["Identifiers"]), &[], 0.2)), ), + ( + "and_exclude", + Identifier::default() + .with_filter(with(Some(&["AWS"]), &["Credentials"], 0.1).and(&with( + Some(&["AWS"]), + &[], + 0.1, + ))) + .with_boundaryless_filter(cli.clone()), + ), ]) } @@ -141,8 +152,15 @@ fn rust_finds_what_python_finds() { let mut differences: BTreeMap = BTreeMap::new(); let mut unicode_differences = 0; let mut shown = 0; + let mut jsons = 0; + let mut trailer = None; for line in std::io::BufReader::new(file).lines() { let record: Value = serde_json::from_str(&line.expect("line")).expect("JSON line"); + if let Some(counts) = record.get("trailer") { + trailer = Some(counts.clone()); + continue; + } + assert!(trailer.is_none(), "records after the trailer in {path}"); let text = record["text"].as_str().expect("text"); let kind = record["kind"].as_str().expect("kind"); texts += 1; @@ -187,6 +205,7 @@ fn rust_finds_what_python_finds() { } if let Some(json) = record.get("json").and_then(Value::as_str) { compared += 1; + jsons += 1; let mut actual = String::new(); pywhat::identify::write_identify_json( &mut actual, @@ -224,6 +243,12 @@ fn rust_finds_what_python_finds() { ); assert!(texts > 0, "no texts in {path}"); assert!(compared > texts, "{compared} comparisons for {texts} texts"); + // scripts/golden.py ends the recording with its counts: a cut recording, or one whose + // JSON was left out, fails here + let trailer = trailer.unwrap_or_else(|| panic!("{path} has no trailer: cut off?")); + assert_eq!(trailer["texts"], texts, "texts recorded and read"); + assert_eq!(trailer["json"], jsons, "JSON records recorded and read"); + assert!(jsons > 0, "no JSON records in {path}"); assert_eq!(total, 0, "Rust and Python differ: {differences:?}"); } @@ -266,7 +291,7 @@ fn report(expected: &Value, actual: &Value) { } #[cfg(feature = "files")] -/// File and directory inputs, recorded by `scripts/golden.py --files-out`. Directory +/// File and directory inputs, recorded by `scripts/golden.py` in target/golden-files.jsonl. Directory /// locations are compared without regard to their order (Python's glob uses the order of /// the file system, Rust sorts by name). #[test] @@ -278,9 +303,16 @@ fn rust_reads_files_like_python() { let file = std::fs::File::open(&path).expect("PYWHAT_GOLDEN_FILES file"); let identifiers = identifiers(); let mut cases = 0; + let mut jsons = 0; + let mut trailer = None; let mut differences = Vec::new(); for line in std::io::BufReader::new(file).lines() { let record: Value = serde_json::from_str(&line.expect("line")).expect("JSON line"); + if let Some(counts) = record.get("trailer") { + trailer = Some(counts.clone()); + continue; + } + assert!(trailer.is_none(), "records after the trailer in {path}"); let inputs: Vec = record["inputs"] .as_array() .expect("inputs") @@ -343,6 +375,7 @@ fn rust_reads_files_like_python() { } } if let Some(json) = record.get("json").and_then(Value::as_str) { + jsons += 1; let actual = identified.to_json(); if actual != json { let at = actual @@ -366,5 +399,9 @@ fn rust_reads_files_like_python() { differences.len() ); assert!(cases > 0); + let trailer = trailer.unwrap_or_else(|| panic!("{path} has no trailer: cut off?")); + assert_eq!(trailer["cases"], cases, "cases recorded and read"); + assert_eq!(trailer["json"], jsons, "JSON records recorded and read"); + assert!(jsons > 0, "no JSON records in {path}"); assert!(differences.is_empty(), "{} differences", differences.len()); } diff --git a/tests/test_validators.py b/tests/test_validators.py index bfa0dd0..5101b3e 100644 --- a/tests/test_validators.py +++ b/tests/test_validators.py @@ -67,8 +67,10 @@ def names(text, identifier=ALL): ), ( facebook_access_token, - ["EAARE0ZATe"], - ["EAHCHERBOTIC", "EA0123ABCDEF", "EAabcdef0123", "EA"], + # The first two characters are skipped, whatever their length in bytes (the + # same in Rust) + ["EAARE0ZATe", "\u20acAAb1C", "\u00e9\u00e9Ab1"], + ["EAHCHERBOTIC", "EA0123ABCDEF", "EAabcdef0123", "EA", "Ab1", "\u20ac"], ), ], )