Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
Please read our wiki here:
https://github.com/bee-san/pyWhat/wiki/Adding-your-own-Regex

New regexes go into the file of their area in `pywhat/Data/regexes/`, and must work the same in
Python and in Rust: [docs/adding-regexes.md](docs/adding-regexes.md) is the checklist (test the
regex in regex101's Rust flavour, examples, checksums, rarity, the `Charset Only` tag, false
positives), and its tests run in CI.

The wiki (https://github.com/bee-san/pyWhat/wiki/Adding-your-own-Regex) predates the area files
and the Rust crate: where it differs, follow docs/adding-regexes.md.

To pick the rarity of a new regex, `python scripts/rarity_score.py 'REGEX'` estimates it from the regex.

Every regex needs at least one valid example in its `"Examples"`, as the examples are run as tests. Please add invalid examples too, like values that look similar but must not match.
Expand Down
13 changes: 9 additions & 4 deletions docs/adding-regexes.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,10 @@ every new regex has to pass. The tests check most of it.
different characters in the two engines (docs/rust.md, "Differences from Python").
- **No look-arounds or back-references** unless there is no other way. The `regex` crate
doesn't have them; such a regex runs on fancy-regex, a backtracking engine that is slower
and has a backtracking limit. If one is needed, add the regex to `NOT_ON_THE_REGEX_CRATE`
in `tests/examples.rs`, with the engine the test reports, and say why in the pull request.
and has a backtracking limit. If one is needed, add the regex to your area's block of
`AREAS_NOT_ON_THE_REGEX_CRATE` in `tests/examples.rs`, with the engine the test reports, and
say why in the pull request. If `tests/charset_only.rs` can't build its DFA, decide the tag
in your area's block of `AREAS_BY_HAND` there, with a reason.
- No nested quantifiers that can backtrack a lot, such as `(a+)+`.
- **Make it specific.** A regex that only checks which characters a text has and how long it
is (`[0-9a-f]{40}`) matches every SHA-1 hash, and Ciphey and other tools run pyWhat on
Expand Down Expand Up @@ -127,7 +129,9 @@ which no other area's pull request changes:
if the regex already fixes part of the checksum, the share is higher).

Validators use the Python standard library and pure Rust only: the crate builds for wasm32 and
the wheel has no dependencies for them. (Keccak-256 for EIP-55 isn't in `hashlib`, whose
the wheel has no dependencies for them. Don't add crates to `Cargo.toml` or Python packages for
them without the maintainers' agreement: two areas that each add one conflict in `Cargo.toml` and
`Cargo.lock`. (Keccak-256 for EIP-55 isn't in `hashlib`, whose
`sha3_256` is NIST SHA-3; write it out or leave the checksum out.) Add an invalid example
with a wrong checksum, and document the validator in its module's docstring rather than in the
table above.
Expand All @@ -148,7 +152,8 @@ regexes with a rarity of 0.1 or more by default.

`tests/charset_only.rs` computes, from each regex's DFA, the largest share of random strings of
nine alphabets (digits, hex, HEX, lower, upper, alphanumeric, Base64, Base64url, Base58) and of
every length up to 200 that match it, after its checksum. A regex that more than 1 in 1,000 of
every length up to 200 (or 50 past the regex's shortest match, if that is longer) that match it,
after its checksum. A regex that more than 1 in 1,000 of
them match has to have the tag `Charset Only`, and only those may have it. When the test fails,
it says which regex needs the tag added or removed. Ciphey ignores these regexes when it asks
whether a whole text is a known format, and `pywhat --exclude "Charset Only"` leaves them out.
Expand Down
15 changes: 8 additions & 7 deletions docs/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ it to `target/bench.txt`: 4.2 MB, 115,270 lines), with the CLI's defaults:

| | Wall time | CPU time | JSON |
|---|---|---|---|
| `pywhat --json --no-progress target/bench.txt` (Python) | 25.0 s | 19.2 s user | 296,955,557 bytes |
| `pywhat --json --no-progress target/bench.txt` (Python) | 25.0 s | 19.2 s user | 296,955,554 bytes |
| `cargo run --release --example bench_file -- target/bench.txt --json out.json` (Rust) | 4.8–8.5 s | 3.2–3.4 s user | the same bytes (`cmp`) |

`cargo run --release --example bench_file -- FILE` prints the time per regex. Most of the Rust
Expand Down Expand Up @@ -201,12 +201,13 @@ whole text is, and filters the matches. With pywhat:
descriptions don't matter for plaintext detection). Its web build adds `wasm` instead of
`parallel`.
2. `use lemmeknow::{Data, Identifier}` becomes `use pywhat::{Identifier, RegexEntry}`, and
`use lemmeknow::Identifier` becomes `use pywhat::Identifier`, in the 15 files that use it:
`checkers/lemmeknow_checker.rs`, `checkers/checker_type.rs`, `checkers/athena.rs`,
`checkers/code_checker.rs`, `checkers/default_checker.rs`, `checkers/english.rs`,
`checkers/json_checker.rs`, `checkers/password.rs`, `checkers/regex_checker.rs`,
`checkers/wait_athena.rs`, `checkers/wordlist.rs`, `decoders/jwt_decoder.rs`,
`config/mod.rs`, `api_library_input_struct.rs` and `detection.rs` (`grep -rln lemmeknow src`).
`use lemmeknow::Identifier` becomes `use pywhat::Identifier`, in the 14 files that use it
(`grep -rl 'lemmeknow::' src`): `checkers/lemmeknow_checker.rs`, `checkers/checker_type.rs`,
`checkers/athena.rs`, `checkers/code_checker.rs`, `checkers/default_checker.rs`,
`checkers/english.rs`, `checkers/json_checker.rs`, `checkers/password.rs`,
`checkers/regex_checker.rs`, `checkers/wait_athena.rs`, `checkers/wordlist.rs`,
`decoders/jwt_decoder.rs`, `config/mod.rs` and `api_library_input_struct.rs`. `detection.rs`
has no `use lemmeknow` line; its change is step 6.
`Checker::lemmeknow_config` keeps its name and holds a `pywhat::Identifier`, which is
`Clone + Send + Sync`; the checkers that don't identify anything keep
`lemmeknow_config: Identifier::default()`.
Expand Down
6 changes: 4 additions & 2 deletions pywhat/Data/regex.json
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,8 @@
],
"Invalid": [
"bitcoincash:qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr",
"qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr"
"qzlg6uvceehgzgtz6phmvy8gtdqyt6vf359at4n3lr",
"BITCOINCASH:QPM2QSZNH\u212aS23Z7629MMS6S4CWEF74VCWVY22GDX6A"
]
}
},
Expand Down Expand Up @@ -3210,7 +3211,8 @@
"Valid": [
"4111111111111111",
"4607 0000 0000 0009",
"4111\u00a01111\u00a01111\u00a01111"
"4111\u00a01111\u00a01111\u00a01111",
"4111\u30001111\u30001111\u30001111"
],
"Invalid": [
"4111111111111112"
Expand Down
2 changes: 1 addition & 1 deletion pywhat/validators/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
A regex can only check which characters a match has and how long it is, so it also
matches random text with the right characters: 1 in 10 random 16-digit numbers that start
with 4 are Visa card numbers to a regex. The formats below have a checksum, which a match
must pass. A regex says which one in its "Validator" in regex.json, and
must pass. A regex says which one in its "Validator" (in regex.json or its area's file), and
pywhat.processors.ValidatorProcessor filters out the matches whose checksum is wrong.

* luhn: the Luhn algorithm (ISO/IEC 7812) of payment card numbers, 1 in 10 random numbers
Expand Down
15 changes: 15 additions & 0 deletions scripts/golden.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,15 @@ def options():
"dist": Distribution(FINANCE | BTC),
"boundaryless": Filter({"Tags": ["Identifiers"], "MinRarity": 0.2}),
},
# & excludes only what both sides exclude (_intersection): all 9 AWS regexes.
# Excluding what either side excludes would drop the 2 that are Credentials too.
"and_exclude": {
"dist": Distribution(
Filter({"Tags": ["AWS"], "ExcludeTags": ["Credentials"]})
& Filter({"Tags": ["AWS"]})
),
"boundaryless": cli_boundaryless,
},
}


Expand Down Expand Up @@ -281,6 +290,7 @@ def file_stage(out, files_dir: str, repo_dir: str):
([f"{files_dir}/binary.png", f"{files_dir}/utf16le-bom.txt"], False),
([repo_dir], False),
]
cases_written = jsons = 0
for inputs, include_filenames in cases:
for name, identifier in identifiers.items():
identified = identifier.identify_inputs(
Expand All @@ -304,7 +314,11 @@ def file_stage(out, files_dir: str, repo_dir: str):
# Byte for byte, where the order of the files is not the file system's
if not any(os.path.isdir(i) for i in inputs):
record["json"] = json.dumps(identified)
jsons += 1
out.write(json.dumps(record) + "\n")
cases_written += 1
# What was written, so that tests/golden.rs notices a cut or partial recording
out.write(json.dumps({"trailer": {"cases": cases_written, "json": jsons}}) + "\n")


# Where the recordings go: target/, which cargo and git ignore
Expand Down Expand Up @@ -365,6 +379,7 @@ def main():
full += 1
out.write(json.dumps(record) + "\n")
count += 1
out.write(json.dumps({"trailer": {"texts": count, "json": full}}) + "\n")
print(f"golden: {count} texts in {OUT} ({os.path.getsize(OUT)} bytes)")


Expand Down
128 changes: 97 additions & 31 deletions src/ranking.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
//! `pywhat/ranking.py`.

use crate::identify::Match;
use crate::validators::is_python_space;
use std::cmp::Ordering;

/// What to sort the matches by (`pywhat.Keys`).
Expand Down Expand Up @@ -77,14 +78,16 @@ pub enum Top {
#[default]
All,
Number(usize),
/// A percentage, as written (digits with an optional decimal point).
/// A percentage, as written (digits with an optional decimal point), as [`Top::parse`]
/// returns it. [`Top::of`] shows all matches for a text that isn't such a number.
Percent(String),
}

impl Top {
/// `parse_top()`: "10", "5%", "1.5 %" or "all". `Err` holds Python's message.
pub fn parse(text: &str) -> Result<Top, String> {
let text = text.trim();
// str.strip() and \s: Python's whitespace, which includes U+001C to U+001F
let text = text.trim_matches(is_python_space);
let error = || {
format!(
"Invalid top '{text}', use a number of matches such as 10, a percentage of them \
Expand All @@ -107,7 +110,7 @@ impl Top {
let Some(number) = text.strip_suffix('%') else {
return Err(error());
};
let number = number.trim_end();
let number = number.trim_end_matches(is_python_space);
let valid = match number.split_once('.') {
Some((whole, fraction)) => {
whole.bytes().all(|b| b.is_ascii_digit())
Expand All @@ -131,39 +134,75 @@ impl Top {
match self {
Top::All => total,
Top::Number(number) => (*number).min(total),
Top::Percent(percent) => {
// ceil(total * percent / 100), exact: with the first 30 significant digits
// of the percentage, the error is below 1 match for any usize total, and
// a nonzero rest still rounds up
let (numerator, scale, rest) = decimal(percent);
let denominator = 100 * 10u128.pow(scale);
let product = (total as u128) * numerator;
let mut shown = product.div_ceil(denominator);
if rest && product.is_multiple_of(denominator) {
shown += 1;
}
usize::try_from(shown).unwrap_or(usize::MAX).min(total)
}
Top::Percent(percent) => percent_of(total, percent),
}
}
}

/// A percentage such as "1.25" as (125, 2, false): its digits up to the 30th significant
/// one, the number of them after the point, and whether nonzero digits were cut off.
/// Percentages are at most 100, so 3 digits before the point and 27 after fit in a u128
/// whatever `total` is multiplied in.
fn decimal(text: &str) -> (u128, u32, bool) {
let (whole, fraction) = text.split_once('.').unwrap_or((text, ""));
let whole = whole.trim_start_matches('0');
let kept = fraction.len().min(27);
let digits = format!("{whole}{}", &fraction[..kept]);
let numerator = if digits.is_empty() {
0
/// `min(total, math.ceil(total * percent / 100))` as Python computes it: `Decimal` rounds the
/// product to 28 significant digits (its default context, half to even), and the division by
/// 100 and the ceiling are exact. A `percent` that isn't digits with an optional point shows
/// all matches.
fn percent_of(total: usize, percent: &str) -> usize {
let (whole, fraction) = percent.split_once('.').unwrap_or((percent, ""));
let is_digits = |part: &str| part.bytes().all(|b| b.is_ascii_digit());
if whole.is_empty() && fraction.is_empty() || !is_digits(whole) || !is_digits(fraction) {
return total;
}
// total × percent × 10^scale, exactly, as decimal digits
let product = times(&format!("{whole}{fraction}"), total as u64);
let product = product.trim_start_matches('0');
if product.is_empty() {
return 0;
}
let (kept, dropped) = product.split_at(product.len().min(28));
let mut coefficient: u128 = kept.parse().expect("at most 28 digits");
let half = match dropped.as_bytes().first() {
None => Ordering::Less,
Some(&first) => match first.cmp(&b'5') {
Ordering::Equal if dropped[1..].bytes().any(|b| b != b'0') => Ordering::Greater,
order => order,
},
};
if half == Ordering::Greater || half == Ordering::Equal && coefficient % 2 == 1 {
coefficient += 1;
}
// The rounded product is coefficient × 10^(dropped − scale); divided by 100, rounded up
let exponent = dropped.len() as i64 - fraction.len() as i64 - 2;
let shown = if exponent >= 0 {
u32::try_from(exponent)
.ok()
.and_then(|e| 10u128.checked_pow(e))
.and_then(|power| coefficient.checked_mul(power))
.unwrap_or(u128::MAX)
} else {
digits.parse::<u128>().unwrap_or(u128::MAX)
match u32::try_from(-exponent)
.ok()
.and_then(|e| 10u128.checked_pow(e))
{
Some(power) => coefficient.div_ceil(power),
// 10^39 or more: above the coefficient, which is not 0
None => 1,
}
};
let rest = fraction[kept..].bytes().any(|b| b != b'0');
(numerator, kept as u32, rest)
usize::try_from(shown).unwrap_or(usize::MAX).min(total)
}

/// The decimal digits of `digits × by`.
fn times(digits: &str, by: u64) -> String {
let mut out = Vec::with_capacity(digits.len() + 20);
let mut carry: u128 = 0;
for digit in digits.bytes().rev() {
let value = u128::from(digit - b'0') * u128::from(by) + carry;
out.push(b'0' + (value % 10) as u8);
carry = value / 10;
}
while carry > 0 {
out.push(b'0' + (carry % 10) as u8);
carry /= 10;
}
out.reverse();
String::from_utf8(out).expect("digits")
}

/// Whether a decimal number (digits, an optional point and digits) is above 0 and at most
Expand Down Expand Up @@ -224,7 +263,8 @@ mod tests {

#[test]
fn long_tops_like_python() {
// Python's Decimal has no limit on the digits
// Python parses a percentage of any length; Top.of() rounds total × percent to 28
// significant digits (Decimal's default context)
let tiny = format!(".{}1%", "0".repeat(38));
assert!(Top::parse(&tiny).is_ok());
assert_eq!(Top::parse(&tiny).unwrap().of(1000), 1);
Expand All @@ -244,6 +284,32 @@ mod tests {
);
}

#[test]
fn percentages_of_like_python() {
let of = |percent: &str, total| Top::parse(percent).unwrap().of(total);
// Exact where 28 digits are enough, also for the largest totals
assert_eq!(of("50.000000000000000000%", usize::MAX), usize::MAX / 2 + 1);
assert_eq!(of("1.5000000000000000000000000001%", 999), 15);
assert_eq!(of("33.333333333333333333334%", 3), 2);
// Where they aren't, Python's rounding: 3 × 33.33…334 = 100.00…002 has 31 digits,
// and rounds to 100
assert_eq!(of("33.3333333333333333333333333334%", 3), 1);
assert_eq!(of("1.0000000000000000000000000001%", 100), 1);
assert_eq!(of("1.0000000000000000000000000001%", 1_000_000), 10_000);
// Ties round to even: 100 × 1.00…005 keeps 28 digits, 100.0…00 (even), and
// 100 × 1.00…015 becomes 100.0…02 (from the odd …01)
assert_eq!(of("1.0000000000000000000000000005%", 100), 1);
assert_eq!(of("1.0000000000000000000000000015%", 100), 2);
// str.strip() and \s are Python's whitespace, U+001C to U+001F included
assert_eq!(Top::parse("\u{1c}5%"), Ok(Top::Percent("5".into())));
assert_eq!(Top::parse("5\u{1f}%"), Ok(Top::Percent("5".into())));
assert_eq!(Top::parse("5\u{3000}%"), Ok(Top::Percent("5".into())));
// A Percent that parse() doesn't return shows everything, without a panic
for odd in ["\u{e9}5", "", ".", "5..5", "x"] {
assert_eq!(Top::Percent(odd.into()).of(10), 10, "{odd:?}");
}
}

#[test]
fn top_of() {
assert_eq!(Top::Percent("1.1".into()).of(1000), 11);
Expand Down
15 changes: 13 additions & 2 deletions src/validators/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ pub fn luhn(number: &str) -> bool {
}

/// Python's `str.isspace()`.
fn is_python_space(c: char) -> bool {
pub(crate) fn is_python_space(c: char) -> bool {
matches!(
c,
'\t' | '\n'
Expand Down Expand Up @@ -181,7 +181,8 @@ pub fn tc_kimlik(number: &str) -> bool {
/// letter, an uppercase letter and a digit, as 190 or more random letters and digits have.
/// Decoder output is often all one case.
pub fn facebook_access_token(token: &str) -> bool {
let rest = token.get(2..).unwrap_or("");
// Python's token[2:]: after two characters, not two bytes
let rest = token.char_indices().nth(2).map_or("", |(i, _)| &token[i..]);
rest.bytes().any(|b| b.is_ascii_lowercase())
&& rest.bytes().any(|b| b.is_ascii_uppercase())
&& rest.bytes().any(|b| b.is_ascii_digit())
Expand All @@ -191,6 +192,16 @@ pub fn facebook_access_token(token: &str) -> bool {
mod tests {
use super::*;

#[test]
fn facebook_access_token_skips_two_characters() {
// As Python's token[2:]: the first two characters, whatever their length in bytes
assert!(facebook_access_token("\u{20ac}AAb1C"));
assert!(facebook_access_token("\u{e9}\u{e9}Ab1"));
assert!(!facebook_access_token("Ab1"));
assert!(!facebook_access_token("\u{20ac}"));
assert!(!facebook_access_token("EAab1\u{20ac}"));
}

#[test]
fn validators() {
assert!(
Expand Down
Loading
Loading