Repository navigation
Masking follow-ups for 0.15.6: XML bodies, wallet tokens, %.f, pci CVV, speed (#160054) - #70
Merged
Merged
Conversation
KNET's KPay gateway takes XML, and Connect logs every request body through redact_body into http.request.body.content. On 0.15.5 neither redact_body nor mask_sensitive_data read an element, so <password>…</password> shipped in clear. redact_body now reads an element as a key: its local name, whatever its namespace prefix and attributes, case-insensitively. A card, CVV or SAD name masks its text by that type; a credential name -- one redact_url reads as a credential param (_is_credential_key), or a secret body key -- as a credential, the label when it holds a card-number run, as a JSON or form value is. The text runs to its end tag, children and all, and is masked as it decodes (a CDATA section's content, entities resolved); entity-encoded text in any other leaf goes through the body rules as it decodes and is written back escaped. A start tag no end tag closes is no element: a route in a message (/v1/cards/<str:token>/) has that shape. The credential text rules gain the element form (rule 2), in every pack, for a string that never went through redact_body, with the keywords the other credential rules use. It runs before them, so a token they write inside an element is never hashed again with it, and an Authorization element carries the header's token (scheme and credential). A form value in redact_body and a credential value in the text rules now end at an end tag: in <note>password=abc</note> the value took `</note` with it, and the next pass hashed the token again. test_masking_secrets pinned that for `<a href="/cb?password=…">x</a>`, whose value ate `</a`; the tag now survives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pack
An Apple Pay or Google Pay token carries the device card number and the
payment cryptogram, encrypted. Under a credential key -- MPGS's
paymentToken, Google Pay's tokenizationData.token, TAP's token_data -- it
was hashed; under any other key, or in text, it shipped whole, and KPay
puts Apple's paymentData in <udf9>.
It is found by its shape, never by a key name alone, so an unrelated
paymentData ({"amount": …}) reads as before:
- Apple Pay: an object whose version is EC_v1 or RSA_v1, with the
encrypted data. The whole object is the label: its signature and header
are useless without data.
- Google Pay: an object whose protocolVersion is ECv1, ECv2 or
ECv2SigningOnly, with a signedMessage.
The key walk gives the label to a token under any key, as a mapping or as
JSON text, and to JSON text holding one deeper under a key that would hash
it whole or show it. Rule 2, in every pack, does to a token written in text
as JSON or a Python repr, or as JSON in a JSON string -- quoted where it
stood as a JSON value, so JSON stays JSON -- before any credential rule can
hash it. redact_body runs it first, and so on an XML element's text such as
KPay's udf9, entities decoded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A record whose number masking turned into text is rendered from the masked arguments, that number's conversion read as %s, and the filter counts the template's conversions to find it. _CONVERSION did not read `%.f`, a precision with no digits (CPython's %.0f), so the count slipped, the %d meant for the masked number met text, and the record became [MASKING-FAILED: TypeError] instead of rendering (#160054; 0.15.4 dropped that line). The whole conversion spec now follows CPython's printf grammar: `%%` alone is a literal percent (a % after a flag, width, key or precision is one CPython refuses); a mapping key runs to the parenthesis that balances its first, as CPython counts them; then flags #0- +, a width * or digits, a precision `.` followed by * or zero or more digits, one of h, l or L, and one of d i o u x X e E f F g G c r s a. hypothesis is not a test dependency, so tests/test_masking_printf.py is a generated table: every flag, width, precision, length modifier and conversion, positional and keyed, rendered by the filter before a number that masks into text, and compared with CPython's own rendering of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… card With the pci pack, the bare-digit CVV rule masked every 3-4 digit run in a line that mentioned a card anywhere -- a "card", "pan" or "cvv" word, or a truncation: `card list returned 200 in 350 ms` became `card list returned [CVV-MASKED] in [CVV-MASKED] ms`, and `declined with code 051` lost its code (#160054). A CVV is worth nothing without its card, and is written beside it. Rule 20 now reads only the fields written after a card number or its truncation -- up to three, joined by whitespace, `|`, `,`, `;` or `:` -- and masks each that is three or four digits alone: `4111111111111111 123`, `411111******1111|12/27|123`, a CSV row. Which of two such fields is the CVV (PAN MMYY CVV, PAN CVV MMYY) no rule can tell, so neither ships. A date (12/27), a field after an expiry word (exp 1227), a month or year of one or two digits, and a field that runs on into a date, a time or an amount (2026-09-30, 10:30, 100.000) stay. Rule 21 reads a CVV after its word with a sign or a word or two between (`the cvv is 123`), which the keyed rules do not read; it is tried near the CVV words, as they are. The card-context precondition, the only one any rule had, is gone, and Rule.precondition with it. Tests that pinned the over-masking change: - test_masking_filter: `1123 4567 8912 34567891` kept its first two groups as [CVV-MASKED]; they follow no card and now read through. The strict xfail test_over_masked_because_of_cvv passes, so its rows moved into NOT_MASKED, as its comment asked. - test_masking_bare_pan: `<PAN> ref 123` masked the reference through "card context"; a CVV right after the PAN still masks, and the reference reads through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…el in text `https://9923960000004314@host/` shipped whole in free text, while redact_url masked the same URL (#160054). The userinfo text rule (rule 14) masked a password, or a token alone, and left a user alone to the other rules, since `ssh://git@host` names an account; a saved card's sixteen-digit gateway token sits there too. Before a host with a TLD the email rule took it, which with a keyset was a keyed hash of a value holding a card number. A user alone that is shaped like a card number or holds a card-number run (as it decodes) is now [SECRET-MASKED], in every pack, before the phone, email and card rules run. Any other user alone, and an email, reads as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntainer
`password=%(pw)s` rendered with a list or dict printed `password=['a',
'b']` (#160054): the credential text rules never start a value at an
opening bracket -- one opens a label masking wrote, or JSON the key walk
re-serialised -- and the key `pw` names nothing, so the container shipped
whole. The same held for `password=%s` with a list, and for a repr of a
mapping that is no dict (`{'password': ['s3cret']}`).
Rule 6 takes the whole container after a credential key and `=` or `:` --
a list, dict, tuple or set, its strings read whole so a bracket in one ends
nothing, nested up to three deep, or the rest of the line where it is never
closed -- and masks it as a credential's value: its token, or the label. A
repr's quoted key keeps its value a string. It leaves alone what the key
walk already masked: a JSON key's container (the saved card under `token`
shows its own number, brand and expiry), and a container holding only
masking's own output, as a record's credential field renders it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l's label `Authorization: Bearer 411111******1111` was hashed whole with a keyset: a token of the scheme and the truncation (#160054). A bare truncation as a credential is already the label -- ten digits of a saved card's gateway token, never passed through as masking's own output -- but with a scheme word before it, mask_by_field_type read neither a truncation nor a placeholder, and tokenized it. A truncation after a scheme word, as `Bearer ****` is read as a placeholder after one, is now [SECRET-MASKED] on every path: the key walk, the Authorization text rules, an Authorization element, mask_secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a scheme word, redact_body masks a form value the text rules left (a label with more after it, a lone quote), and the scheme word's bare-space rule (rule 11) then read `password=<token>` as its credential and hashed it again: a token of a token (#160054). It was pinned as a strict xfail, since skipping `key=<token>` there would let a credential glued to a key name through (`Token abc123abc123api_key=…`). Rule 11 reads a credential only where the value holds a digit, and here the digits are masking's own. It now leaves a value whose digits are all in masking's output -- a whole token, a label, a truncation -- as it is, and still hashes one whose own characters carry digits, the glued credential included. The xfail's shapes join the text-rules-then-redact_body-twice test, and a test pins the glued credential. Fuzzing that composition turned up two more readings that disagreed on a second pass, from this release's end-tag change: rule 5's quoted value ran across an end tag that redact_body's form value now stops at, and redact_body decoded an XML leaf's entities in a form body the first pass had read as written. A quoted value now stops at an end tag, as the unquoted one does; only JSON or XML written with entities is decoded (`&` is a form's separator either way). On 280k generated shapes the composition is now a fixed point; 0.15.5 missed 240 of 120k. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is masked in the record
`logger.info("cvv=%s", "123")` shipped masked, since the formatter's
processor masks the formatted line whole, but after MaskPIIFilter ran
`record.getMessage()` still read `cvv=123` (#160054): the filter masks a
template and its arguments apart, and neither `cvv=%s` nor `123` holds
anything to mask on its own. A handler that never calls format() printed
it, and Sentry's logging integration and handleError read it.
The filter now also renders a record that keeps its template from its
masked arguments, and when masking that line changes it -- a value split
between the two, a CVV after a card number the template holds -- takes the
render path the record takes when its template changes: rendered from the
arguments masked with every pack, masked whole, its arguments (). A record
whose rendered line holds nothing more to mask keeps its template and
arguments, and Sentry's grouping by them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er Authorization
Ottu PG now sends `Authorization: API-Key <merchant key>` on its calls into
Connect, and its own filter, now deleted, masked a bare `API-Key <key>` in
text. ecsctx 0.15.5 masked only the key there, beside the scheme word
(`API-Key ptok:v1:…`): a token of the key alone, which is not the token the
header carries under its key or after `Authorization:` -- the scheme and
the key together, mask_secret("API-Key <key>").
After the `API-Key` scheme word standing on its own, matched
case-insensitively, the credential bare-space rule (rule 11) now masks the
scheme and the key as one value, with that rule's value ends (its
delimiters, escape-aware quotes): `sending API-Key abc123XYZ to core`
carries the header's token, or [SECRET-MASKED]. A header name ending in the
word (`X-API-Key <key>`) is no scheme, and its key alone is masked, as
before; no other word is read as a scheme.
test_masking_filter pinned `api-key abcd1234` and `API-Key <hex>` as
`API-Key [SECRET-MASKED]`, the scheme word kept beside the label; they are
the label alone now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… hash 0.15.5 masked a rich structured event 1.4-1.7x slower than 0.15.4 (#160054). The cost was not the rules that ran: scalar_rules() and _distinct_gates() were lru_caches keyed on the rule tuple, and hashing a tuple of rules hashes every rule -- and a compiled re.Pattern hashes its whole program, on every call. Every string a record's walk reached paid it: 6.5 µs for 0.15.4's seven default rules, 10.8 µs for 0.15.5's eleven, and 32 µs for this release's default pack, whose wallet rule is a large pattern. Both are now cached on the tuple's identity, as the known-clean sets already were (the tuple is kept in the entry, so its id is never reused), which is all scalar_rules() promised: a stable object per set. A combined literal gate -- one precompiled search of every rule's trigger literals before the per-rule pre-checks -- was built and measured, and not kept. On typical log strings (ids, codes, amounts, URLs, uncached) it cost 4.9 µs a string against 3.9 without it in the default pack (case- insensitive alternation tries every alternative at every position); run over the lowercased text instead it saved 6% in the default pack and cost 6-78% with pci, whose shapes (IBAN prefixes, card runs, three digits) make a long alternation. The substring pre-checks it would replace are already cheap. Medians of three alternating runs of the #160054 benchmark, py3.12, no keyset (0.15.4 vs this branch): event_dict_us 261 vs 141 in the default pack, 340 vs 163 with pci and financial_ids -- 0.54x and 0.48x. tests/test_masking_golden.py runs a fixed corpus -- the suite's tables, the benchmarks' inputs and seeded fragments -- through every masking path, in the default pack and every pack, with a fixed keyset and without one, against outputs captured before this change: they are byte-identical. A test pins that a rule set is never hashed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dard do Where they contradicted the code or the Ottu services' standard (#160054): - Message style: Mistake #2 called printf-style messages wrong and every example logged a snake_case "event name". The standard, which Connect and Ottu PG gate, is lowercase prose with a constant template and lazy %s arguments; the event is named by ecs_event=. Every example follows it. - Expiry: the README listed expiry keys among the masked ones, and the root CLAUDE.md among the key names masked in every service; they are safe keys (patterns.SAFE_KEYS), never masked, and can be listed. - Auth headers: tokenized as credentials (mask_secret), scheme and credential together, never "truncated". - Shipping: the generic stdout -> docker -> logs-{PROJECT}-{ENV} guidance stays, beside what Ottu does: JSON log files Vector tails, daily logs-YYYY.MM.DD indices, logs-pci-<env> for a PCI service. - EventSpec.required: its comment said the validator enforces it; nothing does. The comment only changes. - Level table: auth, retryable and customer-config failures were info, but the contract validator reports a failure outcome below warning; they are warning. And this release's behaviour, where the README lists what redact_body and the packs mask: XML bodies and the credential element rule, wallet tokens (the whole token is the label, found by shape), the narrowed pci CVV rule, the credential container, the API-Key scheme, a card-shaped user alone in a URL, a truncation after a scheme, a value split between a template and its argument, CPython's %-grammar, the fixed point the accepted residual was; ecsctx/CLAUDE.md's rule numbers and the identity-keyed caches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ey=value 83773e4 read the bare `API-Key` scheme inside the bare-space credential rule, so its guards held: a value was masked only with a digit and eight characters or more. `API-Key abcdefghijklmnop` and `API-Key ab1cd2` shipped in clear, and about 1 in 1,550 of Ottu PG's 44-character Fernet keys has no digit. The filter Ottu PG deleted masked any value after `API-Key`. The scheme word is the evidence. It now has its own rule (rule 5): after `API-Key` standing alone, case-insensitively and not ending a header name (`X-API-Key`), any value to its delimiter -- the credential value ends, with escape-aware quotes -- and the scheme and the value are one value, mask_secret("API-Key <value>"), the token `Authorization: API-Key <value>` and the key walk give. It runs before the key=value rule, which masked a `password=` inside the value first, where it never does after `Authorization:`: then the two forms carried different tokens. A form field masking already wrote after the scheme (`API-Key password=<token>`) is left, never hashed again. Bearer and the other schemes keep their guards ("Bearer of bad news"). Rules from 5 on are renumbered, in the code and the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
@claude review |
|
Claude finished @jab3z's task in 37s —— View job Review of PR #70 (masking follow-ups for 0.15.6)I read the diff for Overall: the design is sound. XML leaves use the same key classification as JSON and form fields. Wallet tokens are detected by shape. The Things to consider
Minor
VerdictNo blockers from reading the code. Before merge I'd check that CI is green and that the benchmark covers args-bearing records (item 1). Items 2–5 are optional hardening or tests. |
Rule 3 paired each credential start tag with its end tag by a lazy regex, and redact_body searched for one from each start tag: a text holding many start tags that nothing closes took quadratic time -- 160 KB of `<password>` twelve seconds in the text rule, 0.6 s in redact_body. A text's end tags are now indexed once, when the first element needs one, and a start tag's is found by its name in that index; the rule is that walk, handed the text from its first `<`. The rule's output is unchanged: the old regex and the walk agreed on 120,000 generated shapes, and the golden corpus holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`<cvv>123</cvv>` and `<pin>1234</pin>` in a message went out in clear in every pack, and a test card that fails Luhn in `<cardNumber>` whole under `pci`: the text rules read no element but a credential's. Rule 3 now masks an element named as the key walk names a card, a CVV or other SAD, in every pack, as the key walk masks that key's value: a CVV's or SAD's text, children and all, its label; a card's text a card value -- a card number truncated, what a card key refuses [CARD-MASKED] -- and a card object (CyberSource's `<card>`) field by field, so `<accountNumber>` is truncated and an expiry reads through. It runs before the credential element, now rule 4, so a `<cvv>` inside `<password>` or `<token>` is its label and the credential's token is taken over the masked text: it was a keyed hash of text holding the CVV. Both rules are the same walk (4ff17e7). redact_body reads a body's elements with that walk, first. It masked a card element's text whole as one card value, which left a `<cvv>` inside `<card>` in clear, and a CVV inside a credential's element went into the credential's token. Rules 3 to 23 are now 4 to 24. The golden corpus is recaptured for the 12 outputs this changes (ECSCTX_WRITE_GOLDEN=1): three texts' `<cvv>` elements, in each mode and pack, are now [CVV-MASKED]; one held a card number the default pack left in clear. The README notes the classification redact_body shares with a URL's query: `<auth>`, `<user>`, `<sign…>` and `<key>` are credentials. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umber run is the label With `pci`, `4111111111111111@example.com` in text got a keyed hash from the email rule, and `call +4111111111111111 now` one from the phone rule, which runs before the card rule: a keyed hash of a value holding a card number, the pattern PCI DSS FAQ 1117 warns about, where the key walk gives both their labels. `mask_by_field_type` now checks an email and a phone number for a card-number run as it checks a credential and a payment id (`_PAN_CHECKED_TYPES`), on every path, so both are [EMAIL-MASKED] and [PHONE-MASKED]. A phone number is judged as the key walk judges a phone field, as written: a "+"-led run of up to E.164's 15 digits is the number and keeps its token, so real international numbers are unchanged (+44 20 7946 0958, +86 138 0013 8000). Without `pci` both keep their tokens: a default-pack service receives no card numbers. The golden corpus is recaptured for the 4 outputs this changes, all with every pack and a keyset: three emails a URL's userinfo holds a card number in (`https://4111111111111111@h.example/` after a digit, which the userinfo rule never starts at), and one email holding a 14-digit run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…holder can join a value Review of #70: every record with arguments had its rendering masked a second time, to find a value split between the template and an argument (`cvv=%s` with `123`). On bench_mask.py that put a record with arguments at 1.21x 0.15.4 in the default pack and 1.23x with pci (8.13 and 8.19 us against 6.73 and 6.67, medians of five alternating runs); on a real record, whose rendering the known-clean cache has not seen, the pass costs more. The rendering is now masked again only where a placeholder stands beside what a content rule reads across it (`_may_join`, cached per template): a word a rule keys a value on within reach before it -- a credential's or scheme's, a CVV's, a card's, a payment id's (`cvv=%s`, `Bearer %s`, `the cvv is %s`) -- structure beside it (`=`, `:`, `@`, a quote, a bracket, digits), or another placeholder (`card %s %s`, a card number and its CVV, which a guard on credential and CVV words alone would have left `card 411111******1111 123` under pci). Between plain words (`user %s paid %s KWD for %s`) nothing can join, and it is rendered only to find arguments that no longer fit. `_mask_value` also checks a value for a wallet token inside its type's branch, one call fewer per value. Now 7.43 and 7.57 us against 6.73 and 6.76: 1.10x and 1.12x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as a query param is Review of #70: redact_body reads an element's name as redact_url reads a param's, by substring, so `<author>`, `<design>`, `<monkey>`, `<userType>` and `<keyword>` are masked as credentials. The cost of failing closed is accepted; a test pins it for both readers, and the README says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #70: an XML start tag ended at its first `>`, so in `<password hint="a>b">x</password>` the rest of the attribute went into the element's text: masked, but hashed with the credential, whose token was not the key walk's. A start tag's attributes are now read with their quoted values (`_ATTRIBUTES`), in the element walk and in a card object's markup. A quote nothing closes before the next tag still leaves the first `>` to end the tag, as before -- never no start tag, which would leave the element's text in clear. Linear: each piece of an attribute starts with a character no other can, and a quoted value never crosses a `<`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… time Review of #70: redact_body reads `key` and `token` elements as credentials, and loggable_body and loggable_request_body redact a body before they cap it (as Connect's caller does), so every byte a gateway sends reaches it. Since 4ff17e7 a body's end tags are indexed once; this pins it for those names and for many distinct ones. Measured: 1 MB of them in 0.5 s and 2 MB in 1.1 s, a 1.2 MB form body in 0.4 s and 2.4 MB in 0.8 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aracter Review of #70: `_FORM_VALUE` stopped a form value at an end tag with a lookahead before every character, which made a 64 KB form body's field pass twice as slow as 0.15.5's plain character class: redact_body took 7.57 ms on it against 6.27 ms (1.21x), and 22.3 ms against 19.5 ms (1.14x) with credentials in every field, medians of three alternating runs. It is now unrolled, the lookahead tried only at a `<`: the same matches on 300,000 generated shapes, and 6.77 ms against 6.23 ms (1.09x) and 19.47 ms against 19.33 ms (1.01x). What remains is the wallet pass's pre-check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he marker Review of #70: `_render` reads only TypeError, ValueError and KeyError as "does not fit". An argument whose `__str__` raises anything else fails in `_mask_args` first, inside the filter's catch-all, so the record is `[MASKING-FAILED: RuntimeError]` with no arguments, through the filter and through a logger, in both packs and for templates the join guard renders and skips alike -- never an exception out of the log call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… their label (ECSCTX_MASK_VALUE_RULES)
Some values are known by their shape, not by the key they sit under, and
which shapes those are is a service's to say. A service now lists value
rules in ECSCTX_MASK_VALUE_RULES, resolved as ECSCTX_MASK_SAFE_KEYS is:
configure_masking_value_rules(), then the Django setting, then the
environment variable (dotted import paths, comma-separated). An item is a
rule object or a dotted path to a rule or to a collection of them; one that
is not makes the call raise, and from the setting or env var is dropped
with a warning and fails the boot check while the rest apply.
A rule (`ValueRule`, or anything shaped like one) has a `field_type`, the
label a matching value becomes (make_label), `matches(value)`, called with
a mapping or with JSON text the engine parsed once, and optional `hints`,
literal strings its text holds one of, so text holding none is never
parsed to ask. The engine asks the rules in force about every mapping the
key walk meets under any key, every string that is JSON text (and JSON text
holding a match deeper, under a key that would hash it), every JSON object
written in text -- rule 2, now generic: JSON, a Python repr or JSON in a
JSON string, an object no rule matches read member by member -- and every
such object in a body, first in redact_body, JSON in an XML element's text
included. A match is the rule's label, never hashed, left as it is by a
later pass. Nothing configured, nothing runs: the rules are looked up only
when a mapping, JSON text or text holding `{` asks, at most once a call.
The wallet shapes run through this hook as a built-in rule, unchanged: the
golden corpus is byte-identical. The next commit moves them out of core.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n core
Treating Apple Pay and Google Pay payment data as Sensitive Authentication
Data is Ottu's decision, not something a generic logging library imposes on
every service. The two shapes move out of the masking core into
ecsctx.contrib.ottu.masking as WALLET_RULES -- value rules (the previous
commit's hook) with the same shapes as before: an Apple Pay `paymentData`
(`version` EC_v1 or RSA_v1, with `data`) and a Google Pay token
(`protocolVersion` ECv1, ECv2 or ECv2SigningOnly, with `signedMessage`),
the whole object [SAD-MASKED], an unrelated `paymentData` readable.
A service opts in beside its safe keys:
ECSCTX_MASK_VALUE_RULES = [*WALLET_RULES]
# or ECSCTX_MASK_VALUE_RULES=ecsctx.contrib.ottu.masking.WALLET_RULES
Without it, core masks no wallet token: an Ottu service must add the
setting when it takes this release. Nothing under ecsctx/masking or
ecsctx/contrib/net names a wallet shape any more, and with no value rules
configured none of that code runs.
The wallet tests move to tests/test_contrib_ottu_wallets.py and run with
WALLET_RULES configured, plus one that a token is not labelled with nothing
configured. The golden corpus holds wallet tokens; its harness now
configures WALLET_RULES as an Ottu service does, and its outputs are
byte-identical: nothing is recaptured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ecsctx 0.15.6: the masking follow-ups in Redmine #160054, and a leak found in the #159487 final round. It adds a public setting (
ECSCTX_MASK_VALUE_RULES) and moves wallet masking behind it, so it is a minor bump (0.16.0) rather than a patch.The leak: XML bodies.
redact_body. KNET's KPay sends XML with<password>(the tranportal password) on inquiry and refund, and the Apple Pay token in<udf9>.redact_bodyand the text rules now read XML elements with the same key classification as JSON and form fields:<password>becomes a token, and element text that is JSON gets the JSON rules.<cvv>123</cvv>and<pin>1234</pin>become their labels, and<cardNumber>is truncated, including a test card that fails Luhn, with or withoutpci. A card object (CyberSource's<card>) is read field by field.redact_bodyshares that walk. It used to mask a whole card element as one value, which left a<cvv>inside<card>in clear.<password>,<token>) becomes its label first. The credential's token is then taken over the masked text.Value rules (new setting): wallet data is now an Ottu opt-in.
ECSCTX_MASK_VALUE_RULES, resolved likeECSCTX_MASK_SAFE_KEYS:configure_masking_value_rules(), then the Django setting, then the env var, which takes comma-separated dotted paths.ecsctx.masking.ValueRule) has:field_type, which becomes its label;matches(mapping);hints, literal strings its text must hold before it is parsed.redact_bodybodies, including an XML element's text.ecsctx.contrib.ottu.masking.WALLET_RULES, with the same shapes as before: an Apple PaypaymentData(versionEC_v1/RSA_v1 withdata) and a Google Pay token (protocolVersionwithsignedMessage). The whole object becomes[SAD-MASKED], and an unrelatedpaymentDatastays readable.ECSCTX_MASK_VALUE_RULES = [*WALLET_RULES], next toECSCTX_MASK_SAFE_KEYS, or setECSCTX_MASK_VALUE_RULES=ecsctx.contrib.ottu.masking.WALLET_RULES. Without it, this release masks no wallet token.Under
pci, an email or phone number that holds a card-number run becomes the label, as under its key:4111111111111111@example.combecomes[EMAIL-MASKED]and+4111111111111111becomes[PHONE-MASKED]. Before, each was a keyed hash (PCI DSS FAQ 1117). A phone number is judged as a phone field is: a+-led number of up to E.164's 15 digits keeps its token. Withoutpci, both keep their tokens.#160054:
%.f(a precision with no digits) no longer turns a record into[MASKING-FAILED: TypeError]. The filter now reads a %-template as CPython does, backed by a property test.pci, the bare-digit CVV rule reads only a CVV beside its card:card list returned 200 in 350 msand decline code051stay whole.Bearer <truncation>gives the label;API-Key <key>in free text is one masked value, likeAuthorization:. Ottu PG now sends that scheme.Review hardening (#70 review):
<password>took 12 s in the text rule.redact_bodyredacts beforeloggable_bodycaps, so every byte reaches it; 1 MB of unclosed<key>/<token>tags now takes 0.5 s, and 2 MB takes 1.1 s.cvv=%s,Bearer %s,the cvv is %s);card %s %s).>inside a quoted attribute (<password hint="a>b">) belongs to the attribute. A quote that nothing closes still falls back to the first>.<author>,<design>,<monkey>,<userType>and<keyword>are therefore credentials. A test pins this, and the README says so.<.__str__raisesRuntimeErrorproduces[MASKING-FAILED: RuntimeError], never an exception.Speed. The 0.15.5 slowdown was
lru_cachehashing every compiled pattern on each call. Caches keyed by identity fix it. Medians of 5 alternating runs ofbench_mask.py:WALLET_RULESpci,financial_idspciA 64 KB form body in
redact_bodytakes 6.77 ms against 0.15.5's 6.23 ms, and 19.47 ms against 19.33 ms with a credential in every field.Docs.
required=, the level table, and the XML rules.WALLET_RULESas the service's opt-in.Behaviour changes to know
WALLET_RULESinECSCTX_MASK_VALUE_RULES.auth,user,sign…orkeyare treated as credentials inredact_body, as the same names are in a URL's query (redact_url). The match is by substring, so<author>counts too. In a form body, only the secret body keys are credentials.<cvv>,<pin>and<cardNumber>elements in free text are masked in every pack.pci, a bare 3–4 digit field right after a card is masked, and an email or phone number holding a 12+ digit run becomes its label. A phone number up to 15 digits after+keeps its token.API-Keymasks its scheme word along with the key, with no digit or length guard.encodedModel) have no shape to detect. They're left to their field's key.Test plan
<cvv>outputs the XML card rule changes and the 4 email outputs thepciemail check changes;WALLET_RULESconfigured.redact_bodyare each a fixed point, no<cvv>123</cvv>survives either, and the wallet shapes behave the same with the rules configured. Their composition is a fixed point too, except for 2 shapes in 20,000 in one seed, where a>after a label is dropped on a second pass. Both are pre-existing at 11f5ec3, and nothing is left unmasked.🤖 Generated with Claude Code