Keep-rules follow-ups: unforgeable placeholders, CVV floor in text, card numbers in token text (0.17.1, #160102)

CLAUDE.md

@@ -24,6 +24,6 @@ ECS-compliant structured logging with W3C Trace Context support. Framework-agnos
 - Django's `LogContextBinder` must be imported explicitly (not in `__init__.py`) to avoid circular imports
 - `LoggingContextMiddleware` must be placed AFTER auth middleware to capture user_id
 - CVV never carries a token, even when PII is configured — PCI forbids storing CVV in any form, so `[CVV-MASKED]` is always the literal, final output
-- Card number **content** rules (and the bare 3-4 digit CVV rules, which read only a CVV beside a card number or its truncation, or after a CVV word) are the opt-in `pci` pack (`get_logging_config(masking_packs=("pci", "financial_ids"))` or `ECSCTX_MASKING_PACKS`): a PCI service that does not enable it gets no PAN content scanning. The keyed CVV text rules (`cvv=123`, `"securityCode": "123"`, `CVV 123`) are `default`: a CVV must not ship from any service, and Connect (default pack) receives the CVV a saved-card payment sends. Key names (`card`, `pan`, `cvv`, …) are masked in every service; expiry keys are safe keys, never masked. A value a service's label rule matches (`ECSCTX_MASK_VALUE_RULES`, found by shape) is its label in every pack, and one its keep rule matches ships as sent -- rules asked in list order, the first match winning; nothing kept under a CVV/SAD key or container (the floor), nor a match holding a card/CVV/SAD key or a card number (the guard); in text a kept value is a `[KEPT-…-MASKED]` placeholder while the content rules run. Core names no shape itself -- Ottu's wallet tokens are `ecsctx.contrib.ottu.masking.WALLET_RULES` (keep rules: Apple Pay and Google Pay tokens ship as sent, decided for #159487; `WALLET_SAD_RULES` labels them as 0.16.0 did), each service's to list
+- Card number **content** rules (and the bare 3-4 digit CVV rules, which read only a CVV beside a card number or its truncation, or after a CVV word) are the opt-in `pci` pack (`get_logging_config(masking_packs=("pci", "financial_ids"))` or `ECSCTX_MASKING_PACKS`): a PCI service that does not enable it gets no PAN content scanning. The keyed CVV text rules (`cvv=123`, `"securityCode": "123"`, `CVV 123`) are `default`: a CVV must not ship from any service, and Connect (default pack) receives the CVV a saved-card payment sends. Key names (`card`, `pan`, `cvv`, …) are masked in every service; expiry keys are safe keys, never masked. A value a service's label rule matches (`ECSCTX_MASK_VALUE_RULES`, found by shape) is its label in every pack, and one its keep rule matches ships as sent -- rules asked in list order, the first match winning, though a value inside a keep match whose own first match is a label rule (one listed before any keep rule matching it) is labelled and the keep refused; nothing kept under a CVV/SAD key or container (the floor), nor a match holding a card/CVV/SAD key or a card number (the guard); in text a kept value is a `[KEPT-…-MASKED]` placeholder while the content rules run. Core names no shape itself -- Ottu's wallet tokens are `ecsctx.contrib.ottu.masking.WALLET_RULES` (keep rules: Apple Pay and Google Pay tokens ship as sent, decided for #159487, each slot text in its alphabet; `WALLET_SAD_RULES` labels them as 0.16.0 did), each service's to list
 - Card numbers truncate to first 6 + last 4 for 15–19 digits and last 4 only below 15 (`411111******1111`, bare, PCI DSS 3.5.1, FAQ 1091) — no token alongside, independent of PII configuration, and a second masking pass is a noop
 - `install_maskers()` only sweeps handlers that exist at call time (no stdlib patching) — a handler built after the call needs another call

ecsctx/CLAUDE.md

@@ -28,7 +28,7 @@ Core structured logging module with ECS field mapping and PII/PCI masking.
 
 ## Footguns
 - `SAFE_KEYS` whitelist (`masking/patterns.py`) prevents masking of non-PII fields containing "name" etc. (e.g., `service_name`, `pathname`); checked before every other key rule. It holds only names that mean the same in every service — never a service's own vocabulary. A service lists its own in `ECSCTX_MASK_SAFE_KEYS` / `configure_masking_safe_keys()` (`masking/config.py`, resolved like the packs, passed to `classify_key` as `safe`); anything the classifier calls a card, CVV, SAD or credential is refused (`never_safe` asks `classify_key`; expiry is not refused, since nothing masks it). A listed key also frees a digits-only reference number of up to 14 digits from the content rules when the key is not PII on its own and not inside a card/credential/CVV/SAD container (15-19 digits always truncate). Ottu's list is `ecsctx.contrib.ottu.masking.SAFE_KEYS`, opted into by each service, never installed implicitly
-- Value rules (`masking/value_rules.py`): a service's own shapes of value, masked as their label (label rules) or shipped as sent (keep rules) wherever they are logged; ecsctx names none. `ECSCTX_MASK_VALUE_RULES` / `configure_masking_value_rules()` (`masking/config.py`, resolved like the safe keys: rule objects, or dotted import paths to a rule or a collection, comma-separated in the env var; an item that is not one raises from the call, is dropped with a warning from the setting or env var, and fails the boot check). A label rule has `field_type` (the label, `make_label`), `matches(mapping)` and optional `hints` (literal strings its text holds one of: a rule is never asked about text holding none, `applicable`; the key walk of JSON text passes on only those, `_mask_json_text`). A keep rule (`KeepRule`, or `keep is True` with a callable `matches`) writes no label: what it matches ships as sent. One list, asked in its order, the first match winning (`_first_match`; each rule's kind is cached on the tuple's identity, `_split`). The engine asks them about every mapping the key walk meets under any key, every JSON-text string (`_value_ruling` -> label, `KEEP` or None; JSON text holding a label match deeper under a key that would hash it is the label, `label_within`, which stops at a kept mapping: JSON text holding a kept value deeper is still hashed whole), every JSON object in text (rule 2 for label rules, `_OBJECT_TEXT`: JSON, a repr, or JSON in a JSON string, three levels deep; an object no rule matches is read member by member) and every such object in a body (`redact_body` -> `patterns.mask_objects`, first). A label match is the rule's label, never hashed. A keep match is a deep copy in the key walk, not descended into; in text `mask_by_patterns` replaces it with a `[KEPT-<letters>-MASKED]` placeholder before every rule (`stash_kept`/`kept_spans`, top down: a label match is no span, a CVV/SAD key right before an object -- quoted, `key=`, an XML start tag -- floors it and all inside it) and restores each placeholder still there; a keep decision on text rests on the text as written (`json_as_written` refuses a key written twice, `_written_as_parsed` keeps a repr only where it renders back exactly); restored text is not recorded clean, and text holding `[KEPT-` is masked as before. Never kept: under a CVV/SAD key, inside a CVV/SAD container (a listed key there reads through with the label rules only, `_without_keep`), the `value` of a pair labelled as one (`_floored`); nor a match holding a card/CVV/SAD key, a `{name, value}` pair labelled as one, or a card number at any depth, JSON text inside included -- keys, pairs and whole leaves only, a CVV inside a longer string is the matcher's to keep out (`holds_card_data`: a string leaf is a card number when `pan_shaped`, except 13 bare digits from a 1 -- epoch milliseconds, Google Pay's `keyExpiration`, which `pan_shaped` reads as one), which is walked as if nothing matched. A keep matcher that raises is no match (it runs in `redact_url`/`redact_body`/`mask_card_value`); a label matcher's error leaves the marker. `is_kept`/`mask_outside_kept` (`ecsctx.masking`) serve a service's own pass after ecsctx's. Configuring the value rules empties `patterns._clean`. Nothing configured, nothing runs: the rules are looked up only when a mapping, JSON text or text holding `{` asks, at most once a call in the key walk (`_Pass.values`).
+- Value rules (`masking/value_rules.py`): a service's own shapes of value, masked as their label (label rules) or shipped as sent (keep rules) wherever they are logged; ecsctx names none. `ECSCTX_MASK_VALUE_RULES` / `configure_masking_value_rules()` (`masking/config.py`, resolved like the safe keys: rule objects, or dotted import paths to a rule or a collection, comma-separated in the env var; an item that is not one -- a rule with both a `field_type` and `keep = True` included -- raises from the call, is dropped with a warning from the setting or env var, and fails the boot check). A label rule has `field_type` (the label, `make_label`), `matches(mapping)` and optional `hints` (literal strings its text holds one of: a rule is never asked about text holding none, `applicable`; the key walk of JSON text passes on only those, `_mask_json_text`). A keep rule (`KeepRule`, or `keep is True` with a callable `matches`) writes no label: what it matches ships as sent. One list, asked in its order, the first match winning (`_first_match`; each rule's kind is cached on the tuple's identity, `_split`) -- for the value itself: a value strictly inside a keep match whose own first match is a label rule (listed before any keep rule matching it) is labelled and the keep refused (`keep_refused`: the guard, or `label_within` over its members, asked only with a label rule among the rules, a label rule's error refusing the keep and raising nothing; called from `ruling` and rule 2). The engine asks them about every mapping the key walk meets under any key, every JSON-text string (`_value_ruling` -> label, `KEEP` or None; JSON text holding a label match deeper under a key that would hash it is the label, `label_within`, which looks into a keep match like any mapping, a kept one holding no label: JSON text holding a kept value deeper is still hashed whole), every JSON object in text (rule 2 for label rules, `_OBJECT_TEXT`: JSON, a repr, or JSON in a JSON string, three levels deep; an object no rule matches is read member by member) and every such object in a body (`redact_body` -> `patterns.mask_objects`, first). A label match is the rule's label, never hashed. A keep match is a deep copy in the key walk, not descended into; in text `mask_by_patterns` replaces it with a `[KEPT-<letters>-MASKED]` placeholder before every rule (`stash_kept`/`kept_spans`, top down: a label match is no span; and no span at all in text that names a CVV/SAD key anywhere -- `floored_text`, read only once a span is found: a quoted key then `:`/`=`/`=>`, a bare key then `:`/`=>`, a key and `=` where a key starts (not base64 padding in a string), an XML tag, or a `{name, value}` pair's label, `classify_key(..., ALL_PACKS, safe keys)`, as written and entity-decoded; fails closed, a sibling's value too, since the text rules read objects only three levels deep; `redact_body` passes it to an entity-encoded leaf, `_encoded_leaf(keep=False)`) and restores each placeholder still there; a placeholder is named from a blake2b digest of its value (`_placeholder`, 14 letters: the same value one name in any text, so a credential hashed around it carries that value in its token, and `restore` fills only its own names, so a name written into the text restores nothing; two values whose names collide are masked as without a keep rule); a keep decision on text rests on the text as written (`json_as_written` refuses a key written twice, `_written_as_parsed` keeps a repr only where it renders back exactly); restored text is not recorded clean, and text holding `[KEPT-` is masked as before. Never kept: under a CVV/SAD key, inside a CVV/SAD container (a listed key there reads through with the label rules only, `_without_keep`), the `value` of a pair labelled as one (`_floored`); nor a match holding a card/CVV/SAD key, a `{name, value}` pair labelled as one, or a card number at any depth, JSON text inside included -- keys, pairs, whole leaves, and a card number's digits inside a text leaf that is no JSON text (`patterns.holds_card_run`: `holds_pan_run`, 12+ digits joined by single separators, as the pci rules read a value, one linear pass with a bytes-translate fast path for base64; not read in a leaf that is a whole hex id of 24+ characters holding a hex letter, `_HEX_ID` fullmatch -- Apple Pay's 64-hex ids; Ottu's base64 slots refuse hex alone, 1.1% of which hold a Luhn run -- nor a leaf that is epoch milliseconds); a CVV or other short value inside a longer string is the matcher's to keep out (`holds_card_data`: a string leaf is a card number when `pan_shaped`, either way except 13 bare digits from a 1 or 2 -- epoch milliseconds, Google Pay's `keyExpiration`, which `pan_shaped` reads as one), which is walked as if nothing matched. A keep matcher that raises is no match (it runs in `redact_url`/`redact_body`/`mask_card_value`); a label matcher's error leaves the marker. `is_kept`/`mask_outside_kept` (`ecsctx.masking`) serve a service's own pass after ecsctx's. Configuring the value rules empties `patterns._clean`. Nothing configured, nothing runs: the rules are looked up only when a mapping, JSON text or text holding `{` asks, at most once a call in the key walk (`_Pass.values`).
 - CVV never goes through tokenization at all, even when PII is configured — the CVV rules return the literal `[CVV-MASKED]` directly. PCI forbids storing CVV in any form, not even as an HMAC digest.
 - SSN/PEM tokens are computed over a normalized value (digits-only for SSNs, base64-body-only for PEM), not the raw matched text — otherwise the same value in a different grouping/wrapping would produce a different token and break correlation. Card numbers carry no token at all: they truncate to first 6 + last 4 from 15 digits up and last 4 only below (`411111******1111`, bare, PCI DSS 3.5.1, FAQ 1091), separators stripped, independent of PII configuration. The star run is the marker: `already_masked()` and `_text_has_card_context()` both recognise that shape, since there is no wrapper left to read. The card rule (`_CardRun`) reads a whole run of digit groups at once: where a separator admits one card or a card beside another number, Luhn decides, and every Luhn-valid 12-19-digit stretch along the groups lies inside a truncated span (overlapping ones merged), so no output shows more than its first six and last four -- except over digits it leaves as something else's: an IBAN's (`_iban_end`: its country's registered length from `_IBAN_LENGTHS`, in groups of four joined by single spaces, check digits holding; after a letter bank code only if the IBAN ends where the run does), a word's own or a truncation's last four (unless they start a Luhn-valid card in card-style groups, which makes them a card's), a phone number's after "+" (at most 15 digits, E.164, even a card's), a range's across a Unicode dash (no stretch crosses it; a Unicode dash is a separator only when the whole run is card-style groups), and a canonical UUID's (`_UUID`; the run is read again without them, `_truncated_outside`). Accepted residuals follow from those: a card glued to a word or to `****` in non-card-style groups, one split by a range's dash, a card completing an IBAN of its country's exact length with check digits holding, a card typed into a UUID's own groups (`41111111-1111-1111-abcd-ef0123456789` stays whole in free text, and under a name key with a keyset it is hashed), and the phone rule (it runs first) taking phone-shaped digits at a card's end (ten before a Unicode dash, or its last digits with those after them). A later pass leaves a truncation's first six and last four alone. A card key never shows those residuals: it refuses instead, since `mask_card_value` shows its scan only while fewer than twelve digits show outside the truncations the card rule writes, however they are joined -- double spaces, commas, another masker's `X`s (`_digits_beside_truncations`, group by group: a search for the shape from every star of a long run was quadratic). Nor may twelve digits show in a row, however joined (`_card_digits_showing`, one running count): bare digits, a truncation's last four and the next one's first six count together, as a card number could be read across them; a truncation's own stars separate its first six from its last four; a word ends a row, but a run of `X`/`x` joins digits as stars do (another masker's `4508750XXX001019`). So `4111111111  111111******1111`, `400001******5018  00  000009******0001` (the Maestro 501800000009), `********4111  ********1111  1111`, `02313379 ********1262`, `411111******3782XX82246310005`, `512345******0008 12 25 512345******0008` and `4111111111111111 12345678901` are refused. The one residual left: a short group beside a truncation, under twelve digits with it (`450875******1019 000` could be a 19-digit number another masker cut oddly).
 - No dict-level "already masked" marker: a dict re-passed to `_mask_dict()` (e.g. `mask_sensitive_data` running twice in one chain) is fully re-traversed. Idempotency lives in the leaf checks instead — `mask_by_field_type` leaves an already-`[LABEL…]` value untouched via `already_masked()`, and a second content-regex pass over masked markers is a noop, so `_mask_string` always runs the full pass with no early-exit (a whole-string `already_masked()` gate there would let a coincidental `-MASKED]` fragment suppress masking for real PII in the same string). Only `filter()` marks anything, and only on the `LogRecord` (`mark_object_as_masked`/`is_masked_object`, a plain attribute that never ships as output), so a record reaching two handlers skips masking on the second pass.