Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9cfa1ea
fix(net): redact_body and the text rules mask an XML body's credentials
jab3z Sep 30, 2026
d30ff2f
fix(masking): a wallet token's payment data is [SAD-MASKED] in every …
jab3z Sep 30, 2026
4da024d
fix(masking): the filter reads a %-style template as CPython's % does
jab3z Sep 30, 2026
957f8d6
fix(masking): the pci bare-digit CVV rule reads only a CVV beside its…
jab3z Sep 30, 2026
1103bfd
fix(masking): a card-shaped user alone in a URL's userinfo is the lab…
jab3z Sep 30, 2026
4f6759a
fix(masking): a credential word before a container masks the whole co…
jab3z Sep 30, 2026
79193e0
fix(masking): a card truncation after an auth scheme is the credentia…
jab3z Sep 30, 2026
6815f03
fix(masking): the text rules then redact_body never hash a token again
jab3z Sep 30, 2026
a4c86bb
fix(masking): a value split between a record's template and argument …
jab3z Sep 30, 2026
83773e4
fix(masking): a bare API-Key scheme and its key are one value, as aft…
jab3z Sep 30, 2026
fd9599c
fix(masking): cache a rule set's derived tuples by identity, never by…
jab3z Sep 30, 2026
edc9111
fix(docs): the README and CLAUDE.md say what the code and Ottu's stan…
jab3z Sep 30, 2026
11f5ec3
fix(masking): any key after a bare API-Key scheme is masked, before k…
jab3z Sep 30, 2026
4ff17e7
fix(masking): an XML element's end tag is found once per text
jab3z Sep 30, 2026
81675ea
fix(masking): XML card, CVV and SAD elements in free text
jab3z Sep 30, 2026
41f759b
fix(masking): under pci, an email or phone number that holds a card-n…
jab3z Sep 30, 2026
1d319f3
fix(masking): a record's rendering is masked again only where a place…
jab3z Sep 30, 2026
fe921a5
test(net): an element name holding a credential hint is a credential,…
jab3z Sep 30, 2026
492d64b
fix(masking): a `>` inside a quoted attribute value is the attribute's
jab3z Sep 30, 2026
3849f97
test(net): many unclosed <key> and <token> tags in a body take linear…
jab3z Sep 30, 2026
3d60f6b
fix(net): a form value's end-tag check runs at a `<`, not at every ch…
jab3z Sep 30, 2026
ff84e8a
test(masking): an argument whose __str__ raises RuntimeError leaves t…
jab3z Sep 30, 2026
0801822
feat(masking): value rules, a service's own shapes of value masked as…
jab3z Sep 30, 2026
5a84d12
feat(contrib): the wallet shapes are Ottu's WALLET_RULES, no longer i…
jab3z Sep 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ ECS-compliant structured logging with W3C Trace Context support. Framework-agnos
- Django's `LogContextBinder` must be imported explicitly (not in `__init__.py`) to avoid circular imports
- `LoggingContextMiddleware` must be placed AFTER auth middleware to capture user_id
- CVV never carries a token, even when PII is configured — PCI forbids storing CVV in any form, so `[CVV-MASKED]` is always the literal, final output
- Card number **content** rules (and the bare 3-4 digit CVV rule, which needs card context) are the opt-in `pci` pack (`get_logging_config(masking_packs=("pci", "financial_ids"))` or `ECSCTX_MASKING_PACKS`): a PCI service that does not enable it gets no PAN content scanning. The keyed CVV text rules (`cvv=123`, `"securityCode": "123"`, `CVV 123`) are `default`: a CVV must not ship from any service, and Connect (default pack) receives the CVV a saved-card payment sends. Key names (`card`, `pan`, `cvv`, `expiry`, …) are masked in every service
- Card number **content** rules (and the bare 3-4 digit CVV rules, which read only a CVV beside a card number or its truncation, or after a CVV word) are the opt-in `pci` pack (`get_logging_config(masking_packs=("pci", "financial_ids"))` or `ECSCTX_MASKING_PACKS`): a PCI service that does not enable it gets no PAN content scanning. The keyed CVV text rules (`cvv=123`, `"securityCode": "123"`, `CVV 123`) are `default`: a CVV must not ship from any service, and Connect (default pack) receives the CVV a saved-card payment sends. Key names (`card`, `pan`, `cvv`, …) are masked in every service; expiry keys are safe keys, never masked. A value a service's value rule matches (`ECSCTX_MASK_VALUE_RULES`, found by shape) is its label in every pack; core names no shape itself -- Ottu's wallet tokens are `ecsctx.contrib.ottu.masking.WALLET_RULES`, each service's to list
- Card numbers truncate to first 6 + last 4 for 15–19 digits and last 4 only below 15 (`411111******1111`, bare, PCI DSS 3.5.1, FAQ 1091) — no token alongside, independent of PII configuration, and a second masking pass is a noop
- `install_maskers()` only sweeps handlers that exist at call time (no stdlib patching) — a handler built after the call needs another call
368 changes: 262 additions & 106 deletions README.md

Large diffs are not rendered by default.

11 changes: 6 additions & 5 deletions ecsctx/CLAUDE.md

Large diffs are not rendered by default.

7 changes: 6 additions & 1 deletion ecsctx/contrib/django/checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,13 +215,18 @@ def find_masking_errors(logging_config: dict[str, Any]) -> list[str]:
The entry points below all go through this, so none of them can pass while
the other half is broken.
"""
from ecsctx.masking.config import masking_pack_errors, masking_safe_key_errors
from ecsctx.masking.config import (
masking_pack_errors,
masking_safe_key_errors,
masking_value_rule_errors,
)

return (
find_masking_config_errors(logging_config)
+ find_unmasked_live_handlers(logging_config)
+ masking_pack_errors()
+ masking_safe_key_errors()
+ masking_value_rule_errors()
)


Expand Down
149 changes: 116 additions & 33 deletions ecsctx/contrib/net/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,24 +23,32 @@
from __future__ import annotations

import contextlib
import html
import json
import os
import re
from collections.abc import Collection
from http import HTTPStatus
from typing import Any
from urllib.parse import unquote_plus, urlparse, urlsplit, urlunsplit
from xml.sax.saxutils import escape as xml_escape

from ecsctx.masking.filters import MaskPIIFilter
from ecsctx.masking.patterns import (
ALL_PACKS,
_XML_START,
_card_field_type,
_closing,
_element_text,
_end_tags,
_mask_card_field,
_mask_userinfo,
classify_key,
holds_pan_run,
mask_card_value,
mask_card_elements,
mask_objects,
mask_secret,
value_rules_in_force,
)
from ecsctx.masking.tokens import _MASKED_VALUE, _PLACEHOLDER, make_label, mask_by_field_type
from ecsctx.masking.tokens import _MASKED_VALUE, _PLACEHOLDER, make_label

_SECRET_LABEL = f"[{make_label('secret')}]"

Expand Down Expand Up @@ -229,7 +237,7 @@ def _get_body_log_cap() -> int:


def _get_compiled() -> tuple:
"""Cached (hint_re, json_re, form_re) for the current key set."""
"""Cached (hint_re, json_re, form_re, element names) for the current key set."""
global _compiled_cache
keys = _get_secret_keys()
cache_key = keys
Expand All @@ -256,10 +264,13 @@ def _get_compiled() -> tuple:
# Only the structure at its two ends stays outside the mask
# (_split_form_value).
form_re = re.compile(
r"\b(" + "|".join(keys) + r")=([^&\s]*)",
r"\b(" + "|".join(keys) + r")=(" + _FORM_VALUE + ")",
re.IGNORECASE,
)
_compiled_cache = (cache_key, (hint_re, json_re, form_re))
# An XML element named by one of them, lowercased: element names are
# matched case-insensitively, as the keys are.
names = frozenset(key.lower() for key in keys)
_compiled_cache = (cache_key, (hint_re, json_re, form_re, names))
return _compiled_cache[1]


Expand All @@ -268,21 +279,12 @@ def _is_credential_key(key: str) -> bool:
return k in _CREDENTIAL_EXACT or any(hint in k for hint in _CREDENTIAL_HINTS)


# What a query or form key names that is masked by its type, whatever the
# value looks like: a CVV and the rest of Sensitive Authentication Data as
# their labels, never a token, and a card number truncated. Classified as the
# key walk classifies a key, every pack on: MIGS's `vpc_CardSecurityCode`, a
# form's `card_number`, `pin`.
_CARD_TYPES = frozenset({"cvv", "sad", "card"})


def _card_field_type(key: str) -> str | None:
field_type = classify_key(key, ALL_PACKS)
return field_type if field_type in _CARD_TYPES else None


def _mask_card_field(value: str, field_type: str) -> str:
return mask_card_value(value) if field_type == "card" else mask_by_field_type(value, field_type)
# A form value: to the next `&` or whitespace, and never into an end tag. In
# `<note>password=abc</note>` the element's text ends there, as the
# credential text rules read it: read on, the value took the tag with it.
# Unrolled: a lookahead at every character of every value made a 64 KB form
# body's field pass twice as slow as 0.15.5's.
_FORM_VALUE = r"[^&\s<]*(?:<(?!/)[^&\s<]*)*"


def redact_url(url: str, *, secrets: Collection[str] | None = None) -> str:
Expand Down Expand Up @@ -386,18 +388,38 @@ def redact_body(text: str) -> str:
unquoted -- so it carries the token the same value gets under a key; one
already masked passes through.

A form value runs to the next ``&`` or whitespace. Only the structure at
its two ends -- a quote or an escaped one, and ``}``, ``]``, ``,``, a
JSON key's ``":``, ``>`` or ``/>`` -- stays as written; the rest is the
value, unescaped first only when it sits between escaped quotes in a
JSON string. A value, JSON or form, that holds a card-number run is the
label. A form field whose key the key rules call a card, CVV or other
SAD is masked by that type, whatever the credential keys: a card number
truncated, a CVV ``[CVV-MASKED]``.
A form value runs to the next ``&``, whitespace or end tag. Only the
structure at its two ends -- a quote or an escaped one, and ``}``,
``]``, ``,``, a JSON key's ``":``, ``>`` or ``/>`` -- stays as written;
the rest is the value, unescaped first only when it sits between escaped
quotes in a JSON string. A value, JSON or form, that holds a card-number
run is the label. A form field whose key the key rules call a card, CVV
or other SAD is masked by that type, whatever the credential keys: a
card number truncated, a CVV ``[CVV-MASKED]``.

An XML element is read as a key: its local name, whatever its namespace
prefix and attributes, classified as a query or form key is. A card, CVV
or SAD name masks its text by that type, as the text rule does
(``mask_card_elements``: a card object field by field); then a
credential name (as ``redact_url`` reads a param's, or one of the secret
keys) masks its text as a credential, the label when it holds a
card-number run, a CVV inside it already its label. The text runs to
the end tag, children and all, and is masked as it decodes (a CDATA
section's content, entities resolved). Any other element's text that is
JSON or XML written with entities goes through these rules as it
decodes, and is written back escaped.

A value a value rule in force matches (``ECSCTX_MASK_VALUE_RULES``), as
JSON anywhere in the body or as JSON in a JSON string, is the rule's
label first, before a credential's rule could hash it.
"""
if "{" in text and (rules := value_rules_in_force()):
text = mask_objects(text, rules)
if "<" in text:
text = _mask_xml_elements(text)
if "=" in text:
text = _FORM_FIELD.sub(_mask_card_form_field, text)
hint_re, json_re, form_re = _get_compiled()
hint_re, json_re, form_re, _names = _get_compiled()
if not hint_re.search(text):
return text
masked = form_re.sub(_mask_form_value, json_re.sub(_mask_json_value, text))
Expand All @@ -411,8 +433,69 @@ def redact_body(text: str) -> str:


# Any form field: its key as a form writes one (`card[number]` too), not
# glued to more of one, and its value up to the next `&` or whitespace.
_FORM_FIELD = re.compile(r"(?<![\w.\-\[\]%])([\w.\-\[\]%]+)=([^&\s]*)")
# glued to more of one, and its value up to the next `&`, whitespace or end
# tag.
_FORM_FIELD = re.compile(r"(?<![\w.\-\[\]%])([\w.\-\[\]%]+)=(" + _FORM_VALUE + ")")

def _is_credential_element(name: str, names: frozenset[str]) -> bool:
"""A credential's element name: one a query param's name would be read
as (`_is_credential_key`), or one of the secret body keys."""
return _is_credential_key(name) or name.lower() in names


def _mask_xml_elements(text: str) -> str:
"""Each element named as a card, CVV or SAD key is, its text masked by
that type, as the text rule masks one (`mask_card_elements`) -- first, so
a CVV inside a credential's element is its label, not part of the text
the credential's token hashes. Then each element named as a credential
key is, its text masked as a credential; and entity-encoded text in any
other leaf, through the body rules as it decodes."""
text = mask_card_elements(text)
names = _get_compiled()[3]
parts, copied, ends = [], 0, None
for start in _XML_START.finditer(text):
if start.start() < copied:
continue # inside an element already masked whole
if ends is None:
ends = _end_tags(text)
# No end tag, no element: a route (`/v1/cards/<str:token>/`) is a
# start tag's shape. Each end tag is found once (`_end_tags`).
if (closing := _closing(ends, start)) is None:
continue
end = closing.start()
if not _is_credential_element(start.group("local"), names):
if (masked := _encoded_leaf(text, start.end(), end)) is not None:
parts += [text[copied : start.end()], masked]
copied = end
continue
value = _element_text(text[start.end() : end])
masked = _SECRET_LABEL if holds_pan_run(value) else mask_secret(value)
if masked != value:
parts += [text[copied : start.end()], masked]
copied = end
if not parts:
return text
parts.append(text[copied:])
return "".join(parts)


def _encoded_leaf(text: str, start: int, end: int) -> str | None:
"""The text of a leaf, ``text[start:end]``, masked by the body rules as
it decodes and escaped again -- when it is JSON or XML written with
entities, and masking changed it. A leaf's raw text needs nothing here,
nor a form body's: the body rules read them where they stand, `&amp;` a
form's separator either way. Decoded, a form value ran on past `&quot;`,
and a leaf the first pass left as a form body read differently on the
next."""
if text.find("<", start, end) != -1 or text.find("&", start, end) == -1:
return None
decoded = html.unescape(text[start:end])
if decoded.lstrip()[:1] not in ("{", "[", "<"):
return None
masked = redact_body(decoded)
if masked == decoded:
return None
return xml_escape(masked)


def _mask_card_form_field(match: re.Match) -> str:
Expand Down
49 changes: 46 additions & 3 deletions ecsctx/contrib/ottu/masking.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
"""Key names Ottu services log that are not PII, for ``ECSCTX_MASK_SAFE_KEYS``.
"""Ottu's masking vocabulary: key names Ottu services log that are not PII
(``SAFE_KEYS``, for ``ECSCTX_MASK_SAFE_KEYS``), and the wallet tokens they
carry, masked by shape (``WALLET_RULES``, for ``ECSCTX_MASK_VALUE_RULES``).

ecsctx's own safe keys hold only names that mean the same in every service. These
are Ottu's vocabulary, where a generic key rule would mask a value that is not
Expand All @@ -12,16 +14,30 @@
so a service pinned to an older ecsctx keeps working, and because listing a key
that is already safe costs nothing.

A service opts in from its settings::
An Apple Pay or Google Pay token carries the device card number and the
payment cryptogram, encrypted: Sensitive Authentication Data, the SAD label
under any key and in any text, never hashed. Which shapes those are is Ottu's
knowledge, not ecsctx's: core names none, and masks one only as a service's
value rule (``ecsctx.masking.value_rules``). ``WALLET_RULES`` are Ottu's.

from ecsctx.contrib.ottu.masking import SAFE_KEYS as OTTU_SAFE_KEYS
A service opts in to both from its settings::

from ecsctx.contrib.ottu.masking import SAFE_KEYS as OTTU_SAFE_KEYS, WALLET_RULES

ECSCTX_MASK_SAFE_KEYS = [*OTTU_SAFE_KEYS, "x-script-name"]
ECSCTX_MASK_VALUE_RULES = [*WALLET_RULES]

or from the environment:
``ECSCTX_MASK_VALUE_RULES=ecsctx.contrib.ottu.masking.WALLET_RULES``.

Nothing installs them implicitly: ``register_ottu()`` registers events, and what a
service masks stays in its own settings.
"""

from typing import Any

from ecsctx.masking.value_rules import ValueRule

SAFE_KEYS = frozenset({
# Ottu PG API fields: a gateway's short name ("mpgs") and flags saying
# whether a card payment needs its CVV.
Expand Down Expand Up @@ -70,3 +86,30 @@
"installation_name",
"tokenization_status",
})


# Apple Pay's PKPaymentToken `paymentData`: its `version`, and the encrypted
# `data`. Its `signature` and `header` are useless without `data`, so the whole
# object is the label, and an unrelated `paymentData` reads through.
_APPLE_PAY_VERSIONS = frozenset({"EC_v1", "RSA_v1"})
# Google Pay's payment method token: its `protocolVersion` and the
# `signedMessage` holding the encrypted payload.
_GOOGLE_PAY_VERSIONS = frozenset({"ECv1", "ECv2", "ECv2SigningOnly"})


def _is_apple_pay_token(value: Any) -> bool:
version = value.get("version")
return isinstance(version, str) and version in _APPLE_PAY_VERSIONS and "data" in value


def _is_google_pay_token(value: Any) -> bool:
protocol = value.get("protocolVersion")
return isinstance(protocol, str) and protocol in _GOOGLE_PAY_VERSIONS and "signedMessage" in value


APPLE_PAY_RULE = ValueRule("sad", _is_apple_pay_token, hints=("EC_v1", "RSA_v1"))
GOOGLE_PAY_RULE = ValueRule("sad", _is_google_pay_token, hints=("ECv1", "ECv2"))
# Found by shape, never by a key name: under `paymentData`, TAP's `token_data`,
# MPGS's `paymentToken` (a JSON string), Google Pay's `tokenizationData.token`,
# KPay's `<udf9>`, or in a message.
WALLET_RULES = (APPLE_PAY_RULE, GOOGLE_PAY_RULE)
8 changes: 5 additions & 3 deletions ecsctx/events/spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,11 @@ class EventSpec:
# A `Reason` subclass, normalised to its members; a tuple of strings still
# works for a set nobody passes as an object.
reasons: _ReasonSet = ()
# Field names this event is expected to carry. Declaration only — the
# runtime validator (#159491) is what enforces them. Declared here so
# EventSpec is not reopened for that ticket.
# Field names this event is expected to carry. Declaration only: nothing
# enforces them. The contract validator (`event_contract`, #159491) checks
# an event's action, its outcome, the level it is logged at and `labels`,
# not these fields; they are rendered into the catalogue's reference page
# ("Carries") for the developer and the reviewer.
required: tuple[str, ...] = ()
optional: tuple[str, ...] = ()
# Connect's LogEvent carried only `level`, described as "the level of the
Expand Down
11 changes: 9 additions & 2 deletions ecsctx/masking/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,18 @@
See ecsctx.masking.patterns for the rules and packs, ecsctx.masking.config
for choosing packs (configure_masking_packs, ECSCTX_MASKING_PACKS), a
service's own safe key names (configure_masking_safe_keys,
ECSCTX_MASK_SAFE_KEYS) and skip paths, and ecsctx.masking.exemptions for the configure_masking()
path-exemption API.
ECSCTX_MASK_SAFE_KEYS), its value rules (configure_masking_value_rules,
ECSCTX_MASK_VALUE_RULES; ecsctx.masking.value_rules) and skip paths, and
ecsctx.masking.exemptions for the configure_masking() path-exemption API.
"""

from ecsctx.masking.config import (
configure_masking_packs,
configure_masking_safe_keys,
configure_masking_value_rules,
get_masking_packs,
get_masking_safe_keys,
get_masking_value_rules,
key_field_type,
)
from ecsctx.masking.exemptions import (
Expand All @@ -31,6 +34,7 @@
from ecsctx.masking.install import install_maskers, uninstall_maskers
from ecsctx.masking.patterns import ALL_PACKS, PACK_NAMES, mask_card_value, mask_secret
from ecsctx.masking.tokens import mask_by_field_type, safe_tokenize
from ecsctx.masking.value_rules import ValueRule

__all__ = [
"ALL_PACKS",
Expand All @@ -39,6 +43,9 @@
"get_masking_packs",
"configure_masking_safe_keys",
"get_masking_safe_keys",
"configure_masking_value_rules",
"get_masking_value_rules",
"ValueRule",
"MaskPIIFilter",
"install_maskers",
"uninstall_maskers",
Expand Down
Loading
Loading