PII scrubbing: safe additive custom regex controls

[unjica]

Roadmap priority

9

Goal

Provide safe, additive custom regex controls for PII scrubbing without allowing projects or organizations to disable mandatory privacy protections.

Custom regex patterns should extend existing scrubbing capabilities rather than replace them.

Current state

• Built-in PII scrubbing shipped in PII scrubbing for ingest and stored telemetry #470 (default ingest scrubbing, SDK opt-in, project deny-keys, phone/card heuristics, audit logs, session email scrubbing, and historical backfill).
• Organization-level PII defaults are tracked in PII scrubbing: organization-level defaults with project overrides #477.
• Projects currently support deny-listed keys but cannot define domain-specific string patterns.
• There is currently no mechanism for safely defining additional custom regex-based scrubbing rules.

Motivation

Built-in heuristics cover common emails, tokens, phones, and payment card patterns. Deny-keys cover known property names.

Some teams need domain-specific string matching that is difficult to express as field names alone, for example:

Internal employee identifiers
National identification numbers
Customer-specific account formats
Legacy application identifiers

Custom regex controls should make these use cases possible while preserving mandatory privacy protections.

Deliverables

Custom regex controls

• Project-level custom regex patterns
• Safe validated storage and APIs (new endpoint or extension of existing PII settings)
• Stable replacement placeholders
• Dashboard UI clearly separating field-name deny-keys from regex patterns
• Documentation and privacy copy updates (including false-positive risk and testing guidance)

Safety

• Pattern count limits
• Pattern length limits
• Compile-cost validation
• ReDoS protections
• Reject unsafe regex flags where applicable
• Stable placeholder behavior across scrub operations

Quality

• Tests for placeholder stability
• Tests for merge behavior with existing PII defaults
• Tests for validation and ReDoS protections
• Optional SDK parity or clearly documented server-only behavior

Merge semantics

Custom regex patterns are additive only.

Built-in defaults
        +
Organization defaults (#477)
        +
Project deny-keys
        +
Custom regex patterns
        ↓
Effective scrubbed output

Custom regex patterns must never:

• disable mandatory built-in protections
• replace existing built-in patterns
• silently remove supported organization or project defaults

Feature-specific merge rules defined by #477 remain compatible with this model.

Technical notes

• Prefer extending existing PII settings models rather than introducing parallel configuration systems.
• Invalid custom patterns must fail validation rather than silently disabling scrubbing behavior.
• Prefer omission over partial scrubbing failures when safety guarantees cannot be maintained.
• Custom regex patterns should be evaluated after built-in protections and deny-keys.
• Organization-level inheritance of custom regex patterns is intentionally deferred to future roadmap work.
• Existing RBAC and authorization rules remain applicable.

Related

• PII scrubbing for ingest and stored telemetry #470 — Project-scoped PII scrubbing (done)
• PII scrubbing: organization-level defaults with project overrides #477 — Organization-level PII defaults
• Organization settings: defaults layer with project overrides #497 — Organization-wide defaults model
• Companion follow-up issues for organization-level inheritance of custom regex (if introduced later)

Out of scope

• Organization-level inheritance of custom regex patterns
• Automatic replacement of built-in PII protections
• Disabling mandatory built-in scrubbing behavior
• Automatic historical backfill when custom patterns change
• RBAC redesign or permissions changes
• Cross-organization sharing of custom regex configurations
• Changing the default built-in pattern set without an explicit product decision

Base branch

develop

Suggested acceptance checks

1. Custom regex patterns are validated safely before storage.
2. Built-in privacy protections remain enabled regardless of custom configuration.
3. Custom regex patterns are applied additively alongside existing deny-keys and organization defaults.
4. Invalid or unsafe patterns are rejected appropriately.
5. Dashboard UI clearly distinguishes deny-keys from regex-based rules.
6. Tests cover merge behavior, validation, ReDoS protections, and placeholder stability.
7. Existing project-scoped PII functionality continues to work when no custom regex patterns are configured.
8. Documentation describes the relationship between built-in protections, organization defaults, project settings, and custom regex controls.

[unjica]

9:Maps to roadmap Custom PII Regexes (priority 9) — no duplicate created.