Skip to content

feat(mcp): reduce PII in $mcp_intent (redaction + stronger prompt) - #4783

Open
gesh wants to merge 6 commits into
mainfrom
posthog/mcp-intent-pii-redaction
Open

feat(mcp): reduce PII in $mcp_intent (redaction + stronger prompt)#4783
gesh wants to merge 6 commits into
mainfrom
posthog/mcp-intent-pii-redaction

Conversation

@gesh

@gesh gesh commented Sep 4, 2026

Copy link
Copy Markdown
Member

Problem

$mcp_intent is free text the calling agent writes into the injected context argument. Agents sometimes narrate personal data (emails, phone numbers, names) into it, and the SDK previously stored it almost verbatim — the only cleanup removed PostHog tokens and large binary blobs, and the "no personal data" instruction was prompt-only, not enforced.

Changes

Two layers, both scoped to @posthog/mcp:

  • Always-on redaction (redactPii in mcp-payloads.ts, applied to userIntent in sanitization.ts): strips structured identifiers — emails, phone numbers, IPv4/IPv6, Luhn-valid card numbers, US SSNs — from $mcp_intent before capture. Covers both the instrument() and PostHogMCP paths via the shared sanitizeEvent. Scoped to the intent only; structured tool arguments/results are untouched, since the same shapes are often legitimate there.
  • Stronger default context prompt (DEFAULT_CONTEXT_PARAMETER_DESCRIPTION): now forbids repeating/paraphrasing/inferring personal data and tells the agent to generalize entities to roles ("a user", "the customer") — covering names, which regex cannot. Tests reference the constant symbolically, so no assertions changed.

Unit tests added for both. All 694 @posthog/mcp unit tests pass; oxlint clean.

Libraries affected

  • @posthog/mcp (not in the list below)

Checklist

  • Tests for new code
  • Accounted for the impact of any changes across different platforms
  • Accounted for backwards compatibility (no API break; redaction changes only the captured $mcp_intent content)
  • Took care not to unnecessarily increase the bundle size (a few regexes + one function)
  • Ran pnpm changeset to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Built with PostHog Desktop (Claude Opus). The human driver chose the scope (redaction limited to the intent, always-on, no config flag), the structured-identifier pattern set, and the final wording of the default prompt. Names/addresses were deliberately left to the prompt layer rather than regex, to avoid over-redacting ordinary prose. tsc/rslib build OOM locally in this environment (pre-existing, unrelated to the diff); verified via the vitest suite and oxlint instead.


Created with PostHog Desktop

gesh added 2 commits September 4, 2026 12:05
$mcp_intent is agent-narrated free text the calling LLM writes into the
injected `context` argument, so it could carry personal data a model read
aloud despite the "no personal data" instruction. Add an always-on redactor
that strips well-defined structured identifiers — emails, phone numbers,
IPv4/IPv6 addresses, Luhn-valid card numbers, and US SSNs — from the intent
before capture.

Scoped to the intent only: structured tool arguments and responses keep the
same shapes since they are often legitimate data there. Best-effort for those
patterns rather than free-form names or addresses, which regex cannot catch
without over-redacting prose. `context: false` and `beforeSend` remain the
ways to drop the field entirely.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472
Rewrite DEFAULT_CONTEXT_PARAMETER_DESCRIPTION so agents are less likely to
write personal data into the injected `context` argument that becomes
$mcp_intent. The privacy rule is now explicit and mandatory, names the
identifiers to avoid (real names, emails, phone numbers, IPs, IDs,
credentials, secrets, copied user/tool text), and tells the agent to refer
to people and accounts by role ('a user', 'the customer') rather than by
identity. The example now models that role-based phrasing.

Pairs with the always-on intent PII redaction as a second, prompt-level
layer. Tests reference the constant symbolically, so no assertions change.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472
@gesh gesh self-assigned this Sep 4, 2026
@gesh
gesh marked this pull request as ready for review September 4, 2026 09:35
@gesh
gesh requested review from a team as code owners September 4, 2026 09:35
@greptile-apps

greptile-apps Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Security Review

The phone-number redaction can leave personal data in $mcp_intent: slash-separated numbers are not matched, and long formatted numbers can be only partially replaced.

Prompt To Fix All With AI
### Issue 1
packages/mcp/src/extensions/mcp-payloads.ts:42
**Phone numbers remain exposed**

Common phone formats are not fully covered by this pattern. For example, `415/555/0142` is left unchanged because `/` is not accepted, while a long formatted number such as `+44 (0) 20 7946 0958` can be only partly replaced, leaving trailing digits in `$mcp_intent`. Expand the candidate handling and ensure that a match consumes the entire phone token.

> **How this was verified:** The shared sanitization path applies this exact regex to every captured intent, and neither `/` nor characters beyond its fixed span can be included in the replacement.

### Issue 2
packages/mcp/src/extensions/mcp-payloads.ts:42
**Timestamps resemble phone numbers**

The permissive phone detector also matches ordinary date-and-time text. For example, `2024-01-15 12:30 UTC` has ten digits and accepted grouping characters, so its `2024-01-15 12` prefix is replaced with `[redacted]`. This corrupts otherwise valid, non-PII intent text; add validation that distinguishes phone syntax from timestamps.

### Issue 3
packages/mcp/src/__tests__/mcp-payloads.test.ts:69-135
**Redaction tests repeat scaffolding**

These new cases repeatedly call `redactPii(input)` and compare one expected string. This violates the repository directive to prefer parameterised tests. Convert the equivalent cases to a table-driven `it.each` test while keeping separately named tests for genuinely distinct behavior; this repository requirement must be satisfied before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "feat(mcp): strengthen default context pr..." | Re-trigger Greptile

Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
Comment thread packages/mcp/src/__tests__/mcp-payloads.test.ts
Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
@veria-ai

veria-ai Bot commented Sep 4, 2026

Copy link
Copy Markdown

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

Review of the redactor surfaced real gaps, now fixed:

- ReDoS: the email pattern used open-ended `+` quantifiers, quadratic on a
  long local-part run with no valid TLD. `$mcp_intent` is attacker-influenceable
  free text seen before truncation, so a pathological intent could stall the
  event loop (~13s at 80k chars). Bound the quantifiers to RFC-ish limits;
  80k chars now runs in ~26ms.
- Under-redaction: normalize horizontal Unicode spaces (NBSP etc.) so
  copy-pasted identifiers still match; accept dot/slash-grouped card numbers
  (Luhn-gated, so no new false positives) which previously leaked or produced a
  garbled partial redaction; accept space/dot-separated SSNs. A bare 9-digit
  number is still not treated as an SSN, to avoid eating numeric IDs.

Adds tests for each. Full @posthog/mcp suite (699) and oxlint pass.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472
@gesh

gesh commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Note

🤖 Automated comment by QA Swarm — not written by a human

Multi-perspective review across two rounds. Findings verified with deterministic reproductions, then fixed.

Verdict: 💬 APPROVE WITH NITS (round 2 @ a764145)

Redaction is correctly scoped to $mcp_intent. Two review rounds found a safety bug, several under-redaction gaps, and (from human + bot review) precision bugs in the IPv6 and phone patterns. All were reproduced and fixed.

Fixed

  • 🟠 ReDoS in the email pattern (O(n²), reachable pre-truncation) → bounded quantifiers (~13s → ~26ms at 80k chars).
  • 🟡 Under-redaction → Unicode (NBSP) spaces normalized; dot/slash-grouped cards (Luhn-gated); space/dot-separated SSNs.
  • 🟠 IPv6 (from @marandaneto) → 2001:db8:: (trailing ::) leaked and std::bad was over-redacted. Reworked into four bounded forms with a (?<![\w:]) boundary: trailing/middle/leading :: redact; hex-looking C++ scope is left intact.
  • 🟡 Phone (from @marandaneto + greptile) → the loose "any 10-15 digits" heuristic over-redacted dates/versions, missed /-grouped numbers, and partially leaked long international numbers. Replaced with structural patterns (NANP 3-3-4 requiring a separator; international requiring a leading +). Dates, versions, bare digit runs, and order IDs are now left intact.
  • ✅ Tests converted to table-driven it.each per the suite's style (greptile).

Remaining best-effort limitations (by design)

  • Non-punycode IDN email domains, non-+ international phone formats, and bare un-punctuated numbers are not caught. Names and postal addresses cannot be matched by regex at all — the injected context prompt is the primary defense; redaction is the backstop.

Reviewers

Reviewer Assessment
🧭 router (sonnet) + 🔎 opus ReDoS + under-redaction gaps, all fixed.
👤 @marandaneto (human) IPv6 :: leak/over-match + phone-as-date — all fixed & resolved.
🤖 greptile / veria Phone precision, test style, email ReDoS — all addressed.

Automated by QA Swarm — not a human review

Fold the near-identical card/phone `replace` callbacks into one
`redactConfirmedCandidate` helper (strip digits, range-check, confirm). The
phone grouping check reuses the digit-count instead of a second regex scan,
and the digit-strip regex is hoisted to a module constant. Pure refactor;
all 699 tests and oxlint pass unchanged.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472
@gesh gesh added the stamphog label Sep 4, 2026

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated advisory code review.

Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
@marandaneto
marandaneto requested a review from a team September 4, 2026 10:23
From PR review (marandaneto, greptile):

- IPv6: `2001:db8::` (trailing `::`) leaked, and hex-looking C++ scope like
  `std::bad` was over-redacted. Rework the pattern into four bounded forms with
  a `(?<![\w:])` boundary on the compressed branches: trailing/middle/leading
  `::` now redact, while `std::bad` (left side is not a valid hex group) is left
  intact.
- Phone: replace the "any 10-15 digits with a separator" heuristic — which
  over-redacted dates (`2024-01-15 12:30`) and dotted versions, missed
  slash-grouped numbers, and left a trailing digit on long international numbers
  — with two structural patterns: a NANP 3-3-4 grouping requiring a real
  separator, and an international number that must start with `+` and a country
  code. Dates, versions, bare digit runs, and order IDs are now left intact.

Convert the redactPii cases to table-driven `it.each` per the suite's style.
Full @posthog/mcp suite (711) and oxlint pass.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472
Comment thread packages/mcp/src/extensions/mcp-payloads.ts Outdated
Review follow-up (veria-ai): the structural NANP pattern required a separator
after the area code, so `(415)555-0142` (parens, no following separator) leaked.
Allow the parenthesized `(415)` form to omit the separator while the bare `415`
form still requires one — so bare digit runs are still never taken for phones.
Test added; full @posthog/mcp suite (712) and oxlint pass.

Generated-By: PostHog Desktop
Task-Id: fc71ed52-e03a-4f5b-a863-acf3b23aa472

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left one last comment otherwise lgtm

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated advisory code review.

// the digits decides — a non-card digit run of the same length is left intact.
result = result.replace(CREDIT_CARD_CANDIDATE_PATTERN, (match) => {
const digits = match.replace(/[ ./-]/g, '')
return digits.length >= 13 && digits.length <= 19 && passesLuhn(digits) ? REDACTED_VALUE : match

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: Adjacent expiry digits prevent card redaction — For Charging card 4111 1111 1111 1111 12/30 for renewal., the greedy candidate includes the expiry month. The combined number fails Luhn, so the entire match is returned unchanged, exposing the complete valid card in $mcp_intent. Recognize the valid card without absorbing adjacent numeric fields. Reproduction: reproduced — the focused pi-review-card.test.ts Vitest regression failed on this head with the complete card retained; the same card without expiry passed, and a temporary 16-digit candidate control made both cases pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants