Repository navigation
0.85.0: expanded redaction rules, saved keys, two-phase scan - #382
Merged
Merged
Conversation
Kinds split into service and personal-information groups (rule view sorts by kind), new matcher shapes, per-track scan hooks and shared helpers. No new rules yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the personal-information track of the expanded redaction catalog: - ID documents: 15-digit resident IDs, passport MRZ (TD3, all five check digits), Chinese passport numbers, Hong Kong / Macao / Taiwan ID cards, mainland travel permits. The 18-digit resident ID rule now also accepts Taiwan residence permits (83) and the 2023 foreign permanent resident card (9 + province + ISO 3166 nationality). - Foreign IDs: US SSN, UK NINO, Singapore NRIC/FIN, Korean RRN, Japanese My Number, Aadhaar (Verhoeff), Brazilian CPF. - IBAN (country length, BBAN structure, MOD 97-10), landlines, international numbers, and +86 / 0086 / grouped mobile numbers. - Street addresses (division table + road + door number), coordinate pairs, licence plates, VINs, IMEI, MAC addresses, public IPs. - Five field-name rules (name, address, birth date, account, health); bare `name` / `address` count only next to another personal field. - Internal IP adds 100.64.0.0/10 and fc00::/7; internal domains add .corp, .home.arpa, .intranet. Numbers are matched only inside JSON strings. Context-only formats require a keyword within 20 characters. Only passport-mrz and iban ship switched on. Lookup tables are embedded text files (about 48 KB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Credential rules (all on by default):
- About fifty prefix-family rules (AI services, dev platforms, cloud,
business) as Matcher::Prefixes, served by a first-byte prefix table
on the token fast path. Other providers' `sk-` prefixes are no longer
reported as `openai-api-key`.
- Structural rules: OpenAI-compatible `sk-` keys, Zhipu, Qianfan,
Azure AD client secrets, Terraform, Telegram bot tokens.
- Position rules that replace only the value: credentials in request
headers, command-line passwords, URL token parameters, chat bot
webhooks, and secret assignments in config files.
- Crypto: BIP39 recovery phrases (English and Simplified Chinese
wordlists, checksum verified), WIF keys and extended private keys via
Base58Check.
Saved secrets: `thinkwatch-saved-secret` matches values saved in the
config (upstream keys and secret headers with ${VAR} resolved, access
keys, proxy passwords, control key, sync credentials) by value, rebuilt
with the runtime on reload. Findings, security events and trial hits
carry a `source` (SavedSource) and no value; the security log stores it
(schema 28). Credential-to-network treats saved secrets as credentials
and checks them against their own hosts. Trial gains `run_with` taking
the gateway's saved set. Control API version 56.
Adds an example that measures false positives and scan time over
source trees.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…redact-rules-expand
Built-in redaction rules now live in crates/tw-guard/data/redact.yaml: id, kind, default switch, label, provider hosts, English and Chinese name and description, and the prefix shapes. build.rs turns it into the same `&'static [Builtin]` constant as before, so the compile-time KEY_MIN / KEY_FIRST tables and every call site stay as they are, and the generated catalog is identical in order, kinds, labels and matchers. Prefix rules get their descriptions generated from their prefixes; adding one touches only the YAML. Rules whose check is in code are looked up by id in `rules::DETECTORS` at compile time, and a test checks the other direction. Provider hosts for the credential-to-network check moved from tools/net.rs into the catalog. The tool-call and content rule files gain `name_zh` / `why_zh` and their group names. Views: `SecurityRuleView` gains `name_zh`, `why_zh`, `about`, `about_zh`; `GuardDetail` gains `kinds` (slug, name, name_zh) in display order, ending with `custom`. The UIs can drop their own rule name and description tables. The zh config manual now uses the Chinese names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every built-in detector now declares anchors next to its implementation (redact/anchor.rs): literal prefixes and keywords, first/last byte and length gates, two-letter suffixes, required/forbidden character classes, and an optional look at the bytes around the token. Prefix-family anchors are derived from each rule's Matcher. Phase 1 (engine::prescan) walks the text once, tokenising exactly like before. Each token is checked against tables compiled from the anchors; literal anchors go through one anchored Aho-Corasick DFA stepped from the token start. The same pass counts runs: streaks of BIP39-shaped words (only streaks with 12+ lowercase words are checked against a 4-letter prefix bitmap of the wordlist), and runs of non-ASCII bytes / \uXXXX escapes for the Chinese wordlist (only streaks of 12+ close runs are decoded). Phase 2 runs the existing validators only on candidates, in the original per-token order; multi-token detectors (grouped card numbers, phone groups, IBAN, MAC, coordinates, plates) treat skipped tokens as the breaks they were before, including flush timing relative to placed-value filtering. Custom rules: "contains" rules join one Aho-Corasick automaton (ASCII case folding only for patterns without k/s, which regex folds to non-ASCII letters); regex rules form one RegexSet per rule set, consulted once per JSON string so only matching rules run find_iter. Rules with Unicode word boundaries stay on the per-rule path. Matching semantics are unchanged. Also: json_string_spans uses memchr, labeled-field separators find the full-width colon separately, the address scan uses the packed (Teddy) searcher, and string spans collected while tokenising are reused. The previous scanner is kept as a test-only oracle (rules/oracle.rs) with a differential test over every rule sample, this repository's sources and randomized bodies (JSON escapes, \n line starts, nested JSON, unquoted numbers, \uXXXX bodies), for defaults, all rules, single rules and custom rules. Mutations of the engine's break/flush/streak logic are caught. examples/redact_corpus gains RX_SET=defaults|all|custom. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/redact-rules-expand
`secrets::Want::finish` dropped a position-rule hit (auth header, URL token, command-line password, config assignment) whenever any other hit overlapped it. A record-only rule (e.g. `jwt` set to record) then let the whole Bearer token go out raw, and a partial match inside the value (a phone number inside a token, an email inside a password) replaced only that slice. Now the placed hit yields only to hits that replace and together cover it entirely; otherwise it stays and `resolve` merges the overlap so the whole value is replaced. The oracle shares `secrets::Want`, so it changes identically; only the call site gains the rule set argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each header, URL-parameter, env-password, assignment and webhook marker scanned the rest of the text for its value, and each curl/mysql marker looked for the end of its command over the whole remainder. A run of repeated markers with no separator (`api-key=api-key=…`) was quadratic: 100 KB took 2 s, 400 KB 39 s. - A value, command or webhook path is looked at for at most 4 KB. The longest real credentials placed there (Azure AD / Google ID JWTs, signed-URL signatures) are two to three KB; anything longer is data. - Each kind remembers how far it has already looked; a marker inside a span already scanned is skipped, so every byte is scanned once per kind. - Option values inside a command are bounded by the command end. A ratio-based timing test (4x the input must cost < 10x the time) covers every marker kind. The oracle shares these functions and visits marker tokens in the same order, so its results change identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`value_after` looked for the closing `\"` of an escaped-quoted value without pairing backslashes, so `DOB: "1990-01-02\` ended the value between the two bytes of `\\` and replacing it broke the request body. Its quote searches also ran past the end of the containing JSON string and could take the value from the next string, swallowing the structure in between. Backslashes are now skipped in pairs, and every search stops at the end of the JSON string that holds the separator. Shared with the oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`…/services/T…/B…/<token>.`, `…/<token>/` and `…/<token>, ok` were not replaced: the last path segment had to be all letters and digits up to the end of the path. Trailing punctuation and slashes are now trimmed and the token is the leading alphanumeric run of the last segment. This intentionally changes what the shared rule finds, so the oracle (which calls the same function) changes identically; the new cases are added to the rule samples that both scanners are compared on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PHP's `json_encode` writes `/` as `\/` by default, which is valid JSON. A request body produced that way slipped past every rule that matched a literal slash: Qianfan `bce-v3\/ALTAK-…\/<hex>` and the Slack, Feishu, DingTalk, WeCom, Discord and Teams webhook paths. `/` is now matched as either `/` or `\/` wherever these rules expect one (`slash_at` / `path_at`), and a webhook path may contain `\/`. The replaced span keeps the escapes. This intentionally widens what the shared rule code finds, so the oracle changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… spaces
- `Authorization: Bearer<TAB><token>` (a raw tab or the `\t` escape in a
request body) was not taken as a header credential: only a space was
accepted after the scheme word.
- `.env`-style assignments (`NAME=value`, no blanks around `=`) missed
unquoted values containing `&,;()[]{}<>` and quoted values containing
spaces. In that form a quoted value now runs to the closing quote on
the same line, and an unquoted one to whitespace or end of line with
those characters included. To stay quiet on code, the wider reading is
not used when brackets do not balance (`f(token=abc)`), the value starts
with a bracket, or it looks like a call (`make_token(1)`); a following
`&name=` is the next form parameter and ends the value. Other forms
(`key: value`, `"key": "value"`, `key = value`) are unchanged.
These change what the shared rule code finds, so the oracle changes
identically; the new cases are added to the samples both scanners are
compared on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saved secrets were matched only on token boundaries, and `.`, `-`, `_` are token bytes, so the exact saved value went out raw in `My key is <value>.`, `MISTRAL_KEY_<value>` or `<value>_old`. The same lookup feeds credential-to-network, which missed it too. A saved value is matched whole, so it can only be part of something longer when an ASCII letter or digit touches it. That is now the only neighbour that rejects a match (a JSON escape before it still counts as a break). The oracle calls the same lookup and changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fan keys The `sk-` + 32 / 48 key shape allowed every `*.aliyuncs.com` host, where anyone can create a writable OSS bucket or a Function Compute endpoint; a key PUT into such a bucket was not cut. It now allows only the API hosts of the services that issue this shape: api.deepseek.com, the DashScope hosts (dashscope, dashscope-intl, dashscope-us .aliyuncs.com) and api.siliconflow.cn / .com. Relay keys of the same shape are allowed to their own upstream host through saved secrets. The Qianfan `bce-v3/ALTAK-` key likewise allowed all of `baidubce.com`, which also hosts user cloud functions; it is now qianfan.baidubce.com. The catalog header documents the rule and lists the remaining entries that still allow a shared domain because the credential legitimately goes to many products or to the user's own subdomain there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`tw-config`'s `host_of("http://[fd00::1]:8080/v1")` returned `[fd00`, so
a key saved for an upstream at an IPv6 literal was treated as sent to a
stranger when it went to its own upstream (the tool check extracts
`fd00::1`). The bracket-aware parser from the tool check is now public
as `tw_guard::tools::net::url_host` and `tw-config` uses it for every
host it derives (upstreams, OAuth endpoints, proxies, sync targets).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a saved secret overlapped a built-in credential match, only the saved hosts were allowed, so saving an `sk-ant-…` key for api.anthropic.com made sending it to console.anthropic.com a cut. A saved secret now allows the union of its own hosts and the hosts of the built-in rules that also matched it; it is still reported as the saved secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dress Access keys and the control key had no allowed hosts, so only loopback was accepted and calling the gateway at its own LAN address (a client on another machine, a remote core) was cut as exfiltration. Their allowed hosts are now the addresses the gateway itself listens on beyond loopback: the gateway bind and, when enabled, the remote control bind (all interface addresses for `all`, the resolved address for a NIC name or a fixed address). Nothing else is widened; the config has no hostname setting, so only addresses are added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rted `more_found` deduplicated plugin-added findings by (rule, masked), and saved secrets always mask to "". With upstream A's key in the client's text, a plugin writing upstream B's key into the request was taken as already reported and never logged. The key now includes which saved secret it is, as `Reported` already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every rule sample is also scanned after a random mutation (case flips, an escape or other character inserted inside an anchor, padding or truncating to length boundaries, adjacent copies, non-ASCII neighbours, a dropped character), plain and inside one or two layers of JSON. The two-phase scanner must agree with the reference on every one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the previous round found three regressions in the linear-time change: - A value longer than 4 KB was given up on and sent whole (header values, URL tokens, `"api_key"`, `DB_PASSWORD=`; with `jwt` set to record-only a 6 KB bearer JWT went out raw). - `command_end` looked only at the first 4 KB of a command, so `curl … -d '<5 KB>' -u admin:<password>` leaked the password. - When the widened `.env` reading failed it still advanced the assignment's "already scanned" position to the end of its run, so a later name in that run was skipped (`login(token=tok,password=…)` leaked the password). The caps are gone: values and commands are read to their real end and replaced whole. Linearity rests on the per-kind progress alone, which already skips markers inside a span that was read. The `.env` scan keeps its own progress, advanced whether or not it matched, and the assignment's progress moves only when that scan succeeds. The timing test gains the failing-`.env` and unterminated-quote cases. Shared with the oracle, which changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`…/services/…/A,https://hooks.slack.com/services/…/B` was read as one path because a comma did not end it; only B was replaced and the second marker fell inside the scanned span. A webhook path now ends at a comma (webhook paths never contain one), so each address is read on its own. Shared with the oracle, which changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Authorization:<TAB><token>` (a raw tab or the `\t` escape in a request body) was missed because only spaces were skipped after the colon; the same blank rule as after the auth scheme now applies. `Proxy-Authorization` is read like `Authorization`, scheme word included. Shared with the oracle, which changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Go's `encoding/json` writes `&` as `&` by default, so `…?state=1&access_token=<token>` in a request body was not taken as a URL token. A parameter name may now follow `&` and be followed by `=`, and the config-assignment rule treats `&name=` as a URL parameter like `&name=`. Shared with the oracle, which changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es inside the saved key The host union added for saved secrets applied to any overlapping built-in match. A relay key saved for its own host, sent with `sb_secret_` or `xoxe-` glued in front, overlapped a Supabase or Slack match and so was allowed to an attacker's Supabase project or Slack webhook. The built-in rule's hosts are now added only when its match lies entirely within the saved value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The widened `NAME=value` reading took code for literal values: `TOKEN=abc123;fi` swallowed `;fi`, and `token=cfg[key2]`, `password=arr[i+1]`, `secret=Map<String,V2>` were replaced. A `;` now ends the value when it closes the run or is followed by a shell keyword (`fi`, `done`, `then`, …), and a value that starts with an identifier followed by `[` or `<` (an index expression or a generic type) is not a literal, like one followed by `(` already was. Shared with the oracle, which changes identically; the cases are added to the rule samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Leaked or mangled forms found in review: a failed `.env` scan hiding the next assignment, values over 4 KB, a curl password after a long `-d`, `&` / `=` in URL parameters, comma-separated Slack webhooks, header spellings and blanks, a 6 KB JWT with `jwt` record-only, and a random marker soup that must stay valid JSON and restore exactly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two-phase scanner was ~80% slower than the old one on prose-like bodies. Profiling showed three causes: - Mnemonic runs were counted by shape only, then every run of 12+ lowercase short words was tokenized a second time to check each word. In English prose nearly every run qualifies, so most of the body was tokenized twice. Membership is now decided inline from the token's character classes (two bitset lookups for lowercase words), and the second pass is gone. - First/last-byte tables and the length table were looked up separately and ANDed, so a group with several anchors (e.g. "ends in m, 18+ chars" plus "ends in Q, any length") matched every word ending in m. The byte tables are now indexed by byte x length bucket, which is exact per anchor. - A bare "token" compared itself against ~20 longer non-secret suffixes byte by byte; a length check now skips them. Also: the non-ASCII run skip reads 8 bytes at a time, the JWT anchor checks segment lengths before handing a token to the decoder, and the BIP39 four-letter prefix table is built at compile time. Results are unchanged (oracle and all tests pass). The redact_corpus example gains chat-en and chat-zh corpora (synthetic, ~1 MiB each). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The quoted `.env` value loop called `quote_at` at every other byte of a backslash run, and `quote_at` walks the rest of the run each time. With the length cap gone that was quadratic: `TOKEN="` followed by a long run of backslashes took 9.3 s for a 400 KB body. Each run is now counted once; what follows it decides whether it closes the value (n - 1 backslashes before the quote, pairs before that), ends the line (an odd run before `n`/`r`), or escapes one character. Same results as before, covered by a new ratio-based timing test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gain Two false-positive guards from the previous round let passwords out: - An identifier followed by `[` or `<` was taken for an index or a generic type, so `S3cret[2024]Xk9mQ2pLz7wR`, `Xk9mQ2pLz7wR<3>` and `Xk9mQ2pLz7wR[ab]` were not replaced. That exclusion now applies only when the leading identifier does not itself look like a credential; when unsure, replace. - A `;` followed by anything starting with a shell keyword ended the value, so `…;then#Q2pL` and `ab;do!Xk9mQ2pLz7wR` leaked the rest. The keyword must now be followed by the end of the value or another `;`. Shared with the oracle, which changes identically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments After `?`, `&` or the escaped `&`, config-assignment stepped aside for every name, but url-token only knows a fixed list (`access_token`, `api_key`, …). `…?a=1&password=<value>` (and the raw `&password=`, which already leaked) was caught by nobody. Config-assignment now yields only for a name in url-token's list while that rule is on; any other name is read as an assignment whose value ends at the next `&` or `&`. Shared with the oracle, which changes identically. Adds the third review's regression tests for these and the previous two commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Version 0.85.0 with release notes for the expanded redaction rules. README describes what outbound redaction recognises now. The data tables derived from libphonenumber and iban.js name their licences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The BIP39 table is built at compile time by splitting on \n; a CRLF checkout left \r in each word and the const evaluation overflowed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate drops the connection once it reads a wrong token, so the test's second frame can meet a closed socket. That is the behaviour under test, not a failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outbound redaction grows from 30 built-in rules to 120, and the scan is reworked so the extra rules cost little. Version 0.85.0; full notes in
release-notes/0.85.0.md.thinkwatch-saved-secretmatches the credentials kept in the configuration (upstream keys, OAuth tokens, AWS keys, credential headers, access keys, proxy passwords, control key, sync credentials), rebuilt on reload.SavedSourceonSecretItem,SecurityEventViewandTrialHit;trial::run_withtakes the saved list; the credential-to-network tool rule knows each saved key's hosts.sk-keys of other services, provider hosts per rule.data/redact.yamlcompiled in bybuild.rs; rule views carry Chinese names and descriptions (name_zh,why_zh,about,about_zh) andGuardDetail.kinds; the tool and content catalogs gain Chinese fields too.RegexSet. The previous scanner is kept as a test oracle with differential and randomized tests.Upgrade: protocol 55 -> 56, request store schema 27 -> 28 (request records and security log rebuilt), redaction kinds regrouped (
api-keysandpersonalremoved), configuration unchanged, no new message codes.Three rounds of adversarial review were applied (leaks through overlapping hits, quadratic position rules, JSON-escape handling, host lists, saved-key boundaries); each finding has a regression test. Release-mode scan time on a 4 MB body with default rules: +4% on source code, +11% on English prose, faster on Chinese chat, versus 0.84.0.
🤖 Generated with Claude Code