diff --git a/.agents/notes/implemented/feature/2026-11-20-gateway-privacy-masking.md b/.agents/notes/implemented/feature/2026-11-20-gateway-privacy-masking.md new file mode 100644 index 000000000..158094b02 --- /dev/null +++ b/.agents/notes/implemented/feature/2026-11-20-gateway-privacy-masking.md @@ -0,0 +1,33 @@ +# Agent Note: 网关推理正文的可逆隐私脱敏 + +Status: implemented + +## Problem + +Agent 发给模型的上下文可能包含工具读到的密钥、令牌和个人信息。已有指纹脱敏负责改写客户端特征,不保留逆向映射,不能在模型复述被遮盖的工具参数时还原原值。见 [Issue #634](https://github.com/aimod-cc/Agent2API/issues/634);界面术语参考 [magpie 的本地脱敏文档](https://usemagpie.ai/docs/zh/masking)。 + +## Decision + +默认不改变网关行为。管理员在「设置 → 安全 → 本地隐私脱敏」分别启用凭据、个人信息分类与自定义词;管理面用受鉴权保护的 `GET/PUT /api/privacy-masking` 热更新整份设置。设置写入本机数据库,不参加云同步。失败时保留先前配置。GET 在同一快照中返回设置及内容 revision,PUT 必须携带该 revision 并在既有配置更新锁内比较后写入;旧页面保存返回 409,不覆盖其他页面刚启用的保护。revision 不写入配置正文,也不扩展云同步范围。保存提示在草稿修改后清除,刷新经既有应用确认框确认才丢弃草稿,冲突时保留编辑内容。 + +脱敏位于三个外部推理端点的鉴权、容量限制之后,以及任何提供商选路、协议翻译、重试和正文观测之前。只访问对应协议已知的文本、工具参数、描述和工具结果。工具描述包括命名空间子工具及 `additional_tools`、`tool_search_output` 中的声明;沿已知声明结构递归,只改写说明性文本,不改工具名、枚举、必填字段或自定义语法。工具参数与结果中的 JSON 文档按值识别凭据键名,保留变量引用、过期时间和编码附件。其他协议字段不会因键名被当作秘密改写;跟踪 ID、鉴权、签名推理和附件不重写。对同一请求同一原值复用随机占位符;响应层在原有记录与协议翻译之后用该请求的内存映射还原,不在日志或库中持久化映射。SSE 分片按选择、内容块和工具调用独立关联,保留事件顺序与协议结束帧。完整结构化输出只解析并还原助手 JSON 正文,实际未发生替换时保留原字节;截断、围栏或不支持该模式的正文保留上游结果并按 JSON 字符串片段还原,不因格式不完整替换为网关错误。流式正文同样按 JSON 字符串片段转义;推理、拒绝说明、指令和工具描述继续按普通文本还原。输出模式在入站首次确定,后续协议翻译或续写不重置。MCP 审批参数、参数增量和完成事件复用同一遍历,标识符不改写。SSE 行边界统一支持 CRLF、LF 和单独 CR,仅流首识别 BOM;真实回环发现原生上游接收层先于包装层丢弃单独 CR 事件,因此该层同步采用字节行边界,保留签名/未知字段及原始换行,跨传输块的 CR 等下一个字节或 EOF 再判;等待帧数或字节数达到上限且仍有待拼接占位符时显式报错,EOF 的自然未完文本仍保留。去重使用入站原始字节摘要;入口及普通 / 原生发送体在合并网关提示词后复用同一映射,包括流式自动续写。Responses 信封回显的 instructions 和 tools 同样按白名单还原;JSON 字符串标量先解码再遮盖,数字、布尔及 null 的文本仍扫描原始表示,防止转义绕过或非字符串值漏判。重复遮盖在计数前排除已拥有占位符,仅新增明文消耗预算。含 `previous_response_id` 或 `conversation` 的 Responses 请求在开启时明确报错,客户端可随请求重发完整 `input`,而不是为还原旧占位符保存跨请求的原值。 + +内置检测移植自 [固定版本的 Magpie 规则表](https://github.com/yetone/magpie/tree/d847b79690cb12951ceefbb53f51993abac76b43/internal/redact)(MIT,许可见 `licenses/magpie-MIT`):58 个确定性识别器包含凭据、钱包格式、校验位和个人信息的上下文排除。保留 UUID 或摘要中的误匹配、游标与工具元素标识,但真实凭据字段和明确配置的自定义词/规则仍可遮盖同形值。基础个人信息与普通附加类别跟随主开关,主目录用户名、登录名/主机名、公网 IP、云存储桶需另行开启。支持整词、大小写选项、日期变体、前缀或正则和精确白名单。对明确的凭据赋值不要求字母数字混合,并整体遮盖含冒号的值;示例域名只按整域或子域匹配,助记词按每段连续词完整遮盖。数字数组区间只从完整合法的 JSON 文档提取,并跳过 JSON 字符串;普通文字中方括号里的个人号码仍遮盖。每段文本只解析和索引一次,候选通过有序区间查询避免二次前缀扫描。修复范围后的扫描从实际消费位置继续,银行卡只在分组边界尝试有效前缀,不截断连续数字。自定义正则只排除零宽命中,不以四字节下限忽略管理员已保存的短规则。Rust 实现不运行外部模型或进程,不迁移 Magpie 的跨请求缓存、模型辅助识别及路由策略。 + +辅助识图子调用继承请求级映射,但用量、取消状态和关联 id 独立;启用脱敏的请求绕过跨请求识图缓存,避免缓存依赖旧映射的结果。内容敏感的标题/压缩分类在公共入口对原文计算,只将分类结果交给 handler,日志和发送链仍使用遮盖后的正文。真实保存发现标题规则写入后缺少运行期快照刷新,因此公共配置变更链同时刷新该快照,使首次保存即可在原文分类时生效,revision 也与盘面同步。标题识别入口把既有行为分类的显示名映射为配置 ID,避免 `Claude Code` 与 `claude-code` 的跨模块契约错位;未知分类不扩大识别。已启动尝试的发送体构造错误由统一帮助函数先定稿状态及原因,再沿原错误返回;未启动的校验路径不调用它。 + +## Alternatives considered + +- 复用现有指纹词表:最少新增界面和数据,但该词表按提供商决定是否改写、可能删除文本,也没有反向映射,与隐私防护的语义不同。 +- 分别在各提供商出站实现:能看到最终协议的每个字段,但覆盖容易随新提供商与重试分叉,也会让未遮盖的入站正文进入现有调试记录。用网关入口及对应出口统一处理,不改变各家鉴权。 +- 只在识图结果包含可见占位符时跳过缓存:可保留更多缓存命中,但结果可能依赖仍在请求内扩展的映射或拼接后文本;启用时统一绕过读写,以额外识图调用换取请求隔离。 +- 仅按方括号字符集合排除数字:扫描快,也无需解析,但会把普通文字中的联系方式误当 JSON;选择先验证完整文档再索引,增加一次线性语法校验以保留正确的个人信息保护。 +- 跨请求维护映射以提升前缀缓存命中:会扩大敏感值的驻留时间和跨请求关联。首版每请求独立随机,牺牲请求间占位符一致性。 + +## Consequences + +这是规则型文本保护,不是文件扫描、远端会话清理或全量数据防泄漏。面板内部助手/管理请求、图片附件、签名或加密块、协议标识不经此变换;开启时独立图像推理端点返回明确错误。被改写的正文需要解析并重新序列化,长请求和响应还原有上限。SSE 遇到可能未完的占位符会在有界队列里等后续分片;截断、改写过的占位符不保证还原。正常回传到客户端的内容会包含原值,对客户端处理与日志仍应按真实数据看待。 + +## Testing + +`node scripts/test-privacy-masking.mjs ` 启动独立网关和回环上游,验证三种协议的 JSON/SSE 文本和工具参数、未开启时的原样路径、出站鉴权不变、错误与请求隔离。纯 Rust 测试验证规则表、代表性正反例、校验位、SSE 出错后的轮询及请求级映射。集成脚本检验追加提示词、原始正文去重、历史请求拦截、高级规则出站、多语言数字密钥与计量值的区分,三种协议的 JSON 工具结果、标识符保留与 Responses 嵌套工具声明,另有关闭脱敏后让出站断言失败的负向控制。复审补充的八组出站用例在旧构建逐项失败,再验证修订后通过;覆盖纯字母/数字密码、完整冒号值、示例域名边界、file_password、同一匹配的多段助记词、三协议 JSON 标量、40,000 次词项及 Responses 信封回显。大输入用例核对真实上游接收到 40,000 个正确占位符及新增网关提示词,并以短响应隔离与匹配预算无关的大响应开销。界面明确将 Luhn 识别范围称为银行卡号,而不泛称银行账号,基础手机号明确限定中国大陆。新增回归覆盖转义 Unicode 的字符串标量、携带同一 revision 的并发保存,以及真实双页面的陈旧草稿写入冲突;取消刷新保留三个草稿区域,确认后才载入最新设置,编辑后不再显示旧的保存成功提示。 三协议 JSON 输出的引号、反斜杠与换行保持可解析,MCP 审批历史及交错调用的 delta/done 分别核对;相邻 IBAN 包含正则确实会吞入后续账号的原分组样例,而不只覆盖两位末组或未分组样例;原生 CR 从实际回环上游返回并保留签名,流等待达到上限的真实连接以错误结束。辅助识图用分块文本拼接后命中词项,连续两次请求各自还原且不复用旧缓存;大网关注入提示词引发遮盖失败时,报表里的实际尝试包含 503 和原因。Windows 调试网关夹具为既有深层辅助调用设置 16 MiB 线程栈,生产配置不变。 方括号个人号码、非完整 JSON 模式正文与无映射 JSON 的保真回归在已发布构建先实际失败;完整 SSE 序列包含结束事件,真实回环按空行派发并合并多行 data,缺失分隔符的负向输入明确报错。MCP 还原断言同时核对上游只有占位符,流错误断言排除客户端 deadline,故意挂起上游只触发客户端超时的负向模式必须失败。 diff --git a/desktop-tauri/i18n/i18n-src/en/settings.json b/desktop-tauri/i18n/i18n-src/en/settings.json index 87f097fd0..bdabe617a 100644 --- a/desktop-tauri/i18n/i18n-src/en/settings.json +++ b/desktop-tauri/i18n/i18n-src/en/settings.json @@ -1,4 +1,38 @@ { + "刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?": "Refreshing will discard your unsaved local privacy masking changes. Continue?", + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失": "Local privacy masking settings changed elsewhere. Refresh before saving again; your draft has been kept.", + "自定义脱敏规则必须是 JSON 数组": "Custom masking rules must be a JSON array", + "美国社会安全号码(SSN)": "US Social Security number (SSN)", + "护照号": "Passport number", + "IBAN": "IBAN", + "生日": "Date of birth", + "MAC 地址": "MAC address", + "序列号": "Serial number", + "主目录用户名": "Home directory username", + "登录用户名和主机名": "Login username and hostname", + "公网 IP": "Public IP address", + "云存储桶": "Cloud storage bucket", + "遮盖邮箱、中国大陆手机号、身份证号和银行卡号": "Mask email, mainland China mobile numbers, national ID and bank card numbers", + "附加个人信息分类": "Additional personal-data categories", + "自定义脱敏词(每行一个)": "Custom masking terms (one per line)", + "高级脱敏规则": "Advanced masking rules", + "自定义词忽略大小写": "Ignore letter case for custom terms", + "自定义词只匹配整词": "Match custom terms as whole words", + "原样放行的值(每行一个)": "Values to leave unchanged (one per line)", + "前缀或正则规则(开启凭据脱敏时生效)": "Prefix or regex rules (when credential masking is on)", + "每条规则填 prefix 或 regex,正则有捕获组时只遮盖第一组。自定义词支持同一天的日期变体。": "Set prefix or regex for each rule. With capture groups, only group 1 is masked. Custom terms recognize alternate spellings of the same date.", + "本地隐私脱敏": "Local privacy masking", + "服务端未返回本地隐私脱敏设置": "The server did not return privacy masking settings", + "本地隐私脱敏设置格式无效": "Invalid privacy masking settings", + "当前后端暂不支持本地隐私脱敏": "This backend does not support local privacy masking", + "自定义脱敏词最多 100 个,每个不超过 256 字节": "Use up to 100 custom terms of at most 256 bytes each", + "设置已保存:从下一个推理请求生效": "Saved. Applies from the next inference request.", + "正在读取设置…": "Loading settings…", + "默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。": "Off by default. Protects Chat Completions, Responses and Messages through this gateway. Text and tool arguments are restored for the client.", + "遮盖 API Key、Token、密码和已识别的私钥": "Mask API keys, tokens, passwords and recognizable private keys", + "遮盖邮箱和中国大陆手机号": "Mask email addresses and mainland China mobile numbers", + "自定义脱敏词(每行一个,精确区分大小写)": "Custom terms (one per line; exact, case-sensitive matches)", + "原值与占位符的映射只在本次请求内存中保留。附件、签名内容和面板内置助手不受此设置保护;启用时网关独立生图请求会报错。规则可能漏判,请用测试数据验证。": "Mappings live only in memory for this request. Attachments, signed content and the panel assistant are not covered. Standalone gateway image-generation requests fail when masking is enabled. Rules can miss data; test with sample values.", "界面渲染": "Interface rendering", "硬件加速": "Hardware acceleration", "未能读取硬件加速设置": "Could not read hardware acceleration settings", diff --git a/desktop-tauri/i18n/i18n-src/ja/settings.json b/desktop-tauri/i18n/i18n-src/ja/settings.json index c8ecd7765..ccd3f4d6a 100644 --- a/desktop-tauri/i18n/i18n-src/ja/settings.json +++ b/desktop-tauri/i18n/i18n-src/ja/settings.json @@ -1,4 +1,38 @@ { + "刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?": "更新すると未保存のローカルプライバシー設定が破棄されます。続行しますか?", + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失": "ローカルプライバシー設定が別の場所で変更されました。更新後に保存してください。下書きは保持されています。", + "自定义脱敏规则必须是 JSON 数组": "カスタムルールは JSON 配列で指定してください", + "美国社会安全号码(SSN)": "米国社会保障番号(SSN)", + "护照号": "パスポート番号", + "IBAN": "IBAN", + "生日": "生年月日", + "MAC 地址": "MAC アドレス", + "序列号": "シリアル番号", + "主目录用户名": "ホームディレクトリのユーザー名", + "登录用户名和主机名": "ログインユーザー名とホスト名", + "公网 IP": "グローバル IP アドレス", + "云存储桶": "クラウドストレージのバケット", + "遮盖邮箱、中国大陆手机号、身份证号和银行卡号": "メールアドレス、中国本土の携帯番号、身分証番号、銀行カード番号を隠す", + "附加个人信息分类": "追加の個人情報カテゴリ", + "自定义脱敏词(每行一个)": "カスタム遮蔽語(1 行に 1 語)", + "高级脱敏规则": "高度なマスキングルール", + "自定义词忽略大小写": "カスタム語の大文字・小文字を区別しない", + "自定义词只匹配整词": "カスタム語を単語単位で照合する", + "原样放行的值(每行一个)": "変更しない値(1 行に 1 件)", + "前缀或正则规则(开启凭据脱敏时生效)": "接頭辞または正規表現(認証情報の遮蔽が有効な場合)", + "每条规则填 prefix 或 regex,正则有捕获组时只遮盖第一组。自定义词支持同一天的日期变体。": "各ルールに prefix または regex を指定します。キャプチャグループがある場合は第1グループを隠します。同じ日付の別表記にも対応します。", + "本地隐私脱敏": "ローカルプライバシーマスキング", + "服务端未返回本地隐私脱敏设置": "サーバーからマスキング設定が返されませんでした", + "本地隐私脱敏设置格式无效": "マスキング設定の形式が無効です", + "当前后端暂不支持本地隐私脱敏": "現在のバックエンドはマスキングに対応していません", + "自定义脱敏词最多 100 个,每个不超过 256 字节": "カスタム語は最大100個、それぞれ256バイト以内です", + "设置已保存:从下一个推理请求生效": "保存しました。次の推論リクエストから適用されます。", + "正在读取设置…": "設定を読み込み中…", + "默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。": "既定ではオフです。このゲートウェイを通る Chat Completions、Responses、Messages を保護し、テキストとツール引数をクライアント向けに復元します。", + "遮盖 API Key、Token、密码和已识别的私钥": "API キー、トークン、パスワード、認識可能な秘密鍵を隠す", + "遮盖邮箱和中国大陆手机号": "メールアドレスと中国本土の携帯番号を隠す", + "自定义脱敏词(每行一个,精确区分大小写)": "カスタム語(1行に1個、大文字と小文字を区別して完全一致)", + "原值与占位符的映射只在本次请求内存中保留。附件、签名内容和面板内置助手不受此设置保护;启用时网关独立生图请求会报错。规则可能漏判,请用测试数据验证。": "対応付けはリクエスト中のメモリのみに保持されます。添付ファイル、署名付きコンテンツ、パネルのアシスタントは対象外です。有効時、単独の画像生成はエラーになります。検出漏れがあり得るためサンプルで確認してください。", "界面渲染": "画面の描画", "硬件加速": "ハードウェアアクセラレーション", "未能读取硬件加速设置": "ハードウェアアクセラレーションの設定を読み込めませんでした", diff --git a/desktop-tauri/i18n/i18n-src/ko/settings.json b/desktop-tauri/i18n/i18n-src/ko/settings.json index a31a4352e..98158fbfd 100644 --- a/desktop-tauri/i18n/i18n-src/ko/settings.json +++ b/desktop-tauri/i18n/i18n-src/ko/settings.json @@ -1,4 +1,38 @@ { + "刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?": "새로 고치면 저장하지 않은 로컬 개인정보 마스킹 변경사항이 삭제됩니다. 계속하시겠습니까?", + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失": "다른 곳에서 로컬 개인정보 마스킹 설정을 변경했습니다. 새로 고침 후 다시 저장하세요. 현재 초안은 유지됩니다.", + "自定义脱敏规则必须是 JSON 数组": "사용자 지정 규칙은 JSON 배열이어야 합니다", + "美国社会安全号码(SSN)": "미국 사회보장번호(SSN)", + "护照号": "여권 번호", + "IBAN": "IBAN", + "生日": "생년월일", + "MAC 地址": "MAC 주소", + "序列号": "일련번호", + "主目录用户名": "홈 디렉터리 사용자 이름", + "登录用户名和主机名": "로그인 사용자 이름과 호스트 이름", + "公网 IP": "공인 IP 주소", + "云存储桶": "클라우드 스토리지 버킷", + "遮盖邮箱、中国大陆手机号、身份证号和银行卡号": "이메일, 중국 본토 휴대전화 번호, 신분증 번호 및 은행 카드 번호 마스킹", + "附加个人信息分类": "추가 개인정보 유형", + "自定义脱敏词(每行一个)": "사용자 지정 마스킹 단어(한 줄에 하나)", + "高级脱敏规则": "고급 마스킹 규칙", + "自定义词忽略大小写": "사용자 지정 단어의 대소문자 무시", + "自定义词只匹配整词": "사용자 지정 단어를 전체 단어로만 일치", + "原样放行的值(每行一个)": "그대로 전달할 값(한 줄에 하나)", + "前缀或正则规则(开启凭据脱敏时生效)": "접두사 또는 정규식(자격 증명 마스킹 사용 시)", + "每条规则填 prefix 或 regex,正则有捕获组时只遮盖第一组。自定义词支持同一天的日期变体。": "규칙마다 prefix 또는 regex를 지정합니다. 캡처 그룹이 있으면 첫 그룹만 마스킹합니다. 같은 날짜의 다른 표기도 지원합니다.", + "本地隐私脱敏": "로컬 개인정보 마스킹", + "服务端未返回本地隐私脱敏设置": "서버가 개인정보 마스킹 설정을 반환하지 않았습니다", + "本地隐私脱敏设置格式无效": "개인정보 마스킹 설정 형식이 잘못되었습니다", + "当前后端暂不支持本地隐私脱敏": "현재 백엔드는 로컬 개인정보 마스킹을 지원하지 않습니다", + "自定义脱敏词最多 100 个,每个不超过 256 字节": "사용자 지정 단어는 최대 100개이며 각 단어는 256바이트 이하여야 합니다", + "设置已保存:从下一个推理请求生效": "저장했습니다. 다음 추론 요청부터 적용됩니다.", + "正在读取设置…": "설정 읽는 중…", + "默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。": "기본값은 꺼짐입니다. 이 게이트웨이를 거치는 Chat Completions, Responses, Messages 요청만 보호하며 클라이언트에 반환하는 텍스트와 도구 인수는 복원합니다.", + "遮盖 API Key、Token、密码和已识别的私钥": "API 키, 토큰, 비밀번호 및 인식 가능한 개인 키 마스킹", + "遮盖邮箱和中国大陆手机号": "이메일 주소 및 중국 본토 휴대전화 번호 마스킹", + "自定义脱敏词(每行一个,精确区分大小写)": "사용자 지정 단어(한 줄에 하나, 대소문자 구분)", + "原值与占位符的映射只在本次请求内存中保留。附件、签名内容和面板内置助手不受此设置保护;启用时网关独立生图请求会报错。规则可能漏判,请用测试数据验证。": "원본과 자리표시자의 매핑은 이 요청의 메모리에만 보관됩니다. 첨부 파일, 서명된 콘텐츠 및 패널 어시스턴트는 보호되지 않습니다. 활성화하면 별도의 이미지 생성 요청은 실패합니다. 누락 가능성이 있으므로 테스트 데이터로 확인하세요.", "界面渲染": "화면 렌더링", "硬件加速": "하드웨어 가속", "未能读取硬件加速设置": "하드웨어 가속 설정을 읽지 못했습니다", diff --git a/desktop-tauri/i18n/i18n-src/pt-BR/settings.json b/desktop-tauri/i18n/i18n-src/pt-BR/settings.json index 260723a59..ae83ec8b8 100644 --- a/desktop-tauri/i18n/i18n-src/pt-BR/settings.json +++ b/desktop-tauri/i18n/i18n-src/pt-BR/settings.json @@ -1,4 +1,38 @@ { + "刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?": "Atualizar descartará as alterações de privacidade local não salvas. Continuar?", + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失": "As configurações de privacidade local foram alteradas em outro lugar. Atualize antes de salvar novamente; o rascunho foi mantido.", + "自定义脱敏规则必须是 JSON 数组": "As regras personalizadas devem ser um array JSON", + "美国社会安全号码(SSN)": "Número de seguridade social dos EUA (SSN)", + "护照号": "Número do passaporte", + "IBAN": "IBAN", + "生日": "Data de nascimento", + "MAC 地址": "Endereço MAC", + "序列号": "Número de série", + "主目录用户名": "Usuário no diretório pessoal", + "登录用户名和主机名": "Usuário de login e nome do host", + "公网 IP": "Endereço IP público", + "云存储桶": "Bucket de armazenamento em nuvem", + "遮盖邮箱、中国大陆手机号、身份证号和银行卡号": "Mascarar e-mail, celulares da China continental, documento de identidade e números de cartões bancários", + "附加个人信息分类": "Categorias adicionais de dados pessoais", + "自定义脱敏词(每行一个)": "Termos personalizados (um por linha)", + "高级脱敏规则": "Regras avançadas de mascaramento", + "自定义词忽略大小写": "Ignorar maiúsculas e minúsculas nos termos", + "自定义词只匹配整词": "Corresponder apenas a palavras inteiras", + "原样放行的值(每行一个)": "Valores a manter sem alteração (um por linha)", + "前缀或正则规则(开启凭据脱敏时生效)": "Regras de prefixo ou regex (com mascaramento de credenciais ativo)", + "每条规则填 prefix 或 regex,正则有捕获组时只遮盖第一组。自定义词支持同一天的日期变体。": "Defina prefix ou regex. Havendo grupos de captura, só o primeiro é mascarado. Termos aceitam grafias diferentes da mesma data.", + "本地隐私脱敏": "Mascaramento local de dados sensíveis", + "服务端未返回本地隐私脱敏设置": "O servidor não retornou as configurações de mascaramento", + "本地隐私脱敏设置格式无效": "Configurações de mascaramento inválidas", + "当前后端暂不支持本地隐私脱敏": "Este servidor não oferece mascaramento local", + "自定义脱敏词最多 100 个,每个不超过 256 字节": "Use até 100 termos personalizados de até 256 bytes cada", + "设置已保存:从下一个推理请求生效": "Configurações salvas. Valem a partir da próxima solicitação de inferência.", + "正在读取设置…": "Carregando configurações…", + "默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。": "Desativado por padrão. Protege Chat Completions, Responses e Messages deste gateway. Textos e argumentos de ferramentas são restaurados para o cliente.", + "遮盖 API Key、Token、密码和已识别的私钥": "Mascarar chaves de API, tokens, senhas e chaves privadas reconhecíveis", + "遮盖邮箱和中国大陆手机号": "Mascarar endereços de e-mail e celulares da China continental", + "自定义脱敏词(每行一个,精确区分大小写)": "Termos personalizados (um por linha; correspondência exata, com distinção de maiúsculas)", + "原值与占位符的映射只在本次请求内存中保留。附件、签名内容和面板内置助手不受此设置保护;启用时网关独立生图请求会报错。规则可能漏判,请用测试数据验证。": "A associação só fica na memória da solicitação. Anexos, conteúdo assinado e o assistente do painel ficam fora da proteção. A geração independente de imagens falha com o mascaramento ativo. As regras podem ter lacunas; valide com dados de teste.", "界面渲染": "Renderização da interface", "硬件加速": "Aceleração de hardware", "未能读取硬件加速设置": "Não foi possível ler as configurações de aceleração de hardware", diff --git a/desktop-tauri/i18n/i18n-src/zh-Hant/settings.json b/desktop-tauri/i18n/i18n-src/zh-Hant/settings.json index c29dbee41..abebdc7a0 100644 --- a/desktop-tauri/i18n/i18n-src/zh-Hant/settings.json +++ b/desktop-tauri/i18n/i18n-src/zh-Hant/settings.json @@ -1,4 +1,38 @@ { + "刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?": "重新整理將捨棄尚未儲存的本機隱私遮蔽修改,是否繼續?", + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失": "本機隱私遮蔽設定已在其他位置修改,請重新整理後再儲存;目前草稿未遺失", + "自定义脱敏规则必须是 JSON 数组": "自訂遮蔽規則必須是 JSON 陣列", + "美国社会安全号码(SSN)": "美國社會安全號碼(SSN)", + "护照号": "護照號碼", + "IBAN": "IBAN", + "生日": "生日", + "MAC 地址": "MAC 位址", + "序列号": "序號", + "主目录用户名": "家目錄使用者名稱", + "登录用户名和主机名": "登入使用者名稱和主機名稱", + "公网 IP": "公網 IP", + "云存储桶": "雲端儲存桶", + "遮盖邮箱、中国大陆手机号、身份证号和银行卡号": "遮蔽電子郵件、中國大陸手機號碼、身分證號及銀行卡號", + "附加个人信息分类": "其他個人資料分類", + "自定义脱敏词(每行一个)": "自訂遮蔽詞(每行一個)", + "高级脱敏规则": "進階遮蔽規則", + "自定义词忽略大小写": "自訂詞不區分大小寫", + "自定义词只匹配整词": "自訂詞只比對完整詞", + "原样放行的值(每行一个)": "原樣放行的值(每行一個)", + "前缀或正则规则(开启凭据脱敏时生效)": "前綴或正規表示式(啟用憑證遮蔽時生效)", + "每条规则填 prefix 或 regex,正则有捕获组时只遮盖第一组。自定义词支持同一天的日期变体。": "每條規則填 prefix 或 regex;有擷取群組時只遮蔽第一組。自訂詞支援同一天的其他日期格式。", + "本地隐私脱敏": "本機隱私遮蔽", + "服务端未返回本地隐私脱敏设置": "伺服器未回傳本機隱私遮蔽設定", + "本地隐私脱敏设置格式无效": "本機隱私遮蔽設定格式無效", + "当前后端暂不支持本地隐私脱敏": "目前的後端不支援本機隱私遮蔽", + "自定义脱敏词最多 100 个,每个不超过 256 字节": "自訂遮蔽詞最多 100 個,每個不超過 256 位元組", + "设置已保存:从下一个推理请求生效": "設定已儲存,下個推論請求起生效", + "正在读取设置…": "正在讀取設定…", + "默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。": "預設關閉。只保護經此閘道的 Chat Completions、Responses 與 Messages,回傳給用戶端的文字和工具參數會還原。", + "遮盖 API Key、Token、密码和已识别的私钥": "遮蔽 API Key、權杖、密碼與可辨識的私鑰", + "遮盖邮箱和中国大陆手机号": "遮蔽電子郵件及中國大陸手機號碼", + "自定义脱敏词(每行一个,精确区分大小写)": "自訂遮蔽詞(每行一個,精確區分大小寫)", + "原值与占位符的映射只在本次请求内存中保留。附件、签名内容和面板内置助手不受此设置保护;启用时网关独立生图请求会报错。规则可能漏判,请用测试数据验证。": "原值與預留位置的對應只保存在本次請求的記憶體中。附件、簽署內容與面板內建助理不受保護。啟用時,獨立產圖請求會回傳錯誤。規則可能漏判,請用測試資料驗證。", "界面渲染": "介面繪製", "硬件加速": "硬體加速", "未能读取硬件加速设置": "無法讀取硬體加速設定", diff --git a/desktop-tauri/src-tauri/Cargo.lock b/desktop-tauri/src-tauri/Cargo.lock index 73e5888c8..ae1af88b4 100644 --- a/desktop-tauri/src-tauri/Cargo.lock +++ b/desktop-tauri/src-tauri/Cargo.lock @@ -84,8 +84,10 @@ dependencies = [ "axum", "base64 0.22.1", "bcrypt", + "bs58", "bytes", "chrono", + "data-encoding", "ed25519-dalek", "flate2", "futures", @@ -99,6 +101,7 @@ dependencies = [ "md-5", "num-bigint", "p256", + "psl", "regex", "reqwest 0.12.28", "ring", @@ -604,6 +607,7 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bf88ba1141d185c399bee5288d850d63b8369520c1eafc32a0430b5b6c287bf4" dependencies = [ + "sha2", "tinyvec", ] @@ -4094,6 +4098,21 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "psl" +version = "2.1.259" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b85209e5945ce6d43ae3ac56491c30ac506db07108346758667a82bb2478c53" +dependencies = [ + "psl-types", +] + +[[package]] +name = "psl-types" +version = "2.0.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33cb294fe86a74cbcf50d4445b37da762029549ebeea341421c7c70370f86cac" + [[package]] name = "pulley-interpreter" version = "36.0.20" diff --git a/desktop-tauri/src-tauri/server/Cargo.toml b/desktop-tauri/src-tauri/server/Cargo.toml index 76ed56ce2..271f5956c 100644 --- a/desktop-tauri/src-tauri/server/Cargo.toml +++ b/desktop-tauri/src-tauri/server/Cargo.toml @@ -122,6 +122,11 @@ bytes = "1" # CatPaw 适配器的内联图片压缩(providers/catpaw/image_compress.rs) image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp"] } base64 = "0.22" +# 入站隐私遮盖仅识别并校验钱包格式,不导出密钥。 +bs58 = { version = "0.5", features = ["check"] } +data-encoding = "2" +# 编译期嵌入 Public Suffix List;识别邮箱不查询网络。 +psl = "2" # Qoder 网页登录的 PKCE verifier / nonce getrandom = "0.2" # P-256 密钥(两家共用一份依赖): diff --git a/desktop-tauri/src-tauri/server/src/server/api/chat.rs b/desktop-tauri/src-tauri/server/src/server/api/chat.rs index 2450e7e9c..3861e5db6 100644 --- a/desktop-tauri/src-tauri/server/src/server/api/chat.rs +++ b/desktop-tauri/src-tauri/server/src/server/api/chat.rs @@ -79,6 +79,7 @@ pub async fn chat_completions( key_scope: Option>, key_identity: Option>, key_source: Option>>, + privacy: Option>, body: Bytes, ) -> Response { let scope = key_scope.map(|Extension(scope)| scope); @@ -117,7 +118,8 @@ pub async fn chat_completions( .unwrap_or(0); let user_agent = pipeline::user_agent_of(&headers); let body_len = body.len(); - let dedupe_key = pipeline::sha256_hex(&body); + let dedupe_key = privacy.as_ref().map(|Extension(context)| context.original_digest.clone()) + .unwrap_or_else(|| pipeline::sha256_hex(&body)); let raw_request = pipeline::capture_raw_body(&body); // ③ 调试落盘:保存最近一次入站请求体,供重放分析(覆盖写) @@ -132,13 +134,30 @@ pub async fn chat_completions( // 下游等级与下游模型名同一时机采集(payload 被就地改写前;改写只动 model, // 两边前后一致,见 client_reasoning_of 的说明) let client_reasoning = pipeline::client_reasoning_of(&payload); - let compact_client = compact_routing::client_for_request(&request_source.behavior_client, &payload, RequestFormat::Chat); - // 标题请求只在不命中压缩时判:两者都是「把这条请求换成便宜模型」,同时命中会互相覆盖。 + let (compact_client, title_client) = if let Some(Extension(context)) = privacy.as_ref() { + (context.compact_client, context.title_client) + } else { + let compact = compact_routing::client_for_request( + &request_source.behavior_client, + &payload, + RequestFormat::Chat, + ); + let title = if compact.is_none() { + title_routing::client_for_request( + &request_source.behavior_client, + &headers, + &payload, + false, + ) + } else { + None + }; + (compact, title) + }; let special = if compact_client.is_some() { super::request_prepare::SpecialRoute::compact(compact_client) } else { - super::request_prepare::SpecialRoute::title(title_routing::client_for_request( - &request_source.behavior_client, &headers, &payload, false)) + super::request_prepare::SpecialRoute::title(title_client) }; let requested_model = match super::request_prepare::resolve_client_model(&state, &mut payload, scope.as_ref(), special).await { Ok(model) => model, @@ -164,7 +183,8 @@ pub async fn chat_completions( ); // 原始请求的去重键已在模型改写前计算。 - let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source))); + let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source)) + .with_privacy_mask(privacy.map(|Extension(context)| context.masking))); telemetry.capture_prompt_request(&request_source, &client_model); // 「进行中」行:模型解析已成功、转发即将开始 —— 先插一条 status=0 的明细, // 让请求日志在响应还没跑完时就能看到这条请求(收尾时由 `record` 的 UPDATE @@ -511,4 +531,4 @@ pub fn spawn_catalog_refresh(state: &ServerState) { crate::server::logging::verbose("[Maintenance]", &error); } }); -} \ No newline at end of file +} diff --git a/desktop-tauri/src-tauri/server/src/server/api/mod.rs b/desktop-tauri/src-tauri/server/src/server/api/mod.rs index af56cae9a..895f8a481 100644 --- a/desktop-tauri/src-tauri/server/src/server/api/mod.rs +++ b/desktop-tauri/src-tauri/server/src/server/api/mod.rs @@ -190,6 +190,7 @@ pub mod routing_flow; pub mod routing_groups; pub mod routing_trace; pub mod routing_workspace; +pub(crate) mod privacy_masking; pub mod sanitize; pub mod scheduled_tasks; pub mod session; diff --git a/desktop-tauri/src-tauri/server/src/server/api/privacy_masking.rs b/desktop-tauri/src-tauri/server/src/server/api/privacy_masking.rs new file mode 100644 index 000000000..ffc2b35f0 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/api/privacy_masking.rs @@ -0,0 +1,232 @@ +//! HTTP 边界:鉴权/容量闸门之后遮盖正文,记账/调试记录之后仅向本次调用方还原。 +use axum::{ + body::{to_bytes, Body, Bytes}, + extract::Request, + http::{header, HeaderValue, Method}, + middleware::Next, + response::Response, +}; +use serde_json::Value; + +use crate::server::{ + config::privacy_masking as config, + core::privacy_masking::{ + stream, + wire::{self, Format}, + RequestMask, Restore, Settings, + }, + errors::{management_error, GatewayError}, + http::ok_json, +}; + +const MAX_BODY: usize = 32 * 1024 * 1024; + +pub(crate) async fn get() -> Response { + match config::snapshot() { + Ok(Some(policy)) => match policy.as_ref() { + Ok(policy) => settings_response(&policy.rules.settings, &policy.revision), + Err(message) => management_error(503, *message), + }, + Ok(None) => settings_response(&Settings::default(), &config::revision(&Value::Null)), + Err(message) => management_error(503, message), + } +} + +fn settings_response(settings: &Settings, revision: &str) -> Response { + match serde_json::to_value(settings) { + Ok(Value::Object(mut value)) => { + value.insert("revision".into(), Value::String(revision.to_owned())); + ok_json(Value::Object(value)) + } + Ok(_) => management_error(500, "隐私脱敏设置格式错误"), + Err(_) => management_error(500, "隐私脱敏设置序列化失败"), + } +} + +pub(crate) async fn put(body: Bytes) -> Response { + if body.len() > 64 * 1024 { + return management_error(413, "隐私脱敏设置过大"); + } + let Ok(Value::Object(mut fields)) = serde_json::from_slice::(&body) else { + return management_error(400, "隐私脱敏设置必须是 JSON 对象"); + }; + let Some(Value::String(revision)) = fields.remove("revision") else { + return management_error(400, "隐私脱敏设置需要 revision,请刷新后再保存"); + }; + if revision.len() != 64 || !revision.bytes().all(|b| b.is_ascii_hexdigit()) { + return management_error(400, "隐私脱敏 revision 无效,请刷新后再保存"); + } + let Ok(settings) = serde_json::from_value::(Value::Object(fields)) else { + return management_error( + 400, + "隐私脱敏设置必须是包含 secrets、personal、terms 的对象", + ); + }; + match config::save(settings, &revision) { + Ok(()) => get().await, + Err(config::SaveError::Storage) => management_error(500, "隐私脱敏设置未能持久化"), + Err(config::SaveError::Conflict) => management_error( + 409, + "本地隐私脱敏设置已在其他位置修改,请刷新后再保存;当前草稿未丢失", + ), + Err(config::SaveError::Invalid(message)) => management_error(400, message), + } +} + +#[derive(Clone)] +pub(crate) struct PrivacyContext { + pub(super) original_digest: String, + pub(super) masking: RequestMask, + pub(super) compact_client: Option<&'static str>, + pub(super) title_client: Option<&'static str>, +} + +pub(crate) async fn protect(request: Request, next: Next) -> Response { + if request.method() != Method::POST { + return next.run(request).await; + } + let format = match request.uri().path() { + "/v1/chat/completions" => Some(Format::Chat), + "/v1/responses" => Some(Format::Responses), + "/v1/messages" => Some(Format::Messages), + "/v1/images/generations" | "/v1/images/edits" | "/mcp/vision" | "/mcp/image-generation" => { + None + } + _ => return next.run(request).await, + }; + let policy = match config::snapshot() { + Ok(None) => return next.run(request).await, + Ok(Some(policy)) => policy, + Err(message) => return error(503, message), + }; + let rules = match policy.as_ref() { + Ok(policy) if policy.rules.enabled() => &policy.rules, + Ok(_) => return next.run(request).await, + Err(message) => return error(503, message), + }; + let Some(format) = format else { + return error(400, "隐私脱敏已开启;此推理端点不支持本地脱敏,未转发"); + }; + if request + .headers() + .get(header::CONTENT_ENCODING) + .is_some_and(|v| v.as_bytes() != b"identity") + { + return error(415, "隐私脱敏仅接受未压缩的 JSON 请求,未转发"); + } + let (mut parts, body) = request.into_parts(); + let original = match to_bytes(body, MAX_BODY).await { + Ok(body) => body, + Err(_) => return error(413, "隐私脱敏无法读取请求或请求超过 32 MiB,未转发"), + }; + let Ok(mut parsed @ Value::Object(_)) = serde_json::from_slice::(&original) else { + return error(400, "隐私脱敏需要 JSON 对象,未转发"); + }; + if matches!(format, Format::Responses) + && ["previous_response_id", "conversation"] + .iter() + .any(|key| parsed.get(key).is_some_and(|v| !v.is_null())) + { + // 上游历史没有本轮的映射。禁止隐式恢复跨请求的原值,完整重发上下文即可继续使用。 + return GatewayError::bad_request( + "隐私脱敏不支持引用上游历史;请随本次请求发送完整 input,未转发", + ) + .with_code("privacy_masking_stateful_responses") + .payload_response(); + } + // Run content-sensitive routing classifiers on the original request before masking. + let behavior_client = crate::server::core::client::classify( + parts + .headers + .get(header::USER_AGENT) + .and_then(|value| value.to_str().ok()) + .unwrap_or(""), + ); + let request_format = match format { + Format::Chat => crate::server::core::compact_routing::RequestFormat::Chat, + Format::Responses => crate::server::core::compact_routing::RequestFormat::Responses, + Format::Messages => crate::server::core::compact_routing::RequestFormat::Anthropic, + }; + let compact_client = crate::server::core::compact_routing::client_for_request( + &behavior_client, + &parsed, + request_format, + ); + let title_client = if compact_client.is_none() { + crate::server::core::title_routing::client_for_request( + &behavior_client, + &parts.headers, + &parsed, + matches!(format, Format::Responses), + ) + } else { + None + }; + let masking = RequestMask::new(rules); + let original_digest = super::pipeline::sha256_hex(&original); + if let Err(err) = masking.apply(format, &mut parsed) { + return err.payload_response(); + } + let Ok(bytes) = serde_json::to_vec(&parsed) else { + return error(503, "隐私脱敏处理失败,未转发"); + }; + if bytes.len() > MAX_BODY { + return error(413, "脱敏后的请求超过 32 MiB,未转发"); + } + parts.headers.remove(header::CONTENT_LENGTH); + parts.extensions.insert(PrivacyContext { + original_digest, + masking: masking.clone(), + compact_client, + title_client, + }); + let upstream = next + .run(Request::from_parts(parts, Body::from(bytes))) + .await; + if !upstream.status().is_success() { + return upstream; + } + restore_response(format, upstream, masking.restore()).await +} + +async fn restore_response(format: Format, response: Response, mut restore: Restore) -> Response { + let (mut parts, body) = response.into_parts(); + let mime = parts + .headers + .get(header::CONTENT_TYPE) + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.split(';').next()) + .unwrap_or(""); + let streaming = mime.eq_ignore_ascii_case("text/event-stream"); + if !streaming && !mime.eq_ignore_ascii_case("application/json") { + return Response::from_parts(parts, body); + } + parts.headers.remove(header::CONTENT_LENGTH); + parts.headers.remove(header::ETAG); + parts + .headers + .insert(header::CACHE_CONTROL, HeaderValue::from_static("no-store")); + if streaming { + return Response::from_parts(parts, stream::wrap(body, format, restore)); + } + let payload = match to_bytes(body, MAX_BODY).await { + Ok(bytes) => bytes, + Err(_) => return error(502, "隐私脱敏响应超过限制"), + }; + let Ok(mut json) = serde_json::from_slice::(&payload) else { + return error(502, "隐私脱敏响应不是合法 JSON"); + }; + if wire::response(format, &mut json, &mut restore).is_err() { + return error(502, "隐私脱敏响应处理失败"); + } + match serde_json::to_vec(&json) { + Ok(bytes) => Response::from_parts(parts, Body::from(bytes)), + Err(_) => error(502, "隐私脱敏响应处理失败"), + } +} + +fn error(status: i32, message: &str) -> Response { + GatewayError::with_status(status, message) + .with_code("privacy_masking") + .payload_response() +} diff --git a/desktop-tauri/src-tauri/server/src/server/api/protocol.rs b/desktop-tauri/src-tauri/server/src/server/api/protocol.rs index 0583d7812..104eddd15 100644 --- a/desktop-tauri/src-tauri/server/src/server/api/protocol.rs +++ b/desktop-tauri/src-tauri/server/src/server/api/protocol.rs @@ -133,6 +133,7 @@ pub async fn responses_endpoint( key_scope: Option>, key_identity: Option>, key_source: Option>>, + privacy: Option>, body: Bytes, ) -> Response { let scope = key_scope.map(|Extension(scope)| scope); @@ -148,13 +149,25 @@ pub async fn responses_endpoint( Err(response) => return response, }; // 协议翻译前识别,保留原始文本块边界。 - let compact_client = compact_routing::client_for_request(&source.behavior_client, &raw, RequestFormat::Responses); - // 标题请求在原始体上判(Responses 的 input 数组形态),且只在不命中压缩时判。 + let (compact_client, title_client) = if let Some(Extension(context)) = privacy.as_ref() { + (context.compact_client, context.title_client) + } else { + let compact = compact_routing::client_for_request( + &source.behavior_client, + &raw, + RequestFormat::Responses, + ); + let title = if compact.is_none() { + title_routing::client_for_request(&source.behavior_client, &headers, &raw, true) + } else { + None + }; + (compact, title) + }; let special = if compact_client.is_some() { super::request_prepare::SpecialRoute::compact(compact_client) } else { - super::request_prepare::SpecialRoute::title(title_routing::client_for_request( - &source.behavior_client, &headers, &raw, true)) + super::request_prepare::SpecialRoute::title(title_client) }; // 原始请求体留一份:回程要用它回显请求侧字段(instructions / tools / …), // 而下面会把 payload 改写成 Chat 形态 @@ -203,7 +216,8 @@ pub async fn responses_endpoint( ), ); pipeline::write_debug_files(&body, "POST", path, &user_agent); - let dedupe_key = pipeline::sha256_hex(&body); + let dedupe_key = privacy.as_ref().map(|Extension(context)| context.original_digest.clone()) + .unwrap_or_else(|| pipeline::sha256_hex(&body)); let raw_request = pipeline::capture_raw_body(&body); drop(body); @@ -212,7 +226,8 @@ pub async fn responses_endpoint( // 模型那一半已在上面判过(分工见 forward_chat 的说明)。 // telemetry 在这里创建(不再由 forward_chat 代建):进行中行要在 // 转发开始前插入,见下方 record_started。 - let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source))); + let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source)) + .with_privacy_mask(privacy.map(|Extension(context)| context.masking))); let client_model = pipeline::model_field_text(&raw); telemetry.capture_prompt_request(&source, &client_model); // 「进行中」行(与 /v1/chat/completions 同一处时点:模型解析成功、 @@ -370,6 +385,7 @@ pub async fn messages_endpoint( key_scope: Option>, key_identity: Option>, key_source: Option>>, + privacy: Option>, body: Bytes, ) -> Response { let scope = key_scope.map(|Extension(scope)| scope); @@ -383,13 +399,25 @@ pub async fn messages_endpoint( }; let stream = raw.get("stream").and_then(Value::as_bool).unwrap_or(false); - let compact_client = compact_routing::client_for_request(&source.behavior_client, &raw, RequestFormat::Anthropic); - // 标题请求在原始体上判(Anthropic 的 content 块形态),只在不命中压缩时判。 + let (compact_client, title_client) = if let Some(Extension(context)) = privacy.as_ref() { + (context.compact_client, context.title_client) + } else { + let compact = compact_routing::client_for_request( + &source.behavior_client, + &raw, + RequestFormat::Anthropic, + ); + let title = if compact.is_none() { + title_routing::client_for_request(&source.behavior_client, &headers, &raw, false) + } else { + None + }; + (compact, title) + }; let special = if compact_client.is_some() { super::request_prepare::SpecialRoute::compact(compact_client) } else { - super::request_prepare::SpecialRoute::title(title_routing::client_for_request( - &source.behavior_client, &headers, &raw, false)) + super::request_prepare::SpecialRoute::title(title_client) }; // 协议翻译 let (native, mut chat_body) = match NativeRequest::prepare(WireProtocol::Anthropic, raw.clone()) { @@ -421,7 +449,8 @@ pub async fn messages_endpoint( ), ); pipeline::write_debug_files(&body, "POST", path, &user_agent); - let dedupe_key = pipeline::sha256_hex(&body); + let dedupe_key = privacy.as_ref().map(|Extension(context)| context.original_digest.clone()) + .unwrap_or_else(|| pipeline::sha256_hex(&body)); let raw_request = pipeline::capture_raw_body(&body); drop(body); @@ -429,7 +458,8 @@ pub async fn messages_endpoint( // 模型那一半已在上面判过(分工见 forward_chat 的说明)。 // telemetry 在这里创建(与 responses_endpoint 同理:进行中行要在 // 转发开始前插入) - let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source))); + let telemetry = Arc::new(RequestTelemetry::with_model_key(key_source.map(|Extension(source)| source)) + .with_privacy_mask(privacy.map(|Extension(context)| context.masking))); let client_model = pipeline::model_field_text(&raw); telemetry.capture_prompt_request(&source, &client_model); // 流式 `message_start` 的输入锚点:上游 usage 要等流末尾那帧才到,先按 @@ -614,4 +644,4 @@ fn anthropic_error_response_with_status(status: u16, error: &GatewayError) -> Re } } response -} \ No newline at end of file +} diff --git a/desktop-tauri/src-tauri/server/src/server/config/mod.rs b/desktop-tauri/src-tauri/server/src/server/config/mod.rs index 2abe65d6f..59a667ed6 100644 --- a/desktop-tauri/src-tauri/server/src/server/config/mod.rs +++ b/desktop-tauri/src-tauri/server/src/server/config/mod.rs @@ -85,6 +85,7 @@ pub(crate) mod parse; pub(crate) mod routing_groups; mod routing_profiles; pub(crate) mod routing_workspace; +pub(crate) mod privacy_masking; mod sanitize_config; pub(crate) mod sql; mod types; @@ -195,6 +196,7 @@ pub struct RuntimeConfig { debug_mode: bool, /// 规则在配置读取 / 保存时编译,请求间共享,同一次转发使用完整快照。 sanitize: SanitizeSettings, + privacy_masking: privacy_masking::Snapshot, /// 开发者角色兼容开关(设置页「网关 → 开发者角色兼容」)。 /// /// 与 `sanitize_fingerprints` 同一理由:转发层**每次发送前**都要判一次 @@ -619,6 +621,7 @@ fn build(raw: Map) -> RuntimeConfig { // 「写坏回落」同一取向 —— 这个开关控制是否把凭据落盘,宁可少采 debug_mode: raw.get(KEY_DEBUG_MODE).and_then(Value::as_bool).unwrap_or(false), sanitize: sanitize_config::from_raw(&raw), + privacy_masking: privacy_masking::from_raw(&raw), // 只有字面 `false` 算关闭:**默认开**(缺失 → true)。改写语义无损 // (developer 与 system 在 OpenAI 语义上等价),而默认关会让发 developer // 角色的客户端(新版 OpenAI SDK、Claude Code 等)一上来就撞不认识该角色 @@ -1322,6 +1325,7 @@ where let previous = next.clone(); let before = synced_fingerprint(&next.raw); mutate(&mut next); + privacy_masking::refresh(&mut next, &previous); // CONFIG 克隆门已释放;本门撤销完成后才允许存储或新的配置发布。 model_key_source::before_change(&previous, &next); plugin_task_grants::before_change(&previous.raw, &next.raw); diff --git a/desktop-tauri/src-tauri/server/src/server/config/mutations.rs b/desktop-tauri/src-tauri/server/src/server/config/mutations.rs index b2172dd0f..0671f3afd 100644 --- a/desktop-tauri/src-tauri/server/src/server/config/mutations.rs +++ b/desktop-tauri/src-tauri/server/src/server/config/mutations.rs @@ -59,6 +59,9 @@ fn try_update_raw_fields_inner( if next.raw.get(super::compact_routes::KEY_COMPACT_ROUTES) != previous.raw.get(super::compact_routes::KEY_COMPACT_ROUTES) { next.compact_routes = super::compact_routes::from_raw(&next.raw); } + if next.raw.get(super::title_routes::KEY_TITLE_ROUTES) != previous.raw.get(super::title_routes::KEY_TITLE_ROUTES) { + next.title_routes = super::title_routes::from_raw(&next.raw); + } if next.raw.get(super::runtime_capacity::KEY_RUNTIME_CAPACITY) != previous.raw.get(super::runtime_capacity::KEY_RUNTIME_CAPACITY) { next.runtime_capacity = super::runtime_capacity::from_raw(&next.raw); } @@ -73,6 +76,7 @@ fn try_update_raw_fields_inner( super::KEY_PROMPT_GATEWAY_TEXT].iter().any(|key| next.raw.get(*key) != previous.raw.get(*key)) { next.prompt = super::prompt_from(&next.raw); } + super::privacy_masking::refresh(&mut next, &previous); if !save_raw(&next.raw) { return Err(storage_error()); } diff --git a/desktop-tauri/src-tauri/server/src/server/config/privacy_masking.rs b/desktop-tauri/src-tauri/server/src/server/config/privacy_masking.rs new file mode 100644 index 000000000..4411782a9 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/config/privacy_masking.rs @@ -0,0 +1,69 @@ +use super::{try_update_raw_field, CONFIG}; +use crate::server::core::privacy_masking::{Rules, Settings}; +use serde_json::{Map, Value}; +use sha2::{Digest, Sha256}; +use std::sync::Arc; + +pub(crate) const KEY: &str = "privacyMasking"; + +#[derive(Debug)] +pub(crate) struct Policy { + pub(crate) rules: Rules, + pub(crate) revision: String, +} +pub(crate) type Snapshot = Option>>; + +pub(crate) enum SaveError { + Invalid(&'static str), + Conflict, + Storage, +} + +// Same content-revision convention as config::prompt_rules. The version is +// derived from the exact stored value, not a second independently read snapshot. +pub(crate) fn revision(value: &Value) -> String { + format!("{:x}", Sha256::digest(value.to_string().as_bytes())) +} + +pub(super) fn from_raw(raw: &Map) -> Snapshot { + raw.get(KEY).map(|value| { + Arc::new( + serde_json::from_value::(value.clone()) + .map_err(|_| "privacyMasking 设置无效") + .and_then(Rules::compile) + .map(|rules| Policy { + rules, + revision: revision(value), + }), + ) + }) +} + +pub(crate) fn snapshot() -> Result { + let config = CONFIG.read().map_err(|_| "隐私脱敏配置不可读")?; + Ok(config.as_ref().and_then(|cfg| cfg.privacy_masking.clone())) +} + +pub(crate) fn save(settings: Settings, expected: &str) -> Result<(), SaveError> { + let rules = Rules::compile(settings).map_err(SaveError::Invalid)?; + let value = serde_json::to_value(&rules.settings).map_err(|_| SaveError::Storage)?; + try_update_raw_field( + KEY, + move |stored| { + // This comparison and mutation both execute under the existing config + // update lock. A conflicting editor never changes disk or live policy. + if revision(stored) != expected { + return Err(SaveError::Conflict); + } + *stored = value; + Ok(()) + }, + || SaveError::Storage, + ) +} + +pub(super) fn refresh(next: &mut super::RuntimeConfig, previous: &super::RuntimeConfig) { + if next.raw.get(KEY) != previous.raw.get(KEY) { + next.privacy_masking = from_raw(&next.raw); + } +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/cloud_sync/snapshot.rs b/desktop-tauri/src-tauri/server/src/server/core/cloud_sync/snapshot.rs index f6241d979..580c3f0c6 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/cloud_sync/snapshot.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/cloud_sync/snapshot.rs @@ -71,6 +71,8 @@ const EXCLUDED_CONFIG_KEYS: &[&str] = &[ // 代理内核的下载出网线路:绑本机网络环境 "kernelProxy", crate::server::config::KEY_INSTALLED_UPDATE_VERSION, + // 自定义隐私词可能包含用户希望仅留在当前网关的名字或代号。 + crate::server::config::privacy_masking::KEY, ]; /// 保留键里**进快照**的项(其余保留键一律不进)。 @@ -529,6 +531,7 @@ mod tests { assert!(!is_synced_kv_key("kernelDir")); // 另一把密钥加密:不同步 assert!(!is_synced_kv_key("githubToken")); + assert!(!is_synced_kv_key(crate::server::config::privacy_masking::KEY)); // 保留键:白名单里的同步,其余不同步 assert!(is_synced_kv_key("agentProfiles")); assert!(is_synced_kv_key("proxyPool")); diff --git a/desktop-tauri/src-tauri/server/src/server/core/mod.rs b/desktop-tauri/src-tauri/server/src/server/core/mod.rs index d595d435b..231830481 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/mod.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/mod.rs @@ -127,6 +127,7 @@ pub(crate) mod routing_groups; pub mod routing_order; pub mod routing_scope; pub(crate) mod runtime_capacity; +pub(crate) mod privacy_masking; pub mod sanitize; pub mod scheduled_tasks; // 侧边栏布局(分组 / 顺序 / 显隐)的持久化:整份 JSON 存 kv 一个键。 diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/custom.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/custom.rs new file mode 100644 index 000000000..01ec8250d --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/custom.rs @@ -0,0 +1,197 @@ +//! Custom literals, date variants and bounded prefix/regex rules (Magpie MIT; see licenses/magpie-MIT). +use super::detectors::{month, month_name}; +use chrono::NaiveDate; +use regex::{Regex, RegexBuilder}; +use serde::{Deserialize, Serialize}; +use std::sync::LazyLock; + +const SEP: &str = r"(?:[ \t]*[-/.、/-][ \t]*|[ \t])"; +const ORD: &str = r"(?:st|nd|rd|th)?"; +static DATE_FORMATS: LazyLock<[Regex; 5]> = LazyLock::new(|| { + [ + format!(r"^([0-9]{{4}}){SEP}([0-9]{{1,2}}){SEP}([0-9]{{1,2}})\.?$"), + r"^([0-9]{4})[ \t]*[年년][ \t]*([0-9]{1,2})[ \t]*[月월][ \t]*([0-9]{1,2})[ \t]*[日号號일]?$".to_owned(), + format!(r"^([0-9]{{1,2}}){SEP}([0-9]{{1,2}}){SEP}([0-9]{{4}})$"), + format!(r"^([A-Za-z]+)\.?[ \t]+([0-9]{{1,2}}){ORD},?[ \t]+([0-9]{{4}})$"), + format!(r"^([0-9]{{1,2}}){ORD}(?:[ \t]+|-)([A-Za-z]+)\.?,?(?:[ \t]+|-)([0-9]{{4}})$"), +].map(|pattern|Regex::new(&pattern).expect("constant date expression")) +}); + +fn date_word(s: &str) -> Option<(i32, u32, u32)> { + for (i, re) in DATE_FORMATS.iter().enumerate() { + let Some(m) = re.captures(s) else { continue }; + let (a, b, c) = (&m[1], &m[2], &m[3]); + let number = |s: &str| s.parse::().unwrap_or(0); + let (y, m, d) = match i { + 0 | 1 => (number(a), number(b), number(c)), + 2 => { + let (p, q, y) = (number(a), number(b), number(c)); + if p == q || (q > 12 && p <= 12) { + (y, p, q) + } else if p > 12 && q <= 12 { + (y, q, p) + } else { + return None; + } + } + 3 => (number(c), month(a)?, number(b)), + _ => (number(c), month(b)?, number(a)), + }; + if !(1000..=2999).contains(&y) { + return None; + } + return NaiveDate::from_ymd_opt(y as i32, m, d).map(|_| (y as i32, m, d)); + } + None +} + +fn date_variants(w: &str) -> Option { + let (y, m, d) = date_word(w)?; + let name = month_name(m); + let short = if m == 9 { "Sept|Sep" } else { &name[..3] }; + let named = format!(r"(?i:{name}|(?:{short})\.?)"); + let (month, day) = ( + if m < 10 { + format!("0?{m}") + } else { + m.to_string() + }, + if d < 10 { + format!("0?{d}") + } else { + d.to_string() + }, + ); + let mut forms = vec![ + format!("{y}{SEP}{month}{SEP}{day}"), + format!(r"{y}[ \t]*年[ \t]*{month}[ \t]*月[ \t]*{day}(?:[ \t]*[日号號])?"), + format!(r"{y}[ \t]*년[ \t]*{month}[ \t]*월[ \t]*{day}(?:[ \t]*일)?"), + format!("{y}{m:02}{d:02}"), + format!(r"{named}[ \t]+{day}{ORD},?[ \t]+{y}"), + format!(r"{day}{ORD}(?:[ \t]+|-){named},?(?:[ \t]+|-){y}"), + ]; + if d > 12 || d == m { + forms.push(format!("{day}{SEP}{month}{SEP}{y}")); + if d != m { + forms.push(format!("{month}{SEP}{day}{SEP}{y}")); + } + } + forms.push(regex::escape(w)); + Some(forms.join("|")) +} + +fn date_boundary(s: &str, a: usize, b: usize) -> bool { + let t = s.as_bytes(); + let digit = |i: usize| t.get(i).is_some_and(u8::is_ascii_digit); + !(a > 0 && (digit(a - 1) || a > 1 && t[a - 1] == b'.' && digit(a - 2))) + && !s[..a] + .chars() + .next_back() + .is_some_and(|p| p.is_ascii_alphabetic() && t[a].is_ascii_alphabetic()) + && !(digit(b) || t.get(b) == Some(&b'.') && digit(b + 1)) +} +fn whole_word(s: &str, a: usize, b: usize) -> bool { + let is_word = |c: char| c.is_alphanumeric() || c == '_'; + let v = &s[a..b]; + let first = v.chars().next().unwrap_or(' '); + let last = v.chars().next_back().unwrap_or(' '); + !(is_word(first) && s[..a].chars().next_back().is_some_and(is_word)) + && !(is_word(last) && s[b..].chars().next().is_some_and(is_word)) + && !(first.is_ascii_digit() + && a >= 2 + && s.as_bytes()[a - 1] == b'.' + && s.as_bytes()[a - 2].is_ascii_digit()) + && !(last.is_ascii_digit() + && s.as_bytes().get(b) == Some(&b'.') + && s.as_bytes().get(b + 1).is_some_and(u8::is_ascii_digit)) +} + +#[derive(Clone)] +pub(super) struct Term { + pattern: Regex, + date: bool, +} +impl Term { + pub(super) fn compile(word: &str, fold: bool) -> Result { + let date = date_variants(word); + let expression = date.clone().unwrap_or_else(|| regex::escape(word)); + Ok(Self { + pattern: RegexBuilder::new(&expression) + .case_insensitive(fold) + .size_limit(512 * 1024) + .build()?, + date: date.is_some(), + }) + } + pub(super) fn spans<'a>( + &'a self, + s: &'a str, + whole: bool, + ) -> impl Iterator + 'a { + self.pattern + .find_iter(s) + .filter(move |m| { + if self.date { + date_boundary(s, m.start(), m.end()) + && (!whole || whole_word(s, m.start(), m.end())) + } else { + !whole || whole_word(s, m.start(), m.end()) + } + }) + .map(|m| (m.start(), m.end())) + } +} + +#[derive(Clone, Default, Eq, PartialEq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct CustomRule { + #[serde(default, skip_serializing_if = "String::is_empty")] + pub(crate) kind: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub(crate) prefix: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub(crate) regex: String, +} +impl CustomRule { + pub(super) fn compile(&mut self) -> Result, &'static str> { + self.prefix = self.prefix.trim().to_owned(); + self.regex = self.regex.trim().to_owned(); + self.kind = self + .kind + .trim() + .chars() + .map(|c| { + if c.is_ascii_alphanumeric() { + c.to_ascii_uppercase() + } else { + '_' + } + }) + .take(24) + .collect::() + .trim_matches('_') + .to_owned(); + if self.prefix.is_empty() && self.regex.is_empty() { + return Ok(None); + } + if self.prefix.is_empty() == self.regex.is_empty() + || (!self.prefix.is_empty() && !(3..=64).contains(&self.prefix.len())) + || self.regex.len() > 300 + { + return Err("自定义规则仅填前缀或正则;前缀 3–64 字节、正则最多 300 字节"); + } + let pattern = if self.prefix.is_empty() { + self.regex.clone() + } else { + format!("{}[A-Za-z0-9_-]{{8,}}", regex::escape(&self.prefix)) + }; + let re = RegexBuilder::new(&pattern) + .size_limit(512 * 1024) + .build() + .map_err(|_| "自定义脱敏正则无效或过于复杂")?; + if re.is_match("") { + return Err("自定义脱敏正则不得匹配空字符串"); + } + Ok(Some(re)) + } +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/bip39english.txt b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/bip39english.txt new file mode 100644 index 000000000..942040ed5 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/bip39english.txt @@ -0,0 +1,2048 @@ +abandon +ability +able +about +above +absent +absorb +abstract +absurd +abuse +access +accident +account +accuse +achieve +acid +acoustic +acquire +across +act +action +actor +actress +actual +adapt +add +addict +address +adjust +admit +adult +advance +advice +aerobic +affair +afford +afraid +again +age +agent +agree +ahead +aim +air +airport +aisle +alarm +album +alcohol +alert +alien +all +alley +allow +almost +alone +alpha +already +also +alter +always +amateur +amazing +among +amount +amused +analyst +anchor +ancient +anger +angle +angry +animal +ankle +announce +annual +another +answer +antenna +antique +anxiety +any +apart +apology +appear +apple +approve +april +arch +arctic +area +arena +argue +arm +armed +armor +army +around +arrange +arrest +arrive +arrow +art +artefact +artist +artwork +ask +aspect +assault +asset +assist +assume +asthma +athlete +atom +attack +attend +attitude +attract +auction +audit +august +aunt +author +auto +autumn +average +avocado +avoid +awake +aware +away +awesome +awful +awkward +axis +baby +bachelor +bacon +badge +bag +balance +balcony +ball +bamboo +banana +banner +bar +barely +bargain +barrel +base +basic +basket +battle +beach +bean +beauty +because +become +beef +before +begin +behave +behind +believe +below +belt +bench +benefit +best +betray +better +between +beyond +bicycle +bid +bike +bind +biology +bird +birth +bitter +black +blade +blame +blanket +blast +bleak +bless +blind +blood +blossom +blouse +blue +blur +blush +board +boat +body +boil +bomb +bone +bonus +book +boost +border +boring +borrow +boss +bottom +bounce +box +boy +bracket +brain +brand +brass +brave +bread +breeze +brick +bridge +brief +bright +bring +brisk +broccoli +broken +bronze +broom +brother +brown +brush +bubble +buddy +budget +buffalo +build +bulb +bulk +bullet +bundle +bunker +burden +burger +burst +bus +business +busy +butter +buyer +buzz +cabbage +cabin +cable +cactus +cage +cake +call +calm +camera +camp +can +canal +cancel +candy +cannon +canoe +canvas +canyon +capable +capital +captain +car +carbon +card +cargo +carpet +carry +cart +case +cash +casino +castle +casual +cat +catalog +catch +category +cattle +caught +cause +caution +cave +ceiling +celery +cement +census +century +cereal +certain +chair +chalk +champion +change +chaos +chapter +charge +chase +chat +cheap +check +cheese +chef +cherry +chest +chicken +chief +child +chimney +choice +choose +chronic +chuckle +chunk +churn +cigar +cinnamon +circle +citizen +city +civil +claim +clap +clarify +claw +clay +clean +clerk +clever +click +client +cliff +climb +clinic +clip +clock +clog +close +cloth +cloud +clown +club +clump +cluster +clutch +coach +coast +coconut +code +coffee +coil +coin +collect +color +column +combine +come +comfort +comic +common +company +concert +conduct +confirm +congress +connect +consider +control +convince +cook +cool +copper +copy +coral +core +corn +correct +cost +cotton +couch +country +couple +course +cousin +cover +coyote +crack +cradle +craft +cram +crane +crash +crater +crawl +crazy +cream +credit +creek +crew +cricket +crime +crisp +critic +crop +cross +crouch +crowd +crucial +cruel +cruise +crumble +crunch +crush +cry +crystal +cube +culture +cup +cupboard +curious +current +curtain +curve +cushion +custom +cute +cycle +dad +damage +damp +dance +danger +daring +dash +daughter +dawn +day +deal +debate +debris +decade +december +decide +decline +decorate +decrease +deer +defense +define +defy +degree +delay +deliver +demand +demise +denial +dentist +deny +depart +depend +deposit +depth +deputy +derive +describe +desert +design +desk +despair +destroy +detail +detect +develop +device +devote +diagram +dial +diamond +diary +dice +diesel +diet +differ +digital +dignity +dilemma +dinner +dinosaur +direct +dirt +disagree +discover +disease +dish +dismiss +disorder +display +distance +divert +divide +divorce +dizzy +doctor +document +dog +doll +dolphin +domain +donate +donkey +donor +door +dose +double +dove +draft +dragon +drama +drastic +draw +dream +dress +drift +drill +drink +drip +drive +drop +drum +dry +duck +dumb +dune +during +dust +dutch +duty +dwarf +dynamic +eager +eagle +early +earn +earth +easily +east +easy +echo +ecology +economy +edge +edit +educate +effort +egg +eight +either +elbow +elder +electric +elegant +element +elephant +elevator +elite +else +embark +embody +embrace +emerge +emotion +employ +empower +empty +enable +enact +end +endless +endorse +enemy +energy +enforce +engage +engine +enhance +enjoy +enlist +enough +enrich +enroll +ensure +enter +entire +entry +envelope +episode +equal +equip +era +erase +erode +erosion +error +erupt +escape +essay +essence +estate +eternal +ethics +evidence +evil +evoke +evolve +exact +example +excess +exchange +excite +exclude +excuse +execute +exercise +exhaust +exhibit +exile +exist +exit +exotic +expand +expect +expire +explain +expose +express +extend +extra +eye +eyebrow +fabric +face +faculty +fade +faint +faith +fall +false +fame +family +famous +fan +fancy +fantasy +farm +fashion +fat +fatal +father +fatigue +fault +favorite +feature +february +federal +fee +feed +feel +female +fence +festival +fetch +fever +few +fiber +fiction +field +figure +file +film +filter +final +find +fine +finger +finish +fire +firm +first +fiscal +fish +fit +fitness +fix +flag +flame +flash +flat +flavor +flee +flight +flip +float +flock +floor +flower +fluid +flush +fly +foam +focus +fog +foil +fold +follow +food +foot +force +forest +forget +fork +fortune +forum +forward +fossil +foster +found +fox +fragile +frame +frequent +fresh +friend +fringe +frog +front +frost +frown +frozen +fruit +fuel +fun +funny +furnace +fury +future +gadget +gain +galaxy +gallery +game +gap +garage +garbage +garden +garlic +garment +gas +gasp +gate +gather +gauge +gaze +general +genius +genre +gentle +genuine +gesture +ghost +giant +gift +giggle +ginger +giraffe +girl +give +glad +glance +glare +glass +glide +glimpse +globe +gloom +glory +glove +glow +glue +goat +goddess +gold +good +goose +gorilla +gospel +gossip +govern +gown +grab +grace +grain +grant +grape +grass +gravity +great +green +grid +grief +grit +grocery +group +grow +grunt +guard +guess +guide +guilt +guitar +gun +gym +habit +hair +half +hammer +hamster +hand +happy +harbor +hard +harsh +harvest +hat +have +hawk +hazard +head +health +heart +heavy +hedgehog +height +hello +helmet +help +hen +hero +hidden +high +hill +hint +hip +hire +history +hobby +hockey +hold +hole +holiday +hollow +home +honey +hood +hope +horn +horror +horse +hospital +host +hotel +hour +hover +hub +huge +human +humble +humor +hundred +hungry +hunt +hurdle +hurry +hurt +husband +hybrid +ice +icon +idea +identify +idle +ignore +ill +illegal +illness +image +imitate +immense +immune +impact +impose +improve +impulse +inch +include +income +increase +index +indicate +indoor +industry +infant +inflict +inform +inhale +inherit +initial +inject +injury +inmate +inner +innocent +input +inquiry +insane +insect +inside +inspire +install +intact +interest +into +invest +invite +involve +iron +island +isolate +issue +item +ivory +jacket +jaguar +jar +jazz +jealous +jeans +jelly +jewel +job +join +joke +journey +joy +judge +juice +jump +jungle +junior +junk +just +kangaroo +keen +keep +ketchup +key +kick +kid +kidney +kind +kingdom +kiss +kit +kitchen +kite +kitten +kiwi +knee +knife +knock +know +lab +label +labor +ladder +lady +lake +lamp +language +laptop +large +later +latin +laugh +laundry +lava +law +lawn +lawsuit +layer +lazy +leader +leaf +learn +leave +lecture +left +leg +legal +legend +leisure +lemon +lend +length +lens +leopard +lesson +letter +level +liar +liberty +library +license +life +lift +light +like +limb +limit +link +lion +liquid +list +little +live +lizard +load +loan +lobster +local +lock +logic +lonely +long +loop +lottery +loud +lounge +love +loyal +lucky +luggage +lumber +lunar +lunch +luxury +lyrics +machine +mad +magic +magnet +maid +mail +main +major +make +mammal +man +manage +mandate +mango +mansion +manual +maple +marble +march +margin +marine +market +marriage +mask +mass +master +match +material +math +matrix +matter +maximum +maze +meadow +mean +measure +meat +mechanic +medal +media +melody +melt +member +memory +mention +menu +mercy +merge +merit +merry +mesh +message +metal +method +middle +midnight +milk +million +mimic +mind +minimum +minor +minute +miracle +mirror +misery +miss +mistake +mix +mixed +mixture +mobile +model +modify +mom +moment +monitor +monkey +monster +month +moon +moral +more +morning +mosquito +mother +motion +motor +mountain +mouse +move +movie +much +muffin +mule +multiply +muscle +museum +mushroom +music +must +mutual +myself +mystery +myth +naive +name +napkin +narrow +nasty +nation +nature +near +neck +need +negative +neglect +neither +nephew +nerve +nest +net +network +neutral +never +news +next +nice +night +noble +noise +nominee +noodle +normal +north +nose +notable +note +nothing +notice +novel +now +nuclear +number +nurse +nut +oak +obey +object +oblige +obscure +observe +obtain +obvious +occur +ocean +october +odor +off +offer +office +often +oil +okay +old +olive +olympic +omit +once +one +onion +online +only +open +opera +opinion +oppose +option +orange +orbit +orchard +order +ordinary +organ +orient +original +orphan +ostrich +other +outdoor +outer +output +outside +oval +oven +over +own +owner +oxygen +oyster +ozone +pact +paddle +page +pair +palace +palm +panda +panel +panic +panther +paper +parade +parent +park +parrot +party +pass +patch +path +patient +patrol +pattern +pause +pave +payment +peace +peanut +pear +peasant +pelican +pen +penalty +pencil +people +pepper +perfect +permit +person +pet +phone +photo +phrase +physical +piano +picnic +picture +piece +pig +pigeon +pill +pilot +pink +pioneer +pipe +pistol +pitch +pizza +place +planet +plastic +plate +play +please +pledge +pluck +plug +plunge +poem +poet +point +polar +pole +police +pond +pony +pool +popular +portion +position +possible +post +potato +pottery +poverty +powder +power +practice +praise +predict +prefer +prepare +present +pretty +prevent +price +pride +primary +print +priority +prison +private +prize +problem +process +produce +profit +program +project +promote +proof +property +prosper +protect +proud +provide +public +pudding +pull +pulp +pulse +pumpkin +punch +pupil +puppy +purchase +purity +purpose +purse +push +put +puzzle +pyramid +quality +quantum +quarter +question +quick +quit +quiz +quote +rabbit +raccoon +race +rack +radar +radio +rail +rain +raise +rally +ramp +ranch +random +range +rapid +rare +rate +rather +raven +raw +razor +ready +real +reason +rebel +rebuild +recall +receive +recipe +record +recycle +reduce +reflect +reform +refuse +region +regret +regular +reject +relax +release +relief +rely +remain +remember +remind +remove +render +renew +rent +reopen +repair +repeat +replace +report +require +rescue +resemble +resist +resource +response +result +retire +retreat +return +reunion +reveal +review +reward +rhythm +rib +ribbon +rice +rich +ride +ridge +rifle +right +rigid +ring +riot +ripple +risk +ritual +rival +river +road +roast +robot +robust +rocket +romance +roof +rookie +room +rose +rotate +rough +round +route +royal +rubber +rude +rug +rule +run +runway +rural +sad +saddle +sadness +safe +sail +salad +salmon +salon +salt +salute +same +sample +sand +satisfy +satoshi +sauce +sausage +save +say +scale +scan +scare +scatter +scene +scheme +school +science +scissors +scorpion +scout +scrap +screen +script +scrub +sea +search +season +seat +second +secret +section +security +seed +seek +segment +select +sell +seminar +senior +sense +sentence +series +service +session +settle +setup +seven +shadow +shaft +shallow +share +shed +shell +sheriff +shield +shift +shine +ship +shiver +shock +shoe +shoot +shop +short +shoulder +shove +shrimp +shrug +shuffle +shy +sibling +sick +side +siege +sight +sign +silent +silk +silly +silver +similar +simple +since +sing +siren +sister +situate +six +size +skate +sketch +ski +skill +skin +skirt +skull +slab +slam +sleep +slender +slice +slide +slight +slim +slogan +slot +slow +slush +small +smart +smile +smoke +smooth +snack +snake +snap +sniff +snow +soap +soccer +social +sock +soda +soft +solar +soldier +solid +solution +solve +someone +song +soon +sorry +sort +soul +sound +soup +source +south +space +spare +spatial +spawn +speak +special +speed +spell +spend +sphere +spice +spider +spike +spin +spirit +split +spoil +sponsor +spoon +sport +spot +spray +spread +spring +spy +square +squeeze +squirrel +stable +stadium +staff +stage +stairs +stamp +stand +start +state +stay +steak +steel +stem +step +stereo +stick +still +sting +stock +stomach +stone +stool +story +stove +strategy +street +strike +strong +struggle +student +stuff +stumble +style +subject +submit +subway +success +such +sudden +suffer +sugar +suggest +suit +summer +sun +sunny +sunset +super +supply +supreme +sure +surface +surge +surprise +surround +survey +suspect +sustain +swallow +swamp +swap +swarm +swear +sweet +swift +swim +swing +switch +sword +symbol +symptom +syrup +system +table +tackle +tag +tail +talent +talk +tank +tape +target +task +taste +tattoo +taxi +teach +team +tell +ten +tenant +tennis +tent +term +test +text +thank +that +theme +then +theory +there +they +thing +this +thought +three +thrive +throw +thumb +thunder +ticket +tide +tiger +tilt +timber +time +tiny +tip +tired +tissue +title +toast +tobacco +today +toddler +toe +together +toilet +token +tomato +tomorrow +tone +tongue +tonight +tool +tooth +top +topic +topple +torch +tornado +tortoise +toss +total +tourist +toward +tower +town +toy +track +trade +traffic +tragic +train +transfer +trap +trash +travel +tray +treat +tree +trend +trial +tribe +trick +trigger +trim +trip +trophy +trouble +truck +true +truly +trumpet +trust +truth +try +tube +tuition +tumble +tuna +tunnel +turkey +turn +turtle +twelve +twenty +twice +twin +twist +two +type +typical +ugly +umbrella +unable +unaware +uncle +uncover +under +undo +unfair +unfold +unhappy +uniform +unique +unit +universe +unknown +unlock +until +unusual +unveil +update +upgrade +uphold +upon +upper +upset +urban +urge +usage +use +used +useful +useless +usual +utility +vacant +vacuum +vague +valid +valley +valve +van +vanish +vapor +various +vast +vault +vehicle +velvet +vendor +venture +venue +verb +verify +version +very +vessel +veteran +viable +vibrant +vicious +victory +video +view +village +vintage +violin +virtual +virus +visa +visit +visual +vital +vivid +vocal +voice +void +volcano +volume +vote +voyage +wage +wagon +wait +walk +wall +walnut +want +warfare +warm +warrior +wash +wasp +waste +water +wave +way +wealth +weapon +wear +weasel +weather +web +wedding +weekend +weird +welcome +west +wet +whale +what +wheat +wheel +when +where +whip +whisper +wide +width +wife +wild +will +win +window +wine +wing +wink +winner +winter +wire +wisdom +wise +wish +witness +wolf +woman +wonder +wood +wool +word +work +world +worry +worth +wrap +wreck +wrestle +wrist +write +wrong +yard +year +yellow +you +young +youth +zebra +zero +zone +zoo diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/builtins.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/builtins.rs new file mode 100644 index 000000000..b09488025 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/builtins.rs @@ -0,0 +1,124 @@ +//! Rule expressions adapted from yetone/magpie, MIT (see licenses/magpie-MIT). +//! Source: d847b79690cb12951ceefbb53f51993abac76b43, internal/redact/{redact,more,crypto}.go. +//! RE2 ASCII shorthand was made explicit for Rust regex; guards are in guards.rs and wallet.rs. +use super::{Capture, Category, Guard, IdRule, Spec}; + +#[rustfmt::skip] +pub(super) const BUILTINS: &[Spec] = &[ + // redact.go:171 (PRIVATE_KEY) + Spec { category: Category::Secrets, pattern: r#"-----BEGIN (?:[A-Z0-9]+ )*PRIVATE KEY(?: BLOCK)?-----(?s:.)+?-----END (?:[A-Z0-9]+ )*PRIVATE KEY(?: BLOCK)?-----"#, capture: Capture::First, markers: &["PRIVATE KEY"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:172 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"sk-(?:ant-|proj-|or-|svcacct-|admin-)?[A-Za-z0-9_-]{20,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", check: Guard::NotASlug, markers: &["sk-"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:174 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"[a-z]{2,4}_sk_[A-Za-z0-9_-]{20,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", check: Guard::SecretValue, markers: &["_sk_"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:177 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"ark-[A-Za-z0-9][A-Za-z0-9_-]{15,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", check: Guard::SecretValue, markers: &["ark-"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:178 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"(?:gh[pousr]_[A-Za-z0-9]{30,}|github_pat_[A-Za-z0-9_]{40,}|glpat-[A-Za-z0-9_-]{20,})"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["gh", "github_pat_", "glpat-"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:179 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"AIza[0-9A-Za-z_-]{35}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["AIza"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:180 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"xox[abposr]-[0-9A-Za-z-]{10,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["xox"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:181 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"(?:sk|rk)_live_[0-9A-Za-z]{20,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["_live_"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:182 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"(?:AKIA|ASIA)[0-9A-Z]{16}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["AKIA", "ASIA"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:183 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"(?:hf_[A-Za-z0-9]{30,}|gsk_[A-Za-z0-9]{40,}|xai-[A-Za-z0-9]{40,}|npm_[A-Za-z0-9]{36}|pypi-[A-Za-z0-9_-]{50,}|dop_v1_[a-f0-9]{64}|SG\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]{43})"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["hf_", "gsk_", "xai-", "npm_", "pypi-", "dop_v1_", "SG."], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:194 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"cf(?:ut|at|k|oat)_[A-Za-z0-9_-]{40,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["cfut_", "cfat_", "cfk_", "cfoat_"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:195 (API_KEY) + Spec { category: Category::Secrets, pattern: r#"v1\.0-[A-Za-z0-9]{16,}-[A-Za-z0-9]{40,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["v1.0-"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:196 (TOKEN) + Spec { category: Category::Secrets, pattern: r#"eyJhIjoi[A-Za-z0-9+/_-]{40,}={0,2}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["eyJhIjoi"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:199 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?i)x-auth-(?:user-service-)?key["']?[ \t]*[:=][ \t]*["']?([A-Za-z0-9._-]{16,})"#, capture: Capture::First, check: Guard::NotAVariable, fold: &["x-auth-"], id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:204 (TOKEN) + Spec { category: Category::Secrets, pattern: r#"eyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", markers: &["eyJ"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:207 (PASSWORD) + Spec { category: Category::Secrets, pattern: r#"[A-Za-z][A-Za-z0-9+.-]*://[^ \t\n\r\x0C:@/?#"'<>]*:([^ \t\n\r\x0C@/?#"'<>]{3,})@"#, capture: Capture::First, check: Guard::NotAVariable, context: Guard::HiddenInURL, markers: &["://"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:209 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?i)[A-Za-z0-9_.-]*(?:password|passwd|passphrase|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|priv[_-]?key|credential)[A-Za-z0-9_.-]*(?:\\?["'])?[ \t]*[:=][ \t]*(?:\\?["'])?([A-Za-z0-9_\-./+=~!@#%^&*:]{8,})"#, capture: Capture::First, check: Guard::AssignedSecret, context: Guard::FieldAt, field: true, id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:217 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?:^|[^A-Za-z0-9_.-])[A-Za-z0-9_.-]*_(?:KEY|PASS)(?:\\?["'])?[ \t]*[:=][ \t]*(?:\\?["'])?([A-Za-z0-9_\-./+=~!@#%^&*:]{8,})"#, capture: Capture::First, check: Guard::AssignedSecret, context: Guard::NamedSecretAt, field: true, id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:221 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?im)^[ \t]*(?:authorization|cookie)[ \t]*:[ \t]*([^ \t\n\r\x0C][^\r\n]{7,})"#, capture: Capture::First, check: Guard::HeaderValue, id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:226 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?i)(?:^|[^A-Za-z0-9_.-])(?:\\?["'])?key(?:\\?["'])?[ \t]*[:=][ \t]*(?:\\?["'])?([A-Za-z0-9_\-./+=~!@#%^&*:]{16,})"#, capture: Capture::First, check: Guard::AssignedSecret, context: Guard::FieldAt, field: true, id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:233 (PASSWORD) + Spec { category: Category::Secrets, pattern: r#"(?i)(?:密码|密碼|口令|PIN码|PIN碼|PIN 码|安全码|安全碼|パスワード|暗証番号|비밀번호|암호|비번)[ \t]*(?:[::=][ \t]*|[为為是][ \t]*[::]?[ \t]*|は[ \t]*|는[ \t]*)?(?:\\?["'“「])?([A-Za-z0-9_\-./+=~!@#%^&*:]{3,63}[A-Za-z0-9_\-/+=~!@#%^&*:])"#, capture: Capture::First, check: Guard::NotAVariable, field: true, id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // redact.go:238 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?i)(?:密钥|密鑰|秘钥|秘鑰|私钥|私鑰|金钥|金鑰|令牌|权杖|權杖|授权码|授權碼|秘密鍵|トークン|シークレット|APIキー|アクセスキー|토큰|비밀키|API 키|API키)[ \t]*(?:[::=][ \t]*|[为為是][ \t]*[::]?[ \t]*|は[ \t]*|는[ \t]*)?(?:\\?["'“「])?([A-Za-z0-9_\-./+=~!@#%^&*:]{3,63}[A-Za-z0-9_\-/+=~!@#%^&*:])"#, capture: Capture::First, context: Guard::KeyAt, field: true, id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:242 (SECRET) + Spec { category: Category::Secrets, pattern: r#"(?i)(?:password|passwd|passphrase|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|priv[_-]?key|credential|pwd|api[ ]key|access[ ]key|secret[ ]key|key|(?-u:\b)pin(?:[ ]?code)?)(?:\\?["'])?[ \t]*(?:[为為是][ \t]*[::]?|:)[ \t]*(?:\\?["'“「])?([A-Za-z0-9_\-./+=~!@#%^&*:]{3,}[A-Za-z0-9_\-/+=~!@#%^&*:])"#, capture: Capture::First, context: Guard::KeyAt, field: true, fold: &["为", "為", "是", ":"], id_rule: IdRule::Assignment, ..Spec::DEFAULT }, + // redact.go:246 (EMAIL) + Spec { category: Category::Personal, pattern: r#"[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(?:\.[A-Za-z0-9-]+)*\.[A-Za-z]{2,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_%+-", check: Guard::RealEmail, context: Guard::EmailAt, markers: &["@"], ..Spec::DEFAULT }, + // redact.go:247 (ID_CARD) + Spec { category: Category::Personal, pattern: r#"[1-9][0-9]{5}(?:18|19|20)[0-9]{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12][0-9]|3[01])[0-9]{3}[0-9Xx]"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::ChineseID, ..Spec::DEFAULT }, + // redact.go:248 (PHONE) + Spec { category: Category::Personal, pattern: r#"(?:\+?86[- ]?)?1[3-9][0-9]{9}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", context: Guard::PhoneAt, ..Spec::DEFAULT }, + // redact.go:249 (BANK_CARD) + Spec { category: Category::Personal, pattern: r#"[3-6][0-9]{3}(?:[ -]?[0-9]{4}){2,3}(?:[ -]?[0-9]{1,3})?"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::Luhn, ..Spec::DEFAULT }, + // more.go:85 (SSN) + Spec { category: Category::Ssn, pattern: r#"[0-9]{3}-[0-9]{2}-[0-9]{4}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-", check: Guard::SsnOK, markers: &["-"], ..Spec::DEFAULT }, + // more.go:86 (SSN) + Spec { category: Category::Ssn, pattern: r#"(?i:(?-u:\b)ssn|(?-u:\b)social[ \t]+security(?:[ \t]+(?:number|no\.?|#))?)["']?[ \t]*(?:[:#=:][ \t]*)?(?:(?i:is)[ \t]+)?["']?([0-9]{9}|[0-9]{3} [0-9]{2} [0-9]{4})"#, capture: Capture::First, bound: "0123456789", check: Guard::SsnOK, fold: &["ssn", "social"], ..Spec::DEFAULT }, + // more.go:89 (PASSPORT) + Spec { category: Category::Passport, pattern: r#"(?i:(?-u:\b)passport(?:[ \t]*(?:no\.?|number|num|#))?|(?-u:\b)reisepass(?:nummer)?)["']?[ \t]*(?:[:#=:][ \t]*)?(?:(?i:is|was)[ \t]+)?["']?([A-Z0-9]{6,9})|(?:护照(?:号码|号)?|護照(?:號碼|号码|番号)?|パスポート(?:番号)?)[ \t]*(?:[::=][ \t]*)?(?:是[ \t]*|は[ \t]*)?["']?([A-Z0-9]{6,9})"#, capture: Capture::Any, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::PassportOK, fold: &["passport", "reisepass", "护照", "護照", "パスポート"], ..Spec::DEFAULT }, + // more.go:95 (IBAN) + Spec { category: Category::Iban, pattern: r#"[A-Z]{2}[0-9]{2}(?:[A-Z0-9]{11,30}|(?: [A-Z0-9]{4}){2,7}(?: [A-Z0-9]{1,3})?)"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", repair: Guard::IbanAt, ..Spec::DEFAULT }, + // more.go:98 (BIRTHDAY) + Spec { category: Category::Birthday, pattern: r#"(?:(?i:(?-u:\b)date[ \t]+of[ \t]+birth|(?-u:\b)birth[ \t_-]?date|(?-u:\b)birthday|(?-u:\b)dob(?-u:\b)|(?-u:\b)d\.o\.b\.?|(?-u:\b)born(?:[ \t]+on)?|(?-u:\b)geburtsdatum|(?-u:\b)geboren(?:[ \t]+am)?)|生日|出生日期|出生年月日|出生于|出生於|出生|誕生日|生年月日)["']?[ \t]*(?:[:#=:][ \t]*)?(?:(?i:is|was)[ \t]+|是[ \t]*|为[ \t]*|為[ \t]*|は[ \t]*)?["']?([0-9]{4}[-/.][0-9]{1,2}[-/.][0-9]{1,2}|[0-9]{4}[ \t]*年[ \t]*[0-9]{1,2}[ \t]*月[ \t]*[0-9]{1,2}[ \t]*日?|[0-9]{1,2}[-/.][0-9]{1,2}[-/.][0-9]{4}|(?i:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?[ \t]+[0-9]{1,2}(?:st|nd|rd|th)?,?[ \t]+[0-9]{4}|[0-9]{1,2}(?:st|nd|rd|th)?[ \t]+(?i:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?,?[ \t]+[0-9]{4})"#, capture: Capture::First, bound: "0123456789", check: Guard::DateOK, fold: &["birth", "dob", "d.o.b", "born", "geburt", "geboren", "生日", "出生", "誕生日", "生年月日"], ..Spec::DEFAULT }, + // more.go:107 (MAC) + Spec { category: Category::Mac, pattern: r#"[0-9A-Fa-f]{2}(?::[0-9A-Fa-f]{2}){5}|[0-9A-Fa-f]{2}(?:-[0-9A-Fa-f]{2}){5}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789:-", check: Guard::MacOK, ..Spec::DEFAULT }, + // more.go:111 (SERIAL) + Spec { category: Category::Serial, pattern: r#"(?i:(?-u:\b)serial[ \t_-]?(?:number|num|no\.?|#)?|(?-u:\b)s/n|(?-u:\b)sn)(?:[ \t]*\([^)\n]{1,20}\))?["']?[ \t]*[:#=:][ \t]*["']?([A-Za-z0-9][A-Za-z0-9-]{4,39})|(?:序列号|序列號|シリアル番号|(?i:seriennummer))[ \t]*[::=]?[ \t]*["']?([A-Za-z0-9][A-Za-z0-9-]{4,39})"#, capture: Capture::Any, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-", check: Guard::SerialOK, fold: &["serial", "s/n", "sn", "序列", "シリアル", "seriennummer"], ..Spec::DEFAULT }, + // more.go:118 (USERNAME) + Spec { category: Category::Home, pattern: r#"(?:^|[^A-Za-z0-9_.~/\\-])(?:file://)?(?:/mnt/[a-z])?/(?:Users|home)/([^ \t\n\r\x0C/\\:*?"'<>|,;()\[\]{}`$%]{1,64})"#, capture: Capture::First, repair: Guard::HomeAt, fold: &["/users/", "/home/"], ..Spec::DEFAULT }, + // more.go:121 (USERNAME) + Spec { category: Category::Home, pattern: r#"(?:^|[^A-Za-z0-9])[A-Za-z]:(?:\\{1,4}|/)(?i:users|documents and settings)(?:\\{1,4}|/)(?:([A-Za-z0-9._-]+(?: [A-Za-z0-9._-]+)+)(?:\\|/)|([^ \t\n\r\x0C/\\:*?"'<>|,;()\[\]{}`$%]{1,64}))"#, capture: Capture::Any, repair: Guard::HomeAt, fold: &["users", "documents and settings"], ..Spec::DEFAULT }, + // more.go:130 (USERNAME, HOSTNAME) + Spec { category: Category::Userhost, pattern: r#"(?m)(?:^|[^A-Za-z0-9._%+-])([a-z_][a-z0-9_.-]{0,31})@([A-Za-z0-9][A-Za-z0-9-]{0,62}(?:\.[A-Za-z0-9-]{1,63})*)(?::[~/]|:[ \t]*$|[ \t]+[~/]|[ \t]*[$%#>][ \t]|[ \t]*[$%#>]$)"#, capture: Capture::Groups(&[Guard::UserOK, Guard::HostOK]), context: Guard::NotPackagePin, markers: &["@"], ..Spec::DEFAULT }, + // more.go:132 (USERNAME, HOSTNAME) + Spec { category: Category::Userhost, pattern: r#"(?i:(?-u:\b)(?:ssh|scp|sftp|rsync|mosh|ssh-copy-id))(?-u:\b)[^\n@]{0,80}?[ \t"']([a-z_][a-z0-9_.-]{0,31})@([A-Za-z0-9][A-Za-z0-9-]{0,62}(?:\.[A-Za-z0-9-]{1,63})*)"#, capture: Capture::Groups(&[Guard::UserOK, Guard::HostOK]), context: Guard::NotPackagePin, markers: &["@"], ..Spec::DEFAULT }, + // more.go:136 (HOSTNAME) + Spec { category: Category::Userhost, pattern: r#"(?m)^[ \t]*["']?(?:(?i:host[ \t_-]?name|computer[ \t_-]?name|machine[ \t_-]?name|device[ \t_-]?name)|主机名|主機名|計算機名|コンピュータ名|コンピューター名)["']?[ \t]*[:=:][ \t]*(?:["']([A-Za-z0-9][A-Za-z0-9-]{0,62}(?:\.[A-Za-z0-9-]{1,63})*)["'][ \t]*[,;]?|([A-Za-z0-9][A-Za-z0-9-]{0,62}(?:\.[A-Za-z0-9-]{1,63})*))[ \t]*$|^[ \t]+HostName[ \t]+([A-Za-z0-9][A-Za-z0-9-]{0,62}(?:\.[A-Za-z0-9-]{1,63})*)[ \t]*$"#, capture: Capture::Any, check: Guard::LineHostOK, fold: &["name", "主机", "主機", "計算機", "コンピュータ"], ..Spec::DEFAULT }, + // more.go:142 (USERNAME) + Spec { category: Category::Userhost, pattern: r#"(?m)^[ \t]*["']?(?:(?i:user[ \t_-]?name|login[ \t_-]?name|user|login|logname)|用户名|用戶名|ユーザー名)["']?[ \t]*[:=:][ \t]*(?:["']([A-Za-z0-9_][A-Za-z0-9._-]{0,31})["'][ \t]*[,;]?|([A-Za-z0-9_][A-Za-z0-9._-]{0,31}))[ \t]*$|^[ \t]+User[ \t]+([A-Za-z0-9_][A-Za-z0-9._-]{0,31})[ \t]*$"#, capture: Capture::Any, check: Guard::LineUserOK, context: Guard::NotLineIdent, fold: &["user", "login", "logname", "用户", "用戶", "ユーザー"], ..Spec::DEFAULT }, + // more.go:149 (IP) + Spec { category: Category::Ip, pattern: r#"(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])(?:\.(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])){3}"#, capture: Capture::First, context: Guard::Ipv4At, markers: &["."], ..Spec::DEFAULT }, + // more.go:151 (IP) + Spec { category: Category::Ip, pattern: r#"[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{0,4}){2,7}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789:_", check: Guard::Ipv6OK, markers: &[":"], ..Spec::DEFAULT }, + // more.go:156 (BUCKET, BUCKET) + Spec { category: Category::Bucket, pattern: r#"(?i:(?-u:\b)(?:s3a?|s3n|gs|oss|cosn?|obs|r2|ks3|bos|tos|az|azure|wasbs?|abfss?)://)([A-Za-z0-9](?:[A-Za-z0-9._-]{0,61}[A-Za-z0-9])?)(?:@([a-z0-9]{3,24})\.(?:blob|dfs)\.core\.windows\.net)?"#, capture: Capture::Groups(&[Guard::BucketOK, Guard::BucketOK]), markers: &["://"], ..Spec::DEFAULT }, + // more.go:158 (BUCKET) + Spec { category: Category::Bucket, pattern: r#"(?:^|[^A-Za-z0-9.-])([a-z0-9](?:[a-z0-9.-]{0,61}[a-z0-9])?)\.(?:s3(?:[.-](?:dualstack\.)?[a-z0-9-]+)?\.amazonaws\.com(?:\.cn)?|s3-accelerate\.amazonaws\.com|storage\.googleapis\.com|oss-[a-z0-9-]+\.aliyuncs\.com|cos\.[a-z0-9-]+\.myqcloud\.com|obs\.[a-z0-9-]+\.myhuaweicloud\.com|r2\.cloudflarestorage\.com|(?:blob|dfs)\.core\.windows\.net|[a-z0-9-]+\.digitaloceanspaces\.com|tos-[a-z0-9-]+\.volces\.com)"#, capture: Capture::First, check: Guard::BucketOK, fold: &["amazonaws.com", "googleapis.com", "cloud.google.com", "aliyuncs.com", "myqcloud.com", "windows.net", "cloudflarestorage.com", "digitaloceanspaces.com", "myhuaweicloud.com", "volces.com"], ..Spec::DEFAULT }, + // more.go:162 (BUCKET) + Spec { category: Category::Bucket, pattern: r#"(?:^|[^A-Za-z0-9.-])(?:s3(?:[.-](?:dualstack\.)?[a-z0-9-]+)?\.amazonaws\.com(?:\.cn)?|storage\.googleapis\.com|storage\.cloud\.google\.com|[a-z0-9]{3,24}\.blob\.core\.windows\.net|oss-[a-z0-9-]+\.aliyuncs\.com|[a-z0-9-]+\.digitaloceanspaces\.com)/([A-Za-z0-9](?:[A-Za-z0-9._-]{0,61}[A-Za-z0-9])?)"#, capture: Capture::First, check: Guard::BucketOK, fold: &["amazonaws.com", "googleapis.com", "cloud.google.com", "aliyuncs.com", "myqcloud.com", "windows.net", "cloudflarestorage.com", "digitaloceanspaces.com", "myhuaweicloud.com", "volces.com"], ..Spec::DEFAULT }, + // crypto.go:50 (WALLET_KEY) + Spec { category: Category::Secrets, pattern: r#"(?i)0x[0-9a-f]{56,64}|[0-9a-f]{64}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", context: Guard::WalletKeyAt, fold: &["priv", "secret", "signer", "signing", "deployer", "wallet", "pk", "私钥", "私鑰", "秘密鍵", "개인키", "비밀키"], id_rule: IdRule::Credential, ..Spec::DEFAULT }, + // crypto.go:55 (WIF_KEY) + Spec { category: Category::Secrets, pattern: r#"[1-9A-HJ-NP-Za-km-z]{51,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::WifKey, ..Spec::DEFAULT }, + // crypto.go:61 (BIP32_KEY) + Spec { category: Category::Secrets, pattern: r#"[xyztuvYZUV]prv[1-9A-HJ-NP-Za-km-z]{107}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::Bip32Key, markers: &["prv"], ..Spec::DEFAULT }, + // crypto.go:66 (SOLANA_KEY) + Spec { category: Category::Secrets, pattern: r#"[1-9A-HJ-NP-Za-km-z]{85,}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::SolanaKey, ..Spec::DEFAULT }, + // crypto.go:71 (SOLANA_KEYPAIR) + Spec { category: Category::Secrets, pattern: r#"\[[ \t\n\r\x0C]*[0-9]{1,3}(?:[ \t\n\r\x0C]*,[ \t\n\r\x0C]*[0-9]{1,3}){63}[ \t\n\r\x0C]*\]"#, capture: Capture::First, check: Guard::SolanaKeypair, markers: &["["], ..Spec::DEFAULT }, + // crypto.go:76 (NEAR_KEY) + Spec { category: Category::Secrets, pattern: r#"ed25519:[1-9A-HJ-NP-Za-km-z]{86,88}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["ed25519:"], ..Spec::DEFAULT }, + // crypto.go:80 (SUI_KEY) + Spec { category: Category::Secrets, pattern: r#"suiprivkey1[0-9a-z]{58}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["suiprivkey1"], ..Spec::DEFAULT }, + // crypto.go:84 (APTOS_KEY) + Spec { category: Category::Secrets, pattern: r#"(?:ed25519|secp256k1)-priv-0x[0-9a-fA-F]{64}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["-priv-0x"], ..Spec::DEFAULT }, + // crypto.go:89 (STELLAR_KEY) + Spec { category: Category::Secrets, pattern: r#"S[A-Z2-7]{55}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", check: Guard::StellarSeed, ..Spec::DEFAULT }, + // crypto.go:93 (NOSTR_KEY) + Spec { category: Category::Secrets, pattern: r#"nsec1[02-9ac-hj-np-z]{58}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["nsec1"], ..Spec::DEFAULT }, + // crypto.go:97 (AGE_KEY) + Spec { category: Category::Secrets, pattern: r#"AGE-SECRET-KEY-1[0-9A-Z]{58}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["AGE-SECRET-KEY-"], ..Spec::DEFAULT }, + // crypto.go:101 (ZCASH_KEY) + Spec { category: Category::Secrets, pattern: r#"secret-extended-key-(?:main|test)1[0-9a-z]{150,320}"#, capture: Capture::First, bound: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789", markers: &["secret-extended-key-"], ..Spec::DEFAULT }, + // crypto.go:107 (SEED_PHRASE) + Spec { category: Category::Secrets, pattern: r#"(?i)(?:seed|mnemonic|phrase|recovery|助记词|助記詞|シード|ニーモニック|시드|니모닉)[^.\n]{0,24}?(?:\n[ \t]*)?(?-u:\b)([a-z]{3,8}(?:[ \t]+[a-z]{3,8}){11,40})(?-u:\b)"#, capture: Capture::First, repair: Guard::SeedPhraseAt, fold: &["seed", "mnemonic", "phrase", "recovery", "助记词", "助記詞", "シード", "ニーモニック", "시드", "니모닉"], ..Spec::DEFAULT }, +]; diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guard_data.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guard_data.rs new file mode 100644 index 000000000..027373878 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guard_data.rs @@ -0,0 +1,402 @@ +//! Guard dictionaries from the pinned Magpie sources (MIT, see licenses/magpie-MIT). +pub(super) const NOT_NAMES: &[&str] = &[ + "shared", + "public", + "guest", + "default", + "default user", + "all users", + "user", + "username", + "user_name", + "yourname", + "your-name", + "your_name", + "yourusername", + "you", + "me", + "name", + "example", + "foo", + "bar", + "someone", + "john", + "jdoe", + "runner", + "node", + "vscode", + "codespace", + "linuxbrew", + "...", +]; +pub(super) const NOT_USERS: &[&str] = &[ + "git", + "root", + "ec2-user", + "ubuntu", + "admin", + "postgres", + "www-data", + "nobody", + "centos", + "debian", + "fedora", + "azureuser", + "opc", + "core", + "vagrant", + "docker", + "str", + "string", + "int", + "bool", + "any", + "none", + "null", + "nil", + "true", + "false", + "undefined", + "hi", + "hello", + "hey", + "yes", + "no", + "ok", + "okay", + "thanks", + "continue", + "go", + "stop", + "help", + "test", + "y", + "n", +]; +pub(super) const CODE_OBJECTS: &[&str] = &[ + "self", "this", "os", "process", "request", "req", "ctx", "context", "config", "cfg", "conf", + "settings", "env", "args", "opts", "options", "props", "params", "session", "socket", + "platform", "window", "location", "url", "u", "r", "c", "s", "m", "p", "x", "data", "info", + "user", +]; +pub(super) const IDENT_PARTS: &[&str] = &[ + "user", "users", "usr", "username", "name", "names", "login", "logname", "account", "acct", + "owner", "author", "member", "person", "profile", "current", "cur", "default", "target", "new", + "old", "my", "the", "obj", "object", "info", "id", "model", "data", "record", "entity", "row", + "item", "input", "arg", "args", "param", "value", "val", "str", "type", "dto", "db", "auth", + "session", "self", +]; +pub(super) const CHAT_WORDS: &[&str] = &[ + "done", + "why", + "what", + "how", + "sure", + "please", + "retry", + "next", + "again", + "great", + "good", + "nice", + "cool", + "fix", + "go-ahead", + "proceed", + "lgtm", + "required", + "optional", + "unknown", + "anonymous", + "guest", + "current", + "me", +]; +pub(super) const PUBLIC_HOSTS: &[&str] = &[ + "github.com", + "gitlab.com", + "bitbucket.org", + "ssh.dev.azure.com", + "vs-ssh.visualstudio.com", + "localhost", + "host", + "hostname", + "server", + "remote", + "example", + "your-server", + "str", + "string", + "none", + "null", + "true", + "false", + "undefined", +]; +pub(super) const RESOLVERS: &[&str] = &[ + "8.8.8.8", + "8.8.4.4", + "1.1.1.1", + "1.0.0.1", + "9.9.9.9", + "149.112.112.112", + "208.67.222.222", + "208.67.220.220", + "114.114.114.114", + "223.5.5.5", + "223.6.6.6", + "119.29.29.29", + "180.76.76.76", + "2001:4860:4860::8888", + "2001:4860:4860::8844", + "2606:4700:4700::1111", + "2606:4700:4700::1001", +]; +pub(super) const NOT_BUCKETS: &[&str] = &[ + "bucket", + "my-bucket", + "mybucket", + "bucket-name", + "bucketname", + "my-bucket-name", + "your-bucket", + "your-bucket-name", + "yourbucket", + "example-bucket", + "examplebucket", + "amzn-s3-demo-bucket", + "doc-example-bucket", + "s3", + "www", + "storage", + "container", + "mycontainer", + "my-container", + "mystorageaccount", + "account", + "upload", + "download", + "batch", + "b", + "v1", + "v2", + "api", +]; +pub(super) const IBAN_LEN: &[(&str, usize)] = &[ + ("AD", 24), + ("AE", 23), + ("AL", 28), + ("AT", 20), + ("AZ", 28), + ("BA", 20), + ("BE", 16), + ("BG", 22), + ("BH", 22), + ("BI", 27), + ("BR", 29), + ("BY", 28), + ("CH", 21), + ("CR", 22), + ("CY", 28), + ("CZ", 24), + ("DE", 22), + ("DJ", 27), + ("DK", 18), + ("DO", 28), + ("EE", 20), + ("EG", 29), + ("ES", 24), + ("FI", 18), + ("FK", 18), + ("FO", 18), + ("FR", 27), + ("GB", 22), + ("GE", 22), + ("GI", 23), + ("GL", 18), + ("GR", 27), + ("GT", 28), + ("HN", 28), + ("HR", 21), + ("HU", 28), + ("IE", 22), + ("IL", 23), + ("IQ", 23), + ("IS", 26), + ("IT", 27), + ("JO", 30), + ("KW", 30), + ("KZ", 20), + ("LB", 28), + ("LC", 32), + ("LI", 21), + ("LT", 20), + ("LU", 20), + ("LV", 21), + ("LY", 25), + ("MC", 27), + ("MD", 24), + ("ME", 22), + ("MK", 19), + ("MN", 20), + ("MR", 27), + ("MT", 31), + ("MU", 30), + ("NI", 28), + ("NL", 18), + ("NO", 15), + ("OM", 23), + ("PK", 24), + ("PL", 28), + ("PS", 29), + ("PT", 25), + ("QA", 29), + ("RO", 24), + ("RS", 22), + ("RU", 33), + ("SA", 24), + ("SC", 31), + ("SD", 18), + ("SE", 24), + ("SI", 19), + ("SK", 24), + ("SM", 27), + ("SO", 23), + ("ST", 25), + ("SV", 28), + ("TL", 23), + ("TN", 24), + ("TR", 26), + ("UA", 29), + ("VA", 22), + ("VG", 24), + ("XK", 20), + ("YE", 30), +]; +pub(super) const LINKER_WORDS: &[&str] = &["plt", "got", "sec", "iplt", "igot"]; +pub(super) const DOC_FILES: &[&str] = &[ + "agents.md", + "claude.md", + "gemini.md", + "readme.md", + "skill.md", + "changelog.md", + "contributing.md", + "license.md", + "lessons.md", + "todo.md", + "copilot-instructions.md", + "crush.md", + "qwen.md", + "iflow.md", + "warp.md", +]; +pub(super) const LOCAL_TLDS: &[&str] = &[ + "local", + "internal", + "lan", + "corp", + "home", + "intranet", + "localdomain", +]; +pub(super) const FILE_EXT: &[&str] = &[ + "jpg", "jpeg", "png", "gif", "webp", "svg", "heic", "heif", "avif", "bmp", "tif", "tiff", + "ico", "icns", "pdf", "svgz", "apng", "jxl", "psd", "eps", "lottie", "mp4", "m4v", "webm", + "mkv", "avi", "mp3", "m4a", "wav", "ogg", "flac", "aac", "ttf", "otf", "woff", "woff2", "eot", + "json", "yaml", "yml", "toml", "xml", "txt", "csv", "tsv", "log", "html", "htm", "css", "scss", + "less", "js", "mjs", "cjs", "jsx", "ts", "tsx", "doc", "docx", "xls", "xlsx", "ppt", "pptx", + "gz", "tgz", "tar", "rar", "7z", "dmg", +]; +pub(super) const SCALE_EXT: &[&str] = &["zip", "mov", "md", "map"]; +pub(super) const SOURCE_EXT: &[&str] = &[ + "go", "ts", "tsx", "mts", "cts", "js", "jsx", "mjs", "cjs", "py", "pyi", "rb", "rs", "java", + "kt", "kts", "swift", "c", "h", "cc", "cpp", "cxx", "hpp", "hh", "m", "mm", "cs", "fs", "php", + "scala", "dart", "lua", "zig", "nim", "ex", "exs", "erl", "hs", "ml", "clj", "r", "jl", "sh", + "bash", "zsh", "fish", "ps1", "vue", "svelte", "astro", "md", "mdx", "rst", "sql", "proto", + "graphql", "gradle", "html", "css", "scss", "less", "json", "jsonc", "yaml", "yml", "toml", + "xml", "tf", "ipynb", +]; +pub(super) const PATH_ROOTS: &[&str] = &[ + "Users", + "home", + "tmp", + "private", + "var", + "etc", + "run", + "opt", + "usr", + "mnt", + "Volumes", + "root", + "srv", + "data", + "workspace", + "workspaces", + "app", + "Applications", + "Library", + "System", + "proc", + "dev", + "sys", + "nix", +]; +pub(super) const SECRET_WORDS: &[&str] = &[ + "token", + "tokens", + "secret", + "secrets", + "password", + "passwords", + "passwd", + "pwd", + "key", + "keys", + "apikey", + "credential", + "credentials", +]; +pub(super) const PHONE_WORDS: &[&str] = &[ + "phone", + "phones", + "mobile", + "mobiles", + "tel", + "telephone", + "cell", + "cellphone", + "contact", + "contacts", + "whatsapp", + "sms", + "msisdn", + "fax", +]; +pub(super) const PHONE_MARKS: &[&str] = &["手机", "电话", "電話", "联系", "聯繫", "联络"]; +pub(super) const ID_WORDS: &[&str] = &[ + "order", + "id", + "uid", + "no", + "no.", + "#", + "number", + "ticket", + "invoice", + "订单", + "订单号", + "单号", + "编号", + "流水号", + "工单", + "工单号", + "运单号", + "快递单号", + "订单编号", + "交易号", + "序号", + "账单号", + "id号", +]; diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guards.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guards.rs new file mode 100644 index 000000000..0c7d19624 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/guards.rs @@ -0,0 +1,829 @@ +//! Format validation and contextual exclusions, adapted from Magpie (MIT), without storing originals. +use super::guard_data::*; +use super::{wallet, Guard}; +use chrono::{Datelike, NaiveDate, Utc}; +use regex::Regex; +use std::{net::IpAddr, sync::LazyLock}; + +pub(super) fn before(s: &str, a: usize, n: usize) -> usize { + let mut at = a.saturating_sub(n); + while !s.is_char_boundary(at) { + at += 1; + } + at +} + +pub(super) fn adjacent(s: &str, a: usize, b: usize, bound: &str) -> bool { + a.checked_sub(1) + .is_some_and(|i| bound.as_bytes().contains(&s.as_bytes()[i])) + || s.as_bytes() + .get(b) + .is_some_and(|c| bound.as_bytes().contains(c)) +} + +pub(super) fn word_byte(c: u8) -> bool { + c.is_ascii_alphanumeric() || c == b'_' +} +fn numeric(c: Option<&u8>) -> bool { + c.is_some_and(u8::is_ascii_digit) +} + +pub(super) fn repair(guard: Guard, s: &str, a: usize, mut b: usize) -> Option<(usize, usize)> { + match guard { + Guard::IbanAt => loop { + if iban(&s[a..b]) { + break Some((a, b)); + } + b = a + s[a..b].rfind(' ')?; + }, + Guard::HomeAt => { + let name = s[a..b].trim_end_matches('.'); + (!name.is_empty() + && !name.starts_with('.') + && !NOT_NAMES.contains(&name.to_ascii_lowercase().as_str())) + .then_some((a, a + name.len())) + } + Guard::None => Some((a, b)), + _ => None, + } +} + +pub(super) fn accept(guard: Guard, v: &str) -> bool { + if v.is_empty() { + return false; + } + match guard { + Guard::None => true, + Guard::Bip32Key => wallet::bip32(v), + Guard::BucketOK => { + v.len() >= 3 + && !v.bytes().all(|b| b.is_ascii_digit()) + && !NOT_BUCKETS.contains(&v.to_ascii_lowercase().as_str()) + } + Guard::ChineseID => chinese_id(v), + Guard::DateOK => date_ok(v), + Guard::HeaderValue => header(v), + Guard::HostOK => host(v), + Guard::Ipv6OK => { + v.matches(':').count() >= 2 + && v.parse::() + .is_ok_and(|a| a.is_ipv6() && public_ip(a)) + } + Guard::LineHostOK => host(v) && !in_code(v), + Guard::LineUserOK => { + user(v) + && !in_code(v) + && !CHAT_WORDS.contains(&v.to_ascii_lowercase().as_str()) + && !IDENT_PARTS.contains(&v.to_ascii_lowercase().as_str()) + } + Guard::Luhn => luhn(v), + Guard::MacOK => mac(v), + Guard::NotASlug => not_slug(v), + Guard::NotAVariable => not_variable(v), + Guard::PassportOK => { + v.bytes().filter(|c| c.is_ascii_digit()).count() >= 5 && not_repeated(v) + } + Guard::RealEmail => email(v), + Guard::SecretValue => secret(v), + Guard::AssignedSecret => secret_field_value(v) && !path_value(v) && !in_code(v), + Guard::SerialOK => v.len() >= 6 && v.bytes().any(|c| c.is_ascii_digit()) && not_repeated(v), + Guard::SolanaKey => wallet::solana(v), + Guard::SolanaKeypair => wallet::solana_array(v), + Guard::SsnOK => ssn(v), + Guard::StellarSeed => wallet::stellar(v), + Guard::UserOK => user(v), + Guard::WifKey => wallet::wif(v), + _ => false, + } +} + +pub(super) fn context(guard: Guard, s: &str, a: usize, b: usize) -> bool { + match guard { + Guard::None => true, + Guard::EmailAt => { + !s[..a].ends_with('.') + && !(s.as_bytes().get(b) == Some(&b'.') + && s.as_bytes() + .get(b + 1) + .is_some_and(|&c| word_byte(c) || c == b'-')) + && !in_userinfo(s, a) + } + Guard::FieldAt => field_at(s, a, b), + Guard::HiddenInURL => hidden_in_url(s, a, b), + Guard::Ipv4At => ipv4(s, a, b), + Guard::KeyAt => key_at(s, a, b), + Guard::NamedSecretAt => { + field_at(s, a, b) && field_name(s, a).is_none_or(|name| secret_field(name)) + } + Guard::NotLineIdent => { + let words = ident_words(&s[a..b]); + (s[..a].ends_with(['\'', '"'])) + || words.len() < 2 + || !words + .iter() + .any(|w| IDENT_PARTS.contains(&w.to_ascii_lowercase().as_str())) + } + Guard::NotPackagePin => { + !s[b..] + .strip_prefix('@') + .is_some_and(|t| VERSION.is_match(t.trim_end_matches('.'))) + && !(s[..a].ends_with('@') + && VERSION.is_match(&s[a..b]) + && s[a..b].parse::().is_err()) + } + Guard::PhoneAt => phone(s, a, b), + Guard::WalletKeyAt => wallet::wallet_context(s, a), + _ => false, + } +} + +fn not_repeated(v: &str) -> bool { + v.as_bytes().windows(2).any(|p| p[0] != p[1]) +} +fn not_variable(v: &str) -> bool { + !v.starts_with(['$', '%', '{', '<', '[', '*']) + && !v.trim_matches(['*', 'x', 'X', '.']).is_empty() +} +fn secret(v: &str) -> bool { + not_variable(v) + && !v.starts_with("process.env") + && !v.starts_with("os.") + && v.bytes().any(|b| b.is_ascii_digit()) + && v.bytes().any(|b| b.is_ascii_alphabetic()) +} +fn not_slug(v: &str) -> bool { + v.bytes().any(|c| c.is_ascii_uppercase()) + || v.trim_start_matches("sk-") + .bytes() + .all(|b| b.is_ascii_hexdigit() || b == b'-' || b == b'_') + || v.split(['-', '_']).any(|p| p.len() >= 16) +} + +fn chinese_id(v: &str) -> bool { + let bytes = v.as_bytes(); + if bytes.len() != 18 || !bytes[..17].iter().all(u8::is_ascii_digit) { + return false; + } + let sum: usize = bytes[..17] + .iter() + .zip([7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]) + .map(|(b, w)| usize::from(b - b'0') * w) + .sum(); + bytes[17].to_ascii_uppercase() == b"10X98765432"[sum % 11] +} +pub(super) fn card_prefix(s: &str, a: usize, mut b: usize) -> Option<(usize, usize)> { + if luhn(&s[a..b]) { + return Some((a, b)); + } + while b > a { + let separator = s[a..b].rfind([' ', '-'])?; + b = a + separator; + if luhn(&s[a..b]) { + return Some((a, b)); + } + } + None +} + +fn luhn(v: &str) -> bool { + let n = v.bytes().filter(u8::is_ascii_digit).count(); + if !(15..=19).contains(&n) { + return false; + } + v.bytes() + .rev() + .filter(u8::is_ascii_digit) + .enumerate() + .map(|(i, b)| { + let d = b - b'0'; + u16::from(if i % 2 == 0 { + d + } else if d > 4 { + d * 2 - 9 + } else { + d * 2 + }) + }) + .sum::() + % 10 + == 0 +} +fn ssn(v: &str) -> bool { + let s = v.replace([' ', '-'], ""); + s.len() == 9 + && !s.starts_with("000") + && !s.starts_with("666") + && !s.starts_with('9') + && &s[3..5] != "00" + && &s[5..] != "0000" + && !matches!(s.as_str(), "123456789" | "078051120" | "219099999") + && not_repeated(&s) +} +fn iban(v: &str) -> bool { + let s = v.replace(' ', ""); + let Some(country) = s.get(..2) else { + return false; + }; + if !IBAN_LEN + .iter() + .any(|&(key, n)| key == country && n == s.len()) + { + return false; + } + let n = s + .bytes() + .skip(4) + .chain(s.bytes().take(4)) + .try_fold(0_u32, |n, b| match b { + b'0'..=b'9' => Some((n * 10 + u32::from(b - b'0')) % 97), + b'A'..=b'Z' => Some((n * 100 + u32::from(b - b'A') + 10) % 97), + _ => None, + }); + n == Some(1) +} + +const MONTHS: [&str; 12] = [ + "January", + "February", + "March", + "April", + "May", + "June", + "July", + "August", + "September", + "October", + "November", + "December", +]; +pub(in crate::server::core::privacy_masking) fn month(name: &str) -> Option { + let lower = name.trim_end_matches('.').to_ascii_lowercase(); + if lower == "sept" { + return Some(9); + } + MONTHS + .iter() + .position(|n| { + n.eq_ignore_ascii_case(&lower) + || lower.len() == 3 && n[..3].eq_ignore_ascii_case(&lower) + }) + .map(|n| 1 + n as u32) +} +pub(in crate::server::core::privacy_masking) fn month_name(number: u32) -> &'static str { + MONTHS[number as usize - 1] +} +static DATE_NUMBER: LazyLock = + LazyLock::new(|| Regex::new(r"[0-9]+").expect("date pattern")); +static MONTH_WORD: LazyLock = + LazyLock::new(|| Regex::new(r"[A-Za-z]{3,}").expect("month pattern")); +fn date_ok(v: &str) -> bool { + let ns = DATE_NUMBER + .find_iter(v) + .map(|m| m.as_str()) + .collect::>(); + let val = |x: &str| x.parse::().unwrap_or(0); + let today = Utc::now().date_naive(); + let cutoff = today + .with_year(today.year() - 1) + .or_else(|| NaiveDate::from_ymd_opt(today.year() - 1, today.month(), 28)); + let valid = |y: u32, m, d| { + (1900..=2099).contains(&y) + && NaiveDate::from_ymd_opt(y as i32, m, d) + .zip(cutoff) + .is_some_and(|(past, cutoff)| past < cutoff) + }; + match ns.as_slice() { + [x, y] => MONTH_WORD + .find_iter(v) + .find_map(|m| month(m.as_str())) + .is_some_and(|m| { + if x.len() == 4 { + valid(val(x), m, val(y)) + } else { + valid(val(y), m, val(x)) + } + }), + [x, y, z] if x.len() == 4 => valid(val(x), val(y), val(z)), + [x, y, z] => valid(val(z), val(x), val(y)) || valid(val(z), val(y), val(x)), + _ => false, + } +} +fn mac(v: &str) -> bool { + let l = v.replace('-', ":").to_ascii_lowercase(); + !matches!( + l.as_str(), + "00:00:00:00:00:00" + | "ff:ff:ff:ff:ff:ff" + | "00:11:22:33:44:55" + | "aa:bb:cc:dd:ee:ff" + | "01:23:45:67:89:ab" + | "12:34:56:78:9a:bc" + ) && l.bytes().any(|b| matches!(b, b'a'..=b'f')) +} +fn user(v: &str) -> bool { + let l = v.to_ascii_lowercase(); + l.len() >= 2 && !NOT_USERS.contains(&l.as_str()) && !NOT_NAMES.contains(&l.as_str()) +} +fn in_code(v: &str) -> bool { + let l = v.to_ascii_lowercase(); + l.split_once('.').is_some_and(|(first, rest)| { + let last = rest.rsplit('.').next().unwrap_or(rest); + CODE_OBJECTS.contains(&first) + || NOT_USERS.contains(&last) + || NOT_NAMES.contains(&last) + || PUBLIC_HOSTS.contains(&last) + || matches!(last, "login" | "username") + }) +} +fn placeholder_host(host: &str) -> bool { + [ + "example.com", + "example.org", + "example.net", + ".example", + ".test", + ".invalid", + ".localhost", + ] + .iter() + .any(|a| { + let domain = a.trim_start_matches('.'); + host == domain + || host + .strip_suffix(domain) + .is_some_and(|prefix| prefix.ends_with('.')) + }) +} +fn host(v: &str) -> bool { + let l = v.trim_end_matches('.').to_ascii_lowercase(); + l.len() >= 2 + && !PUBLIC_HOSTS.contains(&l.as_str()) + && !["process.", "os.", "env."].iter().any(|p| l.starts_with(p)) + && l.parse::().is_err() + && !placeholder_host(&l) +} +fn public_ip(ip: IpAddr) -> bool { + let ip = ip.to_canonical(); + if ip.is_loopback() + || ip.is_multicast() + || ip.is_unspecified() + || RESOLVERS.contains(&ip.to_string().as_str()) + { + return false; + } + match ip { + IpAddr::V4(v) => { + let [a, b, c, _] = v.octets(); + !v.is_private() + && !v.is_link_local() + && !v.is_documentation() + && a != 0 + && a < 240 + && !(a == 100 && (64..=127).contains(&b)) + && !(a == 192 && b == 0 && c == 0) + && !(a == 198 && matches!(b, 18 | 19)) + } + IpAddr::V6(v) => { + let [first, second, ..] = v.segments(); + (first & 0xe000) == 0x2000 && !(first == 0x2001 && second == 0xdb8) + } + } +} +fn ipv4(s: &str, a: usize, b: usize) -> bool { + let before = s[before(s, a, 16)..a] + .trim_end_matches([' ', '\t', ':', '=', '"', '\'']) + .to_ascii_lowercase(); + if ["version", "ver", "release", "build", "rev", "版本"] + .iter() + .any(|w| { + before + .strip_suffix(w) + .is_some_and(|p| p.as_bytes().last().is_none_or(|&c| !word_byte(c))) + }) + { + return false; + } + if adjacent( + s, + a, + b, + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_-", + ) || s[..a].ends_with(['.', '=', '~', '^']) + || (s.as_bytes().get(b) == Some(&b'.') && numeric(s.as_bytes().get(b + 1))) + || (s[..a].ends_with('/') + && a >= 2 + && (word_byte(s.as_bytes()[a - 2]) || matches!(s.as_bytes()[a - 2], b'.' | b'-'))) + { + return false; + } + s[a..b].parse().is_ok_and(public_ip) +} + +static VERSION: LazyLock = LazyLock::new(|| { + Regex::new(r"^(?:[~^]|[<>=!~]=?|==)?[vV]?[0-9]+(?:\.[0-9]+)+(?:[-+._]?[A-Za-z0-9.-]+)?$") + .expect("version pattern") +}); +static TIMESTAMP: LazyLock = LazyLock::new(|| { + Regex::new(r"^[0-9]{4}-[0-9]{1,2}-[0-9]{1,2}[Tt][0-9]{1,2}(?::[0-9]{1,2}){0,2}(?:\.[0-9]+)?(?:[Zz]|[+-][0-9]{2}:?[0-9]{2})?$").expect("timestamp pattern") +}); +static LINE_REF: LazyLock = + LazyLock::new(|| Regex::new(r"^L?[0-9]+(?:[-–:]L?[0-9]+)*[.,;:]?$").expect("line pattern")); +static SKILL_SLUG: LazyLock = + LazyLock::new(|| Regex::new(r"^[a-z0-9]+(?:-[a-z0-9]+)+$").expect("slug pattern")); +static SCALE: LazyLock = + LazyLock::new(|| Regex::new(r"^[0-9]+(?:\.[0-9]+)?x$").expect("scale pattern")); +fn email(v: &str) -> bool { + let Some((local, raw)) = v.rsplit_once('@') else { + return false; + }; + if local.eq_ignore_ascii_case("git") + || local.eq_ignore_ascii_case("noreply") + || local.ends_with('@') + { + return false; + } + let lower = raw.to_ascii_lowercase(); + let mut host = lower.as_str(); + let Some((stem, tld)) = host.rsplit_once('.') else { + return false; + }; + if SCALE_EXT.contains(&tld) && SCALE.is_match(stem) { + return false; + } + while let Some((stem, tld)) = host.rsplit_once('.') { + if FILE_EXT.contains(&tld) { + host = stem; + } else { + break; + } + } + let Some((_, tld)) = host.rsplit_once('.') else { + return false; + }; + !host.starts_with("noreply") + && !DOC_FILES.contains(&host) + && !placeholder_host(host) + && !host.split('.').all(|s| LINKER_WORDS.contains(&s)) + && (LOCAL_TLDS.contains(&tld) || psl::suffix(tld.as_bytes()).is_some_and(|s| s.is_known())) +} +fn in_userinfo(s: &str, a: usize) -> bool { + let prefix = &s[..a]; + let Some((scheme, authority)) = prefix.rsplit_once("://") else { + return false; + }; + !authority.contains([ + ' ', '\t', '\r', '\n', '/', '?', '#', '"', '\'', '<', '>', '@', '\x60', + ]) && scheme + .chars() + .next_back() + .is_some_and(|c| c.is_ascii_alphanumeric()) +} +fn hidden_in_url(s: &str, a: usize, b: usize) -> bool { + let pw = &s[a..b]; + let Some((_, name)) = s[..a].strip_suffix(':').and_then(|p| p.rsplit_once("//")) else { + return true; + }; + name != pw + && !s[b..] + .strip_prefix('@') + .unwrap_or("") + .split([ + ' ', '\t', '\r', '\n', '"', '\'', '<', '>', '\x60', '\\', ')', ']', '}', + ]) + .next() + .unwrap_or("") + .split(['/', ':', '.', '?', '#', '&', '=', ';', ',', '@']) + .any(|part| part == pw) +} +fn header(v: &str) -> bool { + if !not_variable(v) { + return false; + } + let fields: Vec<_> = v.split_whitespace().collect(); + if fields.len() == 1 + || (fields.len() == 2 && fields[0].bytes().all(|x| x.is_ascii_alphabetic())) + { + let x = fields[fields.len() - 1]; + return x.len() >= 8 && not_variable(x) && x.bytes().any(|b| !b.is_ascii_alphabetic()); + } + v.contains('=') + && v.split(';') + .map(str::trim) + .filter(|x| !x.is_empty()) + .all(|t| { + t.split_once('=') + .is_some_and(|(k, _)| !k.is_empty() && !k.contains([' ', '\t'])) + }) +} +fn path_value(v: &str) -> bool { + !v.contains(['+', '=']) + && (["~/", "./", "../"] + .iter() + .any(|p| v.starts_with(p) && v.len() > p.len()) + || v.strip_prefix('/') + .and_then(|s| s.split_once('/')) + .is_some_and(|(root, rest)| PATH_ROOTS.contains(&root) && !rest.is_empty())) +} +pub(super) fn ident_words(v: &str) -> Vec<&str> { + let mut out = Vec::new(); + for part in v.split(['_', '.', '-']).filter(|x| !x.is_empty()) { + let mut start = 0; + let b = part.as_bytes(); + for i in 1..b.len() { + if b[i].is_ascii_uppercase() && b[i - 1].is_ascii_lowercase() { + out.push(&part[start..i]); + start = i; + } + } + out.push(&part[start..]); + } + out +} +fn names_a_secret(v: &str) -> bool { + v.bytes().all(word_byte_or_dot) + && (v.as_bytes().first().is_some_and(u8::is_ascii_lowercase) || v.contains(['_', '.'])) + && { + let words = ident_words(v); + words.len() >= 2 + && words.iter().any(|w| { + SECRET_WORDS.contains( + &w.trim_end_matches(|c: char| c.is_ascii_digit()) + .to_ascii_lowercase() + .as_str(), + ) + }) + } +} +fn word_byte_or_dot(b: u8) -> bool { + word_byte(b) || b == b'.' +} +pub(super) fn field_name(s: &str, a: usize) -> Option<&str> { + let start = s[..a].trim_end_matches([' ', '\t', '"', '\'', '\\', '“', '「']); + if let Some(xml) = start.strip_suffix('>') { + return xml + .rsplit_once('<') + .and_then(|(_, tag)| tag.split_ascii_whitespace().next()) + .filter(|x| { + !x.is_empty() + && x.bytes() + .all(|b| word_byte_or_dot(b) || matches!(b, b':' | b'-')) + }); + } + let end = start + .strip_suffix([':', '=', ':'])? + .trim_end_matches([' ', '\t', '"', '\'', '\\']); + let field = end + .rsplit(|c: char| { + !(c.is_ascii_alphanumeric() + || matches!(c, '_' | '.' | '-') + || ('\u{4e00}'..='\u{9fff}').contains(&c)) + }) + .next()?; + (!field.is_empty()).then_some(field) +} +static SECRET_FIELD: LazyLock = LazyLock::new(|| { + Regex::new(r"(?i)password|passwd|passphrase|secret|token|api[_-]?key|access[_-]?key|private[_-]?key|priv[_-]?key|credential|authori[sz]ation|cookie|^key$|(?:^|_)pass$").expect("secret field pattern") +}); +pub(super) fn secret_field(name: &str) -> bool { + let lower = name.to_ascii_lowercase(); + if lower.ends_with("_id") + || lower.ends_with("keyid") + || lower.ends_with("key_id") + || lower.ends_with("_count") + || lower.ends_with("tokens") + || matches!( + lower.as_str(), + "max_tokens" + | "token_count" + | "token_type" + | "public_key" + | "publickey" + | "accesskeyid" + ) + { + return false; + } + SECRET_FIELD.is_match(name) + || name.ends_with("_KEY") + && !["public", "cache", "sort", "license", "lookup"] + .iter() + .any(|w| lower.contains(w)) +} +// Magpie's structured-secret value guard: references and expiry values are metadata. +pub(super) fn secret_field_value(value: &str) -> bool { + let value = value.trim(); + !value.is_empty() && not_variable(value) && !TIMESTAMP.is_match(value) +} + +fn field_at(s: &str, a: usize, b: usize) -> bool { + let v = &s[a..b]; + let prospective = field_name(s, a); + if prospective.is_some_and(|f| { + f.rsplit_once('.') + .is_some_and(|(_, x)| SOURCE_EXT.contains(&x.to_ascii_lowercase().as_str())) + || matches!( + f.to_ascii_lowercase().as_str(), + "access_key_id" | "accesskeyid" | "token_count" | "public_key" | "publickey" + ) + }) { + return false; + } + let after = s[b..].trim_start_matches([' ', '\t']); + let code = v.bytes().all(word_byte_or_dot) + && (after.starts_with('(') + || !s[..a].ends_with(['"', '\'']) + && s[..a].trim_end().ends_with(':') + && after.starts_with([';', ',', ')', '}', ']', '|', '?', '&', '<', '>'])); + let skill = SKILL_SLUG.is_match(v) + && !not_slug(v) + && s[..a] + .strip_suffix(':') + .and_then(|p| { + p.rsplit(|c: char| !(c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')) + .next() + }) + .is_some_and(|p| SKILL_SLUG.is_match(p)); + !code + && !skill + && !v.starts_with('=') + && (!LINE_REF.is_match(v) + || prospective.is_some_and(secret_field) && v.bytes().all(|byte| byte.is_ascii_digit())) + && !VERSION.is_match(v) + && !TIMESTAMP.is_match(v) + && !path_value(v) + && !names_a_secret(v) +} +// Units from Magpie's keyAt (redact.go). A labelled number with one +// of these suffixes describes a quantity, not a credential. +const QUANTITY_UNITS: &[&str] = &[ + "位", + "比特", + "字节", + "字節", + "个", + "個", + "次", + "秒", + "毫秒", + "分钟", + "分鐘", + "小时", + "小時", + "天", + "年", + "bit", + "byte", + "ms", + "sec", + "min", + "ビット", + "バイト", + "비트", + "바이트", +]; + +fn key_at(s: &str, a: usize, b: usize) -> bool { + let v = &s[a..b]; + if path_value(v) || VERSION.is_match(v) || TIMESTAMP.is_match(v) { + return false; + } + if v.bytes().all(|byte| byte.is_ascii_digit()) { + // The generic assignment rule excludes stand-alone numbers as line + // references. Here, an explicit separator distinguishes numeric keys. + let before = s[..a].trim_end_matches([' ', '\t', '"', '\'', '“', '「', '\\']); + let after = s[b..].trim_start_matches([' ', '\t']); + return v.len() >= 6 + && before.ends_with(['为', '為', '是', ':', ':', '=', 'は', '는']) + && !QUANTITY_UNITS.iter().any(|unit| after.starts_with(unit)); + } + field_at(s, a, b) + && not_variable(v) + && v.bytes().any(|byte| byte.is_ascii_alphabetic()) + && (v.bytes().any(|byte| byte.is_ascii_digit()) || v.len() >= 16) +} +static NEXT_FIELD: LazyLock = LazyLock::new(|| { + Regex::new(r"^(?:[A-Z_][A-Z0-9_]*|[a-z_][a-z0-9_-]*|--?[A-Za-z][A-Za-z0-9_-]*)=[^=]") + .expect("assignment pattern") +}); +pub(super) fn next_field(s: &str, a: usize, b: usize) -> bool { + s[..a].ends_with([' ', '\t']) && NEXT_FIELD.is_match(&s[a..b]) +} +fn phone(s: &str, a: usize, b: usize) -> bool { + if b - a > 11 { + return true; + } + let name_is_phone = |w: &str| { + PHONE_MARKS.iter().any(|m| w.contains(m)) + || ident_words(w) + .iter() + .any(|p| PHONE_WORDS.contains(&p.to_ascii_lowercase().as_str())) + }; + if let Some(name) = field_name(s, a) { + return name_is_phone(name) + || name.eq_ignore_ascii_case("number") + || s[before(s, a, 48)..a] + .split([' ', '\t']) + .rev() + .skip(1) + .take(2) + .any(name_is_phone); + } + let prefix = s[before(s, a, 32)..a] + .trim_end_matches([' ', '\t', '(', '"', '\'', '“', '「', '是', '为', '為']) + .to_ascii_lowercase(); + if prefix.ends_with('#') { + return false; + } + for w in ID_WORDS { + if let Some(before) = prefix.strip_suffix(w) { + if w.is_ascii() && before.as_bytes().last().is_some_and(|&b| word_byte(b)) { + continue; + } + return matches!(*w, "no" | "no." | "number") + && before + .split_ascii_whitespace() + .next_back() + .is_some_and(name_is_phone); + } + } + true +} +pub(super) fn numeric_arrays(s: &str) -> Vec<(usize, usize)> { + // Exempt numeric data only in a complete JSON document, not prose brackets. + if !s.trim_start().starts_with(['[', '{']) + || serde_json::from_str::(s).is_err() + { + return Vec::new(); + } + let mut ranges = Vec::new(); + let mut start = None; + let (mut quoted, mut escaped) = (false, false); + for (i, byte) in s.bytes().enumerate() { + if quoted { + if escaped { + escaped = false; + } else if byte == b'\\' { + escaped = true; + } else if byte == b'"' { + quoted = false; + } + continue; + } + if byte == b'"' { + quoted = true; + start = None; + continue; + } + match byte { + b'[' => start = Some(i), + b']' => { + if let Some(begin) = start.take() { + ranges.push((begin, i + 1)); + } + } + b'0'..=b'9' + | b',' + | b'.' + | b'-' + | b'+' + | b'e' + | b'E' + | b' ' + | b'\t' + | b'\n' + | b'\r' => {} + _ => start = None, + } + } + ranges +} +pub(super) fn part_of_number_or_id( + s: &str, + a: usize, + b: usize, + numeric_arrays: &[(usize, usize)], +) -> bool { + let raw = s.as_bytes(); + if (numeric(raw.get(a)) && a >= 2 && raw[a - 1] == b'.' && numeric(raw.get(a - 2))) + || (numeric(raw.get(b - 1)) && raw.get(b) == Some(&b'.') && numeric(raw.get(b + 1))) + { + return true; + } + if !s[a..b].bytes().all(|x| x.is_ascii_digit()) { + return false; + } + let in_array = numeric_arrays + .partition_point(|&(start, _)| start <= a) + .checked_sub(1) + .is_some_and(|index| numeric_arrays[index].1 >= b); + if in_array { + return true; + } + let after = s[b..].trim_start(); + let before = s[..a].trim_end(); + // Plain strings can contain JSON documents; unquoted numeric fields are data. + (after.is_empty() || after.starts_with([',', '}', ']'])) + && before + .strip_suffix(':') + .is_some_and(|key| key.trim_end().ends_with('"')) +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/ids.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/ids.rs new file mode 100644 index 000000000..13e63c214 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/ids.rs @@ -0,0 +1,264 @@ +//! Identifier exclusions adapted from Magpie's ids.go (MIT; see licenses/magpie-MIT). +use std::sync::LazyLock; + +use regex::Regex; + +use super::guards::{adjacent, before, field_name, ident_words, word_byte}; + +#[derive(Clone, Copy)] +pub(super) enum IdRule { + Preserve, + Assignment, + Credential, +} + +#[derive(Clone, Copy, PartialEq)] +enum Shape { + Identifier, + Digest, +} + +static UUID: LazyLock = LazyLock::new(|| { + Regex::new(r"^(?i:[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12})$").expect("UUID shape") +}); + +fn hex_shape(value: &str) -> Option { + if value.len() == 36 && UUID.is_match(value) { + return Some(Shape::Identifier); + } + if value.len() >= 32 && value.bytes().all(|b| b.is_ascii_hexdigit()) { + return Some(Shape::Identifier); + } + if !(16..32).contains(&value.len()) { + return None; + } + let (mut lower, mut upper, mut digits) = (0, 0, 0); + for byte in value.bytes() { + match byte { + b'0'..=b'9' => digits += 1, + b'a'..=b'f' => lower += 1, + b'A'..=b'F' => upper += 1, + _ => return None, + } + } + let min = 3.max(value.len() / 8); + ((lower == 0 || upper == 0) && lower + upper >= min && digits >= min).then_some(Shape::Digest) +} + +fn hex_or_hyphen(byte: u8) -> bool { + byte.is_ascii_hexdigit() || byte == b'-' +} + +struct HexRun { + left: usize, + right: usize, + id: Option<(usize, usize, Shape)>, +} + +// Cache the last run per rule: a long run with many candidate hits is scanned once. +#[derive(Default)] +pub(super) struct IdFilter { + run: Option, +} + +impl IdFilter { + fn around(&mut self, text: &str, a: usize, b: usize) -> Option<(usize, usize, Shape)> { + if let Some(run) = self + .run + .as_ref() + .filter(|run| a >= run.left && b <= run.right) + { + return run.id.filter(|&(first, last, _)| a >= first && b <= last); + } + if !text[a..b].bytes().all(hex_or_hyphen) { + return None; + } + let raw = text.as_bytes(); + let left = raw[..a] + .iter() + .rposition(|&c| !hex_or_hyphen(c)) + .map_or(0, |n| n + 1); + let right = raw[b..] + .iter() + .position(|&c| !hex_or_hyphen(c)) + .map_or(raw.len(), |n| b + n); + let first = right - text[left..right].trim_start_matches('-').len(); + let last = first + text[first..right].trim_end_matches('-').len(); + let shape = hex_shape(&text[first..last]).and_then(|kind| { + if kind == Shape::Digest + && adjacent( + text, + first, + last, + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_", + ) + { + None + } else { + Some((first, last, kind)) + } + }); + self.run = Some(HexRun { + left, + right, + id: shape, + }); + shape.filter(|&(first, last, _)| a >= first && b <= last) + } + + pub(super) fn keep(&mut self, rule: IdRule, text: &str, a: usize, b: usize) -> bool { + let field = || field_name(text, a); + if element_token(text, a, b, field) { + return !named_credential(text, a, field()); + } + let Some((left, right, shape)) = self.around(text, a, b) else { + return false; + }; + if a != left || b != right { + return true; + } + match rule { + IdRule::Preserve => true, + IdRule::Credential => false, + IdRule::Assignment => { + let name = field(); + !(shape == Shape::Digest && name.is_none()) && !named_credential(text, a, name) + } + } + } +} + +pub(super) fn tool_lookup(name: &str, value: &str) -> bool { + let value = value.trim(); + (hex_shape(value).is_some() || element_token(value, 0, value.len(), || Some(name))) + && !credential_name(name) +} + +fn element_field(name: &str) -> bool { + let words = ident_words(name); + words.len() == 2 + && words[0].eq_ignore_ascii_case("element") + && words[1].eq_ignore_ascii_case("token") +} + +fn element_token<'a>(s: &str, a: usize, b: usize, field: impl FnOnce() -> Option<&'a str>) -> bool { + let raw = s.as_bytes(); + if b - a < 9 || raw[a] != b's' || a > 0 && word_byte(raw[a - 1]) { + return false; + } + let stem = a + 9; + if !raw[a + 1..stem] + .iter() + .all(|c| matches!(c, b'0'..=b'9' | b'a'..=b'f')) + { + return false; + } + let end = if raw.get(stem) == Some(&b':') { + let n = raw[stem + 1..] + .iter() + .take_while(|c| c.is_ascii_digit()) + .count(); + if n == 0 { + return false; + } + stem + 1 + n + } else { + stem + }; + (b == stem || b == end) + && !raw.get(end).is_some_and(|&c| word_byte(c)) + && (end > stem || field().is_some_and(element_field)) +} + +fn named_credential(s: &str, a: usize, name: Option<&str>) -> bool { + let Some(name) = name else { + return false; + }; + if !name.eq_ignore_ascii_case("key") { + return credential_name(name); + } + let lower = s[before(s, a, 24)..a].to_ascii_lowercase(); + lower.rsplit_once("key").is_some_and(|(prefix, _)| { + let prefix = prefix.trim_end_matches([' ', '\t']); + [ + "api", "access", "secret", "private", "signing", "license", "licence", + ] + .iter() + .any(|part| { + prefix + .strip_suffix(part) + .is_some_and(|rest| rest.as_bytes().last().is_none_or(|&c| !word_byte(c))) + }) + }) +} + +const NON_CREDENTIAL: &[&str] = &[ + "page", + "next", + "cursor", + "continuation", + "csrf", + "xsrf", + "trace", + "request", + "idempotency", + "sync", + "nonce", + "hash", + "digest", + "checksum", + "sha", + "fingerprint", + "etag", + "cache", + "commit", + "object", + "blob", + "file", + "content", + "revision", + "rev", + "partition", + "shard", + "bucket", + "dedup", + "dedupe", + "lookup", + "row", + "sort", +]; + +fn credential_name(name: &str) -> bool { + let words = ident_words(name); + if words.is_empty() + || name.eq_ignore_ascii_case("key") + || element_field(name) + || words.last().is_some_and(|last| { + ["id", "ids", "uuid", "guid"] + .iter() + .any(|word| last.eq_ignore_ascii_case(word)) + }) + { + return false; + } + // "file_password" is still a password. A password's hash/digest is not. + let password = words.iter().any(|word| { + ["password", "passwd", "passphrase", "pass", "pwd"] + .iter() + .any(|secret| word.eq_ignore_ascii_case(secret)) + }); + if password + && !words.iter().any(|word| { + ["hash", "digest", "checksum", "sha", "fingerprint", "etag"] + .iter() + .any(|data| word.eq_ignore_ascii_case(data)) + }) + { + return true; + } + !words.iter().any(|word| { + NON_CREDENTIAL + .iter() + .any(|keep| word.eq_ignore_ascii_case(keep)) + }) +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/mod.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/mod.rs new file mode 100644 index 000000000..fa16f2e07 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/mod.rs @@ -0,0 +1,267 @@ +//! Deterministic Magpie detectors; no model, network, global originals or restoration state. +mod builtins; +mod guard_data; +mod guards; +mod ids; +mod wallet; + +use std::{collections::BTreeMap, sync::LazyLock}; + +pub(super) use guards::{month, month_name}; +use ids::{IdFilter, IdRule}; +use regex::{Captures, Match, Regex}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, Ord, PartialEq, PartialOrd, Serialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum Category { + Secrets, + Personal, + Ssn, + Passport, + Iban, + Birthday, + Mac, + Serial, + Home, + Userhost, + Ip, + Bucket, +} + +impl Category { + pub(super) fn optional(self) -> bool { + !matches!(self, Self::Secrets | Self::Personal) + } + fn enabled(self, secrets: bool, personal: bool, overrides: &BTreeMap) -> bool { + match self { + Self::Secrets => secrets, + Self::Personal => personal, + other => { + personal + && overrides.get(&other).copied().unwrap_or(!matches!( + other, + Self::Home | Self::Userhost | Self::Ip | Self::Bucket + )) + } + } + } +} + +#[derive(Clone, Copy)] +enum Capture { + First, + Any, + Groups(&'static [Guard]), +} + +#[derive(Clone, Copy)] +enum Guard { + None, + Bip32Key, + BucketOK, + ChineseID, + DateOK, + EmailAt, + FieldAt, + HeaderValue, + HiddenInURL, + HomeAt, + HostOK, + IbanAt, + Ipv4At, + Ipv6OK, + KeyAt, + LineHostOK, + LineUserOK, + Luhn, + MacOK, + NamedSecretAt, + NotASlug, + NotAVariable, + NotLineIdent, + NotPackagePin, + PassportOK, + PhoneAt, + RealEmail, + SecretValue, + AssignedSecret, + SeedPhraseAt, + SerialOK, + SolanaKey, + SolanaKeypair, + SsnOK, + StellarSeed, + UserOK, + WalletKeyAt, + WifKey, +} + +#[derive(Clone, Copy)] +struct Spec { + category: Category, + pattern: &'static str, + capture: Capture, + id_rule: IdRule, + bound: &'static str, + check: Guard, + context: Guard, + repair: Guard, + field: bool, + markers: &'static [&'static str], + fold: &'static [&'static str], +} + +impl Spec { + const DEFAULT: Self = Self { + category: Category::Secrets, + pattern: "", + capture: Capture::First, + id_rule: IdRule::Preserve, + bound: "", + check: Guard::None, + context: Guard::None, + repair: Guard::None, + field: false, + markers: &[], + fold: &[], + }; +} + +static RULES: LazyLock, regex::Error>> = LazyLock::new(|| { + builtins::BUILTINS + .iter() + .map(|&spec| Regex::new(spec.pattern).map(|re| (spec, re))) + .collect() +}); + +pub(super) fn ready() -> bool { + RULES.is_ok() +} + +fn captures( + spec: Spec, + caps: &Captures<'_>, + mut visit: impl FnMut(Match<'_>, Guard) -> Result<(), ()>, +) -> Result<(), ()> { + match spec.capture { + Capture::Groups(guards) => { + for (index, guard) in guards.iter().copied().enumerate() { + if let Some(hit) = caps.get(index + 1) { + visit(hit, guard)?; + } + } + } + Capture::Any => { + if let Some(hit) = caps.iter().skip(1).flatten().next() { + visit(hit, spec.check)?; + } + } + Capture::First => { + if let Some(hit) = caps.get(if caps.len() > 1 { 1 } else { 0 }) { + visit(hit, spec.check)?; + } + } + } + Ok(()) +} + +pub(super) fn scan( + text: &str, + secrets: bool, + personal: bool, + overrides: &BTreeMap, + mut visit: impl FnMut((usize, usize)) -> Result<(), ()>, +) -> Result<(), ()> { + if !(secrets || personal) || text.len() < 3 { + return Ok(()); + } + let rules = RULES.as_ref().map_err(|_| ())?; + let number_arrays = guards::numeric_arrays(text); + let lower = std::cell::OnceCell::new(); + for (spec, regex) in rules { + if !spec.category.enabled(secrets, personal, overrides) + || (!spec.markers.is_empty() && !spec.markers.iter().any(|m| text.contains(m))) + || (!spec.fold.is_empty() + && !spec + .fold + .iter() + .any(|m| lower.get_or_init(|| text.to_ascii_lowercase()).contains(m))) + { + continue; + } + let mut id_filter = IdFilter::default(); + let mut from = 0; + while let Some(caps) = regex.captures_at(text, from) { + let Some(full) = caps.get(0) else { break }; + let full_end = full.end(); + let mut resume_at = full_end; + from = full_end; + if from == full.start() { + return Err(()); + } + if spec.field { + if let Some(hit) = caps.get(1).filter(|m| { + m.start() > full.start() && guards::next_field(text, m.start(), m.end()) + }) { + from = hit.start(); + continue; + } + } + captures(*spec, &caps, |hit, check| { + let (a, b) = (hit.start(), hit.end()); + let spans = if matches!(spec.repair, Guard::SeedPhraseAt) { + wallet::phrases(text, a, b) + } else if matches!(check, Guard::Luhn) { + guards::card_prefix(text, a, b).into_iter().collect() + } else { + guards::repair(spec.repair, text, a, b) + .into_iter() + .collect() + }; + for (a, b) in spans { + if (matches!(spec.repair, Guard::IbanAt) || matches!(check, Guard::Luhn)) + && b < resume_at + && b > full.start() + { + resume_at = b; + } + if a >= b || guards::adjacent(text, a, b, spec.bound) { + continue; + } + let value = &text[a..b]; + if !guards::accept(check, value) + || !guards::context(spec.context, text, a, b) + || id_filter.keep(spec.id_rule, text, a, b) + || (spec.category != Category::Secrets + && spec.category != Category::Ip + && guards::part_of_number_or_id(text, a, b, &number_arrays)) + { + continue; + } + visit((a, b))?; + } + Ok(()) + })?; + if resume_at < full_end && resume_at > full.start() { + from = resume_at; + } + } + } + Ok(()) +} + +pub(super) fn is_secret_field(key: &str) -> bool { + guards::secret_field(key) +} + +pub(super) fn secret_field_value(value: &str) -> bool { + guards::secret_field_value(value) +} + +pub(super) fn tool_lookup(name: &str, value: &str) -> bool { + ids::tool_lookup(name, value) +} + +#[cfg(test)] +mod tests; diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/tests.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/tests.rs new file mode 100644 index 000000000..21893fc08 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/tests.rs @@ -0,0 +1,255 @@ +use super::*; + +#[test] +fn magpie_rule_table_is_complete_and_compiles() { + let rules = RULES.as_ref().unwrap(); + assert_eq!(rules.len(), 58); + for (s, re) in rules { + assert!(!re.is_match(""), "{:?}", s.category); + } +} + +fn ranges(text: &str, secrets: bool, personal: bool, extra: bool) -> Vec<&str> { + let mut options = BTreeMap::new(); + if extra { + for kind in [ + Category::Home, + Category::Userhost, + Category::Ip, + Category::Bucket, + ] { + options.insert(kind, true); + } + } + let mut out = Vec::new(); + scan(text, secrets, personal, &options, |(a, b)| { + out.push(&text[a..b]); + Ok(()) + }) + .unwrap(); + out +} + +#[test] +fn personal_formats_and_contextual_exclusions() { + // Examples are synthetic or well-known public format test vectors; not real account data. + let number = ["139", "0000", "9931"].join(""); + for prefix in ["+86", "+86 ", "+86-", "86 ", ""] { + let phone = format!("{prefix}{number}"); + assert!(ranges(&format!("联系手机{phone}"), false, true, false).contains(&phone.as_str())); + } + let card = ["4111", "1111", "1111", "1111"].join(" "); + let id = ["110105", "19491231", "002X"].join(""); + let ssn = ["202", "17", "5060"].join("-"); + let mac = ["3e", "5e", "9f", "74", "81", "fa"].join(":"); + for (source, value) in [ + ( + "test email joe@relay.internal".to_owned(), + "joe@relay.internal".to_owned(), + ), + (format!("身份证: {id}"), id), + (format!("银行卡: {card}"), card), + (format!("SSN: {ssn}"), ssn), + ("Passport No. ABC543210".into(), "ABC543210".into()), + ( + "IBAN: DE89370400440532013000".into(), + "DE89370400440532013000".into(), + ), + ("DOB: 1990-05-20".into(), "1990-05-20".into()), + (format!("MAC: {mac}"), mac), + ("Serial Number: TQ-904959".into(), "TQ-904959".into()), + ] { + assert!( + ranges(&source, false, true, false).contains(&value.as_str()), + "unrecognized fixture: {source}" + ); + } + for clear in [ + format!("订单号:{number}"), + format!("0.{number}"), + "git@github.com:repo/project".into(), + "fixture@example.test".into(), + "443:443".into(), + "0.12.2".into(), + "a@fixture.json".into(), + "release/bld.123".into(), + ] { + assert!( + ranges(&clear, false, true, true).is_empty(), + "false positive: {clear}" + ); + } +} + +#[test] +fn optional_infrastructure_detectors_require_opt_in() { + let public_ip = ["74", "88", "22", "99"].join("."); + let cases = [ + ( + "/home/cedar-agent/work".to_owned(), + "cedar-agent".to_owned(), + ), + ( + "ssh qaoperator@relay-node-78".to_owned(), + "qaoperator".to_owned(), + ), + ( + "ssh qaoperator@relay-node-78".to_owned(), + "relay-node-78".to_owned(), + ), + (format!("public address: {public_ip}"), public_ip), + ( + "s3://cold-archive-project-4242/objects".to_owned(), + "cold-archive-project-4242".to_owned(), + ), + ]; + for (text, value) in cases { + assert!( + !ranges(&text, false, true, false).contains(&value.as_str()), + "optional detector enabled by default" + ); + assert!( + ranges(&text, false, true, true).contains(&value.as_str()), + "unrecognized infrastructure fixture: {text}" + ); + } + for s in [ + "10.0.0.5", + "192.0.2.2", + "8.8.8.8", + "s3://my-bucket/objects", + "/home/runner/work", + "git@github.com:repo/project", + ] { + assert!( + ranges(s, false, true, true).is_empty(), + "false positive: {s}" + ); + } +} + +#[test] +fn luhn_group_prefix_and_full_length_account_numbers_are_checked() { + let leaked = "4111111111111111 123"; + let hits = ranges(leaked, false, true, false); + assert_eq!(hits, vec!["4111111111111111"]); + let prefix = ["4111", "1111", "1111", "1111", "11"].join(""); + let full = (0..=9) + .map(|digit| format!("{prefix}{digit}")) + .find(|value| ranges(value, false, true, false).contains(&value.as_str())); + assert!( + full.is_some(), + "valid complete 19-digit card vector was not recognized" + ); +} + +#[test] +fn numeric_arrays_are_indexed_once_and_preserved() { + let values = vec!["13800138000"; 2048].join(","); + let input = format!("{{\"phone_numbers\":[{values}]}}"); + assert!( + ranges(&input, false, true, false).is_empty(), + "numbers inside a JSON numeric array are values, not text phones" + ); + assert_eq!(super::guards::numeric_arrays(&input).len(), 1); + let nested = format!("[[],[{values}],[{values}]]"); + assert!(ranges(&nested, false, true, false).is_empty()); + assert_eq!(super::guards::numeric_arrays(&nested).len(), 3); +} + +#[test] +fn numeric_exemptions_require_json_and_never_ignore_quoted_text() { + let phone = ["139", "0000", "9931"].join(""); + for input in [ + format!("phone [{phone}]"), + format!("notify [{phone}, {phone}]"), + serde_json::json!({"note": format!("[{phone}]")}).to_string(), + ] { + assert!(super::guards::numeric_arrays(&input).is_empty()); + assert!( + ranges(&input, false, true, false).contains(&phone.as_str()), + "unmasked prose: {input}" + ); + } + let number = phone.parse::().unwrap(); + let data = serde_json::json!({"data": [[number], [number]]}).to_string(); + assert_eq!(super::guards::numeric_arrays(&data).len(), 2); + assert!(ranges(&data, false, true, false).is_empty()); +} +#[test] +fn repaired_iban_match_resumes_before_the_unconsumed_suffix() { + let iban = "DE89 3704 0044 0532 0130 00"; + for (first, second) in [ + (iban, iban), + ("BE68 5390 0754 7034", "DE89 3704 0044 0532 0130 00"), + ] { + let input = format!("{first} {second}"); + let hits = ranges(&input, false, true, false); + assert_eq!( + hits.iter().filter(|&&hit| hit == first).count(), + if first == second { 2 } else { 1 } + ); + assert!(hits.contains(&second), "second grouped IBAN was skipped"); + } +} + +#[test] +fn credential_families_and_checksum_guards() { + let vendor: Vec = [ + ("sk-proj-", 22), + ("ark-", 21), + ("oc_sk_", 22), + ("ghp_", 36), + ("github_pat_", 46), + ("glpat-", 22), + ("AIza", 35), + ("xoxb-", 24), + ("rk_live_", 24), + ("hf_", 30), + ("gsk_", 40), + ("xai-", 40), + ("npm_", 36), + ("pypi-", 50), + ("cfut_", 40), + ("cfat_", 40), + ("cfk_", 40), + ("cfoat_", 40), + ("eyJhIjoi", 44), + ] + .into_iter() + .map(|(prefix, n)| format!("{prefix}{}", "xQ12Z".repeat(10).get(..n).unwrap())) + .collect(); + for value in vendor { + assert!( + ranges(&value, true, false, false).contains(&value.as_str()), + "unrecognized built-in credential family {}", + &value[..5] + ); + } + let derive = ed25519_dalek::SigningKey::from_bytes(&[0x53; 32]).to_keypair_bytes(); + let wif = bs58::encode([&[0x80][..], &[0x34; 32], &[1]].concat()) + .with_check() + .into_string(); + let invalid = format!( + "{}{}", + &wif[..wif.len() - 1], + if wif.ends_with('A') { 'B' } else { 'A' } + ); + assert!(ranges(&wif, true, false, false).contains(&wif.as_str())); + assert!(ranges(&invalid, true, false, false).is_empty()); + let solana = bs58::encode(derive).into_string(); + assert!(ranges(&solana, true, false, false).contains(&solana.as_str())); + let key_array = serde_json::to_string(derive.as_slice()).unwrap(); + assert!(ranges(&key_array, true, false, false).contains(&key_array.as_str())); + let key_hex = "0x".to_owned() + &"b5".repeat(32); + assert!( + ranges(&format!("signing key {key_hex}"), true, false, false).contains(&key_hex.as_str()) + ); + assert!(ranges(&format!("wallet tx hash {key_hex}"), true, false, false).is_empty()); + let pem = "-----BEGIN PRIVATE KEY-----\nfixture-only-data\n-----END PRIVATE KEY-----"; + assert!(ranges(pem, true, false, false).contains(&pem)); + assert!(ranges("密码:Se9a1ch", true, false, false).contains(&"Se9a1ch")); + let words = + "abandon ability able about above absent absorb abstract absurd abuse access accident"; + assert!(ranges(&format!("mnemonic: {words}"), true, false, false).contains(&words)); +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/wallet.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/wallet.rs new file mode 100644 index 000000000..22c239993 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/detectors/wallet.rs @@ -0,0 +1,154 @@ +//! Key shape/checksum verification based on Magpie crypto.go (MIT; licenses/magpie-MIT). +use ed25519_dalek::SigningKey; +use std::{collections::HashSet, sync::LazyLock}; + +pub(super) fn wif(v: &str) -> bool { + if !matches!(v.len(), 51 | 52) { + return false; + } + bs58::decode(v) + .with_check(None) + .into_vec() + .is_ok_and(|p| p.len() == 33 || p.len() == 34 && p[33] == 1) +} + +pub(super) fn bip32(v: &str) -> bool { + bs58::decode(v) + .with_check(None) + .into_vec() + .is_ok_and(|p| p.len() == 78) +} + +fn solana_bytes(data: &[u8]) -> bool { + let Ok(bytes) = <&[u8; 64]>::try_from(data) else { + return false; + }; + SigningKey::from_keypair_bytes(bytes).is_ok() +} + +pub(super) fn solana(v: &str) -> bool { + (85..=88).contains(&v.len()) && bs58::decode(v).into_vec().is_ok_and(|p| solana_bytes(&p)) +} + +pub(super) fn solana_array(v: &str) -> bool { + v.trim_matches(|c: char| c == '[' || c == ']' || c.is_ascii_whitespace()) + .split(',') + .map(|part| part.trim().parse::()) + .collect::, _>>() + .is_ok_and(|p| solana_bytes(&p)) +} + +pub(super) fn stellar(v: &str) -> bool { + let Ok(bin) = data_encoding::BASE32_NOPAD.decode(v.as_bytes()) else { + return false; + }; + if bin.len() != 35 || bin[0] != (18 << 3) { + return false; + } + let crc = bin[..33].iter().fold(0_u16, |mut state, byte| { + state ^= u16::from(*byte) << 8; + for _ in 0..8 { + state = (state << 1) ^ if state & 0x8000 != 0 { 0x1021 } else { 0 }; + } + state + }); + bin[33..] == crc.to_le_bytes() +} + +pub(super) fn wallet_context(text: &str, start: usize) -> bool { + let start_line = text[..start].rfind('\n').map_or(0, |n| n + 1); + let from = super::guards::before(text, start, 48).max(start_line); + let prefix = text[from..start].to_ascii_lowercase(); + let cue = [ + "priv", + "secret", + "signer", + "signing", + "deployer", + "wallet", + "pk", + "私钥", + "私鑰", + "秘密鍵", + "개인키", + "비밀키", + ] + .into_iter() + .flat_map(|cue| { + let prefix = prefix.as_str(); + prefix.match_indices(cue).filter(move |&(a, _)| { + cue != "pk" + || !super::guards::adjacent( + prefix, + a, + a + 2, + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz", + ) + }) + }) + .max_by_key(|&(a, _)| a); + let Some((a, cue)) = cue else { return false }; + let middle = &prefix[a + cue.len()..]; + if [ + "hash", + "sha", + "digest", + "sum", + "pub", + "addr", + "tx", + "fingerprint", + ] + .iter() + .any(|mark| middle.contains(mark)) + { + return false; + } + !middle + .trim_start_matches(['_', '-', ' ']) + .strip_prefix("id") + .is_some_and(|suffix| { + suffix + .as_bytes() + .first() + .is_none_or(|b| !b.is_ascii_alphabetic()) + }) +} + +static WORDS: LazyLock> = LazyLock::new(|| { + include_str!("bip39english.txt") + .split_ascii_whitespace() + .collect() +}); + +pub(super) fn phrases(text: &str, start: usize, end: usize) -> Vec<(usize, usize)> { + let mut found = Vec::new(); + let (mut run_start, mut run_end, mut words) = (start, start, 0); + let mut at = start; + while at < end { + while at < end && matches!(text.as_bytes()[at], b' ' | b'\t') { + at += 1; + } + if at == end { + break; + } + let stop = text[at..end].find([' ', '\t']).map_or(end, |n| at + n); + if WORDS.contains(text[at..stop].to_ascii_lowercase().as_str()) { + if words == 0 { + run_start = at; + } + words += 1; + run_end = stop; + } else { + if words >= 12 { + found.push((run_start, run_end)); + } + words = 0; + } + at = stop; + } + if words >= 12 { + found.push((run_start, run_end)); + } + found +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/mod.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/mod.rs new file mode 100644 index 000000000..84501e2a2 --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/mod.rs @@ -0,0 +1,538 @@ +//! 内存内的推理正文脱敏。只接受协议层挑出的文本;不接触鉴权、路由或持久化。 + +use std::{ + collections::{BTreeMap, HashMap}, + fmt::Write, + sync::{Arc, LazyLock, Mutex}, +}; + +use regex::Regex; +use serde::{Deserialize, Serialize}; + +mod custom; +mod detectors; +use detectors::Category; + +pub(crate) mod stream; +pub(crate) mod wire; + +const PREFIX: &str = "{{A2A_"; +const SUFFIX: &str = "}}"; +const TOKEN_LEN: usize = PREFIX.len() + 32 + SUFFIX.len(); +const MAX_VALUES: usize = 4096; +const MAX_MATCHES: usize = 65_536; +const MAX_VALUE_BYTES: usize = 64 * 1024; +const MAX_EXPANSION: usize = 16 * 1024 * 1024; +const MAX_MAPPED_BYTES: usize = 16 * 1024 * 1024; + +#[derive(Clone, Default, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct Settings { + pub(crate) secrets: bool, + pub(crate) personal: bool, + pub(crate) terms: Vec, + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + kinds: BTreeMap, + #[serde(default, skip_serializing_if = "is_false")] + terms_any_case: bool, + #[serde(default, skip_serializing_if = "is_false")] + terms_whole: bool, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + rules: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + allow: Vec, +} + +fn is_false(value: &bool) -> bool { + !*value +} + +#[derive(Clone, Default)] +pub(crate) struct Rules { + pub(crate) settings: Settings, + terms: Vec, + custom: Vec, +} + +impl std::fmt::Debug for Rules { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PrivacyRules") + .field("enabled", &self.enabled()) + .finish_non_exhaustive() + } +} + +impl Rules { + pub(crate) fn enabled(&self) -> bool { + self.settings.secrets || self.settings.personal || !self.terms.is_empty() + } + + pub(crate) fn compile(mut settings: Settings) -> Result { + for term in &mut settings.terms { + *term = term.trim().to_owned(); + } + settings.terms.retain(|s| !s.is_empty()); + settings.terms.sort(); + settings.terms.dedup(); + if settings.terms.len() > 100 || settings.terms.iter().any(|s| s.len() > 256) { + return Err("terms 最多 100 项,每项最多 256 字节"); + } + if !settings.kinds.keys().copied().all(Category::optional) { + return Err("无效的个人信息分类"); + } + if (settings.secrets || settings.personal) && !detectors::ready() { + return Err("内置脱敏规则初始化失败"); + } + let terms = settings + .terms + .iter() + .map(|term| custom::Term::compile(term, settings.terms_any_case)) + .collect::, _>>() + .map_err(|_| "脱敏词规则编译失败")?; + for allowed in &mut settings.allow { + *allowed = allowed.trim().to_lowercase(); + } + settings.allow.retain(|v| !v.is_empty()); + settings.allow.sort(); + settings.allow.dedup(); + if settings.allow.len() > 100 || settings.allow.iter().any(|v| v.len() > 256) { + return Err("白名单最多 100 项,每项最多 256 字节"); + } + if settings.rules.len() > 32 { + return Err("自定义脱敏规则最多 32 条"); + } + let mut configured = Vec::new(); + let mut custom = Vec::new(); + for mut rule in std::mem::take(&mut settings.rules) { + if let Some(compiled) = rule.compile()? { + if !configured.contains(&rule) { + configured.push(rule); + custom.push(compiled); + } + } + } + settings.rules = configured; + Ok(Self { + settings, + terms, + custom, + }) + } +} + +static PLACEHOLDER: LazyLock = + LazyLock::new(|| Regex::new(r"\{\{A2A_[0-9a-f]{32}\}\}").unwrap()); + +pub(super) fn is_secret_field(name: &str) -> bool { + detectors::is_secret_field(name) +} + +// Encoded attachments can also occur as strings in tool output. Keep only +// the data-URL form, not arbitrary prose beginning with "data:". +fn encoded_attachment(text: &str) -> bool { + text.strip_prefix("data:") + .and_then(|rest| rest.split_once(',')) + .and_then(|(head, data)| head.strip_suffix(";base64").map(|media| (media, data))) + .is_some_and(|(media, data)| { + media.contains('/') + && !media.bytes().any(|b| b.is_ascii_whitespace()) + && !data.is_empty() + && data + .bytes() + .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'+' | b'/' | b'=')) + }) +} + +pub(crate) struct Mask { + rules: Rules, + by_value: HashMap, + by_token: HashMap, + mapped_bytes: usize, + json_output: Option, +} + +impl Mask { + pub(crate) fn new(rules: &Rules) -> Self { + Self { + rules: rules.clone(), + by_value: HashMap::new(), + by_token: HashMap::new(), + mapped_bytes: 0, + json_output: None, + } + } + + fn token(&mut self, value: &str) -> Result<&str, ()> { + if value.len() > MAX_VALUE_BYTES { + return Err(()); + } + if !self.by_value.contains_key(value) { + if self.by_value.len() >= MAX_VALUES + || self.mapped_bytes + value.len() > MAX_MAPPED_BYTES + { + return Err(()); + } + let token = loop { + let mut random = [0; 16]; + getrandom::getrandom(&mut random).map_err(|_| ())?; + let mut token = String::with_capacity(TOKEN_LEN); + token.push_str(PREFIX); + for byte in random { + write!(token, "{byte:02x}").map_err(|_| ())?; + } + token.push_str(SUFFIX); + if !self.by_token.contains_key(&token) { + break token; + } + }; + self.by_token.insert(token.clone(), value.to_owned()); + self.by_value.insert(value.to_owned(), token); + self.mapped_bytes += value.len(); + } + self.by_value.get(value).map(String::as_str).ok_or(()) + } + + pub(super) fn apply(&mut self, text: &mut String, secret_field: bool) -> Result<(), ()> { + if encoded_attachment(text) { + return Ok(()); + } + // The egress pass must not remask placeholders that this request already owns. + let retained: Vec<_> = PLACEHOLDER + .find_iter(text) + .filter(|hit| self.by_token.contains_key(hit.as_str())) + .map(|hit| (hit.start(), hit.end())) + .collect(); + let owned = |(a, b): (usize, usize)| { + let index = retained.partition_point(|&(_, end)| end <= a); + retained + .get(index) + .is_some_and(|&(start, end)| start <= a && b <= end) + }; + let mut ranges = Vec::new(); + let mut collect = |items: &mut dyn Iterator| { + for pair in items { + if pair.0 >= pair.1 || owned(pair) { + continue; + } + if ranges.len() >= MAX_MATCHES { + return Err(()); + } + ranges.push(pair); + } + Ok(()) + }; + let settings = &self.rules.settings; + for term in &self.rules.terms { + collect(&mut term.spans(text, settings.terms_whole))?; + } + let literal_count = ranges.len(); + let mut record = |pair: (usize, usize)| { + if pair.0 >= pair.1 || owned(pair) { + return Ok(()); + } + if ranges.len() >= MAX_MATCHES { + return Err(()); + } + ranges.push(pair); + Ok(()) + }; + if secret_field && settings.secrets && detectors::secret_field_value(text) { + record((0, text.len()))?; + } else { + detectors::scan( + text, + settings.secrets, + settings.personal, + &settings.kinds, + &mut record, + )?; + if settings.secrets { + for re in &self.rules.custom { + for m in re + .captures_iter(text) + .filter_map(|m| m.get(if m.len() > 1 { 1 } else { 0 })) + { + record((m.start(), m.end()))?; + } + } + } + } + if !settings.allow.is_empty() { + let mut index = 0; + ranges.retain(|&(a, b)| { + index += 1; + index <= literal_count + || settings + .allow + .binary_search(&text[a..b].to_lowercase()) + .is_err() + }); + } + if ranges.is_empty() { + return Ok(()); + } + ranges.sort_unstable(); + // Overlaps must not expose the unmatched end of a secret (for example if a + // custom term overlaps a key rather than lying wholly inside it). + let mut merged = Vec::<(usize, usize)>::new(); + for (start, end) in ranges { + match merged.last_mut() { + Some((_, previous_end)) if start < *previous_end => { + *previous_end = end.max(*previous_end) + } + _ => merged.push((start, end)), + } + } + + // Subtract owned placeholders from matched ranges, but still mask any new + // plaintext a match spans (e.g. a freshly appended credential on the same line). + let mut out = String::with_capacity(text.len()); + let mut at = 0; + for (start, end) in merged { + out.push_str(&text[at..start]); + let first = retained.partition_point(|&(_, stop)| stop <= start); + let mut begin = start; + for &(a, b) in &retained[first..] { + if a >= end { + break; + } + if begin < a { + out.push_str(self.token(&text[begin..a])?); + } + // Copy only the part within this merged range. The rest remains in the source. + out.push_str(&text[begin.max(a)..end.min(b)]); + begin = end.min(b); + } + if begin < end { + out.push_str(self.token(&text[begin..end])?); + } + at = end; + } + out.push_str(&text[at..]); + *text = out; + Ok(()) + } + + #[cfg(test)] + fn into_restore(self) -> Option { + (!self.by_token.is_empty()).then(|| RequestMask(Arc::new(Mutex::new(self))).restore()) + } +} + +/// 单次请求的不透明句柄。只在鉴权后创建,不进入日志或遥测快照。 +#[derive(Clone)] +pub(crate) struct RequestMask(Arc>); + +impl RequestMask { + pub(crate) fn new(rules: &Rules) -> Self { + Self(Arc::new(Mutex::new(Mask::new(rules)))) + } + + pub(crate) fn apply( + &self, + format: wire::Format, + body: &mut serde_json::Value, + ) -> Result<(), crate::server::errors::GatewayError> { + let mut mask = self.0.lock().map_err(|_| unavailable())?; + wire::request(format, body, &mut mask).map_err(|_| unavailable()) + } + + // A stream can start an automatic continuation after the HTTP handler returns. + // Keep the same mapping alive until both sides of the request are done. + pub(crate) fn restore(&self) -> Restore { + Restore { + request: self.clone(), + budget: MAX_EXPANSION, + } + } +} + +pub(crate) fn unavailable() -> crate::server::errors::GatewayError { + crate::server::errors::GatewayError::with_status(503, "隐私脱敏处理失败,未转发该请求") + .with_code("privacy_masking") +} + +pub(crate) struct Restore { + // Never derive Debug, and never put this map into an extension visible to metrics/loggers. + request: RequestMask, + budget: usize, +} + +impl Restore { + pub(super) fn json_output(&self) -> bool { + self.request + .0 + .lock() + .is_ok_and(|mask| mask.json_output.unwrap_or(false)) + } + + /// Only wait for a prefix of a token actually emitted for this HTTP request. + pub(super) fn partial_len(&self, text: &str) -> usize { + let Ok(mask) = self.request.0.lock() else { + return 0; + }; + if mask.by_token.is_empty() { + return 0; + } + let max = text.len().min(TOKEN_LEN - 1); + (1..=max) + .rev() + .find(|&n| { + text.get(text.len() - n..).is_some_and(|tail| { + (n <= PREFIX.len() && PREFIX.starts_with(tail)) + || (tail.starts_with(PREFIX) + && mask.by_token.keys().any(|token| token.starts_with(tail))) + }) + }) + .unwrap_or(0) + } + + pub(super) fn apply(&mut self, text: &mut String, json_fragment: bool) -> Result<(), ()> { + let mask = self.request.0.lock().map_err(|_| ())?; + let mut out = String::new(); + let mut at = 0; + for hit in PLACEHOLDER.find_iter(text) { + let Some(value) = mask.by_token.get(hit.as_str()) else { + continue; + }; + out.push_str(&text[at..hit.start()]); + let encoded; + let value = if json_fragment { + // Nested tool arguments are themselves JSON strings on the wire. + encoded = serde_json::to_string(value).map_err(|_| ())?; + &encoded[1..encoded.len() - 1] + } else { + value.as_str() + }; + self.budget = self.budget.checked_sub(value.len()).ok_or(())?; + out.push_str(value); + at = hit.end(); + } + if at != 0 { + out.push_str(&text[at..]); + *text = out; + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn whole_word_mode_also_bounds_date_variants() { + let rules = Rules::compile(Settings { + terms: vec!["1990-05-20".into()], + terms_whole: true, + ..Settings::default() + }) + .unwrap(); + let mut mask = Mask::new(&rules); + let original = "item1990-05-20draft | 1990-05-20"; + let mut payload = serde_json::json!({"messages":[{"role":"user","content":original}]}); + wire::request(wire::Format::Chat, &mut payload, &mut mask).unwrap(); + let redacted = payload + .pointer("/messages/0/content") + .and_then(serde_json::Value::as_str) + .unwrap(); + assert!(redacted.starts_with("item1990-05-20draft | ")); + assert_eq!(PLACEHOLDER.find_iter(redacted).count(), 1); + let mut restored = redacted.to_owned(); + mask.into_restore() + .unwrap() + .apply(&mut restored, false) + .unwrap(); + assert_eq!(restored, original); + } + + #[test] + fn structured_json_output_restores_quotes_as_json_values() { + let secret = "A\"B"; + let rules = Rules::compile(Settings { + terms: vec![secret.into()], + ..Settings::default() + }) + .unwrap(); + let mut mask = Mask::new(&rules); + let mut request = serde_json::json!({"model":"fixture","response_format":{"type":"json_object"},"messages":[{"role":"user","content":secret}]}); + wire::request(wire::Format::Chat, &mut request, &mut mask).unwrap(); + let mut translated = serde_json::json!({"input":[]}); + wire::request(wire::Format::Responses, &mut translated, &mut mask).unwrap(); + let token = request + .pointer("/messages/0/content") + .and_then(serde_json::Value::as_str) + .unwrap(); + let mut response = serde_json::json!({"choices":[{"message":{"role":"assistant","content":format!("{{\"name\":{}}}", serde_json::to_string(token).unwrap()),"reasoning_content":format!("reasoning {token}")}}]}); + let mut restore = mask.into_restore().unwrap(); + wire::response(wire::Format::Chat, &mut response, &mut restore).unwrap(); + let text = response + .pointer("/choices/0/message/content") + .and_then(serde_json::Value::as_str) + .unwrap(); + let json: serde_json::Value = serde_json::from_str(text).unwrap(); + assert_eq!(json["name"], secret); + assert_eq!( + response["choices"][0]["message"]["reasoning_content"], + format!("reasoning {secret}") + ); + } + + #[test] + fn categories_overlap_reuse_and_cross_request_isolation() { + let rules = Rules::compile(Settings { + secrets: true, + personal: true, + terms: vec!["nightjar".into(), "lp-example".into()], + ..Settings::default() + }) + .unwrap(); + let email = "alice@mailprobe.internal"; + let phone = "+8613900009931"; + let password = "Example1234"; + let key = concat!("sk-", "lp-example", "ABC123def456GHI789"); + let input = format!("nightjar {email} {phone} DB_PASSWORD={password} {key} lp-example"); + let mut mask = Mask::new(&rules); + let mut redacted = input.clone(); + mask.apply(&mut redacted, false).unwrap(); + let tokens = PLACEHOLDER + .find_iter(&redacted) + .map(|m| m.as_str()) + .collect::>(); + assert_eq!(tokens.len(), 6); + for expected in ["nightjar", email, phone, password, key, "lp-example"] { + assert!( + !redacted.contains(expected), + "expected range remained: {expected}" + ); + assert!( + input.contains(expected), + "fixture omitted expected range: {expected}" + ); + assert!( + mask.by_value.contains_key(expected), + "range had no mapping: {expected}" + ); + } + let mut repeated = input.to_owned(); + mask.apply(&mut repeated, false).unwrap(); + assert_eq!(repeated, redacted); + let mut second = Mask::new(&rules); + second.apply(&mut repeated, false).unwrap(); + let mut other = input.to_owned(); + second.apply(&mut other, false).unwrap(); + assert_ne!(redacted, other); + let mut saved = redacted.clone(); + second + .into_restore() + .unwrap() + .apply(&mut saved, false) + .unwrap(); + assert_eq!(saved, redacted); + mask.into_restore() + .unwrap() + .apply(&mut redacted, false) + .unwrap(); + assert_eq!(redacted, input); + } +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/stream.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/stream.rs new file mode 100644 index 000000000..0c28fe72f --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/stream.rs @@ -0,0 +1,450 @@ +//! Bounded SSE framing and per-choice/per-call restoration. The event envelope is kept +//! (event name, id, retry, comments and [DONE]); only whitelisted JSON string fields change. +use std::collections::{HashMap, VecDeque}; +use std::io; + +use axum::body::{Body, BodyDataStream}; +use bytes::{Bytes, BytesMut}; +use futures::{stream, StreamExt}; +use serde_json::Value; + +use super::{ + wire::{self, Format, Track}, + Restore, +}; + +const MAX_FRAME: usize = 4 * 1024 * 1024; +const MAX_WAIT_BYTES: usize = 2 * 1024 * 1024; +const MAX_WAIT_FRAMES: usize = 64; +fn line_parts(text: &str) -> Vec<(usize, usize, usize)> { + let bytes = text.as_bytes(); + let mut parts = Vec::new(); + let mut start = 0; + let mut at = 0; + while at < bytes.len() { + if bytes[at] == b'\r' || bytes[at] == b'\n' { + let content_end = at; + if bytes[at] == b'\r' && bytes.get(at + 1) == Some(&b'\n') { + at += 2; + } else { + at += 1; + } + parts.push((start, content_end, at)); + start = at; + } else { + at += 1; + } + } + if start < bytes.len() { + parts.push((start, bytes.len(), bytes.len())); + } + parts +} + +pub(crate) fn wrap(body: Body, format: Format, restore: Restore) -> Body { + let state = Decoder { + source: body.into_data_stream(), + format, + restore, + input: BytesMut::new(), + offset: 0, + line_start: 0, + first_frame: true, + waiting: Vec::new(), + waiting_bytes: 0, + fragments: HashMap::new(), + output: VecDeque::new(), + eof: false, + failed: false, + }; + Body::from_stream(stream::unfold(state, |mut state| async move { + match state.next_frame().await { + Ok(Some(frame)) => Some((Ok(frame), state)), + Ok(None) => None, + Err(error) => { + state.failed = true; + state.offset = 0; + state.line_start = 0; + state.output.clear(); + state.input.clear(); + state.waiting.clear(); + state.fragments.clear(); + Some((Err(error), state)) + } + } + })) +} + +fn invalid() -> io::Error { + io::Error::new( + io::ErrorKind::InvalidData, + "privacy masking: SSE output limit or invalid frame", + ) +} + +struct Frame { + original: Bytes, + json: Option, + changed: bool, + leading_bom: bool, +} + +impl Frame { + fn parse(original: Bytes, first: bool) -> Self { + let json = std::str::from_utf8(&original).ok().and_then(|raw| { + let text = if first { + raw.strip_prefix('\u{feff}').unwrap_or(raw) + } else { + raw + }; + let data = line_parts(text) + .into_iter() + .filter_map(|(start, end, _)| { + let line = &text[start..end]; + line.strip_prefix("data:") + .map(|part| part.strip_prefix(' ').unwrap_or(part)) + }) + .collect::>(); + if data.len() == 1 { + serde_json::from_str(data[0]).ok() + } else if !data.is_empty() { + serde_json::from_str(&data.join("\n")).ok() + } else { + None + } + }); + let leading_bom = first && original.starts_with(&[0xef, 0xbb, 0xbf]); + Self { + original, + json, + changed: false, + leading_bom, + } + } + + fn encode(self) -> io::Result { + if !self.changed { + return Ok(self.original); + } + let payload = serde_json::to_string(self.json.as_ref().ok_or_else(invalid)?) + .map_err(|_| invalid())?; + let old = std::str::from_utf8(&self.original).map_err(|_| invalid())?; + let mut out = String::with_capacity(old.len()); + let mut wrote = false; + for (start, content_end, end) in line_parts(old) { + let bom = start == 0 && self.leading_bom; + let content = &old[start + if bom { '\u{feff}'.len_utf8() } else { 0 }..content_end]; + if content.starts_with("data:") { + if wrote { + continue; + } + wrote = true; + if bom { + out.push('\u{feff}'); + } + out.push_str("data: "); + out.push_str(&payload); + out.push_str(&old[content_end..end]); + } else { + out.push_str(&old[start..end]); + } + } + Ok(Bytes::from(out)) + } +} + +struct Pending { + frame: usize, + path: String, + suffix: String, +} + +struct Decoder { + source: BodyDataStream, + format: Format, + restore: Restore, + input: BytesMut, + offset: usize, + line_start: usize, + first_frame: bool, + waiting: Vec, + waiting_bytes: usize, + fragments: HashMap, + output: VecDeque, + eof: bool, + failed: bool, +} + +impl Decoder { + async fn next_frame(&mut self) -> io::Result> { + if self.failed { + return Ok(None); + } + loop { + if let Some(frame) = self.output.pop_front() { + return Ok(Some(frame)); + } + if let Some(frame) = self.take_frame()? { + self.push(frame)?; + continue; + } + if self.eof { + if !self.input.is_empty() { + let tail = self.input.split().freeze(); + self.offset = 0; + self.line_start = 0; + self.push(tail)?; + } + self.flush()?; + return Ok(self.output.pop_front()); + } + match self.source.next().await { + Some(Ok(bytes)) => self.input.extend_from_slice(&bytes), + Some(Err(error)) => return Err(io::Error::other(error)), + None => self.eof = true, + } + } + } + + fn take_frame(&mut self) -> io::Result> { + let mut at = self.offset; + while at < self.input.len() { + if self.input[at] != b'\r' && self.input[at] != b'\n' { + at += 1; + continue; + } + if self.input[at] == b'\r' && at + 1 == self.input.len() && !self.eof { + self.offset = at; + return Ok(None); + } + let line_end = at; + let next = if self.input[at] == b'\r' && self.input.get(at + 1) == Some(&b'\n') { + at + 2 + } else { + at + 1 + }; + if next > MAX_FRAME { + return Err(invalid()); + } + if line_end == self.line_start { + let frame = self.input.split_to(next).freeze(); + self.offset = 0; + self.line_start = 0; + return Ok(Some(frame)); + } + self.line_start = next; + self.offset = next; + at = next; + } + if self.input.len() > MAX_FRAME { + return Err(invalid()); + } + self.offset = self.input.len(); + Ok(None) + } + + fn push(&mut self, bytes: Bytes) -> io::Result<()> { + let mut frame = Frame::parse(bytes, self.first_frame); + self.first_frame = false; + let seq = self.waiting.len(); + let mut terminal = false; + if let Some(value) = frame.json.as_mut() { + let original = value.clone(); + wire::response(self.format, value, &mut self.restore).map_err(|_| invalid())?; + terminal = matches!( + value.get("type").and_then(Value::as_str), + Some( + "message_stop" + | "response.completed" + | "response.failed" + | "response.incomplete" + | "error" + ) + ); + for delta in wire::deltas(self.format, value, self.restore.json_output()) { + let Some(Value::String(field)) = value.pointer_mut(&delta.path) else { + continue; + }; + if field.is_empty() { + continue; + } + let mut joined = self + .fragments + .remove(&delta.track) + .map_or_else(String::new, |previous| previous.suffix); + joined.push_str(field); + let n = self.restore.partial_len(&joined); + let tail = joined.split_off(joined.len() - n); + self.restore + .apply(&mut joined, delta.json) + .map_err(|_| invalid())?; + *field = joined; + if !tail.is_empty() { + self.fragments.insert( + delta.track, + Pending { + frame: seq, + path: delta.path, + suffix: tail, + }, + ); + } + } + frame.changed = *value != original; + } else if let Ok(frame_text) = std::str::from_utf8(&frame.original) { + terminal = frame_text.lines().any(|line| { + line.strip_prefix("data:") + .is_some_and(|s| s.trim() == "[DONE]") + }); + } + self.waiting_bytes += frame.original.len(); + self.waiting.push(frame); + let limit_reached = + self.waiting.len() >= MAX_WAIT_FRAMES || self.waiting_bytes >= MAX_WAIT_BYTES; + if limit_reached && !self.fragments.is_empty() && !terminal { + return Err(invalid()); + } + if self.fragments.is_empty() || terminal || limit_reached { + self.flush()?; + } + Ok(()) + } + + fn flush(&mut self) -> io::Result<()> { + // A naturally unfinished prefix is ordinary text: never drop it at EOF or at the cap. + for pending in self.fragments.drain().map(|(_, pending)| pending) { + let frame = self.waiting.get_mut(pending.frame).ok_or_else(invalid)?; + let field = frame + .json + .as_mut() + .and_then(|v| v.pointer_mut(&pending.path)); + let Some(Value::String(field)) = field else { + return Err(invalid()); + }; + field.push_str(&pending.suffix); + frame.changed = true; + } + for frame in self.waiting.drain(..) { + self.output.push_back(frame.encode()?); + } + self.waiting_bytes = 0; + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::server::core::privacy_masking::{RequestMask, Rules}; + + fn mapped_token() -> (super::super::RequestMask, String) { + let rules = crate::server::core::privacy_masking::Rules::compile( + crate::server::core::privacy_masking::Settings { + terms: vec!["stream-fixture".into()], + ..Default::default() + }, + ) + .unwrap(); + let mask = super::super::RequestMask::new(&rules); + let mut value = + serde_json::json!({"messages":[{"role":"user","content":"stream-fixture"}]}); + mask.apply(Format::Chat, &mut value).unwrap(); + let token = value + .pointer("/messages/0/content") + .unwrap() + .as_str() + .unwrap() + .to_owned(); + (mask, token) + } + + async fn collect( + source: impl futures::Stream> + Send + 'static, + restore: Restore, + ) -> io::Result { + let mut output = Vec::new(); + let mut wrapped = wrap(Body::from_stream(source), Format::Chat, restore).into_data_stream(); + while let Some(part) = wrapped.next().await { + output.extend_from_slice(&part.map_err(io::Error::other)?); + } + Ok(String::from_utf8(output).unwrap()) + } + + #[tokio::test] + async fn stream_start_bom_and_split_crlf_restore_first_frame() { + let (mask, token) = mapped_token(); + let json = serde_json::json!({"choices":[{"message":{"content":token}}]}).to_string(); + let body = format!("\u{feff}data: {json}\r\n\r\n"); + let bytes = body.as_bytes(); + let split = bytes.iter().rposition(|&byte| byte == b'\r').unwrap() + 1; + let pieces = vec![ + Ok(Bytes::copy_from_slice(&bytes[..split])), + Ok(Bytes::copy_from_slice(&bytes[split..])), + ]; + let result = collect(stream::iter(pieces), mask.restore()).await.unwrap(); + assert!(result.starts_with('\u{feff}')); + assert!(result.contains("stream-fixture")); + assert!(result.ends_with("\r\n\r\n")); + } + + #[tokio::test] + async fn later_bom_line_is_preserved_when_another_data_line_changes() { + let (mask, token) = mapped_token(); + let json = serde_json::json!({"choices":[{"message":{"content":token}}]}).to_string(); + let ignored = format!("{}data: ignored", char::from_u32(0xfeff).unwrap()); + let body = format!(": first\r\n\r\n{ignored}\r\ndata: {json}\r\n\r\n"); + let result = collect(stream::iter([Ok(Bytes::from(body))]), mask.restore()) + .await + .unwrap(); + assert!(result.contains(&ignored)); + assert!(result.contains("stream-fixture")); + } + + #[tokio::test] + async fn lone_cr_is_a_valid_sse_line_and_frame_terminator() { + let (mask, token) = mapped_token(); + let json = serde_json::json!({"choices":[{"message":{"content":token}}]}).to_string(); + let body = format!("data: {json}\r\r"); + let result = collect(stream::iter([Ok(Bytes::from(body))]), mask.restore()) + .await + .unwrap(); + assert!(result.contains("stream-fixture")); + assert!(result.ends_with("\r\r")); + } + + #[tokio::test] + async fn waiting_cap_errors_instead_of_flushing_partial_token() { + let (mask, token) = mapped_token(); + let split = token.len() / 2; + let event = serde_json::json!({"type":"response.output_text.delta","output_index":0,"content_index":0,"delta":&token[..split]}).to_string(); + let mut pieces: Vec> = + vec![Ok(Bytes::from(format!("data: {event}\r\n\r\n")))]; + pieces.extend( + (0..(MAX_WAIT_FRAMES - 1)).map(|_| Ok(Bytes::from_static(b": keepalive\r\n\r\n"))), + ); + let mut wrapped = wrap( + Body::from_stream(stream::iter(pieces)), + Format::Responses, + mask.restore(), + ) + .into_data_stream(); + assert!(wrapped.next().await.is_some_and(|event| event.is_err())); + assert!(wrapped.next().await.is_none()); + } + + #[tokio::test] + async fn errored_stream_fuses_after_partial_frame() { + let source = Body::from_stream(stream::iter([ + Ok(Bytes::from_static(b"data: {\"incomplete\":")), + Err(io::Error::other("synthetic source error")), + ])); + let mut wrapped = wrap( + source, + Format::Chat, + RequestMask::new(&Rules::default()).restore(), + ) + .into_data_stream(); + assert!(wrapped.next().await.is_some_and(|event| event.is_err())); + assert!(wrapped.next().await.is_none()); + } +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/wire.rs b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/wire.rs new file mode 100644 index 000000000..0c08edc4b --- /dev/null +++ b/desktop-tauri/src-tauri/server/src/server/core/privacy_masking/wire.rs @@ -0,0 +1,449 @@ +//! 协议字段白名单:层级明确,避免遍历 ID、签名、附件、认证或任意 metadata。 +use serde_json::{Map, Value}; + +use super::{is_secret_field, Mask, Restore}; + +#[derive(Clone, Copy)] +pub(crate) enum Format { + Chat, + Responses, + Messages, +} + +type Result = std::result::Result<(), ()>; + +fn each(value: Option<&mut Value>, mut visit: impl FnMut(&mut Value) -> Result) -> Result { + if let Some(Value::Array(items)) = value { + for item in items { + visit(item)?; + } + } + Ok(()) +} + +fn text(value: Option<&mut Value>, mut visit: impl FnMut(&mut String) -> Result) -> Result { + if let Some(Value::String(value)) = value { + visit(value)?; + } + Ok(()) +} + +fn kind(value: &Value) -> &str { + value.get("type").and_then(Value::as_str).unwrap_or("") +} + +enum Transform<'a> { + Mask(&'a mut Mask), + Restore(&'a mut Restore), +} + +impl Transform<'_> { + fn text(&mut self, value: Option<&mut Value>) -> Result { + text(value, |s| match self { + Self::Mask(mask) => mask.apply(s, false), + Self::Restore(restore) => restore.apply(s, false), + }) + } + + // Only assistant output follows the declared structured-output format; + // descriptions, reasoning and refusal strings remain ordinary text. + fn output_text(&mut self, value: Option<&mut Value>) -> Result { + if let Self::Restore(restore) = self { + if restore.json_output() { + return text(value, |s| { + if s.is_empty() { + return Ok(()); + } + let Ok(mut decoded) = serde_json::from_str::(s) else { + // A truncated or unsupported JSON-mode response still belongs + // to the caller. Preserve it and escape any restored fragments. + return restore.apply(s, true); + }; + let original = decoded.clone(); + restore_json(restore, &mut decoded)?; + if original != decoded { + *s = serde_json::to_string(&decoded).map_err(|_| ())?; + } + Ok(()) + }); + } + } + self.text(value) + } + + fn output_content(&mut self, value: Option<&mut Value>) -> Result { + match value { + Some(value @ Value::String(_)) => self.output_text(Some(value)), + Some(Value::Array(blocks)) => blocks.iter_mut().try_for_each(|block| self.block(block)), + _ => Ok(()), + } + } + + fn arguments(&mut self, value: Option<&mut Value>) -> Result { + let Some(value) = value else { return Ok(()) }; + if let Value::String(s) = value { + if let Ok(mut decoded @ (Value::Object(_) | Value::Array(_) | Value::String(_))) = + serde_json::from_str::(s) + { + let original = decoded.clone(); + self.tool_values(&mut decoded, None, false)?; + if original != decoded { + *s = serde_json::to_string(&decoded).map_err(|_| ())?; + } + } else { + self.text(Some(value))?; + } + } else { + self.tool_values(value, None, false)?; + } + Ok(()) + } + + fn tool_values(&mut self, value: &mut Value, field: Option<&str>, in_secret: bool) -> Result { + let secret_field = in_secret || field.is_some_and(is_secret_field); + match value { + Value::String(s) => match self { + Self::Mask(mask) => { + let named = secret_field + && !field.is_some_and(|key| super::detectors::tool_lookup(key, s)); + mask.apply(s, named) + } + Self::Restore(restore) => restore.apply(s, false), + }, + Value::Array(items) => items + .iter_mut() + .try_for_each(|v| self.tool_values(v, field, secret_field)), + Value::Object(fields) => fields + .iter_mut() + .try_for_each(|(key, v)| self.tool_values(v, Some(key), secret_field)), + _ => Ok(()), + } + } + + fn content(&mut self, value: Option<&mut Value>) -> Result { + match value { + Some(value @ Value::String(_)) => self.text(Some(value)), + Some(Value::Array(blocks)) => blocks.iter_mut().try_for_each(|block| self.block(block)), + _ => Ok(()), + } + } + + // Tool results can carry a JSON document in a string or a protocol text block. + // Reuse the argument walker for the document, not for the surrounding envelope. + fn result_content(&mut self, value: Option<&mut Value>) -> Result { + match value { + Some(value @ Value::String(_)) => self.arguments(Some(value)), + Some(Value::Array(blocks)) => { + blocks.iter_mut().try_for_each(|block| match kind(block) { + "text" | "input_text" | "output_text" => self.arguments(block.get_mut("text")), + _ => self.block(block), + }) + } + _ => Ok(()), + } + } + + fn block(&mut self, block: &mut Value) -> Result { + match kind(block) { + "text" | "output_text" => self.output_text(block.get_mut("text")), + "input_text" => self.text(block.get_mut("text")), + "refusal" => self.text(block.get_mut("refusal")), + "tool_use" | "server_tool_use" | "mcp_tool_use" => { + self.arguments(block.get_mut("input")) + } + "mcp_approval_request" => self.arguments(block.get_mut("arguments")), + "tool_result" | "mcp_tool_result" => self.result_content(block.get_mut("content")), + "function_call" | "mcp_call" => self.arguments(block.get_mut("arguments")), + "custom_tool_call" => self.text(block.get_mut("input")), + "function_call_output" | "custom_tool_call_output" => { + self.result_content(block.get_mut("output")) + } + // Responses Lite 和工具搜索把声明放在 input 项里,不只在顶层 tools。 + item if item.eq_ignore_ascii_case("additional_tools") + || item == "tool_search_output" => + { + each(block.get_mut("tools"), |tool| self.tool_definition(tool)) + } + "message" => self.message(block), + "" if block.get("role").is_some() => self.message(block), + // Images, audio, documents, thinking/signature/encrypted_content are opaque. + _ => Ok(()), + } + } + + fn message(&mut self, message: &mut Value) -> Result { + if matches!( + message.get("role").and_then(Value::as_str), + Some("tool" | "function") + ) { + self.result_content(message.get_mut("content"))?; + } else { + self.output_content(message.get_mut("content"))?; + } + self.text(message.get_mut("refusal"))?; + self.text(message.get_mut("reasoning_content"))?; + self.arguments(message.pointer_mut("/function_call/arguments"))?; + each(message.get_mut("tool_calls"), |call| { + self.arguments(call.pointer_mut("/function/arguments"))?; + self.text(call.pointer_mut("/custom/input")) + }) + } + + fn tool_definition(&mut self, tool: &mut Value) -> Result { + self.text(tool.get_mut("description"))?; + self.text(tool.pointer_mut("/function/description"))?; + for path in ["/parameters", "/input_schema", "/function/parameters"] { + if let Some(schema) = tool.pointer_mut(path) { + self.schema(schema)?; + } + } + // 命名空间只递归子工具声明;名字、格式约束和语法不参加规则扫描。 + if kind(tool).eq_ignore_ascii_case("namespace") { + each(tool.get_mut("tools"), |child| self.tool_definition(child))?; + } + Ok(()) + } + + fn schema(&mut self, schema: &mut Value) -> Result { + self.text(schema.get_mut("description"))?; + self.text(schema.get_mut("title"))?; + for key in [ + "properties", + "patternProperties", + "$defs", + "definitions", + "dependentSchemas", + ] { + if let Some(Value::Object(fields)) = schema.get_mut(key) { + for field in fields.values_mut() { + self.schema(field)?; + } + } + } + for key in ["allOf", "anyOf", "oneOf", "prefixItems"] { + each(schema.get_mut(key), |v| self.schema(v))?; + } + for key in [ + "items", + "additionalProperties", + "not", + "if", + "then", + "else", + "contains", + ] { + if let Some(Value::Object(field)) = schema.get_mut(key) { + self.schema_object(field)?; + } + } + Ok(()) + } + + fn schema_object(&mut self, value: &mut serde_json::Map) -> Result { + let mut object = Value::Object(std::mem::take(value)); + self.schema(&mut object)?; + if let Value::Object(inner) = object { + *value = inner; + } + Ok(()) + } +} + +fn restore_json(restore: &mut Restore, value: &mut Value) -> Result { + match value { + Value::String(text) => restore.apply(text, false), + Value::Array(items) => items + .iter_mut() + .try_for_each(|item| restore_json(restore, item)), + Value::Object(fields) => { + let old = std::mem::take(fields); + let mut restored = Map::new(); + for (mut key, mut item) in old { + restore.apply(&mut key, false)?; + restore_json(restore, &mut item)?; + if restored.insert(key, item).is_some() { + return Err(()); + } + } + *fields = restored; + Ok(()) + } + _ => Ok(()), + } +} +pub(crate) fn request(format: Format, payload: &mut Value, mask: &mut Mask) -> Result { + // Capture the client mode once. Later protocol translation or continuation + // may omit it but must not change how the public response is restored. + mask.json_output.get_or_insert_with(|| match format { + Format::Chat => payload + .pointer("/response_format/type") + .and_then(Value::as_str) + .is_some_and(|kind| kind == "json_object" || kind == "json_schema"), + Format::Responses => payload + .pointer("/text/format/type") + .and_then(Value::as_str) + .is_some_and(|kind| kind == "json_object" || kind == "json_schema"), + Format::Messages => payload + .pointer("/output_config/format/type") + .and_then(Value::as_str) + .is_some_and(|kind| kind == "json_schema"), + }); + let mut transform = Transform::Mask(mask); + match format { + Format::Chat | Format::Messages => { + each(payload.get_mut("messages"), |message| { + transform.message(message) + })?; + if matches!(format, Format::Messages) { + transform.content(payload.get_mut("system"))?; + } + each(payload.get_mut("functions"), |tool| { + transform.tool_definition(tool) + })?; + } + Format::Responses => { + transform.content(payload.get_mut("input"))?; + transform.text(payload.get_mut("instructions"))?; + } + } + each(payload.get_mut("tools"), |tool| { + transform.tool_definition(tool) + }) +} + +pub(crate) fn response(format: Format, payload: &mut Value, restore: &mut Restore) -> Result { + let mut transform = Transform::Restore(restore); + match format { + Format::Chat => each(payload.get_mut("choices"), |choice| { + match choice.get_mut("message") { + Some(message) => transform.message(message), + None => Ok(()), + } + }), + Format::Messages => { + transform.content(payload.get_mut("content"))?; + transform.content(payload.pointer_mut("/message/content"))?; + if let Some(block) = payload.get_mut("content_block") { + transform.block(block)?; + } + Ok(()) + } + Format::Responses => { + transform.content(payload.get_mut("output"))?; + transform.content(payload.pointer_mut("/response/output"))?; + transform.text(payload.get_mut("instructions"))?; + transform.text(payload.pointer_mut("/response/instructions"))?; + each(payload.get_mut("tools"), |tool| { + transform.tool_definition(tool) + })?; + each(payload.pointer_mut("/response/tools"), |tool| { + transform.tool_definition(tool) + })?; + for key in ["item", "part"] { + if let Some(block) = payload.get_mut(key) { + transform.block(block)?; + } + } + match kind(payload) { + "response.output_text.done" => transform.output_text(payload.get_mut("text")), + "response.refusal.done" => transform.text(payload.get_mut("refusal")), + "response.function_call_arguments.done" | "response.mcp_call_arguments.done" => { + transform.arguments(payload.get_mut("arguments")) + } + "response.custom_tool_call_input.done" => transform.text(payload.get_mut("input")), + _ => Ok(()), + } + } + } +} + +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] +pub(super) struct Track(u64, u64, &'static str); + +pub(super) struct Delta { + pub(super) track: Track, + pub(super) path: String, + pub(super) json: bool, +} + +fn index(value: &Value, field: &str, fallback: u64) -> u64 { + value.get(field).and_then(Value::as_u64).unwrap_or(fallback) +} + +pub(super) fn deltas(format: Format, event: &Value, json_output: bool) -> Vec { + let mut result = Vec::new(); + let mut add = |track: Track, path: String, json: bool| { + if event.pointer(&path).is_some_and(Value::is_string) { + result.push(Delta { track, path, json }); + } + }; + match format { + Format::Chat => { + if let Some(Value::Array(choices)) = event.get("choices") { + for (i, choice) in choices.iter().enumerate() { + let n = index(choice, "index", i as u64); + let path = format!("/choices/{i}/delta"); + for field in ["content", "refusal", "reasoning_content"] { + add( + Track(n, 0, field), + format!("{path}/{field}"), + field == "content" && json_output, + ); + } + add( + Track(n, 0, "legacy_arguments"), + format!("{path}/function_call/arguments"), + true, + ); + if let Some(Value::Array(calls)) = choice.pointer("/delta/tool_calls") { + for (j, call) in calls.iter().enumerate() { + let c = index(call, "index", j as u64); + add( + Track(n, c, "arguments"), + format!("{path}/tool_calls/{j}/function/arguments"), + true, + ); + add( + Track(n, c, "custom"), + format!("{path}/tool_calls/{j}/custom/input"), + false, + ); + } + } + } + } + } + Format::Messages if kind(event) == "content_block_delta" => { + let n = index(event, "index", 0); + match event.pointer("/delta/type").and_then(Value::as_str) { + Some("text_delta") => add(Track(n, 0, "text"), "/delta/text".into(), json_output), + Some("input_json_delta") => { + add(Track(n, 0, "arguments"), "/delta/partial_json".into(), true) + } + _ => {} + } + } + Format::Responses => { + let n = index(event, "output_index", 0); + let c = index(event, "content_index", 0); + match kind(event) { + "response.output_text.delta" => { + add(Track(n, c, "text"), "/delta".into(), json_output) + } + "response.refusal.delta" => add(Track(n, c, "refusal"), "/delta".into(), false), + "response.function_call_arguments.delta" => { + add(Track(n, c, "arguments"), "/delta".into(), true) + } + "response.mcp_call_arguments.delta" => { + add(Track(n, c, "mcp_arguments"), "/delta".into(), true) + } + "response.custom_tool_call_input.delta" => { + add(Track(n, c, "custom"), "/delta".into(), false) + } + _ => {} + } + } + _ => {} + } + result +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/title_routing/mod.rs b/desktop-tauri/src-tauri/server/src/server/core/title_routing/mod.rs index 08709e904..832bdd4e4 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/title_routing/mod.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/title_routing/mod.rs @@ -168,6 +168,13 @@ pub fn client_for_request( body: &Value, responses: bool, ) -> Option<&'static str> { + // Behavioral classification uses display names, while stored route clients use IDs. + let client = match client { + "Claude Code" => CLIENT_CLAUDE_CODE, + "Codex" => CLIENT_CODEX, + "ZCode" => CLIENT_ZCODE, + other => other, + }; let settings = title_routes::active()?; // 这家客户端得有一条真会命中的规则(自己开着、勾了它、配了模型)才值得往下判 —— // 提示词判据要遍历消息结构,不便宜。 diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/native.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/native.rs index c45ebdeb6..c0dd1bcbe 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/native.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/native.rs @@ -192,7 +192,7 @@ async fn collect_native( capture.push(&bytes); } buffer.extend_from_slice(&bytes); - while let Some(end) = frame_end(&buffer) { + while let Some(end) = frame_end(&buffer, false) { if end > MAX_EVENT_BYTES { return Err(GatewayError::with_status( 502, @@ -215,6 +215,19 @@ async fn collect_native( )); } } + while let Some(end) = frame_end(&buffer, true) { + if end > MAX_EVENT_BYTES { + return Err(GatewayError::with_status( + 502, + "上游原生事件超过 8 MiB,已停止接收", + )); + } + let frame: Vec<_> = buffer.drain(..end).collect(); + if let Some(event) = event_json(&frame)? { + observer.observe(protocol, &event, telemetry); + collector.push(&event, frame.len())?; + } + } if !buffer.is_empty() { return Err(GatewayError::with_status( 502, @@ -224,26 +237,44 @@ async fn collect_native( collector.finish() } -fn frame_end(buffer: &[u8]) -> Option { - let mut start = 0; - for (index, byte) in buffer.iter().enumerate() { - if *byte != b'\n' { - continue; - } - let line = &buffer[start..index]; - if line.is_empty() || line == b"\r" { - return Some(index + 1); +// Keep byte ranges so model rewriting preserves CR, LF and CRLF exactly. +fn line_parts(bytes: &[u8]) -> Vec<(usize, usize, usize)> { + let (mut start, mut at) = (0, 0); + let mut parts = Vec::new(); + while at < bytes.len() { + if matches!(bytes[at], b'\r' | b'\n') { + let end = at; + at += if bytes[at] == b'\r' && bytes.get(at + 1) == Some(&b'\n') { + 2 + } else { + 1 + }; + parts.push((start, end, at)); + start = at; + } else { + at += 1; } - start = index + 1; } - None + if start < bytes.len() { + parts.push((start, bytes.len(), bytes.len())); + } + parts +} +fn frame_end(buffer: &[u8], eof: bool) -> Option { + line_parts(buffer) + .into_iter() + .find_map(|(start, end, next)| { + (start == end + && next > end + && !(!eof && buffer[end] == b'\r' && next == end + 1 && next == buffer.len())) + .then_some(next) + }) } - fn event_json(bytes: &[u8]) -> Result, GatewayError> { let mut data = Vec::new(); let mut has_data = false; - for line in bytes.split(|byte| *byte == b'\n') { - let line = line.strip_suffix(b"\r").unwrap_or(line); + for (start, end, _) in line_parts(bytes) { + let line = &bytes[start..end]; if let Some(value) = line.strip_prefix(b"data:") { if has_data { data.push(b'\n'); @@ -378,8 +409,8 @@ impl NativeStream { return; } let mut event_name = String::new(); - for line in bytes.split(|byte| *byte == b'\n') { - let line = line.strip_suffix(b"\r").unwrap_or(line); + for (start, end, _) in line_parts(&bytes) { + let line = &bytes[start..end]; if let Some(value) = line.strip_prefix(b"event:") { event_name = String::from_utf8_lossy(value).trim().to_string(); } @@ -447,45 +478,52 @@ impl NativeStream { out.extend_from_slice(format!("event: {kind}\n").as_bytes()); } let mut written = false; - for line in bytes.split_inclusive(|byte| *byte == b'\n') { + for (start, end, next) in line_parts(&bytes) { + let line = &bytes[start..end]; if line.starts_with(b"data:") { if !written { out.extend_from_slice(b"data: "); out.extend_from_slice(&encoded); - if line.ends_with(b"\r\n") { - out.extend_from_slice(b"\r\n"); - } else if line.ends_with(b"\n") { - out.push(b'\n'); - } + out.extend_from_slice(&bytes[end..next]); written = true; } } else { - out.extend_from_slice(line); + out.extend_from_slice(&bytes[start..next]); } } self.pending.push_back(Bytes::from(out)); } - fn feed(&mut self, bytes: &[u8]) -> Result<(), io::Error> { + fn feed(&mut self, bytes: &[u8], eof: bool) -> Result<(), io::Error> { self.buffer.extend_from_slice(bytes); let mut start = 0; let mut line_start = self.line_start; let mut frames = Vec::new(); - for (index, byte) in self.buffer.iter().enumerate().skip(self.scan_from) { - if *byte != b'\n' { + let mut at = self.scan_from; + while at < self.buffer.len() { + if !matches!(self.buffer[at], b'\r' | b'\n') { + at += 1; continue; } - let line = &self.buffer[line_start..index]; - if line.is_empty() || line == b"\r" { - if index + 1 - start > MAX_EVENT_BYTES { + if self.buffer[at] == b'\r' && at + 1 == self.buffer.len() && !eof { + break; + } + let end = at; + at += if self.buffer[at] == b'\r' && self.buffer.get(at + 1) == Some(&b'\n') { + 2 + } else { + 1 + }; + if end == line_start { + if at - start > MAX_EVENT_BYTES { return Err(io::Error::other("上游原生事件超过 8 MiB,已停止接收")); } - frames.push(self.buffer[start..=index].to_vec()); - start = index + 1; + frames.push(self.buffer[start..at].to_vec()); + start = at; } - line_start = index + 1; + line_start = at; } - self.scan_from = self.buffer.len().saturating_sub(start); + self.scan_from = at.saturating_sub(start); self.line_start = line_start.saturating_sub(start); if start > 0 { self.buffer.drain(..start); @@ -529,12 +567,15 @@ impl Stream for NativeStream { if !bytes.is_empty() { self.telemetry.note_first_frame(); } - if let Err(error) = self.feed(&bytes) { + if let Err(error) = self.feed(&bytes, false) { self.fail(error); } } Poll::Ready(Some(Err(error))) => self.fail(error), Poll::Ready(None) => { + if let Err(error) = self.feed(&[], true) { + self.fail(error); + } self.ended = true; if !self.buffer.is_empty() { self.buffer.clear(); diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/native/tests.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/native/tests.rs index 7247bfe5a..4f4d21888 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/native/tests.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/native/tests.rs @@ -102,6 +102,28 @@ async fn signed_anthropic_stream_is_byte_exact_at_every_split() { } } +#[tokio::test] +async fn lone_cr_survives_transport_splits_and_model_rewriting() { + let text = anthropic().replace('\n', "\r"); + for size in [1, 2, 7, 113] { + let telemetry = Arc::new(RequestTelemetry::new()); + let output = drain(stream_with( + source(chunks(&text, size)), + WireProtocol::Anthropic, + "alias", + telemetry.clone(), + )) + .await; + let output = String::from_utf8(output).unwrap(); + let expected = text.replace("\"model\":\"wire\"", "\"model\":\"alias\""); + assert_eq!( + output, expected, + "complete SSE sequence changed at chunk size {size}" + ); + assert_eq!(telemetry.snapshot().total_tokens, 68); + assert!(telemetry.snapshot().error.is_none()); + } +} #[tokio::test] async fn unicode_and_crlf_survive_one_byte_chunks_and_model_aliasing() { let text = anthropic().replace('\n', "\r\n"); diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/native_payload.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/native_payload.rs index 62faa51d6..37b66cc36 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/native_payload.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/native_payload.rs @@ -66,7 +66,7 @@ pub(super) fn prepare( provider: &str, account: Option<&Value>, degraded: bool, -) -> super::payload::SendBody<'static> { +) -> Result, GatewayError> { let mut body = native.routed(ctx.body); if ctx.compact_routed || ctx.title_routed { crate::server::core::compact_routing::parameters::clear_source_reasoning(&mut body); @@ -162,11 +162,19 @@ pub(super) fn prepare( } } } - super::payload::SendBody { + if let Some(masking) = ctx.telemetry.privacy_mask() { + use crate::server::core::privacy_masking::wire::Format; + masking.apply(match native.protocol { + WireProtocol::Chat => Format::Chat, + WireProtocol::Responses => Format::Responses, + WireProtocol::Anthropic => Format::Messages, + }, &mut body)?; + } + Ok(super::payload::SendBody { body: std::borrow::Cow::Owned(body), wire_model, wire_reasoning: None, - } + }) } fn apply_prompt( diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/payload.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/payload.rs index bbd3804d8..87d5bee95 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/payload.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/payload.rs @@ -201,7 +201,7 @@ pub(super) async fn send_body<'a>( provider_id: &str, account: Option<&Value>, degraded: bool, -) -> SendBody<'a> { +) -> Result, crate::server::errors::GatewayError> { if ctx.images.is_some() { let requested = ctx.body.get("model").and_then(Value::as_str).unwrap_or(""); let wire = crate::server::core::providers::catalog::wire_target_for_provider( @@ -211,7 +211,7 @@ pub(super) async fn send_body<'a>( rewrite_model(&mut body, requested, &wire.model, provider_id); ctx.telemetry.note_upstream_model(&wire.model); ctx.telemetry.note_upstream_reasoning(None); - return SendBody { body, wire_model: wire.model, wire_reasoning: None }; + return Ok(SendBody { body, wire_model: wire.model, wire_reasoning: None }); } // ── ⓪ 角色归一层(开发者角色兼容开关)─────────────────────────── // 在所有改写**之前**:先把 developer 角色归一成 system,后面各层(提示词、 @@ -346,6 +346,9 @@ pub(super) async fn send_body<'a>( None => body, } }; + if let Some(masking) = ctx.telemetry.privacy_mask() { + masking.apply(crate::server::core::privacy_masking::wire::Format::Chat, body.to_mut())?; + } let requested = body .get("model") .and_then(Value::as_str) @@ -355,7 +358,7 @@ pub(super) async fn send_body<'a>( if requested.is_empty() { // 没有 model 字段:不改写,也没有可用的冷却键(空串,与改造前一致) ctx.telemetry.note_upstream_reasoning(None); - return SendBody { body, wire_model: requested, wire_reasoning: None }; + return Ok(SendBody { body, wire_model: requested, wire_reasoning: None }); } // 一次解析出两个属性:该家要收的名字 + 跟着那条映射走的思考等级 // (同源,见模块头「思考等级绑定为什么也在这一步」) @@ -378,7 +381,7 @@ pub(super) async fn send_body<'a>( // 未注入时读取原有客户端参数;绑定成功时以覆盖后的等级为准。 let upstream_reasoning = injected.or_else(|| outbound_reasoning_of(provider_id, &body)); ctx.telemetry.note_upstream_reasoning(upstream_reasoning.clone()); - SendBody { body, wire_model: wire.model, wire_reasoning: upstream_reasoning } + Ok(SendBody { body, wire_model: wire.model, wire_reasoning: upstream_reasoning }) } /// 承载家的 [`ProviderAdapter::outbound_reasoning`](读发送体里随行的等级)。 @@ -594,4 +597,4 @@ pub(super) async fn apply_reasoning( None } } -} \ No newline at end of file +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/provider_loop.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/provider_loop.rs index 2453e1601..5c466b505 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/provider_loop.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/provider_loop.rs @@ -348,6 +348,14 @@ pub(super) fn transient_retry_advice(status: u16, remaining: usize) -> Option(ctx: &ProviderContext<'_>, result: Result) -> Result { + result.inspect_err(|error| { + ctx.telemetry.finish_last_attempt(Some(i64::from(error.status_code)), Some(&error.message)); + }) +} + /// 传输层失败(DNS / 代理 / 连接)的统一退避建议:与瞬时 HTTP 错误同一套设置。 /// /// 原因取错误自带的简短形态(`UpstreamRequestError::reason`):连接超时与 @@ -1066,7 +1074,8 @@ async fn attempt_queue( let cache_key = (provider_id, account_pool.clone(), selected_model.clone(), degraded); if !send_cache.contains_key(&cache_key) { let built = if let Some(native) = subscription_native { - let mut prepared = super::native_payload::prepare(ctx, native, provider_id, target.account.as_ref(), degraded); + let mut prepared = finish_preparation(ctx, super::native_payload::prepare( + ctx, native, provider_id, target.account.as_ref(), degraded))?; let requested = model_of(ctx.body); let wire = crate::server::core::providers::catalog::wire_target_for_provider(&requested, provider_id, target.account.as_ref()); if ctx.direct_model.is_none() && target.account.as_ref().and_then(crate::server::core::routing_groups::candidate_model).is_none() { @@ -1078,7 +1087,7 @@ async fn attempt_queue( } prepared } else { - send_body(ctx, provider_id, target.account.as_ref(), degraded).await + finish_preparation(ctx, send_body(ctx, provider_id, target.account.as_ref(), degraded).await)? }; send_cache.insert(cache_key.clone(), built); } @@ -1989,11 +1998,15 @@ async fn attempt_custom( // **即将发送**。`send_body` 对自定义 id 是零改写(它的模型名改写只认 // modelRules),处理结果就是「提示词/脱敏后的客户端请求体」—— 自定义 // 语义的改写(映射 alias → 真名、思考等级)在 forward 里做。 - let native = if super::native_payload::eligible(ctx, &provider_id)? { ctx.native } else { None }; - let send = match native { + let native = if finish_preparation(ctx, super::native_payload::eligible(ctx, &provider_id))? { + ctx.native + } else { + None + }; + let send = finish_preparation(ctx, match native { Some(native) => super::native_payload::prepare(ctx, native, &provider_id, target.account.as_ref(), degraded), None => send_body(ctx, &provider_id, target.account.as_ref(), degraded).await, - }; + })?; // 「上游模型」列以**真名**为准:send_body 对自定义 id 是零改写(它记的 // 是请求名),真名的解析与改写发生在 forward 内部 —— 这里按同源解析 // 覆盖一次(note_upstream_model 是覆盖式,最后一次为准;空串被内部过滤)。 @@ -2244,7 +2257,7 @@ async fn attempt_stateful( // `degraded` 由调用方(账号循环)给出:本路径**没有**就地补救(有状态 // provider 一次转发就是一个会话轮次,没有「换提示词重发」这一步), // 但降级期内(状态机已生效)首发的提示词也要跟着换。 - let send = send_body(ctx, provider_id, target.account.as_ref(), degraded).await; + let send = send_body(ctx, provider_id, target.account.as_ref(), degraded).await?; // 工具失败标记由实际出站桥接消费,不能提前清掉。 let web_tools = crate::server::config::web_tools::enabled() && adapter.list_models().iter() .any(|model| super::web_tools::Profile::from_model(model).is_some()); @@ -2299,7 +2312,8 @@ async fn attempt_stateful( crate::server::core::plugin_host::script::native::protocol(provider_id, wire_model) == Some(native.protocol) && native.protocol != crate::server::core::protocol::native::WireProtocol::Chat }) { - let prepared = super::native_payload::prepare(ctx, native, provider_id, target.account.as_ref(), degraded); + let prepared = finish_preparation(ctx, super::native_payload::prepare( + ctx, native, provider_id, target.account.as_ref(), degraded))?; crate::server::core::plugin_host::script::native::forward(provider_id, target.account_id.as_deref().unwrap_or(""), prepared.body.into_owned(), ctx.client_headers, target.proxy.clone(), native.protocol, &model, ctx.stream, ctx.telemetry).await @@ -2678,4 +2692,4 @@ mod local_failure { assert_eq!(502, error.status_code); assert_eq!(None, error.retry_after_secs, "内容型不该带 Retry-After"); } -} \ No newline at end of file +} diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/usage.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/usage.rs index 1752bd234..e9b1d2a80 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/usage.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/usage.rs @@ -487,6 +487,7 @@ pub struct RequestTelemetry { /// 请求安全状态独立于统计快照;attempt、首响及用量清理均不能替换或解除它。 replay_guard: Arc, model_key: Option>, + privacy_mask: Option, prompt_request: Mutex>, catpaw_output: Mutex>>, catpaw_output_taken: std::sync::atomic::AtomicBool, @@ -519,6 +520,15 @@ impl Default for RequestTelemetry { } impl RequestTelemetry { + pub(crate) fn with_privacy_mask(mut self, state: Option) -> Self { + self.privacy_mask = state; + self + } + + pub(super) fn privacy_mask(&self) -> Option<&crate::server::core::privacy_masking::RequestMask> { + self.privacy_mask.as_ref() + } + pub fn new() -> Self { Self::with_replay_guard(Arc::new(super::replay_guard::ReplayGuard::new())) } @@ -528,6 +538,7 @@ impl RequestTelemetry { inner: Mutex::new(TelemetrySnapshot::default()), replay_guard, model_key: None, + privacy_mask: None, prompt_request: Mutex::new(None), catpaw_output: Mutex::new(None), catpaw_output_taken: std::sync::atomic::AtomicBool::new(false), @@ -555,6 +566,7 @@ impl RequestTelemetry { pub(crate) fn child_with_id(&self) -> Self { let mut telemetry = Self::with_replay_guard(Arc::clone(&self.replay_guard)); telemetry.model_key = self.model_key.clone(); + telemetry.privacy_mask = self.privacy_mask.clone(); telemetry.prompt_request = Mutex::new(self.prompt_request()); telemetry.ensure_id(&super::request::new_request_id()); telemetry diff --git a/desktop-tauri/src-tauri/server/src/server/core/upstream/vision_compat/assist.rs b/desktop-tauri/src-tauri/server/src/server/core/upstream/vision_compat/assist.rs index 23440953c..aa4908593 100644 --- a/desktop-tauri/src-tauri/server/src/server/core/upstream/vision_compat/assist.rs +++ b/desktop-tauri/src-tauri/server/src/server/core/upstream/vision_compat/assist.rs @@ -136,7 +136,8 @@ pub(super) async fn run(service: &UpstreamService, ctx: &ProviderContext<'_>, or if ctx.telemetry.is_cancelled() { return Err(super::super::cancellation::cancelled_error()); } let key = cache::key(&[&request_fingerprint, &vision.revision, &target.provider, &target.model, &target.reasoning, &image.reference, image.detail.as_deref().unwrap_or("")]); - if rule.reuse_results { + let request_is_masked = ctx.telemetry.privacy_mask().is_some(); + if rule.reuse_results && !request_is_masked { if let Some(text) = cache::get(&key) { detail["cachedImages"] = (detail["cachedImages"].as_u64().unwrap_or(0) + 1).into(); descriptions.push(text); @@ -162,7 +163,7 @@ pub(super) async fn run(service: &UpstreamService, ctx: &ProviderContext<'_>, or GatewayError::with_status(error.status_code, format!("辅助识图失败:{};未删除图片或切换处理方式", error.message)).with_code("vision_assist_failed") })?; - if rule.reuse_results { cache::put(key, text.clone()); } + if rule.reuse_results && !request_is_masked { cache::put(key, text.clone()); } descriptions.push(text); } let mut body = content::replace(ctx.body, &images, &descriptions)?; diff --git a/desktop-tauri/src-tauri/server/src/server/http.rs b/desktop-tauri/src-tauri/server/src/server/http.rs index 95e7dff13..68916e643 100644 --- a/desktop-tauri/src-tauri/server/src/server/http.rs +++ b/desktop-tauri/src-tauri/server/src/server/http.rs @@ -711,6 +711,7 @@ pub fn panel_router(state: ServerState) -> Router { // 「安装」同样能 spawn 任意二进制,敏感度与 /api/plugins/install 同级。 .route("/api/plugins/release", get(api::plugins::release)) .route("/api/plugins/install-from-release", post(api::plugins::install_from_release)) + .route("/api/privacy-masking", get(api::privacy_masking::get).put(api::privacy_masking::put)) // 出站词表与提供商范围修改请求正文,必须沿用管理面鉴权。 .route( "/api/sanitize", @@ -1034,6 +1035,7 @@ pub fn gateway_router(state: ServerState) -> Router { .route("/mcp/image-generation", post(api::tool_runtime::mcp_image_generation) .get(api::tool_runtime::mcp_unsupported).delete(api::tool_runtime::mcp_unsupported)) .route("/mcp/image-generation/files/{id}", get(api::tool_images::download)) + .layer(middleware::from_fn(api::privacy_masking::protect)) .layer(middleware::from_fn(api::capacity_admission::admit)) .layer(middleware::from_fn(require_api_key)) .with_state(state); diff --git a/desktop-tauri/src-tauri/server/src/web_shim.rs b/desktop-tauri/src-tauri/server/src/web_shim.rs index 270da75f3..579d7353c 100644 --- a/desktop-tauri/src-tauri/server/src/web_shim.rs +++ b/desktop-tauri/src-tauri/server/src/web_shim.rs @@ -2197,6 +2197,9 @@ pub fn shim_js() -> &'static str { return call('POST', '/api/scheduled-tasks/' + encodeURIComponent(String(id || '')) + '/run', {}); }, + // ── Magpie 风格本地隐私脱敏 ── + getPrivacyMasking: function () { return call('GET', '/api/privacy-masking'); }, + savePrivacyMasking: function (settings) { return call('PUT', '/api/privacy-masking', settings || {}); }, // ── 软件更新(检查 / 下载 / 容器内更新;只有「跑安装程序」是桌面专有)── checkUpdate: function () { return invokeShell('check_update'); }, getUpdateStatus: function () { return call('GET', '/api/update/status'); }, diff --git a/desktop-tauri/src-tauri/src/bridge.rs b/desktop-tauri/src-tauri/src/bridge.rs index c72469d49..671bbf548 100644 --- a/desktop-tauri/src-tauri/src/bridge.rs +++ b/desktop-tauri/src-tauri/src/bridge.rs @@ -738,6 +738,10 @@ const BRIDGE_JS: &str = r#" saveSanitize: patch => call('PUT', '/api/sanitize', typeof patch === 'boolean' ? { sanitizeBlacklistFingerprints: patch } : patch), + // ── Magpie 风格本地隐私脱敏 ── + getPrivacyMasking: () => call('GET', '/api/privacy-masking'), + savePrivacyMasking: settings => call('PUT', '/api/privacy-masking', settings), + // ── 开发者角色兼容开关 ── // 与 getSanitize / saveSanitize 同形:GET 读、PUT 写,响应体就是新状态。 // 与 web_shim.rs 的同名方法逐字同形(漏一侧的症状见下面 getCors 那段)。 diff --git a/desktop-tauri/ui-islands/src/islands/settings-model.ts b/desktop-tauri/ui-islands/src/islands/settings-model.ts index 68a7b68fe..6960d0e7c 100644 --- a/desktop-tauri/ui-islands/src/islands/settings-model.ts +++ b/desktop-tauri/ui-islands/src/islands/settings-model.ts @@ -342,6 +342,8 @@ export type SettingsBridge = { saveDebug(enabled: boolean): Promise getSanitize(): Promise saveSanitize(patch: boolean | SanitizePatch): Promise + getPrivacyMasking?(): Promise + savePrivacyMasking?(settings: { revision: string; secrets: boolean; personal: boolean; terms: string[]; kinds?: Record; terms_any_case?: boolean; terms_whole?: boolean; allow?: string[]; rules?: unknown[] }): Promise getWebToolEmulation?(): Promise saveWebToolEmulation?(enabled: boolean): Promise getDeveloperRoleCompat(): Promise diff --git a/desktop-tauri/ui-islands/src/islands/settings-page.tsx b/desktop-tauri/ui-islands/src/islands/settings-page.tsx index 1c7a34abe..d0fa8045f 100644 --- a/desktop-tauri/ui-islands/src/islands/settings-page.tsx +++ b/desktop-tauri/ui-islands/src/islands/settings-page.tsx @@ -123,6 +123,7 @@ import { GraphicsSettingsPanel } from './settings-graphics' import { DbRestoreDialog } from './settings-db-restore' import { DatabaseBackupDialogs, DatabaseBackupPanel } from './settings-database-backup' import { SanitizePanel } from './settings-sanitize' +import { PrivacyMaskingPanel } from './settings-privacy-masking' import { WebToolsPanel } from './settings-web-tools' import { ClientIdentificationPanel } from '../../../ui-shared/client-identification-panel' import { ProviderSettingsPane } from './settings-providers' @@ -1071,6 +1072,7 @@ function SecurityPane({ snap }: { snap: SettingsSnapshot }) { return ( <> + } className='panel'> >; terms_any_case?: boolean; terms_whole?: boolean; allow?: string[]; rules?: Rule[] } +const kinds: readonly [ExtraKind, string, boolean][] = [ + ['ssn', '美国社会安全号码(SSN)', true], ['passport', '护照号', true], ['iban', 'IBAN', true], + ['birthday', '生日', true], ['mac', 'MAC 地址', true], ['serial', '序列号', true], + ['home', '主目录用户名', false], ['userhost', '登录用户名和主机名', false], ['ip', '公网 IP', false], ['bucket', '云存储桶', false], +] +function normalizeLines(value: string) { + return [...new Set(value.split(/\r?\n/).map(line => line.trim()).filter(Boolean))] +} +const initial: Policy = { revision: '', secrets: false, personal: false, terms: [] } +function parse(value: unknown): Policy { + if (!value || typeof value !== 'object') throw new Error(t('服务端未返回本地隐私脱敏设置')) + const x = value as Record + if (typeof x.secrets !== 'boolean' || typeof x.personal !== 'boolean' + || !Array.isArray(x.terms) || !x.terms.every(term => typeof term === 'string') + || typeof x.revision !== 'string' || !/^[0-9a-f]{64}$/.test(x.revision)) { + throw new Error(t('本地隐私脱敏设置格式无效')) + } + return x as Policy +} +const ruleText = (policy: Policy) => JSON.stringify(policy.rules ?? [], null, 2) + +export function PrivacyMaskingPanel() { + const [saved, setSaved] = React.useState(null) + const [draft, setDraft] = React.useState(initial) + const [allowText, setAllowText] = React.useState('') + const [rulesText, setRulesText] = React.useState('[]') + const [loading, setLoading] = React.useState(true) + const [busy, setBusy] = React.useState(false) + const [error, setError] = React.useState('') + const [notice, setNotice] = React.useState('') + const pending = React.useRef(false) + const editDraft = (update: (current: Policy) => Policy) => { setError(''); setDraft(update) } + const editAllowText = (value: string) => { setError(''); setAllowText(value) } + const editRulesText = (value: string) => { setError(''); setRulesText(value) } + + const load = React.useCallback(async () => { + if (pending.current) return + pending.current = true + setLoading(true) + setBusy(true) + setError('') + setNotice('') + try { + const api = shared().workbuddyDesktop + if (!api?.getPrivacyMasking) throw new Error(t('当前后端暂不支持本地隐私脱敏')) + const value = parse(await api.getPrivacyMasking()) + setSaved(value) + setDraft(value) + setAllowText((value.allow ?? []).join('\n')) + setRulesText(ruleText(value)) + } catch (failure) { setSaved(null); setError(errorMessage(failure)) } + finally { pending.current = false; setLoading(false); setBusy(false) } + }, []) + React.useEffect(() => { void load() }, [load]) + + async function save() { + if (pending.current || !saved) return + const terms = [...new Set(draft.terms.map(term => term.trim()).filter(Boolean))] + const encoder = new TextEncoder() + if (terms.length > 100 || terms.some(term => encoder.encode(term).length > 256)) { + setError(t('自定义脱敏词最多 100 个,每个不超过 256 字节')) + return + } + pending.current = true + setBusy(true) + setError('') + setNotice('') + try { + const api = shared().workbuddyDesktop + if (!api?.savePrivacyMasking) throw new Error(t('当前后端暂不支持本地隐私脱敏')) + let rules: unknown + try { rules = JSON.parse(rulesText || '[]') } + catch { throw new Error(t('自定义脱敏规则必须是 JSON 数组')) } + if (!Array.isArray(rules)) throw new Error(t('自定义脱敏规则必须是 JSON 数组')) + const value = parse(await api.savePrivacyMasking({ ...draft, terms, allow: normalizeLines(allowText), rules, revision: saved.revision })) + setSaved(value) + setDraft(value) + setAllowText((value.allow ?? []).join('\n')) + setRulesText(ruleText(value)) + setNotice(t('设置已保存:从下一个推理请求生效')) + } catch (failure) { setError(t(errorMessage(failure))) } + finally { pending.current = false; setBusy(false) } + } + + async function refresh() { + if (pending.current) return + if (dirty) { + pending.current = true + setBusy(true) + try { + const ask = window.wbConfirm?.ask + if (!ask) throw new Error(t('确认弹窗不可用,请重新打开页面')) + if (!await ask({ title: t('放弃未保存的修改?'), + text: t('刷新将丢弃尚未保存的本地隐私脱敏修改,是否继续?'), + okText: t('放弃修改并刷新'), cancelText: t('取消'), bodyClass: '' })) return + } catch (failure) { setError(t(errorMessage(failure))); return } + finally { pending.current = false; setBusy(false) } + } + await load() + } + + const active = !!saved && (saved.secrets || saved.personal || saved.terms.length > 0) + const dirty = !!saved && (JSON.stringify(saved) !== JSON.stringify(draft) + || allowText !== (saved.allow ?? []).join('\n') || rulesText !== ruleText(saved)) + const locked = loading || busy || !saved + const labelId = React.useId() + React.useEffect(() => { if (dirty) setNotice('') }, [dirty]) + return } className='panel' id='settings-privacy-masking'> + +
+

{t('本地隐私脱敏')}

+ {loading ? t('正在读取设置…') : saved ? t(active ? '已开启' : '已关闭') : t('不可用')} +
+ +
+ +

+ {t('默认关闭。仅保护经此网关的 Chat Completions、Responses 和 Messages,返回给客户端的文本和工具参数会还原。')} +

+ + + {draft.personal ?
+ {t('附加个人信息分类')} + {kinds.map(([key, label, fallback]) => )} +
: null} +
+ +