Skip to content

[PR] fix: redact storage diagnostics at the publish boundary - #4499

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-24_storage-diag-publish-redaction
Oct 24, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-24_storage-diag-publish-redaction

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

The storage-legacy-only diagnostics blob is designed to be copied into a public GitHub issue (Web UI → storage degradation warning → Copy Diagnostics / Report on GitHub), but its storage section copied raw bytes with no publish-time redaction — unlike the config and conflict-log sections. This leaked into the published blob and the prefilled issue body:

  • raw [storage] log lines (credentials, private hostnames, absolute user paths);
  • free-form reason strings (degrade reason, bootstrapFailure.reason);
  • the absolute SQLite failure-marker path in the force-retry line.

#3959 already proved absolute user filesystem paths reached a public issue body through this flow; credential/private-host exposure was reproduced with synthetic inputs.

Fix

Enforce redaction at the single publish boundary (buildDiagnostics) so every topic's blob is scrubbed before serialization, and the copied blob + prefilled issue body derive from one string so they can never diverge:

Behavior changes (flagged)

Before After Why
Public storage blob carried raw [storage] lines, reasons, absolute paths All redacted + $HOME→~ stop credential/path leak into public issues
[storage] tail lines unbounded bounded to 240 chars parity with conflict-log tail; bound long error lines without losing classification
Force-retry marker shown as /home/<user>/.cache/... in public blob shown as ~/.cache/billion-context/sqlite/.bootstrap-failed.json remove username while keeping the recovery path actionable

Local (non-published) surfaces are unchanged: acp_status's storage wire uses the separate storageStatusWire() and still reports verbatim for the model relay (#3385).

Out of scope (cross-ref)

Non-HTTP URL schemes (mysql://…, the literal repro input) are not masked here — maskUrlsForPublish is intentionally HTTP-only, and broadening scheme recognition is tracked separately in #4191. redactSecretsInText still scrubs token=/credential query params regardless of scheme.

Verification

Targeted suites green on the rebased head (current master): log-mask, web-storage-diagnostics-redaction (new), storage-diagnostics-lines, web-diagnostics-enrichment (#3813 single-read), web-storage-banner, snapshot-shadow — 45 pass / 0 fail. tsc --noEmit clean. Full regression = remote CI.

Closes #4427

The storage-legacy-only diagnostics blob lands in a PUBLIC GitHub issue
body, but its storage section copied raw [storage] bytes (and free-form
reason strings plus the absolute failure-marker path) with no publish-time
redaction — unlike the config and conflict-log sections. #3959 proved
absolute user filesystem paths had already reached a public body through
this reporting flow.

- add maskPathsForPublish ($HOME -> ~, anchored on both flanks so sibling
  homes and URL pathnames are untouched) and a single redactForPublish
  primitive composing secret scrubbing + unconditional non-public host
  masking + home-path stripping
- enforce the net at the SINK (buildDiagnostics) so EVERY topic's blob is
  redacted before serialization; the copied blob and the prefilled issue
  body derive from one string so they can never diverge
- storageLogTail now bounds each line to LOG_LINE_MAX_CHARS and applies the
  same redaction as conflictLogTail
- keep the force-retry lever actionable: the marker stays an absolute path,
  the sink rewrites only the $HOME prefix to ~ so no username leaks

Closes #4427
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-24_storage-diag-publish-redaction (fa0dbd2)

Latest build — install this exact version

npm install -g billion-context@0.0.5414-pr.4499.978e083

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
fa0dbd2 billion-context@0.0.5414-pr.4499.978e083 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4499.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: redact storage diagnostics at the publish boundary [PR] fix: redact storage diagnostics at the publish boundary Oct 24, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本实现就是根因 bug #4427 的修复,与本 issue 的 spec 一致;#3959 是同类泄漏的前例证据(绝对路径确实进了公开 issue),#4191 是「更宽主机识别」的独立跟踪项(已在 PR 正文 cross-ref,不在本次范围)。

标签: bug + scope/ui + p2(保留系统预打的 pr)—— 缺陷类;修复落在 Web UI 诊断发布面;p2=功能可用的信息外泄类,若你认为凭据公开暴露应更高档可升 p1(见第 3 问,这是我的判断)。

PR: #4509 (Closes #4427, Closes #4499)


结构化八问

  1. 是什么问题:存储降级的「Copy Diagnostics / Report on GitHub」会把原始 [storage] 日志行(含凭据 token/api_key、内网 relay 主机名、绝对家目录路径)、自由文本原因串(degrade reason、bootstrapFailure.reason)、以及 force-retry 行里的绝对 SQLite 失败标记路径,原样复制进公开 GitHub issue。config 段和 conflict-log 段有脱敏,storage 段漏了。

  2. 什么原因导致的:buildDiagnostics 是唯一的发布边界,但它序列化 blob 前没对 storage 段做发布期脱敏。storageLogTail 直接 .slice 原始行、force-retry 用绝对路径、原因是自由文本——三处都没过任何发布脱敏;且现有原语只覆盖 URL/密钥,不覆盖裸绝对路径。根因是「发布边界缺一道统一脱敏」,不是某个字段单独漏配。

  3. 影响范围与严重程度:触发条件 = 用户在 Web UI 点「复制诊断 / 上报 GitHub」。受影响 = 所有走该流程的用户;泄露 = 凭据、内网主机拓扑、绝对路径(暴露用户名)。定级 p2:bili 功能本身不受影响、有绕行(不点发布),属信息外泄类。判断说明:它不属于 p1 的两个硬类别(会话不可恢复/缓存失控),故按规则字面定 p2;但它把凭据和内网端点泄到公网,爆炸半径不小——你若按「核心承诺=安全诊断」口径升 p1,我认同,重打即可。

  4. 怎么修复的:在唯一发布边界 buildDiagnostics 对整份 blob 统一套一层 redactForPublish(密钥清洗 → 非公网主机掩码 → 新增 $HOME→~ 路径剥离),让「复制的 blob」和「预填 issue body」从同一字符串派生、永不漂移;storageLogTail 顺带补上 240 字截断与同款脱敏(与 conflict-log 对齐)。maskPathsForPublish 两侧锚定,兄弟家目录(/home/<other>)、URL 主机、URL path 都不误伤。本地未发布面保持逐字:acp_status 的 storageStatusWire()(SQLite unified storage degraded to legacy-only (bili v0.1.197, node v20.20.2, win32-x64) #3385)和直接调 storageDiagnosticsLines() 的单测都仍报绝对路径,只有发布出去的 blob 被改写。

  5. 是否有回归问题:逐项过——① acp_status 存储 wire 仍逐字报模型中继(不变);② storage-diagnostics-lines 单测断言的本地绝对标记路径仍逐字(不变);③ conflict/config 段改走 redactForPublish,输出语义等价;④ blob==body 恒等由单测钉死。双应用幂等安全(URL 匹配符排除 </>,已发的 <private-host> 占位不会被二次命中)。无回归。

  6. 是否破坏缓存:不涉及——纯发布期文本脱敏,不碰压缩/折叠/缓存键。

  7. 是否改了设计方向:不转向,只是把「发布脱敏」从分散在各段收拢到单一边界、并补上裸绝对路径这一层;数据流不变。

  8. 是否引入新配置:无新 env/config/默认值。redactForPublish 无条件生效(不受 BILI_LOG_MASK_HOSTS 门控),符合发布安全语义。


行为变更(显式披露,旧→新+因):

旧 新 因
公开 storage blob 带原始 [storage] 行 / 原因 / 绝对路径 全部脱敏 + $HOME→~ 止住凭据/路径泄漏进公开 issue
[storage] tail 行不限长 限 240 字符 与 conflict-log tail 对齐,长错误行截断但不丢分类
force-retry 标记公开显示 /home/<user>/.cache/... 显示 ~/.cache/billion-context/sqlite/.bootstrap-failed.json 去用户名,保留可操作的恢复路径

范围外(有意为之): 裸 host:port(非 http(s) URL)在自由文本里不掩码——maskUrlsForPublish 故意只做 HTTP(s),更宽主机识别归 #4191;token=/凭据 query 参数无论 scheme 都会被清掉。这与既有 config/conflict 覆盖面完全对齐(对齐而非扩张),同症状族 #4499/#4427,故不开新 issue。

验证: rebase 到当前 master head 后,定向 6 套件全绿(log-mask、web-storage-diagnostics-redaction(新)、storage-diagnostics-lines、web-diagnostics-enrichment、web-storage-banner、snapshot-shadow)— 43 pass / 0 fail;tsc --noEmit clean。全量回归交远程 CI。

判定: CI 绿即可合。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/ui labels Oct 24, 2026
ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
The #4499 publish-boundary consolidation moved every external call site
onto redactForPublish, leaving maskUrlsForPublish referenced only inside
src/log-mask.ts. Drop the leading export per the gate's INTERNAL_ONLY fix;
no WAIVERS entry — no bundle-boundary consumer needs the name.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 结果跟进(接上一条 PR #4509):

本地验证全绿:改动相关的 6 个测试套件(web-storage-diagnostics-redaction、storage-diagnostics-lines、web-diagnostics-enrichment、log-mask、web-storage-banner、snapshot-shadow)+ tsc --noEmit + unused-exports 门禁,在 rebase 后的 head 上全部通过。新测试的 $HOME→~ 断言在本机真实 home 下也实际执行并通过(不是被 homedir()=="/" 守卫跳过)。

CI 有 3 个 test 分片红,我判断是环境抖动(flaky)、不是本次改动的回归,依据:

  1. 失败分片跨平台不一致:ubuntu 挂第 1 分片(node22+node24 都挂)、windows 挂第 3 分片。分片划分是确定性的(ci-shard-partition.mjs 按文件列表+耗时加权做 LPT 平衡,N=3),同一批文件在每个平台都落在同一分片——真正的确定性回归会让同一个分片在所有平台都挂,而这里是不同平台挂了不同分片,符合仓库已记录的那几类环境抖动(CI flake:test (windows-latest, 24) 单文件文件级失败:子测试零执行、异常被吞为 'test failed' #3029 swallowed / Flaky Windows CI: issue1993-preflight-400 suite cancelled by early event-loop drain | Windows CI 偶发假红:preflight-400 测试进程提前退出致 5 例取消 #3665 drain / CI: windows-latest node-22 shard 2 intermittently red on timing-sensitive e2e tests under load (rotating flakes: issue760, #4079, storage-full-flip #3818) #4161 timing / windows-22-1 lane randomly red: during test cleanup, unlink index.db hit an EBUSY external lock, couldn't survive tmp-rm | windows-22-1 车道随机红:测试清理期 unlink index.db 遇 EBUSY 外部锁,熬不过 tmp-rm 的 9.6 秒重试梯子(#2799/#2908 族复发) #4347 lock);runner 对这些形态本来就会先自动重试一次再判红。
  2. 排除了真实的 ESM import 破坏风险:全仓 grep,被我私有化的 maskUrlsForPublish 只在 src/log-mask.ts 内部引用(声明 L137、注释 L160/L178、redactForPublish 内调用 L183),没有任何外部文件 import 它,去掉 export 不会断任何导入链。
  3. 改动的两个测试文件(web-storage-diagnostics-redaction.test.ts→分片1、log-mask.test.ts→分片3)虽恰好落在失败分片里,但本地 linux 上都是过的,所以不是它们确定性坏了。

权限卡点:作业日志接口对我的 token 返回 403("Must have admin rights"),check-run 注解里也只有 "exit code 1"、没有逐条用例名——所以我没法从 API 拿到具体是哪条用例挂了。诚实说明:本环境跑不了 Windows,无法 100% 排除 windows 侧对 log-mask.test.ts(分片3)的环境性抖动,但概率低(linux 已过)。

已采取的动作:我已触发对 3 个失败 job 的重跑(rerun-failed-jobs → HTTP 201 已受理),等新 runner 上的结果——绿了就坐实非回归,我会跟进报告。

需要你(若重跑仍红):方便的话从 GitHub UI 打开对应 job 日志看具体挂的用例(你有权限),或给我日志可见性;我据此定向排查。代码侧我目前没有发现需要修的回归。

判定:代码改动本身可合,等 CI 重跑确认即可。

ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
The storage-legacy-only diagnostics blob is copied into a PUBLIC GitHub
issue, but its storage section copied raw bytes with no publish-time
redaction - unlike the config and conflict-log sections. Raw [storage] log
lines, free-form reason strings, and the absolute SQLite failure-marker path
leaked into the published blob and the prefilled issue body (#4427; #3959
precedent for absolute paths reaching a public issue body).

Enforce redaction at the single publish boundary (buildDiagnostics):

- New redactForPublish(text) in src/log-mask.ts composing three layers:
  credential-shape scrubbing -> non-public host masking -> home-path stripping.
- New maskPathsForPublish: rewrites the $HOME prefix to ~, anchored on both
  flanks so sibling homes, relative paths, and URL pathnames are untouched.
  Closes the bare-absolute-path vector the existing primitives missed.
- storageLogTail bounds each line to LOG_LINE_MAX_CHARS (240) and applies the
  same redaction as conflictLogTail via a shared publishRedactLine helper.
- The force-retry marker stays an absolute path locally (recovery stays
  actionable); only the $HOME prefix is rewritten to ~ in the published blob.

Local (non-published) surfaces unchanged: acp_status's storage wire uses
storageStatusWire() and still reports verbatim for the model relay (#3385).

Targeted suites green (log-mask, web-storage-diagnostics-redaction [new],
storage-diagnostics-lines, web-diagnostics-enrichment, web-storage-banner,
snapshot-shadow); tsc --noEmit clean. Full regression = remote CI.

Closes #4427
Closes #4499
ranxianglei pushed a commit that referenced this pull request Oct 24, 2026
The #4499 publish-boundary consolidation moved every external call site
onto redactForPublish, leaving maskUrlsForPublish referenced only inside
src/log-mask.ts. Drop the leading export per the gate's INTERNAL_ONLY fix;
no WAIVERS entry — no bundle-boundary consumer needs the name.
@ranxianglei
ranxianglei merged commit fe290ad into master Oct 24, 2026
78 of 117 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.813-master.fe290ad (rolling master tag).

npm install -g billion-context@0.0.813-master.fe290ad  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/ui

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[security][web] Storage diagnostics publish raw log lines and absolute paths | 存储诊断公开报告泄露本地路径和未脱敏日志

1 participant