Repository navigation
[PR] fix: redact storage diagnostics at the publish boundary - #4499
Conversation
The storage-legacy-only diagnostics blob lands in a PUBLIC GitHub issue body, but its storage section copied raw [storage] bytes (and free-form reason strings plus the absolute failure-marker path) with no publish-time redaction — unlike the config and conflict-log sections. #3959 proved absolute user filesystem paths had already reached a public body through this reporting flow. - add maskPathsForPublish ($HOME -> ~, anchored on both flanks so sibling homes and URL pathnames are untouched) and a single redactForPublish primitive composing secret scrubbing + unconditional non-public host masking + home-path stripping - enforce the net at the SINK (buildDiagnostics) so EVERY topic's blob is redacted before serialization; the copied blob and the prefilled issue body derive from one string so they can never diverge - storageLogTail now bounds each line to LOG_LINE_MAX_CHARS and applies the same redaction as conflictLogTail - keep the force-retry lever actionable: the marker stays an absolute path, the sink rewrites only the $HOME prefix to ~ so no username leaks Closes #4427
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5414-pr.4499.978e083This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4499.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 —— 本实现就是根因 bug #4427 的修复,与本 issue 的 spec 一致;#3959 是同类泄漏的前例证据(绝对路径确实进了公开 issue),#4191 是「更宽主机识别」的独立跟踪项(已在 PR 正文 cross-ref,不在本次范围)。 标签: PR: #4509 (Closes #4427, Closes #4499) 结构化八问
行为变更(显式披露,旧→新+因):
范围外(有意为之): 裸 验证: rebase 到当前 master head 后,定向 6 套件全绿( 判定: CI 绿即可合。 |
The #4499 publish-boundary consolidation moved every external call site onto redactForPublish, leaving maskUrlsForPublish referenced only inside src/log-mask.ts. Drop the leading export per the gate's INTERNAL_ONLY fix; no WAIVERS entry — no bundle-boundary consumer needs the name.
(ai: vllm-qwen/qwen3.8-27b) CI 结果跟进(接上一条 PR #4509): 本地验证全绿:改动相关的 6 个测试套件(web-storage-diagnostics-redaction、storage-diagnostics-lines、web-diagnostics-enrichment、log-mask、web-storage-banner、snapshot-shadow)+ CI 有 3 个 test 分片红,我判断是环境抖动(flaky)、不是本次改动的回归,依据:
权限卡点:作业日志接口对我的 token 返回 403("Must have admin rights"),check-run 注解里也只有 "exit code 1"、没有逐条用例名——所以我没法从 API 拿到具体是哪条用例挂了。诚实说明:本环境跑不了 Windows,无法 100% 排除 windows 侧对 已采取的动作:我已触发对 3 个失败 job 的重跑(rerun-failed-jobs → HTTP 201 已受理),等新 runner 上的结果——绿了就坐实非回归,我会跟进报告。 需要你(若重跑仍红):方便的话从 GitHub UI 打开对应 job 日志看具体挂的用例(你有权限),或给我日志可见性;我据此定向排查。代码侧我目前没有发现需要修的回归。 判定:代码改动本身可合,等 CI 重跑确认即可。 |
The storage-legacy-only diagnostics blob is copied into a PUBLIC GitHub issue, but its storage section copied raw bytes with no publish-time redaction - unlike the config and conflict-log sections. Raw [storage] log lines, free-form reason strings, and the absolute SQLite failure-marker path leaked into the published blob and the prefilled issue body (#4427; #3959 precedent for absolute paths reaching a public issue body). Enforce redaction at the single publish boundary (buildDiagnostics): - New redactForPublish(text) in src/log-mask.ts composing three layers: credential-shape scrubbing -> non-public host masking -> home-path stripping. - New maskPathsForPublish: rewrites the $HOME prefix to ~, anchored on both flanks so sibling homes, relative paths, and URL pathnames are untouched. Closes the bare-absolute-path vector the existing primitives missed. - storageLogTail bounds each line to LOG_LINE_MAX_CHARS (240) and applies the same redaction as conflictLogTail via a shared publishRedactLine helper. - The force-retry marker stays an absolute path locally (recovery stays actionable); only the $HOME prefix is rewritten to ~ in the published blob. Local (non-published) surfaces unchanged: acp_status's storage wire uses storageStatusWire() and still reports verbatim for the model relay (#3385). Targeted suites green (log-mask, web-storage-diagnostics-redaction [new], storage-diagnostics-lines, web-diagnostics-enrichment, web-storage-banner, snapshot-shadow); tsc --noEmit clean. Full regression = remote CI. Closes #4427 Closes #4499
The #4499 publish-boundary consolidation moved every external call site onto redactForPublish, leaving maskUrlsForPublish referenced only inside src/log-mask.ts. Drop the leading export per the gate's INTERNAL_ONLY fix; no WAIVERS entry — no bundle-boundary consumer needs the name.
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.813-master.fe290ad # this exact build
npm install -g billion-context@master # always the newest merged build
|
What & why
The
storage-legacy-onlydiagnostics blob is designed to be copied into a public GitHub issue (Web UI → storage degradation warning → Copy Diagnostics / Report on GitHub), but its storage section copied raw bytes with no publish-time redaction — unlike the config and conflict-log sections. This leaked into the published blob and the prefilled issue body:[storage]log lines (credentials, private hostnames, absolute user paths);degrade reason,bootstrapFailure.reason);#3959 already proved absolute user filesystem paths reached a public issue body through this flow; credential/private-host exposure was reproduced with synthetic inputs.
Fix
Enforce redaction at the single publish boundary (
buildDiagnostics) so every topic's blob is scrubbed before serialization, and the copied blob + prefilled issue body derive from one string so they can never diverge:redactForPublish(text)primitive insrc/log-mask.tscomposing three layers: secret scrubbing (redactSecretsInText) → non-public host masking (maskUrlsForPublish) → home-path stripping (newmaskPathsForPublish).maskPathsForPublish: rewrites$HOME→~, anchored on both flanks so sibling homes (/home/<other>), URL hosts, and URL pathnames are untouched. This closes the bare-absolute-path vector the existing primitives did not cover (SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959).storageLogTailnow bounds each line toLOG_LINE_MAX_CHARS(240) and applies the same redaction asconflictLogTail.$HOMEprefix to~, so no username leaks publicly.Behavior changes (flagged)
[storage]lines, reasons, absolute paths$HOME→~[storage]tail lines unbounded/home/<user>/.cache/...in public blob~/.cache/billion-context/sqlite/.bootstrap-failed.jsonLocal (non-published) surfaces are unchanged:
acp_status's storage wire uses the separatestorageStatusWire()and still reports verbatim for the model relay (#3385).Out of scope (cross-ref)
Non-HTTP URL schemes (
mysql://…, the literal repro input) are not masked here —maskUrlsForPublishis intentionally HTTP-only, and broadening scheme recognition is tracked separately in #4191.redactSecretsInTextstill scrubstoken=/credential query params regardless of scheme.Verification
Targeted suites green on the rebased head (current master):
log-mask,web-storage-diagnostics-redaction(new),storage-diagnostics-lines,web-diagnostics-enrichment(#3813 single-read),web-storage-banner,snapshot-shadow— 45 pass / 0 fail.tsc --noEmitclean. Full regression = remote CI.Closes #4427