Repository navigation
[PR] fix: redact storage diagnostics at the publish boundary - #4509
ranxianglei wants to merge 9 commits into
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.9853-pr.4509.991f29fThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4509.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: N/A —— 既有 PR 的 CI 自动修复车道(第 1/2 次尝试),未新立条目。
结构化八问:
验证:本地跑 |
(ai: vllm-qwen/qwen3.8-27b) 查重: 同家族非重复 —— 与 PR #4499(同标题、同修复对象、早约 70 分钟创建)是同一机制的两个 PR:#4509 正文显式 "Closes #4499"(取代关系),#4499 的 CI 测试矩阵为红,#4509 门禁全绿且测试更完整。按先来后到本应保留先到的,但后到者显式声明取代且质量占优,建议合 #4509、由 owner 关闭 #4499(我不执行关闭)。父 issue #4427 是被修对象,不构成重复。 标签: bug + p1 + scope/ui(凭据/私有主机名/用户主目录路径经「公开诊断报告」流程泄进公共 GitHub issue = 安全泄露类,按 owner 规则泄露类一律 p1;缺陷面在 Web 诊断面板的发布边界) 独立审核结论(diff 逐段通读,关键路径 grep/本地实测验证,未照抄 issue 侧表述):
验证与处置记录:
CI 状态:head 判定:方向正确(单一发布边界、疏不堵)、回归面逐项清点完毕、测试有牙、行为变更全部如实披露、out-of-scope 边界(HTTP-only 主机识别,同 #4191 口径)合理。可合;合并后请 owner 关闭被取代的 #4499。 |
The storage-legacy-only diagnostics blob is copied into a PUBLIC GitHub issue, but its storage section copied raw bytes with no publish-time redaction - unlike the config and conflict-log sections. Raw [storage] log lines, free-form reason strings, and the absolute SQLite failure-marker path leaked into the published blob and the prefilled issue body (#4427; #3959 precedent for absolute paths reaching a public issue body). Enforce redaction at the single publish boundary (buildDiagnostics): - New redactForPublish(text) in src/log-mask.ts composing three layers: credential-shape scrubbing -> non-public host masking -> home-path stripping. - New maskPathsForPublish: rewrites the $HOME prefix to ~, anchored on both flanks so sibling homes, relative paths, and URL pathnames are untouched. Closes the bare-absolute-path vector the existing primitives missed. - storageLogTail bounds each line to LOG_LINE_MAX_CHARS (240) and applies the same redaction as conflictLogTail via a shared publishRedactLine helper. - The force-retry marker stays an absolute path locally (recovery stays actionable); only the $HOME prefix is rewritten to ~ in the published blob. Local (non-published) surfaces unchanged: acp_status's storage wire uses storageStatusWire() and still reports verbatim for the model relay (#3385). Targeted suites green (log-mask, web-storage-diagnostics-redaction [new], storage-diagnostics-lines, web-diagnostics-enrichment, web-storage-banner, snapshot-shadow); tsc --noEmit clean. Full regression = remote CI. Closes #4427 Closes #4499
The #4499 publish-boundary consolidation moved every external call site onto redactForPublish, leaving maskUrlsForPublish referenced only inside src/log-mask.ts. Drop the leading export per the gate's INTERNAL_ONLY fix; no WAIVERS entry — no bundle-boundary consumer needs the name.
…ign docs with anchor reality Review of the #4427 publish-boundary redaction found the maskPathsForPublish doc claiming the anchors "never corrupt a neighbour token", while dot/hyphen sibling homes (${HOME}.bak, ${HOME}-bob) ARE rewritten to ~… — a safe-side over-mask (no leak), but not "untouched" as documented. Tightening the right flank to spare them would under-mask a real home followed by punctuation (e.g. sentence-final period), so the behaviour stays as-is: the doc now states the actual contract and two assertions pin the over-masking so a future "fix" cannot silently flip it into an under-mask.
…ate) Master-side gap: ad98e7e (#4491) hand-wrote the pluginLaneServerHistory rows into CONFIGURATION.md/.zh-CN.md without a YAML seed entry, so the docs-sync generator drops them and every PR merge commit goes red. Add the missing seed entry and regenerate; this normalizes the two hand-written rows to generator output (inline-code backticks dropped from descriptions, which the seed schema forbids; ZH default cell unified to the shared English form used by sibling rows; env-map rows added). Docs-only wording change, no runtime behavior touched. Branch rebased onto current master as part of landing the self-consistent tree.
fa1d924 to
38f2bde
Compare
(ai: vllm-qwen/qwen3.8-27b) 查重: 按失败签名(pluginLaneServerHistory / docs-sync / 缺失种子条目)检索了 open+closed 的 issue 与 PR——只有原始功能条目 #4491/#4505,没有针对此缺口的既有修复 → 直接修(自动修复车道 2/2)。 本轮红灯的根因不在这个 PR,是 master 上留下的洞。已修复并推送到 PR 分支( 结构化八问:
验证:本地连跑两次 generate(第二次零 diff,幂等 ✓)、check-coverage 通过(180 个 env 名 + 58 个顶层键全部有文档)、unused-exports 保持 clean。等新 head 上 CI 重跑,docs-sync 预期转绿、上一轮修好的 unused-exports 保持绿。CI 转绿后即可人工审阅合并。 |
(ai: vllm-qwen/qwen3.8-27b) 补充一条(透明性说明):修 docs-sync 时发现仓库里已存在一个针对同一根因的分支 |
(ai: vllm-qwen/qwen3.8-27b) CI 跟进(head
|
(ai: vllm-qwen/qwen3.8-27b) 收工状态更新(本 PR 审核结论此前已发:可合;此处只补阻塞解除后的最新进展):
|
The suite baseline preload (preload-storage-baseline.mjs) pins BILI_STORAGE_UNIFIED="legacy" for the broad corpus. After #6480 flipped the storage default to pure-sqlite, the notice-registry suite's visibleNotices assertions expect the pure-sqlite default notice, which is gated on unifiedSelectionIsSqlite (unified "sqlite" or absent) — the preload's "legacy" made the notice silent, so the suite was born-red on master. Pin the selection explicitly (#4011 lesson: a test that depends on the production default passes locally but fails in CI).
# Conflicts: # CONFIGURATION.md # CONFIGURATION.zh-CN.md # src/log-mask.ts # src/web/diagnostics.ts # tests/web-storage-diagnostics-redaction.test.ts # website/config-reference/rollout.yaml
[bot] 🏷 Merge conflict resolved — the PR is now mergeable. I merged Key finding: this PR is a parallel implementation of the same fix master already shipped as #4427, and master's version is an evolved superset. All of master's relevant commits postdate the PR's merge base (verified none are in the merge base). Every one of the PR's 4 commits is fully superseded:
Resolution: took the master side for all 7 touched files (6 conflicted + the auto-merged Result: the merged tree is byte-identical to Recommendation for the owner: since the PR is 100% superseded by master's #4427 work, you can either close it as superseded or merge it as a no-op. I've left it open for your call. 八问(中文审核清单)
判定:冲突已解决,PR 现可合并(mergeable = true)。因 PR 被 master 的 #4427 工作完全取代,建议 owner 将其关闭为 superseded,或作为 no-op 直接合并。 |
…test The dsh plugin lane (ci-e2e-dsh.yml) runs node --import tsx --test WITHOUT the preload-storage-baseline.mjs preload, so BILI_STORAGE_UNIFIED is unset and the storage mode defaults to 'full'. Under the pure-sqlite default flip (5a80e2b, #6480), 'full' mode turns the JSON envelope writer OFF — so the .content-store.json sidecar is never written, and the test's readdirSync(namespace) hits ENOENT. Pin BILI_STORAGE_JSON_WRITER=on in the test so the content-store sidecar is written regardless of the storage mode. This keeps the corrupted-store detection path exercised in both full and legacy modes. Closes: dsh plugin lane red on master (run 43532980164, job 132255459726)
(ai: vllm-qwen/qwen3.8-27b) CI red fix — inherited failure, merged ci-fixes per policy. The 5 failing test shards on this PR (head 3682334) are all inherited reds, not caused by this PR's code: the PR's tree is byte-identical to master, and each failing shard matches a documented inherited-red signature in the CI 总控 (#4599):
Per owner policy (2026-10-24 CI 失败协调): master is red → inherited failure → Action taken this run:
After the merge the PR branch tree == ci-fixes tree; its diff vs current master is only the 2 CI test-pin files ( Note: this PR's content remains a no-op fully superseded by master #4427 (from the prior conflict-resolution run) — owner may close as superseded or merge as a no-op. 八问
判定: 已按政策 merge ci-fixes 并推送(b1220b7),CI 将在新 head 重跑。visibleNotices 红由 ci-fixes 钉修复;EBUSY win22 家族是哨兵在 ci-fixes 上收敛中的 handle 泄漏调查(非本车道职责)。本 PR 内容仍是被 master #4427 完全取代的 no-op,建议 owner 决定关单(已取代)或作为 no-op 合并。 |
What & why
The
storage-legacy-onlydiagnostics blob is designed to be copied into a public GitHub issue (Web UI → storage degradation warning → Copy Diagnostics / Report on GitHub), but its storage section copied raw bytes with no publish-time redaction — unlike the config and conflict-log sections. Raw[storage]log lines (credentials, private hostnames, absolute user paths), free-form reason strings (degrade reason,bootstrapFailure.reason), and the absolute SQLite failure-marker path in the force-retry line leaked into the published blob and the prefilled issue body. #3959 already proved absolute user filesystem paths reached a public issue body through this flow; credential/private-host exposure was reproduced with synthetic inputs.Fix
Enforce redaction at the single publish boundary (
buildDiagnostics) so every topic's blob is scrubbed before serialization, and the copied blob + prefilled issue body derive from one string so they can never diverge:redactForPublish(text)primitive insrc/log-mask.tscomposing three layers: secret scrubbing (redactSecretsInText) → non-public host masking (maskUrlsForPublish) → home-path stripping (newmaskPathsForPublish).maskPathsForPublish: rewrites$HOME→~, anchored on both flanks so sibling homes (/home/<other>), relative paths, and URL pathnames are untouched. Closes the bare-absolute-path vector the existing primitives did not cover (SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db corrupted | SQLite unified storage degraded to legacy-only (bili v0.1.202, node v24.21.0, darwin-x64) index.db损坏 #3959).storageLogTailnow bounds each line toLOG_LINE_MAX_CHARS(240) and applies the same redaction asconflictLogTailvia a sharedpublishRedactLinehelper.$HOMEprefix is rewritten to~at the publish boundary, so no username leaks publicly.Behavior changes (flagged)
[storage]lines, reasons, absolute paths$HOME→~[storage]tail lines unbounded/home/<user>/.cache/...in the public blob~/.cache/billion-context/sqlite/.bootstrap-failed.jsonLocal (non-published) surfaces are unchanged:
acp_status's storage wire uses the separatestorageStatusWire()and still reports verbatim for the model relay (#3385);storageDiagnosticsLines()called directly (as its unit test does) still emits the absolute marker path — only the published blob is rewritten.Out of scope (cross-ref)
Non-HTTP URL schemes and bare
host:portliterals in free-form prose are not masked here —maskUrlsForPublishis intentionally HTTP-only (broader host recognition is tracked separately in #4191);redactSecretsInTextstill scrubs credential query params regardless of scheme. This matches the existing config/conflict-section coverage exactly (parity, not expansion).Verification
Targeted suites green on the rebased head (current master):
log-mask,web-storage-diagnostics-redaction(new),storage-diagnostics-lines,web-diagnostics-enrichment(#3813 single-read),web-storage-banner,snapshot-shadow— 43 pass / 0 fail.tsc --noEmitclean. Full regression = remote CI.Closes #4427
Closes #4499