Skip to content

[PR] fix: make upstream TLS trust failures self-diagnosable; add network.caFile (#3776) - #3786

Merged
ranxianglei merged 5 commits into
masterfrom
2026-10-19_tls-mitm-diagnostics
Oct 19, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-10-19_tls-mitm-diagnostics

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #3776

What & why

Security software doing HTTPS scanning (Kaspersky "encrypted connection scanning", enterprise DLP) or a corporate MITM proxy re-signs the upstream with its own CA. Browsers trust it via the OS certificate store; Node's fetch does not read the OS store — only its built-in roots plus NODE_EXTRA_CA_CERTS. So bili's own upstream fetch fails with SELF_SIGNED_CERT_IN_CHAIN, the upstream circuit opens after 3 pre-response failures, and the client only ever sees a generic 502 (reported case: #3773 — Windows + dsh-desktop + Kaspersky; the user spent ~a day diagnosing from nothing visible client-side).

One mechanism family (MITM trust + error transparency), three sub-defects, one PR:

  1. Actionable TLS-trust hint on every channel. The trust-code set (7 codes incl. SELF_SIGNED_CERT_IN_CHAIN, UNABLE_TO_GET_ISSUER_CERT_LOCALLY) and the remediation text now live once in src/upstream-fail.ts. formatUpstreamError appends the remediation for trust-code TLS failures, so the log line, the 502 detail, and the circuit/alert detail all carry it; preflight's summary-transport note reuses the same helpers (its local code set was deleted); the panel's tls alert banner (en+zh) names the scenario too.
  2. network.caFile / BILI_CA_FILE — a PEM file whose roots are merged into bili's OWN upstream fetch trust (both the direct dispatcher in fetch-util.ts and the proxied ProxyAgent in upstream-proxy.ts). Explicit undici connect.ca REPLACES default trust, so the implementation merges built-in roots + NODE_EXTRA_CA_CERTS + the file (strictly additive over today's default); dispatcher caches key on a TTL-cached CA identity (path+mtime) so file edits rotate within ~30 s. Unset ⇒ dispatchers omit connect entirely ⇒ byte-identical behavior.
    Audit finding from the issue (sub-defect 2): combined-ca.pem (fix(ca): merge OS trust store into combined-ca.pem (#1807) #1808 OS-store merge) is consumed ONLY by launcher lanes (child-client env); bili's own process never points at it — by design, since bili also serves non-launcher installs. That is exactly why the Kaspersky root sitting in the Windows store was untrusted here. Auto-trusting the OS store inside bili's process would be a default trust-surface change (owner decision), deliberately NOT done; this knob is the explicit escape hatch.
  3. Structured 502 diagnostic. The two relay throw sites attach {kind, code, hint} to the error (non-enumerable symbol); the dispatch catch adds an additive upstreamFailure field to the 502 JSON body when present. Clients can now self-diagnose (kind=tls + remediation hint) without reading bili.log. Body is byte-identical for every failure that isn't structured.

config surface

  • Added: network.caFile (string, default unset) ↔ env BILI_CA_FILE (env wins). Documented in CONFIGURATION.md en+zh (network table row, env mapping row, env detail entry, network-block prose).
  • Why the existing surface cannot express it: NODE_EXTRA_CA_CERTS is a process-global environment variable — users on host-managed installs (dsh profile bundles, opencode/pi trees) often cannot control the proxy process's environment at all, and there was no per-install file knob. Considered and rejected: (a) auto-merging the OS store like fix(ca): merge OS trust store into combined-ca.pem (#1807) #1808 — silently expands the trust surface of every install (product/security decision, not an agent call); (b) reusing combined-ca.pem — launcher-lane artifact by design (dsh plugin-market installs never auto-update: the refresh chain's only trigger is a global bili self-update #1196 contract), unavailable/irrelevant for market-only installs.
  • Migration: none — opt-in; unset keeps today's exact behavior (dispatchers omit connect, Node default trust untouched).

Behavior changes (old → new)

  1. Trust-code TLS failure hint (log / 502 detail / circuit+alert detail): old trailing hint — CA/proxy MITM config → same prefix plus appended remediation: …; the presented certificate chain is not trusted by this Node.js process — usually security software doing HTTPS/encrypted-connection scanning (Kaspersky, enterprise DLP) or a corporate MITM proxy re-signing the upstream. Fix: export that scanner/vendor root CA to a PEM file and point NODE_EXTRA_CA_CERTS=/path/to/ca.pem at it (or set bili config "network.caFile"), or exclude node.exe from the software's HTTPS inspection. Non-trust TLS failures (e.g. EPROTO) keep the old hint unchanged.
  2. Preflight summary-transport caNote: old This looks like TLS certificate interception (corporate proxy/MITM): install the corporate root CA for Node.js via NODE_EXTRA_CA_CERTS=/path/to/ca.pem, or run node with --use-system-ca. → new — + the shared remediation above. Dropped the --use-system-ca suggestion (flag-based, unusable for a running daemon) in favor of the file-based fixes.
  3. Panel tls alert banner (en+zh): original clause kept, scanner-remediation clause appended.
  4. 502 JSON body: gains ADDITIVE upstreamFailure: {kind, code?, hint} on structured upstream failures (fetch failure / circuit open); absent otherwise — existing consumers see unchanged bodies.
  5. New opt-in knob as above; the unset path is byte-identical (proven by the e2e A/B test).

Tests

New hermetic suite tests/issue3776-tls-mitm-diagnostics.test.ts: unit (hint present for trust codes / absent for EPROTO; structured attachment round-trip + non-enumerability; CA merge never replaces; unset ⇒ undefined) + in-process proxy e2e with a real self-signed MITM https upstream: untrusted ⇒ 502 carrying upstreamFailure{kind:"tls", code, hint}; the same upstream trusted via BILI_CA_FILE ⇒ 200 completion. tests/preflight-transport-recovery.test.ts assertion updated to the shared remediation text (the old pinned note string no longer exists by design).

Rebased onto current master (original base was 80 commits behind; conflicts in relay.ts/config.ts/docs were union-of-intent resolutions with #3701's WS byte-shed work — the structured-failure change lands in the non-oversize branch only, the ws-shed path stays verbatim). Local on the rebased head: typecheck clean, build green (both dist gates), 95/95 across the 10 affected suites incl. issue3701-ws-byte-budget. Full-suite verdict: CI.

…aFile (#3776)

Security software doing HTTPS scanning (Kaspersky, enterprise DLP) or a corporate MITM proxy re-signs the upstream with its own CA. Browsers trust it via the OS certificate store, but bili's own Node fetch reads only its built-in roots plus NODE_EXTRA_CA_CERTS, so these users hit SELF_SIGNED_CERT_IN_CHAIN, the upstream circuit opens after 3 pre-response failures, and the client only ever sees a generic 502 (#3773/#3776).

- Trust-code TLS failures carry a concrete remediation hint on every channel rendering formatUpstreamError (log, 502 detail, circuit/alert detail) plus the panel tls alert banner (en+zh): export the scanner/vendor root to PEM -> NODE_EXTRA_CA_CERTS or network.caFile, or exclude node.exe from scanning. The trust-code set and remediation text live once in src/upstream-fail.ts; preflight reuses them.

- New network.caFile / BILI_CA_FILE knob: a PEM file whose roots are MERGED into bili's own upstream fetch trust (direct + proxied dispatchers), never replacing built-ins or NODE_EXTRA_CA_CERTS. Unset = byte-identical behavior; dispatcher caches key on CA identity so edits apply on rotation.

- The dispatch-catch 502 body gains an additive upstreamFailure{kind,code,hint} field when the failure is structured (upstream request failed / circuit open), so clients can self-diagnose without reading bili.log. Absent for everything else.

Docs: CONFIGURATION.md en+zh. Tests: new hermetic suite (unit + in-process proxy e2e: untrusted MITM upstream => structured 502; same upstream trusted via caFile => 200); preflight transport-recovery assertion updated to the shared remediation.
@ranxianglei ranxianglei changed the title fix: make upstream TLS trust failures self-diagnosable; add network.caFile (#3776) [PR] fix: make upstream TLS trust failures self-diagnosable; add network.caFile (#3776) Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_tls-mitm-diagnostics (a28f0a2)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4497-pr.3786.a4f49e8

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3786.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 按失败签名(SELF_SIGNED_CERT_IN_CHAIN / network.caFile / BILI_CA_FILE / circuit 502 结构化出口)检索 open+closed issue 与 PR:唯一相关链条是来源 #3773(已关闭,用户自定位根因)→ 跟踪单 #3776(本 PR 关闭它),无其他在途同机制修复。

标签: bug、dsh、scope/wiring、p2(缺陷类;复现客户端 dsh-desktop;改动在代理上游出口/协议接线层;条件触发需 MITM 软件在环,与 #3776 定级一致)。

审核进行中,先报一个已本地复现的硬伤(完整八问结论随后):

CI 必红 ×3 —— 新配置键没进 YAML seed。 本仓配置面纪律:website/config-reference/*.yaml 是唯一事实源,CONFIGURATION.md/zh-CN 由 tools/gen-config-docs.mjs 生成,CI 三道闸都会拦——

  • docs-sync:重新生成后与提交物 diff → 本 PR 手改的 .md 表格行会被生成器剥掉(本地复现:剥 4 行);
  • docs-coverage:BILI_CA_FILE 在 src/knobs.ts 被读但无 seed 条目 → "UNDOCUMENTED env vars"(本地复现 FAIL);
  • config-surface-guard:network.caFile 无 seed 条目 → "1 config-file field(s) lack a seed entry"(本地复现 FAIL)。

修法明确:补 process.yaml 的 caFile seed 条目 + 跑生成器重出文档,我直接在本 PR 分支上修(独立标注 commit)。另外代码细读中还发现一个 Windows 兼容性小缺陷(NODE_EXTRA_CA_CERTS 路径分隔符),一并处理后再交最终八问。

@ranxianglei ranxianglei added bug Something isn't working dsh p2 scope/wiring labels Oct 19, 2026
awork added 3 commits October 19, 2026 14:27
…-coverage/config-surface-guard) (#3776)

Review fix for PR #3786: the new knob was documented by hand-editing the
GENERATED artifacts (CONFIGURATION.md / zh-CN) while the single source of
truth website/config-reference/*.yaml stayed untouched, which trips all three
config-doc CI gates (docs-sync strips the hand-added rows on regeneration;
docs-coverage flags BILI_CA_FILE undocumented; config-surface-guard flags
network.caFile without a seed entry). Add the seed entry and regenerate;
the committed table rows now match the generator output byte-for-byte.
…n merge + circuit-kind coverage (#3776)

Review fixes for PR #3786:

- upstream-ca.ts: the merged trust list re-parses NODE_EXTRA_CA_CERTS itself
  (an explicit undici connect.ca replaces the default store, so every source
  Node would already have trusted must be folded in). It split on ':' only,
  which on Windows shreds drive-letter paths (C:\certs\ca.pem -> "C" +
  "\certs\ca.pem") and silently drops those certs with a warn log. Mirror
  Node's own win32 separator handling (';' also separates there).
- tests: pin the previously uncovered corners of sub-defects 2 and 3 —
  NODE_EXTRA_CA_CERTS entries land in the merged list, and
  upstreamCircuitLastFailureKind reports the opener's kind only while the
  circuit is open (the kind source of the circuit-open egress hint).
Review fix for PR #3786: the interface is only consumed inside
src/upstream-proxy.ts as the return type of describeUpstreamFailure —
no other module imports it, so the unused-exports gate fails the PR
head. Drop the leading export (INTERNAL_ONLY); callers keep the shape
via structural inference, no behavior or typing change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

进度汇报(上一轮长轮询 CI 被看门狗重启,现在恢复):审核已完成,发现的小问题已按「小问题直接修」在 PR 分支上补了 3 个独立标注 commit(均已 push,head db1e13589):

  1. fa86bdd2e — 首条回复报的 CI 必红 ×3 的根治:network.caFile seed 补进唯一事实源 website/config-reference/process.yaml(条目插在 network.wsOutboundBytes 之后,描述沿用原 .md 措辞去反引号),跑 tools/gen-config-docs.mjs generate 重出文档;本地复跑三道闸全绿,且生成位置与作者手插的行一致(幂等验证过)。
  2. 93288dc6a — 两个小缺陷 + 两个测试缺口:
    • src/upstream-ca.ts 合并信任列表时重解析 NODE_EXTRA_CA_CERTS 只按 : 切分,Windows 上会把盘符路径切碎静默丢证书 → 改为 win32 下 ;/: 都作分隔(与 Node 自身语义取超集,两边都不丢);
    • 补测:upstreamCircuitLastFailureKind(circuit-open egress hint 的 kind 来源)此前零覆盖;NODE_EXTRA_CA_CERTS 半边的 CA 合并此前未测。
  3. db1e13589 — CI unused-exports 闸红:本 PR 新增的 UpstreamFailureParts 接口只在本模块内当返回类型用、无处 import → 去掉导出(INTERNAL_ONLY),调用方经结构推断不受影响。

另外给 PR 描述补了缺失的 <!-- ework-agent-pr --> 标记(防后续唤醒误判为外部 PR)。

CI 现状(head db1e13589):静态闸全绿(docs-sync / docs-coverage / config-surface-guard / unused-exports / version-guard / kernel-guard / lane-guard / test-import-graph / lock-sync 等 13 项 success);test 分片多数已过,windows-latest/node22/shard-1 红了——正在拉日志定位(本 PR 恰好动了 Windows 相关路径,必须排除是我的 separator 修法引入的回归);ubuntu-latest/node24/shard-1 还在跑。定位结果 + 最终八问随后给出。

…A merge (#3776)

Review fix for PR #3786 (replaces the separator heuristic from
93288dc, which this commit supersedes): Node does NOT split
NODE_EXTRA_CA_CERTS on any separator — verified against Node 22, the
entire value is loaded as ONE file (multi-cert files carry several PEM
blocks). The previous ':' / win32 ';:' splits diverged from Node on
every platform and shredded drive-letter paths on Windows (which also
broke the env-extra test there). Read the value as a single path so the
merge trusts exactly what Node would already have trusted; pin the
no-split semantics with an extra assertion.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立复核完成,判定:可合并。 CI 在 head a28f0a23a 全绿(静态闸 + 全部 test 分片;e2e 真客户端 lane 按惯例 env-gated 跳过)。审核中发现的 4 个问题已按规则直接以标注 commit 推入本 PR 分支并吸收进 CI(见下)。以下为独立重审结论(未照抄原 issue 分析)。

结构化八问

  1. 是什么问题:上游 TLS 信任失败时(典型场景:Kaspersky 等安全软件做 HTTPS 扫描/MITM),用户只拿到一句模糊日志和一个无结构化原因的通用 502,且没有任何显式配置入口指向厂商根证书——[Bug][dsh] New desktop build V4.1-Flash1017: all model requests fail with 502 when the plugin is enabled; disabling the plugin restores normal use | 新桌面端开插件后全部请求 502,关插件即恢复 #3773 的用户为此自诊断了约一天。三个子缺陷属同一机制家族:提示文本不具体、无显式 CA 配置项、熔断原因不在出口外露。
  2. 什么原因导致:Node 的 fetch 只信任内置 Mozilla 证书包 + NODE_EXTRA_CA_CERTS,不读操作系统证书存储;MITM 根证书只装在系统 store 时就报 SELF_SIGNED_CERT_IN_CHAIN,连续 3 次 pre-response 失败后熔断 OPEN,出口只剩通用 502。根因在 Node 的信任模型(不是 bili 的 bug);bili 的职责边界 = 透明化(让错误可自诊断)+ 显式逃生门(network.caFile),不动默认信任行为(自动读 OS store 已被 owner 决策排除)。方案方向正确。
  3. 影响范围与严重程度:仅影响链路中存在 MITM/证书拦截的子集用户(企业 DLP、安全套件环境);caFile 未设置时所有改动 byte-identical,普通用户零影响。p2 定级吻合:能用但需绕行,本 PR 把绕行成本从「读源码」降到一行配置。
  4. 怎么修复的(机制层):① 提示单一来源——TLS 信任错误码集合(7 码)+ 修复指引文本只存在于 src/upstream-fail.ts,日志行、502 detail、熔断告警、preflight 诊断四处复用同一函数;② network.caFile/BILI_CA_FILE 合并进 undici connect.ca——严格 additive:显式 connect.ca 会替换默认 trust store,所以必须同时 fold in 内置 roots 和 NODE_EXTRA_CA_CERTS,文件轮转经 ~30s TTL identity 缓存自动生效;③ relay 两个 throw 点经 non-enumerable symbol 附 {kind, code, hint},dispatch catch 对 502 JSON body additive 追加 upstreamFailure 字段。
  5. 是否有回归问题:逐项审过触碰路径的既有行为,无无意回归。关键证据:caFile 未设置时 dispatcher 缓存 key 退化为旧 key 的超集且 Agent 构造完全 omit connect(已对 HEAD 代码逐行确认)→ 默认信任 byte-identical;非 trust-code 错误的包装输出 byte-identical;502 body 仅在结构化字段存在时才 spread;preflight 原 7 码集合移入共享常量后成员等价;唯一语义变化是 preflight caNote 不再提 --use-system-ca(flag 建议对常驻 daemon 进程不可执行,作者已在 PR 描述披露 old→new);panel banner en/zh 各追加一句,i18n parity 测试绿。plugin/proxy 双压缩模式不受影响——改动在出站失败面与 bili 自身上游请求,与压缩模式的消息重建正交。
  6. 是否破坏缓存:否。dispatcher 缓存语义保持(key 为超集);prefix cache、fold 游标、去重键均不受影响;CA identity TTL 缓存是新增内部缓存,仅 caFile 配置时活跃。
  7. 是否改了设计方向:否。纯 additive 诊断面 + opt-in 配置项,架构边界与数据流无转向。
  8. 是否引入新配置:是——network.caFile(string,默认 unset)↔ BILI_CA_FILE(env 优先,knobs.ts 既有惯例),CONFIGURATION.md en+zh 已文档化(经 docs seed 修复后由生成器产出)。迁移:无需,unset = 旧行为。

审核发现及处置(均为标注 commit,head a28f0a23a)

  • F1 [拦→修] fa86bdd2e:新配置项最初只手工改了生成物(CONFIGURATION.md/zh-CN),单一来源 website/config-reference/process.yaml 未登记 → CI 三闸红(docs-sync/docs-coverage/config-surface-guard,均本地复现)。补 seed + 生成器重跑,幂等性验证通过。
  • F2 [修] 93288dc6a → 终版 a28f0a23a:CA 合并重解析 NODE_EXTRA_CA_CERTS 时按分隔符切分有误。我第一版改成 win32 按 ;/: 切分,被 Windows CI 当场证伪(唯一失败用例钉死:盘符路径被切成 C 与 \Users\... 两段双双 ENOENT);随后查 Node 源码并在 Node 22 实证:Node 对该变量不做任何平台上的分隔符切分,整个值就是一个文件路径(多证书应放同一文件的多个 PEM 块)。终版 = 按单一路径直读、与 Node 信任语义精确对齐,并加负断言钉死「不得切分」。
  • F3 [修] 93288dc6a:两处零覆盖补齐——熔断出口 kind 来源(upstreamCircuitLastFailureKind)与 CA 合并中 NODE_EXTRA_CA_CERTS 半边。
  • F4 [修] db1e13589:PR 新增的 UpstreamFailureParts 接口无处 import → unused-exports 闸红,移除 export(仅模块内使用)。
  • 另补了 PR 描述缺失的 <!-- ework-agent-pr --> 标记(防后续唤醒误判)。

观察(不拦,记录在册):① proxy lane(ProxyAgent 的 connect.ca)无 e2e,现有 A/B 覆盖 direct lane——undici 标准 API 用法 + direct lane 全链证明,风险低但留缺口;② dispatcher Map 每个 CA identity 增一个 entry 且不关闭旧 Agent(复用既有 timeout-key 模式,rotation 罕见,影响有限)。两者均属后续范畴,不塞入本 PR。

行为变更(作者已披露,复核确认):trust-code TLS 错误追加修复指引后缀;preflight caNote 去掉 --use-system-ca;panel tls banner 加一句(en+zh);502 body 可能 additive 携带 upstreamFailure{kind, code?, hint};新 opt-in knob 的 unset 路径 byte-identical(e2e A/B 钉住)。

判定:可合并。 head a28f0a23a CI 全绿(含我引入又修复的 Windows 回归转绿),本地 typecheck/build/定向套件全绿。合并 human-only,请 @ranxianglei 自行合并:#3786

@ranxianglei
ranxianglei merged commit 7446891 into master Oct 19, 2026
48 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.621-master.7446891 (rolling master tag).

npm install -g billion-context@0.0.621-master.7446891  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p2 scope/wiring

Projects

None yet

1 participant