Skip to content

[PR] fix: preserve Node default trust when adding an upstream CA - #7004

Closed
Afloat16 wants to merge 3 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_preserve-default-ca-trust
Closed

Afloat16 wants to merge 3 commits into
ranxianglei:masterfrom
Afloat16:2026-11-17_preserve-default-ca-trust

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

Adding network.caFile rebuilds outbound trust from bundled Mozilla roots. That drops system/runtime CAs Node already trusts and can make existing HTTPS upstreams fail merely because another CA was added. It also restores bundled roots that an explicit runtime default store excluded.

Change

Merge against tls.getCACertificates("default") on Node versions that expose it. This preserves the effective trust selected by Node, including opted-in system CAs. Older Node versions keep the existing bundled-root fallback. Configured CA files, cache behavior, certificate verification, and the unset path remain intact.

Verification

  • Base: 301e1a5459a77d5578206927c8062a58381af642 (master, 0.1.215).
  • New regression tests: 4 fail / 1 passes on the base; 5 pass with this fix.
  • Tests cover default-only CA preservation, a real HTTPS handshake, excluded-root preservation, the unchanged unset path, and a Linux child process started with --use-system-ca and a fixture SSL_CERT_FILE.
  • New regressions, existing TLS diagnostics, and real proxy/TLS checks: 27 tests passed, including rejection of untrusted and wrong-host certificates.
  • npm run typecheck, npm run build, final tsc --noEmit --project tsconfig.json, and git diff --check: passed.
  • Tested on Node 24.22.1. The Node 20 fallback is retained but was not executed locally.
  • No kernel, wire, configuration-schema, release, or dependency changes.

Reference: Node's effective CA API.

Closes #7003

@ranxianglei ranxianglei changed the title fix: preserve Node default trust when adding an upstream CA [PR] fix: preserve Node default trust when adding an upstream CA Nov 17, 2026
@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Nov 17, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review: fix is correct, tests verified — CI red is two distinct causes, one of which is a one-line registration in your branch.

Verified locally (Node 22.23.2, linux):

  • tests/upstream-ca-default-trust.test.ts: 5/5 pass on head ccfd44a05b, 4/5 fail on base 301e1a5459 (only the unset-path test passes on base) — the tests have teeth.
  • API contract: tls.getCACertificates landed in Node v22.15.0 (tls: implement tls.getCACertificates() nodejs/node#57107) — the "Node 22.15+" comment is accurate; the rootCertificates fallback is still needed because engines is node >=20.
  • Non-blocking note: on Node ≥22.15, getCACertificates("default") already includes NODE_EXTRA_CA_CERTS, so the explicit readPem(nodeExtra) merge duplicates those PEMs in the ca array — harmless (OpenSSL dedupes) and still required on the Node 20 fallback path. No change requested.

CI on head ccfd44a05b — 12 reds, two causes:

  1. This PR's own (all 11 test shards): the SPAWN_RE tripwire — the new test file spawns a child process (node --use-system-ca --import tsx -e, linux-only) but is not registered in LIVE_FILES, so every shard fails at the gate before any test runs:

    process-spawning test files missing from LIVE_FILES in scripts/live-files.mjs: tests/upstream-ca-default-trust.test.ts. Register them there after reviewing their concurrency assumptions (tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248).

    Registration entry (append to the LIVE_FILES array in scripts/live-files.mjs):

        // #7003: the only real child is ONE `node --use-system-ca --import tsx -e`
        // (linux-only, skipped elsewhere) that reads fixture CAs and prints — no
        // ports, no network I/O, 10s timeout; the in-process HTTPS rows are
        // listen(0) — the entry is for the tripwire alone, like #3890/#5114
        "tests/upstream-ca-default-trust.test.ts",

    (The in-process HTTPS rows use listen(0), so no port-band slot is consumed; the entry exists for the tripwire alone, like [Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890/Running tests locally writes test sessions into the real ~/.local/billion-context (polluting the production index.db | 本地跑测试会把测试会话写进真实的 ~/.local/billion-context(污染生产 index.db 和 sessions 目录) #5114.)

  2. Inherited from master (301e1a5459 is itself red, per CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599): the codex 命中率问题分析 #2929 fold probe (3–4 shards) + architecture-docs (module map missing src/responses-system-sections.ts). Both already fixed in ci-fixes 7247410976; ci-fixes→master is open as [PR] ci-fixes: converge master CI fixes (#2929 fold probe schedule-independence + architecture-docs module map) #6974.

Pushing the fix is blocked: I prepared both changes locally — the LIVE_FILES registration (commit 9a21a645a4) plus a clean ci-fixes merge (re-verified 5/5 post-merge) — but the PR head is on your fork and "Allow edits by maintainers" is off, so my push is rejected (permission denied). Two ways forward, your choice:

Once CI is green on the updated head, this is §7.4 auto-merge eligible (single module, regression test reproducing the bug, no kernel/wire/config/persistence changes, Closes #7003).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — no other open issue/PR targets this thread (searched all open [PR] rows: only #7004 matches the CA/trust topic; #7003 is the source issue this PR closes). Also distinct from the launcher MITM CA work (#1808/#5809, combined-ca.pem in src/ca.ts): this PR is about the proxy's own outbound trust in src/upstream-ca.ts, as the source issue notes.

Code review: correct root-cause fix. The root cause is real: upstreamExtraCa() builds an explicit ca list for undici's connect.ca (src/fetch-util.ts:101-102) and the proxy's two TLS legs (src/upstream-proxy.ts:425-426, 445-447) — and an explicit ca REPLACES Node's default trust store. So the merge base must be what Node actually trusts by default, not the bundled-only tls.rootCertificates. On Node 22.15+ that is tls.getCACertificates("default") (bundled + NODE_EXTRA_CA_CERTS + opted-in system CAs + setDefaultCACertificates overrides). I verified on Node 22.23.2 that the "default" store already includes NODE_EXTRA_CA_CERTS, so the env-file merge at src/upstream-ca.ts:60-65 now duplicates those entries — harmless for TLS (duplicate PEMs in ca are fine), only the logged cert count is slightly inflated. The Node 20 fallback (engines >= 20) is retained and byte-identical to the old path; the unset path (no connect override) is untouched.

Test validity: the tests have teeth. I checked out the base's src/upstream-ca.ts and re-ran the new file: 4 fail / 1 pass on base, 5/5 with the fix — exactly the PR's claim. Existing suites on the fix head: tests/issue3776-tls-mitm-diagnostics.test.ts 13/13, tests/upstream-proxy-ca-trust.test.ts 9/9 (27 total, matching the PR body).

CI reds — all 12 accounted for (verified live on head ccfd44a):

What I staged (I cannot push to your fork — permission denied, and "Allow edits by maintainers" is off): the fix is two additive commits on top of your head, pushed as branch 2026-11-17_pr7004-ci-tripwire (head 479fcb1) on ranxianglei/billion-context:

  1. 9a21a645a4 — registers the new suite in LIVE_FILES (appended, per the table's slot rule). Verified on this head: the SPAWN_RE tripwire passes and the full 27-test set passes locally.
  2. 479fcb1b31 — merges ci-fixes (clears the inherited reds).

Your options, in order of least friction:

  • Enable "Allow edits by maintainers" on the PR — I push the two commits straight to your branch and CI re-runs; or
  • Apply the registration yourself (5 lines, appended to LIVE_FILES):
     // Native-zstd absence is simulated in an isolated Node child; it binds
     // no sockets and changes only that child's built-in module exports.
     "tests/content-encoding-zstd.test.ts",
+    // #7003: the only real child is ONE `node --use-system-ca --import tsx -e`
+    // (linux-only, skipped elsewhere) that reads fixture CAs and prints — no
+    // ports, no network I/O, 10s timeout; the in-process HTTPS rows are
+    // listen(0) — the entry is for the tripwire alone, like #3890/#5114
+    "tests/upstream-ca-default-trust.test.ts",
 ];

and merge upstream's ci-fixes (or wait for master to absorb it).

标签 (mirror label API unavailable — for the owner): bug, scope/wiring, p2 — p2 because it only bites when network.caFile is set AND the instance relies on system/programmatic default trust (workaround: add the system CA to the caFile); not client-specific (bili's own outbound fetch, all lanes).


结构化八问 (PR review 结论)

  1. 是什么问题:network.caFile/BILI_CA_FILE 设置后,bili 用「捆绑 Mozilla 根 + 附加文件」重建出站 TLS 的 CA 列表传给 undici connect.ca——显式 ca 会替换 Node 默认信任库,导致 --use-system-ca/NODE_USE_SYSTEM_CA 启用的系统 CA 和 setDefaultCACertificates 覆盖全部丢失,Node 原本信任的上游直接 SELF_SIGNED_CERT_IN_CHAIN;同时用户显式排除的捆绑根又被带回来。
  2. 什么原因导致的:合并基座用了 tls.rootCertificates(仅捆绑根),而不是 Node 的有效默认库。Node 22.15+ 的有效库由 tls.getCACertificates("default") 暴露(捆绑 + NODE_EXTRA_CA_CERTS + 启用时的系统 CA + 程序化覆盖),旧代码没用上。
  3. 影响范围与严重程度:仅影响设置了 network.caFile 且依赖系统/程序化默认信任的实例(企业 DLP/Kaspersky + --use-system-ca 场景);未设置路径与之前逐字节一致。p2:能用(未设置路径不受影响;受影响场景可把系统 CA 加进 caFile 绕行),但有明显问题。
  4. 怎么修复的:合并基座改为 tls.getCACertificates("default")(API 可用时);老 Node(20.x,engines >= 20)保留原 rootCertificates 回退;附加文件合并与未设置路径不动。修在决策点(信任列表构造处),不是掩码。
  5. 是否有回归问题:既有行为逐项审过——未设置路径(undefined、不加 connect 覆盖)不变;merge-not-replace 语义保持(既有 issue3776 套件 13/13 绿);CA 身份缓存(path+mtime)轮转逻辑不变;代理两条 TLS 腿(requestTls/proxyTls)与裸 CONNECT 路径消费同一函数,行为一致。一个已知无害副作用:22.15+ 上 NODE_EXTRA_CA_CERTS 证书会在合并列表里出现两次(默认库已含,代码又追加文件)——重复 PEM 不影响 TLS 验证,仅日志 cert 计数略增。
  6. 是否破坏缓存:不涉及——未触碰缓存/重试/游标/去重键逻辑。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

结论:代码与测试达合并质量(根因修对、测试有牙、27/27 定向测试本地绿);PR 目前只被 ① LIVE_FILES 注册(5 行)和 ② master 继承红卡住,两者都有上面验证过的具体路径。我无法推你的 fork(permission denied + maintainer edits 关)——请开启 "Allow edits by maintainers" 让我把准备好的两个 commit 直接推上你的分支,或按上面 diff 自行应用 + 合并 ci-fixes。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status clarification (two review replies landed from parallel sessions — this one is current):

@Afloat16

Copy link
Copy Markdown
Contributor Author

(ai: OpenAI/gpt-6-astra)

Applied the LIVE_FILES registration on the existing PR branch in b20d51a. The only child is a Linux-only Node process that reads private fixture CA files, binds no ports and exits within 10 seconds; the in-process HTTPS cases use listen(0). The repository's global SPAWN_RE check now reports no unregistered files, and all 27 targeted CA/TLS tests pass again.

The current head has completed CI: the three test shards (Ubuntu 22/24 shard 3 and Windows 22 shard 4) and architecture-docs remain red; the static gates pass. The unrelated ci-fixes branch has not been merged into this change.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Re-verified live state: the event's 11 red shards are stale — 4 reds remain, all inherited from master, and I'm blocked on push access to your fork.

Live state on the current head b20d51ae07 (run 44027821597, single API fetch). The head moved since the event because you applied the LIVE_FILES registration (floor 7), so the 11-shard tripwire abort is gone. What's actually red now:

  • 4 reds: test (ubuntu,22,3), test (ubuntu,24,3), test (windows,22,4), and architecture-docs. The other 26 checks are green (static gates, mysql-contract, plan, version-guard, …).
  • All 4 are inherited from master, not your code. The 3 test shards each fail on the same single test — #2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable (tests/issue2929-codex-fold-double-miss.test.ts, error: scenario miss: no ephemeral nudge in the pre-fold request (inputs[6])); I pulled the ubuntu-22/24 logs and both show this exact failure. architecture-docs fails because reference/architecture.md is missing the src/responses-system-sections.ts entry.
  • Current master 6a641f3785 is fully green (run 44043502486, conclusion success) and already contains both fixes (the ci-fixes work has landed). So the PR only needs to pick up master.

The fix — verified locally, not pushed (I can't reach your fork): I merged origin/master into the PR head on a scratch branch: clean, zero conflicts. On that merged tree the #2929 test passes (1/1) and your CA suite passes (5/5). A master merge turns all 4 reds green.

The blocker: the PR head is on Afloat16/billion-context with "Allow edits by maintainers" off (maintainer_can_modify: false), so I cannot push the merge to your branch — the same wall floors 3–5 hit.

Two ways to clear it (your call):

  • A (least friction for you): toggle "Allow edits by maintainers" on this PR — I push the master merge and CI goes green in one cycle.
  • B (do it yourself): on your branch run git fetch origin master && git merge origin/master && git push — the same clean merge I verified.

Once green this is §7.4 auto-merge eligible (single module, regression test reproducing the bug, no kernel/wire/config/persistence change, Closes #7003).


结构化八问

  1. 是什么问题:PR [PR] fix: preserve Node default trust when adding an upstream CA #7004 的 CI 在 4 个检查上红(3 个 test 分片 + architecture-docs),挡住合并。
  2. 什么原因导致的:4 个红全部继承自 master,非 PR 自身代码——3 个 test 分片都挂在同一个 codex 命中率问题分析 #2929 fold-probe 锚点漂移(tests/issue2929-codex-fold-double-miss.test.ts),architecture-docs 挂在 reference/architecture.md 缺 src/responses-system-sections.ts 条目;PR 自身那个 CI 问题(LIVE_FILES tripwire)你已在 b20d51ae07 修掉。
  3. 影响范围与严重程度:只挡本 PR 合并,无用户面影响(PR 自身代码正确、定向测试 5/5 绿)。纯 CI 阻塞,非功能缺陷。
  4. 怎么修复的:把当前 master(6a641f3785,已含两处修复、整体绿)merge 进 PR 分支;本地已验证 merge 零冲突、codex 命中率问题分析 #2929 测试 1/1、CA 套件 5/5。
  5. 是否有回归问题:无——merge 只带入 master 已合入的修复,PR 自身代码逐字不动。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:无。
  8. 是否引入了新的配置:无。

卡点:PR head 在你的 fork 且 maintainer edits 关,我推不上去。请开 "Allow edits by maintainers"(我推 merge,一轮 CI 转绿)或自行 git merge origin/master 推上。

Merge upstream master 6a641f3, including the architecture module-map and schedule-independent ranxianglei#2929 probe fixes.

Preserve the existing PR head and exact contribution patch. Validation of this refreshed head runs in upstream CI; no local test rerun is claimed.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: 可以收(approve)。 方向正确,把"配了 caFile 反而信任集变小"的不一致修掉了。

核实过的逻辑链:

  • 原来基础信任集是 tls.rootCertificates(只有 Node 内置 Mozilla 根)+ caFile。但不配 caFile 时走的是 Node 生效信任集(22.15+ 可含系统信任库/程序化设置的根)——于是"不配能通、配了反而不通"。改成 tls.getCACertificates("default") 后两条路径对齐,老 Node 回退内置,降级链完整。
  • tests/upstream-ca-default-trust.test.ts(126 行)覆盖了 mock/回退两条路。

两个不阻塞的小瑕疵,记录在案:

  1. caFile 与 NODE_EXTRA_CA_CERTS 内容相同时合并结果有重复条目——对 TLS 校验无害(验证去重),可不改。
  2. 缓存 key(upstreamCaIdentity)只追踪 caFile 文件身份;若宿主代码运行时调用 tls.setDefaultCACertificates() 改变生效信任集,缓存不会失效。极罕见场景,加个注释即可。

@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #7101 (batch-2) — this PR's content (and its review-patch fix where applicable) is now merged via #7101. Closing to avoid a duplicate merge. 本 PR 的内容(含评审补丁,如适用)已通过 #7101 合并,关闭以免重复合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adding an upstream CA drops Node system and runtime default trust

2 participants