Skip to content

[Bug] CA rotation retains obsolete direct and proxy dispatchers #7348

Closed

Description

@Afloat16

Description

Changing network.caFile / BILI_CA_FILE, or rotating that file after its identity refresh, selects a new dispatcher but leaves every dispatcher for previous CA identities strongly referenced in the module cache. Both directDispatchers in src/fetch-util.ts and dispatcherCache in src/upstream-proxy.ts keep growing across rotations, retaining obsolete Agents, their CA material and origin pools.

Reproduction:

  1. Select one CA file and make a direct request through fetchWithTimeout, or obtain and use proxyDispatcher.
  2. Select another CA file and make another request. Repeat with additional identities.
  3. The current master chooses a new Agent, but all earlier Agents still report closed === false; no production cache retirement runs. Returning to an old identity retrieves its old cached Agent.

Expected: release cache references to retired identities and gracefully close their pools, allowing already in-flight responses to finish. Reuse for the current CA identity must still work across timeout/proxy-route variants.

A focused regression against 1e357bf2d6632bfbd2a74ffb70ecee1af1ec7cc3 reproduces this in four behavior assertions: direct cache, proxy cache, an active direct HTTP response, and an active HTTP CONNECT response. All four fail before the fix and pass with graceful retirement. The active streams finish normally after retirement; a new request uses the new identity.

bili version

0.1.216 (source package version on tested master 1e357bf2d6632bfbd2a74ffb70ecee1af1ec7cc3).

Usage mode

Other / not sure: deterministic transport-level reproduction shared by direct and proxied upstream callers.

Client and version

Node.js 24.22.3 test harness; no live host-client session.

Operating system

Linux 6.18.57 x86_64.

Upstream and model

Local HTTP and CONNECT fixtures. No model is needed to reproduce dispatcher retention.

Configuration

Only the existing BILI_CA_FILE selection changes between fixture PEM files. No new configuration surface is proposed.

Verification

  • Original behavior: 4 regression failures because the old dispatchers stay open.
  • Fixed behavior: 30 targeted tests pass, including the new lifecycle regressions, existing TLS trust/rejection checks, CA identity checks and upstream timeout coverage.
  • npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph, git diff --check: pass.
  • The authoritative full-suite verdict remains the repository CI on the PR head.

Triage

Priority: p1 under the repository's cache-leak rule. Scope: scope/other. Topic: N (server-side miscellany). Cross-client transport code; no single host-lane label.

Related but distinct: #3786 introduced identity-keyed pools, #6096 refreshed identity on a path change, and #7004 preserved Node's effective default trust. None retires pools from old identities.

Activity

  1. Afloat16 commented on Nov 20, 2026

    @Afloat16
    ContributorAuthor

    (ai: OpenAI/GPT-6 Astra Pro)

    Topic N — server-side miscellany; cross-lane (no host-specific label). Priority p1 under the cache-leak rule; scope/other.

    Implemented in PR #7349: rotating the upstream CA identity now retires both direct and proxy dispatcher pools while allowing in-flight responses to finish. The 30 targeted transport, TLS and timeout tests pass; the full regression verdict is the PR's remote CI.

  2. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Duplicate screening (查重): Not a duplicate. The only items on "CA rotation retains obsolete dispatchers" are this issue (#7348) and the fix PR (#7349). Related-but-distinct: #3786 introduced network.caFile + the identity-keyed pools; #6096 refreshes the CA identity on a path change; #6583 (CA roots ignored for proxied TLS) and #6095 (temporary reuse of previous trust material) are different bugs. None of them retires pools from old identities — that missing retirement is exactly what #7348/#7349 address.

    I verified the claim independently (read the code, checked out the PR head, ran the regression tests against both head and base) rather than relying on the summary:

    • The bug is real. directDispatchers (src/fetch-util.ts:93) and dispatcherCache (src/upstream-proxy.ts:66) both key entries on the CA identity (upstreamCaIdentity()) but only ever ADD a new identity's pools — they never remove old ones. On a CA rotation (file mtime change or path switch) a fresh Agent/ProxyAgent is created while every prior identity's dispatcher stays strongly referenced in the module Map for the process lifetime, retaining its CA material, origin pools and open sockets. There is no production retirement path — only the test-only _reset*ForTest hooks close them.
    • The fix is correct and complete. PR [PR] fix: retire upstream dispatchers when CA identity changes #7349 adds a "current CA identity" tracker to each cache; when upstreamCaIdentity() changes it calls close() (graceful — waits for in-flight responses, does not destroy()) on the old identity's pools and clears the map, then updates the tracker. Reuse within the current identity across timeout/route variants is untouched. I grepped src/ — these are the only two CA-keyed dispatcher caches, so nothing is missed.
    • The tests have teeth. On the PR head, tests/upstream-ca-dispatcher-retirement.test.ts passes 4/4. On the base (1e357bf2) with the same test file dropped in, it fails 4/4. So the regressions genuinely reproduce the bug and are pinned by the fix. The two in-flight cases use real loopback HTTP/CONNECT fixtures and assert the retired pool is closed but not destroyed until its last response finishes.
    • Diff is clean (3 files: the two caches + the new test; no unrelated changes, no lock churn). CI on the head is in progress — static-gates green, test shards running, 0 failed so far; the full-suite verdict is still pending.

    结构化八问:

    1. 是什么问题:配置了 network.caFile/BILI_CA_FILE 后,每次 CA 身份轮换(文件 mtime 变化或路径切换),两个 dispatcher 缓存都为"新身份"新建池,但旧身份的所有 Agent/ProxyAgent 被模块级 Map 永久强引用,连同 CA 材料、origin 池、已打开连接一起只增不减。
    2. 什么原因导致的:两个缓存按 CA 身份做 key([Bug][tls] Proxy upstream TLS broken by security-software HTTPS scanning (Kaspersky MITM): client sees only a generic 502 | 安全软件HTTPS扫描致代理上游TLS失败,客户端只见泛502 #3776 为支持信任轮换而加),但只实现了"换身份→建新池",缺了"换身份→退旧池"——没有任何生产路径会 close+删除旧身份的池,只有测试用的 _reset*ForTest 钩子会。
    3. 影响范围与严重程度:触发面窄——只有配置了 caFile 且发生轮换的部署才受影响(设一次 CA 永不轮换的不泄漏);但一旦轮换,每轮换一次泄漏一批池,进程生命周期内只增不减,频繁自动轮换(如身份刷新)会持续增长直至耗尽内存/连接。按 owner 缓存泄露规则(只增不减的失控增长)定 p1;实际严重度随轮换频率放大。
    4. 怎么修复的:两个缓存各加一个"当前 CA 身份"跟踪变量;upstreamCaIdentity() 返回值变化时,用 close()(优雅:等待 in-flight 响应完成,不 destroy())关闭并清空旧身份全部池,再更新跟踪身份。当前身份内的 timeout/route 变体复用逻辑不变。
    5. 是否有回归问题:逐项过既有行为——(a) 同身份内 timeout/route 变体复用:保持;(b) [Bug][tls] Proxy upstream TLS broken by security-software HTTPS scanning (Kaspersky MITM): client sees only a generic 502 | 安全软件HTTPS扫描致代理上游TLS失败,客户端只见泛502 #3776 的 CA 身份 keying:保持;(c) in-flight 响应:优雅完成(close 不 destroy,测试用真实 HTTP/CONNECT 验证);(d) _resetFetchUtilForTest/resetProxyCache:等价委托,行为不变;(e) 未配置 CA 的默认身份("none"):正常缓存;(f) 回到旧身份:重建新池(有意变更——旧行为正是泄漏根源,新行为是正确退役,已测试钉住)。无意外回归。
    6. 是否破坏缓存:不涉及 prompt cache / 增量游标 / 去重键。只改 dispatcher 池生命周期;CA 身份变化时池被重建属预期。
    7. 是否改了设计方向:否。信任构造、CA 身份刷新策略(30s TTL)、默认超时、wire payload、配置面均未动;只补上缺失的池退役逻辑。
    8. 是否引入了新的配置:否。无新 env/config/默认值。

    判定:可合并 —— 根因修复正确、测试有牙、diff 干净、未发现回归。剩余:CI 全量绿 + owner 合并。

  3. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Status update (live-verified): the fix is complete and the PR is ready for owner merge.

    结构化八问:

    1. 是什么问题:CA 身份轮换后,两个 dispatcher 缓存(directDispatchers / dispatcherCache)对旧身份的 Agent/ProxyAgent 永久强引用,连同 CA 材料、origin 池、已打开连接只增不减。
    2. 什么原因导致的:缓存按 CA 身份做 key 但只建不退,缺"换身份→退旧池"这一步(生产路径无 close+删除,仅测试钩子有)。
    3. 影响范围与严重程度:触发面窄——需配置 caFile 且发生轮换的部署才受影响;一旦轮换,每轮泄漏一批池,进程生命周期内只增不减。按缓存泄露规则定 p1,实际严重度随轮换频率放大。
    4. 怎么修复的:每个缓存加"当前 CA 身份"跟踪变量;身份变化时用 close()(优雅等待 in-flight 响应完成,不 destroy())关闭并清空旧身份全部池,再更新跟踪。当前身份内的 timeout/route 变体复用不变。
    5. 是否有回归问题:逐项过——同身份内复用保持、[Bug][tls] Proxy upstream TLS broken by security-software HTTPS scanning (Kaspersky MITM): client sees only a generic 502 | 安全软件HTTPS扫描致代理上游TLS失败,客户端只见泛502 #3776 的 CA keying 保持、in-flight 优雅完成、reset 钩子等价委托、未配置 CA 的默认身份("none")正常、回到旧身份重建新池(有意变更)。无意外回归。
    6. 是否破坏缓存:不涉及 prompt cache / 增量游标 / 去重键。
    7. 是否改了设计方向:否。只补上缺失的池退役逻辑。
    8. 是否引入了新的配置:否。

    判定:可合并,等待 owner 合并 —— 修复正确、测试有牙、CI 全量绿(30/30)、diff 干净、无回归。剩余:owner 合并(agent 不合并 PR)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions