Skip to content

ci: per-file zone-port bands for live suites + bounded TOCTOU retries (#3182/#3184/#3188) - #3197

Merged
ranxianglei merged 3 commits into
masterfrom
fix/3182-zone-slot-bands
Oct 15, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
fix/3182-zone-slot-bands

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3182, closes #3184, closes #3188 (forensics + plan landed there; this PR delivers A+B+C+D).

Root cause (from #3182/#3188, verified on the empty PR #3190)

Failure shapes seen in the empty-PR proof (#3190): listen EADDRINUSE 127.0.0.1:18787 (port-zone-live), stolen freePort() picks (claude-native squatter), stolen stub ports (auto-restart).

Fix (ework's A+B+C+D plan, implemented)

Piece Change
A — per-file bands scripts/live-files.mjs exports the shared LIVE_FILES table; zonePortForSlot(slot) = 18787 + (slot+1)*32. First band sits ABOVE the default base, so a spawn that (wrongly) ignores the env lands on 18787 and collides with nobody's assertions — it fails loudly in its own suite. 38 slots, max band 20003 < 32768 (ephemeral range).
A — injection tests/preload-zone-slot.mjs (CI-only): node --test runs each file in its own process, so argv identifies this file; LIVE_FILES members get BILI_ZONE_PORT in-process, inherited by every spawned daemon. Plain npm test is untouched (no preload → default base).
A — runner wiring scripts/run-ci-tests.mjs imports the shared table and adds the preload to PRELOAD_ARGS, so the diagnostic re-runs (#3029) see the same environment.
B — assertions follow the band port-zone-live / claude-native shadow the imported ZONE_PORT_BASE with resolveZonePortBase(); claude-native's explicit child envs spread ZONE_ENV. Product side needed zero changes — BILI_ZONE_PORT override already existed (src/config.ts:2154).
C — env passthrough covered by the spread (above) — every explicit-env spawn site in the two files now carries the band.
D — bounded TOCTOU retries claude-native squatter setup and the two auto-restart #3044 scenarios retry ≤2 extra attempts on infra-shaped failures only (stolen freePort pick; stub exit 91 = EADDRINUSE marker added to the ready-stub). A product regression fails all attempts identically — nothing is masked.

Local verification (this box carries real legacy daemons on 18787/18831/18834 — an adversarial environment)

  • 3 shards run concurrently, exactly the CI shape but on one machine: zero EADDRINUSE anywhere.
  • 9-file port-family set at concurrency 4 with live envs (ACP_TEST_CLAUDE_NATIVE=1): no cross-file collisions; port-zone-live in isolation: master fails 7/9 (legacy daemons own the default base), this branch 2/9 — both residuals are same-file teardown lag under a concurrent foreign full-suite run (0.7ms instant EADDRINUSE / 121s starved test), not collisions.
  • Unit + family regressions green: port-zone, port-owner, listen-policy, auto-restart 16/16, zone-live unit rows.

Guards

live-files.mjs is plain .mjs (no TS, no import-graph impact); zonePortForSlot/LIVE_FILES have two importers each (unused-exports clean); lane-guard unaffected (no workflow change).

…ent"

This reverts commit aedd35c, reversing
changes made to ec7dde9.
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: fix/3182-zone-slot-bands (fcee879)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3699-pr.3197.b7086e6

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3197.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei force-pushed the fix/3182-zone-slot-bands branch from 8bed320 to bea9327 Compare October 14, 2026 16:05
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 对应已立项的 #3182/#3184/#3188(均已关闭待本 PR 交付);检索 open+closed issue/PR 未发现其他针对同一失败签名(EADDRINUSE 18787 / own-lane attach 误拒 / stub exit 1)的在途修复。

这是 [PR] 审核任务(分支为上一会话的交付物,我不重做,只做核对)。正在逐段核对代码与 PR 声明、查 CI 状态。先报备一个初步发现:分支里混入了一个与端口 band 修复无关的 revert(撤销 #3189 的 pages enablement: true,pages.yml -7 行),PR 描述未提及——正在查它的来源,结论会写进最终八问。

…/dist hygiene (#3182/#3184/#3188)

Root cause of the 7-red test matrix: #3145 sharding concentrated the port
families into shards 2/3 while #2816 concurrency let them bind
simultaneously on one runner — cross-file 18787 collisions and shared
ephemeral-port TOCTOU.

Fixes:
- scripts/live-files.mjs: shared LIVE_FILES table + zonePortForSlot bands
  (18787+(slot+1)*32; first band above the default base so bare 18787 is
  never a CI target)
- tests/preload-zone-slot.mjs: CI-only preload assigns each live file its
  band via BILI_ZONE_PORT (product support already in src/config.ts)
- run-ci-tests.mjs: preload injected for live suites
- port-zone-live: band-aware base + waitForBandFree at sandbox entry
  (teardown SIGKILL returns before kernel release) + full XDG isolation
  (a dev box's real config/index.db makes children queue on real locks —
  the #3044 fail-fast then never fires) + squatter leak closes on failure
- claude-native: band-aware helpers (ZONE_BAND_BASE from process.env;
  planClaudeNativeBootstrap({}) stays on the exported default), squatter
  test flipped to the #2877 own-lane refusal contract (was in the original
  red set — #2877 superseded #1660's attach-by-default), foreign-listener
  squatter kills connections (unhandled ECONNRESET crashed the runner),
  freePort TOCTOU bounded retry
- auto-restart: stubs exit 91 on EADDRINUSE with bounded scenario retries;
  SIGTERM-survivor escalation test is POSIX-only (Windows terminates
  uncatchably — was a w24 red)
- ensureDistBuilt (claude-native/port-zone-live/index-entry-guard): rebuild
  when src is newer than dist — a stale dist exercises last week's product
  code (the #3044 'hang' was exactly this)
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

核对完成(逐段读 diff × 最新 head、既有行为逐项清点、远程 CI 全量结论)。说明:分支在初审后 force-push(bea9327 → 1a320fb,commit message 改为「…+ deterministic teardown/dist hygiene」,新增 index-entry-guard 改动、port-zone-live 扩大隔离、claude-native 断言翻转),以下按最新 head 审。

结构化八问

1. 是什么问题:分片 + concurrency=4 之后,同一 runner 上多个会 spawn 真实 daemon 的 e2e 套件抢同一个默认 zone 基端口 18787 和 ephemeral 端口池(freePort bind-0-close 的 TOCTOU),间歇产生 EADDRINUSE / attach 误判 / stub 误报——#3145 之后每个 PR 都带同一批红 test leg,而 master push 不跑 test lane(guards 是 PR-gated,test job 被 needs 连带跳过),全程无信号。

2. 什么原因导致的:三层叠加——(a) #3145 LPT 分片把端口家族 live 套件集中进同一片;(b) #2816 固定 c=4,不同文件的 daemon 时间上共存;(c) #3055 加了对 18787 的硬断言,把原本静默的互相污染变成显式红。根因:zone 基端口是进程级单例资源,CI 调度却把文件当独立并行单元——资源模型与调度模型不一致。次因:teardown SIGKILL 返回时内核尚未释放 listener,下一场景首绑踩窗。

3. 影响范围与严重程度:只影响 CI 信号(以及本地并发跑套件的 dev 机);产品行为零变化(src/ 零改动)。严重度:开发流程级——所有 PR 的 test leg 间歇红、required status checks 被 flake 挡合并,真回归无法从噪声里分辨;非用户面问题。

4. 怎么修复的:策略 = 「每文件独占端口带 + 断言跟随带 + 仅对基础设施形失败做有界重试」:

  • A:scripts/live-files.mjs 共享 LIVE_FILES 表,zonePortForSlot(slot) = 18787 + (slot+1)*32,38 slot 最大 20003 < ephemeral 区;tests/preload-zone-slot.mjs(CI-only --import preload)利用 node --test 每文件独立子进程的 argv 识别本文件并注入 BILI_ZONE_PORT(spawned daemon 继承);run-ci-tests.mjs 把它加进 PRELOAD_ARGS,主阶段/flake 重试/CI flake:test (windows-latest, 24) 单文件文件级失败:子测试零执行、异常被吞为 'test failed' #3029 隔离诊断重跑三者同环境。第一带在默认基之上起跳:漏继承 env 的 spawn 落到 18787,撞不到任何断言,在自己套件里大声失败。
  • B/C:port-zone-live/claude-native 用 resolveZonePortBase() 跟随带;claude-native 四处显式 env spawn 点 spread ZONE_ENV(产品侧零改动,override 早已存在于 src/config.ts)。
  • D:claude-native squatter 对 freePort 被偷重试 ≤2 次;auto-restart 两个 autoRestartOnUpdate + occupied target port -> orphan process leak: new instance silently hops to the next port, old instance never exits #3044 场景对 stub exit 91(EADDRINUSE 标记,ready-stub 新增)重试 ≤2 次。产品回归三次同形失败,不掩盖任何事。
  • force-push 新增的伴随卫生项(PR 正文未列,均属同失败域):三个测试文件的 ensureDistBuilt 从「缺失才重建」改为「stale 也重建」(src/ mtime vs dist);port-zone-live 的 withSandbox XDG 隔离从 state 扩到 state+config+data+cache(dev 机真实 index.db 锁竞争曾致 120s 挂起);每场景前 waitForBandFree 有界等 10s(teardown SIGKILL 窗口);squatter close 移入 finally;claude-native squatter 加 connection-destroy(修未处理 ECONNRESET 崩进程,w22 红)。

5. 是否有回归问题:逐项清点过 diff 触碰路径的既有行为——

6. 是否破坏缓存:不涉及(无缓存/游标/去重键机制;BILI_ZONE_PORT 仅是测试环境的既有读取点,产品解析链未变)。

7. 是否改了设计方向:不涉及——无架构转向,本质是把资源隔离粒度对齐到调度粒度(compression/wire/update 路径零触碰)。

8. 是否引入了新的配置:无新产品配置。BILI_ZONE_PORT override 本就存在且已文档化(docs-coverage/config-surface-guard 绿);本 PR 只在 CI 测试环境使用它,无新 env、无新字段。

发现清单(处置二选一)

判定

机制成立、实现与声明一致(外加已披露的卫生项)、远程 CI 在 head 1a320fb 上 39/39 全绿(12 test leg = 4 平台 × 3 分片 + static-gates ×4 + 全部 guards/e2e lanes)。处理完 pages revert 一项后即可合(摘除,或你明确确认保留并补一句 PR 描述);小项 1/2 建议合并前顺手收掉,不收则 follow-up。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立验证(来自 #3206 排查,同失败家族):

ranxianglei pushed a commit that referenced this pull request Oct 14, 2026
This head went red on three windows lanes with two pre-existing failure
signatures that PR #3197 already fixes on its branch:

- auto-restart "survives SIGTERM" stub: POSIX-only scenario — Windows
  terminates on SIGTERM uncatchably, so the stub can never model a
  SIGTERM-surviving replacement (w22-2 + w24-2 reds). Skip on win32.
- claude-native "occupied stable port": the accepted probe socket's
  ECONNRESET was unhandled and crashed the test process (w24-3 red).
  Destroy connections eagerly.

Both hunks are verbatim mirrors of #3197's fixes so the branches stay
trivially reconcilable when the owner consolidates them.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

交叉链接(来自 #3215 审核):同场红风暴出现了第二个在途修复——PR #3215(2026-10-14_ci-port-zone-red-storm,Closes #3212),改的是与本 PR 相同的两个测试文件。它的「查重: 无」漏掉了本 PR(本 PR 建于 15:57Z,#3212 立案于 19:10Z)。按政策两者都不擅自关闭,合并载体由你拍板,要点:

#3215 侧的完整八问与协调建议见该线程;两边结论一致:最终只留一个交付载体。

@ranxianglei
ranxianglei merged commit b01f96d into master Oct 15, 2026
23 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.432-master.b01f96d (rolling master tag).

npm install -g billion-context@0.0.432-master.b01f96d  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment