Skip to content

ci: shard the test suite across the matrix to cut wall time ~3x (#3143) - #3145

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-14_ci-shard-test-suite
Oct 14, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-14_ci-shard-test-suite

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3143

What & why

CI wall time is dominated by the single test leg: ~656 top-level .test.ts files (~190k LOC) serialize through --test-concurrency=4 on one 4-core runner. Measured on the last full-green run: windows(22) 1225s ≈ 20 min (long pole), windows(24) 1100s, ubuntu(22/24) ~1030s. The suite grew past #2816's ~613-file baseline and outgrew one host; concurrency=4 already saturates the cores, so a single leg cannot go faster.

Fix — shard across the matrix (zero coverage loss)

Add a shard: [1,2,3] dimension to the test job so each leg runs a balanced slice. scripts/run-ci-tests.mjs partitions the file list deterministically and totally: LPT bin-packing over a line-count weight puts every file in exactly one shard, so union of all shards == allFiles — nothing is skipped, and adding a file only re-bins, never drops another. Balance targets the heaviest shard (the wall clock), so the 7k-line outlier doesn't pile onto one leg.

Same tests, same assertions, same env (ACP_TEST_CLAUDE_NATIVE), same OS × node matrix. Expected wall: ~20 → ~7 min (÷3). To go faster later, add entries to the shard list and bump the /3 denominator together (they must stay equal).

Quality (the "质量不能掉" premise)

New surface

One internal CI variable CI_TEST_SHARD="i/N" — read by scripts/run-ci-tests.mjs, set only by the workflow matrix; unset = run everything, so local npm test is unchanged. Not a bili runtime/config knob; documented in reference/testing.md.

The suite outgrew one runner: ~656 top-level files (~190k LOC) serialized
through --test-concurrency=4 on a single 4-core leg took ~17-20 min (windows
slowest; measured 1225s on the last full green run). Add a `shard` matrix
dimension to the test job so each leg runs a balanced slice; run-ci-tests.mjs
now partitions deterministically + totally (LPT bin-packing over file size)
so the union of all shards == allFiles and no test is skipped. Same tests,
assertions, env, OS and node versions — zero coverage loss. Wall drops ~3x
(~20 -> ~7 min).
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-14_ci-shard-test-suite (b8cad88)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3585-pr.3145.418ec36

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3145.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论:分片机制成立、零覆盖损失已证、墙时间实测降 ~2.3x;2 条红腿全部归类为既有 Windows 环境型 flake(与本 PR 无因果),已触发 rerun,转绿即可合并。

先说我独立做过的验证(不复述 PR 描述):

  • 分区正确性:把 PR head 的 parseShard/partitionAll 原样抽出,对真实文件集实跑:N=1..4 全部满足「划全(total)+ 互斥(disjoint)+ 确定性 + 与输入顺序无关」——跨 Windows/Linux runner 分区必然一致。对 PR 写作时的 656 文件集,N=3 负载 [63544, 63524, 63547]、skew 0.036%,与 PR 描述数字逐一复现;union(1..3) == allFiles。
  • 活体验证:本次 CI 12 条腿每条都打印 [ci-tests] shard i/3: 222/662 files (union of all shards == allFiles — nothing skipped)——不变量在全部腿上成立。(注:PR 写作时是 656 个文件,master 此后长了 6 个到 662,只重装箱、不丢件。)
  • tripwire 范围:unregistered-spawner guard 在代码里位于 shard 选择之前,仍扫全集,任何一条腿都能抓住未登记 spawner。
  • 本地不受影响:CI_TEST_SHARD 未设置时 selectedFiles == allFiles,本地 npm test 行为零变化。
  • 墙时间实测(本次 run):ubuntu 腿 315–428s,windows 腿 381–542s,long pole 542s ≈ 9 min,对比分片前 windows(22) 1225s ≈ 20 min → 实际 ~2.3x;「~7 min」估计对 windows 偏乐观但方向正确。

2 条红腿(均在 Windows)逐一定性,均非本 PR 引入:

  1. test (windows-latest, 24, 2):tests/fork-nudge-deadzone.test.ts:209「child turn must log its nudge decision」——收到响应后立即 readFileSync(bili.log),而 src/logger.ts:173 是异步追加写,高负载下日志还没落盘;nudge 行实际出现在 stderr(nudge idle: usage=31% (9213/30000)),行为正确,纯读时序竞态。
  2. 同腿:tests/storage-engine-proofs.test.ts responses-wire 子测试 teardown,rmrf EPERM(Temp 下 golden-responses-* 目录)超出 tests/tmp-rm.ts ~9.8s 重试预算——外部 AV/索引器锁,与 fix: survive a corrupt unified index by degrading to legacy-only (#2771) + storage-engine proof battery #2772/Windows CI: storage-engine-proofs 套件清理 temp 目录遇 EBUSY/EPERM,windows test leg 随机红 #2799/test(windows): driveWire golden-harness 清理期 index.db EBUSY 复发(#2799 族,#2808 后残留) #2908 同族;同文件其余兄弟子测试全过。
  3. test (windows-latest, 22, 3):tests/preflight-hold.test.ts:163,tHeaderMs=1724ms ≥ SLOW_MS=1500——进程内代理的绝对墙钟余量被高负载事件循环 stall 吃光;该子测试其余断言(hold 生效、keep-alive、真实流内容)全部成立。

判定依据:两个失败套件在此前 master 连续 8 次全量 run(含 Windows)均通过;本 PR 不触碰任何测试与服务端代码;分片只改变哪些文件同机共跑,且单机压力下降(656→222 文件/腿);两条红腿日志中多处 EPERM 锁症状指向当天 runner 环境抖动。三点均为既有脆弱断言,已立项 #3147(带来源标记,含三个子缺陷与建议修法),不塞进本 PR。

八问:

  1. 是什么问题:CI 墙时间被单条 test 腿垄断(660+ 个测试文件在一台 4 核机器上串行 ~20 min);本 PR 把套件按矩阵切成 3 份平衡分片并行跑。
  2. 什么原因导致的:套件规模长过单宿主承载(诊断为什么个别ci要跑10分钟以上,是否可以优化 #2816 基线 ~613 文件 → 现 662),concurrency=4 已打满核心,单机再快不了——容量问题,不是效率问题。
  3. 影响范围与严重程度:只影响 CI 墙时间与资源占用;零覆盖损失(分区划全已证 + 12 腿活体不变量);不影响任何运行时行为。
  4. 怎么修复的:LPT bin-packing 按文件行数权重做确定性划分(划全+互斥+顺序无关),每腿跑自己那片;tripwire 保持全局;CI flake:test (windows-latest, 24) 单文件文件级失败:子测试零执行、异常被吞为 'test failed' #3029 flake-retry 语义不变、作用域收窄到本片。
  5. 是否有回归问题:逐项审过既有行为——本地 npm test(不变)、tripwire 范围(不变)、claude-native e2e(ACP_TEST_CLAUDE_NATIVE 各腿都设,文件恰好落一片、执行一次)、static-gates 与 9 个 guard job(不动)、每腿 concurrency=4(不动)、TAP 吞错/重试路径(逻辑未动仅作用域变小)、fail-fast:false(既有)。一处弱于描述:shard 1 集中了 19/37 个 LIVE_FILES,按脚本头自己的公式算片内两-live 并发概率 ~4.6%,高于分片前的 ~1.9%——「稳定性 neutral-to-positive」字面成立(shard 3 只有 5 个),但对 shard 1 比说的弱;此类碰撞从未在 CI 实表现为红,整体风险低。
  6. 是否破坏缓存:不涉及(纯 CI 调度变更)。
  7. 是否改了设计方向:没有。os×node 矩阵 + hermetic 脚本的结构保留,只加一个 shard 维度。
  8. 是否引入了新的配置:一个内部 CI 变量 CI_TEST_SHARD="i/N",只由 workflow matrix 设置,未设置=全跑,不是 bili 运行时旋钮;已在 reference/testing.md 记录。

剩余风险(供你判断):

  • 「matrix.shard 条目数 == 分母 /N」目前靠注释维持:减腿不改分母会静默漏测(我实测 legs=2/denom=3 → 220 个文件不被任何腿执行);增腿超分母则 parseShard 直接 die(响亮失败)。危险方向是静默丢覆盖。如需要可加机械守卫(脚本或 workflow 里 assert 分母==len(matrix.shard))——属新增 CI 面,留你拍板。
  • 成本面:test 腿 4→12,每腿固定开销(npm install + kernel/pi-subagents pretest build)×3 并行重复——墙时间 ÷~3,总分钟数大致持平略升。

收工判定:机制成立、「质量不能掉」前提已证(零跳过、断言未动、tripwire 全局)。rerun 已触发(run 38525560473 的 rerun-failed-jobs,HTTP 201),预期转绿;若你接受 flake 归类(依据如上)也可直接合并,后续加固走 #3147。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

rerun 结果:全绿,flake 定性坐实。

触发的 rerun(run 38525560473)已完成:completed success,12 条 test 腿全部通过——包括此前红的 test (windows-latest, 24, 2) 和 test (windows-latest, 22, 3)。同一份代码、同一分区,重跑即过 → 确认是当天 Windows runner 环境抖动(日志刷盘竞态 / EPERM 锁超预算 / 事件循环 stall)打穿既有脆弱断言,不是分片引入的回归。三个脆弱点已在 #3147 立项跟进。

最终判定:可合并。 机制(确定性 LPT 分区、零覆盖损失、tripwire 全局、#3029 语义保留)独立验证成立;墙时间实测 ~20 min → long pole ~9 min;CI 全绿。合并动作留给你。

@ranxianglei
ranxianglei merged commit 42881c6 into master Oct 14, 2026
63 of 65 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.405-master.42881c6 (rolling master tag).

npm install -g billion-context@0.0.405-master.42881c6  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Oct 14, 2026
…#3182/#3184/#3188)

#3145 sharded the test matrix and concentrated the port-family live suites
into the same shards; with #2816's intra-shard file concurrency of 4 the
files fight over the default zone base 18787 and the shared ephemeral port
pool. port-zone-live pins 18787 (EADDRINUSE), claude-native/auto-restart
lose freePort() picks to concurrent binders (TOCTOU), and master push CI
skips the test lanes entirely so nobody saw it — every PR since #3145 shows
the same 7 red legs (#3182/#3184/#3188, empty-PR proof in #3190).

- scripts/live-files.mjs: shared LIVE_FILES table + ZONE_SLOT_BAND=32;
  zonePortForSlot(slot) = 18787 + (slot+1)*32 (first band above the default
  base so an env-ignoring spawn lands on 18787 and collides with nobody's
  assertions — it fails loudly in its own suite). Max band 20003 < 32768.
- tests/preload-zone-slot.mjs: CI-only preload; node --test runs each file
  in its own process, so scanning argv for tests/*.test.ts identifies this
  file; sets BILI_ZONE_PORT for LIVE_FILES members. npm test is untouched.
- scripts/run-ci-tests.mjs: import the shared table; add the preload to
  PRELOAD_ARGS (main phase + diagnostic re-runs see the same environment).
- port-zone-live/claude-native: shadow the imported ZONE_PORT_BASE with
  resolveZonePortBase() so every assertion follows the injected band;
  claude-native child envs spread ZONE_ENV.
- claude-native squatter + auto-restart stub tests: bounded retries on the
  infra shapes (stolen freePort pick / stub exit 91 on EADDRINUSE) so a
  shared-runner race is retried, never masked as a product failure.

Local proof (this box runs real legacy daemons on 18787/18831/18834):
master fails zone-live 7/9 in isolation; this branch fails 2/9, both
starvation-shaped under a concurrent foreign full-suite run, and the
9-file port-family set at concurrency 4 shows zero cross-file collisions.
ranxianglei added a commit that referenced this pull request Oct 14, 2026
…/dist hygiene (#3182/#3184/#3188)

Root cause of the 7-red test matrix: #3145 sharding concentrated the port
families into shards 2/3 while #2816 concurrency let them bind
simultaneously on one runner — cross-file 18787 collisions and shared
ephemeral-port TOCTOU.

Fixes:
- scripts/live-files.mjs: shared LIVE_FILES table + zonePortForSlot bands
  (18787+(slot+1)*32; first band above the default base so bare 18787 is
  never a CI target)
- tests/preload-zone-slot.mjs: CI-only preload assigns each live file its
  band via BILI_ZONE_PORT (product support already in src/config.ts)
- run-ci-tests.mjs: preload injected for live suites
- port-zone-live: band-aware base + waitForBandFree at sandbox entry
  (teardown SIGKILL returns before kernel release) + full XDG isolation
  (a dev box's real config/index.db makes children queue on real locks —
  the #3044 fail-fast then never fires) + squatter leak closes on failure
- claude-native: band-aware helpers (ZONE_BAND_BASE from process.env;
  planClaudeNativeBootstrap({}) stays on the exported default), squatter
  test flipped to the #2877 own-lane refusal contract (was in the original
  red set — #2877 superseded #1660's attach-by-default), foreign-listener
  squatter kills connections (unhandled ECONNRESET crashed the runner),
  freePort TOCTOU bounded retry
- auto-restart: stubs exit 91 on EADDRINUSE with bounded scenario retries;
  SIGTERM-survivor escalation test is POSIX-only (Windows terminates
  uncatchably — was a w24 red)
- ensureDistBuilt (claude-native/port-zone-live/index-entry-guard): rebuild
  when src is newer than dist — a stale dist exercises last week's product
  code (the #3044 'hang' was exactly this)
LoveDoLove pushed a commit to LoveDoLove-Forked-Projects/billion-context that referenced this pull request Oct 15, 2026
…ascade-skip the legs

The test job needs 9 guards; 4 are pull_request-only (version/kernel/
pi-subagents/lane-guard). A skipped need cascades the skip to dependents
(GitHub-documented behavior) — the workflow comment's 'skipped needs count
as satisfied' never held. Result: every master push ran CI with
'test: skipped' (e.g. run 38665531939), so master has had NO post-merge
unit-test signal — the hole that let ranxianglei#3145 (port collisions), ranxianglei#3027
(golden drift) and ranxianglei#3035 slip red to master unnoticed.

Fix: explicit 'if: !failure() && !cancelled()' on the test job:
- pull_request: guards run; a guard failure still skips the legs
  (required checks pending, merge blocked — ranxianglei#2637 fail-fast tiering intact)
- push/dispatch: PR-only guards skip harmlessly, the 12 shard legs RUN
  and master gets a real post-merge signal

Cost: bounded by the existing per-ref concurrency cancel (owner-approved
2026-10-10) — rapid merge sequences only pay for the last push.
@ranxianglei
ranxianglei deleted the 2026-10-14_ci-shard-test-suite branch October 15, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci已经运行超过20分钟了 检查为什么并作优化

1 participant