Repository navigation
ci: shard the test suite across the matrix to cut wall time ~3x (#3143) - #3145
Conversation
The suite outgrew one runner: ~656 top-level files (~190k LOC) serialized through --test-concurrency=4 on a single 4-core leg took ~17-20 min (windows slowest; measured 1225s on the last full green run). Add a `shard` matrix dimension to the test job so each leg runs a balanced slice; run-ci-tests.mjs now partitions deterministically + totally (LPT bin-packing over file size) so the union of all shards == allFiles and no test is skipped. Same tests, assertions, env, OS and node versions — zero coverage loss. Wall drops ~3x (~20 -> ~7 min).
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3585-pr.3145.418ec36This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3145.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 审核结论:分片机制成立、零覆盖损失已证、墙时间实测降 ~2.3x;2 条红腿全部归类为既有 Windows 环境型 flake(与本 PR 无因果),已触发 rerun,转绿即可合并。 先说我独立做过的验证(不复述 PR 描述):
2 条红腿(均在 Windows)逐一定性,均非本 PR 引入:
判定依据:两个失败套件在此前 master 连续 8 次全量 run(含 Windows)均通过;本 PR 不触碰任何测试与服务端代码;分片只改变哪些文件同机共跑,且单机压力下降(656→222 文件/腿);两条红腿日志中多处 EPERM 锁症状指向当天 runner 环境抖动。三点均为既有脆弱断言,已立项 #3147(带来源标记,含三个子缺陷与建议修法),不塞进本 PR。 八问:
剩余风险(供你判断):
收工判定:机制成立、「质量不能掉」前提已证(零跳过、断言未动、tripwire 全局)。rerun 已触发(run 38525560473 的 rerun-failed-jobs,HTTP 201),预期转绿;若你接受 flake 归类(依据如上)也可直接合并,后续加固走 #3147。 |
(ai: vllm-qwen/qwen3.8-27b) rerun 结果:全绿,flake 定性坐实。 触发的 rerun(run 38525560473)已完成: 最终判定:可合并。 机制(确定性 LPT 分区、零覆盖损失、tripwire 全局、#3029 语义保留)独立验证成立;墙时间实测 ~20 min → long pole ~9 min;CI 全绿。合并动作留给你。 |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.405-master.42881c6 # this exact build
npm install -g billion-context@master # always the newest merged build
|
…#3182/#3184/#3188) #3145 sharded the test matrix and concentrated the port-family live suites into the same shards; with #2816's intra-shard file concurrency of 4 the files fight over the default zone base 18787 and the shared ephemeral port pool. port-zone-live pins 18787 (EADDRINUSE), claude-native/auto-restart lose freePort() picks to concurrent binders (TOCTOU), and master push CI skips the test lanes entirely so nobody saw it — every PR since #3145 shows the same 7 red legs (#3182/#3184/#3188, empty-PR proof in #3190). - scripts/live-files.mjs: shared LIVE_FILES table + ZONE_SLOT_BAND=32; zonePortForSlot(slot) = 18787 + (slot+1)*32 (first band above the default base so an env-ignoring spawn lands on 18787 and collides with nobody's assertions — it fails loudly in its own suite). Max band 20003 < 32768. - tests/preload-zone-slot.mjs: CI-only preload; node --test runs each file in its own process, so scanning argv for tests/*.test.ts identifies this file; sets BILI_ZONE_PORT for LIVE_FILES members. npm test is untouched. - scripts/run-ci-tests.mjs: import the shared table; add the preload to PRELOAD_ARGS (main phase + diagnostic re-runs see the same environment). - port-zone-live/claude-native: shadow the imported ZONE_PORT_BASE with resolveZonePortBase() so every assertion follows the injected band; claude-native child envs spread ZONE_ENV. - claude-native squatter + auto-restart stub tests: bounded retries on the infra shapes (stolen freePort pick / stub exit 91 on EADDRINUSE) so a shared-runner race is retried, never masked as a product failure. Local proof (this box runs real legacy daemons on 18787/18831/18834): master fails zone-live 7/9 in isolation; this branch fails 2/9, both starvation-shaped under a concurrent foreign full-suite run, and the 9-file port-family set at concurrency 4 shows zero cross-file collisions.
…/dist hygiene (#3182/#3184/#3188) Root cause of the 7-red test matrix: #3145 sharding concentrated the port families into shards 2/3 while #2816 concurrency let them bind simultaneously on one runner — cross-file 18787 collisions and shared ephemeral-port TOCTOU. Fixes: - scripts/live-files.mjs: shared LIVE_FILES table + zonePortForSlot bands (18787+(slot+1)*32; first band above the default base so bare 18787 is never a CI target) - tests/preload-zone-slot.mjs: CI-only preload assigns each live file its band via BILI_ZONE_PORT (product support already in src/config.ts) - run-ci-tests.mjs: preload injected for live suites - port-zone-live: band-aware base + waitForBandFree at sandbox entry (teardown SIGKILL returns before kernel release) + full XDG isolation (a dev box's real config/index.db makes children queue on real locks — the #3044 fail-fast then never fires) + squatter leak closes on failure - claude-native: band-aware helpers (ZONE_BAND_BASE from process.env; planClaudeNativeBootstrap({}) stays on the exported default), squatter test flipped to the #2877 own-lane refusal contract (was in the original red set — #2877 superseded #1660's attach-by-default), foreign-listener squatter kills connections (unhandled ECONNRESET crashed the runner), freePort TOCTOU bounded retry - auto-restart: stubs exit 91 on EADDRINUSE with bounded scenario retries; SIGTERM-survivor escalation test is POSIX-only (Windows terminates uncatchably — was a w24 red) - ensureDistBuilt (claude-native/port-zone-live/index-entry-guard): rebuild when src is newer than dist — a stale dist exercises last week's product code (the #3044 'hang' was exactly this)
…ascade-skip the legs The test job needs 9 guards; 4 are pull_request-only (version/kernel/ pi-subagents/lane-guard). A skipped need cascades the skip to dependents (GitHub-documented behavior) — the workflow comment's 'skipped needs count as satisfied' never held. Result: every master push ran CI with 'test: skipped' (e.g. run 38665531939), so master has had NO post-merge unit-test signal — the hole that let ranxianglei#3145 (port collisions), ranxianglei#3027 (golden drift) and ranxianglei#3035 slip red to master unnoticed. Fix: explicit 'if: !failure() && !cancelled()' on the test job: - pull_request: guards run; a guard failure still skips the legs (required checks pending, merge blocked — ranxianglei#2637 fail-fast tiering intact) - push/dispatch: PR-only guards skip harmlessly, the 12 shard legs RUN and master gets a real post-merge signal Cost: bounded by the existing per-ref concurrency cancel (owner-approved 2026-10-10) — rapid merge sequences only pay for the last push.
Closes #3143
What & why
CI wall time is dominated by the single
testleg: ~656 top-level.test.tsfiles (~190k LOC) serialize through--test-concurrency=4on one 4-core runner. Measured on the last full-green run: windows(22) 1225s ≈ 20 min (long pole), windows(24) 1100s, ubuntu(22/24) ~1030s. The suite grew past #2816's ~613-file baseline and outgrew one host; concurrency=4 already saturates the cores, so a single leg cannot go faster.Fix — shard across the matrix (zero coverage loss)
Add a
shard: [1,2,3]dimension to thetestjob so each leg runs a balanced slice.scripts/run-ci-tests.mjspartitions the file list deterministically and totally: LPT bin-packing over a line-count weight puts every file in exactly one shard, so union of all shards == allFiles — nothing is skipped, and adding a file only re-bins, never drops another. Balance targets the heaviest shard (the wall clock), so the 7k-line outlier doesn't pile onto one leg.Same tests, same assertions, same env (
ACP_TEST_CLAUDE_NATIVE), same OS × node matrix. Expected wall: ~20 → ~7 min (÷3). To go faster later, add entries to theshardlist and bump the/3denominator together (they must stay equal).Quality (the "质量不能掉" premise)
docs-coverage,config-surface-guard) all green. Full-suite verdict comes from the CItestlanes on this head.New surface
One internal CI variable
CI_TEST_SHARD="i/N"— read byscripts/run-ci-tests.mjs, set only by the workflow matrix; unset = run everything, so localnpm testis unchanged. Not a bili runtime/config knob; documented inreference/testing.md.