Repository navigation
[PR] CI: build once, reuse everywhere — cache kernel/pi-subagents dist by content hash so legs stop rebuilding 6 minutes of artifacts every run - #4377
Merged
Conversation
The suite grew ~399 -> ~860 files in three weeks while the core matrix shape stayed frozen at its ~650-file design point (#3295), so its heaviest legs became the per-PR wall-clock floor: the ubuntu-24 single-leg whole suite measured ~21 min and the windows 2-shard halves ~13-17 min (up to ~35 min tails on flake re-runs). All three core combos now shard /3. partitionAll stays total for any i/N, so the union of shards is still every file — zero coverage loss; the wall drops to the slowest shard (~7-12 min instead of 20-25). The win node-24 legs stay FULL-scope only. Check-run names are unchanged (explicit job name template), and all nine core leg names already sit in master's required-checks list.
…lding 6 minutes of artifacts every run (#4360)
Contributor
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.757-master.bb85f94 # this exact build
npm install -g billion-context@master # always the newest merged build
|
This was referenced Oct 23, 2026
Closed
ranxianglei
pushed a commit
that referenced
this pull request
Oct 23, 2026
Line-count LPT balanced bytes, not seconds: process-spawning suites cost orders of magnitude more per line than pure assertion files, and the 'perfectly balanced' layouts still produced 5min..37min legs from equal byte counts (windows 22-2/22-3 vs 22-1/22-4 on the same run). With #4384's linger cut landed the per-file wall times are stable enough to weight the partition by MEASURED duration: - scripts/ci-shard-partition.mjs: fileWeight = last measured ms per file (scripts/ci-shard-weights.json), new files fall back to lineCount x msPerLineFallback (median ms/line); same deterministic total LPT otherwise. - scripts/ci-shard-weights.json: 840-file table measured post-#4384 (single-file node --test runs, VM, ~66min serial total); local-env hangs capped at the 150s timeout and listed under timedOut, local rc=1 env-fails excluded. Regenerate via scripts/measure-test-durations.mjs when shard balance degrades. - run-ci-tests.mjs delegates partitionAll to the module (this script has top-level side effects and cannot be imported by tests). - ci.yml: windows-22 CORE 3->5 shards, windows FULL 3->5 (both node versions) — measured max shard drops 1984s->794s serial (concurrency c=4 per shard => ~7min worst-leg target incl. setup). Closes #4360 (with #4377's dist cache + #4384's linger cut, the three levers of the 8-minute goal)
This was referenced Oct 23, 2026
ranxianglei
pushed a commit
that referenced
this pull request
Oct 23, 2026
Line-count LPT balanced bytes, not seconds: process-spawning suites cost orders of magnitude more per line than pure assertion files, and the 'perfectly balanced' layouts still produced 5min..37min legs from equal byte counts (windows 22-2/22-3 vs 22-1/22-4 on the same run). With #4384's linger cut landed the per-file wall times are stable enough to weight the partition by MEASURED duration: - scripts/ci-shard-partition.mjs: fileWeight = last measured ms per file (scripts/ci-shard-weights.json), new files fall back to lineCount x msPerLineFallback (median ms/line); same deterministic total LPT otherwise. - scripts/ci-shard-weights.json: 840-file table measured post-#4384 (single-file node --test runs, VM, ~66min serial total); local-env hangs capped at the 150s timeout and listed under timedOut, local rc=1 env-fails excluded. Regenerate via scripts/measure-test-durations.mjs when shard balance degrades. - run-ci-tests.mjs delegates partitionAll to the module (this script has top-level side effects and cannot be imported by tests). - ci.yml: windows-22 CORE 3->5 shards, windows FULL 3->5 (both node versions) — measured max shard drops 1984s->794s serial (concurrency c=4 per shard => ~7min worst-leg target incl. setup). Four NEW check names appear (windows 22/24 shards 4/5); branch protection must add them before they gate merges. Cherry-picked from closed PR #4361 (commit 0569781, stranded when the PR was superseded by #4377; picked up per owner invitation in the #4361 thread). The sibling LIVE_FILES commit (d06d192) is already on master via #4416 and is not included. Closes #4437
ranxianglei
added a commit
that referenced
this pull request
Oct 24, 2026
…n guard fired on restored artifacts The #4377 cache skip left kernel/dist with save-time mtimes while the checkout rewrote kernel/src mtimes; kernelDistStale() compares mtimes, so the first real cache-HIT PR (this one — no kernel changes) made the golden-storage drives refuse to run on 5 legs. Touch the dist tree on every HIT (metadata-only; bytes are already marker-verified).
ranxianglei
added a commit
that referenced
this pull request
Oct 24, 2026
…n guard fired on restored artifacts The #4377 cache skip left kernel/dist with save-time mtimes while the checkout rewrote kernel/src mtimes; kernelDistStale() compares mtimes, so the first real cache-HIT PR (this one — no kernel changes) made the golden-storage drives refuse to run on 5 legs. Touch the dist tree on every HIT (metadata-only; bytes are already marker-verified).
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4360(lever 1)。叠在 #4355 之上(本分支基于
2026-10-23_ci-core-matrix-rebalance的 head e2e26b1),先合 #4355 再合本 PR。现象
CI 的 test 车道(尤其 Windows 腿)每条腿在跑任何测试之前,都要把
kernel/和pi-subagents/的 dist 从零全量重建一遍 —— Windows 上一次就是 5~6.5 分钟。这些构建产物只由源码和构建配置决定,内容没变也得重编,白白吃掉了每条腿近三分之一的耗时,Windows 腿因此拖到 20+ 分钟。修法
核心思路:构建产物是源码的纯函数,那就把"这份 dist 对应哪份源码"记下来,下次先对账、对上了就跳过重建。
scripts/ci-pretest-cache.mjs,接进npm run pretest(原来直接跑两条npm --prefix … run build,现在换成这个脚本;MISS 时执行的还是一模一样的构建命令)。.pretest-cache-marker。pretest-cache: HIT),不一致或没有 marker → 老老实实重建(pretest-cache: MISS → rebuilding)。ci.yml的 test 车道在npm install之后、跑测试之前,用actions/cache按同样的输入哈希(hashFiles,和脚本里哈希的输入一一对应,注释里写明了要保持同步)恢复两个 dist 目录;构建发生在run-ci-tests.mjs内部的 pretest 步骤里,job 成功后 actions/cache 自动按精确 key 存回去。按操作系统分 key(不跨系统共享 dist),不用 restore-keys(近似命中没有意义,marker 校验也会拒掉)。这样本地热跑
npm test同样受益(marker 校验不依赖 GitHub 缓存)。本地验证(三个场景)
① 冷启动(无 dist → MISS → 真实构建):
② 热启动(无改动 → HIT → 秒过):
③ 篡改(给 kernel/src/index.ts 追加一行注释 → 只有 kernel 重建,pi-subagents 不受牵连):
还原该文件后再跑 → kernel 重建、marker 回到最初的
87aaa7dbfc37(哈希确定性验证)。(本机 rebuild 快是因为机器快;Windows CI 上同样的重建正是每次 5~6.5 分钟 —— 那就是每条热腿省下的时间。)
测试与门禁: 新增
tests/ci-pretest-cache.test.ts(7/7 通过:冷/热/篡改生命周期、marker 逐字节判定、每类输入变更都改变哈希、哈希确定性+平台种子、缺输入报错、构建失败传播且不盖章);tests/test-suite-lock.test.ts19/19 不受影响;npm run typecheck干净;unused-exports / test-import-graph 门禁干净;ci.yml 通过 YAML 解析。注意
npm run build全量重建,没有 pretest 接缝)、pretypecheck(typecheck 车道仍每次重建)。这些留给 CI Windows test lane takes 20+ minutes: every leg does a full rebuild of kernel/pi-subagents dist — the sharding is way | CI windows 测试车道 20+ 分钟:每条腿都全量重建 kernel/pi-subagents dist,分片也太粗 #4360 后续杠杆。