Skip to content

fix: attribute + one-shot flake recovery for swallowed file-level CI test failures - #3036

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-13_ci-flake-attribution
Oct 13, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-13_ci-flake-attribution

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3029

What & why

test (windows-latest, node24) sporadically fails ONE file at file level with zero subtests executed and the only message 'test failed' — the real exception is invisible, so triage becomes guesswork (repro run 38381197626). It's proven a flake (green on rerun / green on real Windows), not a regression. This PR fixes the diagnostic gap so the next occurrence is attributable and self-recovers when it is a transient.

Root cause (verified locally)

node --test runs each file as a child process. When a child dies silently — non-zero exit with no stderr (OOM kill, tsx/esbuild worker death, race exit under --test-concurrency) — the runner collapses it to one top-level not ok <file> whose YAML payload is only error: 'test failed', zero subtests. A reporter swap cannot recover an error the child never emitted, and the old execFileSync(stdio:"inherit") + empty catch discarded even the exit code/signal — leaving nothing to read. (A clean load-time throw does print its stack; this shape is specifically the silent-death class.)

Fix (scripts/run-ci-tests.mjs only)

On failure the runner now:

  1. captures exit code + signal (previously discarded by the empty catch);
  2. logs a resource snapshot (free/total mem, loadavg, node version, platform) — directly serves the suspected node24-win + tsx + concurrency OOM angle;
  3. surfaces child stderr when present (a V8/native fatal such as OOM Reached heap limit lands there);
  4. detects the swallowed shape (top-level not ok … *.test.ts + error: 'test failed'; successful files promote their tests by name and never emit a file-level entry) and re-runs each such file in isolation → distinguishes "reproduces solo = deterministic break" from "passes solo = concurrency/resource flake";
  5. retries the whole phase once and exits 0 only if that retry is green (a confirmed transient), else stays red.

Safety: concrete assertion failures (subtests ran and reported a real message) are not the swallowed shape → they fail hard immediately with no retry, so a real regression is never masked.

Behavior changes (flagged)

  • Test step now forces --test-reporter=tap: Windows log format spec → tap (Linux was already tap). No test-semantics change; makes the TAP stream parseable cross-platform and surfaces clean-throw stacks.
  • Swallowed-shape failures now get one bounded recovery (isolated re-run + full retry → exit 0 if green) instead of failing immediately. Concrete assertion failures are unchanged (fail immediately).
  • Exit code/signal + resource snapshot are now logged on failure (were discarded).
  • No new config/env; --test-concurrency stays 4. The upstream storage-baseline preload (--import ./tests/preload-storage-baseline.mjs) is preserved and shared with the isolated re-run so it sees the same environment.

Why script-level retry instead of nick-fields/retry

The core pain is attribution, which a blind workflow rerun does not provide. Script-level keeps it dependency-free, gives per-file attribution + memory data, and mirrors the accepted "green full rerun = resolved" criterion. If you'd prefer the zero-JS workflow action instead, it's a trivial swap — say the word.

Testing

Validated end-to-end against the real script in a hermetic sandbox (tsx installed, all LIVE_FILES stubbed):

  • transient flaky file → detected, isolated re-run passed, full retry green → FLAKE CONFIRMED, exit 0;
  • deterministic silent death → isolated REPRODUCED, retry red → exit 1 (not masked);
  • concrete assertion failure → WITHOUT auto-retry, no isolate/retry → exit 1.

Full-suite verdict = remote CI on the pushed head (local full-suite runs are out of scope per testing discipline).

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-13_ci-flake-attribution (158f34e)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3433-pr.3036.45d32fe

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3036.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,正在按 review 规范走:先审原 issue #3029 方向,再逐行读 diff、核对 CI 状态,稍后给结构化八问结论。

…test failures (#3029)

When a test child dies silently (non-zero exit, no stderr — e.g. OOM kill, tsx/esbuild worker death, race exit under concurrency), node --test collapses it to one file-level 'not ok <file>' whose only payload is error:'test failed' and zero subtests; no reporter can surface an error the child never emitted. CI was left unable to attribute the failure (#3029).

run-ci-tests.mjs now, on failure:
- captures exit code + signal (previously discarded by an empty catch);
- logs a resource snapshot (free/total mem, loadavg, node version, platform);
- surfaces child stderr when present (may carry a V8/native fatal such as OOM 'Reached heap limit');
- detects the swallowed shape (top-level 'not ok … *.test.ts' + error:'test failed') and re-runs each such file in isolation for attribution;
- retries the whole phase ONCE and exits 0 only if that retry is green (a confirmed transient). Concrete assertion failures are NOT the swallowed shape and still fail hard immediately with no retry, so real regressions are never masked.

Behavior changes (flagged):
- The test step now forces --test-reporter=tap (Windows log format spec->tap; Linux was already tap). No test-semantics change.
- Swallowed-shape failures get one bounded recovery (isolated re-run + full retry -> exit 0 if green) instead of failing immediately. Concrete assertion failures are unchanged (fail immediately).
- No new config/env added; concurrency stays 4.
@ranxianglei
ranxianglei force-pushed the 2026-10-13_ci-flake-attribution branch from 90c5af6 to 158f34e Compare October 13, 2026 12:03
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 结论:可合并(merge 为 human-only,请自行合并)。代码审查通过,已 rebase 到最新 master 的 head 158f34e18 上 19/19 checks 全绿。

Review 期间执行的动作

  1. rebase 到当前 master:审查期间 master 合入了 test: regen storage-render goldens for the #3023 cache-ledger client-insert attribution #3035(storage-render golden regen)——这正是旧 head CI 全红的根因(见下)。本 PR 只动 scripts/run-ci-tests.mjs,与 master 新增变更零文件重叠,clean rebase 后 force-with-lease 推送。
  2. 触发 rerun-failed-jobs:新 head 上 windows-22 lane 出现一次 live e2e 超时(首次数据点,rerun 即绿,见下)。

验证过程(证据,非「应该没问题」)

  • TAP 形状本地实测(node v22.23.2,pass/fail/silent/throw 四种子进程):通过的文件和带具体断言失败的文件都按测试名提升到顶层、不产生 file-level 条目;silent-death 子进程恰好产生 not ok <file> + error: 'test failed' + 零 subtests——检测前提成立。
  • 解析器模拟:混合场景只检出 silent-death 文件、不检出具体断言失败 → 安全门(真实回归永不进重试路径)成立。
  • 安全门生产双重实证:旧 head 上 storage-render golden 失配(具体断言失败)四条 lane 全部立即 hard-fail、无重试;新 head 上 claude-native e2e 超时同样走无重试路径(job 在 phase 结束后 ~5s 即终止,无第二个 phase)——真实失败从未被掩盖。
  • 下游契约:ci.yml 只消费退出码(0/1 契约不变)、无日志解析;--test-concurrency 保持 4;LIVE_FILES tripwire 不变;preload-storage-baseline 主 phase 与隔离重跑共享,环境一致。

CI 时间线

小发现(判定记录,不立项)

load-time throw 的子进程也会匹配 swallowed 形状(子在 TAP 完成前死亡)→ 会先进隔离重试再红。这是安全的:确定性问题重试后照样红,只是检测延迟一个 phase;注释措辞比实际略强,但结论成立。属于形状检测的固有取舍,不构成缺陷。

后续建议

若 claude-native hook e2e 在 windows(任意 lane)再次超时,建议届时单独立 issue(症状/机制均不同于 #3029:可见超时 vs 不可见的 file-level 死亡;2026-10-13 同症状同 issue 规则下它不属于本 issue 家族)。目前仅一次数据点,不足以立项。

结构化八问

  1. 是什么问题:CI 测试 lane 偶发以「文件级」失败变红——只有一个文件条目、零 subtests、唯一信息 test failed,无堆栈无退出码,归因只能靠猜(CI flake:test (windows-latest, 24) 单文件文件级失败:子测试零执行、异常被吞为 'test failed' #3029,windows-node24 复现)。
  2. 什么原因导致:node --test 每文件一个子进程;子进程静默死亡(非零退出 + 无 stderr,如 OOM/worker 死/race exit)时,runner 把它坍缩成顶层 not ok <file> + error:'test failed';旧脚本 execFileSync(stdio:"inherit") + 空 catch 连 exit code/signal 也丢弃 → 现场无任何可读信息。
  3. 影响范围与严重程度:仅 CI 诊断缺口。偶发单 lane 红、重跑即绿;不影响包功能与终端用户。严重度 = triage 效率问题(猜测 vs 归因+自愈),非功能缺陷。
  4. 怎么修复的:脚本层(scripts/run-ci-tests.mjs 单文件)可观测性增强 + 一次性有界自愈:保留 exit/signal → 失败时资源快照(内存/loadavg/node 版本/平台)→ 呈现子进程 stderr → 识别 swallowed 形状 → 逐文件隔离重跑(区分「确定性坏」vs「并发/资源 flake」)→ 整 phase 重试一次,绿才 exit 0。安全边界:有 subtests 的真实断言失败永不进恢复路径,立即 hard-fail,真实回归永不被掩盖。
  5. 是否有回归问题:无。逐项核过:退出码契约(ci.yml 只消费 0/1)不变;并发保持 4;LIVE_FILES tripwire 不变;preload 环境主/隔离一致;Windows 日志格式 spec→tap 为已披露的行为变更(无测试语义变化,使 TAP 跨平台可解析);生产双实证确认真实失败仍立即失败。高危模式(语义漂移/fail-fast 变吞错/阈值漂移)逐项过,均未命中。
  6. 是否破坏缓存:不涉及(不触碰任何缓存/重试游标/去重键机制)。
  7. 是否改了设计方向:否。仅给 CI 测试 runner 增加观测与有界自愈,架构边界不变;script-level 而非 workflow 盲目重跑的理由(归因优先)成立。
  8. 是否引入新配置:无。无新 env/config/默认值;--test-concurrency 保持 4。

判定:可合并。 唯一剩余步骤是人工合并(PR #3036,head 158f34e18)。

@ranxianglei
ranxianglei merged commit 4b3b75f into master Oct 13, 2026
36 of 37 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.377-master.4b3b75f (rolling master tag).

npm install -g billion-context@0.0.377-master.4b3b75f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei
ranxianglei deleted the 2026-10-13_ci-flake-attribution branch October 15, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI flake:test (windows-latest, 24) 单文件文件级失败:子测试零执行、异常被吞为 'test failed'

1 participant