Repository navigation
fix: attribute + one-shot flake recovery for swallowed file-level CI test failures - #3036
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3433-pr.3036.45d32feThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3036.tgz
npm install -g packageInstall instructions are refreshed on each push. |
…test failures (#3029) When a test child dies silently (non-zero exit, no stderr — e.g. OOM kill, tsx/esbuild worker death, race exit under concurrency), node --test collapses it to one file-level 'not ok <file>' whose only payload is error:'test failed' and zero subtests; no reporter can surface an error the child never emitted. CI was left unable to attribute the failure (#3029). run-ci-tests.mjs now, on failure: - captures exit code + signal (previously discarded by an empty catch); - logs a resource snapshot (free/total mem, loadavg, node version, platform); - surfaces child stderr when present (may carry a V8/native fatal such as OOM 'Reached heap limit'); - detects the swallowed shape (top-level 'not ok … *.test.ts' + error:'test failed') and re-runs each such file in isolation for attribution; - retries the whole phase ONCE and exits 0 only if that retry is green (a confirmed transient). Concrete assertion failures are NOT the swallowed shape and still fail hard immediately with no retry, so real regressions are never masked. Behavior changes (flagged): - The test step now forces --test-reporter=tap (Windows log format spec->tap; Linux was already tap). No test-semantics change. - Swallowed-shape failures get one bounded recovery (isolated re-run + full retry -> exit 0 if green) instead of failing immediately. Concrete assertion failures are unchanged (fail immediately). - No new config/env added; concurrency stays 4.
90c5af6 to
158f34e
Compare
(ai: vllm-qwen/qwen3.8-27b) Review 结论:可合并(merge 为 human-only,请自行合并)。代码审查通过,已 rebase 到最新 master 的 head Review 期间执行的动作
验证过程(证据,非「应该没问题」)
CI 时间线
小发现(判定记录,不立项)load-time 后续建议若 claude-native hook e2e 在 windows(任意 lane)再次超时,建议届时单独立 issue(症状/机制均不同于 #3029:可见超时 vs 不可见的 file-level 死亡;2026-10-13 同症状同 issue 规则下它不属于本 issue 家族)。目前仅一次数据点,不足以立项。 结构化八问
判定:可合并。 唯一剩余步骤是人工合并(PR #3036,head |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.377-master.4b3b75f # this exact build
npm install -g billion-context@master # always the newest merged build
|
Closes #3029
What & why
test (windows-latest, node24)sporadically fails ONE file at file level with zero subtests executed and the only message'test failed'— the real exception is invisible, so triage becomes guesswork (repro run 38381197626). It's proven a flake (green on rerun / green on real Windows), not a regression. This PR fixes the diagnostic gap so the next occurrence is attributable and self-recovers when it is a transient.Root cause (verified locally)
node --testruns each file as a child process. When a child dies silently — non-zero exit with no stderr (OOM kill, tsx/esbuild worker death, race exit under--test-concurrency) — the runner collapses it to one top-levelnot ok <file>whose YAML payload is onlyerror: 'test failed', zero subtests. A reporter swap cannot recover an error the child never emitted, and the oldexecFileSync(stdio:"inherit")+ emptycatchdiscarded even the exit code/signal — leaving nothing to read. (A clean load-timethrowdoes print its stack; this shape is specifically the silent-death class.)Fix (
scripts/run-ci-tests.mjsonly)On failure the runner now:
Reached heap limitlands there);not ok … *.test.ts+error: 'test failed'; successful files promote their tests by name and never emit a file-level entry) and re-runs each such file in isolation → distinguishes "reproduces solo = deterministic break" from "passes solo = concurrency/resource flake";Safety: concrete assertion failures (subtests ran and reported a real message) are not the swallowed shape → they fail hard immediately with no retry, so a real regression is never masked.
Behavior changes (flagged)
--test-reporter=tap: Windows log format spec → tap (Linux was already tap). No test-semantics change; makes the TAP stream parseable cross-platform and surfaces clean-throw stacks.--test-concurrencystays 4. The upstream storage-baseline preload (--import ./tests/preload-storage-baseline.mjs) is preserved and shared with the isolated re-run so it sees the same environment.Why script-level retry instead of
nick-fields/retryThe core pain is attribution, which a blind workflow rerun does not provide. Script-level keeps it dependency-free, gives per-file attribution + memory data, and mirrors the accepted "green full rerun = resolved" criterion. If you'd prefer the zero-JS workflow action instead, it's a trivial swap — say the word.
Testing
Validated end-to-end against the real script in a hermetic sandbox (tsx installed, all LIVE_FILES stubbed):
FLAKE CONFIRMED, exit 0;WITHOUT auto-retry, no isolate/retry → exit 1.Full-suite verdict = remote CI on the pushed head (local full-suite runs are out of scope per testing discipline).