Skip to content

fix(tests): extend tmp-rm EBUSY retry budget for c=4 Windows teardown load (#2908) - #2914

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-12_rmrf-ebusy-budget-c4
Oct 12, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-12_rmrf-ebusy-budget-c4

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Closes #2908

What

win-latest leg of PR #2890 failed exactly one case — EBUSY: resource busy or locked, unlink '…\golden-chat-*\sessions\index.db' during golden-harness teardown (6468/6492 pass, 1 fail, 23 skip). ubuntu-22/24 + win24 were all green (including the anthropic/responses/google wire variants of the same harness), and rerun-failed-jobs made the whole run pass → non-deterministic, runner-load related. Same failure domain as the closed #2799 (fixed by #2808), with a residual window.

Triage (reproduce → layer → verdict)

Fix

Right-size the external-lock budget rather than mask anything: extend the RETRY_DELAYS_MS ladder in tests/tmp-rm.ts. It is only paid when a race actually happens — zero effect on the happy path.

Behavior change (old → new)

  • tmp-rm tear-down retry budget: [25,50,100,200,400,800,1600] (~3.2s total, 8 attempts) → [25,50,100,200,400,800,1600,3200,3200] (~9.6s total, 10 attempts). Reason: c=4 load pushed transient-lock duration past the pre-c=4 budget (this ladder was last tuned at ~3.2s by release v0.1.180 #1910). Worst case adds ~6.4s to a single teardown, and only when it actually hits a lock race (rare).
  • Matching golden pin updated in tests/tmp-rm.test.ts: storm case now expects 10 calls and ≥9400ms (was 8 / ≥3100ms).

Not taken (and why)

Verification

  • Local targeted runs: node --import tsx --test tests/tmp-rm.test.ts → 6/6 pass (storm case paid the full ~9.6s ladder); tests/storage-engine-proofs.test.ts → 12/12 pass including the four wire-parity tests that drive driveWire→rmrf. Linux cannot reproduce the Windows EBUSY, so local green = no-regression only.
  • Definitive confirmation requires watching the next several win-latest CI legs (the flake is non-deterministic by nature).

Scope

Single module (tests/tmp-rm.ts + its test). No wire / config / config-surface / kernel / persistence change. Pure test-infra hardening.

… load (#2908)

#2816's c=4 concurrent test scheduling raised per-runner load so a transient Windows AV/indexer lock on sessions/index.db can outlast the old ~3.2s rmrf window (win-latest leg of PR #2890: EBUSY unlink index.db). A probe confirmed the golden harness holds no self-handle at rmrf time (flushAllSessions closes the store's shadow first), so this re-sizes the external-lock budget rather than masking an in-process leak. Ladder 25..1600ms (~3.2s/8 attempts) -> 25..3200ms (~7.6s/10 attempts); only paid when a race actually happens.
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-12_rmrf-ebusy-budget-c4 (c3dafdb)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3242-pr.2914.5e6c72f

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2914.tgz
npm install -g package

Install instructions are refreshed on each push.

…), not ~7.6s

The #2908 disclosure under-stated the new budget: [25..1600,3200,3200]
sums to 9575ms. Tighten the storm-case pin from >=7400 to >=9400 so a
dropped rung (e.g. the 1600 step: 7975ms) can no longer slip through.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成:方向正确、triage 独立复核成立;顺手修了一处披露算术错误(commit c3dafdb + PR 描述已同步更正)。等新 head 的 CI 四腿绿即可合并。

独立复核(triage 三步,审核姿态重走一遍)

子缺陷 #1(driveWire finally 未关最终 store)—— 维持「已证伪」结论,代码路径逐行核过:

Review 中发现并已直接修掉的小问题(仅报告不是合法处置)

新梯 [25,50,100,200,400,800,1600,3200,3200] 总和是 9575ms ≈ 9.6s,不是披露写的 ~7.6s(差整整 2s;worst-case 增量实为 ~6.4s 而非 ~4s)。连带后果:tmp-rm.test.ts 的黄金钉 >= 7400 偏松——删掉 1600 这一档后总和 7975ms 仍能过 >= 7400,即钉抓不住「少一档」的回归。已在 PR 分支补一个标注清晰的 commit:

  • c3dafdb68:断言收紧为 >= 9400(任何单档被删都会红)+ 注释更正为 ~9.6s;PR 描述中三处数字(~7.6s/~4s/≥7400ms)已同步 PATCH 更正。
  • 本地定向验证:node --import tsx --test tests/tmp-rm.test.ts → 6/6 通过,storm 用例实测付满 ~9.6s 全梯(整文件 ~10s,与 9575ms 吻合)。

回归审查(既有行为清点)

diff 触碰的既有行为只有一项:teardown 重试预算(全仓 ~643 个 rmrf 调用点共用)。逐项判定:

  • happy path(无瞬态错误):零变化,不付任何 sleep;
  • 失败路径:放弃时机 ~3.2s/8 次 → ~9.6s/10 次,错误仍原样上抛——无 fail-fast 变吞错、无输出格式漂移;唯一副作用是未来若真出现句柄泄漏类回归,CI 信号每处延后最多 ~6.4s,不掩盖正确性问题;
  • 高危域(自动更新/dsh/lockfile/systemd/网络):未触碰;wire/config/kernel/persistence/product 代码零改动。

结构化八问

  1. 是什么问题:win-latest 腿 golden-harness 清理期偶发 EBUSY(Windows CI: storage-engine-proofs 套件清理 temp 目录遇 EBUSY/EPERM,windows test leg 随机红 #2799 族,fix(test): release self-held handles before temp-tree teardown — Windows EBUSY/EPERM (#2799) #2808 后残留窗口),任意 PR 有概率假红 + fleet 级重跑成本。
  2. 什么原因导致的:诊断为什么个别ci要跑10分钟以上,是否可以优化 #2816/fix(ci): explicit concurrency=4 + parallel static-gates — CI total 16min→11m43s (monitored) (#2816) #2848 的 c=4 并发调度抬高单 runner 负载,Windows Defender/索引器对 index.db 的瞬态锁时长超过了 release v0.1.180 #1910 时代按 ~3.2s 校准的重试预算;非自持句柄(已证伪)、非同进程 writer 饥饿(不适用)。
  3. 影响范围与严重程度:触发面 = 任意 PR 的 windows-latest test 腿;严重度 = CI 假红 + 重跑成本,无产品影响。
  4. 怎么修复的:按 c=4 现实重新标定外部锁预算(RETRY_DELAYS_MS 加两档 3200ms),只在真正撞锁时付出,不掩盖任何进程内缺陷。
  5. 是否有回归问题:无(wire/config/阈值语义均未动;唯一时序变化即本 PR 披露的预算本身,见上)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有——纯 test-infra 加固,同一失效域的既有机制内调参。
  8. 是否引入了新的配置:没有(梯子常量在测试模块内部,非 config surface)。

CI 与判定

  • 原 head 625b766a4:除 win22 外全绿(win22 被我推修复 commit 时自动取消,属正常 supersede);新 head c3dafdb68:guard/static-gates 全绿,四个 test 腿运行中。
  • 判定:可合并(请人工操作),以新 head 四腿绿为准。flake 本身的最终确认按 PR 所述需观察后续若干轮 win-latest 腿——单轮绿不能证明概率性修复,若后续再撞 EBUSY,说明外部锁时长分布比预估更肥,届时按 release v0.1.180 #1910 同款方式再升一档即可,机制不变。

@ranxianglei
ranxianglei merged commit 18776d5 into master Oct 12, 2026
18 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.323-master.18776d5 (rolling master tag).

npm install -g billion-context@0.0.323-master.18776d5  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant