Skip to content

fix(test): release self-held handles before temp-tree teardown — Windows EBUSY/EPERM (#2799) - #2808

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-11_test-teardown-self-handles
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-11_test-teardown-self-handles

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #2799

Root cause

The two red legs were facets of one failure domain — test teardown deleting a tree its own process still holds handles into — plus one instance that was already fixed upstream:

R2 (golden-chat-* EPERM) — new root cause found while working this issue. driveWire (tests/golden-storage-drive.ts) points XDG_STATE_HOME at its disposable tmp before startServer; startServer unconditionally calls configureLogger(opts.logFile ?? defaultLogFile()), and defaultLogFile() resolves XDG_STATE_HOME at call time (src/paths.ts), so the tee logger's process-lifetime WriteStream lands at <tmp>/billion-context/bili.log — inside the very tree finally's rmrf(tmp) deletes. On Windows a process cannot unlink a file it holds open, so the delete fails EPERM regardless of the retry budget (backoff only helps against external transient locks such as AV/indexer — that is what the existing ~3.2s ladder in tests/tmp-rm.ts defends against). Empirically confirmed on Linux: after any driveWire run, /proc/self/fd shows our own fd pointing into <deleted tmp>/billion-context/bili.log.

R1 (bili-engproof-corrupt-* index.db EBUSY) — already fixed upstream before this PR. Every sqlite handle in that suite is now closed before cleanup: 54f21d3 (PR #2781, open-phase + read-only stores) and 9365fb2 (PR #2788, failed-open handle close). The CI red predates both merges; the suite passes locally 10/10 on current master, and the Windows rename guard (tests/storage-engine-proofs.test.ts:380-382) pins "no live handle inside the dir". Recorded as the third checklist item in #2799; no code change needed here.

Changes (all test-hygiene; no product behavior)

  • src/logger.ts: new _releaseLogFileForTest() seam — synchronously closes the held log fd and clears its path. Neither existing exit works for this: closeLogger() keeps the path set (the next log() reopens into the deleted tree) and configureLogger('off') drops the stream without ever closing it (fd lingers until GC). The close is synchronous because an async end()/destroy() cannot complete while a teardown retry sleeps via Atomics.wait (blocked main thread ⇒ libuv never services the close within the retry window).
  • tests/tmp-rm.ts: rmrf() releases the self-held log handle when the current log path lies under the deletion target — a systemic guard for every importer. A family scan found ~20 suites that point XDG_STATE_HOME at their scratch tree around in-process servers; all of them using the shared helper are covered automatically.
  • tests/golden-storage-drive.ts: producer fix — logFile: 'off' so the harness never points the persistent logger into a disposable tree; plus explicit store.cancelAll()/closeQuietly() in finally instead of relying on flushAllSessions()' shadow-close side effect (an early exit mid-drive previously left index.db open, pinning the tree on Windows AND masking the real failure with a cleanup error).
  • tests/mode-upgrade-proof.test.ts: switched its local bare-rmSync rmrf to the shared retried helper.
  • tests/tmp-rm.test.ts: deterministic regression pin (guard fires, path cleared, deleted tree not resurrected by later logging).

Behavior changes (test-observable only)

  • After a teardown releases the self-held handle, file logging stops until the next startServer reconfigures (stderr continues). Old behavior: the logger kept holding the deleted tree's path (on Windows the deletion then failed). No product code path changes — the seam is test-only and only invoked from rmrf().
  • Golden-drive suites no longer create bili.log inside their tmp trees (it was deleted along with the tree anyway; nothing consumed it).
  • mode-upgrade-proof teardown gains the existing transient-retry semantics (was bare rmSync) — strictly more robust.

Config surface

None — no new env vars, config fields, or CLI flags.

Verification

Issue checklist status (#2799)

  • corrupt-index unlinkSync EBUSY — already fixed upstream (54f21d3 + 9365fb2); verified green on current master
  • golden-storage-drive.ts:559 rmrf EPERM — fixed (producer fix + systemic guard above)
  • 54f21d3 open-phase instance — recorded, stays fixed

Watch item (not blocking): a few suites set XDG_STATE_HOME + start in-process servers but their exact call ordering wasn't verified line-by-line (dump-mask, issue2421-dump-body-buffer, acp-cache-diff, auto-restart, loop-strict-echo* — the latter likely safe since startServer precedes the env set there). If any goes red on Windows with the same signature, triage it against this PR's guard; if one uses a non-shared local rmrf, switching it over is the mechanical follow-up.

…ows EBUSY/EPERM (#2799)

Two distinct self-held-handle mechanisms made Windows CI teardown reds
non-deterministic (R1: unlink index.db EBUSY in storage-engine-proofs;
R2: rmrf golden-chat-* EPERM). Backoff cannot absorb a lock held by the
deleting process itself, so release such handles up front:

- src/logger.ts: _releaseLogFileForTest() seam — synchronously close the
  tee-log fd and clear its path (closeLogger() keeps the path → next log()
  reopens into the deleted tree; configureLogger("off") leaks the fd).
- tests/tmp-rm.ts: rmrf() releases the handle when the current log path
  lies under the deletion target — systemic guard for every importer.
- tests/golden-storage-drive.ts: the producer fix — driveWire pointed
  XDG_STATE_HOME at its disposable tmp BEFORE startServer, landing the
  default log file INSIDE the tree finally deletes; set logFile:"off"
  and close the drive store explicitly in finally instead of relying on
  flushAllSessions()'s shadow-close side effect.
- tests/mode-upgrade-proof.test.ts: switch its local bare-rmSync rmrf to
  the shared retried helper.
- tests/tmp-rm.test.ts: deterministic regression pin for the guard
  (release + path-clear + no resurrection of the deleted tree).
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-11_test-teardown-self-handles (1388294)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3108-pr.2808.ac64325

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2808.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已拉取分支核对代码并确认 CI 状态。这是本 agent 自己开的 PR(#2808,Closes #2799),按 [PR] 流程只核对、不重做。

分支:2026-10-11_test-teardown-self-handles(单 commit 974ba6437,base master,mergeable=true / behind —— BEHIND 非阻塞)。5 文件,+91/-5。

CI(截至本条):全部 guard 绿(version-guard / unused-exports / kernel-guard / docs-sync / config-surface-guard / pi-subagents-guard / lane-guard / lock-sync / docs-coverage / build-artifact);4 个 test 矩阵(windows-latest 22/24、ubuntu-latest 22/24)仍在跑。测试落地后再补一条最终确认。

代码核对(逐文件,diff 与描述一致):

  • src/logger.ts 新增 _releaseLogFileForTest() 测试缝:同步关 tee-log fd 并清空路径。模块状态变量名(stream/streamFd/logPath/bytesWritten/reopenWarned)逐一核对无误。关键竞态专门查了:openStream 的 open 回调带 if (stream === s) 守卫,release 先把 stream 置 undefined,所以迟到的 open 不会把 streamFd 写回;且 rmrf 是同步的、紧跟 release 同块执行(Atomics.wait 阻塞主线程),in-flight 的 open 只可能在删除完成后才解析——要么目录已删致 open 失败走 error 自毁,要么早已完成由 closeSync 收掉。两条路都覆盖,无泄漏窗口。
  • tests/tmp-rm.ts:rmrf() 顶部加 releaseSelfHeldLogHandle(target)——当前 log 路径落在删除目标内才释放,面向所有 importer 的系统性守卫;包含判断(lf !== root && !lf.startsWith(root+sep))标准正确;重试阶梯未动。
  • tests/golden-storage-drive.ts:生产者修法 logFile:"off"(harness 不再把常驻日志指进一次性树)+ finally 显式 store?.cancelAll(); store?.closeQuietly();。重载路径会重新赋值 store,故 finally 关的是当前实例,无重复关同一实例(closeQuietly 幂等)。
  • tests/mode-upgrade-proof.test.ts:本地裸 rmSync 换共享重试助手;fs 仍被使用(mkdtempSync),无 unused-import 回归。
  • tests/tmp-rm.test.ts:确定性回归钉(释放 + 清路径 + 删除后不再复活树),真驱动机制而非恒真断言。

结构化八问:

  1. 是什么问题:两条 Windows CI 红腿同属一个失效域——测试 teardown 去删一个自己进程还握着句柄的临时树(R1 storage-engine-proofs 的 index.db EBUSY;R2 golden-chat-* EPERM);退避/重试吸收不了"删除进程自身持有的锁"。
  2. 什么原因导致:R2 新根因——driveWire 在 startServer 前就把 XDG_STATE_HOME 指到一次性 tmp,而 defaultLogFile() 调用时才解析 XDG,tee logger 的进程级 WriteStream 落进 finally 要删的那棵树,自持 fd → Windows unlink EPERM。R1 上游已修(54f21d389 + 9365fb216,test(storage): close index.db handles before rmrf — fix Windows EBUSY in storage-engine-proofs (#2772 follow-up) #2781/fix(storage): close the SQLite handle when UnifiedStore.open fails on a corrupt index (#2771 follow-up) #2788),sqlite 句柄清理前全关。
  3. 影响范围与严重程度:仅测试卫生,影响 Windows CI 稳定性(teardown 红),无产品/运行时影响(src 侧只是新增下划线前缀的测试专用导出)。严重度:CI 抖动,非用户可见。
  4. 怎么修复的:从源头释放自持句柄,而非掩盖 EPERM——logger 加同步关闭缝、rmrf 加系统性前置释放守卫、drive 层生产者改 logFile:"off" 并显式关 store、mode-upgrade 换共享助手、补确定性回归钉。
  5. 是否有回归:审过触碰路径既有行为——logger 状态重置语义、golden drive 的 store 关闭时序(重载路径重赋值,无重复关)、mode-upgrade 的 fs 导入(仍用)、tmp-rm 重试阶梯(未动)均保持;唯一行为变化是 teardown 释放后文件日志暂停到下次 startServer(stderr 续),已在 PR 披露且仅测试可观测。高危模式(fail-fast 变吞错、阈值/时序/输出格式漂移)逐项过,产品码零命中。
  6. 是否破坏缓存:不涉及——未触碰任何缓存/重试游标/去重键逻辑。
  7. 是否改了设计方向:不涉及——仅加测试缝 + 系统性守卫,架构/边界/数据流不变。
  8. 是否引入新配置:不涉及——无新 env/config/CLI flag(logFile:"off" 是 harness 的 proxy option,非配置面)。

判定:代码审查干净,与描述一致,修法打在根因上;等 4 个 test 矩阵转绿即可合并,无需改动。测试落地我再补最终确认。

… warn still fires (#2808)

The shared rmrf() guard added in #2808 releases the self-held tee-log handle AND
clears the log path before deleting (#2799). The reopen-failure rotation test
removes its own live-log tree mid-test (via rmrf) to force an ENOTDIR reopen; after
the guard clears logPath, the post-clobber log() hit the !logPath early-return
instead of attempting the reopen, so the degradation [warn] never fired (0 !== 1)
and the test went red on every platform.

Re-point the logger at the now-invalid path after the clobber so the reopen attempt
still happens and degrades with exactly one [warn]. No production code or behavior
changes; the rmrf guard and its no-resurrection pin are unchanged.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

PR #2808 的 CI 红了,我定位到一个真实回归并已修复,往 PR 分支补了一个 additive commit。

先分清两条红腿——性质完全不同:

① 真实回归(本 PR 引入)— 已修
tests/logger-rotation.test.ts 的 reopen failure: degrades to stderr-only with one [warn], no crash 挂:exactly one [warn] per degradation episode → 0 !== 1。

  • 是什么问题:该测试用「删掉自己正在写的日志树、再在原目录位置放一个普通文件」来强制下一次 reopen 命中 ENOTDIR,从而验证 logger 降级时恰好打一条 [warn]。
  • 什么原因导致:本 PR 给共享 rmrf() 加的守卫(Windows CI: storage-engine-proofs 套件清理 temp 目录遇 EBUSY/EPERM,windows test leg 随机红 #2799)删除前先调 _releaseLogFileForTest()——它不只释放句柄,还清空了 logPath。测试中途那次 rmrf(dir) 触发守卫清路径后,紧接着的 log() 走了 if (!logPath) return undefined 早退,压根没发起 reopen,降级 [warn] 永远打不出来 → 0≠1。我在本地用未改代码复现过(同样 0!==1)。
  • 怎么修复的:clobber 之后加一行 configureLogger(file),把 logger 重新指回那个已被破坏的路径,让 reopen 尝试重新发生并降级出一条 [warn]。exactly-one 语义不受影响(release 已重置 reopenWarned,后续 log 不重复告警)。
  • 为什么不是把守卫直接从 rmrf() 里摘掉:摘掉会重新引入 Windows EPERM——这个测试本身就是「持着日志句柄中途删自己的树」的场景,另有约 19 个 in-process server 套件同构。守卫方向是对的,错的只是这一个测试对旧行为(rmrf 后句柄仍在)的隐性依赖。因此保留守卫 + no-resurrection pin(tests/tmp-rm.test.ts 第 7 条仍绿),只改这一处测试。

② 环境抖动(与本 PR 无关)
tests/dump-mask.test.ts 的 ACP_DUMP_BODY=1 works without --debug (#276) 报 502 !== 200,耗时 35.9s。CI 日志是 [dump] SSE stream dump failed … ENOSPC: no space left on device + upstream connect-timeout … UND_ERR_CONNECT_TIMEOUT——runner 磁盘在并行重负载下写满。这条靠 CI 重跑 即可清除,不动 dump-mask.test.ts。

回归审查(逐条既有行为):本次改动只落在 tests/logger-rotation.test.ts 一个文件(+3/-1),零生产代码、零 wire/message-shape、零 config schema/persistence、零 kernel。logger 的降级行为(外部破坏→恰好一条 warn)保持不变;no-resurrection 保证不变。本地定向验证 logger-rotation.test.ts 3/3、tmp-rm.test.ts 7/7 全绿。

八问结论:

  1. 是什么问题:ubuntu-22/24 test job 两条红腿——一条真回归(rotation 降级测试),一条 ENOSPC 抖动。
  2. 根因:共享 rmrf() 守卫清 logPath 后,测试后续 log 早退、不再发起 reopen。
  3. 影响范围/严重度:① 阻断合并(该测试在任何平台都红);② 瞬时环境性,自清。
  4. 修法:clobber 后重指向失效路径,恢复 reopen 尝试。
  5. 回归:无——仅测试文件,生产/wire/config/kernel 均未触碰;no-resurrection 不变。
  6. 缓存:不涉及(未碰缓存/重试/游标/去重键)。
  7. 设计方向:不涉及(纯测试修正,守卫设计原样保留)。
  8. 新配置:不涉及(无新增 env/config/默认值)。

收工判定:真实回归已修并推到 PR 分支(commit 138829426,叠在原 head 974ba6437 上,fast-forward、无 force),CI 会重跑;②那条 ENOSPC 若再冒头属 runner 资源问题,建议直接 re-run 而非改码。矩阵转绿即可合。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows CI: storage-engine-proofs 套件清理 temp 目录遇 EBUSY/EPERM,windows test leg 随机红

1 participant