Skip to content

[PR] fix(storage): stop a running background quick_check on exit instead of waiting for it - #7260

Open
mariopasquali wants to merge 3 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-18_quickcheck-shutdown
Open

mariopasquali wants to merge 3 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-18_quickcheck-shutdown

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

Closes #6987.

Cause. A backtrace of the hung process shows the main thread in process.exit() → Environment::Exit → stop_sub_worker_contexts → Worker::JoinThread, and the integrity worker inside sqlite3BtreeIntegrityCheck (one sqlite3_step). V8 cannot terminate a thread inside a native call, and node:sqlite has no sqlite3_interrupt, so exit waits for the whole scan (about 45 min on 10 GB). unref() (#6989) does not help: it only affects whether the event loop stays alive, not the join in process.exit(). I tested #6989 on the same store and it hangs the same way.

Fix. On Node, the scan runs in a child process (process.execPath -e) instead of a worker thread. An exit handler kills it with SIGKILL; it holds only a read-only connection. The verdict, the cache write, the alarms and the #5940 stderr filter work as before. Bun keeps the worker thread, because the execPath of a compiled host is the host binary.

Tests

  • New tests/issue6987-quickcheck-shutdown.test.ts: holds the scan inside one endless sqlite3_step (a recursive CTE through a test-only SQL seam), calls process.exit(0), and requires a prompt exit and the scan process to be gone. With the worker thread it hangs until the test kills it after 60 s.
  • 10.7 GB store, 4 CPUs, 5 GiB RAM, with [PR] perf(storage): start a large SQLite store without rewriting every session row #6850 added so the relay can start: SIGTERM 20 s after the relay was ready, with the scan running → exit code 0 after 5.3 s, scan process gone. Before: SIGKILL ~300 s after SIGTERM.
  • Integrity tests 33/33; full suite 12,620 tests, 0 failures, 75 skipped; typecheck, unused-exports, architecture-docs and dist-external checks clean.

mariopasquali and others added 2 commits November 18, 2026 19:21
…f waiting for it (ranxianglei#6987)

The background quick_check ran in a worker thread. The scan is one synchronous sqlite3_step that cannot be interrupted (node:sqlite has no sqlite3_interrupt), and process.exit() joins every worker thread before the process ends. A relay stopped during the scan of a large store (about 45 min for 10 GB) therefore stayed in exit until the supervisor killed it.

On Node the scan now runs in a child process that is killed when the relay exits. Bun keeps the worker thread, because a compiled host's execPath is not a runtime that can run the scan source.

Closes ranxianglei#6987
…LIVE_FILES

The new tests/issue6987-quickcheck-shutdown.test.ts spawnSyncs a bare node child, which trips the SPAWN_RE gate in scripts/run-ci-tests.mjs. It was never added to LIVE_FILES in scripts/live-files.mjs, so the tripwire (which scans the full file set in every shard) failed all 11 test shards before a single test ran.

Concurrency reviewed: the spawn is synchronous (blocks the shard runner), the child runs under a per-run mkdtemp dir, binds no port and does no network I/O — no zone band needed, entry is for the tripwire alone, like ranxianglei#3890/ranxianglei#5114/ranxianglei#5940.
@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Nov 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 This PR is the actual fix for #6987 (shutdown hang during the background quick_check). It is distinct from #6989 (the unref() attempt) — the author tested #6989 on the same store and confirmed it does NOT fix the hang, because unref() only affects event-loop liveness, not the Worker::JoinThread inside process.exit(). No other open PR targets this.

Review verdict: mergeable (CI green). I read the full diff, verified the root cause against the code, ran the targeted suites locally, and confirmed the full CI is green on the head.

Root cause (verified). The SIGTERM path ends in process.exit(0) (src/server.ts:1770, after the session flush). process.exit() → Environment::Exit → stop_sub_worker_contexts → Worker::JoinThread, and the integrity worker is parked inside one native sqlite3_step (the whole quick_check scan). V8 cannot preempt a thread inside a native call, and node:sqlite exposes no sqlite3_interrupt, so exit blocks for the entire scan (~45 min on 10 GB) until the supervisor's SIGKILL. I confirmed a worker stuck in a JS loop does NOT hang exit — the pathology is native-call-specific, matching the reported backtrace.

Fix (correct). On Node the scan now runs in a child process (process.execPath -e) instead of a worker thread; a one-time process.on("exit") hook SIGKILLs it (it holds only a read-only connection). Bun keeps the worker thread (a compiled host's execPath is the host binary, which can't run the scan source). The scan logic is factored into a shared CHECK_CORE (runCheck(path, sql)); the worker and child carriers wrap it with postMessage / stdout-JSON respectively. --experimental-sqlite is forwarded via execArgv (and inherited via env), so it works on both flag-requiring and flag-free Node.

Verification.

  • New test tests/issue6987-quickcheck-shutdown.test.ts: holds the scan in one endless sqlite3_step (a recursive CTE via a test-only SQL seam), calls process.exit(0), requires a prompt exit and the scan process gone. Passes locally (2.3 s); with the old worker-thread path it hangs until the 60 s timeout.
  • All 42 integrity/storage tests pass; typecheck, dist-external, and dist-import-annotations clean.
  • CI was red, now green. The only failure was the new test not being registered in LIVE_FILES (scripts/live-files.mjs) — the tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248 tripwire failed all 11 test shards before a single test ran. That registration is now in place (commit 4f9c7091ddf); I confirmed the tripwire passes and the full CI on head 4f9c7091ddf is 48 checks / 0 failures.
  • Labeled the PR bug / p2 / scope/storage (judged on its own diff: a storage-layer shutdown fix, not a data-loss or cache-leak class).

八问(中文):

  1. 是什么问题: 大存储(10 GB)后台 quick_check 扫描运行时,relay 收 SIGTERM 后 flush 完成却卡死不退出,直到 watchdog SIGKILL(~5 分钟)。
  2. 什么原因导致的: 扫描跑在 worker thread 里、卡在单个原生 sqlite3_step;V8 无法抢占原生调用中的线程,而 process.exit() 会先 join 所有 worker thread;node:sqlite 无 sqlite3_interrupt 可打断,于是 exit 阻塞在整次扫描上。[PR] fix: unref in-flight integrity check workers on shutdown #6989 的 unref() 只影响事件循环存活、不影响 exit 的 join,救不了。
  3. 影响范围与严重程度: 所有用 node:sqlite 跑后台 quick_check 的大存储(超 sync 阈值)relay;SIGTERM/优雅退出时进程不退出、需外部 SIGKILL。严重度 p2——运行时无影响,仅优雅停机受损。
  4. 怎么修复的: Node 上把扫描从 worker thread 改为子进程(process.execPath -e),注册一次性 process.on("exit") 钩子在退出时 SIGKILL 扫描子进程(只持只读连接);Bun 保持 worker thread(编译宿主 execPath 是宿主二进制、跑不了扫描源码);扫描逻辑抽成共享 CHECK_CORE,worker/child 两载体分别包 postMessage / stdout-JSON。
  5. 是否有回归问题: 审过既有行为——判定/缓存/告警/stderr 过滤([Bug][Pi native] SQLite ExperimentalWarning from background integrity worker leaks into Pi TUI #5940)语义不变(仅载体从 worker 换 child);finish 现用 settled 标志保证幂等(原靠 inflight.has 判重,语义等价、更干净);Bun 路径不变(仍 worker、仍不捕 stderr);--experimental-sqlite 经 execArgv 转发 + env 继承,Node 22.23+/24 无需 flag 亦工作。无回归。
  6. 是否破坏缓存: 不涉及——判定缓存(writeIntegrityCache)逻辑未动,仍仅 ok 时写。
  7. 是否改了设计方向: 不涉及架构转向——仍是"后台只读连接跑 quick_check、结果仅作告警",仅执行载体从 worker thread 换 child process(针对 Node 的 exit-join 病理)。
  8. 是否引入了新的配置: 无新 env/config/默认值;新增两个测试 seam(setIntegrityCheckSqlForTests / integrityCheckPidForTests)仅测试用、非配置面。

Conclusion: mergeable. Root cause correctly diagnosed, fix minimal and correct for both runtimes, regression surface reviewed clean, CI green on the head. I can't merge (human-only) — please merge when ready: #7260

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: anthropic/claude)

CI fixed — all 11 red test shards are green now. New head: 4f9c7091ddf (one additive commit on top of your 9b39579870).

What was wrong. The new tests/issue6987-quickcheck-shutdown.test.ts uses spawnSync, which trips the SPAWN_RE gate in scripts/run-ci-tests.mjs — but the file was never registered in LIVE_FILES (scripts/live-files.mjs). That tripwire scans the full test-file set in every shard before running a single test, so all 11 test shards died in ~18s with process-spawning test files missing from LIVE_FILES. It was a registration omission, not a bug in your test or the storage fix.

The fix. One line (+ comment) appended to LIVE_FILES — append-only so no zone-slot renumbering. Concurrency reviewed per #1248: the spawn is synchronous (spawnSync blocks the shard runner), runs under a per-run mkdtemp dir, binds no port, does no network I/O → no zone port band needed; the entry is for the tripwire alone, exactly like the sibling #3890/#5114/#5940 entries.

Verified before pushing. Tripwire logic passes locally (0 unregistered spawners); the new test passes (2.3s, well under its 60s budget); all 22 integrity-related tests pass. Post-push CI on 4f9c7091ddf: 27 success + 15 skipped, 0 failures (a transient duplicate-trigger wave cancelled mid-flight and re-ran green — final state is green).


八问

  1. 是什么问题:PR 新增测试文件 tests/issue6987-quickcheck-shutdown.test.ts 用了 spawnSync,但漏了在 scripts/live-files.mjs 的 LIVE_FILES 表登记,CI 的 SPAWN_RE 绊线在每个 shard 跑任何测试前全量扫描,直接让 11 个 test shard 在 ~18s 内全红。
  2. 什么原因导致的:作者加了会 spawn 进程的测试套件,但没按仓库规矩(tests/claude-native.test.ts linux live-E2E subtests fail/hang in shared multi-agent sandboxes #1248)登记;绊线的设计就是"未登记的 spawner 一律红"。根因是登记缺失,不是测试逻辑或 storage 修复本身有问题。
  3. 影响范围与严重程度:仅卡本 PR 的 CI(11 个 test shard 全红,PR 无法合入);不影响任何运行时行为,非产品缺陷。
  4. 怎么修复的:在 LIVE_FILES 末尾追加该文件(append-only,不打乱 zone slot 编号),附并发评审注释——同步 spawnSync、每次运行独立 mkdtemp、不绑端口、无网络 I/O,故无需 zone 端口带,纯为绊线登记(同 [Bug] Windows: Proxy falsely detects parent-gone and self-terminates on every forwarded request (same live PID triggered | [Bug] Windows:代理每转发一次请求就误判 parent-gone 自杀(同一存活 PID 在 2.5h 内触发 12 次) #3890/Running tests locally writes test sessions into the real ~/.local/billion-context (polluting the production index.db | 本地跑测试会把测试会话写进真实的 ~/.local/billion-context(污染生产 index.db 和 sessions 目录) #5114/[Bug][Pi native] SQLite ExperimentalWarning from background integrity worker leaks into Pi TUI #5940)。
  5. 是否有回归问题:无。只动 CI 登记表(追加一行+注释),不碰任何运行时/wire/存储逻辑;本地已验证绊线通过、新测试通过、22 个 integrity 相关测试全绿。
  6. 是否破坏缓存:不涉及(无缓存/重试/游标/去重逻辑变更)。
  7. 是否改了设计方向:不涉及(纯 CI 登记,无架构/数据流变化)。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

判定:已修复并推送到 PR 分支(最终 head 4f9c7091ddf,无 CI-hold 标记),CI 全绿,可合入。

…olution for ranxianglei#7260)

Conflict in scripts/live-files.mjs: both sides appended a tripwire-only LIVE_FILES entry at the list tail (master: tests/storage-integrity-cache.test.ts from ranxianglei#7156; this branch: tests/issue6987-quickcheck-shutdown.test.ts from ranxianglei#6987). Resolved by union — master's entry keeps its index (stable zone slot), this branch's entry takes the new tail slot. src/storage/integrity-check.ts auto-merged: the ranxianglei#6987 worker-to-child-process scan refactor and master's ranxianglei#7156 foreign-writer ok-cache gating coexist without overlap.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: anthropic/claude)

Merge conflict resolved — the PR is back to a clean, mergeable state. New head: 41cd3e1d28b (a merge of origin/master into the branch; your commits are untouched, no rebase/force-push).

The conflict. Master advanced past your branch point and merged #7156/#7173, which touched the same two files this PR touches. Only one real conflict: scripts/live-files.mjs — both sides had appended a tripwire-only LIVE_FILES entry at the list tail (master: tests/storage-integrity-cache.test.ts from #7156; this branch: tests/issue6987-quickcheck-shutdown.test.ts from #6987). Resolved by union: master's entry keeps its index (its zone slot is unchanged), this branch's entry takes the new tail slot. src/storage/integrity-check.ts auto-merged — the #6987 worker→child-process scan refactor and master's #7156 foreign-writer ok-cache gating live in different regions and don't overlap; I verified the merged signature, the single caller (src/storage/store.ts:771), and the verdict path by hand.

Verified before pushing. typecheck clean, build clean, SPAWN_RE tripwire passes (0 unregistered spawners), and both conflict-adjacent suites pass (14/14: the #6987 quickcheck-shutdown test + master's new #7156 storage-integrity-cache test).

CI on 41cd3e1d28b. All 14 required test shards green; every other check green/skipped (36 success + 12 skipped, 0 failures).

Note on mergeable_state: blocked. This is pre-existing and not caused by this change: branch protection requires test (windows-latest, 24, 1/2/3), but the core CI matrix (default for normal PRs) deliberately runs windows-24 legs as FULL-only — so those 3 required contexts never produce check runs on a core-scope PR. The three PRs merged into master yesterday (#7412/#7404/#7400) were merged in exactly this same state (verified: their heads also have zero windows-24 check runs). I did not add ci:full — that would override the core-as-default cost design and deviate from how these PRs are actually being merged. If the windows-24 requirement is intentional, the core matrix or the required-contexts list needs reconciling; if it's not, the required contexts should drop windows-24. Flagging for your call, not blocking here.


八问

  1. 是什么问题:PR 分支基于旧 master,master 前进后(合入 [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156/[PR] fix: #7156 storage-corruption fallout — webSummaryRows json_valid guard, .chk close-time gate, first-write corruption classification #7173 等)与 base 分支产生冲突(merge state dirty),PR 无法合并。
  2. 什么原因导致的:分叉点后 master 修改了本 PR 也触碰的 scripts/live-files.mjs 和 src/storage/integrity-check.ts;两边都在 live-files.mjs 列表末尾追加条目,在同一位置相撞形成冲突。
  3. 影响范围与严重程度:仅影响本 PR 的可合并性(dirty 期间无法合并);不影响任何运行时行为。
  4. 怎么修复的:把 origin/master 合并进 PR 分支。唯一冲突 live-files.mjs 按并集解决——master 的条目保留原索引(zone slot 不变),本分支条目追加到新的末尾位置;integrity-check.ts 自动合并(Stop hangs after the session flush while the startup background quick_check runs on a large store #6987 worker→子进程重构与 master [Bug][storage] Takeover spawns before reclaim: old and new sidecars simultaneously hold index.db, causing corruption; em | [Bug][storage] 接管时先 spawn 后回收:新旧侧车同时持有 index.db 引发损坏;空 payload_json 又让历史会话静默变 0 #7156 foreign-writer 缓存门控位于不同区域、不重叠),合并后的签名/调用点/裁决逻辑已逐一核对。
  5. 是否有回归问题:无。本地 typecheck/build 干净、SPAWN_RE 绊线通过(0 未登记 spawner)、两个冲突相邻测试套件全过(14/14);新 head 上 14 个必需 test shard 全绿,其余 check 全绿/跳过(36 success + 12 skipped,0 失败)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(纯冲突解决,无架构/数据流变化)。
  8. 是否引入了新的配置:不涉及。

判定:冲突已解决、CI 全绿,PR 恢复可合并状态(与昨日已合入的 #7412/#7404/#7400 同一状态),可合入。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop hangs after the session flush while the startup background quick_check runs on a large store

2 participants