Skip to content

fix: flush durable state before spawning self-restart replacement (#1724) - #1742

Merged
ranxianglei merged 1 commit into
masterfrom
2026-09-30_restart-flush-before-spawn
Sep 30, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-09-30_restart-flush-before-spawn

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Part of #1724 (batch 2 — root-cause fix for the self-restart path). Not closing. Batch 1 (data-layer safety nets) is #1737; web-scoping + state-partitioning follow separately.

What & why

On an auto-update self-restart (--auto-restart-on-update, #811), performSelfRestart previously did: drain → spawn the replacement → wait for it to bind the port → flush this process's durable state in finish(). The replacement hydrates the shared store as soon as it boots, so during that readiness wait it read a stale snapshot, and the old process's final flush was never read back — exactly the lost-update / prefix-cache-busting race in the issue's timeline (new process listening before the old one flushed).

This PR flips the order: once the drain has confirmed state is frozen (listener closed, in-flight 0), it now flushes durable state to disk before spawning the replacement. The port is already released post-close, so this adds no handover delay; every abort path is unchanged (the flush is an idempotent write, in-memory state untouched).

Second gap fixed while here: defaultFinish() omitted flushPrefixAffinity(). Its debounced persist timer is unref()'d, so any affinity writes still inside the debounce window were silently dropped on self-restart. Added — aligning the self-restart durable-state set with what the normal shutdown path persists.

Behavior change (flagged)

  • Durable-state flush now precedes the replacement spawn (was: only in finish, post-spawn). Opt-in path only (--auto-restart-on-update / ACP_AUTO_RESTART_ON_UPDATE / config autoRestartOnUpdate); default-OFF behavior is unchanged.
  • No wire/message-shape change. No config-surface change (adds one injectable test seam preSpawnFlush; reuses existing spawnImpl/finish seam pattern). No acp-kernel pin change.

Scope note (honest boundary)

This fixes the one restart path bili fully owns (#811 self-restart). The issue's observed incident logged parent-gone, i.e. the launcher/watchdog route, where hosts like dsh drive their bundled proxy's lifecycle through their own channel (#991) — that ordering is host-controlled and cannot be fully fixed from inside bili. Those paths are mitigated by the data-layer guards in #1737 (prefix-affinity union) plus the existing #405 session counter guard; a universal cross-host drain barrier needs host coordination and is left as a separate design decision.

Pre-flight

  • typecheck ✓ · full suite 3340 tests / 0 fail (17 e2e-skipped) · build ✓ — run on current master 27c8774a.
  • Regression test: performSelfRestart: durable state flush runs before the replacement spawns (#1724) asserts the pre-spawn flush strictly precedes the spawn (fails on the pre-fix code, where the only flush ran in finish after the child booted).

)

performSelfRestart (#811 opt-in self-restart) previously spawned the replacement and waited for it to bind the port (i.e. hydrate the shared store) BEFORE flushing this process's durable state in finish() — so the new process hydrated a stale snapshot and the old process's final flush was never read back: the lost-update / prefix-cache-busting race observed across an auto-update restart.

Flush conversations + sessions + prefix-affinity to disk after the drain (state frozen: listener closed, in-flight 0) and before spawning the replacement; the port is already released so this adds no handover delay and abort paths are unchanged. Also add the missing flushPrefixAffinity() to defaultFinish (its debounced timer is unref'd, so pending affinity writes were silently dropped on self-restart), aligning the self-restart durable-state set with the normal shutdown path.

Behavior change: durable-state flush now precedes the replacement spawn (was: only in finish, post-spawn). Opt-in path only (--auto-restart-on-update); default-OFF behavior unchanged. No wire/message-shape or config-surface change.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-30_restart-flush-before-spawn (36893d3)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1690-pr.1742.48bcdb9

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1742.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到 — 这是 agent 提交的 [PR](ework-agent-pr 标记),我不会重做,按流程走 review:拉分支核对代码、确认 CI、给出结论。先审原 issue #1724 的方向与根因定位,再走回归审查。稍后回帖。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 完成,结论:可以合并。以下是按审查优先级走的完整记录。

核对方式:拉取分支 2026-09-30_restart-flush-before-spawn @ 36893d38(base = 当前 master 27c8774a,无 stale-base;单 commit、+60/-0、2 文件)逐行审代码 + 本地定向验证,未重做工作。CI 在 head commit 上 12/12 全绿(ubuntu/windows × node 22/24 全量测试矩阵、codex/opencode/pi/dsh 四条真实 lane e2e、hermetic registry e2e、version-guard、build-artifact)。

① 原 issue 方向关:#1724 的根因是「新进程 hydrate 早于旧进程最终 flush」的重启竞态。本 PR 修的是 bili 完全自控的 #811 自重启路径——把 flush 提前到 spawn 之前,消除该路径上的乱序,是机制级修复(排序),不是症状掩盖。范围边界诚实:issue 观测事故走 parent-gone(host 驱动)路线,bili 内部修不到,PR 明确不假装堵上(靠 #1737 数据层守卫缓解),与 #1724 floor 3 定的三轨方案一致。方向成立。

② 回归审查(既有行为清单)——diff 只碰 src/restart.ts,逐项判定:

  • sanity 门 / drain 循环 / marker+cooldown / spawn / readiness 轮询:全部未动,flush 插在 drain 确认(closed && in-flight=0)之后、marker/spawn 之前。✅ 保持
  • abort 路径(no-entry / spawn 异常 / readiness 超时→resumeListening):flush 发生在其中三条之前,但它是当前内存态的幂等写穿,不动内存;老进程恢复服务后照常继续。最坏情形 = 修复前行为(stale snapshot),不更差。✅ 保持
  • flush 失败处理:catch + warn + 继续,与该文件既有宽容风格一致(writeLastRestart best-effort、writeSnapshot 内部 catch)。失败时退化为修复前语义,不会卡死更新。✅ 合理
  • defaultFinish 补 flushPrefixAffinity():已核对正常 shutdown 路径(server.ts finishShutdown = affinity + 清实例文件 + unregister + exit,shutdown 里 conversations + sessions)——自重启退出与正常退出的持久化集合现在精确对齐。这是修复(defaultFinish 原先漏掉 affinity,其 debounce timer 是 unref 的,窗口内写入静默丢),已披露。✅ 有意变更,已标注
  • 双 flush(pre-spawn + finish):幂等,finish 那次捕获 readiness 等待期的任何变化(实际不会有),防御性冗余无害。✅
  • 时序:flush 给「drain 完→子进程 bind」的拒连窗口加了本地 fs 写时间(通常 ms 级);该窗口本就横跨 spawn+boot(秒级),端口已在 close 时释放。PR 说「不增加 handover 延迟」对端口成立,实测影响可忽略。✅
  • 默认 OFF 路径、wire/message-shape、config surface(仅加内部可选 seam preSpawnFlush,沿用既有 spawnImpl/finish seam 模式)、acp-kernel pin、version:均未动。✅

③ 代码关:flushPrefixAffinity() 是同步实现(清 timer + 立即 atomic tmp+rename 写,affinity-persist.ts:53-59),defaultPreSpawnFlush 里不 await 正确;seam 类型注释带 #1724 出处;用 log() 非 console.error;无 as-any/ts-ignore。无问题。

④ 测试有效性:新测试不是假测试——我做了 teeth 验证:把 src/restart.ts 换回修复前版本、保留新测试跑,结果 test 7 fail(events=["spawn"],pre-flush 从未发生);换回 PR 版本 8/8 全过。断言 deepEqual(events, ["pre-flush","spawn"]) 是真牙,bug 回归时抓得住。小注:测试钉住的是 seam 顺序而非 defaultPreSpawnFlush 的具体内容(它调哪三个 flush)——与本文件既有的 seam 注入测试风格一致,default 是 3 行镜像 shutdown 的函数,目检可证,可接受,不算缺口。本地另跑:typecheck ✓ · auto-restart 全套 8/8 ✓ · build ✓。

⑤ diff 整洁度:2 文件 +60/-0,每一行都和本 PR 目的相关,无跑题/空行/生成物。干净。

自动合并门核对:纯 fix、单模块、有回归测试且已验证有牙、rebased head 上绿、不碰 config schema/persistence/wire/cross-repo/update.ts/kernel pin —— 各项满足;合并动作本身留给你(系统规则禁止 agent merge)。

一句话中文摘要:修复了自动更新自重启(#811)路径上「新进程先 hydrate、旧进程后 flush」的状态丢失竞态——把 durable state flush 提前到 spawn 替换进程之前并补上 defaultFinish 漏掉的 prefix-affinity 持久化,仅影响 opt-in 自重启路径、默认行为不变,回归测试已验证对修复前代码会挂,CI 全绿,可以合并:#1742

@ranxianglei
ranxianglei merged commit 36b7f2a into master Sep 30, 2026
12 checks passed
AEmbers pushed a commit to AEmbers/billion-context that referenced this pull request Sep 30, 2026
…ianglei#1724)

Record in TECHNICAL-NOTES (en/zh-CN) that all instances share one state dir whose control plane is lane-aware but whose data plane is not partitioned — no per-session owner, no per-lane isolation. State the two consequences (cross-instance session visibility in every instance's web UI; the restart drain race and its ranxianglei#1737/ranxianglei#1742/ranxianglei#405 mitigations) and the current security posture: only the admin loopback gate + filesystem permissions, i.e. no per-session authorization. Sufficient single-user, insufficient multi-user (must partition). Issue direction ranxianglei#5. Docs-only.
@ranxianglei
ranxianglei deleted the 2026-09-30_restart-flush-before-spawn branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant