Skip to content

[PR] served-markers: graceful dispose records stoppedAt — restart flows no longer raise split-brain alarms (#3923) - #4989

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-27_split-brain-graceful-stop
Oct 28, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-27_split-brain-graceful-stop

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Problem

During the v0.1.209 release-gate run (lane A), the gate's intentional proxy restart logged a false positive:

[split-brain] session pfa-… committed by BOTH this instance (pid 160827, …) and another live instance (pid 154838, …) within 10m

Mechanics: the old proxy was SIGTERM'd; its dispose() writes a final marker (intentional — peers should attribute its last commits). The successor rebinds ~3.5 s later and scans peer markers: isPidAlive(154838) is false, but the marker mtime is inside PEER_FILE_FRESH_MS (30 s), so the dead predecessor is counted "live" and an error is raised.

The freshness fallback exists for platforms where the pid check cannot see the peer (cross-namespace) — it should only rescue crashed peers whose death we cannot prove, not a predecessor that told us it stopped.

Fix (src/served-markers.ts)

  • WriterMarkerFile gains optional stoppedAt?: number — set only by a graceful dispose() on its final marker write.
  • findContestedWriters skips any marker carrying stoppedAt, unconditionally — before the liveness test. A recorded graceful stop is positively-known death: never a concurrent writer, even while the file is fresh or the pid has since been reused (which also removes a latent pid-reuse false positive).
  • Crash path unchanged: no stoppedAt → existing freshness fallback intact.

Tests (tests/served-markers.test.ts)

  • new: graceful dispose records stoppedAt; successor committing the same session seconds later logs nothing (the exact gate-restart sequence).
  • new: stoppedAt skips even when the peer pid is alive (stop outranks liveness).
  • pre-existing freshness test pins the crash path (hand-written marker without stoppedAt).

All 11 served-markers tests + 5 split-brain-surface tests pass; tsc --noEmit clean.

Notes

… longer raise split-brain alarms (#3923)

A SIGTERM'd proxy's final marker write (dispose flush) lands with a fresh
mtime seconds before its successor rebinds; findContestedWriters' 30s
freshness fallback then counts the dead predecessor as live and logs a
false [split-brain] error (observed in release-gate lane A proxy restart).

Fix: dispose() stamps stoppedAt on the final marker. Peers skip markers
carrying stoppedAt unconditionally — a recorded graceful stop is
positively-known death and must not contest, even while the file is fresh
or the pid has been reused. The freshness fallback is unchanged for
crashed peers (no stoppedAt), preserving cross-platform pid invisibility
coverage. Two regression tests pin both paths.
@ranxianglei ranxianglei changed the title served-markers: graceful dispose records stoppedAt — restart flows no longer raise split-brain alarms (#3923) [PR] served-markers: graceful dispose records stoppedAt — restart flows no longer raise split-brain alarms (#3923) Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-27_split-brain-graceful-stop (523cbdd)

Latest build — install this exact version

npm install -g billion-context@0.0.6575-pr.4989.fbe45b1

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
523cbdd billion-context@0.0.6575-pr.4989.fbe45b1 build
1acf5fb billion-context@0.0.6551-pr.4989.5c7d788 build
b5ec3bf billion-context@0.0.6520-pr.4989.f808619 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4989.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 无既有 issue/PR 针对「重启流程对已优雅停机的前任误报 split-brain」这一缺陷。同家族相关项:#3923(本功能的源头 issue,已关闭)、#4626/#4630(同一 split-brain 呈现面,但症状与机制都不同——那边是「横幅在 peer 消失后永久残留」,修的是 stale storage.splitBrain 不清除;本 PR 是「重启时误报新告警」,修的是 freshness fallback 把已知死亡的前任当活实例)。两者互补:#4630 清掉已浮现的旧横幅,本 PR 让重启场景不再产生假横幅。

标签: bug + scope/storage + p3(写者标记文件/会话状态持久化面的观测性误报;无功能损伤、纯日志+面板噪音)。

审核结论(八问)

  1. 是什么问题:双实例共用 sessions dir 的拓扑下,故意的代理重启(SIGTERM 杀旧进程 → ~3.5s 后重绑)会打出 [split-brain] … committed by BOTH … error——release-gate lane A 实测到的假阳性。
  2. 什么原因导致的:freshness fallback(PEER_FILE_FRESH_MS = 30s,为跨命名空间看不到 pid 的平台兜底)无法区分两种死法:「崩溃、死亡无法证明」(该救)vs「优雅停机、死亡是正知识」(不该救)。graceful dispose() 写最终标记时没有留下任何「我已停止」信号,于是死后 30s 内仍被计为 live。
  3. 影响范围与严重程度:仅观测面(error 日志行 + web 面板 splitBrain contest + onContest seam);模块设计即纯 observability,无路由/锁/服务行为变化。触发条件=共享 sessions dir 双实例 + graceful 重启流。严重度 p3:日志噪音/误报警,无功能损坏;另消除一个潜在 pid 复用假阳性(前任 pid 在 30s 窗口内被其他进程复用)。
  4. 怎么修复的:WriterMarkerFile 增加可选 stoppedAt?: number,只由 dispose() 的最终 marker 写入时置位(全仓唯一调用点在 src/server.ts finishShutdown(),SIGTERM drain 路径,故存活实例不可能带上该字段);findContestedWriters 在 liveness 测试之前无条件跳过携带 stoppedAt 的 marker。崩溃路径零改动:无 stoppedAt → 原 freshness fallback 原样保留。
  5. 是否有回归问题:审过的既有行为逐项——① 崩溃兜底(dead-pid + fresh mtime 仍计 live)由既有测试 dead-pid peer counts while its marker is fresh… 钉住(手写 marker 无 stoppedAt),未变;② 竞争窗 10min / flush 节流 5s / reap 1h 全部未动;③ reapStaleWriterMarkers 照常回收优雅停机的 marker(pid dead + mtime>1h),无文件泄漏;④ verdict TTL / ack 逻辑未触碰。行为变更披露(Deliverables feat: OpenAI Responses API support (/v1/responses) #4):旧行为=已知死亡的前任在重启后 30s 内仍可能触发 [split-brain] error + 面板横幅 → 新行为=永不触发;这正是修复目标,CHANGELOG 已带条目。
  6. 是否破坏缓存:不涉及(前缀缓存/重试/游标/去重键均不受影响;marker 文件不是缓存)。
  7. 是否改了设计方向:没有——仍是纯 observability;marker schema 只做加性演进,双向混版本兼容:老 build 读新 marker 忽略未知字段(行为与从前完全一致),新 build 读老 marker 走原逻辑。
  8. 是否引入了新的配置:无(stoppedAt 是 marker 文件内部字段,非 env/config/CLI 面)。

验证

判定:可合并(待 CI 全绿收口;代码与回归审查均通过,无需改动)。

@ranxianglei ranxianglei added bug Something isn't working p3 scope/storage labels Oct 27, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) Merge-guard sync done: merged origin/master (9b6ec2345) into the branch as 1acf5fbbc (clean, no conflicts — zero file overlap), pushed to 2026-10-27_split-brain-graceful-stop; all checks success/skipped on the new head, mergeable state cleared from behind.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

收口更新(审核结论见上一条评论,此处只报最终状态):

  • 分支已同步:merge-guard 已把 origin/master(9b6ec2345)合入 PR 分支(1acf5fbbc,零文件冲突),「behind」状态清除。
  • CI 全绿:新 head 1acf5fbb 上 30 项检查全部完成——success 18 / skipped 12 / failed 0,无 pending。
  • 判定维持:可合并。代码审查、回归清点、本地定向测试(16/16 + tsc clean)、CI 全量四项均已通过,无需改动;按规则合并由人工执行。

@ranxianglei
ranxianglei merged commit 09e33bf into master Oct 28, 2026
21 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.891-master.09e33bf (rolling master tag).

npm install -g billion-context@0.0.891-master.09e33bf  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant