Repository navigation
[PR] fix: stale split-brain notice clears itself once the peer stops contesting (#4626) - #4630
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6237-pr.4630.e8c58b5This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4630.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 — open+closed 检索 "split-brain stale notice clears" / "4626" / "split-brain" 均无并行条目;仅相关的已关闭 #3923 是该功能(split-brain 检测本身)的原始 issue,本 PR 修的是它的「判决冻结」缺陷,机制不同。 独立审核完成(未照抄 issue/PR 侧结论;全 diff 5 文件逐段读完 + 机制源码交叉验证 + 本地定向复跑)。 本地验证(PR head c962f32)
回归清点(②,触路径既有行为逐项)
高危模式逐项过:无常量复制(复用同一 结构化八问
测试有效性(④):G3 双向都有牙(新鲜墓碑必须保留=负向;dead+stale 必须清=正向;且断言 info 日志带全 id);G3-negative 钉住 legacy 与 live-peer 两个不触发面;web e2e 用真实 marker 证据走完整 HTTP 链路证明无 ack 自清(fixture pid 从可能存活的 4242 换成超出 pid_max 的 999_999_999,确定性更好)。这些测试在原 bug 下会红,不是恒真。 CI 分账:除 判定:代码可合。 mergeable_state:blocked 纯粹来自 master 红风暴的 required-checks 死锁(等 #4606/owner 决策),本分支无需任何改动。master 转绿后 retrigger CI 即可合并。 |
|
Thanks for turning this around. I checked Two things I'd want settled before release:
Also: I'm on 0.1.208, which doesn't contain the seam yet — is this slated for the next release? Happy to re-run my restart-handover repro (supervisor restart → fresh tombstone → notice persists) against the prerelease or the released build and report back with the log lines. One nit on tests: a case pinning whichever fail-safe direction you pick (unreadable writers dir) would keep it from regressing later. |
(ai: vllm-qwen/qwen3.8-27b) CI auto-fix lane (attempt 1/2): windows-latest 22/3 已修,push 查重:不重复——此红不是本 PR diff 引入,是继承自 master(master bb7c249 同一子测试同样失败)。 八问结论
备注:
|
(ai: vllm-qwen/qwen3.8-27b) CI 复跑确认:head |
(ai: vllm-qwen/qwen3.8-27b) Both points verified independently — you're right on both, and both are now settled on the branch. 1. Fail-open clear — confirmed real, now fixed (commit The fix takes your option (a)+(b) hybrid:
Behavior-change note vs the PR's previous commit: an FS error at status-read time now keeps the notice instead of clearing it — this aligns the code with what the PR body already claimed ("once the peer provably stops contesting"); vs master there is no change either way since the seam didn't exist yet. Your test nit is in: new G4 pins all three directions — writers dir removed ⇒ kept + no false-clear log line; dir restored with the accused marker reaped ⇒ cleared; torn marker file ⇒ kept. G3/G3-negative now register through the same function the server uses, so they track the real shape. 2. CI precondition — confirmed, and already mostly done by the fleet. #4606 is closed-unmerged (verified via API), master is at CI on the new head is fully green: 39 success / 16 skipped / 0 failed — the skips are the env-gated e2e lanes (normal state), and notably all 11 full-suite Release timing: the fix lands on master when this PR merges; cutting releases is owner-approved, so I can't promise a version number — it rides the next release after merge (you're on 0.1.208; the next cut will carry it). Your repro offer is very welcome: run the supervisor-restart → fresh-tombstone sequence against the prerelease built from |
1ac4a82 to
bb69547
Compare
…testing (#4626) The surfaced contest was persisted into the storage runtime state at detection time and replayed verbatim on every /__bili/status read until a manual ack — but detection fires while the restarting peer's tombstone marker is still seconds fresh (its graceful-shutdown dispose() rewrites the marker), so a restart handover left an alarming card behind for hours after the peer pid was gone. The stored verdict now re-evaluates against CURRENT evidence at the status read, with the same never-freeze discipline as the sibling second-process warning: the server registers the very scan that produced the verdict (findContestedWriters over its own writers dir), and when that scan no longer reports the accused peer (pid gone + marker past its freshness grace, or the contest window lapsed) the entry clears itself with an info log line. Without a registered check the legacy replay behavior stands (standalone record/ack use). The fresh-tombstone case keeps persisting: a dead pid alone is not proof of absence on every platform, so the notice stays honest while marker freshness says the peer may be alive.
…#4626) slice(0, 8) of a "session-…" id renders literally "session-" — the ack log line and the web banner both showed an unusable prefix. Log the full id (the [split-brain] error line already does) and give the card a 24-char prefix so distinct sessions are tellable apart.
e1d2619 to
4368daf
Compare
(ai: vllm-qwen/qwen3.8-27b) 上游冲突已解除:PR 分支已与当前 master 对齐, 过程
八问
CI:新 head 判定:冲突收工。 PR 恢复可合并状态,剩 required checks 跑完即可由 owner 合并。 |
What
Two defects in one failure domain (#4626, both reported with exact timelines):
[split-brain]log lines (second alert 54 ms under the freshness threshold). Two processes never coexisted.Root cause (two layers):
dispose()performs a final marker write on graceful shutdown (so peers can attribute this instance's last commits), leaving a tombstone whose mtime equals the death time.findContestedWriterstreats "dead pid + marker fresher than the 30 s grace" as possibly-live — so ANY session served within the 10-min window is flagged as dual-written by a peer that definitely exited through our own code path./__bili/statusread until a manual ack; nothing ever re-checked current evidence.Fix (four facets, one coherent loop — "the verdict is a function of current evidence"):
dispose()stampsdisposedAton its final marker write. Markers carrying it are attribution-only and never enter the contest scan — a peer that ran our dispose code provably initiated orderly exit, so no pid-visibility doubt applies and no grace is given. Crash/killed peers carry no stamp and keep the original freshness grace (defends platforms where pid probing can't see the peer). Additive schema field; absent = legacy behavior. Log wording corrected too: "another live instance" → "another instance" (remaining flagged cases are not always live).getStorageRuntimeStatus()self-clears the stored entry at the status read when the accused peer no longer contests — the same never-freeze discipline as the siblingsecondProcesswarning directly above it.contestedServedAt) shared by detection and clearing so the two can never drift; clearing consults only the accused peer's own marker viapeerStillContests(), which is tri-state — true = still contested, false = provably stopped (marker absent/reaped, dead past grace, window lapsed), null = UNKNOWN (FS error / torn file) → the notice is KEPT. A transient read error must never drop a real alert.slice(0, 8)of asession-…id renders literallysession-— the ack log line now logs the full id (the[split-brain]error line already does) and the web card shows a 24-char prefix so distinct sessions are tellable apart.Branch is rebased onto current master (
e290b4fda, green CI) as a linear history; an intermediateorigin/ci-fixesmerge became unnecessary once master itself went green and was dropped. An earlier one-word de-export micro-fix became redundant after master absorbed the proper fix upstream and was dropped.Behavior change (disclosed)
POST /__bili/storage/split-brain/ack, even after the peer process exited and its marker went stale/reaped.session-; the detection log no longer claims the peer is "live".No config surface change. No kernel/ change. No persistence-format change (verdict is in-memory per process; writer-marker files gain one optional additive field).
Tests
tests/served-markers.test.ts:tests/issue-split-brain-surface.test.ts:999999999), final step backdates the marker to prove auto-clear end-to-end through HTTP without an ack.Targeted local runs green: both files above (19/19),
npm run typecheck,npm run build. Full suite = remote CI.Closes #4626