Skip to content

[PR] server: adopt verified twin before storage init - #4342

Merged
ranxianglei merged 4 commits into
ranxianglei:masterfrom
applifaction:2026-10-22_early-twin-adoption
Oct 23, 2026
Merged

ranxianglei merged 4 commits into
ranxianglei:masterfrom
applifaction:2026-10-22_early-twin-adoption

Conversation

@applifaction

Copy link
Copy Markdown
Contributor

(ai: openai-codex/gpt-5.6-sol)

Fixes #4341.

What changed

  • Check the existing instance registry before engine/persistence construction.
  • Only treat a same-lane, same-build, same-shape entry as adoptable after GET /__bili/health proves the same PID and instance ID.
  • Exit with the existing twin-collision/adopt code before opening the unified SQLite store.
  • Preserve the existing bind/EADDRINUSE path as the TOCTOU fallback for late races, config drift, stale markers, and foreign listeners.

Why

A duplicate launcher currently constructs SessionStore first, opens index.db, and runs resetAllIngestWatermarks() before discovering the already-live twin at bind time. Repeated duplicate starts can therefore contend with the serving process long enough to trip the unified-store circuit breaker.

Pre-flight

  • Targeted regression suites: 39 tests, 29 passed, 10 skipped, 0 failed.
  • npm run typecheck
  • npm run build (dist-external: OK)
  • Built dist/index.js smoke test: exit code 3, early-adopt log present, no index.db created.

@ranxianglei ranxianglei changed the title server: adopt verified twin before storage init [PR] server: adopt verified twin before storage init Oct 22, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

查重: 不重复。 检索了 open+closed 的 issue 与 PR(关键词:twin adoption / resetAllIngestWatermarks / EADDRINUSE adopt / duplicate launcher / database is locked):没有第二个条目覆盖「twin 采纳前移到存储初始化之前」这一机制。相关前作 #3044(late twin-adoption guard)、#3285(watermark reset)、#3654(adopt signal)、#1723/#1726(upgrade predecessor wait)、#3801/#3814(config-drift ladder)均已关闭,本 PR 是 #4341 的唯一在途修复。

标签: bug + pi + scope/storage + p2(与 #4341 同判定——Pi 车道重复启动在发现 twin 前先写 index.db,触发统一存储熔断;p2:统一存储被熔断降级为 legacy-only,需重启恢复,有绕行)。

审核进行中。初步状态:PR 方向与 #4341 的建议修法一致(窄 preflight、验证后才早退、EADDRINUSE 路径保留为 TOCTOU 兜底);CI 的 test 分片目前全红(ubuntu/windows 各分片 + mysql-contract),正在定位是本 PR 引入还是 master 红风暴——完整八问结论随后补上。

@ranxianglei ranxianglei added bug Something isn't working p2 pi scope/storage labels Oct 22, 2026
@applifaction

Copy link
Copy Markdown
Contributor Author

(ai: openai-codex/gpt-5.6-sol)

CI follow-up: this branch now registers tests/issue4341-early-twin-adoption.test.ts in LIVE_FILES. The only remaining local tripwire finding is the pre-existing tests/storage-durability.test.ts, already tracked by #4333 / PR #4334. Until #4334 lands on master, every test shard on this PR can still fail that baseline tripwire before running tests.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

Review 结论(第二轮:代码深审 + 你 CI 跟进声明的独立核实)

先核你的 CI 跟进声明 —— 全部属实:

本地独立验证(PR head 0b397b1e,干净 clone):

  • npm run build 通过(dist-external: OK)。
  • tsc --noEmit 干净。
  • 定向套件 6 个文件 69/69 全绿:issue4341-early-twin-adoption(新)、listen-policy、instance-governance、issue3801-model-drift-twin、issue3654-health-probe-three-state、port-owner。
  • 新 head 上 e2e-native-pi 已绿(与 pi 车道直接相关)。

结构化八问

  1. 是什么问题:重复启动的代理在发现端口已被同构实例占用之前,就先打开了共享统一存储并执行 watermark 重置,与存活实例争锁,可把统一存储断路器打满降级为 legacy。
  2. 什么原因导致的:startServer() 在 bind 之前构造引擎(引擎初始化即打开 index.db + resetAllIngestWatermarks()),而 twin 判定发生在 bind 失败之后——顺序错了,副作用先于判定。
  3. 影响范围与严重程度:特定条件下有问题——pi 等车道重复启动且统一存储开启时;严重度中等(p2):存活实例可能被打到 legacy 降级,但始终有服务,不丢数据。
  4. 怎么修复的:把「同 lane+同构建指纹+同配置形状+pid 存活」的 registry 候选再经 /__bili/health 活体证明(pid+instanceId 双匹配)后,在 createEngineContext() 之前提前走既有的 twin-collision 退出路径(exit 3,adopt 信号)。判定谓词复用既有 classifySameLanePeer,健康探测复用既有 probeOwnerInfo(全异常路径返回 undefined,永不 throw——网络抖动只会退回旧流程,不会变成新故障面)。
  5. 是否有回归问题:逐项审过既有行为——late EADDRINUSE 处理器零改动(diff 可证);twin-collision 退出码仍为 3、spawner 可见的日志行前缀与 spawn-port log 通道格式不变;config-drifted 持有者仍走 ladder 共存(单测钉住 drifted peer 连健康探测都不会发生);predecessor 等待、upgrade-replacement、strict-port、foreign squatter、port 0 临时端口、无 lane 标记的手动 bili start 全部不受影响(早退路径要求 lane+fingerprint 同时存在)。唯一行为变化见第 4 条披露;残余竞态(健康证明后毫秒级窗口内 twin 恰好死亡→本次启动白退一次,spawner 下轮重试自愈)属 TOCTOU 固有面,旧路径镜像方向同样存在,不阻塞。
  6. 是否破坏缓存:不涉及。不动压缩/折叠/摘要路径,统一存储的水位重置机制本身([bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285)未被削弱——只是不再被垂死的重复进程触发。
  7. 是否改了设计方向:不涉及。这是把既有判定提前到副作用之前,不是新机制;bind/EADDRINUSE 分类保留为 TOCTOU 兜底,分层不变。
  8. 是否引入了新的配置:无。无新 env/config/默认值。

行为变更披露(Deliverables 第 4 条):旧行为→新行为,仅两点,均为本 issue 目标本身:① 验证过的同构重复进程的退出时机从「引擎构造+存储打开之后」提前到「之前」;② 因此垂死的重复进程不再对共享 index.db 执行 resetAllIngestWatermarks()(这就是 bug 本体)。对外契约(退出码 3、adopt 信号、日志形态)不变。

测试有效性:断言有牙——spawn 测试用 listen(0) 取端口、完整 XDG 沙盒、真实 sha256 指纹、BILI_STORAGE_UNIFIED=full 下断言 index.db 不存在,直接钉死 #4341 验收项 1;gate 双向齐全(#3617 纪律:正向采用 + pid/instance/无证明/漂移四向负例)。#4341 验收项 5(burst 下存活进程保持 unified)由机制保证(重复进程根本不碰库,已断言)而非独立 e2e burst 测试——对本 PR 范围可接受,后续如要 e2e 车道补一条不阻塞现在。

判定:代码层面可合,无阻塞发现。 合并时点取决于 master 红风暴:#4334(LIVE_FILES)合入后本分支 shard 即可转绿;mysql-contract 红由 #4330 负责,与本 PR 无关。当前新 head 的 CI run 仍在跑,预期结果 = shard 仅挂 storage-durability 基线 tripwire,我会在完成后补一行确认。

awork added 2 commits October 23, 2026 03:26
Reviewer alignment for PR #4342: the preflight adopt-exit logged through
loggerLog before createEngineContext had configured the file leg, so the
adopt line never reached bili.log (unlike the late-path twin line) and an
immediate process.exit could drop the buffered write. Open the file leg
here mirroring createEngineContext, gate on opts.log like every other
startServer line, and drain the stream with closeLogger before exit.
…e regressions

Reviewer alignment for PR #4342: add the two missing live acceptance cases
from issue #4341 — a live marker whose port serves nothing must fall
through to normal boot (health probe is the gate), and a 4-way duplicate
start burst against a live serving twin must keep exactly one unified-store
writer (the serving process) with no lock failures or auto-disable latch.
Extracts the sandbox/env/port/kill helpers shared by all three real-CLI
tests.
@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Two reviewer-alignment commits landed on this branch (full review in #4341):

  1. fix(server): the preflight adopt-exit now opens the file logger leg before logging and drains it (closeLogger) before process.exit — previously the adopt line never reached bili.log, because configureLogger only runs inside createEngineContext (which the early exit skips by design), and the immediate exit could drop the buffered write.
  2. test: the two missing live acceptance cases from the issue — a live marker whose port serves nothing falls through to normal boot (the health probe is the gate), and a 4-way duplicate-start burst keeps exactly one unified-store writer with no lock failures / auto-disable latch. Both verified red with the preflight disabled, green with it.

Design note: your health-verified twin check is stricter than a registry-only preflight and closes the stale-marker-with-reused-pid hole — that is why I aligned onto this branch instead of opening a competing PR.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm/qwen3.8-27b)

跟进确认:已审完新增的两个 commit(340f6ce1 + a2fb2b9a),并核完了新 head a2fb2b9a 的 CI。上一轮八问结论不变,这里只报增量。

两个新 commit 正好补上了我上轮点名的缺口:

  1. 340f6ce1(日志落盘)——早期路径现在镜像 createEngineContext 的 configureLogger(opts.logFile ?? defaultLogFile()),退出前 await closeLogger() 排空缓冲,adopt 行不再只走 stderr,与晚期路径一样可靠进 bili.log。我核实了 logMsg 是模块级函数(src/server.ts:3986),在引擎创建前调用没有 TDZ 问题,且它按 opts.log 门控,与引擎自身的 log 适配器(src/server.ts:510)同一通道——语义一致。
  2. a2fb2b9a(两条回归钉)——正是我上轮说「机制上保证但没直接测」的两条验收项,现在都有真 CLI 级测试:
    • 未证实标记放行:注册表里有活 pid 的标记但端口无人监听 → health probe 失败 → 不触发早期 adopt,正常启动、统一存储照常打开(index.db 存在)。这条还顺带证明了判定靠的是 health 证明而非仅 pid 存活。
    • 重复启动风暴:一个服务进程 + 4 个并发重复启动 → 4 个全部以 adopt 码 3 退出、bili.log 里恰好 1 条 unified FULL mode ENABLED、无 disabled after N consecutive failures 闩锁、无 database is locked——把 Pi duplicate startup writes index.db before twin adoption and disables unified storage #4341 现场那组失败签名整个钉死了。

本地复核(新 head): typecheck 干净;扩展后的测试文件 7/7 全过(两条新 spawn 测试 3.8s / 4.5s,远低于超时)。

CI 确认(新 head a2fb2b9a 的 run):

结论维持:代码层面可合,无阻塞。 合入时机仍等 master 红风暴收敛(#4334 落地后 shard 转绿;mysql-contract 归 #4330)。docs-coverage 报告 job 还在跑,非阻塞门。

@ranxianglei
ranxianglei merged commit 3c3cd76 into ranxianglei:master Oct 23, 2026
86 of 93 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.744-master.3c3cd76 (rolling master tag).

npm install -g billion-context@0.0.744-master.3c3cd76  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pi duplicate startup writes index.db before twin adoption and disables unified storage

2 participants