Skip to content

fix: refuse concurrent launches on a live shared overlay instead of clobbering it (#1952) - #1969

Merged
ranxianglei merged 3 commits into
masterfrom
2026-10-03_codex-overlay-busy-lock
Oct 3, 2026
Merged

ranxianglei merged 3 commits into
masterfrom
2026-10-03_codex-overlay-busy-lock

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1952

What / why

Two bili codex launches sharing one real Codex home both target the fixed ${CODEX_HOME}-bili overlay. The second launch detected the live first holder (.bili-launch.pid) but only printed a warning, then rewrote the first launcher's generated .env (proxy routing) and MCP config.toml — last writer won. The detection was advisory only: no ownership, and even two FIRST launches racing had a check-then-write window.

Fix

Exclusive lease on the overlay, acquired before any refresh/config write:

  • acquireOverlayLease() — atomic mkdir of <overlay>/.bili-launch.lock/ + owner record (pid + random token). Live foreign holder → OverlayBusyError before any write. Dead holder → stale lease reclaimed automatically (crash residue). Partially written owner record → treated as busy (conservative); manual unlock = delete the lock dir. The legacy .bili-launch.pid marker still acts as a liveness hint, so this build also refuses to touch an overlay held by a pre-[Bug][Codex launcher] 检测到存活启动器后仍覆盖共享 overlay 的代理与 MCP 配置 #1952 launcher.
  • runLaunch aborts cleanly on busy: logs the reason, stops THIS launch's own proxy child, exits 1. A plain throw would leak the detached proxy — the prepare step runs after ensureProxyRunning, outside the cleanup finally. The lease is released in that same finally.
  • Sits in the shared refreshOverlayHome, so every overlay lane gets the guard (codex + dsh) — one fix at the mechanism, matching the issue's acceptance criteria.

Behavior change (old → new)

scenario old new
2nd concurrent launch, same home warns to stderr, then clobbers the 1st launcher's .env / MCP config / pid marker (last writer wins) hard-refuses: clear error naming the holder pid, its own proxy stopped, exit 1, zero writes
relaunch after previous exit works works (lease released in finally)
crashed previous launch (stale lock/marker) works (dead pid ignored) works (dead-pid lease auto-reclaimed)
PID-reuse edge n/a a reused live PID can false-positive as busy → documented manual unlock (rm -rf <home>-bili/.bili-launch.lock)

No config surface added — no new env vars / config fields / CLI flags. No persistence-format change (the lock dir is transient runtime state inside the existing overlay). Wire protocol untouched. Not auto-merge eligible (§7.4: launcher lifecycle territory) — needs human review.

Tests

New tests/overlay-busy-lock.test.ts (10 tests): OP repro via legacy marker (byte-identical files after refusal), foreign live lease, partial owner record, stale lease reclaim, stale marker alone, same-process roundtrip, token-verified release, six-process race (exactly one winner), dsh-lane parity, runLaunch(codex) integration (proxy stopped, client never spawned, exit 1, message carries holder pid).

Targeted local runs (CI does the full suite): overlay-busy-lock 10/10 · codex-overlay-env 14/14 · launcher-sqlite-overlay 17/17 · instance-governance 14/14 · launcher 264/265 (1 pre-existing env skip: needs built dist). Typecheck (src) clean; build green.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-03_codex-overlay-busy-lock (83174c7)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1985-pr.1969.a0962b6

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1969.tgz
npm install -g package

Install instructions are refreshed on each push.

ework-agent and others added 3 commits October 3, 2026 15:51
…lobbering it (#1952)

Two bili launches sharing one real home both target the fixed `${home}-bili`
overlay. The second launch detected the live first holder (.bili-launch.pid)
but only warned, then rewrote the first launcher's generated .env (proxy
routing) and MCP config.toml — last writer won. The detection was advisory:
no ownership, and even two FIRST launches racing had a check-then-write window.

Replace the advisory check with an exclusive lease:

- acquireOverlayLease(): atomic mkdir of <overlay>/.bili-launch.lock/ plus an
  owner record (pid + random token). A live foreign holder throws
  OverlayBusyError BEFORE any refresh/config write; a dead holder's lease is
  reclaimed (crash residue); a partially written owner record reads BUSY
  (manual unlock = delete the lock dir). The legacy .bili-launch.pid marker
  still acts as a liveness hint, so this build also protects overlays held by
  pre-#1952 launchers.
- runLaunch aborts cleanly on busy: logs the reason, stops THIS launch's own
  proxy child (a plain throw would leak it — prepare runs after
  ensureProxyRunning), exits 1. The lease is released in the same finally
  that stops the proxy.
- Sits in the shared refreshOverlayHome, so every overlay lane gets the guard
  (codex + dsh) — one fix at the mechanism.

Behavior change: a second concurrent launch of the same home now hard-refuses
(exit 1, zero writes) instead of warning and clobbering the first launcher's
config. Sequential relaunches are unaffected (lease released on exit; stale
leases auto-reclaimed).

Tests: tests/overlay-busy-lock.test.ts — OP repro via legacy marker, foreign
live lease, partial owner record, stale reclaim (lease + marker),
same-process roundtrip, token-verified release, six-process race (exactly one
winner), dsh lane parity, runLaunch integration (proxy stopped, client not
spawned, exit 1).
…infers CJS and esbuild rejects its top-level await (children exit 1 before printing; red on all 4 CI lanes and locally)
@ranxianglei
ranxianglei force-pushed the 2026-10-03_codex-overlay-busy-lock branch from 4440e86 to 83174c7 Compare October 3, 2026 07:53
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ## PR #1969 review 结论(第二轮:rebase 后复核)

按 [PR] 流程核对完毕:拉取分支逐行审代码、确认 CI、本地对合并后树重跑定向验证。分支未被我改动(见文末说明)。

审查动作与证据

  • 方向关(原 issue [Bug][Codex launcher] 检测到存活启动器后仍覆盖共享 overlay 的代理与 MCP 配置 #1952):修的是机制(共享 overlay 无排他所有权 → 原子租约),不是症状(警告文案)。OP 三条验收标准逐条满足:两个存活启动器不能互相覆盖配置 ✓(拒绝后 .env/config.toml/pid marker 字节级断言)、存活检测保护完整 refresh+config 事务 ✓(租约在任何写入前取得,src/launcher.ts:1956)、未走 per-launch home 而是显式单写者策略并给出理由 ✓。
  • 代码关(逐点核验):
    • 租约获取点在 refreshOverlayHome 顶部、merge/write 之前;前置的 mkdirSync(overlay,{recursive}) 无破坏性(外部持有者存在时目录必已存在;两个首启动竞态时双方 mkdir 后再原子仲裁,最坏只留一个空目录)。
    • 锁目录不会泄漏进用户真实 home:overlay→real 合并循环只在 realEntries.has(entry) 为真时写回(src/launcher.ts:2025),而 .bili-launch.lock 位于 ${home}-bili(home 的兄弟目录),不可能出现在真实 home 中。
    • 模块级 singleton 租约安全:dsh/codex 两条 lane 是 runLaunch 内基于 client 名的 else-if 分支,单次启动互斥,每进程至多一个租约;同进程重入走 token 刷新路径。goose lane 有意不复用 refreshOverlayHome,不受影响。
    • runLaunch 的两个 catch 只捕 OverlayBusyError(其他错误照常上抛),abortLaunchOnBusyOverlay 显式 return 防止穿透到 client spawn——直接 throw 会泄漏 detached proxy 的路径已封死。
    • 跨版本兼容:遗留 .bili-launch.pid 保留为存活提示,新版拒绝触碰旧版持有者的 overlay(方向正确;反向覆盖不可避免,需整体升级)。
  • 回归关(fix(launcher): second launch keeps the overlay-created SQLite database in the active CODEX_HOME (#1951) #1964 交互):fix(launcher): second launch keeps the overlay-created SQLite database in the active CODEX_HOME (#1951) #1964 改的是同一函数内 SQLite copy-inventory(合并树 ~L1904-1960 区域),与本 PR 的租约 hunk(函数顶部)文本不相交、语义正交(门在写入前,inventory 在拷贝阶段);fix: resolve compaction ownership for pi virtual-model selections (#1961) #1962/fix(#1957): invalidate stale health verdicts across the respawn boundary #1966 不触碰 launcher.ts。在合并后的完整树上重跑验证通过(见下)。
  • 测试有效性关:10 个测试全部有牙——OP 复现用字节级文件断言(回归时抓得住)、外部租约断言零写入(.env 不存在)、六进程真竞态断言恰好一个 WIN、token 篡改后 release 不动锁、runLaunch(codex) 集成断言 exit=[1]+client 未 spawn+proxy 被停+错误信息含持有者 pid。非恒真、非 mock-only。

结构化七问

  1. 是什么问题:同一真实 home 的第二个并发 bili 启动,把第一个存活启动器的代理路由(.env)和 MCP 配置(config.toml)静默覆盖,last-writer-wins,第一个实例的路由失效。
  2. 什么原因导致的:共享 overlay ${home}-bili 只有建议性 pid 提示(kill(pid,0) 检查后继续执行全部写入),没有排他所有权;且「先查后写」本身有竞态窗口,连两个首启动都可能同时看到"无持有者"。
  3. 怎么修复的:在共享机制点 refreshOverlayHome 顶部取得原子排他租约(<overlay>/.bili-launch.lock/ 原子 mkdir + {pid, randomUUID} owner 记录),先于任何写入;存活外部持有者 → OverlayBusyError 零写入;死持有者 → stale 自动回收;半写记录 → 保守判忙。runLaunch 遇忙:报错(含持有者 pid)→ 停自己的 proxy → exit 1,租约在同一 finally 释放。codex + dsh 两 lane 一次全覆盖。
  4. 是否有回归问题:逐项审过既有行为——正常退出后顺序重启 ✓、崩溃后重启(stale 自动回收,含遗留 marker)✓、同进程重入 ✓、dsh lane 除忙拒外不变 ✓、manageRouting=false 共享 .env 分支 / SQLite 合并(Windows: bili omp 覆盖层 symlink 全部 EPERM 被裸 catch{} 吞掉 → 每次启动都是全新未配置(即 #289 的 Windows 真因) #381/Windows bili codex:SQLite 覆盖层拆分主库与 WAL,导致数据库损坏及已提交写入丢失 #1917)/ symlink 重指均未触碰 ✓、fail-fast 方向无吞错 ✓、与 fix(launcher): second launch keeps the overlay-created SQLite database in the active CODEX_HOME (#1951) #1964 的 copy-inventory 变更正交(合并树上重验)✓。一处有意行为变更(PR 正文已披露):第二个并发启动从「警告后覆盖」变为「硬拒 exit 1」——这正是修复本身,非漂移。
  5. 是否破坏缓存:不涉及。prefix cache、会话持久化、kernel ref 空间均不受影响;锁目录是 overlay 内瞬态运行时状态,不进持久化格式,且已验证不会回流到用户真实 home。
  6. 是否改了设计方向:无架构转向——保持「固定 overlay + 每 home 单写者」,把建议性提示升级为真正的排他所有权。
  7. 是否引入了新的配置:无。不新增 env/config/CLI flag;手动解锁 = 删除 <home>-bili/.bili-launch.lock(错误信息中已写明)。

验证状态

  • head 83174c7d 已在当前 master(03d27f96)之上,CI 13/13 全绿(test ubuntu/windows 22+24、codex overlay e2e、四个 real-client lane、registry e2e×2、build-artifact、version-guard)。
  • 我对与 head 树完全一致的重基树本地重跑:typecheck(src)干净;定向套件 overlay-busy-lock 10/10(独立重复 2 次)+ codex-overlay-env + launcher-sqlite-overlay + instance-governance + launcher 共 321/321;build 绿。(本地首轮并行跑 tsc+tsup+tests 时出现过 1 例瞬时失败,后续 3 轮干净运行零复现,CI 亦绿——判定为本机资源争用,非代码缺陷。)

过程说明

本轮我原本要把分支 rebase 到最新 master 再推回;推送时发现远端分支已被并发工作先行 rebase,且与我本地重基结果树完全一致(git diff 为空)——故未做任何推送,远端状态即最终状态。

判定:可合。 §7.4 launcher 生命周期类,不满足 auto-merge,请人工合并:#1969

@ranxianglei
ranxianglei merged commit 603d26c into master Oct 3, 2026
13 checks passed
ranxianglei added a commit that referenced this pull request Oct 3, 2026
…o the real codex home (#1965)

finalizeCodexHome excluded the legacy pid marker but not the lease
DIRECTORY from #1969. The normal path deletes it in releaseOverlayLease
before finalize runs, but a failed release (Windows EBUSY-class) leaves
the dir behind and finalize would merge it — and its owner.json — into
the REAL home, violating #1965's acceptance that lock metadata never
writes back. Exclude it like the other launcher metadata; regression
test plants a stale lease dir and asserts the db still merges back.
@ranxianglei
ranxianglei deleted the 2026-10-03_codex-overlay-busy-lock branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Codex launcher] 检测到存活启动器后仍覆盖共享 overlay 的代理与 MCP 配置

1 participant