Skip to content

fix(launcher): write the codex overlay back into the real home on clean exit (#1965) - #1974

Merged
ranxianglei merged 4 commits into
masterfrom
2026-10-03_codex-exit-writeback
Oct 3, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-10-03_codex-exit-writeback

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1965

What changed

1. Exit-time merge-back (finalizeCodexHome, new). bili codex runs the client against <CODEX_HOME>-bili, but nothing folded the overlay back until the NEXT bili start — so a native codex launched right after bili could not see this run's committed SQLite records. On normal launcher exit (after the client has left, while this launch still holds the overlay) the overlay now merges back into the real home:

Failures keep the data recoverable in the overlay, are reported loudly, and the next launch's startup merge retries — no silent overwrite of either side's committed data.

2. Labeled follow-up found while working this issue (same mechanism family: overlay↔home sync safety). prepareCodexHome only refresh-protected config.toml on launches that regenerate it, so an MCP launch's owned config.toml could survive into a later non-MCP launch (MCP disabled / local or private upstream) and be merged — MCP block and all — into the user's REAL config.toml. It is now protected on every launch (symmetric to .env) and non-MCP launches restore the shared link, mirroring the .env handling exactly. Own commit + regression test.

Behavior changes (old → new)

  • Exit: previously nothing was written back until the next bili start; now a clean exit writes the run's records straight into the real home (native Codex can read them immediately). Guarantee boundary: NORMAL client exit only — a killed/crashed launcher skips finally; its data stays recoverable in the overlay and is folded by the next start. Concurrent native writes: Windows open-file lock → rename fails → rollback + retry-next-start; POSIX rename-under-open-handle → the Windows bili codex:SQLite 覆盖层拆分主库与 WAL,导致数据库损坏及已提交写入丢失 #1917 divergence contract applies (both generations survive, loser as .bili-conflict).
  • Startup: a non-MCP launch no longer lets a stale owned config.toml bake into the user's real config; it re-shares the real file instead. Previously reachable with default settings: any MCP launch followed by any launch with MCP off.
  • Scope note: finalize also writes back private entries that do NOT shadow a real-home entry (startup never merged those). Deliberate: after a clean exit the real home is authoritative for native Codex too.

Verification

  • Targeted suites green locally: tests/launcher-sqlite-overlay.test.ts (23 tests incl. 6 new finalize tests: fresh home, existing db, concurrent-native divergence with both generations preserved, blocked-slot failure + retry, crashed-client WAL unit move, missing/untouched no-op) and tests/codex-overlay-env.test.ts (7 incl. the config.toml regression test). Full matrix left to CI per local-testing discipline.
  • New gated e2e (ACP_TEST_E2E_FAKE=1, deterministic fake upstream, zero tokens; runs per-PR via ci-e2e-fake.yml with codex pinned 0.147.0): one real turn through dist/index.js codex → the real home immediately carries state_5.sqlite (quick_check ok, ≥1 thread row), no .bili-* metadata, config.toml byte-identical → then NATIVE codex exec resume --last with NO bili recovers the session (the original turn's prompt is replayed from the recovered thread, asserted against the fake's request oracle). Verified locally against the same codex pin; the rest of the e2e-codex-fake suite (incl. the bili codex 报错 MCP client for codex_apps failed to start 仍然存在 #1802 launcher test) passes alongside it.
  • Merge-compatibility with open PR fix: copy a freshly-merged overlay db back into the active home (#1951) #1955 checked in BOTH orders: byte-identical merged tree, all 39 targeted tests green. fix: copy a freshly-merged overlay db back into the active home (#1951) #1955 makes the next-launch import robust to inventory staleness; this PR makes that next launch mostly unnecessary. Complementary, no interaction defect.
  • typecheck + tsup build green.

Out of scope (flagged): the DSH lane has no known top-level SQLite sets and is unchanged; Windows-specific open-file-lock behavior is exercised via the blocked-slot simulation (directory stand-in, same technique as the existing startup-path test) plus CI.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-03_codex-exit-writeback (3afda32)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1996-pr.1974.0e8ed02

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1974.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

PR review complete — I rebased the branch onto current master and it is now mergeable; CI is re-running on the new head e1850bc9.

What I did this round

结构化七问

  1. 是什么问题:codex 持久 overlay($CODEX_HOME-bili)只在下次 bili 启动时合并回真实 home——运行期间写入的会话库和文件对原生 codex(以及用户)要等下一次 bili 启动才可见;非 MCP 启动还可能把上一次 MCP 启动遗留的生成版 config.toml 合并进用户的真实配置。
  2. 什么原因导致的:overlay 按设计是持久的(注入零文件化:废除 overlay/临时目录机制(#531 设计返工) #535/native模式下自动更新问题需要解决 #991 谱系:绝对路径引用不能悬空),而唯一回写路径 refreshOverlayHome 只挂在启动时——退出路径根本不存在,"startup-only 同步"是未文档化的契约。
  3. 怎么修复的:新增 finalizeCodexHome(),在 runLaunch 的 finally 中(proxy 已停后)执行:普通条目走既有 mergeOverlayEntry,SQLite 集合作为单位走既有 mergeSqliteSet;排除生成文件与 .bili-* 元数据;失败保持 overlay 内完整可重放 + 大声报错 + 下次启动重试(启动路径原样保留,覆盖 SIGKILL 崩溃边界)。第二个提交把 config.toml 提升为与 .env 对称的常驻保护项,非 MCP 启动恢复共享链接而非合并陈旧副本。
  4. 是否有回归问题:逐项审过既有行为——bili codex 报错 MCP client for codex_apps failed to start 仍然存在 #1802 .env 保护通道(生成 .env 永不出 overlay,测试钉住);共享链接语义(finalize 跳过 symlink/hardlink,由下次启动修复,Windows: bili omp 覆盖层 symlink 全部 EPERM 被裸 catch{} 吞掉 → 每次启动都是全新未配置(即 #289 的 Windows 真因) #381 write-through 不回流);Windows bili codex:SQLite 覆盖层拆分主库与 WAL,导致数据库损坏及已提交写入丢失 #1917/sqlite overlay (#1918): steady-state false-positive divergence — spurious warning + unbounded .bili-conflict accumulation every launch #1919 stale-drop/conflict/rollback(原函数复用,冲突保留有测试);[Bug][Codex launcher] 首次生成的 SQLite 数据库在第二次启动时从 active CODEX_HOME 遗漏 #1951 二次启动导入(rebase 已纳入 master 修复,本 PR 测试独立断言);directUrl 降级(无 overlay 则 finalize 不触发);DSH/goose/opencode 通道(未触碰);wire 协议零改动。一处不对称提请作者确认(非阻塞):非 MCP 启动时 runLaunch 传给 finalize 的排除清单只有 [".env"],而 prepare/refresh 的保护清单恒为 [".env","config.toml"]。当前代码下 prepare 保证非 MCP 启动结束时 config.toml 必为共享链接或不存在,故不可达;但若 codex 运行中重写自己的 config.toml,退出路径会把它合并回真实 home、而下一次启动路径不会——两条路径语义分叉。建议对齐两份清单(一行改动)或在 PR 里写明这是有意为之。
  5. 是否破坏缓存:不涉及——纯 launcher/home-dir 文件机制,无缓存/游标/去重键。
  6. 是否改了设计方向:没有——同一 overlay 架构,补上对称的退出期回写,复用同一套原语,启动契约不变。已披露的行为变更(PR 正文)确认成立:① 干净退出即回写(新增能力);② 非 MCP 启动不再把陈旧 owned config.toml 合并进真实 home(旧值:startup 合并,mtime 仲裁+conflict 残留 → 新值:保护+重新共享,修的是 真实 models.yml 可被塞进死端口代理地址且永不解包:"real file never touched" 前提被嵌套 merge-back 递归与遗留插件写破,裸 omp 永久拨死号 #410 类配置漂移,方向正确)。
  7. 是否引入了新的配置:无——没有新 env/config 字段/CLI flag。

判定

代码与测试质量达标,方向正确,根因修复(补退出路径)而非堵症状。CI 在 e1850bc9 上跑完后转绿即可合并(人工合并)。上面第 4 问的 config.toml 清单不对称请作者给个说法——要么一行对齐,要么一句话说明有意为之;不影响本次合并决策的话我这边没有其他拦点。

ework-agent and others added 4 commits October 3, 2026 16:59
…an exit (#1965)

bili codex ran its client against <CODEX_HOME>-bili but only the NEXT bili start attempted to fold the overlay back, so a native codex launched right after bili could not see this run's committed SQLite records. Add finalizeCodexHome(): on normal launcher exit (after the client has left and while this launch still holds the overlay) it merges each SQLite set (main+WAL/SHM/journal as one generation, via the existing mergeSqliteSet stale-check/conflict/rollback machinery) and every other private top-level entry (mtime-adjudicated, conflict-preserving mergeOverlayEntry) back into the real home, excluding generated files (.env / MCP config.toml), bili metadata and drafts. Shared entries are skipped (already in the real home). Failures keep the data in the overlay for the next launch's startup merge and are reported loudly.
…al home (#1965)

Labeled follow-up found while working #1965, same mechanism family (overlay generated-file protection). prepareCodexHome only refresh-protected config.toml on launches that regenerate it, so an MCP launch's owned config.toml survived into a later non-MCP launch (BILI_LAUNCHER_PLUGIN=0 or local/private upstream), where refresh treated it as user data and merged it — MCP block and all — into the user's real config.toml. Protect config.toml on every launch (symmetric to .env) and make non-MCP launches restore the shared link exactly like the .env path does.
…ession resume

Gated by ACP_TEST_E2E_FAKE=1 (deterministic fake upstream, zero tokens; runs per-PR via ci-e2e-fake.yml with codex pinned 0.147.0). One real turn through dist/index.js codex, then: the real home immediately carries state_5.sqlite (quick_check ok, >=1 thread row), no .bili-* metadata, config.toml byte-identical; then a NATIVE codex exec resume --last (no bili) recovers the session — the original turn's prompt is replayed from the recovered thread, asserted against the fake's request oracle.
…o the real codex home (#1965)

finalizeCodexHome excluded the legacy pid marker but not the lease
DIRECTORY from #1969. The normal path deletes it in releaseOverlayLease
before finalize runs, but a failed release (Windows EBUSY-class) leaves
the dir behind and finalize would merge it — and its owner.json — into
the REAL home, violating #1965's acceptance that lock metadata never
writes back. Exclude it like the other launcher metadata; regression
test plants a stale lease dir and asserts the db still merges back.
@ranxianglei
ranxianglei force-pushed the 2026-10-03_codex-exit-writeback branch from e1850bc to 3afda32 Compare October 3, 2026 09:00
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

第二轮 review 完成——作者在我上一轮之后把分支重新 rebase 到了最新 master(此时 master 已并入 #1969 codex overlay 租约锁、#1981 logger drain、#1963),并追加了一个针对跨 PR 交互的修复提交 3afda32c。我已审完这个新状态。

新提交审查(3afda32c)

#1969 给 codex overlay 引入了 .bili-launch.lock 租约目录,而本 PR 的 finalizeCodexHome 原本只排除了旧的 .bili-launch.pid 文件标记——若租约释放失败(Windows EBUSY 类),残留目录会被 finalize 连同其中的 owner.json 一起合并进真实 home,违反 #1965 "锁元数据永不回写" 的验收标准。新提交在 isExcluded 中加一行排除 path.basename(overlayLeaseDir(overlay)),回归测试植入陈旧租约目录后断言 db 照常合并、真实 home 无任何 .bili-* 泄漏。验证:① 排除逻辑正确且与既有元数据排除同列;② 正常路径下 releaseOverlayLease() 在 finally 中先于 finalizeCodexHome 执行(src/launcher.ts:4762 → 4770),提交信息与代码事实相符;③ 测试断言有牙(同时钉住"db 仍合并"和"无元数据泄漏")。通过。

本轮验证

  • CI:3afda32c 上 14/14 全绿(test ubuntu/windows × 22/24、real codex vs fake upstream(含 [Codex launcher] 退出时未将 overlay SQLite 数据库回写到原始 CODEX_HOME #1965 三阶段 e2e)、codex overlay MCP 注册、dsh/opencode/pi lane、hermetic registry e2e × 双平台、build-artifact、version-guard)。
  • PR 状态:mergeable: true,mergeable_state: clean。
  • 我上一轮做的本地定向验证(39/39 测试 + typecheck + build)针对的是内容相同的三个核心提交(rebase 前),本轮新增内容仅上述一行排除 + 一个测试,已由 CI 全量覆盖。

结构化七问(更新版)

  1. 是什么问题:同前——codex overlay 缺退出期回写路径,运行数据要等下次 bili 启动才对原生 codex 可见;另有一个跨 PR 交互:fix: refuse concurrent launches on a live shared overlay instead of clobbering it (#1952) #1969 新引入的租约锁目录会随退出回写泄漏进真实 home。
  2. 什么原因导致的:同前(startup-only 同步是未文档化契约);交互缺陷原因是两个 PR 并行开发,finalize 的排除清单写于租约锁落地之前。
  3. 怎么修复的:同前;新增一行按名称排除租约目录(与 pid 标记、origin 文件同一机制层),不留平行逻辑。
  4. 是否有回归问题:上一轮清单维持不变(全部保持);新增项——租约释放失败残留场景由新测试钉住,正常场景释放先于 finalize,无行为漂移。仍有一处待作者表态(上轮第 4 问):非 MCP 启动时 finalize 的生成文件排除清单只有 [".env"],而 prepare/refresh 恒保护 [".env","config.toml"];当前代码下 prepare 保证该分支结束时 config.toml 必为共享链接或不存在,故不可达,但两条路径语义分叉。作者本轮未处理也未说明——请明确:有意为之(一句话理由)或一行对齐。此为合并前唯一开放项,不阻塞代码质量判定。
  5. 是否破坏缓存:不涉及。
  6. 是否改了设计方向:没有;披露的行为变更(干净退出回写、非 MCP 不再回并陈旧 config.toml)确认成立且方向正确。
  7. 是否引入了新的配置:无。

判定

可合并——CI 全绿、mergeable_state: clean、根因修复、回归面已逐项审过。唯一遗留:上面第 4 问的 config.toml 清单不对称,请作者二选一表态(对齐 or 说明意图);若 owner 认为现状可接受,可直接人工合并:#1974

@ranxianglei
ranxianglei merged commit ffd949b into master Oct 3, 2026
14 checks passed
@ranxianglei
ranxianglei deleted the 2026-10-03_codex-exit-writeback branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Codex launcher] 退出时未将 overlay SQLite 数据库回写到原始 CODEX_HOME

1 participant