Skip to content

fix(#2195): provenance-first generation winner in codex overlay SQLite merge-back - #2207

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-06_sqlite-wal-generation-winner
Oct 6, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-06_sqlite-wal-generation-winner

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

On Windows, two sequential bili codex exec runs lost the second run's thread into .bili-conflict instead of merging it back into the real home's active state_5.sqlite. Root cause (verified against e5a0066de): mergeSqliteSet chose the winner by main-db mtime only (rMain.mtimeMs >= oMain.mtimeMs ? "real" : "overlay") and only afterwards checked the loser against the copy-time origin snapshot. But a WAL-only commit leaves the main db's bytes AND mtime untouched, and a file copy can carry the source mtime forward (issue measured identical mtime_ns on both sides) — so equal main mtimes deterministically selected the STALE generation, and the stale-check could only decide warn-vs-silent for the already-wrong loser.

Fix (mechanism-level)

  • mergeSqliteSet now classifies BOTH sides against the copy-time origin snapshot (sqliteSideVsOrigin, replacing the loser-only sqliteLoserIsStaleCopy):
    • one side unmodified + other committed → committed side wins, unmodified side dropped silently (no warning, no conflict file);
    • both unchanged → byte-equivalent generations, fixed pick, silent de-duplication;
    • both independently advanced → true Windows bili codex:SQLite 覆盖层拆分主库与 WAL,导致数据库损坏及已提交写入丢失 #1917 divergence → loud warning + whole-group .bili-conflict preservation, main-mtime tiebreak (unchanged);
    • no usable snapshot (first launch, upgrade mid-cycle, manual deletion) → conservative mtime fallback, nothing claimed proven newer.
  • Commit detection excludes -shm (rebuildable coordination file — merely OPENING a WAL-mode db creates a 0-byte -wal + -shm; verified empirically). A zero-byte newly-present -wal is likewise treated as a connection artifact, not a commit. This removes a second spurious-conflict path in the same mechanism family: the old staleness check counted shm drift as "not stale" and produced conflict files for mere opens.
  • Whole-set (main/WAL/journal) unit movement, rename-failure rollback, and recoverable conflict groups are all preserved unchanged.
  • runLaunch now releases the overlay lease after finalizeGooseHome/finalizeCodexHome instead of before — the exit merge-back runs while this launch still holds the exclusive lease, closing the window where a concurrent launch could acquire the overlay mid-finalize (issue fix-point 4).

Behavior changes (old → new)

  1. Winner selection: main-db mtime only → provenance-first, mtime only as documented fallback. Flips the outcome exactly in the reported failure shape (WAL-only commit + equal main mtimes: stale-real-wins → committed-side-wins). In the both-unchanged steady state the winner is now deterministically real (was: whatever the mtimes said); end state is identical there (silent drop + re-copy).
  2. Stale-drop predicate: -shm byte drift no longer blocks the silent drop (old code warned + quarantined on shm-only drift; reproduced: new unit test fails on pre-fix code for this reason too).
  3. Warning wording: "kept the newer generation" → "kept the winning generation" (cosmetic; distinct/diverge substrings kept for existing consumers/tests).
  4. Exit ordering: lease release moved after the exit merge-backs (narrower concurrency window; no user-visible change in the normal single-launch flow).
  5. Applies to every consumer of mergeSqliteSet: codex exit finalize AND startup fold, and the dsh startup fold (same mechanism, same direction of improvement).

Config surface

None. No config field, env var, or CLI flag added/renamed/re-semantics'd.

Regression coverage

  • tests/launcher-sqlite-overlay.test.ts: 5 new deterministic tests using subprocess-created uncheckpointed WAL commits (no pre-merge db open, no manual making-the-winner-newer; both mains pinned to a fixed equal epoch per the issue's sanctioned approach): overlay-WAL-only wins silently through BOTH entry points (exit finalize + startup fold), mirror case, SHM-only drift ≠ commit, true both-sides divergence warns + preserves recoverable group (recovered exactly like the issue's manual step 6), missing-origin stays conservative. All run PRAGMA integrity_check on the active db. Verified they fail on pre-fix code (tests 1–3 red on e5a0066de, green after fix); all 30 file tests pass.
  • tests/e2e/e2e-codex-fake.test.ts: new hermetic real-codex test — three sequential bili codex exec launches + native codex exec resume --last, asserting every thread lands in the real home's active db (delta +1 per launch), the NEXT active overlay carries all previous threads while in flight, zero .bili-conflict files, and native resume restores launch 3's history (the issue left codex resume recovery unverified). Ran green locally with codex-cli 0.147.0 (same pin as CI): threads table = 3 rows, no conflict files.
  • .github/workflows/ci-windows-codex.yml: new hosted-Windows job running the #2195|#1965 lanes of that suite (codex pinned 0.147.0 like ci-e2e-fake.yml). The failure condition is Windows-specific (timestamp-preserving copies) and NOT reproducible on POSIX — this is the real-machine gate; the unit suite is its deterministic cross-platform twin.

Local gates: npm run typecheck clean; targeted suites green (launcher-sqlite-overlay 30/30, launcher+codex-overlay-env+overlay-busy-lock 295/295, e2e-fake #2195|#1965 2/2). Full regression delegated to CI per local-testing discipline.

Hot-file note: base is current master (verified 0 behind at commit time). Open PRs touching src/launcher.ts (#2194 dsh spawn retry, #2178 ACP_PORT pin) sit in disjoint regions (dsh lane / proxy port logic); if either lands first, resolve by union of intent and re-run the targeted suites.

Closes #2195

…e merge

A WAL-only commit leaves the main db's bytes AND mtime untouched, and a
file copy can carry the source mtime forward (measured on Windows), so the
old main-mtime-only rule selected the STALE generation whenever the newer
commit sat in an uncheckpointed WAL with equal main mtimes: sequential
bili launches quarantined the new thread into .bili-conflict instead of
merging it back.

mergeSqliteSet now classifies each side against the copy-time origin
snapshot (#1919): unmodified-vs-committed decides the winner outright;
mtime stays in play only when both sides advanced independently (true
#1917 divergence) or no usable snapshot exists (conservative). -shm is
excluded from commit detection (rebuildable coordination file; merely
opening the db creates a 0-byte -wal + -shm), so connection artifacts no
longer read as commits. Whole-set movement, rollback and recoverable
conflict groups are unchanged.

runLaunch now releases the overlay lease AFTER the exit merge-backs, so a
concurrent launch cannot acquire the overlay mid-finalize.

Regression: deterministic unit tests (subprocess-created uncheckpointed
WAL commits, pinned equal main mtimes; fail on pre-fix code) plus the
real-codex two-sequential-launch e2e, now also running on a hosted
Windows runner where the failure condition actually occurs.
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-06_sqlite-wal-generation-winner (20fbd62)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2267-pr.2207.ac6e025

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2207.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #2207 审查结论(fix #2195:codex overlay SQLite merge-back 代际胜者选择)

我按 review 优先级走了五关:审原 issue → 回归审查 → 代码主关 → 测试有效性 → diff 整洁度,并在 head commit 20fbd624c 上独立核对了 CI。结构化七问如下。

1. 是什么问题
顺序两次 bili codex exec(无并发 codex)时,codex 的线程被隔离进 .bili-conflict,而不是合并回真实 home——即归并时选错了"哪一代是新的"。

2. 什么原因导致的
mergeSqliteSet 只按主库 mtime 选胜者(rMain.mtimeMs >= oMain.mtimeMs ? "real" : "overlay")。而 WAL-only 提交不改变主库的字节和 mtime;Windows 文件拷贝又会把源 mtime 原样带过来 → 两边主库 mtime 相等时确定性地选了旧的那一代。原来的 stale 检查只能决定"告警还是静默",纠正不了已经选错的胜者。这是根因层(决策点),不是表面现象。

3. 怎么修复的
改成来源优先:退出前把两边各自与 copy-on-launch 时的 origin 快照对比(新函数 sqliteSideVsOrigin 取代只判输家的 sqliteLoserIsStaleCopy)——一边未变+另一边已提交→已提交方直接胜出;两边都未变→幂等去重;两边都变了(真并发分歧)或无可用快照(unknown)→才回退 mtime 并保留双方。同时把 -shm 排除出提交判定(可重建协调文件,仅打开库就会产生 0 字节 -wal+-shm),并把 overlay 租约释放挪到退出 finalize 之后。修在决策点而非中途过滤字节,符合"根因不是堵症状"。

4. 是否有回归问题
逐项审过 diff 触碰路径的既有行为:稳态单侧提交→正确胜者、输家字节等于 origin(⊆胜者)故静默清除零冲突;双未变→去重到 real;真并发分歧→mtime 决胜 + 响亮告警 + 整组保留(与旧 #1917 路径完全一致);任一无快照→mtime 兜底(保守,等价旧行为)。物理搬迁 / 改名失败回滚 / 可恢复冲突组 / sharedLink 迁移 / orphan 处理全部未动。唯一输出变化:告警文案 newer generation→winning generation(保留 diverge/distinct 子串,消费方与测试匹配不受影响,PR 已披露)。未发现回归。

5. 是否破坏缓存
不涉及。改的是 codex 外部 SQLite 库的代际归并判定,不碰 ACP 会话缓存、压缩增量游标、去重键或重试策略;prefix cache 与会话身份绑定逻辑均未触碰。

6. 是否改了设计方向
不涉及架构转向。仍在同一模块(src/launcher.ts),数据流不变(copy-on-launch → overlay 运行 → 退出归并回真实 home),只是把"哪一代权威"从 mtime 换成 provenance-first——这正是 issue 里 owner 明确指定的方案。

7. 是否引入了新的配置
不涉及。无新 env/config/CLI/默认值;PR 亦声明无配置面变更。


测试有效性(质量关):单测用真实 node:sqlite 子进程构造未 checkpoint 的 WAL 提交,pinMainsEqual 确定性模拟 Windows 等 mtime,断言落到实际文件字节 / 行数 / integrity_check / 冲突文件有无,前三个用例在修复前代码上会红(作者已验证);e2e 走真实 bili codex exec×3 + 原生 codex exec resume --last,按线程数增量断言落入 active 库且零 .bili-conflict,并补验了 issue 里没验证过的 resume 恢复路径;新增 hosted-Windows CI job(codex 钉 0.147.0)让失败条件真正能复现的平台进入门禁。都是真测试、有牙、确定性,无 mock 假断言。

diff 整洁度:恰好 4 个文件(launcher 修复 + 2 个测试 + 1 个 workflow),无锁文件 / 生成物 / 无关改动。唯一一处顺带小改:e2e 既有 #1965 用例紧邻处一行类型收窄 cast((p as { text: unknown }).text),语义中性、同文件顺手修,非跑题噪音。

CI:我在 head 20fbd624c 独立确认 21/21 checks 全绿、mergeable=true、base=master、0 behind(与楼内 bot 报告一致)。

判定:可合并(mergeable)。 根因定位准确、四个修复点全部落实、物理安全机制完整保留、单元 / 真机 e2e / Windows CI 三层有牙测试齐备、diff 干净、CI 全绿、无配置 / 线协议 / kernel 面变更、无数据丢失路径。合并动作按规则由你来执行,我不代合:#2207

@ranxianglei
ranxianglei merged commit f85ccfc into master Oct 6, 2026
21 checks passed
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.56-master.f85ccfc (rolling master tag).

npm install -g billion-context@0.0.56-master.f85ccfc  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Windows Codex] WAL-only 提交时 SQLite 合并选错世代:顺序启动新增 thread 被移入 conflict

1 participant