Skip to content

[PR] fix(incremental): keep message arrival timestamps across a crash before the SQLite save - #6818

Open
mariopasquali wants to merge 5 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-15_incremental-timestamps
Open

mariopasquali wants to merge 5 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-15_incremental-timestamps

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

On the incremental lane the server acknowledges a turn (appliedSeq) before the debounced SQLite save. The kernel's arrival timestamps for the new messages exist only in memory until that save. If the process stops in between, the plugin never resends those messages, and after the restart they get a new timestamp from the reseed instead of their real arrival time.

What this PR does:

  • Before a turn is acknowledged, the arrival timestamps of its new messages are written to a small per-session journal in the session extension table (content-addressed chunks plus a manifest, in one transaction). If that write fails, the turn is not acknowledged and the plugin reseeds on its next request.
  • After a restart, timestamps from the journal are restored for messages that don't have one yet. Existing stamps and message bodies are not changed.
  • Journal entries are removed only after the SQLite rows carry the same timestamp. This also works after the warm canonical was evicted, and when a forward failed after the save was already scheduled.
  • Side requests without kernel timestamps are acknowledged as before.
  • Cold views take a row's timestamp only when the kernel ledger proves it, and refuse a row timestamp that conflicts with the ledger. Sampled verification now also compares the timestamp column. Before, a changed timestamp did not count as drift.

Tests: new tests/incremental-first-seen.test.ts, plus new cases in tests/opencode-rolling-cache-marks.test.ts and tests/storage-batched-verification.test.ts. 71/71 pass in those four files (including tests/incremental-lane.test.ts). On master, 4 of the 5 new verification tests fail; the other new tests need the new module. Typecheck and the unused-export check pass. Full suite: 12,267 tests, 12,191 pass, 74 skipped, 1 fail (tests/wire-exit-matrix.test.ts OpenAI SSE abort timing; it passes 3/3 when run alone, and it doesn't touch the incremental lane). On our machine, tests/issue4612-gc-online-guard.test.ts hangs inside the full suite on master too; it passes when run alone on both.

@ranxianglei
ranxianglei force-pushed the 2026-11-15_incremental-timestamps branch from 40ca070 to 7688d22 Compare November 15, 2026 11:17
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Merge conflict resolved — PR rebased onto current master, force-pushed.

What was done this run:

  1. Rebased the 4 PR commits from mariopasquali:2026-11-15_incremental-timestamps (old head 40ca070629, base f21b601893) onto origin/master 26d9bcc649 (20 commits ahead of the old base).
  2. One conflict, in src/persist.ts writeNow() full-build path: master's [PR] release v0.1.215 #6793-review added a durabilityArms.delete(session.id) clear there, while this PR refactors the same region (refcounted saveWritesInFlight Map + beginSaveWrite/endSaveWrite replacing the Set add/catch-delete, armed saveTimers cancel, post-processing moved inside the try). Resolved as the union of both intents — kept master's durability-arm clear (comment included) at the top of the path, kept the PR's refcounted structure. The two mechanisms are orthogonal (one clears the durability arm, the other tracks in-flight saves and drains flush waiters), so no semantic choice was needed.
  3. Verified (local, targeted): npm run typecheck clean; the PR's four test files 71/71 green (incremental-first-seen, incremental-lane, opencode-rolling-cache-marks, storage-batched-verification); tests/issue3406-writenow-concurrent.test.ts (pins the writeNow concurrency semantics of the resolved region) green. Two local failures in tests/issue3157-persist-durability.test.ts are a local-env artifact, not a regression: without CI's preload-storage-baseline.mjs the pure-sqlite default turns the JSON envelope writer off, so the legacy file the test asserts never appears — both pass with the CI preload, and they fail identically on plain master locally.
  4. ci-fixes check: no ci-fixes-only fix is missing from master (master self-healed; the net diff is master's newer commits only) → no ci-fixes merge needed.
  5. Pushed with force-with-lease: 40ca070629 → 7688d22f7c. Verified live: mergeable: true, state blocked (was dirty) — awaiting human merge; CI will run on the new head.

七问:

  1. 是什么问题:PR 分支与 master 冲突(merge state dirty),无法合并。
  2. 什么原因导致的:PR 基于 f21b601,master 其后前进 20 个提交;[PR] release v0.1.215 #6793-review 在 src/persist.ts 的 writeNow() 全量构建路径加了 durabilityArms 清理,与本 PR 重构同一路径的改动(saveWritesInFlight 由 Set 改引用计数 Map + beginSaveWrite/endSaveWrite)文本重叠,产生冲突。
  3. 影响范围与严重程度:仅影响本 PR 的可合并性;不影响任何用户面或生产行为。
  4. 怎么修复的:rebase 到最新 master(26d9bcc);唯一冲突取两侧意图的并集——保留 master 的 [PR] release v0.1.215 #6793 durabilityArms 清理 + 本 PR 的引用计数 save-write 结构,两机制正交,无语义取舍。
  5. 是否有回归问题:typecheck 干净;PR 四个测试文件 71/71;钉住 writeNow 并发语义的 issue3406 测试绿;两侧机制均原样保留,无语义漂移。[bug] 会话状态写入缺 fsync:内核崩溃后留下 size 正常但 VDL=0 的全零状态文件,loadSync 永久抛错 → 502 acp-proxy failure 且不自愈(含 7 处 tmp+rename 写路径盘点) #3157 本地两红为本地环境假象(缺 CI preload → pure-sqlite 默认关 JSON writer),带 CI preload 即绿,且 master 本地同样红,与本 rebase 无关。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/游标/去重键)。
  7. 是否改了设计方向:不涉及(纯冲突解决,零内容改动;除冲突区外 PR 对 master 的 diff 与 rebase 前一致)。

判定:冲突已解,PR 回到 blocked(等人工合并),收工。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Duplicate screening: Not a duplicate. No other open/closed issue or PR targets "a crash between the incremental-lane ack and the debounced SQLite save loses the kernel's in-memory arrival timestamps." Adjacent-but-different: #6681 (large seed envelopes), #5745 (repeated-message identities, closed), #4881 (forward-fail canonical rollback), #6716 (receipt-timestamp perf), #6247 (extension timestamps across archive restore) — each is a different symptom/mechanism.

Merge status (verified live): The head (7688d22f) was rebased onto current master by the author. The persist.ts writeNow conflict present on the earlier head (40ca0706) is resolved: the new head carries both master's durabilityArms changes (#6780) and the PR's beginSaveWrite/endSaveWrite/landed/timer-cancel rewrite, and a test-merge of master is clean. CI: 0 failed, 9 success, 14 skipped, 7 still queued (mergeable_state: blocked only because checks are in flight).

Local verification (on the PR head):

  • 4 touched test files (incremental-first-seen, incremental-lane, opencode-rolling-cache-marks, storage-batched-verification): 71/71 pass.
  • npm run typecheck: clean. node scripts/check-unused-exports.mjs: clean (12 waived).
  • Side note (not a PR regression): tests/issue3157-persist-durability.test.ts fails in my sandbox but passes with BILI_STORAGE_UNIFIED=legacy — it's environment-dependent on the default storage mode and fails identically on the PR head and master here.

结构化八问(审核结论)

  1. 是什么问题:增量车道在确认一轮(x-bili-applied-seq)之前,内核新消息的到达时间戳只存在于内存;确认与去抖 SQLite 落盘之间若进程崩溃,重启后这些消息拿到的是 reseed 时间戳而非真实到达时间。
  2. 什么原因导致的:持久化顺序错位——ack 先于时间戳落盘。时间戳的「权威事实」在内核内存里,但 ack 已承诺「这一轮被接受」,两者之间没有持久化屏障。
  3. 影响范围与严重程度:仅增量车道(pluginIncrementalTurns 开启 + 统一存储可用)的用户,且仅在「ack 之后、SQLite 落盘之前崩溃」这一窄窗口触发;后果是时间戳漂移(消息时间被改成重启时间),不是数据丢失,也不影响压缩正确性——p2。
  4. 怎么修复的:把时间戳持久化提前到 ack 之前——确认一轮前先把它的新消息时间戳写进会话扩展表里的按会话 journal(内容寻址分块 + manifest,单事务);写失败则不 ack,插件下轮 reseed。重启后对尚无时间戳的消息从 journal 恢复;journal 条目只在 SQLite 行带上同一时间戳后才删除。同时修了 master 上一个真实漏洞:flushSync 取消去抖 timer 后 waiter 永不排空(PR 用 settle(ok) 包裹所有返回、按 saveWritesInFlight 计数排空),以及 writeNow 现在会取消已武装的去抖 timer。
  5. 是否有回归问题:审过的既有行为——(a) afterNextFlush 三个调用方(增量成功/失败路径、spillBetweenTurns)在新语义下都更保守(journal 保留 / spill 跳过),无回归;(b) 冷视图加载的 throw 被 viewFor 的 try/catch 兜住 → 回退 legacy 文件,不 brick 会话;(c) 行 ts 列只在内核账本证明时才采信、冲突则拒收,不会把错误时间戳灌进冷视图;(d) messageSignature 签名前剥 ts、serializeForStore 从不把 ts 写进内容字节 → 无假漂移。未发现回归。
  6. 是否破坏缓存:不破坏。时间戳是行级元数据,不进前缀缓存的内容字节;serializeForStore 明确把 ts 留在专用列、绝不进内容字节(存储内容与 pre-feature 逐字节一致);render 路径只读存储值、从不 Date.now() 的前缀缓存不变量未被触碰。
  7. 是否改了设计方向:无架构转向。在既有「统一存储 + 会话扩展表」边界内加了一条持久化屏障。唯一行为变更:增量 probe 在 turn-in-flight(receiptReady=false)时现在返回 appliedSeq 0 而非当前 msgs.length——更正确(ack 未发前不应报告已应用)。
  8. 是否引入了新的配置:无新 env/config/CLI flag。复用既有 BILI_MESSAGE_TIMESTAMPS(默认开)与统一存储开关;journal 走会话扩展表,无新存储面。

判定:方向正确(修根因——持久化顺序,不是掩症状),代码与测试扎实(71/71 真测试,含并发/回滚/预算/驱逐矩阵),无回归、无缓存破坏。PR 已 rebase 到当前 master、无冲突、CI 无失败(7 项在跑)。可合(等 CI 全绿 + 人工过目)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

ranxianglei pushed a commit that referenced this pull request Nov 15, 2026
…#6818)

The architecture-docs CI lane (tools/check-architecture-docs.mjs) requires
every src/ module to be listed in reference/architecture.md. PR #6818 added
src/incremental-first-seen.ts but only updated the pi.ts row, so the new
module's entry was missing and the lane failed. Add the row in alphabetical
order (before incremental-lane.ts).
ranxianglei pushed a commit that referenced this pull request Nov 17, 2026
….md (#6818)

The post-#6592/#6880 re-merge (2d53c13) was based on e45ad01, which predates the 25a89b1 fix that listed src/incremental-first-seen.ts, so the entry was lost and the architecture-docs CI lane failed on 6d2e2a1. Re-add the row (alphabetical, before incremental-lane.ts). Guard passes locally.
ranxianglei added a commit that referenced this pull request Nov 18, 2026
[PR] wave-11 batch: 7 community fixes (opencode provider, preflight epoch, output budget, +4) — #6797/#6818 excluded

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants