Skip to content

[PR] fix: watermark-HIT window save must not retire the verified prefix (#7346) - #7357

Merged
ranxianglei merged 5 commits into
masterfrom
2026-11-20_issue7346-pf-window-ledger
Nov 21, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-11-20_issue7346-pf-window-ledger

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

On a watermark-HIT turn the server pins only this turn's ~2-row window (snapshotMessages(session, wpHit.windowCores)). snapshotMessages stamps the pin raw (lastMessagesFolded=false), so the save presents the window as the complete raw log and the #3285 authoritative-set sweep retires every ledger row the window omits — a 100k-session prefix drained to the ~2-row window each turn, with the upstream prefix cache lost on every full-list turn.

Fix

Mark the pin non-raw on HIT turns (lastMessagesFolded=true) in src/server/prepare-openai.ts (the only watermark-HIT pin path — the watermark fastpath is openai-wire only, KDD #15). persist.ts then stops passing the raw tail, so hasRawTail=false → completeLog=false → the #3285 sweep never fires. The window is still ingested through the bounded envelope view; a genuine truncation (raw tail) still sweeps.

Leg 2 (not reproducible in master)

The issue's second leg names hydrateSlimCcrStore / slim.ts / the incremental list too short to overlap the ledger log line — none exist in current master. The described desync (empty CCR envelope → null hydration → persistent DIGEST DRIFT → completeLog sweep on every save) is preempted by the liveCcr override at src/storage/shadow.ts:504: both digest legs flow through the same override, so an empty-but-defined contentStore is consistent on both legs and no drift latches. Reported as not-reproducible; no change made for leg 2.

Regression

tests/issue7346-pf-window-ledger.test.ts pins both directions (the sweep only runs on the full path, so the watermark is cleared to force it):

  • a non-raw window save (the fix) appends the window and keeps the verified prefix — no sweep;
  • a raw window save (the pre-fix shape) still sweeps the prefix away — a genuine truncation is not masked.

Behavior change

On a HIT turn the handoff-reply stash rows are now skipped (src/handoff-response.ts:149 guards on lastMessagesFolded !== true) — consistent with the existing folded semantics (a bounded folded re-render is a pruned view, not the raw tail). The reply is still persisted via metadata.handoffResponse and re-sent by the client. No change on non-HIT turns.

Closes #7346

… the verified prefix

On a fastpath HIT the server pins only this turn's ~2-row window
(snapshotMessages(session, wpHit.windowCores)). snapshotMessages stamps the
pin raw (lastMessagesFolded=false), so the save presents the window as the
complete raw log and the #3285 authoritative-set sweep retires every ledger
row the window omits — a 100k-session prefix drained to the ~2-row window
each turn, with the prefix cache lost on every full-list turn.

Mark the pin non-raw on HIT turns (lastMessagesFolded=true) so persist.ts
stops passing the raw tail and the sweep never fires. The window is still
ingested through the bounded envelope view; a genuine truncation (raw tail)
still sweeps.

Regression: tests/issue7346-pf-window-ledger.test.ts pins both directions —
a non-raw window save keeps the verified prefix, a raw window save still
sweeps it.

Fixes #7346
@ranxianglei ranxianglei added bug Something isn't working p1 labels Nov 20, 2026
@ranxianglei ranxianglei changed the title fix: watermark-HIT window save must not retire the verified prefix (#7346) [PR] fix: watermark-HIT window save must not retire the verified prefix (#7346) Nov 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Verified PR #7357 (the #7346 fix) — clean, correct, CI green.

What I verified

Two minor points (not blockers)

  1. The regression test calls shadow.record directly with the two rawTail shapes — it pins the storage-layer sweep/no-sweep contract but does not exercise the prepare-openai.ts fix line itself, so it would not catch a regression where that one line is removed. The fix is a single line and the behavior it produces is pinned, so this is acceptable; a full-path (prepare → persist → shadow) test would be more robust.
  2. The disclosed behavior change: on a HIT turn the handoffReply stash rows are now skipped (src/server/handoff-response.ts:149 guards on lastMessagesFolded !== true); the reply is still persisted via metadata.handoffResponse (line 176) and re-sent by the client. Net effect: an idle session that ends on a HIT turn keeps its last reply in metadata.handoffResponse but not as log rows until the next re-send. Consistent with the existing folded semantics; reasonable.

8问

  1. 是什么问题:watermark-HIT 轮服务端只钉住本轮 ~2 行窗口并把它标成 raw,保存时把窗口当成完整 raw log,[bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 权威集清扫把窗口没覆盖到的所有已验证前缀行全部退役——100k 会话的账本每轮被抽干到 ~2 行,上游前缀缓存每轮全量列表轮都丢。
  2. 什么原因导致的:snapshotMessages(src/session.ts:1750)钉 pin 时恒置 lastMessagesFolded=false(raw);HIT 轮钉的是 ~2 行窗口,persist.ts:3081 据此 tail=lastMessages 把它当 raw tail 传下去,[bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 清扫把它当完整 log。窗口是有界尾,不是 raw log,但 raw 标记让保存把它呈现成了 raw log。
  3. 影响范围与严重程度:影响 watermark-HIT 轮(仅 openai-wire,KDD docs: env vars primary config method + cleanup #15)上的长会话(100k+)。严重度高:已验证前缀每轮被抽干,上游前缀缓存丢失(成本+延迟)——属缓存泄露类(p1)。
  4. 怎么修复的:在唯一的水位 HIT 钉 pin 路径(prepare-openai.ts)把 pin 标成 non-raw(lastMessagesFolded=true),persist.ts 随之不再传 raw tail → completeLog=false → 清扫不触发;窗口仍经有界 envelope 视图入库,真截断(raw tail)仍会清扫。
  5. 是否有回归问题:披露的行为变更——HIT 轮跳过 handoffReply stash 行(handoff-response.ts:149 以 lastMessagesFolded !== true 守门),reply 仍经 metadata.handoffResponse 持久化并由客户端重发;非 HIT 轮无变化。清扫只在 full 路径触发,修复把 HIT 轮路由到有界视图,真截断不被掩盖(测试双向钉死)。
  6. 是否破坏缓存:不破坏——本修复正是修一个缓存泄露(前缀缓存每 HIT 轮被丢),恢复已验证前缀。
  7. 是否改了设计方向:没有。1 行修复,在既有 folded/raw 语义内,无架构变更。
  8. 是否引入了新的配置:没有。

判定:PR 干净、修复正确且最小、回归测试钉住了关键行为、CI 全绿。blocked 是 required-review 门禁而非 CI 红,可交由 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Follow-up: filed drive-by issue #7374 (labels: scope/ci, p3) for the test-gap noted in point 1 above — the #7346 regression test simulates the fix's downstream effect (rawTail=undefined ⇒ no sweep) but does not exercise the prepare-openai.ts fix line, so it would not fail if that line were removed. Tracked separately so it isn't lost; not a blocker for merging #7357 (the fix is a single line and the storage contract is pinned).

ci-check added 2 commits November 20, 2026 17:03
…the verified prefix

The sibling test (issue7346-pf-window-ledger.test.ts) pins the #3285
sweep/no-sweep contract by calling shadow.record directly with the two
rawTail shapes, but never drives prepareOpenai — deleting the fix line
'if (wpHit !== undefined) session.lastMessagesFolded = true' would leave
the suite green.

This test drives a real watermark-HIT turn through the full proxy in
shadow mode (sidecar -> full resend -> HIT) and asserts the verified
prefix survives in the ledger after the HIT turn. Verified: passes with
the fix line, fails without it (got 2 live rows = the swept window).

Closes #7374
…rmSync (win22 EBUSY)

The fullpath regression opened a shadow-enabled SessionStore but only
called cancelAll() at teardown, which cancels pending writes WITHOUT
closing the unified sidecar's SQLite handle. Windows cannot delete a
directory while index.db is open, so the temp-dir rmSync failed with
EBUSY on the windows-latest shard (job 136191107534). closeQuietly()
(the documented short-lived-store close) releases the handle before the
cleanup unlink. Idempotent, never throws; test logic unchanged.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Fixed the red test (windows-latest, 22, 5) shard on this PR.

Diagnosis — the shard's only failure was this PR's own new regression test #7346 full-path: a watermark-HIT window save keeps the verified prefix in the ledger (tests/issue7346-pf-window-ledger-fullpath.test.ts). It was not a logic failure — it failed in teardown with EBUSY: resource busy or locked, unlink '…\sessions\index.db'. The test opens a shadow-enabled SessionStore (unified storage → index.db via node:sqlite) but only called store.cancelAll() at teardown, which cancels pending writes without closing the sidecar's SQLite handle. Windows cannot delete a directory while index.db is open, so the temp-dir rmSync failed. The CI log's own #4347 note confirms it: the same-runner retry reproduced the teardown-lock → a deterministic handle leak, not an AV/indexer flake.

Fix — added h0.store.closeQuietly() after closeServers (tests/issue7346-pf-window-ledger-fullpath.test.ts:193). That is the documented short-lived-store close (cancel pending writes, then release the shadow handle) and it runs before the cleanup rmSync. Idempotent, never throws; the test's assertion logic is untouched.

Verified — targeted local run of both #7346 tests passes (Linux; the EBUSY itself is Windows-only). Pushed as the final head 6b611620d59 (no CI-hold marker) → full CI pass now running (13 shards in progress, 0 failed so far; the previously-red windows shard is in progress).

七问:

  1. 是什么问题: PR 自带的 Watermark-HIT turns drain the msgid pin ledger — 100k-session prefix rows retired, KDD#21 tail form degrades to full form every turn #7346 全路径回归测试在 windows-latest(22,5) 分片 teardown 阶段失败——不是断言失败,是清理临时目录时 unlink index.db 报 EBUSY(文件被占用)。
  2. 什么原因导致的: 测试开了一个 shadow 模式的 SessionStore(统一存储 → node:sqlite 打开 index.db),teardown 只调了 store.cancelAll()——它只取消挂起的写、不关 sidecar 的 SQLite 句柄。Windows 在 index.db 仍打开时无法删除目录,于是 rmSync 失败;CI 日志显示同一 runner 重试也复现(确定性句柄泄漏,非 AV/索引器抖动)。
  3. 影响范围与严重程度: 仅这一个测试文件的 Windows 分片 teardown;不触碰任何产品行为、其他分片或已发布代码。属测试自身清理缺陷,严重程度低(只挡 CI 绿,不挡功能)。
  4. 怎么修复的: 在 closeServers 之后、rmSync 之前补一句 h0.store.closeQuietly()——代码里为"短生命周期 store"文档化的关闭方法(先 cancelAll 再关 shadow 句柄),幂等、不抛错;测试断言逻辑一行未动。
  5. 是否有回归问题: 无。改动只在测试 teardown 路径新增一个幂等清理调用,不触碰任何 src/ 产品代码、wire、配置或持久化格式;两个 Watermark-HIT turns drain the msgid pin ledger — 100k-session prefix rows retired, KDD#21 tail form degrades to full form every turn #7346 测试本地定向跑均绿。
  6. 是否破坏缓存: 不涉及。
  7. 是否改了设计方向: 不涉及。

判定: 修复已作为最终 head 推送,全量 CI 运行中,先前红的 windows 分片已转为运行态;本轮无需进一步动作。

…assertion is not masked by win22 EBUSY

The prior teardown (closeQuietly after closeServers, inside the try block)
only runs when the test PASSES. When the assertion fails (the fix line is
removed — the whole point of this regression test), the throw skips
closeQuietly, the shadow still holds the index.db lock, and the rmSync in
the finally block fails with EBUSY on Windows, masking the real assertion
error.

Move closeQuietly into the finally block (guaranteed to run on both the
passing and failing paths) and use rmBestEffort (bounded retries, swallow
persistent EBUSY — the #3397/#4986/#6780 pattern) so the assertion error
propagates cleanly and the temp dir is best-effort cleaned up.
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-20_issue7346-pf-window-ledger (d281569)

Latest build — install this exact version

npm install -g billion-context@0.0.11138-pr.7357.18ca6cf

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
d281569 billion-context@0.0.11138-pr.7357.18ca6cf build
74c6ece billion-context@0.0.11077-pr.7357.d078942-a2 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7357.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — leg-1 fix is correct and well-tested. One scope flag on the issue (not a code defect).

The change (src/server/prepare-openai.ts:672): after snapshotMessages pins the HIT window, if (wpHit !== undefined) session.lastMessagesFolded = true; marks the pin non-raw before ingestRowsNow, so the #3285 authoritative-set sweep sees a pruned view instead of a "complete raw log" and stops retiring the verified prefix. Miss turns are untouched — a raw full-log save still sweeps, so genuine truncation is still handled.

Tests are solid: the sibling test pins the sweep/no-sweep contract at the storage layer (rawTail omitted vs = window), and the fullpath test drives a real watermark-HIT turn through prepareOpenai, so it fails if the fix line is removed.

Scope flag for #7346: the issue names TWO legs; this PR fixes leg 1 (raw-tail). Leg 2 — the false DIGEST DRIFT on plain/empty-envelope sessions via hydrateSlimCcrStore — is not here (that function doesn't exist in master yet), and the issue says leg 2 forces the sweep even with leg 1 fixed. I can see leg 2 is exactly what big PR #6785 lands: it touches src/storage/slim.ts, src/persist.ts, src/storage/shadow.ts and adds tests/issue5874-ccr-slim.test.ts (the very test #7346 names for leg 2). So: merge #7357 now (leg 1 is a real, correct, tested fix) but keep #7346 open until #6785 also lands.

Housekeeping: 45 behind master (update); build-artifact is the stale pre-token-fix 404, re-running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 root-cause

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watermark-HIT turns drain the msgid pin ledger — 100k-session prefix rows retired, KDD#21 tail form degrades to full form every turn

1 participant