Skip to content

Watermark-HIT turns drain the msgid pin ledger — 100k-session prefix rows retired, KDD#21 tail form degrades to full form every turn #7346

Closed

Description

@ranxianglei

Phenomenon

100k-message 2x2 pinned session: messages table oscillates 100k live rows ↔ 2 live rows every turn; every tail-form request logs incremental list too short to overlap the ledger (list=3, new cores=~T, view=T, ledger=2); body alternates full-list/tail form; ingest is path=full on every turn.

Root cause — two legs, both force the #3285 authoritative-set sweep on window saves

  1. Window presented as raw tail. On a fastpath HIT the server snapshots only the ~2-row tail window (snapshotMessages(session, wpHit.windowCores)). snapshotMessages claims rawness (lastMessagesFolded=false), so the save presents the window as the complete raw log and the sweep retires every ledger row the window omits. Fix: mark the tail non-raw on HIT turns (lastMessagesFolded=true) — the sweep then sees a pruned view, not a log.
  2. False DIGEST DRIFT on plain sessions (slim.ts). Every session persists a slim CCR envelope with zero declared refs (server.ts materializes an empty content store for all sessions). hydrateSlimCcrStore returned null for empty envelope + empty ledger ("file lane stays authority"), desyncing the unified replay from the live view on every sampled verify → session marked drifted → completeLog = hasRawTail || drifted = true on EVERY save → sweep runs even when the raw-tail leg is fixed. Fix: hydrate it as the COMPLETE empty store (the key-set coverage guard above already rejects envelopes that declare refs the ledger lacks).

Fix

Both legs, with regression tests: tests/issue7064-pf-window-ledger.test.ts (HIT window save must not retire the verified prefix; genuine truncation still sweeps) and tests/issue5874-ccr-slim.test.ts (empty envelope + ledger hydrates the empty store).

Impact

  • Performance: full-list body + 100k id validation + path=full ingest every other turn; upstream prefix cache loss on full turns.
  • Data integrity (reversible): messages.msg_id is the single id fact source ([PR] release review #5826); rows recover via retire≠delete, but a crash at low water loses the identity baseline until re-anchor.

Activity

  1. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Dedup: not a duplicate. Sibling #7345 (open) is a different symptom (fastpath never hits) with a different mechanism (id-namespace mismatch in the head check). #7346 is a follow-up that only manifests once #7345 is fixed — the fastpath must HIT for the drain to occur. No open PR targets this thread.

    Fix (leg 1 — confirmed real in master): On a watermark-HIT turn, snapshotMessages(session, wpHit.windowCores) pins only the ~2-row window and stamps it raw (lastMessagesFolded=false). The save then presents the window as the complete raw log, and the #3285 authoritative-set sweep retires every verified prefix row the window omits — a 100k-session ledger drained to the ~2-row window each turn. The fix marks the pin non-raw on HIT turns (lastMessagesFolded=true) in src/server/prepare-openai.ts (the only watermark-HIT pin path — the fastpath is openai-wire only, KDD #15), so persist.ts stops passing the raw tail and the sweep never fires. A genuine truncation (raw tail) still sweeps.

    Leg 2 — not reproducible in master: the named symbols (hydrateSlimCcrStore, slim.ts, the incremental list too short to overlap the ledger log line, pluginMsgIdIncremental, "KDD#21") do not exist in current master. The described desync (empty CCR envelope → null hydration → persistent DIGEST DRIFT → completeLog sweep on every save) is preempted by the liveCcr override at src/storage/shadow.ts:504 — both digest legs flow through the same override, so an empty-but-defined contentStore is consistent on both legs and no drift latches. No change made for leg 2.

    PR: #7357 (labels: bug, p1)

    Labels: bug, scope/storage, p1 (100k-session prefix cache loss on every full-list turn — cache-hit-collapse class per the 2026-10-19 p1 ruling).


    八问

    1. 是什么问题:watermark 快路径命中时,服务端只钉住本轮 ~2 行窗口并标为 raw,保存时把窗口当成完整日志,[bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 权威集清扫把窗口没带的前缀行全部退休——100k 会话的账本每轮被抽干到 ~2 行,前缀缓存在全量轮丢失。
    2. 什么原因导致的:snapshotMessages(src/session.ts:1750)无条件把钉住的消息标为 raw(lastMessagesFolded=false),但 HIT 轮的窗口只是新尾巴、不是完整日志;persist.ts:3085 见 raw 就把窗口当完整日志传给存储层(shadow.ts:495 hasRawTail=true),全量路径下 completeLog=true(shadow.ts:549)触发 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 清扫(store.ts:1035)。
    3. 影响范围与严重程度:所有 watermark 快路径命中的 openai 线会话(2x2 pin 车道,依赖 Watermark fastpath never hits on 2x2 pinned sessions — head check compares message ids across two id namespaces #7345 修好后才命中);100k 级会话每轮全量 body + 100k id 校验 + path=full ingest,上游前缀缓存在全量轮丢失——缓存命中崩坏类,p1。数据可逆(retire≠delete),但低水位崩溃会丢身份基线直到重新锚定。
    4. 怎么修复的:HIT 轮把窗口钉为非 raw(lastMessagesFolded=true),persist.ts 就不再传 raw tail,hasRawTail=false → completeLog=false → 清扫不触发;窗口仍经有界 envelope 视图入库,真截断(raw tail)照常清扫。
    5. 是否有回归问题:审了 lastMessagesFolded 的全部消费方——persist.ts:3085/3269(tail 计算)、persist.ts:4406(envelope 持久化)、export.ts:147(handoff 渲染)、handoff-response.ts:149(reply stash)、ingest.ts:547(tailFolded 标记)、plugin.ts:1880(fork 子会话)——语义一致(有界折叠视图不是 raw tail)。唯一行为变更:HIT 轮的 handoff-reply stash 行现在跳过(handoff-response.ts:149 本就按 lastMessagesFolded !== true 守卫),reply 仍经 metadata.handoffResponse 持久化并由客户端重发;非 HIT 轮无变化。
    6. 是否破坏缓存:不破坏——修复正是恢复前缀缓存命中(HIT 轮不再因全量 body 丢缓存);存储层的水位/增量游标不受影响(窗口仍正常入库,水位照常推进)。
    7. 是否改了设计方向:没有——沿用既有的 folded 语义(foldResidentTail 对空闲会话已是同一做法,persist.ts:3714),只是把 HIT 轮的窗口钉进同一语义。
    8. 是否引入了新的配置:没有。

    收工判定:leg 1 修复 + 双向回归测试(tests/issue7346-pf-window-ledger.test.ts:非 raw 窗口保存保留前缀、raw 窗口保存照常清扫)已交付(PR #7357,typecheck 绿、定向测试 28/28 过);leg 2 在 master 上不可复现(具名符号不存在、liveCcr 覆盖已抢占所述失步),未改代码,如实报告。

  2. added 5 commits that reference this issue on Nov 20, 2026
  3. added a commit that references this issue on Nov 21, 2026
    c7e6e37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions