Skip to content

Watermark fastpath never hits on 2x2 pinned sessions — head check compares message ids across two id namespaces #7345

Closed

Description

@ranxianglei

Phenomenon

In a plugin session with both 2x2 pin lanes on (pluginLaneServerHistory + pluginMsgIdIncremental), EVERY turn logs fastpath miss (head-prefix-drift) and the request falls back to the full legacy path. Field data: 100k-message session, ~16-20s per turn, steady state.

Trigger

Any session where stage-0 pins message ids (host msgid / ledger id namespace) — exactly the traffic the watermark fastpath was designed to accelerate.

Root cause

src/server/fastpath.ts head check compared a.id !== b.id between the fresh head cores (content-hash ids minted by openaiToCore, h_…) and the stashed seed cores. The stash re-seeds from the POST-PIN cores (prepare-openai rebinds originalMessages to s0.pinned before finalizeWatermarkTurn), whose ids are host/ledger msgids. The two namespaces are structurally different on every turn of a pinned session, so the comparison can never pass. Pre-pin (no 2x2) traffic keeps content-hash ids and hits fine — which is why unit tests with rollout flags off stayed green.

Fix

Exclude the id field from the head comparison (content/role inequality still caught; shape via head-shape; remint-with-identical-bytes is not a rewrite). Regression test in tests/fastpath.test.ts plays the field sequence (seed with host-id list → tail-only turns must HIT; in-place head edit still drifts).

Impact

Performance: pinned sessions pay full parse + re-anchor every turn instead of the O(window) fastpath. No data-loss.

Activity

  1. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 Searched open+closed issues and PRs for the watermark fastpath head-check / id-namespace bug — no existing item targets this. Nearest neighbors are distinct facets: #7219 caps the x-bili-pf-tail-ids header at 2 KiB; #5427 is the original foundation increment. No open [PR] row targets this thread.

    标签: bug / p2 / scope/wiring / dsh / perf — a defect in the request-pipeline fastpath (scope/wiring); it is a performance miss (perf), not data loss, so p2 (usable-but-slow, not unusable); dsh is the primary lane the fastpath was built for.

    Confirmed the root cause by tracing the code (not just the report):

    • src/server/fastpath.ts:335 compares a.id !== b.id || JSON.stringify(a) !== JSON.stringify(b) between the fresh head cores and the stashed seed cores.
    • Fresh head cores come from openaiToCore(headFragment) → content-hash ids (h_…).
    • The seed is set by finalizeWatermarkTurn from legacyCores: originalMessages (src/server/prepare-openai.ts:675), and originalMessages is the POST-PIN array — applyMsgIdPin (src/msgid-pin.ts:43) rewrites every id to the host/ledger msgid when pluginLaneServerHistory is on (prepare-openai.ts:207-212).
    • So on every turn of a 2x2 pinned session, a.id (content-hash) vs b.id (host msgid) is structurally different → head-prefix-drift every turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine — which is why the existing unit tests (seeded via the raw openaiToCore oracle, no pin) stayed green.

    Plan: exclude the id field from the head comparison (content/role/shape inequality still caught), add a regression test that seeds with host-id cores so tail-only turns must HIT and an in-place head edit still drifts. Will run the targeted fastpath suite locally, then open a PR against #7345.

  2. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Fixed and PR'd: #7352

    Root cause (verified by code trace + regression test):
    The watermark fastpath head check (src/server/fastpath.ts:335) compared a.id !== b.id between the fresh head cores (content-hash ids minted by openaiToCore, h_…) and the stashed seed cores. In a pinned session (2x2: pluginLaneServerHistory + pluginMsgIdIncremental), prepare-openai rebinds originalMessages to the POST-PIN cores before finalizeWatermarkTurn, so the seed carries host/ledger msgids while a fresh parse mints content-hash ids. The two id namespaces differ structurally on every turn, so the comparison could never pass → permanent head-prefix-drift miss → full legacy parse every turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine (which is why the existing unit tests with rollout flags off stayed green).

    Fix: exclude the id field from the head comparison. The id is not a content signal — content/role inequality is still caught (id-stripped JSON equality), and a remint with identical bytes is not a rewrite. Non-pinned behavior is unchanged (when a.id === b.id, full-JSON equality iff id-stripped equality).

    Verification:

    • New regression test (tests/fastpath.test.ts) plays the field sequence: seed with a host-id list → tail-only growth must HIT (was head-prefix-drift); in-place head edit still drifts.
    • Confirmed RED without the fix (test fails), GREEN with it (20/20 pass).
    • typecheck + build clean.

    Labels (judged on the PR diff): bug p2 scope/wiring perf dsh.

    CI: the initial runs were mass-cancelled (concurrency race on the pull_request group); re-triggered with a no-op commit, will confirm green.


    八问(中文结构化结论)

    1. 是什么问题:水位快路径(full-resend watermark)在 2x2 钉住会话(pluginLaneServerHistory + pluginMsgIdIncremental 同开)上永远不命中,每轮都回退到全量 legacy 解析,10 万消息会话每轮 16-20s。
    2. 什么原因导致的:头部校验拿新解析的头部核(openaiToCore 铸的内容哈希 id,h_…)和存下的种子核比 id。钉住会话里 prepare-openai 在 finalizeWatermarkTurn 前把 originalMessages 重绑到钉住后的核(id 被 applyMsgIdPin 改写成宿主 msgid),所以种子核带宿主 msgid、新解析带内容哈希 id——两套 id 命名空间在钉住会话里结构性地每轮都不同,比较永远过不了。未钉住流量两边都是内容哈希 id,所以能命中(单测关灰度、不钉住,因此一直是绿的)。
    3. 影响范围与严重程度:受影响面 = 所有 2x2 钉住会话(正是快路径设计要加速的流量,dsh 插件会话为主)。严重程度 = 性能退化(每轮付全量解析+重锚定,而非 O(window) 快路径),无数据丢失、无可用性缺陷,故 p2。
    4. 怎么修复的:头部比较里排除 id 字段(id 不是内容信号)。内容/角色不等仍由去 id 后的 JSON 相等性抓到;同字节重铸不算改写。未钉住行为不变(a.id===b.id 时全 JSON 相等 ⟺ 去 id 相等)。
    5. 是否有回归问题:审过既有行为——未钉住流量(内容哈希 id 两边一致)行为完全不变;钉住会话从「每轮 miss」变为「命中快路径」,最终核与 legacy 路径一致(头部校验仍验内容/角色相等,只忽略 id 命名空间差异)。无 fail-fast 变吞错、无阈值/时序/输出格式漂移。新增回归测试双向钉住(尾部增长命中 + 头部原地编辑仍漂移)。
    6. 是否破坏缓存:不涉及。快路径命中/回退的判定逻辑未变,只是头部 id 比较从「永远 miss」修正为「按内容判定」;水位戳(x-bili-pf-seq/count/tail-ids)协议未动。
    7. 是否改了设计方向:不涉及。没有架构/边界/数据流转向,只是把快路径的头部校验修回到它本来的判定意图(按内容而非 id 命名空间)。
    8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

    判定:修复完整、回归测试双向钉住、typecheck+build 干净、diff 收敛(fastpath.ts +17/-1、fastpath.test.ts +67),可合。等 CI 绿后收工。

  3. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    CI update on PR #7352 (re-triggered run, head ae774d7):

    • Main CI (ci.yml): GREEN — all test lanes passed (windows-latest + ubuntu-latest, node 22/24, all shards).
    • All E2E workflows: GREEN — e2e-local-registry, e2e-native-dsh, e2e-native-opencode, e2e-fake-codex, e2e-native-pi, e2e-pi-codex-ws, e2e-codex-persona, e2e-dsh-plugin-lane, E2E (real codex), CI Windows codex overlay.
    • build-artifact (PR Build Artifact): RED — npm publish failed with E404 ('billion-context@0.0.11056-pr.7352.700bd52' is not in this registry). Unusual for a publish (should create a new version, not 404). The build itself succeeded (npm run build passed); only the publish step failed. Likely a transient npm registry issue — will re-run the job to confirm.

    Core test lanes are all green. The build-artifact check is a convenience workflow (builds + publishes the PR artifact for testing); the fix itself is verified by the test lanes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions