You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Watermark fastpath never hits on 2x2 pinned sessions — head check compares message ids across two id namespaces #7345
In a plugin session with both 2x2 pin lanes on (pluginLaneServerHistory + pluginMsgIdIncremental), EVERY turn logs fastpath miss (head-prefix-drift) and the request falls back to the full legacy path. Field data: 100k-message session, ~16-20s per turn, steady state.
Trigger
Any session where stage-0 pins message ids (host msgid / ledger id namespace) — exactly the traffic the watermark fastpath was designed to accelerate.
Root cause
src/server/fastpath.ts head check compared a.id !== b.id between the fresh head cores (content-hash ids minted by openaiToCore, h_…) and the stashed seed cores. The stash re-seeds from the POST-PIN cores (prepare-openai rebinds originalMessages to s0.pinned before finalizeWatermarkTurn), whose ids are host/ledger msgids. The two namespaces are structurally different on every turn of a pinned session, so the comparison can never pass. Pre-pin (no 2x2) traffic keeps content-hash ids and hits fine — which is why unit tests with rollout flags off stayed green.
Fix
Exclude the id field from the head comparison (content/role inequality still caught; shape via head-shape; remint-with-identical-bytes is not a rewrite). Regression test in tests/fastpath.test.ts plays the field sequence (seed with host-id list → tail-only turns must HIT; in-place head edit still drifts).
Impact
Performance: pinned sessions pay full parse + re-anchor every turn instead of the O(window) fastpath. No data-loss.
查重: 不重复。 Searched open+closed issues and PRs for the watermark fastpath head-check / id-namespace bug — no existing item targets this. Nearest neighbors are distinct facets: #7219 caps the x-bili-pf-tail-ids header at 2 KiB; #5427 is the original foundation increment. No open [PR] row targets this thread.
标签: bug / p2 / scope/wiring / dsh / perf — a defect in the request-pipeline fastpath (scope/wiring); it is a performance miss (perf), not data loss, so p2 (usable-but-slow, not unusable); dsh is the primary lane the fastpath was built for.
Confirmed the root cause by tracing the code (not just the report):
src/server/fastpath.ts:335 compares a.id !== b.id || JSON.stringify(a) !== JSON.stringify(b) between the fresh head cores and the stashed seed cores.
Fresh head cores come from openaiToCore(headFragment) → content-hash ids (h_…).
The seed is set by finalizeWatermarkTurn from legacyCores: originalMessages (src/server/prepare-openai.ts:675), and originalMessages is the POST-PIN array — applyMsgIdPin (src/msgid-pin.ts:43) rewrites every id to the host/ledger msgid when pluginLaneServerHistory is on (prepare-openai.ts:207-212).
So on every turn of a 2x2 pinned session, a.id (content-hash) vs b.id (host msgid) is structurally different → head-prefix-drift every turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine — which is why the existing unit tests (seeded via the raw openaiToCore oracle, no pin) stayed green.
Plan: exclude the id field from the head comparison (content/role/shape inequality still caught), add a regression test that seeds with host-id cores so tail-only turns must HIT and an in-place head edit still drifts. Will run the targeted fastpath suite locally, then open a PR against #7345.
Root cause (verified by code trace + regression test):
The watermark fastpath head check (src/server/fastpath.ts:335) compared a.id !== b.id between the fresh head cores (content-hash ids minted by openaiToCore, h_…) and the stashed seed cores. In a pinned session (2x2: pluginLaneServerHistory + pluginMsgIdIncremental), prepare-openai rebinds originalMessages to the POST-PIN cores before finalizeWatermarkTurn, so the seed carries host/ledger msgids while a fresh parse mints content-hash ids. The two id namespaces differ structurally on every turn, so the comparison could never pass → permanent head-prefix-drift miss → full legacy parse every turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine (which is why the existing unit tests with rollout flags off stayed green).
Fix: exclude the id field from the head comparison. The id is not a content signal — content/role inequality is still caught (id-stripped JSON equality), and a remint with identical bytes is not a rewrite. Non-pinned behavior is unchanged (when a.id === b.id, full-JSON equality iff id-stripped equality).
Verification:
New regression test (tests/fastpath.test.ts) plays the field sequence: seed with a host-id list → tail-only growth must HIT (was head-prefix-drift); in-place head edit still drifts.
Confirmed RED without the fix (test fails), GREEN with it (20/20 pass).
typecheck + build clean.
Labels (judged on the PR diff):bugp2scope/wiringperfdsh.
CI: the initial runs were mass-cancelled (concurrency race on the pull_request group); re-triggered with a no-op commit, will confirm green.
CI update on PR #7352 (re-triggered run, head ae774d7):
Main CI (ci.yml): GREEN — all test lanes passed (windows-latest + ubuntu-latest, node 22/24, all shards).
All E2E workflows: GREEN — e2e-local-registry, e2e-native-dsh, e2e-native-opencode, e2e-fake-codex, e2e-native-pi, e2e-pi-codex-ws, e2e-codex-persona, e2e-dsh-plugin-lane, E2E (real codex), CI Windows codex overlay.
build-artifact (PR Build Artifact): RED — npm publish failed with E404 ('billion-context@0.0.11056-pr.7352.700bd52' is not in this registry). Unusual for a publish (should create a new version, not 404). The build itself succeeded (npm run build passed); only the publish step failed. Likely a transient npm registry issue — will re-run the job to confirm.
Core test lanes are all green. The build-artifact check is a convenience workflow (builds + publishes the PR artifact for testing); the fix itself is verified by the test lanes.
Activity
ranxianglei commentedon Nov 20, 2026
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)
查重: 不重复。 Searched open+closed issues and PRs for the watermark fastpath head-check / id-namespace bug — no existing item targets this. Nearest neighbors are distinct facets: #7219 caps the
x-bili-pf-tail-idsheader at 2 KiB; #5427 is the original foundation increment. No open[PR]row targets this thread.标签: bug / p2 / scope/wiring / dsh / perf — a defect in the request-pipeline fastpath (scope/wiring); it is a performance miss (perf), not data loss, so p2 (usable-but-slow, not unusable); dsh is the primary lane the fastpath was built for.
Confirmed the root cause by tracing the code (not just the report):
src/server/fastpath.ts:335comparesa.id !== b.id || JSON.stringify(a) !== JSON.stringify(b)between the fresh head cores and the stashed seed cores.openaiToCore(headFragment)→ content-hash ids (h_…).finalizeWatermarkTurnfromlegacyCores: originalMessages(src/server/prepare-openai.ts:675), andoriginalMessagesis the POST-PIN array —applyMsgIdPin(src/msgid-pin.ts:43) rewrites everyidto the host/ledger msgid whenpluginLaneServerHistoryis on (prepare-openai.ts:207-212).a.id(content-hash) vsb.id(host msgid) is structurally different →head-prefix-driftevery turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine — which is why the existing unit tests (seeded via the rawopenaiToCoreoracle, no pin) stayed green.Plan: exclude the
idfield from the head comparison (content/role/shape inequality still caught), add a regression test that seeds with host-id cores so tail-only turns must HIT and an in-place head edit still drifts. Will run the targeted fastpath suite locally, then open a PR against #7345.ranxianglei commentedon Nov 20, 2026
(ai: vllm-qwen/qwen3.8-27b)
Fixed and PR'd: #7352
Root cause (verified by code trace + regression test):
The watermark fastpath head check (
src/server/fastpath.ts:335) compareda.id !== b.idbetween the fresh head cores (content-hash ids minted byopenaiToCore,h_…) and the stashed seed cores. In a pinned session (2x2:pluginLaneServerHistory+pluginMsgIdIncremental),prepare-openairebindsoriginalMessagesto the POST-PIN cores beforefinalizeWatermarkTurn, so the seed carries host/ledger msgids while a fresh parse mints content-hash ids. The two id namespaces differ structurally on every turn, so the comparison could never pass → permanenthead-prefix-driftmiss → full legacy parse every turn. Pre-pin traffic keeps content-hash ids on both sides and hits fine (which is why the existing unit tests with rollout flags off stayed green).Fix: exclude the
idfield from the head comparison. The id is not a content signal — content/role inequality is still caught (id-stripped JSON equality), and a remint with identical bytes is not a rewrite. Non-pinned behavior is unchanged (whena.id === b.id, full-JSON equality iff id-stripped equality).Verification:
tests/fastpath.test.ts) plays the field sequence: seed with a host-id list → tail-only growth must HIT (washead-prefix-drift); in-place head edit still drifts.Labels (judged on the PR diff):
bugp2scope/wiringperfdsh.CI: the initial runs were mass-cancelled (concurrency race on the
pull_requestgroup); re-triggered with a no-op commit, will confirm green.八问(中文结构化结论)
h_…)和存下的种子核比 id。钉住会话里 prepare-openai 在 finalizeWatermarkTurn 前把 originalMessages 重绑到钉住后的核(id 被 applyMsgIdPin 改写成宿主 msgid),所以种子核带宿主 msgid、新解析带内容哈希 id——两套 id 命名空间在钉住会话里结构性地每轮都不同,比较永远过不了。未钉住流量两边都是内容哈希 id,所以能命中(单测关灰度、不钉住,因此一直是绿的)。判定:修复完整、回归测试双向钉住、typecheck+build 干净、diff 收敛(fastpath.ts +17/-1、fastpath.test.ts +67),可合。等 CI 绿后收工。
ranxianglei commentedon Nov 20, 2026
(ai: vllm-qwen/qwen3.8-27b)
CI update on PR #7352 (re-triggered run, head
ae774d7):build-artifact(PR Build Artifact): RED —npm publishfailed with E404 ('billion-context@0.0.11056-pr.7352.700bd52' is not in this registry). Unusual for a publish (should create a new version, not 404). The build itself succeeded (npm run buildpassed); only the publish step failed. Likely a transient npm registry issue — will re-run the job to confirm.Core test lanes are all green. The
build-artifactcheck is a convenience workflow (builds + publishes the PR artifact for testing); the fix itself is verified by the test lanes.docs(fastpath): the pinned-id head-check comments cite #7345, not #7347
fix(fastpath): exclude id from watermark head check for pinned sessio…