Skip to content

[PR] fix(fastpath): exclude id from watermark head check for pinned sessions (#7345) - #7352

Open
ranxianglei wants to merge 2 commits into
masterfrom
2026-11-20_fastpath-pinned-head-id
Open

ranxianglei wants to merge 2 commits into
masterfrom
2026-11-20_fastpath-pinned-head-id

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Summary

The watermark fastpath (full-resend watermark, KDD #15) never hits on 2x2 pinned sessions (pluginLaneServerHistory + pluginMsgIdIncremental on) — exactly the traffic it was designed to accelerate. Every turn logs fastpath miss (head-prefix-drift) and falls back to the full legacy parse.

Root cause

src/server/fastpath.ts head check compared a.id !== b.id between the fresh head cores (content-hash ids minted by openaiToCore, h_…) and the stashed seed cores. In a pinned session, prepare-openai rebinds originalMessages to the POST-PIN cores before finalizeWatermarkTurn (the pin re-keys every id to the host/ledger msgid), so the seed carries host msgids while a fresh openaiToCore parse mints content-hash ids. The two id namespaces differ structurally on every turn of a pinned session, so the comparison can never pass. Pre-pin (no 2x2) traffic keeps content-hash ids on both sides and hits fine — which is why the existing unit tests (rollout flags off, no pin) stayed green.

Fix

Exclude the id field from the head comparison. The id is not a content signal: content and role inequality are still caught (id-stripped JSON equality), and a remint with identical bytes is not a rewrite. Non-pinned behavior is unchanged — when a.id === b.id, full-JSON equality holds iff the id-stripped equality holds.

Behavior change

None to the final cores. This restores the fastpath's intended behavior for pinned sessions: previously they always missed (full parse + re-anchor every turn); now they hit the O(window) fastpath. The final cores are identical to the legacy path (the head check still verifies content/role equality; only the id-namespace difference is now ignored, which is correct because the id is derived from the content or stable with it). No wire change, no config change, no data-loss.

Tests

  • New regression test in tests/fastpath.test.ts plays the field sequence: seed with a host-id list, then a tail-only growth turn must HIT (was head-prefix-drift), and an in-place head edit must still drift.
  • Confirmed RED without the fix (the new test fails), GREEN with it (20/20 pass).
  • typecheck + build clean.

Fixes #7345

…ns (#7345)

The watermark fastpath head check compared a.id !== b.id between the fresh
head cores (content-hash ids minted by openaiToCore, h_…) and the stashed
seed cores. In a pinned session (pluginLaneServerHistory +
pluginMsgIdIncremental — the 2x2 traffic the fastpath exists to accelerate)
prepare-openai rebinds originalMessages to the POST-PIN cores before
finalizeWatermarkTurn, so the seed carries host/ledger msgids while a fresh
openaiToCore parse mints content-hash ids. The two id namespaces differ
structurally on every turn, so the comparison could never pass: every turn
of a pinned session logged fastpath miss (head-prefix-drift) and fell back
to the full legacy parse.

The id is not a content signal: exclude it from the head comparison (content
and role inequality are still caught, and a remint with identical bytes is
not a rewrite). Non-pinned traffic keeps content-hash ids on both sides, so
when a.id === b.id the full-JSON equality holds iff the id-stripped equality
holds — existing behavior is unchanged.

Regression test plays the field sequence: seed with a host-id list, then a
tail-only growth turn must HIT (was head-prefix-drift) and an in-place head
edit must still drift.

Fixes #7345
@ranxianglei ranxianglei changed the title fix(fastpath): exclude id from watermark head check for pinned sessions (#7345) [PR] fix(fastpath): exclude id from watermark head check for pinned sessions (#7345) Nov 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Review of PR #7352 — fix(fastpath): exclude id from watermark head check for pinned sessions

I pulled the branch, verified the code against the production path, and ran the targeted checks. The fix is correct and minimal.

What I verified

  • Targeted test tests/fastpath.test.ts: 20/20 pass, including the new regression test (pinned seed + tail-only growth HITS; in-place head edit still drifts).
  • RED check: with the fix reverted, the new test FAILS (expected: 'hit', actual: 'head-prefix-drift') — confirming it reproduces the reported bug.
  • Typecheck: clean (exit 0).
  • Branch freshness: merge-base == master HEAD (0 commits behind, no master churn on the touched files). No rebase needed.
  • CI: still in progress at last snapshot (13 guard checks queued; test/static-gates/red-signal show cancelled — looks like a superseded re-run). I'll confirm the final verdict once the run completes.

Why the fix is correct
The head check compares fresh head cores against the stashed seed cores. openaiToCore is deterministic per element: id = content-hash base + _N (intra-call occurrence order) and wireIndex = array position. For head-prefix elements, a full-body parse (the seed) and a head-fragment parse (the fresh) produce identical wireIndex and identical pre-pin ids. In a pinned session the only difference is the re-keyed id (host msgid vs content-hash) — exactly what the fix ignores. The id-stripped comparison still catches every content/role/shape change (the test's turn 3 proves it), and non-pinned behavior is unchanged (same content → same content-hash id, so the id-difference case can't occur there).

I also confirmed the test faithfully reproduces the production seeding path: src/server/prepare-openai.ts:207-212 does applyMsgIdPin(openaiToCore(parsed).msgs, ids) on a miss in a pinned session — identical to the test's runPinnedTurn.

标签: bug + p2 + scope/kernel — 水位快路径在固定会话上从不命中,属上下文管理的性能退化(非正确性缺陷),定级 p2。


结构化八问

  1. 是什么问题:水位快路径(KDD docs: env vars primary config method + cleanup #15 全量重发水位)在 2x2 固定会话(pluginLaneServerHistory + pluginMsgIdIncremental 都开)上从不命中——而这正是它要加速的流量。每轮都记 fastpath miss (head-prefix-drift),回退到全量 legacy 解析。

  2. 什么原因导致的:head 检查用 a.id !== b.id 比较新解析的 head cores 与暂存的 seed cores。固定会话里,prepare-openai 在 finalizeWatermarkTurn 前把 originalMessages 重绑到 pin 后的 cores(pin 把每个 id 重键为宿主/账本 msgid),于是 seed 带宿主 msgid、新解析铸内容 hash id——两个 id 命名空间在固定会话的每一轮都结构性不同,比较永远过不了。pre-pin 流量两边都是内容 hash id,命中正常,所以既有单测(灰度关、无 pin)一直是绿的。

  3. 影响范围与严重程度:影响 2x2 固定会话(插件车道,两特性都开)。严重程度:性能退化——每轮做全量 O(n) legacy 解析 + 重新锚定,而非 O(window) 快路径。不是正确性问题(最终 cores 与 legacy 路径完全一致,会话工作正常,只是更慢)。定级 p2(能用但有明显性能问题)。

  4. 怎么修复的:从 head 比较中排除 id 字段。id 不是内容信号——内容/角色/形状不等仍由「去 id 后的 JSON 相等」抓住;字节相同的重铸不算改写。非固定行为不变(当 a.id === b.id 时,全 JSON 相等 ⇔ 去 id 相等)。

  5. 是否有回归问题:无。唯一行为变化是「id 不同但内容相同」现在命中(原来 miss)——正是固定会话场景。去 id 比较仍抓住任何内容/角色/形状变化(新测试 turn 3 证明:就地 head 编辑 → 仍 drift)。非固定行为不变(同内容 → 同内容 hash id,该 case 不会出现)。无 wire/config/持久化/kernel 变更。已核实测试忠实复现生产 seed 路径(prepare-openai.ts:207-212 固定会话 miss 时做 applyMsgIdPin(openaiToCore(parsed).msgs, ids),与测试一致),且 openaiToCore 逐元素确定(对 head 前缀元素,全量解析与 head 片段解析的 wireIndex 与 pre-pin id 都相同)。

  6. 是否破坏缓存:否。只改快路径的 head 校验(命中与否),不碰前缀缓存、重试、增量游标、去重键。命中时最终 cores 与 legacy 路径一致,出站字节不变,上游前缀缓存不受影响。

  7. 是否改了设计方向:否。这是恢复快路径(KDD docs: env vars primary config method + cleanup #15)对固定会话的既定行为,无架构/边界/数据流转向。

  8. 是否引入了新的配置:否。无新 env/config/默认值。

判定:修复正确、最小、测试充分(含 RED→GREEN 回归测试)。CI 跑绿后即可合并(合并由人来,我不合)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

Follow-up: the PR head moved while I was reviewing — new commit ae774d7dea5 is a pure CI re-trigger (ci: re-trigger (initial runs mass-cancelled by concurrency race)), with zero code changes (src/server/fastpath.ts and tests/fastpath.test.ts are byte-identical to the commit I reviewed). So the review above stands unchanged.

The earlier "cancelled" test/static-gates/red-signal checks were the superseded first run; CI is now re-running cleanly on the new head. I'm watching for the final verdict and will confirm here once it lands.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b)

CI landed green. On head ae774d7dea5: 0 failed, 0 pending — all 30 checks passed or skipped as expected (12 success, 18 skipped). The skips are the gate-gated e2e lanes plus guards that correctly no-op for this diff (kernel-guard/config-surface-guard skip — no kernel/ or config-surface change; lock-sync/version-guard/lane-guard/docs guards ran clean).

Final status

  • Code review: ✅ correct, minimal, well-tested (see the 8-question review above).
  • Targeted test + RED check + typecheck: ✅ all green locally.
  • Branch: ✅ current with master (no rebase needed).
  • CI: ✅ green on the pushed head.
  • Labels: bug + p2 + scope/kernel.

The PR is ready to merge — merging is human-only, so it's in your hands: #7352

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策
(ai: vllm-qwen/qwen3.8-27b)

build-artifact red = npm publish permission 404 — not a code issue, needs owner to fix NPM_TOKEN

Bottom line: this PR's code builds successfully. The red is an npm publish registry-permission 404 that I cannot fix on the PR branch. It needs the owner to fix the NPM_TOKEN / npm account permission.

Diagnosis (each point verified live this run)

  1. Code builds fine: the build-artifact job's npm install + npm run build both pass (dist-ready: OK, 46-file tarball produced). The fastpath change in this PR has no problem.
  2. Sole failure = npm publish: npm error 404 Not Found - PUT https://registry.npmjs.org/billion-context. npm returns 404 (not 401 ENEEDAUTH) for an authenticated account that lacks publish rights to the package — i.e. the account behind NPM_TOKEN is not a maintainer of billion-context.
  3. Not transient: I re-ran the failed job on the same head (attempt 3, ~1h later, version -a3) and it reproduced the byte-identical 404.
  4. Systemic: the pr rolling tag is still at 0.0.11045-pr.7336.3b3c789 (published 06:02Z) — no same-repo PR has published in ~3h (every successful publish advances that tag). PR [PR] fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) #7336 still succeeded at 06:02Z, so something changed in NPM_TOKEN or its account's permission after ~06:02Z.
  5. Registry itself is healthy: GET registry.npmjs.org/billion-context → HTTP 200; package exists (9989 versions), maintainer is ranxianglei.

Options (need owner action)

  • A (recommended): check the repo's NPM_TOKEN secret — confirm it points to an npm token belonging to an account with publish rights on billion-context (i.e. ranxianglei, or an account added as a collaborator). If it was rotated to a token without publish rights, restore a token that has them.
  • B: if the token's account is correct, re-add that account as a maintainer/collaborator on npm (npm owner add <account> billion-context).
  • C: if you suspect a transient npm-side fault on this specific package, wait for npm to recover and re-run; but with two 404s and a healthy read path, this is unlikely — not the preferred option.

Default (if undecided): go with A — it's the only path that restores build-artifact, and it doesn't touch this PR's code.

Impact on this PR

  • This PR's code and tests are unaffected (build succeeded); the red is purely the CI publish-step infra problem.
  • Until NPM_TOKEN is fixed, every same-repo PR's build-artifact will be red (inherited). Once the token is fixed, a re-run goes green with no code change on this branch.
  • I made no changes to the PR branch (working tree clean) and pushed no commits.

八问

  1. 是什么问题:PR [PR] fix(fastpath): exclude id from watermark head check for pinned sessions (#7345) #7352 的 build-artifact check 红,但红在 npm publish 步骤的 registry 权限 404,不是本 PR 代码问题。
  2. 什么原因导致的:NPM_TOKEN 所属 npm 账号对 billion-context 无发布权限(npm 对无权限发布返回 404);该权限在 ~06:02Z(PR [PR] fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) #7336 最后一次成功 publish)之后失效,疑似 token 被轮换或账号 maintainer 权限被移除。
  3. 影响范围与严重程度:所有 same-repo PR 的 build-artifact 发布环节(约 06:02Z 起);严重程度为 CI 信号级(artifact 无法发布/安装),非用户侧功能问题;本 PR 代码本身可用。
  4. 怎么修复的:本次未修复(无法在 PR 分支修 registry 权限)——需 owner 按选项 A/B 修 NPM_TOKEN 或恢复 npm 账号发布权,修好后重跑 build-artifact 即绿。
  5. 是否有回归问题:不涉及——未改动任何代码/配置/CI 文件,无行为变更。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:本 PR 代码侧 ready(构建成功、无代码改动);build-artifact 红为 NPM_TOKEN 权限基础设施问题,已打 needs-decision 标签,等 owner 按选项 A(推荐)处理 NPM_TOKEN 后重跑即绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

1 similar comment
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

ranxianglei pushed a commit that referenced this pull request Nov 21, 2026
The #7352 absorption (cherry-pick of ba4d5a6) carried the original PR's
wrong issue reference: the pinned-session id-namespace issue is #7345
("Watermark fastpath never hits on 2x2 pinned sessions"); #7347 is the
unrelated storage-poison-row-isolation root-cause issue. Comment and test
name only — no behavior change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watermark fastpath never hits on 2x2 pinned sessions — head check compares message ids across two id namespaces

1 participant