Skip to content

[PR] fix: evict watermark sessions incrementally at cache capacity - #6054

Merged
ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-watermark-cache-eviction
Nov 10, 2026
Merged

ranxianglei merged 2 commits into
ranxianglei:masterfrom
Afloat16:2026-11-09_fix-watermark-cache-eviction

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Problem

Admitting session 1,025 clears every existing watermark handshake.

Changes

Evict only the least recently used session and refresh recency when a known session is captured or used.

Regression coverage

A full 1,024-session table retains all but the oldest entry, refreshes recency on emit/capture/learn, and does not allocate slots for unknown reads.

Validation

Passed against exact fork commit ffb5a3dcf0ecbf07a6d28e3db54a64bd05193fa8 in an isolated GitHub Actions checkout: validation job. The targeted regression and related suites, npm run typecheck, and npm run build all passed on Node 22. This is targeted verification, not the full upstream CI matrix.

node --import ./tests/preload-storage-baseline.mjs --import ./tests/preload-test-env.mjs --import tsx --test tests/watermark-cache-eviction.test.ts tests/pf-channel.test.ts tests/msgid-rule-parity.test.ts
npm run typecheck
npm run build

Related work

Inspected pending protocol PRs #5652 and #5855; both leave the full-clear capacity behavior unchanged.

Closes #6044.

@ranxianglei ranxianglei changed the title fix: evict watermark sessions incrementally at cache capacity [PR] fix: evict watermark sessions incrementally at cache capacity Nov 9, 2026
@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Nov 9, 2026
The ranxianglei#6044 fix replaced the clear-everything-at-capacity behavior with
single-entry LRU eviction, but the MAX_TRACKED comment still described
the old 'same hygiene' rationale without naming the new policy. One-line
comment clarification; no behavioral change.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: no duplicate. Searched open+closed issues/PRs for the same failure (watermark handshake table full-clear at the 1024-session cap): only #6044 reports it and only this PR fixes it. Related-but-distinct threads, checked:

Labels applied: bug, p2, scope/wiring, cache — p2 because the trigger is conditional (≥1024 concurrent tracked conversations on one proxy process, watermark cohort only) and self-heals within one round; not a usability break.

Verification status (live, as of this comment):

  • Local targeted run on PR head ffb5a3dcf0: tests/watermark-cache-eviction.test.ts + tests/pf-channel.test.ts + tests/msgid-rule-parity.test.ts → 18/18 pass; npm run typecheck clean.
  • Upstream CI (mirror cache): 0 failed, core test lanes still running; mergeable_state: blocked pending checks/review.
  • Test-quality read: the new test has teeth — it pins exact LRU eviction order (oldest untouched entry evicted, not oldest ever), recency refresh on all three touch points (emit/capture/learn), and that unknown reads allocate nothing (the implicit size-cap proof). A regression back to full-clear would fail retained == 1024 immediately.

Full structured review (regression audit vs KDD #15, hot-file coordination with #5855) follows once CI lands.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review complete — code is ready to merge. One note first: while this review was running, the branch gained 168e8ca1cd (docs: state the LRU policy at the MAX_TRACKED constant, pushed by a parallel agent run at ~15:31 UTC). I re-ran everything at the new head; all findings below apply to 168e8ca1cd.

① Direction (issue check). #6044's defect is real and the fix hits the root cause, not a symptom: the capacity handler was tracked.clear() inside takeSlot (src/agent/pf-channel.ts:54) — admitting session 1,025 wiped every live handshake. Replacing the handler with single-entry LRU eviction removes the producer of the mass loss at the decision point. LRU matches repo precedent (prefix-affinity.ts lastSeen-LRU, served-markers.ts bounded eviction), and KDD #15 makes the consequence safe by construction: an evicted session degrades one round to the legacy full-resend path, then re-stamps from the next response.

② Regression audit (existing behaviors of the touched paths).

  • Memory bound: table stays ≤ 1024 — one entry deleted before each capacity insert. Kept.
  • Non-evicted sessions: same slot object is re-inserted, so t/ids survive touches byte-for-byte; wire output for them is unchanged (all 17 pre-existing pf-channel.test.ts rows pass unmodified).
  • pfRequestHeaders went from pure read to read+recency-refresh: audited all 3 call sites (native-intercept.ts:1668, :1900, :1919) — none relies on side-effect-free reads; unknown sids still allocate nothing (no new growth path).
  • Wire shape: identical headers/values; only which sessions keep their stamps differs. No protocol change.
  • Both compression modes: channel is client-side native-lane machinery, orthogonal to plugin/proxy mode. No interaction.
  • High-risk pattern scan: no fail-fast→swallow shift, no threshold drift (MAX_TRACKED stays 1024), no output-format change, no cache/retry invalidation.
  • Freshness vs master: master advanced past the PR base (now 4446301708). Only shared file is native-intercept.ts, where master's delta ([Bug][omp lane] lane port moves under live omp hosts, and the host's cached proxy hop survives re-targeting — every Anthropic turn dies (connection refused / SELF_SIGNED_CERT_IN_CHAIN) #4566 hop fix) leaves every pf call site untouched. I test-merged the PR onto current master: clean auto-merge, targeted suites 18/18 + typecheck green on the merged tree.

③ Code. Map insertion order as the recency order is the right O(1) mechanism; delete+set on touch is correct and spec-guaranteed. The oldest !== undefined guard is defensive but harmless. Naming/style match the file.

④ Test validity. Real behavioral test with teeth: exact LRU eviction order (oldest untouched entry, not oldest ever), recency refresh pinned on all three touch points (emit/capture/learn), unknown-read non-allocation, full-table sweep retained == 1024. A regression to full-clear fails immediately. Follows repo conventions (unique sid prefix, .ts import suffix, node:test).

⑤/⑤b Diff & quality (external bar). 2 files, +53/−4 across two commits, every line on-purpose; no dead code, no magic numbers, no new deps, no style drift. Clean.

Behavior change disclosure (old → new): at the 1024-session cap, old = admit-new clears ALL handshakes (every active session loses the fastpath for a round); new = exactly ONE least-recently-used session is evicted, and emit/capture/learn refresh recency. Intended by #6044; strictly less disruptive, self-healing within one round per KDD #15.

Coordination: #5855 (v6 reland, open) still carries the old tracked.clear() verbatim — expect a small textual conflict in pf-channel.ts when it rebases after this lands; semantic interaction is nil (its Tracked.v6 field rides the same slot object, preserved by delete+set). Merge this small PR first. #5652 is closed/reverted — no action.

CI (live-verified just now): full matrix on head 168e8ca1cd — 59 success / 29 skipped / 0 failed. PR state: open, mergeable, behind master only (no conflicts; verified mergeable-by-test above). Auto-merge gate §7.4: eligible (single module, regression test, no config/wire/kernel/load-bearing changes, pure fix:, refs #6044, clean diff) — merge itself stays human-only.


结构化八问(中文)

  1. 是什么问题:水位握手表(客户端 pf-channel,上限 1024 个会话)在满表时接纳新会话会把整张表清空,所有已建立握手的会话同时失去水位快路径。
  2. 什么原因导致的:容量策略最初实现为 tracked.clear()(最省事的有界手段),没有区分「新会话准入」与「既有会话访问」,缺少增量淘汰设计。
  3. 影响范围与严重程度:仅影响开启水位快路径的安装(默认 5% 灰度 + 手动 opt-in),且需同一 proxy 进程并发 ≥1024 个被跟踪会话才触发;被清空的会话回退 legacy 全量重发一轮后自动恢复。严重度 p2——高并发条件下有明显性能/成本问题,但自愈、不阻断使用。
  4. 怎么修复的:把容量策略从「清全表」改为「只淘汰一个最久未用(LRU)」;利用 Map 插入序作为最近使用序——emit/capture/learn 三个触点都 delete+set 把槽位移到尾部,满表准入时淘汰头部;未知读不分配槽位。内存上界仍为 1024。
  5. 是否有回归问题:逐项审过——内存上界(保持)、非淘汰会话的 stamp/id 语义(不变,同一槽位对象重插)、wire 形状(不变)、3 个调用点返回值形状(既有 17 条测试全过)、淘汰后的降级路径(KDD docs: env vars primary config method + cleanup #15 一轮收敛不变)、双压缩模式无交互、master 新合入的 native-intercept 改动不触碰 pf 调用点(已实测合并到当前 master:自动合并干净,定向测试 18/18 + typecheck 绿)。未发现回归。
  6. 是否破坏缓存:不影响前缀缓存/重试策略;变化的只是哪些会话保留水位握手(原来全丢 → 现在只丢单个 LRU)。被淘汰会话快路径失效一轮,下一轮响应重新盖戳即恢复。
  7. 是否改了设计方向:没有。同一机制内的局部修复(表容量策略),LRU 与仓库内 prefix-affinity/served-markers 先例一致,无架构/边界/数据流变化。
  8. 是否引入新配置:没有。无新 env/config/默认值;MAX_TRACKED 保持 1024。

判定:可合并。 CI 全绿(59 success / 0 failed),代码质量达外部贡献标准,方向正确、回归面已逐项清点。等 owner 手动合入即可;#5855 后续 rebase 时留意 pf-channel.ts 的小冲突。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Closing summary — review of this PR is complete; see the structured review comment above for the full ①–⑤b walkthrough and the Chinese 八问. What was done and the outcome:

  1. Duplicate screening: no duplicate — closest neighbors ([Bug][dsh] watermark fastpath never hits: the client never sends x-bili-pf-count because pfRequestHeaders is handed the pre-merge headers object #5813/[PR] fix(agent): pass merged headers to pfRequestHeaders so watermark count is emitted (#5813) #5815, [PR] feat(fastpath): v6 composite-identity incremental fastpath lane (full rework, #5576 line voided) #5652 closed/reverted, open [PR] Re-land v6 composite-identity fastpath lane — inverse of #5854 revert (base for snapshot-centric rework, #5643) #5855 v6 reland) are related-but-distinct; this thread's fix stands alone.
  2. Labels applied to the PR: bug, p2, scope/wiring, cache (judged on this diff: conditional-trigger performance defect in the client watermark channel, self-heals within one round per KDD docs: env vars primary config method + cleanup #15).
  3. Verification: targeted suites (watermark-cache-eviction + pf-channel + msgid-rule-parity) 18/18, typecheck and build green at head 168e8ca1cd; test-merge onto current master (4446301708) is clean with the same suites green on the merged tree; CI full matrix on the new head: 59 success / 29 skipped / 0 failed.
  4. Behavior change disclosed: at the 1024-session cap, old = admit-new clears ALL watermark handshakes; new = exactly ONE least-recently-used session evicted + recency refresh on emit/capture/learn. Intended by [Bug] Evict watermark sessions incrementally at cache capacity #6044.
  5. Coordination note for owner: [PR] Re-land v6 composite-identity fastpath lane — inverse of #5854 revert (base for snapshot-centric rework, #5643) #5855 still carries the old tracked.clear() — expect a small textual conflict in src/agent/pf-channel.ts when it rebases after this lands; merge this first.

Outcome: 判定可合并 — code meets the external-contribution bar, no regressions found across the audited behaviors. Merge stays human-only: #6054

@ranxianglei
ranxianglei merged commit d17cd58 into ranxianglei:master Nov 10, 2026
88 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1146-master.d17cd58 (rolling master tag).

npm install -g billion-context@0.0.1146-master.d17cd58  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cache p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Evict watermark sessions incrementally at cache capacity

2 participants