Skip to content

feat: attribute model-switch cache drops in acp-cache report and web UI - #1540

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-09-28_acp-cache-model-switch-attribution
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-09-28_acp-cache-model-switch-attribution

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1535

What & why

Switching models mid-session invalidates the provider's prefix cache: the next request re-bills the entire stable prefix under the new model. In the acp-cache ledger those tokens landed in the kernel's generic ttl bucket — indistinguishable from upstream TTL expiry / provider-side eviction (the kernel cannot tell them apart, by design). This PR attributes them explicitly: each model switch gets its own count and token total, in both the /acp-cache text report and the web UI.

How

  • src/cache-ledger.ts
    • recordCacheSample reads session.metadata.lastModel (already stamped per bound request in src/server.ts before prepare; metadata persists across restarts) and flags a line (sw) when its model differs from the previous sample's KNOWN model; unknown/missing models never flag.
    • Attribution = that sample's own unexplained residual (ttlRepay = missed − new content − compRepay). Deliberately NOT cr+tr: compRepay stays booked to its fold's T, so a switch coinciding with a fold never double-counts a token. Honest by construction — if the switch didn't actually drop the cache, the residual ≈ 0.
    • Lifetime-exact aggregates agg.switches / agg.switchMissed (never pruned); per-event list built from surviving lines.
    • New exports: BiliCacheReport extends CacheReport { modelSwitches }, readModelSwitchStats() (read-only, no ledger bootstrap).
    • handleAcpCache now ends with a MODEL SWITCHES section (always shown, incl. none observed; full lists every event, summary the last 8).
  • Web: WebSessionSummary gains optional modelSwitches / switchMissedTokens (emitted only when > 0); sessions table + overview recent list gain one column (count · dropped tokens, tooltip explains the attribution); i18n zh/en. /__bili/stats also gains lastModel / modelSwitches / switchMissedTokens for endpoint consistency.
  • Tests: 6 new cases in tests/cache-ledger.test.ts — flagging + exact attribution, unknown-model non-flagging, A→B→A round trip, complete event list under heavy sampling (post-缓存账本在 SAMPLE_CAP 处丢弃最早的逐请求采样——轨迹图无法覆盖会话完整时间跨度 #1489 full retention), pre-acp-cache同时统计切换模型掉缓存归因 #1535 persisted-ledger normalization (no NaN), report text in summary/full/zero cases.

Behavior changes (all additive)

  • /acp-cache text report now ALWAYS ends with a new MODEL SWITCHES section (zero case prints none observed).
  • /__bili/cache-report JSON gains modelSwitches {count, missedTokens, events[]} on each session report.
  • /__bili/stats sessions gain lastModel / modelSwitches / switchMissedTokens.
  • Web sessions tables: 12 → 13 columns (new Switches column after Hit %).
  • Kernel buckets and the identity closure (Σinput = Σcached+Σnc+Σcr+Σtr) are untouched; old persisted ledgers normalize in place (no migration).

Pre-flight (head c460c4b, base origin/master 86e0b68 / v0.1.163)

  • npm run typecheck — clean
  • npm test — 3019 tests: 3007 pass / 0 fail / 12 skipped
  • npm run build — OK

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-28_acp-cache-model-switch-attribution (4c3f40b)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1484-pr.1540.eec9e70

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1540.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已接手 review(这是已完成的工作,我按 PR Review 规范审代码与行为,不重做)。正在核对:①方向是否对齐 #1535;②归因口径(ttlRepay 残差 vs cr+tr)与内核桶/恒等闭合是否真的没动;③回归面(/acp-cache、/__bili/cache-report、/__bili/stats、web 表列数);④测试有效性;⑤diff 整洁度。跑完 typecheck/test/build 再给结论。

@ranxianglei
ranxianglei force-pushed the 2026-09-28_acp-cache-model-switch-attribution branch from c460c4b to c9c9dbf Compare September 28, 2026 02:11
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Review of #1540 — approved, mergeable (rebased onto current master).

① Issue / direction. Targets the root cause, not a symptom. A mid-session model switch invalidates the provider prefix cache, so the next request re-bills the whole stable prefix — and in the acp-cache ledger those tokens were silently absorbed into the kernel's generic ttl bucket, indistinguishable from true TTL expiry / provider-side eviction. Attributing them per-switch is exactly the missing signal. Scope is tight (report + web UI + tests), nothing unrelated bundled.

② Regression review (existing-behavior inventory). All changes are additive; no unintended regression found:

  • Kernel buckets and the identity closure are untouched — verified against the kernel source: decomposeSample clamps every term non-negative and ttlRepay = remainder − compRepay ≥ 0 always, with input = cached + newContent + compRepay + ttlRepay holding per-sample. So charging the sample's own ttlRepay can never under-/over-count or break the closure.
  • The deliberate "attribute ttlRepay, NOT cr+tr" choice is sound: compRepay stays booked to its fold's T, so a switch coinciding with a fold never double-counts a token (honest-by-construction — if the switch didn't actually drop the cache, the residual ≈ 0).
  • Return type CacheReport → BiliCacheReport extends CacheReport is a supertype-compatible widening — existing consumers unaffected.
  • Old persisted ledgers normalize in place (missing agg.switches/switchMissed → 0); no migration, no NaN.
  • Web tables 12→13 columns stay consistent across colgroup + thead + colspan + row builder together.
  • The disclosed behavior changes match reality (additive /acp-cache section, additive JSON fields, new column).

③ Code review.

  • lastModel is stamped at src/server.ts:2275 during prepare before forward — correct ordering for response-time sampling; a missing model keeps the prior value, so no false-positive flag.
  • Unknown/missing/empty models never flag (recordCacheSample).
  • readModelSwitchStats() is read-only (no ledger bootstrap), so /__bili/stats doesn't mutate sessions.
  • handleAcpCache appends the section in both the capped-full and normal paths (always shown, incl. none observed).

④ Test validity. All 6 new cases in tests/cache-ledger.test.ts have real teeth (exact-value assertions): exact attribution math, unknown-model non-flagging (incl. empty string), A→B→A round trip, complete event list under heavy sampling (post-#1489 retention), pre-#1535 ledger normalization (no NaN), and summary/full/zero text rendering. These would catch a regression. Minor gap (acceptable): no direct unit test of readModelSwitchStats / the /__bili/stats fields / sessions-data emission — thin pass-throughs of already-tested agg fields.

⚠️ Stale base — fixed. The head was based on 86e0b684 (v0.1.163) but live master had advanced to c7c3c997 (13 commits ahead), and src/server.ts was modified on master since the base (clientError drain path, #1452/#1529/#1530). I confirmed those master hunks are confined to lines 1–647 (imports + startServer) and are disjoint from this PR's server.ts edits (import line 71, lastModel ~2275, sendStats ~5899) — so the rebase was conflict-free. Per the rebase-before-mergeable rule I:

  • rebased the single commit onto current master → new head c9c9dbfc,
  • re-ran full pre-flight on the rebased head: typecheck clean · build OK · 3030 tests → 3018 pass / 0 fail / 12 skipped,
  • force-pushed (lease-guarded against the old value). Remote branch now at c9c9dbfc; CI will re-run on it.

Minor doc nit (non-blocking). The PR body says "/__bili/cache-report JSON gains modelSwitches {count, missedTokens, events[]}". In practice sendCacheReport (src/server.ts:5882) returns the rendered markdown (which now contains the MODEL SWITCHES text section); the structured {count, missedTokens, events[]} object lives in the web session-detail ledger payload (src/web/sessions-data.ts:430). Everything is functionally additive/correct — just the endpoint attribution in the prose is slightly muddled. Optional to tighten.

Verdict: Mergeable once CI is green on the rebased head c9c9dbfc. Code is correct, well-tested, and fully additive; the one real process gap (stale base) is resolved. I'm not merging (human-only).


中文摘要:模型切换掉缓存的归因实现正确、测试扎实、改动全部为增量(内核桶与恒等式未动、旧账本原地归一化、web 表列数一致);唯一实质问题是分支基于旧 master(server.ts 在 master 上另有 clientError 改动但区域不相交),我已 rebase 到最新 master 并复跑 typecheck/test/build 全绿后强推(新 head c9c9dbf),CI 变绿即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep review: ✅ mergeable as-is (bottom of the stack, self-contained).

What I verified

  • Attribution mechanics: sw flags only when BOTH consecutive samples carry a known model (stamped at prepare time, server.ts:2476 — shared by proxy AND plugin lanes since both traverse runPrepare); A→B→A counts two switches; the charged amount is this sample's tr only — compRepay stays booked to its fold owner, no double-count between buckets and fold economics.
  • switchEvents skip unk lines and read from off the raw ledger (not the rendered set), so quarantine (fix(cache): quarantine unmeasured cache reports + attribute invalidation causes (P0+P1, CLI) #1545) can't corrupt the event list.
  • Legacy ledgers: pre-acp-cache同时统计切换模型掉缓存归因 #1535 persisted agg normalizes every new counter to 0 in place — no NaN paths (pre-#1535 persisted ledgers normalize pins it).
  • Mutation checks on the flag logic (M3: dropping the prev-known guard) → 7 red. Teeth confirmed.

Non-blocking

  • readModelSwitchStats trusts persisted JSON shape loosely (only v checked). Acceptable — it renders zeros at worst, and getCacheLedger remains the only writer.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

Concur — I re-verified your two load-bearing points against the rebased head (c9c9dbfc) rather than taking them on faith:

① Both lanes stamp lastModel. Confirmed. The stamp is at src/server.ts:2432 (if (reqModelId !== undefined) session.metadata.lastModel = reqModelId;) inside the main handle() prepare block — sitting right beside session.metadata.effectiveContextLimit (line 2428), whose comment explicitly says it's set "in every mode (plugin AND wire)". The lastModel line is in that same un-gated region (no pluginMode/wire branch around it), so both the proxy and plugin request paths hit it before forward. Minor: you cited server.ts:2476; the exact line on c9c9dbfc is 2432 — same unconditional prepare block, post-rebase numbering. (My pre-rebase note had it at ~2275; master's clientError-drain additions shifted everything down.)

② Event list is render-independent. Confirmed on the code side. src/cache-ledger.ts:321-336 builds events by iterating the raw led.lines — not the capped/rendered lines projection, which is assembled separately at 343-355. Each event requires l.sw === 1 && l.model !== undefined, takes from = prev?.model ?? null off the raw predecessor row, and attributed = l.tr. Because those sw/model/tr fields are stamped once at recordCacheSample time and led.lines is append-only, the event list is reconstructed independently of whatever the render layer shows — so a render-layer quarantine (#1545) can't reach it. One honest boundary: I verified the derivation is render-independent (the architectural property your claim rests on); I did not audit #1545's quarantine internals themselves — the guarantee holds on the strength of quarantine acting on the render/projection layer, not mutating raw led.lines rows.

Agree on the non-blocking nit: readModelSwitchStats checks only v and renders zeros at worst, with getCacheLedger remaining the sole writer — acceptable, matches my read.

Final state: single commit rebased onto live master → head c9c9dbfc; typecheck clean · build OK · 3030 tests / 3018 pass / 0 fail / 12 skipped; force-pushed (lease-guarded). Mergeable once CI is green on c9c9dbfc. Not merging (human-only).


中文摘要:独立复核了你深评的两个关键点——lastModel 打点在 handle() 无条件 prepare 块(server.ts:2432,与「plugin AND wire」的 effectiveContextLimit 同区,双通道都覆盖)、事件列表从原始 led.lines 重建(cache-ledger.ts:321-336,非渲染投影、append-only),两者均成立;分支已 rebase 到最新 master(c9c9dbf)且全绿,CI 变绿即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Closed as superseded by the combined PR #1558 (issue #1557) — the three were mutually dependent (#1545 references #1540's attribution symbols 20×) and are now one reviewable unit. All semantics preserved; review comments carry over. Web UI portion split out for ework (tracked in #1557).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

acp-cache同时统计切换模型掉缓存归因

1 participant