Repository navigation
feat: stable head-system anchor for prefix-cache preservation (#1085) - #1092
Conversation
📦 Built Package ArtifactBranch: Option A — Install from npm PR tag (recommended)npm install -g billion-context@pr-1092Each push to this PR publishes a new version under the Option B — Download artifact
tar xzf billion-context-pr1092.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 Review complete — verified, one small fix pushed, ready for human merge. Per the [PR] wake rule I did not redo the work — I pulled the branch, reviewed the full diff against current master, and independently re-ran the pre-flight. Branch state
Code verification (beyond CI)
Pre-flight re-run on the rebased head
Issue found & fixed directly on the branch (commit
Non-blocking observations (report only)
中文摘要:审查了 #1085 的 head-system 锚定 PR(逐路径核对了四条线路的锚点注入、note/nudge 顺序、失败路径与持久化,并在 rebase 后的分支上重跑 typecheck/test/build 全绿);修复了文档中三行表格多余的行首空格(含两行无关行的无谓改动,已恢复与 master 一致并推回分支);功能设计合理、默认关闭,可以合并(按 §7.4,wire-shape 变更需人工合并:https://github.com/ranxianglei/billion-context/pull/1092)。 |
|
I think this feature needs to be taken further. You can refer to headroom's CacheAligner functionality, at: https://github.com/headroomlabs-ai/headroom |
|
For example:
At least these three features can be enhanced. Of course, the specific evaluation requires you to read headroom's code. |
[bot] 🏷 @BsoBird Thanks — before answering I read the headroom source ( 1. Volatile sub-segment detection — real gap, but the "graft" is bigger than it looks. Confirmed the failure mode against #1092's code: a per-turn-volatile system (client injecting a timestamp / session id / memory usage) makes What the headroom code actually offers: The key design constraint if we build it: masking must gate change detection, not content forwarding. Forwarding fresh volatile values each turn changes nothing — any changed byte at position P invalidates everything downstream of P, and system sits at byte zero. The mechanism is therefore: compare masked fingerprints (volatile spans blanked); masked-equal ⇒ no semantic change ⇒ keep forwarding the frozen anchor bytes, emit no note. Consequence to accept: the model sees a stale timestamp indefinitely, which argues for a bounded refresh (refresh the anchor after N turns / T idle minutes, accepting one deliberate miss) — a product call. And the matcher must be conservative: a false positive that masks a genuine AGENTS.md edit would hide a real semantic change behind the frozen anchor; the safe rule is to suppress a change only when every diff span falls inside a recognized volatile pattern. (headroom bans regex in that codebase for cold-start reasons; bili has no such constraint, so the set can be broader — monotonic counters, hex digests.) 2. Change attribution report — valid, cheap, strong precedent on both sides. headroom's 3. Feeding volatile detection into absorb priority — premise doesn't hold, I'd push back. Tool results mid-history are immutable across turns: produced once, re-sent byte-identical by the client. They cannot cause prefix drift — unlike the head system, which the client re-renders fresh each turn. The actual drift sources are: head system (#1092), client-side history rewrites (what #1001 detects), and tool-list/schema changes (client-controlled). Timestamped log content inside a tool result is stable turn-over-turn, so it does zero incremental cache damage beyond what the content already costs. Worse, folding anything mid-history busts the cache at the fold point by construction — that's inherent to compression (compress = a deliberate cache reset of the compressed span, prefix before it preserved). Absorb/fold prioritization serves window pressure, not cache stability; coupling volatile-content scoring into acp-kernel absorb priority mixes those concerns and lands as a cross-repo kernel change (manual/human territory per §7.4). For what it's worth, headroom's own answer to "the prefix went cold" is its cold-prefix recompaction hook (recompact the whole prefix once after TTL lapse) — not volatile-aware absorption either. Recommendation: ship points 1+2 as one follow-up PR, separate from #1092 — shared volatile-pattern library; masked comparison gates the anchor's change-detection while the frozen anchor keeps flowing; attribution warning rides the existing @ranxianglei — this is feature-class, so it's your call per triage rules. Happy to spec/implement the 1+2 follow-up once you confirm direction here or open the issue. 中文摘要:读了 headroom 的 cache_stabilization 源码后确认——其 E5/E6 检测器均为纯观测、绝不改写请求字节(volatile_detector 只报 timestamp/UUID/ID 字段三类模式并建议客户把易变内容移出缓存前缀;drift_detector 按会话对 system/tools/前 3 条消息做 SHA-256 指纹并点名漂移维度),所以"嫁接"的只是模式分类法,修复逻辑需新写;结论:第 1、2 点成立且有双侧先例,建议合并为一个后续 PR(掩码比较仅用于变更判定、冻结锚定字节继续原样转发、归因告警挂现有 [acp-usage]/cache-warn 通道、加有界锚刷新配置项),第 3 点前提不成立(历史中部的 tool 结果跨轮逐字节不变、不会造成前缀漂移,fold 本身就在折叠点破缓存,且属跨仓内核改动),不予采纳。 |
|
Follow-up to #1085 / #1092. Two gaps found while comparing against opencode's cache work (anomalyco/opencode#29672, #43507): 1. Volatile-subfield masking before change detection
Net effect for volatile-headed clients: a guaranteed full cache bust every 9 turns, plus a growing note tail — strictly worse than no anchoring. Proposal: compute a masked fingerprint for comparison only (normalize timestamp/UUID/monotonic-counter-shaped substrings to placeholders). 2. Anthropic-wire cache_control regression testUpdate notes are appended as trailing user messages — the same position as the opencode plugin notices that broke prompt caching in opencode#43507 (breakpoint landed on a message that can never prefix-match). bili's breakpoint placement is explicit ( Request: add an e2e on the anthropic wire with a client-sent Non-goals
|
|
so what do u think |
…otes (#1092) Session D of the #1085 e2e: a client sending cache_control breakpoints on the system block and an early history user block gets exactly those same logical blocks marked on every turn (identity-keyed carry-over through the kernel round-trip), no breakpoint ever lands on an injected [System context update] note, and steady turns keep the note count at one. Closes the opencode#43507-class regression gap flagged in review.
`[bot] 🏷 My take, point by point: 1. Volatile-subfield masking — agreed, and your framing is exactly rightMasking for comparison only while Three design constraints I'd bake into any implementation:
One open product question to flag for @ranxianglei: with masking on, the model sees the frozen volatile values indefinitely (a timestamp from turn 1 on turn 500). Acceptable for most ambient-instruction cases, but a bounded anchor refresh (age/turn cap → one deliberate miss) may want to exist as an escape hatch. My recommendation: don't build it until someone actually hits the staleness problem. Status: this is feature-class work — I won't start implementing until the owner confirms direction (and opens the issue, per repo rules). If confirmed, I'd scope it as ONE follow-up PR behind 2. Anthropic-wire cache_control regression test — done, pushedAdded as Session D of
I also read the mechanism first, and the opencode#43507 class of failure can't occur structurally in bili: notes/nudges are plain string-content user messages appended after the kernel round-trip, and Pre-flight on the pushed head: typecheck ✅, full suite 2172 pass / 0 fail / 2 pre-existing skips ✅. Merge stays with a human per §7.4. Non-goalsAgree with both: reordering client system content is the client's job (bili forwards client bytes verbatim — that's the wire-fidelity invariant), and the churn guard's replace-on-trip behavior is fine once volatile noise stops triggering it. 中文摘要:已把 anthropic 线 cache_control 断点稳定性的回归测试补进 PR 分支(c0b041c,断点按消息身份而非索引保持、永不落在注入的更新备注上,全量测试通过);volatile 子段掩码方案我认可且给出了三条实现约束(保守抑制规则、移植 headroom 三族模式库、归因日志同 PR 交付),但属新功能,需 owner 确认方向并开 issue 后再动手。 |
审查结论:HOLD 维持(与 PR 自标一致);前提更正 + 两块可拆走的 salvage深审(base 9c01dbe 对拍、锚机制逐路径走查、9/9 单测 + e2e OpenAI/Responses 复核)后的结论:维持 hold,不合。但封存理由里有一处前提需要更正,且有两块资产值得在重设计时拆走。 前提更正:#1094-hold 的"同一替换点"冲突不属于本 PRBsoBird 在 #1094 上的 hold 分析(同一替换点、互斥不变式:都在到达时替换 tool-result 字节)针对的是 #1098(pre-crush)vs #1097——不是 #1092。本 PR 的锚替换的是 head-system 字节,不是 tool-result 字节;且 #1097 自己的文本是建在 #1092 的产物之上的:设计约束 4 明说 fetched content 走 trailing request-only message(与 nudge、 真正要为 #1097 后重设计记录的摩擦点:锚 state 在 Responses 身份修复(commit 04c39e4,
|
|
@ework — please re-review the current head c0b041c in full. Additional context since your last pass: an independent deep review is at #1092 (comment) (verdict: hold maintained, with a premise correction re #1098-vs-#1097 and a salvage list; the instructions-drift bug is now #1102). Note BsoBird's CacheAligner suggestions above are also worth folding into your assessment. |
Scope of this pass
Independent verification of the deep review's claimsI checked each one against the code rather than taking them on trust:
Nothing in this pass contradicts the deep review or changes the verdict. The premise correction (#1094-hold's replacement-point conflict targets #1098-vs-#1097, not this PR) is accepted — the anchor replaces head-system bytes, not tool-result bytes, and #1097 is built on top of this PR's artifacts, so complementary-not-competing holds. Salvage status
Folding in the CacheAligner/volatile-masking threadUnder HOLD this becomes part of the redesign brief instead of a follow-up against this PR. My prior constraints stand: conservative suppression rule (suppress a turn only when every diff span falls inside a recognized volatile pattern), port headroom E5's minimal 3-family taxonomy (ISO-8601 byte-position timestamps / UUIDv4 / id-named fields — no regex, no fuzzy scoring), attribution warn-once log in the same change. One interaction worth recording for the redesign: masking makes churn-guard trips rare in practice, which simultaneously mitigates the churn-replace full-miss concern above — volatile masking and the PR remains held per your verdict; merge is human-only regardless (§7.4 wire-shape change). 中文摘要:已对 c0b041c 完成完整复审(功能代码与首轮一致、CI 绿、单测 9/9 复跑通过),并逐条独立核实了深审的全部论断(subagentNamespace 纠缠、metadata 存储位置、plugin 模式缺守卫、豁免清单、失败路径安全)——均属实,结论同意维持 HOLD;salvage 第一项已落地为独立 PR #1103(无锚版 anthropic 断点稳定性 e2e,master 上直接通过,解锁 #1097 风险项),第二项(systemNotes 通道形状)记入重设计要点、不单独发死代码。 |
Anchor-free e2e salvaged from held PR ranxianglei#1092 per review direction: client-sent cache_control breakpoints (system block + early history block) must ride the same logical blocks on every turn — identity-keyed carry-over through the kernel round-trip, invariant under legitimate head changes and history growth, never migrating to newer blocks. Prerequisite coverage named in ranxianglei#1097's risk section.
|
独立审查结论:approve(rebase 后合并,wire-shape 变更按 §7.4 仍需人工)。 我把 PR head 语义审查(通过)
|
When a client's head system/instructions block changes mid-session (e.g. an embedded AGENTS.md is edited), the forwarded prefix diverges at byte zero and the provider's KV prefix cache misses in full. With stableSystemAnchor enabled (default off; env BILI_STABLE_SYSTEM_ANCHOR=1 or config file), bili freezes each session's first-seen head-system text as a sticky anchor and re-sends those bytes verbatim; detected changes are appended as trailing '[System context update]' user messages carrying the new full text, each superseding the previous ones. The anchor and append-only log live in session metadata (persisted, compaction-proof) and are re-injected every turn across all four wire surfaces (anthropic/openai/google/responses) including internal compress-loop rounds. Churn guard: more than 8 distinct changes replace the anchor outright (one deliberate cache miss). Excluded: title-gen micro-requests, Responses compaction-trigger requests, auto-mode classifier requests. Clients implementing their own variant of this mechanism need no extra injection — their in-history updates pass through as ordinary history (covered by test). On the responses wire, anchoring keys sessions on conversation identity alone so instruction drift no longer forks into orphaned |sub:<fp> sessions.
…otes (#1092) Session D of the #1085 e2e: a client sending cache_control breakpoints on the system block and an early history user block gets exactly those same logical blocks marked on every turn (identity-keyed carry-over through the kernel round-trip), no breakpoint ever lands on an injected [System context update] note, and steady turns keep the note count at one. Closes the opencode#43507-class regression gap flagged in review.
Owner conclusion: proxy-side anchoring is a best-effort wire-layer fallback, not the root fix (that belongs client-side). Re-scoped accordingly: - Plain-proxy mode only: plugin-mode agents (x-bili-plugin) own their context management and are never anchored — avoids double-processing clients that already inject cache-friendly updates (claude-code style). Gates in prepareAnthropic/prepareOpenai/prepareGoogle/prepareResponses plus the Responses identity-derivation site. - Diff-based notes instead of full-text resend: localized head changes (file-style edits sharing >=70% of lines, LCS ratio) append a compact line diff (- removed / + added, 2-line context) against the version in effect so far; each note composes sequentially onto the previous one. - Non-localized changes (structural reshuffle, tool-def churn, timestamped banners, heads >400 lines) are adopted outright: one deliberate cache miss beats appending noise that would mislead the model about its instructions. - Removal keeps flowing as the anchor with a removal note (previously fell into the replace path). - Tests: unit suite rewritten for diff semantics (hunk format, sequential composition, non-localized adoption, churn cap, interop zero-injection); e2e heads are now multi-line AGENTS.md-style texts differing in one line, plus a plugin-mode negative session asserting verbatim passthrough. - Docs: CONFIGURATION.md/.zh-CN.md rows updated with the fallback positioning, mode restriction, eligibility thresholds, and the cache_control residual limitation.
1eb4ade to
df499d1
Compare
[bot] 🏷 Rebase completed — branch What was doneRebased the 4-commit branch onto current master (
Verified in the committed tree: zero conflict markers, hunk shape at Verification (rebased head
|
[bot] 🏷 Addendum: CI on |
…otes (ranxianglei#1092) Session D of the ranxianglei#1085 e2e: a client sending cache_control breakpoints on the system block and an early history user block gets exactly those same logical blocks marked on every turn (identity-keyed carry-over through the kernel round-trip), no breakpoint ever lands on an injected [System context update] note, and steady turns keep the note count at one. Closes the opencode#43507-class regression gap flagged in review.
Closes #1085
Problem
Every conversation starts with the client's head system block, which typically embeds volatile files (AGENTS.md & co.). When those files change mid-session, the re-sent conversation diverges at byte zero and the upstream provider's KV prefix cache misses in full — the failure mode reported in #1085.
Scope (owner conclusion)
This is a best-effort wire-layer fallback, not the root fix — the root fix belongs client-side (the client owns its history and decides how instruction changes are presented; anomalyco/opencode does this in the client). The proxy can only stop itself from letting a changed head invalidate the whole cached prefix. Per that scoping, anchoring is plain-proxy mode only: plugin-mode agents (
x-bili-plugin) own their context management and are never anchored, so clients that already inject cache-friendly updates are not double-processed.Approach
Freeze each session's first-seen head system/instructions text as a sticky anchor and keep forwarding those bytes verbatim. A detected change is appended instead as a trailing
[System context update] …user message; each note composes sequentially onto the previous one.-removed /+added, 2 lines of context).session.metadata(persisted across restarts, immune to compression/compaction) and are re-injected every turn on all four wire surfaces (anthropic / openai / google / responses), including internal compress-loop rounds.instructionschange forks into a fresh|sub:<fp>session via kernelsubagentNamespaceand orphans the anchor state exactly when needed.What changed
src/system-anchor.ts(new):reconcileSystemAnchor()+ exporteddiffLines()(LCS line diff with hunk rendering); constantsANCHOR_MAX_NOTES=8,DIFF_MIN_SHARED=0.7,DIFF_MAX_LINES=400.src/server.ts: per-surface hooks at the system-text extraction points, all gatedstableSystemAnchor && !pluginMode; anchor flows into rebuilt payloads, token estimates, and loop rounds automatically.src/config.ts: opt-instableSystemAnchor(file key) +BILI_STABLE_SYSTEM_ANCHORenv, default OFF, env wins.src/loop/*: adapters accept optionalnotes, appended as trailing user messages inbuildRequest.tests/system-anchor.test.ts(unit: capture / no-op repeat / localized diff + sequential composition / non-localized adoption / removal / churn guard / persistence round-trip / surface independence / third-party interop zero-injection) andtests/e2e-stable-system-anchor.test.ts(real proxy + mock upstream: OpenAI V1→V2→V2 byte-stable prefix with one diff note; interop session zero-injection; Responses wire lifecycle; anthropic-wire cache_control breakpoint stability — breakpoints ride the same logical blocks across turns and never land on a note, per feat: stable head-system anchor for prefix-cache preservation (#1085) #1092 review; plugin-mode negative: verbatim passthrough, zero notes).Known residual limitation
Client-placed
cache_controlbreakpoints may still misalign after a head swap — the proxy cannot move the client's breakpoints; only the head-swap itself is guaranteed stable.Rebase onto master c9e4572 (2026-09-23, head df499d1)
Rebased per review; four conflicts resolved:
src/server.ts): nested union with Responses instructions fingerprint should be an allowlist (codex/claude only), not a default — generic relay clients fork on every system-prompt change #1106 — anchor ON and non-plugin request → verbatim identity (anchor absorbs drift); anchor OFF or plugin request →instructionsFingerprintAppliesgating exactly as on master.prepareAnthropicPrepared: union of master'simageBillingForthird arg and this PR'ssystemNotes.withConversationIdNotewrap and this PR'spickAdapter(…, systemNotes)argument.config.ts/ both CONFIGURATION docs: union of adjacent fields/table rows, deduplicated.Behavior interaction with #1106 (flagged for the merge decision)
With
stableSystemAnchorON, #1106's persona split for claude-over-Responses (and the codex root lane) is bypassed: an instructions change under one session id no longer forks a|sub:<fp>session — the anchor absorbs it instead (localized change → diff note; non-localized → one deliberate anchor swap = one cache miss). That is the natural consequence of the "drift = same conversation" semantics anchoring assumes; neither PR reviewed this combination explicitly. With the knob OFF (default), #1106 behavior is unchanged.Pre-flight (rebased onto current master c9e4572)
npm run typecheck✅npm test✅ 2282 pass / 0 fail / 2 skipped (pre-existing skips)npm run build✅Wire-shape change behind an off-by-default knob → human review/merge per §7.4.