Agent sibling sessions miss shared prefix without prompt_cache_breakpoint

[steve8697]

Summary

Agent clients (Hermes, OpenCode, Pi, etc.) almost never send prompt_cache_breakpoint. On current master, that means sibling conversations that share a long system/tools head always miss and re-prefill the whole head.

The official catalog already has the right hit path (shared_stable_prefix). The gap is the default OpenAI write candidate, which lands on the last content part (usually the unique user turn), not on the stable system block.

This is not a request to merge #73 / #90 as a second store. It is a request to place one automatic write candidate at the rendered system-block frontier, so the existing planner can publish a shared prefix that those clients actually share.

Measured on RTX 5090, Qwen3.8-27B NVFP4, master 3d9fda2

Serve flags (exclusive one-GPU lane): --max-concurrency 2 --kv-dtype int8 --vision --host-kv-mib 16384 --host-state-slots 8 --device-state-slots 2 --max-shared-prefixes 4 --max-private-continuations 4 --spec mtp --draft-tokens 3 --lm-head-draft, ctx/kv 245760.

Chat Completions, reasoning_effort=low, max_tokens=8. Authoritative numbers from the serve done line.

scenario	reuse=	cache/prompt	TTFT
same conversation, next turn (~2k)	private_turn_closure	2188/2209	92 ms (cold 559 ms)
16k shared system, no breakpoint, two sibling users	both root	0/16484	2363 / 2363 ms
same 16k system + prompt_cache_breakpoint on the system part	shared_stable_prefix	16466/16485	91 ms
C=2 concurrent siblings after that prefix was published	both shared_stable_prefix	16466	89 / 223 ms
same conversation, reasoning_effort low → xhigh	root	0	347 ms
3×32k private then return to A	private_turn_closure	32776/32797	125 ms (cold 5365 ms)

So: same-turn reuse is already excellent. Shared-prefix reuse is also excellent once a legal write exists. Default OpenAI implicit write does not create that write for the agent-sibling shape.

Why the default implicit candidate misses

docs/serving.md (OpenAI prompt caching): omitted prompt_cache_options creates a default implicit candidate at the latest representable content boundary. In apply_openai_prompt_cache_policy that is the last content part of the last turn — for a normal chat request, the unique user message.

Request A is then system(16k) + user("SIBA") published as one prefix. Request B is system(16k) + user("SIBB"). Exact prefix identity diverges at the last user tokens, so B is root even though 16k tokens are identical.

allow_engine_automatic_shared_prefixes is also forced off after OpenAI policy is applied (openai_common.cpp), which is consistent with “OpenAI already defined the write policy” — but that policy’s implicit target is the wrong frontier for this workload.

Fork #90 (seed_prefix / --prefix-cache-mib) solved the same gap with a separate GPU seed store. That is a second catalog and would fight the current planner. The behavior we want is only: capture at the system-block frontier using the existing shared-prefix machinery.

Suggestion (smallest change)

Keep the OpenAI explicit-breakpoint contract. Add an extra default write candidate when Chat Completions / Responses has a leading system (or tools+system) block and the client did not already mark it:

1. After rendering, take the token frontier just past the leading system / tools block (the same frontier feat(runtime): cross-request prefix seeding via a content-addressed seed store #90 used).
2. If that frontier is above a floor (256 tokens, or whatever the value model already uses), emit a SharedStablePrefix candidate with DefaultAutomatic / RequestedAutomatic evidence.
3. Do not disable the existing last-content implicit candidate; a same-turn continuation still wants a later frontier.
4. Reads of an already-published prefix should keep working without the client repeating a marker (this already matches serving.md).

That is one extra legal write (still ≤ 4). It does not add PrefixSeedStore, --prefix-cache-mib, --kv-host-cache-mib, or overlay.

Optional serve flag if you want it opt-in: --auto-system-shared-prefix default on for agent serving, off for strict OpenAI-implicit-only.

What we are not asking for

• Do not resurrect Content-addressed host KV cache: prefix restore and trajectory branching #73’s separate content-addressed host KV store next to --host-kv-mib. Same-axis, two eviction policies.
• Do not resurrect feat(runtime): cross-request prefix seeding via a content-addressed seed store #90’s device arena as a third sequence-state home.
• Overlay vision residency is a different VRAM trade; not required for this.

Private continuation + host-kv already covers “same conversation came back.” The remaining hole at C=2 is two chats, one long head, no client marker.

Happy to re-run the sibling probe on a patch.

[steve8697]

Follow-up, separate from the system-frontier write candidate:

Would you consider a later NVMe cold tier under the same Device → pinned-Host planner (parked checkpoints only, not live decode)? Consumer boxes with DDR5 + NVMe can keep more 80–120k agent sessions than --host-kv-mib alone, which raises restore probability at C=2; it does not raise simultaneous decode concurrency.

Opened as #143 so this issue can stay about the automatic system-block write.

[Astrangemaninhere]

Hi — this is exactly the gap we hit too (agent siblings re-prefilling a long system head). Before writing a PR, checking the direction with you:

• Keep the OpenAI explicit-breakpoint contract and the existing last-content implicit candidate.
• Add one extra automatic write candidate at the leading system/developer frontier: after OpenAI policy applies, if the first turn is system/developer and the client did not mark a boundary there, set its cache_boundary_after with DefaultAutomatic evidence.
• Opt-out flag --no-auto-system-shared-prefix (default on).

The engine's existing shared-prefix machinery then publishes it (shared_stable_prefix), and sibling sessions reuse it without any client marker. No second store, no new catalog.

If this matches what you had in mind, I'll open the PR (with the issue linked). If you'd prefer a token-count floor or a different frontier (e.g. tools+system), say the word.

[steve8697]

Yes — that is exactly the shape we had in mind. Thank you for checking before opening a PR.

A few preferences, none of them blockers:

• Keep both writes. Last-content implicit (same-turn continuation) plus one extra candidate at the leading stable block. Do not replace the existing implicit target.
• Frontier. system / developer is the right default. If tools render before that block in the Qwen template, the candidate should sit after the combined tools+system prefix — that is the actual shared head for agent clients. If tools are absent or already covered by an explicit breakpoint, system/developer alone is enough.
• Floor. Skip the extra write below ~256 tokens (or whatever floor the shared-prefix value model already uses). Tiny system prompts are not worth a catalog slot; our miss was 8k–16k Hermes-style heads.
• Flag. --no-auto-system-shared-prefix with the extra write on by default is the right polarity for this workload. Clients that want strict OpenAI-implicit-only can opt out.

Please go ahead and open the PR linked here. Happy to re-run the 16k sibling probe (no client breakpoint → expect shared_stable_prefix) against a patch.

[FallingSnow]

I think I'm also running into this issue. I just started using Hermes and noticed it was really slow between turns, something VSCode's copilot isn't. I looked at llama-swap and it seems cache is always missing:

Here it is when I use VSCode's copilot:

Notice how the cached column is empty when using Hermes. Built from commit 863aa8a.