Design: content-addressed host KV cache — retrospective for #73, relation to #51/#64

[iamwavecut]

Design doc for #73, written after the fact (the PR skipped this step; future work starts here). Overlaps #51/#64 — a synthesis proposal is at the end.

Problem

1. Any prefix lost from device residency is re-prefilled: 53 s at 118k (int8, 5090), 234 s at 134k on a 24 GiB card. 2. Trajectories forking from a shared prefix (subagents) pay the full prefix per branch.

Design (implemented in #73, opt-in --kv-host-cache-mib N, N=0 inert)

1. Chained page keys: key[i] = fold(key[i-1], token_i) over ids, types, all MRoPE axes, and per-item media SHA-256, per KV page. One 64-bit match certifies the whole prefix — the causal validity KV needs. Salted by kv-dtype/spec config + format version.
2. Pinned segment store: budget-bounded chunks; page dedup by key with refcounts; whole-segment LRU (never punctures a chain); shared pages pinned during staging so eviction cannot race a save.
3. Anchors at terminal + rewrite-checkpoint frontiers: GDN linear state, MTP watermark, boundary hidden — the sound fork points for hybrid attention.
4. Hostpack kernels: batched gather/scatter between pool planes and packed staging borrowed from the idle workspace arena.
5. Engine seam: ContentRestore reuse path planned only when resident paths miss; save at terminal completion; decode never touches the machinery.

Numbers: 118 828 tokens cold 53 379 ms → restore 422 ms (126×), byte-identical; branch switch over a shared prefix ≈ 100 ms; production soak on the sm_86 fork at 251×.

Planned follow-ups (in order; each lands only after feedback here)

1. Storage-provider split: interface (present/reserve/read/write/seal/evict/anchors, byte spans) + pinned impl → enables a file-backed provider (mmap NVMe), cache survives restarts. Matches feat(engine): host-backed KV cache - park evicted lanes to pinned RAM #64's provider direction.
2. Restore-before-planning (adopted from feat(engine): host-backed KV cache - park evicted lanes to pinned RAM #64's integration): rehydrate the lane before request planning so the unmodified planner picks append/checkpoint paths; deletes the ContentRestore special cases.
3. Interval anchors (every K pages) for mid-generation forking.

Relation to #64 (park evicted lanes)

Complementary shapes of one problem: #64 parks whole lanes by prefix identity (precise state, simple, no dedup/branching, slab per max-sequence); #73 stores content-addressed pages (dedup, branching, survives prompt rebuilds, needs retokenization stability). A workable synthesis: #64's park/restore-before-planning as the fast path, backed by a shared content-addressed page store as the substrate. Happy to converge on whichever shape you prefer.

Open questions

1. One host-cache substrate or two features?
2. Is the retokenization caveat acceptable, or should ledger token ids be the only key source?
3. Budget flag naming vs feat(engine): host-backed KV cache - park evicted lanes to pinned RAM #64's --host-kv-cache.

[iamwavecut]

Follow-up from review: --kv-host-cache-mib + --spec dflash is now rejected at startup (c9f70ab) instead of silently leaving the cache null. Implementing the real DFlash path belongs here as a design item: anchors would need to carry the drafter context alongside the GDN bundle (its backend KV span and dflash_context_frontier), and ContentRestore would need restore-or-invalidate semantics for the drafter mirroring what b1cf980 established for the lane rewrite checkpoint — a stale drafter context after a restore is the same class of foreign-state bug. Needs DFlash artifacts on hand to verify honestly, so it is deferred rather than bundled into #73.

[iamwavecut]

Two more production-facing design items surfaced by an external code review, parking them here per process:

Async save / off-stream writeback. save() currently drives the full D2H of not-yet-stored pages on the shared execution stream at terminal completion (first save of a ~120k int8 trajectory ≈ 8.6 GB ≈ 200–300 ms). Invisible at --max-concurrency 1; at 3+ lanes it is a TBT spike for neighbors whose decode steps queue behind the copies. Candidate shape: a dedicated copy stream + event-ordered staging against the workspace arena, or deferring the save until the lane is idle.

Admission policy for saves. Everything terminal is saved today, so one-shot long prompts consume budget and LRU-pressure the reusable branching roots. Candidate signals: save only rewrite-checkpoint anchors by default and terminals above a reuse-probability heuristic (e.g. seen-prefix counter), or a minimum-depth threshold; needs eviction-rate/hit-rate stats (now exported) to tune honestly.

[iamwavecut]

One more design item from a concurrency-focused review of #73 (max-concurrency 8 modeling): in-flight request coalescing for identical payloads. A cold burst of N identical requests is the feature's worst case today — no coalescing exists, prefill is lane-exclusive, so TTFT of the k-th request is k × prefill even though save dedup later collapses all N trajectories to one stored segment + one anchor. Candidate shape: key in-flight prefills by the content chain's final key and let identical waiters subscribe to the first lane's completion (then take the resident/checkpoint reuse path), rather than each paying a full prefill. Related but separate from interval anchors and async save already listed here; needs care with sampling-config divergence (identical prompt, different sampler = not coalescible for generation, only for prefill state).

[iamwavecut]

Update: in-flight coalescing of identical prompts landed on the #73 branch (293c25d5..753c64fc) rather than staying a design item — admission holds twins by content identity while a matching lane is active, and the lane's prompt-boundary anchor is published into the host cache at prefill completion (or on demand once it is generating), turning the burst into one prefill plus exact-frontier restores. Concurrency-8 numbers in the PR body: identical 256-token-generation burst went 39–49 s → 7.7–12.8 s walls. The remaining #75 items (async save, save-admission, interval anchors, DFlash path) are unchanged.

[gzenz]

Following up on the synthesis proposal.

We landed the budget-bounded pinned store (your follow-up #1, storage-provider split) as a 2nd commit on #64: one pinned buffer of --host-kv-cache-mib N (single cudaHostAlloc, coalescing free-range list); each parked entry takes only its real bytes (page count + hidden + GDN tail). Whole-entry LRU, fit-driven, with a contiguity-correct satisfiability guard (an unsatisfiable park bails out rather than wiping the cache).

Kept: session-identity keying; park/restore-before-planning (your follow-up #2, already in #64's integration). HostKvProvider is the storage seam you sketched.

Deferred: the content-addressed page substrate (chained keys, page dedup/refcounts, segment LRU, anchors, hostpack kernels) - the more complete solution for branching/subagents. It stays out of scope until forking is a real workload. The two compose: budget store as fast path, substrate as generalization.

Trade-off: footprint vs max concurrent large entries; a session larger than the budget falls back to re-prefill. We size the start script to 29645 MiB (28.95 GB) on the 49 GB box - same pinned footprint as today's 3-entry config - so the budget model holds strictly more concurrent sessions (small sessions take only their real size).

Open questions:

• Please provide Docker images #3 (flag naming): resolved - --host-kv-cache-mib N (MiB budget, 0 = off).
• feat: Memory-adaptive startup with context fallback #1 (one substrate or two): shipping the pinned budget store as the first substrate; the content-addressed store is the second; they compose.
• feat: add --kv-dtype auto fallback to int8-group64 on OOM #2 (retokenization caveat): deferred with the substrate.

We'll revisit the substrate design in this thread once forking becomes a real workload.