Future: NVMe cold tier under Device/Host planner for parked continuations

[steve8697]

Related: #142 (system-frontier write candidate). This is a later capacity idea, not a substitute for that serve-layer fix.

What we are asking

Would you consider, after the current Device + pinned-Host catalog is settled, a cold NVMe tier under the same planner — not a second cache product.

Today the degradation graph is already monotonic:

Device replica → pinned Host State/KV → drop checkpoint → evict owner

A natural extra step is:

Device (hot) → pinned DDR5 Host (warm, --host-kv-mib) → NVMe (cold) → evict

vLLM/LMCache, TensorRT-LLM host offload, and NVIDIA ICMSP all use this shape for parked context. NInfer is stricter (exact frontier + full StateImage + GDN recurrent bundle), so a disk tier only makes sense if it stores the same checkpoint object the Host extent store already knows, not raw KV pages alone.

What it would help, and what it would not

Measured on this box (RTX 5090 32 GiB, WSL2 64 GiB DDR5, C=2, INT8, Qwen3.8-27B NVFP4):

already true today	SSD/DDR5 would not change
Live decode stays on Device	Tokens/s of an active lane
--max-concurrency 2 is the GPU lane cap	Two 150k chats decoding at once (KV pool is still one 245k Device pool)
Same-turn private_turn_closure is already ~90–125 ms	First-token of a hot continuation

gap	why a cold tier might help
Returning to a parked 80–120k agent chat after other sessions filled Device+Host	Avoid 20–50 s root re-prefill. Fork #73 already showed 118k Device miss → host restore 422 ms vs 53 s. NVMe would be slower than pinned DDR, but still far cheaper than prefill if sequential restore is ~1–3 s for a few GiB.
More retained session count at C=2	C=2 only has 2 active + a handful of Device/Host checkpoints. Extra cold slots raise the chance the next Hermes/OpenCode window is a restore instead of root. That is reuse probability, not live batch width.
Shared system prefix once #142 exists	One published 8–30k shared_stable_prefix could sit in Host or NVMe and seed many siblings without occupying Device KV.

So: more parked conversations survive, not higher simultaneous decode concurrency. Worth saying explicitly so this is not read as “SSD makes C=8 on 32 GiB”.

Constraints that match current contracts

From docs/serving.md and resource-scheduling-and-context-cache.md:

• A reusable checkpoint is KV and complete continuation state (Linear Attention image, MTP watermark, boundary hidden). Disk must round-trip that bundle or the restore is unsound (this is exactly the class of bug #73 hit when only part of the slot was restored).
• Pressure already compares immediate transfer work vs later recovery / re-prefill. A disk target is another cost coefficient (context-cost-transfer), not a new policy language.
• Reads stay exact-identity. No approximate / block-hash prefix cache.
• Live decode must never stream KV from NVMe. Cold tier is admission-time materialize-to-Device/Host, then run.
• Opt-in, default off: --cold-kv-dir PATH --cold-kv-gib N. Empty path = current two-tier behavior.
• On WSL2, the directory must be Linux ext4 (~/…), not /mnt/c NTFS.

Suggested shape (if you ever do it)

Keep one catalog. Add one placement:

tier	media	role
Device	GPU HBM	active + hottest checkpoints
Host	pinned DDR5 (--host-kv-mib)	warm park, sub-second restore
Cold	NVMe files / extent log	overflow of Host, sequential dump/restore of sealed checkpoints

Evict Host → Cold as a planner successor of “Device → Host-only”. Restore Cold → Device (or Host) as NeedsTransfer. If restore cost ≥ predicted prefill, skip and root (same degrade-instead-of-crash rule as #73 after the stale-plan fix).

Do not bolt on #73’s separate content-addressed store next to HostKVArena. Same axis, two LRUs.

Why we are filing this now

Single-user C=2 agent serving (Hermes / OpenCode / Pi) already wants more than 2–4 retained 100k histories. Pinned 16 GiB Host is the right warm tier; consumer NVMe is the only place left once that fills. Happy to prototype measurements on 5090 if a disk placement exists; not asking for it before #142.

No local numbers for NVMe restore yet — we only have Device/Host. This is a design vote, not a perf claim.

[Astrangemaninhere]

Hi — we implemented this proposal as PR #149 (ColdPolicy::Disk).

Scope matches the design here: parked cold pages spill to per-layer files (compressed slots, slot * stride offsets), the device slot pool becomes a working set, and the warm path restores before decode. Opt-in via --cold-policy disk --cold-disk-path ..., default off (two-tier behavior unchanged). Functional verification on RTX 5090: egress mirrored 17 pages (1.3 MB/layer, exact stride), restore + prefix reuse work (4.3k prompt 2.5 s first / 0.9 s reuse).

@Neroued — happy to adjust scope or fold this into a different shape if you'd prefer; the PR is rebased on current master and clean.

[steve8697]

Thank you for turning this into a concrete patch so quickly, and for measuring it on a 5090.

We saw PR #149 was closed unmerged. We do not know the maintainer’s reason and are not asking to reopen that branch as-is.

What we still think is worth keeping on the table, if the direction is acceptable:

• One extra placement on the existing Device → pinned-Host degradation graph (parked checkpoints only, default off). Not a second catalog.
• A narrow change is easier to review than a branch that also carries YaRN, DFlash2, per-layer tables, and the broader cold-pool work. Those may be valuable on their own; they are not required for this issue.
• The 4.3k / 2.5s → 0.9s run is a useful functional smoke. The case that motivated Future: NVMe cold tier under Device/Host planner for parked continuations #143 is a parked 80–120k agent session returning after Device+Host are full, versus a 20–50s root re-prefill.
• One caveat from the PR discussion matters for our workload: if a catalogued shared_stable_prefix cannot round-trip through cold pages, sibling reuse (Agent sibling sessions miss shared prefix without prompt_cache_breakpoint #142) and the disk tier would not compose. Exact identity + full StateImage still has to survive the file.

Deferring to @Neroued on whether ColdPolicy::Disk is the right substrate, or a Host-extent successor. Either way we are glad the idea is implementable; no rush on our side.

[Astrangemaninhere]

@steve8697 Since this feature seems not planned for inclusion here,I've implemented it in my fork:https://github.com/Astrangemaninhere/ninfer-fusion
You may check it if you like.

[steve8697]

Thanks for the pointer to ninfer-fusion.

We will look at the Disk path there when we have a spare GPU window. Daily serving here stays on the existing :9810 fork (seed_prefix / content_restore) until something like this lands on official master in a narrow form — we are not switching the daily lane onto a third tree.

If we do exercise the cold tier, the number we care about is still a parked 80–120k return versus root re-prefill, not the 4.3k smoke. No rush.