[rcfa]

Scope first

Like #171 and #176, this is general server code, not Strix specific: it applies to any model whose memory needs context
checkpoints (recurrent/hybrid and SWA models). I am opening it here because this fork syncs from ggml-org directly now
that halo-box/llama.cpp is archived, and because it matters most on the boxes this fork targets, where the GPU and the
host share one pool of RAM. If you would rather not carry it here, please close it; as an agent I cannot open it in
ggml-org.

This PR was written by an AI agent (Claude Code), working for @rcfa; what was and was not verified is listed below.

Overview

Context checkpoints hold the part of a slot's memory that cannot be rolled back token by token. They are written once,
when a prompt is processed, and read back only to rewind the slot when a later prompt diverges. Today they are kept in
host memory, up to --ctx-checkpoints (default 32) per slot. On a hybrid model with a large recurrent state that is
most of a busy slot's memory: on Qwen3.8-Flash-Next each checkpoint is 112.6 MiB whatever the position, so 4 slots x 32
checkpoints can hold 14 GiB.

server : keep context checkpoints in unlinked files (--ctx-checkpoints-dir). With --ctx-checkpoints-dir PATH,
each new checkpoint moves its target and draft state into a file in PATH that is unlinked at once and kept by its
descriptor. Nothing is left behind when the checkpoint is dropped, or when the server exits or crashes, and there is no
cleanup to run at startup.

• Copies of a checkpoint (the prompt cache) share the file.
• The state is read back to rewind a slot (one checkpoint, ~50-100 ms from NVMe) and to write the slot save file.
  Checkpoints restored from a slot file are moved out again one by one, so a restore does not need them all in memory.
• The small speculative state (data_spec) stays in memory.
• On filesystems with their own cache (ZFS ARC, the page cache), recently used checkpoints are still served from RAM
  while RAM is free, and evicted under pressure instead of staying pinned in the server.
• Without the flag nothing changes. On Windows the flag has no effect (the data stays in memory).

Measurements

Device:     Ryzen AI 9 HX PRO 370 / Radeon 890M (gfx1150, Strix Point - NOT Strix Halo) / Minisforum N5 PRO, BIOS 1.05
Memory:     96 GB DDR5-5600 (2 x 48 GB SO-DIMM, ECC)
Kernel:     7.1.5+kali-amd64; Vulkan RADV, Mesa 26.1.6
Build:      -DGGML_VULKAN=ON -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_BUILD_RPATH_USE_ORIGIN=ON (gcc 16.2)
Baseline:   deb5e2f50 (master), built and run in the same session
Change:     e616f82c5
Model:      Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, MTP draft (shared head), mmproj, -np 2 -kvu -c 32768
Checkpoint: 112.6 MiB each (the same at 23 tokens and at 70k)
Files on:   ZFS (raidz2 of 7 NVMe SSDs as special vdev)

Master vs this PR. One scenario, three servers in turn: master, this PR without the flag, this PR with
--ctx-checkpoints-dir. A, B, C are 12000 tokens each; 16 greedy tokens per request.

request	what it exercises	tokens computed: master / PR / PR + flag	greedy tokens
r1 A + q1	prefill, checkpoints made	12019 / 12019 / 12019	same
r2 A + q1 + answer + q2	append	20 / 20 / 20	same
r3 A + q3	rewind to a checkpoint	514 / 514 / 514	same
r4 A + q4 after slot save, erase, restore	rewind to a checkpoint from the slot file	514 / 514 / 514	same
r5 B, r6 C (unpinned)	A goes to the RAM prompt cache with its checkpoints	12019 / 12019 / 12019 each	same
r7 A + q2 (unpinned)	A comes back from the cache and rewinds	516 / 516 / 516	same

Server anonymous RSS (master / PR / PR + flag): 1095 / 1104 / 871 MiB after r3 (2 checkpoints), 2060 / 2069 /
1385 MiB at the end (6 checkpoints: 675 MiB = 6 x 112.6 MiB). With the flag the checkpoint directory looked empty
throughout (the files are unlinked), and it was still empty after kill -9 of that server.

In production (this commit on our production stack: #149's disk prompt cache, #171, #176), with a real workload: an
agent holding two conversations of 51k and 53k tokens. After it had re-read them, 19 checkpoints (2.1 GiB) were in
files and the server's anonymous RSS was 1.1 GiB; with checkpoints in memory it would have been about 3.2 GiB (earlier
the same agent's slots held 34 checkpoints, 5.4 GiB). Prompt processing ran at 149-151 t/s, as before.

With #149 (disk prompt cache) two more places read checkpoints: the cache entry is written from the checkpoint files,
and checkpoints restored from an entry are moved out again. We carry those two hunks on top of #149 and can add them to
whichever of the two PRs merges second.

Correctness

• The same greedy tokens and the same number of tokens computed for every request, with and without the flag (table).
• The rewinds compute a few hundred tokens instead of 12000, so they do use a checkpoint, also one restored from a slot
  file and one restored from the prompt cache.
• After kill -9 the checkpoint directory is empty and the space is freed (the files are unlinked).
• No ggml op, model graph or compute path changes, so test-backend-ops, the corpus comparison and llama-bench do not
  apply.

Not verified

• The tools/server pytest suite was not run (it needs Python packages and model downloads we have not installed).
• Only a hybrid recurrent model (Qwen3.8-Flash-Next); no SWA-only model.
• Only Linux (Vulkan on gfx1150). Not built on Windows or macOS; the Windows path keeps the data in memory.
• One open descriptor per checkpoint: up to n_parallel x --ctx-checkpoints, plus the copies the RAM prompt cache
  keeps. Fine under the usual 1024 limit for 4 x 32, but a large -np with the RAM cache may need a higher limit.
• The RAM prompt cache still counts a moved checkpoint at its full size against --cache-ram (unchanged, conservative).

Generated with Claude Code

[rcfa]

Thanks for rebasing. The rebased branch does not compile against the new master: common_prompt_checkpoint::load_tgt() / load_dft() return bool there, and the file-backed branch still aborted on its own and ended in return; (g++: return-statement with no value). 325d369 returns the result like the in-memory branch, so master's caller decides: abort for a checkpoint of this server, fall back to reprocessing for one restored from a slot file. Written by an AI agent (Claude Code) for @rcfa.

Verified on gfx1150 (Radeon 890M, Vulkan), Qwen3.8-Flash-Next IQ3_XXS with MTP: ckpt-spill-test.py, 12k-token prompts, 7 requests (an append, two rewinds to checkpoints, one of them after a slot save and restore, and a prompt cache round trip), RAM prompt cache. Without and with --ctx-checkpoints-dir: the same tokens computed and the same greedy tokens in all 7; anonymous RSS at the end 2052 vs 1374 MiB (6 x 112.6 MiB in files); after kill -9 no files are left in the directory. Not verified: Strix Halo, Windows.

Unrelated to this PR: the base 93e6d02 does not build examples/speculative-prefill (common_batch_clear / common_batch_add are gone); master fixed it in #182. llama-server builds.