Halogen prompt cache: measured entry capacity Cmax and the prefix registry; expose cacheMb on the Flash lane

[mecattaf]

What the cache is doing today

MEASURED, curl -s http://worker:8731/cache at 23:17 CEST:

{"entries":8,"bytes":931117216,"last_entry_bytes":116389652,"max_entries":8,
 "hits":2455,"misses":415,"stores":3279,"evicted":3271,
 "prompt_tokens_saved":25777598,"hit_rate":0.8554}

MEASURED, curl -s http://worker:8731/health: prompt_cache {enabled: true, cap_mb: 110, mode: 2, snapshot_align: 0, bitwise_identical_to_cold: false}, slots: 4, slot_ctx: 262144, context: 262144, version.api 0.7.0.

3,271 evictions against 3,279 stores is a cache that stores each job's prompt, never gets a second call on it, and evicts it. MEASURED over all 137 completed halogen-v2 outputs in /home/tom/today/review/2026-09-19/halogen-v2/: cached_tokens is greater than zero on 137 of 137 calls, and the total cached is 107,956 tokens, a mean of 788 per call. 788 tokens is exactly /home/tom/today/drain/bench/prompt-v1.txt, which is 3,056 bytes and byte identical across every call. The batch shares its system prefix and nothing else.

Why this is worth fixing but is not the throughput lever

MEASURED over those same 137 outputs: 1,242,271 prompt tokens, 173,833 completion tokens, 4,143 seconds of call time. At the serving rates the server reports, that is 1,053 seconds of prefill at 1,180 tok/s plus 2,897 seconds of decode at 60 tok/s, which totals 3,950 seconds and closes to the measured wall to within 4.6 percent.

Decode is about 73 percent of a serial night. Prefill is about 27 percent. Prompt caching attacks the smaller share. It is still worth building, because the coding loop this design was written for reuses a large static prefix across hundreds of calls, which the harvest batch does not.

What the probe measured

One 22 call probe on an idle engine, held under a single worker-gpu lease so nothing could interleave and pollute the /cache deltas. Receipt: /home/tom/today/review/2026-09-19/kv-probe.json. Verdict: /home/tom/today/review/2026-09-19/kv-probe-verdict.md.

leg	what it answers	result
A1 at 4k, 16k, 32k, 64k	cold baseline, misses +1	4k: 3.38 s, 16k: 9.81 s, 32k: 17.16 s, 64k-char slice (43,538 prompt tokens): 33.26 s; cached_tokens absent on every cold call
A2, identical repeat immediately after	how many positions one entry actually held	2,586 of 2,593; 11,367 of 11,374; 21,330 of 21,337; 43,531 of 43,538 (0.12 to 0.19 s per call)
B, same system bytes, fresh tail	how much of the prefix a fresh call resumes. This is the number the registry depends on	2,574; 11,355; 21,318; 43,519 (0.4 to 0.69 s per call). Cmax = 43,519
C at 16k, nine distinct stores then repeat B	whether a prefix survives nine stores through an 8 entry LRU	did not survive: B2 after nine foreign prompts reports no cached_tokens and takes 9.53 s (each small prompt stores 2 entries, so 9 prompts are 18 stores through an 8-entry LRU; a prefix survives at most about 3 foreign prompts)

The six acceptance checks, all computed from the stored /cache pairs:

1. prompt_tokens_saved delta across A2 equals cached_tokens reported by A2: FAILS on timing only: the saved delta lands on the following call (A2 delta 0, B delta equals A2 cached_tokens at every size)
2. hits +1 and misses +0 across A2 and across B, misses +1 across A1: FAILS on the same one-call lag; the hits and misses sum is right across each A1, A2, B triple
3. stores delta equals evicted delta on every call after the first: passes at every size
4. entries stays 8 throughout: passes (8 throughout)
5. last_entry_bytes after the 4k store versus after the 64k store: 116,389,652 at every size, a slab; only cached_tokens answers. If it varies with size, entries are content sized and bytes per position is directly readable. If it stays 116,389,652, that field is a slab size or stale and only cached_tokens answers.
6. refused stays 0: passes (0 throughout, including the 43,538-token store). An increment on the 32k or 64k store is the 110 MB wall naming itself.

Measured entry capacity: Cmax = 43,519 tokens (leg B by size: 2,574; 11,355; 21,318; 43,519; 0.4 to 0.69 s per resumed call). last_entry_bytes by size: 116,389,652 at 4k, 16k, 32k and 64k, so the field is a slab size and only cached_tokens answers.

The registry Tom asked for

Built tonight at /home/tom/prefix-registry, content addressed, with objects/<aa>/<sha256>.txt holding the exact bytes of messages[0].content, meta/<sha>.json, measurements/<sha>-<ts>.json, and bin/{prefix-add.py,prefix-warm.py,prefix-mirror.sh}. Three rules and they are the whole design: the object file is the exact bytes, the sha is verified at use and the call fails closed on a mismatch, and a measurement belongs to a server build so /health is re-read every time. The mirror is rsync -a --delete from coordinator to worker only, and it is provenance, not a cache Halogen reads.

Honest correction to the original ask: the coordinator's 1 TB SSD buys nothing here. The registry is text and its total size is megabytes. MEASURED free space: coordinator root 439 GB, /home 246 GB, worker root 411 GB. The disk was never the constraint; the 8 entry LRU is. Graduation to /var/lib is named and not done.

Ask

Expose a cacheMb option on the Flash lane in /home/tom/mecattaf/dotfiles/modules/halogen.nix. The precedent already exists one file over: /home/tom/mecattaf/dotfiles/modules/strix.nix line 140 sets HALOGEN_CACHE_MB = "8192". The Flash lane has no equivalent, so cap_mb sits at the image default of 110 and cannot be raised without rebuilding the container, which is not a thing this fleet can do. Urgency follows the verdict: at Cmax below 4,000 the design does not reach past the system prompt on this server and the option is the only path forward.

tier: Opus

[mecattaf]

2026-09-20 one-shot migration: ~/prefix-registry moved to /.local/state/prefix-registry/ (README and bin paths repointed; MEASURED nothing in dotfiles or a user unit declared the old path: grep -rn 'home/tom/prefix-registry|/prefix-registry' over ~/mecattaf/dotfiles = 0 hits, ~/today/drain = 0 hits, ~/.config/systemd/user = 0 hits, and systemctl --user show '' -p ExecStart | grep -c prefix-registry = 0; all 14 hits were inside ~/prefix-registry itself). Repointed: README.md layout header, bin/prefix-add.py (docstring + REGISTRY default), bin/prefix-warm.py (REGISTRY default), bin/prefix-mirror.sh line 12 (REG). Left as-is on purpose: bin/prefix-mirror.sh line 14 REMOTE_PATH, which is a path on the worker host, not in this home, and the worker was not migrated; and the historical records in meta/.json, measurements/*.json and mirror.log, which describe past runs. Log: ~/today/review/2026-09-20/migration/LOG.md