halogen: expose HALOGEN_CACHE_ENTRIES as `services.halogen.cacheEntries` on the Flash lane

[mecattaf]

The prompt cache on the Flash lane holds 8 entries and there is no way to change
that from nix. Measured on 2026-09-20, 8 entries is the binding constraint on the
CUBS prefix design, and the entry SIZE is not. This asks for one option, mirroring
the existing promptCache option directly above it.

What is measured

GET http://worker:8731/cache, 2026-09-20 05:0x CEST:

entries 8, bytes 931117216, last_entry_bytes 116389652, max_entries 8,
hits 2959, misses 437, stores 3821, evicted 3813, prompt_tokens_saved 26600455,
store_ms_total 157113.0, restore_ms_total 13476.2, refused 0,
rows_copied 1455, hit_rate 0.8713

stores - evicted = 3821 - 3813 = 8, exactly max_entries. Strict LRU, every
store past the eighth evicts one.

The KV probe of 2026-09-19 (receipt at
/home/tom/today/review/2026-09-19/kv-probe-verdict.md, tally job
01a0bc01-910d-7262-bcfe-cee00930bfca) measured two things that matter here.

The entry is big enough. Leg B at 64k returned cached_tokens 43,519 with a
fresh tail on the same system bytes, held in one entry whose reported size is
116,389,652 bytes. That is at most 2,674 bytes per token, so a 30,000-token repo map
is at most 80 MB and fits one 110 MB entry with 30 percent of headroom. The probe's
own verdict rule therefore does NOT ask for a cacheMb option: Cmax 43,519 is
well past the 24,000-token CUBS budget. refused stayed 0 at every size.

The entry count is not. Leg C sent nine foreign prompts with a different system
message between two identical leg B calls. The second came back with no cached
tokens at all, against 11,355 on the first. Each request stores one or two entries,
because render_prompt sets two snapshot offsets, SNAP at the end of the rendered
history and SNAP2 at the end of the system block. At two stores per request, about
four foreign requests turn over all eight slots. The probe measured that nine
certainly does; the minimum was not measured.

Why this is the thing that bites

The CUBS coding loop is one fixed repo map plus one short tail per issue. Recomputing
a 24,000-token prefix costs 19.0 s and a 30,000-token one 23.4 s, from a least
squares fit over the probe's four cold legs (prefill_s = 1.52 + tokens / 1370).
Resuming either from the cache costs 0.2 to 0.7 s.

So every time a harvest job, a drain, or an OCR call lands on the capacity-1
worker-gpu pool between two CUBS issues, the loop pays 19 seconds. Over a
100-issue pass that is the difference between a 30-second loop and a 55-second one.

Eight entries is a 2026-era default for a server expected to hold a handful of
conversations. It is the wrong size for a fleet that interleaves one large static
prefix with unrelated traffic on a single serial pool.

Batch ordering by prefix is the free mitigation and is already the documented rule in
/home/tom/prefix-registry/README.md. It is not sufficient on its own: it requires
every producer on the pool to cooperate, and a single high-priority interrupt breaks
it silently, costing 19 seconds with no error anywhere.

What is being asked

One option in modules/halogen.nix, next to promptCache at lines 308 to 322 and
following the same shape as kvSlots at 298 to 306:

cacheEntries = lib.mkOption {
  type = lib.types.nullOr lib.types.ints.positive;
  default = null;
  description = ''
    HALOGEN_CACHE_ENTRIES, the number of prompt-cache entries kept. null
    leaves the image's own value, 8. Each entry costs up to cap_mb (110 MB
    on the Flash lane), so this is a memory decision: 24 entries is about
    2.6 GiB.
  '';
};

and the matching line beside the existing HALOGEN_KV_SLOTS and
HALOGEN_PROMPT_CACHE assignments around line 464 to 468:

// lib.optionalAttrs (cfg.cacheEntries != null) {
  HALOGEN_CACHE_ENTRIES = toString cfg.cacheEntries;
}

No default change. null keeps today's behaviour exactly.

Precedent

modules/strix.nix line 140 already sets HALOGEN_CACHE_MB = "8192" on the 27B
alternate, coordinator only, with the comment that it exists "instead of auto-sizing
from MemAvailable at startup, which on a desktop would claim whatever the TTS, ASR
and browser happen not to be using at that moment". The same reasoning applies to
entry count on the Flash lane, on a worker whose MemFree is currently 1.4 GiB.

One thing to check before merging

The captured 0.7.0 frontend at
notes/references/devlogs/2h26/huion-notepad-pipeline/huion/ocr/runs/halogen-vanilla-2026-09-14/audit/serve_api.py
names HALOGEN_CACHE_ENTRIES in the cache_stats comment at line 963, describing
field 4 of the CSTAT reply as that variable. That is good evidence the variable
exists and is read. It is not proof that the image honours it when set, because the
live container's environment does not set it (read from /proc/406303/environ on
the worker: no HALOGEN_CACHE_ENTRIES, no HALOGEN_CACHE_MB, no cache path of any
kind). Confirm against the image's FLAGS surface before wiring the option, the same
way promptCache was confirmed.

Not being asked

• A cacheMb option. The probe says the cap is not the wall.
• Any change to promptCache. Mode 2 is correct here; mode 1 would zero the hit.
• Disk persistence of KV. The 0.7.0 image registers six routes, none of them a save
  or restore, and its frontend states at line 1400 that "nothing is persisted here".
  That is an upstream feature request against Peonist, not a config option.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com