Design: on-demand vision residency (overlay) — retrospective for #72

[iamwavecut]

Design doc for #72, written after the fact (the PR skipped this step; future work starts here). Feedback shapes the PR before any merge decision.

Problem

--vision permanently spends resident VRAM that otherwise belongs to the KV pool: tower weights + the vision share of startup workspace/transient reservations, paid even when images are rare. Measured cost at 32 GiB / bf16 KV: ~36k tokens of context.

Design (implemented in #72, opt-in, defaults bit-for-bit unchanged)

1. Eviction currency = read-only weights. lm_head, token embedding, draft/MTP heads sit on VMM-backed 16-MiB chunks at stable VAs with pinned host mirrors. A window unmaps the minimal chunk prefix and remaps the same physical handles into a pre-reserved staging VA: zero device allocations inside the transaction, restore is an idempotent H2D re-upload, CUDA graphs stay captured. Read-only weights need no D2H backup — that is why they are the currency, not KV.
2. Block-streamed encode. Tower weights stream through a ~20 MiB ping-pong on a copy stream, overlapped with block compute; the borrowed footprint stays near the workspace size.
3. Exclusive residency gate. Concurrent text lanes wait and resume unharmed; their KV is never touched. Resident workspace envelope becomes text-only; vision workspace/output serve from the evicted staging.
4. Budget policy = fit-and-scale. --vision-max-merged N converts to smart-resize pixel ceilings, so oversized media downscales at preprocessing instead of being rejected (adopted from feat(serve): bound each image with a Vision-token budget #61's policy direction).
5. Targets. 27B family + 35B-A3B (endpoint ladder only; MoE blocks stay unranked — the four endpoint groups already provide far more evictable tail than any window needs).

Numbers (RTX 5090, driver 580.65, CUDA 13.1)

• bf16 max context: 171 648 → 207 936 (+21.1%); int8: both at native cap, free-after-startup 2.45 → 4.80 GiB.
• Window ≈ 400 ms per image on a 24 GiB sm_86 fork of the same code vs ~3.3 s vision TTFT; completions byte-identical between modes.

Open questions

1. Naming/UX: --vision-residency overlay vs folding into --vision=overlay?
2. Is the endpoint-only ladder acceptable long-term, or should decoder blocks get ranks now?
3. Any target you would rather keep resident-only?

[iamwavecut]

Design item from external review, parking here: encode_items_overlay allocates a fresh pinned staging buffer (~125 MB for a full-window image) inside every overlay window via cudaHostAlloc, per item per request. A reusable pinned pool sized to the largest window (allocated once at overlay setup, alongside the existing weight mirrors) would remove the per-request allocation cost and the transient host-RAM spike; needs care only around teardown ordering with the VMM pool.

[Neroued]

Thank, the idea is very good. However, fc5c483 reduced vision workspace size, so it no longer meaningfully squeezes kv capacity. I don’t think the additional complexity is necessary for now.