[Piggidragon]

Ports the phase-aware workspace (#36, GenerelSchwerz#88) and the live-context workspace (#40) of GenerelSchwerz/llama.cpp, by Anbeeld, onto main with the lanes of --split-overlap. Both options are off by default and leave the outputs unchanged:

• --phase-aware-workspace reserves the compute buffers only for generation batches until a prompt comes. An MTP draft context computes in the compute buffers of its target instead of its own.
• --live-context-workspace sizes the attention part of the compute buffers for the used part of the KV cache instead of the whole cache.

On the 27B + MTP production setup, sharing with the draft lowers the VRAM peak by 106 MiB per GPU at every context length. At short context both options together save 212 MiB. Prompt processing is 1–2% slower. Details: docs/compute-workspace.md.

Change

ggml (8d9e11b0e), from GS

• Resizable gallocr: ggml_gallocr_set_resizable(galloc, owner).
  • It takes the size of its last reserve, and shrinks on request.
  • With an owner, a second gallocr borrows the first one's buffers. The shared buffers have the larger of the two requirements, and shrink only after both have reserved again.
  • A generation counter shows when graph addresses became invalid.
• Scheduler wrappers for the same.
• Meta buffers: the borrower gets an alias with its own tensor mapping (ggml_backend_meta_buffer_alias).
• ggml_backend_trim_pools: a meta-aware pool trim.
• tests/test-alloc-resizable.cpp: the original allocator tests, with their dummy backend. 25 pass.

CUDA (ca2a953bb)

• Pool trim: idle VMM pools are unmapped. The address range stays reserved.
• CUDA graphs: the backend's captured graphs are dropped on trim. They hold pool addresses, so a replay would hit unmapped memory. The original did not drop them.

llama (3973fec5b)

• Reserve plan per scheduler lane:
  • tokens: the generation size or n_ubatch;
  • KV cells: in power-of-two steps from 1/8 of the cache.
  • Each lane keeps its own reservation state.
• Lanes:
  • Lane 1 is reserved again only for batches of at least 64 tokens, since it computes only those.
  • The MTP draft borrows lane 0.
• Waits between target and draft (workspace_acquire): before a context allocates, sets inputs, computes or reserves, it waits for the other one if that one has graphs in flight. synchronize() covers both lanes.
• Generation change: when the other context has allocated the shared buffers again, the graphs are dropped and reserved again.
• Hysteresis:
  • A smaller reservation takes effect only after 64 calls in a row that need less. Growth takes effect at once.
  • Without it, every prompt after a short generation allocated twice, and a 256-cell start cost 8% of a 10k prompt (4 reallocations, ~160 ms each).
  • This tuning is mine; GS shrank at once and started at 256 cells.
• Bounded reserve contexts for the KV cache, ISWA, hybrid, hybrid ISWA and the hybrid indexer memory. For qwen4exp the indexer cache and its pools follow the attention cache: without that, a bounded reserve asserted in qwen4exp.cpp.
• Server: it trims the device pools when no slot is busy.
• API: llama_attach_shared_workspace, llama_contexts_share_workspace and llama_trim_transient_memory in llama-ext.h.
• Memory breakdown: it does not count borrowed buffers twice.

Results

Qwen3.8-27B UD-Q4_K_S + MTP, RTX 4070 + RTX 3060, -sm tensor -ts 0.625,0.375 -as 1,3 --split-overlap -ub 256, 2 slots of 131072 tokens, greedy:

	default	--phase-aware-workspace	both
VRAM after load, 4070 / 3060 (MiB)	11309 / 11497	10985 / 11173	10985 / 11173
VRAM peak, 1k prompt	11421 / 11603	11321 / 11503	11209 / 11391
VRAM peak, 10k prompt	11509 / 11691	11403 / 11583	11291 / 11471
VRAM peak, 95k prompt	11513 / 11695	11407 / 11589	11431 / 11639
pp 10k / 95k (t/s)	991 / 733	972 / 728	973 / 723
pp 10k after a 300-token generation	1005	996	999
tg 10k (t/s)	44.9	44.9	44.7

• Outputs: all single-slot requests, including a 300-token generation (which shrinks the buffers), are identical.
• Two slots at once: the outputs differ between runs of the default build as well, because the batches depend on when the requests arrive. The variant from the workspace runs equals one of the default runs.
• 95k with both options: 24/50 MiB above phase-aware alone. Most likely the CUDA driver keeps memory after the reallocations.

Review (reviewer-xhigh)

• [medium] fixed (59aca16d9): the kept scheduler had the hash-set size it was created with. Backend samplers, LoRA adapters or training set after creation made the graph larger, and the next reserve aborted. The context now creates a new scheduler when graph_max_nodes() exceeds it. A borrower whose owner did that keeps the old buffers until its next reserve, then borrows the new ones. Verified with the reviewer's reproducer (samplers set after the first decode: abort before, now fine), and with an owner that grows while attached: logits hash equal to the unattached run, llama and qwen35.
• [low] documented: a borrower that needs more than its owner and fails to allocate it makes the owner fail as well, until one of them reserves successfully. The original allocator keeps the failed requirement on purpose (pinned by its tests). The MTP draft never needs more than its target, so this cannot happen in the shipped flow. llama_attach_shared_workspace documents it.
• Checked without findings: two contexts sharing the workspace over llama, qwen35, gemma3, qwen4exp, glm4moe, deepseek4 and mamba2, on 1 and 2 GPUs, Meta and Meta with lanes, with shrinks, refills and borrower-driven growth.
  • The logits are bit-identical to the run without the options.
  • compute-sanitizer is clean.
  • With the generation check disabled the same run segfaults, so the check is load-bearing.

Merge with main

main (PR 18, --allreduce) is merged into the branch. The README feature list is now a table, with a row for these options.

Tests

• test-llama-archs with new configs <GPU> -paw -lcw, Meta -paw -lcw and Meta -so -paw -lcw: everything passes with seed 42. maion-coder MoE fails with some seeds on main as well.
• test-alloc-resizable: 25 pass.

🤖 Generated with Claude Code