[AaronStGeorge]

Problem

On hrx-graph-develop-v2, standard GPU model loading makes the native weight representation resident up front. When a kernel requests a repacked layout, HRX materializes an additional GPU allocation while keeping the original native weights resident. A repacked weight therefore occupies both native and repacked storage, even if no selected kernel needs the native layout. The original allocation is unconditional; the repack is added on demand.

This duplication consumes memory that should be available for KV cache and execution buffers. On the tested 24 GiB RX 7900 XTX, Qwen3.8-27B Q4_K_M (16.55 GiB GGUF) and Mistral-Small-3.2-24B Q4_K_M (13.35 GiB GGUF) cannot complete Lemonade chat-long-output at context 4096 with the tested HRX offload settings, despite their model files being substantially smaller than the GPU's capacity. Baseline failures explicitly report HSA resource exhaustion while allocating the transient arena.

Solution

Materialize weights on demand and keep each requested representation resident. Ordinary imported GGUF weights describe borrowed, immutable source bytes without eagerly creating a GPU copy. When a selected command first requests a weight layout, the cache materializes that representation, allocates its GPU storage, and uploads it. Later requests and prefill/decode transitions reuse the cached allocation. Unused layouts are never materialized. Existing directly bindable coherent storage is reused, and uploaded device weights, CPU weights, and HRX host buffers retain their native-only eligibility rules.

When a second layout of the same source region is actually materialized, log both layouts and their resident byte cost once on insertion. This identifies kernels that require duplicate weight storage and should converge on a common layout. First-use packing, allocation, and upload still cost time; multiple layouts remain resident if the selected kernels require them.

We also tested unloading weights when demand changed and reloading them for the next phase, in both synchronous and asynchronous designs. Repeated rematerialization proved too slow for retained-model serving. One measured prefill/decode cycle performed 442 materializations totaling 8.142 GB: about 46.54 ms allocating, 634.73 ms uploading, and 352.81 ms in other rematerialization work, or roughly 1.03 seconds altogether. Releasing the old allocations took only about 0.63 ms. Allocation alone was not the dominant cost; repeatedly rebuilding and uploading the weights was. This revision therefore removes transition eviction, retirement polling, weight submission tracking, demand revisions, and obsolete-owner invalidation. The earlier experiments and their evidence remain preserved.

Prepared and recorded programs retain ordinary shared allocation leases. Mutable host staging, imported writeback, direct-input completion, and backend teardown keep their existing ownership rules. Validated graph-instance reuse avoids repeating full binding work on stable executions; mutable-source rebinding returns to full lookup. Both routes share the prepared cache and executor, and the cheaper cache-miss metadata/hash path is retained. A guard-only ablation measured a warm-decode slowdown, so the corrected fast path remains. This PR changes neither HRX runtime nor kernels.

Validation

Exact comparison: candidate 463fd0b285835d36cf15544b7bc121fc37400ca7 versus base f6d215d01be5c9b28350f21cf310629a6dff4b5b, both with HRX 82214dfb029975205d1c3a59164b678e2e21fc41, matching build options and frozen runtime dependencies, on RX 7900 XTX (gfx1100). No HRX runtime or kernel changes are included by this PR.

• 970 distinct HRX checks passed across graph replay and direct execution at 827a95c86; all 224 backend checks passed again after each of the small watch-capture and binding-construction follow-ups. An initial overlap-test failure was an obsolete zero-common-cache-build assertion; the corrected test verifies three shared entries and native-entry reuse. Original failure evidence is retained.
• The new graph-instance test covers distinct physical bindings, alternating instances, canonical UID stability, freeing the original graph arena, mutable offset changes and rebinding, and a new UID reusing the same node array.
• Nine final loader checks pass: AUTO/mmap and native loading in both replay modes, with a Vulkan perplexity reference. HRX/Vulkan PPL ratios are 1.002696 for AUTO and 0.999910 for native loading in both modes.
• Six paired warm tg128 invocations per model (five repetitions, settled mean of repetitions 2-5): Llama +0.146% (95% interval +0.045% to +0.248%); Qwen -0.040% (-0.200% to +0.121%). These intervals do not establish exactly zero overhead.
• All 30 pinned models were attempted: 28 completed six paired comparisons with identical response text and token counts. Qwen3.8-27B and Mistral-Small-3.2-24B failed on both builds and in independent confirmations. Baseline logs explicitly report HSA resource exhaustion; candidate logs report weight materialization failure, consistent with the prior capacity observations. They have no valid performance comparison.
• No generation or total backend request-time interval resolves a slowdown. This is not an all-model equivalence pass. Phi Mini repeats a small prefill loss: -1.175% initially, -1.280% in a fixed independent repeat (about 0.3-0.4 ms). Diagnostics localize that difference mostly to waiting for GPU work, not repeated materialization or substantial binding overhead. Qwen4 Instruct's small prefill flag did not repeat; PromptBridge's repeat is inconclusive. All original and repeat results are retained below.
• Qwen3.5-2B shows a large early-request prefill flag (-23.939%). An actual-Lemonade checkpoint probe identifies most extra host time as checkpoint vector initialization: second retained request, 40.4 MB resize takes 15.670 ms / 9,832 minor page faults versus baseline 0.934 ms / zero faults. Later cycles have zero faults and roughly 1 ms resize on both builds. Selected command programs match, and the candidate does no weight acquisition/allocation or replay building after warmup. This explains the dominant host gap, not every part of the timing difference. A separately declared five-warmup control is reported below.

The fixed uninstrumented follow-up uses five warmups and then three measured requests, six AB/BA pairs, and the mean of all three measured requests. Generation +0.058% [-0.966%, +1.092%]; prefill -0.435% [-4.757%, +4.084%]; backend TTFT +0.394% [-4.797%, +5.869%]; total backend request -0.050% [-1.030%, +0.939%]. All paired text and token counts match; frozen runtime hashes are unchanged. This is a different warmup protocol and does not replace the original one-warmup result.

All-model primary measurements and fixed independent repeats

Model / measurement	Generation t/s, base -> candidate	Change (95% interval)	Prefill change (95% interval)	Backend request time change (95% interval)
minicpm3-4b	67.835 -> 67.690	-0.214% [-0.749%, +0.323%]	-0.309% [-0.995%, +0.381%]	+0.216% [-0.317%, +0.752%]
llama-3.1-8b	123.220 -> 124.267	+0.845% [-0.770%, +2.486%]	+2.237% [+1.588%, +2.890%]	-0.876% [-2.424%, +0.696%]
qwen3-30b-a3b-instruct-2507	205.431 -> 208.489	+1.480% [-0.804%, +3.816%]	-0.117% [-0.919%, +0.691%]	-1.400% [-3.542%, +0.790%]
qwen3.5-35b-a3b-mtp	78.399 -> 78.343	-0.072% [-0.396%, +0.253%]	-0.498% [-1.650%, +0.667%]	+0.098% [-0.210%, +0.407%]
gemma-4-12b	62.423 -> 62.634	+0.338% [-0.449%, +1.131%]	+0.164% [-1.621%, +1.982%]	-0.331% [-1.129%, +0.474%]
minicpm5-1b	514.185 -> 518.157	+0.773% [+0.217%, +1.333%]	+0.182% [-1.194%, +1.578%]	-0.755% [-1.304%, -0.203%]
minicpm-v-4.6	370.746 -> 378.595	+2.127% [+0.703%, +3.570%]	+1.105% [-7.218%, +10.174%]	-2.025% [-3.424%, -0.607%]
minicpm-v-4.6-thinking	366.794 -> 377.289	+2.888% [-0.032%, +5.893%]	-0.493% [-3.798%, +2.925%]	-2.664% [-5.360%, +0.110%]
qwen3-0.6b	428.809 -> 429.365	+0.130% [-1.377%, +1.660%]	-2.479% [-11.771%, +7.792%]	-0.082% [-1.697%, +1.560%]
gemma-3-270m-it	412.265 -> 407.287	-1.233% [-3.867%, +1.473%]	-2.014% [-4.886%, +0.945%]	+1.374% [-1.357%, +4.180%]
qwen3-1.7b	307.690 -> 309.010	+0.435% [-1.699%, +2.615%]	-0.404% [-2.870%, +2.125%]	-0.416% [-2.489%, +1.700%]
qwen3-4b	174.811 -> 178.567	+2.135% [+0.371%, +3.931%]	-0.344% [-1.591%, +0.919%]	-2.042% [-3.725%, -0.329%]
qwen3-4b-instruct-2507	175.419 -> 176.399	+0.555% [-1.742%, +2.905%]	-0.675% [-1.082%, -0.267%]	-0.518% [-2.734%, +1.748%]
bonsai-8b	161.981 -> 160.777	-0.739% [-1.677%, +0.207%]	-0.331% [-0.864%, +0.204%]	+0.735% [-0.186%, +1.666%]
bonsai-4b	217.650 -> 219.594	+0.894% [-1.938%, +3.809%]	-0.164% [-1.307%, +0.992%]	-0.863% [-3.592%, +1.942%]
bonsai-1.7b	339.840 -> 339.696	-0.046% [-2.915%, +2.907%]	-4.951% [-13.271%, +4.167%]	+0.152% [-2.663%, +3.048%]
phi-4-mini-instruct	183.123 -> 183.793	+0.368% [-0.776%, +1.524%]	-1.175% [-2.010%, -0.333%]	-0.335% [-1.437%, +0.780%]
lfm2-1.2b	517.282 -> 518.328	+0.202% [+0.020%, +0.384%]	-0.291% [-2.018%, +1.467%]	-0.183% [-0.320%, -0.046%]
lfm2.5-1.2b-instruct	517.659 -> 520.137	+0.479% [+0.105%, +0.854%]	+0.607% [-2.412%, +3.719%]	-0.482% [-0.807%, -0.155%]
promptbridge-0.6b-alpha	417.393 -> 418.858	+0.349% [-1.079%, +1.798%]	-3.395% [-4.602%, -2.174%]	-0.265% [-1.672%, +1.163%]
jan-nano-128k	175.796 -> 178.985	+1.811% [-0.911%, +4.608%]	-0.195% [-1.111%, +0.730%]	-1.735% [-4.299%, +0.898%]
jan-v1-4b	175.474 -> 176.941	+0.823% [-1.961%, +3.687%]	+0.911% [-1.720%, +3.613%]	-0.823% [-3.542%, +1.972%]
llama-3.2-1b-instruct	467.456 -> 468.632	+0.254% [-0.653%, +1.169%]	-0.597% [-2.692%, +1.544%]	-0.236% [-1.141%, +0.676%]
llama-3.2-3b-instruct	223.864 -> 227.879	+1.780% [+0.018%, +3.574%]	-0.640% [-2.015%, +0.754%]	-1.700% [-3.346%, -0.025%]
smollm3-3b-128k	208.526 -> 207.881	-0.310% [-1.793%, +1.196%]	-1.179% [-4.409%, +2.160%]	+0.334% [-1.125%, +1.814%]
ministral-3-3b-instruct-2512	211.574 -> 211.936	+0.169% [-1.197%, +1.554%]	+0.057% [-1.928%, +2.083%]	-0.165% [-1.502%, +1.189%]
qwen3.5-0.8b	376.024 -> 377.633	+0.428% [-0.517%, +1.381%]	-1.033% [-5.641%, +3.801%]	-0.335% [-1.177%, +0.515%]
qwen3.5-2b	255.372 -> 259.899	+1.775% [-1.141%, +4.776%]	-23.939% [-32.536%, -14.246%]	-0.438% [-3.140%, +2.340%]
qwen3.8-27b	Unavailable	baseline/candidate failure; see notes	-	-
mistral-small-3.2-24b	Unavailable	baseline/candidate failure; see notes	-	-
qwen3-4b-instruct-2507 (independent repeat)	177.953 -> 177.453	-0.284% [-2.864%, +2.365%]	+0.352% [-0.335%, +1.043%]	+0.261% [-2.273%, +2.861%]
phi-4-mini-instruct (independent repeat)	184.105 -> 185.066	+0.531% [-1.507%, +2.612%]	-1.280% [-2.315%, -0.234%]	-0.492% [-2.477%, +1.533%]
promptbridge-0.6b-alpha (independent repeat)	417.169 -> 415.810	-0.339% [-2.660%, +2.039%]	-1.723% [-5.351%, +2.045%]	+0.390% [-1.898%, +2.731%]
gemma-3-270m-it (independent repeat)	409.854 -> 412.327	+0.603% [-0.186%, +1.399%]	+0.454% [-1.694%, +2.649%]	-0.586% [-1.453%, +0.289%]

The Lemonade protocol uses version 2026.43.3, chat-long-output, context 4096, explicit --spec-type none, --no-cache-prompt, --no-reload, one warmup and three retained requests per independent invocation. Six alternating baseline/candidate pairs per model use retained positions 2/3 as the primary estimate; position 1 remains reported separately. Reported intervals are nominal paired 95% intervals, not proof of exactly zero overhead. Instrumentation is excluded from timing runs.

Keeping multiple demanded layouts can increase steady-state GPU memory. First-use packing/allocation/upload costs remain. This change makes those costs persistent cache misses instead of recurring phase-transition work. The separate all-consumer row64 kernel experiment is not included.

Earlier asynchronous, synchronous and trusted-reuse experiments and their validation artifacts remain preserved. Historical full-tier perplexity/performance results do not establish acceptance of this revision; the broader fresh full-tier/PPL work remains tracked separately. Measurements cover standard text inference on this gfx1100 GPU, with speculative decoding disabled; they do not cover MTP modes or multimodal prompts. This PR stays a draft for human review.

Requirements

• I have read and agree with the contributing guidelines - human review/sign-off remains pending for this draft.
• AI usage disclosure: YES. Codex assisted implementation, tests, investigation, and this draft description under the contributor's direction. The contributor remains responsible for reviewing the changes and satisfying the repository's contribution requirements before merge readiness.