qwen36: Vulkan expert tier, and staged device-local uploads for cards without Resizable BAR

[crichalchemist]

Summary

Two things, one PR because the second is what makes the first measurable on most older discrete cards.

Qwen3.6 Vulkan expert tier. make qwen36 VK=1 builds the existing qwen36_tier.c against the shared Vulkan backend. The placement logic (warmstart, heat file, upload queue, hit/miss accounting) is unchanged; a compile-time shim of twelve one-line wrappers selects CUDA or Vulkan (#if COLI_CUDA / #elif COLI_VULKAN, CUDA wins if both). On Vulkan the tier is single-device and fills once at warmstart: the Vulkan weight arena never reclaims a freed slice, so runtime LFRU swaps would leak one expert of VRAM each. Heat still accumulates and saves, so the second run starts hot. Budget is VK_EXPERT_GB (auto = device budget minus 1 GB). The engine names its backend in the banner and, on a build without the tier, says so once when COLI_VULKAN is set instead of silently ignoring it (refs #894).

Staged device-local uploads in backend_vulkan.c. Without Resizable BAR the HOST_VISIBLE|DEVICE_LOCAL type is a ~256 MB window and everything past it silently spills to system RAM, so the tier reports resident experts that every access fetches over PCIe. When the host-visible slice is under a quarter of VRAM (or COLI_VK_STAGED=1), resident weights now go to a DEVICE_LOCAL-only arena through a host staging buffer and vkCmdCopyBuffer. Scratches, the KV mirror and readbacks keep their memory types. Queue submits and arena allocation are mutex-protected so the tier's upload thread is safe next to the decode thread.

Validation

Hardware: AMD Radeon RX 580 8 GB (Polaris, gfx803, 256 MB host-visible BAR), Mesa 25.2.8 RADV, i7-7700K, 62 GB RAM, Ubuntu 24.04. First validation of the Vulkan backend on Polaris.

• Backend exactness harness (-DVK_TEST), mapped vs staged: 50/50 maxrel values identical; staged batched int4 matmuls ~4× faster on this card (0.17 vs 0.66 ms/matmul).
• New gate tests/test_qwen36_tier_vk (part of make check; skips without VK=1 or a device): GPU output vs CPU int4 reference maxrel 6.7e-06, misses return no mask bit, residency frozen after warmstart.
• Engine, Qwen3.6-35B-A3B int4-gs64, 15-token prompt, N_NEW=64, 8 threads, greedy, frozen HEAT_FILE on runs 2–3 and the mapped arm, GPU clocks not pinned (no root):

arm	tok/s	TTFT	VRAM-resident	VRAM hit
CPU only	0.63	44.7 s	—	—
Vulkan, cold heat	6.40	1.50 s	3,663 / 10,240	36.1 %
Vulkan, warm heat (2 runs)	6.38 / 5.60	1.48 / 1.50 s	3,655 / 10,240	96.8 %
Vulkan, mapped path (COLI_VK_STAGED=0)	2.44	2.48 s	3,655 / 10,240	96.8 %

Runs 2 and 3 are token-identical; CPU vs GPU output is token-identical on this prompt. Measured on the tree rebased onto current dev (tier commit bb16ab3); an earlier measurement of the same code before the rebase read 7.5 / 4.2 tok/s for the warm and mapped arms, so treat the absolute numbers as ±15 % (clocks unpinned) and the CPU→Vulkan and mapped→staged ratios as the result.

• Tiny-fixture token-exact check 16/16 on both the plain and the VK=1 build (done manually: there is no qwen36-tiny-check target).
• make check green; make qwen36, make qwen36 VK=1, make colibri VK=1 build with 0 warnings from the touched files. CI's Vulkan job now also builds qwen36 VK=1 and runs the tier gate on Lavapipe.

Compatibility

• Default CPU build unchanged and dependency-free (ldd qwen36 links no libvulkan without VK=1). VK=1 is opt-in.
• CUDA tier: allocation-identical (the Vulkan take buffer is compiled out), but one shared-logic fix affects it: the uploader decremented the queue count at dequeue, before the upload ran, so qt_fill_wait could return before the last upload finished while the engine frees the RAM int8 copies right after. Deterministic on the slower staged path (5/5), latent on CUDA. Now an in-flight completion counter; qt_fill_wait returns strictly later.
• Finding I could not resolve: with staged uploads, results computed from a freshly created device-local block differ slightly and non-deterministically unless the block is filled once with vkCmdFillBuffer on creation. The fill value is irrelevant (0x00 and 0xFF measured identical), so it is the first GPU-side touch of the allocation, not a read of filler bytes. The fill is in; the cause is open (no GPU validation layers on my box). Repro: skip the fill and run the harness twice with COLI_VK_STAGED=1.
• Pre-existing, not fixed here (happy to file issues): qt_issue strides G.is_x by di*8*D while it is sized 32*D, so a two-GPU CUDA run with a wide top-k can overrun; qt_shutdown never signals cv_take, so a thread parked in qt_fill_wait would not see th_stop.
• Rebased onto current dev (f58a267). fix(qwen36): the VRAM tier now promotes int8 experts instead of reserving for nothing #1334's int8 expert promotion is kept verbatim and its uploads go through the shim as fmt 1 on both backends; the int8-on-Vulkan path, which neither side had alone, gets its own gate (test_qwen36_tier_vk_int8, runs on Lavapipe in CI). The whole validation above was re-run on the rebased tree.
• Docs: docs/qwen36-cuda-tier.md → docs/qwen36-tier.md (link updated), docs/vulkan.md, docs/ENVIRONMENT.md (COLI_VK_STAGED, VK_EXPERT_GB), CHANGELOG.

🤖 Generated with Claude Code

https://claude.ai/code/session_019BacNGNxAJ1M57UdYE2M3N

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[crichalchemist]

Rebased onto current dev (merge commit 3fb950d, plus one test commit b0503af); GitHub reports the PR mergeable again.

What the merge had to absorb, all inside the backend shim this branch introduces: the fp8 streaming mode, the resident lm_head and DeltaNet projections, automatic placement, and the cudaMalloc-granularity accounting. The shim gained three entry points (be_fp8_set_lut, be_trunk_upload, be_trunk_matmul). On CUDA they map to the existing calls; on Vulkan they refuse with one stderr line and those pieces stay on the CPU path, since backend_vulkan has no matmul over an already-uploaded tensor wired to the tier yet. docs/qwen36-cuda-tier.md is restored as dev has it; docs/qwen36-tier.md now covers only what differs on Vulkan.

The one test change: since d0a382d an expert is charged at allocator granularity, so test_qwen36_tier_vk's hand-computed two-expert budget admitted zero. Same adjustment 823caa8 made to the other tier tests.

Verified locally: the CPU build and the eleven other qwen36 test binaries; and, new since the first push, make qwen36 VK=1 against MoltenVK on macOS 13 x86_64 (2017 iMac, Radeon Pro 580 — the same Polaris10 silicon as the RX 580 rows in the doc, here under Metal rather than RADV), where both Vulkan tier tests run for real (not the skip path) and pass in int4 and int8 mode. Here the Vulkan (Lavapipe) and Qwen3.6 oracle jobs are green; the remaining jobs are still running as I write this.

(Edited: an earlier version of this comment said "Apple GPU"; the device is the AMD Radeon Pro 580.)

Ready for review whenever someone has the time.

[crichalchemist]

Token-exact end to end on real Vulkan hardware, not only the unit tests.

Setup: the ci.yml Qwen3.6 tiny oracle (make_qwen36_tiny.py --ref-mode full, convert_qwen36.py), run through make qwen36 VK=1 on the branch as it now stands (278a267), on a Radeon Pro 580 under MoltenVK (macOS 13 x86_64; the same Polaris10 silicon as the RX 580 rows in docs/qwen36-tier.md, which is this machine booted into Linux). COLI_DENSE_I8=0 SNAP=qwen36_tiny_c COLI_VULKAN=1 VK_EXPERT_GB=auto ./qwen36 8 8 qwen36_tiny/ref_full.json, cap 8 = n_experts so the tier engages.

run	tokens	residency	VRAM hit rate
CPU baseline	16/16	—	—
Vulkan, warmstart, staged uploads (default)	16/16	64/64	100 %
Vulkan, mapped path (COLI_VK_STAGED=0)	16/16	64/64	100 %
Vulkan, int4 conversion (--ebits 4)	16/16	64/64	100 %
Vulkan, QT_NO_WARMSTART=1, 400-token synthetic reference	n/a (ids are synthetic)	64/64 after fill	86 % during the fill

One thing worth knowing for anyone repeating this: on the stock 16-token reference, QT_NO_WARMSTART=1 reports 0 uploads. That is the process finishing in ~8 ms, before the uploader thread lands a single staged copy, not a tier fault; the 400-token row is the same mode given time to fill. The no-warmstart path also never promotes an int8 container on any backend, because the decode path in qwen36.c only calls qt_note for experts that have a packed g4 copy — pre-existing and outside this PR.

The two test_qwen36_tier_vk* unit tests run for real on the same device (not the skip path) and pass in both modes; the CPU build and the eleven other qwen36 test binaries are unchanged.

[crichalchemist]

Pushed 241f7f3: the Vulkan shim now has a gate that runs without a device. tests/qwen36_fake_vulkan.h is the Vulkan twin of qwen36_fake_cuda.h (every coli_vk_* symbol the tier links against, recording what it receives), and test_qwen36_tier_vk_fake drives the shim through it: single device with COLI_GPUS ignored, the VK_EXPERT_GB budget and its 4 GB fallback without VK_EXT_memory_budget, int4-gs64 arriving as fmt 4 / grp 64 at half a byte per element, fill-once under 64 ticks with a far hotter non-resident (no swap, no free), issue/take through ybuf with the weighted accumulation checked lane by lane, a failed take leaving the output alone, and the fp8 and trunk refusals landing on the CPU path without touching the backend.

It is collected by TEST_RULES, so it lands in make check and test-asan on Linux, macOS and Windows alongside the CUDA fake tests; the numerics keep their real-device gate in test_qwen36_tier_vk. Two mutants (a take that always fails; QT_SWAPS=1 on the Vulkan shim) trip one and seven assertions respectively, and it is clean under ASan+UBSan.

Also note #1390, which carries the qt_fill_wait in-flight fix on its own against dev; once it lands I will rebase this branch onto it and drop the copy here.

[JustVugg]

Worth another forward-merge. The real conflicts are exactly where your shim lives: c/qwen36_tier.c has moved 3 commits and c/qwen36_tier.h 2, and those three are #1388 (QT_MAX_ROWS, placement pin, ownership-based int8 free), #1404 (int8 experts on the decode path) and #1390 (qt_fill_wait residency). Plus c/Makefile 30 and c/qwen36.c 8.

Not trivial, but you absorbed an identical round on 2026-09-07 already. Nothing has overtaken it either: #1513 is VK RAM reservation on integrated GPUs and #1454 is the HIP path, neither is a Vulkan expert tier for qwen36.

The token-exact results on real Radeon hardware and the device-free fake-Vulkan gate are what make this reviewable without the card in hand. Please keep both through the rebase.

[crichalchemist]

Forward-merged onto dev (merge commit 032d8df), and the PR is mergeable again. Six follow-up commits since, described below.

Against merge-base f18a54a with dev at cc756a6: the three PRs you named are 6 commits on c/qwen36_tier.c and 4 on c/qwen36_tier.h, plus 13 on c/qwen36.c and 49 on c/Makefile. Eleven conflict hunks across four files; c/qwen36.c merged clean.

What the conflicts were. #1390 landed with reworded comments, so this branch's copy of the qt_fill_wait fix is gone in favour of yours, as promised. #1388's QT_MAX_ROWS replaces the literal 32 in the issue path and the per-device row arrays, while the backend-neutral QtTensor stays, since your side of those lines still spells ColiCudaTensor. In c/Makefile, your new prerequisites (expert_ffn.h, omp_tune.h, kv_prefix.h) with this branch's $(VK_OBJ)/$(VK_SPV) re-applied, and the tier condition is your ifneq over CUDA/CUDA_DLL/HIP/HIP_DLL with the VK=1 branch kept after it. docs/qwen36.md keeps your expert-kernel section and the ROCm paragraph with the VK=1 sentence restored.

Two things the clean merge hid. Neither is a defect on dev; both are the cost of this branch making qwen36_tier.c compile two ways, and both were this branch's to absorb.

1. The dense-trunk block for the Qwen3.8 trunk — qt_dense_init, qt_dense_matmul, dense_free_all — calls coli_cuda_tensor_upload / coli_cuda_matmul / coli_cuda_tensor_free directly. On dev that is simply correct: the file is only ever compiled with CUDA. Here it meant the file no longer built without it, and nothing conflicted because the calls landed in a region this branch had not touched. They now go through the same be_trunk_upload / be_trunk_matmul / be_free shim as lm_head and the DeltaNet projections, so on Vulkan qt_dense_init returns -1 and qwen38_core.h keeps the weight on the CPU, as it already does for a failed upload. coli_cuda_group_stats is left alone — it is already inside #ifdef COLI_CUDA.

2. make qwen38 VK=1 failed at link with undefined coli_vk_*. This branch's VK=1 arm is what makes QWEN36_TIER_SRC non-empty — on dev it is empty under VK=1 — so the qwen38 rule and the two qwen36 tokenizer tests needed $(VK_OBJ) the moment they met this branch. Fixed in c161d58 for every target that links the tier source. The CI Vulkan job builds only qwen36 and the two test_qwen36_tier_vk* binaries with VK=1, so it could not have reached this on either side; one of the tokenizer tests had the same gap on this branch since before the last merge.

One latent item, not a live bug: G.ybuf was the last literal 32 in the file and is now sized by QT_MAX_ROWS. It is correct at the current value — it would only overflow if that constant were raised, which is the defect #1339 was filed to prevent.

Evidence, re-measured on the merged code rather than carried over.

make check passes: Python suite 970 tests, OK (skipped=105), and make -C c test-c exits 0. Clean at 0 warnings for make qwen36, make qwen36 VK=1, and make qwen38 VK=1. CI was green on all 28 checks at the merge head a1a8dfd; the six follow-ups below are pushed with this comment and re-run it. (Locally the suite shows two warnings that are on dev unchanged and only appear on an x86_64 Mac — qwen38_core.h workers and test_olmoe_matmul_q.c failures; #1599 carries those, out of this PR's scope.)

Token-exact on the Radeon Pro 580 via MoltenVK (macOS 13 x86_64 — the same Polaris10 silicon as the RX 580 rows in docs/qwen36-tier.md, that box booted into Linux). Every run below logs [VK] ready: AMD Radeon Pro 580, so none of them quietly fell back to the CPU:

run	tokens	residency	VRAM hit rate
CPU baseline	16/16	—	—
Vulkan, warmstart, staged uploads (default)	16/16	64/64	100 %
Vulkan, mapped path (COLI_VK_STAGED=0)	16/16	64/64	100 %
Vulkan, int4 conversion (--ebits 4)	16/16	64/64	100 %

Stronger than the match count: all five 16-token runs, including QT_NO_WARMSTART=1, emit a byte-identical C engine : token-id line, so the backends agree with each other and not merely with the reference.

One row from my earlier comment is out of date since #1404. I noted then that an int8 container never promotes on the decode path — true at the time, and the reason #1391 was filed. #1404 fixed it, and it works through the Vulkan tier: on a 256-token reference (make_qwen36_tiny.py --max-new 256), QT_NO_WARMSTART=1 fills from cold to 64/64 resident at a 58.3 % hit rate, against 64/64 and 100 % for the warm run, with both emitting identical tokens. On the stock 16-token reference it still reports 0 uploads — the process finishes before the uploader lands a staged copy — so anyone repeating this should use the longer reference for anything about fill.

The device-free gate you asked me to keep is intact and now also covers the dense handles: the handle is refused, none is registered, the matmul returns 0 and the backend is never touched. Mutating the Vulkan be_trunk_upload to claim success trips both new assertions. To be clear about what that evidence covers — qwen36 never calls qt_dense_*, so the token-exact runs above say nothing about the dense conversion; the fake gate plus that mutation is its evidence.

Under the Khronos validation layer. Everything above was then run again with VK_LAYER_KHRONOS_validation injected by the loader (LunarG SDK 1.4.357.1, no code change), which sees what token-exactness cannot. It reported exactly two things, identically on every run, and both are now fixed — four small commits, two of them this branch's and two that predate it:

• VUID-vkDestroyDevice-device-05137, 21 leaked objects at shutdown. Twelve were this branch's: qt_shutdown joined the uploader and tore the backend down with every resident expert still uploaded — CUDA's device reset had been hiding it — and the staged device-local arena chain was never freed (71991f6, c81a933). Releasing them at shutdown took away the observable your two qwen36 CUDA tier: qt_shutdown never signals cv_take, so pthread_join can hang if a group is open #1340 tests used — the abandoned swap's victim still holding its tensor after qt_shutdown returned — so both now pin abandonment through what still tells it apart from an executed swap (no upload driven, the incoming expert neither queued nor resident) and pin the release itself; disabling the abandonment branch fails both. On CUDA this frees explicitly what the device reset used to reclaim. The other nine predate the branch: the rmsnorm pipeline set and its two descriptor pools were never destroyed, one of the pools being a local that build_pipeline handed back and the caller dropped (cb70b7a). Fixed here because the branch already edits coli_vk_shutdown and a clean layer run is part of its evidence.
• VUID-VkDeviceCreateInfo-pProperties-04451: a portability implementation must have VK_KHR_portability_subset enabled, and MoltenVK is one. Also predates the branch; enabled on both devices when advertised, a no-op on RADV, NVIDIA and Lavapipe, which never list it (bcdee83).

After those, both tier tests and the int8, int4, mapped and no-warmstart oracles report zero validation errors, warnings and VUIDs, still token-exact, still on the Radeon. The Lavapipe job could run the layer too (apt install vulkan-validationlayers, one env var); I have not touched CI in this PR, but would add it if you want it.

[crichalchemist]

One more layer, since the last comment covered only the core checks: the same six runs — both tier tests and the int8, mapped, int4 and 256-token no-warmstart oracles — under the Khronos synchronization validation (validate_sync, duplicates unlimited) on the Radeon Pro 580 via MoltenVK. Zero hazards in every one, and the C engine : token-id lines are byte-identical to the layer-free runs. So the uploader thread's staged copies, the expert-group issue/take path and the shutdown release are hazard-free as far as the layer can see.

What the layer did flag is in the harness and on dev, not in this branch: write-after-write in the harness's own throughput loops and six buffers coli_vk_shutdown never freed. Those are in a separate PR against dev so this one stays about the tier; the eight objects that PR leaves alone are the rmsnorm pipeline set and pools that cb70b7a here already destroys.

One caveat for anyone repeating this: SDK 1.4.357.1's submit-time synchronization validation segfaults inside the layer at vkDeviceWaitIdle when qt_shutdown runs from atexit — after the run is complete, but before stdio flushes, so the match lines vanish. Run the engine under a pty or set VK_LAYER_SYNCVAL_SUBMIT_TIME_VALIDATION=false; the tier tests do not hit it.

[crichalchemist]

Pushed one more commit: xf_mode() now turns the planar-int4 expert_ffn.h kernel off under COLI_VULKAN=1 the way it already did under COLI_CUDA=1.

Found while A/B-ing on the real qwen36-35b-a3b-colibri-i4-gs64 container (Radeon Pro 580 through MoltenVK, COLI_VULKAN=1 VK_EXPERT_GB=auto): bcdee83 segfaulted on the first token. lldb put it in the tier's int8 CPU fallback, matmul_qe(g, xs, e->g, …) at qwen36.c:2266 → matmul_q_gs with q == NULL. The chain: the gate was CUDA-only, so under Vulkan xf_mode() stayed on and every slot was allocated pw-only (g, g4 NULL); tier_warmstart offered NULL weights (if (!wg) continue, and the "N in VRAM" line counts the plan, not uploads); qt_issue returned an empty mask; the int8 fallback dereferenced NULL, and slot_ensure_int8() could not rebuild without g4. The tiny fixture never sees it because its group size keeps the planar kernel off, which is why the oracle passed here on 2026-09-07.

Evidence on this box, same container and prompt, 128 greedy tokens against the CPU ids:

• bcdee83: exit 139 at token 1 (also with the CACHE_ROUTE branch reverted: pristine reproduces).
• bcdee83 + QWEN_EXPERT_KERNEL=0: 3302/10240 uploads, 128/128 ids, 2.42 tok/s, VRAM hit rate 32.1%.
• this commit, no workaround: 3302/10240 uploads, 128/128 ids, exit 0 (3.47 tok/s, VRAM hit rate 32.1%). make check on the branch: exit 0, the same 10 pre-existing warnings as bcdee83..

dev carries the same CUDA-only gate, so the bug is latent there and would go live with this PR; the fix is on this branch for that reason. Docs: the QWEN_EXPERT_KERNEL row in docs/ENVIRONMENT.md and the sentence in docs/qwen36.md now say "CUDA or Vulkan".

[crichalchemist]

Forward-merged onto current dev (tip a2e578c), head is now 27188fa. It's a merge rather than a rebase, so the earlier review threads still line up.

How the conflicts were resolved

• Shutdown (fix(cuda): release Qwen tier resources on shutdown #1678): dev's resident-expert release in qt_shutdown replaces this branch's 71991f6. I dropped the old trailing release loop because after the merge it would run after G.slot is freed. The Vulkan shim only adds free(G.ybuf).
• qt_take (fix(cuda): propagate Qwen expert collection failures #1682): kept dev's two-phase drain. The Vulkan backend passes G.ybuf as the take target.
• Batched dense/dnproj (qwen36: offload batched attention projections to CUDA #1674/qwen36: batch resident DeltaNet input projections #1677): be_trunk_matmul now takes S. On Vulkan it still refuses, so the trunk stays on the CPU path as before.
• [CUDA] mode line ([Bug]: Windows CUDA_DLL build of qwen36 is broken in four independent ways (Makefile gating, missing symbol, missing marker string, cuda_binary() checks the wrong engine) #1533): stays under COLI_CUDA so doctor.py keeps finding it.
• coli_vk_shutdown: keeps both this branch's staging/upload teardown and dev's qprep/norm frees, with no double destroys.
• Two new commits:
  • f5ca8ef: the expert-collection failure message names the backend (qwen36.c and qwen38_core.h).
  • 3b7aa52: links the tier and Vulkan objects into test_qwen38_tokenizer, _idot and _native_weights, which otherwise fail to link under VK=1.

Validation (iMac 2017, Radeon Pro 580, MoltenVK, macOS 13, x86_64)

• make check at 27188fa: passes, 0 compiler warnings.
• make test-c VK=1: every target that still fails also fails on unmodified dev (33422c3). Those are colibri.c tests and kimi rules that don't link backend_vulkan.o under VK=1. I'll send that as a separate PR, since it isn't this branch's.
• test_qwen36_tier_vk and test_qwen36_tier_vk_int8 on the device: pass.
• Tiny Qwen3.6 oracle (16 tokens; int8, int8 with COLI_VK_STAGED=0, int4): 16/16 against torch, ids identical to the CPU run. At 256 tokens, CPU and Vulkan both match torch on 70/256, with identical ids. The Vulkan run's VRAM hit rate was 97.6 %.
• Khronos core validation: 0 VUIDs, errors or warnings across the tier tests and the four oracle modes.
• Sync validation: 0 hazards. With VK_LAYER_SYNCVAL_SUBMIT_TIME_VALIDATION=true, the layer itself segfaults at exit inside SyncValidator::PostCallRecordDeviceWaitIdle (null deref in SyncExecScope::MakeSrc, called from coli_vk_shutdown's vkDeviceWaitIdle). With submit-time validation off, the same run is clean and 16/16.
• Real 35B int4 gs64 container, 64 greedy tokens:
  • With COLI_DENSE_IDOT=0 QWEN_DENSE_BATCH=0, the merged head's Vulkan run, the pre-merge head's (48ac375) Vulkan run and both heads' CPU runs produce identical ids, 64/64.
  • Under dev's defaults (COLI_DENSE_IDOT=1), the Vulkan run diverges from the CPU run at the first generated token.
  • Per-layer dumps (DUMP_LAYERS) locate the difference. The Vulkan run is bit-identical to CPU with QWEN_EXPERT_KERNEL=0 (the int8-unpack path the tier uses for misses) until the first GPU-computed expert. From there the end-of-stack drift (cos 0.989) is the same size as the drift between the two CPU expert kernels under the same trunk.
  • So I read it as the integer-dot trunk amplifying small differences, not the tier computing something different.
  • Caveat: the tiny-oracle and validation-layer runs above used COLI_DENSE_I8=0, so they don't exercise COLI_DENSE_IDOT.