qwen36 tier: fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU by kreuzzelg · Pull Request #1374 · JustVugg/colibri

kreuzzelg · 2026-09-06T20:39:26Z

Four tier-side changes that came out of wiring the tier into the Qwen3.8 engine (that engine hook follows as its own PR once it shows a win; this part stands on its own and two of the four affect qwen36 today). Stacked on #1361, which is stacked on #1360: the diff is the last four commits.

1. fp8 streaming mode (qt_init_fp8). A mode for an engine whose experts do not all live in RAM: cap < n_experts is accepted, weight format 8 is e4m3 bytes plus the checkpoint's [ceil(O/128), ceil(I/128)] block scales -- exactly what #817's tensor_upload(fmt=8) and grouped_*_f8w kernels take, no new kernel. Staging copies slab and scales unchanged (no XOR), the tier keeps no pointer into the engine's slot (the slot is recycled by the next token), and promotion happens at qt_note time only: room -> in, otherwise a budget-neutral swap against the coldest resident on the home device when the newcomer is hotter. The Qwen3.6 modes are untouched (cap == n_experts still required there). tests/test_qwen36_tier_fp8.c pins all of it on the fake backend.

2. VRAM charged at cudaMalloc granularity. Measured with cudaMemGetInfo over 256 allocations each: above 1 MiB the allocator rounds to multiples of 2 MiB, above 512 KiB to one 1 MiB page, below that to 8 KiB steps with about a sixteenth of overhead (96 KiB -> 104 KiB, 384 KiB -> 416 KiB, 400 B -> 8 KiB). Charged by payload, an fp8 expert of three 1.56 MiB matrices looked like 4.69 MiB and took 6.03 MiB: a 6.5 GB budget filled an 8 GB card to the last megabyte and the uploader hit "tensor allocation: out of memory" eight times before its stop-trying fallback shrank the budget. dev_alloc_footprint() now charges what the allocator takes, so the planned count is the resident count. Qwen3.6 is not affected in practice: its int4 matrices are exactly 512 KiB and are served exactly (the new rule charges its gs64 expert 1.79 MiB against ~1.69 MiB actual, measured 2,782 resident on a 4.9 GiB budget). Recovering the fp8 loss would need an arena per device; not in this PR.

3. Thread affinity. With OMP_PROC_BIND set, libgomp binds the initial thread to place 0 before main(), and a pthread inherits the CPU mask of the thread that creates it. The tier's uploader thread and the CUDA runtime's own threads were therefore jailed on the OpenMP master's core -- observed as three threads with the master's one-CPU mask in /proc/<pid>/task/*/status. qt_init now widens the caller's mask to every online CPU around CUDA init and thread creation and restores it afterwards (raw sched_*affinity syscalls, Linux only, no _GNU_SOURCE because tests #include this file after the engine's headers). test_qwen36_tier_fp8 binds its own thread to CPU 0 the way libgomp would and checks that the uploader is not jailed and the caller's mask comes back. Measured effect on Qwen3.8 was within noise; on qwen36 not measured separately. It is the right shape regardless.

4. COLI_GPU. resource_plan.py writes the singular for a one-device plan and colibri.c reads it; the tier only read COLI_GPUS, so coli chat --gpu 1 landed on every visible device. Alias added.

Tests: the seven tier tests pass, ASan/UBSan clean on the fp8 test; make qwen36 CUDA=1 and the CPU builds of qwen36/qwen38 build; test_makefile_cuda_scope and test_family_registry pass.

Two things I found while measuring, not part of this PR: the docs' run line (OMP_PROC_BIND=close without OMP_PLACES) packs libgomp's threads onto SMT siblings on my Threadripper (12 threads on 6 cores); OMP_PLACES=cores gives +30 % on qwen36 (11.0 -> 14.4 tok/s, one card, cold). And coli plan names devices by nvidia-smi index while the tier selects by CUDA ordinal; on two unequal cards they differ (see #1361). Both could be small follow-ups if you want them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

…n leak they found Stacked on JustVugg#1344. Its three regression tests each pin one scenario; this adds the rules those scenarios are instances of, checked directly against tests/qwen36_fake_cuda.h in the plain CPU build: 1. Budget accounting balances. On every device, bytes in use never exceed the budget and, once the queue is drained, equal exactly resident experts x bytes per expert: every reservation is consumed by an upload or handed back. Checked on one and two devices, across an LFRU swap (budget-neutral by construction, so `used` must not move), and across the path where a planned expert is reported without weights. 2. Shutdown wakes every waiter at once. All four cv_take sleepers -- the uploader's victim wait, qt_note_block, qt_note_planned, qt_fill_wait -- are parked simultaneously behind a full queue and an open group; qt_shutdown has to bring every one of them home, under a watchdog thread rather than alarm() so it runs on MinGW too. Afterwards no slot may still read as queued and the abandoned swaps' victim keeps its tensor and its resident flag. 3. Issue geometry under random routing. Random resident sets, random K up to the row limit, one to three devices, 200 seeds each: every device block inside the replica buffer, pairwise disjoint, at its device's slot; the mask names exactly the routed experts that were resident on a device whose issue succeeded; hits + misses add up to everything routed. The row limit is read from the array the rows index (sizeof G.is_k[0] / sizeof G.is_k[0][0]), so the stride cannot drift from it without failing here. Under test-asan this is a fuzz for the JustVugg#1339 class. ## What the first rule found qt_plan_fill reserves budget and sets planned=1; qt_note_planned returned early on NULL weights without undoing either, and the warmstart did not call it at all when the loader came back empty. The bytes stayed out of the budget for the life of the process and "if(resident||queued||planned) continue" never reconsidered the expert. JustVugg#1331 was this leak for every expert of an int8 container; the class survived its fix. Fix: qt_note_planned hands the reservation back when it receives no weights, and tier_warmstart reports every planned expert, with or without them. Without the fix the new test fails eight checks (all this leak) and passes clean under ASan; with it, all green. ## Verified - test_qwen36_tier_invariants: ok; under ASan+UBSan: 0 diagnostics - without the fix: 8 FAIL, 0 sanitizer diagnostics (a red test, not a crash) - the four JustVugg#1344 tests, test_qwen36_ctx and the qwen36 build unchanged: ok - one lesson kept in the file: the watchdog's first draft passed its timeout by pointer into the arming function's frame -- ASan flagged the stack-use-after-return (JustVugg#1277's class) in the test itself before it could flag anything in the tier Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

qt_fill_wait() returns when the queue is empty, but the expert the uploader dequeued last is still queued=1 until its upload returns. The residency check right after it raced that upload and failed about one run in fifteen locally ("expert did not become resident during warmstart") -- a red that says nothing about JustVugg#1339. Poll until no slot is queued before asserting. The same two-line wait is what test_qwen36_tier_invariants uses (WAIT_IDLE). Noted on JustVugg#1344 as review feedback; carried here so the stacked PRs stop rolling dice in CI. Drop this commit if JustVugg#1344 takes the fix first. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

The lm_head GEMV (248320x2048, one call per token at token END, outside the per-layer latency chain) cost 13.8-14.8 ms/token on CPU — 18% of decode. Park the dense-i8 rows + per-row scales on a CUDA device and run them through coli_cuda_matmul (fmt=1, identical y[o]=acc*sc[o] semantics; greedy output bit-identical across all runs). qt_init removes the lm_head device from the expert PLACEMENT list (not just budget=0: home() would still hash experts onto it and the hit rate collapsed 89.9->49.5% in the first attempt) while keeping its CUDA context; if it is the only device, experts stay and lm_head shares it. Measured (gs64, heat-warm, 200 tokens): CPU lm_head 14.8 ms/tok 81.0 total 11.05 tok/s lm_head on Quadro 9.9 ms/tok 76.2 total 11.69 tok/s lm_head on 3070 3.9 ms/tok 72.7 total 12.21 tok/s <- best, single GPU (experts + lm_head share the 3070; token end is idle time) (cherry picked from commit 734e1e3d77344e3bc9613d6fbc6c752746b6a518)

(cherry picked from commit 4064493, ohne die versehentlich committeten Binaries c/qwen36 und c/tools/bench_dnproj; qt_init-Signatur und Device-Auswahl auf den JustVugg#1344-Stand aufgeloest: die COLI_PLACE-Devices werden nach der COLI_GPUS-/Auto-Auswahl und vor der Leer-Pruefung ergaenzt. tests/qwen36_fake_cuda.h bekommt coli_cuda_matmul als aufzeichnenden Stub, damit die Tier-Tests weiter im CPU-Build linken.)

The hand-written COLI_PLACE list from R4 is a measurement tool; nobody running a 6 GB card should have to work out that 1.2 GB of dense trunk is worth more than 800 experts (JustVugg#1040). The engine knows every size involved, so it decides. Rule: bytes saved on the memory bus per token, per byte of VRAM spent. Dense weights are read every token (value 1.0 per byte); an expert with the probability p_e that a token routes to it -- its heat share when a HEAT_FILE exists, topk/n_experts otherwise -- and the CPU fallback reads the int8 slot, twice the VRAM bytes of an int4 expert (value 2*p_e; 1*p_e on an int8 container). A trunk item goes to the device with the most room if its value beats that of the coldest experts it would push out, i.e. the tail of the heat order that still fits today. Without heat the tail is worth 2*topk/n_experts per byte and the trunk always wins; with heat, a card whose marginal expert is routed on more than every second token keeps its experts -- the R4 measurement, where moving all projections onto one near-full card cost 0.7 GB of hot experts and the 13 ms it had saved. Mechanics: - qt_trunk_offer(component, layer, bytes) before qt_init: the engine offers lmhead once and dnproj per DeltaNet layer, sizes from the same dense-i8 entries the uploads use. No entries (dense-i8 off) -> no offer. - qt_init prices and places, writes the decision into the table qt_place_of() already reads, and subtracts placed bytes from that device's expert budget. The explicit list gets the same subtraction now; with COLI_PLACE="dnproj=0" the projections used to come out of the expert cache unannounced. - COLI_PLACE unset or "auto" -> automatic; a list -> obeyed, auto off; "off" -> nothing placed. In auto mode the R4 role-split reservation is skipped on purpose: one card shares trunk and experts. - Weight format, exp_bytes and the heat table move ahead of the role-split block in qt_init, and the parse state is re-derived per init. tests/test_qwen36_tier_autoplace.c on the fake backend (free VRAM is what the test says): fits -> placed and budget shrinks by exactly the trunk; too big -> CPU; heat verdicts both ways at p_marginal 0.67 and 0.17, with and without a heat file, and "room nobody would use goes to the trunk"; explicit list obeyed and charged; off; two devices greedy by room with per-device budgets; nothing offered -> no-op. All six tier tests green, ASan/UBSan clean on this and the invariants test, CPU and CUDA builds ok. Not yet: dnout/attnproj (offered components other than lmhead/dnproj are ignored), and the calibration on real cards that decides whether the factor 2 and the greedy-by-room device choice survive contact with a Quadro/3070 pair and a 6 GB card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

…with the first calibration Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

cudaGetDeviceProperties says ordinal 0 is the RTX 3070 and ordinal 1 the Quadro RTX 4000 on the calibration box (CUDA orders fastest-first, nvidia-smi by PCI bus, and the two disagree there). The rows labelled "RTX 3070" were measured on the Quadro and the row labelled "Quadro" on the 3070. Numbers unchanged; names swapped; the sentence that explained the Quadro being the faster card is gone, because it was the 3070. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

qwen36 keeps every expert in RAM and lets the tier retain raw pointers into slots that are never recycled; `cap == n_experts` guards exactly that. Qwen3.8 streams 24 576 experts of 4.7 MiB through a per-layer LRU whose slots ARE recycled, so a retained pointer would dangle by the next token, and 121 GB of experts do not fit beside the model anyway. qt_init_fp8(nl, ne, D, Ih, cap, topk, e4m3_lut) starts the tier in a mode that owns what it uploads: - cap may be smaller than n_experts; - weight format 8: e4m3 bytes as they came from the checkpoint, one f32 scale per 128x128 block per matrix -- the layout JustVugg#817's kernels and tensor_upload(fmt=8) already agree on, so no kernel changes; the decode table is published to the backend at init and the tier refuses to start without it; - staging copies the bytes unchanged (no XOR) and the three block-scale tables, inside the qt_note call while the engine's slot is live, and the slot pointers are dropped before qt_note returns; - therefore a promotion can only happen when the bytes pass by: the LFRU decision moves from the periodic tick into qt_note -- a non-resident expert noted on a full device evicts the coldest resident there when the shared admission rule says it is hotter, as a budget-neutral swap; the tick keeps only the decay. The int4/int8 modes are untouched: same code paths, same `cap == n_experts` refusal, the six existing tier tests unchanged and green. tests/test_qwen36_tier_fp8.c on the fake backend (which now counts fmt 8 at one byte per element and stubs the LUT publish): starts with cap 4 of 16, exp_bytes = 3*D*Ih + three block tables + slack, three fmt=8 uploads per expert with the bytes intact, no pointer survives a note, a cold newcomer is refused on a full device, a hot one is swapped in budget-neutrally, hits and misses count, shutdown clears the mode, a missing LUT refuses the mode, the int4 mode still refuses cap < n_experts. Clean under ASan/UBSan, as are the invariants and placement tests; CPU and CUDA builds of qwen36 ok. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

Budget accounting. The tier charged an expert by its payload. cudaMalloc rounds an allocation above 1 MiB up to a multiple of 2 MiB and one at or below 1 MiB up to 1 MiB (measured with cudaMemGetInfo: 1,638,400 B -> 2 MiB, 819,200 B -> 1 MiB, 400 B -> 10 KiB). In the fp8 streaming mode an expert of three 1.56 MiB matrices was charged 4.69 MiB and took 6.03 MiB, so a 6.5 GB budget filled an 8 GB card to the last megabyte: eight "tensor allocation: out of memory" lines before the uploader's stop-trying fallback shrank the budget to what had fit (1,279 experts, 7.77 GB on the card). exp_bytes is now three weight footprints plus three scale footprints, so the planned count is the resident count. Qwen3.6 is not affected: its int4 matrices are exactly 512 KiB and the allocator serves those exactly, so the payload accounting was already right there. Recovering the fp8 loss needs an arena per device; not done here. COLI_GPU. resource_plan.py writes COLI_GPU for a one-device plan and colibri.c reads it; the tier only read COLI_GPUS, so a `coli chat --gpu 1` landed on every visible device. The tier now takes COLI_GPU when COLI_GPUS is unset. Tests: the two tier tests that compute exp_bytes use the tier's own dev_alloc_footprint. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

…d CUDA threads With OMP_PROC_BIND set, libgomp binds the initial thread to place 0 before main() runs, and a pthread inherits the CPU mask of the thread that creates it. The tier's uploader thread and the CUDA runtime's own threads were thus jailed on the OpenMP master's core: every staging copy and driver call competed with the master's share of each expert matmul, and the whole team waited for it. On Qwen3.8 (cap 128, one 8 GB card, 12 threads) the CPU time per remaining expert rose 64 % while the tier was on -- 3.7 ms against 2.25 ms -- which ate the whole gain of computing 45 % (one card) to 59 % (two cards) of the routed experts on the GPU: 0.32/0.31 tok/s against 0.32 on the CPU alone. Fewer OpenMP threads did not help, because the contention is on the master's core whatever the team size. qt_init now widens the calling thread's mask to every online CPU while it initializes CUDA and creates the uploader, and restores the caller's mask afterwards. Raw sched_{get,set}affinity syscalls, Linux only, no _GNU_SOURCE (the file is #included by tests after the engine's headers). The uploader records how many CPUs it may run on; test_qwen36_tier_fp8 binds its own thread to CPU 0 the way libgomp would, starts the tier, and checks that the uploader is not jailed and that the caller's mask comes back. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

…urve The first footprint rule charged every allocation at or below 1 MiB a full 1 MiB. That is what a 768 KiB request takes, but not what Qwen3.6's 512 KiB int4 matrices or its 64 KiB gs64 scale tables take: measured over 256 allocations each, small requests round to 8 KiB steps with roughly a sixteenth of overhead (96 KiB -> 104 KiB, 384 KiB -> 416 KiB), requests above 512 KiB take one 1 MiB page, and above 1 MiB multiples of 2 MiB. With the coarse rule a Qwen3.6 gs64 expert was charged 6 MiB instead of its 1.7 MiB and a 4.9 GiB budget planned 836 experts where 2,800 fit (measured: 2,782 resident after the change). The fp8 Qwen3.8 expert is unchanged (three 2 MiB matrices plus three 8 KiB scale tables). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

…tor footprint test_qwen36_tier_int8 compared exp_bytes against the payload formulas and test_qwen36_tier_shutdown sized its one-expert budget from them; both broke when exp_bytes started charging what cudaMalloc takes. They now use the tier's dev_alloc_footprint like the other tier tests. CI caught it; I had run only the five tests I had touched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

kreuzzelg and others added 15 commits September 6, 2026 22:26

qwen36 tier: qt_stats names the trunk share; docs: placement section …

046dec6

…with the first calibration Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

docs(qwen36 tier): 5 GB budget calibration point

112a53a

docs(qwen36 tier): Quadro and two-card calibration, the two-card caveat

371d35d

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

docs(qwen36 tier): Ollama reference row for the placement calibration

181d722

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a

JustVugg merged commit f2853a0 into JustVugg:dev Sep 7, 2026
25 checks passed

kreuzzelg mentioned this pull request Sep 10, 2026

qwen38: CUDA VRAM tier for Qwen3.8-Flash-Next -- hot experts, and the dense trunk as int8 residents (+43 % on one 8 GB card, +60 % on two) #1424

Merged

JustVugg mentioned this pull request Sep 13, 2026

Release 1.11.0 #1467

Merged

Navigation Menu

Repository navigation

qwen36 tier: fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU - #1374

qwen36 tier: fp8 streaming mode, VRAM accounting at allocator granularity, thread affinity, COLI_GPU#1374
JustVugg merged 15 commits into
JustVugg:devfrom
kreuzzelg:qwen36-tier-fp8-fixes

kreuzzelg commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kreuzzelg commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants