Qwen3.8-Flash-Next: is a GPU backend (CUDA/HIP) planned for the qwen38 engine? (offer: gfx1151 / ROCm 7.2.4 validation + benchmarks)

[simongonzalezdc]

Thanks for colibri — running frontier MoE on hardware you already own is exactly the right idea, and v1.10.1 runs Flash-Next here on day one.

Setup: Qwen3.8-Flash-Next-FP8 (official checkpoint), colibri v1.10.1 linux-x86_64 release, Strix Halo (Ryzen AI Max 395, gfx1151 iGPU, 96 GB unified), ROCm 7.2.4, model on external storage.

Measured (per the benchmark bar in CONTRIBUTING):

• coli serve --ram 20 --auto-tier --policy experimental-fast
• cold: 49 completion tokens in 248.6 s (~0.20 tok/s); warm: 47 tokens in 215.3 s (~0.22 tok/s); single request, n=1 each (smoke-scale, not a benchmark claim)
• resident matrices 10.88 GiB, load 15.3 s, answers correct, doctor reports "GPU detected but the engine is CPU-only" and "RAM budget cannot hold one expert slot per sparse layer"
• the box co-runs other resident services, so the RAM budget above is what was safe to give — expert residency is effectively 0% and decode is disk-bound, which matches the doctor's placement warning exactly

Question: c/Makefile builds the qwen38 engine with NOCUDA_CFLAGS (the CUDA/HIP backend is wired for the glm/qwen36/dsv4 families), so on this class of APU the Flash-Next path is CPU-only today. Is a GPU tier for qwen38 on the roadmap?

Offer: we have exactly the rig the HIP path was tested on class-wise (gfx1151, ROCm 7.2.4 — very close to the doc's tested 7.2), a working HIP toolchain, and we're set up to run reproducible measurements (commit, exact commands, hardware, warm-up policy, run count, median). Happy to validate a qwen38 HIP/CUDA build, file structured reports, or help however is useful. No pressure either way — the CPU multitier path alone is already impressive.

[JustVugg]

Thanks for the measured numbers and the offer, both are useful. Your read is right: qwen38 is CPU-only today, and on your box the 0.2 tok/s is the RAM budget (20 GB against a 10.9 GiB resident trunk leaves no expert slots), not the GPU.

Status: there is a concrete plan for a GPU expert tier for Qwen3.8-Flash-Next in #1045 (kreuzzelg's item C: the #713 tier mechanics plus a block-scale-absorbing fp8 kernel for the 128x128 weight_scale_inv layout). I have just answered there that nobody else is on the GPU side, so that is the work to follow. HIP on gfx1151 with ROCm 7.2.4 is exactly the validation that path will need, since the tier has only been measured on CUDA so far; when the PR is up I will ping you here for the structured runs.

Until then, one thing that is measurable now on your hardware: with --ram raised so at least one expert slot per sparse layer fits (doctor tells you the number), the CPU path stops being disk-bound and you get the real CPU ceiling of the box, which is the baseline the GPU tier will be compared against.

[JustVugg]

Update: #1424 is merged into dev. Qwen3.8-Flash-Next now has the GPU tier through qwen36_tier.c (hot experts in VRAM in fp8 streaming mode, and the dense trunk as int8 residents), measured +43% on one 8 GB card and +60% on two, greedy output identical to the CPU BF16 run. It is built with the shared backend, so HIP=1 HIP_ARCH=gfx1151 compiles the same code; whether it runs correctly on RDNA 3.5 is exactly what your offer would settle. If you try it, the correctness line the benchmark protocol now asks for (greedy output identical to Q38_TRUNK_GPU=0 on the same prompt) is the one to include; #1499 taught us what a fast wrong backend looks like.

[DebugSultan]

Running qwen38 on a 2×16 GB consumer Blackwell rig (RTX 5070 Ti + 5060 Ti, CUDA 13) from a downstream fork (DebugSultan/colibri, branch qwen38cuda). Before #1424 landed we'd built a different answer to the same memory-wall problem — routed experts as an on-disk int4 (gs64) container + a fixed arena of VRAM-resident copies with heat-ranked promotion, and the bf16 dense side resident as its own fmt=9 — and we merged upstream/dev today with both tiers living in one engine, one env choosing (Q38_TIER_TRUNK=1 selects the #1424 streaming-fp8 + int8-trunk tier, default is ours). Your new device trap test caught a real divergence in our fork (bf16 decoded in weight_at while the host predicate refused it); fixed properly, make cuda-test is green here.

First A/B on our rig (short-prompt greedy decode, single run each, both without expert pinning):

arena tier (ours)	trunk tier (#1424)
3.20 tok/s	2.12 tok/s
dense 104.6 ms/fwd	dense 56.5 ms/fwd
deltanet 71.1 ms/fwd	deltanet 37.6 ms/fwd
expert-read 52.4 ms/fwd	expert-read 142.5 ms/fwd (62.6 % hit, 11 302 CPU misses)

The int8 resident trunk eliminates exactly the dense-side CPU fallback we'd just spent a night attributing; the int4 arena eliminates exactly the streaming cost. The two tables read like a spec for the hybrid (arena + resident trunk), and qt_trunk_offer is already in the linked binary — we'll know after the paired rerun with pinned experts and a longer prompt.

On the correctness line you asked for: our protocol is fixed-seed greedy text compared bit-identically against a recorded reference run, plus the tiny-fp8 fixture's token-level oracle vs the python reference (cos ≈ 0.99999). Happy to PR back the pieces that look useful — fmt=9 (bf16 dense) and the staging-budget counters on this hardware class are the first candidates.

[DebugSultan]

Follow-up with fresh numbers: the fork has now merged upstream/dev, and we measured your tier, ours, and a hybrid composed of both on the same rig in paired runs. TL;DR: each tier wins on a different half of the model, and the composition beats both.

Setup: RTX 5070 Ti + 5060 Ti (2×16 GB, sm_120), Qwen3.8-Flash-Next 176B MoE. Expert snapshots pinned under --memlock=unlimited (39.7 GB wired, 0 failures), same 274-token prompt, page-cache dropped between runs. Tiers are runtime-selectable in the merged fork: Q38_TIER_TRUNK=0|1|2 (ours / yours / hybrid).

Decode, ms per forward (64-token runs):

timer	ours	your tier	hybrid
resident dense matmul	100.6	42.4	44.9
deltanet	69.9	33.2	32.7
routed experts	33.5	231.9	22.8
TTFT (prefill, CPU)	20.5 s	45.8 s	19.6 s
decode tok/s	3.63	2.85	5.08

What the timers say: your resident-int8 dense trunk is strictly better than our eager bf16 port (100.6 → 42.4 without budget gymnastics), while your expert streaming collapses on a long-prompt working set (~10% residency → 232 ms/fwd on the routed side) — which is exactly the half our pinned arena covers. The hybrid runs our arena over the experts and your qt_* trunk over the dense side: the dispatcher just prefers the trunk's GPU handle, no new compute. The 5 GiB/card floor the arena must leave for the trunk is now the mode-2 default in code, so the composition is correct by configuration.

Honest horizon correction: the +40% above is a short-burst number — at 64 tokens only the hot tail of the expert set gets touched. At a sustained 512-token run the arena walks its cold tail and the advantage compresses to wall 3.99 vs 3.64 tok/s, pure-decode 211.7 vs 235.9 ms/fwd (+11% over ours); the hybrid still leads, and still beats your tier's 2.85 on this workload.

Quality so far: Q38_TRUNK_SELFTEST shows the GPU int8 GEMV matching the CPU int8 path to ~1e-7 rel.err; greedy output vs our tier is bit-identical for the first ~180 tokens of a 512-token run, then diverges benignly in phrasing (same content, no degeneration). A 64-token sample was byte-identical across all three arms.

The fmt=9 conflict from the merge is reconciled too: your weight_at predicate stays exactly as upstream ({0..4}), and bf16 decoding now lives in a dedicated branch of the matmul kernel with a device-side trap test for unsupported formats. If a bf16 path back upstream is useful we'd rather PR it than carry a divergence — and we think the hybrid tier is worth a discussion: the measurements say the two designs are complements, not competitors.

Thanks for #1424 — the trunk was worth the merge.

[kreuzzelg]

@DebugSultan thank you, this is the kind of measurement the tier needed and could not get on my two 8 GB cards: a 2x16 GB rig, pinned experts, paired runs, and an honest 512-token correction. Three reactions.

The composition is the right reading, and it is what the numbers on my side already said. In #1424's table the expert tier on its own moved nothing (1.34 -> 1.33 tok/s) while the trunk moved everything (+34 %); the tier's value was the 6 % it added on top of the trunk at 22 % residency. Your arena gives the routed side what heat-promotion at touch time cannot on this model -- Qwen3.8 has no hot set that carries over between prompts (measured in #1424: chance-level overlap between two runs), so a streaming tier only ever holds the recent tail. A pinned set with enough VRAM behind it is the other answer, and 232 -> 23 ms on the routed side is that answer. Two complements, not two competitors; I would sign that sentence.

One thing to keep visible in the hybrid: what the arena holds is a different model. Your experts are an int4 gs64 container, the trunk tier reads the FP8 checkpoint as shipped. On Qwen3.6 that int4 gs64 quantization costs 2.5 % perplexity against Ollama's Q4_K_M and 2.4 % against int8 on the same tokens (#1370, five rows), and on Qwen3.8 even the int8 trunk cost 1.9 % per row until the group scales in #1552 brought it to 0.4 %. So the hybrid's speed comes with a quality line that "greedy identical to our own tier for 180 tokens" does not measure -- both arms carry the same int4 experts. The line worth adding is the one the protocol asks for: greedy output against Q38_TRUNK_GPU=0 on the FP8 checkpoint (the BF16/FP8 CPU reference), and if you can spare an hour, a perplexity run on a fixed text against the same reference (07_Tests/UserTest/ppl_vs_ollama/ppl_colibri.py in my workspace does it, 16 x 512 tokens; I can post the token file). If int4 gs64 experts cost Qwen3.8 what they cost Qwen3.6, that is the number a user needs next to 5.08 tok/s.

On bringing it upstream. The two halves have natural homes: the resident-copy arena is the "arena per device" follow-up I named in #1424 (cudaMalloc granularity wastes 22 % of the fp8 experts' VRAM; a fixed pool fixes that too), and qt_trunk_offer/qt_dense_* are already generic. fmt=9 (bf16 dense) as a PR on its own would be the cheapest first piece. For the arena I would suggest the tier's shape rather than a parallel tier: a pinned set is a qt_note_planned whose entries never get evicted, and the fake-backend tests in tests/test_qwen36_tier_* would then cover it without a card. Happy to review either, and to run whatever you send on the 3070/Quadro pair for a second data point.

[DebugSultan]

@kreuzzelg thanks — three of your points land squarely, and there is fresh data from last night that belongs in this thread.

The S == 1 gate on the dense GPU path was conservatism, not a kernel limit — and I removed it. quant_matmul already launches grid (O, S), y[s*O+o] is the same row-major layout the CPU produces, and coli_cuda_matmul has always taken S. On a rig where prefill is 19.7 s of CPU BF16 GEMM, the pinned BF16 reference costs real wall time, so: a qt_dense_matmul_batch wrapper + an opt-in dispatcher branch (Q38_TRUNK_PREFILL=1, default keeps your semantics). Measured at 3006 tokens: TTFT 271.9 → 223.3 s (-17.9%), resident-mm 105 → 57 s, and greedy output bit-identical to the CPU BF16 reference at that context; the Q38_TRUNK_SELFTEST batch path is 330/331 matrices at ~1e-7 rel.err. The remaining dense time is placement-bounded (p_marginal ~0.15 says the small matrices should stay on CPU). Happy to shape this however you want it upstream — including keeping it opt-in.

The quality point is correct and I accept it. Greedy-vs-greedy across two tiers that share the same int4 experts measures the trunk, not the container — it cannot see the int4 gs64 cost at all. The arms you asked for are fair and I owe them: greedy against Q38_TRUNK_GPU=0 on the FP8 checkpoint, and a perplexity run on a fixed text against the same reference. I'll get those measurements to you as soon as I have the chance — posting the token file here would let me run your exact 16×512 protocol and keep the numbers comparable across rigs.

Honest correction to my own battery, before you build on it: last night's rig turned out to be measuring under memory pressure — zram/swap churn and file-page refaults (PLE preads and non-pinned warm experts) were contaminating A/B deltas. Same binary, same prompt, only the swap state changed: hybrid wall 3.99 → 4.61 tok/s, and the no-trunk arm moved the other way without the valve (3.64 → 2.73). So treat the +11%/+40% decode numbers as pending a rebaseline with the memory regime declared and reclaim counters logged per run — I'd rather flag it myself than have it found later. (Your chance-level hot-set overlap from #1424 fits what I'm seeing: the engine re-reads ~1 GB/s of NVMe mid-run on warm expert refaults; the arena's pinned set is precisely why it survives that.)

On upstream: agreed on both halves. fmt=9 (bf16 dense) as its own first PR is the cheapest piece — I'll open it. And framing the arena as qt_note_planned entries that never evict is better than a parallel tier: it inherits the fake-backend tests and the cudaMalloc-granularity fix falls out of the pool shape for free. and the second data point from your 3070/Quadro pair is genuinely welcome — no need to wait for a PR to spend that offer: the fork is public and DebugSultan/colibri @ qwen38cuda already carries both pieces — the resident int8 trunk (Q38_TIER_TRUNK) and the batched-prefill gate removal (Q38_TRUNK_PREFILL=1, with the Q38_TRUNK_SELFTEST batch line). The prefill path needs only the trunk placement, not our int4 containers, and on your cards the arena just settles at a lower residency. If a -17.9% TTFT at ~3k tokens replicates on Ampere+Quadro, that is worth knowing before shaping the upstream version.

---

EDIT 2026-09-19 — the rebaseline I promised above is done, and it corrects two things I said. Same comment edited rather than a new one, so the thread stays readable.

1. Fixed memory regime, and this time the regime is part of the measurement. Prod stopped (GPUs at 2 MiB), swapoff -a, preflight verifying SwapTotal=0 kB, and per-arm evidence: pswpin/pswpout delta 0 pages on all four arms, read_bytes 2.4–3.8 MB per run (i.e. no disk I/O during the measurement — 34.70 GB of experts wired with mlock, 0 lock failures), storage probe 5.73 → 5.68 GB/s across the battery, and parallel-batches = 1258/1258/2056/1258, all > 0, so no arm sat in the cap < topk cliff.

2. The decode numbers I asked you to treat as pending did not survive. 64 tokens, CACHE=8192 BITS=8 OMP=20, 2k prompt, same read-only heat file for every arm: ours 0.2193 tok/s — hybrid (Q38_TIER_TRUNK=2) +0.69% — hybrid_pf (+Q38_TRUNK_PREFILL=1) +0.17% — trunk (Q38_TIER_TRUNK=1) −12.39%. The first two are noise, and the counters say so unambiguously: across ours, hybrid and hybrid_pf, weight-ranges, coalesced-gate-up, prefetched, parallel-batches, cache hit rate (92.1%) and the GPU/CPU split (70.1% on GPU) are identical digit for digit. So: the +40%/+11% hybrid gains were an artifact of the contaminated regime — please do not carry them forward.

3. The part that concerns your tier directly, and it is my mistake to report. In that battery your VRAM expert tier showed VRAM hit rate 0.0% | LFRU swaps 0 in every trunk arm — 553 tensors and ~4.0 GB resident in trunk, 16590+16707 tensors and 14.24+14.34 GB in hybrid_pf, and zero hits in all of them. It was not a policy that never promotes. It was unreachable by construction on my side: my battery sets Q38_INT4_SNAP, q38_load_expert opens with if (q38_try_load_int4_expert(...)) return;, so every expert lands in i4_slab and fp8_slab stays NULL — and my note hook requires a native FP8 slot (ex->fp8_slab && ex->gate.data == ex->fp8_slab), so qt_note() was never called once. q_skips 0 rules out queue congestion. Positive control, same arm with only Q38_INT4_SNAP removed: resident 115/24576, uploads 115, hit rate 4.9% — the tier wakes up exactly as designed.

Consequence, stated plainly: that −12.39% measures your dense int8 trunk (553 of 553 offered matrices resident as int8, genuinely active in every arm) with your expert tier switched off. On experts, the two designs were never compared. Please do not read it as a verdict on your expert tier — it isn't one, and I would rather correct that here than have it quoted somewhere as if it were.

For what it is worth, the dense int8 trunk is the best dense path measured on this rig (42.4+33.2 vs 100.6+69.9 ms/fwd), and that result is unaffected by any of the above.

4. Method note, since it cost me an arm. I first hypothesised the starving guard came from the mapping branch in q38_load_native_fp8_ranges and predicted COLI_MAP_EXPERTS=0 would wake the tier. Control run: no change, miss(CPU) 3360 identical bit for bit. The chain was coherent and simply not the one being taken. A coherent code path is not a measurement, and the control has to be run on the flag you believe is causal.

The fmt=9 (bf16) PR is now open: #1620, against dev @ 5a0b725, with the test green. It also carries a small NULL-memcpy fix the test caught in coli_cuda_tensor_update().

[kreuzzelg]

@DebugSultan the second data point, RTX 3070 (sm_86), the 315-token prompt from the #1424 table, cap 224, Ollama stopped, one process at a time. Two binaries: your fork at qwen38cuda (Q38_TIER_TRUNK=1, 5 tokens), and the same gate removal on top of upstream dev + #1552 (qt_dense_matmul_batch, 30 tokens):

	your fork	upstream + the same change
TTFT, prefill through the trunk off	58.1 s	178.6 s
TTFT, prefill through the trunk on	62.4 s (+7 %)	111.1 s (-38 %)
prefill resident-mm, off -> on	6.4 -> 10.4 s	76.4 -> 13.1 s
prefill DeltaNet projections, off -> on	7.8 -> 9.6 s	50.7 -> 11.0 s
greedy text vs the BF16 CPU run	5 tokens equal	30 tokens identical in both modes

Two things fall out.

1. On upstream the gate removal is a large win at 315 tokens, and it is yours. -38 % TTFT, text identical, selftest 553/553 -- so I have it as a commit on top of #1552 (default on, Q38_TRUNK_PREFILL=0 keeps the BF16 CPU prefill), attributed to this thread, and will open it once #1552 is in. What remains of upstream's prompt time is the routed experts on the CPU (50 s GEMV, 34 s reads), the trunk is no longer in it.

2. Your fork's CPU prefill is three times faster than upstream's, and that is the bigger number in this table. 58 s against 179 s with the trunk prefill off: prefill resident-mm 6.4 s against 76 s, DeltaNet 7.8 against 51 s, routed experts 11.6 against 50 s. That is not the GPU, it is your configurable prefill batch rows (upstream fixes 32), the larger workspace and the parallelised attention prefill. It also explains why the trunk prefill loses on your side at this length (+4 s): with the CPU GEMM that fast, 553 uploads of 315 rows plus a GEMV-shaped kernel cost more than they save, and the -18 % only appears once the prompt is long enough (your 3k). On upstream the same knob wins from the first token because the CPU baseline is slow. If you PR the prefill batching, I would expect it to be the more valuable of the two pieces for everyone on upstream, and the trunk prefill then becomes the long-prompt complement.

On test_bf16_cuda.cu: runs clean here (65536/65536 patterns bit-exact, 0 failures) after the one-line change I left on #1620. And the token file: I will get it to you; it is the wikitext-2 test split tokenised with the checkpoint's tokenizer, 297k ids, so tokenize_text.py from my harness on wiki.test.raw reproduces it exactly if a file is inconvenient.