qwen36: offer the rest of the dense trunk to the VRAM placer (dnout, attnproj, shexp)

[JustVugg]

Follows the M10 datapoint in #1652, which showed the ceiling on a CPU without AVX2: with every expert VRAM-resident, DeltaNet 169 ms and attention 55 ms of a 384 ms token stayed on the CPU, because qwen36 offered the placer only lm_head and the DeltaNet input projections.

Where the time is

Measured on a 16-core CPU (ds, 8 threads), the DeltaNet stack per decoded token: 37.5 ms = input projections 23.4 | out_proj and norm 8.3 | convolution 3.3 | recurrence 2.4. The matmuls are the cost, not the recurrence, so the trunk is what goes to VRAM, and the tier already knew how: the placer prices any component by name, and qt_dense_init/qt_dense_matmul are what Qwen3.8 runs its whole trunk on. qwen36.c simply never offered these.

What is offered now

component	what	per layer (35B)	total
dnout	DeltaNet out_proj	8.4 MB	252 MB
attnproj	q, k, v, o of an attention layer, one item	27.3 MB	273 MB
shexp	the shared expert's gate, up, down, every layer	3.1 MB	126 MB

Offer order after lmhead and dnproj, each in layer order, so a partial placement is whole layers (#1361). A component is offered only when every one of its matrices has a dense-i8 copy. On decode a placed matrix answers from VRAM with one GEMV; a prompt batch keeps the batched CPU matmul. Every matrix falls back on its own: a failing handle is turned off by the tier and matmul_d takes over, so the CPU path is exactly the one that runs today when nothing is placed or the build has no CUDA. COLI_PLACE accepts the three new names like the old ones (dnout=0,attnproj=0,shexp=0, per layer, + splits).

Verified without a GPU

• tests/test_qwen36_trunk_dense.c on the fake CUDA backend (in make check): the offers carry the right names, layers and bytes; a component with one matrix missing its dense-i8 copy is not offered; eight matrices placed and uploaded once each; every GEMV from VRAM equals matmul_d on the CPU (relative gap <= 2e-7).
• CPU path: qwen36 exact on the tiny fixture and byte-identical in chat (ds); decode timings unchanged (36.97 vs 37.5 ms DeltaNet per token, noise); make check 1,292 tests green.

What needs a GPU

The measurement. Same command as today, the placement is automatic:

COLI_CUDA=1 COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto COLI_TIMERS=1 ./qwen36 256 4 prompt.txt

The [place] and [dense] lines at startup say what landed; the [timers] lines give the per-phase split; COLI_PLACE=off is the A/B control, and the output must stay byte-identical between the two. Asking the reporter of #1652 (4x M10) and anyone with an 8 GB card to run it before this is merged.

[JustVugg]

@kreuzzelg if the 3070 is free: this is the single-card case the tier doc's calibration table was built on (auto placed lm_head + dnproj there, +49% warm). On this branch the placer also gets dnout, attnproj and shexp, about 650 MB more of int8; on an 8 GB card it will displace some experts, and the pricing rule decides. The two numbers worth having are warm tok/s with auto against COLI_PLACE=off on the same build, plus the [place] lines, with byte-identical output between the two. No hurry, and thank you.

[kreuzzelg]

@JustVugg measured on the 3070 alone, dev 4e28e39 (#1657 in, #1753 head), per-row int4 container, 49-token prompt, 200 new tokens, COLI_GPUS=0 CUDA_EXPERT_GB=auto, 12 cores OMP_PLACES=cores, warm = HEAT_FILE from a first run. Short version: the three new components are worth +17 % over the old placement and +60 % over off on this card, but auto delivers none of it, because the startup probe withdraws the trunk.

COLI_PLACE	trunk in VRAM	experts resident	hit rate	warm tok/s	TTFT
off	--	3,836	91.1-91.5 %	13.47 / 14.45 / 14.48	1.0-1.2 s
auto as shipped	withdrawn by the probe	3,836	90.9-91.6 %	13.61 / 12.96 / 15.02 / 14.74	1.0 s
lmhead=0,dnproj=0 (the old auto)	1.18 GB	3,093	85.6-86.1 %	19.12 / 19.28	1.2 s
all five by hand	1.79 GB	2,717	81.9-82.2 %	22.36 / 22.56	1.4 s
auto + COLI_TRUNK_PROBE=0	1.79 GB	2,717	82.5 %	22.22	1.4 s

Cold (no heat file): off 10.53, auto (withdrawn) 11.16 tok/s, both 38 % hit, output byte-identical. Peak VRAM 6.38 / 6.47 / 6.51 GB for off / old / full.

The probe times the L3, not the memory bus. [place] lines of every auto run:

[place] auto: dev 0 holds 1828.5 MB of trunk (lmhead yes, 30 dnproj layers), 4.30 GB left for experts
[place] probe: one [8192 x 2048] int8 GEMV takes 0.166 ms on CUDA dev 0, 0.095 ms on the CPU -> trunk stays on the CPU
[place] trunk stays on the CPU (measured slower than the CPU): 1.79 GB of VRAM back to the experts

0.095 ms for 16 MiB is 176 GB/s, which no DRAM on this box delivers: trunk_probe_gpu_wins hits the same dn_qkv matrix 13 times, 12 threads see 1.4 MiB each, and on a Zen 2 that sits in the CCX's 16 MiB L3. Decode pays the bus: the timers show 0.7 ms per DeltaNet projection (dn-sub: proj 21 ms over 30 layers). One-line fix, tried locally: let the CPU side rotate over all thirty dn_qkv matrices of the same shape (720 MB, no cache holds that; the GPU side can stay on one handle, the 3070's 4 MB L2 never held 16 MB anyway):

[place] probe: one [8192 x 2048] int8 GEMV takes 0.163 ms on CUDA dev 0, 0.355 ms on the CPU (rotating over 30 matrices) -> trunk to VRAM

0.355 ms is 47 GB/s, the real number, and auto then lands the trunk: 21.3 / 18.9 tok/s. The M10 verdict in #1652 stays intact (68.8 vs 41.7 ms is a 1.6x loss, far above cache effects), but a probe that reads one 16 MB matrix in a loop will fail any CPU with >= 16 MiB L3 per core group, which is every Zen 2+ and most Xeons. Patch is 12 lines, happy to PR it or leave it to you.

Per component, heat file frozen (chmod a-w), two runs each, decode timers vs off:

COLI_PLACE	resident	tok/s a / b	what moved (ms/token)	prefill at 49 tokens	two runs byte-identical
off	3,836	14.4 / 14.5	reference: deltanet 31.9, attention 8.5, shared 4.0, lm_head 11.4, norm+out 6.6	dn 411, attn 53, sh 64, head 11 ms	yes
lmhead=0,dnproj=0	3,055	18.1 / 19.0	deltanet -> 21.8, lm_head -> 4.4	head -> 4	yes
dnout=0	3,642	15.4 / 15.5	norm+out 6.6 -> 3.1	dn +56 ms	yes
attnproj=0	3,629	15.4 / 15.4	attention 8.5 -> 4.7	attn 53 -> 131 ms	yes
shexp=0	3,715	14.6 / 14.3	shared 4.0 -> 7.8, cpu-miss 7.0 -> 10.7, take 2.3 -> 0.2	sh 64 -> 408 ms	yes
all five	2,717	21.4 / 21.1	sum of the above, cpu-miss 12.3	dn 681, attn 132, sh 345	no (diverge at token ~35, LFRU swaps 10 vs 12)

Two things in there:

1. shexp loses on this card, both ways. Three GEMVs of ~1 MB per layer, 120 synchronous launches per token: the shared expert goes from 4.0 to 7.8 ms/token. And it sits between qt_issue and qt_take, where it used to hide the expert group's latency; on the same device it now serialises with it (take 2.3 -> 0.2, cpu-miss +3.7 ms at nearly the same resident set). dnout (-3.5 ms) and attnproj (-3.7 ms) carry the full list past that. I would drop shexp from the offer, or fuse gate+up into one launch and price the launch count, before it goes to a card weaker than a 3070.

2. Prompt rows do go to the card. "A prompt batch keeps the batched CPU matmul" holds for dnproj, but shexp and dnout call qtd() per prompt row (qwen36.c around 2197/2482/2689) and attnproj calls qtd_batch(), and all three are slower than the int8 idot GEMM on 12 cores (qwen38: int8 trunk with integer dot products and a vector FP8 expert kernel #1668): TTFT 1.0 -> 1.4 s at 49 tokens, i.e. ~7 ms per prompt token for shexp, 1.6 for attnproj, 1.1 for dnout. Same shape as the trunk-prefill crossing in Qwen3.8-Flash-Next: is a GPU backend (CUDA/HIP) planned for the qwen38 engine? (offer: gfx1151 / ROCm 7.2.4 validation + benchmarks) #1306: a per-row dense kernel streams the weights once per row. At 2k prompt tokens that is seconds; S > 1 -> matmul_d for these three would give the PR text back its meaning.

Byte-identity, honestly: without a frozen heat file no two warm runs agree, not even off against off (HEAT_FILE is rewritten after every run, the resident set shifts, and a resident expert on the GPU kernel and a miss on the CPU int8 slot do not sum in the same order). With the heat frozen every configuration reproduces itself exactly, except the full list under the tightest budget, where the LFRU count differed between two identical runs. off against any placed variant diverges at token 30-35 on this container, from the changed resident set alone. So on an 8 GB card, where the trunk displaces experts, off = auto byte-identity is not a measurable property; what is measurable is that each placed component is deterministic and that the cold pair (same resident set, trunk on the CPU in both) was identical.

Cosmetic: the [place] auto: summary counts the new components' bytes (1828.5 MB) but names only lm_head and dnproj; the [dense] 190 trunk matrices line comes later. Logs and runner under 07_Tests/UserTest/r6_1657/ in my workspace if you want any of the raw files.