[kreuzzelg]

The second chain, as agreed on #1992: Qwen3.8 Flash Next on the table #1992 brought (cuda_chain.h), qwen38_chain.h's frames and contract.

What it is

c/qwen38_cuda_chain.h. Per layer, one frame on the card: the previous block's MoE output joining the four hyper-connection streams (routed + sigmoid(gate) x shared, then hyper += inject x block), the gated residual's read (per-stream norm, the low-rank down/up pair, the stream mix, the inject weights: CC_EW_HC_*), the gated attention with QSA (the index keys into the device cache, each completed block's pooled key computed once, the indexer's top-k selection per query row, the attention over the listed positions with the output gate) or the Gated DeltaNet (the convolution with its ring, the recurrence with its state, the sigmoid-gated norm), the write-back, the second gated residual, the router logits; waited for. The host routes and runs the routed experts through the fp8 tier (q38_moe_ex routed only) and hands the sum up; the shared expert runs on the card meanwhile (frame A2). At the PLE layer the n-gram rows come from the host (disk reads, the history replayed as q38_ple does) and the key and value projections, the gate and the dilated convolution with its ring run on the device. The last frame runs the final mixer and lm_head; the streams come back when the MTP head needs them.

The matrices. The tier's int8 copies where the placer put them (Q38_TRUNK_GPU, COLI_PLACE); every other matrix of a layer, the final mixer and lm_head the chain uploads itself (Q38Weight.cc): the trunk's int8 rows when the CPU holds them (the same bits), else the BF16/F32 rows as f32. So the chain runs whatever the placement did; what it adds in VRAM are the matrices Q38_TRUNK_MIN_KB never offers (the gated residuals' low-rank pairs, the DeltaNet b/a rows, the PLE projections: 174 MiB on the 35B-class trunk) and the state (381 MiB: the K/V, index-key and pooled-key mirrors, the DeltaNet state and conv rings, the PLE ring).

State as qwen36's chain: the host's K/V and index caches canonical behind a watermark, the pooled block keys behind one of their own; the DeltaNet state, conv rings and PLE ring on the device while the chain runs, pulled before the host reads them (a pinned snapshot, the prefix cache) and pushed after it writes them (a reset as zeros, a restored snapshot as an upload). A verify runs row-wise (the warp GEMV computes each row on its own: a decode step's bits) and copies the DeltaNet state, the conv rings and the PLE ring after each of its rows into a device slot; a rejected draft swaps slot r in (q38cc_rollback) as the CPU swaps its own. A lost device zeroes the state, rebuilds it on the CPU from the prefix record (the PLE history replayed) and runs there from then on. Q38_DN_GPU's per-layer step is not set up beside the chain; KV_SLOTS > 1 keeps the chain off.

The table gains cc_qsa and cc_ple (chain_qsa.comp and chain_ple.comp ported line for line, appended to the table: an older DLL's shorter table is refused by the size check) and the recurrence at KD 4 (the tiny fixture's); the fake carries both. One fix in q38_forward: its final lm_head was bound to chain_logit under COLI_VULKAN only, so a second chain backend recomputed the logits from the CPU's mixed; it is bound for either now.

Measured (RTX 3070 alone, the FP8 checkpoint, the trunk's 553 matrices as int8 in VRAM, the tier at 16 % hits behind it, the usual 49-token prompt, 100 new tokens, two runs each)

	TTFT	tok/s, the run	ms per token	resident-mm / forward	DeltaNet	QSA	PLE	lm_head	shared
Q38_DN_GPU=1 (#1870, the per-layer step)	19.0-19.1 s	1.22 / 1.26	612-632	172 ms	161	23	12	26	23
COLI_CUDA_CHAIN=1	17.7-17.9 s	1.41 / 1.43	526-535	74 ms (the chain waits 68)	in the chain	in the chain	in the chain	10	0.4

The chain takes the whole dense part at once: -15 % per token and 1.2 s off the first token. The token stays the experts': about 1 s of expert reads per forward (24,576 FP8 experts behind a 4 GiB trunk on 8 GB) and 290 ms of routed experts, which no chain touches -- the trunk's weight is what you said on #1992 it would be. On a card with room for the experts the dense share would be the visible one. The chain is deterministic run to run; its text opens as the per-layer path's and diverges after about ninety characters (rounding order: float trees in the norms, the online softmax).

Verified

• tests/test_qwen38_cuda_chain.c (make qwen38-cuda-chain-check, in CI beside qwen38-dn-gpu-check): the tiny FP8 fixture's oracle on the fake tier, the chain's ops from the fake's host-side table -- the chain on (8/8 tokens, cosine 0.99998), a frame failing mid-run (the state rebuilt on the CPU, the run finishing there within the limits), the per-matrix path, and @mfethe1's case from qwen36: every layer as one chain on the CUDA device (COLI_CUDA_CHAIN=1), shaped as vk_chain.h -- 30 -> 18.7 ms a token on an RTX 3070 #1992: the fixture with its MTP head under Q38_MTP_FORCE=reject, every draft rejected and rolled back on the device, within the limits (removing the rollback's buffer swap fails it, 6/8 tokens).
• tests/test_cuda_chain.cu on the 3070: the QSA block keys and selections (ties as the CPU sorts, the causal case) and the PLE gate and convolution against their references, with everything from qwen36: every layer as one chain on the CUDA device (COLI_CUDA_CHAIN=1), shaped as vk_chain.h -- 30 -> 18.7 ms a token on an RTX 3070 #1992.
• Builds: CPU-only, CUDA=1; test_makefile_deps.py, test_backend_loader_header_parity.py, test_makefile_vk_obj.py green (no new export: the table grew, the symbol did not).
• Not run on the real checkpoint: --pin, the prompt cache and MTP drafting under the chain (each pulls or invalidates as the Vulkan chain does; the fixture covers the rejection path), and a second card.

Thanks to Claude Code for the help along the way.