Repository navigation
Conversation
…IN_EXPERTS) The tier's host path (qt_issue / qt_take) was built for an engine whose residual lives on the host: the row into pinned memory, one 256-thread block per output row per expert, the result back, a wait -- every layer, 9.9 ms of a chain token's 18.7. With the chain the row is already on the device, so the chain asks the tier for the residents' tensors (qt_resident_open: hits and heat counted, the group held open against eviction and in-place swaps until the frame they ran in has been waited for; qt_resident_close) and runs them itself through a new op of the chain's table, `experts`: two launches per row on the chain's stream, behind the shared expert -- a warp per output row, 16 bytes a lane, silu(gate x) * up x for every resident into a scratch, then every output row summed over the residents with the router's weights, the way gemv_i8_rows reads the trunk. int8, int4 per row and int4 grouped with gs a multiple of 32 are taken, gate/up and down may differ (--down-bits); a refusal launches nothing and does not lose the device, the route turns itself off and the layer goes the host path. The CPU's share of the row is computed meanwhile as before and comes up through the pinned buffer; the combine adds the device's sum and the host's first. One device only: with the tier on two cards the host path stays. The descriptors travel as a by-value kernel parameter (ptxas: 0 bytes stack, 2.9 KB cmem), so no staging copy can outlive its host buffer and no pageable source forces a stream sync. tests/test_qwen36_cuda_chain.c runs the route through the fake's `experts` (the kernels' arithmetic on the fake's int8 copies; the slots offered to the tier, QT_UPLOAD_SYNC=1) against the CPU path, and COLI_CUDA_CHAIN_EXPERTS=0. tests/test_cuda_chain.cu checks the op on a card: per-row int4, grouped gate/up over an int8 down, one expert alone, a gs-16 refusal that leaves the device to the chain, the counter. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y nibbles The experts case of tests/test_cuda_chain.cu packed two's-complement nibbles; the host's int4 is offset-binary (value + 8) and the upload turns it signed (offset_to_signed_s4), so the device read every weight shifted. With the container's packing the three cases match the reference to 1e-4 on the 3070. qwen36-cuda-tier.md gets the measured rows: 18.9 -> 16.4 ms a decode token, TTFT 1.05 -> 0.81 s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Thanks for this. It no longer merges into dev: the conflicts are in |
|
@JustVugg Thanks for the look. Before I rebase: dev has had
dev's path is faster here (and its first token 0.35 s against 0.82), so I am closing this one. One thing came out of the measurement: with a layer's experts partly on the host, the default hung at the second growth of the decode attention's scratch ( Thanks to Claude Code for the help along the way. |
The follow-up #1992 named as its first lever: the routed experts were 9.9 of the chain's 18.7 ms a token, and @m4t5k4 saw 9.3 ms of them with every expert resident. The tier's host path (
qt_issue/qt_take) was built for an engine whose residual lives on the host: the row into pinned memory, one 256-thread block per output row per expert, the result back, a wait -- every layer. With the chain the row is already on the device.What it is
The route. The chain asks the tier for the residents' tensors (
qt_resident_open: the hits and their heat counted as before, the group held open against eviction and in-place swaps until the frame they ran in has been waited for;qt_resident_close) and runs them itself through a new op of the chain's table,experts: two launches per row on the chain's stream, right behind the shared expert -- a warp per output row, 16 bytes a lane,silu(gate x) * up xfor every resident into a scratch, then every output row summed over the residents with the router's weights, the waygemv_i8_rowsreads the trunk. int8, int4 per row and int4 grouped (gs a multiple of 32, so a 16-byte load never straddles a group) are taken, gate/up and down may differ (--down-bits). The CPU's share of the row is computed meanwhile as before and comes up through the pinned buffer; the combine adds the device's sum and the host's first. Nothing crosses the bus for the residents and the layer has no wait of its own: the next layer's A1 wait covers it.Where it stops. One device only -- with the tier on two cards the residents of the other card would have to cross, so the host path stays (
qt_resident_capable). A format the kernels decline launches nothing, does not lose the device, and turns the route off from that layer on (the message names the layer).COLI_CUDA_CHAIN_EXPERTS=0keeps the host path for an A/B. The descriptors travel as a by-value kernel parameter (ptxas: 0 bytes stack, 2.9 KB cmem): no staging copy that could outlive its host buffer, no stream sync for a pageable source. No new export: the table grew, the symbol did not; an older DLL's shorter table is refused by the size check as before.Measured (RTX 3070 alone, Qwen3.6-35B-A3B int4,
COLI_PLACE=auto, the tier at 84 % hits, the usual 49-token prompt, 200 new tokens, two runs each)COLI_CUDA_CHAIN=1, residents through the host pathCOLI_CUDA_CHAIN=1, residents on the chain's stream-13 % a token and a quarter off the first token (the prompt's rows take the route too). What is left of the host's part is the CPU's misses (16 % of the experts, 4.8 ms, overlapping the device's frame); the device's wait grew by the residents' kernels (435 MB of expert rows a token in about 1.7 ms). The text opens as the host path's and diverges after about 150 characters, as two orders of the K summands do.
Verified
tests/test_qwen36_cuda_chain.c(CI): the route through the fake'sexperts(the kernels' arithmetic on the fake's int8 copies; the slots offered to the tier,QT_UPLOAD_SYNC=1) against the CPU path -- the prompt, three decode tokens, the state read back, the counters; andCOLI_CUDA_CHAIN_EXPERTS=0. Every earlier check unchanged.tests/test_cuda_chain.cuon the 3070 (make cuda-chain-check CUDA=1): three per-row int4 experts of one row at an offset, one alone, grouped gate/up (gs 32) over an int8 down, a gs-16 refusal that leaves the device to the chain, the counter; everything from qwen36: every layer as one chain on the CUDA device (COLI_CUDA_CHAIN=1), shaped as vk_chain.h -- 30 -> 18.7 ms a token on an RTX 3070 #1992 and qwen38: every layer as one chain on the CUDA device (COLI_CUDA_CHAIN=1), on the qwen36 chain's table #2089.CUDA=1;test_backend_loader.py,test_backend_loader_header_parity.py,test_makefile_vk_obj.pygreen.test_qwen38_cuda_chainunchanged (qwen38's chain does not take the route yet: its fp8 experts are not among the formats).Thanks to Claude Code for the help along the way.