Repository navigation
Conversation
…ogress) A prompt's layers now walk a group of up to two chunks' rows: the residual stream, its coefficients and the embedding staging are as tall as the group, every other row buffer stays a block (a chunk) tall, and a layer runs its attention half then its feed-forward half over each block. The experts are placed for the group, so one ring pass serves both blocks. Measured on the RTX 5090 at 475 W, NVFP4 pack, fp16 KV: pp32768 (28,270 tokens) 3,933 t/s against main's 3,493-3,586, i.e. +10-13% from one expert stream instead of two. pp16384 unchanged within drift. Not ready for review: - When the group does not fit the lend, the fallback is broken: a 32,768-token run failed with 'out of device memory for 13 rows' and then 'tier settle copy sync: an illegal memory access', so the failed lend/buffer path needs fixing before any PR. - Numeric parity is not exact for multi-block prompts. A 20,000-token prompt is bit-identical through layer 1, with xlat identical and moe_lat differing by 2e-9 in 0.002% of values, amplifying to the last row's logits; a 17,000-token prompt differs from layer 1 on because its 616-row tail is a short chunk in the chunked read and a ringed block here (a deliberate placement change). - The peer/second-card plan and the two-block group's quality gate (--quality KLD/PPL) are untested.
With the layer loop running attention then feed-forward per block, Xn only ever holds one block, so it is sized by the block instead of the group; the two-chunk group then fits the lend at 28,270 tokens (10.86 GiB asked, the buffer set and the ring both land). Instrumentation confirms the premise too: 41 ring starts for a 28,270-token prompt, one per tier layer, against 82 for the same prompt read as two chunks. It is still slower. Clean ABBA on the RTX 5090 at 475 W, pp32768: group 3,447 / 3,490 t/s against main 3,675 / 3,675, 5.6% down. Halving the expert stream does not show in wall time, so the stream was already overlapped with compute in the chunked read; the added cost is the second block's MoE pass per layer (its stream sync, routing copy to the host and pair list), which the block-major loop doubles. Kept for the record, not for merging.
673ef5d to
85e318c
Compare
|
Closing this as a negative result, with the numbers, because the premise turned out to be wrong and the branch is slower. The premise held. A prompt's layers can walk a group of two chunks with the experts placed once: instrumented ring starts are 41 for a 28,270-token prompt (one per tier layer) against 82 for the same prompt read as two chunks, and with the second residual stream sized by the block instead of the group the two-chunk group now fits the lend (10.86 GiB asked; buffer set and ring both land). It is still slower. Clean ABBA, RTX 5090 at 475 W,
5.6% down. Halving the expert stream (56.6 GiB per pass, ~1.23 s at the RAM's 47 GB/s) does not appear in wall time, so in the chunked read that stream is already overlapped with compute rather than exposed. What the group does add is a second MoE pass per layer — its stream sync, its routing copy to the host, its pair list — and that costs more than the stream it saves. Rebased onto current main first (the tier's |
Draft: the gain is measured, the failure path and the parity gate are not done.
What it does
A prompt's layers walk a group of up to two chunks' rows. The residual stream (
X/Xn), its hyper-connection coefficients and the embedding staging are as tall as the group; every other row buffer stays one block (a chunk) tall. A layer runs its attention half over each block (writing the post-attention stream into the second buffer) and then its feed-forward half over each block (writing the layer's output back), so no whole-group swap is needed. The experts are placed for the group, so one ring pass serves both blocks instead of one pass per chunk.Measured
RTX 5090 at 475 W,
models/Hy4-flash-NVFP4.basalt, fp16 KV,basalt-bench -p N:The engine log confirms the group was granted, not fallen back:
prompt group 28269 rows, 16384 a block (partial lend)(11.09 GiB lent, 1,887 of 4,879 slots kept).Known issues (why this is a draft)
out of device memory for 13 rowsand thentier settle copy sync: an illegal memory access was encountered/tier settle out sync …. When the group does not fit, the release/retry sequence leaves the tier in a bad state instead of falling back to two chunks.xlatis identical;moe_latdiffers by 2e-9 in 0.002% of values, amplifying to the last row's logits. 17,000 tokens: differs from layer 1 because its 616-row tail is a short chunk in the chunked read (CPU/bus staging) and a ringed block here — a deliberate placement change. Single-block prompts are token-identical.moe_lat-1(25,165,824 vs 30,720,000), i.e. it did not read the same prompt.--qualityKLD/PPL) and no two-card run yet; the second card's away plan is now the group's, andstage_awaystill fires per block.Next
Fix the lend-failure path first, then determinism, then the quality gate and the two-card plan.