Skip to content

hy4flash: a prompt runs a group of two chunks layer-major (draft) - #470

Closed
giveen wants to merge 2 commits into
jesdga95:mainfrom
giveen:perf/prefill-layer-major
Closed

giveen wants to merge 2 commits into
jesdga95:mainfrom
giveen:perf/prefill-layer-major

Conversation

@giveen

@giveen giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Draft: the gain is measured, the failure path and the parity gate are not done.

What it does

A prompt's layers walk a group of up to two chunks' rows. The residual stream (X/Xn), its hyper-connection coefficients and the embedding staging are as tall as the group; every other row buffer stays one block (a chunk) tall. A layer runs its attention half over each block (writing the post-attention stream into the second buffer) and then its feed-forward half over each block (writing the layer's output back), so no whole-group swap is needed. The experts are placed for the group, so one ring pass serves both blocks instead of one pass per chunk.

Measured

RTX 5090 at 475 W, models/Hy4-flash-NVFP4.basalt, fp16 KV, basalt-bench -p N:

prefill main this branch
pp16384 (14,155 tokens) 3,497–3,516 3,557
pp32768 (28,270 tokens) 3,493–3,586 3,933 (+10–13%)

The engine log confirms the group was granted, not fallen back: prompt group 28269 rows, 16384 a block (partial lend) (11.09 GiB lent, 1,887 of 4,879 slots kept).

Known issues (why this is a draft)

  1. The failed-lend path is broken. A 32,768-token run, with less free VRAM after the preceding runs, failed with out of device memory for 13 rows and then tier settle copy sync: an illegal memory access was encountered / tier settle out sync …. When the group does not fit, the release/retry sequence leaves the tier in a bad state instead of falling back to two chunks.
  2. Parity is not exact for multi-block prompts. 20,000 tokens: layers 0 and 1 are bit-identical and xlat is identical; moe_lat differs by 2e-9 in 0.002% of values, amplifying to the last row's logits. 17,000 tokens: differs from layer 1 because its 616-row tail is a short chunk in the chunked read (CPU/bus staging) and a ringed block here — a deliberate placement change. Single-block prompts are token-identical.
  3. Run-to-run determinism is unproven: a repeat run of the same binary dumped a different row count for moe_lat-1 (25,165,824 vs 30,720,000), i.e. it did not read the same prompt.
  4. No quality gate (--quality KLD/PPL) and no two-card run yet; the second card's away plan is now the group's, and stage_away still fires per block.

Next

Fix the lend-failure path first, then determinism, then the quality gate and the two-card plan.

…ogress)

A prompt's layers now walk a group of up to two chunks' rows: the residual stream, its coefficients and the embedding staging are as tall as the group, every other row buffer stays a block (a chunk) tall, and a layer runs its attention half then its feed-forward half over each block. The experts are placed for the group, so one ring pass serves both blocks.

Measured on the RTX 5090 at 475 W, NVFP4 pack, fp16 KV: pp32768 (28,270 tokens) 3,933 t/s against main's 3,493-3,586, i.e. +10-13% from one expert stream instead of two. pp16384 unchanged within drift.

Not ready for review:
- When the group does not fit the lend, the fallback is broken: a 32,768-token run failed with 'out of device memory for 13 rows' and then 'tier settle copy sync: an illegal memory access', so the failed lend/buffer path needs fixing before any PR.
- Numeric parity is not exact for multi-block prompts. A 20,000-token prompt is bit-identical through layer 1, with xlat identical and moe_lat differing by 2e-9 in 0.002% of values, amplifying to the last row's logits; a 17,000-token prompt differs from layer 1 on because its 616-row tail is a short chunk in the chunked read and a ringed block here (a deliberate placement change).
- The peer/second-card plan and the two-block group's quality gate (--quality KLD/PPL) are untested.
With the layer loop running attention then feed-forward per block, Xn only ever holds one block, so it is sized by the block instead of the group; the two-chunk group then fits the lend at 28,270 tokens (10.86 GiB asked, the buffer set and the ring both land). Instrumentation confirms the premise too: 41 ring starts for a 28,270-token prompt, one per tier layer, against 82 for the same prompt read as two chunks.

It is still slower. Clean ABBA on the RTX 5090 at 475 W, pp32768: group 3,447 / 3,490 t/s against main 3,675 / 3,675, 5.6% down. Halving the expert stream does not show in wall time, so the stream was already overlapped with compute in the chunked read; the added cost is the second block's MoE pass per layer (its stream sync, routing copy to the host and pair list), which the block-major loop doubles.

Kept for the record, not for merging.
@giveen
giveen force-pushed the perf/prefill-layer-major branch from 673ef5d to 85e318c Compare November 19, 2026 22:40
@giveen

giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Closing this as a negative result, with the numbers, because the premise turned out to be wrong and the branch is slower.

The premise held. A prompt's layers can walk a group of two chunks with the experts placed once: instrumented ring starts are 41 for a 28,270-token prompt (one per tier layer) against 82 for the same prompt read as two chunks, and with the second residual stream sized by the block instead of the group the two-chunk group now fits the lend (10.86 GiB asked; buffer set and ring both land).

It is still slower. Clean ABBA, RTX 5090 at 475 W, pp32768 (28,270 tokens), same build except the branch:

run 1 run 2
group 3,447 3,490
main 3,675 3,675

5.6% down. Halving the expert stream (56.6 GiB per pass, ~1.23 s at the RAM's 47 GB/s) does not appear in wall time, so in the chunked read that stream is already overlapped with compute rather than exposed. What the group does add is a second MoE pass per layer — its stream sync, its routing copy to the host, its pair list — and that costs more than the stream it saves.

Rebased onto current main first (the tier's reclaim(restore=true) and the bounded lends), so this is not a stale comparison. The two commits are kept on the branch and as patches; I do not think it is worth merging.

@giveen giveen closed this Nov 19, 2026
@giveen
giveen deleted the perf/prefill-layer-major branch November 19, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant