Skip to content

hy4flash: a prompt that overruns the chunk cap by a step still reads in one chunk - #469

Open
giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:perf/prefill-one-chunk
Open

giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:perf/prefill-one-chunk

Conversation

@giveen

@giveen giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

What changed

The Hy4 prompt path picked its chunk as a power of two, so a prompt a few tokens over 16,384 read as 16,384 + a tail. The tail is a second run_rows call, and a call streams every expert of every layer again: on a 16,409-token prompt the 25-row tail cost 278 ms of expert stream and left the expert ring, so those rows fell below kRingRows and took the short-chunk path (2,695 rows staged from RAM over the bus, 2,821 on the CPU).

open_prompt now carries min(total, kMaxChunk + kAttRows) rows instead of rounding up to a power of two, and the memory fallback steps to kMaxChunk before halving. The margin is exactly one attention step, so the merged tail is the step the two chunks already ran: the step sequence inside the chunk is unchanged, and every prompt at or below 16,384 runs the same run_rows calls as before (only its buffer layout is now sized to the prompt instead of the next power of two).

Numbers

RTX 5090 at 475 W, models/Hy4-flash-NVFP4.basalt, fp16 KV, basalt-bench --model ... -p N --seed 1, two runs each:

prefill tokens chunks before after
pp16384 14,155 1 3,771 3,771
pp19000 16,409 16,384 + 25 → 1 3,495 ± 62 3,862 ± 40 (+10.5%)
pp32768 28,270 2 3,829 3,840

Engine log, 16,409 tokens: the expert stream drops 1,251 → 1,124 ms and the tail's bus/CPU staging disappears.

Quality

No row arithmetic changes: the same run_rows calls with the same row counts, only the chunk count for prompts that overran the cap by less than a step (and the buffer size, which does not reach the kernels' shapes beyond arows_ = min(R, 2048)). Decode and drafting are untouched.

Tried and not taken

Reading a 28,270-token prompt in one chunk: the loan lands (13.34 GiB lent, 853 MiB free) and then the prompt path faults in a buffer sized for the 16,384-token chunk (prefill mmq: quantize: operation not supported on global/shared address space). That is the gate on the larger gain (~13% at 28k from a single stream) and it is a resizing job, not a chunking rule.

Docs: docs/models/hy4-flash.md, Long prompts says the cap and the one-step margin.

…in one chunk

The chunk cap was a power of two, so a prompt a few tokens over 16,384 read as 16,384 + a tail. The tail is a second call, and a call streams every expert of every layer again: on a 16,409-token prompt the tail of 25 rows cost 278 ms of expert stream and left the ring, so those rows took the short-chunk path (2,695 rows from RAM over the bus, 2,821 on the CPU). A chunk now carries up to kMaxChunk + one attention step, which swallows such a tail; the split it replaces was on a step boundary, so the steps inside the one chunk are the ones the two chunks ran.

pp19000 (16,409 tokens), RTX 5090 at 475 W, fp16 KV, NVFP4 pack: 3,495 +- 62 -> 3,862 +- 40 t/s (two runs each), +10.5%. pp16384 3,771 -> 3,771 and pp32768 3,829 -> 3,840, unchanged (one chunk already, two chunks either way).

Reading a 28,270-token prompt in one chunk was tried and does not work yet: the lend lands (13.34 GiB lent, 853 MiB free) but the prompt path faults in a fixed buffer sized for the 16,384-token chunk ('prefill mmq: quantize: operation not supported on global/shared address space').

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant