Repository navigation
Conversation
…in one chunk
The chunk cap was a power of two, so a prompt a few tokens over 16,384 read as 16,384 + a tail. The tail is a second call, and a call streams every expert of every layer again: on a 16,409-token prompt the tail of 25 rows cost 278 ms of expert stream and left the ring, so those rows took the short-chunk path (2,695 rows from RAM over the bus, 2,821 on the CPU). A chunk now carries up to kMaxChunk + one attention step, which swallows such a tail; the split it replaces was on a step boundary, so the steps inside the one chunk are the ones the two chunks ran.
pp19000 (16,409 tokens), RTX 5090 at 475 W, fp16 KV, NVFP4 pack: 3,495 +- 62 -> 3,862 +- 40 t/s (two runs each), +10.5%. pp16384 3,771 -> 3,771 and pp32768 3,829 -> 3,840, unchanged (one chunk already, two chunks either way).
Reading a 28,270-token prompt in one chunk was tried and does not work yet: the lend lands (13.34 GiB lent, 853 MiB free) but the prompt path faults in a fixed buffer sized for the 16,384-token chunk ('prefill mmq: quantize: operation not supported on global/shared address space').
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The Hy4 prompt path picked its chunk as a power of two, so a prompt a few tokens over 16,384 read as 16,384 + a tail. The tail is a second
run_rowscall, and a call streams every expert of every layer again: on a 16,409-token prompt the 25-row tail cost 278 ms of expert stream and left the expert ring, so those rows fell belowkRingRowsand took the short-chunk path (2,695 rows staged from RAM over the bus, 2,821 on the CPU).open_promptnow carriesmin(total, kMaxChunk + kAttRows)rows instead of rounding up to a power of two, and the memory fallback steps tokMaxChunkbefore halving. The margin is exactly one attention step, so the merged tail is the step the two chunks already ran: the step sequence inside the chunk is unchanged, and every prompt at or below 16,384 runs the samerun_rowscalls as before (only its buffer layout is now sized to the prompt instead of the next power of two).Numbers
RTX 5090 at 475 W,
models/Hy4-flash-NVFP4.basalt, fp16 KV,basalt-bench --model ... -p N --seed 1, two runs each:Engine log, 16,409 tokens: the expert stream drops 1,251 → 1,124 ms and the tail's bus/CPU staging disappears.
Quality
No row arithmetic changes: the same
run_rowscalls with the same row counts, only the chunk count for prompts that overran the cap by less than a step (and the buffer size, which does not reach the kernels' shapes beyondarows_ = min(R, 2048)). Decode and drafting are untouched.Tried and not taken
Reading a 28,270-token prompt in one chunk: the loan lands (13.34 GiB lent, 853 MiB free) and then the prompt path faults in a buffer sized for the 16,384-token chunk (
prefill mmq: quantize: operation not supported on global/shared address space). That is the gate on the larger gain (~13% at 28k from a single stream) and it is a resizing job, not a chunking rule.Docs:
docs/models/hy4-flash.md, Long prompts says the cap and the one-step margin.