Repository navigation
qsa : track a sequence's prefix while other sequences share the unified cache - #116
1 commit merged into
Conversation
|
Tested on Strix Halo (gfx1151), master
I couldn't reproduce the speedup. In a Could you share the exact server flags and request sequence you measured with? With that I'll verify it. Comment generated via automated review of clanker. |
…ed cache The incremental QSA state (qsa_prefix: one sequence's cells in position order, the pooled indexer-key cache, the compact metadata) could only be rebuilt when the tracked sequence was the only occupant of the cache: qsa_recover() gave up at the first cell of another sequence. A multi-slot server with a unified KV pool therefore never regained the incremental path once a second conversation had cells in the pool. - qsa_rebuild_prefix() (src/prefix.h, unit-tested in test-qsa-recover) skips other sequences' cells; holes, shared cells and duplicate positions are still refused. - qsa_apply() recovers the prefix of whichever single-sequence ubatch comes next, including right on a switch between sequences, and tries a sequence that cannot form a prefix only once until the state is invalidated again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assisted-by: Claude Opus 5.5
|
Thanks for testing. Your sequence does not reach this path because of llama-server's default Two more points:
What we measured with (
Every request goes to
Results on master 4e0d917 (gfx1150, RADV), decode t/s:
In steps 1 and 3, all three builds give the same tokens with bit-identical top-5 log-probabilities. Your sequence, with A = 22.5k tokens, B = 4k tokens, and A ext = A + its output + 512 tokens, 64 tokens generated each time. Server:
With the default flags, B is cleared before A ext starts, so both builds take the incremental path. In our run, A ext came back from the RAM prompt cache (22563 tokens reused) rather than being reprocessed; either way, A is alone in the KV cache at that point. With On HIP, master vs master + #116 should show the same split in steps 3 and 4, or in your sequence with Assisted-by: Claude Opus 5.5 |
dzannotti
closed this pull request by merging all changes
into
halo-box:master
in
6aa81b8
…mages An mrope image puts all its cells at one position, so qsa_apply rejected its ubatch and qsa_recover refused its cells: after the first image in a conversation every later decode and verify step re-pooled all indexer blocks on the GPU and rebuilt the block metadata with the per-cell scan on the host, both in proportion to the context. The tracked prefix now keeps the cells in the mask's order (position, then y, then x), the order the selection ranks them in once an image is present, and the ranks of a 1-D ubatch after the image are the rank blocks the selection uses. Tails and block starts are ranks; a block's pooled key takes the rope position the selection gives it (its first cell's key for rank blocks, the block start otherwise). Image ubatches are tracked but do not match, so they keep the scan. A prefix with a position hole and no image still matches no selection layout and stays invalid, the prefill scoring bound stays position-only, and a ranked prefix only matches while its sequence is all the cache holds. Recovery (qsa_rebuild_prefix, which skips the cells of other sequences since halo-box#116) orders the cells of the sequence the same way: a counting sort by position, then y and x among an image's cells. It builds the prefix through apply, so a recovered prefix has the same ranks, keys and blocks as one kept ubatch by ubatch. A cell shared with another sequence is still refused. A ranked prefix is used only while its sequence is alone in the cache; with other sequences resident the selection does not rank, so those ubatches keep the scan. Text-only caches keep rank == position and are unchanged. Assisted-by: Claude
2 participants
Overview
The incremental QSA state is
qsa_prefix: the tracked sequence's cells in position order, the pooled indexer-keycache and the compact block metadata. It could be rebuilt only when the tracked sequence was the only occupant of
the cache:
qsa_recover()gave up at the first cell of another sequence. So a multi-slot server with a unified KVcache (
-np N -kvu) lost the incremental path as soon as a second conversation had cells in the cache, and nevergot it back. After that, every ubatch pooled the whole cache again.
qsa_rebuild_prefix()(src/prefix.h) skips the cells of other sequences. It still refuses holes, shared cellsand duplicate positions. The new unit test
tests/test-qsa-recover.cppcovers these cases.qsa_apply()follows whichever single-sequence ubatch comes next: when another sequence takes over, it rebuildsthe prefix from that sequence's cells. A sequence whose cells cannot form a prefix is tried only once, until the
state is invalidated again.
Measurements
On master the incremental QSA cache runs only when the QSA layers are on ROCm, so on our Vulkan machine this PR alone
changes nothing. With the cache off, #117's build (this PR + the Vulkan gate) is bit-identical to master on the four
repository corpora, each as is (570-876 tokens) and repeated to 12.7k-53.4k tokens. We compared the prompt tokens, the
complete next-token distributions after prefill, and 64 greedy steps with top-20 log-probabilities. The same held on
7a9196d, 12643af and 307c50d.
The code this PR changes is backend-independent cache bookkeeping. We exercised it on Vulkan with #117, which enables
the incremental cache there, and measured this PR's own share by building #117's gate with and without it.
llama-server -c 65536 -np 4 -kvu --cache-ram 0 --no-cache-idle-slots, no MTP.16k-token conversations.
All three generate the same 64 tokens, with bit-identical top-5 log-probabilities. Without this PR,
qsa_recover()gives up at the first cell of another conversation, so next to the residents the cache is not used at all.
--no-cache-idle-slotsmatters. By default, llama-server with-kvusaves every idle slot to the RAM prompt cache andclears it from the KV cache whenever a new task starts. A conversation then rarely sits next to another one's cells,
except while requests overlap.
Correctness:
test-qsa-recover: PASS (4e0d917). Corpora: bit-identical to master, see above. No ggml op is touched.Additional information
whole-cache path. This PR only makes sure the next single-sequence ubatch gets the incremental path back.
Requirements
QSA incremental cache. Our measurements come from Strix Point (gfx1150), not Strix Halo.
every measurement. Every commit carries
Assisted-by: Claude Opus 5.5.gfx1150, and ROCm's GTT limit cannot hold this model on our machine. The code was exercised on Vulkan only, with
qsa : enable the incremental QSA cache on Vulkan #117's switch.