Repository navigation
qwen38: the MTP head scores a list of frequent tokens - #2118
Conversation
|
Thanks for this. It no longer merges cleanly into dev: the conflicts are in c/idot.h c/qwen38.c c/qwen38_core.h. Could you rebase it onto the current dev? Once it merges, I will review it and put it in the next CI batch. |
205dff7 to
66cb78e
Compare
|
Rebased this one, too. |
66cb78e to
5fc0b8d
Compare
|
Rebased onto the current dev (0a0f66b) again. The full check passes. |
|
Thanks for this. I measured it before the batch (#2284) and had to leave it out: on the real model the head's drafts are almost never accepted with the list. Qwen3.8 Flash Next, the FP8 release with the int4-g64 sidecar, on the CPU (i5-13500,
Decode went from 6.62-7.19 tok/s on dev to 2.90-3.19 with the list (prompt 17.6-18.1 against 14.1-14.9), a warm-up and three runs each. The list itself looks right ( A test that would have caught it: the draft of Once it holds the acceptance of dev, I would like to measure it again; the idea of scoring the frequent tokens only is a good one for the CPU. |
|
Rebased onto the current dev (0a0f66b) again. The full check passes. |
|
Note Written by Claude Opus 5.5 for @simaotwx. Thanks, that was it. The head read the listed rows of the The new commit adds Qwen3.8 Flash Next (FP8 with the int4-g64 sidecar), CPU only (
The output with the list is identical to the output without it on all three. The experts were read |
On the CPU the MTP head scored all 248,320 tokens of the vocabulary for every draft. Now it scores a list: the 16000 most used tokens (Q38_MTP_VOCAB), the tokens almost as frequent as the 16000th, and the tokens of the conversation. Q38_MTP_VOCAB=0 scores the whole vocabulary as before. The token counts behind the list are saved to a file in the model directory (COLI_VOCAB_USAGE sets another file) and read by the next run. The head keeps scoring the whole vocabulary where it has another way to do it: on the dense chain the device scores it, and Q38_DRAFT_HEAD=int4, Q38_MTP_DUMP and a counts file that is the expert history's file all turn the list off. This change adds draft_vocab.h (the counts, the list and the counts file) and helpers in own_file.h to write a state file as a whole and to compare two paths. The draft paths of the head in qwen38_core.h return the draft instead of its logits and take it from q38_mtp_best_token, which scores the listed rows with the new matmul_q_idot_rows in idot.h. The lm_head keeps its int8 rows in host RAM when COLI_VK_DENSE_HOST releases the host copies, because the CPU scores the list from them, and the RAM plan counts them. The docs, two new test programs, a unit test case and a dense-only Vulkan case cover it. A draft is a proposal that the verify step checks, so it does not need the score of every token. A token missing from the list can only turn an accepted draft into a rejected one, and the rows of the rarely used tokens were read for every draft. Measured with an earlier version of this change on an AMD Radeon 8060S: 21.75 tok/s with the list off, 22.82 with 16000 tokens (+4.9 %), 22.07 with 1000. Greedy output was identical with and without the list. The output does not change, and the dense chain's device scoring is left as it is. Test: tests/test_draft_vocab and tests/test_own_file (both ok) Test: tests/test_qwen38_draft_head (34 checks passed) Test: tests/test_qwen38_idot (11 checks passed) Test: tests/test_idot_r16 (0 of 132 cases differ) Test: python -m unittest tests.test_resource_plan (122 tests, OK) Test: qwen38_mtp_harness.py --skip-head, 3 tiny fixtures (33 checks each) Test: the same with Q38_MTP_VOCAB=8 (33 checks each, as plain decoding) Test: vulkan_engines.sh dense-only-qwen38 on Lavapipe (14 cases OK) Test: qwen38 MTP cases of vulkan_engines.sh qwen-spec on Lavapipe (12 OK) Test: make -C c check, CPU build (exit 0; 2802 Python tests, 251 skipped)
The MTP head scored its list of frequent tokens from the int8 rows of the lm_head read as plain rows, one row after the other. On a build with AVX2 or AVX-512 the engine interleaves those rows by 16 for the CPU once they are loaded, so the list was scored from bytes of other rows. Now a listed token gets the score that the whole lm_head gives it, on every build. The draft is the best listed token again. This change adds i8r16_row, which reads one row out of the interleaved layout, makes i8r16_unpack use it, and adds matmul_q_idot_rows_r16, which unpacks each listed row and scores it with the row kernel. The head picks it when its rows are interleaved. The draft head test gets a model whose rows are interleaved the way the engine does it and checks the scores and the draft against the whole vocabulary. On the real model the drafts were almost never accepted with the list (0 to 1.6 % against 79 to 91 % without it), while the list itself held the picks. The scores read the wrong layout. The test of the head only used plain rows, so it did not see this. The list can only help the CPU when its scores are the ones the whole head gives, and the new test fails as soon as a layout of the rows is read the wrong way. Test: make tests/test_qwen38_draft_head with -march=native (all ok, including the two new checks); the same two checks fail without the fix Test: make tests/test_idot_r16 (0 of 132 cases differ, round trip exact) Test: make tests/test_draft_vocab (ok) Test: make tests/test_qwen38_draft_head without -march=native (all ok)
d59aef0 to
737c537
Compare
|
Thanks for the fix, it holds up. I measured it on the real model before the batch: Qwen3.8 FP8 with the int4-g64 experts, on the CPU (i5-13500), dev against dev with this PR, two rounds alternated, four turns each. Decode went from 6.77-7.44 tok/s (mean 7.07) to 7.30-7.65 (mean 7.48); the drafts accepted on the first two turns were 83.2% and 91.0% on dev, 78.6% and 87.9% with the list; the greedy answers were identical (md5) on two prompts. It is in the batch #2330 and goes into dev when that CI is green. |
Note
Written by Claude (Opus 5.5 and Sonnet 5.5) for @simaotwx.
The MTP head scored all 248,320 tokens for every draft. It now scores the 16000 most used tokens (
Q38_MTP_VOCAB), the tokens almost as frequent as the 16000th, and the tokens of the conversation.Q38_MTP_VOCAB=0scores the whole vocabulary as before.The counts are saved to a file in the model directory (
COLI_VOCAB_USAGEsets another file) and read by the next run. The run never overwrites a file that is not a counts file. The list is off whenCOLI_VOCAB_USAGEandCOLI_USAGEare the same file, withQ38_DRAFT_HEAD=int4, and while the dense chain is on, where the device scores the whole vocabulary. The int8 rows of thelm_headare not released from host RAM withCOLI_VK_DENSE_HOST=0, because the CPU scores the list from them.The verify step still checks every token, so the output does not change.
Measured with an earlier version of this change on an AMD Radeon 8060S (Strix Halo): 21.75 tok/s with the list off, 22.82 with 16000 tokens (+4.9 %), 22.07 with 1000. Greedy output was identical with and without the list.
Tested with unit tests for the list, the draft, counts file, file paths and RAM plan, the output-equals-plain-decoding checks of the tiny-model MTP test, and the dense-only and MTP cases of the Vulkan tests on a software device.