Repository navigation
batch: #2324-#2329, #2331, #2334, #2335 and eight external PRs - #2330
Merged
Merged
Conversation
Added details about legacy prompt handling and token ID requirements for the GLM engine.
On the CPU the MTP head scored all 248,320 tokens of the vocabulary for every draft. Now it scores a list: the 16000 most used tokens (Q38_MTP_VOCAB), the tokens almost as frequent as the 16000th, and the tokens of the conversation. Q38_MTP_VOCAB=0 scores the whole vocabulary as before. The token counts behind the list are saved to a file in the model directory (COLI_VOCAB_USAGE sets another file) and read by the next run. The head keeps scoring the whole vocabulary where it has another way to do it: on the dense chain the device scores it, and Q38_DRAFT_HEAD=int4, Q38_MTP_DUMP and a counts file that is the expert history's file all turn the list off. This change adds draft_vocab.h (the counts, the list and the counts file) and helpers in own_file.h to write a state file as a whole and to compare two paths. The draft paths of the head in qwen38_core.h return the draft instead of its logits and take it from q38_mtp_best_token, which scores the listed rows with the new matmul_q_idot_rows in idot.h. The lm_head keeps its int8 rows in host RAM when COLI_VK_DENSE_HOST releases the host copies, because the CPU scores the list from them, and the RAM plan counts them. The docs, two new test programs, a unit test case and a dense-only Vulkan case cover it. A draft is a proposal that the verify step checks, so it does not need the score of every token. A token missing from the list can only turn an accepted draft into a rejected one, and the rows of the rarely used tokens were read for every draft. Measured with an earlier version of this change on an AMD Radeon 8060S: 21.75 tok/s with the list off, 22.82 with 16000 tokens (+4.9 %), 22.07 with 1000. Greedy output was identical with and without the list. The output does not change, and the dense chain's device scoring is left as it is. Test: tests/test_draft_vocab and tests/test_own_file (both ok) Test: tests/test_qwen38_draft_head (34 checks passed) Test: tests/test_qwen38_idot (11 checks passed) Test: tests/test_idot_r16 (0 of 132 cases differ) Test: python -m unittest tests.test_resource_plan (122 tests, OK) Test: qwen38_mtp_harness.py --skip-head, 3 tiny fixtures (33 checks each) Test: the same with Q38_MTP_VOCAB=8 (33 checks each, as plain decoding) Test: vulkan_engines.sh dense-only-qwen38 on Lavapipe (14 cases OK) Test: qwen38 MTP cases of vulkan_engines.sh qwen-spec on Lavapipe (12 OK) Test: make -C c check, CPU build (exit 0; 2802 Python tests, 251 skipped)
The MTP head scored its list of frequent tokens from the int8 rows of the lm_head read as plain rows, one row after the other. On a build with AVX2 or AVX-512 the engine interleaves those rows by 16 for the CPU once they are loaded, so the list was scored from bytes of other rows. Now a listed token gets the score that the whole lm_head gives it, on every build. The draft is the best listed token again. This change adds i8r16_row, which reads one row out of the interleaved layout, makes i8r16_unpack use it, and adds matmul_q_idot_rows_r16, which unpacks each listed row and scores it with the row kernel. The head picks it when its rows are interleaved. The draft head test gets a model whose rows are interleaved the way the engine does it and checks the scores and the draft against the whole vocabulary. On the real model the drafts were almost never accepted with the list (0 to 1.6 % against 79 to 91 % without it), while the list itself held the picks. The scores read the wrong layout. The test of the head only used plain rows, so it did not see this. The list can only help the CPU when its scores are the ones the whole head gives, and the new test fails as soon as a layout of the rows is read the wrong way. Test: make tests/test_qwen38_draft_head with -march=native (all ok, including the two new checks); the same two checks fail without the fix Test: make tests/test_idot_r16 (0 of 132 cases differ, round trip exact) Test: make tests/test_draft_vocab (ok) Test: make tests/test_qwen38_draft_head without -march=native (all ok)
GLM-5.3 Flash's layer-0 v_proj has 148 BF16 weights of 1e20..1e33, so int4 g64 scales reach 1e34. chain_gemm.comp and qmatmul_coop.comp hold code * scale in f16, which overflows to inf: the output column becomes NaN and prefill logs "non-finite logits at layer 3". upload_tensor_pool now flags a matrix whose scale times the format's largest code exceeds 65504 (ColiVkTensor.wide) and keeps it on the fp32 GEMM/GEMV path. Threadripper PRO 5955WX, RTX 3090 24 GB and RTX 4090 24 GB, Windows 11, NVIDIA 617.14, GCC 16.2.0, glslc 16.6.0, make VK=1: 0 warnings. GLM-5.3 Flash on the 3090, T=0, 24 tokens, "List the first twelve prime numbers": before "!!!!!!!!" with "[router] non-finite logits"; after "2, 3, 5, 7, 11, 13, 17, 19,". test_vk_memtype, test_vk_tier (also host-pairs) and test_glm53_vk_f32 pass on both GPUs; test_vk_chain fails the same 18 cases with and without this change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…not by a stream sync inside a frame cc_attn grew the split decode attention's scratch (adec) with a cudaStreamSynchronize before the free. Inside a frame that sync waits on everything queued on the stream -- with the routed experts in the frame (COLI_CUDA_CHAIN_MOE) that includes the layer before's cc_wait_host kernel, whose host side (q36cc_dm_serve) comes only after this layer was recorded. The device spins on the host's word, the host on the device: a deadlock at the first regrowth past 128 positions where a layer has experts on the host (Qwen3.6-35B int4 on an RTX 3070 with 82 % of the decode's experts resident, the usual 49-token prompt: every run stopped at token ~145, position 193, with the GPU at 100 %). The resize now retires the old scratch into a short list that is freed after the next frame wait or finish (the stream has passed every frame that could read it by then), and at shutdown. A fifth retirement before any wait falls back to the sync, which no step reaches (each regrowth is 1.5x). Both cc_attn and the windowed variant. RTX 3070, Qwen3.6-35B-A3B int4, 200 and 600 new tokens: no hang, 14.5-14.7 ms a decode token (68-69 tok/s) against 16.0-17.2 with COLI_CUDA_CHAIN_MOE=0; the greedy text as before. test_qwen36_cuda_chain and cuda-chain-check unchanged.
…fest root is refused in 26 ms, not 16 s qpack_object_has_duplicate_keys compared every pair of keys with strcmp, n^2 / 2 comparisons. The files table it guards is capped at 8192 keys first (at the cap, ~0.8 s here), but the manifest root, the layout root and each section object reach the scan with no count check at all: the only bound on their keys is the 16 MiB JSON cap, about 1.4 million short ones. Measured on dev 36b713e with qpack-inspect: a manifest root of 100,000 keys took 18.0 s of CPU to refuse, 200,000 took 62 s; 1.4 million would be most of an hour, all of it before the container is refused. The keys are now copied to an array of pointers, sorted with qsort and compared neighbour to neighbour: n log n. The same inputs on this commit: 0.01-0.03 s each. If the pointer array cannot be allocated the pairwise scan still answers, so memory only ever changes the time, never the verdict. What is accepted and what is refused does not change. tests/test_qpack.c: a files table whose repeated key has other keys between the copies is refused (the sorted scan must find non-neighbours; passes before and after), and a 100,000-key manifest root with its first key repeated as its last is refused within 4 s of CPU time -- 15.9 s and a failed check against the previous qpack.c, 0.026 s here (0.104 s under ASan+UBSan). clock(), not wall time, so a loaded host does not fail it.
…test_v4_hybrid_policy run once per `make check`, not twice TEST_BINS has been derived from the tests' build rules since 1d8d19b ("build: derive TEST_BINS from the build rules, not from the .c files"): every tests/test_*$(EXE) rule that is not in TEST_EXCLUDE is a gate. Two later additions appended their tests to TEST_BINS by hand as well -- bc44ff0 the three V4 infrastructure tests under COLI_V4_SUPPORTED, 049d444 test_exact_dot under LINUX -- and all four have unconditional rules, so on the hosts those blocks apply to each was built once and run twice per `make test`. The two lines go; nothing else changes: make -s --eval='print-bins: ; @echo $(TEST_BINS)' print-bins | tr ' ' '\n' | sort | uniq -d lists the four on dev 36b713e (226 entries, 222 distinct) and nothing here (222 entries, 222 distinct, the same 222). The four already ran on every platform through their rules, so no host gains or loses a gate.
… preamble The pi example only names glm-5.2-colibri. Note that the same provider entry serves the other engines by swapping the model id, and that pi's preamble (system prompt, tool catalog, project instructions, ~5,000 tokens) has to fit the served context, with a measured pi 1.4.0 / Qwen3.6-35B-A3B data point.
…s every forward fell back to the host)
… (between two chunks) and with the device tiers; the gateway logs a client gone mid-turn
…ween chunks of 8 rows)
… crashed compiling the shader on a Radeon 780M, #2289); a Vulkan build says when COLI_VULKAN leaves the device unused
…ar ties that move from CPU to CPU)
…; float16 serve gates for qwen36
…commit too GlobalMemoryStatusEx's ullAvailPhys is the physical memory free now; whether the next malloc succeeds depends on ullAvailPageFile, the commit still grantable (RAM + page file, less what every process has committed). #1375 bounded the planner and compat_meminfo_gb by it; compat_meminfo (kimi_k3's expert-cache budget, the telemetry) and deepseek_v4's coli_v4_os_available_memory (its RAM plan) still read ullAvailPhys alone. On a laptop with WSL running: 6.9 GB physical free, 2.8 GB of commit, so a budget from the first was refused by Windows. Both now take the smaller of the two. tests/test_windows_commit_avail_source.py: every C reader of ullAvailPhys reads ullAvailPageFile beside it (it names compat.h and deepseek_v4.c without this change).
…-initializer warnings in the other engines)
…the loop's bits chain_moe_gv.comp's lanes walked a row's 16-byte steps in a loop that waited for each load in turn: on a Radeon 8060S (RADV) a Qwen3.6 layer's routed experts took 160 us at one row, 78 GB/s of the pairs' weights. For int4 per group (fmt 4) with groups of whole steps, where the row is a whole number of steps a lane at coli_vk_gv_lanes' lanes (up to 16; Qwen3.6's 2048 and 512 inputs: eight), moe_gv_pipe builds the shader with HQ (and the steps a lane in NQ): a lane loads every step and scale first, gate's and up's together, then sums them. In that path and in the fmt 4 loop each step's product is rounded before it is added (precise), so both give the same bits: tests/test_vk_chain runs Qwen3.6's shapes, D 2048 and the clamped SwiGLU both ways and compares them bit for bit, and the tier's small experts (the loop) keep the chain's. The fmt 4 loop also takes gate and up as two loops instead of one interleaved (160 -> 119 us alone). Formats 1, 2 and 7 are untouched. COLI_VK_MOE_GV_HQ=0 keeps the loop. Radeon 8060S, COLI_VK_CHAIN_BENCH=moe at Qwen3.6's shapes (K 8 of 256), us per layer: - one row 160 -> 100, two 282 -> 166, three 385 -> 239 Qwen3.6-35B-A3B int4-g64, Vulkan chain, tier 24 GB, 512 new tokens on an English, a German and a code prompt, two rounds alternated: 45.0 -> 50.3 tok/s, the same text (md5). This brings #1952's one-wave kernel (each lane's loads ahead of its sums) into vkc_moe as asked there, on chain_moe_gv.comp's lanes so a row keeps its bits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loat16 or a float32 run); the kv16 rules list no headers
…tier's start calls it; gcc 14 and later refuse the implicit declaration)
JustVugg
force-pushed
the
batch/kv-oom-2026-10-23
branch
from
October 23, 2026 18:18
78f0d30 to
3fe0952
Compare
JustVugg
changed the title
…o both runs verify the same steps and dump the same logit rows
…un with Q36_KV=f32
…_VK_ATTN mirror, GLM chain, split, .coli_kv v4, mux adoption; oracle runs keep float32 unless KV16 is set)
…I_VK_ATTN, split, serve with one and two slots)
JustVugg
force-pushed
the
batch/kv-oom-2026-10-23
branch
from
October 23, 2026 19:41
3fe0952 to
690cdc7
Compare
JustVugg
changed the title
This was referenced Oct 23, 2026
…nversations' places closes the places (states home) instead of turning the chain off
…a pin another conversation saved rewound one whose window uploads had left holes); Muse Spark three-slot gate
gcc fuses y += a*x only from -O2 on. In the ASan build (-O1) the scalar tail and the test's float loop rounded twice while the F16C body used vfmadd, so test_kv16 found 67 rows that differ. MATMUL_F32_MADD fuses where the target has FMA and does not where it has none, at every optimisation level; at -O3 the bits are the ones gcc already gave.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One CI for eighteen PRs (one CI at a time): this branch is dev with each of them merged as it is, in this order. When it is green this PR is merged with its merge commit, which brings every one of them into dev.
Ours:
External:
make check, not twice #2312 Makefile: four tests run once permake check, not twiceChecked before this CI, on the merge: every engine builds with
VK=1with no warnings;make checkpasses (gcc 15); on Lavapipe thedecide,muxand qwen chain families ofvulkan_engines.shpass (the qwen chain family up to its GGUF gates, which need gguf-py on that box); on an RTX PRO 4000 the CUDA chain's own check, the float16 serve gates and the external PRs' measurements:test_vk_chainpasses on a Radeon 780M.