Sync gfx11 with fork master (2026-09-20) - #110
Conversation
* metal : fix glu dispatch with ne00 = 1 * tests : disable ill-defined tests
* metal : support n_kv_max sparse mask hint in flash attention vec kernel
- add kernel_flash_attn_ext_vec_idx: compacts finite mask entries into
a per-row index list (Hillis-Steele scan, one threadgroup per row)
- extend vec FA kernel with optional sparse index gathering (FC slot 5)
- add host-side gate: sparse path when n_kv_max > 0, mask present,
supported head sizes / KV types, n_kv_max <= 4096
- new buffer region extra_idx for the index list
- pipeline getter extended with has_sparse param
- add test cases: head sizes, quant types, nb>1, nr23 variants,
sinks, ALiBi, softcap, permute, v_view_of_k, no-mask fallback
Note: multi-row (nb*nr23[1] > 1) cases still failing - rid mapping
in the store phase needs revisiting for the sparse path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fix sparse flash attention row addressing
- kernel_flash_attn_ext_vec_idx: mask param is half* but nb31 is a byte
stride, so the per-row mask offset was scaled by 2x; cast to char*
before applying the byte strides
- kernel_flash_attn_ext_vec: sparse pidx param is char* so the per-row
element offset was under-scaled by sizeof(int); scale it by sizeof(int)
to get the correct byte offset
- fixes the multi-row (nb*nr23[1] > 1) sparse flash attention failures
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* cont : use sparse vec FA for prefill
* metal : single-pass flash attention sparse index compaction
The idx kernel previously read the mask row twice: once to count the finite
entries (for the prefix scan) and again to recover their positions. Since the
kernel is memory-bound, this doubled the mask traffic.
Keep the finite positions in a per-thread register array during the count
pass and write them out directly, avoiding the second mask read. A dense
mask with more than NLOCAL finite entries in a slice falls back to re-reading
the mask to write the remaining positions.
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* tests : add perf cases for sparse flash attention prefill
Measure the sparse vec FA kernel across KV sizes, n_kv_max hints and batch
sizes. Run with:
./build/bin/test-backend-ops -b MTL0 -o FLASH_ATTN_EXT -p "n_kv_max=[1-9]" perf
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* qwen4 : enable sparse attention
* cont : adjust nsg
* cont : sync test-backend-ops
* cont : disable Qwen4 for now
* cont : clean-up + tests
* server : use pytest-xdist for server tests This commit adds pytest-xdist to the server tests. This is pytest plugin that distributes test execution across multiple CPU cores. Assisted-by: pi:llama.cpp/qwen3.8-27B Refs: ggml-org#26734 (comment) * remove server_base_port and BASE_PORT * use worksteal and pytest builting tmp_path
* snapdragon: update CI script to use new snapdragon/run.py * snapdragon: update build.py to not set +x on /lib
…ns (speculative decoding/MTP) (ggml-org#26477) * opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations * opencl: guard q4_K/q6_K tiled_ns convert-kernel registration for non-Adreno build * opencl: gate q4_K MUL_MAT+GLU fusion dispatch to Adreno * opencl: require the noshuffle weight layout in the q4_K GLU fusion gate * opencl: do not take the vectorized f16 mrow GEMV path on an unaligned row stride * opencl: pass the new get_scale_min_k4 stride argument at the row-major call sites * opencl: enable the q4_K split-K decode GEMV only where it is measured to win * opencl: record the X1-85 split-K datapoint (neutral, exclusion confirmed) * opencl: restrict the tiled lm_head/embed GEMV default to X2E/A8X * opencl: fix q4_K variant kernels to read the transposed scales layout * opencl: keep the flat-GEMV large-m escape opt-in * opencl: guard the o4 GEMV store against the rounded-up dispatch tail * opencl: restore the tiled q4_K/q6_K layout on tensor read-back * opencl: split-K for the q8_0 decode GEMV at small M * opencl: keep the q6_K noshuffle correctness escape ahead of the opt-in gate
Fuse RMS_NORM+MUL+ADD and ADD+ADD under GGML_SYCL_ENABLE_FUSION. ADD+ADD uses the same binbcast indexing and type matrix as standalone add() (f32, f16, f16/f32, i32, i16, bf16, including broadcast and non-contiguous). Unsupported combinations fall back to two add() launches.
* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (ggml-org#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Let llama_print_build_info write to a caller-provided FILE* instead of hardcoding stderr. The parameter defaults to stderr so existing callers keep their current behavior. The version command in llama-app now passes stdout, so plain version output goes to stdout where users expect it. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…8271) Use std::error_code overloads of fs::current_path() and fs::directory_iterator in ggml_backend_load_best() so an inaccessible search path (WebDAV mount, removed CWD) is skipped instead of terminating the process with an uncaught filesystem_error. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
This commit adds a cmake version configuration file to replace the current compile definition solution for the version. The motivation for this change is that I made a mistake and did not take into consideration that the compile definition means that this will become a compiler flag for all sources in the target. This means that when a version update happens that will recompile all sources in the target even if they have not changed. Refs: ggml-org#28278
* model, mtmd: fix gemma4 vision handling * nits
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
…rg#28127) * model: add Tencent Hy 4 (hy_v4) preview architecture support Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4): Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry Register hy_v4 GGUF constants, arch enum, and writer support Implement the hy-v4 model graph, hparams, vocab and context changes Register the new arch in llama-arch and models registry Extend arch tests to cover hy_v4 Assisted by Claude Opus 5 * Update convert_hf_to_gguf_update.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Update conversion/base.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py * model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview * n_layer_all --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* move more jobs to ccache-buckets * add venv deps * also jq
* opencl: add Adreno xmem SDPA path Assisted-by: Codex * Removed the Adreno-specific queue profiling override * Clean up formatting * 修复数值误差优化gqa/mask attn Assisted-by: Codex * add env GGML_OPENCL_XMEM_SDPA Assisted-by: OpenAI Codex --------- Co-authored-by: happyyzy <happyyzy@users.noreply.github.com>
* opencl: fix warnings * opencl: fix warnings for non adreno
ggml-org#28993) * gguf : align the data section relative to the GGUF start, not the file gguf_init_from_file_ptr reads a GGUF from the current file position, but padded the data section from file offset 0, so a GGUF embedded at an offset that is not a multiple of the alignment loaded without error and returned wrong tensor data. Also adds llama_adapter_lora_init_from_file_ptr, and disables mmap with a warning when an embedded data section is not aligned, instead of asserting in ggml. Assisted-by: Claude Opus 5 * llama : load lora from path through the FILE* variant The test now checks that mmap is disabled only for an unaligned offset. Assisted-by: Claude Fable 5.1 * Update ggml/src/gguf.cpp Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Update include/llama.h Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * llama : error on unaligned mmap of an embedded GGUF, drop test-load-file-ptr --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
…g#29008) * chat : add message delimiters to the DeepSeek V3.2/V4 parser Assisted-by: Claude Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ci : add API/ABI check to make-release workflow [no ci] This commit adds an API/ABI compatibility check to the make-release workflow. The motivation for this to allow us to detect any potential breaking changes in API/ABI compatibility between releases and fail the the release if there are any. The workflow can be triggered manually as before and this check can be skipped if needed as it does take some time which might be useful when doing a dry-run and not specifically interested in the API/ABI check. By default this will check the current release against the latest release, but this can also be configured in the workflow, or in the script run on the command line, to check a different tag. * add check for minor version bumps [no ci] This commit also changes the build type to be RelWithDebInfo so that the reported information is more useful.
…org#27985) * ui: fix accidentally removed reasoning menu in single model mode on desktop * ui: formatting task run to fix storybook test * ui: mount the add menu reasoning submenu outside router mode only The models selector already owns the reasoning submenu in router mode, so the add menu only mounts it in single model mode. The first enabled item of the add menu is now the reasoning submenu, the accessibility story expects it. --------- Co-authored-by: Ben Babik <work@benjaminbabik.com> Co-authored-by: Pascal <admin@serveurperso.com>
…org#29009) * Update to openvino-2026.4 * Update OV docs * ggml-openvino : fix clangd and MSVC warnings * fix int to ptr cast, more internal linkage enforcement, and avoiding duplicate switch case --------- Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
* first fix * removed unnecessary declarations
required for qwen35moe if MTP tensors are fused but not loaded
Co-Authored-By: Claude <noreply@anthropic.com>
Sync fork master with upstream (2026-09-17)
Assisted-by: Claude Opus
Annieren
left a comment
Annieren
left a comment
There was a problem hiding this comment.
I reviewed ggml_cuda mmvq path, looks good to me.
|
MMQ also looks good. Let me run some models to confirm. @Annieren there is one conflict in test-backend-ops.cpp on flush mode, do we want to keep both? |
My PR#107 adds if (flush_cache) ggml_backend_graph_compute(backend, gf_flush); at the top of the loop, and a break on flush_iters at the do { } while (total_time_us < 1000*1000); |
|
I have resolved the conflicts. |
Benchmark Results
There are several models that have reproducible regression, will check them tomorrow. LFM2 remains the clearest reproducible regression greater than 2% on comparable work. |
mmvdq dispatch moved to mmvq.cu, so ggml-cuda.cu no longer references it. Assisted-by: Claude Opus
Source comparison and regression causesInvestigation of latest LFM2 pp4096: confirmed MoE reduction fusion regressionIntroducing commit: The new matcher in An isolated diagnostic library disables only this matcher, preserving other optimizations and link inputs:
The sequence was master / diagnostic / diagnostic / master / latest; each invocation used 20 timed repetitions and the original pp4096 parameters. Disabling the fusion recovers 2.40% and 2.73% in the paired comparisons. Against the fresh latest control, master averages -2.16% and the diagnostic +0.35%. This isolates the measured gap to the new fusion for the tested workload. The initial kernel profiles support this result: the dominant Q4_K tile widths and call counts stayed the same; the new reduction consumes 132.00 ms over 680 calls, versus approximately 78.65 ms for the old multiply/reduction combination. Those aggregates include warmup and profiling overhead, so they are explanatory evidence rather than timed-region regression measurements. Exact low-level reasons for the kernel's poor performance still require hardware counters; the functional cause is established by the selective ablation. A production fix should gate or tune this fusion for affected HIP/prefill shapes, then validate correctness and other MoE workloads. Disabling all fusions would be unnecessarily broad. The commit message mentions Evidence: ablation results, diagnostic patch, introducing kernel change, and Qwen3.5 Q4_0: normalization changes performance and generated workCommit
Master-sync implements the corrected formula as RMS_NORM followed by SCALE, adding two graph operations per recurrent layer. HIP does not fuse this RMS_NORM + SCALE pair. The arithmetic change can change logits, generated text, and speculative acceptance. Restoring only the old normalization in an isolated master-sync libllama changes MTP prose from 83.20 to 87.07 tok/s (+4.65%), acceptance from 43.55% to 45.29%, and output length from 142 to 143. It does not restore latest's 132-token response or 48.75% acceptance. Thus normalization has an experimentally demonstrated effect on this application metric; it is not the only cross-build behavior difference, and the result is not a fixed-output speed measurement. The same intervention makes MTP code slower (125.30 to 115.76 tok/s), with acceptance dropping from 82.46% to 71.96%. The effect depends on prompt and generated content. Reverting the correctness fix would be the wrong general remedy. A performance improvement should preserve the corrected formula, for example by fusing its normalization and scale, and evaluate MTP on controlled output/acceptance as well as real prompts. For the borderline baseline code-prefill result, this intervention improves 529.78 to 533.19 tok/s (+0.64%). That does not establish the cause of the earlier -2.16% magnitude. The 32-token prompt is short, and the earlier four pairs did not all exceed 2%. Treat its exact attribution as unresolved. Evidence: normalization patch, metric comparison, and the invocation directories with preserved server logs and outputs. Each invocation uses five repetitions per prompt and the separate n_max=1 identity check. Gemma VLM: changed image budget and attention semanticsCommit
Commit The old repeat findings (E4B -4.18%, E2B OCR -6.50%, 26B-A4B -12.19%, 31B -3.60%) therefore compare different internal work. Increasing the image token count increases vision-encoder and language-model prefill work; throughput in tokens/s need not remain constant because those costs do not scale uniformly with the reported prompt-token count. This does not rule out additional kernel performance differences, especially for MoE variants. The repeated +105 prompt-token change in the synthetic cases is consistent with 252 versus 357 image tokens. With a 48-pixel merged-patch stride, the shared 1024x800 image aligns/downsizes to 864x672 under the old cap (18x14=252 tokens), versus 1008x816 with the new cap (21x17=357 tokens). A runtime budget intervention checks the causal link without changing the model code. Evidence: introducing commit, original per-model reports, and the image-budget check below. Gemma 31B stallThe single interrupted request has no GPU fault report or captured device stack identifying a cause, and the next two master-sync attempts passed. Code comparison alone cannot attribute that stall to a particular commit. It remains separate from the reproducible throughput findings. See analysis notes, reproduction instructions, and diagnostic library hashes. Fresh Qwen controls on halo41
The normalization intervention removes most of the throughput gap, while preserving master-sync's remaining code. Exact cross-build output equality is not restored. All six metric rows and associated identity checks pass in each invocation. The fresh baseline-code prefill comparison is latest 530.9617 versus master 529.7838 tok/s, only -0.2218%. The earlier borderline -2.16% on halo05 is not a consistent >2% result across these checks. Source attribution of that small prefill difference is unresolved. See three-way comparison. Qwen localization cross-checkA further diagnostic restores the entire latest qwen35.cpp while retaining master-sync's other code. It produces the same 143-token prose response and 45.29% acceptance as the normalization-only diagnostic, at 87.09 tok/s. The remaining Qwen model-file changes, including QKV construction/order, therefore do not explain the residual output difference in this case. That difference lies outside this model file and remains unattributed. All six rows and identity checks pass. See |
Do we want to merge this PR first and create separated tickets to invest these issues? |
20 participants
Merge fork master into gfx11, bringing in the upstream sync from PR #109 (ggml-org master through 972d231).
Conflict resolution kept the fork behavior and took upstream where it supersedes it:
Also restores the local
Iconstant in mmq-vec-dot.cuh that the RDNA3.5 J=128 static_assert depends on, which the upstream refactor had removed.Validation: CPU test targets build; HIP gfx1151 test-backend-ops builds; MUL_MAT on gfx1151 passes 1621/1621.