Skip to content

Rebase onto upstream llama.cpp c114bd4c5 (2026-10-14) - #178

Closed
gaetan-puleo wants to merge 401 commits into
masterfrom
sync/upstream-2026-10-14-c114bd4c5
Closed

Rebase onto upstream llama.cpp c114bd4c5 (2026-10-14)#178
gaetan-puleo wants to merge 401 commits into
masterfrom
sync/upstream-2026-10-14-c114bd4c5

Conversation

@gaetan-puleo

@gaetan-puleo gaetan-puleo commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Overview

First sync of the fork by rebase instead of merge: the fork's commits are replayed onto upstream ggml-org/llama.cpp c114bd4c5 (2026-10-14), 381 upstream commits past the fork's content base 86ea01d05 (2026-09-29). It also moves the sync docs and skills to the rebase workflow.

An agent (Claude) wrote this sync and this description; what was and was not verified is listed below.

How the rebase was done

  1. Every non-merge fork commit (git rev-list --no-merges deb5e2f50 --not upstream/master) was replayed onto c114bd4c5, conflicts resolved fork first, adapted to upstream's current API.
  2. A rebase drops merge commits, and part of the fork's work only lived in them (sync resolutions, --lazy-mode on-direct from Merge halo-box/llama.cpp master into strix (2026-09-15) #64, the per-area Vulkan choices of the 2026-09-29 sync). The reconciliation commit brings the tree to a 3-way merge of deb5e2f50 and c114bd4c5 from 86ea01d05, conflicts resolved fork first.
  3. forklines.py (skill) audits the fork lines: the ones still missing are explained below.
  4. Build fixes for HIP and Vulkan in their own commits.

Intermediate commits do not all build on their own (their merge-time adaptations come back in the reconciliation commit). The final tree builds and is the one validated. git bisect inside this series is unreliable.

Fork-first decisions to review

Kept from the fork, upstream changes to these not taken:

Area Kept Upstream not taken
src/models/qwen4exp.cpp (+ its struct in models.h, MTP tensor names, converter) fork MTP, QSA, PLE on disk, gfx1151 prefill ggml-org#29761 (upstream qwen4exp MTP), ggml-org#29825 (halved indexer score memory), ggml-org#29824, ggml-org#29622, ggml-org#28569, ggml-org#29599, ggml-org#29751, ggml-org#29819, ggml-org#29958, ggml-org#29994, ggml-org#30017, ggml-org#30042, ggml-org#30097, ggml-org#30160, ggml-org#30257, ggml-org#30397, ggml-org#30445, ggml-org#30480
ggml-cuda/gated_delta_net.cu fork RDNA3.5 warp counts, chunked kernel, fused decode ggml-org#30087 (four state columns per warp), ggml-org#29353 (GDN chunked kernel)
ggml-cuda/top-k.cu fork HIP RDNA3.5 wg/radix paths (stable ties for QSA) ggml-org#28713, ggml-org#29792 (top-k restructuring); the CCCL >= 3.4.3 race fix is taken
ggml-cuda/mmq-config-rdna3-5.cuh fork tile table ggml-org#28195 (AMD's own gfx1151 retune)
Vulkan flash attention shaders strix kernels (as decided in the 2026-09-29 sync) ggml-org#29639, ggml-org#30003 (sparse FA for quantized K/V, coopmat2)

Upstream taken over the fork:

Ported onto upstream's API (fork behaviour unchanged, please review):

Verification

  • CPU, HIP (ROCm 10, gfx1151) and Vulkan build.
  • ctest -L main on CPU: 64/65 pass; test-tokenizers-ggml-vocabs fails without git lfs (invalid magic characters: 'vers'), not a sync failure. test-llama-archs generates every arch, including GLM5-Next (k-pool) and qwen4exp.
  • forklines.py 86ea01d05 deb5e2f50 ... c114bd4c5: remaining missing lines are the replacements listed above.
  • GPU validation (validate.sh, HIP + Vulkan, base deb5e2f50): running, results will be added here. Models: Qwen3.8-Flash-Next (qwen4exp), GLM-5.3-Flash (glm5-next), Qwen3.8-27B (qwen35), Ornith-1.5-35B-A3B (qwen35moe), Ling-3.0-tiny, plus Qwen3.5-0.8B in every quant and Granite 1B-A400M.
  • Perplexity / KLD and llama-bench against deb5e2f50: part of the run above.

How to land

Not with the merge button. Once approved and validated, a maintainer runs:

git fetch origin
git push --force-with-lease=master:deb5e2f50 origin origin/sync/upstream-2026-10-14-c114bd4c5:master

then closes this PR. Every open PR and every branch made from the old master then needs git rebase --onto origin/master deb5e2f50 <branch> (see SYNC.md). Open PRs affected: #165, #166, #169, #171, #172, #175, #176. #169 is another upstream sync (re-open of #155) that this one replaces.

Docs

SYNC.md, AGENTS.md (rule 4) and the upstream-sync / upstream-sync-verify skills now describe the rebase workflow (phase 2 rewritten). sync.sh merge/verify/summary belong to the old merge workflow and are not used by a rebase sync; sync.sh report, forklines.py and validate.sh still are.

https://claude.ai/code/session_01Qr1p3WtS7Gr8dSEFcgaTLc

gaetan-puleo and others added 30 commits October 14, 2026 18:43
@gaetan-puleo
hip: select RDNA3.5 mul_mat_id tile width by rows-per-expert range
b5f6357 
The previous rule only matched exactly 16/32/64 rows per expert (ubatch
512/1024/2048 with 8 active experts); any other ubatch size, and the
remainder ubatch of every prompt, still fell back to the generic J=128 tiled
grid, which launches ncols_max/J*n_experts mostly-empty blocks.

Use ranges instead, and allow the routed-compact kernel for every selected J:
  Q8_0:            rows <= 12 -> J16, <= 64 -> J48, else J128
  Q4_K/Q5_K/Q6_K:  rows <= 12 -> J16, <= 24 -> J32, <= 48 -> J48,
                   <= 64 -> J32, else J128
The 16/32/64 choices are unchanged from the previous commit. Ranges were
picked from a J sweep over 64..4096 tokens on gfx1151 and validated
end-to-end.

Qwen3.6-35B-A3B on gfx1151, -b 2048 -ub 2048 unless noted, vs cutoff:
                   PP128   PP256   PP384   PP768   PP1536  PP4096@ub4096
  UD-Q8_K_XL      +39.7%  +61.1%  +21.5%  +17.2%  +12.4%   +3.9%
  UD-Q4_K_XL      +93.5%  +75.9%  +60.1%  +39.1%  +18.4%   +2.4%
(PP512/1024/2048 as in the previous commit.)
test-backend-ops MUL_MAT_ID: 914/914 passed (adds 4- and 128-row cases).
@gaetan-puleo
CUDA/HIP: use WMMA flash attention for D=256 prefill on RDNA3.5
bdeb626 
On gfx1151 the D=256 flash-attention path always fell back to the tile
kernel, which is FMA-bound (~8 TFLOPS) and dominates prefill time at deep
context (68% of PP time at 32k, 81% at 64k for Qwen3.5-0.8B).

- Enable AMD WMMA device code for DKQ=256 (previously DKQ>128 was excluded).
- Add a RDNA ncols=64 D=256 config tuned for the 256-VGPR budget:
  nwarps=8, nstages=1, nbatch_fa=32, nbatch_K2=128, nbatch_V2=64,
  nbatch_combine=64. Naive settings spill heavily and lose to the tile kernel.
- Dispatch to MMA only when Q->ne[1]*gqa_ratio_eff > 32; the tile kernel is
  still faster for small batches (nb 4/8).
- Fix a latent tile_Q reuse race in the combine loop: when
  DV/2 > nbatch_combine, warps read tile_Q columns written by other warps
  and the buffer is rewritten by the next batch without a barrier.

Qwen3.5-0.8B UD-Q8_K_XL, gfx1151, PP512:
  depth 0:     9361 -> 9473 t/s  (+1.2%)
  depth 32768: 3211 -> 4105 t/s  (+27.9%)
  depth 65536: 1927 -> 2632 t/s  (+36.6%)
TG unchanged. test-backend-ops FLASH_ATTN_EXT: 2939/2939 passed.
@gaetan-puleo
llama: synchronize before mutating graph state
e952926
@gaetan-puleo
tests: avoid ambiguous top-k MoE scores
b6765cd
@gaetan-puleo
hip: optimize RDNA3.5 MoE inference paths
5f7cbd9
@gaetan-puleo
hip: address PR review correctness issues
68ad200 
Constrain architecture-specific paths, validate fused GDN use counts, narrow graph synchronization, remove disabled experiments, and strengthen fusion tests.

Assisted-by: OpenCode
@gaetan-puleo
hip: tune Ling MoE inference on RDNA3.5
67e93a6
@gaetan-puleo
hip: fix RDNA3.5 fused quantization edge cases
99cf804
@dzannotti @claude @gaetan-puleo
qwen4exp: --ngram-on-disk, forge ngram on ssd (#3)
42f1e32 
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Marshall @gaetan-puleo
qwen4exp: MTP draft head for speculative decoding (#9)
c752bb3 
Co-authored-by: Marshall <assistant@llama.cpp>
@dzannotti @gaetan-puleo
qwen4exp: fix --ngram-on-disk, which crashes on every model (#8)
9943b67
Marshall @claude @gaetan-puleo
server : add a hidden preset option to omit a model from GET /models (
935a676 
#11)

Co-authored-by: Marshall <assistant@llama.cpp>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Marshall @gaetan-puleo
speculative-prefill : port upstream PR 27692
a356cdf 
A small draft model prefills the prompt and decodes a few lookahead tokens. The
softmax attention of those tokens over the prompt is captured with a ggml eval
callback, smoothed, max-reduced over heads and layers and pooled into chunks.
Only the top scoring chunks go into the target model's KV cache. This is lossy.

Adds llama_set_eval_callback, common/speculative-prefill, the --spec-prefill-*
options, the standalone example and the llama-server hookup. Not ported from
that PR: the llama-bench changes and the python eval scripts.

Measured on Strix Halo / Vulkan with a Qwen3.5-2B estimator, TTFT at p = 0.15:
Qwen3.8-27B 5604 -> 1587 ms, Flash-Next FP4 4920 -> 1645 ms, and 54.9 -> 11.9 s
on a 12k prompt. Decode rate is unchanged. See docs/speculative-prefill.md.

Assisted-by: Claude Opus 5
Marshall @claude @gaetan-puleo
vulkan: driver-gate the coopmat LDS stride pad (#14)
25f0abb 
Co-authored-by: Marshall <assistant@llama.cpp>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Marshall @gaetan-puleo
Halo/vulkan topk radix (#17)
b82b3a3 
Co-authored-by: Marshall <assistant@llama.cpp>
@Nathanw1014 @gaetan-puleo
vulkan: replace the EngramHalo radix top-k with upstream's radix select
a8475b5 
This reverts the Vulkan part of "Halo/vulkan topk radix (#17)" (ca062e9),
which ported the EngramHalo HIP radix kernel as topk_radix.comp, so that the
next commit can carry upstream's top_k radix select (ggml-org#28032)
unchanged. One implementation of the op, the one upstream settled on, keeps
this tree closest to upstream and lets the deterministic slot assignment that
follows apply as-is.

The two shaders solve the same problem; the difference that matters for
Qwen 3.8 Flash-Next is order: topk_radix.comp assigns output slots with
atomicAdd, so the selected indices come out in a scheduling-dependent order,
and the sparse attention sums them in that order (see the deterministic-scan
commit for the measured effect). Upstream's variant also carries the QSA
indexer fusion.

Co-authored-by: Claude (Opus 5)
@0cc4m @gaetan-puleo
vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next (ggm…
3311813 
…l-org#28032)

* vulkan: add top-k radix sort shader for k >= 1024

* add Qwen 3.8 Flash Next top-k tests

* add top-k qsa fusion

* clean up code

(cherry picked from commit daef7b6)
@Nathanw1014 @gaetan-puleo
vulkan: deterministic slot assignment in top_k radix select
88c1cdd 
The radix select (ggml-org#28032) assigned output slots with atomicAdd, so identical
runs emitted the selected indices in a scheduling-dependent order and, at the
tie boundary, could drop a different tied element. On Qwen 3.8 Flash-Next the
sparse attention sums the selection in that order, the ULP noise cascades
through the QSA layers, and identical greedy requests above 2051 prompt tokens
diverge (32k prompt x4: 2 to 3 distinct on every unfixed build, upstream
included).

Replace both atomicAdd slot counters with a per-chunk subgroup-ballot exclusive
scan, walking the columns in ascending index order: the output order is fixed
(ascending, which also helps gather locality) and the tie fill keeps the
lowest-indexed tied elements. The histogram atomicAdd is unchanged.

Verified on gfx1151: test-backend-ops TOP_K 453/453; 32k x4 byte-identical
(was 2 to 3 distinct); 1024 x16 byte-identical; layer 3 and layer 47 top-k
dumps identical in set and order across runs; prefill 337.5 vs 339.6 t/s and
decode 26.3 vs 26.2 t/s at 16k depth (within noise).

Co-authored-by: Claude (Opus 5)
(cherry picked from commit fe6620c88c599b2f3a6e5ff4a6b99bb4773be38e)
@danielhanchen @gaetan-puleo
models: use flash-linear-attention's l2norm for gated delta net q/k
3d6971c 
The GDN q/k normalization is defined by flash-linear-attention as

    l2norm(x) = x * rsqrt(sum(x*x) + eps)

with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.

The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.

transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.

eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.

ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.

No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).

(cherry picked from commit 0dd72fd)
@Nathanw1014 @gaetan-puleo
kv-cache: zero freed cells so masked-out rows never carry stale K/V
592a5ca 
On gfx11 (RDNA3) WMMA, x + (-0.0) is not exact in f16, so a fully masked
flash-attention column with P == +0.0 still leaks the sign of whatever V the
cell last held into the accumulator: the output of a request depends on what
the previous request left in the cache. On a 1024-token prompt with 16
identical greedy requests, Qwen2.5-7B (f16 KV) alternates between two
continuations and Qwen3.8 Flash-Next gives 5-6 distinct ones.

Rather than patching the shader (which costs 8-18% dense prefill at depth on
gfx1151 by its mere presence in flash_attn_cm1.comp), keep the invariant that
a free cell is always zero: the KV buffers are cleared at construction, and
every cell that becomes free is memset again, in every layer, when it is
freed (seq_rm, seq_keep; clear() wipes up to a per-stream high-water mark, and
stream copies mirror the mark). Caches that mirror another cache's cells (the
Flash-Next indexer) register with the owner and are zeroed with it. On UMA the
Vulkan backend's tensor memset is a host memset; it happens once per free,
never per token, and never as a graph node, so graph reuse cannot replay it.

Not covered: masked-out cells that belong to another sequence in a unified
multi-sequence cache; only a shader-side fix covers that case.

Gate (16 identical greedy requests, 1024-token prompt, 129 tokens, per-position
top-8 logprob streams compared): 16/16 identical; the fresh-server output is
unchanged byte for byte.

Co-authored-by: Claude (Opus 5)
@gaetan-puleo
hip: fix ROCm 7.2 Flash Attention Werror build
0399d7a
@gaetan-puleo
hip: avoid compiling RDNA3.5 Q8 FA on CDNA
cc4cbc2
@gaetan-puleo
readme : add Halo Box fork section (#19)
7620f12
@gaetan-puleo
Update README to clarify features and fix formatting
6c22fb7 
Removed duplicate entry for 'hidden' server preset option and made minor adjustments to the list formatting.
@0cc4m @fairydreaming
@sszymczy @SimonTeixidor @wjinxu @CISC @ruixiang63 @arthw @ynankani @nikwen @jadenmach2 @ServeurpersoCom @wanghqc @itsnotoger @ggerganov @bartowski1182 @ozgursoy @JonathanC-ARM @luhenry @JamesFranc @ngxson @danielhanchen @forforever73 @jhen0409 @lstolcman @philip-jingxin @Titaniumtown @anujj @trivikram-reddy1 @alanhc @madsmtm Marshall @max-krasnyansky @ac-mmi Acmmi @am17an @angt @superm1 @init-22 @cqderek @geckguy @YanissAmz @Eurekatic @RaulAbejonDelgado @tdakhran @Nathanw1014 @tannerbruhn @danbev @tboinovski1 @kbenkhaled @SergeySklyarov @johnkarlhill @newjordan @taronaeo @ravi9 @cabelo @eapache @Little0o0 @intel00000 @happyyzy @nachobh @nicois @gaetan-puleo
Halo/upstream sync 2026 09 05 (#20)
4422659 
Signed-off-by: ynankani <ynankani@nvidia.com>
Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Ruben Ortlam <rortlam@redhat.com>
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Simon Teixidor <simon@flaskpost.me>
Co-authored-by: 王金旭 <105263726+wjinxu@users.noreply.github.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Ruixiang Wang <wangruixiang07@outlook.com>
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com>
Co-authored-by: ynankani <ynankani@nvidia.com>
Co-authored-by: Niklas Wenzel <dev@nikwen.de>
Co-authored-by: Jaden_Mach <88880593+jadenmach2@users.noreply.github.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: itsnotoger <19309683+itsnotoger@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Bartowski <3266127+bartowski1182@users.noreply.github.com>
Co-authored-by: Buğra Özgürsoy <13810383+ozgursoy@users.noreply.github.com>
Co-authored-by: Jonathan Clohessy <jonathan.clohessy@arm.com>
Co-authored-by: Ludovic Henry <git@ludovic.dev>
Co-authored-by: James Francis <6763899+JamesFranc@users.noreply.github.com>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com>
Co-authored-by: Jhen-Jie Hong <iainst0409@gmail.com>
Co-authored-by: Lukasz Stolcman <4583553+lstolcman@users.noreply.github.com>
Co-authored-by: Jingxin (Philip) Li <philipaslee@gmail.com>
Co-authored-by: Titaniumtown <titaniumtown@proton.me>
Co-authored-by: anujj <ajalota@nvidia.com>
Co-authored-by: Trivikram Reddy <127072883+trivikram-reddy1@users.noreply.github.com>
Co-authored-by: Alan Tseng <alanhc.tseng1999@gmail.com>
Co-authored-by: Mads Marquart <mads@marquart.dk>
Co-authored-by: Marshall <assistant@llama.cpp>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Co-authored-by: Aman Chadha(IVIXMMI) <79802170+ac-mmi@users.noreply.github.com>
Co-authored-by: Acmmi <acmmi@Acmmis-MacBook-Air.local>
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Mario Limonciello <mario.limonciello@amd.com>
Co-authored-by: Isaac <34376531+init-22@users.noreply.github.com>
Co-authored-by: cqderek <cqiang@qti.qualcomm.com>
Co-authored-by: Abhiram <78226909+geckguy@users.noreply.github.com>
Co-authored-by: Yaniss Amazouz <yaniss91600@gmail.com>
Co-authored-by: Eurekatic <eurekatic@eurekatic.eu>
Co-authored-by: RaulAbejonDelgado <raul.abejon.delgado@gmail.com>
Co-authored-by: Tarek Dakhran <tarek@liquid.ai>
Co-authored-by: Nathan Wilson <67372905+Nathanw1014@users.noreply.github.com>
Co-authored-by: Tanner Bruhn <66120666+tannerbruhn@users.noreply.github.com>
Co-authored-by: tannerbruhn <tannerbruhn@users.noreply.github.com>
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
Co-authored-by: Todor Boinovski <todorb@qti.qualcomm.com>
Co-authored-by: kbenkhaled <khalilbenkhaled01@gmail.com>
Co-authored-by: Sergey Sklyarov <sergey.sklyarov@gmail.com>
Co-authored-by: Ozymandias_EBON <112784549+johnkarlhill@users.noreply.github.com>
Co-authored-by: Frosty40 <newjordan@gmail.com>
Co-authored-by: Aaron Teo <aaron.teo1@ibm.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@opensuse.org>
Co-authored-by: Evan Huus <eapache@gmail.com>
Co-authored-by: HongHuang <honghuang2000@outlook.com>
Co-authored-by: Tom Tan <29201606+intel00000@users.noreply.github.com>
Co-authored-by: yzyyzyhhh <96101183+happyyzy@users.noreply.github.com>
Co-authored-by: happyyzy <happyyzy@users.noreply.github.com>
Co-authored-by: nachobh <ibenitoherrero@gmail.com>
Co-authored-by: Nick Farrell <nick.farrell@aiven.io>
@gaetan-puleo
readme : merge the halo-box/llama.cpp README into this fork's
e271ea3 
Keep this fork's README as the base and take from the parent what it lacked: the
Halo Box image, the two-fork table that says which repo to use, and a feature
table. The table is now split into what is inherited from halo-box/llama.cpp and
what is Strix Halo only, and lists the ROCm/HIP work carried here with its
escape-hatch environment variables.

Assisted-by: Claude Fable 5.1
Claude-Session: https://claude.ai/code/session_01LTnaB4ucGc5vkzctYns4Nq
@gaetan-puleo
readme : align the Halo Box section with strix-llama.cpp (#21)
de9ed96
@gaetan-puleo
readme : adopt the strix-llama.cpp README for this fork (#22)
5a83ffa
@gaetan-puleo @am17an @pwilkin
Halo/scheduler uma ring 27311 (#23)
a50bc8e 
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Piotr Wilkin <ilintar@gmail.com>
Marshall @claude @gaetan-puleo
ROCmFPx: port the AMD-native FPx tensor types (CPU + Vulkan)
908b07f 
Ports the experimental ROCmFPx quantization formats from
ciru-ai/ROCmFPX @ 54f5fe0 onto current master. That fork has no shared
git history and is based on an older llama.cpp, so this is a hand-port of
the feature onto upstream's current structure rather than a merge.

Six GGUF types are registered in a high ID range (100-104, 107) so
upstream type IDs stay free:

  Q4_0_ROCMFP4       4.50 bpw, dual UE4M3 half-block scales
  Q4_0_ROCMFP4_FAST  4.25 bpw, single scale per block
  Q3_0_ROCMFPX       3.50 bpw
  Q6_0_ROCMFPX       6.50 bpw
  Q8_0_ROCMFPX       8.25 bpw
  Q2_0_ROCMFPX       2.50 bpw (CPU only)

Included:
  - reference codecs in ggml/rocmfp4 and ggml/rocmfpx
  - ggml type traits, quantize dispatch, row-data validation
  - CPU vec_dot/from_float for all six types
  - Vulkan: dequant, get_rows, mul_mat_vec, mul_mm, coopmat2 dequant,
    copy_to/from_quant, set_rows and same-type block copy for the five
    types above
  - llama ftypes and the per-tensor quant recipes (ROCmFP4 LEAN/COHERENT/
    STRIX variants, ROCmFPx AGENT/LEAN routing), llama-quantize presets
  - gguf-py type/ftype constants and dequant classes
  - FPx types added to the test-backend-ops sweep

Two bugs in the upstream fork are fixed rather than carried over:
  - the Vulkan fp6 path used a 32-byte unpacked block layout and stored
    decoded int8 values, which does not match the 24-byte 6-bit packing
    written by the CPU encoder. Reimplemented against the real layout.
  - the CPU fp6 vec_dot decoded the sign|0 code as -0 instead of -32,
    disagreeing with rocmfpx_decode_fp6_code() and costing ~2% error on
    every fp6 mat-vec.

Not included: HIP/CUDA and OpenCL kernels, the q8_1 integer-dot (MMQ)
paths, flash-attention with FPx KV, and the Vulkan fp2 kernels. The HIP
sources ship with the reference codecs but are not built.

Verified on Radeon 8060S (RADV STRIX_HALO): test-backend-ops passes the
full suite, with 857 ROCmFPx cases green. Round-tripped Qwen3-1.7B
through llama-quantize to ROCMFP4_FAST/Q8_0/Q6_0/Q3_0_ROCMFPX and got
coherent generation on both Vulkan and CPU.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 8610c8b)
@gaetan-puleo
sync: the RDNA3.5 fused-quantize MMVQ leaves the shared-expert fusion…
640440b 
… to upstream's kernel

test-backend-ops MUL_MAT_ID_SHARED(type=q8_0,n=1) failed on HIP: the fork's fused-quantize path took the
call and ignored the shared expert.

Assisted-by: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01Qr1p3WtS7Gr8dSEFcgaTLc
@gaetan-puleo
skills : validate.sh runs the KLD without mmap
13b16cf 
With mmap, an 80 GB model sits in the page cache and in GPU memory at once: the long-context KLD of
Qwen3.8-Flash-Next was OOM-killed on a 128 GB Strix Halo. llama-bench already used --load-mode none.

Assisted-by: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01Qr1p3WtS7Gr8dSEFcgaTLc
@gaetan-puleo
skills : validate.sh shortens the long KLD of a model that fills the GPU
7a964d1 
llama-perplexity keeps n_ctx x n_vocab logits in RAM: at 16384 tokens and a 248k vocab that is 16 GB on
top of an 80 GB model, and Qwen3.8-Flash-Next was OOM-killed on a 128 GB Strix Halo, base and candidate
alike. A model above LONG_CTX_BIG_GPU_MIB (64 GiB) of GPU memory at 2048 tokens runs it at LONG_CTX_BIG
(4096) tokens.

Assisted-by: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01Qr1p3WtS7Gr8dSEFcgaTLc
@gaetan-puleo
sync : join the rebased history onto the old master
e0187a3 
The tree is the rebased one (7a964d1) unchanged. The second parent keeps the
old master (deb5e2f) in the history, so master fast-forwards to this commit
without a force push. git log --first-parent shows the fork commits rebased on
upstream c114bd4.

Claude-Session: https://claude.ai/code/session_01Qr1p3WtS7Gr8dSEFcgaTLc
@gaetan-puleo

gaetan-puleo commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Closed: the sync will be redone from scratch.

@gaetan-puleo
gaetan-puleo deleted the sync/upstream-2026-10-14-c114bd4c5 branch October 14, 2026 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pwilkin pwilkin Awaiting requested review from pwilkin pwilkin will be requested when the pull request is marked ready for review pwilkin is a code owner

Assignees

No one assigned

Labels

android Apple Metal Ascend NPU build conversion CUDA devops documentation Improvements or additions to documentation ggml Hexagon IBM zDNN jinja parser model mtmd OpenCL OpenVINO server/ui server SYCL testing vendor Vulkan WebGPU

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.