llama : qwen35/qwen35moe GDN graph - 2-D output projection, in-place recurrent state, masked h_nextn fix - #57
Open
SimonTeixidor wants to merge 2 commits into
Open
SimonTeixidor wants to merge 2 commits into
SimonTeixidor wants to merge 2 commits into
Conversation
build_layer_attn_linear reshapes the gated-norm result to [d, n_seq_tokens, n_seqs] before the ssm_out projection and then flattens the projection result to [n_embd, n_seq_tokens*n_seqs] on the very next line. The 3-D shape therefore carries no information past the mul_mat, but it does decide how the mul_mat is dispatched: src1 with ne12 = n_seqs is a batch of n_seqs matmuls of ne11 = n_seq_tokens columns each, and every one of them re-reads the whole ssm_out weight. Reshaping to [d, n_seq_tokens*n_seqs] instead gives a single matmul of ne11 = n_seq_tokens*n_seqs columns over one weight read. Mathematically identical: mul_mat is per-column, attn_out_norm is contiguous (the 3-D reshape already required that), and the columns are in the same order in both layouts, so the flattened call computes the same columns in the same order. The reshape_2d after the projection becomes shape-preserving. At n_seqs = 6 this is the difference between a batched call and six unbatched ones on every backend, over 48 of these projections per decode step on the 27B models. Note this only helps n_seqs > 1: one sequence with n_seq_tokens speculative tokens already arrives as a single ne11 = n_seq_tokens matmul. Only qwen35 and qwen35moe are changed, the architectures this was tested on. Assisted-by: Claude Opus 5
…e cache build_rs gains an allow_inplace option: when the sequence-copy mapping is the identity, the state input is a view of the cache slot rather than a get_rows copy. qwen35 and qwen35moe both opt in, since they build the same Gated DeltaNet graph through llm_build_delta_net_base. s_copy_main_identity joins the graph-reuse key: a seq_cp between steps changes whether the mapping is the identity, so a graph built for one must not be reused for the other. test-recurrent-state-rollback gains a check that decodes two sequences in shared ubatches while seq_cp swaps them, forks one and rolls it back, with graph reuse on. The logits are compared with a context without graph reuse (bit-exact), one with n_rs_seq = 1 and one that decodes one sequence per ubatch. It uses fresh random weights at scale 1: at the 0.01 scale of the generated models the recurrent branch does not change the logits, so a wrong state would not be seen. The check runs for qwen35 and qwen35moe, and the test now also runs on the generated qwen35moe model. Assisted-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MkuKwgPNgpAg7EbxKR2iWC
dzannotti
requested changes
Sep 15, 2026
Collaborator
There was a problem hiding this comment.
Validation: a Release HIP gfx1151 build completed.
Benchmark reproduced on the PR's Qwen3.6-35B-A3B target: tg128 at depth 32000 averaged 48.92 t/s on current master and 49.95 t/s on this PR across six alternating samples per arm: +2.11%. This closely matches the reported +2.28% result.
Correctness/test request: add a regression that interleaves two sequences with seq_cp/rollback and graph reuse, alongside the multi-row MTP case. Cover every changed architecture rather than only the reported Qwen variants.
Scope: this mixes a general GDN graph optimization across four architectures, a recurrent-state change, and an MTP correctness fix. It is not Strix-specific; split it and route the generic pieces through halo-box/llama.cpp/upstream.
comment generated by my clanker Codex
SimonTeixidor
force-pushed
the
pr/qwen35-gdn-graph
branch
from
September 15, 2026 19:49
c976b39 to
e3f31cf
Compare
Author
|
I reduced the scope of this one to the two model architectures that I have been optimizing a lot. I'm trying to push this upstream as well, first step is here: ggml-org#28265. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
pr/qwen35-gdn-graph(c976b39c4), 3 commits onhalo/master69946438a.Overview
b009b7972Flatten the GDN output projection input to 2-D: onessm_outmatmul overn_seq_tokens*n_seqscolumns instead ofn_seqsmatmuls that each read the whole weight. Same columns, same order; onlyn_seqs > 1changes.e3f31cf6dRun gated_delta_net in place on the recurrent state cache (qwen35, qwen35moe).build_rsgainsallow_inplace: with an identity sequence-copy mapping the state input is a view of the cache rows, not aget_rowscopy;s_copy_main_identityjoins the graph-reuse key. Off while rollback snapshots are kept (n_rs_seq > 0). Adds aseq_cp/graph-reuse check totest-recurrent-state-rollback(qwen35, qwen35moe), against no reuse,n_rs_seq = 1and one sequence per ubatch.Measurements
Stock:
654803517, same flags, image and session. Mirrored blocks, n = 4 per build.Batched decode at npl 1-2 and every other pp2048 / tg128 cell (depths 0-64000) are unresolved.
Correctness:
test-backend-ops test -b ROCm0: GATED_DELTA_NET 50/50, SSM_CONV 45/45. No ggml code changes.Raw results
Prompt processing:
llama-bench -m MODEL -p 2048 -d 0,12000,32000,64000 -b 1024 -ub 1024 -n 0 -r 2 -ngl 99 -fa on -ctk f16 -ctv f16 --load-mode none -o jsonl. Token generation: same with-p 0 -n 128 -d 0,32000.Qwen3.8-27B, prompt processing:
Qwen3.6-35B-A3B, prompt processing:
Qwen3.8-27B, token generation:
Qwen3.6-35B-A3B, token generation:
Batched decode:
llama-batched-bench -c 8192 -b 4096 -ub 512 -npp 512 -ntg 128 -npl ..., decode throughput fromspeed_tg, one sample per invocation.Qwen3.8-27B,
-npl 1,2,4,6,8(blocks A/C: stock, this PR, matmul PR; B/D reversed):Qwen3.6-35B-A3B,
-npl 1,6(A/C stock first, B/D PR first):Correctness:
llama-debug --save-logits -b 16384 -ub 512(prompt token ids and complete final logits, float32 x 248320);llama-perplexity --kl-divergence-base(all scored positions; log-probs quantised to 16 bits within 16 nats of the max, so identity is strong but not complete);llama-servergreedy/completion, 256 tokens (128 on the 13480-token prompt),n_probs 10. Same-b/-ub, context, KV types and FA on both builds. The single-sequence checks cannot see commit 1 (n_seqs = 1makes the reshape a no-op); the-b 2048 -ub 2048 -c 512run givesn_seqs = 4.llama-debug --save-logits,-b 16384 -ub 512: prompt token ids + complete final logits (float32 x 248320)llama-perplexity -c 512 -b/-ub 512 --chunks 4, KLD-base files (all scored positions)-b/-ub 2048(n_seq = 4, 4 sequences per ubatch, exercises commit 1)llama-servergreedy/completion, 256 tokens (128 on the long prompt),n_probs 10On the 35B the 4-sequence KLD-base files are also byte-identical to the 1-sequence ones on both builds; on the 27B they differ (PPL 5.4312 vs 5.4246), as batch-width kernel choice predicts.
Requirements
654803517. The branch was then rebased onto69946438a, whose 6 new commits touch onlyggml/rocmfpx, one Vulkan shader andtests/CMakeLists.txt; not re-measured.GDN_STATE_CPY, mentioned in a comment): not built; ROCm only.seq_cpbetween steps: no dedicated test;--parallel > 1server decoding not run.llama-benchy: not run (MTP drafting usesn_rs_seq > 0, where in-place state is off).test-backend-opssuite: not run (no ggml change).-r 2over two mirrored blocks, not one-r 4).https://claude.ai/code/session_01MkuKwgPNgpAg7EbxKR2iWC