Skip to content

Serve a pruned MoE GGUF by the routed experts its header declares - #561

Merged
jianc99 merged 9 commits into
incoai:mainfrom
jianc99:release/13-pruned-experts
Oct 23, 2026
Merged

jianc99 merged 9 commits into
incoai:mainfrom
jianc99:release/13-pruned-experts

Conversation

@jianc99

@jianc99 jianc99 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Supersedes #536 (thanks @wmTJc9IK0Q); closes #527.

Where #536 served the header's count when the family listed it ({512,
256} for Qwen3.8-Flash-Next), this serves every count up to the family's
that the MoE kernels route and the router's 8-column tiles hold, which
the installer derives from the header into the config, so a start reads
one layout.

A family's layout fixed its routed experts, and every start held a
GGUF's header to that count (gguf::requireMetadata), so a release that
keeps fewer experts than its family was refused by the installer's model
check and at every start. ISTA-DASLab's Coder release of
Qwen3.8-Flash-Next (Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) keeps 256 of
its 512 routed experts, 10 still taken per token, and stopped at "GGUF
metadata does not match the target: expert_count 256 (expected 512)"
(#527). #536 first served it by a list of the counts each family states.

The routing, grouping and expert kernels, the expert store, the planner
and the memory plan already take the count at run time. The installer
now derives it into a MoE GGUF's config.json, as it derives every other
size the descriptor reads (gguf.model_config: num_experts, the header's
expert_count), and describeSourceModel puts it in the family's layout
before the descriptor is made (ggufRoutedExperts, from the text config
and the family's layout). Sizing, planning and loading read that one
layout, and the model check takes the same path. A count is served when
it is:

  • at most the family's, as a pruned release holds no more experts than
    its family;
  • a shape the MoE kernels route (ops::MoeShape::valid), from the experts
    each token takes to SPLASH_MOE_WIDE_EXPERT_SLOTS;
  • a multiple of 8: the router is a float projection, and addGgufFloat
    refuses one whose columns do not fill whole 8-column tiles.

Qwen3.8-Flash-Next so serves 16 to 512 experts in steps of 8, and
Qwen3.6-35B-A3B and Qwen3.5-122B-A10B 8 to 256. A refusal names the
family and that range ("text config num_experts: GGUF 264,
Qwen3.6-35B-A3B a multiple of 8 from 8 to 256"). Every other dimension
is the family's, and every start's planner still holds the header's
expert_count and each router and expert tensor to the count, so a config
that disagrees with its GGUF is refused. A config without the count,
which an earlier release derived, states the family's; an MLX target's
count must still be the family's. No runtime header, kernel or op
changes, and DEVELOPMENT.md states the rule under GGUF targets.

The one side effect: install/gguf.py's hash keys the derived metadata,
so each installed GGUF derives it again locally at its first start after
the upgrade, which prints "the GGUF metadata adapter changed". Nothing
is downloaded, and the digests of unchanged files come from their
records. If the derivation fails, the start keeps the installed
assembly, whose config without the count states the family's.

Tests: model-configuration's testPrunedExperts runs the start path
(inspectModelRoot): a Qwen3.8-Flash-Next GGUF config of 256 experts is
the family's layout but for its experts, with the family's draft; 16 and
512 are served, 8, 12 and 520 are refused with "a multiple of 8 from 16
to 512" and true as not a number, and a config without the count states
512; a Qwen3.6-35B-A3B GGUF serves 8 and 128 and refuses 264 ("from 8 to
256"), and an MLX one of 128 is refused. testConfigurationCheck finds a
config and metadata pruned to 256 experts, as the Coder's are, to be
Qwen3.8-Flash-Next with and without the draft, and refuses that config
with the unpruned header (expert_count 512 (expected 256)); the test's
MoE GGUF configs, the 35B's and the 122B's, state the 256 experts the
installer now derives. test_gguf_metadata derives the config and the
metadata from one header in each case: expert_count 128 is
Qwen3.6-35B-A3B, 264 is refused, and the Flash-Next fixture config is
the one derived from its fixture header.

At the top of this stack, on the 64 GB M5 Pro, make all and
check-native-build build without a warning; check-source,
test-engine-cpu (model configuration with its pruned experts) and
check-python (1321 tests OK, 27 skipped; ruff clean) pass; and test-real
passes on the Qwen3.6-35B-A3B UD-Q4_K_M and Ornith-1.5-122B UD-Q2_K_XL
GGUFs. On the M5 Max (36 GB), ISTA-DASLab's Qwen3.8-Flash-Next Q2_0
passes test-http-real. Earlier, on the M5 Max with #539's previous head
and the same changes, make check and test-sanitizers passed, and so did
test-http-real on the Coder IQ1_M
(ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M), also with
--verify-rows all.

Before that, with the change on 1.4.0 (1482f6b), on the M5 Max (36 GB),
prepare, run offline over the Coder IQ1_M and ISTA Q2_0 installations
made by 1.4.0, printed "the GGUF metadata adapter changed" for each and
exited 0. Of each installation only config.json, its copy under
tokenizer/ and model.json's records of them changed, the config gaining
num_experts 256 and 512 respectively, and each then started to Ready.

Stacked on #560; review the commit after it.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

zhijian-liu and others added 9 commits October 21, 2026 20:27
llama.cpp's gguf-split writes the first part of a split GGUF with the
metadata alone, ending at its header: its data section is empty and the
padding before it is not written. The parser required that padding and
refused the part as truncated. A part of no tensors that ends at its
header now has an empty data section. unsloth's Ornith-1.5-122B GGUFs
are split this way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A family of the Qwen3.6-35B-A3B's architecture and layers at a larger
size (hidden 3072, 48 layers, 32 query heads over 2 KV heads, 64 GDN
value heads, experts of 1024). Both name the qwen3_5_moe_text model
type and the qwen35moe GGUF architecture, so a config's sizes pick its
family. Its draft is Ornith-1.5-122B's DFlash2, trained to fill a block
of 16: a draft's block may now be the kernels' 8 rows or more, of which
a cycle drafts the first.

Families and kernel shapes are data, so another size of an architecture
is one value, and a shape no family had its kernels and one table row:

- A layout type is an architecture, whose code its families share. A
  family is a value of it, named on the shared base layout, and every
  family is one entry of model::kFamilies, which refusals, family lookup
  and the tests walk. The 122B is kQwen3_5_122BLayout, a
  Qwen3_6MoeLayout with no code of its own.
- A config's family is the one of its model type whose seven sizes it
  holds. A config of none is refused as it differs from the family it
  matches furthest.
- GDN and attention kernels are chosen from one table of compiled
  shapes each (kCompiledGdn, kCompiledAttention), their names built
  from a base and the shape's suffix. New rows: GDN for 64 value heads
  (_vh64) and attention for 16 query heads per KV head (_kv2_g16).
- The kernel tests take every family's GDN and attention shapes from
  kFamilies (dev/tests/engine/TestFamilies.hpp), so a new family's
  kernels are tested with no edit to them.

A dense paged-attention tile runs in the fewest parts that divide its
eight rows and leave each at most 64 fused rows, the 256-thread tile's
(SPLASH_ATTENTION_TILE_PARTS, splash_attention_tile_part): one part up
to eight query heads per KV head, two up to sixteen, four up to
thirty-two. The kernels, host plans and tests read the one rule;
today's tiles are one part and unchanged.

Measured on an M5 Max (40 cores), interleaved rounds of the attention
sweep, which now takes the 122b shape: the two 64-row parts of a group
of sixteen ran its 32K and 128K prefill and its 128K verify 20-40%
faster than one 128-row tile of 512 threads (32K prefill 67 against
95 ms, 128K verify 1.21 against 1.70 ms), both parts reading the same
pages, the second mostly from cache. Splitting further was slower at
long histories: sixteen in four parts (128K verify, four lanes, 5.18
against 4.54 ms) and eight in two (128K prefill 172 against 134 ms).
Outputs are bit-identical across part counts.

Checked on an M5 Max (128 GB) with unsloth/Ornith-1.5-122B-GGUF:UD-Q4_K_XL
and its experts streamed: the runtime oracle (fp32 and bf16 state), the
kernel tests under shader validation, the native CPU and Python checks,
and text, tool calls and vision over HTTP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A draft's rows after its anchor read the target's token table row of the
mask token. Qwen3.5-122B-A10B's row, which Ornith-1.5-122B inherits, is
of order 1e-6, below what the fp16 block scales of GGUF Q8_0 and the
K-quants hold, so a GGUF of the family reads zeros there, and the
draft's acceptance on code and reasoning halves (about 2.3 tokens per
cycle against 4.9 on the MLX target, which keeps the row).

A DFlash2 release may now ship the mask token's BF16 embedding beside
its weights, mask_embedding.safetensors holding one tensor
mask_embedding of the hidden size. Installation fetches it with the
weights; the draft checkpoint keeps it in model.bin after the
selector's codebooks; and after each draft input gather one dispatch
(draft_mask_rows) sets every lane's mask rows to it. A draft without
the file loads and runs as before.

With the file beside incoai/Ornith-1.5-122B-DFlash2's weights,
unsloth/Ornith-1.5-122B-GGUF:UD-Q4_K_XL drafts 4.7 tokens per cycle on
the same coding prompts with every row verified. Resident on an M5 Max,
one lane of the backend benchmark decodes 150 tok/s against 71 before
(6.40 tokens per cycle against 3.05), and one coding stream over HTTP
105 tok/s against 40.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment on the parser's rule for a part of no tensors, which may end at
its header unpadded, and the comment on its test named the program that
writes such files. They now describe the layout itself: a split GGUF whose
first part holds the metadata alone, its data section empty.

Tests: none; comments only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The family's draft width, layout and arena sizing had no test of their
own, and the choice of a family by its sizes had none for a config that
differs from every family in the first size.

These tests come from ezoushen's version of this family (incoai#529), adapted
to the family as a value of the MoE architecture (kQwen3_5_122BLayout):
- draft-attention runs the draft kernels at the 122B draft's width, 3072
  (dynamic width 768);
- model-execution-plans holds every family's layout to requireQwenLayout
  (forEachFamily), and sizes the 122B's arenas and checks its lane
  scratch at Apple9, 10 and 11 with 12, 20 and 40 cores: loadedModel
  takes a family's layout and draft, by default its architecture's first
  family's;
- model-configuration refuses the Qwen3.6-35B-A3B's config at a hidden
  size of neither MoE family, 4096, as it differs from the first family
  of its model type (Qwen3.6-35B-A3B 2048). The 122B's config at the
  35B's 40 layers is the refusal incoai#539 tests.
incoai#529's tests of an exact draft block and of a refused 122B GGUF do not
apply: this family drafts the first 8 rows of its block of 16 and serves
its GGUF.

Tests: on the M5 Pro (64 GB), at the head of this series and with incoai#536
and the other fixes since 1.4.0 merged on top of it, make all and
check-native-build build without a warning, model-configuration and
model-execution-plans pass (test-engine-cpu), and so does draft-attention
under MTL_SHADER_VALIDATION=1.

Co-authored-by: ezoushen <ezoushen@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each of the eighteen dense split and reduce entries passed its tile's
parts by hand, SPLASH_ATTENTION_TILE_PARTS(Group) beside the group it
restates, so an entry could compile with a part count that disagrees with
the host's grid (PagedAttention.cpp tileParts) or with its own reduce:
only a count that does not divide the tile's rows, or leaves a part more
than 64 fused rows, fails to build. The prefill and verify reduce phases
also each carried the same steps from a fused row of the whole tile to
its part: find the part, write zeros for one past the active rows (which
a verify tile never has) and reduce the row within it.

The split and reduce macros take their 1.4.0 arguments again; the verify
split's scratch and tile and the prefill split phase derive the parts from
the group, and the reduce phases pass them to splash_attention_reduce_row.
That helper, the two dense reduce phases' only one, now takes the whole
tile and its slot stride and resolves the row's part itself, so each phase
calls it as it did before the parts. The comment on the partial slots names
the part in their layout. Every kernel computes what it did.

Tests: on the M5 Max (36 GB), this change and its parent, each built with
the same FNV-1a digests printed by two tests for this comparison alone,
gave the same bits:
- q8_flash_attention_metal_test printed the same 370 lines, against its
  own library and against splash.metallib. Among them are the digests of
  each of its 159 cases' output, and of the split's partials and
  statistics in the 103 that run one: 37 cases for each dense group, of
  six, eight and sixteen (the last's tiles in two parts), and 48 for the
  sparse group of twelve.
- The runtime oracle on the Qwen3.8-27B UD-Q4_K_M (a group of six), with
  int8 and with bf16 KV, passed and printed the same digests of its first
  prefill's and verify's logits, and otherwise the same lines, timings
  aside.
Served by each build, every row verified and on the GPU alone, the
Ornith-1.5-122B UD-Q2_K_XL (a group of sixteen), with int8 and with bf16
KV, and the Qwen3.8-Flash-Next ISTA Q2_0 start and answer three greedy
chats with the same text and token counts from both. With the next two
commits on top, make all check-native-build builds without a warning,
and under MTL_SHADER_VALIDATION=1 the q8 flash attention, q8 chunked
prefill and paged attention plan tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README: the models table's row read "Qwen3.5-122B-A10B, such as
  Ornith-1.5-122B", but the draft the installer pairs with the family is
  Ornith-1.5-122B's, and no draft is released for the base model; the
  row names the model it serves, Ornith-1.5-122B. Its MLX cell, empty
  before, names ornith-ai/Ornith-1.5-122B-MLX-4bit, which the family
  loads (Qwen3_6MoeLayout::loadsSafetensors) and which passed
  test-real's oracle and the HTTP smoke on the M5 Max (36 GB), text
  only: its index names a vision shard, model-vision.safetensors, the
  release has not published.
- DEVELOPMENT.md, Drafts: the release's files were config.json and the
  weights, "only those files" downloaded, while a later paragraph had
  installation fetch mask_embedding.safetensors too. The first paragraph
  now lists it, the later one says what the draft does with it, and its
  2.3 and 4.7 tokens a cycle give their setup (UD-Q4_K_XL on an M5 Max,
  every row verified).
- The model-configuration test said the 122B's hidden size names its
  family, which its sizes do; the attention sweep's comment gave three
  of its four layer counts; the plan test called its grid's x, KV head
  parts, the KV head; upstream.py's comment on the family-agnostic mask
  file justified it by one family, now pointing to Drafts. A reflow
  (ModelDescriptor.mm) was ragged and gguf_file_test.cpp lacked a blank
  line between two functions.

Tests: comments and docs only, beside a renamed variable and message in
paged_attention_plan_test. On the M5 Pro (64 GB), with the next commit on
top, make all check-native-build builds without a warning; check-source,
test-engine-cpu (gguf_file_test and attention-sweep's argument checks
among it) and check-python (1319 tests OK, 27 skipped; compileall, ruff
check and ruff format clean) pass; and under MTL_SHADER_VALIDATION=1 the
paged attention plan test passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every family has a vision parity fixture beside its config fixtures, and
test-real, so also the release check, chooses it by the installation's
family. The Qwen3.5-122B-A10B had none: on an installation of it that
serves vision, vision_encoder_test stopped at "fixture grid is invalid",
and the encoder's 3072-wide merger had no parity check.

dev/tools/vision_reference.py wrote it, at the default 96x128 image as
for the 27B and Flash-Next, from unsloth/Ornith-1.5-122B-GGUF at
745bd742, whose BF16 projector (mmproj-BF16.gguf) installation chooses:
12 merged tokens of 3072 fp32 values.

Tests: on the M5 Max (36 GB), vision_encoder_test of that installation
(UD-Q2_K_XL with its projector) passes against it under
MTL_SHADER_VALIDATION=1: relative error 0.0071 and worst row error 0.0157
against bars of 0.03 and 0.15, deterministic, the 128x128 serving grid
included. Its pixels.bin is byte for byte the 27B's and Flash-Next's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A family's layout fixed its routed experts, and every start held a
GGUF's header to that count (gguf::requireMetadata), so a release that
keeps fewer experts than its family was refused by the installer's model
check and at every start. ISTA-DASLab's Coder release of
Qwen3.8-Flash-Next (Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) keeps 256 of
its 512 routed experts, 10 still taken per token, and stopped at "GGUF
metadata does not match the target: expert_count 256 (expected 512)"
(incoai#527). incoai#536 first served it by a list of the counts each family states.

The routing, grouping and expert kernels, the expert store, the planner
and the memory plan already take the count at run time. The installer
now derives it into a MoE GGUF's config.json, as it derives every other
size the descriptor reads (gguf.model_config: num_experts, the header's
expert_count), and describeSourceModel puts it in the family's layout
before the descriptor is made (ggufRoutedExperts, from the text config
and the family's layout). Sizing, planning and loading read that one
layout, and the model check takes the same path. A count is served when
it is:

- at most the family's, as a pruned release holds no more experts than
  its family;
- a shape the MoE kernels route (ops::MoeShape::valid), from the experts
  each token takes to SPLASH_MOE_WIDE_EXPERT_SLOTS;
- a multiple of 8: the router is a float projection, and addGgufFloat
  refuses one whose columns do not fill whole 8-column tiles.

Qwen3.8-Flash-Next so serves 16 to 512 experts in steps of 8, and
Qwen3.6-35B-A3B and Qwen3.5-122B-A10B 8 to 256. A refusal names the
family and that range ("text config num_experts: GGUF 264,
Qwen3.6-35B-A3B a multiple of 8 from 8 to 256"). Every other dimension
is the family's, and every start's planner still holds the header's
expert_count and each router and expert tensor to the count, so a config
that disagrees with its GGUF is refused. A config without the count,
which an earlier release derived, states the family's; an MLX target's
count must still be the family's. No runtime header, kernel or op
changes, and DEVELOPMENT.md states the rule under GGUF targets.

The one side effect: install/gguf.py's hash keys the derived metadata,
so each installed GGUF derives it again locally at its first start after
the upgrade, which prints "the GGUF metadata adapter changed". Nothing
is downloaded, and the digests of unchanged files come from their
records. If the derivation fails, the start keeps the installed
assembly, whose config without the count states the family's.

Tests: model-configuration's testPrunedExperts runs the start path
(inspectModelRoot): a Qwen3.8-Flash-Next GGUF config of 256 experts is
the family's layout but for its experts, with the family's draft; 16 and
512 are served, 8, 12 and 520 are refused with "a multiple of 8 from 16
to 512" and true as not a number, and a config without the count states
512; a Qwen3.6-35B-A3B GGUF serves 8 and 128 and refuses 264 ("from 8 to
256"), and an MLX one of 128 is refused. testConfigurationCheck finds a
config and metadata pruned to 256 experts, as the Coder's are, to be
Qwen3.8-Flash-Next with and without the draft, and refuses that config
with the unpruned header (expert_count 512 (expected 256)); the test's
MoE GGUF configs, the 35B's and the 122B's, state the 256 experts the
installer now derives. test_gguf_metadata derives the config and the
metadata from one header in each case: expert_count 128 is
Qwen3.6-35B-A3B, 264 is refused, and the Flash-Next fixture config is
the one derived from its fixture header.

At the top of this stack, on the 64 GB M5 Pro, make all and
check-native-build build without a warning; check-source,
test-engine-cpu (model configuration with its pruned experts) and
check-python (1321 tests OK, 27 skipped; ruff clean) pass; and test-real
passes on the Qwen3.6-35B-A3B UD-Q4_K_M and Ornith-1.5-122B UD-Q2_K_XL
GGUFs. On the M5 Max (36 GB), ISTA-DASLab's Qwen3.8-Flash-Next Q2_0
passes test-http-real. Earlier, on the M5 Max with incoai#539's previous head
and the same changes, make check and test-sanitizers passed, and so did
test-http-real on the Coder IQ1_M
(ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M), also with
--verify-rows all.

Before that, with the change on 1.4.0 (1482f6b), on the M5 Max (36 GB),
prepare, run offline over the Coder IQ1_M and ISTA Q2_0 installations
made by 1.4.0, printed "the GGUF metadata adapter changed" for each and
exited 0. Of each installation only config.json, its copy under
tokenizer/ and model.json's records of them changed, the config gaining
num_experts 256 and 512 respectively, and each then started to Ready.

Co-authored-by: wmTJc9IK0Q <171362836+wmTJc9IK0Q@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jianc99
jianc99 merged commit 720f5fd into incoai:main Oct 23, 2026
5 checks passed
snagnever added a commit to snagnever/macstudio-local-llm that referenced this pull request Oct 27, 2026
…lease; t5r runs its 8K and 256K bands and the battery at 262144

Splash 1.4.1 (2026-10-26) is the first release with incoai/splash#561. It installs
beside 1.4.0 in ~/.local/opt/qwen38/splash-v1.4.1; k2a stays on 1.4.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen-3.8 Next: Support reduced expert count?

2 participants