Repository navigation
Support expert-pruned MoE GGUF variants of a family - #536
Closed
wmTJc9IK0Q wants to merge 1 commit into
Closed
wmTJc9IK0Q wants to merge 1 commit into
wmTJc9IK0Q wants to merge 1 commit into
Conversation
A family's layout pinned the routed experts' count, so a GGUF whose header declares a supported pruned count, such as the Qwen3.8-Flash-Next GSQ-RCO Coder release's 256 of 512, was refused at every start (expert_count 256 (expected 512)). A family now states the expert counts its kernels plan (QwenTargetDimensions::supportedExperts), its layout's among them, and the descriptor takes the count the GGUF's own header declares, only if it is one of them (adaptedTargetLayout). Everything the MoE shape feeds -- the planner's tensor shapes, the expert store, the memory plan -- flows from the adapted layout, whose count the narrow routing path holds (SPLASH_MOE_EXPERT_SLOTS). Every other check stays strict: a count of no supported variant keeps the layout and is refused, as is another expert count each token takes.
Contributor
Author
|
Benchmarked via oMLX's benchmark tool on a M3 Max 48GB: |
wmTJc9IK0Q
marked this pull request as ready for review
October 21, 2026 16:59
|
I can confirm this works well! Runs fast on an M4 Max 64GB: |
jianc99
added a commit
that referenced
this pull request
Oct 23, 2026
The family's draft width, layout and arena sizing had no test of their own, and the choice of a family by its sizes had none for a config that differs from every family in the first size. These tests come from ezoushen's version of this family (#529), adapted to the family as a value of the MoE architecture (kQwen3_5_122BLayout): - draft-attention runs the draft kernels at the 122B draft's width, 3072 (dynamic width 768); - model-execution-plans holds every family's layout to requireQwenLayout (forEachFamily), and sizes the 122B's arenas and checks its lane scratch at Apple9, 10 and 11 with 12, 20 and 40 cores: loadedModel takes a family's layout and draft, by default its architecture's first family's; - model-configuration refuses the Qwen3.6-35B-A3B's config at a hidden size of neither MoE family, 4096, as it differs from the first family of its model type (Qwen3.6-35B-A3B 2048). The 122B's config at the 35B's 40 layers is the refusal #539 tests. #529's tests of an exact draft block and of a refused 122B GGUF do not apply: this family drafts the first 8 rows of its block of 16 and serves its GGUF. Tests: on the M5 Pro (64 GB), at the head of this series and with #536 and the other fixes since 1.4.0 merged on top of it, make all and check-native-build build without a warning, model-configuration and model-execution-plans pass (test-engine-cpu), and so does draft-attention under MTL_SHADER_VALIDATION=1. Co-authored-by: ezoushen <ezoushen@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jianc99
added a commit
that referenced
this pull request
Oct 23, 2026
A family's layout fixed its routed experts, and every start held a GGUF's header to that count (gguf::requireMetadata), so a release that keeps fewer experts than its family was refused by the installer's model check and at every start. ISTA-DASLab's Coder release of Qwen3.8-Flash-Next (Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) keeps 256 of its 512 routed experts, 10 still taken per token, and stopped at "GGUF metadata does not match the target: expert_count 256 (expected 512)" (#527). #536 first served it by a list of the counts each family states. The routing, grouping and expert kernels, the expert store, the planner and the memory plan already take the count at run time. The installer now derives it into a MoE GGUF's config.json, as it derives every other size the descriptor reads (gguf.model_config: num_experts, the header's expert_count), and describeSourceModel puts it in the family's layout before the descriptor is made (ggufRoutedExperts, from the text config and the family's layout). Sizing, planning and loading read that one layout, and the model check takes the same path. A count is served when it is: - at most the family's, as a pruned release holds no more experts than its family; - a shape the MoE kernels route (ops::MoeShape::valid), from the experts each token takes to SPLASH_MOE_WIDE_EXPERT_SLOTS; - a multiple of 8: the router is a float projection, and addGgufFloat refuses one whose columns do not fill whole 8-column tiles. Qwen3.8-Flash-Next so serves 16 to 512 experts in steps of 8, and Qwen3.6-35B-A3B and Qwen3.5-122B-A10B 8 to 256. A refusal names the family and that range ("text config num_experts: GGUF 264, Qwen3.6-35B-A3B a multiple of 8 from 8 to 256"). Every other dimension is the family's, and every start's planner still holds the header's expert_count and each router and expert tensor to the count, so a config that disagrees with its GGUF is refused. A config without the count, which an earlier release derived, states the family's; an MLX target's count must still be the family's. No runtime header, kernel or op changes, and DEVELOPMENT.md states the rule under GGUF targets. The one side effect: install/gguf.py's hash keys the derived metadata, so each installed GGUF derives it again locally at its first start after the upgrade, which prints "the GGUF metadata adapter changed". Nothing is downloaded, and the digests of unchanged files come from their records. If the derivation fails, the start keeps the installed assembly, whose config without the count states the family's. Tests: model-configuration's testPrunedExperts runs the start path (inspectModelRoot): a Qwen3.8-Flash-Next GGUF config of 256 experts is the family's layout but for its experts, with the family's draft; 16 and 512 are served, 8, 12 and 520 are refused with "a multiple of 8 from 16 to 512" and true as not a number, and a config without the count states 512; a Qwen3.6-35B-A3B GGUF serves 8 and 128 and refuses 264 ("from 8 to 256"), and an MLX one of 128 is refused. testConfigurationCheck finds a config and metadata pruned to 256 experts, as the Coder's are, to be Qwen3.8-Flash-Next with and without the draft, and refuses that config with the unpruned header (expert_count 512 (expected 256)); the test's MoE GGUF configs, the 35B's and the 122B's, state the 256 experts the installer now derives. test_gguf_metadata derives the config and the metadata from one header in each case: expert_count 128 is Qwen3.6-35B-A3B, 264 is refused, and the Flash-Next fixture config is the one derived from its fixture header. At the top of this stack, on the 64 GB M5 Pro, make all and check-native-build build without a warning; check-source, test-engine-cpu (model configuration with its pruned experts) and check-python (1321 tests OK, 27 skipped; ruff clean) pass; and test-real passes on the Qwen3.6-35B-A3B UD-Q4_K_M and Ornith-1.5-122B UD-Q2_K_XL GGUFs. On the M5 Max (36 GB), ISTA-DASLab's Qwen3.8-Flash-Next Q2_0 passes test-http-real. Earlier, on the M5 Max with #539's previous head and the same changes, make check and test-sanitizers passed, and so did test-http-real on the Coder IQ1_M (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M), also with --verify-rows all. Before that, with the change on 1.4.0 (1482f6b), on the M5 Max (36 GB), prepare, run offline over the Coder IQ1_M and ISTA Q2_0 installations made by 1.4.0, printed "the GGUF metadata adapter changed" for each and exited 0. Of each installation only config.json, its copy under tokenizer/ and model.json's records of them changed, the config gaining num_experts 256 and 512 respectively, and each then started to Ready. Co-authored-by: wmTJc9IK0Q <171362836+wmTJc9IK0Q@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
Thanks @wmTJc9IK0Q! This landed in #561, with you as co-author: instead of a list of supported counts, Splash now takes the routed expert count from the GGUF header and serves any count the MoE kernels support. Closing as superseded. |
Contributor
Author
|
@jianc99 great, thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A family's layout pins the routed experts' count at compile time (
Qwen4ExpLayoutsetsexperts = 512), and every start validates a GGUF's header against it (gguf::requireMetadata,runtime/model/GgufImage.cpp). The expert-pruned Coder release of Qwen3.8-Flash-Next (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF) declaresqwen4exp.expert_count = 256— it removes half of the routed experts (10 still active per token) — and is refused:Change
QwenTargetDimensions::supportedExperts: the expert counts a family's GGUFs whose MoE the kernels plan, the layout's among them; empty for a family of one shape.Qwen4ExpLayoutdeclares{512, 256}.adaptedTargetLayout(ModelDescriptor.mm): the descriptor takes the expert count the GGUF's own header declares, only if it is one of the supported counts — at start (inspectModelRoot, from the installed GGUF), at the installer'smodel-check(inspectSourceConfiguration, from the copied scalar metadata), and so for what sizing plans and what loads.SPLASH_MOE_EXPERT_SLOTS = 256); no kernel changes.Verification
build/engine-tests/model-configuration: PASS, with new cases — the 256-expert header accepted,expert_count 999andexpert_used_count 16refused.build/engine-tests/gguf-planner,runtime-resources,qwen4-kernels: PASS.dev.tests.install.test_gguf_metadata,test_upstream): 84 OK.build/splash model-checkagainst the Coder IQ1_M's real header scalars: accepted ({"family":"Qwen3.8-Flash-Next"});expert_count 999andexpert_used_count 16refused.Need help on this PR? Tag
@codesmith-botwith what you need.