Repository navigation
Follow up the Qwen3.5-122B-A10B family's tests, kernels and docs - #560
Merged
Merged
Conversation
llama.cpp's gguf-split writes the first part of a split GGUF with the metadata alone, ending at its header: its data section is empty and the padding before it is not written. The parser required that padding and refused the part as truncated. A part of no tensors that ends at its header now has an empty data section. unsloth's Ornith-1.5-122B GGUFs are split this way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A family of the Qwen3.6-35B-A3B's architecture and layers at a larger size (hidden 3072, 48 layers, 32 query heads over 2 KV heads, 64 GDN value heads, experts of 1024). Both name the qwen3_5_moe_text model type and the qwen35moe GGUF architecture, so a config's sizes pick its family. Its draft is Ornith-1.5-122B's DFlash2, trained to fill a block of 16: a draft's block may now be the kernels' 8 rows or more, of which a cycle drafts the first. Families and kernel shapes are data, so another size of an architecture is one value, and a shape no family had its kernels and one table row: - A layout type is an architecture, whose code its families share. A family is a value of it, named on the shared base layout, and every family is one entry of model::kFamilies, which refusals, family lookup and the tests walk. The 122B is kQwen3_5_122BLayout, a Qwen3_6MoeLayout with no code of its own. - A config's family is the one of its model type whose seven sizes it holds. A config of none is refused as it differs from the family it matches furthest. - GDN and attention kernels are chosen from one table of compiled shapes each (kCompiledGdn, kCompiledAttention), their names built from a base and the shape's suffix. New rows: GDN for 64 value heads (_vh64) and attention for 16 query heads per KV head (_kv2_g16). - The kernel tests take every family's GDN and attention shapes from kFamilies (dev/tests/engine/TestFamilies.hpp), so a new family's kernels are tested with no edit to them. A dense paged-attention tile runs in the fewest parts that divide its eight rows and leave each at most 64 fused rows, the 256-thread tile's (SPLASH_ATTENTION_TILE_PARTS, splash_attention_tile_part): one part up to eight query heads per KV head, two up to sixteen, four up to thirty-two. The kernels, host plans and tests read the one rule; today's tiles are one part and unchanged. Measured on an M5 Max (40 cores), interleaved rounds of the attention sweep, which now takes the 122b shape: the two 64-row parts of a group of sixteen ran its 32K and 128K prefill and its 128K verify 20-40% faster than one 128-row tile of 512 threads (32K prefill 67 against 95 ms, 128K verify 1.21 against 1.70 ms), both parts reading the same pages, the second mostly from cache. Splitting further was slower at long histories: sixteen in four parts (128K verify, four lanes, 5.18 against 4.54 ms) and eight in two (128K prefill 172 against 134 ms). Outputs are bit-identical across part counts. Checked on an M5 Max (128 GB) with unsloth/Ornith-1.5-122B-GGUF:UD-Q4_K_XL and its experts streamed: the runtime oracle (fp32 and bf16 state), the kernel tests under shader validation, the native CPU and Python checks, and text, tool calls and vision over HTTP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A draft's rows after its anchor read the target's token table row of the mask token. Qwen3.5-122B-A10B's row, which Ornith-1.5-122B inherits, is of order 1e-6, below what the fp16 block scales of GGUF Q8_0 and the K-quants hold, so a GGUF of the family reads zeros there, and the draft's acceptance on code and reasoning halves (about 2.3 tokens per cycle against 4.9 on the MLX target, which keeps the row). A DFlash2 release may now ship the mask token's BF16 embedding beside its weights, mask_embedding.safetensors holding one tensor mask_embedding of the hidden size. Installation fetches it with the weights; the draft checkpoint keeps it in model.bin after the selector's codebooks; and after each draft input gather one dispatch (draft_mask_rows) sets every lane's mask rows to it. A draft without the file loads and runs as before. With the file beside incoai/Ornith-1.5-122B-DFlash2's weights, unsloth/Ornith-1.5-122B-GGUF:UD-Q4_K_XL drafts 4.7 tokens per cycle on the same coding prompts with every row verified. Resident on an M5 Max, one lane of the backend benchmark decodes 150 tok/s against 71 before (6.40 tokens per cycle against 3.05), and one coding stream over HTTP 105 tok/s against 40. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment on the parser's rule for a part of no tensors, which may end at its header unpadded, and the comment on its test named the program that writes such files. They now describe the layout itself: a split GGUF whose first part holds the metadata alone, its data section empty. Tests: none; comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The family's draft width, layout and arena sizing had no test of their own, and the choice of a family by its sizes had none for a config that differs from every family in the first size. These tests come from ezoushen's version of this family (incoai#529), adapted to the family as a value of the MoE architecture (kQwen3_5_122BLayout): - draft-attention runs the draft kernels at the 122B draft's width, 3072 (dynamic width 768); - model-execution-plans holds every family's layout to requireQwenLayout (forEachFamily), and sizes the 122B's arenas and checks its lane scratch at Apple9, 10 and 11 with 12, 20 and 40 cores: loadedModel takes a family's layout and draft, by default its architecture's first family's; - model-configuration refuses the Qwen3.6-35B-A3B's config at a hidden size of neither MoE family, 4096, as it differs from the first family of its model type (Qwen3.6-35B-A3B 2048). The 122B's config at the 35B's 40 layers is the refusal incoai#539 tests. incoai#529's tests of an exact draft block and of a refused 122B GGUF do not apply: this family drafts the first 8 rows of its block of 16 and serves its GGUF. Tests: on the M5 Pro (64 GB), at the head of this series and with incoai#536 and the other fixes since 1.4.0 merged on top of it, make all and check-native-build build without a warning, model-configuration and model-execution-plans pass (test-engine-cpu), and so does draft-attention under MTL_SHADER_VALIDATION=1. Co-authored-by: ezoushen <ezoushen@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each of the eighteen dense split and reduce entries passed its tile's parts by hand, SPLASH_ATTENTION_TILE_PARTS(Group) beside the group it restates, so an entry could compile with a part count that disagrees with the host's grid (PagedAttention.cpp tileParts) or with its own reduce: only a count that does not divide the tile's rows, or leaves a part more than 64 fused rows, fails to build. The prefill and verify reduce phases also each carried the same steps from a fused row of the whole tile to its part: find the part, write zeros for one past the active rows (which a verify tile never has) and reduce the row within it. The split and reduce macros take their 1.4.0 arguments again; the verify split's scratch and tile and the prefill split phase derive the parts from the group, and the reduce phases pass them to splash_attention_reduce_row. That helper, the two dense reduce phases' only one, now takes the whole tile and its slot stride and resolves the row's part itself, so each phase calls it as it did before the parts. The comment on the partial slots names the part in their layout. Every kernel computes what it did. Tests: on the M5 Max (36 GB), this change and its parent, each built with the same FNV-1a digests printed by two tests for this comparison alone, gave the same bits: - q8_flash_attention_metal_test printed the same 370 lines, against its own library and against splash.metallib. Among them are the digests of each of its 159 cases' output, and of the split's partials and statistics in the 103 that run one: 37 cases for each dense group, of six, eight and sixteen (the last's tiles in two parts), and 48 for the sparse group of twelve. - The runtime oracle on the Qwen3.8-27B UD-Q4_K_M (a group of six), with int8 and with bf16 KV, passed and printed the same digests of its first prefill's and verify's logits, and otherwise the same lines, timings aside. Served by each build, every row verified and on the GPU alone, the Ornith-1.5-122B UD-Q2_K_XL (a group of sixteen), with int8 and with bf16 KV, and the Qwen3.8-Flash-Next ISTA Q2_0 start and answer three greedy chats with the same text and token counts from both. With the next two commits on top, make all check-native-build builds without a warning, and under MTL_SHADER_VALIDATION=1 the q8 flash attention, q8 chunked prefill and paged attention plan tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README: the models table's row read "Qwen3.5-122B-A10B, such as Ornith-1.5-122B", but the draft the installer pairs with the family is Ornith-1.5-122B's, and no draft is released for the base model; the row names the model it serves, Ornith-1.5-122B. Its MLX cell, empty before, names ornith-ai/Ornith-1.5-122B-MLX-4bit, which the family loads (Qwen3_6MoeLayout::loadsSafetensors) and which passed test-real's oracle and the HTTP smoke on the M5 Max (36 GB), text only: its index names a vision shard, model-vision.safetensors, the release has not published. - DEVELOPMENT.md, Drafts: the release's files were config.json and the weights, "only those files" downloaded, while a later paragraph had installation fetch mask_embedding.safetensors too. The first paragraph now lists it, the later one says what the draft does with it, and its 2.3 and 4.7 tokens a cycle give their setup (UD-Q4_K_XL on an M5 Max, every row verified). - The model-configuration test said the 122B's hidden size names its family, which its sizes do; the attention sweep's comment gave three of its four layer counts; the plan test called its grid's x, KV head parts, the KV head; upstream.py's comment on the family-agnostic mask file justified it by one family, now pointing to Drafts. A reflow (ModelDescriptor.mm) was ragged and gguf_file_test.cpp lacked a blank line between two functions. Tests: comments and docs only, beside a renamed variable and message in paged_attention_plan_test. On the M5 Pro (64 GB), with the next commit on top, make all check-native-build builds without a warning; check-source, test-engine-cpu (gguf_file_test and attention-sweep's argument checks among it) and check-python (1319 tests OK, 27 skipped; compileall, ruff check and ruff format clean) pass; and under MTL_SHADER_VALIDATION=1 the paged attention plan test passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every family has a vision parity fixture beside its config fixtures, and test-real, so also the release check, chooses it by the installation's family. The Qwen3.5-122B-A10B had none: on an installation of it that serves vision, vision_encoder_test stopped at "fixture grid is invalid", and the encoder's 3072-wide merger had no parity check. dev/tools/vision_reference.py wrote it, at the default 96x128 image as for the 27B and Flash-Next, from unsloth/Ornith-1.5-122B-GGUF at 745bd742, whose BF16 projector (mmproj-BF16.gguf) installation chooses: 12 merged tokens of 3072 fp32 values. Tests: on the M5 Max (36 GB), vision_encoder_test of that installation (UD-Q2_K_XL with its projector) passes against it under MTL_SHADER_VALIDATION=1: relative error 0.0071 and worst row error 0.0157 against bars of 0.03 and 0.15, deterministic, the 128x128 serving grid included. Its pixels.bin is byte for byte the 27B's and Flash-Next's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five follow-ups to the Qwen3.5-122B-A10B family.
Describe a split GGUF's metadata-only first part by its layout
The comments on the parser's rule for a part of no tensors, which may end at its header unpadded, and on its test named the program that writes such files. They now describe the layout itself: a split GGUF whose first part holds the metadata alone, its data section empty. Comments only.
Test the Qwen3.5-122B-A10B's draft width, layout and arenas
The family's draft width, layout and arena sizing had no test of their own, and the choice of a family by its sizes had none for a config that differs from every family in the first size. These tests come from @ezoushen's version of the family (#529), adapted to the family as a value of the MoE architecture (kQwen3_5_122BLayout):
Two of #529's tests, of an exact draft block and of a refused 122B GGUF, do not apply: this family drafts the first 8 rows of its block of 16 and serves its GGUF.
Derive a dense attention tile's parts from its group in the kernels
Each of the eighteen dense split and reduce entries passed its tile's parts by hand, SPLASH_ATTENTION_TILE_PARTS(Group) beside the group it restates, so an entry could compile with a part count that disagrees with the host's grid (PagedAttention.cpp tileParts) or with its own reduce. The split and reduce macros take their 1.4.0 arguments again; the verify split's scratch and tile and the prefill split phase derive the parts from the group, and splash_attention_reduce_row, the two dense reduce phases' only helper, takes the whole tile and its slot stride and resolves the row's part itself. Every kernel computes what it did. On the M5 Max (36 GB), with FNV-1a digests added to two tests for this comparison alone, this change and its parent gave the same bits: the q8 flash attention test's 159 cases' output and, in the 103 that split, the partials and statistics; and the runtime oracle's first prefill and verify logits on the Qwen3.8-27B UD-Q4_K_M with int8 and with bf16 KV. Served by each build with every row verified, on the GPU alone, the Ornith-1.5-122B UD-Q2_K_XL (a group of sixteen), with int8 and with bf16 KV, and ISTA-DASLab's Qwen3.8-Flash-Next Q2_0 answer three greedy chats with the same text and token counts.
Bring the docs and comments up to date with the 122B family
The README's row read "Qwen3.5-122B-A10B, such as Ornith-1.5-122B", but the draft the installer pairs with the family is Ornith-1.5-122B's, and no draft is released for the base model; the row now names Ornith-1.5-122B, and its MLX cell, empty in #539, names ornith-ai/Ornith-1.5-122B-MLX-4bit, which the family loads and which passed test-real's oracle and the HTTP smoke on the M5 Max (36 GB), text only: its index names a vision shard, model-vision.safetensors, the release has not published. DEVELOPMENT.md's Drafts now lists mask_embedding.safetensors among a release's files, says what the draft does with it, and gives the setup of its tokens-a-cycle figures. Comments in the model-configuration and plan tests, the attention sweep and upstream.py that had fallen behind the family are corrected.
Add the Qwen3.5-122B-A10B's vision parity fixture
Every family has a vision parity fixture, which test-real, and so the release check, chooses by the installation's family. The Qwen3.5-122B-A10B had none: on an installation of it that serves vision, vision_encoder_test stopped at "fixture grid is invalid", and the encoder's 3072-wide merger had no parity check. dev/tools/vision_reference.py wrote it at the default 96x128 image from unsloth/Ornith-1.5-122B-GGUF at 745bd742, whose BF16 projector installation chooses. On the M5 Max (36 GB), the UD-Q2_K_XL with its projector passes against it under MTL_SHADER_VALIDATION=1: relative error 0.0071 and worst row error 0.0157 against bars of 0.03 and 0.15, the 128x128 serving grid included.
Tests: at the top of this stack (release/20), which also holds the fixes stacked on this one:
Earlier, on #539's previous head with the same changes (the same Metal code and splash.metallib), on the M5 Max (36 GB): make check and test-sanitizers pass (1321 Python tests, 38 Metal tests, 36 sanitizer runs); the 27B GGUF's full release-check passes, its ABBA against 1.4.0 within 0.33% on all 14 metrics; the Ornith-1.5-122B MLX 4-bit, language only, passes three oracle runs, verify-models (57 images) and test-http-real, by default and with --verify-rows all, and its draft gives 5.19 tokens a cycle with --verify-rows all (4.08 for the UD-Q2_K_XL GGUF on the same three prompts).
Stacked on #539; review the five commits after 4dd195e.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.