Repository navigation
Support Ornith-1.5-122B (Qwen3.5-122B-A10B) and its DFlash2 draft #528
Description
Summary
ornith-ai/Ornith-1.5-122B-MLX-4bit (Qwen3.5-122B-A10B: hidden size 3072, 48 layers, 16 key / 64 value GDN heads, 32 query / 2 KV attention heads) does not load today: inspectModelRoot refuses it at hidden_size, and its attention shape (16 query heads per KV head) has no kernel. incoai/Ornith-1.5-122B-DFlash2 already exists, but no family pairs it. I have it loading and serving with that draft on top of current main (1.4.0), and would like to offer it as a PR: ezoushen/splash ornith-122b-1.4.0, 4 commits.
The approach, for agreement before review
Qwen3_6MoeLayoutkeeps its type and its default (the 35B). It takes a geometry value with two instances, chosen by exacthidden_sizeand layer count; any other config still falls back to the 35B and keeps its refusal text. The 122B's GGUF is refused until a GGUF has run.- Kernels: GDN
vh64, draft convolution at width 3072 (already generic in 1.4.0), and attentionkv2_g16. The paged-attention tile's softmax lanes (4 lanes x 128 fused rows = 512) exceed its 256 threads, so the softmax takes a second pass over its lanes at the call site; a single-pass tile calls the old function unchanged. - No existing kernel changes: against a build of
main(1482f6b), 1816 kernels are AIR-identical, 26 are new, 0 differ (dev/tools/kernel_identity.py). - The draft was trained for blocks of 16 and the engine drafts 8 rows, so the layout records the block a family's draft was trained for and checks the draft's
config.jsonagainst it.
Measurements (the branch alone on main)
Apple M5 Max, 128 GB, macOS 27.0; Xcode 26.6; official MLX 4-bit target and the incoai draft, --language-only --verify-rows all --max-memory 80G. Weights load in 16 s; context 256K; no engine restarts. Greedy, thinking off.
| Splash | mlx-lm 0.32.2, no draft | |
|---|---|---|
| Decode, counting / code / prose, tok/s | 232 / 157 / 81 | 57-66 |
| Cold prefill at 0.8K / 6K / 25K tokens, tok/s | 1986 / 1739 / 925 | 1022 / 905 / 926 |
| 4 concurrent counting streams, aggregate tok/s | 208 |
Medians of three trials, taken in a second pass when the machine was calm (load average 4-8, swap usage unchanged); the three 25K cold-prefill TTFTs were 21, 27 and 36 s. The mlx-lm figures were measured earlier on the same machine, not re-run.
Quality, same prompts as an mlx-lm run of the same checkpoint: 10/10 on ten arithmetic word problems on both; HumanEval first 40 problems, executed: 36/40 here (fails 10, 17, 32, 38) against 37/40 on mlx-lm (fails 10, 32, 38); the extra failure is a solution that misses the empty-string case. The sets are small.
What is not covered
- Vision: the official MLX repo's index names
model-vision.safetensorsbut does not contain it, so it serves text only (--language-only). - GGUF, thinking mode, and long-context quality are not measured; prefill at 25K is on par with mlx-lm, not faster.
- The draft's first 8 rows of a 16-row block are what run; the acceptance cost of that is unmeasured (cumulative draft acceptance over the run above: 0.61 of drafted tokens).
Would you take this as a PR, and is a geometry value on the existing layout the shape you want for a second MoE size?
Activity
studer commentedon Oct 21, 2026
Thanks for contributing this and documenting the measurements and limitations so carefully. I use Splash with Ornith 35B through Unsloth, and I’d love to see Ornith 122B running on my 64 GB Mac with DFlash2 and SSD streaming, along the lines of Flash-Next. That would be a dream setup.
I appreciate the time you’re putting into this, and I’m happy to support the effort however I can.
ezoushen commentedon Oct 22, 2026
Thanks @studer, that's kind of you, and a 64 GB setup is exactly the case I'd like to hear about.
You can try it now from my fork: branch
ornith-122b-1.4.0of https://github.com/ezoushen/splash, the branch behind #529. As of today it also has a faster attention path for the 122B (on an M5 Max, about 1.6x on a cold 64K prompt and 1.9x when adding 4K tokens at 128K context, same output).The DFlash2 draft is paired automatically. On a 64 GB Mac,
--expert-memorystreams the experts from the SSD: by default it keeps what fits in memory, or you can set a cap such as--expert-memory 24G. As a rough guide, on a 128 GB M5 Max with the expert store capped at 24G, the process used about 32 GiB and decoded about 44 / 17 / 9 tok/s on counting / code / prose prompts; a real 64 GB machine has less page cache, so expect it to be slower. Those numbers predate today's attention change. I build with Xcode 26.6.Any feedback is welcome: speed, memory pressure, crashes or odd output. Your Mac model and the flags you used would help me reproduce it.
studer commentedon Oct 22, 2026
Thanks again for making this available. I’ve now built and tested
82368b006e1043478f88a47559da62d7078436d0on an Apple M5 Max (40 GPU cores, 64 GiB), macOS 27.0.1, Xcode 27.1 with the Metal Toolchain. The official MLX 4-bit 122B and paired Inco DFlash2 serve successfully through our existing Unsloth integration.Here are short text-only measurements, with greedy decoding and thinking disabled. Each table entry is the median of two replies to that prompt; policy order was A/B/C/C/B/A. Counting 1..40, a loop-based Python factorial function (zero returns 1; negative input raises ValueError), and a roughly 90-word explanation of source verification ran in that order after each reload. All had a 192-token output cap and tools disabled. Each request used a unique ignored system marker; native counters confirmed zero prefix reuse; the OS page cache was not purged. Tokens/s are the native decode rate; first-text times run from the HTTPS request through Unsloth to the first visible answer text, excluding model loading.
The resident expert arenas were 22.72/35.12/41.91 GiB respectively (4,596/7,103/8,478 slots out of 12,288). The 24G-capped and auto-experts/44G profiles stayed at normal sampled memory pressure. With the default overall budget, all sampled requests reported warning pressure and system swap use rose by about 882 MiB during the first pass, remaining elevated afterwards; no critical pressure was observed. That profile increased code throughput but didn’t improve prose throughput over auto/44G. Counting’s first text was also slower, despite faster subsequent generation.
Common serving flags were
--max-context 32768 --verify-rows all --kv-format int8 --state-format fp32 --persistent-cache --max-cache-disk 16G. The first two profiles used--max-memory 44Gwith--expert-memory 24Gorauto; the third used--expert-memory autoand omitted--max-memory. Actual serving used the prepared selection and binary from this branch behind Unsloth, rather than a separate standalone HTTP benchmark. Model and prefix-cache storage were on the local data SSD.All 18 short requests completed without crashes. Counting was exact, the factorial functions handled zero/positive/negative inputs correctly, and the prose showed no obvious corruption on manual inspection. This was a small feasibility/speed check, not a broad quality benchmark or maximum-context stress test. Native expert-read and accepted-draft counters confirmed streaming/speculation were active. Happy to share more detailed measurements if useful, and thank you again for the work.
studer commentedon Oct 22, 2026
A follow-up from the same 64 GiB machine: a lower-bit target makes a substantial difference with your branch. Thanks again for enabling us to try this.
Tested through our existing Unsloth interface/API on an Apple M5 Max, 40 GPU cores, 64 GiB, macOS 27.0.1, Xcode 27.1/Metal Toolchain. Same
ezoushen/splashcommit82368b006e1043478f88a47559da62d7078436d0behind #529, withincoai/Ornith-1.5-122B-DFlash2(9baefc1bec85c08193ab442e0ca5f7057669b61d). No further engine changes.Targets:
ornith-ai/Ornith-1.5-122B-MLX-4bit, as the fresh control.araysuter/Ornith-1.5-122B-MLX-mixed-2bit, revision848e5a0d53f457d213bb43e554012722f3f3fc60: 40.62 GB of weight files; routed experts 2-bit affine/group 64, other projections 4-bit, routers/gates 8-bit.araysuter/Ornith-1.5-122B-MLX-mixed-2bit-down3, revision0705915b20c311aee268664d85c54c340e3b1ccb: 45.45 GB; expert down projections use 3-bit. Both exports include vision assets and exclude native MTP weights; the existing external DFlash2 was used.Common flags:
--expert-memory auto --max-memory 44G --max-context 32768 --verify-rows all --kv-format int8 --state-format fp32 --persistent-cache --max-cache-disk 16G. Model files and dedicated prefix caches are on the local data SSD. Native prepared selections are served behind Unsloth. Greedy decoding, thinking explicitly off.One reply per prompt for the fresh 4-bit control; medians of two replies per prompt for each lower-bit target. Counting 1..40, loop-based Python factorial (zero returns 1; negative input raises ValueError), and roughly 90-word source-verification prose ran in that order, with a 192-token output cap and tools disabled. Counting on mixed 2/3-bit varied substantially: 107→226 tok/s, first text 3.68→0.50s. Each speed request used a unique ignored system marker; native counters confirmed zero prefix reuse. The OS page cache was not purged. Tokens/s are native decode rates; first-text times measure the HTTPS request through Unsloth to the first visible answer text, excluding model loading. These are descriptive whole-configuration measurements, not a controlled isolation of quantisation speed.
The mixed 2-bit target reported
streamed_experts: null, selecting fully resident model weights rather than expert swapping. Mixed 2/3-bit had 11,160/12,288 expert slots (90.8% capacity), with all 11,160 observed resident by the end; the 4-bit control had 7,103 slots. DFlash counters confirmed accepted tokens in both lower-bit runs (mixed 2-bit: 438/1,001 across the six speed replies, strongly prompt-dependent). Sampled pressure stayed normal, with no crashes. The largest observed system-swap increase was 0.5 MiB during the mixed 2-bit trial; the other two trials showed no increase. There was already about 3,472 MiB of system swap allocated before the tests, which was not reset. Native Metal allocation is not the total OS footprint, and these short samples do not establish maximum-context safety.In a separate cache check using a synthetic 10.2K-token prefix, mixed 2-bit reused 4,096 tokens on a growing-chat follow-up, then 10,208/10,224 after reloading the model (first visible text 4.44s cold, 2.79s warm, 0.23s after reload). Mixed 2/3-bit also reused 4,096 tokens on its follow-up. The reload timing starts after model loading; this was a worker-reload check, not an OS reboot or a cold-SSD measurement.
Keeping most or all experts resident looks promising for speed on 64 GiB. Happy to provide more detailed performance, memory or caching measurements if useful, and thank you again for the work.
jianc99 commentedon Oct 23, 2026
Ornith-1.5-122B is now supported on main (#539, #560): the unsloth GGUF, and
ornith-ai/Ornith-1.5-122B-MLX-4bitwith--language-onlyuntil its vision shard is published. Thanks @ezoushen for raising this, and @studer for testing!