Supported models

This page summarizes model-family support in the source tree. The runtime source of truth is the code, not this prose page:

detection: src/models/detection.rs
ModelType enum and module exports: src/models/mod.rs
loading policy: src/model_metadata.rs
VLM loading routes: src/loading/vlm*.rs

ModelType spans text and non-VLM language models, VLM variants, a speech-to-text encoder-decoder (Whisper), a text-to-speech model (Kokoro), and the embedding families served through /v1/embeddings. These are architecture/runtime variants, not a guarantee that every checkpoint under a marketing family name is supported.

Text and hybrid model families

Implemented model families include:

Llama-family and Mistral-style dense decoders (the shared llama3 decoder reads rope_scaling and implements three of its schemes: an absent block and "default" keep the plain base^(2i/d) frequencies, "linear" divides positions by factor, and "llama3" builds the NTK-by-parts banded frequency table every Llama 3.1 / 3.2 / 3.3 checkpoint declares, which is then used identically on the graph, batched-decode, paged-decode and tensor-parallel paths. Before #1355 the block was parsed and dropped, and the key holding the scheme name was read under the wrong spelling on top of that, so every Llama 3.x checkpoint rotated its low-frequency bands up to 8x (32x on 3.2-1B/3B) too fast past a few thousand tokens. Short prompts cannot see the difference: the scaled and unscaled tables agree closely at low positions, so a six-token greedy diff passes either way. A scheme this decoder does not implement (yarn, dynamic, longrope) prints one warning naming the model and the scheme and then decodes on the plain table, which is what it did before the block was read at all; that is deliberately not a load error, because the same args carry the text_config of several VLMs and at least one shipping checkpoint declares "dynamic" there. Reading the block in the shared decoder is a deliberate divergence from mlx-vlm for five VLM families, recorded here because feature parity with mlx-vlm is a stated project principle. mlx-vlm's language.py for idefics2, idefics3 / SmolVLM, internvl_chat, llava / llava_next and pixtral builds a plain nn.RoPE(dims, traditional, base) and never calls initialize_rope, so a text_config scaling block is dropped there (llava and pixtral honor linear only); mistral3 and mllama already call initialize_rope, so those two agree either way. mlxcel applies the declared scheme in all of them, because the HuggingFace definition these checkpoints were trained under does: Idefics3Model wraps LlamaModel, and LlamaModel applies rope_scaling. So the divergence is mlx-vlm dropping the block, not mlxcel inventing one. It is observable on one local checkpoint, models/idefics3-8b-llama3-4bit, whose text_config declares "rope_type": "llama3" with factor: 8: a long-context continuation from it changes with this decoder and does not change under mlx-vlm.)
IQuest-Coder V1 (iquestcoder, IQuestCoderForCausalLM: the 7B / 14B / 40B Base, Instruct and Thinking checkpoints are the shared llama3 decoder under a different label, RMSNorm plus GQA with an explicit head_dim, SwiGLU, an untied lm_head, RoPE at base 500000 with no scaling, and no attention or MLP bias. Its config schema adds three keys Llama does not have, and the whole equivalence rests on them being inert, so get_model_type refuses at load rather than ignoring them: a non-null clip_qkv clamps Q, K and V before attention and the shared decoder has no such clamp, and use_sliding_window with a non-null sliding_window windows every layer from max_window_layers on while the shared decoder attends over the full prefix. Every published checkpoint ships clip_qkv: null, use_sliding_window: false, sliding_window: null, max_window_layers: 0, so all of them route. The family ships a SentencePiece tokenizer.model with no tokenizer.json and an explicit "add_prefix_space": false, which mlxcel now honors by clearing add_dummy_prefix in the SentencePiece model it loads (src/tokenizer/spm_proto.rs). Note that this tokenizer.model is a SentencePiece BPE model, so a fast tokenizer converted from it by transformers can split a word differently: the converter reconstructs BPE merges from piece scores and does not always recover the trained order. Compare token ids, not just decoded text, when diffing this family against a reference stack. All three eos_token_id entries, </s> (2), <|im_end|> (75864) and <|endoftext|> (75869), reach the stop set through generation_config.json. Validated against mlx-community/IQuest-Coder-V1-7B-Instruct-8bit. Pipeline parallelism uses the Llama stage family; tensor parallelism is refused, because the TP Llama runtime is gated on the architecture string and this family has not been validated on a multi-rank host.)
IQuest-Coder Loop (iquestloopcoder, IQuestLoopCoderForCausalLM: the 40B-Loop Instruct and Thinking checkpoints. A Llama-shaped dense decoder whose 80 layers are run twice over the same tokens with the same weights (loop_num, only 2 is implemented and anything else is refused at detection). Pass 1 is plain causal attention. In pass 2 each layer's attention output is a per-head sigmoid-gated mix of a global branch, attending the K/V pass 1 produced for that layer, and a local branch, attending pass 2's own K/V through a loop_window_size (64) sliding window; the gate is sigmoid(q2 . gate_w[h] + gate_b[h]) computed from the post-RoPE pass-2 query, with model.gate_projections.{i} the only pass-2-specific weight and the only tensor in the checkpoint that is never quantized. Each layer therefore owns two caches, a dense one for pass 1 and a rotating 64-entry one for pass 2, which is why the family declares SequenceStateLayout::model_owned and keeps its state in a per-SequenceId ModelOwnedSequenceState rather than in the generator's flat Vec<KVCache>. Prefix and prompt caching are excluded for this family: the scheduler's prompt-cache donate and adopt paths both bail on a model-owned backend and record PromptCacheRejectReason::ModelOwnedState, so a request records a structural skip rather than a misleading miss. Snapshot reuse, padded prefill, batched decode and tensor parallelism are likewise off; padded prefill in particular is refused because pad tokens appended to a rotating cache that has already wrapped cannot be trimmed back out. Validated against mlx-community/IQuest-Coder-V1-40B-Loop-Instruct-4bit on CUDA (sm_121) against an independent float32 NumPy oracle written from the checkpoint's own modeling_iquestloopcoder.py: identical argmax on a 38-token and a 218-token prompt, an exact 12-token greedy continuation, and KL(oracle || mlxcel) = 0.0047 at 218 tokens (tests/iquestloopcoder_parity.rs). Note for anyone extending that comparison: it cannot validate the sliding window. Every layer's gate bias is +2.0, so the gate sits at ~0.88 and the local branch contributes ~12%; removing the window entirely moves the 218-token logits by mean 5.0e-02, five times less than mlxcel's own f16 deviation from the f32 oracle, and changes neither the argmax nor the top-10 order. The window is pinned instead by unit tests that isolate the local branch. Two runtime limits worth knowing before diffing this family against another stack. First, decode continuations past 64 tokens will not match transformers running the checkpoint's own modeling_iquestloopcoder.py: that file's cached decode path seeds its local cache with every prompt token and never trims it to loop_window_size, so its window stays as wide as the prompt, and it writes that cache from K/V recomputed after the layer already ran. mlxcel implements the windowed semantics the model was trained under (_forward_loop), so prefill logits agree exactly and decode diverges after the first token on a long prompt. Second, --max-kv-size does not bound the pass-1 cache: the trimmer walks the pool's per-layer caches, which are empty for a model-owned family, so the dense pass-1 cache grows with the sequence. That is shared with every model-owned family, but it bites hardest here because this one's pass-1 cache carries no window of its own. KV cache modes are FP16 only; --kv-cache-mode is refused with a warning rather than silently ignored. The tokenizer is the same SentencePiece tokenizer.model plus added_tokens.json layout as iquestcoder, with no tokenizer.json and add_prefix_space: false; the 19 "special": true added tokens (the ChatML, FIM and repo markers) round-trip in both directions, while the 8 "special": false ones (<think>, </think> and the six tool tags) decode correctly but do not fold back into a single id on encode, which is a property of the shared SentencePiece loader and is observable only on a tool-calling prompt.)
Llama 4 text (supports_snapshot_reuse() since #1335: the chunked/full-attention iGQA cache pattern donates a model-state snapshot for exact-prefix, and truncating while every chunked layer's front is untrimmed, multi-turn prompt-cache reuse; see Exact-prefix snapshots for recurrent state)
Qwen 2 / 2.5 / 3 / 3.5 / 3.6 / 3.8, Qwen MoE, Qwen3 Next. Qwen 3.6 uses the qwen3_5_moe architecture path; Qwen 3.8 uses qwen3_5, so mlxcel arch presents both as public-version aliases of their Qwen 3.5-family entries rather than as duplicate architectures.
Gemma 1 / 2 / 3 / 3n / 4 text variants (gemma2 is the one member of this group with an attention-mask contract worth naming. The generation paths (CLI text prefill, VLM embeddings prefill, chunked prefill) call the model with mask == None and expect the model to build its own causal prefill mask, the way create_attention_mask(h, cache[0], return_array=True) does in the reference. Gemma 2 also sets attn_logit_softcapping, so a null mask lands in the softcap SDPA composite (compiled_softcap_sdpa), which applies no causality of its own; the model therefore routes a maskless prefill through causal_attention carrying that softcap value rather than around it, and an explicit caller mask still takes the masked call. Validation checkpoint: mlx-community/gemma-2-2b-it-4bit, token-exact against the mlx-lm 0.31.3 greedy reference on a 44-token prompt and pinned by tests/causal_prefill_greedy_parity.rs, which also asserts the prefill is causal so the contract cannot silently regress. A short prompt does not exercise this: for the last query position a causal and a bidirectional mask select the same keys, so a six-token prompt answers correctly either way. The same module backs the PaliGemma 2 VLM path in src/loading/vlm_siglip.rs; that path supplies its own additive 4D mask built by prepare_inputs_for_multimodal, so it takes the explicit-mask branch and is unchanged, verified byte-identical on mlx-community/paligemma2-3b-ft-docci-448-6bit. Gemma 3 from 4B up declares rope_scaling: {"rope_type": "linear", "factor": 8.0} in its text_config, and that factor applies to the global-attention layers only: the sliding layers keep rope_local_base_freq unscaled, which is how the reference builds them. mlxcel parsed the block and never read it until #1340, so every layer rotated at scale 1.0 and the global layers saw positions eight times larger than training. The 1B checkpoint declares no block and is unaffected, which is what makes it the control: its output is byte-identical across that change. Note the deliberate asymmetry with the shared Llama decoder described above. A rope_type this path does not implement is a load error on Gemma 3, where the Llama path only warns, because no Gemma 3 checkpoint routes an arbitrary text_config into these args the way a VLM does into the Llama ones, so failing loudly here cannot strand a model that used to load. Gemma 3 also answers supports_snapshot_reuse() since #1335, donating a model-state snapshot of its sliding/full Cache state for exact-prefix, and truncating while every sliding layer is still unwrapped, multi-turn prompt-cache reuse; see Exact-prefix snapshots for recurrent state.)
Phi, Phi-3, Phi-3 Small, PhiMoE (the phi3 decoder also serves Phi-3.5 and the Phi-4 text backbone; checkpoints declaring rope_scaling.type longrope or su carry two frequency tables and pick between them by position, see Phi-3 / Phi-4 LongRoPE position scaling)
Phixtral (phi-msft with num_local_experts, mlabonne: a Mixtral-style sparse MoE on the Phi-2 parallel-residual backbone. There is no phixtral model_type to key on. mlabonne/phixtral-4x2_8 declares phi-msft, the same string dense Phi-2 declares, and upstream mlx-lm reaches its phixtral implementation through a MODEL_REMAPPING rename table rather than the config value, so an arm keyed on the literal "phixtral" could never fire and this tree's existing phi-msft arm pointed at the dense decoder. The arm is therefore discriminated rather than added, on num_local_experts (> 1 is sparse, absent or 1 is dense), which is the same signal upstream's config carries. The checkpoint is neither Llama-named nor named like this tree's dense Phi loader: it ships the original Microsoft layout throughout (transformer.embd.wte, transformer.h.{i}.ln, transformer.h.{i}.mixer.Wqkv, mixer.out_proj, moe.gate, and an output head split into lm_head.ln plus lm_head.linear, with the final norm living inside the head rather than at the top of the transformer), so none of it overlaps phi and it is a separate module. Upstream's own ModelArgs names num_vocab / model_dim / num_heads / num_layers, none of which a phixtral config.json contains (it writes vocab_size / n_embd / n_head / n_layer), so from_dict drops all four and upstream silently runs on its defaults; that is correct only because phixtral-4x2_8 is built on Phi-2, whose dimensions are exactly those defaults, and a differently-sized phixtral would load upstream at the wrong shape. This loader reads the spellings the checkpoint uses and accepts upstream's as aliases. Three behaviours are load-bearing and invisible to a shape check. Attention runs in float32: upstream casts the queries immediately before the SDPA call and casts back immediately after, which promotes the whole score computation, and it is required rather than decorative, since Phi-2 carries large outlier activations and this checkpoint ships float16 whose 65504 ceiling the q @ k^T products reach in the deep layers. Running the scores in f16 produces NaN from layer 30 of 32 on a four-token prompt, with layers 0 through 29 tracking the reference to three decimal places first, so the only symptom is a late and total one. The router softmaxes the k gathered logits, not the full expert row: selecting on the raw logits, gathering those logits, and only then softmaxing normalizes by the sum over the selected k, while softmaxing the row first and gathering afterwards divides by the sum over all num_local_experts and leaves every output finite. And the top-k is taken as argpartition(-gates, kth = k - 1) followed by the FIRST k, upstream's orientation, rather than the equivalent-looking argpartition(gates, kth = n - k) plus the last k, because on this checkpoint the two disagree: every gate row of phixtral-4x2_8 is identical across all four experts in every one of its 32 layers (verified by dequantizing the router), so each token's four logits are exactly tied, the routing weights are always a uniform 1/k, and which experts run is decided entirely by how argpartition breaks the tie. Upstream's form selects experts 0 and 1 there and the mirrored form selects 2 and 3, which is a different output rather than a reordering of the same one and moves every layer's MoE result by ~0.5%. The experts themselves are a two-projection fc2(gelu(fc1(x))) MLP rather than a gated SwiGLU triple, so SwitchGLU does not apply, and they are built with bias=True, so each projection carries a [num_local_experts, out_features] bias plane that the shared SwitchLinear does not implement; the per-expert row is gathered with take along the expert axis and added, which is what upstream's x + expand_dims(bias[indices], -2) does, and a checkpoint missing that plane is rejected at load rather than loaded without it. GELU is the shared erf-based gelu_approx rather than the tanh approximation upstream's nn.GELU(approx="precise") evaluates; the substitution was measured on this checkpoint rather than assumed, moving a layer's MoE output by 0.025% and producing identical generated tokens over a 30-token greedy decode, which is the same conclusion the gpt2 port reached. Loading rejects an architecture scalar that could size an allocation or divide (zero checks precede the divisibility check, since 0.is_multiple_of(0) is true in Rust); a num_experts_per_tok outside 1..=num_local_experts, which would put argpartition(kth = k - 1) out of range; a rotary width that is zero, odd, or wider than the head; a non-positive or non-finite layer_norm_epsilon or rotary base; and any tensor whose real shape disagrees with the config, including an expert stack with fewer planes than num_local_experts (the router emits indices below that count and the gather behind gather_mm / gather_qmm does not range-check a positive index). Validation checkpoint: mlabonne/phixtral-4x2_8 converted locally with mlx_lm convert --hf-path mlabonne/phixtral-4x2_8 -q --q-bits 4 --q-group-size 64 (4.1 GB, 32 layers, 4 experts top-2, 681 tensors), since the upstream repo ships fp16 only. Greedy decode is token-exact against the mlx-lm reference over 30 tokens, and a single-token prefill reproduces its top-5 logits.)
Mixtral and other MoE families
DBRX (dbrx, Databricks: a 132B/36B-active sparse MoE decoder whose deltas from the Mixtral-shaped path are all structural rather than algorithmic. Config geometry is nested under attn_config and ffn_config instead of the flat Llama fields, and both blocks are a full PretrainedConfig dump in published checkpoints, so the config structs declare only the fields that matter and let serde drop the dozens of generation-time keys (top_k, temperature, id2label) that share the block. Attention is one fused Wqkv whose output is clamped to [-clip_qkv, +clip_qkv] before an uneven three-way split at (d_model, d_model + head_dim * kv_n_heads): on dbrx-instruct that is 48 query heads and 8 KV heads of width 128, so the offsets are (6144, 7168) and an even three-way split would produce correctly shaped tensors assembled from the wrong channels. clip_qkv is parsed as an optional float rather than a defaulted one, because upstream writes it as the JSON integer 8 and a config that omits it means "no clipping"; a defaulted 0.0 would clamp every activation to zero and yield a model that loads and emits noise. The block layout is "norm-attn-norm": norm_1 feeds attention, the residual is added, and norm_2 of that residual feeds the FFN whose output is added back to the un-normalized residual, so the tensor the FFN reads and the tensor it adds to are different. Normalization is nn.LayerNorm without bias (not RMSNorm) at MLX's default 1e-5, since DBRX configs carry no epsilon field. Routed experts are named w1 (gate), v1 (up) and w2 (down) under transformer.blocks.{i}.ffn.experts.{e}, which is the shared switch_layers per-expert layout with different leaf names; upstream's sanitize splits a stacked experts.mlp.* tensor, but mlx-community conversions run that split before quantizing, so the loader consumes the already-split per-expert form and never slices a packed quantized tensor (which would be unsound for gather_qmm). Upstream softmaxes over all experts, selects top-k, then divides by the L1 norm of the selected scores (moe_normalize_expert_weights: 1); softmax is monotonic and that composition reduces exactly to a softmax over the top-k logits, which is the shorter route taken here. The layer stack is transformer.blocks with a transformer.wte token table that published conversions leave unquantized even when every projection is 4-bit, and eos_token_id is null in every published config, so the model falls back to the <|endoftext|> id 100257 that lives only in tokenizer_config.json. Like every other text family here it satisfies the maskless-prefill contract by routing a multi-token prefill with no caller mask through causal_attention. Not wired for tensor parallel: the fused Wqkv packs query and KV heads into one tensor, so a row split at any offset a generic plan would pick lands inside the Q block rather than between whole heads. Validation checkpoint: mlx-community/dbrx-instruct-4bit.)
DeepSeek v1 / v2 / v3 / v3.2 (deepseek, deepseek_v2, deepseek_v3, deepseek_v32: sparse Mixture-of-Experts decoders, with Multi-head Latent Attention from v2 onward. The four are separate implementations that share only the generic core layers, so a change to one does not reach the others. One contract does bind all four: the generation paths (CLI text prefill, VLM embeddings prefill, chunked prefill) call the model with mask == None and expect the model to build its own causal prefill mask, the way create_attention_mask(h, cache[0]) does in the reference. deepseek and deepseek_v2 satisfy it by routing a maskless multi-token prefill through causal_attention; deepseek_v3 and deepseek_v32 add create_causal_mask into the additive pe_scores mask their MLA path already carries. Validation checkpoint: mlx-community/DeepSeek-V2-Lite-Chat-4bit-mlx for deepseek_v2, token-exact against the mlx-lm 0.31.3 greedy reference and pinned by tests/deepseek_v2_greedy_parity.rs, which also asserts the prefill is causal so the contract above cannot silently regress. The same deepseek_v2 decoder is exercised from the other side by mlx-community/deepseek-vl2-small-4bit, and deepseek by the two DeepSeek-OCR checkpoints. No DeepSeek V3 or V3.2 checkpoint fits the development machine, so those two families carry code inspection and unit tests rather than a real-checkpoint gate. DeepSeek V4 is NOT a fifth member of this implementation family: it drops MLA and plain residuals entirely and has its own entry below.)
DeepSeek V4 (deepseek_v4, DeepSeek-V4-Flash: a genuinely new architecture, not a V3 variant, ported from the in-tree references/mlx-vlm reference after an earlier V3-wrapper attempt (PR #592) proved unable to load a real checkpoint. The state carried between blocks is rank-4 [B, L, hc_mult, D]: HyperConnections collapse it per sublayer through learned gates whose mixing matrix is softmaxed and then Sinkhorn-normalised for hc_sinkhorn_iters (20) alternating column/row rounds in float32, and re-expand the sublayer output into the widened residual; both the iteration count and the normalisation order change the output while keeping it finite and plausible. Attention runs one shared 512-wide KV head (num_key_value_heads == 1) across 64 query heads inside a 128-token rotating window, with weightless Q RMSNorm after the head reshape, learned per-head sinks in every softmax, an inverse RoPE applied to the attention OUTPUT before the grouped wo_a MultiLinear projection, and full-head mx.fast.rope-style rotation whose frequency table is inf-padded so only the trailing qk_rope_head_dim lanes rotate. Per layer, compress_ratios selects local (0), compressed (128), or sparse-compressed (4) attention: a Compressor pools each ratio-sized window with a learned ape and a window softmax, RMS-norms, and Yarn-ropes the pooled row at compress_rope_theta with the frequency table divided by the ratio; ratio 4 runs an overlap mode (out_dim = 2 * head_dim, first feature half shifted one window back with a zero/-inf prefix) that is NOT interchangeable with the simple mode, and the shift applies within each ready batch, so decode- or chunk-boundary-completed windows legitimately see the prefix instead of their predecessor. Sparse layers add a HiSA Indexer that owns its own compressor and returns the top-index_topk (512) pooled rows through a coarse block-mean stage plus fine rescoring (index_block 64 and index_keep 16 are dataclass defaults absent from the real config.json), feeding a split softmax over the local and gathered pooled KV that shares one log-normalizer with the sinks. The MoE routes the first num_hash_layers (3) layers by a tid2eid token-id lookup (indices from the raw input ids, weights from the logits, which is why input_ids threads through every block), scores with sqrt(softplus(logits)) in float32, selects non-hash layers on bias-corrected scores while weighting from the unbiased ones (the same silent-misweighting contract as bailing_moe / afmoe / klear), and clamps both the routed and shared expert SwiGLU at swiglu_limit (10.0); every layer is MoE. Heterogeneous per-layer state (rotating window plus one or two pooling caches) rides in model-owned sequence state like Gemma 3 / AFMoE. The checkpoint declares per-module quantization overrides: the routed experts are mxfp4 at group size 32 under an affine/64 top level, and the loader reads the override table rather than trusting the defaults. mtp.* tensors (num_nextn_predict_layers) are dropped at load; MTP drafting, the reference's fused HC Metal kernel, and sharding are follow-ups. Validation checkpoint: mlx-community/DeepSeek-V4-Flash-4bit (43 layers, ~151 GB on disk), load-parity gated by a strict weight-coverage check inside from_weights and generation-gated by three #[ignore]d real-checkpoint tests in tests/deepseek_v4_real_model.rs: a 24-token greedy decode that must name Paris for a capital-of-France prompt, since a fluent-but-wrong answer there is the signature of a silently misweighted component (Sinkhorn order, the overlap compressor, a selection-vs-weighting gate contract) rather than a load failure; a 48-token decode built to cross a ratio-4 and a ratio-128 pooling-window boundary; and a >2100-token prompt that pushes a ratio-4 layer's pooled count past index_topk (512) into the sparse split-softmax and the batched HiSA selection path, which still has to retrieve the right fact (the Pacific Ocean) from early in the context.)
dots.llm1 (dots1, rednote: a DeepSeek-V3-style Mixture-of-Experts without MLA. Standard multi-head attention with per-head Q/K RMSNorm, a dense first layer (first_k_dense_replace), then sigmoid-routed experts that select on gate.weight logits plus an e_score_correction_bias, with a single always-on shared expert. Validated against mlx-community/dots.llm1.inst-mixed-4-6bit, a mixed 4/6-bit export whose v_proj and down_proj tensors are 6-bit while the rest are 4-bit; the unified loaders detect the per-tensor bit width from shape.)
Ling / Bailing MoE (bailing_moe, Ant Group: a DeepSeek-shaped sparse decoder, RMSNorm before attention and before the FFN, grouped-query attention with RoPE, num_experts routed SwiGLU experts selected per token, and one always-on shared expert, so the expert machinery is the shared switch_layers path unchanged. What is not shared with DeepSeek is where the port's work went. The checkpoint uses GPT-NeoX naming rather than Llama naming: the token table is model.word_embeddings, the output projection is attention.dense, and Q, K and V arrive fused in one attention.query_key_value matrix that on inclusionAI/Ling-lite-1.5 is [3072, 2048], which is (16 + 2 * 4) * 128 for 16 query heads and 4 KV heads of width 128; the split offsets are therefore (2048, 2560), the uneven GQA form rather than an even three-way split, and MLX's slice clamps an out-of-range stop instead of throwing, so the even split would produce tensors of exactly the right shape assembled from the wrong channels. The router is renamed by upstream mlx-lm's sanitize from mlp.gate.weight to mlp.gate.gate_proj.weight, which collides with the gate_proj every routed expert also has; the loader resolves the router from two fully anchored keys instead of matching a suffix, so a raw inclusionAI export and an mlx-lm conversion both load and neither can read an expert's SwiGLU gate as the router. The raw checkpoint stores each routed expert separately at mlp.experts.{e}.{gate,up,down}_proj, 64 experts times 3 projections times 28 layers on Ling-lite; the loader joins those individually-keyed tensors into SwitchLinear's stacked [num_experts, out, in] layout at load, checking every expert index rather than only index 0 so a checkpoint missing a late expert cannot register a short stack the router can index past, and an mlx-lm conversion that already ships the pre-stacked mlp.switch_mlp.{proj} form loads unchanged on the same path. Three routing details are load-bearing and invisible downstream: the correction bias is added to a copy of the scores used for SELECTION only, and the returned weights are gathered from the unbiased scores (gathering from the biased copy leaves the output finite and plausible while misweighting every routed contribution); norm_topk_prob is applied only when num_experts_per_tok > 1 and divides by sum + 1e-20 rather than a bare sum; and score_function defaults to "softmax", not to DeepSeek-V3's sigmoid, so reusing the DeepSeek gate as-is would change the routing distribution on every token. moe_router_enable_routed_scaling is dead code upstream (BailingMoeGate stores it and never reads it, and group_expert_select ends with an unconditional multiply), and this port mirrors upstream so a checkpoint decodes here as it does under the reference; the loader prints a diagnostic naming both fields for the one combination where the two readings diverge, a non-unit routed_scaling_factor with the flag false. The shared expert is ONE wide MLP of (moe_shared_expert_intermediate_size or moe_intermediate_size) * num_shared_experts, which is 1408 * 2 = 2816 on Ling-lite, added to the routed mixture at a fixed weight of 1.0 and never packed into the switch tensors; there is no per-shared-expert axis anywhere in the checkpoint. Grouped top-k routing (n_group groups scored by the sum of their top two experts, topk_group kept) and the router expert bias are implemented and unit-tested, but no published Bailing checkpoint reaches either: Ling-lite declares neither field and both default to off, so a passing real-checkpoint run says nothing about them. norm_head is live upstream despite appearing vestigial, and is implemented as an axis-0 L2 normalization of lm_head.weight in float32 with a 1e-7 epsilon cast back to the weight's own dtype; norm_softmax is declared by upstream's ModelArgs and never read, does not appear in the vendored modeling_bailing_moe.py, and is not a named parameter of the vendored config, so a config setting it true is rejected at load rather than parsed and ignored. Loading also rejects an architecture scalar that could size an allocation, divide, or truncate through an as i32 cast (zero checks precede divisibility checks, since 0.is_multiple_of(0) is true in Rust); routing parameters that would put argpartition(kth = n_group - topk_group - 1) out of range or reshape the score row into unequal groups, which upstream checks neither of; a rotary width that is zero, odd or wider than the head, and a non-positive or non-finite rope_theta or rms_norm_eps; a rope_scaling block other than an absent, empty or "default" one, since upstream threads it into initialize_rope while this loader always builds the plain rotation; a quantization block whose bits falls outside 1..=32 or whose group_size is non-positive; the output head's input width against hidden_size, the one axis nothing else in the loader checked, with its row count checked exactly against upstream's own strict nn.Linear(hidden_size, vocab_size) load and the check skipped only when tie_word_embeddings ships no separate lm_head tensor; a quantized token table's own input width, reconstructed the same way, since the shared validate_embedding_table checks the row count unconditionally but the width only on the unquantized path; and any tensor whose real shape disagrees with the config, on both axes, per expert index rather than only the first (a checkpoint missing a late expert would otherwise register a short stack the router can index past), and on the quantized path, where packing compresses the input axis only so the input width is recovered from the scales the way MLX recovers it, scales.shape(-1) * group_size. Each of those is a load-time rejection because an MLX C++ exception crossing the cxx bridge is an uncatchable std::terminate at the first forward pass rather than a Rust error. Validation checkpoint: inclusionAI/Ling-lite-1.5 (16.8B total / 2.75B activated, raw bf16, 5603 tensors, no mlx-community conversion needed). A token-exact comparison against the mlx-lm reference needs --no-chat-template on both sides, because this checkpoint does ship a chat template embedded in tokenizer_config.json while the reference harness feeds the raw prompt.)
Ling / Ring linear-attention MoE (bailing_moe_linear, Ant Group: the bailing_moe sparse block interleaved with gated-linear-attention layers. The FFN half is the dense family's unchanged, down to the mlp.gate.gate_proj router rename, the selection-only expert bias and the single wide shared MLP, so this port reuses those types rather than restating them; every delta is in the attention stack. The linear attention is GLA, not gated-delta: h_t = h_{t-1} * exp(g) + k_t^T v_t with y_t = (q_t h_t) * scale and NO delta term, so reusing gated_delta.rs (which computes delta = (v - k h) * beta and adds k^T delta) yields a model that loads, runs and emits fluent text from the wrong recurrence. The decay g is read from no tensor at all: it is the ALiBi slope schedule for num_attention_heads, negated and scaled by 1 - max(0, layer_idx - 1) / max(1, num_hidden_layers - 1) + 1e-5, so every linear layer decays at its own constant per-head rate and nothing in the checkpoint disagrees if it is computed wrong (the non-power-of-two head-count fallback is implemented and unit-tested but unreachable on the published checkpoint's 16 heads). Because exp(g) does not depend on t, the recurrence has a closed form, and the port carries both evaluations: gla_sequential is upstream's per-token loop transcribed step for step, and gla_chunked is the closed form over 64-token chunks. Chunked is the default as of #1040; set MLXCEL_BAILING_LINEAR_CHUNKED_PREFILL=0 for upstream's arithmetic, which is what a mlx-lm reference diff needs. Chunking measures 2.6x to 4x faster on prefill (0.07s against 0.29s at 101 tokens, 1.07s against 2.81s at 2048 on an M1 Ultra), since the intra-chunk sum lands in a matmul accumulator instead of a bf16 running state. It shipped opt-in first because this stack amplifies a reassociation hard: reordering the sum moves layer 0's mean absolute activation by 0.04%, layer 3 then multiplies its input magnitude by roughly 23x, and the 256-expert top-8 router converts a sub-ulp score difference into a different selected expert, so the two paths decode different continuations from the same checkpoint. #1040 measured which of the two differing answers is better rather than assuming they were equivalent, and teacher-forced perplexity on a WikiText-2 excerpt favours chunked at every window length: 115.83 against 114.09 at 128 tokens, 39.39 against 36.39 at 1024, 155.47 against 151.42 over 8192 in 256-token windows, and 85.96 against 74.14 over 16384 in 512-token windows. The advantage grows with the window within a fixed scoring shape, which is what a compounding bf16 accumulation error predicts, so the faster path is also the more accurate one and the cost of the promotion is only token-for-token comparability with mlx-lm. The same sensitivity means token-exactness against mlx-lm is not a meaningful gate for this family in general: the reference disagrees with ITSELF between its cached and its full-recompute paths, flipping a 0.0625-wide top-2 tie (exactly one bf16 ulp at that magnitude) six tokens into a 12-token prompt. What was validated instead is the layer-by-layer numerical trace, which is stronger: on a 101-token prompt the default path reproduces the reference's mean absolute activation at layer 0 to six significant figures (0.033069 against 0.033069) and at layer 1 to five, confirming the recurrence, the RoPE offset threading, the QK-norm ordering, GroupRMSNorm, the sigmoid output gate and both residuals at once; a single-token prefill reproduces the reference's top-5 logits exactly; and greedy decode matches the reference's full-recompute path for all 40 tokens of a low-entropy counting prompt. GroupRMSNorm is not RMSNorm: it splits the num_heads * head_dim axis into group_norm_size groups (4 on Ring-mini), RMS-normalizes each group WITHOUT a weight, flattens, and only then multiplies by the full-width weight, so substituting a plain RMSNorm changes every linear layer's output while leaving it finite. The linear layers are MHA and the full-attention layers are GQA, which gives one checkpoint two fused-QKV widths and two head widths: on mlx-community/Ring-mini-linear-2.0-4bit a linear layer's attention.query_key_value is [6144, 2048] (3 * 16 * 128) and a global layer's is [3072, 2048] ((16 + 2 * 4) * 128), and upstream's LinearAttention derives hidden_size // num_attention_heads while ignoring any explicit head_dim, so weight validation checks each layer against its own width rather than one shared one; MLX's slice clamps an out-of-range stop instead of throwing, so a single shared width would let a mislabeled layer silently read the wrong channels. The hybrid schedule is (layer_idx + 1) % layer_group_size == 0 plus a tail clause that makes every leftover layer global when the stack does not divide evenly (layers 4, 9, 14 and 19 of 20 on Ring-mini; the tail clause never fires there and is unit-tested separately), and the RoPE offset every linear layer uses is read from a global layer's KV cache, since the recurrence has no position of its own. Upstream hardcodes that lookup as layer_group_size - 1 and would index out of range for a stack shorter than one group; this loader resolves it by scan. The linear layers hold a recurrent state and the global ones a KV cache, which does not fit the trait's homogeneous &mut [KVCache], so the model owns its heterogeneous per-sequence state the way qwen3_next does, with snapshot and restore wired for exact-prefix reuse and batched decode declined. norm_softmax and use_rmsnorm are both declared by upstream's ModelArgs and never read (every normalizer in the file is an RMSNorm or a GroupRMSNorm unconditionally), so a config that sets norm_softmax: true or use_rmsnorm: false is rejected at load rather than parsed and ignored. Unlike the dense family, this one reaches grouped top-k routing on a real checkpoint (n_group 8, topk_group 4, 256 experts), so the guards against an out-of-range argpartition(kth = n_group - topk_group - 1) and an expert count that does not divide into equal groups are live rather than theoretical. The config also carries per-tensor quantization overrides (the router mlp.gate.gate_proj at 8 bits while the rest of the checkpoint is 4-bit); those keys are deliberately not parsed, because UnifiedLinear, SwitchLinear and UnifiedEmbedding each reconcile bits and group size from the tensor shapes they load. Validation checkpoint: mlx-community/Ring-mini-linear-2.0-4bit (16B total / 1.4B activated, 8.6 GB, 20 layers, 256 routed experts plus one shared, 698 tensors).)
Arcee AFMoE / Trinity (afmoe, Arcee: a hybrid sliding/full attention MoE decoder. The issue that requested it described the family as "structurally a qwen3_moe / deepseek MoE derivative with QK-norm", which is true of the expert machinery and wrong about the model class: five features change how every layer is built and four are invisible to a shape check. layer_types interleaves sliding_attention and full_attention (three sliding then one global, 56 times over on Trinity-Nano-Preview), so sliding layers need a RotatingKVCache and global layers a KVCache and the model owns its heterogeneous per-sequence state the way gemma3 does, reusing that module's Cache enum rather than restating it. The schedule comes from the LIST, not a modulus: global_attn_every_n_layers is declared beside it and upstream never reads it, so a port that trusts the modulus agrees with Trinity by coincidence and diverges on any checkpoint whose list is irregular. Full-attention layers are NoPE: upstream builds self.rope only when is_local_attention, so global layers apply no positional encoding at all, and rotating them anyway loads every tensor and is invisible at position 0 where the rotation is the identity, which is why it is pinned by a reference-free unit test that runs a global block at two sequence offsets and asserts the output does not move (and that a sliding block's does). Attention carries a sigmoid output gate applied BEFORE o_proj, so every block has an extra self_attn.gate_proj weight qwen3_moe has no counterpart for, and the name collides with both the SwiGLU gate_proj and the MoE router at mlp.router.gate. There are FOUR norms per layer in a sandwich (input_layernorm, post_attention_layernorm, pre_mlp_layernorm, post_mlp_layernorm), applied so the two "post" norms normalize each branch's OUTPUT before it joins the residual rather than normalizing the residual on the way in; a two-norm pre-norm block loads every tensor the checkpoint ships except two per layer and still generates. When mup_enabled the embeddings are multiplied by sqrt(hidden_size) before the stack, a config flag rather than a tensor, so dropping it shrinks every hidden state 32x on Trinity-Nano and still produces finite logits. head_dim is read rather than derived: Trinity-Nano declares 128 with hidden 1024 and 8 heads so the derived value coincides, which is exactly the coincidence that hides a derived implementation. The first num_dense_layers (2) use a plain SwiGLU MLP. Routing is sigmoid-scored with the expert_bias added to a SELECTION-only copy while the returned weights are gathered from the unbiased scores (the same silent-misweighting contract bailing_moe documents), then route_norm renormalization and a route_scale multiply (2.826); the routing weights stay float32 through the combine because AfmoeMoE.__call__ never casts them back, so this family deliberately does not use the shared moe_weighted_sum helper, which casts them down first. Grouped routing exists but n_group is 1 on every published checkpoint, so it is unit-tested rather than exercised. Validation is by numerical trace, not token-exactness, and that is not a shortcut. Trinity-Nano-Preview amplifies a perturbation roughly twofold per layer across its 56 layers: rounding the routing weights to bf16, the smallest change an equally-valid implementation could make, changes the REFERENCE's own top-5 logits from [64, 68, 3049, 88, 89] to [1265, 9370, 11041, 6569, 8650] with no token in common, so no two implementations that differ by a single rounding can agree on output. What was verified instead is stronger per unit of evidence: on both a 1-token and a 6-token prompt, layers 0 and 1 (the dense prefix, which has no MoE) reproduce the reference's mean absolute activation EXACTLY to six significant figures at every stage (ln, attn, mlp, and the layer output), which confirms the muP scale, the embeddings, the full sliding-attention block including RoPE at nonzero positions and the sigmoid gate, the sliding-window prefill mask, all four norms, both residuals and the dense SwiGLU MLP on real weights; and at layer 2, the first MoE layer, the routing indices and weights match the reference exactly ([98, 18, 63, 17, 64, 99, 34, 81], weights summing to route_scale). Loading rejects an architecture scalar that could size an allocation or divide, a layer_types list shorter than the stack (which would silently make the uncovered tail global) or containing an undefined kind, a zero sliding_window when a layer slides, an odd head_dim, a num_experts_per_tok outside 1..=num_experts, the grouped-routing parameters that would put argpartition out of range, a non-default rope_scaling block, and any tensor whose real shape disagrees with the config including an expert stack with fewer planes than num_experts. Validation checkpoint: mlx-community/Trinity-Nano-Preview-4bit (3.2 GB, 56 layers, 128 experts top-8 plus one shared, 2389 tensors). Note this is a preview checkpoint of limited quality: it emits repetitive degenerate text under the mlx-lm reference as well as here, so fluency is not a usable signal for it. The family also answers supports_snapshot_reuse() since #1335, reusing gemma3's shared Cache serializer to donate a model-state snapshot for exact-prefix, and truncating while every sliding layer is still unwrapped, multi-turn prompt-cache reuse; see Exact-prefix snapshots for recurrent state.)
Klear (Klear, Kuaishou: a Qwen3-shaped sparse MoE decoder whose shared expert is BLENDED with the routed mixture rather than added. The backbone is unremarkable and is what the issue described: GQA with per-head QK-RMSNorm, RMSNorm layer norms, RoPE over the full head width, optional attention_bias, routed top-k experts over the shared switch_layers path plus a shared expert. Three things are not. First, the model_type is capital-K "Klear", not "klear"; upstream mlx-lm has to ship Klear.py and a byte-identical klear.py to cover both spellings, and this tree needs only the lowercase detection arm because get_model_type lowercases model_type before matching, which is asserted directly through the real detection entry point rather than through a helper. Second, and the reason a qwen3_moe-shaped port would be silently wrong, KlearSparseMoeBlock does not add the shared expert: it learns a per-token 2-way softmax over an mlp.coefficient head (a [2, hidden] projection WITH a [2] bias, since upstream's nn.Linear(hidden, 2) defaults to bias=True) and mixes the branches as y = y_experts * coef[..., :1] + shared * coef[..., 1:]. A plain add misweights every token's output while leaving it finite and the text fluent, so the blend is pinned by a unit test that zeroes the routed branch and injects a known constant into the shared one, making the blended and added readings numerically distinguishable, and a second test that varies the coefficient logits to confirm the weights are read from the checkpoint and that index 1 (not 0) weights the shared branch. Third, routing activates with SIGMOID rather than softmax, and the expert_bias is added to a copy used for SELECTION only while the returned scores are gathered from the unbiased weights, the same silent-misweighting contract bailing_moe documents; norm_topk_prob then renormalizes and, unlike the neighbouring DeepSeek-derived families, nothing scales the result. That last point deserves naming: the published checkpoint declares routed_scaling_factor: 2.5, which looks exactly like the knob those families apply, and upstream's ModelArgs does not declare the field at all, so it is dropped on parse and KlearSparseMoeBlock scales nothing. This port mirrors upstream rather than guessing, and prints a diagnostic naming the field at load so the choice is visible instead of invisible; applying it would multiply every routed contribution by 2.5 against the reference. The sparse schedule is layer_idx not in mlp_only_layers and num_experts > 0 and (layer_idx + 1) % decoder_sparse_step == 0, which on the published checkpoint (decoder_sparse_step: 1, empty mlp_only_layers) makes every one of the 32 layers sparse; a zero step is rejected because upstream would divide by it, and an mlp_only_layers entry past the end of the stack is rejected too. Loading also rejects an architecture scalar that could size an allocation or divide (zero checks precede the divisibility check, since 0.is_multiple_of(0) is true in Rust), a num_experts_per_tok outside 1..=num_experts (which would put argpartition(kth = k - 1) out of range), an odd head width, a non-positive or non-finite rope_theta or rms_norm_eps, a rope_scaling block other than an absent, empty or "default" one (upstream does not declare the field and so drops a scaled block silently; rejecting is the safer reading), and any tensor whose real shape disagrees with the config, including an expert stack with fewer planes than num_experts and a missing mlp.coefficient. Validation checkpoint: Kwai-Klear/Klear-46B-A2.5B-Instruct converted locally to 4-bit (24 GB, 32 layers, 256 experts top-8 plus one shared, 1351 tensors), since mlx-community/Klear-46B-A2.5B-Instruct-4bit is an empty repo and only the 3-bit conversion has weights. A 5-token prefill reproduces the mlx-lm reference's top-5 logits EXACTLY, values included, and greedy decode is token-exact over 20 tokens. One caveat worth knowing: that exactness holds with MLXCEL_FUSED_MOE=0. The default-on fused MoE decode kernel (issue #268) engages only at single-token decode and is not byte-identical here, so with it enabled the model picks a different but equally coherent continuation from a low-confidence position; the 5-token prefill, where the kernel does not engage, is exact either way.)
Mistral 4 (mistral4, Mistral Small 4: a DeepSeek-V3-style Multi-Latent Attention decoder with compressed query and KV LoRA projections (q_lora_rank 1024, kv_lora_rank 256), split rope/nope query-key head dims, and a separate value head dim, paired with a softmax-routed Mixture-of-Experts (128 routed experts plus one always-on shared expert, 4 active per token, norm_topk_prob) and Llama-4 position-dependent attention scaling. The only public checkpoint is the Mistral Small 4 119B vision model, whose text_config.model_type is mistral4; mlxcel detects it as a Mistral 3 VLM (Pixtral vision tower) on the Mistral4 text backbone. Validated against mlx-community/Mistral-Small-4-119B-2603-4bit for both text and image-plus-text.)
Cohere / Cohere2
Command MoE (cohere2_moe, Cohere: the Cohere2 parallel-residual backbone (interleaved sliding/global attention, conditional RoPE, logit scaling, tied embeddings) with the dense FFN replaced by a sparse Mixture-of-Experts. The router scores on mlp.gate logits activated in f32 by sigmoid (default) or softmax, takes the top-k, and gathers the activated values at the selected experts (not a fresh softmax over the k logits), with optional norm_topk_prob renormalization clamped by 1e-12. Always-on shared experts (moe_num_shared_experts) run a dense SwiGLU on the same input and combine by average (default (y + y_s) / 2) or sum. An optional dense-FFN prefix (first_k_dense_replace, width prefix_dense_intermediate_size) and an optional RMSNorm mode (rms_norm_eps, else LayerNorm) round out the deltas versus dense cohere2. The full runtime (detection, metadata, loaded-model dispatch, CLI and server generation) is wired; real-checkpoint validation is pending a public cohere2_moe checkpoint.)
MiniMax-M3 (minimax_m3, MiniMax: a hybrid dense/MoE decoder that follows MiniMax-M2 (minimax) but swaps in Gemma-style RMSNorm (weight+1) everywhere including per-head Q/K norm, clamp-SwiGLU (swigluoai) experts and dense MLPs, partial RoPE (rotary_dim of head_dim), and a sigmoid router with a selection-only routing bias plus one shared expert. The per-layer plan comes from moe_layer_freq (leading dense layers, then MoE); routed experts load the Mixtral-style block_sparse_moe.experts.{i}.w1/w3/w2 checkpoint naming (auto-falling back to gate_proj/up_proj/down_proj), and the shared expert is always a separate block_sparse_moe.shared_experts MLP added to the routed mixture at a fixed weight of 1.0, never packed into the switch tensors. The sparse layers (sparse_attention_config.sparse_attention_freq) add a block-sparse "MSA" indexer with an MQA-style index: per-head index_q_proj query heads score against a single shared index_k_proj key head, each normed (Gemma norm) and RoPE'd, top-sparse_topk_blocks selection over sparse_block_size key blocks scored by their max token score, expanded to an additive block mask on top of the causal mask, with the shared index key cached alongside the regular K (requires sparse_index_dim == head_dim, else the indexer falls back to dense). When the selected window still covers at least half the live cache, attention stays dense; when sparse_topk_blocks covers every block, the block mask is all zeros and attention is bit-identical to dense; a zero sparse_block_size or sparse_topk_blocks is rejected at load time instead of dividing by zero or mis-indexing the block partition. Only the text decoder is implemented; the VL wrapper and MTP head are out of scope (MTP tensors are skipped on load). The only public checkpoint is the 427B MiniMaxAI/MiniMax-M3 VL repo, which exceeds the development machine's memory, so validation is synthetic reduced-config unit tests plus a real-text_config parse test; real-checkpoint load/generate carries over to a follow-up runtime validation.)
InternLM 2 / 3 (both families share one dynamic-NTK rotary schedule, src/models/dynamic_ntk_rope.rs: positions are never scaled, and the rotary base is recomputed per forward from the live sequence length, departing from rope_theta only once the sequence passes max_position_embeddings. That is what the checkpoints' own remote code implements, via transformers' _compute_dynamic_ntk_parameters; a linear block is the mirror image, dividing the position by factor and leaving the base alone. Like every other text family here they satisfy the maskless-prefill contract by routing a multi-token prefill with no caller mask through causal_attention. Validation checkpoint: mlx-community/internlm3-8b-instruct-4bit, token-exact on a 56-token prompt against a corrected mlx-lm 0.31.3 greedy reference, and pinned by tests/causal_prefill_greedy_parity.rs. The correction matters and the qualifier is not decorative: stock mlx-lm 0.31.3 computes rope_scale = 1 / factor if rope_type == "linear" else 2.0 for both InternLM families, so on a dynamic checkpoint it rotates every token at twice its true position, and mlxcel matched it id for id until #1324 because the same expression had been ported verbatim. The reference this entry now claims parity against is that same mlx-lm with only its per-layer rope module replaced by the schedule above; ids from mlxcel's own output were deliberately not used. That parity holds on identical prompt ids, which is the only comparison that isolates the model: this checkpoint ships a tokenizer.json whose byte-fallback splitting disagrees with its own tokenizer.model, and mlxcel prefers tokenizer.json while the reference harness loads the repo's InternLM3Tokenizer under trust_remote_code. Both tokenizations round-trip to the same text, so an end-to-end CLI comparison against mlx-lm compares two different prompt id sequences and diverges for reasons that have nothing to do with the decoder. InternLM2's own checkpoint, mlx-community/internlm2_5-7b-chat-4bit, is held to a weaker standard, and issue #1320 records exactly which one: a 34021-token prompt enters the dynamic branch and rescales to base_eff=1077737.0, which is the closed form above evaluated at that length (f64 gives 1077736.99 there, so the f32 the code computes agrees with the exact value to every digit the log prints; the length is what has to be quoted carefully, since the base moves about 62 per token at that point and 34020 gives 1077674.91). What that does not establish is the rotation itself. The accompanying byte-identity runs, an 89-token prompt and the 34021-token one, compare this tree against the same tree with only the schedule's debug diagnostic removed, so they are a control showing the diagnostic is inert, not a parity result. No corrected long-context reference exists for this family the way it does for InternLM3 above, so the long run's standing claim is that the branch is entered and computes the documented base, and no more.)
GLM 4, GLM MoE, GLM MoE Lite (glm4_moe_lite, GLM-4.7-Flash; its next-token-prediction layer is served as a separate MTP drafter, see the speculative-decoding table), GLM MoE DSA
ERNIE 4.5 and ERNIE 4.5 MoE
Hunyuan dense and MoE variants (hunyuan_v1_dense and hunyuan_moe, Tencent: separate implementations sharing only the generic core layers, both with per-head Q/K RMSNorm applied after the rotation and a DynamicNTK-alpha rope base folded in once at load (rope_theta * alpha^(d/(d-2))) rather than recomputed per forward; the MoE adds softmax routing over num_experts with per-layer moe_topk, an optional shared MLP gated by use_mixed_mlp_moe, and optional cross-layer attention (use_cla) where a layer reuses an earlier layer's K/V but still runs its own SDPA, which is why the mask policy has to live in the attention leaf rather than in the CLA plumbing above it. Both satisfy the maskless-prefill contract by routing a multi-token prefill with no caller mask through causal_attention. Validation checkpoint for the dense family: tencent/Hunyuan-1.8B-Instruct as a 4-bit MLX conversion, token-exact against the mlx-lm 0.31.3 greedy reference on a 36-token chat-templated prompt and pinned by tests/causal_prefill_greedy_parity.rs. That checkpoint degenerates on raw text completion under the reference itself, so a coherent continuation needs the chat template; the parity gate is on prompt ids either way. The MoE family has no runnable reference: the only local checkpoint is mlx-community/Hunyuan-A13B-Instruct-4bit at 42 GB and its bf16 sibling does not fit the development machine, so it carries the reference-free causality gate only, opt-in behind MLXCEL_TEST_HUNYUAN_MOE=1 so a 42 GB load stays off the default test run.)
IBM Granite dense (granite)
IBM Granite 4.x hybrid (granitemoehybrid: interleaves Mamba2 SSM and GQA attention layers by layer_types, applies the four Granite scalar multipliers (embedding, attention, residual, logits), and defaults to NoPE attention. The dense-MLP mode is validated against mlx-community/granite-4.0-h-350m-4bit; the MoE mode (block_sparse_moe + shared_mlp) is implemented but awaits a public MLX checkpoint to validate. The non-hybrid granitemoe variant is not yet ported.)
BitNet b1.58 (bitnet, Microsoft: a Llama-style transformer whose every projection is a BitLinear with 1.58-bit ternary weights ({-1, 0, +1}) packed 4-per-uint8 and scaled by a single per-tensor weight_scale. A custom Metal kernel (bitlinear_matmul) multiplies directly on the packed bytes, so the unpacked weights never materialize. Two extra sub-norms (attn_sub_norm before o_proj, ffn_sub_norm inside the MLP) and a squared-ReLU MLP (relu2(gate) * up). Runs in native bf16 (its squared-ReLU overflows f16), bypassing the Apple-Silicon f16 conversion. Validated against mlx-community/bitnet-b1.58-2B-4T and its -4bit variant, which additionally affine-quantizes the embedding/lm_head to 4-bit (the BitLinear weights stay ternary); keeping the whole model bf16 also keeps that 4-bit dequant dtype-consistent.)
ExaOne / ExaOne 4 / ExaOne MoE / Solar Open
OLMo / OLMo2 / OLMo3 / OLMoE
OpenELM (openelm, Apple: a dense decoder whose defining feature is layer-wise scaling, so almost nothing about a block's width is a single global number. num_query_heads, num_kv_heads and ffn_multipliers are per-layer lists read by layer id. On OpenELM-1_1B-Instruct the query head count climbs 16 to 20 to 24 to 28 to 32 across the 28 layers while the KV head count climbs 4 to 8, so the GQA grouping, the fused qkv_proj output width and the out_proj input width all move layer to layer; a loader that reads head counts once and reuses them builds layer 0's geometry for the whole stack and fails on the real checkpoint. FFN width is not in the config at all: each layer computes make_divisible(ffn_multipliers[i] * model_dim, ffn_dim_divisor), a rounding helper carried over from the original TensorFlow MobileNet source, whose two load-bearing details are that the floor is divisor itself (upstream's min_value defaults to it) and that a rounded value falling below 0.9 * v gets one more divisor added back, so on the 1.1B checkpoint the widths run 1024, 1280, ..., 4608, ..., 8192. The rest is familiar: QK-RMSNorm over the head dimension (the qwen3 pattern, applied before the transpose since RMSNorm normalizes the last axis either way), GQA, SwiGLU through a single fused proj_1 split in half with the first half as the gate, and share_input_output_layers as the OpenELM spelling of tied embeddings, so published checkpoints ship no lm_head. The fused QKV is a plain channel split at (n_heads * head_dim, (n_heads + n_kv_heads) * head_dim), not the head-major interleave of the GPT-NeoX family. Loading rejects a per-layer list shorter than the declared stack, a head ratio that is not a whole GQA grouping, and a non-positive FFN multiplier, because an MLX C++ exception crossing the cxx bridge is an uncatchable std::terminate at the first forward pass rather than a Rust error. Like every other text family here it satisfies the maskless-prefill contract by routing a multi-token prefill with no caller mask through causal_attention. Not wired for tensor parallel: a generic shard plan assumes one head count for the whole stack, which layer-wise scaling contradicts. Validation checkpoint: mlx-community/OpenELM-1_1B-Instruct-4bit.)
StarCoder2, StableLM, SmolLM3, Baichuan, MiniCPM, MiniCPM3, MiniMax, Ministral3, Nemotron, Nemotron-NAS, Step 3.5, MiMo
Mellum / Mellum 2 (mellum, JetBrains code model: sliding/full hybrid attention driven by layer_types, with QK-RMSNorm and a sparse softmax-routed MoE (norm_topk_prob) in every layer. Full-attention layers use YaRN-scaled RoPE plus a standard KVCache; sliding-attention layers use default RoPE plus a RotatingKVCache windowed to sliding_window. Supports tied and untied LM heads. Validated against JetBrains/Mellum2-12B-A2.5B-Base.)
Laguna XS 2.1 / XS.2 / S 2.1 (laguna, Poolside code models: a pre-norm decoder with QK-RMSNorm, a hybrid of full_attention layers (YaRN over the first partial_rotary_factor share of each head, explicit attention_factor, dense KVCache) and sliding_attention layers (plain RoPE at base 10000, RotatingKVCache windowed to sliding_window 512), per-layer query-head counts from num_attention_heads_per_layer, a per-head softplus attention output gate (g_proj, gating: "per-head"), an optional per-head attention sink on sliding layers, a dense SwiGLU MLP on layer 0, and elsewhere a sigmoid-routed (or sqrtsoftplus) top-k MoE whose expert selection adds e_score_correction_bias while the weights stay bias-free, renormalized and scaled by moe_routed_scaling_factor, plus an always-on shared expert. The published NVFP4 checkpoints keep the experts and the shared expert in the compressed-tensors nvfp4-pack-quantized layout (weight_packed / weight_scale / weight_global_scale); the loader reinterprets the packed E2M1 bytes as MLX native NVFP4 words, keeps every E4M3 block scale byte for byte, and applies 1 / weight_global_scale per expert after the routed matmul (the SwitchLinear global_scale sidecar) rather than folding it into the 3-mantissa-bit block scales. Pre-stacked MLX conversions (switch_mlp.*, a fused gate_up_proj is split) and per-expert bf16 exports load through the same path. The chat template emits the tokenizer's own BOS spelling (〈|EOS|〉, id 2, which is also an EOS) first, so the tokenize sites suppress add_special_tokens for a prompt that starts with the tokenizer's BOS string (MlxcelTokenizer::prompt_carries_bos); generation stops on ids 2 and 24 (</assistant>, a plain added token), and <think> / </think> are the reasoning markers. The YaRN block's attention_factor is used when the config carries one and derived from factor otherwise, which is what transformers' _compute_yarn_parameters does and therefore what the checkpoint's own modeling_laguna.py inherits; mlx-lm's port drops the key and always derives, so Laguna-XS.2-4bit, which declares attention_factor: 1.0 where the derived value would be 1.3466, rotates differently under the two runtimes. XS 2.1 declares the derived value and agrees either way. Not wired for tensor parallel. Validated against poolside/Laguna-XS-2.1-NVFP4 (native compressed-tensors NVFP4, 234 planes transcoded at load), mlx-community/Laguna-XS.2-4bit (pre-stacked affine, per-layer 8-bit router planes reconciled from the tensor shapes) and poolside/Laguna-S-2.1-NVFP4 (48 layers, 256 experts at top-10). S 2.1 mixes expert formats in one checkpoint: layers 1 to 39 carry NVFP4 routed experts (117 planes), layers 40 to 47 keep their routed experts in bf16, and the shared experts are bf16 throughout, so a load exercises both the NVFP4 and the per-expert bf16 paths. It generates fluent Python with finite logits. Its load-time resident figure (49.4 GiB) counts only the transcoded planes, since the bf16 weights realize on the first forward, so plan for about 93 GiB and a 128 GB host. The external reference is the checkpoint's own modeling_laguna.py, run out of band in float32 by scripts/laguna_oracle_trace.py, which decodes the NVFP4 planes exactly and writes the examples/logit_trace format. Against it on XS 2.1, mlxcel (bf16) has no top-1 disagreement at a position the reference had decided (top-two gap of 2 or more): 0 of 75 at width 1 and 0 of 296 at width 8, both behind 512 tokens of context, and 0 of 107 at width 256 in a window that starts with the BOS token. A window without the 〈|EOS|〉 BOS degrades in the reference as well (perplexity about 950), and the only decided disagreement at width 256, 1 of 93, is in such a window. The reference implements neither sqrtsoftplus routing nor the sliding-layer sinks (it allocates sink but never reads it), and no published checkpoint enables either, so those two paths are covered by synthetic-config tests only.)
Inkling (inkling_mm_model and inkling): the text backbone combines hybrid sliding/global NoPE attention, learned banded relative-position bias and log-position temperature, four f32 short-convolution states per layer, dense-first SwiGLU, logsigmoid-softmax routed and shared experts, muP-scaled logits, and padded-vocabulary trimming. Image input is supported through both the CLI --image path and OpenAI-compatible server image_url content blocks when the checkpoint carries vision_config and model.visual.* weights. The HMLP tower maps row-major 40x40 RGB tiles duplicated across two temporal slots directly to text-width soft tokens; exact-width images intentionally receive a trailing all-padding tile column, and each rendered image placeholder expands to the number of tiles preprocessing actually produced before ordered feature scatter. Audio input is supported for checkpoints that also carry audio_config and model.audio.* weights through CLI --audio and server input_audio, alone or with still images. The bounded host path decodes, downmixes, and resamples WAV input to 16 kHz, produces reference-compatible dMel IDs, scatters image rows first and audio rows second, and keeps every multimodal request on classic prepared-embedding prefill. Video input is supported through CLI --video and server video_url: sampled frames default to 2 fps, odd sequences repeat their final frame, and at most 16 evenly spaced adjacent pairs are allocated request-wide without crossing clip boundaries. Each clip restarts its timestamp origin, and timestamps use actual probed source-frame time after sampling. Before decoding, the request is capped at eight clips, 32 unique selected frames, and 512 MiB of decoded pixel storage; only selected frames are decoded and retained. The second frame replaces temporal slot 1 only in the suffix of HMLP rows belonging to video tiles. Companion still images remain ahead of complete timestamped Inkling content parts inside the current user turn and retain duplicated temporal planes. Native chained MTP is available for B = 1 with the original model.mtp.layers.* tensors for standalone text targets and text-only requests on HMLP-wrapped checkpoints. Speculative decode uses the wrapper's text backbone, while image-, audio-, and video-bearing requests stay on the classic prepared-embedding prefill path before continuing through the shared text target. MTP shares the target embedding, final norm, and LM head while restoring and replaying exact KV plus four-convolution state after verification. The loader accepts original bf16/f32, native ModelOpt NVFP4, and pre-converted affine MLX text weights, plus dense or affine-quantized HMLP projections and dense or affine-quantized audio embeddings. Fused kernels, padded batching, and image-feature caching remain separate work. Validation covers f64-referenced dMel extraction and quantization, summed-channel audio embeddings, deterministic HMLP planning/folding, tiling/normalization, placeholder cardinality, image-then-audio scatter, current-user audio insertion with history, clip-local adjacent-frame selection, temporal suffix splicing, pre-decode resource caps, timestamp drift, raw-weight detection and normalization, the synthetic text graph, caches, sanitizer, chat markers, deterministic tiny-model greedy MTP parity, exact snapshot/replay, wrapper dispatch, and prepared-prefill preservation. The public 153.5 GB target, roughly 170.7 GB native NVFP4 checkpoint, and 4.5 GB MTP shard were not available on the validation host, so real image/audio/video answer quality, MTP throughput, and acceptance length remain unmeasured.)
TeleChat3 (telechat3, TeleAI: structurally a stock Llama decoder (RMSNorm, GQA, SwiGLU, untied lm_head), so what matters is the one thing that is not stock. Telechat3-36B-Thinking ships rope_scaling.rope_type = "telechat3-yarn" with factor: 4.0, original_max_position_embeddings: 8192, beta_fast: 32 and beta_slow: 1. Upstream routes that into the same YarnRoPE it uses for "yarn" and "deepseek_yarn": the vendor prefix names the checkpoint that produced the config, not a different algorithm, so the shared YaRN reader accepts all three spellings. This is why the family does not reuse the llama3 decoder. llama3::ModelArgs reads its rope_scaling field since #1355, but only for the default, linear and llama3 schemes, and it carries no beta_fast / beta_slow, so routing TeleChat3 through that path would warn once and then rotate at the unscaled base while every tensor kept the right shape; YaRN and default RoPE agree closely at small offsets and only diverge past original_max_position_embeddings, so no short-prompt output could expose it. TeleChat3 also keeps rope_theta at the top level of config.json while the shared reader looks for it inside the scaling block (defaulting to 500000, not this family's 1000000), so the base is injected into the block before the table is built; a table built at the wrong base is finite, correctly shaped and wrong. attention_bias puts bias terms on q/k/v/o and is plumbed because the family declares it, but the published 36B checkpoint sets it false, so it is not what distinguishes this family in practice. Like every other text family here it satisfies the maskless-prefill contract by routing a multi-token prefill with no caller mask through causal_attention. Not wired for tensor parallel, for the same reason it does not reuse the llama3 decoder: the TP runtime parses config.json straight into llama3::ModelArgs, which has no YaRN branch, so a sharded TeleChat3 would drop the scaling. Validation checkpoint: mlx-community/Telechat3-36B-Thinking-4bit.)
Apertus (apertus, Swiss AI: Llama-style dense transformer with an xIELU activation MLP (no gate), QK-norm, llama3 RoPE scaling, and untied embeddings)
Seed-OSS (seed_oss, ByteDance: plain Llama-style dense transformer with a standard SwiGLU MLP and standard residuals. The only deltas are a split attention bias (attention_bias on q/k/v, attention_out_bias on o_proj), an explicit head_dim, untied embeddings, and a {"rope_type": "default"} rope_scaling that applies no scaling. Validated against mlx-community/Seed-OSS-36B-Instruct-4bit.)
Mamba, Mamba2, RWKV7, Jamba, Nemotron-H
Recurrent Gemma / Griffin (recurrent_gemma, also accepted as griffin): the RG-LRU recurrent block interleaved with windowed local attention on the pattern config.block_types declares, which is the repeating unit rather than the per-layer list and is indexed block_types[i % len] (recurrentgemma-9b declares ["recurrent", "recurrent", "attention"] over 38 layers, giving 26 recurrent and 12 attention). Three things this port got wrong until #1687 are worth naming because each fails in a different way. The depthwise conv takes MLX's NLC [B, L, C], not PyTorch's [B, C, L]; the transposed call raises "input channels must be divisible by the number of groups" on every forward, since MLX reads the channel count off the last axis. RoPE is applied after the transpose to [B, H, L, D], not before, because fast_rope takes token positions from the second-to-last axis and rotating [B, L, H, D] makes the head index the position. And iterating block_types directly instead of 0..num_hidden_layers builds one layer per pattern entry, which loads cleanly, runs, and emits finite nonsense. Validated checkpoint: alpindale/recurrentgemma-9b (17.96 GB bf16, 38 layers, lru_width 4096, conv1d_width 4, 16 query heads over 1 KV head, attention_window_size 2048, logits_soft_cap 30.0). On that checkpoint greedy decode matches mlx-lm token for token over 60 tokens of a chat prompt; a plain completion prompt agrees for the first several tokens and then diverges, which is the bf16 to f16 weight conversion this runtime applies on Apple Silicon and not a porting difference. google/recurrentgemma-2b and -2b-it are gated on HuggingFace, so the 9b mirror is what the port is qualified against. Decode throughput is currently about 19% of mlx-lm on M5 Max (5.65 against 29.48 tok/s at 512 prompt tokens) and prefill about 53% (864 against 1645.73), which is a known gap tracked in #1689 and not a correctness problem.
Falcon-H1 (TII: runs a Mamba2 SSM mixer and GQA attention in parallel within each block, summing both outputs; the MUP channel multipliers are pre-folded into the MLX weights)
LFM2 and LFM2-MoE (Liquid Foundation Models: short-convolution and attention hybrid; the MoE variant routes through sigmoid-gated experts)
LFM2-VL (lfm2_vl / lfm2-vl): a SigLIP2-style packed-patch vision tower (native variable resolution, per-image bicubically-resampled position grid) + a pixel-unshuffle (space-to-depth) projector into the LFM2 hybrid text backbone. Each image is smart-resized so its post-downsample token count lands in [min_image_tokens, max_image_tokens], packed at its native patch count (no padding), and its ceil(h/f)*ceil(w/f) projected tokens replace the <image> placeholder. When processor_config.json enables do_image_splitting, large images are resized onto the reference aspect-ratio tile grid, emitted row-major with <|img_row_r_col_c|> framing and an optional <|img_thumbnail|> view, while small images keep the single-view path.
PLaMo 2 (Preferred Networks: interleaves Mamba SSM and GQA attention layers by index; each block carries normformer-style pre/post offset RMSNorms, and the Mamba mixer derives B/C/dt from a post-conv projection). The architecture is validated against the mlx-lm reference at the token-id level (tests/plamo2_parity.rs). CLI text generation additionally needs support for PLaMo's custom PlamoTokenizer (the tokenizer.jsonl Unigram format), which the Rust tokenizer loader does not yet read.
Kimi Linear, LongCat Flash, LongCat Flash N-gram
Kimi K3 text backbone (kimi_k3 with text_config.model_type: kimi_linear, Moonshot: the 2.8T-parameter, 93-layer successor to Kimi Linear, ported as the text decoder only. Relative to the kimi_linear module it keeps the absorbed NoPE MLA and the grouped sigmoid router and adds five mechanisms. The Kimi Delta Attention layers (69 of 93; a layer is KDA when its 1-based index is in linear_attn_config.kda_layers, MLA otherwise, so the MLA layers are 4, 8, ..., 92, 93) project q/k/v with one fused qkv_proj and one depthwise qkv_conv over the concatenated 3 * 96 * 128 channels, gate the normalized output with a full-rank g_proj (use_full_rank_gate), and run the gated delta recurrence with the gate g = exp(gate_lower_bound * sigmoid(exp(A_log) * (a + dt_bias))) at the published gate_lower_bound = -5.0, which keeps every decay inside (e^-5, 1) where the softplus gate of Kimi Linear can decay a state to zero; A_log ships as 128 entries for 96 heads and is sliced at sanitize. The MLA layers compress the query through q-LoRA (q_a_proj at rank 1536, an RMSNorm at eps 1e-6, q_b_proj), apply no rotation to the q_pe / k_pe halves (mla_use_nope, rejected when false), and multiply the attention output by sigmoid(g_proj(x)) before o_proj (mla_use_output_gate). Every gated MLP uses the SiTU activation (beta * tanh(gate / beta) * sigmoid(gate)) * (linear_beta * tanh(up / linear_beta)) in f32 with activation_situ_beta = 4.0 and activation_situ_linear_beta = 25.0 (a null linear_beta drops the input tanh; any hidden_act other than situ is refused at load). Layers 1 through 92 are a latent MoE: routed_expert_down_proj takes the 7168-wide hidden state to routed_expert_hidden_size = 3584, 16 of 896 experts run on that latent through the shared SwitchGLU path with the SiTU activation, the weighted sum passes through routed_expert_norm (latent_moe_use_norm) and routed_expert_up_proj back to 7168, and two always-on shared experts (gate_proj / up_proj at 2 * 3072 rows) run on the full hidden state; layer 0 is the dense 33792-wide SiTU MLP (first_k_dense_replace = 1). Attention Residuals (attn_res_block_size = 12) freeze the running residual every 12 layers (layer 0 stores the embeddings) and feed each attention and MLP sublayer a softmax mix of the frozen blocks and the current partial sum, scored per token by res_norm.weight * res_proj.weight against the RMS-normalized states in f32, with a model-level output_attn_res_{proj,norm} mix after the last layer; a null block size gives the ordinary pre-norm residual. The published checkpoint is compressed-tensors mxfp4-pack-quantized: the 896 per-layer experts ship as uint8 weight_packed and uint8 E8M0 weight_scale planes at group size 32, which the sanitizer stacks per layer and reinterprets as MLX's mxfp4 layout (uint32 packed codes, uint8 block scales, no biases, [896, 3072, 448] for gate_proj / up_proj and [896, 3584, 384] for down_proj) so gather_qmm reads them without a conversion step; everything the quantization config ignores (attention, shared experts, the dense MLP, the router, lm_head) stays bf16, and the checkpoint is treated as quantized for the Apple Silicon dtype policy so those bf16 planes are not converted to f16 around the packed experts. The fused single-token MoE kernel is affine-only, so the mxfp4 experts always take the gather path. Weight keys arrive under a language_model. prefix that is stripped; model.mtp* and any layer at or past num_hidden_layers are dropped, block_sparse_moe.* is renamed to mlp.*, and on a checkpoint loaded as the text backbone (no vision_config, or no vision_tower.* tensors) the vision_tower.* and mm_projector.* keys are dropped too. The published checkpoint, which has both, is detected as the Kimi K3 VLM instead and routes its vision keys to the MoonViT3D tower (see the vision-language section); the text side is the same module either way. The tokenizer and the native XTML chat renderer are described under Tiktoken vocabularies. The full model needs about 1.4 TB at 4 bits, more than twice the memory of the largest single Apple Silicon host, so only a pipeline across at least three 512 GB hosts could hold it; that path does not run yet, because this port adds the per-layer partition profile (KDA, MLA, dense and MoE layers priced separately, see distributed inference) but no pipeline stage executor, which #1734 tracks. What runs today is single-process on a layer-truncated local copy whose config.json lowers num_hidden_layers, where the load, shape, causality and finite-logit gates are checked. Prefill runs in one pass: the model resets its KDA and MLA state on any multi-token forward that carries no sequence id, so it opts out of chunked prefill rather than answering a long prompt from its last chunk.)
GPT-2 (gpt2, OpenAI: the original decoder-only transformer, and the first family in this tree to use learned absolute position embeddings. wpe(arange(seq_len) + cache_offset) is added to the wte token embeddings at the input boundary and there is no RoPE anywhere, so the KV-cache offset enters the graph through the position lookup rather than through a rotation. The rest of the block is LayerNorm with bias (not RMSNorm), one fused c_attn QKV projection split three ways on the last axis, multi-head attention (n_kv_heads == n_head), a tanh-approximate GELU MLP at four times the model width, and a tied output head (wte.as_linear; the checkpoint has no lm_head tensor). Raw HuggingFace exports store c_attn / c_proj / c_fc in the Conv1D [in, out] layout, which the loader transposes to [out, in]; the transpose applies to weights only, never to the 1-D bias vectors, and the decision is taken from the c_attn weight shape so an already-sanitized MLX conversion loads unchanged. h.N.attn.bias is HuggingFace's [1, 1, n_ctx, n_ctx] registered causal-mask buffer rather than a projection bias; it is dropped at load and never reaches a linear layer. Validated checkpoints: openai-community/gpt2 (f32, 124M, bare keys), distilbert/distilgpt2 (transformer.-prefixed keys) and mlx-community/gpt2-base-mlx (model.-prefixed, already transposed); all three generate the same text for the same prompt. No GPT-2 checkpoint ships a chat template, so the CLI renders the prompt verbatim and --no-chat-template is optional. The wpe table has a hard 1024 rows: the default -n -1 budget already resolves to n_positions - prompt_len, but an explicit -n N or a prompt longer than the context runs past the table, and every token past it is embedded at the last row (the loader clamps and warns once during generation rather than indexing out of bounds). Loading separately rejects a checkpoint whose wpe weight has fewer rows than config.json's n_positions claims, and rejects a c_attn / c_proj / c_fc shape that disagrees with the layout Gpt2Layout detected from layer 0, since the position clamp and the Conv1D transpose both trust those config-derived and layer-zero-derived bounds for every later lookup.)
GPT-BigCode (gpt_bigcode, BigCode: the StarCoder / SantaCoder code models, a GPT-2 derivative whose defining change is Multi-Query Attention. With multi_query: true a single KV head is shared by every query head, so the fused c_attn projection produces n_embd + 2 * kv_dim features rather than 3 * n_embd and is split at [n_embd, n_embd + kv_dim], which is not an even three-way split; on bigcode/gpt_bigcode-santacoder that is 2048 + 2 * 128 = 2304 for 16 query heads of width 128 over one KV head. MQA maps onto the existing GQA attention path with n_kv_heads = 1, so no new attention code is involved. The rest of the block follows GPT-2: learned absolute position embeddings added to the token embeddings at the input boundary with no RoPE anywhere, LayerNorm with bias, and a tied output head. Unlike GPT-2 there is no Conv1D weight layout. HuggingFace GPTBigCode builds its projections with nn.Linear, so c_attn / c_proj / c_fc are already stored [out, in] (transformer.h.0.attn.c_attn.weight is [2304, 2048], where the GPT-2 equivalent is [768, 2304]) and the loader must not transpose them; a weight that arrives in the transposed orientation is rejected by name rather than accepted. The MLP width comes from n_inner, not a hardcoded 4 * n_embd, and there is no h.N.attn.bias causal-mask buffer to strip because HuggingFace registers this family's mask non-persistently. Loading rejects a wpe or wte table with fewer rows than config.json's n_positions / vocab_size claims, a projection or norm whose shape disagrees with the config in any layer, and an untied config with no lm_head tensor, since the position clamp and the token lookup both trust config-derived bounds and an embedding gather does not range-check a positive index. A quantized c_attn is packed on the input axis only, so its output row count is checked against the config-derived width on the same path as an unquantized checkpoint; without that check, a 4-bit or 8-bit conversion whose config.json omits multi_query (which then defaults to true) against a packed weight sized for full multi-head attention would leave every slice in bounds while Attention::forward's config-derived offsets read K and V from the wrong channels. Validated checkpoint: bigcode/gpt_bigcode-santacoder (f16, 1.1B). Code checkpoints ship no chat template, so the CLI renders the prompt verbatim.)
GPT-NeoX (gpt_neox, EleutherAI: the decoder behind the Pythia suite and the NeoX-derived checkpoints. Three things define it. First, the fused query_key_value projection is interleaved per head: it is [3 * hidden_size, hidden_size] in plain nn.Linear orientation, but its output is not three contiguous hidden_size blocks. The projection is reshaped to (..., num_heads, 3 * head_dim) and split on that axis, so the layout is head-major, [q_i | k_i | v_i] per head; on EleutherAI/pythia-1b that is 8 heads of width 256 contributing 768 contiguous channels each. A flat three-way split of the last axis (the GPT-2 and GPT-BigCode pattern) yields Q, K and V of exactly the right shape assembled from the wrong channels, so nothing throws and the model still emits fluent English out of a scrambled attention; the unit tests therefore pin the layout by channel value, not by shape. Second, partial RoPE: only int(head_dim * rotary_pct) channels of each head rotate and the rest pass through untouched (64 of 256 for Pythia's rotary_pct 0.25), expressed through the dims argument of fast_rope with traditional = false. Third, the optional parallel residual: with use_parallel_residual: true (every Pythia checkpoint) the attention and MLP sub-layers both read the same pre-norm input and their outputs are summed into the residual, x + attn(input_layernorm(x)) + mlp(post_attention_layernorm(x)); with false the block is the ordinary chained form where the MLP norm reads the post-attention residual. Both layouts are implemented, and running the wrong one is a silent quality regression rather than a crash. The rest of the block is LayerNorm with bias (not RMSNorm), no learned position embeddings at all, an MLP of dense_4h_to_h(gelu(dense_h_to_4h(x))) with no gate/up pattern, and an untied embed_out head. HuggingFace registers three per-layer PyTorch buffers that reach the checkpoint and are dropped at load: attention.bias (a [1, 1, n, n] causal mask whose name collides with the projection-bias namespace, exactly as GPT-2's h.N.attn.bias does), attention.masked_bias, and attention.rotary_emb.inv_freq (a precomputed RoPE table that fast_rope recomputes from rotary_emb_base). Both the raw HuggingFace key layout (gpt_neox.layers.N, top-level embed_out) and the layout produced by upstream mlx-lm's sanitize (model.h.N, model.embed_out) load. Loading rejects an embed_in table with fewer rows than config.json's vocab_size claims, since an embedding gather wraps a negative index but does not range-check a positive one; a projection, bias or norm whose shape disagrees with the config in any layer, including on the quantized path, where packing compresses the input axis only and the output row count is still the width the fused reshape depends on; a rotary_pct whose rotary dimension count is zero, negative, odd or larger than head_dim, because MLX enforces exactly that contract on rope's dims argument by throwing, and an MLX C++ exception crossing the cxx bridge is an uncatchable std::terminate at the first forward pass rather than a load error; a non-positive or non-finite rotary_emb_base, which would make every rotated channel NaN; a non-finite, negative or zero layer_norm_eps, since fast::layer_norm never looks at eps and computes x * rsqrt(mean(x^2) + eps), so a bad value turns every hidden state into NaN without anything throwing and the checkpoint would otherwise load cleanly and generate uniform garbage; and a quantization block whose bits falls outside 1..=32 or whose group_size is non-positive, since both reach quantized_matmul unreconciled and MLX's own check on the derived unpacked width divides by zero or collapses to zero before throwing, the same uncatchable-std::terminate shape the rope guard exists to prevent. Validated checkpoint: EleutherAI/pythia-1b (f16, 244 tensors). Pythia ships no chat template, so the CLI renders the prompt verbatim and --no-chat-template is optional.)
GPT-OSS
Kyutai Helium (helium, Kyutai: a dense Llama-shaped decoder. RMSNorm before attention and before the MLP, grouped-query attention, a SwiGLU MLP over gate_proj / up_proj / down_proj, no QK-norm, no MoE, no sliding window, and Llama's own weight key names, so the decoder block, attention and MLP are the existing dense path unchanged. The one architectural difference is the RoPE convention: upstream builds nn.RoPE(head_dim, traditional=True, base=rope_theta), so Helium rotates interleaved channel pairs (2i, 2i+1) where every other Llama-family model in this tree rotates the split-half pairs (i, i + dims/2). The two produce identically shaped tensors from identical weights, so running the wrong one is a silent quality regression rather than a crash, and no shape assertion can catch it. The flag is threaded from the loader through to every RoPE call the family can reach, including the batched decode path; it also disables the two fused quantized RoPE fast paths (MLXCEL_ENABLE_FUSED_QKV_SPLIT_ROPE and MLXCEL_ENABLE_FUSED_CAUSAL_PREFILL_ATTENTION), whose C++ launchers hardcode the split-half rotation and take no flag, so a quantized Helium would otherwise be mis-rotated under either of those opt-in gates. Tensor parallelism is deliberately refused: Helium's convention is fixed in upstream code rather than declared in its config, so the published config.json carries no rope_traditional key and the loader supplies it during config conversion, while the TP runtime builds its per-rank model by parsing config.json straight into the shared Llama config and never goes through that conversion. head_dim comes from hidden_size / num_attention_heads as upstream does, and a config whose declared head_dim disagrees is rejected rather than one being silently preferred. Loading also rejects an embedding table or output head with fewer rows than config.json's vocab_size claims, since an embedding gather wraps a negative index but does not range-check a positive one; a projection, bias or norm whose shape disagrees with the config in any layer, on both axes and including on the quantized path, where packing compresses the input axis only so the output row count is still the width the fused QKV concatenation and the attention reshape depend on and the input width is recovered from the scales the way MLX recovers it, scales.shape(-1) * group_size; quantization biases whose shape disagrees with the scales they are the zero points for; an attention block whose q_proj, k_proj and v_proj disagree on whether they are quantized, on whether they carry affine biases, or on whether they carry a dense bias, since the fused loader decides from q_proj alone and silently drops a whole bias set when one is missing; an odd or non-positive head width, because MLX enforces exactly that contract on rope's dims argument by throwing and an MLX C++ exception crossing the cxx bridge is an uncatchable std::terminate at the first forward pass rather than a load error; a non-positive or non-finite rope_theta, which would make every rotated channel NaN; a non-finite, negative or zero rms_norm_eps (Helium's own 1e-08 is unusually small for this family and stays accepted); and a quantization block whose bits falls outside 1..=32 or whose group_size is non-positive. Stop tokens come from config.json's eos_token_id, since Helium's tokenizer_config.json declares neither an EOS token nor a chat template. Validated checkpoint: kyutai/helium-1-preview-2b (mlx-community 4-bit, 559 tensors). Helium 1 preview is an English/French base model and ships no chat template, so the CLI renders the prompt verbatim.)
Youtu-LLM (youtu / youtu_llm, Tencent: the text-only decoder that also serves as Youtu-VL's text tower, so this is a route onto an existing implementation rather than a new architecture. Multi-head Latent Attention in the DeepSeek-V2 layout: a LoRA-compressed query (q_a_proj into q_a_layernorm into q_b_proj) split into 128 non-positional and 64 rotary dimensions, a kv_a_proj_with_mqa that yields a 512-dimension latent plus one 64-dimension rotary key shared by all heads, and a kv_b_proj decomposed at load into the per-head embed_q / unembed_out pair. Dense SwiGLU MLP, tied word embeddings, no MoE. tencent/Youtu-LLM-2B declares model_type: "youtu" and one community conversion declares youtu_llm; both were rejected at detection before this entry existed. A third conversion, mlx-community/Youtu-LLM-2B-4bit, relabels itself deepseek_v2 for mlx-lm compatibility and deliberately keeps the DeepSeek-V2 route: greedy decode of that checkpoint through the DeepSeek-V2 decoder was measured against an mlx-lm deepseek_v2 oracle on the same weights and tracks it, so detection decides on the label alone and does not inspect architectures. Two config keys that the Youtu-VL checkpoint never exercises are honored on this route. rope_interleave selects the interleaved rather than half-split rotary pair layout, and it reaches the rope call instead of being assumed. A rope_scaling block is accepted only when it reduces to the identity, which is what the published {"type": "yarn", "factor": 1.0, "mscale_all_dim": 0} block is: at factor 1 the extrapolation and interpolation frequencies coincide and the attention mscale is 1. A factor above 1 asks for a frequency interpolation this decoder does not implement and is refused at load, rather than decoding on the plain table and returning fluent but positionally wrong long-context text. q_lora_rank is optional, so a variant that projects the query directly through q_proj parses. One property of this checkpoint is worth knowing before comparing it against a reference: on raw completion prompts it is close to undecided at many positions (median top-1 to top-2 logit gap around 0.6 over a 126-token passage, with roughly one position in eight an exact tie in bf16), so greedy output is not reproducible across implementations, and mlx-lm's own bf16 and float32 runs agree on only 91 percent of those positions. Through its chat template, where the model is in distribution, the median gap rises to about 3.9 and greedy output is stable; parity comparisons should use the chat template.)

Many of these families have checkpoint-specific config or weight-layout requirements. If a checkpoint fails detection or loading, inspect its config.json::model_type first and compare it with src/models/detection.rs.

Block-diffusion text models

Family	`model_type` key	Notes
DiffusionGemma	`diffusion_gemma` / `diffusion_gemma_text`	Block-diffusion on a Gemma 4 MoE backbone. Generates a canvas of tokens per block through iterative denoising rather than token-by-token left-to-right decoding. CLI (`mlxcel generate`) supports text and image input (`--image <path>`, repeatable). Served in `mlxcel-server` (serial, batch-1 by design) via `/v1/chat/completions` and `/v1/completions`; image input follows the standard `image_url` content part format, all declared images must resolve, and the pending single-stream queue honors `--max-queue-depth`. See Block-diffusion generation.
LLaDA-2 MoE	`llada2_moe`	Masked-diffusion LM with a DeepSeek-V3-style MoE FFN (sigmoid-routed grouped experts on `gate.weight` + `expert_bias`, one always-on shared expert, `first_k_dense_replace` dense layers). Decoder-only transformer with fully bidirectional attention (no causal mask), fused QKV projection, per-head QK-norm, and partial RoPE. Generates by semi-autoregressive block-wise unmasking of `<

DiffusionGemma uses a two-phase forward pass: an encoder prefill that caches the prompt into dense FP16 KV caches, then a canvas loop that attends bidirectionally within each output block while attending causally to the cached prefix. Load detection accepts model_type: "diffusion_gemma" (outer config) and model_type: "diffusion_gemma_text" (inner text_config). The fused MoE gate_up_proj weights are split at load time. When a vision tower is present in the checkpoint, its weights are loaded and wired for image input; checkpoints without vision weights fall back to text-only mode without error.

Vision-language and multimodal variants

Implemented VLM variants include:

Gemma 3 VL, Gemma 3n VL, Gemma 4 VL
Gemma 4 Unified (gemma4_unified): encoder-free text + image + audio + video. Patch-projection vision embedder and waveform-chunk audio path feed the shared Gemma 4 backbone, with blockwise bidirectional attention over image/video token spans during prefill; audio tokens stay outside those blocks and keep the plain causal rows. Video is handled as images-per-frame: frames are extracted with ffmpeg (uniform sampling, default 2.0 fps), patchified through the same vision embedder with a per-frame vision_soft_tokens_per_video_frame budget (70), and scattered into video_token_id placeholder spans. Video is available on both the CLI (--video) and the server (video_url content blocks). The prompt grows by ~70 soft tokens per sampled frame, so a multi-second clip at the default fps expands past the model's 1024-token sliding window. That over-window single-pass prefill is handled correctly: the windowed prefill mask spans the full prompt (the rotating cache keeps every prefill key and only trims to the window for the decode step), so long clips decode coherently without lowering --fps. Video and audio may be sent in the same prompt on this family, and only on this family: --video together with --audio on the CLI, a video_url content block together with an input_audio block on the server. The frame runs and the audio run expand into disjoint placeholder ids and scatter through the same merge_multimodal call, so a clip's soundtrack and its pictures reach the backbone together. The one-audio-clip-per-request limit is unchanged, and every other family (Gemma 4 VL, Kimi-VL, Inkling, Qwen-VL) still refuses the combination: the server answers Combined video and audio inputs are not supported as a 400 invalid_request_error, and the CLI exits with Combined --video and --audio inputs are not supported yet.
Gemma 4 VL audio: the Gemma 4 VL checkpoints that ship a Conformer audio tower (for example the e2b / e4b instruct models) take spoken audio from the CLI with --audio <path> and transcribe or answer questions about it. Input audio is resampled to 16 kHz before the Conformer encoder, so a clip at any source rate produces the encoder frame count the duration-based audio-token budget expects. Server-side input_audio in POST /v1/chat/completions is also supported: the audio block is spliced inside the last user turn, before the <turn|> end-of-turn marker that Gemma 4 uses (id 106).
Llama 4 VLM
Llama 3.2 Vision (mllama): a Llama-3 text backbone whose layers at cross_attention_layers (e.g. [3, 8, 13, 18, 23, 28, 33, 38]) are gated cross-attention adapters attending to a tiled ViT tower. The image processor picks an optimal tile arrangement (up to max_num_tiles, default 4) from the supported aspect ratios, resizes and pads each image into 560x560 tiles, and emits aspect-ratio ids/masks. The tower adds gated tile + position embeddings, runs a local then a global (gated) transformer, and concatenates a set of intermediate hidden states; a multi_modal_projector maps those vision_output_dim features into the text hidden size. Unlike the LLaVA-style VLMs, image features are not merged into the token stream: they are held as cross_attention_states and consulted through the gated cross-attention layers (with per-head q_norm/k_norm). Text-only prompts leave the cross-attention layers as pass-throughs.
LLaVA and LLaVA-Bunny
Aya Vision and PaliGemma
Cohere Compass / North-Micro-Vision (cohere_compass): CohereLabs' small generative VLM. The vision half is a Qwen3-VL tower down to the config keys (Conv3d patch embed at patch 16 / temporal 2, a 48x48 learned position grid bilinearly resampled per image, 2D vision RoPE, cu_seqlens-packed attention, a 2x2 spatial merger, and three DeepStack mergers at blocks 8 / 16 / 24 injected after the first three decoder layers), so the encoder, image processor, prompt expansion and weight-prefix remap are the Qwen3-VL ones reused unchanged. The text half is a Command-style parallel decoder: one input LayerNorm feeds both the attention and the SwiGLU MLP and both land on the residual, there is no post_attention_layernorm, embeddings are tied, and the head output is multiplied by logit_scale (0.25). Its positional encoding is per layer type, keyed off rope_parameters: the sliding_attention layers rotate with the interleaved 3-axis MRoPE (section [24, 20, 20], base 50000, split rotate-half) and see at most sliding_window (4096) keys, while the full_attention layers (indices 3, 7, 11, ... of 28) have a JSON null entry, meaning no positional encoding at all rather than a fallback to the default RoPE, and attend the whole prefix. norm_type: "rms_norm" is also implemented; a transformer_block_type: "sequential" or a non-interleaved mrope_section is refused at load rather than silently run as something else. The resize bounds come from the nested image_processor block of processor_config.json (min_pixels 16384, max_pixels 3868706), which is what HF's AutoProcessor builds the image processor from; the standalone preprocessor_config.json on the published checkpoint disagrees (65536 / 16777216) and is only the fallback. The distinction is not cosmetic: the stale bound upscales anything under 256x256, so a 224x224 input would carry 64 image tokens instead of 49 and desynchronize the prompt from the reference. Validated against mlx-community/North-Micro-Vision-Instruct-4bit (2.5B, text 4-bit affine group 64, vision tower bf16). Video input (<|VIDEO_PAD|>) and the CohereCompassTextForSequenceClassification reranker head are out of scope.
- Vision parity note (#1738): the stage observer in src/vision/encoders/qwen3_vl.rs plus scripts/tools/qwen3_vl_stage_oracle_compare.py localizes the remaining image-path gap against transformers 5.18.0.dev0 (df04b012229d50d2b6dfba32c61c3057c3a40ea1) on tests/fixtures/test_image_shapes.png and a float32 copy of mlx-community/North-Micro-Vision-Instruct-4bit. The Rust processor's raw T,C patch rows are semantically equivalent to transformers' C,T view after reshape/permute (input_patches_processor_ct_order max 2.98e-08, RMSE 5.42e-09; raw T,C max 1.49, RMSE 0.464), patch embed is within f32 rounding (max 1.14e-05, RMSE 2.04e-07), the learned position grid is now within f32 rounding after keeping the interpolation multiply/add in float32 and casting back before the residual add (max 5.72e-05, RMSE 2.84e-07), and 2D vision RoPE is exact (max 0). The first remaining beyond-f32 residual appears at block 0 MLP (max 0.0311, mean 0.00143, RMSE 0.00228) and then accumulates through the tower; the three DeepStack tensors measure max/RMSE 0.0122/0.00188, 0.0170/0.00206, and 0.0243/0.00209, while post-merger is max 0.0961, mean 0.00602, RMSE 0.00810. The real greedy single-image prompt Describe this image in detail. still begins The image features... under mlxcel after this fix, while the recorded transformers oracle remains The image displays..., so image-conditioned token exactness is not claimed.
Pixtral and Mistral 3 VLM wrappers (Mistral 3 VLM supports both the standard Llama/Mistral text backbone and the Mistral4 MLA+MoE backbone; text and image-plus-text are validated on both, including the Mistral Small 4 119B checkpoint). Images keep their aspect ratio: each is downscaled so its longest side fits size.longest_edge and rounded up to a whole number of merged patches (patch_size * spatial_merge_size, so Pixtral rounds to 16 and Mistral 3 to 28), never upscaled. The [IMG] placeholder then expands to one patch token per merged cell laid out row by row, with [IMG_BREAK] between rows and [IMG_END] after the last, so non-square inputs keep the spatial structure the models were trained on. The encoder builds its 2D-RoPE grid per image from the actual patch shape, and multi-image batches with differing sizes run the tower once per image.
Qwen2-VL, Qwen2.5-VL, Qwen3-VL, and Qwen3-VL MoE: text, image, and video inputs are wired on both the CLI (--video) and OpenAI-compatible server (video_url content blocks). Video frames are decoded through the shared ffmpeg pipeline, sampled under Qwen processor frame bounds (min_frames and max_frames from processor_config.json when present, upstream defaults otherwise), padded only to temporal_patch_size, and rejected instead of silently clamped when a request exceeds max_frames. Server requests that omit a per-video FPS use the processor sidecar's fps; the CLI keeps its explicit --fps setting, whose default is 2.0. Mixed image/video prompts preserve the rendered placeholder order; a malformed prompt/media cardinality mismatch fails before generation.
Qwen3.5-VL, Qwen3.6-VL MoE, and Qwen3.8-VL (qwen3_5, qwen3_5_moe): the Qwen 3.5-family paths serve all three public generations. Qwen3.8 declares model_type: "qwen3_5" / Qwen3_5ForConditionalGeneration and is architecturally identical to Qwen3.5 at the same size, down to a byte-identical weight-map key set, so no separate family exists in the tree. The stack is a hybrid text backbone (three gated-delta linear-attention layers for every full-attention layer, full_attention_interval: 4) under the Qwen3-VL vision tower, with interleaved MRoPE over an [11, 11, 10] section. Validated checkpoint: mlx-community/Qwen3.8-27B-4bit (16 GB, 4-bit affine, 64 layers, head_dim 256, 24 Q / 4 KV heads, vocab 248320 over a 248077-entry tokenizer). CLI text, image, and video, server text, image, and video, and multi-turn prompt-cache reuse all work at the code path level; decode measured 18.21 tok/s median over three text/image runs (spread 0.04) on Apple Silicon under background load, which is an order-of-magnitude baseline rather than a benchmark record. Video support shares the Qwen-VL path above: decoded clips produce video_token_id spans, Qwen visual grids keep a real temporal axis, and the MRoPE span scanner consumes both image and video token runs. Treat actual ffmpeg-backed Qwen3.8 video Q&A on target hardware as separate runtime qualification unless that exact checkpoint/run is cited. The three config keys the generation added (output_gate_type, rope_parameters.mrope_interleaved, and the top-level language_model_only) are read at load time and rejected with a named error when they ask for behavior mlxcel does not implement (output_gate_type: "sigmoid", mrope_interleaved: false, language_model_only: true), rather than being dropped silently. vision_start_token_id is required from the config for this family and never defaulted: the whole family uses 248053, and a wrong start id mis-segments MRoPE vision spans without failing. Not supported on this family: MTP speculative decoding (the mlx-community conversions drop the mtp.* tensors and publish the drafter separately as a qwen3_5_mtp checkpoint, #1165) and audio. Despite the family naming Qwen3.8-27B is not an omni model: config.json carries no audio_config, audio_tower, thinker_config, or talker_config. The upstream Qwen/Qwen3.8-27B repo's own weight map holds model.language_model.*, model.visual.*, mtp.*, and lm_head.weight; the validated mlx-community/Qwen3.8-27B-4bit conversion is a different shape, language_model.* (with lm_head.* folded inside that prefix) and vision_tower.*, and drops mtp.* entirely. The vendor fine-grained FP8 builds (Qwen/Qwen3.8-27B-FP8 and siblings) are supported: they declare quantization_config: {"quant_method": "fp8", "fmt": "e4m3", "weight_block_size": [128, 128]} and store each converted projection as raw E4M3 bytes in *.weight plus a bf16 inverse scale per 128x128 block in *.weight_scale_inv. The loader reconstructs each pair (decode(byte) * block_scale) and requantizes it to MLX-native mxfp8 at load, one tensor at a time, so the reconstruction stays transient; the served model is 8 bits per weight plus one E8M0 exponent per 32 values, roughly the checkpoint's own footprint rather than a dequantized copy. Tensors with no sidecar (norms, embeddings, conv1d, A_log, dt_bias, and the whole vision tower, which these releases list under modules_to_not_convert) stay dense. Any other weight_block_size, a non-E4M3 fmt, and per-tensor FP8 with no weight_block_size at all are named load errors rather than a silently mis-scaled model. The ModelOpt NVFP4 builds remain out of scope: they use a different sidecar layout (weight_scale / weight_scale_2). MTP is unaffected by this, since the family does not support MTP speculative decoding here in any case. One footprint number is worth planning around because head_dim: 256 is unusually large: measured KV state on the 27B is about 87 MiB of length-independent recurrent and conv state plus 64.1 KiB per token, against a 64.0 KiB (65,536 B/token = 16 full-attention layers x 4 KV heads x 256 head_dim x 2 x 2 bytes) architectural minimum for those layers; the two agree to within about 0.16%, so nothing is meaningfully over-allocated and the 48 gated-delta layers cost only the fixed term. At the native 262,144-token context that is 16 GiB of KV for a single sequence, on top of the weights.
PaddleOCR-VL (paddleocr_vl): document-OCR VLM pairing a NaViT dynamic-resolution SigLIP-style vision encoder (Conv2d patch embedding, cached bilinearly interpolated learned position embeddings for repeated page grids, 2D vision RoPE, cu_seqlens-packed attention with uniform and length-bucketed batched fast paths, and a spatial-merge projector) with a lightweight ERNIE-4.5 text decoder that uses MRoPE. Best for plain OCR, tables, formulas, and chart understanding.
Step-3.7 (step3p7): StepFun's multimodal model. A 47-block perception_encoder ViT tower (Conv2d patchify, learned absolute position embeddings bilinearly resized for the patch grid, 2D vision RoPE, LayerScale, quick-GELU MLP) feeds two stride-2 conv downsamplers and a linear projector into the text hidden size. Each image runs a base pass (728 px, 52x52 grid collapsed to 169 tokens) and, when larger than 728 px, a tiled-patch pass (504 px windows, each 36x36 grid collapsed to 81 tokens); features are ordered patches-first then base per image and scattered into <im_patch> placeholders. The text decoder is the Step-3.5 MoE stack (text_config.model_type: step3p5), reused as the backbone. Text-only prompts work through the same checkpoint.
dots.ocr (dots_ocr): document-OCR VLM (rednote-hilab) pairing a 42-block dots_vit dynamic-resolution ViT with a Qwen2 text decoder. The tower shares the Qwen2-VL vision machinery (cu_seqlens-packed block-diagonal attention, 2D vision RoPE, merge-block patch ordering) but uses RMSNorm blocks, a SwiGLU vision MLP, a patch embed with bias and a trailing RMSNorm, and a post_trunk_norm; its merger projects each 2x2 patch block straight to the text width (no separate connector). Plain 1D RoPE text decode, no MRoPE. Best for layout analysis (bbox + category JSON), plain OCR, tables, formulas, and Markdown conversion.
Falcon-OCR (falcon_ocr): TII's 300M early-fusion document-OCR VLM, and the one VLM in the tree with no vision tower at all. Each 16x16 RGB patch is flattened and pushed through a single linear img_projector straight into the token stream, and one 22-layer decoder reads image and text together under a hybrid mask that is bidirectional inside every image block (<|image_cls|>, four register tokens, and the patch tokens) and causal everywhere else. The decoder is Llama-derived with five differences: a fused wqkv, weightless RMSNorm everywhere except the final norm (including the per-head Q/K norms, which is why the checkpoint ships only five tensors per layer), per-head learned attention sinks, a squared-ReLU gated MLP over a row-interleaved fused w13 that the loader de-interleaves, and a 3-D rotary that puts a 1-D temporal rotary on the low half of each head and a 2-D per-head spatial rotary (driven by the shipped freqs_cis_golden table) on the high half. An entire image block collapses onto one temporal position, so intra-image geometry is carried by the spatial rotary alone. The checkpoint uses the raw dim / n_layers / ffn_dim config keys; the HF spellings are also accepted. Chunked prefill is disabled for this family, matching upstream. Prompts are expanded at the token level: image blocks are prepended (or substituted for an existing <|image|> placeholder) and <|OCR_PLAIN|> is appended when the prompt does not already end with it, since the model needs the task token to transcribe rather than describe. The layout-aware second stage (category routing, nested-box suppression, region cropping, per-category instructions) is wired to the CLI through --layout-detections; see Falcon-OCR layout-aware OCR below. The PP-DocLayoutV3 detector that feeds it is a separate architecture and is not included, so the detections are an input rather than something mlxcel produces.
Jina VLM (jvlm, also accepted as jina_vlm): Jina AI's small multilingual VLM. A 27-layer SigLIP-so400m-class ViT (linear patch embedding over already-patchified 14x14 pixels, learned absolute positions, no class token, fused attn.qkv, tanh-GELU MLP) feeds a Molmo-style connector: features from two intermediate layers are concatenated on the channel axis (vit_layers: [-4, -10], 2 x 1152 = 2304 wide), two learned pad_embed rows are added where the coverage mask reports padding, the 27x27 patch grid is zero-padded to 28x28 and pooled 2x2 by a cross-attention block whose single query is the mean of each window, and a SwiGLU projector maps the result into the 2048-wide text stream. vit_layers resolves against a hidden-state list that includes the post_lnorm output, so [-4, -10] selects the outputs of layers 24 and 18 and the last two blocks are never loaded. The text decoder is Qwen2-class with an OLMo tensor layout: fused attn.qkv, GQA 16/8 at head_dim 128, per-head Q/K RMSNorm, a fused ffn.gate_up ordered [up, gate], RoPE theta 1e6, no biases, and an untied lm_head. The vocabulary is split into a 151936-row base table plus a 128-row extension holding <im_start> / <im_end> / <im_patch> / <im_col> / <|image|> / <im_slice>. The image processor is Molmo's overlap-and-resize cropper: smart_resize to a patch-aligned size inside the pixel budget, then a stretch to a crop tiling with 4-patch overlap margins, 378x378 crops, and a thumbnail emitted first, with the token block laid out as <im_start> [<im_patch> * w <im_col>] * h <im_end>. Prompts follow the checkpoint's own chat template ( User: <image> text Assistant:, with the leading space gated by always_start_with_space) and carry the BOS the reference processor prepends. The MLX conversion drops upstream's chat_template.jinja, so that template is also built in and used when the checkpoint declares none, which the server needs for text-only turns (under the generic User:/Assistant: fallback the model answers "17" to "What is the capital of France?"). Config is nested (text_config.block_config.attn_config, vision_config.vl_connector_config) rather than flat HF keys, and the released MLX conversion is mixed precision: language layer 0, every norm, the embedding tables, the ViT patch embedding and positional table, and vision_model.layers.N.ffn.down stay bf16 while the rest is 4-bit affine. Validated on jinaai/jina-vlm-mlx.
GLM-4V (glm4v): GLM-4V ViT vision tower (3D patch embedding, bilinear-resampled learned position embeddings, Conv2d spatial downsample, SwiGLU patch merger) plus a GLM-4 text backbone driven by sectioned even/odd MRoPE. Reuses the shared Qwen-VL image processor and prompt/token plumbing.
GLM-4V MoE (glm4v_moe): GLM-4.5V-class variant reusing the GLM-4V ViT vision tower with a GLM-4 MoE text backbone (grouped noaux_tc routing, shared experts, first_k_dense_replace dense layers) driven by sectioned half-split MRoPE. Reuses the GLM-4 MoE machinery and the shared Qwen-VL runtime.
Granite Vision (granite_vision, or llava_next with a granite text config): IBM's document VLM. A SigLIP vision tower with four intermediate feature taps (concatenated on the channel axis) feeds a 2-layer GELU projector; a learned image_newline embedding, LLaVA-Next AnyRes multi-tile preprocessing (image_grid_pinpoints), and a dense Granite text backbone (embedding / attention / residual / logits multipliers) complete the model. Both config.json spellings route to the same loader. Best for document understanding, charts, and tables.
Granite 4 Vision (granite4_vision): IBM's document VLM with multi-depth visual injection. A SigLIP tower feeds eight window-QFormer projectors (a QFormer with self+cross attention per 4x4/8x8 window, with deepstack mean-pool and spatial strided-offset query downsamplers) whose packed outputs are added into the residual stream at eight different depths of a Granite 4 hybrid (granitemoehybrid) text backbone during prefill, rather than merged once. Reuses the shared AnyRes tiling and the four Granite scalar multipliers. Best for document understanding.
DeepSeek-OCR (deepseekocr): DeepSeek's document-OCR VLM. Each view runs a SAM-style ViT-B (windowed / global attention with a decomposed relative-position bias, a two-conv neck, and a two-stage stride-2 conv compressor) and a CLIP-style ViT-L that ingests the SAM grid as its patch embeddings; the two token sets are concatenated on the channel axis, projected to the decoder width, and laid out as a 2D tile mosaic with learned image_newline columns and a trailing view_separator. A small DeepSeek MoE decoder (12 layers, 64 routed + 2 shared experts, standard attention, reusing deepseek) reads the mosaic. The processor pads a global 1024 view and, for larger images, adds a closest-aspect-ratio grid of 640 tiles. Best for plain text, markdown, HTML tables, and grounding boxes.
DeepSeek-OCR 2 (deepseekocr_2): the second-generation document-OCR VLM. It keeps DeepSeek-OCR's SAM-style ViT-B (with the compressor emitting 896 channels) but replaces the CLIP stage with a Qwen2-0.5B-shaped query resampler: the SAM grid is concatenated with a learnable query bank and run through 24 GQA + rotary + SwiGLU layers under a mixed mask (image tokens bidirectional among themselves, queries causal and attending to all image tokens); only the query outputs are projected to the decoder width. There is no channel-concat fusion and no image_newline mosaic, so features are flat runs assembled per image as [tiles, global, view_separator]. The processor tiles every image with 768 tiles (closest-aspect-ratio grid of 1..6) plus the padded global 1024 view. The same DeepSeek MoE decoder reads the features. Best for plain text, markdown, HTML tables, and grounding boxes.
Unlimited-OCR (unlimited-ocr): baidu's document-OCR VLM (baidu/Unlimited-OCR), built on the DeepSeek-OCR stack. The SAM ViT-B + CLIP-L/14-224 encoders, the linear projector, the image_newline / view_separator mosaic, and the 12-layer DeepSeek MoE decoder (64 routed + 2 shared experts) are the same as DeepSeek-OCR, so the whole vision path and processor are reused. The one new piece is the decode cache: attention runs against a per-layer ring sliding KV cache that keeps the full prefill KV (prompt + image tokens) permanently and rotates only the most recent sliding_window_size (128) decode tokens, with absolute RoPE positions that keep increasing past the window. A single literal <image> covers all pages of a multi-page input. Best for long-document OCR where decode runs well past the window without unbounded KV growth.
DeepSeek-VL2 (deepseek_vl_v2): DeepSeek's general vision-language model. Each view runs a SigLIP-style ViT (no CLS token, a patch_size Conv2d patch embed folded into a linear, a learned absolute position embedding, fused-qkv full attention, GELU-tanh MLP, and a trailing LayerNorm); the downsample_mlp_gelu projector folds each 2x2 patch block into one channel-outermost feature vector and maps it to the decoder width with a two-layer GELU MLP. Features are laid out as a 2D tile mosaic with learned image_newline columns and a view_separator, ordered [global, view_separator, local]. A DeepSeek-V2 MoE decoder (MLA attention, softmax-gated routed experts plus shared experts, reusing deepseek_v2) reads the mosaic. The processor pads a global view and, for at most two images, adds a best-fit grid of local tiles chosen from candidate_resolutions; three or more images use a single tile each. Best for image description, grounding, and document understanding.
ERNIE-4.5 MoE VL (ernie4_5_moe_vl): Baidu's vision-language MoE. A DFNRope ViT (linear patch embedding over 588-wide merge-window rows, 2D vision RoPE, cu_seqlens-packed attention, quick_gelu MLP) feeds a variable-resolution resampler (2x2 spatial fold, temporal pair fold with single-frame duplication, GELU MLP stacks, RMSNorm) whose rows replace the <|IMAGE_PLACEHOLDER|> tokens. The text decoder extends ERNIE-4.5 MoE with modality-split expert banks: separate text and multimodal routers and expert stacks selected per token by token type, with a correction bias that shifts expert selection but never the mixing weights, plus a fused shared-experts MLP. Position encoding is interleaved 3D MRoPE ([T, H, W] axes assigned per frequency index, adjacent-pair rotation) that degenerates to traditional RoPE for text. Validated against mlx-community/ERNIE-4.5-VL-28B-A3B-Thinking-4bit (28B-A3B; text and resampler 4-bit, vision tower bf16). Best for general image chat and grounded reasoning; the checkpoint is a thinking variant and emits reasoning before the answer.
Qwen3-Omni MoE (qwen3_omni_moe, thinker): Alibaba's omni-modal MoE. Stage 1 covers the thinker: text output conditioned on text, image, and audio inputs. The vision tower and MoE text decoder are the Qwen3-VL-MoE stack (DeepStack feature injection, interleaved MRoPE) reused unchanged; the new audio tower converts 16 kHz audio to a 128-bin log-mel spectrogram, downsamples it through three stride-2 convolutions (13 output frames per second of audio), and runs 32 windowed-attention encoder layers whose output rows scatter into the token stream exactly like image features. Audio arrives via --audio file.wav on the CLI (combinable with --image). Stage 2 adds speech output: mlxcel generate --output-audio out.wav runs the talker and code2wav after text generation and writes 24 kHz mono PCM16. The talker is a 20-layer Qwen3-MoE codec decoder conditioned on the projected thinker token embeddings of the chat-role segments; per frame it emits the first of 16 codebooks and a 5-layer code predictor fills in the residual 15, then the code2wav vocoder (causal pre-transformer, ConvNeXt upsampling, BigVGAN-style SnakeBeta decoder) renders 1920 samples per 12.5 Hz frame. --speaker selects the voice (ethan default; chelsie and aiden also ship in the released checkpoints). The speech stack loads lazily and only when requested, so text/vision use keeps its memory footprint; speech currently requires a text-only, chat-templated prompt. Validated against mlx-community/Qwen3-Omni-30B-A3B-Instruct-4bit (text and talker 4-bit; vision, audio tower, code predictor, and code2wav bf16).
Hunyuan-VL (hunyuan_vl, e.g. HunyuanOCR): Tencent's vision-language family. A ViT with a per-patch conv embedding, bilinearly interpolated learned position embeddings, and full attention over the packed patch sequence feeds a perceive merger: a stride-2 conv pair over the raster grid, a learned image_newline column, a linear to the decoder width, and learned image_begin / image_end rows. Per image that yields mh * (mw + 1) + 2 feature rows, matching the prompt placeholder count exactly. The decoder is the Hunyuan dense stack (per-head Q/K RMSNorm after the rotation, DynamicNTK-alpha rope base) with XD-RoPE at prefill: 4D [P, T, H, W] position ids split across the frequency dims, degenerating to the standard rotation for text; decode uses sequential positions. Validated against hadeseus/HunyuanOCR-mlx-4bit (text 4-bit, vision bf16). Best for OCR: text spotting, document parsing, and grounded extraction.
MiniMax-M3-VL (minimax_m3_vl): MiniMax's vision-language model on top of the MiniMax-M3 text backbone. A CLIP-style ViT (hidden 1280, 16 heads, 32 layers, patch 14, pre_layrnorm, LayerNorm + exact-GELU blocks with separate q/k/v/out projections) runs native-resolution packing: Qwen2-VL-style dynamic smart_resize to patch_size * spatial_merge_size = 28-aligned dimensions, image_grid_thw patchify, per-image cu_seqlens variable-length attention, and 3D (t, h, w) vision RoPE (temporal axis inert for images, trailing dims unrotated). A two-stage projector then maps features to the text width: a per-patch multi_modal_projector (linear_1 -> GELU -> linear_2, into projection_dim 6144) followed by a patch_merge_mlp that folds each spatial_merge_size^2 = 4 adjacent patches (linear_1 [6144, 24576] -> GELU -> linear_2). Each ]<]image[>[ placeholder expands to grid_t * (h/2) * (w/2) tokens (vision_start 200029, vision_end 200030), which the merged features replace LLaVA-style; the MiniMax-M3 hybrid dense/MoE decoder then runs its standard partial 1D RoPE. The vision tower runs in f32 (non-quantized in the checkpoint) while the text tower may be quantized in community exports. Image and multi-image inputs are wired end to end (CLI and server); video is out of scope for this port. The only public checkpoint (MiniMaxAI/MiniMax-M3, 427B) exceeds the development machine, so image Q&A parity is deferred to a runtime validation once a fitting quantized conversion exists; the merge gate is the unit tests plus the real nested-config parse.
Muse Glimmer (muse_glimmer, MuseGlimmerForConditionalGeneration): Meta's 30B VLM, supporting the pinned public BF16 checkpoint meta-models/Muse-Glimmer-30B revision 97c77dff50b2797bcc558fa2d909761dbc575c59 and local path models/mlx/muse-glimmer-30b, plus the MLX affine-Q4 checkpoint mlx-community/Muse-Glimmer-30B-4bit revision 3e7677d7a40d348a3daba263a2b1c0aa41910710 and local path models/mlx/muse-glimmer-30b-4bit. The BF16 weights occupy 59,553,253,376 tensor bytes. The Q4 conversion occupies 19,414,521,856 bytes, packs the text embedding, decoder projections, untied LM head, vision adapter, and vision projection at 4 bits with group size 64, while keeping the 50-layer vision tower dense. The text decoder exposes a 131072-token context window and owns a mixed cache: sliding layers keep a 2048-token rotating KV window while full-attention layers grow across the prompt/context. The vision path supports text-only, single-image, and multi-image CLI/server requests. Each image is smart-resized on the 28-pixel grid, normalized with mean = std = 0.5, duplicated for temporal patch size 2, capped at 4096 merged visual tokens, encoded by the 50-layer Muse tower, pixel-shuffled 2x2, and projected into the 6656-wide text stream. The pinned template's one <|patch|> marker per image expands to image-start, the required number of patch tokens, and image-end before ordered feature scatter. Chat uses the pinned chat_template.jinja; reasoning_strength accepts low, medium, high, and xhigh with high as the default, and tool calls use ATEM. Generation defaults are temperature = 1.0, top_p = 0.95, top_k = 64, and EOS ids 200001 / 200008. OpenAI Chat Completions, Responses, Anthropic-compatible routes, classic CLI generation, continuous batching, streaming, expanded usage accounting, text/image/multi-image, ATEM replay, and reasoning output are wired. Speculative decoding runs on mlxcel-server with the published DFlash assistant drafter (meta-models/Muse-Glimmer-30B-assistant, model_type: muse_glimmer_assistant) on text-only requests at B = 1; see the speculative decoding table below. Unsupported paths are video, quantized vision-tower weights, Turbo/INT8 KV quantization, any other drafter (MTP or a non-Muse DFlash-family drafter), LoRA/adapters, TP, PP, XLA/IREE/OpenXLA, and distributed/disaggregated serving. Quantization support is checkpoint-format specific; the pinned mlx-community affine-Q4 layout is the qualified contract, not arbitrary GGUF or external packings.

The 2026-08-11 real-checkpoint gate ran on Linux/aarch64 with an NVIDIA GB10 (CUDA 13.0, driver 580.173.02). Greedy CLI text generation was coherent at 4.25 decode tok/s; a 2204-token prompt crossed the sliding window at 46.47 prefill tok/s; one- and two-image prompts grounded the orange fixture; and server requests returned expanded prompt usage, ATEM calls, tool-result replay, and isolated answers under scheduler parallelism 1 and 2. During the cold two-image concurrency gate, system MemAvailable fell by at most 59.608 GiB and process VmHWM reached 4.136 GiB. MLX's allocator counters report zero on this CUDA backend, so those OS measurements are recorded explicitly rather than presented as allocator/device-only memory.

The 2026-08-12 affine-Q4 smoke gate used the pinned mlx-community revision on the same GB10. A 69-token text prompt prefixed at 7.22 tok/s and decoded 64 greedy tokens at 13.21 tok/s on the first measurement. A warm rerun reached the final answer The capital of France is Paris. after 152 generated tokens, prefilling at 12.43 tok/s and decoding at 13.34 tok/s. A real image expanded to 64 patch tokens, produced an accurate solid orange-red description, and measured 5.80 tok/s over the 130-token multimodal prefill plus 13.15 decode tok/s; that cold image run included first-use CUDA kernel compilation. This is approximately 3.1x the BF16 text decode rate above. The CLI also separates Muse's to=self reasoning from to=user content, hiding reasoning and envelope tokens unless --show-reasoning is requested. A clean final-source release rebuild repeated the same answer with no leaked control tokens at 12.97 decode tok/s; its cold prefill was 3.15 tok/s, showing why cold and warm prefill are reported separately.
FastVLM (llava_qwen2 / fastvlm): Apple's low-latency VLM. A FastViTHD hybrid encoder runs entirely on channels-last maps: a conv stem, three RepMixer stages (depthwise token mixing plus a BatchNorm ConvFFN), two attention stages with channel LayerNorm and head_dim 32 self-attention, inter-stage large-kernel PatchEmbed downsamples, RepCPE position encoders, and a squeeze-excite conv_exp head. Each 1024x1024 pad-to-square image becomes a (16, 16, 3072) map flattened to 256 tokens, projected to the Qwen2 decoder width by an mlp2x_gelu MLP. The <image> placeholder is the fixed -200 sentinel (not a vocabulary token); the runtime splices one sentinel per image, expands it to 256 tokens, and scatters the image embeddings (LLaVA merge). The text decoder is stock Qwen2, reused unchanged. The loader accepts both the genuine (apple/FastVLM-0.5B) and converted (mlx-community/FastVLM-0.5B-bf16) weight layouts. Best for fast image description and grounded chat.
GLM-OCR (glm_ocr): document-OCR sibling of GLM-4V. A 24-block ViT (3D patch embedding, per-head q/k RMSNorm on the packed cu_seqlens attention, 2D vision RoPE, Conv2d spatial downsample, SwiGLU patch merger) feeds a 16-layer GLM-4 text decoder driven by full-width even/odd MRoPE (rope_parameters with mrope_section [16, 24, 24], partial_rotary_factor 1.0). The tower has no learned position embedding or post-conv norm, and the loader drops the next-n prediction (MTP) layer. Patches are reordered from the processor's raster order into spatial-merge-window order so the rotary, downsample, and merged-token grid stay spatially aligned (OCR reads scrambled patches wrong). Best for plain text, tables, and formula recognition.
Youtu-VL (youtu_vl): SigLIP2 windowed-attention vision tower feeding the Youtu MLA text decoder. The same decoder loads on its own for the text-only Youtu-LLM checkpoint; see the Youtu-LLM entry under Text and hybrid model families.
Kimi-VL and Kimi-VL 2.5 (kimi_vl / kimi_k25): MoonViT native-resolution vision encoder (Conv2d patch embedding, learned plus bicubically-interpolated 2D position embedding, a shared 2D rotary embedding, block-diagonal cross-image attention, and spatial_merge_size patch merging) feeding a LayerNorm -> Linear -> GELU -> Linear connector into a DeepSeek-V3-style MoE text backbone. Detected, loaded, and served end to end: the safetensors directory loader wires the MoonViT tower, the connector, and the DeepSeek-V3 MoE backbone; the native-resolution processor patchifies each image; and the runtime expands each <|media_pad|> placeholder into (h/merge) * (w/merge) tokens before the merged vision features replace them. Video is also supported, on both the CLI (--video) and the server (video_url content blocks): frames are extracted with ffmpeg (uniform sampling, default 2.0 fps, matching the Gemma 4 video path) and patchified per frame, and the encoder treats a clip as a 3D (t, h, w) patch grid rather than t independent images. A computed (not learned) temporal sinusoid is added to the tiled 2D position table, the 2D rotary tables repeat per frame, attention over a clip is one block-diagonal segment spanning all its frames, and the patch merger mean-pools over the temporal axis before the usual spatial_merge_size 2x2 merge, so the placeholder expansion (h/merge) * (w/merge) is unchanged by t. A single-frame clip is equivalent to the same frame processed as an image once the frame-0 temporal constant folds into the patch-embed bias.
Kimi K3 (kimi_k3 with vision_config and vision_tower.* tensors, KimiK3ForConditionalGeneration): the MoonViT3D native-resolution tower (vision_config.model_type: moonvit3d) in front of the Kimi K3 text backbone described in the text section. Images go through the navit rule of the checkpoint's preprocessor_config.json: s = min(1, sqrt(65536 / patches), 7168 / w, 7168 / h), a bicubic resize to floor(w * s) x floor(h * s), the alpha channel flattened after the resize onto an 8 px chessboard (white 255, gray 180, white top-left), black padding on the right and bottom to multiples of 28, (x / 255 - 0.5) / 0.5 normalization, and a [gh * gw, 3, 14, 14] patchify, so a 4000x3000 photo keeps its size, pads to 4004x3024 and costs 216 * 286 / 4 = 15444 tokens while a 100x100 icon pads to 112x112 and costs 16. The tower is a 27-block ViT with RMSNorm at eps 2^-7, a fused wqkv whose qkv_hidden_size (1536, 12 heads of 128) is wider than the 1024 hidden size, the MoonViT 2D rotary table (32 frequencies alternating x and y, applied as interleaved pairs, shared with the Kimi-VL encoder), a tanh-GELU MLP, a 64x64 learnable position grid resampled bilinearly with half-pixel sampling to each image's patch grid (Kimi-VL's is bicubic) plus a fixed sincos time table for multi-frame grids (t > 1 only), and attention that runs per image rather than through a block-diagonal mask. After the final norm the sd2_tpool merger groups each 2x2 patch block and averages over frames, and the patchmergerv2 projector (Linear(4096, 4096) without bias, erf GELU, Linear(4096, 7168) without bias, RMSNorm at 1e-5) produces one 7168-wide row per merged token. Each image renders as <|media_begin|>image {w}x{h}<|media_content|> followed by gh * gw / 4 <|media_pad|> control ids and <|media_end|>, with w x h the original size; the XTML renderer emits that block for every image_url content part on /v1/chat/completions (sized from the image header before the render), mlxcel generate --image splices the same block in front of the raw prompt, and the projected rows replace the <|media_pad|> positions (id 163605). A prompt whose placeholder runs disagree with the grids the processor produced is refused rather than scattered. The tower and projector are bf16 in the published checkpoint and stay bf16 on Apple Silicon because the checkpoint counts as quantized for the dtype policy (its mxfp4 experts are declared as a compressed-tensors quantization_config), the same decision the text backbone makes. One request's images together may ask for at most 32768 media tokens (MLXCEL_KIMI_K3_MAX_MEDIA_TOKENS raises or lowers it), because the navit rule alone lets one image reach nearly 17000 and the tower's attention is quadratic in each image's patch count; images are preprocessed, run and projected one at a time rather than as one batch tensor. An image_url part on a tool message is refused, since a run of tool results is reordered to the assistant's call order while images stay in wire order. Validation: the seven unit tests named in #1342 plus an #[ignore]d kimi_k3_tower_real_weights harness that runs the real tower and projector shards (00095 and 00096 of the published layout, about 1.5 GB) on tests/fixtures/kimi_k3_vision/navit_probe.png, a 303x181 crop that pads on both axes and repeats no patch, against tests/fixtures/kimi_k3_vision/reference.json, a dump tests/fixtures/kimi_k3_vision/generate_reference.py computes in numpy from the same shards without touching the Rust code. The full model is not runnable on one host (see the text entry and #1734), so the end-to-end path is exercised on a layer-truncated local copy as a load, placeholder-count and finite-logits gate only; the answer such a copy produces means nothing. Video (<|video_pad|>, temporal_merge_kernel_size, sample_fps) is out of scope, as the published processor rejects it; a video_url block reaches this family through the frames fallback as ordered images.
LocateAnything (locateanything): NVIDIA's generative grounding VLM. The same MoonViT native-resolution tower Kimi-VL uses, reused unchanged apart from two documented deltas its upstream implementation carries (LayerNorm eps 1e-5 instead of 1e-6, and the tanh-approximate GELU in the block MLP instead of the exact erf form), feeding a LayerNorm -> Linear -> GELU -> Linear connector into a Qwen2 text decoder. The connector normalizes over the flattened merged patch (vision_hidden * merge_h * merge_w, 4608 for the released 2x2 checkpoint) rather than over the vision hidden dim, which is where it differs from the Kimi-VL projector. The image processor is also its own: each side is resized up to the next multiple of merge * patch (Kimi-VL centre-crops down instead), normalization is the plain mean = std = 0.5 rescale, and the per-image patch budget is 25600. Each image expands to <img> + <IMG_CONTEXT> * (grid_h * grid_w / (merge_h * merge_w)) + </img>, and the connector's feature rows replace the <IMG_CONTEXT> (151665) positions LLaVA-style. Grounding output is ordinary text: the model interleaves <ref>/</ref>, <box>/</box>, and the 1001 coordinate tokens <0>..<1000> (ids 151677..152677) into its answer, so plain autoregressive decode is sufficient and no special detokenization runs. Two things about the released conversion needed handling beyond the model graph. It is a mixed_4_8 quantization, not uniform 4-bit (the model card explains why: pure 4-bit on the tied embed_tokens destroys coordinate-token precision), so 18 of the 36 layers store v_proj at 8 bits while q_proj / k_proj stay at 4. The shared fused QKV loader handles that itself (issue #1090): a layer whose three planes disagree on (bits, group_size) keeps them separate in the widths the checkpoint stored them at, rather than dequantizing them (which this loader used to do, at about 190 MB) or requantizing the narrow ones onto a grid they do not land on. And it ships only the slow-tokenizer files (vocab.json + merges.txt), never an exported tokenizer.json, so the tokenizer is reconstructed the way transformers' Qwen2Converter does: byte-level BPE, NFC normalizer, the Qwen2Tokenizer pre-tokenize regex under an isolated Split, and ByteLevel decode/post-process. Validated against mlx-community/LocateAnything-3B-4bit: on the model card's own example image the first emitted box is byte-identical to the output it documents, and a referring query resolves to within a few normalized units of the COCO ground-truth box. The checkpoint's parallel box-decoding head (pbd, n_future_tokens = 6) and coordinate-token-to-box post-processing are not implemented; they are a follow-up.
LLM-jp-VL (llmjpvl): the LLM-jp lab's Japanese VLM family. One architecture covers two released checkpoints that differ only in the text backbone llm_config.model_type names: llm-jp/llm-jp-4-vl-9B-beta (llama, hidden 4096, 32 layers, vocab 196608, untied LM head) and llm-jp/Jagle-VL-2.2B-Jagle-FineVision (qwen3, hidden 2048, 28 layers, tied embeddings, per-head q/k RMSNorm). A SigLIP2-so400m tower at vision_backbone.vision_model.* (512 px, patch 16, 1024 patches per tile, 27 layers of width 1152, gelu_pytorch_tanh) runs behind InternVL-style dynamic tiling, and its post_layernorm output is folded 2x2 by pixel_shuffle(0.5) into [tiles, 256, 4608] before the mlp1 connector (LayerNorm -> Linear(4608 -> hidden) -> GELU -> Linear(hidden, hidden)) projects it into the decoder width. Three details separate the runtime from InternVL's, which it otherwise reuses unchanged. The decoder config lives under llm_config, not text_config. The SigLIP tower has no CLS token, so nothing is stripped from the tower output (InternViT's [:, 1:] slice would drop a real patch here); the checkpoint's attention-pooling head.* is unused and dropped at load. And the mlp1 LayerNorm takes torch's default epsilon of 1e-5, because upstream builds a bare nn.LayerNorm(...) there, rather than the 1e-6 in vision_config.layer_norm_eps. Normalization is SigLIP's (mean and std 0.5), not ImageNet's. Each image is framed as <|image_start|> + <|image_pad|> * (256 * tiles) + <|image_end|> spliced into the Harmony user turn where the checkpoint processor's <image> placeholder sits, and the tile budget is recomputed per request from what the prompt leaves of the 4096-token context: max_num = ((model_max_length - text_tokens) // images - 2) // 256 - 1, clamped into [1, max_dynamic_patch], so a short prompt saturates at 12 tiles plus a thumbnail. The two checkpoints disagree on every image and stop id (14 / 15 / 16 and stops 2, 11 for the 9B; 151655 / 151669 / 151670 and stops 151675, 151645, 151672 for Jagle-VL), so all of them are resolved from the checkpoint's own added_tokens_decoder and generation_config.json rather than from constants. The shipped chat_template.jinja ends a generation render at <|start|>assistant; the checkpoint's Python processor appends <|channel|>final<|message|> after it, so mlxcel completes the render the same way on both the CLI and the server, and a bare template render is never fed to the model. select_layer other than -1 (intermediate-layer features) is rejected at load with the value named, because serving last_hidden_state instead would be features the checkpoint was not trained against. Video input is out of scope and not implemented.
GOT-OCR 2.0 (GOT): a 0.58B document-OCR VLM, released as stepfun-ai/GOT-OCR2_0 and converted as mlx-community/GOT-OCR2_0-{bf16,8bit,4bit}. A SAM-style ViT-B tower (windowed / global attention with a decomposed relative-position bias, a two-conv neck, and a two-stage stride-2 conv compressor) runs on a fixed 1024x1024 canvas and emits a 16x16 grid of width 1024, so every page becomes exactly 256 feature rows; a single Linear(1024, 1024) projector maps them into a Qwen2-0.5B decoder (24 layers, width 1024, q/k/v biases, tied embeddings). The tower is the DeepSeek-OCR one reused unchanged, and the decoder is served by the Llama-family backbone. There is no tiling, no aspect-ratio preservation and no thumbnail: the image is stretched to the square canvas with bicubic filtering and normalized with the CLIP mean/std, which is why the image-token count is a constant rather than something derived from the image. Two key layouts load through one canonicalizer: the original's model.vision_tower_high.* / model.mm_projector_vary.* / model.* with the tower neck at its nn.Sequential indices, and the conversions' vision_tower.* / multi_modal_projector.* / language_model.model.* with the neck renamed to conv1 / norm1 / conv2 / norm2; the tied lm_head copy the original ships is dropped at load. The checkpoint ships no chat template, so mlxcel carries the fixed conversation modeling_GOT.py::chat builds, on both front ends: a constant system turn (You should follow the instructions carefully and explain your answers in detail., with the eight spaces of the reference's source indentation), then <img> + <imgpad> * 256 + </img> and the instruction in the user turn. A client-supplied system message is replaced by the fixed one, because the vision features are scattered into a prompt whose prefix the model saw on every training example. The tokenizer is QWen tiktoken (qwen.tiktoken, 151643 ranks plus 217 specials); the loader picks the QWen special table off tokenizer_class rather than the HunYuan default, which is what gives <|im_end|> 151645 and <imgpad> 151859. Stop ids are [151643, 151645]: config.json declares only <|endoftext|>, which an OCR answer never emits, and upstream stops on the <|im_end|> turn separator through a KeywordsStoppingCriteria instead, so without it generation runs to the token cap. One image per request. The four prompting modes are the instruction text itself: OCR: for plain text, OCR with format: for markdown / LaTeX, [x1,y1,x2,y2] OCR with format: for a region (coordinates normalized to 0..1000), and [red] OCR with format: for a color-marked region. Multi-crop OCR and the model card's rendered-HTML helpers are out of scope.
SmolVLM, SmolVLM2, and Idefics3 (smolvlm / idefics3): one shared Idefics3-family runtime, a SigLIP vision tower + pixel-shuffle token compression + a Llama text backbone (SmolLM2 for SmolVLM, Llama-3 for Idefics3-8B-Llama3). Both on-disk layouts load: SmolVLMForConditionalGeneration (model.text_model.* + a top-level lm_head) and Idefics3ForConditionalGeneration (the whole Llama-with-head nested under language_model.*). Image splitting follows preprocessor_config.json: when do_image_splitting is enabled, mlxcel resizes the longest edge to size.longest_edge, clamps oversized intermediate dimensions to 4096, rounds the canvas upward to exact max_image_size.longest_edge tile multiples, bilinearly resizes, emits row-major square crops, and appends the global thumbnail tile last; images that fit one tile use the single aspect-preserved padded tile path. Split prompts match the upstream framing by tokenizing <fake_token_around_image><row_r_col_c> before each tile run, ending each row with a newline, and appending \n<fake_token_around_image><global-img> plus the global run and closing fake marker; single-tile prompts remain <fake_token_around_image><global-img><image>*N<fake_token_around_image>. Each tile contributes exactly num_image_token compressed feature tokens, so the number of <image> ids remains total_tiles * num_image_token with row-major feature order and the global tile last.
Idefics2 (idefics2): SigLIP vision tower + a perceiver-resampler connector (a modality_projection SwiGLU MLP into the text hidden size, then n_latents learned query slots refined by grouped-query cross-attention over the image patches) + a Mistral text backbone. Each image contributes a fixed n_latents (64) compressed feature tokens regardless of the patch grid, and the <image> placeholder is framed by <fake_token_around_image> and expanded to those tokens. This first port feeds a single full-resolution square tile per image; the optional do_image_splitting high-resolution tiling is a documented follow-up.
MiniCPM-O
Moondream 3
Moondream 2 (moondream2 / moondream1): reuses Moondream3's linear-patch ViT vision tower and overlap-crop preprocessor, paired with a Phi-1.5-style dense text decoder (fused QKV, partial rotary embedding, parallel attention/MLP, tanh-GELU) instead of Moondream3's sparse-MoE decoder. Images are split into a resized global crop and a grid of overlapping local crops; the local crop features are trimmed of their overlap margins, stitched, and adaptively average-pooled back to the 27x27 encoder grid before being concatenated with the global features and projected to the text hidden size. The BOS token and the 729 projected image tokens form a bidirectional prefix ahead of the causal text prompt. Checkpoint revisions from 2025-06-21 onwards are trained against the moondream/starmie-v1 tokenizer with Moondream3-style control-token templates (bos = eos = 0), while the official repository still ships the older GPT-2 tokenizer.json next to them; mlxcel detects the revision from the bundled moondream.py, resolves the starmie tokenizer from the Hub (cached after the first fetch, or place starmie's tokenizer.json in the model directory when offline), and keeps the GPT-2 tokenizer with Question:/Answer: framing for the 2025-01-09 .. 2025-04-14 revisions where that contract is the correct one.
Florence-2 (florence2): Microsoft's task-prompted vision foundation model, an encoder-decoder (seq2seq) VLM. A DaViT tower (4 stages, window + grid attention) encodes the 768x768 image into a [1, 576, 1024] feature grid; a learned 2D position embedding, a cosine temporal embedding, and the image_feature_source pooling recipe produce 577 projected tokens that are concatenated in front of the task-prompt embeddings (no image placeholder token). The fused sequence runs through a BART encoder, and a causal BART decoder with encoder cross-attention generates the answer greedily from decoder_start_token_id. The -p/--prompt string selects one of fifteen task modes, for example <CAPTION>, <DETAILED_CAPTION>, <OCR>, <OCR_WITH_REGION>, <OD>, <DENSE_REGION_CAPTION>, <REGION_PROPOSAL>, or <CAPTION_TO_PHRASE_GROUNDING> some text for the seven tasks that take input text; markers are case-insensitive and the angle brackets are optional. Spatial answers are decoded through the checkpoint's 1000-bin <loc_*> tokens and parsed into pixel-space boxes, quad boxes, or polygons against the original image size. Because generation is seq2seq (cross-attention against cached encoder output), the family runs through its own CLI pipeline (mlxcel generate -m <model> --image <img> -p '<OD>') rather than the shared autoregressive loop, and mlxcel-server serves it through a dedicated single-stream seq2seq worker rather than the batched/paged scheduler. Over HTTP, send an OpenAI-compatible POST /v1/chat/completions with one user message whose text is the task prompt (the same string the CLI takes via -p) and one image_url content part; the response's message.content carries the identical human-readable text the CLI prints, and the parsed coordinates additionally arrive as JSON in the mlxcel-specific message.florence2_result extension field (see the extension-field note in responses-api.md). Serving is one request at a time by design in this first landing: the encoder pass has a different cost profile from the decode loop and no batched admission policy has been designed or measured for it, so requests queue on the worker channel and are answered serially; the pending queue honors --max-queue-depth, and no concurrent-throughput property is claimed. Decode is greedy, so sampling parameters and a response_format structured-output constraint are accepted and ignored (the answer shape is fixed by the task marker, and the parsed form is returned in florence2_result instead), and a stream: true request delivers the whole rendered answer as a single delta chunk, because the parsed result only exists after the full decode. The input text of the seven input-taking tasks is validated at the request boundary (2048-byte bound, no control characters, strict <loc_a><loc_b><loc_c><loc_d> form for the region tasks, no angle brackets in the free-text tasks), all declared images must resolve, image payloads pass through the same ImageInputLimits decompression-bomb bounds every other family uses, and server usage.prompt_tokens reports the fused encoder length including projected image tokens. Dense and quantized exports both load, one image per request. Validated on mlx-community/Florence-2-base-ft-bf16 and its 4-bit and 8-bit conversions; the 3-bit and 6-bit base-ft conversions use the same packing and load through the same path. In a quantized export the BART projections, the LM head, the DaViT window and channel attention, the tower MLPs, the shared token table, the BART position tables, and the learned 2D image position tables are all packed; image_projection, the cosine temporal buffer, the layer norms, and the conv stack stay dense, because upstream registers those as raw parameters or convolutions that nn.quantize does not walk. A checkpoint that packs one of them anyway is refused at load with the offending tensor named. The -large-ft releases load but do not generate usable text: -large-ft-bf16 returns a run of <s> for <CAPTION> while <OD> and <OCR> work, and -large-ft-4bit does so for every task. Upstream mlx-vlm reproduces both on the same checkpoints, so this appears to be a property of the published large-ft MLX checkpoint/conversion family rather than of this loader; the focused upstream tracker is Blaizzy/mlx-vlm#1840, and the mlx-community/Florence-2-large-ft-4bit model card records conversion from prince-canuma/Florence-2-large-ft with mlx-vlm 0.1.0. Prefer -base-ft until that issue is resolved.
Phi-3 Vision, Phi4MM, Phi4 SigLIP VLM (all three run the phi3 text decoder, so Phi-3 / Phi-4 LongRoPE position scaling applies to them too)
Molmo2 and Molmo-Point
Nemotron-H Nano Omni: ships a Conformer/Parakeet audio encoder and accepts spoken audio from the CLI with --audio <path>. Input audio is resampled to 16 kHz before the encoder. Server-side input_audio in POST /v1/chat/completions is also supported: the audio block is spliced inside the last user turn, before the <|im_end|> end-of-turn marker that the ChatML template uses (id 151 in the released checkpoint).

Inkling dMel audio

Inkling checkpoints with both the visual shell and audio_config accept one WAV through the CLI's --audio <path> argument and one or more OpenAI-compatible input_audio message parts on the server. Both surfaces may combine audio with still images. The shared host boundary decodes, downmixes, and linearly resamples clips to 16 kHz mono before model-specific feature extraction. A request is limited to 16 clips and five aggregate minutes so the 20 audio rows per second and 6,144-wide prepared embeddings remain bounded.

The frontend uses a periodic 1,600-sample Hann window, an uncentered 1,600-point RFFT every 800 samples, and an 80-row librosa-compatible Slaney filterbank. Magnitudes and filter rows accumulate in f32 before log10; each value is assigned to one of 16 centers spanning -7 through 2. Midpoint boundaries are derived in f64 and rounded downward onto the f32 lattice, with strict greater-than comparisons so exact ties remain in the lower bin. Each mel channel has its own 16-row segment of a 1,280-row embedding table. The 80 selected rows are summed, RMS-normalized, and scattered into one <|unused_200053|> position per valid 50 ms frame, after any image rows have been merged.

When a chat template preserves the audio marker, its single placeholder is expanded in place to the valid frame count. The server's text-only flattening path instead synthesizes the full audio wrapper immediately before the final current-user end marker, without crossing system or history boundaries. Audio-bearing requests retain the prepared embeddings through prefill and are excluded from the text-only Inkling MTP burst path. Deterministic CPU tests cover the f64-derived filterbank and bin boundaries, mixed-length masks, padded-row removal, channel summation, RMS normalization, prompt placement and counts, config loading, weight renames, mixed image/audio scatter, CLI/server dispatch helpers, and speculative fallback. The public approximately 153.5 GB affine Inkling checkpoint was unavailable on the validation host, so these tests establish frontend and control-plane parity without claiming a real-checkpoint transcription run.

Audio/video capability is model-specific. The server request types include image_url, video_url, and input_audio content blocks, but a loaded model must advertise support for the corresponding modality. A request that carries video_url and input_audio together needs more than both of those flags: the model must also merge the two into one prompt, which today only gemma4_unified does. Video frame extraction uses the system ffmpeg/ffprobe binaries at runtime.

Video on a model with no native video path

Two different things answer a clip, and the difference is worth knowing before reading a reply.

Native video means the family consumes the clip as a clip: a 3D patch grid, adjacent-frame temporal planes, or per-frame scatter into video placeholders. Gemma 4 VL, Gemma 4 Unified, Inkling, Kimi-VL / Kimi-VL 2.5 and the Qwen-VL families (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-VL MoE, Qwen3.5/3.6/3.8-VL) all do this, each through its own processor, and none of them is affected by the paragraph below.

The frames fallback is what every other checkpoint with a vision tower does (issue #1322). The clip is decoded at the requested fps, evenly subsampled to at most --video-max-frames frames with the first and last always kept, PNG-encoded, and sent to the model as that many ordinary images in chronological order, preceded by the sentence Here is a video as a sequence of N frames in chronological order. so the model reads them as one clip. Several clips get one sentence each, naming that clip's own frame count and placed immediately ahead of that clip's frames. The CLI renders its --video clips after any --image inputs and before the question, so it produces the same prompt as a server body that lists the same clips ahead of its question; a template without image content items receives the sentences and the question joined with no separator on both fronts. On the CLI the frames are written as PNG files into a private per-run directory under the system temp directory (mode 0700, each file 0600 on Unix) and removed when the run ends. Nothing about the image pipeline changes: the template emits one image placeholder per frame, the vision tower sees stills, and the prompt-cache multimodal digest hashes the frame bytes, so two requests for the same clip at the same fps and frame cap share a prefix and a different clip does not.

What this costs and what it does not buy:

The prompt grows by the model's per-image token cost times the kept frame count, which for a 16-frame default on a family that spends 256 tokens per image is about 4,096 prompt tokens.
The frames spend the per-request image budget (--max-images, default 16), so a clip plus the caller's own pictures can exceed it. The refusal names the frame count, the caller's own image count and the limit.
There is no temporal encoding. The model sees an ordered set of pictures and the sentence saying they are one video; it does not receive frame timestamps or motion between frames. Expect it to describe what changes across the frames, not to reason about speed or duration.

The fallback is on wherever the checkpoint has a vision tower and no native path (Gemma 3, LLaVA, LLaVA-NeXT, Pixtral, SmolVLM / Idefics3, LFM2-VL, InternVL, MiniCPM-o, and so on), on the CLI (--video) and on the server (video_url). Muse Glimmer is the one exception: it refuses --video and video_url by name on both fronts until the family is qualified for multi-image prompts. A checkpoint with no vision tower at all keeps refusing video, as before.

GET /props reports video: true for both kinds, because both accept a video_url block. There is no wire-level flag distinguishing them; the server log line model <id> has no native video path; sending K of N sampled frames from <file> as ordered images is what says the substitution happened.

Supported ffmpeg range: 5.0 (2022) or newer, on both the CLI (--video) and the server (video_url). Both binaries must be on PATH; neither is a build-time dependency, and a missing one produces a named error rather than a crash. The server checks for them once, at startup, when the loaded checkpoint takes video, logs a warning if either is missing, and keeps that answer for the life of the process, so install ffmpeg before starting the server. The floor is set by one flag: the extraction command passes -fps_mode vfr, which ffmpeg added in 5.0 at the same time it deprecated the older -vsync. ffmpeg 8 removed -vsync outright, so the previous spelling made every video request fail at argument parsing, before a frame was decoded (#1172). Because -fps_mode is accepted by every release from 5.0 on, mlxcel passes it unconditionally rather than probing the binary or parsing its version banner. Nothing else in the video path depends on a version-gated feature, so 5.0 is the whole requirement, and there is no upper bound: releases through ffmpeg 9.x work unchanged. On ffmpeg 4.x and older, video input is not supported; upgrade the system binary.

Falcon-OCR layout-aware OCR

Falcon-OCR transcribes a whole page in one pass by default. It also has a two-stage mode: detect the page's layout regions, then OCR each region on its own with the instruction that region's class was trained on, which is what produces per-region text instead of one undifferentiated block.

mlxcel does not ship a document-layout detector. The reference's first stage loads PP-DocLayoutV3 through transformers.AutoModelForObjectDetection; that is a separate object-detection architecture and is not part of this port. Only the second stage is implemented, so the region boxes have to come from somewhere else and are supplied as a file:

mlxcel generate -m models/mlx/falcon-ocr \
  --image page.png \
  --layout-detections regions.json

-p/--prompt is unused on this path (it is ignored with a note if passed): every region's prompt is the OCR instruction its layout class maps to. Each region is cropped from --image, OCRed on its own, and printed with its class, score, and pixel box.

The accepted JSON is the shape mlxcel detect --format json prints, so a detector mlxcel gains later feeds this path without a format change:

{
  "detections": [
    {"label": "title", "confidence": 0.95,
     "box": {"l": 60, "t": 55, "r": 940, "b": 140}},
    {"label": "text", "confidence": 0.90,
     "box": {"l": 60, "t": 295, "r": 940, "b": 450}}
  ]
}

A bare top-level array works too, as does the {"category": ..., "bbox": [l, t, r, b], "score": ...} spelling that mlx-vlm's falcon_ocr/layout.py emits. score is optional and defaults to 1.0. Malformed input is rejected by index (detection #3: the bounding box is empty or inverted ...) before the model is loaded.

Class names route through the reference's LAYOUT_TO_OCR_CATEGORY table: table, formula, caption, footnote, list-item, title, and section-header each select their own instruction, aliases such as abstract, doc_title, and figure_title collapse onto the broader category, and the DocLayNet underscore spellings (list_item, page_header, page_footer, section_header) are accepted alongside the hyphenated ones. Classes that carry no text (picture, figure, chart, seal, image) are skipped, as is any class the table does not know. Boxes more than 80% contained in a strictly larger box are dropped so an inline formula is not transcribed twice, and crops under 16 px on a side (or that would fall under it once the long side is capped at the processor's 1024 px) are skipped. When nothing usable survives, the whole page is OCRed as one plain region, matching the reference. A one-line summary reports the counts, so a region that disappeared is visible rather than silent.

Regions are OCRed and printed in the order the file lists them; nothing reorders them. The reference detector emits reading order, and so does mlxcel detect (top to bottom, then left to right), so the two commands pipe together directly:

mlxcel detect -m models/docling-layout-heron-mlx-bf16 -i page.png --format json > layout.json
mlxcel generate -m models/falcon-ocr -i page.png --layout-detections layout.json

A hand-written file should likewise be supplied in reading order.

One page element can arrive under several labels. A DETR-style detection head takes no per-query argmax, so a heading that scores above the threshold as section_header, title, and page_header is emitted three times, each entry carrying the same box; mlxcel detect prints them together, most confident first. That is the detector's intended output, not a duplicate, so the planner collapses same-box entries to one region rather than OCRing the text once per label. The label it keeps is the highest-confidence one that maps to an OCR category, so a box detected as both picture and text is still read as text instead of being skipped on the un-OCRable label.

Phi-3 / Phi-4 LongRoPE position scaling

microsoft/Phi-3.5-mini-instruct, microsoft/Phi-4-mini-instruct and microsoft/Phi-4-multimodal-instruct declare rope_scaling.type as longrope (older conversions spell it su) and ship two per-dimension factor lists rather than one. The trained model rotates with short_factor while the sequence still fits in original_max_position_embeddings (4096 on every shipped checkpoint) and with long_factor above it. The two lists are far apart: on Phi-3.5-mini the low-frequency long_factor entries reach 64.8 where short_factor stays under 2.9, so applying one list at every position mis-rotates every prompt on the other side of the threshold. mlxcel builds both tables at load time and selects per forward pass. The attention-magnitude scale follows the same table: it is short_mscale or long_mscale when the block gives them, and otherwise sqrt(1 + ln(M / L) / ln(L)) for both. original_max_position_embeddings is read from the rope_scaling block first and then from the top level, which is where the shipped Phi configs actually put it.

The selection is made from the whole prompt, not from the tokens in front of the model at that moment. mlxcel reaches the model with a piece of the prompt in several ways: a prefill chunk (2048 tokens on the CLI, --prefill-chunk-size on the server, default 512), the history-boundary segment the prompt cache splits off to snapshot, and the suffix left after a prompt-cache hit. A per-piece decision would rotate the head of a threshold-crossing prompt with the short table and its tail with the long one, leaving one KV cache holding keys built from two different tables. Measured on a 5136-token prompt, that degenerates the output into repetition within a few tokens, on the CLI at the 2048-token default chunk and on the server at its 512-token one. Every scheduler entry point that runs prefill work for a sequence therefore announces the total prompt length, and each piece resolves the same table from it. The one exception is a padded batched prefill, which starts at offset 0 and spans the longest row in its cohort, so the pass already carries the answer; the shorter rows share that cohort's decision the way they already share its padded mask.

A position-selected table also constrains cross-request KV reuse, which is what the server's prompt cache does. A prefix one request encoded under the short table is not valid input for a longer request that will read it under the long table, and the reverse holds too: restoring across that boundary leaves one cache holding keys built from two tables and reads as fluent repetition, not as an error. Phi3Model therefore answers LanguageModel::rope_table_regime, and the prompt cache folds that tag into its bucket identity (the entry digest, the per-session bucket, and the session-independent radix trie), so a lookup only ever matches entries stored under the same tag. A miss across the boundary falls back to a full prefill under the request's own table, which is what transformers does by resetting its cache. An entry whose generation crossed the boundary mid-decode spans both tables and is never donated at all; it is counted as a rope_regime_mismatch reject in /v1/cache/stats. Sharing within one regime is unchanged, so a conversation whose turns stay on one side of the threshold keeps every bit of its prefix reuse.

mlx-lm is not the oracle for this. SuScaledRotaryEmbedding in mlx_lm/models/rope_utils.py accepts short_factor and short_mscale as arguments and then builds its frequencies from long_factor and its scale from long_mscale alone, so it applies the long table at every position. A token-exact comparison against mlx-lm below 4096 tokens therefore disagrees with the checkpoint; the reference that matches the trained model is Phi3LongRoPEScaledRotaryEmbedding in transformers, which is what mlxcel follows here. Anyone re-checking this path against mlx-lm and "fixing" mlxcel back toward it will reintroduce the defect.

One divergence from transformers remains, and it is in decode rather than prefill. When generation crosses original_max_position_embeddings, transformers drops the KV cache and re-encodes the whole sequence with the long table (prepare_inputs_for_generation, the branch on input_ids.shape[1] >= original_max_position_embeddings + 1 and past_length <= original_max_position_embeddings). mlxcel flips to the long table in place at that step and leaves the short-table keys already in the cache untouched, which is an approximation, not a re-encode. It costs nothing for a prompt and generation that stay on one side of the threshold, which covers the shipped 4096-token trained context and every parity test in this repository. Do not claim transformers parity for a generation that crosses the threshold mid-decode.

Phi-4 Multimodal audio

Phi4MM supports the official microsoft/Phi-4-multimodal-instruct checkpoint at revision 93f923e1a7727d1c4f446756212d9d3e8fcc5d81. The CLI accepts one WAV clip through --audio <path>, optionally together with repeated --image arguments. The OpenAI-compatible server accepts multiple input_audio content blocks and can combine them with image_url blocks.

Numbered placeholders such as <|audio_1|> and <|image_1|> determine the exact feature order. When placeholders are omitted, mlxcel synthesizes them; the server's normalized media representation groups audio before images, so callers that require the original content-part interleaving must include the numbered placeholders explicitly. Missing, extra, malformed, or duplicated audio placeholders are rejected rather than silently reordered.

The shared WAV decoder accepts mono or multi-channel PCM and supplies mono samples to the Phi4MM frontend. Sample-rate handling intentionally follows the pinned processor's integer-ratio policy rather than an ideal rational resampler: native 8 kHz and 16 kHz are unchanged; rates above 16 kHz are downsampled by floor(rate / 16000) and then labeled 16 kHz; rates strictly between 8 kHz and 16 kHz use floor(rate / 8000) and are labeled 8 kHz. Thus, for example, the released processor labels 24 kHz input as 16 kHz without changing its samples. This behavior is required for the official frame count and token-exact output. Rates below 8 kHz are rejected. Clips of 40 seconds or less are recommended for checkpoint-quality parity, and clips over 30 minutes are rejected. Audio-only requests select the speech adapter and speech projection. Mixed image/audio requests select the vision adapter and vision projection, while text-only requests use the base language model. The selected mode remains active through decode and is isolated per request.

Gemma 4 image soft-token budget

Gemma 4's vision front-end is resolution-driven: it works over whatever patch grid the preprocessor hands it, with no fixed feature length. The number of soft tokens an image contributes to the prompt therefore follows directly from the resize target, and the resize target follows from a soft-token budget. That budget is read from the checkpoint's config at load time (280 for every shipped Gemma 4 checkpoint) and can be overridden per request.

This applies to both Gemma 4 architectures: gemma4 (the ViT tower, e.g. gemma-4-e2b-it, gemma-4-31b-it) and gemma4_unified (the encoder-free patch projector, e.g. gemma-4-12b-it).

A larger budget keeps more detail (denser patch grid, better small-text and fine-structure reading) at the cost of a longer prompt and more prefill work. A smaller budget is cheaper and faster. Supported values, shared with the video per-frame budget:

Budget	Meaning
70	Smallest. What `detail: "low"` maps to.
140
280	Checkpoint default for the shipped Gemma 4 models.
560
1120	Largest. What `detail: "high"` maps to.

Any other value is rejected with a 400. The budget is untrusted request input and it scales the resized image and its patch grid, so it is validated against this ladder rather than clamped: a silent clamp would leave the caller believing they got a budget they did not get. 1120 is also the hard ceiling for gemma4_unified, whose learned 2-D position table is exactly 1120 wide; a budget past it would index off the end of that table.

Server

Two optional fields on the image_url content part:

{
  "model": "gemma-4-12b-it-4bit",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Transcribe the small print."},
      {"type": "image_url", "image_url": {
        "url": "data:image/png;base64,...",
        "detail": "high"
      }}
    ]
  }]
}

detail is the OpenAI-standard field. "low" maps to 70, "high" maps to 1120, and "auto" (or an absent field) leaves the checkpoint's configured budget in place. An unknown value is a 400, not a silent fallback to "auto".
max_soft_tokens is an mlxcel extension, not part of the OpenAI spec. It names an exact budget from the ladder above and takes precedence over detail when both are present:
```
{"type": "image_url", "image_url": {"url": "...", "max_soft_tokens": 560}}
```

Both fields are optional and default to absent, so a request that sends only url behaves exactly as it did before these fields existed. The budget applies to the request as a whole; when a request carries several image_url parts they must agree on an explicit budget (parts that set neither field are free to coexist with parts that do). Every non-Gemma-4 model ignores both fields.

Note that the budget is part of the vision-feature cache identity, so the same image requested at two budgets is encoded twice rather than served the wrong cached features.

CLI

mlxcel generate --image-soft-tokens <N> takes the same ladder and the same validation:

./target/release/mlxcel generate -m models/gemma-4-12b-it-4bit \
  --image receipt.png --image-soft-tokens 1120 \
  -p "What is the total?"

Omitting the flag uses the checkpoint's configured budget. The flag applies to --image inputs only; --video frames keep their own (smaller) per-frame budget.

Thinking default for `gemma4_unified`

Gemma 4 Unified (gemma4_unified) ships <|channel> / <channel|> thinking markers in its tokenizer, so the server defaults enable_thinking=true for this family on startup, mirroring ml-explore/mlx-lm#1114. With thinking on, the model writes an internal scratchpad before the visible reply, so simple prompts spend more of the budget on reasoning than on the answer. A one-sentence answer can take roughly 275 completion tokens, and a default max_tokens of 64 to 80 may return an empty content with finish_reason of length because the whole budget went to thinking. Set max_tokens to at least 512 for this family, and higher for multi-sentence answers.

The scratchpad is no longer dropped: Chat Completions surfaces it as reasoning_content and, by default, an identical OpenRouter-compatible reasoning alias on both streaming deltas and non-streaming assistant messages. Both fields are omitted when the model produces no reasoning. Set --reasoning-alias-field none (or MLXCEL_REASONING_ALIAS_FIELD=none) to retain only reasoning_content when response bytes matter. This applies to every thinking family, including Qwen-style <think> models. To turn thinking off rather than only suppress its alias, pass chat_template_kwargs={"enable_thinking": false} per request, or set the server default via --chat-template-kwargs or LLAMA_ARG_CHAT_TEMPLATE_KWARGS. A per-request value always wins over the server default.

CLI reasoning display (`--show-reasoning`)

mlxcel generate and mlxcel run decode with special tokens intact, so a thinking checkpoint's raw channel markers and chain-of-thought would otherwise print to the terminal as if they were the answer. Both commands now hide the reasoning channel by default and print only the final answer, for both Gemma 4's <|channel>thought / <channel|> pair and Qwen-style <think> / </think> models. Pass --show-reasoning to also print the reasoning body (dimmed on a terminal); the raw channel markers never print either way. This is a separate axis from enable_thinking / chat_template_kwargs above: that controls whether the model generates reasoning at all, while --show-reasoning only controls whether the CLI displays reasoning the model already generated.

`thinking` alias for the DeepSeek-V3.2 (`deepseek_v32`) chat template

The upstream deepseek-ai/DeepSeek-V3.2-Exp chat template gates its <think> block on a bare thinking boolean rather than the conventional enable_thinking kwarg that mlxcel forwards by default (verified against deepseek-ai/DeepSeek-V3.2-Exp assets/chat_template.jinja). Because the template only ever reads thinking, enable_thinking alone would have no effect on this family. mlxcel detects the {% if not thinking is defined %} idiom in the loaded template and, when present, also mirrors the fully-resolved enable_thinking value (request override, or the server default) into a thinking key so toggling reasoning works the same way it does for every other thinking-capable model. The detection is based on the template source, not the model name, so any future template that adopts the same idiom is covered automatically. GLM-5.2 (glm_moe_dsa) ships its own chat template that already reads enable_thinking directly and is unaffected. An explicit thinking entry in chat_template_kwargs always overrides the derived alias.

`thinking_mode` alias for chat templates gating on a string variable

Some chat templates check a thinking_mode string (for example thinking_mode == "enabled") instead of the enable_thinking boolean, so mlxcel also detects that identifier in the loaded template and injects thinking_mode: "enabled" whenever the resolved enable_thinking value is true. Nothing is injected when thinking is off, and an explicit thinking_mode entry in chat_template_kwargs always overrides the derived value.

`reasoning_effort` and templates that validate their own kwargs

The OpenAI-standard top-level reasoning_effort request field resolves both whether thinking is enabled and which level value reaches the loaded chat template. After trimming and lowercasing for comparison, none, off, disabled, false, and 0 set enable_thinking=false and are not forwarded as a level; every other non-empty value sets enable_thinking=true and is forwarded verbatim as reasoning_effort when the template reads that name, or as reasoning_strength when the template reads that alias (including Muse Glimmer). Templates that read neither level name still receive the derived enable_thinking switch but no level kwarg. The same resolver accepts reasoning: {"effort": "..."} and boolean reasoning values in compatible extra_body fields.

Precedence remains per key: an explicit per-request chat_template_kwargs entry wins, then the derived request control, then the server-wide --chat-template-kwargs default. The portable request therefore cannot overwrite an explicit enable_thinking, reasoning_effort, or reasoning_strength, while unrelated defaults persist.

The value is passed through verbatim. OpenAI's vocabulary is minimal / low / medium / high, and a checkpoint's vocabulary need not match: Qwen3.8 accepts xhigh / medium / low, so high is valid OpenAI and invalid there while xhigh is the reverse. mlxcel does not translate between the two. The value sets the model's reasoning budget, and picking a different budget than the caller asked for is a silent rewrite the caller cannot detect.

A template that refuses the value calls Jinja's raise_exception, and that now fails the request with 400 carrying the template's own message, which names the set the template accepts. Before, the refusal was swallowed: rendering fell back to a plain User: ... Assistant: prompt with no chat framing, no system message, and no tool declarations, and the model answered from it with 200. Only a server-side WARN recorded the degradation.

The 400 is specific to a deliberate refusal, not to render failures in general. mlxcel tells the two apart at the type level: raise_exception attaches a TemplateRejection sentinel as the error's source, so a template that mlxcel genuinely cannot render (an unimplemented filter, a malformed template) still degrades to the plain prompt exactly as it did before. The type is pub, so it is nameable outside the crate (mlxcel::server::chat_template::TemplateRejection), but its constructor and its field are both private, so nothing outside the render path can construct one and forge a false rejection. The rule applies to any template that validates a caller-supplied value, not just to reasoning_effort: templates that reject an enable_thinking value, a tool-choice value, or an unsupported role behave the same way.

The compiled MiniJinja environment is cached on the ChatTemplateProcessor for the lifetime of the loaded template. The typed-message renderer, raw JSON renderer, history renderer and probe renders all look up the same compiled "chat" template instead of rebuilding the environment, filters and bytecode on each call. The cache stores only the operator/model-supplied template and environment configuration; per-request messages, tools and kwargs are still rebuilt for every render, so request isolation and the TemplateRejection source-chain behavior above remain unchanged.

The Responses API's reasoning.effort feeds this same resolver and remains echoed unchanged on the response object. Its derived controls likewise override server-wide template defaults per key.

`tojson` in chat templates behaves like Python `json.dumps`

Chat templates published on HuggingFace are written against transformers, where Jinja's tojson is CPython's json.dumps with ensure_ascii=False prefilled. mlxcel provides its own tojson filter with those semantics rather than the generic Jinja one, so a template renders the same prompt here that it does under the reference renderer.

The filter accepts ensure_ascii (default false), indent (default none; true means 2), separators (a two element tuple, default (", ", ": "), or (",", ": ") when indent is set), and sort_keys (default false, sorting object keys recursively by Unicode code point). Object keys otherwise keep the order the client sent them in. Floats are formatted like CPython's repr, so an integral float keeps its .0, 1e-05 and 1e+16 use exponent form with a signed two digit exponent, and the non-finite values render bare as NaN / Infinity / -Infinity. Jinja's HTML-safe variant is not applied: <, >, & and ' reach the prompt as themselves, which is what a JSON Schema pattern full of comparison operators needs. An unrecognized keyword argument is still an error rather than being ignored.

Three limits bound the filter, because the value it serializes arrives in the client request body while the arguments come from the template. indent is clamped to 64 columns, nesting deeper than 192 levels is refused, and a single call will not emit more than 64 MiB. All three sit far above any real template: tool schemas render in kilobytes, and the depth ceiling deliberately leaves room above the 128-level limit the request parser itself applies, so that a message tree carrying separately parsed function.arguments cannot trip it. Crossing any of them fails the render, which costs the prompt its tool declarations, so the limits are set to catch runaway amplification rather than to police ordinary input.

Two consequences are worth knowing about. Templates that call tojson with json.dumps keyword arguments (poolside/Laguna-XS-2.1 uses ensure_ascii=False, thinkingmachines/Inkling-Small uses sort_keys=true, separators=(",", ":")) render their tool declarations and tool-call history instead of failing and degrading to the plain User: ... Assistant: prompt. And a bare x | tojson writes {"a": 1, "b": 2}, with the spaces the reference renderer emits, so tool schemas tokenize the way the model provider measured them.

Tiktoken vocabularies: two families, and Kimi K3's native XTML chat format

A checkpoint that ships a .tiktoken BPE vocabulary instead of a tokenizer.json goes through mlxcel's own tiktoken loader, which now serves two families selected from config.json. Anything that is not recognized as Kimi K3 loads as HunYuan and behaves exactly as it did before: the HunYuan pre-tokenization pattern, the five named specials plus <|extra_N|> filler, and the tokenizer_config.json override. A directory whose model_type is kimi_k3, or kimi_linear alongside a tiktoken.model file, loads as Kimi K3: the K3 pre-tokenization pattern (a Han-first alternation with digit runs capped at three), and 256 control tokens above the BPE ranks named from added_tokens_decoder with <|reserved_token_N|> for the unnamed ones.

Kimi K3 also ships no Jinja chat template. Its chat format is XTML, a tag language built from the control tokens <|open|>, <|close|>, <|sep|> and <|end_of_msg|> that the reference implementation renders in code, so mlxcel renders it in code too (src/server/kimi_k3_chat.rs) rather than through the template engine. The renderer emits token ids directly: tag names, attribute names and attribute values are ordinary text segments and only the four structural markers become control ids, so a <|open|> written into a user message is byte tokens and cannot inject structure into the prompt. The rendered ids reach generation through the pre-tokenized request path, and the equivalent text form is kept for diagnostics and for the prompt cache key. That guarantee covers chat message bodies specifically: /v1/completions, /tokenize and mlxcel generate -p still parse control-token spellings out of the raw prompt they are given, which is the same parse_special behavior every other family gets and matches llama-server, so a deployment that exposes those endpoints to the same callers a chat-level system prompt is meant to be trusted over has no boundary there.

Two request features are refused rather than silently mis-rendered on this format: a --prefill-assistant continuation (the XTML generation prompt has no text tail to append to) and declared image, audio or video media. Image support arrives with the vision half of the family.

Loop-detection default-on for the Gemma 4 family

Gemma 4 (31B Dense and 26B-A4B MoE, including the QAT 4-bit checkpoints) has an upstream, weights-level token-repetition collapse documented in google-deepmind/gemma#622 and reproduced across other engines, including vllm-project/vllm#40080. Generation can degenerate into a single repeated token (or short fragment) that fills the token budget, most often inside the thought channel. Tool declarations and json_schema structured-output requests amplify it strongly, and those are the conditions #432's reproduction could trigger reliably, but the defect is upstream and in the weights, so a collapse without either is possible. Sampling penalties do not reliably recover it, because once the logits collapse the top-k candidates are themselves garbage.

mlxcel applies vLLM-style N-gram loop detection to break this, default-on for the Gemma 4 family with no configuration required on tool-shaped requests. For any model in the family (Gemma4, Gemma4VLM, Gemma4Unified), the engine applies the conservative threshold min_pattern_size=1, max_pattern_size=20, min_count=12 when tool declarations reach the rendered prompt (tools non-empty and tool_choice other than none) or message content carries tool_calls / tool_call_id. A genuine repetition collapse on that traffic ends early with finish_reason of stop. A downstream serving app and its users still need no setup and see no toggle for the protected tool-call surface.

Plain Gemma 4 chat, grammar-only structured-output requests, plain completion requests, and requests that declare tools but send tool_choice: "none" and carry no tool-shaped messages keep the disabled baseline. The tool_choice case reads the same helper the template uses to select tools, so a request whose declarations are never rendered is treated exactly like the plain chat it is indistinguishable from. A follow-up turn that replays prior tool_calls or tool results stays amplified regardless of tool_choice, because those fields reach the prompt on the raw-JSON render path.

This is a deliberate trade, not a claim that unprotected collapses cannot happen. Running the detector on plain chat cost a class of false positives that silently truncated correct answers: a markdown table alignment row (| :--- | :--- | :--- | :--- |) tokenizes as one 3-token cell block repeated once per column, so at the previous min_count=4 any table with 4 or more columns was cut mid-row while the client saw a normal stop. Grammar constraints have the same fundamental ambiguity: a schema-valid array can legitimately repeat one value beyond any fixed threshold, so token-level detection cannot tell it from a collapse. An operator who wants blanket coverage can still use MLXCEL_LOOP_DETECTION=on, and the per-request max_pattern_size / min_pattern_size / min_count fields override everything per call. MLXCEL_LOOP_DETECTION=1,20,4 restores the pre-#967 behavior exactly.

Raising min_count from 4 to 12 has two costs, not one. A pattern of size p that would previously have been cut after 4p tokens is now cut after 12p, so a genuine collapse runs 8p tokens longer: 8 extra tokens for the single-token collapse in the known reproduction, 160 at the p = 20 ceiling. Separately, firing at size p now needs p * 12 tail tokens rather than p * 4, so a pattern is undetectable whenever the remaining budget is shorter than p * 12. With thinking_budget_tokens = 128, for example, patterns of p >= 11 can no longer be caught where p * 4 = 44 tokens previously sufficed. Short-budget requests therefore lose coverage of long patterns, though not of the short ones that dominate real collapses.

Grammar-only requests no longer arm the family default. The measured json_schema reproduction—asking gemma-4-12b-it-4bit for an integer array containing 0 repeated 30 times under a schema requiring that array—therefore runs with loop detection disabled and can complete as valid, schema-conformant JSON. The grammar continues to constrain every sampled token; this change removes only the independent token-level early-stop heuristic from that surface.

Explicit overrides remain authoritative. If an operator globally enables loop detection or a client opts in per request, the same legitimate-repetition ambiguity still applies; a client can send max_pattern_size: 0 to disable detection for that call.

Every non-Gemma-4 model defaults to disabled (bit-exact baseline preserved). The behavior is still tunable on top of the default, and neither tuning surface is subject to the gate: a per-request override (the vLLM max_pattern_size / min_pattern_size / min_count fields, including max_pattern_size=0 to opt out) wins over everything, and a global operator override (MLXCEL_LOOP_DETECTION) can tune, force-enable for any model and any request, or force-disable. See Generation loop detection for the field semantics and the full precedence order.

Speech-to-text (ASR)

mlxcel loads Whisper-style encoder-decoder ASR checkpoints (model_type: "whisper") and serves them through the OpenAI audio endpoints. A convolutional audio encoder builds features from a 30-second log-mel window, and an autoregressive text decoder cross-attends to those features as it emits tokens, steered by the multilingual transcribe/translate task tokens.

When the server's loaded checkpoint is detected as Whisper, the speech-to-text slot is populated and POST /v1/audio/transcriptions (transcribe in place) and POST /v1/audio/translations (translate to English) return the recognized text. Uploaded audio is decoded with the shared WAV reader, resampled to 16 kHz, and processed in consecutive 30-second windows. An explicit language hint is honored; otherwise the language is detected from the first decoder step. Token suppression follows the Whisper rules: suppress_blank, the non-speech symbol set, and <|notimestamps|>.

This first port targets non-quantized (fp16/f32) checkpoints with greedy decoding, and the loader accepts both the native MLX and HuggingFace key layouts. Loading a Whisper checkpoint serves speech-to-text only; chat generation is not available in the same process. Beam search, word-level and segment timestamps, quantized checkpoints, and streaming transcription are follow-ups.

Text-to-speech (TTS)

mlxcel loads the Kokoro-82M model (a StyleTTS2 phoneme-to-mel acoustic model with a built-in iSTFTNet vocoder) and serves it through POST /v1/audio/speech. The path is: text to phonemes (grapheme-to-phoneme front-end) to a PLBert text encoder, a duration predictor that expands per-token features to per-frame, F0 (pitch) and energy prosody, and an iSTFTNet decoder that produces a 24 kHz mono waveform directly via an inverse STFT (no separate neural codec).

Detection works without a top-level model_type: the loader recognizes a Kokoro checkpoint by the istftnet config block or the kokoro-v1_0.safetensors weight filename, so -m <kokoro-dir> resolves to the TTS provider. The voice request field selects a pack from voices/<name>.safetensors (54 voices; default af_heart), validated against the available packs with a safe fallback. speed scales the predicted durations (larger is faster and shorter). response_format accepts wav today (returned via the shared WAV writer); other containers are a follow-up.

The grapheme-to-phoneme front-end is a self-contained American-English phonemizer: text is normalized (lower-cased, integers spoken, common punctuation kept), each word is looked up in a bundled lexicon, and out-of-vocabulary words fall back to deterministic letter-to-sound rules. It emits the IPA symbols in Kokoro's vocab and needs no external binary or download. Non-English voices in the checkpoint still load and synthesize, but their phonemes come from the English front-end, so pronunciation quality is limited; per-language g2p (the analogue of upstream Kokoro's misaki[xx] packages) is future work. Like Whisper, the model loads and runs every synthesis on one dedicated MLX worker thread, so loading a Kokoro checkpoint serves text-to-speech only.

Embedding models

Embedding checkpoints are served through POST /v1/embeddings and the offline mlxcel embed command rather than text generation; see Embeddings and reranking for detection rules, pooling, the request schema, the server flags and the checklist for adding a family. Detection recognizes a checkpoint by an encoder-only model_type (bert, xlm-roberta, modernbert, siglip), an embedding architectures[0], a modules.json Pooling entry or a 1_Pooling/config.json, and mlxcel arch lists the variants under the Embedding family. Every family in the table is implemented; its status cell records the validated checkpoint and any input or deployment limits.

Family	`model_type`	`ModelType`	Pooling	Validation checkpoint	Status
BERT / MiniLM encoder	`bert`	`Bert`	mean (from `1_Pooling/config.json`)	`sentence-transformers/all-MiniLM-L6-v2`, `intfloat/multilingual-e5-small`	supported
XLM-RoBERTa encoder	`xlm-roberta`	`XlmRoberta`	cls (from `1_Pooling/config.json`)	`BAAI/bge-m3`	supported
ModernBERT (alternating local/global attention, RoPE, GeGLU)	`modernbert`	`ModernBert`	`mean` (family default; `1_Pooling/config.json` wins)	`nomic-ai/modernbert-embed-base`	supported. `ModernBertModel` and `ModernBertForMaskedLM` load as embedders. nomic's checkpoint is asymmetric and needs the `search_query:` / `search_document:` prefixes in the input text; Matryoshka `dimensions` down to 256 is meaningful. `Alibaba-NLP/gte-reranker-modernbert-base` loads through the sequence-classification head, which `/v1/rerank` consumes; see Reranker models.
SigLIP text tower	`siglip`	`SiglipText`	last position, fixed (no `1_Pooling`)	`google/siglip-base-patch16-224`	supported (text only)
EmbeddingGemma (bidirectional Gemma 3)	`gemma3_text`, `gemma3`	`Gemma3Embedding`	mean, then two bias-free `Dense` projections (768 -> 3072 -> 768)	`mlx-community/embeddinggemma-300m-4bit`	supported
Qwen3-Embedding	`qwen3`	`Qwen3Embedding`	last token (the appended `<	endoftext	>`)
Qwen3-VL-Embedding (multimodal)	`qwen3_vl`	`Qwen3VLEmbedding`	last token (the final newline of the assistant header)	`Qwen/Qwen3-VL-Embedding-2B`	supported (text and image inputs). The generative Qwen3-VL stack (vision tower, DeepStack injection, interleaved M-RoPE) with the head replaced by pooling; the input is wrapped in the checkpoint's own chat template around an instruction system message.
Llama-Nemotron-VL-Embed (SigLIP-400M, pixel-shuffle `mlp1`, bidirectional Llama 3.2 1B)	`llama_nemotron_vl`	`LlamaNemotronVLEmbedding`	mean	`nvidia/llama-nemotron-embed-vl-1b-v2`	supported (text and image inputs). The caller supplies the `query:` / `passage:` prefix; an image item is wrapped server-side in `<img>` / `<IMG_CONTEXT>` / `</img>` at 256 tokens per 512px tile, up to six tiles plus a thumbnail.
ColIdefics3 (late interaction over SmolVLM / Idefics3)	`idefics3`	`ColIdefics3`	none: one 128-dim vector per token	`vidore/ColSmolVLM-Instruct-256M-base` (layout), `vidore/colSmol-256M` merged into it (retrieval quality)	supported (text and image). Multi-vector output; rank with MaxSim. A LoRA-only repository is rejected with a message asking for the merged checkpoint, because mlxcel does not merge adapters. `mask_non_image_embeddings: true` is rejected.
ColQwen2.5 (late interaction over Qwen2.5-VL)	`qwen2_5_vl`, `colqwen2`	`ColQwen25`	none: one 128-dim vector per token	`vidore/colqwen2.5-base` (layout), `vidore/colqwen2.5-v0.2` merged into it (retrieval quality)	supported (text and image). Multi-vector output; rank with MaxSim. Both the `ColQwen2_5` and the native `ColQwen2ForRetrieval` key layouts load.
Bidirectional Llama (LLM2Vec)	`llama_bidirec`, `llama` with `LlamaBidirectionalModel`	`LlamaBidirec`	mean	`nvidia/llama-nemotron-embed-1b-v2`	supported. The Llama 3 decoder run under a padding-only mask, with no head. Prefixes `query:` / `passage:` are caller-side. LLM2Vec checkpoints published as PEFT adapters are rejected at load: merge the adapter into a full `LlamaBidirectionalModel` export first.
Nemotron-3-Embed (Ministral 3)	`ministral3` with `is_causal: false`	`Ministral3Embedding`	mean	`nvidia/Nemotron-3-Embed-1B-BF16`, `mlx-community/Nemotron-3-Embed-1B-BF16-8bit`	supported. The Ministral 3 backbone run bidirectionally, with the Llama 4 per-position attention scaling computed at offset 0. The 8-bit conversion agrees with the bf16 original to cosine 0.9998. Prefixes `query:` / `passage:` are caller-side.
LFM2.5-Embedding (non-causal short conv)	`lfm2` with `Lfm2BidirectionalModel`	`Lfm2Embedding`	CLS (the prepended `<\|startoftext\|>`)	`LiquidAI/LFM2.5-Embedding-350M`	supported. The hybrid short-conv plus attention backbone with both mixers made bidirectional: the attention layers get a padding-only mask, the short conv splits its padding across both sides and zeroes its input at padding positions. Prefixes `query:` / `document:` are caller-side. The ColBERT late-interaction LFM2 checkpoints share this architecture and are rejected at load.

ColIdefics3 and ColQwen2.5 are the two late-interaction (ColBERT-style) families: they keep one L2-normalized 128-dimension vector per token instead of pooling, so /v1/embeddings returns [num_real_tokens, 128] per input and candidates are ranked with MaxSim rather than cosine. Both accept image_url items and embed a rendered document page directly. See Embeddings API for the response shape, the prompt formats and the scoring helper.

BERT and XLM-RoBERTa share one post-LayerNorm encoder over absolute position embeddings (src/models/bert.rs) and differ only in position-id construction, the type_vocab_size / layer_norm_eps / pad_token_id defaults and the checkpoint's weight-key prefix. BertForSequenceClassification and XLMRobertaForSequenceClassification checkpoints (BAAI/bge-reranker-v2-m3) load through the same trunk with the classification head in src/models/bert_heads.rs; they are rerankers, so they are deliberately not embedding variants and detect as SequenceClassifier instead (see Reranker models). See Embeddings API for the prompt prefixes these checkpoints expect.

SigLIP is the one family whose sequence width is fixed rather than derived: every input is truncated to 63 tokens plus the trailing </s> and right-padded to exactly the 64 learned positions, no attention mask is applied, and the vector is the projection head applied to the hidden state at position 63. The pad token and the EOS token are the same id (</s>, 1), which is what makes that slot meaningful for short inputs. Image embeddings through the SigLIP vision tower are not served yet.

image_url items are accepted only by the four multimodal families above (EmbeddingModel::supports_images()): Qwen3-VL-Embedding and Llama-Nemotron-VL-Embed return one pooled vector per image, ColIdefics3 and ColQwen2.5 return one vector per token. Every other family rejects an image item with a 400. All four embed one image per forward pass, so a request mixing text and images runs the text items as sorted micro-batches and each image on its own.

Reranker models

Rerankers score how relevant a document is to a query and are served through POST /v1/rerank and the offline mlxcel rerank command; see Reranking for the prompt formats, the request schema, the server flags and the error table. Three kinds are served, and all three return a probability in [0, 1].

Family	`model_type`	Kind	Score	Validation checkpoint	Status
BERT cross-encoder	`bert` with `BertForSequenceClassification`	`sequence_classifier`	`sigmoid(logit)`	`cross-encoder/ms-marco-MiniLM-L6-v2`	supported. Detected from `-m` alone. `[CLS] query [SEP] document [SEP]` with real segment ids; `max_length` 512 from the absolute position table.
XLM-RoBERTa cross-encoder	`xlm-roberta` with `XLMRobertaForSequenceClassification`	`sequence_classifier`	`sigmoid(logit)`	`BAAI/bge-reranker-v2-m3`	supported. Detected from `-m` alone. Reproduces the model card's published pair `[0.00028, 0.99484]` to within 2e-5.
ModernBERT cross-encoder	`modernbert` with `ModernBertForSequenceClassification`	`sequence_classifier`	`sigmoid(logit)`	`Alibaba-NLP/gte-reranker-modernbert-base`	supported. Detected from `-m` alone. `classifier_pooling` (`cls` or `mean`) decides the pooling before the head.
Qwen3 generative reranker	`qwen3` (`Qwen3ForCausalLM`)	`generative_text`	`sigmoid(logit("yes") - logit("no"))` at the last prompt position	`mlx-community/Qwen3-Reranker-0.6B-4bit`	supported through `--reranker-model` only: the checkpoint is an ordinary chat export and is indistinguishable from one. Uses the model card's exact prompt recipe, left padding and a per-request `instruction`.
Qwen3-VL multimodal reranker	`qwen3_vl` (`Qwen3VLForConditionalGeneration`)	`generative_vl`	the same yes/no read	`Qwen/Qwen3-VL-Reranker-2B`	supported through `--reranker-model` only (text and image documents). Renders the checkpoint's own `additional_chat_templates/reranker.jinja` and reads the answer token ids from `1_LogitScore/config.json`. Image rows are scored one at a time; video documents are out of scope.

Only the cross-encoder kind is detectable from a checkpoint alone, so it is the only one -m can serve; mlxcel arch lists it as SequenceClassifier under the Reranker family. A classifier with more than one output label is rejected at load, and a ForSequenceClassification export on any other family (DebertaV2ForSequenceClassification, for instance) stays on the ordinary dispatch and is reported there. Qwen/Qwen3-VL-Reranker-2B ships a modules.json whose only extra module is 1_LogitScore rather than a Pooling one, which is why detection does not mistake it for an embedding export.

Quantization formats

Format	Status	Notes
FP16 / BF16	supported	BF16 handling is platform/model dependent; Apple Silicon paths commonly convert to FP16 for execution.
4-bit affine MLX checkpoints	supported	Primary path for many `mlx-community` checkpoints. CUDA coverage depends on MLX kernel support for the target GPU.
8-bit affine	supported	Used for weights and/or KV cache depending on path.
NVFP4 / MXFP4 / MXFP8	supported where implemented	Used by specific families such as GPT-OSS and recent quantized checkpoints.

Do not infer quality or speed from the ability to load a quantized checkpoint. Run a smoke test and, for release claims, a benchmark/quality gate.

Muse Glimmer supports both the canonical dense BF16 checkpoint and the pinned mlx-community/Muse-Glimmer-30B-4bit affine-Q4 conversion. The Q4 path normalizes mlx-vlm's language_model.* roots and inherits the root-level quantization contract into the text decoder and fusion projections; the vision tower remains dense. Non-FP16/Turbo KV-cache modes are a separate follow-up because the model owns a mixed 2048-token sliding plus growing full-layer cache.

Distributed support summary

Capability	Current summary
Tensor parallelism	Advertised for selected dense text families such as Llama, Qwen, Gemma text, ERNIE 4.5, and Hunyuan dense. Validate per model/rank count.
Pipeline parallelism	Best validated for Llama-family text models; stage executors exist for more families with less operator coverage.
VLM under TP/PP	Partial. Vision tower / projector partitioning is not uniformly supported.
Disaggregated inference	Infrastructure exists; validate per topology and workload.

Muse Glimmer is single-process for both BF16 and affine Q4: TP, PP, XLA/IREE/OpenXLA, distributed, and disaggregated serving must remain disabled until each path has explicit mixed-cache and multimodal validation.

Speculative decoding

Drafter	Target families	Notes
MTP	Gemma 4 target paths	Available through shared speculative decoding flags, on `mlxcel-server` and on offline `mlxcel generate --draft-kind mtp`.
MTP	Qwen 3.5 / 3.6 / 3.8 text and VLM paths (B = 1)	Requires the split-out `qwen3_5_mtp` drafter checkpoint (for Qwen3.8-27B: `mlx-community/Qwen3.8-27B-MTP-bf16`; the mlx-community target conversions drop the in-model `mtp.*` tensors, so the drafter is always a separate directory). Auto-detected from the drafter's `model_type` with no `--draft-kind`, on `mlxcel-server` and offline `mlxcel generate`. Single-request only: batched (B > 1) MTP windows decline to classic decode. Image/video prefill runs on the target; MTP accelerates only the text decode tail. Metal-only (the temperature-0 exactness contract rests on the Metal chain-parity gated-delta kernel). Whether MTP pays off is hardware-generation-dependent, exactly as for Gemma 4: on M1 Ultra the target's own multi-token verify forward barely amortizes and MTP measures 0.59-0.70x classic; the server's per-hardware default plus the adaptive MTP policy gate this automatically. See `benchmark_results/qwen38-mtp-m1ultra-2026-08-16.md`.
MTP	Inkling text path (B = 1)	Uses `model.mtp.layers.*` from the original checkpoint or a directory containing its `config.json` and `mtp.safetensors`; tensor presence, not `mtp_config` alone, selects the drafter. Both `LoadedModel::Inkling` and the public checkpoint's `LoadedModel::InklingVLM` wrapper accept text-only MTP requests, with the latter decoding through `vlm.text`. Image requests remain on classic HMLP prepared-embedding prefill and do not enter the speculative round loop. For Inkling-Small, fetch only the approximately 4.5 GB head with `mlxcel download thinkingmachines/Inkling-Small --local-dir models/Inkling-Small-mtp --include config.json mtp.safetensors`, then pass `--draft-model models/Inkling-Small-mtp --draft-kind mtp` to either `mlxcel generate` or `mlxcel-server`. The default verify block is `num_nextn_predict_layers + 2` (10 for Inkling-Small), and `--draft-block-size` overrides it. Rollback restores an exact pre-verify snapshot and replays the accepted prefix because the four recurrent convolution states cannot be tail-trimmed. B > 1 declines to classic decode; tree verification is explicitly unavailable and Inkling rounds remain linear. The 153.5 GB target and 4.5 GB head were unavailable for this change, so the real-checkpoint parity, acceptance, and speed claims remain unqualified until those commands are run on a suitable host.
MTP	GLM-4.7-Flash (`glm4_moe_lite`), B = 1	Requires the split-out `glm4_moe_lite_mtp` drafter produced by `mlxcel split-mtp -m <raw zai-org/GLM-4.7-Flash dir> -o <drafter dir> [--q-bits 4]` from the raw checkpoint; the community 4-bit conversions drop the `model.layers.47.*` next-token-prediction tensors, so the drafter is always a separate directory. The drafter is the DeepSeek-V3-style nextn block (`enorm`, `hnorm`, `eh_proj`, one MLA + MoE decoder block, `shared_head.norm`, `shared_head.head`, its own token table) fed with the target's post-final-norm hidden state (measured against the pre-norm residual on the 4-bit pairing: mean accepted length 1.753 versus 1.716 over 128 greedy tokens, `MLXCEL_GLM_MTP_HIDDEN_TAP=pre` selects the loser for re-measurement); the default verify block is `num_nextn_predict_layers + 1` (2). Auto-detected from the drafter's `model_type` with no `--draft-kind`, on `mlxcel-server` and offline `mlxcel generate`. The block-vs-chain exactness probe (`MLXCEL_MTP_ALLOW_INEXACT` to override) gates the pairing per session exactly as for Gemma 4 and Qwen 3.5; on Apple GPU generation 15+ it pins the process to the narrow `qmv` kernel (#1199), so a temperature-0 MTP stream is identical to classic decode in the same process (the server) and to a standalone `mlxcel generate` run only under the same pin (`MLXCEL_QMV_WIDE=0`). This family does not dispatch the fused single-token MoE kernel, so `MLXCEL_FUSED_MOE` does not change its output. Batched (B > 1) windows and requests that adopted a prompt-cache prefix decline to classic decode. Automatic prefix caching and MTP do not combine on this family: the target's K/V for an MTP-served request lives on a model-owned slot rather than in the scheduler's `CachePool`, so such a request donates nothing to the prompt-cache store and a later turn finds no prefix to adopt. Serve without `--model-draft` when prefix reuse across turns matters more than the decode speedup. The resolved `--kv-cache-mode` (and `BatchKvQuantConfig`) applies to that slot as well, so a quantized KV mode is honoured with the drafter attached.
DFlash	Qwen 3.5 text/VLM paths	`mlxcel-server` only. Available through shared speculative decoding flags there; offline `mlxcel generate` rejects a DFlash drafter with a named error because it does not construct the DFlash round loop. Qwen3.8 rides the same `qwen3_5` path but has no published DFlash drafter; pair it with the MTP drafter row above instead (#1165).
DFlash	Muse Glimmer text-only requests (B = 1)	`mlxcel-server` only, through the same `--draft-model` / `--model-draft` flags with the published assistant drafter `meta-models/Muse-Glimmer-30B-assistant` (`model_type: muse_glimmer_assistant`; the kind is auto-detected, so `--draft-kind` is optional and only `dflash` is accepted). The drafter is five sliding-attention layers (32 query / 8 KV heads, per-head q/k norms, RoPE theta 500000, window 2048) fed with the target's residual streams after layers `[1, 13, 25, 37, 49]` through `encoder.fc`, predicting a 16-token block (bonus row plus 15 proposals) in one non-causal forward under a bidirectional sliding mask over absolute positions. It ships no `embed_tokens` and no `lm_head`: it binds the target's raw embedding table (not the target's `embed_norm`-wrapped lookup) and the untied head (with no `output_multiplier` or `final_logit_softcapping` behind it), and its `config.json` carries neither `num_target_layers` nor `vocab_size`, both checked against the bound target instead. The first draft round consumes every prompt row; a prompt longer than 2048 tokens keeps its last 2048 rows of context. The verify width is adaptive: the round loop starts at 4 rows and widens to the requested ceiling (`--draft-block-size`, default the checkpoint's 16) only when a measurement window emits more tokens per millisecond there, the same throughput comparator the MTP loop runs. Measured on an M5 Max with the 4-bit target: at a fixed 16 rows natural-text prompts accept 1.9 proposals per round and decode at 13 to 18 tok/s against 18 tok/s classic, at 4 rows they decode at 34 to 41 tok/s, and a repetitive 2000-token log accepts 14 of 15 proposals per round at 16 rows (94 tok/s), which is what the comparator widens for. The target's verify is one causal forward over the block; the rollback after a partial accept trims every cache, and the sliding caches are armed with a `clamp(block * 8, 32, 128)`-row speculative buffer before the prefill so a block appended across the ring boundary can be rewound. Temperature-0 output is token-identical to classic decode when the block-versus-chain exactness probe passes on the host (`MuseGlimmerTextModel::dflash_exactness_allows`, the same `MLXCEL_MTP_ALLOW_INEXACT` override and `qmv_wide` retry as the MTP and DSpark arms); a failing probe declines the burst to classic decode with the verdict in the log. Because the width is adaptive, the gate probes every width the loop can settle on, the 4-row warm-up depth as well as the requested ceiling, and declines if either is inexact: the forward width selects which quantized-matmul kernel MLX dispatches, so a pass at one width is not evidence about another. The speculative buffer is `clamp(block * 8, 32, 128)` floored at the block size itself, so a `--draft-block-size` above 128 still gets slack at least as wide as the block it verifies. Image requests, B > 1 windows and an adopted prompt-cache prefix decline to classic decode with one log line. Offline `mlxcel generate` rejects the drafter with the same named error as DFlash.
DSpark	LFM2 / LFM2.5 text (B = 1), and text-only requests on LFM2-VL	`mlxcel-server` only, through the same `--draft-model` / `--draft-kind dflash` flags (the kind is auto-detected from the drafter's `Lfm2DSparkDraftModel` architecture and `dflash_config` block, so `--draft-kind` is optional). Pairs each LiquidAI target with its published drafter: `LiquidAI/LFM2.5-2.6B` with `LiquidAI/LFM2.5-2.6B-DSpark`, `LiquidAI/LFM2.5-8B-A1B` (`lfm2_moe`) with `LiquidAI/LFM2.5-8B-A1B-DSpark`, and `LFM2.5-1.2B-Instruct` with its own drafter; the drafter's `num_target_layers`, hidden size and vocabulary are checked against the target before binding and a mismatched pair is rejected by name. The drafter is a 5-layer DFlash backbone fed with five of the target's residual streams plus a rank-256 Markov token-transition head that chains the block's logits into sequential proposals; its confidence head is loaded and never used. Greedy only: a request with `temperature > 0` and `top_k != 1` is served by classic decode with one warning, because the drafter has no stochastic acceptance rule. The default verify width is 8 rows (7 proposals plus the anchor); `--draft-block-size 10` restores the trained width (`block_size 9` in the drafter config counts proposals). Rollback after a partial accept trims the attention KV caches and recomputes each short-conv state from the block's gated input, so the next round continues from exactly the committed prefix. The temperature-0 output is token-identical to classic decode only when the block-versus-chain exactness probe passes on the host (`Lfm2Model::dflash_exactness_allows`, the same `MLXCEL_MTP_ALLOW_INEXACT` override and `qmv_wide` retry as the MTP arms); a failing probe declines the burst to classic decode with the verdict in the log. B > 1 windows decline to classic decode, as does an adopted prompt-cache prefix. Offline `mlxcel generate` rejects the drafter with the same named error as DFlash.
DFlash	Laguna text path (B = 1)	Poolside's `Laguna-*-DFlash` speculators (`poolside/Laguna-XS-2.1-DFlash`, 0.9 GB bf16; `Laguna-S-2.1-DFlash`): `model_type: laguna`, `architectures: ["DFlashLagunaForCausalLM"]`, a fused `self_attn.qkv_proj`, per-head `q_norm` / `k_norm`, a per-head softplus gate `g_proj`, one `aux_hidden_norms.{i}` per captured target layer, and causal sliding-window (512) attention from the 16-token proposal block over the target's captured residual streams (`dflash_config.target_layer_ids`, `[1, 13, 25, 33, 39]` on XS 2.1). The drafter borrows `embed_tokens` and `lm_head` from the target; `draft_vocab_size` must equal the target vocabulary and `num_target_layers` the target depth. Auto-detected from the drafter's `config.json` (a `dflash_config` block or the architecture marker) on `mlxcel-server` and on offline `mlxcel generate`, with or without `--draft-kind dflash`. The forward follows vLLM's `laguna_dflash.py` (vllm-project/vllm#46853): every drafter layer passes the projected context through its own `input_layernorm` before the K/V projection, and each proposal position sees only the 511 positions before it (a per-query window, not a fixed context view). Target rollback trims the dense caches and the sliding `RotatingKVCache`s, which run with `block_size` positions of speculative slack. Single-request only: B > 1 windows decline to classic decode. The pairing engages only when the block-versus-chain exactness probe passes on the host (`LagunaModel::dflash_exactness_allows`, the same `MLXCEL_MTP_ALLOW_INEXACT` override and `qmv_wide` retry as the LFM2 and Muse Glimmer arms); a failing probe declines the server burst to classic decode and makes offline `mlxcel generate --draft-kind dflash` exit with the verdict. On a GB10 with the NVFP4 target the probe fails (107246 of 200704 logit bytes differ at the first verify position), and with the override greedy output equals classic decode wherever the target's top-2 logits are more than one bf16 ulp apart: at an exact or one-ulp tie the `M >= 2` verify kernels and the `M = 1` decode kernel can round the winner differently (one or two such positions per 128 tokens measured, every one at a chain-arm margin of 0.0 or 0.125). Acceptance is text-dependent: on raw code completions the drafter accepts 2.6 to 3.9 proposals per round at `block_size` 16 and 3.6 at `--draft-block-size 8`; inside the `<think>` channel of a chat prompt it accepts 1.1 to 1.8. Decode throughput versus classic decode on the GB10 (same binary, feature off versus on, n=3 per width, 200-token code completions, measured before and after #1795 with the same result) is at or below 1.0x at every block size 2 to 16: 0.52x at block 2, 0.87x at 4, 0.98x at 6, 1.00x at 8 (its range inside the off arm's, so not a demonstrated win), 0.96x at 10, 0.88x at the default block 16, and about 0.4x on thinking-channel prose; there is no sub-8 cliff on the NVFP4 path and no crossover past 8. A verify block costs a fixed 77 ms plus about 3.1 ms per row of device time against a 31 ms graph-replayed classic step, plus 30 to 35 ms of host-side drafter graph construction per round, so even perfect acceptance would cap near 1.8x at block 8 and the measured 3.3 accepted lands at 1.00x. The default block size stays at 16 (a served-width change is #1797); measure before enabling on a new host. Details in the #1351 technical report.

Use auto-detection by default. Override only when you know the target and drafter checkpoint pair are compatible.

A DFlash, DSpark or Muse Glimmer assistant drafter checkpoint is not a standalone model: it ships no embed_tokens and no lm_head, borrowing both from the target when it binds. Passing one to -m is rejected with that explanation rather than with a weight-lookup failure. See speculative-acceptance.md for what the offline path does and does not construct.

Muse Glimmer accepts exactly one drafter, its own assistant: mlxcel-server refuses any other --draft-model, any --draft-kind other than dflash, and draft flags without a drafter, by name at startup.

Known non-goals / caveats

A supported architecture does not imply every community checkpoint variant is supported.
VLM and video/audio paths require additional runtime dependencies and prompt preparation beyond text-only generation.
TurboQuant, TP, PP, and speculative decoding are not uniformly validated for every family.
Muse Glimmer's measured gate above is hardware-specific. CUDA allocator counters are unavailable on the GB10 backend, and Apple-Silicon/Metal remains untested; use the recorded OS memory bounds rather than extrapolating them.
mlxcel arch is the human-readable architecture catalog. mlxcel arch --json emits the machine-readable recipes registry with mlxcel_version, one families entry per loadable ALL_MODEL_TYPES variant plus standalone runtime families such as rt_detr_v2, stable id values, detection keys in model_types, runtime/modalities/output fields, Metal, CUDA and ROCm status, tensor/pipeline parallel flags, speculative drafter support, and supported KV modes. The ROCm column is a first-pass partial for every family until families are validated on AMD under #1801.
The registry is still a family-level contract, not a checkpoint qualification. Use the per-family notes above before treating a model card, quantization, backend, or hardware combination as validated.

Adding support

See Adding a new model for the registration, loading, and test checklist.

FilesExpand file tree

supported-models.md

Latest commit

History