Support Qwen3.8 narrow-IQ1 GGUF packages

[michaelneale]

Qwen3.8 and other models using Unsloth's narrow IQ1 GGUF types can now be inspected, packaged, and built with mesh-llm instead of failing on tensor type 64, 65, or 66.

This imports the reviewed narrow-IQ1 implementation from merged unslothai/llama.cpp#91 (c86ed269986f2dced6325c5c58bda966a2e2ead1) as the foundational first patch in mesh-llm's durable llama.cpp queue. Existing Skippy patches are renumbered without content changes.

Compatibility

• Adds fork-local GGUF types IQ1_XS=64, IQ1_XXS=65, and IQ1_XXXS=66.
• Stock llama.cpp readers still reject these nonstandard files cleanly.
• No Skippy public ABI declarations changed, so the Skippy ABI version is unchanged.
• CPU and CUDA kernels come from the source change. Apple builds retain the normal Metal backend with CPU fallback for narrow-IQ1 operations; throughput requires real-model benchmarking.

Validation

At commit efd440a5:

• scripts/prepare-llama.sh pinned — clean replay of all 19 patches
• final prepared tree matches the independently constructed pin + Unsloth change + existing queue tree
• just build — passed on Apple Silicon, including the dynamic Metal native runtime package
• generated table script executed successfully with NumPy and produced all three expected tables (1024/512/256 entries); output is a generated fragment rather than a whole-file replacement

The real Qwen3.8 layer-package job is intentionally deferred until this branch is available to the HF job via a pushed ref. Family certification remains separate and also needs #1283.

Source: unslothai/llama.cpp#91

Deliberate scope boundary

This import adds native GGUF reader/runtime support required by model packaging and serving. It does not add Rust-side skippy-quantize::QuantType variants for the fork-local ftype IDs; skippy-quantize therefore cannot select these formats for new quantization yet. That is separate from consuming existing narrow-IQ1 GGUF artifacts and should be added with its own Rust API/tests if mesh-llm needs to produce these quantizations.

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

💤 Files selected but had no reviewable changes (19)

• third_party/llama.cpp/patches/0001-Add-narrow-IQ1-quantization-types.patch
• third_party/llama.cpp/patches/0002-Add-staged-model-graph-and-family-support.patch
• third_party/llama.cpp/patches/0003-Add-GLM-DSA-backend-execution-support.patch
• third_party/llama.cpp/patches/0004-Add-Inkling-model-and-multimodal-support.patch
• third_party/llama.cpp/patches/0005-Add-Skippy-public-ABI-surface.patch
• third_party/llama.cpp/patches/0006-Add-Skippy-model-lifecycle-and-package-support.patch
• third_party/llama.cpp/patches/0007-Add-Skippy-session-and-state-management.patch
• third_party/llama.cpp/patches/0008-Add-Skippy-activation-frame-handling.patch
• third_party/llama.cpp/patches/0009-Add-Skippy-staged-execution-paths.patch
• third_party/llama.cpp/patches/0010-Add-Skippy-sampling-and-speculative-decoding.patch
• third_party/llama.cpp/patches/0011-Add-Skippy-tokenization-and-stage-chat.patch
• third_party/llama.cpp/patches/0012-Wire-staged-runtime-builds-and-tests.patch
• third_party/llama.cpp/patches/0013-Pass-generic-chat-template-kwargs-through-Skippy.patch
• third_party/llama.cpp/patches/0014-skippy-remove-legacy-chat-template-ABI.patch
• third_party/llama.cpp/patches/0015-docs-annotate-Skippy-native-API-headers.patch
• third_party/llama.cpp/patches/0016-Fix-staged-runtime-upstream-compatibility.patch
• third_party/llama.cpp/patches/0017-skippy-load-MTP-tensors-for-native-draft-attachment.patch
• third_party/llama.cpp/patches/0018-skippy-select-MTP-source-in-runtime-config.patch
• third_party/llama.cpp/patches/0019-skippy-honor-selected-MTP-source-when-loading.patch

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b1cfea36-2209-4271-abb6-b5dffb09f8ee

📥 Commits

Reviewing files that changed from the base of the PR and between 767edfb and efd440a.

📒 Files selected for processing (19)

• third_party/llama.cpp/patches/0001-Add-narrow-IQ1-quantization-types.patch
• third_party/llama.cpp/patches/0002-Add-staged-model-graph-and-family-support.patch
• third_party/llama.cpp/patches/0003-Add-GLM-DSA-backend-execution-support.patch
• third_party/llama.cpp/patches/0004-Add-Inkling-model-and-multimodal-support.patch
• third_party/llama.cpp/patches/0005-Add-Skippy-public-ABI-surface.patch
• third_party/llama.cpp/patches/0006-Add-Skippy-model-lifecycle-and-package-support.patch
• third_party/llama.cpp/patches/0007-Add-Skippy-session-and-state-management.patch
• third_party/llama.cpp/patches/0008-Add-Skippy-activation-frame-handling.patch
• third_party/llama.cpp/patches/0009-Add-Skippy-staged-execution-paths.patch
• third_party/llama.cpp/patches/0010-Add-Skippy-sampling-and-speculative-decoding.patch
• third_party/llama.cpp/patches/0011-Add-Skippy-tokenization-and-stage-chat.patch
• third_party/llama.cpp/patches/0012-Wire-staged-runtime-builds-and-tests.patch
• third_party/llama.cpp/patches/0013-Pass-generic-chat-template-kwargs-through-Skippy.patch
• third_party/llama.cpp/patches/0014-skippy-remove-legacy-chat-template-ABI.patch
• third_party/llama.cpp/patches/0015-docs-annotate-Skippy-native-API-headers.patch
• third_party/llama.cpp/patches/0016-Fix-staged-runtime-upstream-compatibility.patch
• third_party/llama.cpp/patches/0017-skippy-load-MTP-tensors-for-native-draft-attachment.patch
• third_party/llama.cpp/patches/0018-skippy-select-MTP-source-in-runtime-config.patch
• third_party/llama.cpp/patches/0019-skippy-honor-selected-MTP-source-when-loading.patch

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[michaelneale]

Read the patch rather than just the description. Structure is right, and the renumber-not-regenerate call is the correct one — 18 files with zero content change makes this reviewable, which it would not have been otherwise.

Things I checked and am satisfied by:

• Type IDs are safely placed. 64/65/66 with 42..63 left to upstream, and upstream currently at GGML_TYPE_COUNT = 43 using 42 for Q2_0. So a stock reader rejects these files rather than misreading them, which is the property that matters most for a fork-local format.
• arch-fallback.h covers every arch arm, including __aarch64__. That is what actually makes the Apple path work: the scalar _generic vec_dot is aliased in on ARM rather than left undefined.
• ggml_backend_cpu_device_supports_op adds the three types alongside the existing IQ1_S entry, so CPU advertises support rather than silently declining the op.
• Tables generated from committed .npy grids via iqxs_gen_tables.py rather than pasted. Good — that is auditable.

Two things worth resolving, neither a blocker:

1. QuantType::from_llama_ftype_id does not know 64/65/66.

crates/skippy-quantize/src/types.rs maps ftype ids up to 41 and returns None past that, and FromStr turns that into unsupported quant ftype id {n}. This patch adds LLAMA_FTYPE_MOSTLY_IQ1_XS/XXS/XXXS = 64/65/66 on the C side but nothing on the Rust side learns them.

I do not think this blocks the package job — skippy-quantize is a standalone crate with no in-tree dependents, and the package path goes through skippy-model-package and the ABI, not through QuantType. So the scope is "skippy-quantize CLI cannot name these types", not "packaging breaks". Worth either adding the three variants or noting explicitly that quantize-side support is out of scope, so the next person does not read the C-side enum and assume the Rust side followed.

2. The Metal claim in the description is right but easy to misread.

"Apple builds retain the normal Metal backend with CPU fallback for narrow-IQ1 operations" is accurate. The sharper version, since the patch touches zero ggml-metal files: any tensor stored in these three types has no Metal kernel, so on Apple those specific matmuls execute the scalar ARM path with no SIMD — the upstream description says "no SIMD kernel" explicitly. For UD-Q1_0 that is the expert tensors, which is where the FLOPs are.

I overstated this earlier in the channel as "Macs are blocked", which was wrong, and Mic correctly called it out — the Unsloth guide does document a Mac build. But the throughput question is real and I would not want the benchmark to be a surprise. Suggest the benchmark be sequenced before any large rental, since a Mac-anchored split is the current plan and this is the assumption it rests on.

Neither point needs to hold up the merge. The validation you listed — clean 19-patch replay, tree equivalence against an independently constructed tree, just build on Apple Silicon with the Metal runtime packaged — is the right evidence for a patch-queue import, and tree equivalence in particular is the check I would have asked for.

[michaelneale]

🤖 Blocking follow-up found during review: the imported narrow-IQ1 change has CPU/CUDA implementations but no Metal pipeline cases. On the current pinned tree, ggml_metal_device_supports_op advertises MUL_MAT/MUL_MAT_ID for every source type except NVFP4, while pipeline selection has no cases for types 64–66 and aborts in the default arm. Therefore the successful Apple build does not imply type-66 Metal execution works. I am treating Mac serving as blocked until the PR either adds real Metal kernels or, at minimum, accurately declines these operations so CPU fallback is possible and tested. The latter is correctness-only and likely too slow for the planned Mac-anchored Qwen3.8 split.

[michaelneale]

🤖 Parking note: this PR is off the critical path.

We verified the tensor-type census of every Qwen3.8 GGUF tier by reading GGUF headers directly off HF (range requests, real shard 2 of each):

Tier	Types	Metal
UD-Q1_0 (397 GB)	IQ1_XXXS(66) ×33, Q5_K ×52, Q6_K ×20, Q4_K ×2, F32 ×106	❌
UD-IQ1_S (508 GB)	IQ1_S ×26, Q5_K ×39, Q6_K ×15, Q4_K ×2, F32 ×76	✅
UD-IQ2_XXS (657 GB)	IQ2_XXS ×20, Q5_K ×29, Q6_K ×12, Q4_K ×2, F32 ×60	✅

UD-Q1_0 is the only tier containing types 64/65/66. The lab plan has moved to UD-IQ2_XXS, which needs nothing from this PR — all its types are supported by stock mesh-llm.

Separately, the narrow-IQ1 types have no Metal kernels anywhere — not this branch, not Unsloth's iq1-narrow, not merged Unsloth #91, not upstream. Reproduced on Apple M5 Max against Unsloth's own unmodified branch built per their published Mac instructions (-DGGML_CUDA=OFF):

loaded kernel_mul_mv_iq1_s_f32 ...
loaded kernel_mul_mm_iq1_m_f32 ...
Asserting on type 64
ggml/src/ggml-metal/ggml-metal-device.cpp:959: not implemented
  ggml_abort / ggml_metal_op_mul_mat / ggml_metal_graph_compute
Abort trap: 6   (exit 134)

Metal advertises MUL_MAT for every type except NVFP4 (ggml-metal-device.m:1389), then pipeline selection has no case for 64/65/66 and aborts — so it is a crash, not a CPU fallback.

This PR remains correct and worth keeping: it is a clean import of merged Unsloth #91, CPU + CUDA complete, and the only way anything in our stack can read type 66 at all. It is the required foundation if we later write Metal kernels or want an all-CUDA topology on the 397 GB tier.

It is currently red (CI / Linux, CI Required, plan digest 1dcb7bb2fc478f614e25e1c257e5544bd2d016fc9b8e152be8eb22e3650c1f12). Nobody should spend time on that red build believing it blocks Qwen3.8 work — it does not. Leaving open and parked rather than closed, so the branch stays as the artifact.

The Qwen3.8 blocker that is still live is #1283 (family identity + recurrent cache policy), which is quant-independent.

Context: #1262

[michaelneale]

🤖 Parked / off the Qwen3.8 Mac-split critical path. The UD-Q1_0 package this enabled stores expert tensors as IQ1_XXXS (type 66), but the imported implementation has CPU/CUDA kernels only. The published Mac path aborts because Metal advertises the operation without a matching pipeline. We are switching the planned experiment to a Metal-supported quant tier. Keeping this PR and branch as the foundation for a future all-CUDA use case or proper narrow-IQ1 Metal implementation; it should not be merged as evidence of Mac runtime support.

[michaelneale]

🤖 Closing as not-now rather than asking reviewers/CI to carry an off-path capability. Reopen when there is a concrete all-CUDA consumer or the missing narrow-IQ1 Metal kernels and execution tests are added.