Repository navigation
Integrate upstream Flash-Next MTP support - #8
Conversation
Adds the MTP head's own hyper-connection mixer tensor names and lists the NextN tensors under the qwen4exp architecture.
Adds --spec-type draft-mtp support for Qwen3.8-Flash-Next. The MTP head folds the next token's embedding into the trunk's wide hyper-connection residual, runs one trunk-style block (dense attention + MoE) over it, and collapses the result with its own mixer before reusing the trunk's LM head. - read nextn_predict_layers so n_layer() excludes the MTP block - load the trailing block through the existing trunk path: is_recr() and is_ple() are already false past the trunk, so it needs no special casing - eh_proj fuses the checkpoint's fc_embedding and fc_hidden side by side, so one matmul computes fc_embedding@e + fc_hidden@h - the head carries its own hyper-connection mixer, mirroring the trunk's hc_head_*, which stands in for the output norm qwen4exp does not have - export the wide pre-collapse residual as t_h_nextn from both graphs, so the driver can feed it back for the next draft step - route MTP contexts to a plain KV cache filtered to the trailing layer The draft block attends densely for now: the trunk's QSA only prunes context past a 2048-token budget, so dense is a numerical superset and drafts are verified either way. Indexer tensors are still loaded.
The MTP block is one trunk-shaped block (dense attention + MoE wrapped in hyper-connections) plus a head-level combiner, so once _QwenMtpMixin renames mtp.layers.0.* to the trailing block index its tensors ride the existing qwen4exp mappings unchanged. Two head-level pieces need handling: - fc_embedding and fc_hidden fuse into the eh_proj the shared NextN code expects, since W_e@e + W_h@h == [W_e|W_h] @ concat(e, h) - mtp.hyper_connection_mixer.* is the head's own copy of the trunk's hc_head_* output mixer, unindexed in the checkpoint and per-block in the GGUF compress_ratios is read with length block_count, so it gains a trailing 0 for the MTP block, which attends densely. --no-nextn drops the head; --mtp exports it on its own.
A NextN/MTP draft exported with --mtp carries the token embeddings, output norm and lm head so it can be loaded as a standalone model. For every current sidecar those three tensors are most of the file: ggml-org/Qwen3.8-27B-GGUF mtp-Qwen3.8-27B-Q4_0.gguf is 1.565 GiB, of which 1.332 GiB (85%) is the copy, against 0.223 GiB for the MTP block itself. Add an opt-in --mtp-shared-embd that leaves them out and marks the file with nextn_shared_target_tensors. The loader then resolves those names against the already loaded target model. The graph side needs no change: the nextn blocks of twelve archs already fall back to model.tok_embd and model.output. The borrow is gated on the new key, so a sidecar published before this change cannot reach it and keeps its current behaviour. Shapes are checked against the target and a mismatch is refused, as is loading such a file on its own.
_QwenMtpMixin is not a ModelBase subclass, so it re-declares the attributes it reads off cls for the type checker. filter_tensors reads cls.mtp_shared_embd without a matching declaration, which ty reports as unresolved-attribute. The declaration is a bare annotation, matching no_mtp and mtp_only above it. That creates no class attribute, so it cannot shadow ModelBase.mtp_shared_embd even though the mixin precedes the model class in the MRO; a default value here would have. Assisted-by: Claude
A draft-only export declares the full block count but ships the MTP block alone, so the trunk tensors load as null and only the MTP graph is buildable. Context reservation builds the trunk graph, which walked those nulls and segfaulted. A shared-embedding draft is caught earlier by the borrow check, since it has no token_embd of its own. A self-contained draft keeps one, so it passed that check and reached here. Assisted-by: Claude
Reuses the existing llama_context_params::ctx_other plumbing instead of adding a model_shared load parameter and a nextn_shared_target_tensors metadata key. A draft-only export now simply omits token_embd/output, and the qwen4exp MTP graph resolves them against the target context at graph build, following dflash and gemma4-assistant. Loading such a file on its own reports that it needs -md. Assisted-by: Claude
Assisted-by: Claude
A qwen4exp draft that borrows the target's embeddings sets ctx_other but keeps its own memory, so it must be caught up and rolled back like any other draft. Treating it as memory-shared skipped the catch-up decode and placed every draft token at the same position, which the M-RoPE position check rejects. Assisted-by: Claude
Merge upstream master into the qwen4exp MTP branch. Every conflict is in src/models/qwen4exp.cpp and comes from 41abbfd, which moved the hyper-connection gammas to {n_embd, hc} with TENSOR_ALLOW_RESHAPE for the fused rms_norm+mul path. Each one takes master's shape and keeps this branch's tensor flags. layer.nextn.hc_head_norm needs the same shape but never conflicts, since master has never carried that line. Left flat it aborts at load with GGML_ASSERT(ggml_can_repeat(b, a)) in build_hc_mix, so it is converted here as well. nextn.hnorm stays flat because graph_mtp reshapes before that multiply.
Assisted-by: Claude
Integrate ggml-org/llama.cpp PR 28243 at 6fcaa16. Preserve owner cache and selective CUDA transport changes. Addresses novkien/llama-proxy#470 Assisted-by: Codex
Pre-merge handoffPrivate-fork integration PR #8 implements the owner's explicit Flash-Next MTP merge/rebuild decision. Reviewed head |
Native/RPC delivery and Flash-Next acceptanceFork PR #8 merged as Focused checks passed: 10 changed Python files parse, whitespace/merge ancestry checks, native grammar parser, sampling, and one Qwen4Exp CPU architecture/serialization roundtrip case. The exact existing Flash target/draft GGUF metadata and NextN tensor dimensions were inspected for compatibility. Both hosts staged hash-verified packages with rollback copies. A temporary model-admission fence queued new inference while the normal non-forced unload drained the old model; native/RPC packages were then installed and the admission fence restored to false. The owner's GUI keep-awake hold remains owned by the owner and unchanged. No proxy service restart or agent idle hold was used. Live Remaining: finish Qwen-27B final profile verification on the new binary, including concurrent prefill, then publish consolidated configuration/performance acceptance under ggml-org#470. Source worktrees on native and guest are |
|
Final acceptance completed on rebuilt fork e58b770. Qwen is persistently and actively set to 690k shared KV, b2048/ub32, split29,29,27,15, four slots, MTP n_min1/n_max2/p_min0. Final sequential decode is 46.96/52.94 token/s; four concurrent 7208-token prompts complete without OOM (117.01 s total), and image input correctly returns Red. Original DFlash2 profile on the same binary measures 34.96/56.78 token/s; combined decode is about 15% lower than the selected MTP profile, while DFlash2 remains faster on the code prompt alone. Final MTP settings were restored and read back after comparison. Flash-Next now uses the rebuilt fork with its original MTP head: 127/222 drafts accepted and a coherent completion. All 14 routes have validated persisted NVLink settings; actual source manifests and child environment match. Model admission is unblocked. The owner-controlled keep-awake state is preserved; no agent hold exists. This completes ggml-org#470's authorized scope. Separate catalog-capacity issue ggml-org#494 and native draft-media issue #6 remain open; neither was silently patched or treated as repaired. |
4 participants
Addresses novkien/llama-proxy#470
Integrates ggml-org#28243 at
6fcaa16f4b360649933a54d1f91ad40ed35c0e11, explicitly requested by the owner in Codex on 2026-09-30. Fork base:345b1c5f26f21b0aed458d846aaedb48565d8490. Integration head:8c8edf8f6ceb76b41049948f269d33faa5687769.The true merge is conflict-free and changes exactly the upstream PR's 18 paths (408 additions, 77 deletions). It adds Flash-Next NextN/MTP graph, detached/shared draft tensor loading and conversion support. Generic MTP initialization now loads the actual draft path and limits shared-KV treatment to Gemma assistant. Existing RAM-cache and selective CUDA transport source is unchanged; both immutable input SHAs are ancestors of the integration head. The unrelated custom
llama.cppcheckout remains untouched.Validation before source merge: full delta reviewed; staged diff whitespace check passes; all 10 changed Python files parse; no unmerged entries; owner cache/P2P paths compare byte-identically. Native/RPC rebuild and Flash/Qwen live acceptance follow source merge, as requested, and are not yet claimed complete. Qwen context/speed tuning remains provisional until tested on the rebuilt fork.
AI disclosure: Codex reviewed and integrated the existing upstream PR; no upstream submission is made.