Repository navigation
qwen4exp: dense MTP layer aborts on unused QSA pool input #11
Description
Current status
Accepted on2026-10-03: PR #12 merged at8d75def06, matched native/RPC artifacts deployed, original pipeline-enabled reproduction loads and serves8192+512 through proxy8082. Deployment and acceptance report. Broader optimization and>=200k final capacity remain in #9.
Problem and impact
Current fork master 7e0d6c8 aborts while loading the owner's existing Flash Next IQ3 target with its Q4_K_M MTP draft. This blocks the approved optimization/benchmark campaign in #9. The same model and exact launch configuration load successfully on the previously installed 98b3e09 pair.
Expected and actual behavior
Expected: a Qwen4Exp MTP layer with compress_ratio=0 uses dense attention without constructing unused QSA pool inputs, while QSA-enabled layers retain pooling. The existing detached draft should load and generate through the proxy with two slots and pipeline parallelism enabled.
Actual: native/RPC builds from clean 7e0 master load all weights and reserve the target pipeline scheduler successfully (copies=4, no fallback). After allocating the draft and entering common_context_can_seq_rm(), the process aborts with:
ggml/src/ggml-backend.cpp:345: GGML_ASSERT(buffer) failed
ggml_backend_buffer_is_host
llama_kv_cache::set_input_k_idxs
llama_model_qwen4exp::llm_graph_input_kpool::set_input
llm_graph_result::set_inputs
llama_context::process_ubatch / decode
common_context_can_seq_rm
server_context_impl::load_model
This is not a CUDA OOM. Target reservation and draft allocation completed; the backend buffer pointer for an unused graph input is null.
Reproduction and source evidence
- Four V100-SXM2-16GB CUDA devices plus RTX5060Ti RPC; target GGUF Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS (two shards), existing mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf. Exact artifact hashes are in Research: port V100/SM70 kernel and runtime optimizations into llama.cpp-fork #9's artifact intake report.
- Both arms use c262144, b1024, ub512, parallel2, unified Q8 KV, ngl99, layer split12,8,10,9,10, MTP min1/max3 on CUDA1, unchanged selective P2P and model sampling. Launch through proxy8082; immutable native/RPC manifests identify the exact same source revision and empty patch for each arm.
- 7e0 load job3a149bc46a874d58b3e08105ff797512 failed after42.65seconds with SIGABRT and the stack above. Native CUDA12.9/SM70 build and guest CUDA13.3/SM120a build both succeeded.
- Reinstall the retained 98b3 pair with the exact same route: job0029dd8443844239b801207acc54e53d succeeded in43.05seconds, both slots n_ctx262144, target scheduler copies4 and no pipeline fallback. This isolates the source interval from placement/memory settings.
- The current draft metadata has 49 compress-ratio entries: QSA4 every fourth trunk block, followed by 0 for MTP layer48. Global indexer_kpool remains4 because the trunk entries are nonzero.
- In graph_mtp(),
mctx_hyb->get_idx() && hparams.indexer_kpool > 0constructs/registers pool inputs using the global property. Later build_layer_attn() requireshparams.dsv4_compress_ratios[il] > 0and does not use the pool for layer48. k_idxs is not expanded through a consumer, but llm_graph_input_kpool::set_input() still writes it unconditionally. This explains the null buffer and matches the live stack.
Search: inspected all-state fork issues for kpool/set_input_k_idxs/GGML_ASSERT and PRs for qwen4exp; no matching repair was returned. #9 owns performance work and supplies the evidence; this is a separate startup-correctness prerequisite. The broader upstream QSA/mask optimizations are not reverted by this proposal.
Proposed bounded repair and acceptance
Construct MTP QSA-pool inputs only when the actual selected MTP layer uses QSA, matching build_layer_attn()'s eligibility. Preserve the existing cache/indexer validation for eligible QSA layers. Do not weaken generic buffer assertions or silently skip writes to allocated/required inputs.
- Focused existing native architecture/server fixture reproduces a dense MTP layer with QSA trunk metadata and fails on unmodified master.
- Corrected dense MTP setup/decode succeeds; QSA-enabled MTP and ordinary trunk paths retain their coverage.
- Reviewed small issue-linked PR builds on native and RPC, is merged/deployed as a matched pair, and the exact Flash route loads and generates with two slots/pipeline enabled.
- Preserve upstream optimization behavior for later speed/memory A/B in Research: port V100/SM70 kernel and runtime optimizations into llama.cpp-fork #9; do not label this correctness prerequisite a measured speedup.
Authority and ownership
The owner explicitly approved #9's full implementation, build, separate PRs and real Flash MTP acceptance. This bounded prerequisite repairs the verified startup regression encountered in that authorized path without changing the public API, model, draft depth or behavior. Codex /root coordinates the issue and independent source review; the accepted repair is merged in PR #12 and deployed as matching native/RPC packages at8d75def06; see the acceptance report. The working 98b3 pair is temporarily retained for continued independent configuration measurements, not a final rollback decision.
Activity
novkien commentedon Oct 3, 2026
Work report - prerequisite repair claimed
Codex /root coordinates this bounded correctness prerequisite for the owner's approved Flash optimization in #9. /root/flash_attention owns source changes in isolated branch codex/flash9-dense-mtp at base7e0d6c858; other agents review/investigate only. The selected change gates MTP pool construction on that selected layer's compress ratio, matching its attention path. Generic backend assertions and the target's upstream QSA/mask optimization remain intact.
The before case is a verified SIGABRT on clean7e0; the retained98b3 pair passes with the exact same pipeline-enabled ub512/context/placement. Next: minimal patch and focused maintained tests, independent review, paired native/RPC build and the same proxy startup/MTP request on the repaired revision. The open issue and parent #9 will retain the outcomes; startup acceptance is distinct from optimization performance.
novkien commentedon Oct 3, 2026
Work report - minimal candidate passes live startup and MTP
The one-line candidate adds the selected MTP layer's
compress_ratio > 0to pool-input construction. It leaves the target QSA graph, buffer assertions and public API unchanged. Independent source review agrees with the causal explanation. A focused tiny architecture fixture is being added to verify the dense failure before the patch and both dense/QSA MTP paths after it.Immutable native and RPC packages were built at
7e0d6c85838868ff5fc3541086f656f1bcdf45d6plus source patch SHA2568d7d7fc8a2815789616183f0349229cbee005798337d6c4bbde9b79579da6c7e. Normal proxy unload, matched complete-package exchange and load succeeded in 43.04 seconds at the unchanged pipeline-enabled reproduction configuration. Startup reports target copies4 with no fallback, two slots and all49/49 target layers offloaded. The draft host compute allocation returns from539MiB to281MiB; the target's larger upstream scratch remains unchanged, confirming those are separate observations.Three subsequent proxy8082 SSE requests each processed exactly8192 fresh prompt tokens (cache_n0) and generated512 tokens. All benchmark records are valid: PP405.954/409.648/409.517 and TG79.725/80.811/81.012 tokens/s. Draft acceptance is371/420,372/417,372/417. This is candidate correctness and performance screening, not a speedup claim or merged-revision acceptance.
Next: complete the maintained regression test and independent review, publish/merge the issue-linked prerequisite, then deploy and verify the exact merged native/RPC pair. #9 profiling proceeds with a separately marked diagnostic build. No issue closure is claimed yet.
novkien commentedon Oct 3, 2026
Pre-merge handoff — PR #12
PR: #12
Reviewed head:
6ed6a0621db408f952dd53fce6d8a4eb9ef824feBase:
7e0d6c85838868ff5fc3541086f656f1bcdf45d6The two-file change gates MTP QSA pool-input construction on the selected layer's nonzero compression ratio. Dense MTP no longer registers an unused pool input with a null buffer; QSA-enabled MTP retains its existing index/cache validation. The added architecture fixture covers ordinary Qwen4Exp plus dense and QSA-enabled MTP decode.
Validation:
GGML_ASSERT(buffer)inggml_backend_buffer_is_host, throughset_input_k_idxsduring decode.test-llama-archs -a qwen4exp -v 2passes 3 tests with 0 failures.git diff --checkpassed.The issue remains open pending root's build/deployment verification of the exact merged native/RPC pair and behavior check through proxy 8082. PR checks
model-naming,labeler, andeditorconfigpassed; other optional platform/build workflows were still pending at handoff. No branch protection or repository ruleset requires those checks.llama : avoid unused QSA pool input for dense MTP (#11)
novkien commentedon Oct 3, 2026
Deployment and acceptance - dense MTP startup regression repaired
PR #12 is merged as
8d75def06db97c8736989c795be403faf214a9d4. Both immutable native/RPC packages were built from that revision with an empty source patch, preserving their SM70/CUDA12.9 and SM120a/CUDA13.3 toolchains. The complete29-file native and22-file RPC packages were installed through normal proxy unload and locked atomic directory exchange; retained prior packages provide rollback. Installation verified old fork children had exited, and the actual new process executable and every mapped backend library match their package hashes. Both manifests report the identical merged source ID.The original reproduction settings were restored: c262144, b1024, ub512, two unified-KV slots, q8_0, ts12,8,10,9,10, ngl99, unchanged target/draft, MTP min1/max3 onCUDA1. Normal Load job
55156ffca5504488bdcdd6db9232f1e9succeeded in42.95seconds. Native properties reportb11397-8d75def06and two slots with n_ctx262144. Logs confirm49/49 target and50/50 draft GPU offload, pipeline copies4, no pipeline allocation fallback, and no buffer assertion.A subsequent real proxy8082 SSE request processed8192 new tokens (cache_n0) and emitted512 tokens. Its final timings and usage agree; MTP accepted371 of418 drafted tokens. PP405.740 and TG79.304tokens/s are correctness-run observations, not a performance claim. The generated two-lane benchmark fixture exposed an independent preparation bug before sending any request; that tooling follow-up is recorded under #9/PR#13 and is not mislabeled as a model failure or concurrent-behavior proof.
Maintained regression evidence also satisfies the pre/post criterion: the same generated dense-MTP fixture aborts on unmodified7e0 in set_input_k_idxs, while the merged change passes the ordinary Qwen4Exp architecture/roundtrip, dense MTP and QSA MTP cases (3 tests,0 failures). The target upstream QSA optimizations remain intact. All #11 acceptance criteria are met; close this startup prerequisite. Broader performance, context>=200k, rollback/checkpoint and concurrent behavior acceptance remain in open #9. The owned GPU-task idle lease was released and read back inactive after the checks.