Repository navigation
[model] Nex DFlash draft lacks M-RoPE metadata and rejects cold image positions #6
Description
Current status — deployed and accepted (2026-10-01)
The owner authorized implementation, matched host/RPC build and deployment into the current proxy backend binary paths, end-to-end tests and closure. PR #10 is merged as 98b3e092d34b99222d9144f90e388767f02ec117; both installed manifests and running native/library maps verify this source. Applicable maintained and live acceptance is complete.
The binary infers temporal-only M-RoPE for a legacy DFlash draft only when the multimodal target requires it and the draft has no explicit sections. Original Nex cold image, full media-prefix reuse, text controls and proxy8082 vision passed; original target/mmproj/draft SHA256 values are unchanged. Exact Nex warm/cold generated-token equality retains the owner-accepted boundary recorded in issue #1; the observed variation and unisolated cause remain in the acceptance report.
Final deployment and acceptance evidence. The earlier pending investigation/design/delivery statements below are historical and superseded by this accepted endpoint. Separate Settings/Requests cancellation work belongs to novkien/llama-proxy#506.
Historical investigation and intake
Problem and impact
The configured Nex-N2.5-mini-IQ4_NL route cannot generate from a cold inline image with its existing Qwen3.6-35B-A3B DFlash draft. The native server returns HTTP 500 before any media RAM-cache restore is exercised. This was found while accepting the independent RAM retention repair in fork issue #1.
Expected and actual behavior
A vision-capable target with its draft should either process the supported media positions or give a clear unsupported-configuration diagnostic. Actual draft decode receives position 48 while its cached sequence ends at position 15 and fails the consecutive-position validator.
Reproduction and evidence
- Tested source: fork
95e6565970621aa508e80fd805dcf73af413093c, deployed native/RPC paired build; native executable SHA256cd5d9185c1e781e9716cbd323fb6295ff6b89c14aabc8010ac75f424bfb0ac78. - Target: Nex-N2.5-mini-IQ4_NL, q8_0 KV, three V100 CUDA devices plus RPC0, context 400000, parallel 8, split 32,32,32,4.
- Draft:
Qwen3.6-35B-A3B-DFlash-williamliao-Q8_0.gguf, SHA256fc79ee2c84688674e4f134d5797a6baee4884167659a6eec8866c7789ee1318b, DFlash type, RPC0, max 7/min 1. - Send a deterministic 32x32 RGB P6 PPM as base64 in
/completionsusing the runtimemedia_marker, a chat-like user/assistant prefix, n_predict=16, temperature=0, top_k=1, return_tokens=true. The disposable acceptance fixture records the image SHA and request construction; no private image was used. - Native error: draft
llama_process(ctx_dft)returned -1 for three tokens, offset 0. Sequence memory X=15 and incoming Y=48 violated Y=X+1; the client receiveddecode() failed: failed to process speculative batch. - The failure happened on the first media request, before snapshot restore. Text A/B/A RAM reuse independently produced cache_n=20015/prompt_n=65. Exact text warm/cold output differences are being tested against a RAM-disabled continuation control rather than presumed to be serialization damage.
Prior searches: open and closed upstream issues for DFlash multimodal/positions/M-RoPE and this fork's DFlash/RoPE issues returned no matching report. Fork #5 concerns PP/TG admission and is independent.
Investigation
The old draft is architecture dflash, six blocks, embedding length 2048, target layers [2,7,12,17,23,28,33,38]. It lacks dflash.rope.dimension_sections, so the model uses NEOX with one position per embedding. Pinned image embeddings are skipped by DFlash processing; its cached text positions end at 15, while the target's following text uses temporal position 48. The one-position validator rejects the gap.
The target uses M-RoPE [11,11,10,0]. Current conversion code in conversion/qwen.py explicitly exports temporal-only sections [head_dim/2,0,0,0] for a DFlash draft of an M-RoPE target. The known working Qwen-27B DFlash2 file has [64,0,0,0]. These observations support missing draft export metadata as the cause; a repaired derivative still needs actual validation.
The old draft also lacks dflash.attention.causal, but that is not the proposed fix: DFlash initializes the local value to false and explicitly sets noncausal attention. A scalar causal override does not change the position validator. The generic CLI override does not supply array metadata.
Scope and decision
The owner authorized GPU testing/build/deployment of fork issues #1/#3 and persistent Qwen configuration. This separate finding does not authorize a new runtime-source patch. No backend source or original model artifact was changed for it.
The next bounded test is a derivative GGUF containing the converter-prescribed temporal-only M-RoPE array, retaining all original tensor bytes and other metadata. This is a reversible test artifact; permanent Nex model/config adoption remains a separate decision. If the export/capability premise is wrong or the derivative fails, propose a position-handling backend repair with evidence and obtain explicit owner approval before writing a candidate patch.
Acceptance
- Verify draft head dimension/target capability and the intended export metadata.
- Verify every tensor byte and all other metadata against the original derivative source.
- Repeat cold image and media append/borrowed snapshot/cold reference with no positional error.
- Verify matching text continuation behavior and unchanged target/draft weights.
- Adopt the accepted artifact/config through authorized production delivery, or record an explicit decision to retain the limitation.
Status and next action
Open, Codex root task. Diagnosis is read-only; the media failure is recorded independently of issue #1's RAM capture/lookup repair. Validate the derivative while continuing already-authorized Qwen delivery, then report the concrete result and remaining adoption decision. Do not close this at a metadata hypothesis or a passing text RAM test.
Current authorized delivery (2026-10-01)
The owner now explicitly requests implementation of fork #5 and #6, paired backend build/deployment into the llama.cpp-fork path selected by the current proxy, end-to-end tests for both, and closure after acceptance. At model-use stages force unload and proxy model-request blocking are authorized. Earlier research-only/diagnosis-only scope paragraphs above are historical. Codex root coordinates separate isolated source implementers from published base e58b770; implementation/review/build/deployment/acceptance remain pending. The latest implementation claim comment owns the current handoff.
Activity
novkien commentedon Oct 1, 2026
Implementation claim and owner authorization
The owner explicitly authorized implementing #6 together with #5, end-to-end deployment into the
llama.cpp-forknative/RPC binary stream currently used by the proxy, tests for both issues, and closure after acceptance in Codex on 2026-10-01. At model use the owner authorizes force unload and blocking proxy model requests. Runtime-source repair and deployment are now authorized; the previous diagnosis-only restriction no longer describes this assignment.Accountable coordinator: Codex root task. Source implementer:
/root/issue6, isolated worktree/home/jarvis/.codex/worktrees/llama-fork-issue6, branchcodex/issue6-dflash-mrope, clean basee58b770f3c2346deab036d9e4c3b2ea61ca0bbc5. No competing open fork PR was returned. Root coordinates the independent scheduler repair #5, review, publication and all model/resource operations.Trace the existing legacy DFlash/M-RoPE gap and implement the smallest binary-level compatibility repair supported by actual target/draft capability evidence. Preserve original model tensors/artifacts and text semantics; do not bypass native position validation blindly. Next: focused source checks and a reviewed candidate, paired backend rebuild/deployment, cold image plus media continuation/cache and text acceptance using the current route. The issue remains open until those observations pass; no repair/acceptance is claimed at intake.
novkien commentedon Oct 1, 2026
Work report — legacy DFlash compatibility candidate
Owner and scope: Codex root task, issues #5/#6, authorized binary delivery from published
e58b770f3c2346deab036d9e4c3b2ea61ca0bbc5.Done: Before creating a draft context, the binary infers temporal-only M-RoPE sections for a DFlash draft with no existing sections when the target declares M-RoPE or interleaved M-RoPE sections. Explicit draft sections, text-only targets and DSV4 remain unchanged; invalid rotary widths refuse draft-context initialization. Original GGUF files/tensor bytes are not edited. Existing allocator and RoPE tests now cover compatibility guards, the image-induced text position gap, and numerical temporal-only M-RoPE/NEOX text parity. Independent source review found no blocker; the invalid-dimension diagnostic was corrected to describe refusal rather than successful fallback.
The first valid CPU native run passed allocator tests (53 cases, 342 assertions, zero failures) and RoPE including temporal-only text parity. The final combined build includes added null-pointer/zero-width tests and issue #5; its focused CPU/regression run is pending. No original model/config or installed binary has been changed.
Remaining/next: Reviewed PR/merge and matched host/RPC binary manifests; real legacy Nex+DFlash cold image, media append/borrowed snapshot/cold reference, text continuation controls, and current proxy end-to-end checks. The force-unload/drain dependency is recorded in the #5 work report. Do not close this issue on a unit test or merge alone. Root owns final delivery and acceptance.
novkien commentedon Oct 1, 2026
Work report — reviewed combined candidate ready for publication
Owner/scope: Codex root, native fork #5/#6. Integration branch
codex/issues-5-6-delivery, basee58b770f3c2346deab036d9e4c3b2ea61ca0bbc5. One linked PR will carry the related native binary changes; both issues remain open through deployment/acceptance.Done/evidence: Independent source review approved the final integration after correcting a discovered incompatible-LoRA PP deadlock: DONE_PROMPT waiters no longer seed the next batch group. Native GCC13 build of server and focused test binaries passed. CPU-only allocator passed53 tests/345 assertions; RoPE temporal-only text equivalence and sampler tests passed. All five new completion phase cases passed, four existing n_probs variants passed, speculative with/without-draft parity passed, existing per-request LoRA passed, and the new incompatible-LoRA barrier case passed after sizing its fixture within the model's256-token training context. Earlier fixture failures (oversized prompts and mistaking tokens:[0] in progress events for TG) were corrected in maintained tests, not masked by the runner.
git diff --checkand Python syntax checks pass. Tests use disposable state/local fixture files/loopback ephemeral ports with CUDA hidden. Cached continuation and stream/nonstream equivalence checks are additionally running.Runtime/recovery: Owner approved exceptional proxy Cold restart after force-unload cancellation stalled. The restart cleared accepted/queued stale admissions;8082 reports zero active requests and model gate blocked. This is recovery, not deployment acceptance. No alarm reset or host power cycle occurred. Native installed binaries remain unchanged.
Remaining: Publish/review/merge exact candidate; build and install matching host/RPC packages at the existing proxy-selected binary path, verify manifests/process maps, then real model/proxy acceptance (unequal PP/TG barrier, legacy Nex image/cache/text). RPC VM startup currently fails with kernel AppArmor DENIED exec on the existing root-owned PCI-fix helper; a bounded local allow-rule decision is pending. Proxy Settings cancellation is separately authorized in novkien/llama-proxy#506 and builds on merged ggml-org#408/PR505. Neither a passing CPU suite nor merge closes #5/#6.
Next: Delivery delegate publishes this reviewed content unchanged; root retains paired build/deployment and all remaining acceptance.
novkien commentedon Oct 1, 2026
Work report — focused compatibility checks complete; combined PR prepared
Owner and scope: Codex root task, fork issues #5/#6; publication delegate for the unchanged reviewed integration branch
codex/issues-5-6-deliveryat basee58b770f3c2346deab036d9e4c3b2ea61ca0bbc5.Why: Record the focused evidence supporting the DFlash compatibility change before publishing it with the related scheduler work. Before draft-context creation, the binary infers temporal-only M-RoPE sections only when a legacy DFlash draft has no explicit sections and its target declares M-RoPE sections. Explicit draft metadata, text-only targets, DSV4, and original GGUF tensors remain unchanged; invalid rotary dimensions fail closed.
Done and evidence: The combined GCC 13 build produced the server and focused allocator, RoPE, and sampler binaries. CPU allocator checks passed (53 cases/345 assertions); the RoPE test passed its temporal-only M-RoPE versus NEOX text parity case, and sampler checks passed. Additional cache/text controls passed: four borrowed-snapshot cases, stream/non-stream equivalence, and cache/no-cache continuation. The cache/no-cache case was unskipped only in an out-of-source test wrapper. Python syntax and
git diff --checkpassed. These checks support the metadata guards and text behavior but do not prove cold image handling with the actual Nex/DFlash pair.Remaining: Merge and paired host/RPC installation are pending. Cold image, media append/borrowed-snapshot/cold-reference, text-continuation control, and current-proxy acceptance on the matching deployed binaries remain required; issue #6 stays open until those checks pass.
Next: Publish the reviewed head and hand off the exact PR revision to the root coordinator for paired build, deployment, and real-model acceptance.
novkien commentedon Oct 1, 2026
Work report — combined PR opened; type-check correction pending
Owner and scope: Codex root task, issue #6; combined PR #10 carries the reviewed
2664ff5f1b1a83386a7678dcba8b1e8d768845a7candidate. See the exact-head pre-merge handoff.Done and evidence: The PR links this issue and #5. The focused allocator/RoPE CPU checks and cached text continuation controls passed for the combined candidate; independent source review approved this head. GitHub type-check found two assertion-call errors in #5's new tests; they are recorded on the PR. Other workflow jobs were still running at handoff.
Remaining: Correct and re-review those assertions, let required checks pass, then merge and complete paired host/RPC installation plus Nex DFlash cold-image, media/cache continuation, and text acceptance. This issue remains open until deployed behavior is verified.
Next: Root coordinator owns the exact-head CI repair and remaining delivery/acceptance.
novkien commentedon Oct 1, 2026
Work report — type-check correction pushed to combined PR
PR #10 is updated at exact head
6dbd31a697ea6c402dd47f476a92e912103f4712; see the head-specific report. This fix changes only two #5 test failure paths; the DFlash implementation and native build remain unchanged. The CI-versionty 0.0.84check on the affected files, latestty, andgit diff --checkpassed. GitHub checks for the new head are pending/running. Merge, paired binary deployment, and live Nex DFlash acceptance remain pending; this issue stays open.novkien commentedon Oct 1, 2026
Work report — exact-head handoff ready to merge
PR #10 is reviewed at
6dbd31a697ea6c402dd47f476a92e912103f4712; the pre-merge handoff records scope, tests, checks, and remaining acceptance. Focused allocator/RoPE and cache/text controls passed, and current-head GitHub type-check is green. Broad Linux/Windows CI remains optional and in progress; no branch protection or repository ruleset gate is configured. Root owns merge and then paired binary deployment plus real Nex DFlash image/cache/text acceptance. This issue stays open until deployed behavior passes.server : isolate PP phases and infer DFlash M-RoPE (#5, #6)
novkien commentedon Oct 1, 2026
Work report — DFlash compatibility candidate merged; live acceptance pending
PR #10 merged at reviewed head
6dbd31a697ea6c402dd47f476a92e912103f4712as commit98b3e092d34b99222d9144f90e388767f02ec117; see the merge report. Focused allocator/RoPE and cache/text controls plus exact-head type-check passed. Optional CPU Linux and Server CI jobs remained in progress/queued at merge. No installed binary was changed here. Root owns paired host/RPC deployment, manifest/process verification, and Nex DFlash cold-image/media/cache/text acceptance. This issue remains open until deployed behavior passes.novkien commentedon Oct 1, 2026
Work report — immutable paired build and acceptance window
Owner/scope: Codex root, fork #5/#6, merged PR #10 at
98b3e092d34b99222d9144f90e388767f02ec117. Both issues stay open for binary deployment and real model/proxy acceptance.Done/evidence: The host's new isolated release build includes server, RPC server and existing runtime tools. Its complete34-entry manifest validates, reports
98b3e092…:e3b0c442…(clean source), and the staged server reports build11275/commit98b3e092/GCC13. Only unchanged architecture-specific GGML libraries are reused from the verified e58 dependency package; all affected llama/common/server libraries are rebuilt. Original installed binaries remain e58.The RPC VM was off. Owner separately approved (1) a local AppArmor read/execute allow for the existing root-owned PCI helper and (2) repairing the QEMU hook after its callback into virsh deadlocked libvirt. The reviewed replacement validates the guest XML against sysfs GPU/audio identities without re-entering libvirt; valid PCI passes, mismatched PCI fails. The helper remains unchanged. Exact old hook/profile backups remain local. The VM starts successfully and the guest-only build is running against exact merged98b3e092 with its own GGML package.
Shared GPU tasks caused host/control reboots and interrupted agents/build preparation. Root retained the published/tested source, resumed the guest stage and obtained the owner's explicit decision to prioritize fork #5/#6 deployment/tests before releasing the GPU. The normal proxy force-unload of the now-renamed
Qwen-Translatorreturnedok:true,status:unloaded,force:true; the model gate is blocked, no native child remains, and all four V100s are empty. This is the current acceptance window, not permission to change other model configurations or reset alarms/power-cycle the host.Remaining/next: Verify/stage guest manifest matching98b3e092, snapshot exact installed bins on both nodes, install the full pair at the actual
/home/jarvis/llama.cpp-fork/build/binpaths, then confirm loaded executable/library maps and affected real model behavior through8082. Functional CPU phase/LoRA/sampler/speculation/cache checks already passed; no GPU speedup is asserted from those tests. Separate Settings/Requests cancellation work continues in proxy ggml-org#506. Root owns remaining delivery/acceptance and closure.novkien commentedon Oct 1, 2026
Work report — matched native pair installed; real model acceptance started
Owner/scope: Codex root, authorized delivery of #5/#6 from merged PR #10, exact revision
98b3e092d34b99222d9144f90e388767f02ec117.Installed evidence: Both actual
/home/jarvis/llama.cpp-fork/build/bindirectories now validate the same clean source identity98b3e092…:e3b0c442…. Host manifest has 34 entries; RPC guest manifest has 28. Each architecture retains its own verified GGML dependencies. A guarded same-filesystem atomic directory exchange preserved the exact previously installed e58 binaries inbuild/bin.before-issues5-6-98b3e092-20261001on each node. No physical source checkout or model artifact was overwritten.The normal controller woke the host after idle shutdown. An intermediate manual-load job correctly failed on the deliberately temporary e58/98 pair mismatch; no model inference was accepted from that mismatched pair. After both installs and manifest verification, a fresh Qwen-27B load job
76caed3df38b44c59eddda388f228ca5succeeded. Native/propsreportsb11275-98b3e092d. Its running PID5749 uses the actual installed executable and all nine mapped project libraries hash-match the host manifest; RPC PID3846 uses the installed guest executable on50052. The requested model gate is blocked.Remaining/next: Real unequal multi-chunk PP cohort/TG barrier and late-arrival checks with MTP, two synchronized proxy8082 requests, followed by legacy Nex+DFlash cold-image/media/cache/text acceptance and unchanged original-weight hashes. Functional native CPU tests already passed; no new GPU performance claim is made. These issues remain open until their applicable live behavior is verified. Root owns acceptance and closure; Settings/Requests cancellation continues separately in proxy ggml-org#506.
novkien commentedon Oct 1, 2026
Acceptance and closure — deployed #5/#6
Owner/scope: Codex root. The owner explicitly authorized implementation, paired build/deployment into the backend path used by the current proxy, end-to-end tests and closure. Related PR #10 is merged as
98b3e092d34b99222d9144f90e388767f02ec117.Deployment: Both actual host and RPC-guest
build/binmanifests validate the same clean98b3 source identity, with34 and28 entries respectively. Exact prior e58 installed directories are backed up on each node. Normal proxy loads succeeded for Qwen-27B and Nex. Running executable and all mapped project libraries match the installed manifests;/propsreportsb11275-98b3e092d. Original model/route settings were preserved. No proxy source activation is claimed by this native binary deployment; separate cancellation feature ggml-org#506 still needs its own delivery.#5 live acceptance: Unequal Qwen prompts processed8,339 and4,247 tokens in9 and7 distinct PP progress positions. Neither emitted a real TG token before both PP totals completed. Each then generated64 tokens, with MTP draft counts50/49. A live native slot snapshot observed A/B in TG (
n_decoded=1) while admitted C was processing with zero PP tokens; C progressed and completed after the cohort drained. Two synchronized OpenAI streams through8082 both returned HTTP200, generated reasoning deltas and[DONE]. The initial observer's prompt-field assumption was corrected to explicit native slot IDs, and the proxy fixture counts reasoning as well as content. Maintained isolated tests separately cover cancellation, parent/child, incompatible LoRA, cache, sampler, stream/nonstream and speculative behavior. Earlier real-engine comparisons informed the owner's choice; no new performance tuning or speedup claim is included.#6 live acceptance: The original Nex target, mmproj and legacy DFlash draft now process the deterministic32x32 cold image with HTTP200,20 generated tokens and positive DFlash activity (
draft_n=64) instead of the previous position-validator500. The binary logs temporal-only M-RoPE compatibility inference before draft-context initialization. A/B/appendA restores1,036 cached tokens from the1,040-token image prefix and processes7 new tokens; cache-disabled reference reports1,043 processed/zero cached. Text A/B/appendA controls match the cold reference. The actual inline PNG vision request through8082 succeeds and emits generated output. SHA256 of all three original files matches the pre-test record; no GGUF/tensor derivative or configuration replacement was required.Retained output observation: The first strict harness reported one extra failure: warm media tokens differed from the cold reference. Bounded controls repeated both paths: cold repeats equal each other, warm repeats equal each other, with one different generated token at index11. No numerical or cache-state root cause is isolated. This is retained as evidence against the owner's existing Nex RAM acceptance decision, which explicitly removes exact Nex warm/cold token equality as a closure criterion. It does not negate #6's cold-position/media/cache/text/proxy acceptance, and no equality or new numerical fix is claimed.
Disposition: Applicable acceptance for #5 and #6 is complete on the installed matched pair. Close both as completed under the owner's requested endpoint. Unlimited or long TG still delays later PP as documented by the selected drain-to-completion policy; Nex warm/cold variation retains its prior accepted boundary. Root continues the separately authorized Settings/Requests cancellation delivery in proxy ggml-org#506.