[bong-water-water-bong]

Follows the pin-side change (1bit-MONSTER/llama.cpp #137): the next-layer routing predictor that fits Qwen3.8-Flash-Next's hyper-connections. Nothing here changes a running configuration — --moe-predict does nothing without --moe-slots, which this build refuses until the HRX streamer lands.

Why

The prefetch that hides a streamed model's expert reads used the gate-ahead router. On Flash-Next that predictor is wrong by construction: with four 2560-wide residual streams, layer l+1's router input is l+1's own mixer applied to the residual after layer l's MoE, so applying l+1's router to layer l's FFN input mixes through the wrong weights. It was right about two times in three, and every wrong guess is a read from a drive that is already the limit: 133 s per 512-token pass without prefetch, 247 s with depth 1 at 4,608 slots (docs/moe-streaming.md, "Flash-Next, streamed").

What lands

• docs/moe-streaming.md, new subsection "The next layer's own block (Flash-Next; untested on device)": what the predictor computes (l+1's PLE injection, its attention or DeltaNet block, its combine, then its FFN mixer and router; ranking only, deeper than the layer's top-k), why an attention successor needs no state suppression and the recurrent/conv/PLE histories do (scratch), the flag and its measurement variables (_DEPTH, _MARGIN, _MIN, _LAYERS, _OUT, _STATS), and the explicit statement that it is untested on device and what is missing (the prefetch consumer, i.e. the streamer). The "Next" list item 3 now points at the built path.
• docs/serve.md: the synopsis line and a short paragraph under the streaming section, with the same "untested on device" note.
• app/serve.cpp: --moe-predict off|router|block, refused if the value is unknown, passed through as ONEBIT_MOE_PREDICT beside ONEBIT_MOE_PREFETCH inside the ONEBIT_MOE_STREAMING_HRX block (the only place a streamed run reads it).

Verification

Docs-only + one option in serve; the engine is not built here (the HRX toolchain is not part of this host-side run, and no device run was made — that is the next step after the lane opens). The pin-side branch is built and its unit test passes on this box.

Device plan (from the private notes, in this order)

1. ONEBIT_MOE_PREDICT=router vs block, resident Flash-Next, hit rate of the next layer's experts; falsifier: block not >= 90 %.
2. Streamed, --moe-slots 4608 and 9216, --moe-prefetch 1 --moe-predict block: pass time against the 133 s baseline; falsifier: not below it at 4,608 slots (a correct predictor with no drive slack — then gate on resident share).
3. Long context 64K/128K for the indexer's per-layer cost.

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit b27c692)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 137 - Partially compliant Compliant requirements: MoE expert slot grouping by size classes implemented Memory usage reduced from 34.4 GiB to 27.0 GiB for Flash-Next (measured on Strix Halo) --moe-predict flag added with support for off, router, and block options Documentation updated in docs/moe-streaming.md and docs/serve.md New predictor implementation for Flash-Next's hyper-connections Non-compliant requirements: The PR does not include the actual HRX streamer that consumes the prefetch predictions (as noted in docs) Requires further human verification: Verification that the new predictor actually works correctly on device (since it's marked as untested) Confirmation that the memory savings are consistent with the claimed 27.0 GiB for Flash-Next

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Untested Prediction Method The new --moe-predict block functionality is explicitly marked as "untested on device" in the documentation. This is a significant concern because the prediction method changes how experts are prefetched, which could affect correctness and performance. The PR description states that this is a "next-layer routing predictor that fits Qwen3.8-Flash-Next's hyper-connections" but without device testing, there's no guarantee that the implementation will work as expected in real hardware conditions. ### The next layer's own block (Flash-Next; untested on device) The gate-ahead predictor above does not fit Qwen3.8-Flash-Next: its layers keep four 2560-wide residual streams (hyper-connections), so layer l+1's router input is l+1's own mixer applied to the residual *after* layer l's MoE, and applying l+1's router to layer l's FFN input mixes through the wrong weights. The measured result is in "Flash-Next, streamed" below: right about two times in three, and a loss on a drive that is already the limit. The predictor that does fit runs layer l+1's own token-mixer block on the current residual, before layer l's MoE output exists: l+1's PLE injection, its attention or DeltaNet block, its combine, then its FFN mixer and its router. Only the ranking of the router logits is used, and more candidates than the layer routes to are kept so the near misses are read too. The output of the model does not change: the block only decides which experts a prefetch reads, and whatever a token really routes to is still waited for. State writes of that speculative pass are suppressed where the exact pass would read them: the DeltaNet recurrent state and the conv histories are written into scratch tensors, and an attention successor needs nothing, because the exact block rewrites the same cache cell before anything reads it. The speculative nodes are also left out of the fused-op manifest, so they cannot make a fused dispatch claim the wrong tensor. `--moe-predict off|router|block` (env `ONEBIT_MOE_PREDICT`) selects it, and takes effect with `--moe-prefetch 1` once the HRX llama.cpp runs the streamer; `router` keeps the cheap residual-only form for comparison. `ONEBIT_MOE_PREDICT_DEPTH`, `_MARGIN` (read a deeper candidate only while its logit is within the given distance of the best one), `_MIN`, `_LAYERS` and `_OUT` (a trace of predictions and the routing that was really used) tune and measure it, and `ONEBIT_MOE_PREDICT_STATS=1` prints the hit rate per layer when the process exits. The graph side is `src/onebit-moe-predict.*` plus one hook in the `qwen4exp` graph in the llama.cpp pin, and it is **untested on device**: the predictor, its readback and the hit-rate table are built and unit-tested host-side (`tests/test-onebit-moe-predict.cpp`), and the prefetch consumer (the streamer's in-graph op) is the step that needs the GPU lane. Missing Validation for --moe-predict The --moe-predict flag is added to the command-line interface but there's no validation that it's only used with --moe-slots and --moe-prefetch. According to the documentation, --moe-predict only takes effect with --moe-prefetch 1 and requires the HRX streamer to consume it. The current implementation allows setting --moe-predict without these flags, which could lead to unexpected behavior or runtime errors. else if (a == "--moe-predict") { o.moe_predict = next(); if (o.moe_predict != "off" && o.moe_predict != "router" && o.moe_predict != "block") { throw std::runtime_error("--moe-predict takes off, router or block, not '" + o.moe_predict + "'"); } }

[bong-water-water-bong]

Review (orchestrator): serve flag passthrough + docs, pairs with fork #137; draft until that one's device numbers are in, then both merge with the pin.

[github-actions]

Persistent review updated to latest commit b27c692

[github-actions]

Persistent review updated to latest commit b27c692