HRX: report or refuse CPU fallback (strict mode); Flash-Next UD-IQ4_XS 18x slower than a peer engine

HRX: report or refuse CPU fallback (strict mode); Flash-Next UD-IQ4_XS runs 18x slower than a peer engine

When part of a graph silently runs on the CPU, nothing tells us. On Qwen3.8 Flash-Next UD-IQ4_XS, HRX is
far behind another engine on the same box, and the host was busy during the run. We cannot tell from the
logs which ops ran where.

Measurement (strixhalo, GPU only, 2026-10-11)

Engine 84140c7 (llama.cpp 0e255d77, hrx-system 98d05d94), llama-bench -ngl 99 -fa 1 -dev HRX0 -ub 512,
same GGUF for both engines, box.lock held, coder-llm and 1bit-canonical stopped:

engine	prefill	decode
gufo 8e0fb1d (mikealanni/gufo, MIT, HIP)	pp16384 1,679 ± 2 tok/s	tg128 33.8 tok/s
ours, HRX0	pp4096 92 tok/s	tg64 4.8 tok/s

Our pp16384 x3 arm did not finish: it reached Tctl 92 C after 17 min and was stopped.
Load average was 6-8 during the HRX runs, against 1-3 for gufo.
For contrast, on Qwen3.8-27B UD-Q4_K_XL gufo is 1.9x faster at prefill (609-614 vs 323.8 tok/s) and decode is level (about 12 tok/s). The Flash-Next gap is of a different order.

What the support query says

test-backend-ops support -b HRX0 reports MUL_MAT and MUL_MAT_ID with type_a=iq4_xs as SUPPORTED
(3 of 3 MUL_MAT_ID cases), so the expert matmuls are not obviously rejected. GET_ROWS iq4_xs with
be1=7 is NOT SUPPORTED. Where the host time goes is therefore still open: other ops placed on CPU,
graph splits, or slow supported kernels.

Ask

A strict mode, for example GGML_HRX_STRICT=1: fail with the op name, type and shape instead of placing an
op on the CPU. Precedent: an open RK3588 NPU runtime (poad42/opennpu_rk3588) never falls back
silently and errors on unsupported ops.
Without strict mode, one line at load listing every op type placed off HRX, with its count, plus the graph split count.
Use 1 and 2 to explain the Flash-Next gap, and check how many graph splits Flash-Next produces per token on HRX.

Related: #389 (prompt batches above 512 fail with res = -3). Evidence on strixhalo:
~/wt/gufo-bench/results/flashnext-* (command, Tctl, load, memory per arm).

Status: no code change yet; a way to answer question 3 today with what ggml already has.

ggml's scheduler can already print where every node runs. Run the same Flash-Next bench with GGML_SCHED_DEBUG=2. It prints each graph split with its backend and the nodes assigned to it, so a single decode step shows the split count and every op placed on the CPU. That covers ask 2 for diagnosis and tells us whether ask 1 is about a few ops or many.

What the support query already rules in and out:

GET_ROWS iq4_xs NOT SUPPORTED is real but cheap. It is the token-embedding lookup: one row per token, so it costs a split, not compute.
MUL_MAT and MUL_MAT_ID on iq4_xs report supported, so if the expert matmuls stay on HRX, the 18x gap is either many splits per token (each a GPU-CPU round trip, which would also explain the load of 6-8) or a slow supported kernel.

Next step: one decode token and one 512-token prompt batch under GGML_SCHED_DEBUG=2, counting splits and CPU-placed nodes by op type. If splits dominate, the fix is HRX coverage for those ops. If they don't, profile the iq4_xs MUL_MAT_ID kernel against the Q4_K one on the same model shape. A strict mode, ask 1, is still worth adding as a guard either way.

Related: #389 caps prompt batches at 512, which also limits prefill here.

serve /metrics: only what Lemonade cannot see

serve /metrics: drop graph splits; parse backend lines with a substri…

The 18x figure compares two different prompt lengths, and the HRX arm never finished

Before the gap is explained, the measurement needs a caveat. From the stored
evidence in ~/wt/gufo-bench/results/:

arm	prompt	prefill	decode	outcome
gufo (ROCm/HIP)	pp16384	1679.19 ± 2.37	tg128 33.79 ± 0.18	rc=0, peak Tctl 80 C
gufo	pp32768	1653.29 ± 2.55	tg1 33.93	rc=0, peak Tctl 82 C
ours, HRX0	pp16384	—	—	ABORTED, Tctl 92 C, rc=143
ours, HRX0	pp4096	91.97	tg64 4.76	rc=0, peak Tctl 88 C

So the ~18x in the title is gufo pp16384 (1679) against ours pp4096 (92).
Those are not the same measurement: prefill throughput generally rises with prompt
length, which favours gufo, and our own pp16384 arm aborted on thermals rather
than completing. At matched context the decode gap is 33.79 / 4.76 ≈ 7.1x.

The gap is real — the load averages support it (6-8 during our runs against 1-3 for
gufo) — but the number to chase is the like-for-like one, and an HRX pp16384 run has
not completed yet.

Where the host time likely goes

test-backend-ops support already gives the strongest candidate: GET_ROWS iq4_xs
with be1=7 is NOT SUPPORTED, while MUL_MAT / MUL_MAT_ID with type_a=iq4_xs
are supported (3 of 3 MUL_MAT_ID cases). So the expert matmul is on HRX but the
expert gather is not.

That is consistent with the rest of the evidence: this is a 176.94 B A3B MoE
(87.25 GiB), so the expert gather runs per token, and moving it to the CPU would
depress both prefill and decode and drive host load up — exactly the signature seen.

The mechanism that makes it silent is in device_supports_op()
(ggml/src/ggml-hrx/ggml-hrx.cpp): the backend deliberately declines ops it cannot
execute, and ggml then places them on the CPU with no log line anywhere. There is
also a hard edge: if the tensor is already in HRX memory, the same function
returns eager_capability_declared(op->op) unconditionally — the node then cannot
migrate to the CPU and the whole graph fails with GGML_STATUS_FAILED (ret = -3),
which is exactly what #389 was.

A second silent-fallback site, found while root-causing #389: the prefill matmul
matchers stop at 2048 tokens, so on the 27B the qkv projection goes to the CPU at
-ub 4096. At the -ub 512 used for these Flash-Next runs that limit is not
reached, so it is not the cause of this gap — but it is another unreported
off-HRX placement.

On the ask

Both asks remain unfulfilled and both look implementable now that the failure paths
are understood — a strict mode (GGML_HRX_STRICT=1) that errors with the op name,
type and shape instead of placing an op on the CPU, and a one-line load summary of
every op type placed off HRX with its count plus the graph split count. The split
count is already computed (the #389 work counted 97 per ubatch on the failing
qwen35 case), and the unsupported-node diagnostics already carry op name, shape and
inputs — they are simply discarded in the fallback path instead of reported.

Related: #389 (now root-caused, fix in progress).

Not verified: that GET_ROWS iq4_xs is the dominant off-HRX cost for Flash-Next.
That needs test-backend-ops plus a profile of an HRX Flash-Next run; the above is
the candidate best supported by the existing evidence, not a measurement.

Asks 1 and 2 are implemented and measured on real models; ask 3 is narrowed but not closed

The instrument is merged on the fork: llama.cpp PR #102, squash-merged as 1a5dcba2 on 1bit/hrx-vulkan-patched. (The engine still pins ad18320a, so the tool is not in engine builds until the next bump — that is expected, not a gap.)

Ask 2 — the placement report, on real models (gfx1151, Tctl/load recorded per arm, max 62 C, nothing throttled)

model	placement	throughput
Qwen3.8-27B dense (`qwen35`)	`3655 nodes, 0 off HRX (none), 1 graph splits` — prompt and decode	pp512 332.79, tg32 11.32 tok/s
Ornith-1.5-35B Q4_K_M (`qwen35moe`, 256 experts / 8 used)	`3727 nodes, 0 off HRX (none), 1 graph splits`	pp512 1151.89, tg32 37.20 tok/s

That second row is the substantive finding: a per-token expert gather runs fully on HRX with a single graph split, so "the MoE forces a CPU split per token" is not true on this backend in general. It also means the 7.1x like-for-like decode gap is not explained by generic MoE splits.

Ask 1 — strict mode, demonstrated by an actual refusal

GGML_HRX_STRICT=1 on the 27B at -ub 512: exit 0, no abort (correct — nothing is off HRX). At -ub 4096, the refusal the issue asked for:

placement_observer: GGML_HRX_STRICT=1: "model.input_embed" is GET_ROWS f32 ne=[5120,4096,1,1] and is not placed on HRX;
438 of 3655 nodes off HRX (MUL_MAT x 389, GET_ROWS x 1, SSM_CONV x 48), 806 splits

exit 134 (SIGABRT), no core file written. Off by default: unset/0 prints nothing and changes no placement decision.

Ask 3 — narrowed, still open

test-backend-ops support -b HRX0 confirms the issue's own lead precisely: GET_ROWS(type=iq4_xs, …, be1=7) NOT SUPPORTED, while GET_ROWS be1=1 and MUL_MAT_ID iq4_xs are SUPPORTED. That one op is the only confirmed iq4_xs coverage gap, so the remaining split story is specific, not general.

Flash-Next itself was not run, deliberately: it is ~88 GB across three parts against 41 GB free / 68 GB available RAM with 27.7 GiB of GTT already held by two resident llama-servers, so loading it risked OOM-killing another agent's server. The next measurement to close ask 3 is named rather than implied: GGML_HRX_REPORT=2 / STRICT=1 on Flash-Next UD-IQ4_XS on a box with ~90 GB free RAM, looking for GET_ROWS iq4_xs be1=7 in the decode graph and how many splits it costs. If it never appears there, the gap is a slow supported kernel rather than coverage — which is the other branch of the issue's own question.

Also worth carrying forward from the thread, independent of this work: the 18x headline compares gufo pp16384 (1679 tok/s) against our pp4096 (92 tok/s), and our pp16384 arm aborted on thermals; the like-for-like decode gap is ~7.1x.

Evidence: team/evidence/390-strict-mode-and-placement-report.md §4 plus ten 390-strixhalo-*.log raw arms; mechanism/local validation in the same file §1-3.

Follow-up: the instrument is now on its way into the engine pin

PR #424 moves third_party/llama.cpp to 1a5dcba2 (the one commit that adds the strict mode and placement report), so asks 1 and 2 become available in engine builds rather than only on the fork. registry/architectures.json is regenerated with it, and the pair (1a5dcba2 + the pinned hrx-system) is being built with ONEBIT_HRX=ON to confirm before merge.

Still open, unchanged: ask 3. Flash-Next is ~88 GB and the Strix Halo box currently reports 122 GB total / ~69 GB available with 27 GiB of GTT already held by two resident llama-servers that are not ours to stop, so loading it would risk OOM-killing another session's service. The measurement to close it is unchanged: GGML_HRX_REPORT=2 / STRICT=1 on Flash-Next UD-IQ4_XS in a ~90 GB-free window, looking for GET_ROWS iq4_xs be1=7 in the decode graph and its split cost — if it never appears, the gap is a slow supported kernel rather than coverage.

Mechanism result (small-model proxy): the failing case is batched `GET_ROWS`, not IQ4_XS

Asks 1 and 2 are now shipped: PR #424 put the strict mode and placement report into the engine pin (built from a fresh tree at that exact pair, exit 0; op gate 1141/1141; GGML_HRX_REPORT=1 -v prints 986 nodes, 0 off HRX (none), 1 graph splits; STRICT=1 does not fire when nothing is off HRX; unset is silent).

For ask 3, a cheap proxy was run rather than waiting for an 88 GB load: Qwen3-0.6B-iq4_xs.gguf vs Qwen3-0.6B-Q4_K_M.gguf on HRX0 with the placement report. It is a proxy — a 0.6B dense model, not the 177B-A3B MoE — and it must not be read as the Flash-Next explanation.

Two findings that do generalize

1. The unsupported case is batched GET_ROWS, independent of the quant.

case	verdict
`GET_ROWS(iq4_xs, be1=1)`	SUPPORTED
`GET_ROWS(iq4_xs, be1=7)`	NOT SUPPORTED
`GET_ROWS(q8_0, be1=1)`	SUPPORTED
`GET_ROWS(q8_0, be1=7)`	NOT SUPPORTED

be1/be2 are batch dimensions in the test generator (tests/test-backend-ops.cpp:2263: source [n,m,be1,be2], index [r,be1,be2]). So the issue's phrase "GET_ROWS iq4_xs be1=7 … NOT SUPPORTED" names a real failing case but attributes it to the quant — the quant is not the variable, the batch dimension is. q8_0 fails the identical case. That is a sharper statement of the gap than the issue starts with, and it matters for where a fix would go.

2. Placement is not the cost on this shape. Both quants, prompt and decode graph: 986 nodes, 0 off HRX (none), 1 graph splits. Nothing about the token gather falls back, for either quant. Meanwhile the prefill difference between the two quant mixes is −21.6 % (pp512 14799.54 vs 18878.45) at identical placement — so on this backend a ~22 % prefill delta comes from kernel throughput between quant mixes, with no split involved. That is the honest shape of "quant choice costs, fallback doesn't" on a model where nothing falls back.

What it explicitly does not show

The -iq4_xs file does not put IQ4_XS where a GET_ROWS happens: its token_embd.weight (and output.weight) are Q8_0; the 196 IQ4_XS tensors are block matmul weights. So this model cannot exercise GET_ROWS iq4_xs at all, and the run says nothing about Flash-Next's gather. It also says nothing about the 18x headline or the ~7.1x like-for-like decode gap.

Remaining unknown, and the next measurement

If some graph in Flash-Next issues a batched GET_ROWS (be1 > 1), HRX declines it and it goes to the CPU; the 0.6B dense graph has no batched gather, so this proxy cannot see it. Next measurement is unchanged: GGML_HRX_REPORT=2 on Flash-Next UD-IQ4_XS (qwen4exp, 512 experts / 10 used) in a window with ~90 GB free RAM, looking for a GET_ROWS with be1 > 1 in the decode graph and its split cost. If it does not appear there either, the gap is a slow supported kernel rather than coverage.

Evidence: team/evidence/390-iq4-xs-proxy.md, 390-engine-pin.md and the raw logs beside each.

Ask 3: the gap is batch-size scaling of our prefill, not CPU fallback — measured like-for-like

Asks 1 and 2 are shipped in the engine pin (PR #424, 2df1a9b): built from a fresh tree at that exact pair, op gate 1141/1141, GGML_HRX_REPORT=1 -v printing the placement line, STRICT=1 correctly not firing when nothing is off HRX, unset silent.

The thread already concedes the 18x was not like-for-like (gufo pp16384 vs our pp4096, our pp16384 arm aborted on thermals, decode ~7.1x). So the comparison was re-run at matched prompt lengths on the same box, against the same peer engine the issue names.

Matched-length, same model, same box: Qwen3.8-27B UD-Q4_K_XL (`qwen35` dense)

prompt	ours (HRX0) pp	gufo pp	ratio	ours tg128	gufo tg128
512	329.70	405.22	1.23×	11.43	11.26
2048	332.87	572.34	1.72×	11.35	11.46
4096	324.70	614.54	1.89×	11.39	11.36

All contended (load 11, 27 GiB GTT held by resident services, MemAvail 65–71 G, peak Tctl 73–91 °C; the 91 °C arm is why pp8192 was not attempted). -ngl 99 -fa on -ub 512.

Two things to take from this:

The 1.9× is reproduced like-for-like at pp4096, which is what the issue's pp16384 comparison was really showing — so the number was right even though the comparison was not.
The shape is the finding. gufo's prefill rises 405 → 572 → 615 (+52 %) as the prompt grows; ours is flat 330 → 333 → 325 (±1 %). The ratio therefore grows with prompt length. Decode is level within noise (11.4 vs 11.3–11.5), exactly as the issue says.

Coverage is not the cause — three models, prompt length 512 → 4096

model	placement at pp512 / pp2048 / pp4096
Qwen3.8-27B dense	`3655 nodes, 0 off HRX (none), 1 graph split` at all three
Ornith-1.5-35B (`qwen35moe`, 256 experts)	`3727 nodes, 0 off HRX (none), 1 split`
Qwen3.6-35B-A3B MoE	`3727 nodes, 0 off HRX (none), 1 split` at all three; pp 1075.95 / 961.51 / 963.02

So "how many graph splits per token on HRX" is one, with nothing off HRX, on every model measurable here and at every prompt length. The issue's premise that the gap is the expert gather falling to the CPU is not what happens on this backend; the only confirmed coverage gap remains the batched GET_ROWS (be1 > 1) I reported earlier, which these graphs do not issue.

Cross-check against a peer backend on the MoE

Qwen3.6-35B-A3B MoE, HRX0 vs the resident canonical Vulkan build (a peer backend on this box, not gufo): prefill 961.51 vs 1081.32 (pp2048) and 963.02 vs 1039.51 (pp4096) — within 8–12 % — while decode is ~1.4× behind (36.2–36.8 vs 51.2–51.9). The Vulkan pp512 arm aborted at 92 °C 15 s in and is recorded, not retried.

Where that leaves Flash-Next — and what stays unknown

Flash-Next itself is still not measured. The only quant on this box is the 88 GB UD-IQ4_XS; the box reports 122 GB total / ~69 GB available with 27 GiB of GTT held by services that are not ours to stop. Loading it would risk OOM-killing another session's server, so I did not.
A matched MoE comparison against gufo is impossible here: gufo cannot load any qwen35moe model at all — Error creating Qwen GPU executor: Missing required Qwen tensor: blk.0.ffn_gate.weight — because src/models/qwen/weights.cpp:246 requires a dense ffn_gate.weight per layer and its MoE executor is specific to qwen4exp. Recorded so nobody retries it blind.
None of this quantifies Flash-Next's 92 tok/s pp4096 or 4.76 tok/s tg64, and it should not be read as its explanation.

The actionable conclusion

On everything measurable, our prefill is single-split with zero off-HRX ops at every prompt length, yet it does not improve with batch size the way both peer engines do — gufo gains 52 % from pp512 to pp4096, we gain nothing. That points at the batch-scaling of our prefill path (kernel/dispatch efficiency at larger -ub), not at coverage, and it is a much more specific target than the issue started with. Next steps, in order: (1) Flash-Next's own GGML_HRX_REPORT=2 run on a box with ~90 GB free, looking for any off-HRX op at pp4096; (2) profile the prefill kernels against batch size on the 27B where the curve is flat.

Evidence: team/evidence/390-like-for-like.md with 15 raw logs, plus 390-iq4-xs-proxy.md and 390-engine-pin.md.

serve: GET /metrics with the engine-only series Lemonade cannot see (…

Engine side: #442 (1bit/strict-device).

The llama.cpp half of this issue is already in the pin (#441 pins 8d9ce2c1a, which carries fork #102): ggml-hrx's scheduler knows the backend of every node, prints one line when GGML_HRX_REPORT is set, and aborts on the first node off HRX with GGML_HRX_STRICT. What was missing is the engine asking for either, so nothing reached the user.

Against the three asks:

Strict mode - 1bit serve --strict-device: serve sets GGML_HRX_STRICT=1, the backend aborts at load on the first node off HRX (its line names the node, op, type and shape), and serve exits non-zero with its own line instead of "backend did not become ready". If a report still names off-HRX nodes after a backend reached ready, serve refuses with the same numbers. Needs --device hrx or auto.
One line at load - --placement-report: serve prints
```
1bit serve: HRX placement: 34 of 512 nodes off HRX (GET_ROWS x 4, MUL_MAT x 30), 7 graph splits
```
(or all 512 nodes on HRX, 1 graph splits), and the same numbers are onebit_cpu_fallback_ops{op} in /metrics.
Explain the Flash-Next gap - not done here: that needs the GPU queue. This PR is the instrument: with --placement-report the per-token graph split count and the off-HRX op types are visible, and the GET_ROWS iq4_xs be1=7 case the issue mentions is exactly what the report would name.

One thing to know before reading the line on the box: llama-server drops the library's INFO lines below -lv 4 (common/log.cpp maps GGML_LOG_LEVEL_INFO to its trace level), so the backend's report does not reach serve at the default verbosity - that is why --placement-report passes -lv 4 for that load, and why step 2 is a flag rather than the default. If the report should be visible without a flag, the one-line fix is in the fork: log the placement line at WARN (as the program-cache flush line already is) instead of GGML_LOG_INFO. Say the word and I will do that as a separate fork PR plus a pin bump.

CPU-only test: tests/hrx_placement_test.cpp, ctest hrx_placement, in the CI build list. Full suite here: 32/32 pass, including metrics_serve, which already asserted onebit_cpu_fallback_ops from a fake backend's line.

Not verified: an actual HRX run (GPU queue), so the strict refusal is verified only through serve's own path and the parser in unit tests.

serve: report off-HRX placement at load, and --strict-device to refus…

Hygiene pass (2026-10-18): asks 1 and 2 are merged and verified on main; ask 3 is not done. Leaving this open, with the state written down here.

Merged. #442 (5516e16, "serve: report off-HRX placement at load, and --strict-device") is in origin/main, and the llama.cpp half is in the pin main carries (#441, llama.cpp 8d9ce2c1a = fork #102 with GGML_HRX_REPORT / GGML_HRX_STRICT).

Ask 1, strict mode: 1bit serve --strict-device sets GGML_HRX_STRICT=1; the backend aborts at load on the first node off HRX and serve exits non-zero with its own line instead of "backend did not become ready".
Ask 2, one line at load: --placement-report prints 1bit serve: HRX placement: N of M nodes off HRX (...), K graph splits, and the same numbers are onebit_cpu_fallback_ops{op} in /metrics.
Both documented in docs/serve.md "Off-HRX placement: --placement-report, --strict-device".

Verified here, CPU only (no GPU, no strixhalo job): took origin/main into a scratch tree, compiled tests/hrx_placement_test.cpp against app/serve_metrics.h with just -I app -I <nlohmann 3.11.3 single-include>, and ran it - hrx_placement: OK, exit 0. It is a real ctest: CMakeLists.txt has add_test(NAME hrx_placement ...) and .github/workflows/ci.yml lists hrx_placement_test in the build step. What that does not cover is the GPU path itself (a live load that actually goes partly to the CPU); that is a box job.

Ask 3 is still open and needs the GPU queue and the Flash-Next model: how many graph splits Flash-Next produces per token on HRX, and whether the GET_ROWS iq4_xs be1=7 NOT SUPPORTED case from test-backend-ops support is what the report names. The instrument exists now:

1bit serve -m <flash-next gguf> --device hrx --placement-report

should print the split count and the off-HRX op types for that load, and --strict-device will say whether the graph is complete on HRX at all. Nothing else is blocking 1 and 2.

The placement half of this issue, on the current line. Short version: on Flash-Next UD-IQ4_XS exactly one node of 7248 is off HRX, and it is a tiny f32 GET_ROWS — so the gap is not a mass CPU fallback.

Build and boot

engine main 8b6a0f4, third_party/hrx-system 2ab311496, third_party/llama.cpp 8d9ce2c1a (llama-bench reports 8d9ce2c1a (10643)), built -DONEBIT_HRX=ON; 1bit sha256 9dc92c08….
This boot: amdgpu.cwsr_enable = 1 (CWSR on, unlike the 2026-10-11 numbers), coder-llm and 1bit-canonical stopped and masked for the window, GTT 120 GiB free, and HRX0 confirmed in the log: - HRX0 : AMD Radeon 8060S Graphics (Node 1) (gfx1151) (122880 MiB free), load_tensors: HRX0 model buffer size = 61222.07 MiB.
1bit serve set its own placement for this model: n_parallel = 4, kv_unified = true, n_ubatch = 512, flash_attn = auto, and memory: need 88.50 GiB (weights 87.25 + KV/state 0.75 + compute 0.50) at ctx 8192, free 118.42 GiB.

Placement (`--placement-report`, GGML_HRX_REPORT=2)

placement_observer: placement: 7248 nodes, 1 off HRX (GET_ROWS x 1), 3 graph splits
1bit serve: HRX placement: 1 of 7248 nodes off HRX (GET_ROWS x 1), 3 graph splits
sched_reserve: graph nodes = 7932 / graph splits = 243

One node in 7248, op type GET_ROWS, count 1. The observer's line is the graph it saw at load (3 splits); the scheduler's own reservation for the 7932-node graph reports 243 splits.

Which node it is (`--strict-device` names it)

placement_observer: GGML_HRX_STRICT=1: "node_1" is GET_ROWS f32 ne=[160,32,1,1] and is not placed on HRX; 1 of 7248 nodes off HRX (GET_ROWS x 1), 3 splits
1bit serve: --strict-device: a backend refused the graph (GGML_HRX_STRICT=1); its own line above names the node

A 32-row gather of 160-wide f32 rows — the router's top-k gather, not the expert gather. The GET_ROWS be1=7 gap that test-backend-ops support reports (and which led this issue) is a different, larger case and does not appear on this model's path: its expert matmuls are MUL_MAT_ID on iq4_xs, which HRX places.

Bench (HRX0, 2 reps)

llama-bench -m <shard1> -ngl 99 -fa on -dev HRX0 -ub 512 -p 512 -n 128 -r 2
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | HRX | ngl 99 | fa 1 | HRX0 | pp512 | 102.93 ± 0.88 |
| qwen4exp A3B IQ4_XS - 4.25 bpw | 87.24 GiB | 176.94 B | HRX | ngl 99 | fa 1 | HRX0 | tg128 |   4.99 ± 0.01 |
build: 8d9ce2c1a (10643)

That is the same order as the issue's pp4096 92 / tg64 4.8 (this run is pp512/tg128, and against gufo's pp16384 1,679 / tg128 33.8).

Which kernel removes the off-HRX node, and the order I would work

GET_ROWS f32 ne=[160,32,1,1] — ggml/src/ggml-hrx/dispatch_registration/common/dispatch-get-rows.cpp claims gathers whose row width and token count fit the Qwen profiles (hidden-size checks, up to 262208 rows); a 160-wide, 32-id f32 gather matches none of them, so the node leaves HRX. The corpus already carries loom_libs/ggml_get_rows_f32 (generic contiguous-row f32) and small_rows_f32.loom (narrow rows), so the cheapest fix is the matcher: accept narrow f32 rows there, and add a narrow-row dispatch if the current tile needs a minimum width. That removes the only off-HRX node on this model.
Prefill throughput. With one node off and 3 splits at load, the prefill gap cannot be fallback; it is kernel throughput and chunk size — the same conclusion HRX: bump llama.cpp to 1a5dcba2 (engine#390 strict mode and placement report) #424 reached on other models. The next measurements are the prefill chunk the engine feeds (gufo chunks up to 8192 tokens; our line is -ub 512, and HRX: prompt batches above 512 tokens fail with res = -3 (ubatch capped at 512) #389 bounds the graph above that) and the iq4_xs matmul tile widths (dispatch-mul-mat.cpp, the q5_k q8 plane wmma / kquant_decode motifs).
GET_ROWS iq4_xs be1=7 — real per test-backend-ops support, but not on this model's path; last.

Caveats

No kernel changes (scoped to the measurement).
Rehearsal with the same flags on a small UD quant (Qwen3-0.6B-UD-Q4_K_XL) prints 986 nodes, 0 off HRX (none), 1 graph splits, so the plumbing is sound and the single node above is model-specific.
One --placement-report load and one --strict-device load; a per-token split count across a long generation was not sampled (the load graph is the 3 above).
llama-bench run directly out of the engine's build sees no device (--list-devices → none) until IREE_HAL_AMDGPU_LIBHSA_PATH points at the TheRock libhsa-runtime64.so.1; with it, HRX0 appears. Worth a line in docs/hrx.md.

🤖 Generated with Claude Code

Move the private HIP kernels pin: m128 dense prefill tile on by defau…

Ask 3 cannot be measured in the window this run is allowed to take: Flash-Next UD-IQ4_XS does not fit the box with the resident services up, and stopping them is outside this run's rules. Recording the arithmetic and the exact commands so the measurement is one authorised window away, rather than re-deriving it. Box state all 90 W cap, PCM pad (23 Oct), Tctl 66 C, nothing loaded by this run.

Why it does not fit (strixhalo, 2026-10-23 05:17)

quantity	value
GTT total (`mem_info_gtt_total`)	128,849,018,880 B = 120.0 GiB
GTT held by resident clients (`mem_info_gtt_used`)	84,390,047,744 B = 78.6 GiB (two `llama-server`s at 24.7 and 13.3 GiB, a third at 13.3, a fourth at 6.7 from `drm-total-gtt` in `/proc/*/fdinfo`)
GTT free	41.4 GiB
system RAM available	26 GiB (124 total, 98 used)
UD-IQ4_XS on disk	3 shards, 10,946,624 + 49,835,229,856 + 43,836,407,744 = 87.3 GiB
the model's own load line, same GGUF, 2026-10-20	`memory: need 88.50 GiB (weights 87.25 + KV/state 0.75 + compute 0.50) at ctx 8192`

41.4 GiB free against a 88.50 GiB need. The 2026-10-20 run in this thread only loaded it because coder-llm and 1bit-canonical were stopped and masked for that window (GTT 120 GiB free in that comment); this run's rules forbid stopping a service, and four resident llama-servers hold 58.75 GiB of GTT between them. Loading anyway would have the driver refuse the allocation at best and put another agent's server in the OOM path at worst, so the attempt was not made.

What the run is ready to do, unchanged from 2026-10-20 (`repaste 1 (paste)` for that table)

Binary: llama-bench 10669, built from third_party/llama.cpp af04c9f22 — the pin main carries — with our own HIP kernels compiled in as the release build does (a main engine rebuild produced the same pair today, 1bit at engine f826cb3, llama.cpp af04c9f2, hrx-system 82804197).

# placement first (GGML_HRX_REPORT=2 prints one line at load; STRICT=1 already refuses off-HRX nodes)
GGML_HRX_REPORT=2 llama-bench -m <shard1> -ngl 99 -fa 1 -dev HRX0 -ub 512 -p 512 -n 0 -r 1
# the issue's line, 3 repeats each
llama-bench -m <shard1> -ngl 99 -fa 1 -dev HRX0 -ub 512 -p 4096 -n 0 -r 3
llama-bench -m <shard1> -ngl 99 -fa 1 -dev HRX0 -ub 512 -p 0 -n 64 -r 3

One correction for whoever takes the window: --mtp is not a llama-bench option — tools/llama-bench/llama-bench.cpp has no mtp at the pin af04c9f2 or at the fork tip 851fa6c8d (--help on both prints nothing matching). --mtp is an engine flag (1bit serve --mtp <head.gguf>, app/serve.cpp), so the MTP arm is a serve-side decode measurement, not a bench row:

1bit serve -m <shard1> --device hrx --ctx-size 8192 --port 8291 --mtp <MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf>
# then one /v1/completions with a 1024-token prompt, max_tokens 256, and take timings.predicted_per_second

The comparison the issue asks for is unchanged: 11 Oct pp4096 92 / tg64 4.8 (ours, HRX0) against gufo pp16384 1,679 / tg128 33.8; the nearest current number on this model is the 2026-10-20 run's pp512 102.93 ± 0.88 / tg128 4.99 ± 0.01 at 8d9ce2c1a (10643), which predates the prefill/decode work merged since. Nothing in this comment supersedes that table.

Leaving this open: asks 1 and 2 are merged (#442, #482), ask 3's remaining item is the Flash-Next measurement above in a ~90 GiB-free window, and the open finding from the thread stays - the gap is prefill throughput on a graph with one off-HRX node, not a mass CPU fallback.

HRX: report or refuse CPU fallback (strict mode); Flash-Next UD-IQ4_XS 18x slower than a peer engine #390

Description

HRX: report or refuse CPU fallback (strict mode); Flash-Next UD-IQ4_XS runs 18x slower than a peer engine

Measurement (strixhalo, GPU only, 2026-10-11)

What the support query says

Ask

Activity

bong-water-water-bong commented on Oct 12, 2026

bong-water-water-bong commented on Oct 12, 2026

The 18x figure compares two different prompt lengths, and the HRX arm never finished

Where the host time likely goes

On the ask

bong-water-water-bong commented on Oct 16, 2026

Asks 1 and 2 are implemented and measured on real models; ask 3 is narrowed but not closed

Ask 2 — the placement report, on real models (gfx1151, Tctl/load recorded per arm, max 62 C, nothing throttled)

Ask 1 — strict mode, demonstrated by an actual refusal

Ask 3 — narrowed, still open

bong-water-water-bong commented on Oct 16, 2026

Follow-up: the instrument is now on its way into the engine pin

bong-water-water-bong commented on Oct 16, 2026

Mechanism result (small-model proxy): the failing case is batched GET_ROWS, not IQ4_XS

Two findings that do generalize

What it explicitly does not show

Remaining unknown, and the next measurement

bong-water-water-bong commented on Oct 16, 2026

Ask 3: the gap is batch-size scaling of our prefill, not CPU fallback — measured like-for-like

Matched-length, same model, same box: Qwen3.8-27B UD-Q4_K_XL (qwen35 dense)

Coverage is not the cause — three models, prompt length 512 → 4096

Cross-check against a peer backend on the MoE

Where that leaves Flash-Next — and what stays unknown

The actionable conclusion

bong-water-water-bong commented on Oct 18, 2026

bong-water-water-bong commented on Oct 18, 2026

bong-water-water-bong commented on Oct 18, 2026

Build and boot

Placement (--placement-report, GGML_HRX_REPORT=2)

Which node it is (--strict-device names it)

Bench (HRX0, 2 reps)

Which kernel removes the off-HRX node, and the order I would work

Caveats

bong-water-water-bong commented on Oct 23, 2026

Why it does not fit (strixhalo, 2026-10-23 05:17)

What the run is ready to do, unchanged from 2026-10-20 (repaste 1 (paste) for that table)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Mechanism result (small-model proxy): the failing case is batched `GET_ROWS`, not IQ4_XS

Matched-length, same model, same box: Qwen3.8-27B UD-Q4_K_XL (`qwen35` dense)

Placement (`--placement-report`, GGML_HRX_REPORT=2)

Which node it is (`--strict-device` names it)

What the run is ready to do, unchanged from 2026-10-20 (`repaste 1 (paste)` for that table)