HRX: prompt batches above 512 tokens fail with res = -3 (ubatch capped at 512)

HRX: prompt batches above 512 tokens fail with `res = -3`

On HRX0, any -ub above 512 fails at the first prompt batch. -ub 512 works.

Repro

strixhalo, engine 84140c7 (llama.cpp 0e255d77, hrx-system 98d05d94, the pins on main as of 2026-10-11), Qwen3.8-27B UD-Q4_K_XL, nothing else on the GPU:

IREE_HAL_AMDGPU_LIBHSA_PATH=/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 \
  build/hrx/llama/bin/llama-bench -m ~/models/qwen38/UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 -fa 1 -dev HRX0 -p 16384 -n 0 -b 2048 -ub 1024 -r 3

test_prompt: failed to decode prompt batch, res = -3
llama_bench: error: failed to run prompt warmup

-b / -ub	result
default (2048 / 512)	works: pp16384 323.80 ± 3.11 tok/s
2048 / 1024	res = -3
2048 / 2048	res = -3
4096 / 4096	res = -3

All runs: Tctl 51-74 C, 82 GB available, 26 GB of GTT held by other servers. No HRX error line is printed before the -3.

Why it matters

It caps prefill at 512-token ubatches. For comparison, gufo (mikealanni/gufo, MIT) on the same file and box runs its dense 27B prefill in batches of up to 4096 tokens and reaches 609-614 tok/s at pp16384, about 1.9x our 323.8. Decode is level at about 12 tok/s.

Evidence

On strixhalo, ~/wt/gufo-bench/results/hrx-27b-pp16384-ub{1024,2048,4096}.{err,meta} (command, Tctl, load, memory per run) and hrx-27b-pp16384-tg128.* for the working baseline.

Status, from reading the pinned llama.cpp line (no new run yet).

res = -3 is llama_decode mapping GGML_STATUS_FAILED from graph compute. In src/llama-context.cpp, the two llama-side paths that set FAILED both log an error first: "failed to apply memory context" and "failed to initialize graph". Neither appears in the report, so the failure most likely comes from the HRX backend's compute call itself.

On that side, GraphExecutionResult in ggml/src/ggml-hrx/runtime/graph-executor.h starts as GGML_STATUS_FAILED. Any early return that does not set it explicitly yields FAILED, and no HRX log line goes with it. That would match the silent -3.

Next step: rerun the -ub 1024 repro with HRX debug logging, and with a breakpoint or log on each return path of GraphExecutor::execute / can_execute. That names the check that rejects graphs above 512 tokens, for example a per-dispatch or workgroup-count bound, or a graph-program match limit. A fix should also make that path log a reason instead of failing silently.

Possibly related: the same session saw Flash-Next UD-IQ4_XS prefill at 92 tok/s on HRX with load 6-8, which suggests some IQ4_XS ops fall back to the CPU. That is a separate question, tracked outside this issue.

Root cause, and why no error line was printed

Two separate defects are behind this issue.

1. A claim that promised a node no matcher could execute

fused-context-claim.h claimed L2_NORM / SOFTPLUS / GATED_DELTA_NET for HRX
at any token count, but the gated-delta-net matchers only took token_count <= 512.
The claim decides placement, so the scheduler placed a node that no matcher could
execute and the whole graph failed. SSM_CONV above 512 tokens had no HRX path at
all (its rollback/generic Loom kernels declare n_t <= 512), so it went to the CPU:
97 graph splits per ubatch.

The failing node, with -v:

graph_compute: unsupported HRX node 2: L2_NORM output=3:f32[128,16,1024,1]<-L2_NORM \
  inputs=[2:f32[128,16,1024,1]<-VIEW] consumers=[16:GATED_DELTA_NET]
process_ubatch: failed to compute graph, compute status: -1
llama_decode: failed to decode, ret = -3

This is the exact hazard the TODO in ggml-hrx.cpp describes: "split this into
placement capability and exact graph execution capability once graph claiming owns
the full decision."

Note eager_capability_declared() lists GGML_OP_L2_NORM unconditionally, and
device_supports_op() returns that unconditionally for a tensor already in HRX
memory — where it cannot migrate to the CPU. So the failure is hard rather than
a silent fallback. That asymmetry is also the other half of #390.

2. The missing log line was `llama-bench`, not ggml

The "no HRX error line is printed before the -3" observation is correct, but the
cause is not in ggml or HRX. tools/llama-bench/llama-bench.cpp installed a null
log callback that discarded every line, errors included:

static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) {
    (void) level;
    (void) text;
}

The HRX diagnostic was emitted correctly all along. It is invisible without -v
only because llama-bench threw it away.

3. A second face at `-ub 4096`

At -ub 4096 on the 27B, the qkv projection went to the CPU (the prefill matmul
matchers stop at 2048 tokens) and the claimed SSM_CONV failed the graph with
unsupported HRX node 11: SSM_CONV output=18:f32[10240,4096,1,1]. A claim cannot
see its consumers, so a claim must also require the producers its matcher looks at
to stay on this backend.

Fix

Branch 1bit/hrx-ub389, five commits, 13 files, +521/−14:

ubatch-token-limits.h (new): shared bounds, so the claim and the matchers cannot
disagree again.
dispatch-ssm-conv-chunked.{h,cpp} (new): a conv prefill above 512 tokens runs as
a chain of at most 512-token dispatches, carrying the conv history forward.
dispatch-row-chunks.{h,cpp} (new): SET_ROWS above its 2048-token range runs as
several dispatches (the KV cache cannot leave HRX).
fused-context-claim.h: bounded GDN claim; SSM_CONV claim keyed on the delta-net
conv tensor names; and a fused_context_claim(op, supported) overload that also
requires the matcher's producers to stay on this backend.
llama-bench.cpp: keep errors (and continuation lines) without -v.

Verification

check	result
Qwen3-0.6B greedy text, ub 512 / 1024 / 2048	bit-identical
Qwen3.8-27B PPL, ub 512 / 1024 / 2048	`5.7532 ± 0.16114` on all three
Qwen3.8-27B multi-seq PPL, b1024/ub512 vs b1024/ub1024	`7.1431 ± 0.27993` on both
Qwen3.8-27B ub 1024	runs to completion (4097 tokens, 19.4 s)
`-ub 4096`	addressed by the producer-stays-on-HRX claim; not yet run-verified

-ub 4096 still needs a confirming run: the verification batch was aborted, so the
fix there is inferred from the commit that targets it rather than measured.

The fix is now open for review: 1bit-MONSTER/llama.cpp#101 (base 1bit/hrx-vulkan-patched, head 1bit/hrx-ub389), 5 commits, 13 files, +521/-14.

Bump HRX: llama.cpp ad18320a (ships the #389 >512-token ubatch fix) (#…

Fixed — and it was a pin gap, not a missing patch

The claim in the thread was right, and it is now confirmed verbatim on hardware. The engine side then turned out to be a stale pin, not missing code.

Root cause, reproduced

Engine-pinned llama.cpp 0e255d77, llama-bench -m Qwen3.8-27B-UD-Q4_K_XL -ngl 99 -fa 1 -dev HRX0 -b 2048 -ub 1024 -p 16384 -n 0:

test_prompt: failed to decode prompt batch, res = -3
llama_bench: error: failed to run prompt warmup

With -v, the error the report noted as missing does appear, and it names the node exactly as predicted:

llama_context: n_batch  = 2048
llama_context: n_ubatch = 1024
graph_compute: unsupported HRX node 2: L2_NORM output=3:f32[128,16,1024,1]<-L2_NORM inputs=[2:f32[128,16,1024,1]<-VIEW] consumers=[16:GATED_DELTA_NET]
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
process_ubatch: failed to compute graph, compute status: -1
llama_decode: failed to decode, ret = -3

So: L2_NORM was claimed at placement while no matcher could execute it at 1024 tokens, and the failure was hard rather than a fallback.

The fix already existed on the fork

llama.cpp PR #101 ("ggml-hrx: run prompt batches above 512 tokens") merged on 2026-10-12 as ad18320a, rewriting fused-context-claim.h to bound L2_NORM/SOFTPLUS/GDN/SSM_CONV by kUbatchMaxTokens, adding ubatch-token-limits.h, chunked SSM_CONV and row-chunk helpers. engine@main still pinned 0e255d77, which is 18 commits behind that merge — so engine users kept hitting the bug the fork had already fixed.

Fixed on hardware, then shipped

Same box, gfx1151, ad18320a (all figures contended: load 10.4-12.4, GTT ~27 GB held by two resident Vulkan servers; no arm throttled or aborted, Tctl 47-62 C):

binary	`-ub`	result
`0e255d77` (engine pin)	1024	`res = -3`, exit 1
`ad18320a`	1024	completes, pp16384 285.24 ± 3.70 tok/s
`ad18320a`	2048	completes, pp16384 220.48 tok/s
`ad18320a`	512	completes, pp16384 306.94 ± 2.42 tok/s (issue's clean baseline: 323.80; the ~5% gap is the contention)

Engine pin moved: #419 merged as be6fd1b, third_party/llama.cpp 0e255d77 → ad18320a. The new pins guard (#421) ran on that PR and passed (hold consistency + LoomC compile), so the pair is verified as buildable: ad18320a + the held hrx-system 98d05d94 builds 420/420 with all 123 corpus .loom files parsed, corpus unmodified.

Caveats, stated rather than buried

The after-arm binary had the (separate) engine#390 placement observer compiled in and disabled; it was verified silent when unset (GGML_HRX_REPORT unset/0 → no output, placement unchanged). So the claim is "ad18320a completes -ub 1024", not "ad18320a with a clean tree".
Throughput numbers are contended and should not be quoted as a clean baseline.
One acceptance item from the thread is still outstanding: the GGML_SCHED_DEBUG=2 placement excerpt for -ub 512/1024, which needs a cool-box slot. It is diagnostic, not part of the fix.
hrx-system is untouched by this (still held at 98d05d94), so nothing else in the engine moved.

Evidence: team/evidence/389-strixhalo-runs.md plus raw 389-strixhalo-{base1024,base1024v,ub1024,ub512,ub2048}.log, including a discarded first run whose LD_LIBRARY_PATH order silently loaded the fixed libraries — recorded so nobody quotes it.

Addendum — the two remaining acceptance items, and the caveat removed

Following the merge, both outstanding items from the thread were run on gfx1151. Max Tctl across these arms was 62 C; nothing was throttled and nothing aborted.

1. GGML_SCHED_DEBUG=2 placement excerpt, as the thread asked. Both prompt sizes show a single split, entirely on HRX:

-p 512  -b 512  -ub 512   : 10 x "## SPLIT #0: HRX0 # 0 inputs";  placement: 3655 nodes, 0 off HRX (none), 1 graph splits
-p 1024 -b 1024 -ub 1024  : 10 x "## SPLIT #0: HRX0 # 0 inputs";  placement: 3655 nodes, 0 off HRX (none), 1 graph splits

2. Clean-binary confirmation, which retires the caveat from the fix comment. The earlier after-arm binary carried the separate engine#390 placement observer (compiled in, disabled). Rebuilt from git archive ad18320a with no observer code at all (verified: 0 observer symbols in libggml-base, 0 strict strings in libggml-hrx):

binary	`-ub`	result
clean `ad18320a`	1024	completes, pp16384 281.81 tok/s, exit 0, Tctl 47→59 C, load 10.3→10.7
clean `ad18320a`	512	completes, pp16384 306.57 tok/s, exit 0, Tctl 58→59 C

So the unqualified claim stands: ad18320a completes -ub 1024 with a clean tree.

Logs: team/evidence/389-strixhalo-{sched512,sched1024,ub1024clean,ub512clean}.log; write-up 389-strixhalo-runs.md.

Navigation Menu

Repository navigation

HRX: prompt batches above 512 tokens fail with res = -3 (ubatch capped at 512) #389