HRX: Qwen3-4B --parallel 4 NaN logits on a never-used slot while another GPU job runs (CWSR off)

1bit-canonical on strixhalo aborted at 2026-10-23 07:45:12 ADT with a whole-vocabulary NaN:

367.05.963.920 E common_sampler_sample: HRX returned NaN logits: count=151936 of 151936,
                 index range [0..151935], contiguous=yes, continuing=no (engine#315)
/home/bcloud/wt/tsvchrx4/src/third_party/llama.cpp/common/sampling.cpp:620: HRX: NaN logits (engine#123)

Every one of the 151,936 logits was NaN, and the whole range was contiguous, so this is not one lane or one row of the output — it is the output of the request as a whole. The boot is CWSR off (/sys/module/amdgpu/parameters/cwsr_enable = 0), so the #315 eviction mechanism does not apply here.

What the slot was doing

07:45:09  slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, f_sim_best = 0.252 (> 0.100 thold), f_keep = 0.472
07:45:09  slot launch_slot_: id 3 | task 187 | processing task, is_child = 0
07:45:11  slot print_timing: id 3 | task 187 | prompt eval 210.74 ms / 175 tokens (830.42 tok/s)
07:45:11  slot print_timing: id 3 | task 187 |        eval 1624.04 ms /  29 tokens (17.86 tok/s)
07:45:11  slot print_timing: id 3 | task 187 |    graphs reused = 190
07:45:11  slot      release: id 3 | task 187 | stop processing: n_tokens = 262
07:45:11  slot get_availabl: id 2 | task -1 | selected slot by LRU, t_last = -1
07:45:11  slot launch_slot_: id 2 | task 218 | processing task, is_child = 0
07:45:12  E common_sampler_sample: HRX returned NaN logits (above)
07:45:18  systemd: Main process exited, code=exited, status=1/FAILURE
07:45:29  systemd: Scheduled restart job, restart counter is at 1  (active again, ready on :8095)

The abort is on slot 2's first task after selected slot by LRU, t_last = -1 — a slot that had never been used in this process (6 h 07 min uptime, 18.8 s CPU) — one second after slot 3 finished a 175-token prompt / 29-token generation normally with 190 reused graphs. Backtrace (the crash handler's): common_sampler_sample <- server_context_impl::post_decode lambda <- iterate <- post_decode <- update_slots <- server_queue::start_loop <- llama_server.

Concurrent GPU work

An hw-gate run was on the same GPU for the whole window: run 39940594897 (hw-gate.yml, head f061e5df = the gpu-kernels 744bd713d968 bump, created 2026-10-23T10:39:01Z, updated 10:46:28Z → 07:39:01–07:46:28 ADT), i.e. it spans the 07:45:12 abort. GTT at the service's restart one line later: GTT 95.1 of 120.0 GiB free, MemAvailable 84.7 GiB.

Environment

Unit: 1bit-canonical.service + drop-in hrx.conf — 1bit serve -m ~/models/Qwen3-4B-Q4_K_M.gguf --device hrx --ctx-size 8192 --parallel 4 (serve turns qwen.attention off and passes -kvu; docs/serve.md, engine serve --parallel on HRX: shared KV cache, no qwen.attention (Qwen3-4B 4 slots 39 -> 153 tok/s) #240), ONEBIT_LLAMA_SERVER=~/.local/share/1bit-hrx-3092596/hrx/llama/bin/llama-server, HSA_USERPTR_FOR_PAGED_MEM=0, Restart=always.
Package: ~/.local/share/1bit-hrx-3092596 (installed 2026-10-19T22:57 ADT, own copy, never rebuilt in place): engine 309259627, llama.cpp 00f2a68de (fork Vulkan: carry ggml-org#28243 (Qwen3.8-Flash-Next MTP) on our fork branch #110), hrx-system 82804197bcbe, gpu-kernels 7562196; built with -DONEBIT_HRX=ON and the private HIP add-on (ONEBIT_GPU_PRIVATE), packed by scripts/package-linux.sh 3092596.
Kernel 7.3.0-rc4-1bit-perfopt, CWSR off, --parallel 4 (n_slots = 4, n_ctx_slot = 8192, kv_unified) as logged at load.
The service restarted itself once (NRestarts=1). coder-llm (Qwen3-Coder-30B-A3B, same package family) has 0 restarts since the 01:38 boot.

One occurrence in about 6.5 h of uptime.

#433 (ZAYA1-8B NaN with a second HRX process on the GPU) was re-checked clean today: 120 fresh loads, 0 NaN, no heavy GPU neighbour. #123 is the guard that turned this into an abort instead of a silently wrong token. The hrx-gdn-* NaN family is the 27B MTP path, a different model and a different graph. --parallel on HRX is per-slot KV with unified cache and qwen.attention dispatch off (engine #240).

Reproduce with 60 requests per arm on this package and on main, each request landing on a fresh/LRU slot (prompts varied so LCP similarity stays below the 0.1 threshold), with and without a GPU neighbour inside the same claim — the neighbour being any heavy job of ours, not the real hardware gate.

Reproduction attempt: 0 NaN in 300 requests across five arms, so this is one occurrence that these arms do not reproduce. Data and the two conditions I could not reproduce below.

Arms (each 60 requests, every request on a never-used or LRU slot)

Two builds, 1bit serve -m ~/models/Qwen3-4B-Q4_K_M.gguf --device hrx --ctx-size 8192, private port and systemd-run --scope -p MemoryMax=8G, HSA_USERPTR_FOR_PAGED_MEM=0 as the unit sets it, all inside one gpu-hold -l "pi issues-a" -t 14 claim per arm pair:

arm	build	`--parallel`	GPU neighbour	requests	OK	never-used slots hit
old-neighbour	`3092596` package (engine `309259627`, llama.cpp `00f2a68de`, gpu-kernels `7562196`)	4	our own Qwen3-0.6B serve, 127 requests	60	60	4 of 4
old-alone	same	4	none	60	60	4 of 4
main-neighbour	release main `7871c71d` (llama.cpp `851fa6c8d8f2`, hrx-system `82804197bcbe`, private HIP add-on `744bd713d968`, `1bit` sha256 `703edb65…`)	4	our own Qwen3-0.6B serve	60	60	4 of 4
main-alone	same	4	none	60	60	4 of 4
main-parallel1	same	1	none	60	60	1 of 1

Method, so the arm shape can be checked: each request carried a 16-character nonce plus random words (~300 prompt tokens, max_tokens 32, temperature 0), so no two requests shared a prefix. The server's own slot log confirms it: every --parallel 4 arm filled all four slots from selected slot by LRU, t_last = -1 first, then rotated by LRU, and no request was ever selected by LCP similarity (selected slot by LCP similarity count = 0 in every arm) — i.e. no prefix reuse at all, which is the state the failing slot 2 was in. The neighbour was our own second HRX serve hammered from a thread (127 requests, 0 NaN), never the real hardware gate.

Environment: strixhalo, kernel 7.3.0-rc4-1bit-perfopt, CWSR off, 90 W cap, PCM pad (23 Oct), Tctl 55–63 °C across the two claims, GTT 34.9 GB of 128.8 GB used and MemAvailable 74 GiB at claim start (the incident's line was GTT 95.1 of 120.0 GiB free, MemAvailable 84.7 GiB). sudo -n dmesg after each claim: no amdgpu fault, reset or ring timeout (the only amdgpu lines are the boot-time REG_WAIT/TMZ ones from 01:37).

What this does and does not rule out

Ruled out as a sufficient condition, on this box, in 300 requests: a never-used slot (4 hit per arm), full --parallel 4 multi-sequence serving, the 2026-10-19 package alone, the current release line alone, and a second model sharing the GPU.

Two conditions of the incident are not in these arms:

The kind of neighbour. Mine was a second serve — decode-heavy, one graph reused. The gate run 39940594897 was test-backend-ops op sweeps plus repeated model loads: many different graphs, many allocations, the kernels this path never touched. That is the closest thing to a difference in mechanism and the next thing to reproduce.
Uptime. The service had been up 6 h 07 min and had served 217 tasks when the whole-vocab NaN appeared; each arm here lived about a minute.

Next probes, in order: (1) same five arms with test-backend-ops -b HRX0 -o MUL_MAT in a loop as the neighbour instead of a second serve; (2) a long soak of the failing configuration (one --parallel 4 serve, hours, unique prompts) with GGML_HRX_NAN_CONTINUE=1 so a NaN is captured with its index range instead of aborting; (3) keep the abort behaviour in the service: the guard printed count=151936 of 151936 … contiguous=yes, which is the diagnostic that makes this class of report actionable.

Recommendation for the orchestrator (owner decides)

Move 1bit-canonical to the release-candidate package when the candidate exists, for currency — the running package is from 2026-10-19 (engine 309259627, llama.cpp 00f2a68de) and predates the current HRX pin, strict placement by default and the warm-up work — but not as a fix for this NaN: 0/300 here attributes nothing to those changes. Rollback stays trivial (the ~/.local/share/1bit-hrx-3092596 directory and the drop-in's ONEBIT_LLAMA_SERVER line), and no agent should touch the service; the owner restarts it.

Status from pi issues-c (the handover of this item) — a reproduction attempt that is still queued, and what
the code says to look at next. No new NaN was produced; nothing here contradicts the 0/300 from the earlier
campaign.

What I built for it

~/wt/issues-c/nan528.sh, run inside one claim, shaped after the failing configuration rather than after a
second serve:

llama-server -m Qwen3-4B-Q4_K_M.gguf -c 8192 --parallel 4 -kvu -ngl 99 -dev HRX0, i.e. the same unified-KV
four-slot shape 1bit serve --device hrx --parallel passes, with GGML_HRX_DISABLE_DISPATCH=qwen.attention
set (that is what app/serve.cpp adds, and it is the path that made per-slot answers bleed before serve --parallel on HRX: shared KV cache, no qwen.attention (Qwen3-4B 4 slots 39 -> 153 tok/s) #240);
HSA_USERPTR_FOR_PAGED_MEM=0 as the unit sets it, and GGML_HRX_NAN_CONTINUE=1 so a NaN is captured with its
index range instead of aborting (the guard at common/sampling.cpp:608 is the diagnostic that made the report
actionable in the first place);
the second job is llama-bench on Qwen3-0.6B in a loop inside the same claim, i.e. a neighbour with many
different graphs, many allocations, and kernels this path never touches - the closest cheap analogue of the
test-backend-ops sweep the hardware gate was running at 07:45 on 23 Oct. The earlier campaign's neighbour was
a second serve (one graph reused), which is the one difference it named as unexplained;
~170-word prompts plus a 16-character nonce and cache_prompt: false, so no request can be picked by LCP
similarity and every request lands on a never-used or LRU slot; 60 requests per arm, two arms (neighbour on /
neighbour off).

Arm 1 (heavy neighbour) has now run on the pin (llama.cpp 75242f31 + hrx-system 82804197bc, our own build,
not the 2026-10-19 package), one gpu-hold claim, label 90 W cap, PCM pad (23 Oct), sudo -n dmesg after the
claim: no amdgpu fault, reset or ring timeout.

arm	requests	NaN	HTTP errors	restarts	slot selection
neighbour = `llama-bench` Qwen3-0.6B loop	60	0	0	0	60/60 by LRU (4 never-used slots, then rotation), 0 by LCP similarity
control, serve alone	60	0	0	0	60/60 by LRU, same shape

Median request 0.7-0.9 s (175-token prompt + 32 generated), i.e. the shape of the incident's requests; both arms
selected every request by LRU and none by LCP similarity, and neither produced a single NaN line with the guard's
index-range diagnostic armed (GGML_HRX_NAN_CONTINUE=1).

So this makes 0 NaN in 120 requests on the current pin, on top of the earlier campaign's 0/300 across five arms
on the 2026-10-19 package and the release line. A 1-in-~500-request event cannot be settled by 120 requests; what
these arms do settle is that the neighbour class the incident blamed (a second serve then, a bench loop now) is
not a sufficient condition, and that the reproduce path is a soak, not another 60-request arm.

What the incident shape still points at

The abort was on slot 2's first task (selected slot by LRU, t_last = -1, one second after slot 3 finished
cleanly with 190 reused graphs), and every one of the 151,936 logits was NaN with a contiguous range. That is not
a wrong-value kernel: the whole output of the request is NaN, which is what an unwritten or poisoned output range
looks like - the logits buffer was allocated, the graph ran, and what came back was not a number. Two mechanisms
in this tree can do that and both are independent of -kvu's correctness:

a transient arena range read before its write in a plan whose dispatch order the recorder did not model -
the class engine hrx: fusion must withdraw dispatches it makes redundant (reader emitted before the fused writer, engine #533) #570 is about (a fusion that makes an already-emitted dispatch redundant leaves a reader in
the program), except that the verifier catches those at plan time when it can see them, and a runtime plan
chosen under a different ubatch shape is where it could slip through;
fresh device memory: a transient arena or a per-slot KV allocation that the pool hands out for the first
time in that process, read before the first write. A never-used slot is exactly the state that makes this
visible, and a heavy neighbour changes which pool pages are fresh - which is why the neighbour matters.

Both are testable without waiting for a second 6.5-hour occurrence: the arm above records the NaN index range and
the slot id, GGML_HRX_LOG_TRANSIENT_REUSE=1 prints both arena sizes for every program (guarded vs stock), and
GGML_HRX_TRANSIENT_REUSE=legacy is the A/B switch for the guarded packer. If an arm does produce a NaN, the next
step is the same request with the stock packer and with the plan dump of that shape - that separates "arena reuse
without a barrier" from "uninitialized fresh memory" in one run.

Recommendation (unchanged, and now with a second reason)

Do not move 1bit-canonical off the 2026-10-19 package for this; move it when the release candidate exists,
and add the soak to the candidate's acceptance instead: GGML_HRX_NAN_CONTINUE=1 on one --parallel 4 serve for
a few hours with unique prompts and a bench neighbour. 60 requests per arm cannot see a 1-in-~500 request event;
a soak with the index-range guard on can, and it costs one claim.

tools: a generated NPU engine leaderboard (#621)

leaderboard: the 27B first-token pair from the #622 trace (#623)

Ran the issue's own repro plan on strixhalo. Two valid arms, four slots each, no NaN.

arm	slots	requests	NaN replies	"NaN logits" in log	GTT before → after
`none`	4 of 4	60	0	0	13 365 248 → 13 402 112
`neighbour` (second HRX server busy for the whole arm)	4 of 4	60	0	0	6 645 248 → 6 400 172 032

Conditions, so this is auditable rather than a claim: package ~/.local/share/1bit-hrx-3092596
(mtime 2026-10-19 22:57), Qwen3-4B-Q4_K_M.gguf, --parallel 4 --ctx-size 8192,
amdgpu.cwsr_enable=0 (matches the report), 24-token greedy prompts, prompts generated from
/dev/urandom so slot LCP similarity stays under the 0.1 reuse threshold, 4 requests in flight per
batch (a sequential loop only ever uses one slot and cannot reach a never-used slot — my first run
proved that by reporting slots_used=1 of 4 and is marked invalid by the harness now).
Logs: ~/lb/528/run-20261027T215046Z-none and ~/lb/528/run-20261027T215131Z-neighbour.

What this does not reproduce: the neighbour here is a second HRX server, not the hardware gate's
own workload (hw-gate run 39940594897 as named in the report). If the trigger is specific to that
workload's shape — a bench holding the GPU's queue rather than a second server sharing GTT — this arm
does not cover it. That is the one remaining difference I can see between the report and this run.

Harness is in the repo convention for reruns; it refuses to report a result unless at least three
distinct slots actually launched. Verdict from me: unreproduced, not fixed — leaving the issue open
rather than closing it on absence of evidence, and if it recurs, the useful datum is which GPU job was
co-resident and whether GTT was still growing at the time.

Navigation Menu

Repository navigation

HRX: Qwen3-4B --parallel 4 NaN logits on a never-used slot while another GPU job runs (CWSR off) #528

What the slot was doing

Concurrent GPU work

Environment

Related

Next

Arms (each 60 requests, every request on a never-used or LRU slot)

What this does and does not rule out

Recommendation for the orchestrator (owner decides)

What I built for it

What the incident shape still points at

Recommendation (unchanged, and now with a second reason)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: Qwen3-4B --parallel 4 NaN logits on a never-used slot while another GPU job runs (CWSR off) #528

Description

What the slot was doing

Concurrent GPU work

Environment

Related

Next

Activity

bong-water-water-bong commented on Oct 23, 2026

Arms (each 60 requests, every request on a never-used or LRU slot)

What this does and does not rule out

Recommendation for the orchestrator (owner decides)

bong-water-water-bong commented on Oct 25, 2026

What I built for it

What the incident shape still points at

Recommendation (unchanged, and now with a second reason)

bong-water-water-bong commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions