HRX: ZAYA1-8B intermittent NaN logits abort the server (common_sampler_sample guard), seen with a second HRX process on the GPU

ZAYA1-8B on HRX intermittently returns NaN logits. The NaN guard in common_sampler_sample (added for #123) then aborts the server rather than sample a wrong token. Seen 3 times on 2026-10-16 while a ZAYA-driven demo agent was running, on a CWSR-off boot (amdgpu.cwsr_enable=0), so this is not the #315 CWSR/eviction mechanism.

What happened

Model: zaya1-8b-Q4_K_M.gguf, 1bit serve -m <model> --device hrx --ctx-size 16384 (4 slots, unified KV).
Build: engine 06cc0ef, llama.cpp fork dd74f6bb0. This is older than the current pin (see "Current pin" below).
Box: Strix Halo, kernel 7.3.0-rc4, booted with amdgpu.cwsr_enable=0, about 77 GiB free. The serve's RSS was about 2 GiB.
3 aborts across separate serve instances. Two hit on the first sample after a fresh load (task 0, 458-token prompt, tool-templated /v1/completions). The third hit on a request that reused a slot by prefix similarity (f_sim_best = 1.000, f_keep = 0.847).
At the same time, a second HRX process was serving ZAYA1-8B on the same GPU (port 8135, another agent's --pm check).

E common_sampler_sample: HRX returned NaN logits at vocab index 0 - refusing to sample a silently wrong token (engine#123)
common/sampling.cpp:588: HRX: NaN logits (engine#123)
#5 ggml_abort ()
#6 common_sampler_sample(common_sampler*, llama_context*, int, bool) ()
#7 server_context_impl::post_decode(...)::{lambda(server_slot&)#2}::operator()
#10 server_context_impl::update_slots()

Not reproduced in isolation

Fresh load, then the demo's exact first request (same system prompt, 4 tools, same world state, temperature 0.2, max_tokens 400), with no other HRX process on the GPU:

build	fresh loads	NaN
llama.cpp `dd74f6bb0` (the demo's build)	10	0
llama.cpp `1a5dcba2` + hrx-system `3319e536` (current pin)	10	0

Current pin

Not yet known whether the current pin faults under the conditions the demo hit. The demo is moving to engine main (llama.cpp 1a5dcba2), and it will count every NaN abort per run (first-request vs prefix-reuse).

Next step

Rerun the same harness with a second ZAYA1-8B HRX serve under load on the same GPU, then with prefix-reuse sequences (the demo's 6-decision loop) on a single serve, to separate the two candidates.

Reran this issue's condition on the current pin: 60 fresh loads, 0 NaN aborts. The earlier attempt in ~/wt/hrx-zaya-nan/ was invalid (its requests went out while llama-server was still loading; all 20 saved replies are {"error":{"message":"model is loading"}}), so this run replaces it with a harness that proves each load was served.

Builds

arms	engine	llama.cpp	hrx-system	`llama-server --version`
a, b (current pin)	`fce3a687c1dd101d39562b268c4ad94efc45e4ca`	`8d9ce2c1aa60ed6b2334df3576e203e40c533a9c`	`d3c731e1d7f47de2d64739b1ec298f0d0fe829d4`	`10643 (8d9ce2c1a)`
c (this issue's pin)	`6dd709f7774bfb74c845d8d5c84b5f93c6852bea`	`1a5dcba2aa6d91dacec3c3c00a37dd77aa9c6084`	`3319e536196b99119f30f78b60f39a340aee6a5d`	`1 (1a5dcba)`

Boot: amdgpu.cwsr_enable=0 (/sys/module/amdgpu/parameters/cwsr_enable = 0). One GPU job at a time; box services untouched; Tctl 65-78 C across the three arms (pause >85, abort >88).

One load

systemd-run --user --scope --collect -p MemoryMax=14G -- \
  1bit serve -m ~/models/zaya1-8b-Q4_K_M.gguf --device hrx --port <n> --ctx-size 16384 --alias zaya1-8b \
             --llama-server <build>/build/hrx/llama/bin/llama-server

then one tool-templated request, exactly as the game sends it:

POST /apply-template   {messages:[SYSTEM, "World state:\n{...}\n\nChoose the next tool."],
                        tools:[mine_block, move, turn, jump], chat_template_kwargs:{enable_thinking:false}}
POST /v1/completions   {model, prompt, max_tokens:400, temperature:0.2, stream:false,
                        stop:["</zyphra_tool_call>","</tool_call>","<|im_end|>","<|endofstate|>"]}

Every load asserts its own shape from its serve log — n_slots = 4, n_ctx_slot = 16384, kv_unified = 'true' — and readiness is GET /health == 200 (llama-server answers 503 until the model is up), not a log grep. The second-process arms keep a Qwen3-0.6B-Q4_K_M.gguf HRX serve resident (MemoryMax 4G) for the whole loop; each arm is 20 fresh loads with teardown in between.

Result

arm	build	GPU condition	loads	OK
a	current pin	alone	20	20
b	current pin	+ resident 0.6B HRX process	20	20
c	issue pin	+ resident 0.6B HRX process	20	20

All 60 loads: HTTP 200, prompt_n = 851, predicted_n = 400 (the cap was reached every time), 4 slots, unified KV, ctx 16384. "void" = a load that never became a data point; a void load is retried once and never counted as OK.

What the harness fixed

readiness by /health, and an unservable load is void (retried once, reported, never OK) — the failure mode of the previous attempt is now impossible;
/apply-template without a prompt is void with the body kept, instead of a KeyError;
NaN is a per-load line-count delta on that load's own log, and the newer pin's count=.. index range [..], contiguous=, continuing= fields are parsed (dry-run on saved data: a good pair -> OK, the invalid pair -> void, the verbatim NaN line and the new diagnostic line -> NaN with the fields);
the slot/KV/ctx shape is asserted per load instead of assumed.

One number in the report does not add up

The abort above carries a 458-token tool-templated prompt. Measured with this GGUF's own template and the game's request:

request	templated prompt
no tools	257 tokens
1 tool	629
2 tools	726
4 tools (the game's)	851

Any request that carries a tool block is >= 629 tokens on this template, so a 458-token prompt cannot have carried one: either that request went out without tools, or the model/template differed. The game overwrites runs/p2/dispatch.jsonl, so the aborting request's bytes are not recoverable now — worth saving the exact request (and the response's timings.prompt_n) with the next abort.

The second-process condition alone does not reproduce on either pin here. The other candidate in the issue is prefix reuse; a decided-next step would be a single serve with the game's 6-decision loop (prompt shapes that reuse the cache, cache_n > 0) at this pin, with GGML_HRX_NAN_CONTINUE=1 so a NaN is logged with its index range instead of aborting.

Harness and per-load logs/responses/summaries: strixhalo:~/wt/hrx-zaya-nan/t433/ (repro.py, out/{a,b,c}-*-summary.json).

🤖 Generated with Claude Code

docs: the 2026-10-17/18 merges (HRX pins, DwarfStar, PM router and sp…

Reran the issue's condition on current `main`: 0 NaN aborts in 120 fresh loads

Two sets, because the HRX pin moved while this ran: #507 (llama.cpp 851fa6c8d8f2 + hrx-system 82804197) merged 07:39Z, after the first build was taken. So set A is the pin that was on main when the run started, set B is current main.

set	engine	llama.cpp	hrx-system	`llama-server --version`
A (pin before #507)	`f826cb3` (main's tip at 04:21 ADT, when it was built)	`af04c9f22`	`82804197`	`1 (af04c9f)`
B (current main)	`716ae1b8` (main's tip at 05:40 ADT)	`851fa6c8d8`	`82804197`	`1 (851fa6c8d)`

Both built from a clean tree: -DONEBIT_HRX=ON with our HIP kernels compiled in as the release build does (45 kernels from the pinned add-on), under thermal-run.sh (4 cores, nice 19).

Box, all 90 W cap, PCM pad (23 Oct): strixhalo, kernel 7.3.0-rc4-1bit-perfopt, amdgpu.cwsr_enable = 0 (CWSR off — the issue's regime), each set run inside gpu-hold -l "pi issues-a" -t 14, one GPU job at a time, no service touched. Tctl 38–60 °C (set A) and 38–67 °C (set B), peak under the 75 °C start gate. sudo -n dmesg after every claim: no amdgpu line after boot — all 86 amdgpu lines on this boot are init lines from 01:37–01:38.

One load

Two 1bit serve instances of zaya1-8b-Q4_K_M.gguf resident on HRX at once (--device hrx --ctx-size 16384, MemoryMax=14G per instance) — the issue's condition — and one fresh load per instance per round: teardown, fresh start, GET /health == 200, then the demo's request verbatim (POST /apply-template with the 4 tools and the world state, then POST /v1/completions, max_tokens 400, temperature 0.2, the issue's stop list). Each load asserts its own shape from its own log — n_slots = 4, n_ctx_slot = 16384, kv_unified = 'true' — and NaN is a line-count delta on that load's log, so a NaN in any request of that load is attributed to it. A load that never answers is void, retried once, and never counted OK.

The first 10 loads of each instance carry one extra /v1/completions with the same body — the prefix-reuse path the issue's third abort hit: the server selects the slot by LCP similarity and serves it from cache.

Result

set	instance	fresh loads	OK
A `af04c9f2`	A	30	30
A `af04c9f2`	B	30	30
B `851fa6c8d`	A	30	30
B `851fa6c8d`	B	30	30
total		120	120

Every one of the 120: prompt_n = 851 — the same figure the issue's own harness measured for this 4-tool request, so no request here went out at the earlier report's impossible 458 tokens — and predicted_n = 400, the cap reached every time, with (slots 4, ctx 16384, unified KV) asserted 60/60 per set.

Slot-reuse requests: 40/40 with cache_n = 847 (20 in each set), i.e. every reuse request was a real prefix-cache hit, and none produced a NaN.

What this settles, and what it does not

It settles the issue's last open question ("not yet known whether the current pin faults under the conditions the demo hit"): on current main, with two ZAYA1-8B HRX processes on the GPU, 0 of 60 fresh loads faulted and 0 of 20 prefix-reuse requests did either — and the same at the pin before the bump.
It does not explain the 2026-10-16 aborts. Nothing here changes that: the aborting request's bytes are not recoverable (the game overwrites its dispatch log), and a 458-token prompt cannot have carried a tool block on this template (≥ 629 tokens with one tool, 851 with four).
The #123 guard stays. It is what turns NaN logits into a refused sample instead of a silently wrong token, and GGML_HRX_NAN_CONTINUE=1 still exists for the next abort that needs its index range captured alongside the request.

Closing on 0/120 with the shas and per-load shape above. Harness and per-load logs/responses: strixhalo:~/wt/issues-a/t433/ — twin.py, claim-{1,2}.log + out/ (set A), claim-{3,4}.log + out-851fa6c8d/ (set B, merged rows in setB-merged.json).

0 NaN in 120 fresh loads on current main (716ae1b8, llama.cpp 851fa6c8d) and on the pin before #507; 40/40 slot-reuse requests served from cache with no NaN; per-load shape (4 slots, ctx 16384, unified KV) asserted and dmesg clean. The #123 guard stays.

docs: the v2026.43 candidate section and the README status table (#525 …

docs: v2026.43 candidate section + README status table (#527)

Navigation Menu

Repository navigation

HRX: ZAYA1-8B intermittent NaN logits abort the server (common_sampler_sample guard), seen with a second HRX process on the GPU #433

What happened

Not reproduced in isolation

Current pin

Next step

Builds

One load

Result

What the harness fixed

One number in the report does not add up

Next

Reran the issue's condition on current `main`: 0 NaN aborts in 120 fresh loads

One load

Result

What this settles, and what it does not

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: ZAYA1-8B intermittent NaN logits abort the server (common_sampler_sample guard), seen with a second HRX process on the GPU #433

Description

What happened

Not reproduced in isolation

Current pin

Next step

Activity

bong-water-water-bong commented on Oct 18, 2026

Builds

One load

Result

What the harness fixed

One number in the report does not add up

Next

bong-water-water-bong commented on Oct 23, 2026

Reran the issue's condition on current main: 0 NaN aborts in 120 fresh loads

One load

Result

What this settles, and what it does not

bong-water-water-bong commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Reran the issue's condition on current `main`: 0 NaN aborts in 120 fresh loads