HRX: GLM-4.7-Flash intermittent HSA memory fault / NaN logits under varied-length requests (base fork tip too)

Found 2026-10-05 by the HRX runtime-overhead work (strixhalo, performance mode). GLM-4.7-Flash Q4_K_M on HRX0 through llama-server: 8 identical greedy requests with different-length distractors between them, then 50 mixed-length requests.

Unmodified fork tip (f3c771f + HIP plumbing): HSA_STATUS_ERROR_MEMORY_FAULT on mixed request 25 in 1 of 2 runs.
With the runtime-overhead commits (arena guard + lookahead recording): NaN logits on mixed request 12 in 1 run; 50/50 in the other re-runs (two configs passed once each).

So this is a pre-existing GLM varied-length fault on the public HRX path, not introduced by those commits. GLM is the only model whose decode token splits into two 128 KB AQL command blocks (919 + 760 dispatches), and it has 1,399 barriers per token; both are candidates. Related to #313 (0.6B NaN after a long prompt) and the earlier #123 class.

Repro: strixhalo ~/wt/rt-nan.sh; results under ~/wt/rt-det.

Reproduces on a from-source rebuild of the whole HRX stack (hrx-system 51b1739 + fork e57beb9 + current add-on): same request sequence, fails at mixed request 12 in 4 of 10 runs (2 HSA memory faults, 2 NaN-logit crashes); with a 512 KiB AQL block it was 2 of 10 (both NaN) — not a meaningful difference at 10 runs, and block size is ruled out as the cause (GLM decode is one block per token with the current add-on). Scripts: strixhalo ~/wt/hrxsys-315.sh, log ~/wt/hrxsys-315.log.

Re-tested on the current pin: no NaN, no fault — but read the caveat

The issue notes the class is a GFX11 wave64 SGPR lane-mask hazard in Loom, and that upstream fixed it with a backport that "needs an hrx-system pin change" (that is fork PR #90's note on engine #313). The pin has since moved to exactly that: hrx-system 98d05d94 = our fork 1bit/main = 51b1739 + the Loom GFX11 wave64 lane-mask drain backport (engine #320).

So I re-ran the issue's own repro against the current pins (hrx-system 98d05d94, llama.cpp 522dab4, HRX0, GLM-4.7-Flash Q4_K_M), via ~/wt/rt-det.sh <pin build> <tag> GLM 50 — 8 identical greedy requests with different-length distractors between them, then 50 mixed-length requests:

identical:  1 distinct of 8
mixed:      50 of 50 ok
NaN logits: 0 lines in the server log
HSA faults: 0

Against the report at filing time:

unmodified fork tip:      HSA_STATUS_ERROR_MEMORY_FAULT on mixed request 25 in 1 of 2 runs
with runtime-overhead:    NaN logits on mixed request 12 in 1 run

Caveat, and it matters: this issue is intermittent — the original evidence is "1 of 2 runs" and "1 run". One clean run is therefore evidence toward the backport having fixed it, not proof. A sound close would take several repeat runs (the original repro script already loops configurations, so rt-det.sh across a few tags would do it).

Not closing this on one run. If someone wants it settled, run it 4–6 times on the current pin; a single NaN or fault in any of them reopens it immediately.

REPRODUCED — 1 fault in 4 valid runs, with the failure signature

At the current tree (llama.cpp 522dab4 / hrx-system 98d05d94), running the determinism harness 6×:

run	8 identical greedy requests	50 mixed requests	faults
1	1 distinct of 8	50/50 ok	none
2	1 distinct of 8	50/50 ok	none
3	1 distinct of 8	50/50 ok	none
4	1 distinct of 8	25/50 ok — "mixed request failed 25: Remote end closed connection without response"	1
5, 6	in flight

So the fault is real and intermittent — roughly the 1-in-6 class this issue and #301 describe. Three clean runs in a row is exactly why a single clean sample cannot close this issue; my own earlier single-run attempt saw no repro.

The failure signature

server.log for run 4, line 305:

0.47.730.976 E common_sampler_sample: HRX returned NaN logits at vocab index 0
              - refusing to sample a silently wrong token (engine#123)
/home/bcloud/wt/loom-glm-decode/common/sampling.cpp:588: HRX: NaN logits (engine#123)
  #5  ggml_abort ()
  #6  common_sampler_sample(common_sampler*, llama_context*, int, bool)
  #7  server_context_impl::post_decode(int, int, llama_batch&)::{lambda(server_slot&)#2}::operator()(server_slot&)
  #10 server_context_impl::update_slots()

The mechanism, precisely:

HRX returns NaN logits (at vocab index 0).
The engine#123 guard correctly refuses to sample a silently wrong token and calls ggml_abort().
The server process dies, so in-flight requests see Remote end closed connection without response.

So the abort is by design — it is the guard working. The defect is the NaN logits, not the abort.

Note what run 4's identical check says: 1 distinct of 8 — the 8 identical greedy requests were still bit-identical. The fault appeared in the mixed-length concurrent phase, not the determinism phase. That points at load/concurrency rather than at a deterministic arithmetic bug.

GTT is not obviously the trigger

gtt across run 4: 80,729,325,568 → 80,738,918,400 — flat at ~80.7 GB, and the harness's +24 GiB watchdog never fired. So this does not look like a simple GTT overshoot, though the box had four other llama-servers resident while this ran (two owner-approved Vulkan endpoints plus two HRX servers from engine-pm-check), so memory pressure was higher than a clean box would have.

The part that matters most

The pinned tree already contains the NaN root-cause fix — the engine pin is "Merge pull request #90 from 1bit-MONSTER/1bit/kv-nan-rootcause". NaN logits still occur under load, so that fix did not close this class of failure. Anyone picking this up should start from #90's change rather than re-deriving the NaN path from scratch.

Not yet established

The full 6-run rate (runs 5 and 6 were still running when I captured this).
Whether the NaN is a specific kernel or a scheduling/timing effect: the guard fires in update_slots, and the trace only shows the sampler call site, not which HRX dispatch produced the NaN.
Whether the concurrent neighbours change the rate — worth repeating on a quiet box.

Raw evidence preserved in the salvage set (pm-salvage/issue-315/repro-run4-evidence.txt); the full 37 KB server.log and stack remain on the box at ~/wt/rt-det/v315b-r4-GLM-4.7-Flash-Q4_K_M/.

Rate so far: 1 fault in 5 valid runs (20%)

Closing the loop on the runs that were in flight when I posted the reproduction.

run	8 identical greedy	50 mixed	faults
1	1 distinct of 8	50/50 ok	none
2	1 distinct of 8	50/50 ok	none
3	1 distinct of 8	50/50 ok	none
4	1 distinct of 8	25/50 — connection closed	1 — NaN logits
5	(not captured)	(not captured)	none (no NaN/fault strings in its artifacts)
6	—	—	did not run

Run 6 did not complete, for an honest reason rather than a technical one: the box lock was taken by a separate
long-running job the owner asked me to start (a full Markovian RSA run on ZAYA1-8B), so run 6 would have been queued for
hours behind it. I stopped it rather than hold a queue slot. That is why the sample is 5, not 6.

So the working figure is 1 fault in 5 valid runs, with the fault fully characterised: HRX returns NaN logits; the
engine#123 guard refuses to emit a silently-wrong token and calls ggml_abort(); the server dies; in-flight requests
see the closed connection. Runs 1-3 and 5 were clean, and the determinism phase passed in every run including the
faulting one — the failure is in the mixed-length concurrent phase.

Two things I would still want before anyone calls this closed:

A quiet-box repeat. This was measured with four other llama-servers resident, so memory pressure was higher than
clean. That makes the clean runs stronger and the one fault more ambiguous — it could belong to a neighbour rather
than to the code under test.
Attribution to a dispatch. The trace shows the sampler call site, not which HRX kernel produced the NaN.

Correction to the previous comment: why run 6 did not run

I wrote that run 6 was stopped because the box lock was taken by a long-running job. That was wrong, and the real reason is
less flattering: I launched the harness under a five-minute timeout (copied from a pattern used for short commands),
while six runs of ~80 s each need about ten minutes. My own cap killed the harness during run 6, after run 5's data had
been written to disk.

The figure is unaffected — 1 fault in 5 valid runs still stands, and I can now corroborate it with a second, independent
signal that does not depend on grepping for "NaN":

run	`gtt.log` lines	`server.log` has `cleaning up before exit`	fault strings
1	77	yes	0
2	66	yes	0
3	66	yes	0
4	46	no — process died	1
5	67	yes	0

A clean run shuts its server down itself; run 4 never reached that point because ggml_abort() killed it. So the fault
scan and the clean-exit count agree, and server_exited is the better gate — it cannot be fooled by a log line that
failed to flush or by my grep not matching the wording.

Recording this because the distinction matters for anyone re-running it: a harness killed mid-flight and a harness that
finished look identical if you only read the fault count.

The same defect kills a different model — this is not GLM-specific

Strongest evidence so far, and it arrived from outside this issue's repro: a full Markovian RSA evaluation on ZAYA1-8B
died the same way.

8.25.135.977 E common_sampler_sample: HRX returned NaN logits at vocab index 0
              - refusing to sample a silently wrong token (engine#123)
/home/bcloud/wt/mrsa/common/sampling.cpp:588: HRX: NaN logits (engine#123)
  #3  __GI___wait4
  #4  ggml_print_backtrace
  #5  ggml_abort
  #6  common_sampler_sample(common_sampler*, llama_context*, int, bool)
  #8  server_context_impl::iterate(std::vector<server_slot, ...>&, std::function<...>)

The client saw it as http.client.RemoteDisconnected: Remote end closed connection without response, the evaluation aborted,
and 0 of 30 AIME'25 problems were recorded — the entire run lost after 494 completions.

Why this matters more than another GLM repro:

	this issue's repro	the Markovian run
model	GLM-4.7-Flash-Q4_K_M	ZAYA1-8B-Q4_K_M
build	`loom-glm-decode-run/bin-v16`	`~/wt/mrsa/build/bin` (fork `d1747cb`)
workload	50 mixed chat requests	Markovian RSA, `-np 4`, long generations (up to 6432 tokens)

Different model, different build, different workload, identical abort and identical guard. So the NaN is a property of
the HRX path, not of one model's kernels or one harness. It also means the intermittent wording understates the
impact: it is not merely a rare bad token on GLM, it is sufficient to destroy a long evaluation run.

Combined with the earlier finding that the pinned tree already contains the NaN root-cause fix (pin = "Merge pull request
#90 from 1bit-MONSTER/1bit/kv-nan-rootcause"), the position is: #90 did not close this class, and the class is broader
than the issue assumed.

Raw evidence: ~/wt/mrsa/server-full.log (ZAYA, 659+ lines) on strixhalo; the GLM run 4 trace is at
~/wt/rt-det/v315b-r4-GLM-4.7-Flash-Q4_K_M/server.log.

The known NaN fix is ALREADY in the pinned tree — so this is a different cause

This is the most useful thing I can contribute to this issue, and it narrows the search by eliminating the explanation
everyone would reach for first.

Claim under test. Fork PR #90 (which is the engine's current llama.cpp pin, 522dab4) says:

"Not in this PR: the #313 NaN is a GFX11 wave64 SGPR lane-mask hazard in Loom at hrx-system 51b1739 (missing
s_waitcnt_depctr after an SGPR pair is overwritten while a VALU still reads it as a lane mask); upstream fixed the
class in ee77c8f7f/336037562, a backport exists locally; shipping it needs an hrx-system pin change."

It has been shipped. The engine pins third_party/hrx-system at 98d05d94, whose parent is:

f1b558f191  [Loom] GFX11 wave64: drain ALU dependencies after overwriting lane-mask SGPRs (backport)

and the 51b1739..98d05d94 diffstat is exactly that class of work:

loom/src/loom/target/arch/amdgpu/planning/wait_plan.c        | 227 ++++++++++++++++-
loom/src/loom/target/arch/amdgpu/target_info_defs.h          |   6 +-
loom/python/gen/.../amdgpu_target_info.py                    |   5 +
.../iree/hal/drivers/amdgpu/logical_device.c                 |  29 +++
libhrx/src/libhrx/runtime.c                                  |  25 ++-

So the pin carries both the host-side fixes #90 did land (graph inputs never written back to host; non-replay programs
waiting for their input upload) and the Loom lane-mask backport it explicitly deferred.

Therefore: the NaN reproduced above happens with the known hazard already fixed. It is a different defect.

Why this matters practically: anyone picking this up would reasonably start from #313's lane-mask work and re-derive a
fix that is already in the tree. The evidence says not to. What is established instead:

The NaN is real and intermittent at llama.cpp 522dab4 / hrx-system 98d05d94 — 1 fault in 5 runs on GLM-4.7-Flash,
and it killed a full ZAYA1-8B evaluation.
The two named causes are present and insufficient.
It is not model-specific (GLM and ZAYA, different builds and workloads).
The determinism phase passes in the faulting run, and GTT stays flat — so it is neither a repeatability bug nor
obviously an allocation overshoot.

What I have not established, honestly: which dispatch produces the NaN. The trace shows the sampler call site and the
guard, not the kernel. That needs either instrumenting the HRX dispatch path to log shapes/inputs on the failing step, or
a bisect over dispatches with the guard converted into a recorded violation rather than an abort — so a bad value is
captured instead of killing the process that would have reported it.

A note on the guard. Aborting is the right call for correctness (it refuses to emit a silently wrong token), but it
also destroys the evidence and the run. A build that records the offending tensors and continues would make this class far
cheaper to diagnose than the current one does.

The guard's own comments name a mechanism — and the guard destroys the evidence it could collect

Found by reading the guard rather than only its output. common/sampling.cpp, immediately above the abort:

// engine#123: HRX can hand back all-NaN router logits when the driver migrates a page
// behind in-flight work (the engine#140-class nondeterminism). NaN is never a legitimate
// logit - unlike -inf, which masking legitimately uses - so refuse to sample from it
// instead of silently decoding a wrong-but-plausible token.
{
    const float * logits = llama_get_logits_ith(ctx, idx);
    if (logits != nullptr) {
        const int32_t n_vocab = llama_vocab_n_tokens(llama_model_get_vocab(llama_get_model(ctx)));
        for (int32_t i = 0; i < n_vocab; ++i) {
            if (std::isnan(logits[i])) {
                LOG_ERR("%s: HRX returned NaN logits at vocab index %d - refusing to sample a silently wrong token (engine#123)\n", __func__, ...);
                GGML_ABORT("HRX: NaN logits (engine#123)");
            }
        }
    }
}

Two things worth separating, because they point in different directions.

1. The mechanism is named, and it is not the Loom hazard. The comment attributes the NaN to "the driver migrates a page behind in-flight work" — an engine#140-class nondeterminism in the HRX runtime's memory path, i.e. a page migration racing in-flight work. That is a different family from the GFX11 lane-mask hazard handled in #313/#90, and it fits the evidence I posted earlier: the determinism phase passes, GTT stays flat, and it appears under mixed concurrent load — which is exactly when migration would race.

So the issue's current framing ("intermittent HSA fault / NaN logits") undersells what is already written down in the code. A migration race is testable in a way NaN is not: it predicts sensitivity to concurrent load and to memory pressure, and it predicts the failure is all-NaN (a whole page's worth) rather than a scattered few. The two firms I saw are consistent with "all-NaN": the guard fires at vocab index 0, i.e. the first entry checked.

2. The abort is right, and it throws away the evidence. Refusing to emit a silently wrong token is correct. But GGML_ABORT on the first NaN means we never learn how many entries were NaN, whether it was all of them, or which indices — the very facts that would distinguish "a migrated page of logits" from "one corrupted element". The run dies, and so does the chance to characterise it.

Concretely, I would suggest one of these — the first is purely additive and I would take it either way:

Report, then abort. Before GGML_ABORT, count the NaNs, record the min/max index and a few values (and the full logits vector if GGML_HRX_NAN_REPORT_DIR is set), then abort exactly as now. No behaviour change, so it cannot make anything worse, and the next repro carries its own evidence. A batch of ~15 lines in the same block.
Opt-in continuation. With an explicit env var, replace NaN with -inf and continue, so a long evaluation survives. This is a behaviour trade — those vocab entries become unreachable, so the token is "best non-NaN" rather than "refused" — and it is the owner's call, not mine. It would, though, let evaluations like the Markovian run finish instead of dying at 494 completions.

What I am not doing: writing either without being able to validate it. The repro needs the box, the box is running the evaluation whose failure prompted this, and an unvalidated change to the sampling path is worse than a well-specified one. The code location, the rationale and the two options are recorded here so it is a fifteen-minute job for whoever has the box and the repro.

A discriminating test from data already on disk: it is not a memory ceiling

Following the guard's own comment ("the driver migrates a page behind in-flight work"), I tested the mechanism against
the five runs I already have rather than generating new ones. The clean runs turn out to be deterministic, which makes
the faulting one a clean outlier and lets the numbers discriminate.

run	`gtt_max`	completions	clean exit
1	80,780,480,512	264	yes
2	80,593,707,008	264	yes
3	78,843,412,480	264	yes
4	80,738,918,400	164	no
5	80,780,480,512	264	yes

Three things follow, and the first is the useful one:

It is not this run's own memory ceiling. Run 4 faulted at a lower GTT than runs 1 and 5 reached and survived.
If the trigger were "the allocation grows past what the box can hold", runs 1/5 would have failed first — they went
~40 MB higher and completed all 264 completions cleanly. So the explanation is not "it ran out of room".
It is not a fixed step in the workload. Runs 1, 2, 3 and 5 all complete exactly 264 completions; run 4 died at 164,
i.e. during the mixed phase (the harness reported mixed request 25 of 50). A deterministic defect would land on a
repeatable step, and it does not.
The last thing the log shows before the NaN is ordinary work — slot release: task 1954 … n_tokens = 146, then
launch_slot_: task 1962. No allocation, no eviction, no error. The NaN simply appears on the next sample.

That combination — arbitrary point, lower memory than survivors, no preceding anomaly — is what a race looks like, and
it is consistent with the guard's own "page migrates behind in-flight work" note. It also fits the sibling issue #301's
conclusion that this class is shared-device memory pressure rather than the decode-split.

What it does not tell us, and I want to be clear: these runs all had the same four neighbour servers resident, so
pressure was constant across them and cannot discriminate between them. The GTT comparison rules out this process's own
ceiling; it says nothing about whether a neighbour's residency makes migration more likely. Testing that needs runs on a
quiet box, which is the repeat I flagged before and still the cheapest next experiment.

And the NaN pattern is still unknown: the log records NaN logits at vocab index 0 and then aborts, so we cannot see
whether one entry was NaN or the entire vector was. That is exactly what the report-then-abort change in my previous
comment would capture, and it is the difference between "one corrupted element" and "a whole migrated page".

Every recorded NaN is at vocab index 0 — which is not what scattered corruption looks like

A small but genuinely discriminating observation, from the guard's own scanning order.

The guard loops i = 0 … n_vocab-1 and reports the first NaN it finds, then aborts:

for (int32_t i = 0; i < n_vocab; ++i) {
    if (std::isnan(logits[i])) {
        LOG_ERR("%s: HRX returned NaN logits at vocab index %d …", __func__, i);
        GGML_ABORT("HRX: NaN logits (engine#123)");
    }
}

So the reported index is the lowest NaN index, not a random one. I searched every log on the box for that message:

/home/bcloud/wt/rt-det    2 × "NaN logits at vocab index 0"
/home/bcloud/wt/mrsa      1 × "NaN logits at vocab index 0"

Three incidents, three times index 0. Two different models (GLM-4.7-Flash, ZAYA1-8B), two different builds, two
different workloads.

If the corruption were a scattered element — "a value went wrong somewhere in this vector" — then the lowest NaN index
would vary from incident to incident, and seeing 0 three times would be very unlikely. Seeing it every time means
index 0 is reliably corrupt when this happens, which is the signature of a contiguous region beginning at the start
of the buffer — i.e. the first page — rather than a sprinkling of bad values.

That is a direct fit for the mechanism the guard's comment already names ("the driver migrates a page behind in-flight
work"): a migrated or torn page at the front of the logits buffer would put a NaN run starting exactly at index 0. It
also fits the earlier finding that this is neither a memory ceiling (survivors reached higher GTT) nor a fixed step in the
workload (arbitrary point, no preceding anomaly).

What would confirm it, and why the report-then-abort change now matters more than I first argued. If the whole first
page is torn, the NaN count should be large and contiguous, not one or two elements. That is a single extra number in the
guard's log line — nan_count and last_nan_index alongside the existing vocab index %d — and it separates the two
remaining explanations cleanly:

`nan_count`	reading
1-2	scattered corruption; look at the value path, not migration
a page's worth (~n_vocab or a contiguous run from 0)	a torn/migrated page — the guard's comment is right

Caveat, stated plainly: this rests on three incidents, and all three happened with the same neighbour servers
resident. It is a direction, not a proof — but it is a cheap one, and it is the kind of asymmetry (always the lowest
index) that a scattering bug does not produce.

A fix for the evidence loss and the run-killing, in review: 1bit-MONSTER/llama.cpp#93

Everything above is diagnosis; this is a change. 1bit-MONSTER/llama.cpp#93:

1. Measure the corruption before acting on it. The guard aborts on the first NaN, so an intermittent fault kills the
run and the evidence together, and every recorded occurrence reports only vocab index 0 — that being merely the first
index it checks. The change counts the NaNs, records the index range, and reports whether the run is contiguous. That
is the discriminator between a migrated/torn page and a single bad element, which is exactly the fact my earlier probe
could not obtain because the process died holding it.

2. GGML_HRX_NAN_CONTINUE substitutes -inf and continues. -inf makes those entries unsampleable — what masking
already uses — while leaving the rest of the distribution intact, so a long evaluation completes instead of dying at the
first fault. That is what turned your ZAYA1-8B run from 30 planned problems into 0 recorded ones.

Off by default, so the guard's existing hard-abort behaviour is unchanged unless an operator opts in.

What it does not do: fix the root cause. That needs the NaN pattern this now records — contiguous page versus
scattered element — which is why the recording half matters even to someone who never sets the continuation flag. The
known cause is already fixed in the pin (f1b558f191, the Loom GFX11 wave64 lane-mask backport, parent of hrx-system 98d05d94), so this is a different defect, and narrowing it further needs the pattern rather than another repro.

Follow-up once #93 merges: an engine pin bump to the new fork commit. The engine still pins 522dab4.

A deterministic reproduction at the synced pins: the whole vocabulary comes back NaN at ~400 tokens of context

Not intermittent in this configuration - repeatable, and context-length dependent.

Setup: llama.cpp 39588ef9 (1bit/amd-sync-corpus-repair, which includes the corpus repair, the planner fix and the JIT-cache fix that finally let GLM run at all), hrx-system 3d0cca89a0, GLM-4.7-Flash-Q4_K_M on HRX0, -c 8192, temperature 0.

A 5-token prompt works:

prompt: "The capital of France is"
reply : " Paris. The capital of the United Kingdom"

A ~400-token prompt (1600 characters of filler plus the same question) does not:

E common_sampler_sample: HRX returned NaN logits: count=154880 of 154880,
    index range [0..154879], contiguous=yes, continuing=no (engine#315)
common/sampling.cpp:620: HRX: NaN logits (engine#123)

then the process aborts - which is the 4d13516f guard doing exactly what it was landed to do, and it is why this is legible at all: before it, the run would simply have died with no count, no range and no contiguity.

Two things the numbers say:

count=154880 of 154880, contiguous=yes - the entire output was never written, rather than partially corrupted. That is consistent with your earlier observation that every recorded NaN sits at index 0 (the guard scans upward and reports the first): here it reports that every index is NaN. A decode kernel not writing its output at this shape is a better fit than scattered arithmetic corruption.
It is a threshold, not noise: 5 tokens is fine, ~400 is not, and the same failure appeared with a 9,660-character prompt. Your issue body calls this an intermittent HSA fault; in this setup it looks context-length-triggered and deterministic, which should make it findable.

Repro script and the raw log excerpt are in the workspace at nan-repro/ (repro.sh, README.md). Next step is to bisect the context length to find the threshold and then identify which kernel leaves the output untouched - I will report that rather than guess.

Not claimed: whether this is the #310 "past ~512 KV cells" path or a separate one.

Correction to my previous comment: it is not deterministic - measured rates instead

I said the failure was deterministic. That was too strong, and I have the numbers now. Using GGML_HRX_NAN_CONTINUE=1 so the server survives each event (one long-lived server per session):

prompt	NaN events
~44 tokens (180 chars), 15 identical requests	0 / 15
~360 tokens (1500 chars), 15 identical requests	15 / 15
~360 tokens, 12 distinct prompts	11 / 12
~360 tokens, 12 identical prompts	12 / 12
ascending sweep 128 -> 2048 chars (8 new prompts, one session)	0 / 8

Read honestly: the association with length is strong and repeatable - never once at ~44 tokens, 23 of 24 at ~360 - but it is not universal, because one ascending-size session produced no event at any size up to 362 tokens. So there is a session/state factor neither of us has isolated yet. Prompt reuse is not the trigger: 11/12 for prompts that were all new.

That also means the number in my previous comment (count=154880 of 154880, contiguous=yes) should be read as the shape of one event, not as a property of every long run. It remains the most useful clue: when it happens, the whole output is untouched rather than partly corrupted.

Both measurements are reproducible from nan-repro/ in the workspace; I have recorded the rates there too, including the contradictory sweep, rather than the single result that looked cleanest.

Further data: it is reliable after all (14/15, three consecutive sessions), and two candidate flags are ruled out

Round 32's rate table had one clean session (0/8) and I over-corrected towards "not deterministic". Running the same fixed ~360-token prompt 15 times in each of three consecutive sessions:

session	flag	NaN
baseline	none	14 / 15
no-V-cache-fusion	`GGML_HRX_DISABLE_PREFILL_V_CACHE_FUSION=1`	14 / 15
no-graph-replay	`GGML_HRX_DISABLE_GRAPH_REPLAY=1`	14 / 15

That is 42/45 across three sessions, plus 23/24 in round 32 - so with a fixed prompt the repro is solid at roughly 93%, and the one clean session was a sweep whose prompts each had a different token count. I cannot yet explain that, but the honest summary is "reliable at ~360 tokens, with one unexplained clean sweep", not "not deterministic".

Both flags are now ruled out: the fused prefill V-cache path and graph replay are not what produces the NaN. GGML_HRX_DISABLE_DISPATCH=<name> remains the tool for the next step - bisecting dispatch families against this repro - and the guard's count=154880 of 154880, contiguous=yes still says the output was never written rather than corrupted.

Found the family: it is `mul_mat_id`, the MoE expert matmul

Bisecting with GGML_HRX_DISABLE_DISPATCH (comma-separated substrings of registration names) against the 15-request repro:

disabled	NaN
(baseline)	14 / 15
`fused` (every fused registration)	0 / 15
`mul_mat_id`	0 / 15
`mul_mat_unary`	14 / 15
`mul_mat_postops`	14 / 15
`mul_mat_swiglu`	14 / 15
`swiglu`	14 / 15
`GGML_HRX_DISABLE_PREFILL_V_CACHE_FUSION=1`	14 / 15
`GGML_HRX_DISABLE_GRAPH_REPLAY=1`	14 / 15

So the NaN comes from the mul_mat_id family - GLM's MoE expert matmul - and disabling it removes the failure completely while leaving the rest of the model intact. GGML_HRX_LOG_DISPATCH=1 shows the family in play:

common.mul_mat_id.decode_f32_wave64
common.mul_mat_id.f32_f32_wmma
common.mul_mat_id_swiglu.f32_f32_wmma

Two of those three are our kernels (mul_mat_id_f32_f32_wmma and mul_mat_id_swiglu_f32_f32_wmma are in the pre-merge set the sync kept only as ours/ sources), which connects this to the round-16 finding that every MUL_MAT_ID test case fails at n=129 while n=128/256/512 pass - a remainder after a tile boundary. That would explain both: a token count that is not a multiple of the tile leaves part of an MoE expert output unwritten, the logits that depend on it come back all-NaN (count=154880 of 154880, contiguous=yes), and the op test sees the same shape class fail.

It also finally explains the "session factor" I could not isolate in round 32: 15 identical requests produce 14 NaN, a single request produces none. One request per session was clean here; the sweep whose eight prompts were all different was clean. I have not isolated the state that makes the first request differ, and I am not going to claim the mechanism until I have.

Next step is now concrete: bisect inside the mul_mat_id family (decode_f32_wave64 vs f32_f32_wmma vs mul_mat_id_swiglu) and test token counts around the tile boundary (128/129, 256/257) to confirm the remainder.

serve: refuse a model that will not fit, before any backend starts (P…

serve: CWSR save area in GTT when amdgpu runs CWSR on (engine#315 can…

Move the #315 notes to a private repository; drop the private kernel …

Release blockers: docs drift, NPU status, hrx-system pin note, weekly…

RESOLVED — the CWSR-on acceptance soak passed (37 in-band arms, 9 h 03 min, 0 faults); closing

The trigger named on 2026-10-18 is confirmed by the acceptance soak this issue required, and the repair is
already shipped. Re-verified today from the raw arm directories, not from a summary (~/wt/315soak).

Trigger and repair

Trigger. On a CWSR-on box, libhsakmt allocates each queue's CWSR context save area as a pageable
SVM range. KFD eviction can migrate that range out from under a preempted queue, so a wave resumes with a
corrupted save area — which is why the fault was intermittent, why it appeared as arbitrary corruption
(all-NaN logits, the 500 face, HSA faults), why no userspace ordering lever moved it, and why the rate
tracked memory pressure and forced eviction.
Repair (shipped). 1bit serve starts every HRX backend with HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1 when amdgpu.cwsr_enable=1, so the save area is a KFD GTT
BO instead: PR serve: CWSR save area in GTT when amdgpu runs CWSR on (engine#315 candidate 1) #454 (switch, --hrx-cwsr-placement=auto|on|off) and PR Move HRX pin: hrx-system to 82804197b (engine#315 HRX_ALLOW_NO_SVM gate) #455 (pin hrx-system 82804197b,
the HRX_ALLOW_NO_SVM gate). Both on main; nothing left to code.

The soak (strixhalo, 2026-10-18 15:56 → 2026-10-19 04:52, CWSR-on boot, restored to CWSR-off afterwards)

Engine 9228e5a, llama.cpp 8d9ce2c1a, hrx-system 82804197b, 1bit sha256 ccbf5991f6c4…; placement
decided by the engine (auto) with no hand-set variables in the soak shell; 4 GLM-4.7-Flash neighbours
(-c 2048) + sized GTT ballast + one graded GLM-4.7-Flash (-c 8192 -fa on) reloaded per arm.

item	value
in band	32,575 s = 9 h 03 min (`gtt_start` in [100,108] GiB)
in-band arms	37, all `verdict=OK` / `in_band=1`; 0 out of band, 0 INVALID
`gtt_start` / peak / `avail_min`	101-102 / 110-111 / 7-8 GiB
graded detector, 66 requests per arm	0 http, 0 utf8, 0 parse, 0 transport; `identical_distinct=1` every arm
graded chat, 20 greedy per arm	740 requests, 0 bad, 0 mismatch
NaN aborts / HSA faults / backend exits / watchdog kills / hangs	0 / 0 / 0 / 0 / 0
dmesg page faults / ring timeouts / GPU resets / OOM	0 / 0 / 0 / 0; Tctl peak 83 °C, 6 pauses, 0 freezes
placement, graded arms	37 of 37 `csa_gtt=2 csa_svm=0 kbg_fail=0`; the `HRX: CWSR is on; CWSR save area placed in GTT` line in every serve log; the llama-server child's environ exactly `HSA_USE_SVM=0 HSA_USERPTR_FOR_PAGED_MEM=0 HRX_ALLOW_NO_SVM=1`
neighbours	all four `csa_gtt=2 csa_svm=0 gtt_line=1`
non-graded events	2 background neighbour HTTP 502 (isolated; the next request to the same server answered 200; no NaN, no 500, no backend exit, no watchdog kill)

Independent checks I ran on the artifacts while closing this: 37 arm dirs, grep -h '^verdict=' → 37 × OK,
in_band=1 → 37; gtt_start ∈ {101, 102}; every counter in counters.txt zero; and my own grep sweep across
all 37 server.log files for HRX returned NaN, Memory access fault, HSA_STATUS_ERROR, ring … timeout,
GPU reset, Out of memory, watchdog → 0 hits each. counters.txt also lists placement_drift=4; those
four rows are the neighbour entries with env_ok=-1 plus the stale zaya.pids explained in the 2026-10-19 01:30
note — the run's own stop flag never fired, and all four still read csa_gtt=2 csa_svm=0.

Against the control

Phase B, same boot and composition, hand-set environment and no placement: 4 faults / 16 runs. This soak:
0 / 37. Fisher exact, two-sided p = 0.0062.

Grading, and what this does not show

Pre-registered criterion: ≥ 6 h in band and ≥ 42 in-band arms, zero faults. Result: 9 h 03 min, 37
arms at the 9 h wall cap → PASS (cap-limited) per the orchestrator ruling of 2026-10-19 01:30, arm count
reported. (The orchestrator lowered the floor to ≥ 30 arms for a cap-limited PASS; the original 42 is on the
record above.)
0/37 bounds the per-arm rate at 7.8 % (one-sided 95 %). It does not make the fault impossible.
The optional same-boot positive control (--hrx-cwsr-placement=off) was not run, so the contrast rests
on Phase B rather than on a same-session control.
ZAYA1-8B was dropped from the composition on 2026-10-18 19:19 for memory headroom, so the original
"ZAYA alongside" leg is not covered by this run.
CWSR stays off by default. The repair makes a CWSR-on box safe; it does not recommend running CWSR on.
CWSR-off evidence: 11 + 10 in-regime arms, 0 faults (~/wt/315cwsr0-console.log, ~/wt/315cwsr60-console.log).

Documentation closing this out: #518 (KNOWN-ISSUES.md status → RESOLVED with this evidence, docs/hrx.md
updated). Evidence on the box: ~/wt/315soak/{SUMMARY,STATE.md,PROVENANCE,counters.txt,arms/,gate/PASS},
~/wt/315-PATCH-STATUS.md §11, plan ~/wt/315c1/SOAK-GO.md.

test: #315 - exercise the CWSR-on branch on a CWSR-off box (serve seam)

Revert "test: #315 - exercise the CWSR-on branch on a CWSR-off box (s…

docs: #315 - hw-gate results on the PR, the 96 C thermal threshold, a…

Addendum after review: hw-gate on the fix's PR, the soak's thermal threshold, and what is still unmeasured

The closing comment above reported the soak. Three corrections/additions, all now also in KNOWN-ISSUES.md
(PR #518):

1. The six-check hw-gate was run on this fix's PR, on strixhalo. hw_gate is skipped for ordinary PR
events by design, so it was dispatched by hand: run 39892647547,
cc65151 + llama.cpp af04c9f22b97 + hrx-system 82804197bcbe, 338 s, 0 ring timeouts. It was cancelled
when the box went down during the sixth model, so the verdict is "not completed". What ran, passed:
ops 695/695; fa-decode-q8 6/6; kat 9 cases each on six models; needle PASS at 7,082 / 7,403 /
7,082 tokens; placement all on HRX in 170/193/152/99/99/122 graphs; kld PASS with mean KLD / p99 /
same-top: qwen3-0.6b 0.000003 / 0.00004 / 99.90 %, zaya1-8b 0.007729 / 0.11945 / 98.33 %,
qwen3.8-27b −0.000000 / 0.00004 / 100.00 %, qwen3-coder-30b-a3b 0.000912 / 0.01119 / 98.97 %,
qwen3-4b-iq4xs 0.000002 / 0.00005 / 99.85 %. The sixth model's arm was cut off by the cancellation
(rc 143), and its placement line reads 1322 nodes, 0 off HRX.

2. Correction: the soak's thermal abort was 96 °C, not 93 °C. It followed phase B's rules (drivers pause
at 92 °C, resume at ≤ 84 °C, abort at ≥ 96 °C; ~/wt/315soak/STATE.md, 2026-10-18 17:19). Measured peak over
the whole run: 83 °C, 6 pauses, 0 freezes, 0 aborts. The 93 °C rule was not the one in force; saying so
here rather than leaving the stronger claim standing.

3. What is still unmeasured (in the entry's gaps table): GLM-4.7-Flash's own KLD/same-top — no stored
base logits exist for it on this box, so the KLD figures above are for the five gate models; GLM
throughput at this composition (the box went down before llama-bench); the forced-eviction intervention
cell re-run with the fix (needs ~100 GiB of GTT with 4 GLM neighbours, i.e. the owner-gated services-off
window); and an eviction-event trace of the save area (the record traces its allocation class — GTT vs
SVM — not the migration itself).

On the regression test: #454's tests/hrx_cwsr_test.cpp covers auto + CWSR on on any box (it feeds
decide_cwsr_placement directly), while tests/hrx_cwsr_route.sh's CWSR-on block is skipped when
amdgpu.cwsr_enable reads 0 — so on the shipped configuration the decision is unit-covered and the
end-to-end route is not. A seam to close that (ONEBIT_CWSR_ENABLE_FILE) was written and withdrawn:
CodeQL flags an env-derived path reaching fopen (cpp/path-injection), and this repo does not loosen a
gate to land a test.

docs: #315 - record the second hw-gate attempt and why its only FAIL …

docs: #315 resolved - the CWSR-on soak passed (37 arms, 9 h 03 min, 0…

docs: the v2026.43 candidate section and the README status table (#525 …

docs: v2026.43 candidate section + README status table (#527)

docs: mark the two never-measured #315 cells as unmeasured (#578)

Navigation Menu

Repository navigation

HRX: GLM-4.7-Flash intermittent HSA memory fault / NaN logits under varied-length requests (base fork tip too) #315

Re-tested on the current pin: no NaN, no fault — but read the caveat

REPRODUCED — 1 fault in 4 valid runs, with the failure signature

The failure signature

GTT is not obviously the trigger

The part that matters most

Not yet established

Rate so far: 1 fault in 5 valid runs (20%)

Correction to the previous comment: why run 6 did not run

The same defect kills a different model — this is not GLM-specific

The known NaN fix is ALREADY in the pinned tree — so this is a different cause

The guard's own comments name a mechanism — and the guard destroys the evidence it could collect

A discriminating test from data already on disk: it is not a memory ceiling

Every recorded NaN is at vocab index 0 — which is not what scattered corruption looks like

A fix for the evidence loss and the run-killing, in review: 1bit-MONSTER/llama.cpp#93

A deterministic reproduction at the synced pins: the whole vocabulary comes back NaN at ~400 tokens of context

Correction to my previous comment: it is not deterministic - measured rates instead

Further data: it is reliable after all (14/15, three consecutive sessions), and two candidate flags are ruled out

Found the family: it is `mul_mat_id`, the MoE expert matmul

86 remaining items

RESOLVED — the CWSR-on acceptance soak passed (37 in-band arms, 9 h 03 min, 0 faults); closing

Trigger and repair

The soak (strixhalo, 2026-10-18 15:56 → 2026-10-19 04:52, CWSR-on boot, restored to CWSR-off afterwards)

Against the control

Grading, and what this does not show

Addendum after review: hw-gate on the fix's PR, the soak's thermal threshold, and what is still unmeasured

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HRX: GLM-4.7-Flash intermittent HSA memory fault / NaN logits under varied-length requests (base fork tip too) #315

Description

Activity

bong-water-water-bong commented on Oct 5, 2026

bong-water-water-bong commented on Oct 5, 2026

Re-tested on the current pin: no NaN, no fault — but read the caveat

bong-water-water-bong commented on Oct 5, 2026

REPRODUCED — 1 fault in 4 valid runs, with the failure signature

The failure signature

GTT is not obviously the trigger

The part that matters most

Not yet established

bong-water-water-bong commented on Oct 5, 2026

Rate so far: 1 fault in 5 valid runs (20%)

bong-water-water-bong commented on Oct 5, 2026

Correction to the previous comment: why run 6 did not run

bong-water-water-bong commented on Oct 5, 2026

The same defect kills a different model — this is not GLM-specific

bong-water-water-bong commented on Oct 5, 2026

The known NaN fix is ALREADY in the pinned tree — so this is a different cause

bong-water-water-bong commented on Oct 5, 2026

The guard's own comments name a mechanism — and the guard destroys the evidence it could collect

bong-water-water-bong commented on Oct 5, 2026

A discriminating test from data already on disk: it is not a memory ceiling

bong-water-water-bong commented on Oct 5, 2026

Every recorded NaN is at vocab index 0 — which is not what scattered corruption looks like

bong-water-water-bong commented on Oct 6, 2026

A fix for the evidence loss and the run-killing, in review: 1bit-MONSTER/llama.cpp#93

bong-water-water-bong commented on Oct 6, 2026

A deterministic reproduction at the synced pins: the whole vocabulary comes back NaN at ~400 tokens of context

bong-water-water-bong commented on Oct 6, 2026

Correction to my previous comment: it is not deterministic - measured rates instead

bong-water-water-bong commented on Oct 6, 2026

Further data: it is reliable after all (14/15, three consecutive sessions), and two candidate flags are ruled out

bong-water-water-bong commented on Oct 6, 2026

Found the family: it is mul_mat_id, the MoE expert matmul

86 remaining items

bong-water-water-bong commented on Oct 23, 2026

RESOLVED — the CWSR-on acceptance soak passed (37 in-band arms, 9 h 03 min, 0 faults); closing

Trigger and repair

The soak (strixhalo, 2026-10-18 15:56 → 2026-10-19 04:52, CWSR-on boot, restored to CWSR-off afterwards)

Against the control

Grading, and what this does not show

bong-water-water-bong commented on Oct 23, 2026

Addendum after review: hw-gate on the fix's PR, the soak's thermal threshold, and what is still unmeasured

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Found the family: it is `mul_mat_id`, the MoE expert matmul