Repository navigation
hrx-backend-test: two fused plans fail the command-program verifier (transient read before write) #533
Description
Two host cases in hrx-backend-test fail the backend's own command-program verifier (not a naming
expectation), on the current pin. Both are fused MoE/GDN paths where a kernel reads a transient value
that the plan writes later, so build_command_program() produces a command program that
verify_command_program() rejects. Found while finishing engine #445's classification at the pin
(llama.cpp af04c9f2, hrx-system 98d05d94 — see the pin note at the end).
common_mul_mat_id_postops_dispatch — mul_mat_id_postops_next_rmsnorm
The plan is 4 dispatches; command 2 is the fused postops+next-rmsnorm kernel with 10 bindings:
DUMP commands: loom_libs:ggml_moe_build_expert_table/b2, loom_libs:ggml_moe_build_expert_partition_table/b2,
loom_libs:ggml_mul_mat_id_postops_next_rmsnorm_f32_f32_wmma/b10
DUMP verify: ok=0 errors=1 arena=17408 allocs=4
DUMP verify-error: command 2 kind=Kernel kernel_id=343748864632602669 bindings=10 deps=1
binding bias value=5 origin=Transient access=Read range=[0, 16384)
reads transient value 5 before write
The next_rmsnorm variant is the only row of that case that fails; the plain
ggml_mul_mat_id_postops_f32_f32_wmma rows verify (ok=1 errors=0 arena=768 allocs=2). So the fused
variant's binding order or its transient declaration is wrong: it reads the bias/next-norm input as a
transient that no earlier command writes.
gdn_selected_rms_q8_dispatch — gated-delta-net selected snapshot + rms-gate + q8
Same assertion (command_program_verifies(commands)), same plan for every table row:
loom_libs:ggml_scale_f32/b2 x4, loom_libs:llm_gated_delta_net_f32_wmma_head128_selected_snapshot_projection_rms_gate_q8/b14,
loom_libs:ggml_mul_mat_f32_f32_wmma/b3, loom_libs:ggml_scale_f32/b2
The failing one is the q8 fused GDN kernel with 14 bindings; the case asserts the q8 alternate and the
consumer wiring up to that point and then fails on verification. (Verifier message for this one still to
be attached — the run stops at the first failing row.)
Why it is filed rather than fixed here
Both are kernel/registration-side (the fused variants' binding and transient contracts), not test
expectations: the plan is what the current registrations produce. They are the only two host rows that
block a green suite — the other ten red rows in #445 are expectation updates (the routes are the ones the
current registrations name for those shapes), and the five #449 guards were re-pointed at the publication
the line actually makes.
Pin note
The two cases cannot be built at the engine pin as pinned: third_party/hrx-system 98d05d94 predates
Loom's multiple_of rename that our corpus requires (fork 37c499d5), so loom-link rejects the corpus;
and ggml/src/ggml-hrx/graph/graph-diagnostics.cpp is missing a PadParams branch in op_params_json()
since ba64834d (PR #114), which is a hard compile error with GGML_HRX=ON. Both are reported separately;
this issue's evidence was taken with hrx-system 82804197b (first pin where the corpus parses).
Activity
bong-water-water-bong commentedon Oct 23, 2026
Two more rows of the same class, found while finishing engine #445's remaining rows at the pin (fork PR 1bit-MONSTER/llama.cpp#129):
qwen_attention_postprocess_dispatch— the three q4_k/q6_k projections go throughggml_kquant_mul_mat_decode_tokens_f32and the fused postprocess kernel follows; the program fails verification with three reads of transient values nothing writes:qwen_routed_gate_up_dispatch— the four-token routed gate/up takesggml_mul_mat_id_decode_f32_wave64and then fails with a forward dependency, which is the more interesting shape:i.e. command 6 reads a transient that command 7 writes: the decode variant's dependency edge points the wrong way (or its
depscount is one short). Both cases are taggedknown_fail("#533")inhrx-backend-test, andcommand_program_verifies()there now printsVERIFY-ERROR: <message>for every error, so this class of failure is diagnosable without instrumenting a case.Host suite totals after the change: 58 passed, 0 failed, 8 known-fail (#533 x3, #534 x1, #445 x3 rows still needing one downstream binding assertion each).
bong-water-water-bong commentedon Oct 23, 2026
Verdict: the model-vs-outputs answer is negative for both families — the verifier is rejecting a declaration, not a miscompute. One case is class (b) declaration, the other is class (a) planner deps; neither changes numbers on the models that use it.
Method: reproduced/classified statically at the newest pin I can fetch from strixhalo (
af04c9f22+hrx-system
82804197), then proved on models with the fused registration disabled vs enabled, comparingper-token top-5 logprobs (bit-exact) plus greedy text and a cold-vs-warm two-turn repeat. All runs
90 W cap, PCM pad (23 Oct).Pin note (blocks step 1 as written): engine main's llama.cpp pin
851fa6c8d8f2(and the7ca8ba5athebrief names) cannot be fetched from strixhalo —
git fetch origin <sha>returnsupload-pack: not our ref,the fork publishes only three heads (
master,1bit/mtp-cum-budget,1bit/kv-q8-salvage), and enginemain's own
third_party/llama.cppsubmodule fetch fails the same way. So the #533 tags and theVERIFY-ERROR:printing (added at that pin) are not available here; the verifier itself is present ataf04c9f22, which is what I classified against.Case 1 —
common_mul_mat_id_postops_dispatch(postops + next_rmsnorm): class (b), declaration(c) binding order: excluded. The fused export declares 10 bindings
[input, expert_table, partition_table, weight, bias, residual_input, residual_output, norm_weight, normalized_output, completion_counters](loom-libs manifest) andmatch_mul_mat_id_postops_dispatchpushes exactly those 10 in that order; the plain export declares 7 and the dispatch pushes 7. No offset.
(b) declaration: confirmed.
has_bias == 0binds the output transient into the slot the exportdeclares
biaswith accessread(andresidual_inputlikewise whenhas_residual == 0). The kernelguards that read behind the
has_bias/has_residualconfig, so the byte is never loaded — but thedeclaration says "read" and the value is written later by the same command, which is exactly the
error's shape. The failing test row is the one with
include_bias = false(
tests/hrx-backend-test.cpp:7124,build_common_mul_mat_id_postops_graph(ctx, false, true, true));the rows that pass have the bias present.
Model evidence (Qwen3-Coder-30B-A3B Q4_K_M): fused family enabled vs
GGML_HRX_DISABLE_DISPATCH=common.mul_mat_id_postops.f32_f32_wmma:Cold-vs-warm (same server, second pass over the same prompts): identical on all three.
Case 2 — GDN
selected_snapshot ... q8: class (a), planner dependenciesThe reported
command 6 reads transient value 33 before write by command 7is a forward edge, i.e. thefused programme's dependency list is short or mis-ordered, not a declaration problem and not a binding
offset. Confirming it needs the newer pin's
VERIFY-ERRORoutput.Model evidence (Qwen3.8-27B UD-Q4_K_XL
--mtp): fused family enabled vsGGML_HRX_DISABLE_DISPATCH=llm.gated_delta_net.selected_snapshot:Cold-vs-warm: identical. So this path is not a source of run-to-run non-determinism on the 27B MTP route
either.
What this means for the box's two open mysteries
Neither family changes numbers on the models that use it, and neither is non-deterministic across a
cold/warm repeat, so #533 is a plan-declaration/ordering defect with no model-visible effect on these
paths — it does not explain the Qwen3.5-4B 1/10 non-determinism or the 27B
--mtpGDN NaN family. TheNaN lead needs a different instrument (catch the poisoning turn, not the steady state).
Fixes implied (not shipped: both are release-candidate-adjacent and neither changes outputs)
has_bias == 0/has_residual == 0, do not bind the output transient into a slotthe export declares
read— bind a dedicated read-only scratch (or teach the declaration that thoseslots are conditionally read). Either is a small change in
dispatch_registration/common/dispatch-mat-id.cppplus the loom-libs manifest bit.selected_snapshotprogramme the edge from its producer ofraw_gate/inputbefore the consumer, i.e. fixdepsfor that dispatch.bong-water-water-bong commentedon Oct 25, 2026
Status at fork tip
e7e5cd08with #133's branch (f7071dde) merged in locally, hrx-system82804197bc, host planner,GGML_HRX=ON:Row 1 —
common_mul_mat_id_postops_dispatch: fixed by #133'sd6d0f715. The exports declare the bias and residual slots with read access, and the matcher was filling an absent slot with the output transient, so the plan read a value nothing had written. Binding a read-only value there is the correct contract; the row passes and its guard is retired in the same commit.Row 2 —
gdn_selected_rms_q8_dispatch: not fixed yet, but its first cause is. #133'sf7071ddecorrects the real ordering bug (the one-token programme's gate producer was dropped fromselected.dispatcheswhile its nodes stayed covered). With it, the plan keeps the producer:and the case now reaches the verifier, which rejects the plan for a second, different reason:
Command 4 is the snapshot
ggml_copy_f32. Its source belongs to the node the fused kernel now covers, so nothing writes it: the fusion is claiming a programme whose rmsnorm/gate output still has a consumer outside it. That is a fusion-eligibility guard, not an ordering fix — refuse the fused programme when any consumer of the rmsnorm output falls outsideselected+rms_match, in the style of the existingvalue_ready/ only-consumer checks.Two other rows are in the same class and need the same treatment in their own matchers:
qwen_attention_postprocess_dispatch(command 0, 12 bindings:query_input/key_input/value_inputread before write) andqwen_routed_gate_up_dispatch(command 3, 4 bindings:inputvalue 16).So #533 needs three guards/fixes, of which #133 lands one row. The
#533markers stay on the three remaining rows.