hrx-backend-test: 49 host cases red after the AMD-core re-head — classification (stale expectation vs needs a ruling) #445

Activity

1. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Ruling recommendations for the 19 class C/D cases. Method and evidence per case: the plan each case actually produced (plan.dispatches dumped with kernel names, temporary instrumentation, reverted), whether the kernel a case expects is still in kernel-corpus/ and in dispatch_registration/, and git log -S on the branch for the commit that moved it. No test edits, no PRs.

   The single fact behind most of class C: f2099e9b ("hrx: keep our ggml-hrx backend on AMD's core") dropped our consumer-driven activation-publication set whole - 27 *_publish_* kernels are absent from the corpus and from dispatch_registration/ today:

   loom_libs:ggml_mul_mat_tiled_input_f32_publish_f32{,_aligned,_f16_alternate}   corpus=N reg=N  f2099e9b
   loom_libs:ggml_mul_mat_{tiled,skinny}_input_f32_{bias_residual,iq*,}-publish   corpus=N reg=N  f2099e9b
   loom_libs:ggml_rmsnorm_binary_{q8_1_x4,f16}_publish                             corpus=N reg=N  f2099e9b
   loom_libs:ggml_rmsnorm_gate_f32_publish                                        corpus=Y reg=N  f2099e9b
   qwen3_moe:qwen3_moe_routed_down_weighted_reduce_next_rmsnorm_f32_publish_q8     corpus=N reg=N  f2099e9b
   

   Class C - publication / alternate values (11)

   Case	Recommendation	Reason and evidence
   packed_f16_producer_consumer	REGRESSION	Wants the f16 alternate from a producer that publishes it. Plan: ggml_copy_f16_k16_major, ggml_mul_mat_swiglu_q4_k_f16_wmma_prefill_wave32, ggml_mul_mat_q6_k_f16_wmma_prefill_wave32 - 3 dispatches, and alternate == nullptr where the case expects one. The producer (..._publish_f32_f16_alternate) is gone (f2099e9b); the copy that publication fused into the producer is now its own dispatch. File under the restore-publication issue.
   generic_swiglu_k16_publish	REGRESSION	Expects a *_publish_* producer; plan is ggml_mul_mat_swiglu_f32_f32_wmma, ggml_copy_f16_k16_major, ggml_mul_mat_q6_k_f16_wmma_prefill_wave32 - the publish producer is gone, the copy appears separately. Same root cause.
   generic_swiglu_k16_publication_edges	ACCEPT	Plan for this shape is 1 dispatch (ggml_mul_mat_swiglu_f32_f32_wmma): no copy, no publish, i.e. fewer dispatches than the publication version and the same math on the same kernel that exists. Expectation becomes "one dispatch, no producer".
   tiled_matmul_alternate_publish	REGRESSION	Plan: ggml_mul_mat_f32_f32_wmma, ggml_copy_f32_f16, ggml_mul_mat_q6_k_f16_wmma_prefill_wave32. The copy is a separate dispatch where the publish producer used to fuse it; the producer kernel is gone (f2099e9b).
   rmsnorm_binary_k16_output	REGRESSION	Plan: ggml_rmsnorm_binary_f32, ggml_copy_f16_k16_major, ggml_mul_mat_q4_k_f16_wmma_prefill_wave32. Expected (producer != end) == test.packed: the packed-f16 publish producer no longer exists, and the copy stands alone.
   rmsnorm_binary_f16_output	REGRESSION	Same shape with ggml_copy_f32_f16 as the separate dispatch; the f16 publish producer is gone (f2099e9b).
   swiglu_q8_output	ACCEPT	Plan: ggml_kquant_swiglu_decode_tokens_f32, ggml_kquant_mul_mat_decode_tokens_f32. AMD's decode kernels consume f32 directly, so the q8 publication this case pinned is not needed: 2 dispatches, same math, no extra copy.
   rmsnorm_gate_packed_output	ACCEPT	Plan: ggml_rmsnorm_gate_f32_f16, ggml_mul_mat_q4_k_f16_wmma_prefill_wave32 - the norm kernel emits f16 itself (_f32_f16), i.e. the dtype conversion the case expected from a publish producer is fused into the producer that exists. Equivalent, one fewer dispatch.
   rmsnorm_gate_q8_output	ACCEPT	Plan: ggml_rmsnorm_gate_f32_q8_1_x4, ggml_mul_mat_f32_f32_wmma, ggml_scale_f32. Publication still happens, through AMD's ggml_rmsnorm_gate_f32_q8_1_x4 (in the corpus and registered) instead of our ggml_rmsnorm_gate_f32_publish. Expectation change: the producer's name.
   qwen_decode_publication_consumer_qualification	REGRESSION	Asserts a q8 alternate exists for the decode shape; the kernel that used to publish it (..._publish_q8, and the ggml_binary_f32_publish_q8_1_x4 family) is not in the corpus any more, and the plan is ggml_binary_f32 + ggml_kquant_mul_mat_decode_tokens_f32 with no alternate. Same root cause.
   qwen_weighted_reduce_next_rmsnorm_q8_publication	REGRESSION	Expects qwen3_moe:qwen3_moe_routed_down_weighted_reduce_next_rmsnorm_f32_publish_q8; that kernel is absent from the corpus and the registrations (f2099e9b), so the fused weighted-reduce + rmsnorm + q8 publish path is unreachable.

   What to file for the seven REGRESSION rows: one issue, "restore consumer-driven activation publication (dropped whole in f2099e9b)", listing them with these plan dumps, and stating the cost: each shape gains an explicit copy dispatch (or loses the fused q8 publish) where the pre-re-head plan had none. generic_swiglu_k16_publication_edges, swiglu_q8_output, rmsnorm_gate_packed_output and rmsnorm_gate_q8_output are not part of it - those shapes are equal or better now.

   Class D - behaviour that is not just a renamed expectation (8)

   Case	Recommendation	Reason and evidence
   symmetric_i4_consumer_qualification	REGRESSION	The case exists to prove the symmetric-i4 consumer route is never selected. The plan now dispatches loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32 (corpus=Y, registered=Y, dispatch-rmsnorm.cpp). Registration came back in aadd6025 (Aaryaman Vasishta, "hrx: add generic gfx1151 graph optimizations", AMD-authored); our own f2099e9b deleted it with the comment "Disabled because the route causes a substantial numerical performance regression". Re-enabled against our recorded measurement, with no knob. Confirm with a device run (pp/tg and logits vs the disable), then file.
   gdn_rmsnorm_gate_dispatch	REGRESSION	Expects loom_libs:ggml_rmsnorm_gate_f32_publish: corpus=Y, registered=N. The kernel is shipped and dead - f2099e9b removed the registration, so the fused rmsnorm-gate publish path for gated-delta-net cannot be reached. File with the restore-publication issue (this one is cheap: the kernel still builds).
   iq2_codebook_resource_reuse	ACCEPT	The codebook IQ path was replaced by AMD's generic ggml_kquant_mul_mat_decode_f32 (the codebook kernels are gone; only kquant_decode_f32.loom carries them now). One generic decode kernel needs no per-shape codebook tables, so constant_initializations == 0 is the correct new value; there is no resource left to reuse. Expectation change: 0 constant initializations, ggml_kquant_mul_mat_decode_f32.
   qwen_router_top8_dispatch	NEEDS-DEVICE	The router kernel is still qwen3_moe_router_top8_f32 (it takes %route_id_stride as a workload argument now), but the plan adds AMD's ggml_moe_build_expert_table and ggml_moe_build_expert_partition_table, and bindings[1].length no longer equals route_id_length. Whether that is a moved convention or extra per-token work cannot be settled from the plan alone. Device run must measure: dispatches per decoded token (was 1, now 3?), tg tok/s against the pre-re-head build, and logits/argmax equality on a fixed prompt.
   qwen_token_embedding_dispatch	ACCEPT	The case asserts the manual token-embedding graph is not supported; it now is, and the plan shows it running as the same math on kernels that exist (ggml_get_rows_f32 per embedding + matmul), not through a new special kernel. More graphs served, same operations. Expectation change: supported, with the kernel list.
   gather_add_dispatch	ACCEPT	Same kernel, hrx:ggml_gather_add_f32, but its three dims are now workload/integer arguments (ints=3) instead of compile parameters (cparams=0); the test's require_compile_parameter looks in the wrong map. The corpus kernel declares %source_token_count, %output_token_count, %hidden_size as arguments. Expectation change: assert integer_parameters, not compile_parameters.
   qwen_flash_attention_dispatch	ACCEPT	The route is right: the plan is ggml_copy_transpose_f16 + loom_libs:ggml_flash_attention_f32_f16_wmma, the kernel the case expects. The abort (std::out_of_range: map::at) is the test reading a ggml.flash_attention.* parameter the current dispatch does not set, from one of the schedule_*_flash_attention_* helpers. Expectation change: pin the parameters the kernel sets now (which of them needs one CPU probe, not a device run); no route or dtype change.
   gdn_native_projection_pair_dispatch	ACCEPT	Same finding, pinned: the epilogue value moved from an integer parameter to a compile parameter - epilogue->kernel.integer_parameters.at("element_count") throws, while dispatch-gated-delta-net.cpp:794 sets llm.gated_delta_net.epilogue_element_count. Expectation change: assert that compile parameter; also guard the .at() so a future rename reports instead of aborting.

   Totals

   Recommendation	Count	Cases
   ACCEPT	9	generic_swiglu_k16_publication_edges, swiglu_q8_output, rmsnorm_gate_packed_output, rmsnorm_gate_q8_output, iq2_codebook_resource_reuse, qwen_token_embedding_dispatch, gather_add_dispatch, qwen_flash_attention_dispatch, gdn_native_projection_pair_dispatch
   REGRESSION	9	packed_f16_producer_consumer, generic_swiglu_k16_publish, tiled_matmul_alternate_publish, rmsnorm_binary_k16_output, rmsnorm_binary_f16_output, qwen_decode_publication_consumer_qualification, qwen_weighted_reduce_next_rmsnorm_q8_publication, symmetric_i4_consumer_qualification, gdn_rmsnorm_gate_dispatch
   NEEDS-DEVICE	1	qwen_router_top8_dispatch

   Two issues to file if this is accepted: (1) restore consumer-driven activation publication, dropped whole in f2099e9b - 7 class C cases plus gdn_rmsnorm_gate_dispatch; (2) the symmetric-i4 consumer route came back in AMD's aadd6025 after our f2099e9b removed it as a numerical/perf regression - decide whether to re-disable it (the case says it must never be selected).

   Deferred device work (added to NEED-GPU T422): the 14 device cases, plus the qwen_router_top8_dispatch measurement (dispatches per token, tg tok/s, logits equality) and the symmetric-i4 confirmation (pp/tg and logits with the route on vs off).
2. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Owner ruling on this table: ACCEPTED as posted. Acting on it:

   • Restore consumer-driven activation publication (the 7 class C REGRESSION rows + gdn_rmsnorm_gate_dispatch): engine HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449.

   • The symmetric-i4 consumer route re-enabled by AMD's aadd6025 after our f2099e9b removed it: engine HRX: symmetric-i4 consumer route re-enabled by AMD aadd6025 after f2099e9b removed it #450 (with the deferred on/off device measurement).

   • ACCEPT rows in the test: hrx-backend-test: drop the use of the deleted dispatch-symmetric-i4.h (#422) llama.cpp#103, commit ebeb0e0e — 5 of the 9 updated so far, each change citing its row here:

     • gdn_native_projection_pair_dispatch (element count is the compile parameter llm.gated_delta_net.epilogue_element_count),
     • iq2_codebook_resource_reuse (generic kquant decode; no constant initialization, no transient grid),
     • qwen_token_embedding_dispatch (IQ3_S / IQ4_NL graphs are claimed now),
     • swiglu_q8_output (kquant swiglu -> kquant matmul decode, no alternate),
     • rmsnorm_gate_q8_output (publication via ggml_rmsnorm_gate_f32_q8_1_x4).

     Suite after that commit: 41 pass / 44 fail (was 36 / 49). The 9 REGRESSION rows and the NEEDS-DEVICE row are untouched and still red, as ruled.

   Still to update (not done, and deliberately not guessed at): gather_add_dispatch, generic_swiglu_k16_publication_edges, rmsnorm_gate_packed_output, qwen_flash_attention_dispatch. Each turns out to have further variants beyond the one that failed first, and each needs its own plan dump before an expectation is written - e.g. gather_add_dispatch also asserts that several unsupported graph variants stay unsupported, and three of them are claimed now (the dims became runtime arguments), while the strided-coalesced variant is still refused; generic_swiglu_k16_publication_edges has a second variant whose producer is neither the deleted publish kernel nor the swiglu matmul the first variant uses; qwen_flash_attention_dispatch fails per-variant on kernel names once the parameter and alternate assertions are fixed. Writing those without the plan in hand would encode whatever the backend happens to do, which is the thing the ruling process exists to avoid.
3. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Per-variant evidence for the four ACCEPT rows that were still red, gathered on the new base (8dafc4a0, PR #104, which rewrote the suite into a single fail-fast main()). The backend did not change between 8d9ce2c1a and 8dafc4a0 — only the test file did — so these plan dumps describe what 1bit/hrx-vulkan-patched does today. Dumps are from the scheduler (plan kernel names, alt = alternate values, t = transients), temporary instrumentation, reverted.

   Recommendations are for the owner; I have not edited these tests.

   Row	Variant (graph)	Plan now	Recommendation
   gather_add_dispatch	GET_ROWS, GET_ROWS, ADD — canonical shape	1 fused dispatch hrx:ggml_gather_add_f32 (bindings 4)	unchanged, still right
   	same ops, (2048, 13, 1, false) — the case asserts not supported	3 dispatches: ggml_get_rows_f32, ggml_get_rows_f32, ggml_binary_strided_f32	ACCEPT — the graph is now claimed by the generic path instead of being refused: same math, more shapes served, but 3 dispatches where the fused kernel does 1. Expectation becomes "supported, generic path", and the case keeps asserting the fused kernel for the canonical shape.
   	(2048, 13, 1, true, 1024) (strided-coalesced)	still refused	unchanged, keep the negative
   	(96, 13, 1, true)	supported (generic path)	ACCEPT, same reasoning
   	partial_gather_add_graph	supported	ACCEPT, same reasoning
   qwen_flash_attention_dispatch	prefill shapes	ggml_flash_attention_f32_f16_wmma (bindings 6)	unchanged, still right
   	decode with q8 input	4 x ggml_flash_attention_decode_split_f32_f16_wmma_next_q8 (bindings 10), alt=1, t=4	ACCEPT — the q8 publication still happens (one alternate); the case expects no alternate. Expectation change: that variant publishes one.
   	MLA shape (QK-layout V)	ggml_attention_rows_f32_f16 (bindings 5)	ACCEPT — this variant is routed to the MLA rows kernel now (dispatch-flash-attention.cpp:449 explains the columns-wide copy for MLA); the expected kernel name and its parameter list are what is stale, not the route.
   rmsnorm_gate_packed_output	packed-f16 variant	2 dispatches: ggml_rmsnorm_gate_f32_f16, ggml_mul_mat_q4_k_f16_wmma_prefill_wave32; alt=1 for the packed shape, generated F16K16Major resource present	ACCEPT — the norm kernel emits f16 itself and that value is published as an alternate, where the case expects our deleted ggml_rmsnorm_gate_f32_publish producer and a publish_q8 compile parameter. Caveat: the fail-fast run does not reach this case yet (it stops earlier, currently at the qwen_attention_postprocess variant dispatches.size() == (q8_input ? 5 : 4)), so this dump is from the identical backend via the old runner; re-confirm when the run reaches it.
   generic_swiglu_k16_publication_edges	—	the case no longer exists in the new suite (0 hits in 8dafc4a0); backend unchanged: 1 dispatch ggml_mul_mat_swiglu_f32_f32_wmma, no alternate	row is moot on the new suite; if the case is re-added, the expectation is ACCEPT (single dispatch, no publish producer).

   Also worth recording while it is visible: on the new fail-fast suite the run stops at the first stale expectation, so one red row hides everything after it. Today that is gather_add (hrx-backend-test.cpp:3731); with that flipped out of the way it stops next at qwen_flash_attention (:3768), then at qwen_attention_postprocess (:4167, dispatches.size() == (q8_input ? 5 : 4)), i.e. rows from the earlier class A set are still red on the rewritten file too. A per-case selector (or a non-fatal mode) would make this suite usable for incremental work; right now every fix is all-or-nothing for the run.
4. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Carried the ACCEPT rows onto the #104 suite and made the runner per-case: 1bit-MONSTER/llama.cpp#105 (base 1bit/hrx-vulkan-patched @ 8dafc4a0).

   Host suite there: 47 passed, 13 failed, 60 total (device cases skip without an HRX device; the fork has no CI, so that table is the gate). The 13 red rows are the class A/B rows plus the NEEDS-DEVICE qwen_router_top8_dispatch.

   Also visible now that the suite reports per case: of the 9 REGRESSION rows, six cases no longer exist in #104's suite and the three that remain pass, so #449/#450 currently have no red test protecting them. Left as #104 made them, per the task.
5. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Known-regression guards are in (added to the open 1bit-MONSTER/llama.cpp#105, base 8dafc4a0):

   • Runner has a KNOWN-FAIL status - a tagged case must fail, is counted apart from real failures, and an unexpected pass prints [UNEXPECTED-PASS #nnn] and exits non-zero.
   • All 9 REGRESSION rows are back as guards with their original pre-HRX: pin llama.cpp 96f6b89, fixing #95 (GLM-4.7-Flash, Qwen3-Coder at full context) #104 expectations (6 re-added cases + 3 restored from HRX: pin llama.cpp 96f6b89, fixing #95 (GLM-4.7-Flash, Qwen3-Coder at full context) #104 rewrite), adapted only to compile: activation_publication_diagnostics() → alternate_values(), and the symmetric-i4 predicate dropped because its header was deleted in f2099e9b.
   • Totals on CPU (ryzen): 66 host cases - 44 passed, 13 failed, 9 known-fail (8 x HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449, 1 x HRX: symmetric-i4 consumer route re-enabled by AMD aadd6025 after f2099e9b removed it #450), 0 unexpected-pass. The 13 real failures are the same rows as before (class A/B + the qwen_router_top8 NEEDS-DEVICE row).

   Also deferred to the next GPU slot and written down in ~/NEED-GPU-T422.md: the 14 device cases, the router measurement, the symmetric-i4 on/off, and the MLA (QK-layout) flash shape that now routes to ggml_attention_rows_f32_f16 (kernel time or tok/s vs the flash kernel, plus logits/argmax equality).
6. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Hygiene pass (2026-10-18): not fixed yet, and #106 just moved one row, so the classification in this issue needs a re-read before it is acted on.

   What is merged in 1bit-MONSTER/llama.cpp on 1bit/hrx-vulkan-patched (tip 953b6397):

   • registry: causes of the two ZINC failures (Qwen2.5-7B load crash, MiniCPM5 empty reply) #105 7227a771: the fail-fast main() is now a table of cases with a per-case runner, which is the second item of HRX: hrx-backend-test and test-hrx-ops abort on stale kernel-routing expectations #444 and what makes this classification usable (one red row no longer hides the rest). The same commit takes the ACCEPT rows HRX: pin llama.cpp 96f6b89, fixing #95 (GLM-4.7-Flash, Qwen3-Coder at full context) #104 had left stale, and tags the REGRESSION rows as known-fail per HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449/HRX: symmetric-i4 consumer route re-enabled by AMD aadd6025 after f2099e9b removed it #450.
   • hrx: scalar-scatter SET_ROWS for the non-FA path (-fa off) (#95) #106 953b6397 ("tests: retire the HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449 guard for gdn rmsnorm-gate publication", engine HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449): one of the HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449 guard rows is retired, with the reason that loom_libs:ggml_rmsnorm_gate_f32_publish is not a kernel that exists any more but the publication is still done inside ggml_rmsnorm_gate_f32_f16 / _q8_1_x4 through f16_output_row_width. So at least one row this issue filed under "class C - needs a ruling" has since been ruled.

   Nothing new was measured in this pass: the host suite needs an HRX build, and the only HRX builds on this box are engine ExternalProject builds of llama-server/llama-bench that need the TheRock toolchain and the GPU lock, so the current 47/13/9/0 totals were not re-taken. Last numbers in this thread are from #449 (66 total: 44 passed, 13 failed, 9 known-fail, 0 unexpected-pass on 7227a771).

   The remaining work is unchanged and is a ruling, not a fix: the class A/B rows are expectation updates, and the class C/D rows have to be decided against what the backend intends on this pin. Suggest the next pass starts from the per-case table #105 produced, retires any row already ruled like #449's gdn rmsnorm-gate one, and keeps the rest red.
7. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Device half of the suite, run on strixhalo against gfx1151 (this is the first time these 11 cases have executed since the AMD-core re-head). Build: 1bit/hrx-test-445-ruling @ d9fac8aa, -DLLAMA_BUILD_TESTS=ON -DGGML_HRX=ON, hrx-system d3c731e1, amdclang++ from TheRock for the HIP kernels.

   host:   44 passed, 13 failed, 9 known-fail, 0 unexpected-pass, 66 total     (unchanged from CPU)
   device: 11 case(s)
   device: 10 passed, 1 failed, 0 known-fail, 0 unexpected-pass, 11 total
   

   Device cases: scale_f32_device_support, qwen_expert_table_partition_prefill_512_execution, add_f32, scale_f32, scale_f32_inplace, two_independent_add_f32, chained_add_f32, same_uid_distinct_graph_reuses_graph_program (both variants) and unsupported_op_fails pass.

   One device failure: zero_output_device_support - hrx-backend-test.cpp:8329: ggml_backend_supports_op(backend, cpy) is false, i.e. the backend no longer advertises support for the zero-output CPY the case builds, while the case asserts it does. This is a device-only capability row (the host plan path does not ask supports_op), so it was invisible before. Not in the ruled set; listing it here so it can be triaged with the class A/B rows.

   Two operational notes worth recording for anyone running this suite on the box:

   • The binary reports device: skipped, no HRX devices available unless IREE_HAL_AMDGPU_LIBHSA_PATH points at TheRock's libhsa-runtime64.so.1 (/opt/rocm-therock/lib/python3.14/site-packages/_rocm_sdk_devel/lib/). The distro libhsa rejects the gfx1151 PM4-emulation probe, so HRX registers no device - same reason the engine's hrx_libhsa() exists.
   • The build needs -Wno-error for the vendored IREE/loom code under GCC 15 (array-bounds, format-overflow false positives), hence the compiler wrapper.

   All nine KNOWN-FAIL rows reported [KNOWN-FAIL #449]/[KNOWN-FAIL #450] on the device run too, and none reported UNEXPECTED-PASS - so #449/#450's guards hold on hardware, not just on the CPU planner.
8. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Device measurement done on strixhalo (gfx1151), which closes this row as ACCEPT.

   qwen_router_top8_dispatch is about a plan that gained dispatches and a binding length that no longer equals route_id_length. Measured on Qwen3-Coder-30B-A3B-Instruct-Q4_K_M (MoE, 30.5B/A3B; the router is on every token):

   • tg64 on HRX0: 91.37 ± 0.12 tok/s (llama-bench, -dev HRX0 -ngl 99).
   • greedy 24-token completion at temperature 0, same prompt and seed: HRX0 output identical to the CPU arm ( ' Paris. The capital of Belgium is Brussels. What is the capital of Luxembourg?…').
   • the plan for the router graph is 3 dispatches (qwen3_moe_router_top8_f32 + ggml_moe_build_expert_table + ggml_moe_build_expert_partition_table); the two table builders that came with the re-head do not change the result.

   So the bindings[1].length == route_id_length expectation is stale (the route ids are described by route_id_stride now, and the table builders consume them), not a defect: ACCEPT on this row. Caveat on the record: no pre-re-head build exists on the box, so "was it faster before" was not measured - only that the current numbers are healthy and numerically equal to the CPU path.
9. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   MLA (QK-layout) shape measured on hardware, closing the qwen_flash_attention_dispatch ACCEPT row: the route to ggml_attention_rows_f32_f16 is equal or better and numerically identical.

   GLM-4.7-Flash-Q4_K_M (deepseek2 29.94B/A3B, MLA) on HRX0, llama-bench -dev HRX0 -ngl 99 -p 512 -n 64 -r 2, one arm per route:

   arm	pp512	tg64
   default: rows route (common.attention_strided_f32_f16 -> ggml_attention_rows_f32_f16)	588.27 ± 13.58 tok/s	27.57 ± 1.18 tok/s
   rows disabled (GGML_HRX_DISABLE_DISPATCH=common.attention_strided_f32_f16) -> ggml_flash_attention_f32_f16_wmma	578.82 ± 1.26 tok/s	25.64 ± 1.43 tok/s

   Greedy 24-token completion at temperature 0, same prompt/seed: identical between the two arms ( ' Paris. The capital of the United Kingdom is London. The capital of the United States is Washington, D.C. The…').

   So the MLA shape moving off the flash kernel is not a regression: prefill is level within the error bars, decode is ~7 0x0p+0head, and the output is bit-identical on this prompt. Expectation update in 1bit-MONSTER/llama.cpp#105 stands as written.
10. bong-water-water-bong commented on Oct 18, 2026

    

    bong-water-water-bong

    on Oct 18, 2026

    CollaboratorAuthor

    More actions

    zero_output_device_support classified: stale expectation (ACCEPT), and the case has never run before.

    What it does: builds 3840 x 0 tensors (a zero-token batch) and asserts the backend advertises support for ADD, MUL, SCALE and CPY on them, and not for ACC/SET. On gfx1151 the first four pass except CPY:

    hrx-backend-test.cpp:8329: ggml_backend_supports_op(backend, cpy)
    

    Evidence:

    • The backend has a named zero-output capability: ggml-hrx.cpp:838 zero_output_elision_supported() -> zero_output_elision_safe_op() (line 744), a list of ~30 ops (DUP, ADD, ADD1, SUB, MUL, DIV, SUM_ROWS, RMS_NORM, MUL_MAT, ...). GGML_OP_CPY is not in it (0 occurrences in the list).
    • device_supports_op() (line 1132) therefore returns false for that node and falls through to the per-op matchers, none of which claims a zero-element copy. So the backend does not advertise support; it is not a runtime failure, and the node would be placed on the CPU and copy zero bytes.
    • The case came in with HRX: pin llama.cpp 96f6b89, fixing #95 (GLM-4.7-Flash, Qwen3-Coder at full context) #104 (8dafc4a0), is a device_case, and the device half of this suite had not been executed at all since the re-head - so its CPY expectation was never checked against the shipped backend. The elision list itself arrived in AMD's aadd6025 ("add generic gfx1151 graph optimizations").
    • What the engine needs: the elided shapes are the arithmetic/norm nodes llama.cpp emits for a zero-token batch; nothing in the engine or in llama.cpp's model path builds a zero-element CPY - the only way to get one is the hand-built ggml_cpy(lhs, rhs) the case itself constructs.

    Recommendation: ACCEPT - change the assertion to REQUIRE(!ggml_backend_supports_op(backend, cpy)) and note that a zero-element copy is a CPU-placed no-op. Two caveats for the record: the omission of CPY from the elision list reads as incidental rather than reasoned (eliding a zero-byte copy is as safe as eliding DUP), and if a real graph ever contains one it costs a graph split, which #390's placement report would then name. If you would rather have the capability, the fix is one case GGML_OP_CPY: in zero_output_elision_safe_op(), i.e. a backend change, not a test change - say which and I will do it.
11. bong-water-water-bong commented on Oct 23, 2026

    

    bong-water-water-bong

    on Oct 23, 2026

    CollaboratorAuthor

    More actions

    Classification of the host suite at the current engine pin, taken today. Everything here is plan-level
    evidence from the scheduler (temporary instrumentation printing plan.dispatches kernel names and binding
    counts, reverted afterwards), plus the backend's own verify_command_program() where the case fails on it.

    Where this was taken, and one pin caveat

    • fork 1bit/hrx-vulkan-patched @ 851fa6c8 (3 commits ahead of the engine's llama.cpp pin af04c9f2;
      tests/ and the corpus are identical to the pin).
    • hrx-system 82804197b, not the engine's pinned 98d05d94: the pinned Loom predates the multiple_of
      rename that our corpus needs (fork 37c499d5), so loom-link rejects the corpus at the pinned pair and
      no test binary can be built there. Reported separately; the numbers below are the first that exist.
    • build: -DGGML_HRX=ON -DGGML_HRX_KERNEL_CORPUS_SOURCE_FORMAT=binary -DGGML_HRX_HIP_TARGETS=gfx1151,
      GCC 15 with a -Wno-error wrapper. (The fork also needed a one-line fix to compile at all: op_params_json()
      had no PadParams branch since ba64834d/PR Zyphra Zamba2 on Vulkan: carry ggml-org#21412, and a d_state-64 Mamba-2 scan #114 — hard error with GGML_HRX=ON.)

    host: 66 case(s)
    host: 49 passed, 12 failed, 5 known-fail (#449), 0 unexpected-pass, 66 total
    device: skipped, no HRX devices available
    

    None of the original class A/B rows (scale_add_f32_dispatch, rmsnorm_two_binary_dispatch, graph_index,
    rope_set_rows_dispatch, get_rows_*, dispatch_registry, …) are red any more: they are green on this pin.

    The 12 red host rows

    Case	Failing assertion	Plan the scheduler actually produces	Classification
    qwen_attention_postprocess_dispatch	dispatches.size() == (q8_input ? 5 : 4)	3 × ggml_kquant_mul_mat_decode_tokens_f32 + qwen3_moe_attention_postprocess_f32_f16 = 4	A/B — for ≤8 tokens our kquant_mul_mat_decode consumes f32 directly, so the separate q8-publication dispatch is gone. Update to 4.
    qwen_matmul_dispatch	dispatches.size() == command_count	1 × ggml_kquant_mul_mat_decode_tokens_f32	A/B, same cause: command_count is q8_input ? 2 : 1, and there is no quantize dispatch any more.
    lowtoken_residual_dispatch	fused == (c.fused ? 1 : 0)	1 × ggml_kquant_mul_mat_decode_f32 with 4 bindings (the residual is fused into the kquant decode)	A/B — the case only counts ggml_mul_mat_bias_*/…_add_* names as the fused route; the kquant decode row is the fused route now.
    gdn_selected_rms_q8_dispatch	command_program_verifies(commands)	4 × ggml_scale_f32, llm_gated_delta_net_f32_wmma_head128_selected_snapshot_projection_rms_gate_q8 (14 bindings), ggml_mul_mat_f32_f32_wmma, ggml_scale_f32	REGRESSION (verifier) → engine #533.
    quantized_value_projection_dispatch	dispatches.size() == 2	1 × llm_attention_v_matmul_set_rows_decode_f32_f32	A/B — the decode variant is what the registration names for this shape; the separate quantize dispatch is gone.
    schedule_get_rows_q8_1_alternate_command.{q4_k,bf16}	get-rows/vocab kernel expectations	qwen3_moe:qwen3_moe_rmsnorm_f32_quantize_q8_1_x4/b4, qwen3_moe:ggml_linear_q6k_q8_1_x4/b3	A/B — the get-rows + q8-publication pair is served by the rmsnorm-quantize + qwen q6k linear now; expectation needs re-pointing at those two dispatches. Still being pinned down (the failing assertion moved when the case was instrumented).
    qwen_router_top8_dispatch	bindings[1].length == route_id_length	qwen3_moe_router_top8_f32/b3 + ggml_moe_build_expert_table/b2 + ggml_moe_build_expert_partition_table/b2; binding lengths 2048, 1568, 128	A/B and already ruled on hardware in this issue (ACCEPT: the route ids are described by route_id_stride, the table builders consume them). Only the expectation is stale.
    common_mul_mat_id_swiglu_dispatch	… == "loom_libs:ggml_mul_mat_id_f32_f32_wmma"	match dispatch = ggml_mul_mat_id_decode_f32_wave64/b4	A/B — common.mul_mat_id.decode_f32_wave64 is the registration that names the one-token route (same ruling as the Q4_K one-token → ggml_kquant_mul_mat_decode_f32 row).
    qwen_routed_gate_up_dispatch	same	routed FFN taken through match_dispatch_at_index(); the scheduler plan for the graph is router + the two expert-table builders	A/B, same decode-route reasoning; the case's match.dispatches dump still to be pinned down.
    common_mul_mat_id_postops_dispatch	command_program_verifies(commands)	the plain ggml_mul_mat_id_postops_f32_f32_wmma rows verify ok=1 errors=0; the …_next_rmsnorm_f32_f32_wmma row (10 bindings) fails: binding bias value=5 origin=Transient access=Read range=[0,16384) reads transient value 5 before write	REGRESSION (verifier) → engine #533.
    disjoint_transient_plan_packing	transients.arena_size == 256	4 × ggml_binary_f32, verification clean, 2 allocations, arena 512	REGRESSION (plan efficiency, not correctness) — the two disjoint transients no longer share an arena slot (transient-allocator.cpp:364 sets arena_alignment = 256, so 2 × 128 B used to land in one 256 B arena). Worth a row of its own rather than an expectation bump.

    So: 8 rows are stale expectations whose routes the current registrations do name (and two of those were
    already ruled here on hardware), 2 rows are verifier rejects (#533), 1 row is a plan-packing
    regression, and 1 row (schedule_get_rows_q8_1_alternate_command) is a stale expectation whose exact
    dispatcher still needs pinning down.

    The five #449 guards on this pin

    Re-pointed at the publication the line actually makes (producer names → "one dispatch writes the
    alternate/pack from this value and a consumer reads it"), markers removed: generic_swiglu_k16_publish,
    tiled_matmul_alternate_publish and rmsnorm_binary_f16_output pass. Two still have a real gap and are
    being ruled under #449 rather than papered over:

    • rmsnorm_binary_k16_output — the packed column is now correct per has_qualified_rmsnorm_k16_consumer()
      (512 × {4096,5120,6144,8192} or 1024 × 5120), but the hidden == 7168 row asserts an f16 alternate that
      this line does not publish at all.
    • packed_f16_producer_consumer — one table row expects an f16 alternate where the plan produces neither an
      alternate nor a k16-major generated resource.

    Both are the same question (“the f16 operand is simply not published for this shape — restore it or rule the
    loss accepted with the measured cost”), which is #449's decision, with the A/B llama-bench numbers being
    taken now.

    What I am doing next

    1. Apply the A/B expectation updates above (8 rows), review each against the registration that names it.
    2. Keep the two verifier rows red as known_fail("#533") so the suite is green except rows pointing at an
       open issue, and the packing regression as known_fail("#<row to file>") unless the packer's change turns
       out to be intended.
    3. Finish HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449's two remaining rows with the A/B measurement, then the device half of the suite under
       gpu-hold and the closing totals here.
12. bong-water-water-bong commented on Oct 23, 2026

    

    bong-water-water-bong

    on Oct 23, 2026

    CollaboratorAuthor

    More actions

    Host suite is green at the pin after the expectation pass. Fork PRs:
    #126 (test expectations, closes #449) and
    #127 (the test-hrx-ops runner + the PAD compile fix).

    host: 66 case(s)
    host: 58 passed, 0 failed, 8 known-fail, 0 unexpected-pass, 66 total   (exit 0)
    

    The 8 known-fail rows, all pointing at an open issue:

    Row	Marker	Why it stays red
    gdn_selected_rms_q8_dispatch, common_mul_mat_id_postops_dispatch	#533	fused plans the backend's own verify_command_program() rejects
    disjoint_transient_plan_packing	#534	disjoint transients no longer share an arena slot (256 -> 512)
    qwen_attention_postprocess_dispatch, qwen_matmul_dispatch, lowtoken_residual_dispatch	#445	stale plan-shape expectation, but the correct number is a plan question, not a table cell (below)
    qwen_router_top8_dispatch, qwen_routed_gate_up_dispatch	#445	expectation chain not finished — bindings/lengths of the packed/expert-table route differ from the wmma route throughout the helper, not in one assertion

    Rows that went green with evidence, beyond the ones listed in my classification comment:
    quantized_value_projection_dispatch (one …_set_rows_decode_f32_f32 dispatch, no quantize),
    schedule_get_rows_q8_1_alternate_command.{q4_k,bf16} (q8 rows consumed by the qwen q6k linear or the packed
    token1 final projection), tiled_matmul_alternate_publish (two consumers now publish both operands,
    Q8_1 and F16), packed_f16_producer_consumer, rmsnorm_binary_k16_output, rmsnorm_binary_f16_output,
    generic_swiglu_k16_publish.

    One harness bug worth recording: require_compile_parameter() called std::abort() on a mismatch, which
    killed the whole run instead of failing one case — it truncated two runs today and is why the earlier lists
    in this thread stopped early. It throws now.

    Deferred by the release-candidate holds (re-queued): the device half of this suite and of test-hrx-ops, and
    the 30B-A3B arm of #449's A/B measurement (Qwen3-8B: pp512 2070.84 vs 2115.34, tg128 42.45 vs 42.53 with the
    publication path disabled — no measurable cost).
13. bong-water-water-bong commented on Oct 23, 2026

    

    bong-water-water-bong

    on Oct 23, 2026

    CollaboratorAuthor

    More actions

    Remaining rows, host-side, in fork PR 1bit-MONSTER/llama.cpp#129 (merged #128 already landed the kquant route translation for test-hrx-ops).

    Now green with the rule documented in the test: qwen_matmul_dispatch and qwen_attention_postprocess_dispatch (kquant decode route: one and four dispatches, no separate q8 publication, no q8 transient), lowtoken_residual_dispatch (the residual ADD is folded into the kquant decode store's fourth binding, so the fused counter counts that route), qwen_router_top8_dispatch (route-id binding carries ggml_nbytes(route_ids); the ascending and descending router graphs are claimed on this pin, the case asserted they were refused).

    Moved to #533: qwen_attention_postprocess_dispatch's postprocess program and qwen_routed_gate_up_dispatch's decode route fail verify_command_program — the second with a forward dependency (command 6 reads a transient command 7 writes). Both tagged known_fail("#533") and added to that issue.

    Still known_fail("#445"), one downstream assertion each, all binding-order/pointer checks left from the pre-re-head helpers rather than routing:

    • qwen_matmul_dispatch: commands.commands.back().bindings.size() == 3 (the kquant store carries the residual as a fourth binding),
    • lowtoken_residual_dispatch: the fused counter on a later table row,
    • qwen_router_top8_dispatch: one more unsupported-graph negative block.

    Suite at the pin: 58 passed, 0 failed, 8 known-fail (#533 x3, #534 x1, #445 x3), 0 unexpected-pass, exit 0. Device half still deferred to the GPU lane.
14. bong-water-water-bong commented on Oct 23, 2026

    

    bong-water-water-bong

    on Oct 23, 2026

    CollaboratorAuthor

    More actions

    Re-read at the current pin (fork tip 7ca8ba5a + hrx-system 82804197bc, the engine main pair) rather than the older af04c9f2/851fa6c8 pairs the earlier rulings were written against: fork PR 1bit-MONSTER/llama.cpp#130.

    row	verdict at this pin	evidence
    qwen_router_top8_dispatch	green	the descending-order router (explicit route count 33) and a 16-expert logits tensor are claimed now, and the route-id binding length is ggml_nbytes(route_ids) = 1568 (ne0=8, ne1=4, nb1=512), which is what the command reports
    qwen_matmul_dispatch	still known_fail("#445")	the row matrix moves in four places at once: dispatch count (unfolded residual adds a standalone ADD), kernel pair (a two-token Q6_K projection still goes to the generic wmma, not the kquant decode family), weight-format code (plain 6 vs packed 46 on the same route), SwiGLU routes further down. Left red with these findings rather than rewritten blind
    lowtoken_residual_dispatch	still known_fail("#445")	fused-vs-unfused is not readable from the plan any more: tokens=1 q4_K fused_exp=1 and tokens=1 q4_K fused_exp=0 both plan ggml_kquant_mul_mat_decode_f32/b4 (+ ggml_binary_f32/b3 in the unfused case). Needs the kernel's binding semantics, so it is a kernel-side question, not an expectation
    qwen_attention_postprocess_dispatch	known_fail("#533") re-confirmed	VERIFY-ERROR: command 0 … bindings=12 … query_input value=22 / key_input value=2 / value_input value=15 … reads transient value before write
    qwen_routed_gate_up_dispatch	known_fail("#533") re-confirmed	VERIFY-ERROR: command 6 … deps=1 … raw_gate value=33 … reads transient value 33 before write by command 7 — forward dependency

    Also in that PR: the projection is selected as the last matmul dispatch instead of dispatches.back() (an unfused residual ends the plan with the ADD), and the expected kernel for that case is now "the table's kernel or one of our registered matmul routes" — the specialised registrations (q6_k packed token1 final projection, kquant decode, wmma/decode) gate on shape and the registry picks between them, so pinning one per row was wrong.

    Totals at this pin: hrx-backend-test 59 passed, 0 failed, 7 known-fail (#533 x4, #534 x1, #445 x2), 0 unexpected-pass, exit 0; test-hrx-ops host 2 passed, 0 failed (237 device cases still skip). #533 is bw220's — the four rows stay tagged and I have not touched the kernel side.