HRX: hrx-backend-test and test-hrx-ops abort on stale kernel-routing expectations #444

Activity

1. bong-water-water-bong commented on Oct 18, 2026

   

   bong-water-water-bong

   on Oct 18, 2026

   CollaboratorAuthor

   More actions

   Hygiene pass (2026-10-18): no fixed issue to close, because the engine's HTTPS CI never builds these two binaries.

   The builds here are cmake -B build -DCMAKE_BUILD_TYPE=Release with no -DONEBIT_HRX=ON (.github/workflows/ci.yml), and the HRX build (cmake/hrx.cmake) is an ExternalProject_Add(llama_hrx ...) whose BUILD_COMMAND builds only llama-server llama-bench. Neither hrx-backend-test nor test-hrx-ops is in an engine target, so no engine PR can go red on them: the fork has no CI and the engine CI does not compile them. The fork pin reached main via #441 without them building, and #451/#453 would too.

   What is actually fixed and merged in 1bit-MONSTER/llama.cpp (branch 1bit/hrx-vulkan-patched, tip 953b6397):

   • HRX: pin llama.cpp 96f6b89, fixing #95 (GLM-4.7-Flash, Qwen3-Coder at full context) #104 8dafc4a050ba: both files build again (engine#422). tests/hrx-backend-test.cpp and tests/test-hrx-ops.cpp no longer include the headers f2099e9b deleted (dispatch-symmetric-i4.h, dispatch-activation-publication.h), and the deleted eligibility predicate is no longer called. Checked at 7227a771 and 953b6397: no include of either header, neither header in ggml/src/ggml-hrx/dispatch_registration/common/, both targets still in tests/CMakeLists.txt.
   • registry: causes of the two ZINC failures (Qwen2.5-7B load crash, MiniCPM5 empty reply) #105 7227a771: the fail-fast main() became a per-case runner, which is item 2 of this issue. A red case now reports itself and the rest still run.

   So the substance of this issue is done in the fork, but it cannot be closed as verified: nothing on the engine side builds hrx-backend-test or test-hrx-ops, so "they build at the pin" has no CI evidence, and the 13 red rows are still red by design (#105 tags them; #445/#449/#450 track the rulings). Closing it here would claim a build nobody has produced at the engine pin.

   Two things would make it closable:

   1. an engine-side target (or a hw_gate step) that configures the pinned llama.cpp with GGML_HRX=ON and builds hrx-backend-test and test-hrx-ops, so the fork's test tree is gated by the engine pin rather than by hand;
   2. the per-case totals recorded per pin in this issue.
2. bong-water-water-bong commented on Oct 23, 2026

   

   bong-water-water-bong

   on Oct 23, 2026

   CollaboratorAuthor

   More actions

   Device pass done at fork f1670b43 + hrx-system 82804197bc (strixhalo gfx1151, gpu-hold -l 'pi issues-b' -t 14, IREE_HAL_AMDGPU_LIBHSA_PATH set, dmesg clean apart from UFW noise).

   Both binaries, at that pin:

   hrx-backend-test   host:   60 passed, 0 failed, 6 known-fail, 0 unexpected-pass, 66 total
                      device: 11 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 11 total
   test-hrx-ops       host:    2 passed, 0 failed, 2 total
                      device: 194 passed, 45 failed, 239 total   (145 s, one process per case)
   

   Two things worth recording from hrx-backend-test: zero_output_device_support now passes (it was the single device failure when the device half last ran), and common_mul_mat_id_postops_dispatch left the known-fail list without tripping UNEXPECTED-PASS — #131's fix landing under it, so the remaining six are #533 ×3, #534 ×1 and #445 ×2.

   test-hrx-ops: 45 of the 45 failures were mine. The route translation merged in #128 was wrong in direction, and the device run says so plainly:

   missing expected kernel: loom_libs:ggml_kquant_mul_mat_decode_tokens_f32
   scheduled kernels:
     qwen3_moe:ggml_quantize_q8_1_x4_f32
     loom_libs:ggml_mul_mat_f32_f32_wmma
   

   The registry publishes q8 and runs the kernel the table already named — the dispatch-kquant-decode.cpp rule describes when the decode family is eligible, not that it wins. So the table was right and #128's translation should not have replaced it. Fix in 1bit-MONSTER/llama.cpp#132: the case now accepts the table's kernel or the kquant family and prints the scheduled sequence when neither is there; plus the genuinely stale names the same log exposed — registered decode/prefill pairs (llm_attention_v_matmul_set_rows_f32_f32_wmma ↔ …_decode_f32_f32, ggml_flash_attention_f32_f16_wmma ↔ ggml_attention_rows_f32_f16, the MLA rows route already ruled in #445), and three routed-MoE shapes whose fused/next-q8 kernels are not in the sequence any more.

   Host-verified: test-hrx-ops 2 passed / 0 failed. The device re-run of the fix is the one remaining item — the GPU lane went to the Qwen3.8-27B leaderboard before I could take it, so #444 stays open until that run reproduces green on hardware, and the closing comment will carry both totals at the then-current sha.
3. bong-water-water-bong

   added a commit that references this issue on Oct 23, 2026

   Bump llama.cpp: 830e2c161c68 (fork #132, test-hrx-ops route expectati…

   ...

   7b6fbbe