Repository navigation
HRX: symmetric-i4 consumer route re-enabled by AMD aadd6025 after f2099e9b removed it #450
Description
hrx-backend-test's symmetric_i4_consumer_qualification exists to assert that the symmetric-i4 consumer route is never selected. On the branch tip it is selected:
placement/plan dump for symmetric_i4_consumer_qualification (host case, no GPU):
loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32
loom_libs:ggml_kquant_mul_mat_decode_tokens_f32
loom_libs:ggml_kquant_mul_mat_decode_tokens_f32
loom_libs:ggml_scale_f32
loom_libs:ggml_scale_f32
Evidence
- Our own removal, with the reason:
f2099e9b("hrx: keep our ggml-hrx backend on AMD's core") deleted
ggml/src/ggml-hrx/dispatch_registration/common/dispatch-symmetric-i4.h, whose two functions were a route
switch that returnedfalseand the eligibility predicate. The file's comment, verbatim:// Disabled because the route causes a substantial numerical performance regression.
git log --oneline --diff-filter=D -- ggml/src/ggml-hrx/dispatch_registration/common/dispatch-symmetric-i4.h
gives exactly that commit, one file changed. - AMD's re-enable:
aadd6025(Aaryaman Vasishta, "hrx: add generic gfx1151 graph optimizations") registers
loom_libs:ggml_rmsnorm_binary_symmetric_i4_k32again in
ggml/src/ggml-hrx/dispatch_registration/common/dispatch-rmsnorm.cpp. The kernel is in the corpus
(loom-libs/ops/rmsnorm_binary_f32.loom, listed inloom-libs/manifest.json), and the plan above shows it
being dispatched. - The pre-re-head dispatcher therefore could not produce this dispatch; the post-re-head one does, on a graph the
test builds precisely because it used to be a symmetric-i4 consumer candidate.
Why it is not just a test update
The route was disabled on the strength of a measurement we recorded; nothing in this tree recovers that knob
(there is no env var or flag that turns the route off - the registration is unconditional). Either the regression
no longer applies on AMD's current kernels (possible: the route's kernel and the corpus both changed since), or it
does.
What a device run must measure (deferred; strixhalo is in #315 phase B)
- Same model, same prompt, same
-ub, twice: with the symmetric-i4 registration present and with it removed
(a one-line local revert is enough for the second arm). - Numbers: pp512 and pp4096 tok/s, tg128 tok/s, and output equality (logits or argmax on a fixed greedy prompt).
- Report the qualified consumers actually hit (the case lists direct, reshape and strided consumers), so the
measurement covers the route the test is about and not only a generic matmul path.
If it is level or better, the case's assertion is obsolete and should be removed with this issue as the record. If
the old measurement still holds, the registration should go back behind a switch, and the case keeps its meaning.
Related: engine #445 (ruling comment), 1bit-MONSTER/llama.cpp#103 (the build/port PR; this case is untouched there).
Activity
bong-water-water-bong commentedon Oct 18, 2026
Guard in place: 1bit-MONSTER/llama.cpp#105 (on the open #105 branch).
symmetric_i4_consumer_qualificationis back with its original pre-#104 expectations, adapted only to compile (the eligibility predicate lived indispatch-symmetric-i4.h, which f2099e9b deleted). It is taggedknown_fail("#450"): it must fail while AMD'saadd6025registration is in the tree, and the runner prints[UNEXPECTED-PASS #450]and exits non-zero if it ever passes - i.e. when the route is removed or the case is retired on purpose.Host suite on CPU: 9 known-fail rows (8 x #449, 1 x #450), 0 unexpected-pass. The on/off device measurement in #450 is still deferred behind the #315 cell; it is recorded in ~/NEED-GPU-T422.md along with the run/measure list.
bong-water-water-bong commentedon Oct 18, 2026
Hygiene pass (2026-10-18): adds no new fix, leaves the state in one place.
Open findings, in the order they can be closed:
7227a77/ hrx-system2ab3114966a4. Baseline on that pin: 44 passed, 13 failed, 9 known-fail, 0 unexpected-pass, 66 total.symmetric_i4_consumer_qualificationis still taggedknown_fail("#450")at the branch tip953b6397, so the runner prints[UNEXPECTED-PASS #450]and exits non-zero the moment AMD'saadd6025registration leaves the tree. That is the signal to retire this issue: it fires by itself.953b6397) retired one HRX: restore consumer-driven activation publication (dropped whole in f2099e9b) #449 guard, so the table needs a re-read.No build or suite run was done in this pass: an HRX build here is the engine
ExternalProject(TheRock toolchain, GPU lock), and the box was busy with anhw_gaterun.One thing worth fixing separately: neither this issue's guard nor #449's is gated by the engine.
.github/workflows/ci.ymlconfigures without-DONEBIT_HRX=ONandcmake/hrx.cmakebuilds onlyllama-server/llama-bench, so no engine PR compileshrx-backend-test; the fork has no CI either. The guards therefore only run when someone builds them by hand, which is how #450 can stay red for a long time unnoticed except in this thread.bong-water-water-bong commentedon Oct 18, 2026
On/off measurement attempted on strixhalo during the T422 slot and could not be made: there is no model on the box that reaches the route.
~/models/fmt/Qwen3-0.6B-q5_k.gguf,-iq4_xs.gguf) are Q8_0 inside - llama-bench reportsqwen3 0.6B Q8_0- and quantizing that fixture with llama-quantize fails.GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32) changes nothing - Qwen3-4B-Q4_K_M tg64 75.26 ± 0.20 vs 74.81 ± 0.07 tok/s, i.e. within noise - so the route is not being exercised at all by Q4_K_M/Q8_0 graphs, and a tok/s comparison would measure nothing.known_fail("#450")guard still reports KNOWN-FAIL on hardware as well as on the CPU planner, so the route being enabled stays visible.To close this properly, #450 needs one of:
GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32as the off arm - about 20 minutes;device_casebuilding the qualifiedrmsnorm_binary -> MUL_MATshapes directly), which needs no model at all - about 40 minutes including the build.Recommendation: 2, because it tests the exact consumers the case lists and does not depend on whatever quants are on the box. The measurement is still outstanding; nothing was concluded about the regression itself.
bong-water-water-bong commentedon Oct 18, 2026
On/off measurement done on strixhalo (gfx1151) with models that actually reach the route. Result: the route is not a regression on the current kernel set — decode is slightly faster with it, and perplexity is identical.
Models:
Qwen3-4B-Q5_K_M.gguf(2.69 GiB, llama-bench reports qwen3 4B Q5_K - Medium) andQwen3-4B-IQ4_XS.gguf(2.11 GiB, IQ4_XS - 4.25 bpw), downloaded for this measurement; both quants qualify under the deleted header's predicate (F32 output, Q5_K/IQ4_XS weight, 256 <= ne[0] <= 32768, 1-16 tokens).Arms, per model: default, and
GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32, everything else identical (-dev HRX0 -ngl 99 -t 4,~/lb/thermal-run.sh,systemd-run --user --scopewith MemoryMax).Numerics (logits level):
llama-perplexityon one fixed 69 KB text,-c 512 --chunks 4, same arms - PPL = 1.0042 ± 0.00072 in both, identical to four decimals, so the route computes the same thing.Device use is confirmed, not assumed: the bench table shows
dev HRX0, ngl 99and the runs reach pp512 ≈ 3000 tok/s on a 4B, against 89.76 tok/s measured for the same model on-ngl 0(CPU) - two orders of magnitude apart, so this is not a silent CPU run.IREE_HAL_AMDGPU_LIBHSA_PATHwas set for every arm (TheRocklibhsa-runtime64.so.1), without which HRX registers no device at all.What this means for the row:
f2099e9b's "substantial numerical performance regression" is no longer observable on AMD's current kernels - at worst it is noise, and on Q5_K it is a small win at decode with identical output. Recommendation: retire theknown_fail("#450")marker and replace the case's "must never be selected" assertion with "may be selected, and its output matches the non-symmetric path", or simply drop the case; the decision is the owner's, and #450 can close as re-measured with no regression.Caveats on the record: one model family (Qwen3-4B), two quants, one box; tg64 is decode with short prompts, which is what the route's 1-16 token qualification covers. Slot released (
~/GPU-TOKENremoved), no processes left, no amdgpu errors/timeouts in dmesg, Tctl back to 38 °C.bong-water-water-bong commentedon Oct 18, 2026
Real-text PPL is in, the KLD arm did not fit the slot - recording both so the next window is planned, not guessed.
Arm A (route ON), held-out natural text: Gutenberg Pride and Prejudice + Moby Dick concatenated (1.97 M chars),
llama-perplexity -c 512 --chunks 20, Qwen3-4B-Q5_K_M on HRX0 (-dev HRX0 -ngl 99 -t 4, thermal-run):That is a real number, unlike the 1.0042 the repetitive 69 KB text gave - the earlier equality test was indeed weak.
Arm B (route OFF) + KLD: started against arm A's logits and reported
for the same 20 chunks, which alone exceeds the 20-minute window I had, so it was stopped at ~3.5 minutes and the three 64-token greedy completions were never reached. Nothing was concluded from this run beyond arm A's PPL.
Two ways to finish, for the next slot:
--chunks 5for the KLD (2560 held-out tokens, ~5 min at the measured pass rate) plus the 3 x 64-token greedy completions (~3 min). Enough to answer "same model on both arms".--chunks 20, ~20 min of KLD plus ~3 min of completions.Slot released: no processes left,
~/GPU-TOKENremoved, no amdgpu errors, Tctl 37 C.bong-water-water-bong commentedon Oct 18, 2026
KLD + greedy check done on hardware, and it closes the last gap: both arms are the same model.
Qwen3-4B-Q5_K_M on gfx1151,
-dev HRX0 -ngl 99 -t 4, route on vs off (GGML_HRX_DISABLE_DISPATCH=common.rmsnorm_binary_symmetric_i4_k32), held-out Gutenberg text (Pride and Prejudice + Moby Dick, 1.97 M chars),-c 512, 5 chunks:Three 64-token greedy completions at temperature 0 (explanatory, code, narrative prompts), one server per arm: byte-identical, 931 bytes each.
For the record, arm A at 20 chunks of the same text gave PPL 16.3451 +- 0.74733, so the earlier 1.0042 came from the repetitive 69 KB file, not from the model.
With the earlier throughput numbers (pp512 3033 vs 2991 tok/s, pp2048 2648 vs 2613, tg64 65.65 vs 63.61, all in favour of the route), the route is established as equal-or-better and numerically identical, so the
known_fail("#450")guard is retired in 1bit-MONSTER/llama.cpp#109: the case now asserts that a qualifying graph selectsggml_rmsnorm_binary_symmetric_i4_k32- and that a graph with no Q5_K/IQ4_XS consumer does not - instead of asserting it is never selected. Its old q8-publication assertions move out to #449's guards, where they belong.Host suite on the current base (
29dd8820): 66 cases, 47 passed, 13 failed, 6 known-fail, 0 unexpected-pass. #450 can close when #109 merges.bong-water-water-bong commentedon Oct 18, 2026
Closed: the symmetric-i4 consumer route is not a regression on the current kernels. Measured on gfx1151 (Qwen3-4B Q5_K_M / IQ4_XS, HRX0): throughput equal or better with the route (Q5_K tg64 65.65 vs 63.61), PPL 16.35 on held-out text, mean KLD 0.000000 (max 0.000056) between route on/off, 3 greedy completions byte-identical. The known-fail guard is retired in 1bit-MONSTER/llama.cpp#109.