Repository navigation
ggml-hrx: update upstream and enable Flint through 32K context - #158
Draft
AaronStGeorge wants to merge 23 commits into
Draft
AaronStGeorge wants to merge 23 commits into
AaronStGeorge wants to merge 23 commits into
Conversation
Proposal for a new AMD/ROCm backend. See RFC discusion: `Insert discussion link`
Co-authored-by: Aaryaman Vasishta <aaryaman.vasishta@amd.com> Co-authored-by: Rob Suderman <rob.suderman@gmail.com> Assisted-by: Codex
Loom kernels failed to parse after HRX replaced the mul predicate with multiple_of. Migrate the kernel contracts and require d8da4382863fef59d2deb914a287e5cda9f3f349, which introduced the canonical predicate spelling. Relevant hrx PR: ROCm/hrx-system#1372
## Overview Qwen3 MoE microbatches that request no output logits produce empty expert views with nonzero slot offsets. HRX rejected these valid GGML views because their offsets exceed the zero-byte source extent, preventing graph matching during prefill and decode-like perplexity evaluation. Skip byte-span bounds checks for empty tensors while preserving storage-root and alias-offset validation. Add a regression test covering all eight empty expert views, cache reuse, and rejection of a changed alias offset. ## Validation - Final source builds successfully; all 7 focused graph matching, cache UID, and zero-output tests pass. - Earlier end-to-end validation on gfx1100: Qwen3-30B-A3B Q4_K_M perplexity with `-c 512 --chunks 2 -b 512 -ub 1` completes with HRX PPL 6.7431 versus Vulkan 6.7573. The original code fails graph matching. - Server boundary checks with `-ub 256`: original backend passes 256 prompt tokens and fails at 257; the repair completes 256, 257, and 513-token requests. These are execution checks, not generated-text quality checks. - gfx1151 is covered by host-side graph construction, not GPU execution. The reported 262144-token all-GPU context could not be tested on the local 24 GiB GPU because its KV cache did not fit. No HRX dependency update is required. ## Requirements - [ ] I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md). - AI usage disclosure: YES. Codex assisted with investigation, implementation, tests, explanatory comments, and PR preparation under human review.
HRX per-axis footprint verification rejects aligned matmul, attention, and normalization packets whose remainder bounds omit the packet width. The stricter view contract prevents flattening from hiding axis overruns. State the existing two- and four-element packet bounds and the guarded query-head bounds before accesses are linearized. Relevant hrx PR: ROCm/hrx-system#1512
## Motivation Qwen3 MoE decode-like perplexity is wrong on gfx1151 while prefill-like is right, and the same binaries are right for both on gfx1100. In https://github.com/ROCm/ggml-staging-automation/actions/runs/38275322264, Qwen3-30B-A3B-Instruct-2507 Q4_K_M measured HRX decode-like PPL 20503 against Vulkan 6.7573 on gfx1151 (`-c 512 --chunks 2 -b 512 -ub 1`); on gfx1100 the CI binaries give 6.7431. The fused decode router (`qwen3_moe_router_projection_top8_fused_decode_f32`) hard-coded four experts per wave when sizing its launch, but the gfx1151 projection template handles two experts per wave. On gfx1151 the kernel therefore launched half the expert tiles it needed. The launch now reads experts-per-wave from the same template the projection uses, so the workgroup count follows the target's tile width. Every other wave64 target resolves to the previous constant of four. This path only runs when the token count is exactly one, so it affects single-token decode and nothing else. It became reachable on 2026-10-12 when #140 let the graph matcher accept these microbatches; before that the regime did not execute on HRX at all. ## Testing **gfx1151, staging CI.** A perplexity-only run of Qwen3-30B-A3B-Instruct-2507 alone, with llama.cpp pinned at this PR's head and hrx-system at the current staging pin 7696b5b3c02a, and the model's `perplexity: fail` relaxation removed so a pass is a strict PASS: https://github.com/ROCm/ggml-staging-automation/actions/runs/38580145237 (branch `ppl/qwen3-30b-router-fix-gfx1151`, dispatched with `skip_benchmarks=["lemonade-benchmark"]`, `model_tier=full`). Result: strict PASS. | Regime | HRX gfx1151, before (run 38275322264) | HRX gfx1151, this PR | Vulkan | | --- | ---: | ---: | ---: | | decode-like, `-ub 1`, 2 chunks | 20503.1356 | 6.7431 | 6.7573 | | prefill-like, `-ub 512`, 32 chunks | 6.0732 | 6.0732 | 6.0628 | **gfx1100, regression.** Built against hrx-system 7696b5b3c02a. Qwen3-30B-A3B-Instruct-2507 Q4_K_M on wiki.test.raw, RX 7900 XTX, before and after this commit, four runs each, identical per-chunk values: | Regime | Before | After | Vulkan | | --- | ---: | ---: | ---: | | decode-like, `-ub 1`, 2 chunks | 6.7431 | 6.7431 | 6.7573 | | prefill-like, `-ub 512`, 32 chunks | 6.0732 | 6.0732 | 6.0760 | gfx1100 is unchanged by construction: the default template returns the old constant, and the measured values agree to the last digit. The 182 kernel specializations from the CI failure logs also compile with this corpus against hrx-system main (2a13b0bc) via `loom-check`. No HRX dependency changes; the minimum HRX commit is unchanged. ## Requirements - [x] I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md). - AI usage disclosure: YES. Codex produced the fix; Claude Code performed the gfx1100 regression validation, set up the gfx1151 CI run, and prepared this PR under human review.
HRX commit 771711d49befd397f10c3c6deb66f1c5c89e5042 removed the Loom link-index builder create/release API, breaking the JIT build. Use allocate/free and require that commit because builders are exclusively owned, while create/retain/release is reserved for reference-counted objects. Relevant hrx PR: ROCm/hrx-system#1597
Load use optimization for scales etc that are re-used when dequantizing several 16x16 blocks destined for WMMA ops. Specifically, dense and paired gate/up tiled matmul currently reload Q5_K/IQ4_XS block metadata for each K32 group. This change loads it once per K256 block and retains it across the eight groups, explicitly inside each matmul kernel. ## Generic preload & dequantize helper functions? This was tried, it ends up being pretty ugly. Each kernel really needs explicitly own what gets retained and for how long to reach peak performance and a generic solution pulls against that. Preloading exactly where and exactly what the kernel wants to when it wants to requires an `if/else` block for quantization types can get a little unwieldy but it all gets stripped out by the compiler. ## Validation - Current inline implementation commit: `4e6fb071096d27af05298157d7d8469cdd329928`. - Base: `hrx-graph-develop-v2` at `dbe1d6e1a27ea78e432f3da84778ecb591f196a6`. - Local HRX SDK: `771711d49befd397f10c3c6deb66f1c5c89e5042`; pinned RX 7900 XTX (`gfx1100`). - Current inline implementation rebuild passed; all 17 CPU-reference cases passed normally and with Loom access + HAL ASAN instrumentation. - All 19 exact captured configurations (9 dense including fallbacks, 10 paired) compile to byte-identical native `.text` compared with the benchmarked template candidates. No performance rerun is claimed for this source-only refactor; the native-code comparison preserves the link to the recorded timings. - Separate candidates previously passed 36 dense/16 gate native compile checks and 18 dense/20 gate exact captured-config checks. No spills; captured Qwen fallback machine code unchanged. - All 30 baseline/dense/gate model perplexity runs matched printed estimates (five models, prefill and decode). These are independent-candidate results, not a combined performance claim. ## Direct kernel measurements Actual Llama 3.2 3B production entry points/configurations, 512 tokens, original model weights and identical deterministic nonzero synthetic activations. Ten alternating before/after process pairs. Batch-normalized host queue-completion time per dispatch, batch16, binding ring8; compilation and warmup excluded. Speedup is before/after throughput; 95% intervals use paired log ratios. | Kernel | Before | After | Speedup (95% CI) | |---|---:|---:|---:| | dense_q5 | 956.7 us | 868.0 us | +10.22% [+9.80, +10.65] | | gate_q5 | 2303.0 us | 1928.9 us | +19.39% [+18.02, +20.79] | | gate_iq4 | 2173.8 us | 1879.6 us | +15.65% [+15.14, +16.16] | Dense shape K8192/N3072; paired shapes K3072/N8192. These differ in shape and epilogue, so this is not an isolation of the cause of their different gains. ## Independent model measurements Original UD-Q4_K_XL models, pp512, ten alternating process pairs with five repetitions each, same GPU/toolchain. Per-model 95% intervals are exploratory and unadjusted. No outliers removed; pilots excluded. Columns compare each change independently against the baseline; gains must not be added together. | Model | Dense change (95% CI) | Gate/up change (95% CI) | |---|---:|---:| | llama-3.2-1b-instruct | +1.14% [+0.14, +2.16] | +7.63% [+6.28, +9.00] | | llama-3.2-3b-instruct | +0.44% [-0.03, +0.92] | +5.58% [+5.19, +5.98] | | qwen3.5-0.8b | -0.21% [-0.83, +0.42] | +1.36% [+0.90, +1.81] | | qwen3.5-2b | +0.23% [-0.36, +0.84] | +3.81% [+3.43, +4.20] | | smollm3-3b-128k | +0.45% [+0.14, +0.75] | +4.46% [+4.14, +4.77] | Every model exercises five affected Q5 dense residual dispatches and nine or ten affected paired Q5/IQ4 dispatches. Decode does not exercise these tiled reuse paths. Dense IQ4 has focused numerical/compiler coverage, not a model-level speedup claim. No broad split-count sweep or other-GPU performance claim. ## ggml-staging-automation Current [ggml-staging-automation CI run 38642793413](https://github.com/ROCm/ggml-staging-automation/actions/runs/38642793413) is queued, with smoke benchmarks and perplexity enabled (`skip_benchmarks=[]`). Staging commit `ce208c65eda6a1ab75d978ed372fb90840d7033f` pins this PR's inline revision `4e6fb0710` and retains HRX `dbda3434868da87540e7e093314c3f4dca867930`. Staging uses a different SDK from the local timings, and no passing staging result is claimed yet. The [earlier run 38635190536](https://github.com/ROCm/ggml-staging-automation/actions/runs/38635190536) targeted the superseded template commit and was canceled. Local raw evidence is retained under `build/decoder-reuse-head-refresh/`: `timing/paired-summary.json`, `timing/evidence.txt`, `kernel-timing/kernel-summary.json`, `combined/correctness-{normal,access}.{json,log}`, and each arm's native reports. These workspace paths are provenance notes, not publicly accessible artifact links. The staging run supplies shareable CI evidence.
## Overview Retain reusable IQ3_S metadata and Q5 headers/payloads across repeated work in HRX kernels, following the format-specific preload/fallback structure introduced in #148. This reduces repeated loads and decoding while preserving generic fallbacks and numerical operation order. The IQ3_S metadata preload uses byte-aligned loads: the metadata at offset 106 in a 110-byte block cannot promise four-byte alignment. This fixes an alignment error found during integration. ## Scope The hidden-2048 RMSNorm retained-input specialization was removed in `ba5cf74512097303b19f5518760ce9a78370fdb7`; `rmsnorm_binary_f32_f16.loom` exactly matches the PR baseline. Generic caller-parameterized RMSNorm unrolling is a separate workstream. This PR now changes only IQ3_S metadata and Q5_K header/payload reuse. ## Reproduced local benchmark results Compared exact revised head `ba5cf74512097303b19f5518760ce9a78370fdb7` against `43b0ec2c41cc671cc0add797a4b8c0aae87b2f1a` (includes #148), on Radeon RX 7900 XTX (gfx1100), with the original pinned HRX `61c152ac4b40dd758803a2cbcd237184a8dfb669` SDK and ROCm/toolchain. The original baseline binary/SDK hashes were verified; the revised candidate was rebuilt with the original build options. All 27 original manifest model files were SHA256-verified. - Clean run: October 21, 2026, 00:08:27–01:14:31 UTC; 756/756 fresh-process calls succeeded. 540 scored calls and 216 control calls cover 54 model/mode comparisons. - Same original protocol: llama-bench pp512/tg128, 8 threads, batch 512, microbatch 512, full GPU offload, 5 inner repetitions per process. Default warmup enabled: a full 512-token prompt pass for prefill and one generated token for decode, outside measured repetitions. Five balanced ABBA blocks give 10 disjoint process pairs per model, with 4 baseline control pairs. Original deterministic model order. - Seven decode gains (+0.73% to +1.71%) survive simultaneous 95% intervals across all 54 comparisons; Holm correction agrees. All seven are positive in both AB/BA order subsets and five-block sensitivity intervals. No prefill change or regression survives the same threshold. Three additional comparisons are nominal improvements only; all remaining results are inconclusive. Inconclusive does not establish equivalence. - Same-arm adjacent control variation reached 2.67% for prefill and 2.74% for decode; maximum absolute before/after baseline drift was 1.36% and 0.90%, respectively. Effects below use paired log ratios, not comparison of unpaired aggregate means. Inner repeats are not independent observations. - An earlier 241-call attempt (29 load failures) overlapped confirmed GPU activity from an independent MTP correctness investigation. It was excluded **in full**, with raw results retained. The full original schedule was restarted after both GPU workstreams confirmed release. All reported results come from the single clean restart; no individual scores were selected and no outliers removed. The shared exclusive GPU lock was held throughout scoring. The same seven models improved in the earlier combined-patch measurement, with effects of similar magnitude. This is a reproduction of the reduced changeset's gains; separate-session differences do not isolate RMSNorm's contribution. <details> <summary>Full 27-model prefill/decode results for revised head (simultaneous 95% confidence intervals)</summary> Rates are arithmetic means of process means. Percentage changes use paired log ratios; intervals use Bonferroni across 54 comparisons. AB/BA order, five-block sensitivity, controls, drift and Holm diagnostics were retained alongside raw measurements. | Model | Mode | Baseline tok/s | Candidate tok/s | Paired delta | Simultaneous 95% CI | |---|---|---:|---:|---:|---:| | qwen3-30b-a3b-instruct-2507 | pp512 | 4661.02 | 4662.73 | +0.037% | [-0.288, +0.362]% | | qwen3-30b-a3b-instruct-2507 | tg128 | 198.89 | 198.44 | -0.226% | [-1.078, +0.633]% | | gemma-4-12b | pp512 | 2675.80 | 2675.59 | -0.008% | [-0.143, +0.128]% | | gemma-4-12b | tg128 | 62.92 | 64.00 | +1.712% | [+1.453, +1.971]% | | llama-3.1-8b | pp512 | 5039.77 | 5035.94 | -0.076% | [-0.312, +0.161]% | | llama-3.1-8b | tg128 | 129.11 | 129.15 | +0.038% | [-0.066, +0.142]% | | minicpm5-1b | pp512 | 29712.67 | 29596.55 | -0.393% | [-1.567, +0.795]% | | minicpm5-1b | tg128 | 537.07 | 537.01 | -0.011% | [-0.702, +0.684]% | | minicpm3-4b | pp512 | 5313.59 | 5300.23 | -0.256% | [-2.739, +2.290]% | | minicpm3-4b | tg128 | 67.34 | 67.32 | -0.023% | [-0.665, +0.623]% | | minicpm-v-4.6 | pp512 | 26316.86 | 26365.37 | +0.184% | [-0.742, +1.119]% | | minicpm-v-4.6 | tg128 | 408.73 | 409.13 | +0.100% | [-0.174, +0.374]% | | minicpm-v-4.6-thinking | pp512 | 26425.27 | 26560.66 | +0.514% | [-0.466, +1.503]% | | minicpm-v-4.6-thinking | tg128 | 409.87 | 409.85 | -0.005% | [-0.290, +0.280]% | | qwen3-0.6b | pp512 | 30620.85 | 30593.63 | -0.088% | [-2.153, +2.021]% | | qwen3-0.6b | tg128 | 460.46 | 460.33 | -0.027% | [-0.602, +0.552]% | | gemma-3-270m-it | pp512 | 52961.82 | 53586.24 | +1.181% | [-0.921, +3.328]% | | gemma-3-270m-it | tg128 | 456.47 | 453.33 | -0.663% | [-4.186, +2.990]% | | qwen3-1.7b | pp512 | 12439.81 | 12377.94 | -0.495% | [-2.034, +1.068]% | | qwen3-1.7b | tg128 | 317.14 | 316.50 | -0.204% | [-0.766, +0.362]% | | qwen3-4b | pp512 | 4723.80 | 4721.72 | -0.045% | [-1.007, +0.927]% | | qwen3-4b | tg128 | 183.51 | 183.67 | +0.086% | [-0.311, +0.484]% | | qwen3-4b-instruct-2507 | pp512 | 8508.20 | 8500.54 | -0.090% | [-0.814, +0.641]% | | qwen3-4b-instruct-2507 | tg128 | 183.73 | 183.65 | -0.047% | [-0.728, +0.638]% | | bonsai-8b | pp512 | 2571.80 | 2572.18 | +0.015% | [-0.363, +0.394]% | | bonsai-8b | tg128 | 158.08 | 158.04 | -0.025% | [-0.345, +0.297]% | | bonsai-4b | pp512 | 5004.24 | 5021.44 | +0.348% | [-1.306, +2.029]% | | bonsai-4b | tg128 | 222.83 | 222.58 | -0.115% | [-1.236, +1.018]% | | bonsai-1.7b | pp512 | 12405.81 | 12413.46 | +0.061% | [-0.977, +1.111]% | | bonsai-1.7b | tg128 | 358.11 | 359.09 | +0.270% | [-1.203, +1.765]% | | phi-4-mini-instruct | pp512 | 9842.45 | 9851.63 | +0.093% | [-0.217, +0.405]% | | phi-4-mini-instruct | tg128 | 189.14 | 191.45 | +1.223% | [+0.922, +1.525]% | | lfm2-1.2b | pp512 | 28715.64 | 28738.45 | +0.081% | [-1.036, +1.212]% | | lfm2-1.2b | tg128 | 523.44 | 523.23 | -0.042% | [-0.239, +0.156]% | | lfm2.5-1.2b-instruct | pp512 | 28544.30 | 28540.05 | -0.018% | [-4.134, +4.274]% | | lfm2.5-1.2b-instruct | tg128 | 523.25 | 523.56 | +0.058% | [-0.323, +0.440]% | | promptbridge-0.6b-alpha | pp512 | 34693.11 | 34681.71 | -0.041% | [-2.906, +2.907]% | | promptbridge-0.6b-alpha | tg128 | 442.55 | 443.57 | +0.230% | [-0.228, +0.691]% | | jan-nano-128k | pp512 | 8523.09 | 8522.00 | -0.013% | [-0.503, +0.479]% | | jan-nano-128k | tg128 | 184.08 | 184.18 | +0.052% | [-0.692, +0.802]% | | jan-v1-4b | pp512 | 8529.04 | 8543.25 | +0.167% | [-0.329, +0.665]% | | jan-v1-4b | tg128 | 183.89 | 184.26 | +0.198% | [-0.374, +0.772]% | | llama-3.2-1b-instruct | pp512 | 19935.02 | 19902.70 | -0.166% | [-2.545, +2.272]% | | llama-3.2-1b-instruct | tg128 | 470.43 | 475.08 | +0.990% | [+0.764, +1.216]% | | llama-3.2-3b-instruct | pp512 | 9236.47 | 9238.47 | +0.022% | [-0.387, +0.432]% | | llama-3.2-3b-instruct | tg128 | 227.63 | 229.28 | +0.725% | [+0.501, +0.950]% | | smollm3-3b-128k | pp512 | 8470.96 | 8477.20 | +0.073% | [-0.324, +0.472]% | | smollm3-3b-128k | tg128 | 217.14 | 218.73 | +0.730% | [+0.558, +0.902]% | | ministral-3-3b-instruct-2512 | pp512 | 11015.70 | 11014.01 | -0.015% | [-0.472, +0.444]% | | ministral-3-3b-instruct-2512 | tg128 | 233.78 | 233.97 | +0.080% | [-0.492, +0.654]% | | qwen3.5-0.8b | pp512 | 24610.85 | 24617.16 | +0.027% | [-1.608, +1.690]% | | qwen3.5-0.8b | tg128 | 387.55 | 390.49 | +0.757% | [+0.400, +1.116]% | | qwen3.5-2b | pp512 | 13809.41 | 13829.31 | +0.144% | [-0.301, +0.591]% | | qwen3.5-2b | tg128 | 263.97 | 267.56 | +1.360% | [+1.003, +1.718]% | </details> ## Correctness and CI provenance For the revised head, the restored RMSNorm kernel was linked and compiled for gfx1100 (H2048,multiply,T9), both benchmark arms passed separate unscored loader/device smoke checks, and the diff passed whitespace checks. An initial smoke check failed to locate libomp because the historical build used the old workspace path; adding the original ROCm llvm/lib directory to both arms' library search path resolved it. No binary or SDK substitution was made. No new full-model perplexity campaign or ggml-staging-automation run was requested or performed. The following evidence is historical and applies to combined commit `01d0a3ad5054231298806d547eae33c283336a70`, which included RMSNorm retention: - [Full ggml-staging-automation CI — successful attempt 3](https://github.com/ROCm/ggml-staging-automation/actions/runs/38806373681/attempts/3). - All 27 expected-pass manifest models passed unchanged local CI-equivalent perplexity on both arms: 108 HRX/Vulkan comparisons, 216 measurements; maximum ratio 1.00655 against threshold 1.10. Prefill c512/b512/ub512/32chunks; decode c512/b512/ub1/2chunks. Expected-fail/report-only models excluded as documented in the frozen manifest. - Captured production/tail checks, alignment fixtures and 36/36 fallback checks passed. Focused checks 52/56 passed; four failures were the same inherited IQ3_S K2048/N256/T32 synthetic mismatch on both arms in normal/access modes, with all 8192 outputs bit-identical. The discrepancy remains unresolved; tolerances were not relaxed. ## Requirements - Contributing guidelines reviewed; draft submitted for human review and ownership of the changes. - AI usage disclosure: YES. Codex assisted with investigation, implementation, testing, statistical analysis, and drafting this PR under the contributor's direction.
## Overview Fix HRX inference for Qwen3.5-35B-A3B-MTP-GGUF, which faults during multi-token prefill and produces decode perplexity around 19.8 million in [the failing run](https://github.com/ROCm/ggml-staging-automation/actions/runs/38806373681). The router packs expert partitions using the legacy 7-bit expert layout, while the generic expert matmul reads a 9-bit layout. This turns valid descriptors into out-of-bounds row offsets. The router also launches only 128 partition-builder threads for a model with 256 experts. Use a shared 9/6/5-bit expert/partition/row-count layout across generic and specialized consumers, including the fused partition producer, and launch one partition-builder thread per supported expert. Update compiler fixtures and CPU references together. Add end-to-end router-to-matmul regressions for 128, 256, and 512 experts with 1, 2, and 512 tokens, including high expert IDs, multirow partitions, and graph replay. ## Validation [Single-model, perplexity-only upstream gfx1151 run](https://github.com/ROCm/ggml-staging-automation/actions/runs/39495823554): **PASS**, with the model's expected-failure exemption removed. Artifact metadata confirms this fix commit and HRX runtime `4341b71ab812e618b92b016176528363718e6667`. | Regime | HRX perplexity | Vulkan perplexity | | --- | ---: | ---: | | Prefill: context/batch/ubatch 512, 32 chunks | 5.4237 | 5.4282 | | Decode: context/batch 512, ubatch 1, 2 chunks | 6.0322 | 6.0373 | Local gfx1100 validation: - All 9 new regression cases pass; the 256-expert one-token regression fails with the original backend. - All 7 existing specialized MoE/router checks and 3 focused dispatch/fused-partition checks pass. - JIT compile checks: 5/5 pass. - Full backend suite: 95/99 pass. The four failures reproduce on unmodified upstream: `rope_set_rows_dispatch`, `generic_swiglu_k16_publication_edges`, `tiled_matmul_alternate_publish`, and `rmsnorm_binary_k16_output`. - Full JIT suite including device materialization: 5/6 pass. The concat topology-range compile failure also reproduces on unmodified upstream. The single-model automation manifest is on a separate validation branch and is not part of this PR. ## Requirements - [ ] I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md). Contributor confirmation pending. - AI usage disclosure: YES. Codex assisted with root-cause investigation, implementation, regression coverage, validation, and this draft PR.
## Motivation Fixes required to bump [`hrx-graph-develop-v2`](https://github.com/AMD-Ecosystem/llama.cpp/tree/hrx-graph-develop-v2) up to hrx-system: ROCm/hrx-system@5f4dfc8. Decode attention writes F32 results into global memory, then Q8 packing reads values written by other lanes. The intervening barrier fenced workgroup memory only. Fence global memory before packing in both reduction paths to prevent stale values from producing nonfinite Q8 scales and NaN perplexity. This appears to be a pre-existing llama.cpp synchronization bug, not a newly introduced HRX regression. The previous green PR105 package uses the same llama.cpp revision and older HRX `4341b71`; it also reproduces the failure locally. Newer HRX may have made the race more common by changing execution timing or generated code, but the controlled local comparison did not establish that: older and newer packages both produced NaNs in all ten alternating fresh-process repetitions. A second local comparison used the September 29 package from https://github.com/ROCm/ggml-staging-automation/actions/runs/36638317205, whose saved report explicitly passed this exact model and checksum with decode PPL 10.0045. That package (HRX `32b86d4`, llama.cpp `2ba2ddc`) and the current unfixed package each failed 10/10 alternating repetitions locally. Its llama.cpp revision differs, so this is a historical package comparison rather than an HRX-only comparison. Both old reduction paths already contain the same workgroup-only pre-packing barriers. Neither local comparison provides evidence of an increased failure rate with newer HRX. No introducing HRX commit is attributed. The HRX PR referenced by the fix commit documents the barrier API, not the introduction of this bug. ### Breaking changes | Breaking hrx-system PR | Fix | | --- | --- | | None identified; pre-existing llama.cpp race | AaronStGeorge@103a47d | ## Testing Tested in `ggml-staging-automation` full all-model CI with llama.cpp `103a47dd80e7` and hrx-system `5f4dfc8731a1`. - Bump PR: ROCm/ggml-staging-automation#108 - Full all-model CI: passed on strixhalo-10, https://github.com/ROCm/ggml-staging-automation/actions/runs/39701697708, staging head `1bad5e0d6acb`. The release benchmark and security checks passed. Llama-3.2-1B-Instruct decode PPL was 10.0408 +/- 1.32270; prefill PPL was 14.4414 +/- 0.45819. Both numerical checks and Vulkan references passed. The staging workflow supplies missing GPU device groups on canary runners and launches benchmarks with refreshed membership; no model expectations were changed. - Focused numerical validation: https://github.com/ROCm/ggml-staging-automation/actions/runs/39519720454. On DT241, DT243, and strixhalo-10, the original package failed 5/5 repetitions per host and the fix passed 5/5. Fixed decode PPL was 10.0408 +/- 1.32270; prefill PPL was 14.4414 +/- 0.45819. Vulkan reference checks passed. The run was red because DT245 could not initialize either package; it produced no model measurements. - Local CPU decode reference: 10.0388 +/- 1.32349. All 37 existing attention tests passed. - Compatibility floor remains `771711d49befd397f10c3c6deb66f1c5c89e5042`. Rebuilt HRX, Loom, the backend, and embedded kernel bytecode at that floor and ran the same fixed decode command successfully: PPL 10.0408 +/- 1.32270. - Reproduction command: `ROCR_VISIBLE_DEVICES=0 llama-perplexity -m Llama-3.2-1B-Instruct-UD-Q4_K_XL.gguf -f wiki.test.raw -c 512 --chunks 2 -b 512 -ub 1 --device HRX0` on gfx1151. Model and corpus are the pinned, checksum-verified inputs in the staging manifest. Device initialization hangs are separate from this numerical fix. DT239's captured stacks block in ROCr `BlitKernel::SubmitLinearCopyCommand` during HRX device-library initialization with both packages, even with Vulkan disabled, before any model executes. The underlying cause of that runtime hang is unresolved. Historical endpoint search for a llama.cpp frequency bisect: the September 7 package cannot execute this model's Q5_K matrix multiplication; September 16 produces PPL 41875.0725; September 23 produces NaNs. None is a locally correct good endpoint. Source history places the workgroup-only pre-packing barriers in `f43c5b4d9` (August 28), the initial Llama support change. This is source-history attribution of the missing fence, not a completed runtime bisect or proof of a later frequency increase.
## Overview The packaged HRX suites fail 16 cases at the current backend pin. This fixes concat launch-bound compilation and updates stale dispatch expectations while retaining numerical comparisons and negative coverage. - Reuse the concat bound fix and dynamic-reuse regression from #155; the existing element limit guarantees at most 4,194,304 workgroups. - Check the current RoPE capacity class, aligned publication kernels, IQ2_S specialization, and low-token RMSNorm selection. - Explicitly verify rejection of in-place RMSNorm, matching the kernels' no-alias contract. - Verify supported Gemma Add-before-Add fusion parameters and add CPU-reference coverage. ## Additional information Validated on gfx1100 with HRX `5f4dfc8731a1b37fdd380c566c7babf7ae3548cd`: **436 passed, 0 failed** across buffer (5), backend (99), JIT (6), and ops (326). CTest wall time was 62.09 seconds with two concurrent suites and internal worker counts 1/4/2/4. Three standard backend CONCAT CPU-comparison cases also passed. Numerical tolerances were unchanged. gfx1151 CI qualification **passed: 436 cases, 0 failures, 59.03 seconds CTest wall time**: [workflow run 40393012701](https://github.com/ROCm/ggml-staging-automation/actions/runs/40393012701), testing this PR's exact backend commit `0230f1b29ee2d009602a7b2c6cbce64f1b1290d2` through the packaged test workflow. The separate packaging integration is ROCm/ggml-staging-automation#110. This PR overlaps only the concat fix/regression in #155; it does not include that PR's recurrent-state changes. ## Requirements - [ ] I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md) (contributor confirmation). - AI usage disclosure: YES. Codex investigated the 16 failing cases through sequential subagents, reviewed the changes, and ran focused and full-suite validation. The user requested this submission.
## Overview Normalize the HRX suite source, target, executable, and CTest names to `hrx-*-test`. Label the four suites `hrx` and put their complete invocation in CTest: workers 1/4/2/4, full case selection and output, a 900-second timeout, and failure on missing-device skips. Suite concurrency stays caller-controlled. Document discovery through `ctest -L '^hrx$' --show-only=json-v1`: test names identify build targets before building, and commands/properties supply packaging and execution metadata afterward. Numerical assertions and case registrations are unchanged. ## Validation - Built all four targets in a separate Release build against HRX `5f4dfc8731a1b37fdd380c566c7babf7ae3548cd`. - Full CTest run with `--parallel 2 --verbose` on gfx1100: all four suites passed (buffer 5, backend 99, Loom JIT 6, ops 326 cases). - Verified JSON discovery contains exactly four renamed suites with the expected executable paths, arguments, timeout, and failure regex. - Missing-device buffer run fails through the registered failure regex (CTest exit 8). - Verified all case registrations and numerical assertions are unchanged; `git diff --check` passes. ## Requirements - Contributing guidelines reviewed; this draft targets the AMD-Ecosystem development fork under explicit user authorization. - AI usage disclosure: YES. Codex implemented the naming/CTest changes and ran validation. Human review remains required before merge.
Use the validated tree from the HRX rebase onto upstream. Preserve both independent normalization tests in tests/test-backend-ops.cpp. Assisted-by: Codex
Support learned attention sinks, sigmoid MoE routing, strided expert IDs, scalar SET_ROWS and CLAMP. Update upstream and HRX API integration and retire cached graph programs safely to bound long-context staging memory. Add operation, routing and cache regressions. Assisted-by: Codex
Fix silent zero weights for IQ2_XXS, IQ2_XS and IQ3_XXS in expert and gather paths. Add CPU-reference regressions across mixed Flint tensor formats. Tracks lemonade-ws-ice.1. Assisted-by: Codex
Add the Q2_K format mapping, block decoding and numerical/dispatch regressions. Full-offload Flint Q2_K perplexity agrees with Vulkan within 0.12 percent. Tracks lemonade-ws-ice.2. Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Import and compare n_offs, rotate only the selected channel range, and copy the unchanged prefix and suffix. Keep unsupported offset fusions on the generic path. Cover normal/NeoX RoPE, scalar and per-token scaling, and MiniCPM3 prefill/decode shapes against CPU. Bead: lemonade-ws-aio.2.3 Assisted-by: Codex
Skip inactive branch storage and scheduling, invalidate cached programs when branch selection changes, and preserve compute masks in graph snapshots. Add host and device regressions for inactive SET_ROWS and stable-UID selection changes. Assisted-by: Codex
Track main-command writes separately from initialization accesses when deciding host staging writeback. Synthetic attention metadata must not overwrite caller-owned inputs after a later batch has populated them. Keep aggregate accesses for uploads, materialization and hazards. Add a replay regression covering deferred input replacement, genuine outputs, in-place updates, and prepared-cache miss/hit paths. Assisted-by: Codex
Keep split-local graph fixtures restricted to their intended nodes while honoring the ggml compute flag contract. Assisted-by: Codex
AaronStGeorge
force-pushed
the
hrx-graph-develop-v2
branch
from
October 30, 2026 16:52
44ee571 to
6851a42
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The earlier HRX queue-epoch incident remains tracked separately, not claimed fixed: 100 bounded exact-package trials and both final native suites passed. An upstream report documents the same signature before this integration, but an identical cause is unproven. This PR remains draft as requested.
Follow-up fixes now included:
Recognize upstream RMS-plus-scale GDN normalization with its additive epsilon semantics. Local Qwen3.5 0.8B/2B and MiniCPM-V prefill/decode perplexities agree with native Vulkan within 0.3%; 12 focused GPU numerical cases pass.
Preserve RoPE offsets and defer unsupported specialized fusions. MiniCPM3 prefill/decode perplexity improves from 423.0835/517.4914 to 14.6335/21.9671, versus Vulkan 14.6675/21.9590.
Honor inactive graph branches and include their compute masks in cached-graph matching. This fixes uninitialized embedding gathers during the server's final four-token prompt batch. All five Flint quants now pass the exact 32-token prompt plus 256 generated tokens locally, with replay enabled and the cache bound retained. The negative-control server run faults before generation; the focused inactive-write regression also fails before the fix.
Preserve caller-owned inputs during attention metadata initialization: initializer-only writes no longer copy stale positions/indices/masks back to host memory. Main-command outputs and in-place writes still copy back. Qwen3-30B decode perplexity improves from 3076.3825 to 6.7750 versus Vulkan 6.7605; prefill remains 6.0760 versus 6.0708. A focused negative-control test demonstrates the old input overwrite and verifies both caller inputs and real outputs after the fix.
The combined source builds and passes 92 host checks plus four focused GPU checks. GPU follow-up checks used gfx1100 with HRX5e6; native CI uses gfx1151 with HRXcf1. The original full-tier baseline used the same models/settings/HRX pin, but reported different host/GPU memory capacity. Local reproductions confirm the MiniCPM3, Qwen decode, and Flint failures independently of that hardware difference. All local follow-up model checks pass with replay enabled and the 256-entry cache bound retained. The final native gfx1151 runs linked above validate these fixes at the same final llama pin and unchanged HRXcf1 revision.
Overview
Enable Flint Q3_K_M inference on HRX and fix the routing corruption and retained staging allocations that prevented correct, sustained inference. The model now completes chat, perplexity, and retrieval checks through 32,768 tokens on an RX 7900 XTX, with paired HRX/Vulkan perplexity differing by less than 1% in the tested runs.
This draft includes the required upstream update to
7aca0d2ea8d92b56911233b822db01f9fbb1ae50. Commit5c3c0db68preserves the existing fork and upstream histories with the tree validated after rebasing the HRX changes. The rebase had one conflict intests/test-backend-ops.cpp; both independent normalization tests were retained. Commit49c2b54f8contains the HRX implementation and regression tests (28 files), separately from the larger upstream update.Changes
1.28e20.SET_ROWSfor transposed KV-cache layouts and contiguous F32CLAMPwith CPU-compatible edge semantics.loom-benchmarkintegration and require HRX revision5e6af0ec1a0c1fd6c5553e7d0e6f2bc7c4930444.Vulkan needed no additional source fixes for Flint; it served as the comparison backend on the updated upstream baseline. Validation used the existing native GGUF, not a JANG/MLX conversion.
Validation
Model:
bartowski/RecursiveSI_flint-GGUF/RecursiveSI_flint-Q3_K_M.gguf, SHA2561d6d5800dbed15e60c6ca8b3af9f131a5570bae3f1b85a552632930dc033a7ea.RX 7900 XTX (gfx1100), 50/50 layers offloaded, F16 K/V, flash attention enabled, batch 512, microbatch 128, 8 CPU threads. Some HRX operations still use CPU fallback. Corpus:
wiki.test.raw, SHA256173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08.Compare backends within each row; the scoring windows differ across context lengths. The 512-token CPU control measured 25.4637.
git diff --checkpassed; packaging the commits preserves the validated source tree.Limits and review notes
Validation stops at 32K, the current HRX flash-attention KV-length limit. The advertised 262,144-token context, context shifting, and concurrent serving are unvalidated. Retrieval is one synthetic prompt per context, not a broad quality evaluation. HRX remains slower than Vulkan. Cache retirement uses an entry-count bound rather than a byte budget and can cause prepared programs to be rebuilt.
Workspace tracking:
lemonade-ws-z0zandlemonade-ws-dcn, with blocker-specific beads and isolated workspace threads. Model files, build products, and local investigation logs are excluded from this PR.Additional low-bit quant validation
Tested only the four smaller Bartowski GGUFs requested, with full layer offload on the same RX 7900 XTX. No larger quant downloads or partial-offload tests. Settings match the 512-token, 16-chunk protocol above.
Two additional blockers were isolated in bead-owned workspace threads and integrated:
lemonade-ws-ice.1: Generic GPU dequantization accepted IQ2_XXS, IQ2_XS, and IQ3_XXS but lacked their FP16/FP32 decode branches, silently yielding zero expert and gather weights. Initial HRX perplexities were 9747.4659 (XXS) and 10823.3029 (XS). Added the missing canonical codebook decoding. The mixed IQ2_XXS file also contains 11 IQ3_XXS expert tensors, so repairing IQ2 alone was insufficient. Negative controls failed before the fix; 103 targeted worker tests passed afterward.lemonade-ws-ice.2: Q2_K lacked format mapping and GPU decoding, causing graph execution failure. Added the mapping, 84-byte block decoding, and dense/expert dispatch coverage. Nine strict numerical GPU cases and 14 host dispatch cases passed. IQ2_M needed no additional source fix.The combined main-workspace build passed 79 GPU numerical tests and 14 host dispatch tests. Four-chunk integrated runs matched the repaired workers for XXS, XS, and Q2_K; the original 16-chunk Q3_K_M control remained unchanged. All four selected quants produced coherent chat answers. Vulkan required no new source changes.
These additional quants were validated at 512-token context only; the earlier 32K results apply to Q3_K_M. Inline codebook decoding prioritizes correctness and is not performance optimized. Model weights and local logs are excluded from the PR.
Requirements