update(rocm): enable fused add-RMSNorm and RoPE-append by default on ROCm

README.md

@@ -16,7 +16,7 @@ Apple Silicon is the primary target. Linux/CUDA is supported as a secondary targ
 
 The current `main` branch is v0.7.0 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the [changelog](CHANGELOG.md) for the release-by-release record.
 
-- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the samplers, the SSM update step, the Mamba1 scan, the decode-MoE kernels, xIELU, the add3 LayerNorm, and the opt-in fused add-RMSNorm and RoPE + append) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps.
+- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the samplers, the SSM update step, the Mamba1 scan, the decode-MoE kernels, xIELU, the add3 LayerNorm, and the fused add-RMSNorm and RoPE + append, which are on by default on ROCm) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps.
 - **Verified `llama-server` compatibility.** A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored.
 - **OpenAI, Anthropic, and Vertex-compatible serving.** Chat Completions, Completions, Responses, Embeddings, Reranking, Audio, and Anthropic Messages are available alongside the native `llama-server` routes. Optional Vertex AI custom-container routing is supported through the standard `AIP_*` variables.
 - **Multi-model routing and live adapters.** Router mode discovers checkpoints from the model store, a model directory, or INI presets; loads them on demand; and bounds the resident set with LRU eviction. Multiple LoRA adapters can remain unfused for per-request or live scale changes, or be fused for zero decode overhead.

benchmarks/rocm_strixhalo-gfx1151_2026-10-07_fused-default-off.csv

@@ -0,0 +1,7 @@
+model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlxcel_version,build_type,max_tokens,prompt,prompt_target_len,mlxcel_commit,mlx_commit,mlx_rocm_overlay_commit,hip_version
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,506.66,1010.54,3382.72,37.84,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,499.94,1024.12,3317.53,38.58,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,504.75,1014.37,3355.18,38.15,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,331.59,1544.08,2694.05,47.51,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,328.36,1559.28,2666.65,48.00,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,333.50,1535.22,2705.67,47.31,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333

benchmarks/rocm_strixhalo-gfx1151_2026-10-07_fused-default-on.csv

@@ -0,0 +1,7 @@
+model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlxcel_version,build_type,max_tokens,prompt,prompt_target_len,mlxcel_commit,mlx_commit,mlx_rocm_overlay_commit,hip_version
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,495.60,1033.08,3380.83,37.86,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,499.68,1024.65,3360.17,38.09,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Meta-Llama-3.1-8B-Instruct-4bit,models/mlx/Meta-Llama-3.1-8B-Instruct-4bit,512,128,504.77,1014.32,3359.47,38.10,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,318.02,1609.96,2672.40,47.90,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,319.59,1602.06,2650.37,48.30,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333
+Qwen2.5-7B-Instruct-4bit,models/mlx/Qwen2.5-7B-Instruct-4bit,512,128,312.08,1640.59,2708.01,47.27,2026-10-07,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57b59629,81ba1c6a,75915908,7.15.26333

docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md

@@ -277,6 +277,6 @@ The Metal traces were produced with the loop in `benchmarks/logit_traces/rocm_gf
 - The second acceptance criterion of #1809 (decided-position mismatch reported per model and width against Metal) is now met for all nine models: fifteen of these sixteen pairs pass and `nemotron-3-nano-30b-a3b` `w1` is inconclusive, with the same checkpoint's `w8` and `w256` pairs passing. The two `w1ctx512` rows, which cover the fused SSM update kernel on Metal, also pass.
 - Kernel against kernel for the SSM update: the `w1ctx512` rows here compare Metal's kernel with ROCm's graph. The HIP port (#2067) added kernel-against-kernel rows, with zero decided-position mismatches for both hybrids, in [rocm-ssm-update-kernel-gfx1151-2026-10-04.md](rocm-ssm-update-kernel-gfx1151-2026-10-04.md#model-logits-kernel-against-kernel).
 - Two MoE implementations in the Nemotron-H pairs: since the HIP fused MoE ports (#2065) ROCm's gate reads `moe_down_kernel_available()` and takes `fused_moe_forward`, as Metal does. New Nemotron-H traces on that path pass against these Metal traces at `w8` and `w1ctx512` with zero decided-position mismatches, in [rocm-fused-moe-gfx1151-2026-10-05.md](rocm-fused-moe-gfx1151-2026-10-05.md#model-logits).
-- The opt-in fused add-RMSNorm and RoPE + append kernels: these traces ran with both off, the default. Their HIP ports (#2063) are byte-identical to the graph they replace: Llama 3.1 and Qwen2.5 traces with them on match the traces with them off byte for byte at `w1`, `w8` and `w1ctx512`, so for the models that call them, turning them on does not change ROCm's side of these comparisons ([rocm-fused-norm-rope-gfx1151-2026-10-05.md](rocm-fused-norm-rope-gfx1151-2026-10-05.md)).
+- The fused add-RMSNorm and RoPE + append kernels: these traces ran with both off, the default at the time. They are on by default on ROCm since lablup/mlxcel#2145. Their HIP ports (#2063) are byte-identical to the graph they replace: Llama 3.1 and Qwen2.5 traces with them on match the traces with them off byte for byte at `w1`, `w8` and `w1ctx512`, so for the models that call them, turning them on does not change ROCm's side of these comparisons ([rocm-fused-norm-rope-gfx1151-2026-10-05.md](rocm-fused-norm-rope-gfx1151-2026-10-05.md)).
 - `src/models/nemotron_h.rs` prints its loading messages to stdout, which puts five non-trace lines into every Nemotron-H trace and stops `compare_logit_traces.py`. Either the loader should log to stderr or the comparison script should skip them.
 - No noise floor was measured; the first run's reasoning for not needing one still applies.

docs/benchmark_results/rocm-fused-norm-rope-gfx1151-2026-10-05.md

@@ -1,6 +1,6 @@
 # HIP ports of the fused add-RMSNorm and RoPE + append kernels on gfx1151 (2026-10-05)
 
-lablup/mlxcel#2063, part of #1814. `fused_add_rms_norm` (#905) and `fused_rope_qk_append` (#905) had Metal and CUDA ports only, so on ROCm `MLXCEL_FUSED_ADD_RMSNORM=1` and `MLXCEL_FUSED_ROPE_APPEND=1` quietly kept the graph path. Both fusions ship off on every backend (`FUSED_ADD_RMSNORM_DEFAULT` and `FUSED_ROPE_APPEND_DEFAULT` in `src/lib/mlxcel-core/src/layers.rs`), and the decode profile ([rocm-decode-profile-gfx1151-2026-09-30.md](rocm-decode-profile-gfx1151-2026-09-30.md)) put the most they could take over at 0.83 to 0.89% of Llama 3.1 decode GPU time. This page records what the ports match, what turning them on does to decode, and why the ROCm default stays off.
+lablup/mlxcel#2063, part of #1814. `fused_add_rms_norm` (#905) and `fused_rope_qk_append` (#905) had Metal and CUDA ports only, so on ROCm `MLXCEL_FUSED_ADD_RMSNORM=1` and `MLXCEL_FUSED_ROPE_APPEND=1` quietly kept the graph path. Both fusions ship off on every backend (`FUSED_ADD_RMSNORM_DEFAULT` and `FUSED_ROPE_APPEND_DEFAULT` in `src/lib/mlxcel-core/src/layers.rs`), and the decode profile ([rocm-decode-profile-gfx1151-2026-09-30.md](rocm-decode-profile-gfx1151-2026-09-30.md)) put the most they could take over at 0.83 to 0.89% of Llama 3.1 decode GPU time. This page records what the ports match, what turning them on does to decode, and the ROCm default: the measurement below recommended keeping it off, and the maintainer then turned both fusions on for ROCm (#2145, see [Decision](#decision-2145)).
 
 ## Environment
 
@@ -61,22 +61,41 @@ How to read it:
 - **Qwen3-30B-A3B** calls neither kernel (`qwen3_moe.rs` has its own block), so its row is a control: +0.3%, noise.
 - **Qwen2.5 7B** is the one model that reaches both ports. Every on arm is above off in 6 of 7 rounds, but the size does not hold still: rounds 1 to 3 gave +0.95 to +3.5 tok/s, rounds 4 to 7 (run about two hours later on the same binary) mostly -0.4 to +1 (with -2.7 for `both` in round 4 and +2.7 for `add` in round 7), and `both` is not above `add`, which it would be if both ports saved time independently. An earlier run on the branch's first commit gave the same picture (7 rounds: off 46.54, add 46.84, rope 46.95, both 47.58 tok/s). The off arm's own spread (45.75 to 46.95) is about as wide as the median gain. A gain of 1 to 2% on Qwen2.5 is plausible from the removed dispatches (the RoPE port alone stands in for three slices, three reshape and transpose pairs and two `fast_rope` calls per layer, and #2099 showed that GPU-time shares understate paths made of many small ops), but this data does not establish it.
 
-## Recommendation
+## Recommendation (#2063)
 
-Keep `FUSED_ADD_RMSNORM_DEFAULT` and `FUSED_ROPE_APPEND_DEFAULT` at `false` on ROCm, as on Metal and CUDA. The measurement does not clearly support a flip: no change on Llama 3.1, and on Qwen2.5 a small positive median whose size changes with the time of day. Because the ports are byte-identical to the graph, opting in on ROCm costs nothing in output, so `MLXCEL_FUSED_ADD_RMSNORM=1 MLXCEL_FUSED_ROPE_APPEND=1` is safe for a deployment that measures a win on its own model. What would justify a ROCm default of on: a longer interleaved run on a quiet GPU (20 or more pairs) that shows Qwen2.5-class models (no `rope_scaling`, so both ports run) gaining at least 1% with the paired differences consistently positive, and Llama-class models not losing.
+The original recommendation, superseded by the decision below: keep `FUSED_ADD_RMSNORM_DEFAULT` and `FUSED_ROPE_APPEND_DEFAULT` at `false` on ROCm, as on Metal and CUDA. The measurement does not clearly support a flip: no change on Llama 3.1, and on Qwen2.5 a small positive median whose size changes with the time of day. Because the ports are byte-identical to the graph, opting in on ROCm costs nothing in output, so `MLXCEL_FUSED_ADD_RMSNORM=1 MLXCEL_FUSED_ROPE_APPEND=1` is safe for a deployment that measures a win on its own model. What would justify a ROCm default of on: a longer interleaved run on a quiet GPU (20 or more pairs) that shows Qwen2.5-class models (no `rope_scaling`, so both ports run) gaining at least 1% with the paired differences consistently positive, and Llama-class models not losing.
+
+## Decision (#2145)
+
+The maintainer turned both fusions on by default on ROCm: the ports cost no accuracy (every on/off trace pair is byte-identical) and no median above was slower. `FUSED_ADD_RMSNORM_DEFAULT` and `FUSED_ROPE_APPEND_DEFAULT` are now `cfg!(feature = "rocm")`, so a ROCm build runs both kernels with no environment variables set, Metal and CUDA builds keep them off, and `MLXCEL_FUSED_ADD_RMSNORM=0` / `MLXCEL_FUSED_ROPE_APPEND=0` still restore the graph. The Llama 3.1 `rope_scaling` bypass notice now prints only when `MLXCEL_FUSED_ROPE_APPEND` is explicitly set to a truthy value, so default ROCm runs stay quiet.
+
+Re-measurement of the default-on build (2026-10-07, commit `57b59629`, the pre-rebase SHA of PR #2189's first commit, on `main` `08e22698`, same host and toolchain as above): `scripts/bench_decode.sh` at pp512/tg128, default (no variables) against `MLXCEL_FUSED_ADD_RMSNORM=0 MLXCEL_FUSED_ROPE_APPEND=0`, arm order alternated per round, every run through `scripts/rocm_gpu_guard.sh` with its host-wide lock (#2146), 12 of 12 clean on the first attempt. Raw rows: `benchmarks/rocm_strixhalo-gfx1151_2026-10-07_fused-default-{on,off}.csv`.
+
+| Model | Arm | Decode tok/s, by round | Median | vs off | Paired (on minus off), by round | Prefill median |
+|---|---|---|---|---|---|---|
+| Llama-3.1-8B | default (on) | 37.86 38.09 38.10 | 38.09 | -0.2% | +0.02 -0.49 -0.05 | 1024.65 |
+| | off | 37.84 38.58 38.15 | 38.15 | | | 1014.37 |
+| Qwen2.5-7B | default (on) | 47.90 48.30 47.27 | 47.90 | +0.8% | +0.39 +0.30 -0.04 | 1609.96 |
+| | off | 47.51 48.00 47.31 | 47.51 | | | 1544.08 |
+
+Same reading as before: Llama 3.1, where only the norm port runs, is flat (its -0.2% comes from one off run at 38.58, above every other run of either arm); Qwen2.5, where both run, is slightly above off in two of three rounds, inside the run-to-run spread. Qwen2.5 prefill was 2.7 to 6.9% higher with the default in every round, a pattern the earlier run did not show and three rounds cannot separate from drift. No result here argues against the decision.
+
+Kernel trace (`rocprofv3 --kernel-trace`, 8-token `mlxcel generate`, no variables set): Qwen2.5 dispatched `fused_add_rms_norm` and `fused_rope_qk_append` 252 times each (28 layers x 9 forward passes) and no graph RoPE kernel; with both variables at `0` it dispatched `rope_single_1d` instead. Llama 3.1 dispatched the norm port 288 times (32 x 9) and kept `rope_single_freqs_1d` / `rope_freqs`. `logit_trace` `w8` traces of both models with the default and with both variables at `0` are byte-identical, file for file.
 
 ## Not covered
 
-Metal and CUDA (not available on this host); their kernel sources and table entries are untouched. Their predicates gained the GPU-device term: GPU behavior is unchanged, and `MLXCEL_DEVICE=cpu` now takes the graph path instead of throwing at the first launch. Gemma and IQuest Loop Coder also call `fused_add_rms_norm` but have no checkpoint here; the Gemma `(1 + w)` convention is covered by the tolerance tests only. Wave64 (CDNA) is untested.
+Metal and CUDA (not available on this host); their kernel sources and table entries are untouched. Their predicates gained the GPU-device term: GPU behavior is unchanged, and `MLXCEL_DEVICE=cpu` now takes the graph path instead of throwing at the first launch. Gemma and IQuest Loop Coder also call `fused_add_rms_norm`, and on ROCm now do so by default, but have no checkpoint here; the Gemma `(1 + w)` convention is covered by the tolerance tests only. Wave64 (CDNA) is untested.
 
 ## Reproducing
 
 ```bash
 cargo build --release --features rocm --bin mlxcel --bin mlxcel-bench-decode --example logit_trace
 cargo test --release --features rocm -p mlxcel-core --lib -- --test-threads=1 fused_norm_parity_tests fused_rope_parity_tests
 cargo test --release --features rocm --test cpu_device_custom_kernel_gates
-MLXCEL_FUSED_ADD_RMSNORM=1 MLXCEL_FUSED_ROPE_APPEND=1 \
-    scripts/rocm_gpu_guard.sh -- scripts/bench_decode.sh models/mlx/Qwen2.5-7B-Instruct-4bit --output /tmp/on.csv
+# On ROCm the fusions are on by default since #2145; the off arm sets both to 0.
+scripts/rocm_gpu_guard.sh -- scripts/bench_decode.sh models/mlx/Qwen2.5-7B-Instruct-4bit --output /tmp/on.csv
+MLXCEL_FUSED_ADD_RMSNORM=0 MLXCEL_FUSED_ROPE_APPEND=0 \
+    scripts/rocm_gpu_guard.sh -- scripts/bench_decode.sh models/mlx/Qwen2.5-7B-Instruct-4bit --output /tmp/off.csv
 ```
 
 The trace commands are in `benchmarks/logit_traces/rocm_gfx1151_3f0e51af/README.md`.

docs/benchmarks.md

@@ -341,7 +341,8 @@ A speedup below 1.00 for either op is the signal to flip
 `FUSED_ADD_RMSNORM_DEFAULT` or `FUSED_ROPE_APPEND_DEFAULT` in
 `src/lib/mlxcel-core/src/layers.rs`, which leaves the kernel available and
 opt-in through `MLXCEL_FUSED_ADD_RMSNORM=1` / `MLXCEL_FUSED_ROPE_APPEND=1`
-instead of removing it.
+instead of removing it. Both constants are `cfg!(feature = "rocm")`: off on
+Metal and CUDA builds, on in a ROCm build (#2145).
 
 The op-level number is a lower bound on the end-to-end effect. Both fusions also
 remove a full-width intermediate from the MLX graph per call, which shows up as