perf(cuda): enable hardware NVFP4/MXFP4 conversion on Blackwell builds

.github/workflows/release.yml

@@ -537,13 +537,25 @@ jobs:
     # 90a is load-bearing for MLX's Hopper quantized kernel gate, and build.rs
     # forwards the list verbatim. The CLI and server (~347 MB each) ship as
     # separate archives, both with the CCCL headers.
+    #
+    # Blackwell stays plain here even though MLX's hardware NVFP4/MXFP4
+    # converters are gated on `__CUDA_ARCH_SPECIFIC__` and a plain target
+    # compiles them out (issue #1934). Naming Blackwell `<sm>a` in this list
+    # would enable them for every translation unit, which costs 12.7 MB of
+    # archive and changes the machine code of every kernel, decode included.
+    # `src/lib/mlx-cpp/CMakeLists.txt` adds the architecture-specific image to
+    # `fp_quantize.cu` alone instead, which is the only translation unit that
+    # can reach those converters, so this list needs no `a` entry to get them.
+    # scripts/ci/check_cuda_arch_lists.py rejects one.
     env:
       MLX_CUDA_ARCHITECTURES: "90a;100;121"
       CLI_ASSET: mlxcel-linux-aarch64-cuda13
       SERVER_ASSET: mlxcel-server-linux-aarch64-cuda13
 
-    # Cold build compiles MLX for three CUDA architectures; generous ceiling so
-    # a from-scratch build is never killed (the cargo cache makes reruns cheap).
+    # Cold build compiles MLX for three CUDA architectures, plus one extra
+    # code-generation pass for `fp_quantize.cu` per Blackwell entry; generous
+    # ceiling so a from-scratch build is never killed (the cargo cache makes
+    # reruns cheap).
     timeout-minutes: 360
 
     steps:
@@ -756,10 +768,16 @@ jobs:
 
     # One fat binary covering Ampere and later. 90a is load-bearing: MLX gates
     # its Hopper quantized kernel on "90a" being in the list (MLX_CUDA_SM90A_ENABLED),
-    # and build.rs forwards MLX_CUDA_ARCHITECTURES verbatim (no auto-suffix). Blackwell
-    # (sm_100/sm_120) has no MLX gate, so plain numbers emit SASS+PTX. sm_87 (Jetson)
-    # and sm_121 (GB10) are aarch64-only and excluded here. The build runner's own GPU
-    # is an RTX 5060 (sm_120), so the smoke test below runs on native SASS.
+    # and build.rs forwards MLX_CUDA_ARCHITECTURES verbatim (no auto-suffix).
+    # Blackwell stays plain for the reason spelled out in the aarch64 job
+    # above: the hardware NVFP4/MXFP4 converters are added to `fp_quantize.cu`
+    # alone by `src/lib/mlx-cpp/CMakeLists.txt` rather than to the whole target.
+    # This corrects the rationale that stood here before, which said Blackwell
+    # had no MLX gate and that plain numbers were therefore enough. There is a
+    # gate (issue #1934); it is simply not answered from this list. sm_87
+    # (Jetson) and sm_121 (GB10) are aarch64-only and excluded here. The build
+    # runner's own GPU is an RTX 5060 (sm_120), so the smoke test below runs on
+    # native SASS.
     env:
       MLX_CUDA_ARCHITECTURES: "80;86;89;90a;100;120"
       # The CLI and server binaries are ~347 MB each, so ship them as separate
@@ -768,9 +786,10 @@ jobs:
       SERVER_ASSET: mlxcel-server-linux-x86_64-cuda13
 
     # A cold build compiles MLX for six CUDA architectures (CUTLASS-heavy
-    # quantized kernels per arch); the persistent cargo cache makes later runs
-    # far cheaper. Generous ceiling so a from-scratch build is not killed (a
-    # cold 6-arch build was observed at ~3h).
+    # quantized kernels per arch), plus one extra pass for `fp_quantize.cu` per
+    # Blackwell entry; the persistent cargo cache makes later runs far cheaper.
+    # Generous ceiling so a from-scratch build is not killed (a cold six-arch
+    # build was observed at ~3h).
     timeout-minutes: 480
 
     steps:

docs/benchmark_results/data/blackwell-arch-specific-gb10-2026-09-21/README.md

@@ -0,0 +1,59 @@
+# Does compiling MLX architecture-specific cost decode? (GB10, 2026-09-21)
+
+Issue #1934. Host: NVIDIA GB10 (sm_121, DGX Spark), Linux aarch64 `7.0.0-1019-nvidia`, CUDA 13.0.88, MLX pin `81ba1c6a`, toolchain 1.97.1. Source tree `01fe5bbc` (this PR's branch, rebased onto `76c849b5`). Pairing: `models/mlx/qwen3.5-4b-4bit` with `models/mlx/qwen3.5-4b-dflash`, draft width 4 and classic. Driver: `harness/ab_interleave.py`, which imports `run_arm` from the #1797 served harness (`data/draft-block-width-gb10-2026-09-20/harness/`) rather than reimplementing the host gate, the NVRM trip wire and the DFlash diagnostics parsing.
+
+## Why interleaved
+
+A sequential A/B on this pairing reported the architecture-specific build about 2.6% slower at width 4 with disjoint ranges. Two arms measured one after the other are only as comparable as the host is still between them, and on this box that is the thing most likely to be false. This driver alternates at round granularity instead: one server start, one measured request, teardown, next binary, and the arm that opens rotates every round, so no binary systematically owns the warm or the cool half of the session.
+
+## Arms
+
+All three binaries are built from the same source tree and differ only in how MLX was compiled.
+
+| arm | `MLX_CUDA_ARCHITECTURES` | `libmlx.a` | cubin images in the binary |
+|---|---|---|---|
+| `121` | `121` | 178,388,958 B | 95 `sm_121` |
+| `121a-real;121` | `121a-real;121` | 191,110,534 B | 95 `sm_121`, 95 `sm_121a` |
+| `121+fp_quantize-a` | `121`, plus the per-source injection | 178,438,494 B | 95 `sm_121`, 1 `sm_121a` |
+
+The third arm is also the experiment's negative control. Its decode kernels are compiled with byte-identical flags to the first arm's, and their disassembly hashes match (`qmv.cu.o` sm_121 section SHA-256 `6eb4cf943169` in both, 3,640,476 lines; `fp_qmv.cu.o` `53faf7a4e1d5`, 65,040 lines), with no second image for the driver to prefer. Any difference it shows cannot be a decode-kernel-speed difference.
+
+## Result: no decode cost survives interleaving
+
+30 arms, 0 errors, `nvrm_delta` 0 everywhere, no CI job overlapping any arm, load1 before each arm between 0.01 and 0.47. All fifteen width-4 runs produced byte-identical text from a 158-token prompt, across all three binaries, so the arms did the same work and what is being compared is how fast they did it.
+
+| width | arm | n | min | max | mean | ratio to `121` | ranges disjoint |
+|---|---|---|---|---|---|---|---|
+| 4 | `121` | 5 | 65.96 | 67.81 | 66.97 | | |
+| 4 | `121a-real;121` | 5 | 66.04 | 68.18 | 67.31 | 1.0052 | no |
+| 4 | `121+fp_quantize-a` | 5 | 65.40 | 67.17 | 66.21 | 0.9887 | no |
+| classic | `121` | 5 | 55.72 | 59.39 | 57.21 | | |
+| classic | `121a-real;121` | 5 | 55.94 | 57.24 | 56.76 | 0.9923 | no |
+| classic | `121+fp_quantize-a` | 5 | 55.09 | 57.25 | 56.34 | 0.9849 | no |
+
+Paired within-round differences against `121`, which removes round-level drift:
+
+| width | arm | deltas (tok/s) | mean |
+|---|---|---|---|
+| 4 | `121a-real;121` | +0.81, +1.67, +1.54, -0.60, -1.67 | +0.54% |
+| 4 | `121+fp_quantize-a` | -0.26, -1.10, +1.21, -1.55, -2.08 | -1.11% |
+| classic | `121a-real;121` | -1.24, +0.22, -0.09, +1.13, -2.22 | -0.73% |
+| classic | `121+fp_quantize-a` | -1.18, -0.63, -0.31, -0.05, -2.14 | -1.48% |
+
+The architecture-specific arm is not slower. At width 4 it is nominally faster, its paired deltas change sign across rounds, and every range overlaps.
+
+What bounds the method is the negative control. `121+fp_quantize-a` runs decode kernels that are the same bytes as `121`, and it measures 1.11% slower at width 4 and 1.48% slower at classic with all five paired deltas negative. So five same-sign deltas at about 1.5% are reachable here with no code difference at all, which is what the earlier 2.6% sequential result has to be read against.
+
+## Reproducing
+
+```bash
+MLXCEL_SWEEP_HARNESS=<repo>/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness \
+python3 harness/ab_interleave.py \
+  --arm "121=<server built at 121>" \
+  --arm "121a-real;121=<server built at 121a-real;121>" \
+  --arm "121+fp_quantize-a=<server built at 121 on this branch>" \
+  --target <models>/qwen3.5-4b-4bit --drafter <models>/qwen3.5-4b-dflash \
+  --rounds 5 --widths 4,classic --n 1 --out ab.jsonl --logdir logs
+```
+
+`ab.jsonl` is one record per arm, carrying the host-gate readings, the server command line, the DFlash diagnostics and the generated text. `logs/` holds each arm's server log; the `CUDA compute capability ...; compiled for [...]` startup line in each one names the architecture list that binary was compiled for, which is how an arm's label is checked against the binary rather than trusted.