From 00f34900adf8b5d12602c3e51a5a2d278818529e Mon Sep 17 00:00:00 2001 From: John Travers Date: Fri, 6 Nov 2026 09:35:46 +0000 Subject: [PATCH 1/2] Ledger: name the order rocFFT's rank-3 work area was measured in rocFFT lists lengths innermost first, so the probe's {T, N, N} held the frequency innermost, not HIP's current (omega, y, x). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XGENPtq9Gp7VCh5QmcDdzQ --- docs/dev/benchmarks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/benchmarks.md b/docs/dev/benchmarks.md index d895fb8cd..47c5543f1 100644 --- a/docs/dev/benchmarks.md +++ b/docs/dev/benchmarks.md @@ -968,7 +968,7 @@ So a `Local` scan on a device now keeps the backend while each point is the last `python -m benchmarks ab --device --length 0.3`, 3 rounds, the same steps and final fields with no difference: `kerr_beam_384` x0.994 (0.984–1.048), `kerr_beam_192` x1.030 (1.001–1.035, a window of two steps). So a size no longer costs more memory for its rank-2 pair, and 768 x 768 x 600 now holds less than 900 x 900 x 600. The size still sets the seconds per GiB. -rocFFT's plans share it as well. On an AMD Instinct MI300X (ROCm 10.0.0, `gfx942`) at `c0bd23f2`, 8 steps, check off, the plans grow by 1.35, 5.07, 5.76 and 7.28 GiB at 384, 750, 768 and 900 x 900 x 600, and a run holds 7.01, 21.95, 23.37 and 30.85 GiB. Asked of rocFFT directly, in place, Z2Z, a rank-3 plan's work area is one state at every N from 320 to 1200 and every T from 512 to 1024. A rank-2 plan batched over T takes one state at 192 and 384, none at 256 and 512, and 1.10, 1.08 and 1.06 states at 640, 768 and 1024. A rank-1 plan takes none. So at 768 x 768 x 600 four areas of their own would hold 22.0 GiB, as the MI325X's 22.02 GiB at `9a5d2346` did, and the shared one holds the largest of them. +rocFFT's plans share it as well. On an AMD Instinct MI300X (ROCm 10.0.0, `gfx942`) at `c0bd23f2`, 8 steps, check off, the plans grow by 1.35, 5.07, 5.76 and 7.28 GiB at 384, 750, 768 and 900 x 900 x 600, and a run holds 7.01, 21.95, 23.37 and 30.85 GiB. Asked of rocFFT directly, in place, Z2Z, a rank-3 plan with the frequency innermost (rocFFT lengths `{T, N, N}`, which rocFFT lists innermost first) has a work area of one state at every N from 320 to 1200 and every T from 512 to 1024. A rank-2 plan batched over T takes one state at 192 and 384, none at 256 and 512, and 1.10, 1.08 and 1.06 states at 640, 768 and 1024. A rank-1 plan takes none. So at 768 x 768 x 600 four areas of their own would hold 22.0 GiB, as the MI325X's 22.02 GiB at `9a5d2346` did, and the shared one holds the largest of them. ### The transverse edge against 0.1.0 From 0ddbd5e31491a659ebceab9792d12220fae767ad Mon Sep 17 00:00:00 2001 From: John Travers Date: Fri, 6 Nov 2026 09:56:20 +0000 Subject: [PATCH 2/2] Hold the volume route's scratch in the registers' order on HIP too #365's order was CUDA-only until rocFFT's two orders were measured. On an AMD Instinct MI325X, ROCm 10.0.0, the orders take the same time a transform (15.31 ms each at 768 x 768 x 600), but the step still gains from the input and fold that no longer cross layouts: 199 against 211 ms at 768 and 52 against 57 ms at 384, three interleaved rounds, every one faster. The device tests pass on that card in the new order. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01XGENPtq9Gp7VCh5QmcDdzQ --- changelog.d/365-volume-order.changed.md | 2 +- docs/dev/backends.md | 2 +- docs/dev/benchmarks.md | 22 ++++++++++++++++++---- native/src/general.cpp | 21 +++++++++++---------- 4 files changed, 31 insertions(+), 16 deletions(-) diff --git a/changelog.d/365-volume-order.changed.md b/changelog.d/365-volume-order.changed.md index ef59b64d5..a2ece6cca 100644 --- a/changelog.d/365-volume-order.changed.md +++ b/changelog.d/365-volume-order.changed.md @@ -1 +1 @@ -On a CUDA build, a Kerr-only free-space envelope's stage holds its transverse scratch in the state's order, so its rank-3 transform keeps frequency contiguous and its input and fold no longer cross layouts. A step on an NVIDIA H200 takes 204 ms at 768 x 768 x 600 where it took 225 ms, and 36 ms at 384 x 384 x 600 where it took 51 ms, with 1.3 GiB less memory there. Results are unchanged to rounding. +On a CUDA or HIP build, a Kerr-only free-space envelope's stage holds its transverse scratch in the state's order, so its rank-3 transform keeps frequency contiguous and its input and fold no longer cross layouts. A step on an NVIDIA H200 takes 204 ms at 768 x 768 x 600 where it took 225 ms, and 36 ms at 384 x 384 x 600 where it took 51 ms, with 1.3 GiB less memory there. On an AMD Instinct MI325X it takes 199 ms where it took 211 at 768 x 768 x 600, and 52 ms where it took 57 at 384 x 384 x 600. Results are unchanged to rounding. diff --git a/docs/dev/backends.md b/docs/dev/backends.md index ce06778a9..7d4acc921 100644 --- a/docs/dev/backends.md +++ b/docs/dev/backends.md @@ -1449,7 +1449,7 @@ free-space envelope whose material response is the Kerr term alone, with one polarisation, on a time grid the spectrum fills (`n_t = n_to = n_omega`, the crop the identity) takes its stage core on a device through one rank-3 transform of the scratch volume each way, forming the polarisation in place between them, instead of the chunks' -transposing pack and crop and their temporal transforms. On CUDA the stage holds that +transposing pack and crop and their temporal transforms. On a device the stage holds that scratch as `(y, x, omega)`, the state's own order, so the transform keeps frequency contiguous and the stage's input and fold stream beside the registers. Every other pass over the scratch forms its own input in `(omega, y, x)`, and an in-stage absorber keeps diff --git a/docs/dev/benchmarks.md b/docs/dev/benchmarks.md index 47c5543f1..4bd416786 100644 --- a/docs/dev/benchmarks.md +++ b/docs/dev/benchmarks.md @@ -968,7 +968,7 @@ So a `Local` scan on a device now keeps the backend while each point is the last `python -m benchmarks ab --device --length 0.3`, 3 rounds, the same steps and final fields with no difference: `kerr_beam_384` x0.994 (0.984–1.048), `kerr_beam_192` x1.030 (1.001–1.035, a window of two steps). So a size no longer costs more memory for its rank-2 pair, and 768 x 768 x 600 now holds less than 900 x 900 x 600. The size still sets the seconds per GiB. -rocFFT's plans share it as well. On an AMD Instinct MI300X (ROCm 10.0.0, `gfx942`) at `c0bd23f2`, 8 steps, check off, the plans grow by 1.35, 5.07, 5.76 and 7.28 GiB at 384, 750, 768 and 900 x 900 x 600, and a run holds 7.01, 21.95, 23.37 and 30.85 GiB. Asked of rocFFT directly, in place, Z2Z, a rank-3 plan with the frequency innermost (rocFFT lengths `{T, N, N}`, which rocFFT lists innermost first) has a work area of one state at every N from 320 to 1200 and every T from 512 to 1024. A rank-2 plan batched over T takes one state at 192 and 384, none at 256 and 512, and 1.10, 1.08 and 1.06 states at 640, 768 and 1024. A rank-1 plan takes none. So at 768 x 768 x 600 four areas of their own would hold 22.0 GiB, as the MI325X's 22.02 GiB at `9a5d2346` did, and the shared one holds the largest of them. +rocFFT's plans share it as well. On an AMD Instinct MI300X (ROCm 10.0.0, `gfx942`) at `c0bd23f2`, 8 steps, check off, the plans grow by 1.35, 5.07, 5.76 and 7.28 GiB at 384, 750, 768 and 900 x 900 x 600, and a run holds 7.01, 21.95, 23.37 and 30.85 GiB. Asked of rocFFT directly, in place, Z2Z, a rank-3 plan with the frequency innermost (rocFFT lengths `{T, N, N}`, which rocFFT lists innermost first) has a work area of one state at every N from 320 to 1200 and every T from 512 to 1024. With x innermost, an AMD Instinct MI325X's plans need one state too, at N = 384, 750, 768, 800 and 900 with T = 600. A rank-2 plan batched over T takes one state at 192 and 384, none at 256 and 512, and 1.10, 1.08 and 1.06 states at 640, 768 and 1024. A rank-1 plan takes none. So at 768 x 768 x 600 four areas of their own would hold 22.0 GiB, as the MI325X's 22.02 GiB at `9a5d2346` did, and the shared one holds the largest of them. ### The transverse edge against 0.1.0 @@ -2475,7 +2475,7 @@ against the build before them: `he_rdw_averaged` 0.988 (0.986–1.007), `rdw` 0. **Scratch-order registers on the volume route**, tried on the A40 and not adopted, take the fold from 407 to 387 ms and the step to 2,538 ms, with the fold's tile staging the state rather than the scratch. A layout change needs its own issue. -**The scratch in the registers' order** ([#365](https://github.com/LupoLab/solprop/issues/365)), on CUDA only. A stage of the volume route views the scratch as `(y, x, omega)`, the state's and the +**The scratch in the registers' order** ([#365](https://github.com/LupoLab/solprop/issues/365)), on a device. A stage of the volume route views the scratch as `(y, x, omega)`, the state's and the registers' order, so its rank-3 pair is cuFFT's frequency-fastest `{N,N,T}`, and its input and fold stream instead of crossing. A probe of the two orders on an NVIDIA H200, CUDA 13.1, T = 600, took `{N,N,T}` 2.49, 11.08, 14.71, 17.62 and 13.81 ms a transform at N = 384, 750, 768, 800 and 900, against 3.74, 14.97, 17.63, 17.97 and 18.96 ms for `{T,N,N}`, and `{N,N,T}` needs no work area at those sizes but 768. A step, from a field-free point with the check off, (24 steps - 8 steps) / 16, three interleaved rounds against the build before it @@ -2490,13 +2490,27 @@ At 768 the transforms account for it: eight of them are 23.4 ms apart in the pro `kerr_beam_384` x0.971 (0.928–1.003), fields to 3.2e-17. On the NVIDIA A40, CUDA 12.8.1, where the orders are within 1 % at 768, two rounds against `29e9e156`, whose native code is the same: 2,535 and 2,578 ms a step before, 2,556 and 2,597 after, at 768; 559 and 560 ms before, 513 and 513 after, at 384; `ab --device` geometric mean 0.998, with `kerr_beam_384` x1.010 (0.997–1.042) and `kerr_beam_192` x0.977 (0.950–1.070). The device tests pass on both cards. With the stage's window read in the old order, nine of them fail across the three -marches, and with the fold reading the old order, three low-storage ones do. HIP keeps the scratch's order until rocFFT's orders are measured. +marches, and with the fold reading the old order, three low-storage ones do. + +**On HIP too.** On an AMD Instinct MI325X, ROCm 10.0.0, `gfx942`, rocFFT's two orders take the same time. In place, Z2Z, T = 600, frequency innermost (rocFFT lengths +`{T,N,N}`, cuFFT's `{N,N,T}`) took 4.07, 15.66, 15.31, 16.72 and 20.94 ms a transform at N = 384, 750, 768, 800 and 900, against 4.10, 16.20, 15.31, 16.69 and 20.35 ms with x innermost. Both +need one state of work area at every one of those sizes. The step gains all the same, from the input and fold that no longer cross layouts. Three interleaved rounds, the same method, +against #542's head (`931ac0dc`): + +| N x N x 600, AMD Instinct MI325X | Before, ms | After, ms | Ratio | A run's memory, GiB | +|---:|---:|---:|---:|---:| +| 384 | 57.30–58.02 | 51.80–52.20 | 0.902 | 7.009 → 7.013 | +| 768 | 210.00–211.22 | 198.87–199.73 | 0.946 | 23.37 → 23.37 | + +At 768 eight transforms are the same in the probe, so all of the step's 11.8 ms is in the passes beside them. `python -m benchmarks ab --device --length 0.3`, 3 rounds: `kerr_beam_384` +x0.978 (0.974–0.982), fields to 1.8e-17. `pytest -m gpu` (251 passed, 1 skipped), the native device tests (31 passed) and the affected modules (172 passed, 3 skipped) pass on that +card in the new order. On a CPU, `python -m benchmarks ab` against the merge base on an Intel Xeon Gold 6240 (nine cores of one socket, bound passive threads, on a machine others shared; three rounds, the same steps): `kerr_beam_192` 0.982 (0.978–0.996) at one thread and 0.974 (0.973–0.987) at nine; `kerr_beam_384` 0.981 (0.981–0.982) at nine, fields to 6.4e-16. `rdw` under `rk` and `multistep-interaction`, one and nine threads, without the host's fold fix, which they do not reach: 1.011, 0.974, 0.996 and 0.999. Before the fold's fix, `radial_filament_256` 0.993 and `planar` 0.980 at nine threads, five rounds, took neither path. -The H200 has since measured the step in the scratch's order, above. An AMD Instinct MI300X has measured what rocFFT's plans hold and what a whole state's collar is worth. No MI325X has run the head. +The H200 has since measured the step in the scratch's order, above. An AMD Instinct MI300X has measured what rocFFT's plans hold and what a whole state's collar is worth, and an MI325X the step and the device tests with HIP in the scratch's new order, above. ## Reference tables diff --git a/native/src/general.cpp b/native/src/general.cpp index 260938716..cc5ae2280 100644 --- a/native/src/general.cpp +++ b/native/src/general.cpp @@ -612,8 +612,8 @@ struct RegisterIndex { // One polarisation: a column is its mode, without a division (a flag of its own, // which the compiler cannot fold back into `column / 1`). bool scalar{}; -#if defined(KOKKOS_ENABLE_CUDA) - // A stage of the volume route on CUDA (`volume_order_`): the scratch is `(y, x, +#if defined(KOKKOS_ENABLE_CUDA) || defined(KOKKOS_ENABLE_HIP) + // A stage of the volume route on a device (`volume_order_`): the scratch is `(y, x, // omega)`, the state's and the registers' own order, so nothing crosses. bool volume{}; #endif @@ -626,7 +626,7 @@ struct RegisterIndex { // The transverse scratch's position of the element, on any build. KOKKOS_INLINE_FUNCTION std::size_t cell(const std::size_t column, const std::size_t omega) const { -#if defined(KOKKOS_ENABLE_CUDA) +#if defined(KOKKOS_ENABLE_CUDA) || defined(KOKKOS_ENABLE_HIP) if (volume) return column * n + omega; #endif @@ -2374,12 +2374,12 @@ class GeneralWorkspace { return {transverse_layout(), problem_.n_pol_state, problem_.n_omega, plane, problem_.n_pol_state == 1}; } - // A stage's scratch positions: the volume route's own order on CUDA, or the scratch's - // (`register_index`). Every other pass over the scratch forms its own input in - // `(omega, y, x)`, since the scratch is dead between them. + // A stage's scratch positions: the volume route's own order on a device, or the + // scratch's (`register_index`). Every other pass over the scratch forms its own input + // in `(omega, y, x)`, since the scratch is dead between them. RegisterIndex stage_index() const noexcept { auto index = register_index(); -#if defined(KOKKOS_ENABLE_CUDA) +#if defined(KOKKOS_ENABLE_CUDA) || defined(KOKKOS_ENABLE_HIP) index.volume = volume_order_; #endif return index; @@ -2714,8 +2714,9 @@ class GeneralWorkspace { // transposes and temporal transforms (`separable_volume_core`). bool volume_kerr_ = false; // The volume route's stages hold the scratch as `(y, x, omega)`, `volume_scratch_`, - // on CUDA (solprop #365): its rank-3 transform is then cuFFT's frequency-fastest - // `{N,N,T}`, and its input and fold no longer cross layouts. + // on a device (solprop #365): its rank-3 transform then keeps frequency contiguous + // (cuFFT's `{N,N,T}`, rocFFT's `{T,N,N}`), and its input and fold no longer cross + // layouts. bool volume_order_ = false; ComplexView3 volume_scratch_; std::unique_ptr volume_forward_, volume_inverse_; @@ -4581,7 +4582,7 @@ void GeneralWorkspace::plan() { !problem_.raman && !problem_.envelope_plasma && !problem_.dispersive && separable_dispersive_ == nullptr && problem_.harmonic_factor == 0.0 && separable_gaps_.empty(); -#if defined(KOKKOS_ENABLE_CUDA) +#if defined(KOKKOS_ENABLE_CUDA) || defined(KOKKOS_ENABLE_HIP) // At 768 x 768 x 600 on an NVIDIA H200, cuFFT's `{N,N,T}` takes 14.7 ms a transform // against 17.6 ms for `{T,N,N}` (solprop #365). An in-stage absorber evaluates its // term from the same input in the scratch's order, so it keeps that order.