Skip to content

The low-storage march on a GPU is 1.35 to 1.67 times a single-transform engine #365

Closed

Description

@jtravs

What happens

On an NVIDIA H200, SEP 28 measured the low-storage march (RK4IP keeping few registers, for field-free planes on large Cartesian grids) at 51.6 ms a step at 384² by 600 and 227 ms at 768² by 600, against 30.8 and 168.7 ms for a dedicated single-transform engine: 1.67 and 1.35 times, on 2026-10-20.

A cuFFT probe found the transforms near their best: one rank-3 transform would save only 1.6 to 2.2 ms an evaluation. The rest is the gather and scatter, 32 ms a step at 768², and a fold near the card's bandwidth, so the volumes each pass reads and writes. A march that touches fewer volumes per pass is the remaining lever.

Why it matters

This is the route for the largest grids, where a GPU's memory bandwidth sets the run time, so the gap is paid at each step.

To do

  • Re-measure both grids on an H200 at the current tree with a per-step timer. The tracker's kerr_beam_384 row (384 x 384 x 750, four steps, 4.16 s on an H200 at 0.1.0) is a whole run, so it is not comparable; what takes the rest of that time is worth knowing.
  • Count volumes read and written per pass, and try a march that touches fewer, on the option list in SEP 28.

What must not change: the march's fields within its parity bounds, and the memory ceiling that is the reason for this route (#353 measures that ceiling).

Done when

The per-step gap to a single-transform engine on an H200 at 768² by 600 is below 1.2 times, or the issue closes with the re-measured gap and the reason nothing further helps.


Checked against commit 8a9f764. Takes over the roadmap's entry "The march's register traffic on devices".

Activity

  1. added
    performanceSlower or larger than it should be, with no wrong result
    area: free spaceRadial, Cartesian, line and planar free-space routes
    area: gpuCUDA and HIP builds: qualification and performance
    pnpsFound by, or would help, pnps (the TG/SD-FROG forward model built on solprop)
    on Nov 1, 2026
  2. jtravs commented on Nov 4, 2026

    @jtravs
    ContributorAuthor

    Re-measured with a per-step timer on the far-field branch (#497), median of steps 3 to 10 of one process, low-storage RK4IP with fixed 0.5 µm steps, fused-silica Kerr beam at 260 nm, field-free planes. The temporal collar stays on, no transverse collar. The spread was under 0.2 % within a process.

    Grid NVIDIA H200 (CUDA 13.1) AMD Instinct MI325X (ROCm 10) SEP 28's H200 march SEP 28's single-transform engine
    384² × 600 53.9 ms 61.0 ms 51.6 ms 30.8 ms
    768² × 600 236.4 ms 222.7 ms 227 ms 168.7 ms
    • Gap on the H200, against the dated engine: 1.75 and 1.40 times. The engine isn't in the repository, so its numbers are from 2026-10-20 and weren't re-run. The target of 1.2 is 202 ms at 768², so this isn't met.
    • Against October's march: about 4 % slower. My case may differ from SEP 28's (I don't know whether its run had the collar), so I wouldn't read that as a regression.
    • Not done: the volume count per pass and any change to the march. This is left open for a decision on whether a march that touches fewer volumes is worth trying. SEP 28's own note says the transforms are near their best and that the gather and scatter are about 32 ms a step at 768², so removing all of that would still leave about 204 ms.
  3. jtravs commented on Nov 4, 2026

    @jtravs
    ContributorAuthor

    Decision (maintainer): try it. This stays open to try a march that touches fewer volumes per pass, as the second to-do above says. A closing note is only acceptable with the re-measured gap and the reason nothing further helps.

    Where to start, from the numbers above:

    • The target needs more than the gather and scatter. Removing all of it (about 32 ms a step at 768²) would still leave about 204 ms against the 202 ms target. So a change has to reduce the fold's traffic too. That's why the volume count per pass comes first.
    • The comparison is dated. The single-transform engine isn't in the repository, so 168.7 ms is from 2026-10-20. Compare the march against its own numbers here (236.4 ms on the H200, 222.7 ms on the MI325X, 53.9 and 61.0 ms at 384²), and against the 202 ms as an absolute figure, unless the engine can be re-run.
    • The measurement script isn't in the tree. It timed backend.propagate_to around steps 3 to 10, with boundary_check=None, fields="none" and a small energy statistic after each step to make sure the step had finished. That statistic isn't free (The energy statistic costs more than a far-field pass on a device, unexplained #503), but it wasn't counted in the step's time.
    • Must not change: the fields within parity bounds, and the memory ceiling that is the reason for this route.

    Neither card has been re-run since #496 and #497. #496 changes what's held (held_plane=False), which could move the numbers.

  4. jtravs commented on Nov 4, 2026

    @jtravs
    ContributorAuthor

    The per-pass budget, from kernel traces. One step at 768 x 768 x 600, 4 steps averaged, 7a224ea5, 2026-11-04. This case takes the volume route (free.volume_kerr appears), which by reading is 30 volumes a step outside cuFFT's and rocFFT's own traffic.

    NVIDIA H200, 238.6 ms AMD Instinct MI325X, 223.7 ms
    8 rank-3 transforms 144 ms (60 %) 112 ms (50 %)
    low_storage.separable_fold 52 ms (22 %) 49 ms (22 %)
    Temporal split collar: 106 chunks of 5 kernels 18 ms (8 %) 40 ms (18 %)
    free.volume_kerr 11 ms 12 ms
    low_storage.separable_input 10 ms 8 ms
    low_storage.separable_norms 1.4 ms 1.2 ms
    • The gather and scatter aren't what's left: this route doesn't have them, so SEP 28's 32 ms doesn't apply to it.
    • The temporal collar is a candidate for the gap to the dated 168.7 ms engine. On the H200 it's 18 ms, and I don't know whether that engine ran with a collar. On the MI325X its chunked rank-1 pair costs about as much as a whole rank-3 transform.
    • The target looks hard. 202 ms on the H200 means losing 36 ms, with 144 ms in transforms that the probe found near their best. My estimate is about 10 % from one plan for the collar, the dead read of the first register in pass 3 and the fold's order. That's a suspicion until tried.
  5. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    Re-measured with #520. At 768 x 768 x 600, by python -m benchmarks.device_split on one runner's second point:

    SEP 28's single-transform engine Now Gap
    NVIDIA H200, step 168.7 ms 220 ms x1.30
    AMD Instinct MI325X, step not measured 208 ms

    Reaching x1.2 would take 202 ms on the H200, and this is how far the leads went:

    What's left is the budget in the last comment: the eight transforms take about 144 ms, and SEP 28's probe found them near their best, and the fold's 52 ms is near the card's bandwidth for those volumes. I have no lead left that keeps the memory ceiling.

    One lead remains outside the step itself. With the default check on, the temporal monitor's sums take 24 ms a step on the H200, beside _finish's 11 to 15 ms. If both read the state, a shared pass might save some of that on a checked run. I haven't checked or tried it, and it wouldn't change this issue's number, which is an unchecked step.

    So I'd close this with the gap at x1.30, under "the re-measured gap and the reason nothing further helps". I'll leave that to you.

  6. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    This corrects my last comment. I said the transforms were near their best and that I had no lead left, and both were wrong. So this stays open, as the maintainer has decided.

    The dedicated single-transform engine's own figures, on an NVIDIA H200 at 768 x 768 x 600, against this route with #520:

    Engine This route
    Step 159.4 ms 220 ms
    Eight rank-3 transforms 117.6 ms about 144 ms, by a kernel trace with fences
    The rest 41.8 ms about 76 ms
    Device memory 21.19 GiB 38.96 GiB

    So memory isn't what limits this route, since the engine holds less. The engine also has no temporal collar in its step, which is up to 18 ms of this route's, and that part is the price of a capability.

    The transforms. The engine's plan keeps the frequency axis contiguous, {N,N,T}. This route's scratch is (omega, y, x), so its plan is {T,N,N}. On an H200, {N,N,T} alone takes 14.7 ms a transform at 768, which is the engine's 117.6 ms for eight. {T,N,N} hasn't been timed alone on that card.

    An interleaved probe on an NVIDIA A40 (CUDA 12.8.1), T = 600, ms a transform:

    N {N,N,T}, work area {T,N,N}, work area {T,N,N} / {N,N,T}
    384 41.92, none 47.90, 1 state 1.143
    750 190.88, none 216.58, 1 state 1.135
    768 232.02, 1 state 230.08, 1 state 0.992
    800 205.64, none 246.39, 1 state 1.198
    900 268.32, none 306.01, 1 state 1.140

    The passes. On a device the state and registers are already frequency-contiguous, and only the scratch differs, so the fold and the input cross two orders. On the same A40 at 768, a plain pass of two reads and a write takes 30.1 ms in one order, 61.9 ms crossing by lines and 117.3 ms crossing along the scratch's rows. That's a bare kernel, not this route's tiled fold, so it shows what crossing can cost and not what the fold would gain.

    The 0.7 % I reported for scratch-order registers was a whole step on an A40, where the transforms are 72 % of it. It also moved the registers towards the scratch, the opposite of what the engine does.

    What to try. Only the volume route's scratch, in the registers' order. The route is already a guarded fast path for an envelope Kerr beam with one polarisation, an identity crop, and no plasma, Raman, dispersion, harmonic or gaps. The state, the registers and every other route keep their layout. I haven't yet checked what else on this route reads the scratch. An H200 probe of both orders comes first, before any code changes, and if the change can't stay inside that route, it isn't worth making.

  7. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    The maintainer has set the bar for the transform order: try it only if an H200 probe supports it, the change stays inside the volume route, and the whole step improves by at least 10 %, the bar #361 used. Otherwise this closes with the measured gap and that reason. #529's shared work area goes ahead either way, since it costs no capability.

  8. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    The transform order, measured against the bar above.

    The probe. On an NVIDIA H200, CUDA 13.1, T = 600, ms a transform, median of interleaved rounds:

    N {N,N,T}, work area {T,N,N}, work area {T,N,N} / {N,N,T}
    384 2.49, none 3.74, 1 state 1.504
    750 11.08, none 14.97, 1 state 1.351
    768 14.71, 1 state 17.63, 1 state 1.198
    800 17.62, none 17.97, 1 state 1.020
    900 13.81, none 18.96, 1 state 1.373

    That supported trying it, so #542 now holds the volume route's scratch as (y, x, omega) on CUDA, the state's and the registers' order: the rank-3 pair is {N,N,T}, and the stage's input and fold stream instead of crossing layouts. It stays inside the route: every other pass over the scratch forms its own input in (omega, y, x), as before, and an in-stage absorber keeps the old order. HIP is unchanged until rocFFT's orders are measured.

    The step, from a field-free point with the check off, (24 steps - 8) / 16, three interleaved rounds against #542 before it:

    N x N x 600, NVIDIA H200 Before, ms After, ms Ratio A run's memory, GiB
    384 50.96–51.17 36.22–36.49 0.713 7.23 → 5.91
    768 225.21–225.78 203.88–204.13 0.905 23.14 → 23.14

    At 768 the transforms account for it: eight are 23.4 ms apart in the probe, and the step gains 21.6 ms. Fields agree to 3.2e-17 on python -m benchmarks ab --device. The device tests pass on the H200 and the NVIDIA A40, and on the A40 two deliberate breaks of the order, the stage's time window and the low-storage fold, fail 9 and 3 of them.

    Against the bar. At 768 the step is 10.5 % faster, or takes 9.5 % less time. That's on the line, and which way it's read decides it, so it's for the maintainer on #542. If it stays, the gap at 768 is 204 ms against the dedicated engine's 159.4 ms, x1.28. If it goes, this closes with 225 ms, x1.41, and that reason. This stays open until then.

  9. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    Decided: the scratch's order stays in #542 on CUDA, as the faster route.

    At 768² by 600 on an NVIDIA H200 a step now takes 204 ms against 225 before. That leaves the gap to the dedicated engine's 159.4 ms at x1.28, so this issue's "below 1.2 times" isn't met yet and it stays open. Two leads remain:

    • HIP's order. HIP keeps (omega, y, x), because rocFFT's two orders haven't been compared. On an AMD Instinct MI300X, ROCm 10.0.0, rocFFT's in-place rank-3 transform with the frequency innermost takes 17.4 ms at 768 and needs one state of work area at every size measured. Corrected 2026-11-06: that is rocFFT lengths {T,N,N}, which rocFFT lists innermost first, so it is the new order, this issue's {N,N,T}, not HIP's current one. A probe of HIP's current order and a step A/B on an AMD card would settle whether HIP takes the new order. That's an AMD-only question, and it doesn't touch the H200 gap.
    • The fold and the other passes beside the transforms. At 768 the eight rank-3 transforms are about 118 ms of the 204 ms step by the probe, so about 86 ms is still in the fold, the input, Kerr and the collar.
  10. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    HIP measured, and it takes the new order too. #547 closes this issue.

    On an AMD Instinct MI325X with ROCm 10.0.0, rocFFT's two orders take the same time per transform: 15.31 ms each at 768 x 768 x 600 and 4.07 against 4.10 ms at 384. Both need one state of work area. The step is still faster in the new order in every round:

    • 384: 51.80–52.20 against 57.30–58.02 ms, x0.90.
    • 768: 198.87–199.73 against 210.00–211.22 ms, x0.95.

    Since the transforms take the same time, that gain comes from the stage's input and fold no longer crossing layouts. ab --device gives kerr_beam_384 x0.978, and the device tests pass on that card in the new order. The volume route now holds the scratch in the registers' order on CUDA and HIP alike.

    Where the target stands. On the NVIDIA H200 a step at 768 x 768 x 600 takes 204 ms, against the 159.4 ms of a dedicated single-transform engine. That is x1.28, against this issue's target of x1.2. What makes up the 204 ms:

    • The eight rank-3 transforms: about 118 ms by the probe.
    • The rest, about 86 ms: work the step has to do. That is the fold (applying the linear propagator and combining the registers), forming each stage's input, the Kerr polarisation, and the temporal collar with its own 1-D transforms. In the earlier 239 ms profile these were about 52, 10, 11 and 18 ms.

    All of those passes are bound by memory traffic, not arithmetic. One read or write of the 5.27 GiB state costs about 1.2 ms at the H200's bandwidth, so 86 ms is roughly 70 state-sized reads or writes per step.

    The remaining lever is fusion, not layout. The step has to move the state fewer times: fold Kerr into the stage input, combine the fold with the next stage's input, or let the FFT library's load and store callbacks apply the propagator phase as the transform reads and writes. That is a redesign of the march. If it's taken up, it needs its own issue with a kernel trace first. Closing this one with the gap at x1.28 and that reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: free spaceRadial, Cartesian, line and planar free-space routesarea: gpuCUDA and HIP builds: qualification and performanceperformanceSlower or larger than it should be, with no wrong resultpnpsFound by, or would help, pnps (the TG/SD-FROG forward model built on solprop)priority: lowWorth doing, no hurry

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions