Skip to content

Hold the volume route's scratch in the registers' order on HIP too - #547

Merged
jtravs merged 2 commits into
mainfrom
perf/hip-volume-order
Nov 6, 2026
Merged

jtravs merged 2 commits into
mainfrom
perf/hip-volume-order

Conversation

@jtravs

@jtravs jtravs commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

#542 held the volume route's scratch in the registers' order, (y, x, omega), on CUDA only, and left HIP in (omega, y, x) until rocFFT's two orders had been measured. They now have been, on an AMD Instinct MI325X, and HIP is faster in the new order. This PR turns it on for HIP and corrects how the ledger labelled the earlier rocFFT probe.

The change

The four KOKKOS_ENABLE_CUDA guards on the order in native/src/general.cpp now also take KOKKOS_ENABLE_HIP. They cover the index's field, cell(), stage_index() and plan(). The comments, docs/dev/backends.md and #365's changelog fragment now say "on a device". There is no other code change.

What was measured

All of this ran on an AMD Instinct MI325X with ROCm 10.0.0, gfx942.

rocFFT's two orders take the same time. Each transform is in place, Z2Z, with T = 600:

N Frequency innermost, ms x innermost, ms
384 4.07 4.10
750 15.66 16.20
768 15.31 15.31
800 16.72 16.69
900 20.94 20.35

Both orders need one state of work area at each of these sizes.

The step still gains, every round. A step is measured from a field-free point with the check off, as (24 steps - 8 steps) / 16. The probe ran three interleaved rounds against #542's head (931ac0dc):

N x N x 600 Before, ms After, ms Per-round ratios A run's memory, GiB
384 57.30–58.02 51.80–52.20 0.902, 0.911, 0.894 7.009 → 7.013
768 210.00–211.22 198.87–199.73 0.944, 0.947, 0.946 23.37 → 23.37

At 768 the eight transforms take the same time in both orders, so the step's 11.8 ms gain comes from the passes beside them. Those are the stage's input and fold, which no longer cross layouts.

A/B and tests. python -m benchmarks ab --device --length 0.3 over 3 rounds gives kerr_beam_384 x0.978 (0.974–0.982), with the fields within 1.8e-17. On that card, in the new order, these pass:

  • pytest -m gpu: 251 passed, 1 skipped.
  • The native device tests: 31 passed.
  • The affected modules: 172 passed, 3 skipped.

The only native difference between this branch and the build that was measured is the comment.

A correction

rocFFT lists lengths innermost first. So the earlier rank-3 probe in the ledger, at lengths {T, N, N}, measured the frequency-innermost order, not HIP's old one. The ledger now names the order and adds the x-innermost work areas. The run-level memory figures stand, because they come from solprop's own plans. The comments on #529, #541 and #365 have been corrected the same way.

#365

The volume route is now in the scratch's order on both device backends. #365's own target, a step within x1.2 of a dedicated single-transform engine on an NVIDIA H200, is not met: it stands at x1.28 (204 against 159.4 ms). The transforms are about 118 ms of that step. The other ~86 ms is work the step has to do: the fold, the stage's input, Kerr and the temporal collar. That work is bound by memory traffic. Closing the gap means fusing those passes so the step moves the state fewer times, which is a redesign of the march, not a layout change. I've written that up on #365 as the remaining lever, and this closes it.

Closes #365.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XGENPtq9Gp7VCh5QmcDdzQ

jtravs and others added 2 commits November 6, 2026 09:35
rocFFT lists lengths innermost first, so the probe's {T, N, N} held the
frequency innermost, not HIP's current (omega, y, x).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPtq9Gp7VCh5QmcDdzQ
#365's order was CUDA-only until rocFFT's two orders were measured. On an
AMD Instinct MI325X, ROCm 10.0.0, the orders take the same time a
transform (15.31 ms each at 768 x 768 x 600), but the step still gains
from the input and fold that no longer cross layouts: 199 against 211 ms
at 768 and 52 against 57 ms at 384, three interleaved rounds, every one
faster. The device tests pass on that card in the new order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPtq9Gp7VCh5QmcDdzQ
Copilot AI balanced review requested due to automatic review settings November 6, 2026 09:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T09:58:56.001809Z 0ddbd5e PR opened
🔒 Security Review ✅ Completed 2026-11-06T10:00:26.604809Z 0ddbd5e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 solprop | 🛠️ Build #35693886 | 📁 Comparing 0ddbd5e against latest (1d785cb)

  🔍 Preview build  

2 files changed
± dev/backends.html
± dev/benchmarks.html

@codecov

codecov Bot commented Nov 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jtravs
jtravs merged commit b8215ca into main Nov 6, 2026
41 checks passed
@jtravs
jtravs deleted the perf/hip-volume-order branch November 6, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The low-storage march on a GPU is 1.35 to 1.67 times a single-transform engine

2 participants