Skip to content

A submission's commit walks only the pages its GPU writes touched, in table order: Ocarina of Time's 25 commits a frame take 2.6 µs each for the tables of running GPU writes instead of 12 - #2458

Open
PucaVaz wants to merge 1 commit into
seedds:mainfrom
PucaVaz:perf-gpu-write-commit

Conversation

@PucaVaz

@PucaVaz PucaVaz commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

What and why

gw_commit puts a submission's GPU-written pages into the tables that uniform shadows ask (g_gw, g_gwx). In a sample of Ocarina of Time in play, 85 of the replay thread's 857 busy samples were on the line that calls it, with no Metal call underneath. The game commits 25 submissions a frame, and each commit paid for two things:

  • The pages kept one by one (g_sbwp, 16,384 slots tagged with the submission) were found again by reading all 16,384 slots (256 KB) in every commit that had any: 17 of the 25 a frame, for about 27 pages each.
  • A page's slot in the tables was a hash of the page number alone, so each page of a write was a cache line of its own in each of the two 1 MB tables. Writes of 64 to 4,095 pages are put page by page; the game has about one a commit, 1,176 pages on average.

The change (dm/gfx/nvn/nvn_mtl.m, 13 lines):

  • The set lists the slots a submission takes (g_sbw_new); gw_commit walks the list. The tables end up the same in any order of the pages.
  • gw_slot is the page's number within its 256 MB region plus a hashed offset of the region, so a run of pages takes neighbouring slots and the pages of one region never share a slot. What a slot means is unchanged. Which pages share a slot does change: scattered pages of different regions as often as before; two long writes in different regions can overlap in a run of slots.

Reference entry: A submission's commit scattered its GPU-written pages over the tables. The game note gets the measurements of this Mac.

Checked

  • Tests: make -C dm/gfx/tests GAME=0400486024DEC000 check_t_sbw 45 checks, 0 failed (10 new); check_t_ubp_running 25 checks, 0 failed; dm/tools/tests/test_docs.py passes. The 10 new checks also pass on untouched main (0 fail): they hold what must not change (which pages read as written, in both tables, and that pages beside a write do not). Two deliberate breaks fail them: a slot from the page's low bits alone fails 1 (the same offsets of a third 256 MB region read as written), a walk that skips one listed slot fails 2. The speed itself has no test.
  • Games run: The Legend of Zelda: Ocarina of Time (0400486024DEC000, v131072), headless, from a save, standing in Kokiri Forest outside Link's house, DM_FPS_HOLD=off, window from 70 s. The host is 7867152 plus this patch, because the packed app's shader libraries are of that chain (the patch applies cleanly there and on main; the tables' code is the same). No new red line (the app's 2,009 missing _b libraries on both sides), 101 to 103 render passes a frame and refused 0 in every residency line on both sides.
  • Devices and OS: MacBook Air 15-inch, M4 (10-core GPU), 24 GB, macOS 27.2.
  • Measurements. The Mac has no fan and identical runs drift (GPU p50 21.2 → 27.3 ms over six runs in a row), so the comparison that counts is the first table: both paths in one process, alternating every 300 frames (DM_AB=300; a test build with a switch between the old and new path, which is not part of this change). Each phase's median without its first quarter; "paired" is each new-path phase against the old-path phases beside it.
Run (phases) gw_commit µs a commit, old → new Submit CPU ms a frame, old → new (paired median) Submit instructions a frame, old → new (paired median) Frame p50 ms, old / new GPU p50 ms, old / new
1 (13) 11.6 → 2.5 5.12 → 4.86 (−0.27) 35.56 M → 34.08 M (−1.40 M) 23.88 / 23.86 22.60 / 22.57
2 (12) 12.1 → 2.6 5.42 → 5.12 (−0.25) 35.91 M → 34.46 M (−1.50 M) 27.29 / 27.29 25.97 / 26.07
3 (12) 12.4 → 2.7 5.46 → 5.22 (−0.19) 35.37 M → 34.41 M (−1.19 M) 26.50 / 26.25 24.61 / 24.64

Separate runs, interleaved before, after, before, … (105 s each, the shipped code, no switch):

Run Frame p50 / p95 ms GPU p50 / p95 ms Submit CPU p50 ms Submit instructions p50
before 1 22.50 / 24.11 21.25 / 22.30 4.95 35.75 M
after 1 25.86 / 28.00 24.54 / 26.37 5.06 34.26 M
before 2 24.84 / 26.75 23.51 / 24.98 5.31 35.59 M
after 2 28.57 / 30.50 27.28 / 28.96 4.86 34.19 M
before 3 25.45 / 28.32 24.02 / 25.97 5.79 35.25 M
after 3 25.70 / 29.27 24.30 / 27.84 5.13 34.13 M

What they show: the submit thread does 1.2 to 1.5 million fewer instructions a frame (about 4 %) and takes 0.19 to 0.27 ms less CPU (about 5 %). The frame time does not change in this scene on this Mac: it waits for the GPU (frame = GPU + about 1.3 ms), and the separate runs' frame and GPU times follow the Mac's temperature, not the change. A gain in frame time needs a scene or a device where the submit thread is the limit; none was measured.

  • scripts/regress.py check: not run. Ocarina of Time is the only game on this Mac and its build's libraries do not match main's chain. The change reaches every NVN game's commit (nvn_mtl.m); it changes no chain input.
  • Not run: the host of main itself in the game (main's nvn_mtl.m with the patch was compiled and run only in the gfx tests); a window; the iPhone; any other game; a game with GPU writes in many 256 MB regions at once; lint (ruff, pyright and the pinned LLVM are not installed here).
  • Open: whether two long page-by-page writes in different regions meet in a run of slots in some game (they would then count for any page hashing there until they complete, as a shared slot always did). Ocarina of Time shows no refused shadow either way.

Docs

  • The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
  • A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change (no chain tool changed)
  • No game files, keys or data extracted from a game in the change or the attachments

… table order: Ocarina of Time's 25 commits a frame take 2.6 µs each for the tables of running GPU writes instead of 12 (0.25 ms and 1.3 million instructions less a frame on the submit thread)

## What and why

`gw_commit` puts a submission's GPU-written pages into the tables that uniform shadows ask (`g_gw`, `g_gwx`). In a `sample` of Ocarina of Time in play, 85 of the replay thread's 857 busy samples were on the line that calls it, with no Metal call underneath. The game commits 25 submissions a frame, and each commit paid for two things:

- The pages kept one by one (`g_sbwp`, 16,384 slots tagged with the submission) were found again by reading all 16,384 slots (256 KB) in every commit that had any: 17 of the 25 a frame, for about 27 pages each.
- A page's slot in the tables was a hash of the page number alone, so each page of a write was a cache line of its own in each of the two 1 MB tables. Writes of 64 to 4,095 pages are put page by page; the game has about one a commit, 1,176 pages on average.

The change (`dm/gfx/nvn/nvn_mtl.m`, 13 lines):

- The set lists the slots a submission takes (`g_sbw_new`); `gw_commit` walks the list. The tables end up the same in any order of the pages.
- `gw_slot` is the page's number within its 256 MB region plus a hashed offset of the region, so a run of pages takes neighbouring slots and the pages of one region never share a slot. What a slot means is unchanged. Which pages share a slot does change: scattered pages of different regions as often as before; two long writes in different regions can overlap in a run of slots.

Reference entry: [A submission's commit scattered its GPU-written pages over the tables](docs/reference/frame-rate.md#a-submissions-commit-scattered-its-gpu-written-pages-over-the-tables). The game note gets the measurements of this Mac.

## Checked

- Tests: `make -C dm/gfx/tests GAME=0400486024DEC000 check_t_sbw` 45 checks, 0 failed (10 new); `check_t_ubp_running` 25 checks, 0 failed; `dm/tools/tests/test_docs.py` passes. The 10 new checks also pass on untouched main (0 fail): they hold what must not change (which pages read as written, in both tables, and that pages beside a write do not). Two deliberate breaks fail them: a slot from the page's low bits alone fails 1 (the same offsets of a third 256 MB region read as written), a walk that skips one listed slot fails 2. The speed itself has no test.
- Games run: The Legend of Zelda: Ocarina of Time (0400486024DEC000, v131072), headless, from a save, standing in Kokiri Forest outside Link's house, `DM_FPS_HOLD=off`, window from 70 s. The host is 7867152 plus this patch, because the packed app's shader libraries are of that chain (the patch applies cleanly there and on main; the tables' code is the same). No new red line (the app's 2,009 missing `_b` libraries on both sides), 101 to 103 render passes a frame and `refused 0` in every residency line on both sides.
- Devices and OS: MacBook Air 15-inch, M4 (10-core GPU), 24 GB, macOS 27.2.
- Measurements. The Mac has no fan and identical runs drift (GPU p50 21.2 → 27.3 ms over six runs in a row), so the comparison that counts is the first table: both paths in one process, alternating every 300 frames (`DM_AB=300`; a test build with a switch between the old and new path, which is not part of this change). Each phase's median without its first quarter; "paired" is each new-path phase against the old-path phases beside it.

| Run (phases) | `gw_commit` µs a commit, old → new | Submit CPU ms a frame, old → new (paired median) | Submit instructions a frame, old → new (paired median) | Frame p50 ms, old / new | GPU p50 ms, old / new |
|---|---|---|---|---|---|
| 1 (13) | 11.6 → 2.5 | 5.12 → 4.86 (−0.27) | 35.56 M → 34.08 M (−1.40 M) | 23.88 / 23.86 | 22.60 / 22.57 |
| 2 (12) | 12.1 → 2.6 | 5.42 → 5.12 (−0.25) | 35.91 M → 34.46 M (−1.50 M) | 27.29 / 27.29 | 25.97 / 26.07 |
| 3 (12) | 12.4 → 2.7 | 5.46 → 5.22 (−0.19) | 35.37 M → 34.41 M (−1.19 M) | 26.50 / 26.25 | 24.61 / 24.64 |

  Separate runs, interleaved before, after, before, … (105 s each, the shipped code, no switch):

| Run | Frame p50 / p95 ms | GPU p50 / p95 ms | Submit CPU p50 ms | Submit instructions p50 |
|---|---|---|---|---|
| before 1 | 22.50 / 24.11 | 21.25 / 22.30 | 4.95 | 35.75 M |
| after 1 | 25.86 / 28.00 | 24.54 / 26.37 | 5.06 | 34.26 M |
| before 2 | 24.84 / 26.75 | 23.51 / 24.98 | 5.31 | 35.59 M |
| after 2 | 28.57 / 30.50 | 27.28 / 28.96 | 4.86 | 34.19 M |
| before 3 | 25.45 / 28.32 | 24.02 / 25.97 | 5.79 | 35.25 M |
| after 3 | 25.70 / 29.27 | 24.30 / 27.84 | 5.13 | 34.13 M |

  What they show: the submit thread does 1.2 to 1.5 million fewer instructions a frame (about 4 %) and takes 0.19 to 0.27 ms less CPU (about 5 %). The frame time does not change in this scene on this Mac: it waits for the GPU (frame = GPU + about 1.3 ms), and the separate runs' frame and GPU times follow the Mac's temperature, not the change. A gain in frame time needs a scene or a device where the submit thread is the limit; none was measured.
- `scripts/regress.py check`: not run. Ocarina of Time is the only game on this Mac and its build's libraries do not match main's chain. The change reaches every NVN game's commit (`nvn_mtl.m`); it changes no chain input.
- Not run: the host of main itself in the game (main's `nvn_mtl.m` with the patch was compiled and run only in the gfx tests); a window; the iPhone; any other game; a game with GPU writes in many 256 MB regions at once; lint (ruff, pyright and the pinned LLVM are not installed here).
- Open: whether two long page-by-page writes in different regions meet in a run of slots in some game (they would then count for any page hashing there until they complete, as a shared slot always did). Ocarina of Time shows no refused shadow either way.

## Docs

- [x] The finding is in its game note or its `docs/reference/<subsystem>.md` entry, and `python3 dm/tools/docs_symptoms.py` was run (the symptom index)
- [ ] A chain tool changed: `dmsh_chain.py --header` was run and `dm/gfx/nvn/dmsh_chain.h` is part of the change (no chain tool changed)
- [x] No game files, keys or data extracted from a game in the change or the attachments
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation game: 0400486024DEC000 A game labels Nov 12, 2026
@dpadr

dpadr commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

We measured this on Ocarina of Time (Switch 2, v131072) on a fanless M2 MacBook Air (16 GB, macOS 27.0), handheld at full resolution (1440x810, MetalFX, 60 fps target, hold off).

Base and changes needed to apply it.

  • Base: our cut7 line, a406a81f: main 7f50222 plus our carries, chain bf6035002b98.
  • gw_commit there has no SBW_GPU filter: report pages still count for later submissions on that base. So the walk puts every listed slot, and we left out t_sbw's report-only check.
  • For the A/B we added a run-level switch, DM_NVN_GW_FAST (default 1; 0 restores the old walk and hash), read once at start. The slot mapping can't change mid-run without losing pages already in the tables, so an in-run DM_AB alternation isn't possible here.
  • On current main (596d719) the code is still in nvn_mtl.m and the PR applies as it is.

A/B.

  • One app; A = old path, B = this PR.
  • Interleaved whole runs ABABAB after a discarded warm-up, Kokiri Forest route to frame 12,000 from a pinned save, three runs a side.
  • CPU clock 2.3–2.8 GHz (thermal level 2, on AC).
A B B − A [95 %]
submit-thread k cycles a draw 19.92 18.97 −0.96 [−3.21, −0.54] (−4.8 %)
submit-thread k instructions a draw 43.46 39.95 −3.51 [−6.54, +1.19] (noise)
submit-thread M cycles a frame 30.19 28.26 −1.93 [−3.91, +1.54] (noise at N = 3)
frame p50 / p95 28.03 / 32.25 28.05 / 32.21 +0.02 / −0.04 (noise)
GPU span p50 25.94 25.89 noise
  • Per-frame totals move with each run's scene and timing (this route runs on the wall clock). So the per-draw rates are the cleaner measure, and the cycles a draw drop clearly, by about 5 %.
  • That's in line with your M4 result (about −4 % instructions and −5 % submit CPU).
  • Frame time doesn't change here: this route is GPU-bound on the M2 (frame ≈ GPU span + about 2 ms).
  • No new red lines. The picture is unchanged (shots at frames 2520, 6120 and 11520 on both sides).

seedds added a commit that referenced this pull request Nov 13, 2026
…noted, not all 16,384 slots

gw_commit read every slot of the storage-write page set (g_sbwp, 16,384 slots, 256 KB) to find the
current submission's pages whenever it had one; Ocarina of Time commits about 30 submissions a frame.
The set now lists the slots a submission takes (g_sbw_list) and the commit reads only those. Which
pages enter the running-write tables (g_gw, g_gwx) is unchanged, and their order does not change the
tables (two pages meeting in a slot mark it shared either way).

Ocarina of Time, Kokiri Forest deck (M4 Pro): 42.8-43.9k -> 38.7-39.3k submit-thread instructions
per draw; PR #2458's measurement of the same list on an M4 Air: gw_commit 12 -> 2.6 us a commit.
Frame time unchanged where the game waits for the GPU. #2458's slot change is not part of this.

t_sbw gains checks of the list and the commit.

Co-authored-by: Stan Bono <3304418+stan69b@users.noreply.github.com>
Co-authored-by: Puca <65235669+PucaVaz@users.noreply.github.com>
@seedds

seedds commented Nov 13, 2026

Copy link
Copy Markdown
Owner

The slot list part of this PR (g_sbw_new: gw_commit walks only the slots this submission took) landed on main as f63ecbd, through #2543, which made the same change. You are credited as co-author, and your gw_commit timing (12 → 2.6 µs a commit) is cited in the reference entry. The gw_slot change (slots by 256 MB region) is not part of it: it changes which pages share a slot, and its benefit has not been measured on its own. If you would like it landed, please rebase this PR onto main with only that part and an A/B of it alone against the list (gw_commit µs a commit and submit CPU a frame), plus a game whose GPU writes span several 256 MB regions.

iDams pushed a commit to iDams/nxnative that referenced this pull request Nov 13, 2026
…noted, not all 16,384 slots

gw_commit read every slot of the storage-write page set (g_sbwp, 16,384 slots, 256 KB) to find the
current submission's pages whenever it had one; Ocarina of Time commits about 30 submissions a frame.
The set now lists the slots a submission takes (g_sbw_list) and the commit reads only those. Which
pages enter the running-write tables (g_gw, g_gwx) is unchanged, and their order does not change the
tables (two pages meeting in a slot mark it shared either way).

Ocarina of Time, Kokiri Forest deck (M4 Pro): 42.8-43.9k -> 38.7-39.3k submit-thread instructions
per draw; PR seedds#2458's measurement of the same list on an M4 Air: gw_commit 12 -> 2.6 us a commit.
Frame time unchanged where the game waits for the GPU. seedds#2458's slot change is not part of this.

t_sbw gains checks of the list and the commit.

Co-authored-by: Stan Bono <3304418+stan69b@users.noreply.github.com>
Co-authored-by: Puca <65235669+PucaVaz@users.noreply.github.com>
(cherry picked from commit f63ecbd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants