[lunks]

What and why

Ocarina of Time (Switch 2, 0400486024DEC000, v131072; profile in #1299) hangs the GPU, and in longer sessions floors go missing. Reports: #1265 (the red line eight minutes in, floors missing), #1326 (its first hang, kIOGPUCommandBufferCallbackErrorHang with a GPU restart for an MMU interrupt; both of its logs have the red line), #1347 (forcing the old table size gives the hang).

Cause: the raw global memory pool table had a fixed 4,096 entries. The game's save load makes 4,097 to 4,522 pools. The host left the highest ones out ([RED] raw global memory: N more memory pools than the 4095 entries of the pool table) and shaders read 0 for addresses in them. agl's task shader 99280b7b8aae78df (and 47f9b38490dcf9a2) reads both ends of a range through raw loads and loops (min(end - start, 256) - 1) >> 5 times, so with both ends 0 it loops 2^27 - 1 times. DM_NVN_HANGLOG named both in the two segments that hung. The lookup itself stays bounded; the zeros reach the game's arithmetic, so every pool has to stay in the table.

Fix: the host sizes the table to its pools, a power of two of at least 4,096 entries up to 2^20, with the count in entry 0's delta word (gmem_map_gpu). The translator's binary search loops log2(count) times from that count (GlobalToStorage.TryGenerateRawStorageOp). The table is rebuilt when pools come or go, which also shrinks it. The red line for pools still left out gives the first address and the size. The translator is rebuilt from these sources and the chain version changes.

Fixes the hang the overflow caused. Does not cover: #1326's second run, which ended in GPU write faults at 15.5 GB of GPU allocations; nor the hang dacONJA reports on #1326 with the grown table in (4,098 ranges, no overflow line, hang several minutes later, GPU about 32 ms a frame), which is still open; nor the memory growth in #1347.

stan69b's oot-b-wip branch (d7f5b40) has the same design; it has no PR, and they said on #1326 to go ahead with ours. Thanks to them for the comparison. Our comment there: #1326 (comment)

Reference entry: More memory pools than the raw pool table had entries.

Checked

• Tests: t_pool_chunks (85 checks: 5,000 pools, every address found as the shader finds it, the table is the least again once they go), t_gmem_raw (new, 61 checks: the translated load on the GPU against the host's table, pools past entry 4,095, 0 outside any pool), test_dmsh_ampere_mem.py (new RawGlobal), every test_dmsh_*.py with the rebuilt translator, test_dmsh_abi.py, test_nsp_chain.py, test_docs.py. lint_c.py tidy on nvn_mtl.m and the two tests: no finding above the baseline. The rest of the Metal suite (make -C dm/gfx/tests run) was not run.
• Games run: Ocarina of Time v131072 on our stack, which carries the same fix in another form (a hint per 2 MiB chunk and a write log with a bit per entry, as our base differs from main). Before: 3 of 8 save loads wedged the GPU and the overflow line was in every run. After: 3 save loads clean and about 15 minutes of play with no hang and no overflow line. This branch itself was not played: its shaders need a full rebuild.
• Devices and OS: Apple M5 Pro, macOS 27.
• Chain change: the translator rebuilt from this branch's sources (dotnet publish, the 5 dlls copied, tool_lock.py --update-lock, dmsh_chain.py --header). Every game's shaders are rebuilt; only shaders with raw global memory change their MSL (Donkey Kong Bananza has two), and I did not compare other games' MSL. regress.py was not run.

Docs

• The finding is in docs/reference/shaders.md and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change
• No game files, keys or data extracted from a game in the change or the attachments

[PucaVaz]

Tested in game on a second machine: Apple M4, 24 GB, macOS 27.2, Xcode 27.2, Ocarina of Time v131072 from a NUP. This is the earlier commit 574a38f (fix/gmem-pool-table), not this PR's rebuilt head.

A new game from the title, through the prologue and the owl, into the first 3D scene:

11.3 s   raw global memory: pool table of 2131 ranges in 4096 entries (pool generation 2139), rebuilt whenever a pool comes or goes
396.0 s  raw global memory: the pool table grows to 8192 entries for 4104 ranges   (near frame 13,200)

• The pools peaked at 4,302, past the old table's 4,095.
• No more pool ranges line, no command buffer error (Hang, PageFault, SubmissionsIgnored), no FATAL.
• The run went on for 165 s and about 3,000 frames after the table grew (16,200 frames, 561 s in all). Then my own test harness stopped it, because macOS had reported memory pressure for 120 s (the game was at 9.9 GB resident, 8.9 GB GPU allocated). It was not a hang.

One non-blocking note from reading the diff: the table is still a new buffer on every rebuild, whenever a pool comes or goes, so at about 4,300 pools that is a 256 KB allocation plus a residency commit each time. A small ring of reused table buffers, each reused once its fence has passed, would avoid that churn. It isn't needed for this fix.

Drafted with Claude; reviewed by me.

[PucaVaz]

A longer run on the same build (574a38f, M4 24 GB), this time into gameplay: a new game through the opening until Link is controllable in his house, where the game autosaved.

• The table grew to 8192 entries at 351 s (4,104 ranges), shrank once pools were freed, and grew again at 571 s (4,108 ranges).
• Memory pools peaked at 4,543 in Link's house.
• No more pool ranges line, no command buffer error (Hang, PageFault, SubmissionsIgnored), no FATAL.
• The run ended at 597 s on my test harness's own 12 GB footprint limit, not a hang. In Link's house the game was at 11.6 GB footprint with 10.5 GB GPU allocated, and 12.8 GB at the stop.

Then the save load itself, the case that wedged in your runs. Same build, "Kokiri Forest, Autosave" loaded through Continue → Load, in handheld mode at upscale 2.0:

• The table grew to 8192 entries for 4,100 ranges at 78.6 s, during the load. Pools went from 3,730 to 4,523, peaking at 4,557 in Link's house.
• About 130 s of gameplay with walking. No overflow line, no Hang, PageFault or SubmissionsIgnored, no FATAL.
• The window was closed by hand at 216 s, with a clean exit.

Drafted with Claude; reviewed by me.

[lunks]

Thanks for testing on the M4 24 GB, including Continue → Load of the Kokiri Forest autosave, the case that wedged for us. On the ring of table buffers: I'd rather keep this PR to the table growth and look at the ring as a follow-up if rebuilding the table shows up in a profile.

[deanpektas27]

Tested on a third machine, and a note on combining this with #1299, which changes the same table.

Setup: Apple M3 Max, 36 GB, macOS 27.0.1, Xcode 27.2. Ocarina of Time v131072 from a NUP, Mac build (--platform mac), windowed, Continue → Load of the Kokiri Forest autosave. The game runs were on my local integration branch: main plus #1299 and 16 other open Ocarina PRs (#1282, #1284, #1285, #1300, #1312, #1320, #1331, #1332, #1343, #1345, #1353–#1357, #1359).

Before (without this PR): 34 s after the load, on walking out of Link's house, [RED] raw global memory: 13 more memory pools than the 4095 entries of the pool table, then kIOGPUCommandBufferCallbackErrorHang, InnocentVictim, SubmissionsIgnored and the GPU-lost alert. Same as #1326's first run.

After (this PR merged as below): the pool table grows to 8192 entries for 4101 ranges right after the load. A 12-minute session (716 s, 20,334 frames, forest and Link's house) had no hang and no command buffer errors. Pools peaked at 4,660. GPU allocations stayed between 11 and 15 GB without a trend and peaked once at 16.3 GB, with no write faults (#1326's second run faulted at 15.5 GB).

Merging with #1299: the two conflict in gmem_map, GlobalToStorage.TryGenerateRawStorageOp, dmsh_abi.h and the tests. #1299 sizes two more things to 4,096 entries, and both break once the table grows:

• Chunk hints. Word 6 of slot chunk % 4096 packs a 12-bit entry index with a 20-bit chunk tag, so an entry past 4,095 cannot be named. In the merge, hints stay in the first 4,096 slots, but word 6 holds the whole entry index and word 7 the 32-bit chunk tag (valid for addresses below 2^53; above that the shader searches). The shader takes a hint whose tag matches its chunk; otherwise it runs this PR's search loop from the count in entry 0's word 4.
• Write-log bitmap. It has DMSH_GMEM_MAP_ENTRIES / 32 words, one bit per entry, so a store through an entry past 4,095 would set a bit outside the log buffer. In the merge, gml_take sizes each log to the table it is taken for (gml_words(map)), releases a free log that is smaller than that, and gml_read walks that many words.

Branch: deanpektas27/nxnative:oot/pr1299-pr1360 is #1299's head merged with this PR's head, with only that resolution on top. It also brings in the 10 main commits #1299 is behind. The translator was rebuilt from that source (.NET SDK 10.0.403), giving chain 02857393f603. The merge commit is the one to review: deanpektas27@7109760

Checked on that branch: test_dmsh_ampere_mem.py (65, including RawGlobal), test_nsp_chain.py, test_xlate_anycpu.py and test_docs.py (31), t_pool_chunks (94), t_gmem_raw (61) and the full make -C dm/gfx/tests run suite all pass. Three new t_pool_chunks checks: a write-log bit for an entry past 4,095 resolves to its pool, the log has one bit per entry of the grown table, and hints in the grown table name the entry the search finds. t_gmem_raw runs the translated shader on the GPU against an 8,192-entry table; its large-pool probe takes the hint path and the small pools take the search. On the integration branch, t_image_usage's "RGBA8 read sees preserved clear bits and the atomic OR" check is flaky with or without these changes (2 of 40 runs failed without them, with a shader build loading the GPU).

This only matters for whichever of #1299 and #1360 merges second; I'm happy to rebase the merge onto that one. #1394, #1404 and #1417 also change gmem_map_gpu and are based on main. Whichever of them lands alongside #1299 needs the same check: no hint index or write-log bit may assume 4,096 entries.

[LucyferDev]

This PR and #1299 (at ee9b776) both change the raw global memory pool lookup, and they don't merge as they stand. Anyone who tests Ocarina of Time on #1299 currently has to leave this fix out. On a 24 GB M4 Pro, that build hit this PR's overflow line at 6:54 into play, hung the GPU at about 9:27, and ended in a kernel panic (report in #1265).

Text conflicts (merging eaa7a1d into #1299's head, both on current main):

• GlobalToStorage.cs: 3 hunks
• nvn_mtl.m: 3 hunks
• dmsh_abi.h, t_pool_chunks.m, test_dmsh_ampere_mem.py: 1 hunk each
• plus the generated chain files

Design conflict. #1299 adds a 2 MiB chunk hint in front of the binary search, and its encoding assumes a table of exactly 4,096 entries:

• Host (nvn_mtl.m, gmem map build): hint = chunk * DMSH_GMEM_MAP_ENTRIES + k, stored in word 6 of entry chunk & (DMSH_GMEM_MAP_ENTRIES - 1). So the entry index k gets 12 bits, and the chunk tag sits above them.
• Shader (GlobalToStorage.cs): indexMask = (1 << GmemMapLog2) - 1 with GmemMapLog2 = 12. The tag check is hint >> GmemMapLog2 == chunk, and the fallback is the fixed 12-step search from 1 << (GmemMapLog2 - 1).

This PR makes the table a power of two from 4,096 up to 2^20 entries, with the count in entry 0's delta low word, and makes the search follow that count. As I read the two together:

1. The search: [WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299's fixed 12 steps reach only the first 4,096 entries, so a merge has to take this PR's count-driven loop.
2. The hint: an entry index of 4,096 or more doesn't fit in the 12-bit field. Its hint then carries a tag that never equals the chunk, so the shader rejects it and falls back to the search. That's correct, but the pools past 4,095 never get a hint. Ocarina has 4,097–4,522 pools after its save load, so the hint stops helping exactly where this PR matters.

A possible combined layout. Words 6 and 7 of an entry are both unused under this PR:

• word 6: the entry index, up to 20 bits
• word 7: the chunk tag
• the hint slot is chunk & (count - 1), with the count from GmemMapCountWord; the host writes the same.

That keeps #1299's NDVE speed-up (title movie 47.6 → 16.67 ms) for any table size. Whether the hint needs to cover more than the first count slots is the authors' call.

I haven't built or played a merged version: the above is from reading both diffs, and from the run in #1265, which has #1299 without this PR. I can test a combined branch on the same Mac if one of you pushes it.

[deanpektas27]

@LucyferDev a combined branch is already up, from my comment above: deanpektas27/nxnative:oot/pr1299-pr1360 (merge commit 7109760). Its parents are the heads you read, #1299 at ee9b776 and this PR at eaa7a1d. It would help if you ran it on your 24 GB M4 Pro, since your #1299-only build is the one that hung at 9:27 and panicked.

It uses your layout with one difference:

• Same: word 6 holds the whole entry index, word 7 the 32-bit chunk tag, and the search follows the count in entry 0's word 4. The write log is also sized to the table (gml_words), since [WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299's bitmap has one bit per entry and a store through an entry past 4,095 would land outside it.
• Different: the hint slot is still chunk % 4096, not chunk & (count - 1). Pools past 4,095 do get hints, because the index has a whole word, but there are only 4,096 slots whatever the table's size.

On whether the hints should cover count slots, I don't have numbers yet. In my 12-minute session the pools spanned 11 to 16 GB. That is more 2 MiB chunks than 4,096 slots hold (8 GB), so some chunks must lose their hint to another and take the search. Following the count would be a small change: the host writes slot chunk & (cap - 1), and the shader masks with the count it already reads for the search. I can push it to the same branch if the authors prefer it, or first measure how many chunks lose their hint in Ocarina.

#1427 (table words rewritten at commit) touches the same table: it repoints argument blocks at gmem_map_gpu() at each commit. Its new test t_gmem_words looks for a pool in only the first DMSH_GMEM_MAP_ENTRIES entries, so with this PR it would miss a pool past entry 4,095.

[seedds]

Thank you, and thanks to everyone whose reports fed into it. The raw pool table overflow is fixed on main (landed on main in b1d7c70 and 6121fd7), so I'm closing this. Final checks on the rebased tree (full shader build, walks, regression) are still running, so please report anything you see.