Standalone Metal probes: heap aliasing, render-target sizes, mesh index ranges, mesh pipeline archive, mesh/task sweep, task-shader hang by nafields · Pull Request #1583 · seedds/nxnative

nafields · 2026-11-06T22:31:27Z

What and why

Several of Ocarina of Time's open questions were being answered with 16 GB game runs, and on 16 GB a run that goes wrong can reset the GPU or panic the machine. Three examples: which render targets can share heap memory, why some task shaders hang the GPU, and whether mesh shaders overflow their stage memory. These probes ask Metal one question each, with no game. The worst any of them does is time out one command buffer (about 22 ms of GPU time).

dm/gfx/tests/probes/, one file per question, with its own Makefile (make; make run runs the four that need no input). Its README gives each probe's question, input and result. docs/agents/04-graphics.md §8 links it. 05-debugging.md would be the natural place, but it is at its 6,000-word limit.

Results on an M3:

Probe	Result
`heap_alias`	Textures at one placement-heap offset share a layout only with the same width, height, bytes per pixel and compression. With those equal, the format doesn't matter (RGBA8, R32Uint, R32Float and Depth32Float all read back 100%). A size, bpp or compression change scrambles the data (0–3% of words match).
`rt_sizes`	At 2048×1152, 855 dynamic-resolution steps give 21 distinct RGBA8 allocation sizes, from `heapTextureSizeAndAlign` rounding. A texture free list keyed on the allocation size would reuse across steps; one keyed on width × height almost never would.
`mesh_index_range`	Mesh primitive indices past the declared vertex count (200 past 64, or all 255) don't hang: 1,000 groups complete in under 5 ms.
`mesh_archive`	A mesh pipeline built through `MTL4Compiler` is stored in a Metal 4 archive and found again (0.18 ms, against a 36 ms build). The host builds mesh pipelines with `newRenderPipelineStateWithMeshDescriptor`, which bypasses the archive.
`mesh_sweep.sh` (`mesh_sweep` + `mesh_guard.py`)	One draw of every translated mesh/task shader on zeroed inputs, under shader validation, plus a guard band. Validation doesn't check static threadgroup arrays: a deliberate overflow wasn't reported. Ocarina: 387 of 500 meshes overflow their stage array before the primitive-memory fix (#1575), 0 after.
`task_hang`	The 8 task shaders that hang the GPU do so on a raw pool-table miss of their range pointer: the loop runs `uint(0-1)>>5` = 134,217,727 times. They complete when the pool is in the table, or when the bind-time pointer site entry is filled.
`raw_sites`	The bind-time pointer site path on the translator's real output: 11 of 11 cases read from the buffer they should.
`rt_reclaim_sim.py`	Render-target memory simulated from a write/release log. On Ocarina, frames 0–4,000, reclaiming targets that are dead by overwrite cuts render targets from about 3.5 GB to about 1.2 GB.

task_hang case C and raw_sites need the bind-time pointer translator (PR to follow). With main's translator, case C hangs like case B.

Checked

Tests: make and make run in dm/gfx/tests/probes; lint_python.py (371 findings, 0 new); test_docs.py (OK).
Games run: none. The game results in the table come from earlier Ocarina of Time (0400486024DEC000) builds and runs.
Devices and OS: M3, macOS 26.
No host, chain or tool change, so no regress.py run. The probes aren't in make -C dm/gfx/tests run or CI.

Docs

Linked from docs/agents/04-graphics.md §8; results in the probes README
No chain tool changed
No game files, keys or data extracted from a game in the change or the attachments

…carina of Time's Kokiri load no longer piles up 3.4 GB of render targets Symptom: at Ocarina of Time's Kokiri Forest load on a 16 GB M2 Pro, the residency set grew from 1.6 to 6.6 GB in about 30 s, 3.2 GB of it render targets (72 → 1,068 textures, 31 → 313 descriptors); swap followed wired (wired 3.0 → 9.8 GB while swap rose 1.1 → 3.2 GB and available memory fell to 15%), and the runs reached the +4 GB swap limit. Cause: the game keeps its render targets as texture objects over one 305 MB virtual pool, reused by overwrite: at f1,200, 781 live targets (2,788 MB of sizes) sat over 74 MB of storage, and 716 of the 828 live ones (2,697 MB) had their whole storage range written by other targets since their own last write. On the console those cost the pool's bytes; here each kept a private Metal texture, and the texture-ID registry (treg_tex) made one for every registered ID, dead or not. Fix (behind DM_NVN_RT_RECLAIM=1): a shared texture is dead once other textures' render passes have covered its whole storage range in its pool since its own last write. Coverage counts only passes whose draws and clears (viewport ∩ scissor) covered their whole attachment; a partial pass covers nothing. A dead texture leaves the sharing table with its serials' mappings; its Metal texture goes to a free list keyed on the full Metal descriptor, is handed to the next texture of that descriptor after the fence of its last use and zero-filled as a new one is, and is released after 15 idle frames. A reclaimed serial's texture ID reads as an empty slot until a render pass writes it again, which notes the serial so its IDs are taken again. Never reclaimed: IMAGE textures, copy / clear destinations, readback sources, anything a pool copy overlaps, sparse pools, textures filled from pool bytes, and textures read before their next write after more than 4 deaths (luminance and bloom chains: 429 textures, 324 MB in a run to f4,978). Checked: Ocarina of Time on an M2 Pro 16 GB, Kokiri Forest route to f7,500. Render targets in the set stayed at 1.1-1.2 GB instead of 3.5 GB through the load, peak wired 7.6-8.8 GB, and swap grew by 0.05-2.4 GB. A third of attachment passes were partial and cover nothing. Shots at f1,500 and f2,500 match a tree without it in scene and brightness (127.4 / 96.3 against 128.9 / 96.1). The free-list key and the aliasing rule rest on seedds#1583's heap_alias and rt_sizes probes: a texture placed over another's memory reads its bytes only with the same width, height, bytes per pixel and compression, and a single dynamic-resolution step scrambles it. Those game runs were on a tree that also carried PRs 1417 and 1484 and the bind-time pointers. Rebased on main, with the switch off by default: host build, host tests, t_alias_takeover (28), t_image_alias (329), test_docs. No unit test drives the reclaim itself yet, and regress.py check was not run (only Ocarina of Time is available here). Reference entry and game note line added; env_switches.tsv lists the switch.

…ex ranges, the mesh pipeline archive, a mesh and task shader sweep, and the task-shader hang, each without the game Symptom: Ocarina of Time's open questions (which render targets can share heap memory, why task shaders hang the GPU, whether mesh shaders overflow their stage memory) were being answered with 16 GB game runs, and a game run that goes wrong can reset the GPU or panic the machine. Cause: there was no way to ask Metal one question at a time. Every answer needed the whole game. Fix: dm/gfx/tests/probes, one file per question, built by its own Makefile (make; make run for the four that need no input). Each probe's question, input and result on an M3 are in its README: - heap_alias: textures at one placement-heap offset share a layout only with the same width, height, bytes per pixel and compression; the pixel format does not matter. - rt_sizes: 855 dynamic-resolution steps at 2048x1152 give 21 distinct RGBA8 allocation sizes. - mesh_index_range: mesh indices past the vertex count do not hang the GPU. - mesh_archive: an MTL4Compiler mesh pipeline is stored in and found in a Metal 4 archive (0.18 ms against a 36 ms build); the host's mesh pipelines bypass it. - mesh_sweep.sh with mesh_sweep and mesh_guard.py: one draw of every translated mesh/task shader on zeroed inputs under shader validation, plus a guard band for threadgroup overflows, which validation does not check. Ocarina: 387 of 500 meshes overflow before the primitive-memory fix, 0 after. - task_hang: the 8 hanging task shaders loop 134,217,727 times when the raw pool table misses their range pointer, and complete when the bind-time site entry is filled. - raw_sites: the bind-time site path, 11 of 11 cases. - rt_reclaim_sim.py: render-target memory from a write/release log, 3.5 GB to about 1.2 GB with reclaim by overwrite. 04-graphics.md section 8 links them (05-debugging.md is at its word limit). Checked: make and make run on an M3 (macOS 26); lint_python.py (0 new); test_docs.py. No host, chain or tool change, so no regress run.

…carina of Time's Kokiri load no longer piles up 3.4 GB of render targets Symptom: at Ocarina of Time's Kokiri Forest load on a 16 GB M2 Pro, the residency set grew from 1.6 to 6.6 GB in about 30 s, 3.2 GB of it render targets (72 → 1,068 textures, 31 → 313 descriptors); swap followed wired (wired 3.0 → 9.8 GB while swap rose 1.1 → 3.2 GB and available memory fell to 15%), and the runs reached the +4 GB swap limit. Cause: the game keeps its render targets as texture objects over one 305 MB virtual pool, reused by overwrite: at f1,200, 781 live targets (2,788 MB of sizes) sat over 74 MB of storage, and 716 of the 828 live ones (2,697 MB) had their whole storage range written by other targets since their own last write. On the console those cost the pool's bytes; here each kept a private Metal texture, and the texture-ID registry (treg_tex) made one for every registered ID, dead or not. Fix (behind DM_NVN_RT_RECLAIM=1): a shared texture is dead once other textures' render passes have covered its whole storage range in its pool since its own last write. Coverage counts only passes whose draws and clears (viewport ∩ scissor) covered their whole attachment; a partial pass covers nothing. A dead texture leaves the sharing table with its serials' mappings; its Metal texture goes to a free list keyed on the full Metal descriptor, is handed to the next texture of that descriptor after the fence of its last use and zero-filled as a new one is, and is released after 15 idle frames. A reclaimed serial's texture ID reads as an empty slot until a render pass writes it again, which notes the serial so its IDs are taken again. Never reclaimed: IMAGE textures, copy / clear destinations, readback sources, anything a pool copy overlaps, sparse pools, textures filled from pool bytes, and textures read before their next write after more than 4 deaths (luminance and bloom chains: 429 textures, 324 MB in a run to f4,978). Checked: Ocarina of Time on an M2 Pro 16 GB, Kokiri Forest route to f7,500. Render targets in the set stayed at 1.1-1.2 GB instead of 3.5 GB through the load, peak wired 7.6-8.8 GB, and swap grew by 0.05-2.4 GB. A third of attachment passes were partial and cover nothing. Shots at f1,500 and f2,500 match a tree without it in scene and brightness (127.4 / 96.3 against 128.9 / 96.1). The free-list key and the aliasing rule rest on seedds#1583's heap_alias and rt_sizes probes: a texture placed over another's memory reads its bytes only with the same width, height, bytes per pixel and compression, and a single dynamic-resolution step scrambles it. Those game runs were on a tree that also carried PRs 1417 and 1484 and the bind-time pointers. Rebased on main, with the switch off by default: host build, host tests, t_alias_takeover (28), t_image_alias (329), test_docs. No unit test drives the reclaim itself yet, and regress.py check was not run (only Ocarina of Time is available here). Reference entry and game note line added; env_switches.tsv lists the switch. On this branch: a render target placed in its GPU-only pool's heap (pool_heap_texture, PR seedds#1453) is excluded from the reclaim, since the free list would hand a texture placed at one pool offset to a texture at another. (cherry picked from commit 4b231ac)

…carina of Time's Kokiri load no longer piles up 3.4 GB of render targets Symptom: at Ocarina of Time's Kokiri Forest load on a 16 GB M2 Pro, the residency set grew from 1.6 to 6.6 GB in about 30 s, 3.2 GB of it render targets (72 → 1,068 textures, 31 → 313 descriptors); swap followed wired (wired 3.0 → 9.8 GB while swap rose 1.1 → 3.2 GB and available memory fell to 15%), and the runs reached the +4 GB swap limit. Cause: the game keeps its render targets as texture objects over one 305 MB virtual pool, reused by overwrite: at f1,200, 781 live targets (2,788 MB of sizes) sat over 74 MB of storage, and 716 of the 828 live ones (2,697 MB) had their whole storage range written by other targets since their own last write. On the console those cost the pool's bytes; here each kept a private Metal texture, and the texture-ID registry (treg_tex) made one for every registered ID, dead or not. Fix (behind DM_NVN_RT_RECLAIM=1): a shared texture is dead once other textures' render passes have covered its whole storage range in its pool since its own last write. Coverage counts only passes whose draws and clears (viewport ∩ scissor) covered their whole attachment; a partial pass covers nothing. A dead texture leaves the sharing table with its serials' mappings; its Metal texture goes to a free list keyed on the full Metal descriptor, is handed to the next texture of that descriptor after the fence of its last use and zero-filled as a new one is, and is released after 15 idle frames. A reclaimed serial's texture ID reads as an empty slot until a render pass writes it again, which notes the serial so its IDs are taken again. Never reclaimed: IMAGE textures, copy / clear destinations, readback sources, anything a pool copy overlaps, sparse pools, textures filled from pool bytes, and textures read before their next write after more than 4 deaths (luminance and bloom chains: 429 textures, 324 MB in a run to f4,978). Checked: Ocarina of Time on an M2 Pro 16 GB, Kokiri Forest route to f7,500. Render targets in the set stayed at 1.1-1.2 GB instead of 3.5 GB through the load, peak wired 7.6-8.8 GB, and swap grew by 0.05-2.4 GB. A third of attachment passes were partial and cover nothing. Shots at f1,500 and f2,500 match a tree without it in scene and brightness (127.4 / 96.3 against 128.9 / 96.1). The free-list key and the aliasing rule rest on seedds#1583's heap_alias and rt_sizes probes: a texture placed over another's memory reads its bytes only with the same width, height, bytes per pixel and compression, and a single dynamic-resolution step scrambles it. Those game runs were on a tree that also carried PRs 1417 and 1484 and the bind-time pointers. Rebased on main, with the switch off by default: host build, host tests, t_alias_takeover (28), t_image_alias (329), test_docs. No unit test drives the reclaim itself yet, and regress.py check was not run (only Ocarina of Time is available here). Reference entry and game note line added; env_switches.tsv lists the switch. On this branch: a render target placed in its GPU-only pool's heap (pool_heap_texture, PR seedds#1453) is excluded from the reclaim, since the free list would hand a texture placed at one pool offset to a texture at another. (cherry picked from commit 4b231ac) (cherry picked from commit 3cde45e)

…carina of Time's Kokiri load no longer piles up 3.4 GB of render targets Symptom: at Ocarina of Time's Kokiri Forest load on a 16 GB M2 Pro, the residency set grew from 1.6 to 6.6 GB in about 30 s, 3.2 GB of it render targets (72 → 1,068 textures, 31 → 313 descriptors); swap followed wired (wired 3.0 → 9.8 GB while swap rose 1.1 → 3.2 GB and available memory fell to 15%), and the runs reached the +4 GB swap limit. Cause: the game keeps its render targets as texture objects over one 305 MB virtual pool, reused by overwrite: at f1,200, 781 live targets (2,788 MB of sizes) sat over 74 MB of storage, and 716 of the 828 live ones (2,697 MB) had their whole storage range written by other targets since their own last write. On the console those cost the pool's bytes; here each kept a private Metal texture, and the texture-ID registry (treg_tex) made one for every registered ID, dead or not. Fix (behind DM_NVN_RT_RECLAIM=1): a shared texture is dead once other textures' render passes have covered its whole storage range in its pool since its own last write. Coverage counts only passes whose draws and clears (viewport ∩ scissor) covered their whole attachment; a partial pass covers nothing. A dead texture leaves the sharing table with its serials' mappings; its Metal texture goes to a free list keyed on the full Metal descriptor, is handed to the next texture of that descriptor after the fence of its last use and zero-filled as a new one is, and is released after 15 idle frames. A reclaimed serial's texture ID reads as an empty slot until a render pass writes it again, which notes the serial so its IDs are taken again. Never reclaimed: IMAGE textures, copy / clear destinations, readback sources, anything a pool copy overlaps, sparse pools, textures filled from pool bytes, and textures read before their next write after more than 4 deaths (luminance and bloom chains: 429 textures, 324 MB in a run to f4,978). Checked: Ocarina of Time on an M2 Pro 16 GB, Kokiri Forest route to f7,500. Render targets in the set stayed at 1.1-1.2 GB instead of 3.5 GB through the load, peak wired 7.6-8.8 GB, and swap grew by 0.05-2.4 GB. A third of attachment passes were partial and cover nothing. Shots at f1,500 and f2,500 match a tree without it in scene and brightness (127.4 / 96.3 against 128.9 / 96.1). The free-list key and the aliasing rule rest on seedds#1583's heap_alias and rt_sizes probes: a texture placed over another's memory reads its bytes only with the same width, height, bytes per pixel and compression, and a single dynamic-resolution step scrambles it. Those game runs were on a tree that also carried PRs 1417 and 1484 and the bind-time pointers. Rebased on main, with the switch off by default: host build, host tests, t_alias_takeover (28), t_image_alias (329), test_docs. No unit test drives the reclaim itself yet, and regress.py check was not run (only Ocarina of Time is available here). Reference entry and game note line added; env_switches.tsv lists the switch. On this branch: a render target placed in its GPU-only pool's heap (pool_heap_texture, PR seedds#1453) is excluded from the reclaim, since the free list would hand a texture placed at one pool offset to a texture at another. (cherry picked from commit 4b231ac) (cherry picked from commit 3cde45e) (cherry picked from commit 9a04e57)

github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation labels Nov 6, 2026

nafields force-pushed the metal-probes branch from 8500122 to 53eaaad Compare November 7, 2026 04:37

PromethOps mentioned this pull request Nov 7, 2026

A GPU-only pool's render targets share one placement heap: Ocarina of Time's 3 GB of render targets fit in 0.7 GB #1453

Closed

2 of 3 tasks

Navigation Menu

Repository navigation

Standalone Metal probes: heap aliasing, render-target sizes, mesh index ranges, mesh pipeline archive, mesh/task sweep, task-shader hang - #1583

Standalone Metal probes: heap aliasing, render-target sizes, mesh index ranges, mesh pipeline archive, mesh/task sweep, task-shader hang#1583
nafields wants to merge 1 commit into
seedds:mainfrom
nafields:metal-probes

nafields commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nafields commented Nov 6, 2026

What and why

Checked

Docs

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants