Mesh stage memory reserves the primitive index bytes for every block the threads write by nafields · Pull Request #1575 · seedds/nxnative

nafields · 2026-11-06T21:57:13Z

What and why

Symptom: white blotches over Kokiri Forest's foliage, walkway and ground in Ocarina of Time. They are the house light tiles: the light-tile pass of task a5288d0a6c4971fd with meshes e8c667282ba7bd9b / d4171cf9adf0504b writes constant white through fragment ea1bf96b72e72476, and its triangles were corrupt.

Cause: the guest writes the U8 index bytes of primitive t = min(count - 1, lane + 32k) for every block. A meshlet count of 0 wraps count - 1, so all 128 lanes write. With max_p 126 the index area was 4 + 378 bytes, so lanes 126 and 127 wrote bytes 2,702 to 2,707: word 676, past the 676-word threadgroup array, into neighbouring workgroups' memory.

Change: memory() in dm/gfx/tools/dmsh_meshstage.py reserves 4 + 3 * blocks * BLOCK bytes for the index bytes. dmsh_chain.h is regenerated. Reference entry: Mesh stage memory reserves the primitive index bytes for every block; game note line added.

This does NOT fix the intermittent GPU hangs seen in buffers with these meshes; those are a separate open issue.

Checked

Tests: test_dmsh_meshstage.py (28, OK; test_mesh_memory updated, a 126-primitive case added), test_nsp_chain.py (OK), test_docs.py (OK).
Standalone: all 500 of Ocarina's mesh shaders run on zeroed inputs, each with a guard band past its stage array (Metal shader validation does not check static threadgroup arrays; a deliberate one-past-the-end write was not reported). Before the fix 387 meshes wrote past the array, every max_p 126 mesh; after it, 0. Meshes with max_p a multiple of 32 were never affected.
Game run: Ocarina of Time (0400486024DEC000), Kokiri Forest, M2 Pro. The blotches are gone and the house doorways and windows glow as intended. Nothing was missing compared with skipping the pass. Only Ocarina was checked in the game.
Chain change, so every game's shaders rebuild. Mesh-shader games in main: Yoshi and the Mysterious Book 0400245021D2C000, Ocarina 0400486024DEC000, Mario Kart World 0400C3F00006E000, Splatoon Raiders 0400380020056000. scripts/regress.py check was not run. regress.py impact origin/main..HEAD: Breath of the Wild, Pokemon Scarlet and Tears of the Kingdom reached through the broad chain-version rule (shader libraries, NVN); Hades not affected; 146 profiles have no usage manifest. dmsh_meshstage.py is not mapped by impact_map.tsv.

Docs

The finding is in its game note and its docs/reference/shaders.md entry, and python3 dm/tools/docs_symptoms.py was run
A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change
No game files, keys or data extracted from a game in the change or the attachments

ycarogabriel-lgtm · 2026-11-07T02:44:31Z

Integration note for this PR with #1590: both change memory() in dmsh_meshstage.py, and their fixes need to be combined rather than choosing either side of the conflict. On a local integration based on main e2b76d4e, with #1575 79b34d15 and #1590 d3536794, I kept #1590's separate vertex and primitive attribute block counts and reserved index space for max(v_blocks, p_blocks) * BLOCK primitives as in this PR. I used the topology's verts count for bytes per primitive, following #1590's point/line support.

test_dmsh_meshstage.py passed all 29 tests on the combined tree, including the 126-primitive case and #1590's separate attribute-block check. This is an integration/unit-test result on Apple M1, macOS 26.6.2; I have not rebuilt the game's shaders or checked the picture or guard bands in game on this combination. The point/line index reservation in particular still needs runtime or focused fixture coverage.

…the threads write: lanes 126 and 127 of a max_p 126 mesh wrote past the stage array Symptom: white blotches over Kokiri Forest's foliage, walkway and ground in Ocarina of Time (the house light tiles). The light-tile pass of task a5288d0a6c4971fd with meshes e8c667282ba7bd9b / d4171cf9adf0504b writes constant white through fragment ea1bf96b72e72476; its triangles were corrupt. Cause: the guest writes the U8 index bytes of primitive t = min(count - 1, lane + 32k) for every block. A meshlet count of 0 wraps count - 1, so all 128 lanes write. With max_p 126 the index area was 4 + 378 bytes, so lanes 126 and 127 wrote bytes 2,702-2,707: word 676, past the 676-word threadgroup array, into neighbouring workgroups' memory. Fix: dmsh_meshstage.memory() reserves 4 + 3 * blocks * 32 bytes for the index bytes. dmsh_chain.h regenerated (chain change: every game's shaders rebuild). Reference entry, game note line, symptom index. Checked: test_dmsh_meshstage.py, test_nsp_chain.py, test_docs.py pass. All 500 of Ocarina's mesh shaders run standalone on zeroed inputs with a guard band past the stage array: 387 meshes wrote past it before (every max_p 126 mesh), 0 after. In the game on an M2 Pro the blotches are gone and the house doorways and windows glow as intended. Does not fix the intermittent GPU hangs seen in buffers with these meshes (open). regress.py check not run; only Ocarina checked in the game.

…the threads write (upstream PR seedds#1575): Kokiri Forest's white blotches over foliage, walkway and ground From seedds/nxnative PR seedds#1575 (a7f945f): lanes 126 and 127 of a max_p 126 mesh wrote past the stage array into neighbouring workgroups' memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SzykpcKm2eZPvPjGPKx1C

…ive block the threads write: lanes 126 and 127 of a max_p 126 mesh no longer write past the stage array, and Ocarina of Time's Kokiri Forest loses its white blotches (chain 337c3caedaa7, PR #1575) Symptom: white blotches over Kokiri Forest's foliage, walkway and ground in Ocarina of Time (the house light tiles). The light-tile pass of task a5288d0a6c4971fd with meshes e8c667282ba7bd9b / d4171cf9adf0504b writes constant white through fragment ea1bf96b72e72476; its triangles were corrupt. Cause: the guest writes the U8 index bytes of primitive t = min(count - 1, lane + 32k) for each of its ceil(max_p / 32) primitive blocks. A meshlet count of 0 wraps count - 1, so all 128 lanes of a max_p 126 mesh write. The index area was 4 + 378 bytes, so lanes 126 and 127 wrote bytes 2,702-2,707: word 676, past the 676-word threadgroup array, into neighbouring workgroups' memory. Fix: dmsh_meshstage.memory() reserves 4 + 3 * ceil(max_p / 32) * 32 bytes for the index bytes. The pull request reserved them for the vertex blocks as well (blocks of max(max_v, max_p)); the meshes of 128 vertices and 64 primitives did not write past 4 + 3 * 64 before the fix, so the guest's loop goes by primitive blocks, and vertex-heavy meshes keep their size. Checks: test_dmsh_meshstage.py, test_nsp_chain.py, test_docs.py, test_dmsh_shared_store.py. The pull request ran all 500 of Ocarina's mesh shaders standalone with a guard band past the stage array (with its larger sizing): 387 wrote past it before, every max_p 126 mesh, 0 after; in the game on an M2 Pro the blotches are gone. Of Ocarina's 500 mesh shaders, the 387 of 64 vertices / 126 primitives take the pull request's size and the 102 with more vertices than primitives keep theirs; the largest is 21,652 bytes, none crosses Metal's 32 KB. Regression: Ocarina of Time only (the one mesh-shader game in the local harness; no smoke route, so the chain check alone): 42,848 of 42,848 shaders build, none newly failing, no interface mismatches. Mario Kart World, Splatoon Raiders and Yoshi rebuild with the chain and were not checked. The intermittent GPU hangs in passes with these meshes stay open. Co-authored-by: nafields <2613057+nafields@users.noreply.github.com>

seedds · 2026-11-07T06:26:19Z

Landed as d365496 (chain 337c3caedaa7), with you credited as co-author. Thank you for the diagnosis and the guard-band run.

One change: the index bytes are reserved for whole primitive blocks, 4 + 3 * ceil(max_p / 32) * 32, rather than for max(max_v, max_p) blocks. Your own guard-band numbers point that way: the meshes of 128 vertices and 64 primitives did not write past 4 + 3 * 64 before the fix, so the guest's loop goes by primitive blocks. Of Ocarina's 500 mesh shaders, the 387 with 64 vertices and 126 primitives get the size you gave them, and the 102 with more vertices than primitives keep their old one. The largest is 21,652 bytes, well under Metal's 32 KB. This also keeps the vertex and primitive block counts separate for #1590.

Checks: the mesh-stage, chain, docs and shared-store tests pass. On Ocarina, a static chain check (no smoke route): 42,848 of 42,848 shaders build, none newly failing, no interface mismatches. The primitive-block size was not run through your guard-band harness. If you still have it, a rerun on the new chain would confirm it. The GPU hangs in these passes stay open, as you noted.

…ck before the primitive blocks, with the point size ahead of the position and the topology from the control section: Ocarina of Time's Poe Sisters and Phantom Ganon paintings take their per-primitive colours again (chain 8900605baef9, from PR #1590) Symptom: a mesh shader's per-primitive outputs read as other values once it has more than 32 vertices. Ocarina of Time's Poe Sisters and Phantom Ganon paintings (ee4ef51a4fddc28e, 83c0dbc51c5481e3: 128 vertices, 64 primitives, a per-primitive colour) came out solid green or magenta (pull request #1590). A point mesh took its point size as its position's x. Cause: the wrapper read the attribute area as blocks of 32 vertices and 32 primitives side by side, each holding the vertex and then the primitive components. The guest writes all its vertex blocks first, then all its primitive blocks, each block only its own SPH's components: ee4ef51a4fddc28e (6 vertex components, 4 primitive) writes vertex v at (v / 32) * 768 + (v % 32) * 4 and primitive p at 3072 + (p / 32) * 512 + (p % 32) * 4. The SPH's point size (byte 0x35 bit 3) is a vertex block's first component: the point mesh 3a6340a04bf09797 writes the size at component 0 and the position at 1-4. Control +0x5f is a primitive's vertices (1 point, 2 line, 3 triangle); Ocarina's 5 mesh shaders with 1 are the 5 that write a point size, the other 495 have 3. Fix: dmsh_meshstage.memory and to_stage take ceil(max_v / 32) vertex blocks, then ceil(max_p / 32) primitive blocks; the point size goes to [[point_size]]; the topology and a primitive's index bytes follow +0x5f (another value stops the shader's build). The primitive index bytes stay reserved for every primitive block (PR #1575), +0x5f bytes a primitive instead of 3. The pull request's index area, +0x5f * max_p, would have undone #1575 and was not taken; nor were its other changes (reviewed on the pull request). Checks: test_dmsh_meshstage.py (30: the pull request's layout and point mesh tests, a point mesh memory test, the Metal compile of each case), test_docs.py, test_nsp_chain.py. The wrapper's reads checked against the guest's address arithmetic in the translated ee4ef51a4fddc28e and 3a6340a04bf09797. Regression: Ocarina of Time, its 500 mesh shaders translated and compiled on this chain (no other shader changes): 500 of 500 build, none refused, the largest stage memory 17,172 bytes (21,652 before). Mario Kart World, Splatoon Raiders and Yoshi not checked (no build here); the paintings not seen in the game; line topology not met. Ocarina's foliage point meshes (3a6340a04bf09797, a203347bb7bdbd03) still fail their pipelines on user(locn5), which their pixel shader reads as a texture coordinate (point sprite coordinate replacement suspected; open). Co-authored-by: ksolenk38ze <352968655+ksolenk38ze@users.noreply.github.com>

…tream landed them: Kokiri Forest's white blotches, Ocarina's Poe Sisters and Phantom Ganon paintings; compute pipelines in archives; picture shape Match screen) into switch2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SzykpcKm2eZPvPjGPKx1C

…s first write stays resident while any of them holds it: releasing one took it out of the residency set (GPU page faults, issue seedds#1326) The serials of one storage share one packaged texture (g_pkg_share). At the GPU's first write over that storage, cpu_unc_drop gives each of them the package's texture itself (tex_put), so they all hold one texture that was in no shared set. The first holder released (tex_put's ghost-ring eviction, serial_put or serial_del, through obj_drop or defer_obj_release), or the package's own release (NvPkg dealloc's res_remove), took it out of the residency set while the other serials still drew with it, and a command buffer reading it faulted. pkg_hand_over enters the handed-over texture in g_alias_set and in g_handed, a list by the package's sharing key, so releasing one holder only drops a reference (alias_has). The package's release and its re-upload's spare leave a handed-over texture resident, mtl_alias_drop_pool retires it with its pool, and tex_upgrade moves its entry. It does not enter g_alias, so a new texture object over the same storage still gets a texture of its own. DM_UNC_SHARE=0 gives back the old hand-over. Ocarina of Time v131072, Mac mini M4 16 GB, macOS 27.2, on main b2d46b8 with seedds#1590, seedds#1557, seedds#1556, seedds#1575, seedds#1522, seedds#1543 and seedds#1390: page faults at 44, 29, 24 and 15.5 minutes, with the missing textures named by DM_RES_CHECK and a diagnostic of where each left the residency set, before two of them, as 260x260 terrain tiles handed over this way and released by the ghost-ring eviction. One binary: with the change, 107 minutes and no fault (86 hand-overs); with DM_UNC_SHARE=0, a fault at 37 minutes with the same kind of tiles named. t_cpu_tex --metal handover_shared: 97 checks, 0 failed; with DM_UNC_SHARE=0, 2 failed. The other GPU suites are unchanged. (cherry picked from commit 8cbc330)

…ive block the threads write: lanes 126 and 127 of a max_p 126 mesh no longer write past the stage array, and Ocarina of Time's Kokiri Forest loses its white blotches (chain 337c3caedaa7, PR seedds#1575) Symptom: white blotches over Kokiri Forest's foliage, walkway and ground in Ocarina of Time (the house light tiles). The light-tile pass of task a5288d0a6c4971fd with meshes e8c667282ba7bd9b / d4171cf9adf0504b writes constant white through fragment ea1bf96b72e72476; its triangles were corrupt. Cause: the guest writes the U8 index bytes of primitive t = min(count - 1, lane + 32k) for each of its ceil(max_p / 32) primitive blocks. A meshlet count of 0 wraps count - 1, so all 128 lanes of a max_p 126 mesh write. The index area was 4 + 378 bytes, so lanes 126 and 127 wrote bytes 2,702-2,707: word 676, past the 676-word threadgroup array, into neighbouring workgroups' memory. Fix: dmsh_meshstage.memory() reserves 4 + 3 * ceil(max_p / 32) * 32 bytes for the index bytes. The pull request reserved them for the vertex blocks as well (blocks of max(max_v, max_p)); the meshes of 128 vertices and 64 primitives did not write past 4 + 3 * 64 before the fix, so the guest's loop goes by primitive blocks, and vertex-heavy meshes keep their size. Checks: test_dmsh_meshstage.py, test_nsp_chain.py, test_docs.py, test_dmsh_shared_store.py. The pull request ran all 500 of Ocarina's mesh shaders standalone with a guard band past the stage array (with its larger sizing): 387 wrote past it before, every max_p 126 mesh, 0 after; in the game on an M2 Pro the blotches are gone. Of Ocarina's 500 mesh shaders, the 387 of 64 vertices / 126 primitives take the pull request's size and the 102 with more vertices than primitives keep theirs; the largest is 21,652 bytes, none crosses Metal's 32 KB. Regression: Ocarina of Time only (the one mesh-shader game in the local harness; no smoke route, so the chain check alone): 42,848 of 42,848 shaders build, none newly failing, no interface mismatches. Mario Kart World, Splatoon Raiders and Yoshi rebuild with the chain and were not checked. The intermittent GPU hangs in passes with these meshes stay open. Co-authored-by: nafields <2613057+nafields@users.noreply.github.com>

…ck before the primitive blocks, with the point size ahead of the position and the topology from the control section: Ocarina of Time's Poe Sisters and Phantom Ganon paintings take their per-primitive colours again (chain 8900605baef9, from PR seedds#1590) Symptom: a mesh shader's per-primitive outputs read as other values once it has more than 32 vertices. Ocarina of Time's Poe Sisters and Phantom Ganon paintings (ee4ef51a4fddc28e, 83c0dbc51c5481e3: 128 vertices, 64 primitives, a per-primitive colour) came out solid green or magenta (pull request seedds#1590). A point mesh took its point size as its position's x. Cause: the wrapper read the attribute area as blocks of 32 vertices and 32 primitives side by side, each holding the vertex and then the primitive components. The guest writes all its vertex blocks first, then all its primitive blocks, each block only its own SPH's components: ee4ef51a4fddc28e (6 vertex components, 4 primitive) writes vertex v at (v / 32) * 768 + (v % 32) * 4 and primitive p at 3072 + (p / 32) * 512 + (p % 32) * 4. The SPH's point size (byte 0x35 bit 3) is a vertex block's first component: the point mesh 3a6340a04bf09797 writes the size at component 0 and the position at 1-4. Control +0x5f is a primitive's vertices (1 point, 2 line, 3 triangle); Ocarina's 5 mesh shaders with 1 are the 5 that write a point size, the other 495 have 3. Fix: dmsh_meshstage.memory and to_stage take ceil(max_v / 32) vertex blocks, then ceil(max_p / 32) primitive blocks; the point size goes to [[point_size]]; the topology and a primitive's index bytes follow +0x5f (another value stops the shader's build). The primitive index bytes stay reserved for every primitive block (PR seedds#1575), +0x5f bytes a primitive instead of 3. The pull request's index area, +0x5f * max_p, would have undone seedds#1575 and was not taken; nor were its other changes (reviewed on the pull request). Checks: test_dmsh_meshstage.py (30: the pull request's layout and point mesh tests, a point mesh memory test, the Metal compile of each case), test_docs.py, test_nsp_chain.py. The wrapper's reads checked against the guest's address arithmetic in the translated ee4ef51a4fddc28e and 3a6340a04bf09797. Regression: Ocarina of Time, its 500 mesh shaders translated and compiled on this chain (no other shader changes): 500 of 500 build, none refused, the largest stage memory 17,172 bytes (21,652 before). Mario Kart World, Splatoon Raiders and Yoshi not checked (no build here); the paintings not seen in the game; line topology not met. Ocarina's foliage point meshes (3a6340a04bf09797, a203347bb7bdbd03) still fail their pipelines on user(locn5), which their pixel shader reads as a texture coordinate (point sprite coordinate replacement suspected; open). Co-authored-by: ksolenk38ze <352968655+ksolenk38ze@users.noreply.github.com>

github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders game: 0400486024DEC000 A game labels Nov 6, 2026

nafields mentioned this pull request Nov 6, 2026

Standalone Metal probes: heap aliasing, render-target sizes, mesh index ranges, mesh pipeline archive, mesh/task sweep, task-shader hang #1583

Open

3 tasks done

bettterlabs mentioned this pull request Nov 7, 2026

The Legend of Zelda: Ocarina of Time (0400486024DEC000): renderer, shader and pipeline archive fixes through the Forest Temple #1590

Closed

nafields force-pushed the mesh-prim-bytes-blocks branch from 79b34d1 to a7f945f Compare November 7, 2026 04:34

seedds closed this Nov 7, 2026

bettterlabs mentioned this pull request Nov 7, 2026

Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it: Ocarina of Time (GPU page faults, #1326) #1644

Closed

2 tasks done

PaulEins mentioned this pull request Nov 7, 2026

Ocarina of Time (0400486024DEC000 v131072, Mac): GPU hang about 10 minutes into play takes WindowServer down #1326

Open

1 task done

Navigation Menu

Repository navigation

Mesh stage memory reserves the primitive index bytes for every block the threads write - #1575

Mesh stage memory reserves the primitive index bytes for every block the threads write#1575
nafields wants to merge 1 commit into
seedds:mainfrom
nafields:mesh-prim-bytes-blocks

nafields commented Nov 6, 2026

Uh oh!

ycarogabriel-lgtm commented Nov 7, 2026

Uh oh!

seedds commented Nov 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nafields commented Nov 6, 2026

What and why

Checked

Docs

Uh oh!

ycarogabriel-lgtm commented Nov 7, 2026

Uh oh!

seedds commented Nov 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants