Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it: Ocarina of Time (GPU page faults, #1326) by bettterlabs · Pull Request #1644 · seedds/nxnative

bettterlabs · 2026-11-07T08:54:01Z

What and why

One cause of the GPU page faults in #1326. It doesn't cover the hangs there, or the faults that follow raw stores into pools the host hadn't learned yet.

The serials of one storage share one packaged texture (g_pkg_share). At the GPU's first write over that storage, cpu_unc_drop gives each of them the package's texture itself (tex_put), so they all hold one texture that is in no shared set. The first holder released (tex_put's ghost-ring eviction, serial_put or serial_del), or the package's own release (NvPkg's dealloc), took it out of the residency set while the other serials still drew with it. A command buffer reading it then faulted.

pkg_hand_over enters the handed-over texture in g_alias_set and in g_handed, a list by the package's sharing key. Releasing one holder then only drops a reference. The package's release, and its re-upload's spare, leave a handed-over texture resident. mtl_alias_drop_pool retires it with its pool, and tex_upgrade moves its entry. It doesn't enter g_alias, so a new texture object over the same storage still gets a texture of its own, as before. DM_UNC_SHARE=0 gives back the old hand-over.

Reference entry: A texture shared by its packaged texture's serials left the residency set with one of them, plus a line in the Ocarina of Time note.

Draft until the game run on current main is in. The build of d777f0a is running. I'll post the result here and mark the PR ready.

Checked

Tests (main d777f0a, Mac mini M4):
- t_cpu_tex --metal with the new handover_shared: 97 checks, 0 failed. With DM_UNC_SHARE=0: 2 failed (the texture isn't in the shared set; it leaves the residency set while another serial holds it).
- make -k -C dm/gfx/tests GAME=0400486024DEC000 run: every test that ran reported 0 failed (142 result lines). Three recipe steps stop the same way on main without the change: they pass $(CURDIR) unquoted, and this checkout's path has spaces. That's the Python steps of check_t_border and check_t_pso_archive (their binaries pass, 44 and 123 checks), and check_t_transform_feedback, which didn't run.
- Tool tests test_docs, test_env_switches, test_gfx_tests_makefile and test_lint_c pass, and env_switches.py --check passes.
- Not run: lint_c.py tidy. This Mac has no LLVM clang-tidy; CI runs it.

Game run: The Legend of Zelda: Ocarina of Time 0400486024DEC000 v131072, Mac build, windowed, DM_RES_CHECK=1, on main b2d46b8 with The Legend of Zelda: Ocarina of Time (0400486024DEC000): renderer, shader and pipeline archive fixes through the Forest Temple #1590, Resolution families take every GPU-written compressible 2D image, not only COMPRESSIBLE|IMAGE: Ocarina of Time's dynamic-resolution depth buffer (COMPRESSIBLE alone, in a virtual pool) kept 15-16 live sizes at one offset as separate 14.5 MB Metal textures #1557, An encoder's end barrier flushes to the device only; the start barriers carry the resource-alias flush: Ocarina of Time's main submission's command allocators no longer grow to 36-80 MB each #1556, Mesh stage memory reserves the primitive index bytes for every block the threads write #1575, perf(pipelines): take compute pipelines from the pipeline archives instead of compiling them at the first dispatch #1522, Ocarina of Time keeps real speed below 60 fps: its frame step follows the measured frame time #1543 and GPU-written storage is tracked as ranges, so a pool that holds both kinds no longer leaves its CPU-written textures at zero #1390 merged locally (chain 89d5ebc3ba0b). Without the change, the page faults came at 44, 29, 24 and 15.5 minutes. A local diagnostic that records where each object left the residency set named the missing textures before two of the faults: 260x260 terrain tiles handed over by cpu_unc_drop and released by the ghost-ring eviction. In one session they were missing for about 2 minutes (50,709 uses) before the first fault.

one binary	change on	`DM_UNC_SHARE=0`
session	107 min (frame 212,338), quit by hand	page fault at 37 min (frame ~59,949)
hand-overs	86 (41 entered, 45 already in)	n/a
missing textures	0	2, both 260x260 tiles handed over by `cpu_unc_drop` and released by the ghost-ring eviction

Device and OS: Mac mini M4 (10-core GPU), 16 GB, macOS 27.2 (26B123).
regress.py: impact names Breath of the Wild, Pokémon Scarlet and Tears of the Kingdom, and every NVN game through the API rule. check wasn't run: this Mac has no builds of those games, and Ocarina of Time has no smoke route.
Not checked:
- The game on current main (above).
- Why the faults began with this stack. Our previous stack ran 95 minutes without one, and its hand-over code is the same by reading.
- The memory cost, measured A/B. A handed-over texture now stays until its pool goes, so the most the change can hold is what it handed over. In the 107-minute run that was 41 distinct textures, 21.7 MB in all, every one 260x260. The process's peak footprint was 8.88 GB in that run and 8.14 GB in the 37-minute DM_UNC_SHARE=0 run, but those runs aren't matched (different play, different lengths).
- With the change, more textures left the residency set while a serial-table entry held them (117 against 5). All the printed ones came from mtl_alias_drop_pool at a pool's finalize, and none was used afterwards.
- Other Macs.

Docs

The finding is in its docs/reference/nvn-objects.md entry and the game note, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
No game files, keys or data extracted from a game in the change or the attachments

…s first write stays resident while any of them holds it: releasing one took it out of the residency set (GPU page faults, issue seedds#1326) The serials of one storage share one packaged texture (g_pkg_share). At the GPU's first write over that storage, cpu_unc_drop gives each of them the package's texture itself (tex_put), so they all hold one texture that was in no shared set. The first holder released (tex_put's ghost-ring eviction, serial_put or serial_del, through obj_drop or defer_obj_release), or the package's own release (NvPkg dealloc's res_remove), took it out of the residency set while the other serials still drew with it, and a command buffer reading it faulted. pkg_hand_over enters the handed-over texture in g_alias_set and in g_handed, a list by the package's sharing key, so releasing one holder only drops a reference (alias_has). The package's release and its re-upload's spare leave a handed-over texture resident, mtl_alias_drop_pool retires it with its pool, and tex_upgrade moves its entry. It does not enter g_alias, so a new texture object over the same storage still gets a texture of its own. DM_UNC_SHARE=0 gives back the old hand-over. Ocarina of Time v131072, Mac mini M4 16 GB, macOS 27.2, on main b2d46b8 with seedds#1590, seedds#1557, seedds#1556, seedds#1575, seedds#1522, seedds#1543 and seedds#1390: page faults at 44, 29, 24 and 15.5 minutes, with the missing textures named by DM_RES_CHECK and a diagnostic of where each left the residency set, before two of them, as 260x260 terrain tiles handed over this way and released by the ghost-ring eviction. One binary: with the change, 107 minutes and no fault (86 hand-overs); with DM_UNC_SHARE=0, a fault at 37 minutes with the same kind of tiles named. t_cpu_tex --metal handover_shared: 97 checks, 0 failed; with DM_UNC_SHARE=0, 2 failed. The other GPU suites are unchanged.

lunks · 2026-11-07T09:00:25Z

We took this onto our downstream stack and are testing it on Ocarina of Time (M5 Pro): no result yet.

…holds it: the texture cpu_unc_drop hands to a package's serials stays resident while any of them draws with it (Ocarina of Time's GPU page faults 15-45 minutes into play, #1326; PR #1644) At the GPU's first write over storage a packaged texture was taken from, every serial of the package, live or finalized in the ghost ring, gets the package's texture itself, and no table owns it. The package's release (NvPkg dealloc) took it out of the residency set at once, and so did any one holder's release (the ghost-ring eviction, serial_put, serial_del), while the others still drew with it. - defer_drain asks whether a serial entry still holds a resource it retires (g_by_serial_held, and g_ghost_held, a new count for the ghost ring). One that does keeps its residency and waits in g_retire_held; the last entry's release retires it again (retire_resume), unless the sharing table owns it by then. - The package's release retires its textures through defer_obj_release instead of res_remove at once. A re-upload does not write into a spare that serials still hold. - alias_take_over no longer scans the ghost ring to choose unref over release: the same rule covers it. - The memory report counts the waiting textures ("retired but held by a serial"). PR #1644 found the cause and measured it (one binary: 107 minutes with no fault against a fault at 37 minutes; the residency diagnostic named 260x260 terrain tiles handed over this way). It kept handed-over textures in a table by storage key until their pool went; this covers every holder instead, with no key collision and nothing kept past the last holder. Checked: t_cpu_tex --metal with the new handover_holders, 98 checks, 0 failed (3 failed on main's host code: the package's release alone took the texture out). The whole gfx test run, test_docs, test_lint_c and test_env_switches pass. Ocarina of Time v131072 on a 16 GB Mac, test hosts on chain 3ee493b2c3a4 (the newest libraries on disk) with DM_RES_CHECK=1, 230 s village walk each: both reach frame 9,000+, no residency or page-fault lines, the same two red lines; at most one texture waited at a time. That walk makes no hand-over, so it shows no regression, not the fault gone. regress.py check not run: main's chain 1d619d0f520b has no libraries for any game here. Co-authored-by: bettterlabs <279109819+bettterlabs@users.noreply.github.com>

seedds · 2026-11-07T12:52:54Z

Thanks: your diagnosis and the A/B found this. It landed as fa9cfb8 with a different mechanism, and you're credited as co-author.

The cause is broader than handed-over textures. Retiring an object took it out of the residency set no matter who else still held it. The new test shows it on main's host code: right after a hand-over, the package's own release alone takes the texture out, before any serial lets go.

So instead of a hand-over table, defer_drain now asks whether a serial entry still holds what it retires:

g_by_serial_held, plus a new count for the ghost ring, g_ghost_held.
If an entry still holds it, the resource stays resident and waits in g_retire_held. The last entry's release retires it.
NvPkg's dealloc goes through the same deferred release.
A re-upload doesn't reuse a spare that serials still hold.
alias_take_over's ghost-ring scan is gone, because the same rule covers it.

Why not keep the hand-over table: it held one texture per storage key until the pool went. When a reused storage slot was handed over a second time with a new texture, the key was already taken, so it fell back to the old hand-over. The new rule keeps nothing past the last holder and needs no DM_UNC_SHARE switch.

Your handover_shared became handover_holders. It hands a package over to two serials, then drops the package, then each serial, the last through the ghost ring. 98 checks pass; on main's host code 3 fail. A 230 s village walk didn't hand anything over, so this hasn't been confirmed against the long-play fault yet. A long session from you or the downstream tester on main would settle it. #1326 stays open for the hangs and the faults this doesn't cover.

…holds it: the texture cpu_unc_drop hands to a package's serials stays resident while any of them draws with it (Ocarina of Time's GPU page faults 15-45 minutes into play, seedds#1326; PR seedds#1644) At the GPU's first write over storage a packaged texture was taken from, every serial of the package, live or finalized in the ghost ring, gets the package's texture itself, and no table owns it. The package's release (NvPkg dealloc) took it out of the residency set at once, and so did any one holder's release (the ghost-ring eviction, serial_put, serial_del), while the others still drew with it. - defer_drain asks whether a serial entry still holds a resource it retires (g_by_serial_held, and g_ghost_held, a new count for the ghost ring). One that does keeps its residency and waits in g_retire_held; the last entry's release retires it again (retire_resume), unless the sharing table owns it by then. - The package's release retires its textures through defer_obj_release instead of res_remove at once. A re-upload does not write into a spare that serials still hold. - alias_take_over no longer scans the ghost ring to choose unref over release: the same rule covers it. - The memory report counts the waiting textures ("retired but held by a serial"). PR seedds#1644 found the cause and measured it (one binary: 107 minutes with no fault against a fault at 37 minutes; the residency diagnostic named 260x260 terrain tiles handed over this way). It kept handed-over textures in a table by storage key until their pool went; this covers every holder instead, with no key collision and nothing kept past the last holder. Checked: t_cpu_tex --metal with the new handover_holders, 98 checks, 0 failed (3 failed on main's host code: the package's release alone took the texture out). The whole gfx test run, test_docs, test_lint_c and test_env_switches pass. Ocarina of Time v131072 on a 16 GB Mac, test hosts on chain 3ee493b2c3a4 (the newest libraries on disk) with DM_RES_CHECK=1, 230 s village walk each: both reach frame 9,000+, no residency or page-fault lines, the same two red lines; at most one texture waited at a time. That walk makes no hand-over, so it shows no regression, not the fault gone. regress.py check not run: main's chain 1d619d0f520b has no libraries for any game here. Co-authored-by: bettterlabs <279109819+bettterlabs@users.noreply.github.com> (cherry picked from commit fa9cfb8)

…heck; seedds#1520 / seedds#1484 landed versions left for the next full merge (NXForge carries their heads) The pick of seedds#1644 kept NXForge's alias_let_go (used at two call sites) with upstream's rule (always a release; residency follows g_retire_held), its newest-first defer_drain order, and the GPU-by-kind memory line with upstream's held count added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK3qWJVi2jG3Gz4i1eY

…holds it: the texture cpu_unc_drop hands to a package's serials stays resident while any of them draws with it (Ocarina of Time's GPU page faults 15-45 minutes into play, seedds#1326; PR seedds#1644) At the GPU's first write over storage a packaged texture was taken from, every serial of the package, live or finalized in the ghost ring, gets the package's texture itself, and no table owns it. The package's release (NvPkg dealloc) took it out of the residency set at once, and so did any one holder's release (the ghost-ring eviction, serial_put, serial_del), while the others still drew with it. - defer_drain asks whether a serial entry still holds a resource it retires (g_by_serial_held, and g_ghost_held, a new count for the ghost ring). One that does keeps its residency and waits in g_retire_held; the last entry's release retires it again (retire_resume), unless the sharing table owns it by then. - The package's release retires its textures through defer_obj_release instead of res_remove at once. A re-upload does not write into a spare that serials still hold. - alias_take_over no longer scans the ghost ring to choose unref over release: the same rule covers it. - The memory report counts the waiting textures ("retired but held by a serial"). PR seedds#1644 found the cause and measured it (one binary: 107 minutes with no fault against a fault at 37 minutes; the residency diagnostic named 260x260 terrain tiles handed over this way). It kept handed-over textures in a table by storage key until their pool went; this covers every holder instead, with no key collision and nothing kept past the last holder. Checked: t_cpu_tex --metal with the new handover_holders, 98 checks, 0 failed (3 failed on main's host code: the package's release alone took the texture out). The whole gfx test run, test_docs, test_lint_c and test_env_switches pass. Ocarina of Time v131072 on a 16 GB Mac, test hosts on chain 3ee493b2c3a4 (the newest libraries on disk) with DM_RES_CHECK=1, 230 s village walk each: both reach frame 9,000+, no residency or page-fault lines, the same two red lines; at most one texture waited at a time. That walk makes no hand-over, so it shows no regression, not the fault gone. regress.py check not run: main's chain 1d619d0f520b has no libraries for any game here. Co-authored-by: bettterlabs <279109819+bettterlabs@users.noreply.github.com> (cherry picked from commit fa9cfb8)

…mmits on top of main, without seedds#1484 and seedds#1644, which main landed as f25fa95 and fa9cfb8 (this merge keeps the earlier branch history reachable, so the push needs no force) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Aax1HnchwCweUDRNWjhHx9

…holds it: the texture cpu_unc_drop hands to a package's serials stays resident while any of them draws with it (Ocarina of Time's GPU page faults 15-45 minutes into play, seedds#1326; PR seedds#1644)

…holds it: the texture cpu_unc_drop hands to a package's serials stays resident while any of them draws with it (Ocarina of Time's GPU page faults 15-45 minutes into play, seedds#1326; PR seedds#1644) At the GPU's first write over storage a packaged texture was taken from, every serial of the package, live or finalized in the ghost ring, gets the package's texture itself, and no table owns it. The package's release (NvPkg dealloc) took it out of the residency set at once, and so did any one holder's release (the ghost-ring eviction, serial_put, serial_del), while the others still drew with it. - defer_drain asks whether a serial entry still holds a resource it retires (g_by_serial_held, and g_ghost_held, a new count for the ghost ring). One that does keeps its residency and waits in g_retire_held; the last entry's release retires it again (retire_resume), unless the sharing table owns it by then. - The package's release retires its textures through defer_obj_release instead of res_remove at once. A re-upload does not write into a spare that serials still hold. - alias_take_over no longer scans the ghost ring to choose unref over release: the same rule covers it. - The memory report counts the waiting textures ("retired but held by a serial"). PR seedds#1644 found the cause and measured it (one binary: 107 minutes with no fault against a fault at 37 minutes; the residency diagnostic named 260x260 terrain tiles handed over this way). It kept handed-over textures in a table by storage key until their pool went; this covers every holder instead, with no key collision and nothing kept past the last holder. Checked: t_cpu_tex --metal with the new handover_holders, 98 checks, 0 failed (3 failed on main's host code: the package's release alone took the texture out). The whole gfx test run, test_docs, test_lint_c and test_env_switches pass. Ocarina of Time v131072 on a 16 GB Mac, test hosts on chain 3ee493b2c3a4 (the newest libraries on disk) with DM_RES_CHECK=1, 230 s village walk each: both reach frame 9,000+, no residency or page-fault lines, the same two red lines; at most one texture waited at a time. That walk makes no hand-over, so it shows no regression, not the fault gone. regress.py check not run: main's chain 1d619d0f520b has no libraries for any game here. Co-authored-by: bettterlabs <279109819+bettterlabs@users.noreply.github.com>

bettterlabs mentioned this pull request Nov 7, 2026

Ocarina of Time (0400486024DEC000 v131072, Mac): GPU hang about 10 minutes into play takes WindowServer down #1326

Open

1 task done

github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation game: 0400486024DEC000 A game labels Nov 7, 2026

bettterlabs changed the title ~~Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it (GPU page faults, #1326)~~ Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it: Ocarina of Time (GPU page faults, #1326) Nov 7, 2026

seedds closed this Nov 7, 2026

satperm mentioned this pull request Nov 7, 2026

Ocarina of Time (0400486024DEC000 v131072, Mac M4 Pro): docked, the 3D picture outdoors turns into flat patches; handheld is fine, MetalFX and HDR change nothing #1653

Open

Aznracer pushed a commit to Aznracer/Mighty-Green-Kokiri-Kid that referenced this pull request Nov 7, 2026

check_updates: watch for seedds#1644 in oot2

9f43e05

sleepyeyedjoe mentioned this pull request Nov 7, 2026

Ocarina of Time (0400486024DEC000 v131072, Mac): ground shadows drop for one frame in Kokiri Forest; the host ignores cross-queue event waits (fixed by #1245) #1703

Closed

1 task done

Navigation Menu

Repository navigation

Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it: Ocarina of Time (GPU page faults, #1326) - #1644

Keep a texture cpu_unc_drop hands to a package's serials resident while any of them holds it: Ocarina of Time (GPU page faults, #1326)#1644
bettterlabs wants to merge 1 commit into
seedds:mainfrom
bettterlabs:unc-handover-residency

bettterlabs commented Nov 7, 2026

Uh oh!

lunks commented Nov 7, 2026

Uh oh!

seedds commented Nov 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bettterlabs commented Nov 7, 2026

What and why

Checked

Docs

Uh oh!

lunks commented Nov 7, 2026

Uh oh!

seedds commented Nov 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants