A GPU memory guard before each residency commit stops the game instead of freezing the Mac by lunks · Pull Request #1391 · seedds/nxnative

lunks · 2026-11-05T22:49:07Z

What and why

GPU memory is wired. When a game's set passes Metal's recommendedMaxWorkingSetSize, IOGPUScheduler's freeToWireMemory fails, the submit thread waits in the kernel where nothing can kill the process, and the Mac freezes until the power button (#1326, @aCidcOde's kernel log: failed to wire down memory ... size 19069665280, 3,351 times, seven minutes after _exit(0); @stan69b's 16 GB M4 run on #1326 ended the same way at 11.8 GB of Metal's 12.1 GB). On an 8 GB Mac, #1319 had WindowServer's watchdog restart the desktop. Nothing in the host looked at the working set.

res_commit now reads the larger of the set's allocatedSize and the device's currentAllocatedSize before it commits (the set's size leaves out what was never committed: a new 64 MB texture adds 0 to it until the commit; the device's counts it from its creation).

From 80% of the working set the submit thread lets go of what can go, at most every 30 frames: idle sharing entries nothing holds, finished deferred releases, finished spares, the staging the upload slots keep.
From about 95%, if that was not enough, the game stops: a [RED] line says what the set holds, and the front end closes the graphics gate, leaves full screen, shows an alert and quits as Cmd-Q does (front_gpu_memory_stop; a windowless run takes the host's graceful exit). front_stop_alert is written so front_gpu_lost (A GPU whose access macOS revoked stops the game instead of leaving it running with nothing drawn #1332) can call it.
A commit that adds to the set past the working set itself is not made. Commits that only remove go through.
res_commit callers can hold g_mu, so it only leaves a level for the submit thread (guard_act); a 100 ms timer reads the sizes too, as a stuck submit thread never comes back to look.

Credit: DM_NVN_RES_LIMIT_MB and the report's distinct pages and texture formats come from @JioChoi's #1319, moved from after the commit to before it. =<MB> stops at that size, =0 turns the guard off. iOS only evicts unless the switch sets a limit: no iPhone has tried the stop.

Checked

Tests (Apple M5 Pro, macOS 27, niced): make -C dm/gfx/tests GAME=010015100B514000 check_t_res_guard 20 checks and the stuck run 2 checks, 0 failed (levels, the commit refused and removals passing, what stays: a live object, the parent of a live sRGB view, a recent lookup; the second look at the stop level; the timer with the lock held); test_docs.py, test_env_switches.py, test_gfx_tests_makefile.py OK; docs_symptoms.py --check current; env_switches.py --check prints the same two lines as unchanged main (DM_NVN_CUT_FIRST DM_NVN_VC_WAYS); lint_c.py tidy on nvn_mtl.m, t_res_guard.m, front_mac.m: 0 pairs above the baseline.
Games run: none for this branch. The change was written and tested on our own stack (stack-next, Ocarina of Time v131072, Apple M5 Pro), not on main. There the clean stop was verified windowless and in full screen (the host leaves full screen, shows the alert and quits as Cmd-Q does), and a low DM_NVN_RES_LIMIT_MB ends the run with the report and exit code 0. The guard is not exact: a load overshot a 6,000 MB test limit to about 7.7 GB before the stop. This port to main (dropping the stack-only g_ghost_active and alias_sweep parts) was only built and unit-tested.
Not run: the rest of the Metal suite, the host tests, regress.py, any game on main, the Mac app's alert path in a window on main, iOS.
Overlap: PRAGMATA's opening load fits an 8 GB Mac's GPU working set: large-pool chunks retire, pools from 128 MB are chunked, and DM_NVN_TEX_MIP_SKIP #1319 (JioChoi) has the same DM_NVN_RES_LIMIT_MB and report code in res_check, res_top, mtl_init and docs/env_switches.tsv. If PRAGMATA's opening load fits an 8 GB Mac's GPU working set: large-pool chunks retire, pools from 128 MB are chunked, and DM_NVN_TEX_MIP_SKIP #1319 lands first, this PR rebases onto it and drops those hunks.

Docs

The finding is in docs/reference/memory.md ("GPU memory past Metal's working set freezes the Mac") and docs_symptoms.py was run
No chain tool changed
No game files, keys or data extracted from a game in the change

…d of freezing the Mac A session of Ocarina of Time ended in a GPU out-of-memory and a stuck game, and seedds#1326 reports a Mac frozen until the power button: GPU memory is wired, IOGPUScheduler's freeToWireMemory fails at recommendedMaxWorkingSetSize, and the submit thread then waits in the kernel where the process cannot be killed. Nothing in the host looked at the working set. res_commit now reads the larger of the set's allocatedSize and the device's currentAllocatedSize before it commits; the set's size leaves out what was never committed (measured: a new 64 MB texture adds 0 to it until the commit), the device's counts it from its creation. From 80% of the working set the submit thread lets go of what can go: idle sharing entries nothing holds, finished deferred releases, spares and the upload slots' staging. From 95%, if that was not enough, the game stops with a [RED] report of what the set holds, by kind. A commit that adds to the set is not made at the working set itself. A 100 ms timer reads the sizes too, as a stuck submit thread never comes back to look. The stop reaches the launcher as a normal exit: the Mac app closes the graphics gate, leaves full screen, shows an alert and quits as Cmd-Q does, the path PucaVaz's PR seedds#1332 takes for a revoked GPU (front_stop_alert is written so front_gpu_lost can call it); without a window the host takes its graceful exit. DM_NVN_RES_LIMIT_MB and the report's distinct pages and texture formats come from JioChoi's PR seedds#1319, moved from after the commit to before it. iOS only evicts unless the switch sets a limit: no iPhone has tried the stop. t_res_guard covers the levels, the commit refused and removals passing, what stays (a live object, the parent of a live sRGB view, a recent lookup), the second look at the stop level, and the timer with the lock held.

…fers no longer pile up when a game stops submitting (upstream PRs seedds#1391 and seedds#1392)

…cy commit stops the game instead of freezing the Mac front_mac.m: seedds#1332's front_gpu_lost now goes through this pull request's front_stop_alert; alias_let_go keeps the submission's finalized serials (g_ghost_active); the memory line keeps NXForge's GPU allocation by kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g

…memory guard) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g

Only the generated symptom index conflicted; regenerated (docs_symptoms.py). Checked: test_docs.py, docs_symptoms.py --check. No code conflict.

…fers between submissions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SzykpcKm2eZPvPjGPKx1C

) replaces the pull request heads Upstream landed Ocarina of Time in main (chain adec7d81d970: "holds 60 fps in Kokiri Forest, 4 GB less") and closed the pull requests it folded in, with reworded and partly different code. Mixing a pull request head with its landed version hunk by hunk silently broke things (two GlobalToStorage tables, a duplicate AF2ip, two mtk_pso definitions), so the rule here is one version per feature: - Pull requests upstream closed (seedds#1299, seedds#1300, seedds#1312, seedds#1343, seedds#1345, seedds#1353, seedds#1354, seedds#1359, seedds#1360, seedds#1373, seedds#1428, seedds#1430, seedds#1434): their hunks are taken out of our side before the three-way merge, so upstream's landed version stands alone. Gone with them: bindless pool table patching (reg_bump), the ghost table's per-submission hold, the growing raw pool table and its write-log sizing, seedds#1428's pool_buf spares (upstream's pool_table_take is the same idea), the ixc-based indirect mesh draw (upstream uses mdx), and their tests (t_pool_table, t_mesh_indirect, t_gmem_raw, finalized_bindless_pool, test_bnsh_nvn2, test_ndve). - Open pull requests that upstream's landing supersedes: seedds#1394 (upstream's mtl_pool_addressed) and seedds#1393 pool_skip_retired (upstream's budgeted pool walk; the profile does not use it) are dropped with their tests and docs. seedds#1368's mesh pipeline prewarm is dropped too: upstream kept the background build in nvn_geometry.inc, and the prewarm's mtk_build / mtk_claim / g_warm_mps went with it. - Kept, as upstream does not have them: NXForge's own features (GPU watchdog, present filters and anti-aliasing, aniso floor, quick settings, pipeline archive compaction, upload breakdown, memory report lines) and the still open pull requests seedds#1304, seedds#1305, seedds#1308, seedds#1320, seedds#1331, seedds#1332, seedds#1355, seedds#1356, seedds#1369, seedds#1376, seedds#1377, seedds#1379, seedds#1382, seedds#1390, seedds#1391, seedds#1392, seedds#1414, seedds#1420, seedds#1436, seedds#1439, seedds#1446, seedds#1453. - Combined where both sides changed the same lines: GW_N 1 << 16 with seedds#1320's wide-write intervals (GW_PAGES = SBW_WPAGES, 128 slots), the CPU upload log with upstream's linear upload route, arch_open_end before the compaction limit, cbn_prog in geo_gs_one (seedds#1382). - Translator: upstream's sources and binaries, plus seedds#1414's InstEmitAmpereMem diff and the LDSM SubgroupBarrier in InstEmitAmpereTensor.cs. The binaries and dmsh_chain.lock are upstream's until the Mac rebuilds them (dotnet publish, tool_lock --update-lock, dmsh_chain --header); dmsh_chain.h is regenerated for now. - RED_ONCE keys: upstream took 153 and 154, so the present anti-aliasing lines move to 156 and 157 and seedds#1420's packaged-entry line to 158 (array 159). - Host: hle_net.c and t_net_eventfd.c are upstream's (its eventfd poll waits out the whole timeout, which our 5d7920c fixed another way). - Kept ours: dm/tools/romfs_image.py and its test, so the 26 GB RomFS image is not packed again. dm/tools tests on Linux: same failures as on 875449e (Linux-only quirks), plus test_nlg4 and test_dmsh_pool_arrays, which fail on upstream main too. Not yet built on the Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g

…atedSize currentAllocatedSize counts each no-copy buffer in full however many cover the same pages: 50 over one 64 MB region add 3,200 MB. In Ocarina of Time's Kokiri Forest on a 24 GB Mac it read 17.3 GB within a minute or a few, with 6.5 GB in the residency set and the driver's In use system memory near 7.7 GB, so the guard stopped the game each time. With its stop raised past Metal's working set, the refused commit was followed by InnocentVictim command buffer errors. The guard now reads the set's own size, so an addition counts from its commit. A large burst is seen one commit later than before, and the timer sees the size at the last commit. Checked: t_res_guard, both runs. Ocarina of Time v131072 on an M6 Mac with 24 GB, on main with seedds#1357, seedds#1390, seedds#1391, seedds#1453 and seedds#1455: about 49 minutes and 170,400 frames through Kokiri Forest and the Great Deku Tree, currentAllocatedSize at about 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Not checked: other games, a smaller Mac.

…size, not currentAllocatedSize) over seedds#1391's guard_used Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SzykpcKm2eZPvPjGPKx1C

# Conflicts: # dm/gfx/nvn/nvn_mtl.m

…atedSize currentAllocatedSize counts each no-copy buffer in full however many cover the same pages: 50 over one 64 MB region add 3,200 MB. In Ocarina of Time's Kokiri Forest on a 24 GB Mac it read 17.3 GB within a minute or a few, with 6.5 GB in the residency set and the driver's In use system memory near 7.7 GB, so the guard stopped the game each time. With its stop raised past Metal's working set, the refused commit was followed by InnocentVictim command buffer errors. The guard now reads the set's own size, so an addition counts from its commit. A large burst is seen one commit later than before, and the timer sees the size at the last commit. Checked: t_res_guard, both runs. Ocarina of Time v131072 on an M6 Mac with 24 GB, on main with seedds#1357, seedds#1390, seedds#1391, seedds#1453 and seedds#1455: about 49 minutes and 170,400 frames through Kokiri Forest and the Great Deku Tree, currentAllocatedSize at about 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Not checked: other games, a smaller Mac.

…a, gmem commit repoint on the test stack seedds#1589 (carrying seedds#1391 + seedds#1482) the GPU memory guard reads the residency set's size and stops before macOS runs out of RAM; hand-merged with seedds#1590's ghost-active alias tables. seedds#1599 soft textures (gated, default off). seedds#1662's tip commit: the ocarina plays into the Mac's microphone (DM_MIC=1). seedds#1427 raw pool table words follow the pools at commit, as a candidate for seedds#1326's page faults (its window is skipped, not faulted: kept for the record). Full-map shader library (42,848), clearing the no-stamp and c1_full red lines. In-game: the GPU page fault at scene changes (seedds#1326) remains on this stack - one session lost the picture at frame 55,940 with the device revoked; the fault predates seedds#1585, which was excluded after the M2 Pro report but is not its cause. Front-ends: host change only, chain 2027aa29b1b7.

… for another one: with no submission open it waits only for the last committed fence, is drained at once off the submit thread, and the residency set's removal is committed then too (PR #1392) Only a submission drained the deferred-release queue (at its start or in its feedback handler), and a finalized pool's buffer waited for the next submission's fence. A game that stops submitting while its other threads go on piles them up: PR #1392 measured Ocarina of Time out of focus going from 11 to 20 GB of GPU allocated in 12 s (its streaming threads remade a virtual pool run of about 10 MB 70 times a second), the likely cause of a session that stayed frozen after a window switch. - mtl_pool_del bumps the pool generation first, then reads g_sub_on (set in mtl_sub_begin before any lookup, cleared once the commit has its fence), with a full fence on each side: a submission either is seen open, and the buffer waits for its fence, or sees the bump and cannot reach the buffer through the table or the per-thread cache. - An entry queued with its fence already passed starts one background drain (defer_kick); entries handed back while g_smu was busy are tried again 10 ms later. - A drain's removals free nothing until the residency set's next commit: a feedback handler, or a submission closing, that finds none open has it committed the same way (defer_idle_commit). Unlike the PR, nothing runs on #1391's 100 ms timer, the guard accounting stays with #1391, and DM_OE_SCRIPT already sets the focus state without a window. Checked: t_pool_retire (new, 11 checks, 20 runs) and t_pool_chunks, t_pool_notes, t_defer_chain, t_gmem_words, t_readback, t_feedback, t_rt_recycle, t_alias_takeover; test_docs.py; lint_c.py tidy at the baseline. Games, base against fix on hosts pinned to the chains on disk: Ocarina of Time with 40 s out of focus (the game stopped submitting but remade no pool run on the village-deck route, so the pile-up was not reproduced; the same red lines, no GPU error, the picture back at focus) and Fire Emblem Engage's 240 s attract demo (about 45 pool deletions, the same pictures, residency commits per 300 frames as before). Co-authored-by: Pedro Nascimento <91118+lunks@users.noreply.github.com>

… for another one: with no submission open it waits only for the last committed fence, is drained at once off the submit thread, and the residency set's removal is committed then too (PR seedds#1392) Only a submission drained the deferred-release queue (at its start or in its feedback handler), and a finalized pool's buffer waited for the next submission's fence. A game that stops submitting while its other threads go on piles them up: PR seedds#1392 measured Ocarina of Time out of focus going from 11 to 20 GB of GPU allocated in 12 s (its streaming threads remade a virtual pool run of about 10 MB 70 times a second), the likely cause of a session that stayed frozen after a window switch. - mtl_pool_del bumps the pool generation first, then reads g_sub_on (set in mtl_sub_begin before any lookup, cleared once the commit has its fence), with a full fence on each side: a submission either is seen open, and the buffer waits for its fence, or sees the bump and cannot reach the buffer through the table or the per-thread cache. - An entry queued with its fence already passed starts one background drain (defer_kick); entries handed back while g_smu was busy are tried again 10 ms later. - A drain's removals free nothing until the residency set's next commit: a feedback handler, or a submission closing, that finds none open has it committed the same way (defer_idle_commit). Unlike the PR, nothing runs on seedds#1391's 100 ms timer, the guard accounting stays with seedds#1391, and DM_OE_SCRIPT already sets the focus state without a window. Checked: t_pool_retire (new, 11 checks, 20 runs) and t_pool_chunks, t_pool_notes, t_defer_chain, t_gmem_words, t_readback, t_feedback, t_rt_recycle, t_alias_takeover; test_docs.py; lint_c.py tidy at the baseline. Games, base against fix on hosts pinned to the chains on disk: Ocarina of Time with 40 s out of focus (the game stopped submitting but remade no pool run on the village-deck route, so the pile-up was not reproduced; the same red lines, no GPU error, the picture back at focus) and Fire Emblem Engage's 240 s attract demo (about 45 pool deletions, the same pictures, residency commits per 300 frames as before). Co-authored-by: Pedro Nascimento <91118+lunks@users.noreply.github.com>

…he game with an alert from about 95%, and follow the system's available memory Net change of PR seedds#1589 at cddc1c0 (seedds/nxnative), which stacks seedds#1391 and seedds#1482, merged by hand: front_stop_alert serves both the GPU-lost alert (exit status 70 kept) and the memory stop; alias_let_go retires as this tree does (g_retire_held covers ghosts); the memory line keeps this tree's form. DM_NVN_RES_LIMIT_MB=0 turns the guard off.

…h residency commit stops the game instead of freezing the Mac Ported onto current main (the PR was written on a base before several upstream changes): - alias_let_go keeps main's rule for an entry leaving the sharing table, defer_obj_release (a texture a finalized object still resolves through g_ghost stays resident through defer_drain / g_retire_held). The PR carried the older serial_holds / defer_obj_unref walk that upstream replaced in PR seedds#1439, so that part is not taken. - front_mac.m: main's front_gpu_lost already was the PR's alert code inline. It is now one helper, front_stop_alert(title, text, status), used by front_gpu_lost (exit status 70, as before) and by the new front_gpu_memory_stop (quits as Cmd-Q). - The memory report line keeps main's extra fields and adds the PR's 'idle' count; Makefile and front.h keep both sides. On this Mac (Apple M4, 16 GB) Metal's working set is 11.84 GB: the guard lets go of idle sharing entries, spare textures and staging blocks from 9.47 GB and stops the game at 11.25 GB; DM_NVN_RES_LIMIT_MB=0 turns it off. Checked: t_res_guard 20 + 2 checks, the neighbouring residency / alias tests, the host builds for Ocarina of Time. Not run: a game, so how often the 80% level fires in play is not measured. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

lunks mentioned this pull request Nov 5, 2026

A game that stops submitting no longer piles up retired pool buffers, the likely cause of the freeze after a window switch #1392

Closed

3 tasks done

github-actions Bot added host The Horizon host: SVCs, services, threads, memory (dm/host) nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) mac The Mac front end and app (dm/mac, front_mac.m) docs Documentation labels Nov 5, 2026

fpedrolucas95 added a commit to fpedrolucas95/nxnative that referenced this pull request Nov 5, 2026

A GPU memory guard before each residency commit, and retired pool buf…

2e220e0

…fers no longer pile up when a game stops submitting (upstream PRs seedds#1391 and seedds#1392)

Merge origin/main into pr/res-guard

c8689d9

Only the generated symptom index conflicted; regenerated (docs_symptoms.py). Checked: test_docs.py, docs_symptoms.py --check. No code conflict.

akshay4 mentioned this pull request Nov 6, 2026

The GPU memory guard reads the residency set's size, not currentAllocatedSize (on #1391) #1482

Open

2 tasks done

This was referenced Nov 6, 2026

DM_POOL_BUDGET_MB makes textures up front against the residency set #1498

Open

Ocarina of Time: fire has little glow and the air no haze compared with console video #1499

Open

Merge remote-tracking branch 'origin/main' into rebase-1391

eae16d5

# Conflicts: # dm/gfx/nvn/nvn_mtl.m

PucaVaz mentioned this pull request Nov 6, 2026

A GPU whose access macOS revoked stops the game instead of leaving it running with nothing drawn #1332

Closed

3 tasks done

fxd0h mentioned this pull request Nov 10, 2026

Ocarina of Time (0400486024DEC000): GPU out of memory on 8 GB MacBook Neo (A18 Pro) — 32 command buffers failed, host exits 134 #1895

Open

1 task done

Navigation Menu

Repository navigation

A GPU memory guard before each residency commit stops the game instead of freezing the Mac - #1391

A GPU memory guard before each residency commit stops the game instead of freezing the Mac#1391
lunks wants to merge 3 commits into
seedds:mainfrom
lunks:pr/res-guard

lunks commented Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lunks commented Nov 5, 2026

What and why

Checked

Docs

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant