Repository navigation
Conversation
…d of freezing the Mac A session of Ocarina of Time ended in a GPU out-of-memory and a stuck game, and seedds#1326 reports a Mac frozen until the power button: GPU memory is wired, IOGPUScheduler's freeToWireMemory fails at recommendedMaxWorkingSetSize, and the submit thread then waits in the kernel where the process cannot be killed. Nothing in the host looked at the working set. res_commit now reads the larger of the set's allocatedSize and the device's currentAllocatedSize before it commits; the set's size leaves out what was never committed (measured: a new 64 MB texture adds 0 to it until the commit), the device's counts it from its creation. From 80% of the working set the submit thread lets go of what can go: idle sharing entries nothing holds, finished deferred releases, spares and the upload slots' staging. From 95%, if that was not enough, the game stops with a [RED] report of what the set holds, by kind. A commit that adds to the set is not made at the working set itself. A 100 ms timer reads the sizes too, as a stuck submit thread never comes back to look. The stop reaches the launcher as a normal exit: the Mac app closes the graphics gate, leaves full screen, shows an alert and quits as Cmd-Q does, the path PucaVaz's PR seedds#1332 takes for a revoked GPU (front_stop_alert is written so front_gpu_lost can call it); without a window the host takes its graceful exit. DM_NVN_RES_LIMIT_MB and the report's distinct pages and texture formats come from JioChoi's PR seedds#1319, moved from after the commit to before it. iOS only evicts unless the switch sets a limit: no iPhone has tried the stop. t_res_guard covers the levels, the commit refused and removals passing, what stays (a live object, the parent of a live sRGB view, a recent lookup), the second look at the stop level, and the timer with the lock held.
Only the generated symptom index conflicted; regenerated (docs_symptoms.py). Checked: test_docs.py, docs_symptoms.py --check. No code conflict.
|
Same gap on an M4 MacBook with 24 GB (Metal's working set 18,186 MB), main 75c1a8c with #1453, #1455, #1332 and #1446, Ocarina of Time from the autosave at Link's house: |
…size, not currentAllocatedSize) over seedds#1391's guard_used Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013SzykpcKm2eZPvPjGPKx1C
# Conflicts: # dm/gfx/nvn/nvn_mtl.m
…atedSize currentAllocatedSize counts each no-copy buffer in full however many cover the same pages: 50 over one 64 MB region add 3,200 MB. In Ocarina of Time's Kokiri Forest on a 24 GB Mac it read 17.3 GB within a minute or a few, with 6.5 GB in the residency set and the driver's In use system memory near 7.7 GB, so the guard stopped the game each time. With its stop raised past Metal's working set, the refused commit was followed by InnocentVictim command buffer errors. The guard now reads the set's own size, so an addition counts from its commit. A large burst is seen one commit later than before, and the timer sees the size at the last commit. Checked: t_res_guard, both runs. Ocarina of Time v131072 on an M6 Mac with 24 GB, on main with seedds#1357, seedds#1390, seedds#1391, seedds#1453 and seedds#1455: about 49 minutes and 170,400 frames through Kokiri Forest and the Great Deku Tree, currentAllocatedSize at about 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Not checked: other games, a smaller Mac.
7e70e0d to
0f15b55
Compare
|
Moved onto #1391's latest head and re-ran |
…a, gmem commit repoint on the test stack seedds#1589 (carrying seedds#1391 + seedds#1482) the GPU memory guard reads the residency set's size and stops before macOS runs out of RAM; hand-merged with seedds#1590's ghost-active alias tables. seedds#1599 soft textures (gated, default off). seedds#1662's tip commit: the ocarina plays into the Mac's microphone (DM_MIC=1). seedds#1427 raw pool table words follow the pools at commit, as a candidate for seedds#1326's page faults (its window is skipped, not faulted: kept for the record). Full-map shader library (42,848), clearing the no-stamp and c1_full red lines. In-game: the GPU page fault at scene changes (seedds#1326) remains on this stack - one session lost the picture at frame 55,940 with the device revoked; the fault predates seedds#1585, which was excluded after the M2 Pro report but is not its cause. Front-ends: host change only, chain 2027aa29b1b7.
…he game with an alert from about 95%, and follow the system's available memory Net change of PR seedds#1589 at cddc1c0 (seedds/nxnative), which stacks seedds#1391 and seedds#1482, merged by hand: front_stop_alert serves both the GPU-lost alert (exit status 70 kept) and the memory stop; alias_let_go retires as this tree does (g_retire_held covers ghosts); the memory line keeps this tree's form. DM_NVN_RES_LIMIT_MB=0 turns the guard off.
…nto main: alias_let_go with main's retire, one stop alert for GPU lost (exit 70) and memory (cmd-Q) Upstream pull requests applied in this commit (their authors' work, brought onto this tree with a three-way merge; the description is in each PR): - seedds#1589 by nafields (open): GPU memory guard, the system's side: stop before the Mac runs out of RAM (kern.memorystatus_level, DM_NVN_SYS_MARGIN_MB; on seedds#1482)
Stacked on #1391, and this is only its last commit. Happy for it to be folded in there instead.
What and why
With #1391, Ocarina of Time stops within minutes on a 24 GB Mac with the "GPU memory is nearly full" alert, while the GPU is well under its limit:
currentAllocatedSizeIn use system memorycurrentAllocatedSizecounts each no-copy buffer in full however many cover the same pages. A small test put 50 buffers over one 64 MB region and it rose 3,200 MB. I haven't confirmed that overlap accounts for the whole gap in this game.docs/reference/memory.mdalready notes this counter reading 16 GB against 3.9 GB invmmap.With the stop raised past the working set, the refused commit was followed by
InnocentVictimcommand buffer errors, so refusing can itself cause a failure.The guard now reads the set's own size.
Trade-off: a large burst is seen one commit later, and the timer only knows the size at the last commit. #1393 reports real out-of-memory errors with 16 GB allocated on a 16 GB Mac, which this doesn't explain, so it needs trying on a smaller Mac.
Checked
check_t_res_guard, both runs;test_docs.py,test_env_switches.py0400486024DEC000v131072, main with Metal compiles take -fbracket-depth=1024: two Ocarina of Time compute shaders nest 259 brackets #1357, GPU-written storage is tracked as ranges, so a pool that holds both kinds no longer leaves its CPU-written textures at zero #1390, A GPU memory guard before each residency commit stops the game instead of freezing the Mac #1391, A GPU-only pool's render targets share one placement heap: Ocarina of Time's 3 GB of render targets fit in 0.7 GB #1453 and Shared textures no live object resolves to are retired when GPU memory runs short: Ocarina of Time no longer freezes after the talk with Saria #1455 on top: about 49 minutes, 170,400 frames, Kokiri Forest and the Great Deku Tree.currentAllocatedSizeabout 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Four earlier runs on the same stack stopped at the guard.regress.pynot run: only the guard's reading changes, and it has no effect below its levels.Docs
docs/reference/memory.mdupdated,docs_symptoms.pyrun