[PucaVaz]

Checked in the game (Ocarina of Time, M4 24 GB, macOS 27.2; details in the update comment): the revoke line once, the picture stops while the game keeps running, the alert, and Quit exits with 0. Rebased on 865cc35f.

Refs #1326

What and why

In #1326 (The Legend of Zelda: Ocarina of Time, 0400486024DEC000 v131072, on a Mac), a GPU hang (run 1, full screen) or GPU page faults (run 2, windowed) were followed by every later upload, segment, submission and display command buffer coming back MTL4CommandQueueErrorAccessRevoked ("Ignored (for causing prior/excessive GPU errors)", kIOGPUCommandBufferCallbackErrorSubmissionsIgnored). The host logged them and went on:

• in run 2, nothing was drawn while the game's sound and logic kept running for about 70 s, until the game aborted itself;
• in run 1, in full screen, WindowServer froze for 40 s and its watchdog killed it.

Cause. macOS revokes a process's GPU access after GPU errors it caused, and from then on runs none of its command buffers. They come back at once with that error, so their feedback still arrives. But the GPU no longer signals the shared event that a game's SyncWait waits on (for up to an hour, mtl_wait_at), and every present takes a drawable that nothing draws.

The change (nvn_mtl.m, front.h, front_mac.m). gpu_lost_check looks for the revoked access in each command buffer's error. The first one:

• stops presents (present_now);
• raises the shared event from the CPU to the last fence whose feedback has come back. It does so again at each later submission's feedback (gpu_lost_release), so no SyncWait waits for a signal that will not come;
• tells the front end once (front_gpu_lost). The Mac app leaves full screen, says why in an alert that points to the log and scripts/bug_report.py, and quits as ⌘Q does. Without a window, the run ends with a FATAL line and an abort (the weak default in nvn_mtl.m).

Why the access being revoked, and not the first timeout or page fault. The two #1326 logs show that the GPU recovers from those on its own:

• Run 2 had innocent-victim errors (code 1, timeout) from 219 s, and the game recovered each time and kept playing. The page fault (code 2) came at fences 110102 and 110108. From fence 110114, two frames later, every command buffer came back code 5 until the game aborted.
• Run 1 had the hang (code 1) in two segment command buffers, then code 5 on every command buffer after them.

Stopping at the first fault would have ended run 2 at 219 s while it was still playable. (My first comment on #1326 said "timeout, page fault or access-revoked"; this is the narrower trigger.)

Two changes from the first commit of this PR:

• The error is looked for underneath. Metal 4 reports a commit's error as MTL4CommandQueueErrorTimeout with no text at the top, and the command buffer's own error under NSUnderlyingErrorKey / NSMultipleUnderlyingErrorsKey. Every logged command buffer error in both Ocarina of Time (0400486024DEC000 v131072, Mac): GPU hang about 10 minutes into play takes WindowServer down #1326 logs has that shape. The check walks those, in MTL4CommandQueueErrorDomain only: an innocent victim's IOGPU error has code 5 too.
• The event is raised at feedback, not at commit. The first commit raised it to g_fence when a submission was committed. A SyncWait could then mark a fence done (post_early moves g_done) before Metal had handed the command buffer back. The next submission could then reset a command allocator Metal still held, because mtl_sub_begin reuses allocators whose fence is at or below g_done. Raising it to g_done at each feedback keeps g_done tied to Metal's feedback.

iPhone. The weak default stays (a FATAL line in log.txt and an abort). The app has no quit of its own to follow, and I cannot try it on a device. Before this, the game ran on with nothing drawn.

Not covered. The Vulkan and OpenGL layers keep their own queues and waits. Their presents go through present_now, though, so a revoked display command buffer still tells the front end.

This complements #1275's GPU hang defences rather than replacing them. Its watchdog fires when a fence has not completed after 4 s. Here the ignored command buffers complete at once, with an error, so it never fires.

Rendering reference entry. The game note dm/games/0400486024DEC000/README.md is not on main (the profile is in #1299), so the entry links the issue, and the note gets no "Found on this game" line here.

Checked

• Tests (Apple silicon, Metal):

  • make -C dm/gfx/tests GAME=010015100B514000 t_gpu_lost && MTL_DEBUG_LAYER=1 ./t_gpu_lost: 14 checks, 0 failed, five times, and three more with DM_NVN_ENC_THREAD=1. It checks:

    • the error's shape: the revoked access is found under Metal's top-level timeout; an innocent victim's, a page fault's or a hang's error is not taken for it;
    • front_gpu_lost is called exactly once over four revoked submissions;
    • a SyncWait on a revoked fence returns at its feedback (1–6 ms), for the later submissions too;
    • a present after it no longer asks for the layer, and its ticket completes.

    With the event release taken out, the four SyncWait checks fail (each waits out its 2 s timeout). With the first commit's top-level-only check, 7 of 14 fail.

  • make -C dm/gfx/tests GAME=010015100B514000 -j4 run: every test passed (exit 0). t_refl skipped 2 checks for sample files that are not on this Mac. This ran before the rebase onto 7249ee9f, which changes only tools and docs. t_gpu_lost was built and run again after it.

  • python3 -m unittest dm/tools/tests/test_docs.py dm/tools/tests/test_env_switches.py: 12 tests OK; python3 dm/tools/env_switches.py --check: OK.

  • front_mac.m compiled with the host's flags (the strong front_gpu_lost is in its object). The full host was not built, and clang-tidy ran in CI: the two findings in t_gpu_lost.m are fixed; the one left is in t_int_float_view.m from main (fixed by Tests check the Metal device, textures, encoders and functions they make before using them #1338).

• Games run: The Legend of Zelda: Ocarina of Time (0400486024DEC000, v131072), on lunks's 574a38f with this change, windowed, with DM_TEST_GPU_LOST=1500 (the imitation; no real revoked access was reproduced). In two runs the [RED] GPU access revoked line came once (at 15.3 s and 8.2 s), the picture stopped while the game kept running at its normal frame times, and the alert came; Quit exited with 0. The alert was hidden behind a background-opened or DM_WINDOW_TOP game window; 750bda24 activates the app and raises the alert above it (compiled; not yet re-run in the game). Leaving full screen was not exercised: both runs were windowed.
• Devices and OS: M4 (24 GB), macOS 27.2, Xcode 27.2.
• scripts/regress.py check --games: not run. My only local game build is busy, and the change's reach is narrow: the new code runs only on a command buffer error, plus one early return in present_now once the access is revoked. On a run without a revoked access, the new work is a flag test per submission and, on a command buffer error, a walk of the errors under it.

Front ends

No Front-ends: line. After the alert, a Mac app exits with 0, as for ⌘Q. Before, a run like this ended in the game's own abort (the launcher returned 134) or in WindowServer's kill. docs/frontends.md documents no exit status for a game run. If a non-zero status would be better for launchers (#1275's watchdog uses 70), that is a small change.

Docs

• The finding is in its docs/reference/rendering.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• DM_TEST_GPU_LOST is in docs/env_switches.tsv
• No game files, keys or data extracted from a game in the change or the attachments

Drafted with Claude; reviewed by me.

🤖 Generated with Claude Code

[PucaVaz]

Status: I'm building Ocarina of Time (0400486024DEC000, v131072 from a NUP, with #1299 and #1312) on an M4 (24 GB), macOS 27.2, Xcode 27.2, to check this change against the real game. I'll post an update here once the build is done and I've played it.

Drafted with Claude; reviewed by me.

[PucaVaz]

clang-tidy: the two findings in this PR's t_gpu_lost.m are fixed (setvbuf, and a NOLINT block for the texture serial_put keeps, as in t_attachment_capacity.m). The one left is in dm/gfx/tests/t_int_float_view.m from main's f083873 (NullablePassedToNonnull), which this PR doesn't touch; #1343's clang-tidy check fails on the same finding.

Drafted with Claude; reviewed by me.

[PucaVaz]

Update after building and playing, as promised: checked in Ocarina of Time on an M4 (24 GB), macOS 27.2.

Setup: lunks's 574a38f (the pool-table fix branch) with this PR's change on top, windowed, DM_TEST_GPU_LOST=1500 so the revocation happens during the title movie.

• [RED] GPU access revoked: the submission command buffer was ignored (fences back 1499 of 1501); … appears once, followed by front end: the GPU no longer runs the game's work; telling the player and quitting. It came at 15.3 s in one run and 8.2 s in the other.
• After it the picture stops while the game keeps running at its normal frame times, so no SyncWait stayed blocked.
• The alert appears. Clicking Quit goes through terminate: and the app exits with code 0.

One fix from the test, now in the PR (750bda24): when the app was opened in the background the alert stayed behind other apps, and with DM_WINDOW_TOP=1 the floating game window covered it. The alert now activates the app and sits above the game window. This last change is compiled but not yet re-run in the game.

On the previous head the build check, host tests, sanitizers and iOS app passed (coverage and tool tests were still running); the one clang-tidy finding left is in t_int_float_view.m from main, which #1338 fixes.

Drafted with Claude; reviewed by me.

[PucaVaz]

Rebased onto main b2d46b86. One conflict, with f956ffb's failed-command-buffer shader report: both feedback handlers now call cbs_feedback first, then gpu_lost_check, so the report is in the log before a windowless abort. The rendering entry now links Ocarina of Time's note, which is on main now, and the note lists it under "Found on this game".

Checked on an M4 (24 GB), macOS 27.2: t_gpu_lost 14 checks, 0 failed (5 runs, plus 3 with DM_NVN_ENC_THREAD=1); with the event release taken out, 4 fail. t_feedback 193/0, t_cbs_fault 14/0, t_present_alloc 13/0. Of the Metal suite, 109 of 110 check targets pass; t_fuse_check fails 2 of 10 the same way with main's nvn_mtl.m, and t_nvn and t_gfx_suspend don't link (dm_vi_frame_queued), which this change doesn't touch. The Mac host builds.

The reports since show the state this PR handles:

• @LucyferDev's log in Test support for The Legend of Zelda: Ocarina of Time #1265 (main 865cc35f, without this PR) has the exact error shape the check looks for: MTL4CommandQueueErrorDomain Code=1 with Code=5 "Ignored (for causing prior/excessive GPU errors) (00000004:kIOGPUCommandBufferCallbackErrorSubmissionsIgnored)" under it (code 5 is MTL4CommandQueueErrorAccessRevoked), then the game kept running with nothing drawn, and a kernel panic about 5 minutes later.
• @stan69b: every later submission came back "Ignored (for causing prior/excessive GPU errors)"; the picture froze, the sound kept going, and the main thread waited forever in h_nvnSyncWait. That SyncWait is what the event release frees.
• @mbanusic (main 671c68e1) and @reefbluesky (main 88c7e03f): submissions ignored after a fault, the picture frozen; on reefbluesky's run the game kept running without drawing for about 94 s, then the whole Mac froze.
• @dacONJA, with this PR cherry-picked (Ocarina of Time (0400486024DEC000 v131072, Mac): GPU hang about 10 minutes into play takes WindowServer down #1326): "the GPU hung and A GPU whose access macOS revoked stops the game instead of leaving it running with nothing drawn #1332's alert fired ("The GPU stopped running…"). The game quit cleanly, with no Mac freeze."

This PR ends the game at the first of those ignored command buffers. It does not fix the hangs and faults that lead there, and the reports don't show whether stopping earlier avoids every later Mac freeze or panic.

#1391's front_stop_alert is this PR's alert code made into a helper. If you'd rather have it shared, I have a commit that makes front_gpu_lost call it, with the helper copied unchanged, so whichever lands second keeps one copy.

[seedds]

Landed as 5636e7a with you as co-author. Thank you, this is good work: the error shape under the top-level timeout, raising the event at feedback rather than at commit, and the test.

One change: after the alert the Mac app now quits with exit status 70 instead of 0 (exit(70) in place of [NSApp terminate:nil]), so a launcher or script that runs it sees a failed run. exit() runs the same atexit handlers as ⌘Q's terminate; the one will-terminate observer, haptics shutdown, is also registered with atexit. docs/frontends.md now lists the Mac app's exit statuses.

Checked on an M4 Pro: t_gpu_lost passed 14 of 14 (also with DM_NVN_ENC_THREAD=1), the Mac host builds, and the docs and switch-table tests pass. regress.py was not run because the new code runs only on command buffer errors.