[crypt0dj]

shaders.md's mesh stage memory entry closes with "This does not fix the intermittent GPU hangs seen in buffers with these meshes; those are a separate open issue." This adds measurements of those hangs to Ocarina of Time's note: how often they come, what makes one of them stop the game, and what the burst lines up with.

MacBook Pro M1 Max 32 GB, macOS 27.0.1 (26A434), chains 3ee52af3ca18 and 094f939bfef9, profile 0400486024DEC000 v131072, Kokiri Forest and the village floor, windowed.

The rate, and the threshold

run	chain / base	hangs	per 1,000 frames	cascaded errors	exit
main bd979ced	3ee52af3ca18	2	0.26	2	0
main bd979ced, longer	3ee52af3ca18	6	0.16	3	0
DM_NVN_TBDR=1	3ee52af3ca18	2	0.27	2	0
earlier session	older	2	0.03	1	0
current main	094f939bfef9	2	0.21	1	0
current main, next session	094f939bfef9	6	5.00	231	70

A hang on its own is survivable and the host recovers, as the reference says: five of six runs lost one to three command buffers and exited 0. The sixth took 6 hangs in about 1,200 frames, roughly twenty times the rate, and that crossed what macOS tolerates:

[RED] GPU access revoked: the submission command buffer was ignored
      (fences back 25170 of 25174); macOS runs none of this process's
      GPU work any more, so the game stops

The host then did exactly what A GPU whose access macOS revoked stops the game prescribes. The 231 error lines are not a loop: 12 are the hangs themselves, 52 are submissions already in flight coming back SubmissionsIgnored while it wound down, and the repeated frame 1469 is the frame counter standing still because no GPU work runs any more. So the interesting number is not the exit but the hang rate: isolated hangs are invisible, a burst is fatal.

What the burst lines up with

Task and mesh pipelines built while the game ran, against a clean run on the same build:

	clean run (~9,600 frames)	burst run (~1,200 frames)
task / mesh pipeline: builds	64	57
per 1,000 frames	6.7	47.5
when	spread out	all 57 before the first hang, 0 after

And the failed command buffer is mesh-enriched. Of its 172 distinct shaders, resolved against the profile's shader_map.tsv:

stage	in the failed buffer	in the profile
pixel	94	22,352
vertex	58	15,851
mesh	16	500
task	3	20
compute	1	4,125

Mesh and task are 11 % of the buffer against 1.3 % of the profile. Among them is the light-tile triple named in the mesh stage memory entry itself — task a5288d0a6c4971fd, meshes d4171cf9adf0504b and e8c667282ba7bd9b, fragment ea1bf96b72e72476:

task / mesh pipeline: task a5288d0a6c4971fd_b mesh d4171cf9adf0504b_b fragment ea1bf96b72e72476_b, 1.1 ms
task / mesh pipeline: task a5288d0a6c4971fd_b mesh e8c667282ba7bd9b_b fragment ea1bf96b72e72476_b, 0.3 ms

What this is not

Correlation only. A command buffer holds whatever the scene drew, so the stage counts are compared against the wrong baseline: the fair one is the mesh share of a command buffer from the same scene that did not hang, and I did not capture one. The blame ranking the host prints (game shaders by failed/ran) does not discriminate either — after three failures the top entries are ordinary Nin_Ounce_NVN2 model pixel shaders at 3/101 to 3/371, which is what a heavy outdoor buffer contains.

Not a regression. The hangs are present on plain main with none of my changes, at the same rate.

Not memory. The clean run held 15,300 MB resident and 12,387 MB GPU-allocated with headroom to the system memory limit none for 9,600 frames and survived; the burst run died at 13,974 MB.

A testable prediction

Mesh pipelines are built with newRenderPipelineStateWithMeshDescriptor and so bypass the pipeline archive (nvn_geometry.inc), which is #1583's point. If the correlation above is causal, archiving them would cut the hang exposure as well as the build cost — a prewarmed scene would not be building 47 pipelines per 1,000 frames while it draws. Untested; I am posting the measurements rather than the conclusion.

🤖 Generated with Claude Code

[T-Damer]

An M2 Max data point for these hangs: the same task shader is in the failed buffer, and here the burst stopped the game twice, the second time about 10 minutes into play.

Machine: MacBook Pro M2 Max (38-core GPU), macOS 26.6.2, Xcode 27.2 (Metal 32023.922). Profile 0400486024DEC000 v131072 from a NUP, Mac build through NX Play 1.8.4, windowed on a 165 Hz external screen (2560x1440 window). Three .zs textures in my NUP were damaged (leading zero pages). I packed them with --allow-damaged and served replacements through DM_ROMFS_OVERLAY, so no damaged bytes reach the game. I don't think they're related, but I'm listing them.

run	base	chain	where	first hang	hangs before revoke	exit
1	main bd979ced	3ee52af3ca18	after ~6 h idle at the title, then a few minutes of play	frame 1,240,915	2 (within ~45 frames)	70, GPU access revoked: the display command buffer was ignored
2	main 62f6a52c with ea0c98df reverted*	c096b1d1dc53	~10 min of play from a fresh boot	frame 19,686 (627 s)	3 (frames 19,686 to 21,229)	70, GPU access revoked: the submission command buffer was ignored

* I reverted ea0c98df only because I suspected it of a black screen at boot. That turned out to be the game waiting for focus (main thread in h_oe_PopMsg after oe focus state → 2), so the revert is probably irrelevant here.

All errors are kIOGPUCommandBufferCallbackErrorHang, followed by SubmissionsIgnored.

Same task shader as yours. Task a5288d0a6c4971fd is in the failed buffer in both runs, as is 21d9f83e6027c565 (the queue 1 SH producer from the game note). In both runs the last pipeline built before the first hang was a task/mesh one:

run 1: task / mesh pipeline: task a5288d0a6c4971fd_b mesh 9df3c6c01fec0193_b fragment 7b0a886903a664b2_b, 9558.6 ms in the background
       [RED] command buffer error details ... Caused GPU Hang Error (00000003:kIOGPUCommandBufferCallbackErrorHang)   ← 2 lines later
run 2: task / mesh pipeline: task a5288d0a6c4971fd_b mesh 9b11e51ca22b3608_b fragment aa3f1dca9e8828f0_b, 1.3 ms in the background
       ... first hang ~80 log lines later; mesh 9df3c6c01fec0193 is in the failed buffer's shader list

All 59 task/mesh pipeline builds of each run came before its first hang, and none after, the same pattern as your burst run. As in your run, the failed submission was passes +96 ... serial table changes +36 both times.

Run 1 also changed the window size right before the hang (drawable 4096x1716, the screen dropping from 165 Hz to 120, then 60 Hz, about 10 s earlier). Run 2 had no resize, so I don't think the resize is the cause.

Not tested yet: whether DM_NVN_SKIP_FS=7b0a886903a664b2 (dropping the draws of the pipeline that finished building right before the run 1 hang) avoids the hang. I'll add the result here. I can rerun with DM_NVN_GPUPROF, DM_NVN_INDLOG, or a branch build on this M2 Max if a particular measurement would help.

[ycarogabriel-lgtm]

Base M1 data on the prediction that archiving mesh pipelines cuts the hang exposure (MacBook Pro M1, 16 GB, macOS 26.6.2; OoT v131072; local branch of open PRs on an older main).

Measured: pipelines built by MTL4Compiler and serialized into a Metal 4 archive were found again in another process (lookup 3.2 ms against 928 ms to compile; a different format missed). In a 300 s second session with those 56 mesh pipelines in the archive: 56 taken from the archive, 0 compiled in game, 0 draws dropped, no frame over 1 s caused by a mesh pipeline.
Measured against the prediction: in the second session a GPU hang still happened (frame 4,591, mesh 3439aaa071d868ae; contained by our guard) with no mesh pipeline built in game before it. In an earlier session with the 58 M rows prewarmed, a hang came at 174 s after only one in-game mesh build. One run each, so this is weak, but it points the same way as lunks on #1605: a freshly built pipeline is not the trigger here.
What did stop it here: ending the command buffer before each a5288d0a6c4971fd draw (4 of 4 clean walks, see #1326).

[crypt0dj]

My prediction in this PR is wrong, and the correlation it rested on is a coincidence. Thanks for measuring it.

@ycarogabriel-lgtm tested the one testable claim I put in the description — that archiving the mesh pipelines would cut the hang exposure — and it does not hold. With 56 mesh pipelines served from a Metal 4 archive, none compiled in game, no draws dropped and no frame over 1 s from a mesh build, a hang still came at frame 4,591; an earlier session hung at 174 s after a single in-game build. One run each, but both point the same way, and the same conclusion was already on #1605: a freshly built pipeline is not the trigger.

So the part of this PR that stands is the rate measurement and the revocation threshold. The part that does not is the inference I drew from it:

from this PR	status
0.03-0.27 hangs per 1,000 frames over six runs, each recovering	stands
a burst of 6 in ~1,200 frames crosses macOS's threshold; the host stops as written	stands
the failed buffer is mesh- and task-enriched (19 of 172 against 520 of 42,851)	stands
in-game mesh pipeline builds line up with the burst (47.5 against 6.7 per 1,000 frames, all before the first hang)	reproduced, not causal
archiving them would therefore cut the exposure	refuted

The fourth row is worth keeping on the record precisely because it reproduces. @T-Damer saw it on an M2 Max — all 59 task/mesh builds of each run before the first hang, none after — and I saw it on an M1 Max. Three machines, same pattern, and it is still not the cause. It looks like both the builds and the first hang simply cluster in the minutes when a scene is first entered. Anyone else matching that pattern should not read it as a lead; I did, and it cost a round of measurement.

What does look causal is the task shader, not the pipeline lifecycle. a5288d0a6c4971fd is in the failed buffer in my run and in both of @T-Damer's, it is the shader whose draws, split into their own command buffer, walked clean 4 of 4 times on @ycarogabriel-lgtm's M1 (#1326), and it is 19 of the 32 slow builds @yoBoss283 timed on an M4 Max (#1720). It is also the light-tile task of the mesh stage memory entry and of the bands @nawangwe just diagnosed in #1844. I have no measurement of my own to add there yet.

Two corrections pushed to the branch, so the note in the repo does not carry the refuted claim:

• the archiving sentence is replaced by the measurement above, and the correlation is marked as a coincidence of the first minutes rather than a lead
• every run in the note is marked as predating 04419bc2 and 7de11da5, so it sat at the dynamic resolution's floor (683x384) throughout. The pass shapes and GPU times a reproduction sees on a later main are not these, which I should have stated in the first place

Happy to rerun anything on the M1 Max (32 GB, macOS 27.0.1) on current main, including with the command-buffer split, if a particular measurement would help.

🤖 Generated with Claude Code