Repository navigation
Closed
Labels
game: 0400486024DEC000A gameA gamemacThe Mac front end and app (dm/mac, front_mac.m)The Mac front end and app (dm/mac, front_mac.m)
Description
vanilla-run2.log
vanilla-impa.log
Summary
On plain main (eed7c42) the game hangs the GPU (kIOGPUCommandBufferCallbackErrorHang → "GPU access revoked") every time at two story moments:
- Kokiri Forest, early NPC conversation: The girl sitting above the shop. Right after the game prompts me to target her and I start talking, as the dialogue camera begins circling her. The frozen frame keeps the NPC drawn; the rest of the scene is gone.
- Hyrule Castle, Impa escorting Link out after meeting Zelda. Here the screen flashes white a few seconds before the hang.
Not intermittent for me: 100% reproducible across ~20 runs and several builds. Related: #1843 (same pipeline found independently), #1830, #1326, #1756, #1788, #1690.
Setup
- Apple M2 Pro (Mac14,12), macOS 26.6.2, Xcode 27.2, Metal compiler 32023.922
- nxnative
eed7c42, shader chain094f939bfef9,build_ipa_from_nsp.py --platform mac-app --gpu g14s, no env changes. Only local change:shader_split.unzsdecompresses in-process, because thezstdsubprocess pipe stalls the split at ~160,400 files under Python 3.15 on this Mac (doesn't change the app). - NUP: base v65536 + update v131072; all 203,786
.zsfiles checked (zstd signature + SHA-256), none damaged. (An earlier update-only NUP had 3 zero-filled textures; the clean dump hangs the same.) - Same hangs with
DM_UPSCALER=gameandmetalfx, handheld mode.
Narrowing it down
I added a small switch, DM_NVN_SKIP_MESHKEYS=<key>[,…], in mesh_tasks_draw that drops task/mesh draws whose task, mesh or pixel key matches (DM_NVN_SKIP_FS doesn't apply to mesh draws, so #1756's planned test with it can't drop these).
DM_NVN_SKIP_MESH=1(all task/mesh draws off) → Kokiri conversation goes through.- Bisecting the task/mesh pipelines (49 in that scene) by key → dropping only mesh
3439aaa071d868ae(taska5288d0a6c4971fd, pixelbd09ab5fc4fd7680/7b0a886903a664b2,Model.Product.110 #BNSH@0x11543000) prevents the Kokiri hang every time; every run that draws it hangs. That pipeline is first built a fraction of a second before the hang. - Dropping only
e8c667282ba7bd9b+d4171cf9adf0504b(Ocarina of Time's intermittent GPU hangs measured: 0.03-0.27 per 1,000 frames, against 5.00 in the run macOS stopped, lining up with mesh pipelines built while the game ran #1756's triple) → still hangs. Dropping only pixel7b0a886903a664b2→ still hangs. - With only
3439aaa071d868aedropped, one session ran 88 minutes with zero GPU errors (Deku Tree → Castle Town), then hung at the Impa escort. There, 11 other newa5288d0a6c4971fdpipelines were in the failed buffer (e.g. meshes57ed4680e3042998,b903546b6b988e5f,d744d81c38c5831a,0c75c8aed6290acf,40a470bb5dbd8f2d). DM_NVN_SKIP_MESHKEYS=a5288d0a6c4971fd(every pipeline with that task) gets through both scenes, with nothing obviously missing on screen.
What the shaders show
- Task
a5288d0a6c4971fd: its tile loops are bounded byc3[193].xy. I loggedS.ubo[3][0]vec4 193 on the host for every a5288 draw: a sane32 × 18(and1 × 1for small passes) right up to the hang, never garbage, so the Hi-Z scan in the task is bounded. (This is the suspectDM_NVN_A5288_C3in Ocarina of Time: diagnostics, off by default: TEXDUMP over a range of frames with depth targets, DM_NVN_A5288_C3 for the task hang #1843 is built to check.) Some draws report0 × 0. - Mesh
3439aaa071d868ae: no loops, barriers or SIMD ops in its body. Its index writes reach 1 word past the stage array when all 128 lanes write (max_p 126, the Mesh stage memory reserves the primitive index bytes for every block the threads write #1575 case). - Pixel
bd09ab5fc4fd7680: ado { … } while (true)loop ofmin(fp_s0[tile].count, fp_c6[113].x) + 1iterations over lights. Loggedfp_c6[113].x= 1000 in every draw. So the hang implies the GPU-read per-tile counts infp_s0are far too high during these scenes, giving up to ~1000 lighting iterations per pixel across the screen, enough to trip the watchdog. The white flash before the Impa hang fits overfull light/tile lists.
Ruled out on this machine
- Batching ([0400486024DEC000] Ocarina of Time (macOS): GPU Hang -> 'GPU access revoked', game quits #1788's suggestion):
DM_NVN_CUT=4 DM_NVN_CUT_BUSY=0still hangs (the failed segment is the main scene pass). - Raw global memory through a pointer read when the draw is bound: the host fills each fixed-offset pointer's pool range before the draw (chain 1ffcb6b77740) #1585 (task range pointer), A memory pool's buffer follows its pages when they are unmapped and mapped again (Monster Jam Steel Titans 2 GPU hang) #1616 (pool buffers after remap), the residency fix (now merged): each tested, still hangs.
- Upscaler choice, damaged textures, memory (footprint ~4 GB at the Kokiri hang, ~11.7 GB peak in the long session).
Workaround
DM_NVN_SKIP_MESHKEYS=a5288d0a6c4971fd (patch to mesh_tasks_draw, 13 lines; happy to open a PR). Logs from vanilla eed7c42 runs of both scenes are available.
Metadata
Metadata
Assignees
Labels
game: 0400486024DEC000A gameA gamemacThe Mac front end and app (dm/mac, front_mac.m)The Mac front end and app (dm/mac, front_mac.m)
Activity
pkimsey commentedon Nov 9, 2026
Ruled out the draw-pass ordering race (#1707): ran the same build with
DM_NVN_BARRIER=full(every pass waits for the previous one, stricter than #1707's object/mesh stages) and no pipelines skipped (DM_NVN_SKIP_MESHKEYSempty). It still hangs (frame 1303), witha5288d0a6c4971fd_bin all 5 failed command buffers (5/1994, top of the ranking). So the bad tile light counts aren't a pass-overlap race on this machine.pkimsey commentedon Nov 9, 2026
Fixed on current
mainfor me:8d5d4f1b(chain0c434f1b68df), vanilla, noDM_NVN_SKIP_MESHKEYS. Both the Kokiri conversation and the Impa escort play through with zero GPU errors, locked 30 fps (GPU p50 ≈ 21 ms). Likely from #1860's uniform bounds (26062f9e) and/or-fwrapv(e4e5ff64). Closing.seedds commentedon Nov 10, 2026
Notes from looking at this on an M6 (main, chain f9f01e4f9994 libraries), for #1326, #1605 and this issue.
The pixel shader's light loop isn't it. In
bd09ab5fc4fd7680the loop runsmin(count, c6[113].x) + 1times, so at most 1,001 per pixel, and nothing in it loops again. Every macOS GPU report attached to these issues that says which stage was busy names the geometry stage (fw_ta_statebusy,fw_cl_stateidle) with restart reason "MMU interrupt", and reefbluesky saw page faults at 0x700 and 0x3400000000, outside every pool. Task and mesh shaders run in that stage; a slow fragment loop would show the fragment stage instead. The taska5288d0a6c4971fd's tile loops are clamped toc3[193](logged sane above). On builds since ffebfd5, mesh3439aaa071d868aehas no loops.An M6 can't show this hang. On the M6, a shader read 4 GB or 1 TB past its buffer, or at address 0x700, returns 0, a wild write is dropped, and the command buffer completes. On M1–M5 the same access gives the page faults above. Saria's opening dialogue drew the
a5288d0a6c4971fd/3439aaa071d868aepipelines with both pixel shaders on the M6, at 13.6 ms of GPU a frame, with no errors. The M6 also accepts mesh vertex indices pastmax_vertices(64 up to 4e9). So results from an M6 say nothing about this bug.Working hypothesis, not confirmed: a shader in those pipelines' task or mesh stage reads or writes a bad device address. Metal's shader validation reports such a load on the M6 (address, buffer, source line), so the next step is a run with validation limited to the
a5288d0a6c4971fdpipelines. Full validation stalls at boot.DM_NVN_SKIP_MESHKEYSstops the hang by not drawing those pipelines; it hides the cause rather than fixing it.ycarogabriel-lgtm commentedon Nov 10, 2026
Base M1 (8-core GPU, 16 GB), macOS 26.6.2: first try at validating only the
a5288d0a6c4971fdpipelines — the per-pipeline switch is ignored without the global layerReproducer: our older local branch (
codex/oot-final6at23272a83, main as ofe2b76d4eplus our picks; its own matching library, chain90e8d82142f4), a save of our own, docked, 60 fps target, scripted input (DM_KEYS=15:A*12/4),DM_NVN_ISOLATE_TASK=0, headless. It hangs ~25–35 s after the save loads in every run so far (5 of 5: 3 windowed, 2 headless; frames ~1418–1600),kIOGPUCommandBufferCallbackErrorHang, then GPU access revoked; each failed command buffer holds the culling taskf75708d176bb6591(10 of 10), and 7 of 10 alsoa5288d0a6c4971fd. No shader of the run failed to load.What we tried, with a local switch that builds the task / mesh pipelines whose task or mesh key is
a5288d0a6c4971fdor3439aaa071d868aethrough theMTLDevicewithMTLMeshRenderPipelineDescriptor.shaderValidation = MTLShaderValidationEnabled(never from or into a pipeline archive) and logs what the returned state reports:MTL_SHADER_VALIDATION: all 14 such pipelines came back withshaderValidationnot enabled, so the field is ignored on this system without the global layer. The run hung as usual (no validation report, which here means nothing).MTL_SHADER_VALIDATION=1: the game boots ("Metal GPU Validation Enabled") but runs far too slowly to reach the hang (about 900 frames in 300 s, frames up to 2.8 s), so noa5288pipeline was built. During boot the layer reported 4,024 times:We can't tell which shader that is yet (the pipelines carry no label). Next on our side: label every pipeline with its shader keys, and with the global layer on, turn validation off for every pipeline outside the list, so the game reaches the scene with only those pipelines validated.
ycarogabriel-lgtm commentedon Nov 10, 2026
Two small corrections to my comment above: the 5 reproducer runs were 3 windowed and 2 headless (I wrote "headless" for all), and the task
a5288d0a6c4971fdis in 7 of the 10 failed command buffers, not "about half" (the culling taskf75708d176bb6591is in 10 of 10). The older branch iscodex/oot-final6at23272a83, main as ofe2b76d4eplus our picks.ycarogabriel-lgtm commentedon Nov 10, 2026
Base M1, follow-up: validation limited to the
a5288d0a6c4971fdpipelines now runs, and the run dies inside Metal's debug layer before the hang pointSame reproducer as above (host
23272a83, its matching library90e8d82142f4, docked, 60 fps target, scripted input,DM_NVN_ISOLATE_TASK=0, headless; it hangs ~25–35 s after the load, 5 of 5 so far). This time withMTL_SHADER_VALIDATION=1loaded globally and, in our local diagnostic build, every pipeline outside the list built withshaderValidation = MTLShaderValidationDisabled(Metal 4 options, legacy and mesh descriptors), the listed task / mesh pipelines built through theMTLDevicewith validation enabled, and every pipeline labelled with its shader keys.a5288d0a6c4971fdwith meshesd4171cf9adf0504bande8c667282ba7bd9b(fragmentea1bf96b72e72476), the returned state reportsshaderValidationenabled, and the first draw of one of them ran (3 workgroups from 0).Invalid device load …, executing fragment function: "MFX_SCALE_SHADER_V1_FRAG", pipelineMFX_ScaleRenderPipelineStateV1), not a game shader.So up to that point the validated
a5288d0a6c4971fdpipelines made no invalid device access, but the run did not reach the moment of the hang. We will look at what the debug layer found in the residency set (a released object would fit it) before trying again.ycarogabriel-lgtm commentedon Nov 10, 2026
Base M1, follow-up 2: the debug-layer crash is a buffer used in a residency set after it was freed (repeatable), and the boot report is in a vertex-pulling pipeline
Same reproducer (host
23272a83, matching library90e8d82142f4, docked, 60 fps target, scripted input,DM_NVN_ISOLATE_TASK=0, headless), withMTL_SHADER_VALIDATION=1.-[MTL4GPUDebugCommandBuffer applyResidencySets:](at frame ~1,192, and between frames 300 and 600 in the other two); the fourth, with every pipeline validated and no labels, ran too slowly to get past frame ~900 in 300 s. WithNSZombieEnabled=YESit came at 14.8 s, on the title, as:So at a commit, a residency set still listed an
MTLBufferthat had already been released. Without the debug layer nothing catches that, and a GPU access through a buffer whose pages are gone would fit the MMU faults in the hang reports. This is on our older host; we have not yet checked whether current main does the same (a run of a main-based host with the layer is next).a5288d0a6c4971fdpipelines made no report before the run ended (it never reached the hang point, ~frame 1,418–1,600 on this save).Invalid device load ... vertex function: "" / buffer: Out of bounds of user address space, length:1, UID07C32766…): with every pipeline labelled, that pipeline isvs=? fs=29e3c9dd92b8a571_b, i.e. a vertex-pulling stand-in (not a library shader, hence no key) with the terrain pixel shader29e3c9dd92b8a571(tera_develop.sharcb). 2,190 such reports in one boot, all in pulling stand-ins (fragment shaders29e3c9dd…,be57fb15…,4dd9070f…,38482ed8…, …), all a 1-byte read "out of bounds of user address space". These are the host's hybrid pulling pipelines reading a vertex stream with no valid address, not the task / mesh stage.ycarogabriel-lgtm commentedon Nov 10, 2026
Base M1, follow-up 3: correction and a narrower reading of the debug-layer crash — the residency set is committed on a completion thread while the submit thread commits command buffers
Same reproducer (host
23272a83, matching library90e8d82142f4; it has now hung in 11 of 11 runs without the layer). We added a local log of every allocation released while the host's residency set (g_res) still listed it, or afterremoveAllocationbut before the set's nextcommit, with the releasing stack.-[IOGPUMetalResidencySet commit](with the layer:-[MTLToolsResidencySet commit]), i.e. the set kept the allocation alive and dropped it when its removal took effect. So "a buffer used after free" was the wrong reading.res_commit ← __up_flush_block_invoke ← -[_MTL4CommitFeedbackDispatch executeBlocksWithCommitFeedback:] ← IOGPUNotificationQueueDispatchAvailableCompletionNotifications. That handler removes the batch's staging buffers from the set and commits the set, while the guest submit thread may be committing command buffers on the queue that uses the same set (rec_commit ← mtl_sub_commit). The debug layer's crash is in-[MTL4GPUDebugCommandBuffer applyResidencySets:]on the submit thread meeting a buffer the other thread's set commit had just released (-[MTLGPUDebugBuffer isKindOfClass:]: message sent to deallocated instance). That fits a race betweenMTLResidencySetcommit on one thread and a queue commit that applies the set on another.res_removeof the staging buffers andres_commit()inup_flush's feedback handler), and a main-based host with main's library did not hang on this route in 2 of 2 runs, so this race alone doesn't explain the difference.Next on our side: serialize the two (no set commit off the submit thread) and see whether the layer still crashes and whether the reproducer still hangs.
ycarogabriel-lgtm commentedon Nov 10, 2026
Base M1, follow-up 4: serializing the residency set commit with the queue commit removes the debug-layer crash but not the hang
Same reproducer (host
23272a83, matching library90e8d82142f4, docked, 60 fps target, scripted input,DM_NVN_ISOLATE_TASK=0, headless). In a local diagnostic build every Metal 4 queue commit now holds the lockres_committakes (g_res_mu), so the upload path's set commit on the completion thread can no longer overlap a command-buffer commit on the submit thread.MTL_SHADER_VALIDATION=1andNSZombieEnabled=YES: the run went its full 120 s to frame ~900 with no crash and no zombie. Without the serialization, the same setup ended in-[MTL4GPUDebugCommandBuffer applyResidencySets:]on a released buffer in 4 of 4 runs that got that far (at 14.8 s with zombies on). So that crash was the race between the two commits.kIOGPUCommandBufferCallbackErrorHang×4, then GPU access revoked), with frame p50 16.6 ms, submit+present p50 0.7–0.8 ms and GPU p50 5.7–5.9 ms in the minute before. The reproducer is now 14 of 14 without the layer.So the residency set race is real (the debug layer was right to complain) but it is not what hangs this save. We'll leave that path there. Note that holding
g_res_muacross a queue commit can deadlock if the commit ever waits for completions while a completion handler wants the lock (see #2098), so it is not a fix to land as is.seedds commentedon Nov 10, 2026
Looked at the M1 follow-ups on an M4 Pro (main at
bbea4c90, Ocarina libraries on chain0c434f1b68df).The residency-set crash is in Metal's validation layer, not a buffer the host freed. A residency set holds a strong reference to every allocation it lists until a commit after its removal. A buffer whose last reference is gone stays alive while it's in the set (checked standalone). A small program reproduces the same crash with correct locking. One thread adds a buffer, commits the set, removes it and commits again. Another thread commits empty
MTL4CommandBuffers on a queue that has the set. WithMTL_SHADER_VALIDATION=1andNSZombieEnabled=YESit stops within milliseconds:The same loop on one thread runs 20,000 iterations without a report, and so does the two-thread version without validation. The debug command buffer reads the set at its commit while the other thread's set commit releases a removed buffer. The host commits its set from more than one thread (deferred releases are committed off the submit thread), which Metal allows. So this crash says nothing about the hang. With validation on, it will keep ending runs early.
The pulling stand-ins' reports are real reads, not false positives.
t_pull, which checks every vertex format through the pulling fetch against Metal's own, runs clean underMTL_SHADER_VALIDATION=1. So the 2,190Out of bounds of user address spaceloads in your boot were real reads. I could not check whether current main still makes them: a validation boot here waits at frame 110 before the title.Your host predates two fixes that can stop this hang.
23272a83is main as ofe2b76d4eplus your picks. It has neithere4e5ff64(translated shaders compile with-fwrapv) nor26062f9e(a read past a uniform binding's size gives 0). Both are in8d5d4f1b, where the original report's scenes played through with no GPU errors. For mesh3439aaa071d868ae,-fwrapvchanges the compiled code a lot (about 21,000 lines of LLVM IR differ). The task's and mesh's raw global loads are checked against the pool table before they read, so an address in no pool reads 0 rather than faulting.@ycarogabriel-lgtm could you run the same save and input on current main, without your picks, using libraries built for its chain? If it still hangs there, please attach the log with the failed command buffers' shader list. A run that validates only the
a5288d0a6c4971fdpipelines would then be the next step. To avoid the crash above, it needs either validation off for the residency set's other users, or every set commit on the submit thread in your diagnostic build.ycarogabriel-lgtm commentedon Nov 10, 2026
Pure
mainon an M1: the hang did not reproduce in 2 runs that went well past the point where our branch hangs every timeYou asked for a run on current
mainwithout our picks. Short version: on6c676444with a full library for that commit, the route that hangs our older branch 14 times out of 14 (all by frame ~1,418-1,600) ran to frame 4,738 and 5,582 with nokIOGPUCommandBufferCallbackErrorHang, no failed command buffer and no[RED]line. Details and caveats below.What ran
6c676444(commit date 2026-11-10 11:21 +0800; origin/main when the build started), no local changesd403eb0683df. Fulldmsh_build.pyrun for that commit (shaders 42848: ok 42848,lost without failing: nothing, stampinputs 4bd48f723876),c1_full.tsv/cb_extent.tsvfrom the same buildDM_FPS_TARGET=60,DM_NVN_ISOLATE_TASK=0,DM_UPSCALER=metalfx, headless (no window)DM_KEYS=15:A*12/4), 355 s per run, empty pipeline bundle, a fresh pipeline archive at the start of run 1 that later runs reusemainhas moved on since: it is at641636e2now, with chaine1d97965d3f3. That includes6c3aeb37(#2077, the plaid terrain after dungeons), which these runs do not have. I did not rebuild for the newer chain.Results
[RED]ea0c98dfbase plus our picks) with the same save and input, 14 of 14 runs hung between frame ~1,418 and ~1,600. A failed command buffer held the task shaderf75708d176bb6591in 10 of 10 failures logged.Numbers from runs 2 and 3
[longframe]lines only, so they are not a mean frame time. The power line gives 15.3 fps averaged over the run.Run 1 note (separate from the hang)
new pipelines 12 ... 315237.5 ms in all.bases compiled 26 / 1304.5 ms. So almost all of that time was spent waiting, not compiling.Caveats
6c676444makes the difference. One candidate isea0c98df(NVN queue dependencies), which our hanging branch does not have, but I have not tested it.revert: the uniform bounds guard is back in this variant (prova 24b),…
seedds commentedon Nov 10, 2026
Closing, as you said: on
8d5d4f1b(chain0c434f1b68df) the Kokiri conversation and the Impa escort play through with no GPU errors, and a run of plainmainon an M1 (6c676444) went well past the frame where the older branch hangs. The likely fixes are 26062f9 (a uniform read past the size the game bound gives 0, from #1860) and e4e5ff6 (-fwrapv). If the hang comes back on currentmain, please reopen with the log'sbuild:line.