Repository navigation
Conversation
…s, instead of only a fence and a few counters In issue seedds#1326 (Ocarina of Time on a Mac) a GPU hang, in seedds#1289 (Splatoon Raiders) a page fault, both ending in the revoked access seedds#1332 now stops on. Four reports across two games agree on the shape of the failure and not one can say which pass or which shader the GPU died in: the host printed the fence, Metal's error and the counters of cctx_snap ("passes +105, serial table changes +44"), and a segment's own command buffer printed no context at all. seedds#1332's description names this as its second piece. So every main command buffer records the render passes encoded into it, from fail_cb_begin at its start (mtl_sub_begin, the one after a segment cut, the one after mdi_wait) to g_stat_pass at its commit, and a ring of the last 256 passes holds each pass's size, attachment counts, draw count and the fragment shader key of its first draw, taken where the encoder label takes it (draw_). When a command buffer comes back with an error, its handler prints the range and the ring entries that fall in it: [RED] the submission command buffer that failed: fence 107793, frame 17964, segment 3, passes 18201..18305 (105 passes) [RED] pass 18201 1920x1080 colors 1 depth 1 draws 342 first fragment shader 09ab31c6 [RED] pass 18202 256x128 colors 1 depth 0 draws 1 first fragment shader 47f9b384 The key is the one shader_map.tsv maps back to the shader. Both failing paths print it, the submission's last command buffer and a segment's, the first four of each; an innocent victim does not, as it names work that did not fail. It is always on, with no switch: the failure it is for appears about twelve minutes into play, and a switch that has to be set before then is a switch nobody has set. A pass costs a few stores and the size of its first attachment, a draw one compare, and nothing is formatted until a command buffer fails. The submit thread writes the ring and a feedback thread reads it without a lock: a torn entry would misname one pass in a report the host is about to stop after, which is worth more than a lock on every pass. A command buffer with no render pass is the common case in those logs, where most failed fences carry "passes +0". It says what it held rather than printing an empty list. A range the ring has lost says that too, with how many passes came after it. t_fail_dump: 10 checks, 0 failed, on an M1 Max. The pass range is the one the command buffer encoded; another command buffer's passes stay out of it; each pass carries its size, its draws and its first fragment key, cut to eight digits; a pass with no draw, a command buffer with no pass and a range the ring has lost each say so. The Mac host builds, and dm/gfx/tests run gives 70,697 checks, 0 failed. regress.py was not run: this Mac's NSP folders are not in the layout its game discovery reads, and the change adds two calls on the pass and draw paths and formats only when a command buffer has already failed. No real GPU fault was reproduced against it, as in seedds#1332. Refs seedds#1326 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Closing this: its premise is wrong. I opened it saying the host cannot name the shader a failed command buffer was running. It can, and has been able to for some time: What is left of this change is the pass range of a failed command buffer and a per-pass breakdown, which The correction is on the issue: #1326 (comment 6708780929). Sorry for the round trip. |
In issue #1326 (Ocarina of Time on a Mac) a GPU hang, in #1289 (Splatoon Raiders) a page fault, both ending in the revoked access #1332 now stops on. Four reports across two games agree on the shape of the failure and not one can say which pass or which shader the GPU died in. The host printed the fence, Metal's error and the counters of
cctx_snap:and a segment's own command buffer printed no context at all. #1332's description names this as its second piece; this is that piece.
What it does
Every main command buffer records the render passes encoded into it, from
fail_cb_beginat its start (mtl_sub_begin, the one after a segment cut, the one aftermdi_wait) tog_stat_passat its commit. A ring of the last 256 passes holds each pass's size, attachment counts, draw count and the fragment shader key of its first draw, taken where the encoder label already takes it indraw_. When a command buffer comes back with an error, its handler prints the range and the ring entries that fall in it:The key is the one
shader_map.tsvmaps back to the shader. Both failing paths print it, the submission's last command buffer and a segment's, the first four of each. An innocent victim does not, as it names work that did not fail.Two cases the logs made clear are worth their own line rather than an empty list:
passes +0: uploads, copies or compute. It says so.Why it is always on
The failure it is for appears about twelve minutes into play. A switch that has to be set before then is a switch nobody has set, and the reporter only finds out it was needed after the run is gone.
A pass costs a few stores and the size of its first attachment; a draw costs one compare; nothing is formatted until a command buffer has already failed. The submit thread writes the ring and a feedback thread reads it without a lock: a torn entry would misname one pass in a report the host is about to stop after, which is worth more than a lock on every pass.
Checked
t_fail_dump: 10 checks, 0 failed on an M1 Max (macOS 27.0.1). The pass range is the one the command buffer encoded; another command buffer's passes stay out of it; each pass carries its size, its draws and its first fragment key cut to eight digits; a pass with no draw, a command buffer with no pass and a range the ring has lost each say so.make -C dm/gfx/tests run: 70,697 checks, 0 failed.Not checked
regress.pywas not run. This Mac's NSP folders are not in the layout its game discovery reads. The change adds two calls on the pass and draw paths and formats only once a command buffer has failed, so its reach is narrow and it cannot change what is drawn.lint-chas not run locally (no clang-tidy 23.1.4 here); CI's ratchet will say whethernvn_mtl.mgained a finding.Refs #1326
🤖 Generated with Claude Code