Skip to content

A command buffer that fails names the passes it held and their shaders, instead of only a fence and a few counters - #1680

Closed
crypt0dj wants to merge 1 commit into
seedds:mainfrom
crypt0dj:gpu-fail-pass-ring
Closed

crypt0dj wants to merge 1 commit into
seedds:mainfrom
crypt0dj:gpu-fail-pass-ring

Conversation

@crypt0dj

@crypt0dj crypt0dj commented Nov 7, 2026

Copy link
Copy Markdown

In issue #1326 (Ocarina of Time on a Mac) a GPU hang, in #1289 (Splatoon Raiders) a page fault, both ending in the revoked access #1332 now stops on. Four reports across two games agree on the shape of the failure and not one can say which pass or which shader the GPU died in. The host printed the fence, Metal's error and the counters of cctx_snap:

[RED] state of the failed submission fence=107793: frame 17964; since the last submission: passes +104, first texture uploads +0, re-uploads +0, serial table changes +43; deferred-release queue 282
[RED] command buffer execution error: The operation couldn't be completed. (MTL4CommandQueueErrorDomain error 1.)

and a segment's own command buffer printed no context at all. #1332's description names this as its second piece; this is that piece.

What it does

Every main command buffer records the render passes encoded into it, from fail_cb_begin at its start (mtl_sub_begin, the one after a segment cut, the one after mdi_wait) to g_stat_pass at its commit. A ring of the last 256 passes holds each pass's size, attachment counts, draw count and the fragment shader key of its first draw, taken where the encoder label already takes it in draw_. When a command buffer comes back with an error, its handler prints the range and the ring entries that fall in it:

[RED] the submission command buffer that failed: fence 107793, frame 17964, segment 3, passes 18201..18305 (105 passes)
[RED]   pass 18201 1920x1080 colors 1 depth 1 draws 342 first fragment shader 09ab31c6
[RED]   pass 18202 256x128 colors 1 depth 0 draws 1 first fragment shader 47f9b384

The key is the one shader_map.tsv maps back to the shader. Both failing paths print it, the submission's last command buffer and a segment's, the first four of each. An innocent victim does not, as it names work that did not fail.

Two cases the logs made clear are worth their own line rather than an empty list:

  • A command buffer with no render pass. Most failed fences in those reports carry passes +0: uploads, copies or compute. It says so.
  • A range the ring has lost, with how many passes were encoded after it.

Why it is always on

The failure it is for appears about twelve minutes into play. A switch that has to be set before then is a switch nobody has set, and the reporter only finds out it was needed after the run is gone.

A pass costs a few stores and the size of its first attachment; a draw costs one compare; nothing is formatted until a command buffer has already failed. The submit thread writes the ring and a feedback thread reads it without a lock: a torn entry would misname one pass in a report the host is about to stop after, which is worth more than a lock on every pass.

Checked

  • t_fail_dump: 10 checks, 0 failed on an M1 Max (macOS 27.0.1). The pass range is the one the command buffer encoded; another command buffer's passes stay out of it; each pass carries its size, its draws and its first fragment key cut to eight digits; a pass with no draw, a command buffer with no pass and a range the ring has lost each say so.
  • The Mac host builds.
  • make -C dm/gfx/tests run: 70,697 checks, 0 failed.

Not checked

Refs #1326

🤖 Generated with Claude Code

…s, instead of only a fence and a few counters

In issue seedds#1326 (Ocarina of Time on a Mac) a GPU hang, in seedds#1289 (Splatoon Raiders) a page fault, both ending in the revoked access seedds#1332 now stops on. Four reports across two games agree on the shape of the failure and not one can say which pass or which shader the GPU died in: the host printed the fence, Metal's error and the counters of cctx_snap ("passes +105, serial table changes +44"), and a segment's own command buffer printed no context at all. seedds#1332's description names this as its second piece.

So every main command buffer records the render passes encoded into it, from fail_cb_begin at its start (mtl_sub_begin, the one after a segment cut, the one after mdi_wait) to g_stat_pass at its commit, and a ring of the last 256 passes holds each pass's size, attachment counts, draw count and the fragment shader key of its first draw, taken where the encoder label takes it (draw_). When a command buffer comes back with an error, its handler prints the range and the ring entries that fall in it:

    [RED] the submission command buffer that failed: fence 107793, frame 17964, segment 3, passes 18201..18305 (105 passes)
    [RED]   pass 18201 1920x1080 colors 1 depth 1 draws 342 first fragment shader 09ab31c6
    [RED]   pass 18202 256x128 colors 1 depth 0 draws 1 first fragment shader 47f9b384

The key is the one shader_map.tsv maps back to the shader. Both failing paths print it, the submission's last command buffer and a segment's, the first four of each; an innocent victim does not, as it names work that did not fail.

It is always on, with no switch: the failure it is for appears about twelve minutes into play, and a switch that has to be set before then is a switch nobody has set. A pass costs a few stores and the size of its first attachment, a draw one compare, and nothing is formatted until a command buffer fails. The submit thread writes the ring and a feedback thread reads it without a lock: a torn entry would misname one pass in a report the host is about to stop after, which is worth more than a lock on every pass.

A command buffer with no render pass is the common case in those logs, where most failed fences carry "passes +0". It says what it held rather than printing an empty list. A range the ring has lost says that too, with how many passes came after it.

t_fail_dump: 10 checks, 0 failed, on an M1 Max. The pass range is the one the command buffer encoded; another command buffer's passes stay out of it; each pass carries its size, its draws and its first fragment key, cut to eight digits; a pass with no draw, a command buffer with no pass and a range the ring has lost each say so. The Mac host builds, and dm/gfx/tests run gives 70,697 checks, 0 failed.

regress.py was not run: this Mac's NSP folders are not in the layout its game discovery reads, and the change adds two calls on the pass and draw paths and formats only when a command buffer has already failed. No real GPU fault was reproduced against it, as in seedds#1332.

Refs seedds#1326

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) label Nov 7, 2026
@crypt0dj

crypt0dj commented Nov 7, 2026

Copy link
Copy Markdown
Author

Closing this: its premise is wrong.

I opened it saying the host cannot name the shader a failed command buffer was running. It can, and has been able to for some time: cbs_feedback in nvn_mtl.m keeps the game shaders each command buffer encoded, prints them when that command buffer fails, and ranks the suspects across failures as key failed/ran. It ran in the very log I used as the motivation here — four lines I read past. That ranking is a better instrument than what this adds, because it separates a shader that is present at every failure from one that merely runs everywhere.

What is left of this change is the pass range of a failed command buffer and a per-pass breakdown, which cbs does not give. That is real but small, and not worth a reviewer's time on top of a wrong premise, so I am not rescoping it into this PR. If the pass range turns out to be wanted later it is a handful of lines and can be proposed on its own.

The correction is on the issue: #1326 (comment 6708780929). Sorry for the round trip.

@crypt0dj crypt0dj closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants