[PucaVaz]

What and why

Refs #1326. This is the second piece I described there: when the GPU hangs or faults, the log now names the work that failed.

Symptom. dacONJA's Ocarina of Time run on #1326 hung the GPU about 7 minutes in. Like every GPU error so far, the log had only [RED] segment command buffer execution error: Error Domain=MTL4CommandQueueErrorDomain Code=1 "(null)" with kIOGPUCommandBufferCallbackErrorHang nested inside. It gave no frame, segment, pass or shader. Naming the shader took a rerun with lunks's DM_NVN_HANGLOG=1 DM_NVN_CUT=1 on his branch.

Cause. Metal 4 reports a GPU error only as the commit feedback's error; it has nothing like Metal 3's encoder execution status. The feedback handlers of the main command buffers kept nothing about what their command buffer held.

Change (nvn_mtl.m, always on):

• Each main command buffer is labelled f<frame> s<segment> p<first pass>-<last pass>, for Xcode GPU captures and Instruments. That covers sub_cut's segments, mtl_sub_commit's last segment and mdi_wait's count wait; the count wait now has a feedback handler for this. With DM_NVN_ENC_THREAD the label goes through the encode thread's record (rec_cb_block).
• Each command buffer's entry keeps the same fields plus its fence, its draw and dispatch counts, and the keys of the game shaders its draws and dispatches ran. Each key is stored once, through a stamp on NvShader, and only the first 32 are kept; the rest are counted. Keys are noted in draw_ (the program, plus task/mesh and geometry-mesh stages), in dispatch and in geo_gs_one.
• The feedback handler keeps its command buffer's entry by value, so a hang reported seconds later, after hundreds more segments, still names it. A ring keeps the last 256 for the ones before it.
• A command buffer whose error is its own prints a header with the error's kind, then its entry and the three before it, oldest first. Innocent victims print nothing (the same InnocentVictim test mtl_sub_commit already uses). The kinds are: a hang, a page fault, out of memory, or ignored once macOS revoked the GPU access, in which case the ones before it are the last the GPU ran.
• An entry prints once while the ring holds it. An ignored command buffer prints only the first time, and there are at most 8 headers per run. From the test:

[RED] failed GPU work: main command buffer #2 f7 s1 p2-2 caused a GPU hang (imitated by DM_TEST_GPU_HANG); the game's shaders it and the command buffers before it ran, oldest first (v vertex, f fragment, g geometry, h hull, d domain, t task, m mesh, c compute):
[RED]   #1 f7 s0 p1-1 (segment of fence 1): draws 2, dispatches 0; v:a0a0a0a0a0a0a0a1 f:a0a0a0a0a0a0a0a2
[RED]   #2 f7 s1 p2-2 (segment of fence 1): draws 1, dispatches 1; c:c0c0c0c0c0c0c0c3 v:b0b0b0b0b0b0b0b1 f:b0b0b0b0b0b0b0b2  <- failed

DM_TEST_GPU_HANG=<n> makes the nth main command buffer report a hang in the shape of #1326's log while its work still runs. That number is the #n in the lines.

Credit. I took these from lunks's DM_NVN_HANGLOG (on fix/gmem-pool-table): the segment tagging, reading the error's kind from its description, and noting shaders at the draw and the dispatch. Printing the keys of the failed work and of the work before it, each key once, follows wk_dump in #1275 (uncraftablexd). Here it is kept per command buffer, not per submission. #1275's watchdog is not part of this PR.

Interplay with #1332. This PR is independent of #1332, and either can land first:

• In the segment and submission feedback handlers, cbn_failed goes right after gpu_tag_done, so it runs before A GPU whose access macOS revoked stops the game instead of leaving it running with nothing drawn #1332's gpu_lost_check. That order matters: without a window, gpu_lost_check ends the run with an abort, and the dump has to come first.
• A trial merge of the two branches (git merge-tree) conflicts only in docs/env_switches.tsv, where DM_TEST_GPU_HANG and A GPU whose access macOS revoked stops the game instead of leaving it running with nothing drawn #1332's DM_TEST_GPU_LOST both go after DM_TEST_GEOMETRY_PRIMITIVE_ID. Keep both rows.

Reference entry: Failed GPU work named by its command buffer. The symptom index is regenerated.

Checked

• Tests:

  • make -C dm/gfx/tests GAME=010015100B514000 check_t_gpu_hang_names: the new test, run with and without DM_NVN_ENC_THREAD, 12 checks each, all pass. It runs one submission of three segments with known shaders, and the second one imitates a hang. The checks cover:

    • the dump names f7 s1 p2-2 with its compute and draw keys, lists the segment before it, and leaves out the one after it;
    • no error and an innocent victim print nothing;
    • a page fault prints only what was not printed yet;
    • an ignored command buffer prints once;
    • a hang reported after the ring has gone round still names its command buffer;
    • the label is on the command buffer.

    6 of the 12 checks fail with the segment's cbn_failed call taken out.

  • make -j8 -C dm/gfx/tests GAME=010015100B514000 run: passes.

  • python3 -m unittest dm/tools/tests/test_docs.py dm/tools/tests/test_env_switches.py dm/tools/tests/test_gfx_tests_makefile.py: OK.

  • clang-tidy (dm/tools/lint_c.py tidy on t_gpu_hang_names.m and nvn_mtl.m) finds nothing above the baseline. I ran LLVM 22.1.1 locally; CI uses 23.1.4.

  • nvn_mtl.m compiles with the host's flags with the same 56 warnings as main.

• Cost when nothing fails, timed by the test on an M4 without the Metal debug layer, with or without the encode thread:

  • about 1.9 ns per draw whose shaders are already noted;
  • 9.5 ns for a program's first draw in a command buffer;
  • 175–190 ns per command buffer for its label and the ring copy.

  Over a 17,000-frame session, Ocarina of Time averaged 531 draws and 84 passes per frame with 4.7 ms of submit time per frame. That puts this under 10 µs per frame, so it stays on with no switch.

• Games run: The Legend of Zelda: Ocarina of Time (0400486024DEC000, v131072), windowed, on lunks's 574a38f with this commit cherry-picked ([WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299 has the profile), with DM_TEST_GPU_HANG=8000. At frame 1155, 41 s in, during the title, the log printed the lines below, using the game's own shader keys. The game had run 1,164 frames with no other new red line, and the run harness stopped it on the line. I did not play further.

  [RED] failed GPU work: main command buffer #8000 f1155 s0 no pass caused a GPU hang (imitated by DM_TEST_GPU_HANG); …
  [RED]   #7997 f1154 s0 p3375-3376 (segment of fence 6931): draws 1, dispatches 14; c:49dff010b9832251_b c:ba079f68ae288dba_b c:a8b4fb48fee0ccf8_b c:e2eebd7ba18fe35f_b c:e6be82d835fbe4d3_b c:9442573ee4ace011_b c:0a56002cdc7e1fce_b c:d15370a382630d4e_b c:7f7ff0f36b54d90f_b c:5a7d69b067a7f011_b c:4f41ccf24f163e0d_b v:8f9728d9b2b88600_b f:29e3c9dd92b8a571_b
  [RED]   #7998 f1154 s1 p3377-3377 (last segment of fence 6931): draws 1, dispatches 0; v:a5b1bd74e47458d7_b f:29e3c9dd92b8a571_b
  [RED]   #7999 f1155 s0 no pass (last segment of fence 6932): draws 0, dispatches 0; no game shaders
  [RED]   #8000 f1155 s0 no pass (last segment of fence 6933): draws 0, dispatches 0; no game shaders  <- failed
  

• Devices and OS: M4 (24 GB), macOS 27.2, Xcode 27.2. Not checked on iOS.

• regress.py was not run. The change adds bookkeeping (a stamp compare per shader of a draw, a label and two entry copies per command buffer) and log lines that print only on a GPU error, and Ocarina of Time's profile is not on main yet ([WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299). No real hang or page fault was reproduced; only the imitation was.

Docs

• The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change
• No game files, keys or data extracted from a game in the change or the attachments

Drafted with Claude; reviewed by me.

🤖 Generated with Claude Code

[PucaVaz]

clang-tidy: the one finding above the baseline is in dm/gfx/tests/t_int_float_view.m from main's f083873 (NullablePassedToNonnull), which this PR doesn't touch; #1338 fixes it. This PR's files have nothing above the baseline.

Drafted with Claude; reviewed by me.

[PucaVaz]

The Tool tests failure was mine: test_dmsh_abi's ratchet counted the "?" placeholder for a shader without a key as a third ABI literal in nvn_mtl.m ("?" is also an ABI kind name). It now reads "unkeyed" (c87cc8c). test_dmsh_abi.py and the whole tool-test loop pass locally, and check_t_gpu_hang_names still gives 12 checks, 0 failed. The clang-tidy finding left is main's t_int_float_view.m, as noted above.

Drafted with Claude; reviewed by me.

[PucaVaz]

Closing: main's f956ffb now names a failed command buffer's game shaders (cbs_feedback), which covers what this PR did.