Repository navigation
Failure report: task / mesh draws with their grids, +pull marks, and DM_NVN_MESHLOG_CB (with an opt-in GPU copy of the task's uniform words) - #1
Open
lunks wants to merge 3 commits into
Conversation
…stuck pipelines Hyrule Warriors: Age of Imprisonment froze the GPU in its first outpost mission while a special attack filled the screen; the allocator wait of the next submission had no timeout, so the host hung in silence. Every blocking GPU wait now runs in 2 s slices, and after DM_NVN_GPU_WATCHDOG seconds (5 by default) with work in flight a [RED] line names the frame, counts and pipelines of the submission the GPU is stuck on, from a per-submission record of its vertex/fragment pairs and compute shaders, and points at DM_NVN_SKIP_FS to play on. The cause is open: the shaders' light-list loops run in the first battle at 1x and 2x with an empty list and never hang (game README, "GPU hang in a battle"). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 7083aa2)
…n record is printed in the command-buffer error path too When macOS aborts a command buffer that ran too long (Ocarina of Time, issue seedds#1326), the fence completes with an error and g_done moves on, so the hang watchdog, which needs a fence that never completes, does not fire. The feedback handlers now print the same record for a submission that had a failure of its own: once its last command buffer has come back, once per fence, for the first 16 fences. A failed segment leaves a mark for the last handler to print. A segment whose handler runs after the last one prints the record itself. The record is now kept per command buffer of the submission, a pair once in each, so a failed segment lists every pipeline it ran. Pairs are in the order they first ran, each marked with the render pass it began in (counted from the submission's first, whose DM_NVN_PASSDUMP number the header gives). Victims and refused submissions are left out, through the filter cbs_feedback already used. The record holds up to 48 pairs and only counts the rest. Its text is built by one formatter that stays inside its buffer; the watchdog uses it too. Main's shader lists name each game shader once per failed command buffer and rank them across failures. This line shows, for the one failed submission, which vertex shader drew with which fragment shader, its compute shaders and the host's builtins, in order, with the failed command buffer marked. t_gpu_watchdog checks the record, its text, both handler orders, once per fence, victims and the watchdog's single report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ulling stand-ins, and DM_NVN_MESHLOG_CB with an opt-in GPU copy of the task's uniform words For issue seedds#1326 (Ocarina of Time's foliage-pass hangs): the foliage task a5288d0a loops forever only if the grid size at c3[193] is garbage when the GPU reads it, and the failure report did not say. Each task / mesh draw of the failed submission now appears in the report with its grid (a GPU-written one's count as the copy holds it when the report prints), a draw through its pulling stand-in is counted and its pair marked +pull, and DM_NVN_MESHLOG_CB=<register>:<offset> adds four words of the task's uniform buffer: as the CPU read them at encoding, where the buffer was bound (guest address, size, Metal GPU address), and the memory as it is when the report prints. DM_NVN_MESHLOG_CB_GPU=1 also ends the render pass before each task draw and has the GPU copy the same words in queue order (0xfeedface if the copy never ran); it splits the pass at every task draw, so it can change whether the hang happens. No cap on any grid, no shader or chain change. Tests: t_gpu_watchdog (the lines), t_mlcb_gpu (the GPU copy sees a write made after encoding).
Amost-
force-pushed
the
pr/gpu-watchdog
branch
from
November 11, 2026 10:14
0694d7e to
67516f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
For seedds#1326 (Ocarina of Time's foliage-pass hangs). seedds read the foliage task
a5288d0a6c4971fd's translation: its loops are finite unless the grid size atc3[193]is garbage, which would explain the hang and the page faults. What is open is whyc3[193]would be wrong when the GPU reads it, and the failure report doesn't say. On top of seedds#1550's record this adds:mdx_runmade holds it when the report prints;+pull;DM_NVN_MESHLOG_CB=<register>:<byte offset>(off by default): four words of the task stage's uniform buffer (0:3088isc3[193]and the next three words), as the CPU read them when the draw was encoded, with where the buffer was bound (guest address, size, Metal GPU address) and the memory as it is when the report prints. A difference is a write after the draw was encoded;DM_NVN_MESHLOG_CB_GPU=1with it: the render pass is ended before each task draw and the GPU copies the same four words in queue order into a host buffer; the report prints them asGPU copy ahead of the draw(0xfeedfaceif that copy never ran). It splits the pass at every task draw, so it can change whether a hang happens; use it to see a value, not to count hangs. Documented in the reference entry.No grid is capped and nothing about drawing changes with the switches off. No shader or chain change.
Reference entry: the failed-submission section of
docs/reference/rendering.md(the new bullet).Checked
make -C dm/gfx/tests GAME=0400486024DEC000 check_t_gpu_watchdog(27 checks: the lines, a pool-backed case with the memory changed after the draw),check_t_mlcb_gpu(10 checks, new: the GPU copy sees a write made after the draw was encoded),check_t_nvn(243),check_t_geometry(1,929),check_t_gmem_words(16).docs_symptoms.py --checkcurrent.regress.pynot run: host diagnostics, off unless the switches are set.Docs
docs/reference/<subsystem>.mdentry, andpython3 dm/tools/docs_symptoms.pywas run (the symptom index)