Skip to content

nvn: implement nvnCommandBufferWaitEvent and per-queue command parking - #1728

Closed
ksolenk38ze wants to merge 1 commit into
seedds:mainfrom
ksolenk38ze:pr/nvn-wait-event-queue-parking
Closed

ksolenk38ze wants to merge 1 commit into
seedds:mainfrom
ksolenk38ze:pr/nvn-wait-event-queue-parking

Conversation

@ksolenk38ze

@ksolenk38ze ksolenk38ze commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What and why

nvnCommandBufferWaitEvent was a no-op stub, and when a game submitted command buffers across multiple NVNqueue instances (q=0 compute/transfer and q=1 graphics), SubmitCommands drained and executed the secondary queue immediately on the calling thread instead of ordering it behind earlier submissions waiting on a shared NVNevent. In The Legend of Zelda: Ocarina of Time (0400486024DEC000), compute skinning, shadow/sky passes, and object updates across queues race ahead of or behind q=1 draws when event waits are ignored, causing one-frame shadow/sky drops, flickering or missing dynamic meshes, and synchronization stalls.

This change:

  • Records P_WAIT_EVENT commands in nvnCommandBufferWaitEvent and P_SIGNAL_EVENT in nvnCommandBufferSignalEvent, tracking whether a command handle contains event waits (has_wait) or signals (has_signal).
  • Parks submissions on a per-queue pending ring (q_submit_or_park) when an unsatisfied P_WAIT_EVENT is encountered, and drains parked queues (q_drain_parked) as soon as a matching SignalEvent or CPU write satisfies the condition.
  • Evaluates P_WAIT_EVENT at replay boundaries in nvn_mtl.m (ev_wait_ready, bounded spin for cross-queue event writes) and wakes parked queues when submission post-op events complete.
  • Documents the mechanism in docs/reference/rendering.md and updates docs/agents/symptoms.md.

Related

Checked

  • Tests: make -C dm/gfx/tests GAME=0400486024DEC000 run (t_pool_notes, t_upgrade_alias, t_image_alias, t_alias_takeover, t_srgb_alias, t_rt_lifetime, t_pool_chunks), python3 dm/tools/docs_symptoms.py
  • Games run, with title ID and version: The Legend of Zelda: Ocarina of Time (0400486024DEC000, v131072, SDK 22.2.8), played through Kokiri Forest, Hyrule Field, Zora's Domain, and Forest Temple at 60 fps.
  • Devices and OS: Apple Silicon Mac, macOS 26.2

Docs

  • The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
  • No game files, keys or data extracted from a game in the change or the attachments

@github-actions github-actions Bot added the nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) label Nov 7, 2026
When a guest records nvnCommandBufferWaitEvent on an NVNevent whose counter is
signaled later on another NVNqueue or from the CPU via nvnEventSignal, the
waiting queue must pause replay at the WaitEvent boundary until the target
event value is reached. Executing past an unsatisfied WaitEvent reads compute
or render outputs before the signaling queue has produced them, while spinning
on the submission lock deadlocks when the signal is submitted later on the
same thread.

Record OP_SIGNAL_EVENT with bit 1 set in nvnCommandBufferWaitEvent, track the
latest GPU-signaled value per event address across in-flight command buffers,
and park the active queue's remaining command stream (including its active
OP_CALL return stack and pending post-submission callbacks) when a WaitEvent
is not yet satisfied. Unpark and drain ready queues when an event is signaled
on the GPU or CPU, or when nvnSyncWait, nvnQueueFlush, nvnQueueFinish, or
nvnQueuePresentTexture flushes pending work.

Verified in The Legend of Zelda: Ocarina of Time (0400486024DEC000) on Apple
M-series (macOS) and with the dm/gfx/tests suite.
@aksmith-git

Copy link
Copy Markdown
Contributor

We are evaluating this against #1245 on an iPad Pro (M5) with Ocarina of Time (0400486024DEC000 v131072), Docked 1536x864.

Data point for why: #1245 as merged on our stack fixes the game's one-frame fog and shadow flicker (#1703), but its chunking costs the iPad. The game makes about 6 submissions a frame; cut at every event signal and wait they become 37 host submissions, and the GPU drains between a chunk and the next while the host commits it: 2.5 ms (p50) of GPU idle inside a 22.9 ms frame, against 0 idle in a 20.4 ms frame with the waits ignored, at the same GPU time. Parking keeps one command buffer per QueueSubmit, which is the shape the iPad needs, so this PR is the one we want to compare. It cherry-picks onto our branch (main 01abf27 base plus our changes) with three small conflicts in nvn_mtl.m (g_queue_post_base, its reset in sub_begin, ev_gpu_land beside our post_run lines) and builds; the device run is queued and I will post the frame numbers here.

Two points to watch from reading it, no result yet:

@ksolenk38ze

Copy link
Copy Markdown
Contributor Author

Thanks @aksmith-git for testing this on the iPad Pro (M5)! Keeping one host submission per QueueSubmit and draining parked queues as soon as nvnEventSignal or an earlier queue's post-op signals the event was specifically aimed at avoiding per-event command buffer chunking overhead while eliminating the cross-queue race (#1703).

Looking forward to your iPad frame numbers, and happy to add a counter/log line on mtl_flush_parked if the 50 ms fallback is ever reached in your runs.

@seedds

seedds commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Thanks. The mechanism landed in ea0c98d (OP_WAIT_EVENT, with queues parked until their producer signals the event). Its cost is a pass and a host submission cut at every signal event, which is what this PR's design of one host submission per QueueSubmit was meant to avoid. That cost is tracked in #2112 and PR #2116. Closing as superseded.

@seedds seedds closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants