Repository navigation
Guest queue scheduler: DM_NVN_CHUNK_MERGE encodes the chunks one pump offers into one command buffer (follow-up to #1245) - #1749
Open
aksmith-git wants to merge 2 commits into
Open
aksmith-git wants to merge 2 commits into
aksmith-git wants to merge 2 commits into
Conversation
Event and sync waits were ignored, allowing consumers to read stale input when another guest queue had not yet submitted its producer. This implements ready-prefix scheduling: a blocked queue yields so its producer can be submitted later, including by the same CPU thread. CPU-visible event values are published after GPU completion. Preserve render targets, conditional rendering, visibility queries and submission-mode timestamps across event boundaries. Reuse command allocators across chunks without resetting GPU-owned data, while keeping the allocator pool bounded. Wait-only submissions retain the producer's completion fence. Validation covers forward and bidirectional dependencies, CPU signals, queue ordering, resource retirement, allocator pressure, both timestamp modes and capture compatibility. Scheduler tests also pass with AddressSanitizer and UndefinedBehaviorSanitizer; GPU tests pass with Metal validation enabled. Front-ends: break: rebuild host and replay tools together for the updated capture layout. Older captures are rejected; raw replay of event/sync waits remains unsupported.
… offers into one command buffer With the event queue ordering a guest submission is cut into chunks at its event waits, signals and recorded fences, and each chunk was a host submission of its own: a command buffer, an allocator lease, a visibility buffer, a completion record, a commit and, with the encode thread, a drain. Ocarina of Time (Switch 2) on an iPad Pro (M5), Docked 1536x864, went from 6 to 37 submissions a frame, with the GPU draining between a chunk and the next while the host committed it: 2.5 ms (p50) of GPU idle inside a 22.9 ms frame against none in a 20.4 ms frame with the waits ignored, at the same GPU time; the 64-draw segment commits fell from 6.4-6.9 to 1-3 a frame, so the 760-draw colour pass was committed whole. DM_NVN_CHUNK_MERGE=1 (off by default; ab with DM_AB): qe_pump keeps a chunk's command buffer open and encodes the chunks it offers next into it (mtl_chunk_end / mtl_chunk_begin: the chunk's own landing and per-chunk state, without a commit or a new buffer, allocator, visibility buffer or completion record). It commits when the pump has no ready step left, before a present, and in submission timestamp mode when the next chunk belongs to another logical submission (one clock per buffer). The open buffer's fence is mtl_pending_fence() = g_fence + 1 for the chunk and for the signal's projection; every CPU wait on a fence runs its pump first under the submission lock, so no thread waits on a fence not yet committed. The order on the Metal queue is the scheduler's as before: a chunk whose wait is not satisfied still waits for its producer's offer. The [nvn] timing line counts the scheduler's chunks per frame and how many were merged. check_t_nvn runs the event tests with a submission per chunk and again with DM_NVN_CHUNK_MERGE=1. Reference entry in docs/reference/nvn-objects.md, the switch in docs/env_switches.tsv.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Follow-up to #1245 (sn0wst0rm), on its head
5269ecc5: this branch is that PR plus one commit,e364b8c6, so the diff here shows #1245 too until it lands. Opened againstmainas the template asks; if it is better folded into #1245 itself, the commit cherry-picks onto5269ecc5as is.Symptom. With the guest event queue ordering, Ocarina of Time (Switch 2) on an iPad Pro (M5), Docked 1536x864, runs with the GPU idle between the scheduler's chunks. The game makes about 6 submissions a frame; cut at its event signals, waits and recorded fences they become 37 host submissions, and the GPU drains between a chunk and the next while the host commits it:
GPU time per frame is the same in both. The frame listing shows 35 chunks for 6 guest submissions, alternating queue 5 (compute) and queue 1 (graphics) about 12 times, cut by 19 signals and 26 waits that were all satisfied by the scheduler before the next chunk; the
DM_NVN_GPUPROFtimeline shows 1.6 ms of idle before the 760-draw colour pass (the segment counter restarts per chunk, so the pass is committed whole where it used to be fed in 64-draw segments) and about 60 smaller gaps at the other chunk boundaries.Cause. Every chunk
qe_pumpoffers is a host submission of its own: a command buffer, an allocator lease, a visibility buffer, a completion record, a commit and, with the encode thread, arec_drain(). It is not a CPU wait on the previous chunk's fence (qe_readonly picks the projected value over the mapped word, and sync waits are satisfied bynvn_qs_scheduled); it is the granularity.Change.
DM_NVN_CHUNK_MERGE=1(off by default;abalternates it withDM_AB):qe_pumpkeeps a chunk's command buffer open and encodes the chunks it offers next into it.mtl_chunk_endlands the chunk's passes, clears, deferred draws, uniform updates and blit encoder (whatmtl_sub_commitdoes before it commits), the queue is sealed as after a commit,mtl_chunk_beginsets the per-chunk state ofsub_beginwithout a new buffer, allocator, visibility buffer, completion record or segment clock. The buffer commits when the pump has no ready step left (every pump ends committed), before a present, and in submission timestamp mode when the next chunk belongs to another logical submission (post_runtakes one clock per buffer; in segment mode chunks of any submission share one). The scheduler records the open buffer's fence asmtl_pending_fence()=g_fence + 1(the conventionOP_FENCE_SYNCand the PASSGPU ring use) for the chunk and for the signal's projection; every CPU wait on a fence runs its pump first under the submission lock, so no thread can wait on a fence not yet committed. Ordering is unchanged: the chunks go onto the one Metal queue in the order the scheduler offered them, and a chunk whose wait is not satisfied still waits for its producer's offer. What changes is that a satisfied wait, a signal or a recorded fence costs no commit, allocator, visibility buffer, completion record or drain.The
[nvn] timingline gainsscheduler chunks per frame N, M of them merged into an open command buffer. Reference entry:docs/reference/nvn-objects.md, "Scheduler chunks committed one by one"; the switch indocs/env_switches.tsv.Checked
GAME=0400C3F00006E000, since this base has no Ocarina of Time profile):t_queue_schedule295 checks / 0 failed,check_t_capture82 / 0,check_t_queue_vis94 / 0,check_t_queue_alloc388 / 0 (32 split chunks, 0 allocator waits),check_t_nvn327 / 0 both ways (it now runs once plain and once withDM_NVN_CHUNK_MERGE=1), andDM_NVN_CHUNK_MERGE=1 DM_NVN_GPU_TS=submission ./t_nvn327 / 0 (the one-clock-per-buffer rule).dm/tools/tests/test_docs.py11 OK;test_env_switches.pyfails on this base as before the change (DM_NVN_CUT_FIRST,DM_TEST_ALLOC_LEGACY, not touched here).make -C dm/host GAME=0400C3F00006E000 dmhostbuilds (0 errors).gpu_ts_hz, the tick rate of raw counter reports (default 614.4 MHz, unchanged): Ocarina of Time reads them at 31.25 MHz, so its wind and leaves moved 19.66 times too fast #1714 and our changes) builds and passes the same tests there (t_nvn336 / 0 both ways); the iPad run of the merged form is queued and I will post its numbers here. The numbers above are the before.regress.py(off by default; the path exists only once a game has recorded an event wait).Docs
docs/reference/nvn-objects.md, andpython3 dm/tools/docs_symptoms.pywas run (the symptom index)