Skip to content

Frame generation: a ring slot is free once the present after its own is taken, not once it has run on the GPU - #2582

Open
stan69b wants to merge 2 commits into
seedds:mainfrom
stan69b:pr/framegen-gpu-time
Open

stan69b wants to merge 2 commits into
seedds:mainfrom
stan69b:pr/framegen-gpu-time

Conversation

@stan69b

@stan69b stan69b commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Depends on #2241 (frame generation). This branch is based on its head, so until #2241 merges the diff below includes it; the change itself is the last commit.

What changes

With frame generation on, a snapshot ring slot's ticket is the present after its own (tk + 1), because the slot's picture is shown again after its generated frame. The game's acquire (mtl_snap_acquire_wait) and the copy (snap_take) waited until present tk + 1 had run on the GPU. That present is committed behind the game's next frame on the same queue, so the game waited for the GPU to finish that frame and lost its frame of overlap with the GPU.

Now the slot is free once the display link (or the present thread) has taken present tk + 1 (snap_slot_free). By then the picture owed after the generated frame has been committed or skipped, and the next copy, committed later on the same queue, runs after it. When the link drops presents after a stall, it also drops the picture still owed. Frame generation stays off by default; with it off, nothing changes.

Before and after

Mac (M1 Pro), Ocarina of Time, Hyrule Field, 30 fps cap, the same route each run, DM_NVN_WAITS=1:

Frame time p50 / p95 Snapshot ring blocked
Frame generation off 33.3 / 33.5 ms —
On, before 33.3 / 36-39 ms 68-89 % of frames, 8-16 ms
On, after 33.3 / 33.5-33.8 ms 0 % in most windows (two windows: 69 % for 11.6 ms, 22 % for 3.6 ms)

iPhone 17, Kakariko Village by day, 30 fps cap, 6 minutes each, unattended and back to back:

Build Frame generation fps Frame time p50 / p95 GPU p50 Interpolation p50
Before on 26.2 37.3 / 48.7 ms 25.8 ms 8.4 ms
After on 24.9 38.5 / 50.4 ms 26.7 ms 8.7 ms
After off 27.9 35.6 / 41.3 ms 32.1 ms —

On the phone the difference is within noise. The 4-7 fps seen on the phone with frame generation on comes from the scene, not from frame generation: Kakariko Village at night ran at 8.5 fps (124 ms frames, every present dropped) with frame generation off as well. That is open and outside this change.

Also tried and left out: frame-generation presents on a queue of their own, so that a present's wait for a free drawable does not hold back the game's next frame. With the present waiting for its snapshot on the GPU, the game blocked about 23 ms a frame on its fences (Mac field, 54-64 ms frames). Waiting for the snapshot on the CPU, it matched this change on the Mac and on the phone (26.2 fps), so it brings nothing.

Checks

  • dm/gfx/tests/t_framegen: a new check holds the Metal 4 queue once the link has taken p2, then asks for p1's slot. Before the fix the acquire is still waiting after 200 ms. After the fix it returns and p4 gets its snapshot. 30 of 30 checks pass with Metal validation. The test's alarm goes from 30 s to 60 s, because the test already took 26.6 s before this check.
  • python3 -m unittest dm/tools/tests/test_docs.py passes.
  • scripts/regress.py check was not run. With frame generation off (the default, and how regress runs), the changed code reduces exactly to the old rule (fg_on() is false), so no game in the regress set reaches it. The Mac and phone runs above are the check for the frame-generation path.

The new reference entry is "A frame-generation ring slot waits for the GPU" in docs/reference/frame-rate.md, with its line in Ocarina of Time's note.

seedds and others added 2 commits November 10, 2026 20:39
…th the 30 fps frame rate): MetalFX's frame interpolator makes the frame halfway between each two presents and the display link shows it between them, so a game held at 30 fps is shown at 60

Ported from cheemu21's Ocarina of Time fork (kid-with-sword-remake 7979a46, X2-X9), the frame generation part only; its pipeline-script capture (DM_PSO_CAPTURE) is left out. The offline archive's .mtl4x unwrap (pso_bundle_unwrap, t_pso_unwrap) comes with it, since the same nvn_mtl.m hunks carry it; build_ipa.sh does not pack .mtl4x yet, so it is idle until an app carries one.

- After each present's snapshot copy, in the same command buffer, MTL4FXFrameInterpolator makes the frame between the previous snapshot and this one into the slot's own texture; the display link shows it on the present's first tick and the present's own picture half an interval later. Depth and motion inputs are constant (far depth, no motion): the interpolator's optical flow finds the motion, so it needs nothing from a game's profile.
- A picture shown smaller than the game draws it is copied at the shown size, a crop alone 1:1; generation pauses while frames drop until three presents in a row come in time; the picture after a generated frame takes a present allocator of its own (3 to 5); the 30 fps cap keeps the 60 Hz link with frame generation on.
- RED_ONCE keys 164-167 (main uses up to 163).

Checked on the Mac (M1 Pro): t_framegen 29 checks, t_present_snap 14, t_present_alloc 13, t_link_ticks 43, t_pso_unwrap 9, all pass with Metal validation; test_docs, test_launcher_settings, test_settings. Tears of the Kingdom 1.4.3 at the Great Sky Island lookout, 30 fps cap with frame generation: interpolation p50 7.8-8.5 ms, p95 14.6-16.4 ms of GPU at 1920x1080 over 1,800 generated frames, the history never started again, one slow display-link callback (77 ms); the picture itself was not checked (the Mac's shader libraries for this chain were incomplete and every run drew black, frame generation on or off).

Front-ends: new: the launcher setting DM_FRAMEGEN (Frame generation), a toggle in Game drawn while DM_FPS_TARGET is 30 (needs=DM_FPS_TARGET=30), NVN games

Co-authored-by: cheemu21 <130556833+cheemu21@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter its own is taken, not once that present has run on the GPU

With frame generation a ring slot's ticket is the present after its own,
because its picture is shown again after its generated frame. The acquire
and the copy waited until that present had run on the GPU, and it is
committed behind the game's next frame on the same queue, so the game lost
its frame of overlap with the GPU. Now the slot is free once the display
link has taken that present: the picture owed after the generated frame has
been committed or skipped by then, and the next copy, committed later on
the same queue, runs after it.

Ocarina of Time on the Mac (M1 Pro, Hyrule Field, 30 fps cap): the
snapshot ring blocked 68-89 % of frames for 8-16 ms, frame time p95
36-39 ms; after it 0 % in most windows, 33.5-33.8 ms (33.5 with frame
generation off). On an iPhone 17 in Kakariko Village by day: no
measurable change (24.9 fps against 26.2).

t_framegen holds the queue once the link has taken p2 and asks for p1's
slot. Its alarm goes from 30 to 60 s: the test took 26.6 s before this
check.
@stan69b

stan69b commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

OoT-fps combined phone build, against the baseline (session A, A-2)

iPhone 17, Ocarina of Time, Kakariko Village by day, unattended drive script, 7 minutes each, back to back (2026-11-13, 22:20 and 22:36). Medians over the gameplay windows (more than 1,000 draws a frame).

baseline (512 ring) combined change
fps 28.6 29.4 +0.8
frame time p50 / p95 34.3 / 40.6 ms 33.6 / 37.1 ms p95 −3.5 ms
NVN submit+present p50 14.13 ms 12.25 ms −13 %
GPU p50 31.2 ms 30.4 ms −3 %
submit thread CPU a frame 13.47 ms at 2.05 GHz 12.00 ms at 1.82 GHz −11 % at a lower clock
instructions per draw 26,790 24,614 −8.1 %
draws per frame 2,680 2,770
share of CPU time on performance cores 49 % 51 %
CPU energy 0.99 W 0.84 W −15 %
residency set 3,499-3,561 MB 3,014-3,174 MB −0.4 GB
system wired 5,230-5,397 MB 4,808-4,950 MB −0.4 GB
memory-pressure warns 1 0
thermal state (max) 1 2
  • Both runs show the same screenshots and the same RED line: the stale pipeline archive message, which both builds print.
  • After each run, env.txt was put back byte for byte, and the save was not touched.
  • The game's real-time threads in the combined run were never demoted: 0 of 300 frames on the acquire, queue replay and encode threads.

Against A-1's targets:

  • Submit ≤ 15 ms: reached.
  • A steady 30 fps: nearly; 29.4 median, p95 37 ms.
  • Performance-core share ≥ 60 %: not reached (51 %).

An earlier pair that cannot be compared: the first pair (21:04-21:19) gave 11.4 fps for the baseline and 29.2 fps for the combined build. That gap is the time of day, not the change: the baseline's save loaded Kakariko at night (GPU p50 60 ms) and the combined build's by day, as B found (#2582). Every run's end.png was checked for time of day, and only day runs are compared above.

The combined build is the one installed on the phone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants