Repository navigation
Frame generation: MetalFX makes a frame between each two presents, a 30 fps game shown at 60 - #2241
Mars-Mah3r wants to merge 1 commit into
Conversation
…th the 30 fps frame rate): MetalFX's frame interpolator makes the frame halfway between each two presents and the display link shows it between them, so a game held at 30 fps is shown at 60 Ported from cheemu21's Ocarina of Time fork (kid-with-sword-remake 7979a46, X2-X9), the frame generation part only; its pipeline-script capture (DM_PSO_CAPTURE) is left out. The offline archive's .mtl4x unwrap (pso_bundle_unwrap, t_pso_unwrap) comes with it, since the same nvn_mtl.m hunks carry it; build_ipa.sh does not pack .mtl4x yet, so it is idle until an app carries one. - After each present's snapshot copy, in the same command buffer, MTL4FXFrameInterpolator makes the frame between the previous snapshot and this one into the slot's own texture; the display link shows it on the present's first tick and the present's own picture half an interval later. Depth and motion inputs are constant (far depth, no motion): the interpolator's optical flow finds the motion, so it needs nothing from a game's profile. - A picture shown smaller than the game draws it is copied at the shown size, a crop alone 1:1; generation pauses while frames drop until three presents in a row come in time; the picture after a generated frame takes a present allocator of its own (3 to 5); the 30 fps cap keeps the 60 Hz link with frame generation on. - RED_ONCE keys 164-167 (main uses up to 163). Checked on the Mac (M1 Pro): t_framegen 29 checks, t_present_snap 14, t_present_alloc 13, t_link_ticks 43, t_pso_unwrap 9, all pass with Metal validation; test_docs, test_launcher_settings, test_settings. Tears of the Kingdom 1.4.3 at the Great Sky Island lookout, 30 fps cap with frame generation: interpolation p50 7.8-8.5 ms, p95 14.6-16.4 ms of GPU at 1920x1080 over 1,800 generated frames, the history never started again, one slow display-link callback (77 ms); the picture itself was not checked (the Mac's shader libraries for this chain were incomplete and every run drew black, frame generation on or off). Front-ends: new: the launcher setting DM_FRAMEGEN (Frame generation), a toggle in Game drawn while DM_FPS_TARGET is 30 (needs=DM_FPS_TARGET=30), NVN games Co-authored-by: cheemu21 <130556833+cheemu21@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
nvn
|
Numbers from a slower Mac: M1 Pro, 16 GB (macOS 26.6.2), Ocarina of Time v131072, this branch merged onto
Merging onto Not checked: how the picture looks in play (artifacts, the HUD), latency, |
|
Follow-up after actually playing it on the M1 Pro (tier 2 + frame generation, a few minutes around the castle): the pacing was fine, but I didn't like the picture and went back to plain 30 fps. What I saw: buildings and other structures shimmer while standing still, and everything turns blurry as soon as the camera moves. Overall the image looked "generated" rather than drawn. The log agrees the feature itself was doing its job: 3,600 generated frames shown, 71 late presents, interpolation at 2.6 ms p50. One caveat: I only played it with |
… the scene before the HUD, its depth and the game's camera (DM_FRAMEGEN, off by default) A prototype for evaluation, not for merging as it is. Between two frames the game draws, the host shows one made by MetalFX's frame interpolator (Metal 4) from the 1440x810 scene before the HUD, the scene's depth and motion vectors reprojected from the game's camera block; the generated scene goes through the game's spatial upscale a second time and is composed under the real frame's HUD, which is never interpolated. The generated frame is presented ahead of its real frame on the present thread (DM_PRESENT_LINK=0), each for half the period; the pacing follows seedds#2241, the presented-time shift and the delayed first engagement an outside patch (framegen-stage1-v4) read as reference. Specific to Ocarina of Time: the camera block's layout and the far-plane test that picks the scene camera, read from one build and not keyed by build ID yet. Played on the Ocarina performance line (cut6) on an M2 MacBook Air: 45-53 shown frames a second from a 30 fps cap, the game's frame rising from about 30 ms to about 37 ms. On main: the host compiles for 0400486024DEC000; no game run and no regress run. With DM_FRAMEGEN unset the hooks return at once. Co-Authored-By: Claude <noreply@anthropic.com>
|
iPhone run of this port (the "Not checked" item): iPhone Duo (iPhone19,4, Apple A20 Pro GPU), OoT v131072, NX2Apple 1.4.0 shared-app build on engine ab41d6a, shader chain cb2e91c138e3. I applied this PR's single commit to that engine snapshot: 25 of 26 hunks applied cleanly; the two remaining conflicts were local context differences (an extra DM_PRESENT_INTERVAL settings row and dm_prof()->present_snap) and were resolved by hand. clang -fsyntax-only with the iOS 27.2 SDK passed, chain V unchanged. Settings: DM_FPS_TARGET=30, DM_FRAMEGEN=1, DM_PRESENT_SNAP=0 (auto-promoted to a ring of 3), DM_PRESENT_UPSCALE=off, DM_UPSCALER=metalfx, Metal HUD on. Result over ~204 s: 5,400 generated frames shown, history restarted 0 times, 3 skipped for a newer present, 31 presents shown without theirs while frames dropped. Interpolator 1920x1080 (depth/motion 960x540), created in 320.8 ms. Interpolation GPU p50 4.1-8.0 ms (steady 5.6-6.6), p95 6.6-13.8 ms @1920x1080. Game frame p50 33.31-33.37 ms, dropped 0 in the last 11 windows, 30 drawn / 60 shown, thermal 0 throughout, no crash. CPU time on performance cores stayed 53-59 %. Two slow display-link callbacks at startup (377.5 ms, 82.6 ms), none afterwards -- the fork's X11 fix (not committing the residency set for generated pictures) is not part of this port, which may explain those two. For contrast with the A18 Pro numbers in #2269: 1920x1080 frame generation cost p50 11.7 ms there and did not fit beside a 30 ms frame; on this A20 Pro it is 4-8 ms and fits without a resolution preset. So the headroom is quite SoC-dependent. Limitation: light/medium scenes, cold start, one session; heavy areas and fast camera turns not measured. Full log available. More context and the 60 fps vs 30 fps comparison in #2269 (comment posted there). Thanks for the port -- it made 30 fps + 60 shown work on this device without doubling the CPU/GPU load. |
…ree to the Kokiri Emerald at the 30 fps cap, with the eight runs behind it, what did not help, frame generation and what is open A dated entry under "For an iPhone or iPad build": the build (smaller textures, staging released, MetalFX for the upscaler, DM_SCREEN_PIXELS=1; branch iphone-exp, chain 39eb2f781d07, PR seedds#2241's frame generation for part of the play), a table of r1_base, r1_reuse0, r1_chunk256, r1_nocomp0, r2_mfx_game, r2_mfx_on, b2_play1 and b2_play2 over frames 6,000-22,500, made from the logs. r2_mfx_on is marked as what it was: MetalFX refused every variant before the square-root shape. Not this: DM_GUEST_FILE_CHUNK=256M (noise) and DM_NVN_FAMILY_NOCOMPRESS=0 (no gain; the rule protects the M1 iPad from a GPU hang). Frame generation: 9,000 frames generated with no restart, the boss at 29.3-29.9 fps shown at 60. Open: the movie cutscenes at 15-18 fps (nn::ndve's network 35-70 ms of GPU), the freeze at a scene state's first visit, the empty CPU profile (seedds#2197) and DM_MEMSTAT on iOS.
…ree to the Kokiri Emerald at the 30 fps cap, with the eight runs behind it, what did not help, frame generation and what is open A dated entry under "For an iPhone or iPad build": the build (smaller textures, staging released, MetalFX for the upscaler, DM_SCREEN_PIXELS=1; branch iphone-exp, chain 39eb2f781d07, PR seedds#2241's frame generation for part of the play), a table of r1_base, r1_reuse0, r1_chunk256, r1_nocomp0, r2_mfx_game, r2_mfx_on, b2_play1 and b2_play2 over frames 6,000-22,500, made from the logs. r2_mfx_on is marked as what it was: MetalFX refused every variant before the square-root shape. Not this: DM_GUEST_FILE_CHUNK=256M (noise) and DM_NVN_FAMILY_NOCOMPRESS=0 (no gain; the rule protects the M1 iPad from a GPU hang). Frame generation: 9,000 frames generated with no restart, the boss at 29.3-29.9 fps shown at 60. Open: the movie cutscenes at 15-18 fps (nn::ndve's network 35-70 ms of GPU), the freeze at a scene state's first visit, the empty CPU profile (seedds#2197) and DM_MEMSTAT on iOS.
…ree to the Kokiri Emerald at the 30 fps cap, with the eight runs behind it, what did not help, frame generation and what is open A dated entry under "For an iPhone or iPad build": the build (smaller textures, staging released, MetalFX for the upscaler, DM_SCREEN_PIXELS=1; branch iphone-exp, chain 39eb2f781d07, PR seedds#2241's frame generation for part of the play), a table of r1_base, r1_reuse0, r1_chunk256, r1_nocomp0, r2_mfx_game, r2_mfx_on, b2_play1 and b2_play2 over frames 6,000-22,500, made from the logs. r2_mfx_on is marked as what it was: MetalFX refused every variant before the square-root shape. Not this: DM_GUEST_FILE_CHUNK=256M (noise) and DM_NVN_FAMILY_NOCOMPRESS=0 (no gain; the rule protects the M1 iPad from a GPU hang). Frame generation: 9,000 frames generated with no restart, the boss at 29.3-29.9 fps shown at 60. Open: the movie cutscenes at 15-18 fps (nn::ndve's network 35-70 ms of GPU), the freeze at a scene state's first visit, the empty CPU profile (seedds#2197) and DM_MEMSTAT on iOS.
…ree to the Kokiri Emerald at the 30 fps cap, with the eight runs behind it, what did not help, frame generation and what is open A dated entry under "For an iPhone or iPad build": the build (smaller textures, staging released, MetalFX for the upscaler, DM_SCREEN_PIXELS=1; branch iphone-exp, chain 39eb2f781d07, PR seedds#2241's frame generation for part of the play), a table of r1_base, r1_reuse0, r1_chunk256, r1_nocomp0, r2_mfx_game, r2_mfx_on, b2_play1 and b2_play2 over frames 6,000-22,500, made from the logs. r2_mfx_on is marked as what it was: MetalFX refused every variant before the square-root shape. Not this: DM_GUEST_FILE_CHUNK=256M (noise) and DM_NVN_FAMILY_NOCOMPRESS=0 (no gain; the rule protects the M1 iPad from a GPU hang). Frame generation: 9,000 frames generated with no restart, the boss at 29.3-29.9 fps shown at 60. Open: the movie cutscenes at 15-18 fps (nn::ndve's network 35-70 ms of GPU), the freeze at a scene state's first visit, the empty CPU profile (seedds#2197) and DM_MEMSTAT on iOS.
…X's frame interpolator makes the frame between each two presents; a game at 60 fps is shown at 120 on a 120 Hz screen, one at 30 at 60 Ported from seedds#2241 (Mars-Mah3r; from cheemu21's Ocarina of Time fork), which shows a 30 fps game at 60. On top of it: - at 60 fps on a screen that shows 120 Hz the display link runs at 120 Hz, so each present has a tick for its generated frame and one for its own picture - a 120 fps target becomes 60 drawn by the game, shown as 120 (dm_fps_state_apply) - variable refresh stays off with it: generated frames go out on the link's fixed ticks - the launcher row is a choice (Off / 2x) offered at every frame rate Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
A follow-up for this PR is in #2582. With frame generation on, the game's acquire of a snapshot ring slot waited until the present after the slot's own had run on the GPU. That present sits behind the game's next frame on the same queue, so the game lost its frame of overlap with the GPU. #2582 frees the slot once the display link has taken that present. Ocarina of Time, 30 fps cap, frame generation on:
The 4-7 fps seen on the iPhone with frame generation was the scene: Kakariko Village at night runs at 8.5 fps with frame generation off too. |
What and why
A game held at 30 fps on a 60 Hz display link shows each frame for two ticks; 60 fps doubles the CPU and GPU work and, on an iPhone, the heat. Frame generation (
DM_FRAMEGEN=1, launcher: Frame generation, drawn while the frame rate is 30 fps) has MetalFX's frame interpolator (MTL4FXFrameInterpolator) make the frame halfway between each two presents, and the display link shows it between them: the game does the work of 30 frames a second and the screen shows 60.This ports the frame generation of cheemu21's Ocarina of Time fork (kid-with-sword-remake
7979a465, builds X2-X9; co-author on the commit) onto main, without the fork's other changes:DM_PSO_CAPTURE). The offline archive's.mtl4xunwrap (pso_bundle_unwrap,t_pso_unwrap) comes along because it shares the samenvn_mtl.mhunks;build_ipa.shdoes not pack.mtl4x, so it is idle until an app carries one.Off by default; with
DM_FRAMEGENunset, the only path that changes is the present allocator index (stillticket % 3) and two bookkeeping fields.Checked
make -C dm/gfx/tests GAME=0100F2C0115B6000 check_t_framegen29 checks,check_t_present_snap14,check_t_present_alloc13,check_t_link_ticks43,check_t_pso_unwrap9, all pass;dm/tools/teststest_docs, test_env_switches, test_launcher_settings, test_settings.regress.py check --games: not run. The feature is off by default and its paths run only withDM_FRAMEGEN=1; the present and display-link tests above cover the shared paths.Front ends
Front-ends: new: the launcher setting DM_FRAMEGEN (Frame generation), a toggle in Game drawn while DM_FPS_TARGET is 30 (needs=DM_FPS_TARGET=30), NVN games
Docs
docs/reference/frame-rate.mdentry, andpython3 dm/tools/docs_symptoms.pywas run🤖 Generated with Claude Code