[dpadr]

A scratch prototype for evaluation, not for merging as it is. It is off unless DM_FRAMEGEN=1 is set, and parts of it are specific to Ocarina of Time (below). It is the same feature as #2241, built on different inputs; this draft exists so the two efforts can find each other.

What was tested where. The code was written and played on our Ocarina of Time performance line (cut6, 31 commits ahead of main); every number below is from that build. This branch is the frame-generation code restacked on main (78671525) plus #2366, and it is untested on main: compile-checked only (make -C dm/host GAME=0400486024DEC000 dmhost, test_game_profile, test_settings, test_dmsh_abi), no game run. The first version of this branch generated no frames on main (found by @devMlGUE, see the comments). Until #2366 merges, this diff includes its changes.

Prerequisites

• The game's upscaler drawn by MetalFX: nvn/upscaler: MetalFX takes an upscaler with square-root colour #2366. The hook is at the end of up_mfx_encode. Ocarina of Time's upscaler is agl_ndsr_vr_light.pck's network, whose input is square-root encoded; main does not draw it with MetalFX, nvn/upscaler: MetalFX takes an upscaler with square-root colour #2366 does, and this branch is stacked on it (not on Ocarina of Time's chained upscaler (agl_ndsr_vr) through MetalFX #1951, the other open route to the same upscaler). The generated frame takes nvn/upscaler: MetalFX takes an upscaler with square-root colour #2366's sharpness blend as the real frame does. With DM_FRAMEGEN=1 and no MetalFX upscale, the log says so in a [RED] line after 600 presents.
• Settings: DM_FRAMEGEN=1 DM_UPSCALER=metalfx DM_OPMODE=0 DM_DYNAMIC_RESOLUTION=0 DM_FPS_TARGET=30 DM_PRESENT_LINK=0 DM_NVN_ENC_THREAD=0, graphics scale tier 0. Scene 1440x810, canvas 1920x1080.
• A check that it is running: the log must have upscaler: first dispatch of ... drawn by MetalFX and a non-zero generated count in the frame generation on: lines.
• Not together with the straight MetalFX path (nvn/upscaler: MetalFX scales straight between the game's images in any shared format (OoT −0.6 to −1.0 ms a frame on an M1) #2559). The hook runs only where MetalFX scales through its own RGBA16F images. Where it scales straight between the game's images (nvn/upscaler: MetalFX scales straight between the game's images in any shared format (OoT −0.6 to −1.0 ms a frame on an M1) #2559, on by default once merged), no frame is generated and the log says so in a [RED] line. Until the hook covers that path, run with DM_NVN_MFX_DIRECT=0 when DM_FRAMEGEN=1.

What it does

Between each two frames the game draws, the host shows one generated frame, to turn a 30 fps source into 60 shown frames a second.

• Input: the scene before the HUD. The hook is the game's own upscale (up_mfx_encode): the 1440x810 scene colour, the scene's depth buffer, and the game's camera.
• Motion from the camera. Motion vectors are reprojected per pixel from the depth and the camera block (this frame's view-projection against the previous view the game keeps in the same block). Where the previous depth disagrees, the pixel gets zero motion and MetalFX's own flow takes over (the test is from Mac: MetalFX frame interpolation (30+30), adapted from MacNativeGames; profile keys scene_upscale / camera_block #301).
• MetalFX frame interpolator (Metal 4) at 1440x810, then the game's spatial upscale a second time on the generated scene.
• The HUD is never interpolated. Pixels the HUD passes changed on the canvas are taken from the real frame when the generated picture is composed.
• Pacing, after Frame generation: MetalFX makes a frame between each two presents, a 30 fps game shown at 60 #2241: the generated frame is presented first and held half a period, then the real frame; own command allocators for the extra present; generation pauses while frames come late.

How it differs from #2241

	#2241	This branch
Input picture	the presented frame, HUD included	the scene before the HUD
Depth	constant far depth	the scene's depth buffer
Motion vectors	none (the interpolator's optical flow alone)	reprojected from the game's camera
HUD	interpolated with the picture	kept from the real frame
Games	any	Ocarina of Time only, so far

Offline, on the same real frames, a fast camera swing (about 100-150 px a frame) comes out sharp with these inputs and smeared with flow alone. #2241's pacing is used here nearly as it is. Two ideas come from an outside patch read as reference (framegen-stage1-v4): the presented-time shift for the held real frame, and not engaging during boot.

In play (M2 MacBook Air, 16 GB; the user, 2026-11-10)

Handheld, 1440x810 scene, dynamic resolution off, MetalFX upscale, DM_FPS_TARGET=30, DM_PRESENT_LINK=0, encode thread off, fullscreen window.

Per 600 shown frames	1st	2nd	3rd
Shown frames a second	44.8	50.5	53.3
Generated	233	284	292
Interval p50 / p95 (ms)	16.67 / 33.33	16.67 / 33.33	16.67 / 33.33
Intervals over 25 ms	188	102	72
Game enqueue to screen, p50 (ms)	52.9	50.8	50.7

• The game's own frame rose to 36.6-37.7 ms p50 (GPU p50 34.5-35.3 ms) from about 30-31 ms without generation: generation costs about 6-7 ms of GPU a real frame, which is why it does not reach 60.
• The user: "overall the framegen is impressive, never got to 60fps but i saw close to it sometimes. there is a slight uncanny soap opera quality to it ... definitely worth continuing to work on".
• No latency baseline yet (the run without generation did not log it).

Known flaws

• Not 60: generation's own GPU cost pushes the game past 33.3 ms. The split of that cost is being measured.
• Uneven cadence (16.7 / 33.3 ms) whenever the source misses 33.3 ms; the generated frame is always the midpoint.
• Fast characters smear, worst when moving toward the camera: there is no per-object motion.
• Particles without depth (fireflies, glows, falling leaves) fade or ghost in generated frames.
• Near foliage under parallax can lose its shape.
• At cuts and the fastest swings the interpolator returns the real frame unchanged.
• Generation starts after 300 upscaled frames (a stopgap for making its pipelines at first use) and needs three on-time frames to resume after a late one.
• Headless runs stall with generation on (two drawables a frame); a window does not.

What is specific to Ocarina of Time

• The camera block's layout (vertex uniform 0, 13,568 bytes; view-projection, projection, previous view, near, far, field of view at fixed float offsets) and the far-plane test that tells the scene camera from the environment probe's. These were read from one build and are not keyed by build ID yet: before anything here leaves scratch they must be, per the per-build rule.
• The assumption that the game's upscale covers whole textures and the HUD is drawn on the upscaled canvas afterwards.

General parts: the interpolator set-up, the motion pass, the HUD mask, the pacing.

Not in this branch

Display-link presents (DM_PRESENT_LINK=1), the encode thread, iOS, any game but this one, tests, docs, a docs/env_switches.tsv row. regress.py was not run: the change is behind a switch that is off by default.

🤖 Generated with Claude Code

[devMlGUE]

I tried this on an M1 Pro, 16 GB (macOS 26.6.2), v131072, the diff applied onto main at 8b55ebb3 (it applies cleanly and builds). It generates no frame here: frame generation on: 600 shown frames at 30.0 a second, 0 generated ... interpolations 0, history restarts 0, no camera 0, no depth 0 in every run. Settings as in the description (DM_FRAMEGEN=1 DM_UPSCALER=metalfx DM_FPS_TARGET=30 DM_PRESENT_LINK=0, encode thread off), bench.py --route field-run, handheld at tier 0, DM_GFX_SCALE=1 and =2, with dynamic resolution on, and docked.

The reason is that the hook is never reached: up_mfx_encode does not run on main with this game, with or without this change.

• The log has upscaler (DM_UPSCALER): metalfx, the game's upscaler shaders (22, from the shader map) ... but never first dispatch, and no upscaler <key>: ... line. The same holds for a 2 h 20 min play session on unmodified main.
• With a temporary log line in up_mfx_wanted: the game dispatches 107 to 112 compute shaders in these runs and none of them is one of the 22 DM_GAME_UPSCALER_KEYS (the ones from agl_ndsr.pck).
• By the shader map, 11 of the dispatched ones come from Ndve-Nvn2-vid_seq5.dat.zs, which also has 22 compute shaders. With the key list pointed at those instead, each reads as neither of the colour shapes MetalFX draws; the game's shader runs, so the swap still does not happen.

So on main the MetalFX setting seems to do nothing for Ocarina of Time on this version, and this draft depends on it. Does your performance line change which pack is recognised, or the colour shapes up_mfx_color accepts? And which settings give the 1440x810 scene: here handheld draws 3D at 1920x1080 at tier 0 and 1280x720 at tier 2.

For comparison, #2241 merged onto the same commit does generate on this Mac (numbers in a comment there).

[devMlGUE]

Correction to my earlier comment: I pointed at the wrong network. The Ndve one is the movie network. The upscaler OoT runs in play comes from agl_ndsr_vr_light.pck, and the key list only takes shaders found in agl_ndsr.pck, so the MetalFX path never sees it.

I tried a host that also accepts the _vr and _vr_light packages. The shaders are recognised then, but it logs "neither of the colour shapes MetalFX draws" for them and the game's shader still runs, so GPU time doesn't move.

For scale: on this M1 Pro that one dispatch (fa3033d05c0a41d6) is about 4.4 ms a frame at 1440x810 and 2.1 ms at 960x540.

[devMlGUE]

One more data point on this, in case it saves someone the experiment. I let MetalFX draw the _vr_light network's dispatch as linear in, linear out (its code has none of the sRGB curve constants, the colour arrives packed as RGB9E5 and leaves as RG11B10F). It works and the picture looked right at a glance, but on this M1 Pro it doesn't pay:

	fps	GPU p50
game's network, 1440x810	26.4	30.5 ms
MetalFX, 1440x810	26.9	32.4 ms
game's network, 960x540	29.7	25.2 ms
MetalFX, 960x540	29.3	27.8 ms

Two runs each, same route. The per-pass profile shows compute dropping about 4 ms a frame, yet the frame's GPU time goes up about 2 ms. I haven't found out why, so I'm leaving it there.

[dpadr]

Thank you for running it, and for tracing it to the hook: you are right, and the fault is in this draft, not in your setup.

Cause. The hook is in up_mfx_encode, and on main that never runs for Ocarina of Time. This game upscales with agl's chained upscaler (agl_ndsr_vr.pck / agl_ndsr_vr_light.pck), and main's game.py only takes upscaler keys from agl_ndsr.pck (22 keys, none of them dispatched here). Our performance line carries two changes that main does not: the key list includes the _vr packs (59 keys), and up_mfx_color accepts the network's square-root-encoded input (squared, then MetalFX in HDR mode). With those, the log has upscaler: first dispatch of fa3033d05c0a41d6_b drawn by MetalFX, 1440x810 → 1920x1080, and the frame-generation hook follows that draw. I ported the frame-generation code to main without them and only compile-checked it, so the draft silently depends on something that is neither on main nor in the diff. Ndve-Nvn2-vid_seq5 is the video's shaders, not the upscaler, which is why pointing the keys there was refused.

The open pull request that brings this upscaler to main is #1951. It does it differently from our line (up_chain_dispatch), so this draft has to be re-hooked onto it rather than just merged with it.

Settings for 1440x810. DM_OPMODE=0 (handheld), DM_DYNAMIC_RESOLUTION=0, DM_GFX_SCALE unset (tier 0), DM_UPSCALER=metalfx, DM_FPS_TARGET=30, DM_PRESENT_LINK=0, DM_NVN_ENC_THREAD=0, a 2560x1440 fullscreen window. On our line that gives a 1440x810 scene (RG11B10Float) upscaled to a 1920x1080 canvas, which the HUD is drawn on. Our line predates 084f5aa, but as far as I can read it that change is to docked only. I have not run a main build, so I can't yet say whether your 1920x1080 at tier 0 is the canvas after the upscale or a real difference; where do you read it?

What I'll change. Restack the draft on main plus #1951 with the hook at its MetalFX draw, run it on main before saying it works, and make the failure loud: a [RED] line when DM_FRAMEGEN=1 and no upscale has been drawn by MetalFX, instead of a quiet "0 generated". I'm not moving the hook to the game's own upscaler: the generated frame needs the same upscale as the real one, and running the game's four-dispatch chain a second time costs more than the interpolation saves. The description's prerequisites are corrected below. Until the restack is pushed, this draft cannot generate on main; #2241 and #2289 are the ones to try there.

[dpadr]

@devMlGUE thank you again for the report. The branch is now restacked on main (78671525) plus #2366, which draws this game's upscaler (the square-root-colour network of agl_ndsr_vr_light.pck) with MetalFX; the frame-generation hook sits at the end of that draw. I first stacked it on #1951 as I wrote above, then moved to #2366: it is clean against main and it is the same shape as the path our numbers were measured on. Until #2366 lands, this pull request's diff includes its changes.

It is untested on main: compile-checked only. We have no shader library for main's chain on our Mac, and you do. If you have the time, could you try it on main + #2366 with the same settings as before (DM_FRAMEGEN=1 DM_UPSCALER=metalfx DM_OPMODE=0 DM_DYNAMIC_RESOLUTION=0 DM_FPS_TARGET=30 DM_PRESENT_LINK=0, encode thread off)? Three log lines tell the story:

• upscaler: first dispatch of ... drawn by MetalFX ... squared on the way in: nvn/upscaler: MetalFX takes an upscaler with square-root colour #2366's path is active;
• frame generation on: 600 shown frames ..., N generated: N above 0 is the answer;
• a new [RED] frame generation (DM_FRAMEGEN=1): ... no upscale of the game's was drawn by MetalFX after 600 presents, if the hook is still not reached.

One thing is new and unseen by us: #2366 softens MetalFX's picture for this game (upscaler_sharpness 0.7), and the generated frame now takes the same blend. If generated frames look crisper or softer than the real ones, that blend is where to look. No hurry, and no obligation.

[devMlGUE]

Thanks, and good to know about the square-root colour. That means my MetalFX numbers above shouldn't be trusted: my quick hack fed that network's input to MetalFX as if it were linear, so both the picture and the timing need redoing with #2366.

I haven't run the re-stacked branch yet. Same problem as you, my shader library is for the older chain (2afd39090b76), so I need to rebuild it for main first. I'll do that and come back with the three log lines you listed, plus fps and GPU time with and without MetalFX on the M1 Pro at the same settings as before.

[devMlGUE]

Ran the re-stacked branch (f83131c) on the M1 Pro. Rebuilt the shader library for its chain (ded41f8d62ed, 16 minutes) and let the pipelines compile once before measuring. Same route as before (standing, then walking in the field), DM_OPMODE=0 DM_DYNAMIC_RESOLUTION=0, encode thread off, three runs each.

Your three lines:

• upscaler: first dispatch of fa3033d05c0a41d6_b drawn by MetalFX, 1440x810 ... → 1920x1080 ..., through RGBA16F, squared on the way in — yes, in every MetalFX run.
• frame generation on: 600 shown frames at 60.0 a second, 300 generated — yes. Over the route the windows range from 51.6 to 60.0 shown fps, 252 to 300 generated per 600, history restarts 0, no camera 0, no depth 0.
• The new [RED] ... no upscale of the game's was drawn by MetalFX line does show up, once per run, but it's a false alarm: it fires during the first 600 presents (loading and menus, before the game dispatches its upscaler at all) and frame generation works fine afterwards. Probably worth counting from the first frame that has an upscale, or from the first in-game frame.

Numbers:

	fps	GPU p50	GPU p95
game's network	26.9–28.6	27.3–27.4 ms	29.4 ms
MetalFX	25.9–31.4	28.9–29.3 ms	31.7–35.1 ms
MetalFX + frame generation	29.7–29.9 real	33.7–34.0 ms	36.8–37.2 ms

(One of the three network runs came out at 43 fps with 13 ms of GPU, clearly not the same scene, so I left it out.)

So with the colour done properly my earlier result still holds on this machine: MetalFX is about 1.5–2 ms slower than the network here, not faster. The M1 Pro sits right at the 33 ms line either way, which is why fps doesn't move. Frame generation adds roughly another 4.5 ms of GPU but holds the 30 real frames, with 0 dropped.

One more thing from the frame generation log: game enqueue to screen for real frames is p50 ~139 ms, p95 ~141 ms. That seems like a lot of latency to me, is it expected with the encode thread off?

I haven't looked at the picture with the 0.7 blend yet, these were unattended runs. I'll play a bit on this build and tell you whether generated frames look different from real ones.

[devMlGUE]

Played it for a few minutes and found something more useful than a picture opinion: with dynamic resolution on, frame generation produces nothing.

My play session had DM_DYNAMIC_RESOLUTION=1 (it's what I normally use), handheld, 30 fps, DM_FRAMEGEN=1 DM_UPSCALER=metalfx DM_PRESENT_LINK=0. Six frame generation on windows, and generated was 0, 0, 0, 0, 64, 0. The "no depth" counter climbed the whole time, 264 → 864 → 1464 → 1766 → 2365, and the screen stayed at 30 shown frames a second. MetalFX itself was working, the log has scalers for 1440x810, 1366x768 and 1280x720.

I then repeated it on the benchmark route to rule out my playing: same settings as the run in my previous comment, only dynamic resolution switched on. Three windows, 0 generated in all of them (interpolations 0, no depth 0 this time, so it doesn't even get that far), while the game went through scalers for 1440x810 down to 960x540. With dynamic resolution off, the same route gives 252–300 generated per 600. So it's the changing input size that breaks it, not the scene.

On the picture, with that caveat (what I was looking at was essentially MetalFX with the 0.7 blend at a dropped resolution, not generated frames): nearby objects look better than before, but distant ones look like modelling clay, soft and with slightly wrong shapes. I'd guess that's the spatial scaler stretching 1280x720 more than the blend, but I haven't separated the two. I'll look again with dynamic resolution off, where frames are actually generated, and compare sharpness 0.7 against 1.0.

[devMlGUE]

Follow-up with dynamic resolution off, same build and settings otherwise (handheld, 30 fps, sharpness left at 0.7).

This is the best the game has looked in any of my tests. Distant objects are well defined now, so the modelling-clay look I described came from the dropped resolution, not from MetalFX or the 0.7 blend. I didn't notice generated frames looking sharper or softer than real ones.

Frame generation was really running this time. After the first window (loading), the next six showed 36 to 55 frames a second with 211 to 281 generated per 600, no depth 0 throughout, 4 history restarts and 2 "no camera" in total. It doesn't hold 60 while playing on this machine the way it did on the benchmark route: 39 to 194 of every 600 intervals were over 25 ms, and "late" generated frames kept adding up (171 by the end). Latency for real frames jumped between about 59 and 140 ms from window to window.

So on an M1 Pro: picture good, pacing uneven, and it needs dynamic resolution off, which is the one setting I'd normally want on for a GPU this size.

[devMlGUE]

A correction to my dynamic resolution report, after reading that benchmark log more carefully: the benchmark run doesn't prove what I said it did. With dynamic resolution on, that run started slower, the first MetalFX upscale only came after the second 600-present window, and the one window left was mostly inside the 300-frame warm-up. So "0 generated" there is just a run that was too short, not a reproduction. Sorry for the noise.

The play session is still solid evidence though: several minutes in game, "no depth" going up by about 600 per window, and 64 generated frames in total. Looking at fg_frame, a frame only counts as having depth when the last depth texture is exactly the size of the scene being upscaled, so my guess is that with dynamic resolution the scene shrinks (1366x768, 1280x720 in my log) while the depth texture it finds stays at another size. I haven't confirmed which size it actually sees. I'll log that on my side and, if it's what it looks like, try a fix and send you the patch.

[dpadr]

Thanks for all of the testing and insights, will roll all of this into the next phase of this project. As it stands it's really tailored to OOT but once this is solid the ideal would be to generalize this (per repo guidelines) so it can use DLSS inputs for games that support that and fallback for games that don't have camera targets like OOT (3 modes).

[devMlGUE]

Found why frame generation did nothing for me with dynamic resolution on, and I have a small patch that fixes it here.

The scene depth is picked as "the last depth attachment at least 1024 wide". On this Mac dynamic resolution takes the scene down to 953x536 and 854x480, so the real scene depth is never a candidate. What's left is a square ~1712x1712 depth texture (the shadow map, I assume), which doesn't match the scene size, so every frame counts as "no depth". That's the 2,365 "no depth" and 64 generated frames in my earlier session.

The patch keeps the last four depth attachments instead and uses the newest one that is exactly the scene's size. The log from a session with it:

scene 953x536 at 0,0, depth used 953x536, recent depth attachments: 953x536 953x536 476x268 476x268
scene 854x480 at 0,0, depth used 854x480, recent depth attachments: 854x480 854x480 427x240 427x240

dynamic resolution on, M1 Pro, 2560x1440 window	before	with the patch
generated frames per 600 shown	0 (64 in the whole session)	157–219 in most windows
"no depth"	climbing ~600 per window	0
shown fps	30	37–44, median 39
history restarts	–	0

That's one play session of about five minutes each side in Hyrule Field, same settings as before (30 fps target, present link off; docked, so the scene starts at 1920x1080 before dynamic resolution drops it), not a benchmark route. Latency stayed where it was without dynamic resolution, 100–150 ms.

Two caveats. There is a second rule in the patch for a depth texture larger than the scene with the same shape (scene drawn into part of it); nothing here exercised it, the exact-size match always won, so feel free to drop it. And the extra log line is only there to show which texture was chosen. The picture itself at 854x480 is not something I'd play with, but that's dynamic resolution on this Mac, not frame generation.

patch against f83131c

diff --git a/dm/gfx/nvn/nvn_framegen.inc b/dm/gfx/nvn/nvn_framegen.inc
index bae56a47..28617e87 100644
--- a/dm/gfx/nvn/nvn_framegen.inc
+++ b/dm/gfx/nvn/nvn_framegen.inc
@@ -25,7 +25,7 @@ static float g_fg_soft;   // the real frame's upscale blends this share of a bil
 static _Atomic int g_fg_hooked;   // the hook was reached: an upscale of the game's was drawn by MetalFX (up_mfx_encode_from -> fg_frame)
 static struct {
     id<MTL4FXFrameInterpolator> fi; NSUInteger iw, ih, ow, oh; MTLPixelFormat cvf; int bad;
-    id<MTLTexture> c[2], d[2], mv, gen, gen_up, cv0, cv, depth_last, win[FG_WIN];
+    id<MTLTexture> c[2], d[2], mv, gen, gen_up, cv0, cv, depth_seen[4], win[FG_WIN];
     uint64_t win_serial[FG_WIN], win_ticket[FG_WIN], win_next;
     id<MTL4CommandAllocator> walloc[FG_WIN]; id<MTL4CommandBuffer> wcb[FG_WIN]; id<MTLBuffer> wpar[FG_WIN];
     id<MTL4ArgumentTable> margs, wargs; id<MTLRenderPipelineState> mv_pso, comp_pso; MTLPixelFormat comp_fmt; id<MTLFunction> vf, f_mv, f_comp;
@@ -51,11 +51,15 @@ static int fg_on(void) {
 }
 // The camera block as the game binds it (encode_args_): the scene camera's binding has far plane 25000; the environment probe's (far 10000, one cube face a frame) is not taken
 static void fg_cam_note(const float *c) { if (!fg_on() || !(c[57] > 20000.0f)) return; memcpy(FG.cam, c, sizeof FG.cam); FG.cam_frame = (long)atomic_load(&g_cur_frame); }
-// Each render pass (td_note): the last large D32F depth attachment before the upscaler is the scene depth
+// Each render pass (td_note): the last few D32F depth attachments, newest first. The scene depth is the newest of them that holds the scene being upscaled
+// (fg_frame); with dynamic resolution the scene is narrower than a shadow map, so "the last large one" is not it
 static void fg_depth_note(MTL4RenderPassDescriptor *pd) {
     if (!g_fg_mode || !pd || !fg_on()) return;
     id<MTLTexture> d = pd.depthAttachment.texture;
-    if (d && d.pixelFormat == MTLPixelFormatDepth32Float && d.textureType == MTLTextureType2D && d.width >= 1024) FG.depth_last = d;
+    if (!d || d.pixelFormat != MTLPixelFormatDepth32Float || d.textureType != MTLTextureType2D || d == FG.depth_seen[0]) return;
+    int n = 3; for (int i = 1; i < 4; i++) if (FG.depth_seen[i] == d) { n = i; break; }
+    for (int i = n; i > 0; i--) FG.depth_seen[i] = FG.depth_seen[i - 1];
+    FG.depth_seen[0] = d;
 }
 static int fg_inv4(const double a_[16], double r[16]) {
     double a[16]; memcpy(a, a_, sizeof a); for (int i = 0; i < 16; i++) r[i] = i % 5 == 0;
@@ -152,7 +156,7 @@ static void fg_barrier(void) {   // as up_mfx_encode: MetalFX opens encoders of
     [ce endEncoding];
 }
 // After the game's upscale of this frame (up_mfx_encode): color = the scene, linear RGBA16F (the scaler's input), out = the canvas the upscaled scene was just drawn into
-static void fg_frame(id<MTL4FXSpatialScaler> sc, id<MTLTexture> color, id<MTLTexture> out, const NVNcopyRegion *orr, NSUInteger cw, NSUInteger ch) {
+static void fg_frame(id<MTL4FXSpatialScaler> sc, id<MTLTexture> color, id<MTLTexture> out, const NVNcopyRegion *orr, NSUInteger cw, NSUInteger ch, NSUInteger ix, NSUInteger iy) {
     if (!fg_on()) return;
     atomic_store(&g_fg_hooked, 1);
     long f = (long)atomic_load(&g_cur_frame); NSUInteger ow = (NSUInteger)orr->width, oh = (NSUInteger)orr->height;
@@ -165,13 +169,19 @@ static void fg_frame(id<MTL4FXSpatialScaler> sc, id<MTLTexture> color, id<MTLTex
     { static int seen; if (seen < 300) { seen++; return; } }
     if (!fg_pipes() || !fg_setup(cw, ch, ow, oh, out.pixelFormat, sc)) return;
     if (g_fg_upto < 1) return;
-    id<MTLTexture> dep = FG.depth_last;
-    int have_d = dep && dep.width == cw && dep.height == ch, have_cam = FG.cam_frame == f, cur = FG.cur, prev = cur ^ 1;
+    id<MTLTexture> dep = nil;   // exactly the scene's size first; else one the scene is a part of that has the scene's shape (not a square shadow map)
+    for (int i = 0; i < 4 && !dep; i++) { id<MTLTexture> d = FG.depth_seen[i]; if (d && d.width == cw && d.height == ch) dep = d; }
+    for (int i = 0; i < 4 && !dep; i++) { id<MTLTexture> d = FG.depth_seen[i]; if (d && d.width >= ix + cw && d.height >= iy + ch && d.width * ch == d.height * cw) dep = d; }
+    int have_d = dep != nil, have_cam = FG.cam_frame == f, cur = FG.cur, prev = cur ^ 1;
     if (!have_d) FG.nodepth++; if (!have_cam) FG.nocam++;
+    { static NSUInteger lw, lh, ldw; static int n; NSUInteger dw = dep ? dep.width : 0;
+      if ((lw != cw || lh != ch || ldw != dw) && n < 40) { n++; lw = cw; lh = ch; ldw = dw; char b[96] = ""; int k = 0;
+          for (int i = 0; i < 4; i++) if (FG.depth_seen[i]) k += snprintf(b + k, sizeof b - k, " %lux%lu", (unsigned long)FG.depth_seen[i].width, (unsigned long)FG.depth_seen[i].height);
+          MLOG("frame generation: scene %lux%lu at %lu,%lu, depth used %lux%lu, recent depth attachments:%s", (unsigned long)cw, (unsigned long)ch, (unsigned long)ix, (unsigned long)iy, (unsigned long)dw, (unsigned long)(dep ? dep.height : 0), b); } }
     MTLOrigin o0 = MTLOriginMake(0, 0, 0);
     id<MTL4ComputeCommandEncoder> b = blit();
     enc_b_copy_texture(b, color, 0, 0, o0, MTLSizeMake(cw, ch, 1), FG.c[cur], 0, 0, o0);
-    if (have_d) enc_b_copy_texture(b, dep, 0, 0, o0, MTLSizeMake(cw, ch, 1), FG.d[cur], 0, 0, o0);
+    if (have_d) enc_b_copy_texture(b, dep, 0, 0, MTLOriginMake(ix, iy, 0), MTLSizeMake(cw, ch, 1), FG.d[cur], 0, 0, o0);
     enc_b_copy_texture(b, out, 0, 0, o0, MTLSizeMake(ow, oh, 1), FG.cv0, 0, 0, o0);
     FG.cv = out; FG.gen_ready = 0;
     int ok = have_d && have_cam && FG.prev_ok && FG.last_frame == f - 1; double m[16];
diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m
index 0d26722c..dc533217 100644
--- a/dm/gfx/nvn/nvn_mtl.m
+++ b/dm/gfx/nvn/nvn_mtl.m
@@ -16320,7 +16320,7 @@ static int up_mfx_encode(NvShader *cs) {
         args_dirty_all(); g_up_mfx_kept++; return 0;   // only the scratch image was scaled; the game's shader can still produce the output
     }
     args_dirty_all();
-    if (!direct) { g_fg_soft = soft; fg_frame(sc, color, out, &orr, cw, ch); }   // scratch (DM_FRAMEGEN)
+    if (!direct) { g_fg_soft = soft; fg_frame(sc, color, out, &orr, cw, ch, (NSUInteger)ir.xoffset, (NSUInteger)ir.yoffset); }   // scratch (DM_FRAMEGEN)
     else if (fg_on()) { static int once; if (!once++) MLOG("[RED] frame generation: MetalFX scales straight between the game's own textures here (%s), which the scratch build does not hook; frames are shown as the game draws them", cs->key); }
     if (!g_up_mfx_n++) MLOG("upscaler: first dispatch of %s drawn by MetalFX, %dx%d at %d,%d of %lux%lu (format %lu) → %dx%d at %d,%d of %lux%lu (format %lu)%s", cs->key, ir.width, ir.height, ir.xoffset, ir.yoffset,
                             (unsigned long)in.width, (unsigned long)in.height, (unsigned long)in.pixelFormat, orr.width, orr.height, orr.xoffset, orr.yoffset, (unsigned long)out.width, (unsigned long)out.height,

[dpadr]

@devMlGUE thank you, that was the cause and your fix is in: 76e6843e keeps the last four depth attachments and uses the newest one of exactly the scene's size, with your log line naming the choice. I left out the second rule (a larger depth texture with an offset), as you suggested, since nothing exercises it yet; it can come back with a case that needs it. Your figures are in the commit message (dynamic resolution on, M1 Pro: 157–219 generated per 600 shown, "no depth" 0, 37–44 shown frames a second).

It is compile-checked only on our side: we still have no shader library for main's chain on this Mac, so your session remains the test.

[devMlGUE]

Played ~42 min on 76e6843 with dynamic resolution on (M1 Pro, handheld, 30 fps target, MetalFX). The depth pick matched the scene size every time while the resolution moved between 854x480 and 1440x810, and "no depth" stayed at 0 for the whole session. Frame gen held a median of ~58 fps shown, and the only red line was the usual one at frame 600 before the first MetalFX upscale.

One thing I noticed: when the resolution switches you can see it, mostly on trees and far away stuff. I think that's just the drop to 854x480 rather than frame gen, but I didn't compare with frame gen off. The fix itself looks good to me, thanks!

[dpadr]

#2578 @devMlGUE successfully tested these together locally but haven't rebased to main yet but this should cut latency in half while framegen is on