Skip to content

Scale decoded NDVE movies on Metal and skip matched reconstruction - #1331

Closed
givethesourceplox wants to merge 2 commits into
seedds:mainfrom
givethesourceplox:fix/ndve-movie-direct
Closed

givethesourceplox wants to merge 2 commits into
seedds:mainfrom
givethesourceplox:fix/ndve-movie-direct

Conversation

@givethesourceplox

@givethesourceplox givethesourceplox commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Ocarina of Time's title movie shows rectangular color blocks. Earlier GPU readbacks found a clean 1920×1080 RGBA frame after AV1 decoding and color conversion; corruption first appeared in the following NDVE intermediate and carried through to the 2560×1440 output.

With DM_NVN_NDVE_MOVIE_DIRECT=1, scale that clean frame into the movie's normal output with a bilinear Metal draw. Match the conversion shader, same-submission lifetime, RGBA8 image formats and observed dimensions. The option is off by default and deliberately substitutes standard spatial scaling for NDVE reconstruction; the original translation error remains open.

This update also recognizes the contiguous 13-dispatch reconstruction sequence. Encode the native draw successfully before skipping its 12 intermediate dispatches and final reconstruction dispatch, while still replaying all bindings. Unknown keys, indirect dispatches, side-effect commands, invalid image bindings or mismatched outputs retain the original path. Compute-hoisted conversion does not initiate the sequence skip. Reset pending state at each submission boundary. The destination requires render-target usage, without unnecessarily demanding shader-write usage on the first frame.

Rendering reference. The small SDK-26 setter declaration now included is the same prerequisite as #1404: runtime calls retain their existing OS-27 availability guard.

Checked

Apple M1 Pro, macOS 26.5.2, Xcode 26.5:

  • Updated t_ndve_movie passes with Metal validation. It checks the 1080p-to-1440p draw, a destination without shader-write usage, output samples at both corners and the center, full-sequence recognition, wrong-key/indirect/missing-image/side-effect rejection, skipped dispatches, and preserved final program/image bindings.
  • Documentation tests: 11 pass. Chain tests: 17 pass. LLVM static analysis of production code and fixture: no new findings. git diff --check passes.
  • Prior local Ocarina integration run: title ID 0400486024DEC000, v1.1.1, title menu reached; captured movie frames had no rectangular corruption with the direct path enabled. This exact standalone PR branch has fixture validation; the game still needs [WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299's bring-up work and other open fixes.
  • No new isolated gameplay/FPS or power measurement, iPhone run or five-game regression report is claimed. This option changes movie reconstruction only and does not solve slow 3D gameplay.
  • Shader chain and shader-library inputs unchanged; no game shader rebuild.

Docs

  • Rendering finding updated; symptom index regenerated.
  • Shader chain unchanged.
  • No game files, keys or extracted data included.

The title movie shows block corruption after NDVE reconstruction. The
decoded RGBA image is clean before the reconstruction dispatches.

Add an opt-in path that scales that image into the normal movie output
with a bilinear Metal draw. Match the observed shader keys and check the
submission and texture layout before replacing the final dispatch.

Document the switch and add a Metal check for the scaled output and
submission lifetime.
@givethesourceplox givethesourceplox changed the title Scale Ocarina's decoded NDVE movie frames directly on Metal Scale decoded NDVE movies on Metal and skip matched reconstruction Nov 5, 2026
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
… skip matched reconstruction

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
) replaces the pull request heads

Upstream landed Ocarina of Time in main (chain adec7d81d970: "holds 60 fps in
Kokiri Forest, 4 GB less") and closed the pull requests it folded in, with
reworded and partly different code. Mixing a pull request head with its landed
version hunk by hunk silently broke things (two GlobalToStorage tables, a
duplicate AF2ip, two mtk_pso definitions), so the rule here is one version per
feature:

- Pull requests upstream closed (seedds#1299, seedds#1300, seedds#1312, seedds#1343, seedds#1345, seedds#1353,
  seedds#1354, seedds#1359, seedds#1360, seedds#1373, seedds#1428, seedds#1430, seedds#1434): their hunks are taken out
  of our side before the three-way merge, so upstream's landed version stands
  alone. Gone with them: bindless pool table patching (reg_bump), the ghost
  table's per-submission hold, the growing raw pool table and its write-log
  sizing, seedds#1428's pool_buf spares (upstream's pool_table_take is the same
  idea), the ixc-based indirect mesh draw (upstream uses mdx), and their tests
  (t_pool_table, t_mesh_indirect, t_gmem_raw, finalized_bindless_pool,
  test_bnsh_nvn2, test_ndve).
- Open pull requests that upstream's landing supersedes: seedds#1394 (upstream's
  mtl_pool_addressed) and seedds#1393 pool_skip_retired (upstream's budgeted pool
  walk; the profile does not use it) are dropped with their tests and docs.
  seedds#1368's mesh pipeline prewarm is dropped too: upstream kept the background
  build in nvn_geometry.inc, and the prewarm's mtk_build / mtk_claim /
  g_warm_mps went with it.
- Kept, as upstream does not have them: NXForge's own features (GPU watchdog,
  present filters and anti-aliasing, aniso floor, quick settings, pipeline
  archive compaction, upload breakdown, memory report lines) and the still
  open pull requests seedds#1304, seedds#1305, seedds#1308, seedds#1320, seedds#1331, seedds#1332, seedds#1355, seedds#1356,
  seedds#1369, seedds#1376, seedds#1377, seedds#1379, seedds#1382, seedds#1390, seedds#1391, seedds#1392, seedds#1414, seedds#1420,
  seedds#1436, seedds#1439, seedds#1446, seedds#1453.
- Combined where both sides changed the same lines: GW_N 1 << 16 with seedds#1320's
  wide-write intervals (GW_PAGES = SBW_WPAGES, 128 slots), the CPU upload log
  with upstream's linear upload route, arch_open_end before the compaction
  limit, cbn_prog in geo_gs_one (seedds#1382).
- Translator: upstream's sources and binaries, plus seedds#1414's
  InstEmitAmpereMem diff and the LDSM SubgroupBarrier in
  InstEmitAmpereTensor.cs. The binaries and dmsh_chain.lock are upstream's
  until the Mac rebuilds them (dotnet publish, tool_lock --update-lock,
  dmsh_chain --header); dmsh_chain.h is regenerated for now.
- RED_ONCE keys: upstream took 153 and 154, so the present anti-aliasing lines
  move to 156 and 157 and seedds#1420's packaged-entry line to 158 (array 159).
- Host: hle_net.c and t_net_eventfd.c are upstream's (its eventfd poll waits
  out the whole timeout, which our 5d7920c fixed another way).
- Kept ours: dm/tools/romfs_image.py and its test, so the 26 GB RomFS image is
  not packed again.

dm/tools tests on Linux: same failures as on 875449e (Linux-only quirks),
plus test_nlg4 and test_dmsh_pool_arrays, which fail on upstream main too.
Not yet built on the Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
Aeonitis pushed a commit to Aeonitis/nxnative that referenced this pull request Nov 6, 2026
…er and shader chain, the branch's host services, movie clock, packing and Ocarina of Time additions

Upstream reached about 60 fps in Kokiri Forest with its own versions of most of the branch's merged PRs, so where both changed the same code upstream's is kept:
- The shader chain is upstream's (translator, lock, dmsh tools, Ryujinx sources), plus the branch's bracket-depth flattening (dmsh_depth.py, test_dmsh_depth.py); chain c732699d4fdb.
  Not carried: the LDSM subgroup fence (seedds#1376, needs a translator rebuild; upstream's HMMA/LDSM rework reports clean movies) and the shared raw-memory lookups (seedds#1405, superseded by upstream's last-entry lookups).
- The renderer (dm/gfx/nvn, dm/gfx/msl) and its tests are upstream's. Not carried: seedds#1390-seedds#1394, seedds#1404, seedds#1408, seedds#1409, seedds#1417, seedds#1418, seedds#1420, seedds#1427, seedds#1320, seedds#1331, seedds#1332, seedds#1368, seedds#1285 renderer parts, pool_skip_retired, and the branch's warning fixes and prewarm progress there. They stay on oot-integration.
- Ocarina of Time's profile is upstream's, with the branch's 12 movie compute rows in pso_manifest.tsv, PLAN.md and PR-AUDIT.md.
- shader_split.py is upstream's plus the BAHS record check (test_split_bahs_records.py).
- Host: upstream's EventFd, PlayReport send, audio lock and vibration answers; the branch's service sessions (hle_sf, hle_prepo), sync waits, AV1 movie clock and parallel decode, diag_abort frame fix and focus control are kept.
- Scripts: shader.noindex, tmp.noindex, notes outside the shader key, bench and regress changes.

Checked: the Mac host builds (GAME=0400486024DEC000); host unit tests pass; the Python tool tests pass on Python 3.13 except test_dmsh_pool_arrays, test_nro_pack, test_nsp_dump and test_nsp_incremental, which fail on upstream main as well. Not run: the full shader build on this chain, gameplay.
Haseenesss pushed a commit to Haseenesss/NX2Apple that referenced this pull request Nov 7, 2026
… running out of focus

Four patches on f25fa956:
- 0092, 0093: seedds/nxnative#1331, the direct NDVE movie scaling, adapted
  to the pin (mtl_tex_gpu_written).
- 0094: the direct scaling is on by default. DM_NVN_NDVE_MOVIE_DIRECT=0
  turns it off.
- 0095: DM_FOCUS_KEEP is on by default, so clicking into another app no
  longer freezes Tears of the Kingdom or Ocarina of Time.
@crypt0dj

crypt0dj commented Nov 8, 2026

Copy link
Copy Markdown

Second M1 Max report, and a warning about rebasing this onto current main

Rebased onto main bd979ced (chain 3ee52af3ca18) and measured on Ocarina of Time 0400486024DEC000 v131072, MacBook Pro M1 Max 32 GB, macOS 27.0.1 (26A434), applegpu_g13s. Windowed, DM_UPSCALER=game, DM_PRESENT_UPSCALE=off, same save point, only the switch differing.

The movie result reproduces, and then some. Movie frames, matched by their 3-pass signature:

n GPU median p95 frame fps
main 571 24.86 ms 25.02 25.00 ms 40.0
DM_NVN_NDVE_MOVIE_DIRECT=1 2687 1.13 ms 1.26 16.66 ms 60.0

−95.5 % of GPU, and the log confirms the path (scaling the decoded RGBA frame directly from 1920x1080 to 2560x1440, skipped 12 reconstruction dispatches). Zero [RED] lines over the run.


The warning. The patch calls mtl_pool_gpu_written(desc->pool) in ndve_scale, which current main does not have: pool_t.gpu_written and g_pool_gpuw_new are gone, replaced by address-range spans (mtl_tex_gpu_written). The literal translation compiles and looks right. It made gameplay unplayable.

Same binary, same save, DM_NVN_NDVE_MOVIE_DIRECT=1 throughout, against a diagnostic build that left only that call out. Frames of 2000+ draws:

with mtl_tex_gpu_written(desc) without
frame time 215.2 ms 33.7 ms
submit thread (@sub_wall_ms) 205.2 ms 9.2 ms
instructions retired a frame (@sub_instr) 1.73e9 6.6e7
GPU 45.9 ms 35.7 ms

The GPU was never the cost. No pipeline was built, no texture uploaded (new pipelines 0 + shader libraries 0, texture uploads 0 first + 0 re-uploads), and page faults, mprotect calls and pages were identical in both (4 / 12 / 138 a frame) — so it is not the write protection gws_stamp installs, which is what I first assumed and had to drop.

The mechanism is the span's lifetime. mtl_tex_gpu_written records the movie's 2560×1440 output, about 14 MB, and the span stays for the rest of the run. gws_owned, which tex_cpu_content calls for every uncompressed 2D texture (nvn_mtl.m:3734, :3753), walks the overlapping spans and scans that range's dirty pages under g_gws_mu on each one. Hundreds of textures a frame, each scanning 14 MB of page state. Outdoor areas are worst because they reach more textures a frame — which is exactly how it shows up in play: interiors fine, exteriors a slideshow.

Dropping the call, rather than translating it, looks like the right port. mtl_tex_gpu_written's only other caller on main is the shader-image case at nvn_mtl.m:8977; no render target is recorded through it. This output is a render target — ndve_scale checks out.usage & MTLTextureUsageRenderTarget before it does anything — so main's render-target path already covers what its storage is worth. With the call gone, the movie keeps 16.7 ms / 60 fps and gameplay returns to 33.7 ms against 29.7 on plain main, over 12,217 frames.

Not checked: picture quality of the bilinear scale against the network, by eye or by measurement; whether the same applies to any other game; and I have not run regress.py check --games.

(The comparison that produced this was between three runs of one binary; happy to attach the per-frame TSVs if useful.)

@crypt0dj

crypt0dj commented Nov 8, 2026

Copy link
Copy Markdown

Follow-up on the one thing my earlier report left open: the picture.

Watched the opening on the M1 Max with DM_NVN_NDVE_MOVIE_DIRECT=1 and saw no quality loss worth reporting. By eye, at 2560×1440, the bilinear scale is not distinguishable from the network's output in normal viewing. This is an eye check by the person running it, not a measurement against a reference frame, so take it for what it is.

One more data point: this ran with DM_UPSCALER=metalfx and DM_PRESENT_UPSCALE=metalfx, not the game scaler my measurements used, and the path still engages (NDVE movie: scaling the decoded RGBA frame directly…, skipped 12 reconstruction dispatches). So the movie path is independent of the upscaler choice — MetalFX replaces the NDSR network, not NDVE — and nobody has to choose between MetalFX and 60 fps movies.

@matijagrcic

Copy link
Copy Markdown

Tested in my NX2Apple 1.2.3-based with DM_NVN_NDVE_MOVIE_DIRECT=1. The movie/title sequence runs around 60 fps, and the log confirms direct scaling of the decoded frame with reconstruction dispatches skipped. This is included alongside #1427, #1707 and #1708 in my working build.

@seedds

seedds commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Thank you for this, and thanks to everyone who measured it on M1 Max and M1 Pro: those reports showed how much the network cost on smaller GPUs.

I'm closing this without landing it. c66a36a takes a different route to the same goal: nn::ndve's movie network now runs as the host's own Metal kernels, on by default (from #1894, reworked). It computes the same picture as the game's network rather than a bilinear scale of the decoded frame. In check mode over 1,500 frames, it matched the game's network in all but 10 texels, each off by 1/1023. On an M4 Pro the title movie's GPU time went from 12.78 to 6.77 ms a frame; on an M1 Pro the network alone went from 36.3 to 10.8 ms (#1894's standalone measurement). That should bring the title and the movies to 60 fps on M1-class Macs, but nobody has measured that in the game yet.

Skipping the network would still be cheaper (about 1 ms), but the picture is no longer the game's, so it's a workaround we'd only keep if the engine fell short. If it still doesn't reach 60 fps on your Mac, please post the frame's GPU time (DM_NVN_GPUPROF=every:600) on #1375 and we'll look again.

@seedds seedds closed this Nov 9, 2026
Fasen24-AI added a commit to Fasen24-AI/swordboiMacNative that referenced this pull request Nov 10, 2026
…d reconstruction

Upstream pull requests applied in this commit (their authors' work, brought onto this tree with a three-way merge; the description is in each PR):
- seedds#1331 by givethesourceplox (closed): Scale decoded NDVE movies on Metal and skip matched reconstruction
Fasen24-AI added a commit to Fasen24-AI/swordboiMacNative that referenced this pull request Nov 10, 2026
…mtl_tex_gpu_written

Upstream pull requests applied in this commit (their authors' work, brought onto this tree with a three-way merge; the description is in each PR):
- seedds#1331 by givethesourceplox (closed): Scale decoded NDVE movies on Metal and skip matched reconstruction
Fasen24-AI added a commit to Fasen24-AI/swordboiMacNative that referenced this pull request Nov 10, 2026
…a 23 was built from

Ours kept where main has not taken the change: the pool test of image_texture (virtual pools), need_make with seedds#2075's stand-in,
the copy absorb beside main's gsb_mark, DM_SHARPEN's copy_t2t_draw_x, the render scale (seedds#1903) and fixed resolution stubs beside
seedds#2020's native output and seedds#2021's render size. Main's own taken for the depth rule (fb_depth_rule), the family textures and the
movie network: the opt-in shortcut of PR seedds#1331 (DM_NVN_NDVE_MOVIE_DIRECT) is removed, main runs the network as host kernels.
Generated files (chain header, lock, translator binaries, symptoms, game-patches) are main's here and rebuilt in the next commit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014eD437mxrMnW2UwLZhpwc7

Upstream pull requests applied in this commit (their authors' work, brought onto this tree with a three-way merge; the description is in each PR):
- seedds#1331 by givethesourceplox (closed): Scale decoded NDVE movies on Metal and skip matched reconstruction
- seedds#1903 by FifteenPen (open): Ocarina of Time takes the launcher's Render scale: the 3D views at that fraction of the game's own size, the dynamic resolution held at its top level (Kokiri Forest docked at 0.75: GPU p50 15.77 → 14.31 ms, 60 fps held)
- seedds#2020 by padsiwono (open): Ocarina of Time v131072: Output Native draws the full output size instead of the game's 80 % frame (`DM_OUTPUT`)
- seedds#2021 by padsiwono (open): Ocarina of Time v131072 renders at the launcher's Resolution (`DM_RENDER_SIZE`), docked
- seedds#2075 by sleepyeyedjoe (closed): Ocarina of Time: a patch of scenery no longer draws dark for one frame when the game gives it a sharper texture (a pool texture the walk does not make holds a stand-in of its own colours until its first read)
Fasen24-AI added a commit to Fasen24-AI/swordboiMacNative that referenced this pull request Nov 10, 2026
…he look-ahead argument went with PR seedds#1331's shortcut)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014eD437mxrMnW2UwLZhpwc7

Upstream pull requests applied in this commit (their authors' work, brought onto this tree with a three-way merge; the description is in each PR):
- seedds#1331 by givethesourceplox (closed): Scale decoded NDVE movies on Metal and skip matched reconstruction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants