Skip to content

Run nn::ndve's movie network as host Metal kernels behind DM_NDVE_ENGINE=1 - #1837

Closed
zephleggett wants to merge 1 commit into
seedds:mainfrom
zephleggett:ndve-engine
Closed

zephleggett wants to merge 1 commit into
seedds:mainfrom
zephleggett:ndve-engine

Conversation

@zephleggett

Copy link
Copy Markdown

What and why

On M1-class Macs, Ocarina of Time's title screen and movies run at about 24 fps. The movie library's NDVE network takes 36 ms of GPU a frame (#1375).

This PR adds an opt-in engine that runs the same network as 10 hand-written Metal kernels. On an M1 Pro it takes 10.8 ms.

  • Off by default. DM_NDVE_ENGINE=1 turns it on.
  • Fail-safe. The hook recognizes the network by its 14 shader keys in order, its weight layout, and its image sizes and formats. On any mismatch, the game's own dispatches run.
  • Main's bits. The engine uses the same HMMA operations as main's dmsh_hmma16816_f16, so it reproduces main's output.

Closes #1375.

Exactness

  • test_ndve_engine.py: all 10 kernels at the network's sizes, on random inputs and weights. They are bit-identical to references built from main's HMMA helper. An f32-accumulator variant fails 6–25 % of each HMMA kernel's outputs, so the test can tell the difference.
  • Main's translated shaders: 2 title-movie frames and 1 random picture. The engine matches at every output that main computes the same way on every run. The remaining outputs vary between main's own runs; see the first finding below.

Two findings for the translator

  1. Lane race. Main's translated network is not deterministic.

    • 10 of its 12 dispatches give different values from run to run.
    • Three end-to-end runs of one frame differ in 42–46 % of pixels, by up to 502 of 1,023 codes.
    • The engine has no race.
  2. Rounding. Main's helper rounds the f16 accumulator after every product, which is how simdgroup_half8x8 behaves on Apple GPUs. Recorded A100 data (Khattak and Mikaitis's tensor core models) shows that HMMA with an F16 accumulator rounds the sum of 8 products once:

    • rounding the sum once matches 5,000 of 5,000 cases;
    • per-product rounding matches 2,420.

    The engine follows main for now. If the helper changes, the test fails until the engine is updated.

Checked

  • Device: M1 Pro (16-core GPU, 16 GB), macOS 26.5.2, Xcode 26.6.

  • Tests:

    • test_ndve_engine.py passes.
    • make -C dm/host/tests run passes.
    • make -C dm/gfx/tests GAME=0400486024DEC000 run passes except t_present_crop, which crashes the same way on unmodified main on this Mac.
    • dm/tools/tests fails the same tests as unmodified main.
  • Speed: measured standalone on one title-movie frame.

    Path GPU per frame
    Translated network 36.3 ms (sum of best-of-10 per dispatch)
    Engine 10.8 ms (best of 19)
  • Reach: regress.py impact lists BotW, Scarlet, TotK and Hades only because nvn_mtl.m's dispatch changed. With the switch unset, the hook returns on its first line. I did not run regress.py check.

  • Not checked on this branch:

    • in-game fps and picture (on a downstream tree with the engine, the title held 60 fps at 12.5 ms of GPU);
    • the iOS compile of ndve_engine.metal;
    • the DM_NVN_ENC_THREAD path;
    • other Macs.

Scope

The change is 853 lines, 361 of them the test. It is a second path for work the translator already does, so CONTRIBUTING asks to agree on scope first. If you would rather settle that in #1375, I'll move it there.

#1331 skips the network instead: it is cheaper, but the picture is not the game's. The two can coexist.

Docs

  • docs/reference/shaders.md: new entry "The NDVE movie network by the host". python3 dm/tools/docs_symptoms.py was run.
  • The game note and docs/env_switches.tsv are updated.
  • No game files, keys or extracted data. The weights come from the game's buffers at run time, and the test generates its own.

…NDVE_ENGINE=1, off by default): Ocarina of Time's title-movie network takes 10.8 ms of GPU instead of 36.3 on an M1 Pro with 16 GPU cores, standalone, with the translated shaders' bits wherever their lane race leaves them stable (issue #1375)

The title and the movies run 14 compute dispatches a frame: the colour conversion 49dff010, eleven HMMA.16816 convolutions of a U-Net over the picture packed 2x2 at 960x540, the last convolution added to the picture (5a7d69b0) and a 4/3 resampler with a small network of its own (4f41ccf2). On an M1 Pro the translated network costs about 38 ms of GPU a frame, so the title runs at about 24 fps (24.8 ms on an M1 Max in the issue). The translated HMMAs still take B through the SIMD-group's scratch and the weights' LDSM and MOVM through shuffles.

nvn_mtl.m's dispatch now hands those dispatches to nvn_ndve.inc when DM_NDVE_ENGINE is 1 (or ab with DM_AB). It recognizes the network by its 14 shader keys in order and its weight table (each dispatch's weight and bias buffers at the network's offsets from the first's), then the picture and image sizes and formats and the resampler's constants; anything else leaves the game's dispatches running, logged once, and a mismatch after some dispatches were skipped turns the host's network off for the rest of the run. It skips the first eleven, runs the whole network at 5a7d69b0 into the image that dispatch writes, from the picture it reads and the weights the game bound, and runs its own resampler at 4f41ccf2. The kernels are dm/gfx/msl/ndve_engine.metal, built into the host with its other shaders (host_msl.py), so nothing compiles at run time. They do the translated shaders' arithmetic in their order: each HMMA as dmsh_build.py's HMMA_BODY does it (simdgroup_half8x8, simdgroup_multiply for a chain's first call, then multiply_accumulate over input channels 0-7 and 8-15), the chain over the taps and then the 16-channel blocks, the epilogue's D + bias in f32 with ReLU and one rounding, the picture's sum in the game's order under safe math.

The translated network has a lane race on main: run twice on the same input, 10 of its 12 dispatches give different values in some outputs (5a7d69b0 in 13.5 % of its pixels over 12 runs); end to end, three runs on one frame differ from each other in 42-46 % of the pixels, by up to 502 of 1,023 codes. The host's network gives at every output a value some run gives, and at outputs that never change, that value. The race's cause in the translation is open.

The matrix hardware rounds simdgroup_half8x8's accumulator after every product (M1 Pro, 40,000 of 40,000 random 8-product cases), and an A100's HMMA with an F16 accumulator rounds the sum of 8 products once (5,000 of 5,000 cases recorded on an A100 in Khattak and Mikaitis's tensor core models, against 2,420 for the per-product rounding). So the translated helper, and the host's network with it, differ from the Switch in the low bits of most HMMAs; shaders.md says what an exact helper would do. The host's network follows the helper, and its test fails when they differ.

Checked: dm/gfx/tests/test_ndve_engine.py on the M1 Pro: every kernel at the network's sizes on random inputs and weights, bit for bit against references that call dmsh_hmma16816_f16 as hmma_msl rewrites it, in PTX's fragment layout with TensorPass's chain flags (all identical; with an f32 accumulator per HMMA instead, 6-25 % of each HMMA kernel's outputs differ), and the kernel names the host binds. Main's 13 translated dispatches (chain c096b1d1dc53) standalone on two title-movie frames and a random picture against the host's: equal at every output that is the same in 6 or 12 runs, and every changing output equals the host's in some run; the resampler identical. make -C dm/host/tests run; make -C dm/gfx/tests GAME=0400486024DEC000 run (t_present_crop crashes in its MetalFX scaler resize check, as on main on this Mac); dm/tools/tests (the same five failures as on main here: test_env_switches, test_nsp_scan, test_tree_tar, test_triage_report, test_usage). Not run: the game itself with the switch on (frame rate, GPU span, the picture compared in the game), other Macs, the iPhone.
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation game: 0400486024DEC000 A game labels Nov 8, 2026
@zephleggett zephleggett closed this by deleting the head repository Nov 8, 2026
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ocarina of Time on an M1 Max: the movie library's NDVE network takes 35 ms of GPU a frame, so movies and the title screen run at ~28 fps

1 participant