Repository navigation
Run nn::ndve's movie network as host Metal kernels behind DM_NDVE_ENGINE=1 - #1837
Closed
zephleggett wants to merge 1 commit into
Closed
zephleggett wants to merge 1 commit into
zephleggett wants to merge 1 commit into
Conversation
…NDVE_ENGINE=1, off by default): Ocarina of Time's title-movie network takes 10.8 ms of GPU instead of 36.3 on an M1 Pro with 16 GPU cores, standalone, with the translated shaders' bits wherever their lane race leaves them stable (issue #1375) The title and the movies run 14 compute dispatches a frame: the colour conversion 49dff010, eleven HMMA.16816 convolutions of a U-Net over the picture packed 2x2 at 960x540, the last convolution added to the picture (5a7d69b0) and a 4/3 resampler with a small network of its own (4f41ccf2). On an M1 Pro the translated network costs about 38 ms of GPU a frame, so the title runs at about 24 fps (24.8 ms on an M1 Max in the issue). The translated HMMAs still take B through the SIMD-group's scratch and the weights' LDSM and MOVM through shuffles. nvn_mtl.m's dispatch now hands those dispatches to nvn_ndve.inc when DM_NDVE_ENGINE is 1 (or ab with DM_AB). It recognizes the network by its 14 shader keys in order and its weight table (each dispatch's weight and bias buffers at the network's offsets from the first's), then the picture and image sizes and formats and the resampler's constants; anything else leaves the game's dispatches running, logged once, and a mismatch after some dispatches were skipped turns the host's network off for the rest of the run. It skips the first eleven, runs the whole network at 5a7d69b0 into the image that dispatch writes, from the picture it reads and the weights the game bound, and runs its own resampler at 4f41ccf2. The kernels are dm/gfx/msl/ndve_engine.metal, built into the host with its other shaders (host_msl.py), so nothing compiles at run time. They do the translated shaders' arithmetic in their order: each HMMA as dmsh_build.py's HMMA_BODY does it (simdgroup_half8x8, simdgroup_multiply for a chain's first call, then multiply_accumulate over input channels 0-7 and 8-15), the chain over the taps and then the 16-channel blocks, the epilogue's D + bias in f32 with ReLU and one rounding, the picture's sum in the game's order under safe math. The translated network has a lane race on main: run twice on the same input, 10 of its 12 dispatches give different values in some outputs (5a7d69b0 in 13.5 % of its pixels over 12 runs); end to end, three runs on one frame differ from each other in 42-46 % of the pixels, by up to 502 of 1,023 codes. The host's network gives at every output a value some run gives, and at outputs that never change, that value. The race's cause in the translation is open. The matrix hardware rounds simdgroup_half8x8's accumulator after every product (M1 Pro, 40,000 of 40,000 random 8-product cases), and an A100's HMMA with an F16 accumulator rounds the sum of 8 products once (5,000 of 5,000 cases recorded on an A100 in Khattak and Mikaitis's tensor core models, against 2,420 for the per-product rounding). So the translated helper, and the host's network with it, differ from the Switch in the low bits of most HMMAs; shaders.md says what an exact helper would do. The host's network follows the helper, and its test fails when they differ. Checked: dm/gfx/tests/test_ndve_engine.py on the M1 Pro: every kernel at the network's sizes on random inputs and weights, bit for bit against references that call dmsh_hmma16816_f16 as hmma_msl rewrites it, in PTX's fragment layout with TensorPass's chain flags (all identical; with an f32 accumulator per HMMA instead, 6-25 % of each HMMA kernel's outputs differ), and the kernel names the host binds. Main's 13 translated dispatches (chain c096b1d1dc53) standalone on two title-movie frames and a random picture against the host's: equal at every output that is the same in 6 or 12 runs, and every changing output equals the host's in some run; the resampler identical. make -C dm/host/tests run; make -C dm/gfx/tests GAME=0400486024DEC000 run (t_present_crop crashes in its MetalFX scaler resize check, as on main on this Mac); dm/tools/tests (the same five failures as on main here: test_env_switches, test_nsp_scan, test_tree_tar, test_triage_report, test_usage). Not run: the game itself with the switch on (frame rate, GPU span, the picture compared in the game), other Macs, the iPhone.
Know-It-All-Brother
added a commit
to Know-It-All-Brother/nxnative
that referenced
this pull request
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
On M1-class Macs, Ocarina of Time's title screen and movies run at about 24 fps. The movie library's NDVE network takes 36 ms of GPU a frame (#1375).
This PR adds an opt-in engine that runs the same network as 10 hand-written Metal kernels. On an M1 Pro it takes 10.8 ms.
DM_NDVE_ENGINE=1turns it on.dmsh_hmma16816_f16, so it reproduces main's output.Closes #1375.
Exactness
test_ndve_engine.py: all 10 kernels at the network's sizes, on random inputs and weights. They are bit-identical to references built from main's HMMA helper. An f32-accumulator variant fails 6–25 % of each HMMA kernel's outputs, so the test can tell the difference.Two findings for the translator
Lane race. Main's translated network is not deterministic.
Rounding. Main's helper rounds the f16 accumulator after every product, which is how
simdgroup_half8x8behaves on Apple GPUs. Recorded A100 data (Khattak and Mikaitis's tensor core models) shows that HMMA with an F16 accumulator rounds the sum of 8 products once:The engine follows main for now. If the helper changes, the test fails until the engine is updated.
Checked
Device: M1 Pro (16-core GPU, 16 GB), macOS 26.5.2, Xcode 26.6.
Tests:
test_ndve_engine.pypasses.make -C dm/host/tests runpasses.make -C dm/gfx/tests GAME=0400486024DEC000 runpasses exceptt_present_crop, which crashes the same way on unmodified main on this Mac.dm/tools/testsfails the same tests as unmodified main.Speed: measured standalone on one title-movie frame.
Reach:
regress.py impactlists BotW, Scarlet, TotK and Hades only becausenvn_mtl.m's dispatch changed. With the switch unset, the hook returns on its first line. I did not runregress.py check.Not checked on this branch:
ndve_engine.metal;DM_NVN_ENC_THREADpath;Scope
The change is 853 lines, 361 of them the test. It is a second path for work the translator already does, so CONTRIBUTING asks to agree on scope first. If you would rather settle that in #1375, I'll move it there.
#1331 skips the network instead: it is cheaper, but the picture is not the game's. The two can coexist.
Docs
docs/reference/shaders.md: new entry "The NDVE movie network by the host".python3 dm/tools/docs_symptoms.pywas run.docs/env_switches.tsvare updated.