Skip to content

Fix Ocarina of Time foliage jitter by preserving live shader inputs - #1343

Closed
nanotchec wants to merge 1 commit into
seedds:mainfrom
nanotchec:codex/fix-live-image-load-inputs
Closed

nanotchec wants to merge 1 commit into
seedds:mainfrom
nanotchec:codex/fix-live-image-load-inputs

Conversation

@nanotchec

@nanotchec nanotchec commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Ryujinx's dead-code removal could detach the inputs of a multi-result instruction when only one of its results lost its last use. If another result remained live, the instruction survived but its coordinate calculations could be removed. This produced image reads with uninitialized coordinates in two compute shaders used by Ocarina of Time's foliage simulation.

Use the existing IsUnused check before recursing into a producer. It checks every destination and preserves instructions with side effects. Cleanup still recurses into fully dead producers.

Checked

  • On upstream 7249ee9, a synthetic Ampere image load loses its coordinates with the original bundled translator in both tested channel orders. The patched bundled translator passes both, along with the fully dead load and atomic controls. All 83 targeted checks passed, including the Ampere translation tests, AnyCPU and chain-header checks. The tool lock matches the bundled tools.
  • Retranslation restores the missing coordinate definitions in both foliage compute shaders. Their reflection matches apart from the chain identifier.
  • Ocarina of Time 1.1.1, Apple M3 Max, macOS 27.0.1. On the same 30 fps route into Kokiri Forest, an overlay of just the two retranslated shaders reduced frame-to-frame color variation by about 80% in the sampled grass region and 84% in the moss region. In the pre-upscale depth buffer, the fraction of pixels with changing depth fell from 29.2% to 8.2% and from 22.2% to 6.7%, respectively. The captures had matching framing, camera motion stayed below the comparison threshold, and control regions stayed stable.
  • A subsequent full rebuild of the local Ocarina of Time build succeeded for all 42,853 shaders. Exactly eight generated MSL sources changed; the other 42,845 were byte-identical. Besides the foliage shaders, this restores previously eliminated inputs and matrix calculations in NDSR/NDVE kernels. The full rebuilt application completed the same docked Kokiri route with normal rendering and no shader/pipeline errors.
  • Restoring the additional NDSR computations adds GPU work: in the sampled warm band, median GPU time was 35.96 ms before and 37.80 ms after. This is a rendering-correctness change. The window was intermittently covered, so the run does not establish displayed-frame cadence.
  • Runtime validation used the local Ocarina of Time integration. The source patch and bundled translator were separately validated on upstream 7249ee9; a full game run and multi-game regression on that upstream base, and comparison with Switch 2, remain pending.

Docs

Updated the shader reference entry and regenerated the symptom index. Regenerated the shader tool lock and chain header for the rebuilt translator.

Dead-code removal could drop the coordinate inputs of an image load while another result was still in use. Reuse the existing liveness test before walking into a producer and cover both channel orders with a synthetic Ampere regression.

The rebuilt translator passes 83 targeted checks. On the same Kokiri Forest route, replacing just the two retranslated shaders reduced frame-to-frame color variation in the sampled grass and moss.
@MetaIX

MetaIX commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

This also fixes Ocarina's title movie. We found the same RemoveNode bug independently, through the movie's upscaler, and patched it the same way (!HasSideEffects(src.AsgOp) && DestHasNoUses(src.AsgOp), equivalent to your IsUnused).

Drafted with Claude; reviewed by me.

@PaulEins

PaulEins commented Nov 5, 2026

Copy link
Copy Markdown

Tested on a real device: the foliage jitter is gone.

Ocarina of Time 0400486024DEC000 v131072 (update-only NUP), Mac app, Apple M4 Pro, 48 GB, macOS 26.7.2, Xcode 27.2. This PR (6941aa84) merged into stan69b's oot-clean (6c71e575) together with #1345, #1332, #1350 and #1376, translator rebuilt, chain ba403debcc6c. 42,853 of 42,854 shaders built.

Without this PR, grass and plants in Kokiri Forest twitched constantly, as if in heavy wind. With it they are calm. Full report in #1265.

Drafted with Claude; reviewed by me.

@PaulEins

PaulEins commented Nov 6, 2026

Copy link
Copy Markdown

Test report: Ocarina of Time (v131072 NUP), 2 hours on an M4 Pro with 48 GB — three crash types (one reproducible with the Ocarina), ground fog, Deku Tree floors

A report from a player, not a developer. About 2 hours of play, from the start until leaving Kokiri Forest. Logs from scripts/bug_report.py and screenshots are attached. No game files, keys or extracted data are attached.

Hardware and software

  • MacBook Pro 14" (Nov 2024), Apple M4 Pro, 48 GB, macOS Tahoe 26.7.2
  • Xcode 27.2 (27B52), Metal Toolchain 27B47
  • Homebrew python 3.15.2, dotnet-sdk (.NET SDK 10.0.403). Venv: lz4, capstone 5.0.9, fonttools, zbic, nsz

Game

0400486024DEC000, update-only NUP v131072 (24.4 GB), no DLC, no prod.keys. The script derives a v131072 profile from #1299's v0 profile.

What was built

stan69b's oot-clean at 6c71e575, with these merged locally:

PR Head Merge
#1345 audio update lock d7a43c2a clean
#1332 GPU-lost stop 7f5f2fcf clean
#1350 GPU timestamps per segment 9e5e8763 clean
#1343 foliage jitter 6941aa84 conflicts only in dmsh_chain.h, dmsh_chain.lock, dmsh_xlate_bin/
#1376 LDSM subgroup fence 2d6f86f8 same

Generated-file conflicts kept oot-clean's side, then the translator was rebuilt as in stan69b's 9bb57c4 (dotnet build … -c Release, three dlls copied, tool_lock.py --update-lock, dmsh_chain.py --header) → chain ba403debcc6c, --check-tools passes. build_ipa_from_nsp.py <nup> --platform mac-app: 42,853 of 42,854 shaders.

env.txt: DM_FULLSCREEN=0, DM_ALIAS_SIZE=0x400000000. Settings at the defaults; mostly Docked, some Handheld.

The 2-hour session ran on this build plus one experimental host change (a tex_fit cube → 2D array view, see "Ground fog" below), which did not change what is seen. All three crashes below happened on that build.

What works well

Crash 1: -[NvPkg buffer]: unrecognized selector (entering a room in the Deku Tree)

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[NvPkg buffer]: unrecognized selector sent to instance 0x7f3192da0'
bt: … objc_exception_throw … ___forwarding___ … tex_by_id+0x360 pool_entry+0x174 pool_gpu+0x308 encode_args_+0x2380 … draw_ … h_nvnQueueSubmitCommands …

It happened after about 38,000 frames in that session. Memory was flat (10.6 GB resident, 9.3 GB GPU) and the game was at 60 fps right up to the crash.

Reading of the source (untested): in tex_mtl_, when the packaged path is not taken but the serial already holds an NvPkg, the line if (have) { … return have; } returns the package object itself, and tex_by_id then calls base.buffer on it. tex_made unwraps an NvPkg, but this return does not. The same line is on main 865cc35f. A possible change:

+    if (have && [have isKindOfClass:[NvPkg class]]) { NvPkg *pk = have; have = pk->view ? pk->view : pk->tex; if (!have) return nil; }
     if (have) { if (t->pool && nvn_sparse_pool(t->pool)) sparse_gpu_look(t, have); return have; }

It is built here but not played yet, so I can't say whether it prevents the crash. Log: nxnative-report-absturz-40min.zip.

Crash 2: reproducible when using the Ocarina (3 times in a row)

Taking out the Ocarina crashed the game immediately, three times in a row:

[longframe] frame 16020 (348.17 s) whole frame 87.4 ms, submit+present 87.3 ms, GPU 22.6 ms; … texture uploads 1003 first + 17 re-uploads, 387.9 MB / 54.2 ms
[RED] BufferInitialize: pool has no storage (pool 0x0 flags 0 offset 0 size 0x3d0)
CRASH sig=11 addr=0x4 pc=h_nvnMemoryPoolGetFlags+0 lr=dm_bigstack_ret+0
bt: main+0x8e51ec main+0x14d2a3c main+0x8c780 main+0x8c614 main+0x436fb4 main+0x435a70 main+0x3b95a14 main+0x367b00 main+0x363d8c main+0x182268 main+0x14bd4 main+0xdd26c0 main+0x3482ec8 sdk+0x1673d0 sdk+0x16eb58
stack scan: [-88]h_nvnBufferInitialize+0x158

The game calls nvnBufferInitialize with a NULL pool, and the host faults reading the pool's flags. In the frames just before, the game uploads 850–1,000 new textures a frame (about 320–400 MB a frame), with frames of 72–87 ms in submit+present. The last memory lines show GPU allocated 12039 MB.

My guess, unverified: one of the game's own allocations failed during that load (guest heap or address space), and the game passed the NULL pool on. Log: nxnative-report-absturz2.zip. I can reproduce this on demand and run it with any switch that would help.

Crash 3: repeated crashes after about 30 minutes

Over the session the game also crashed several times, roughly every 30 minutes, more often towards the end. The window closes and I'm back on the desktop with no message. The logs were only kept for the last two runs, so the ones above are the two I have.

Ground fog renders as see-through, pale patches

In Kokiri Village, where Switch 2 footage of the same spots shows low ground fog, the Mac shows flat pale-blue/white patches with hard polygon edges, and the ground is see-through there. It depends on the camera: from some angles the same spot looks right. Screenshots are attached.

Ruled out: the log had two texture type mismatch lines (shaders 09ab31c69af710c9_b / 9b310b3349825e9a_b, slot 7 declared Metal type 3, texture IDs 906 / 908 are type 5 = cube). tex_fit has no cube → 2D array case, so I added one (ns = 6 for a cube, t.arrayLength * 6 for a cube array). The red lines went away (texture type mismatches no view bridges 0 in total), but the fog patches stayed the same. So that mismatch is not the cause.

Deku Tree: missing floors

  • In places you can look through the floor.
  • After going up a level inside the tree, the floor of that level is sometimes not drawn: an in-between layer is missing.

Sound

Occasional sound problems, mostly in voiced cutscenes. The device side looks clean (0 underruns, audren render time max 0.29 ms of a 5 ms budget), so it may be the game's own Opus streams going idle, as stan69b traced for the opening movie (B-5).

Frame rate and controls

  • Kokiri Village: sluggish. Docked about 37–56 fps (GPU about 18–26 ms), Handheld about 44–60 fps (GPU about 15–23 ms), and the camera and controls feel delayed. The log shows game enqueue to target display p50 58–82 ms with frame latency 1 (window).
  • Deku Tree: 60 fps, GPU about 10 ms, direct and responsive.

Next on this Mac

Playing the build with the NvPkg change above, and the Ocarina case with any switch you suggest. Happy to test branches or PRs.

Drafted with Claude; reviewed by me.
nxnative-report-absturz2.zip
Uploading nxnative-report-absturz-40min.zip…

fpedrolucas95 pushed a commit to fpedrolucas95/nxnative that referenced this pull request Nov 6, 2026
fpedrolucas95 pushed a commit to fpedrolucas95/nxnative that referenced this pull request Nov 6, 2026
…, seedds#1410, seedds#1411, seedds#1412, seedds#1413, seedds#1414, seedds#1329 on top of main's translator fixes): chain c9cf9613657e

Publishes Ryujinx.Graphics.Shader, Ryujinx.Common, Spv.Generator and dmsh_xlate (osx-arm64, framework-dependent, the universal launcher kept) from the merged source; the
PRs' own binaries were built without this tree's translator changes (float atomic adds, F2IP, the growing raw pool table). The lock and dmsh_chain.h are written again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PaulEins

PaulEins commented Nov 6, 2026

Copy link
Copy Markdown

Tested on a real device (M4 Pro 14-core CPU / 20-core GPU, 48 GB, Ocarina of Time v131072, Mac app): the foliage jitter is gone. Before, every plant moved as if in constant heavy wind. Full report in #1265.

Drafted with Claude; reviewed by me.

@seedds

seedds commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Thank you. You found the same root cause as I did: dead-code removal dropping a coordinate of a multi-result instruction. The fix is on main (landed on main in b1d7c70 and 6121fd7), so I'm closing this. Final checks on the rebased tree (full shader build, walks, regression) are still running, so please report anything you see.

@seedds seedds closed this Nov 6, 2026
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
) replaces the pull request heads

Upstream landed Ocarina of Time in main (chain adec7d81d970: "holds 60 fps in
Kokiri Forest, 4 GB less") and closed the pull requests it folded in, with
reworded and partly different code. Mixing a pull request head with its landed
version hunk by hunk silently broke things (two GlobalToStorage tables, a
duplicate AF2ip, two mtk_pso definitions), so the rule here is one version per
feature:

- Pull requests upstream closed (seedds#1299, seedds#1300, seedds#1312, seedds#1343, seedds#1345, seedds#1353,
  seedds#1354, seedds#1359, seedds#1360, seedds#1373, seedds#1428, seedds#1430, seedds#1434): their hunks are taken out
  of our side before the three-way merge, so upstream's landed version stands
  alone. Gone with them: bindless pool table patching (reg_bump), the ghost
  table's per-submission hold, the growing raw pool table and its write-log
  sizing, seedds#1428's pool_buf spares (upstream's pool_table_take is the same
  idea), the ixc-based indirect mesh draw (upstream uses mdx), and their tests
  (t_pool_table, t_mesh_indirect, t_gmem_raw, finalized_bindless_pool,
  test_bnsh_nvn2, test_ndve).
- Open pull requests that upstream's landing supersedes: seedds#1394 (upstream's
  mtl_pool_addressed) and seedds#1393 pool_skip_retired (upstream's budgeted pool
  walk; the profile does not use it) are dropped with their tests and docs.
  seedds#1368's mesh pipeline prewarm is dropped too: upstream kept the background
  build in nvn_geometry.inc, and the prewarm's mtk_build / mtk_claim /
  g_warm_mps went with it.
- Kept, as upstream does not have them: NXForge's own features (GPU watchdog,
  present filters and anti-aliasing, aniso floor, quick settings, pipeline
  archive compaction, upload breakdown, memory report lines) and the still
  open pull requests seedds#1304, seedds#1305, seedds#1308, seedds#1320, seedds#1331, seedds#1332, seedds#1355, seedds#1356,
  seedds#1369, seedds#1376, seedds#1377, seedds#1379, seedds#1382, seedds#1390, seedds#1391, seedds#1392, seedds#1414, seedds#1420,
  seedds#1436, seedds#1439, seedds#1446, seedds#1453.
- Combined where both sides changed the same lines: GW_N 1 << 16 with seedds#1320's
  wide-write intervals (GW_PAGES = SBW_WPAGES, 128 slots), the CPU upload log
  with upstream's linear upload route, arch_open_end before the compaction
  limit, cbn_prog in geo_gs_one (seedds#1382).
- Translator: upstream's sources and binaries, plus seedds#1414's
  InstEmitAmpereMem diff and the LDSM SubgroupBarrier in
  InstEmitAmpereTensor.cs. The binaries and dmsh_chain.lock are upstream's
  until the Mac rebuilds them (dotnet publish, tool_lock --update-lock,
  dmsh_chain --header); dmsh_chain.h is regenerated for now.
- RED_ONCE keys: upstream took 153 and 154, so the present anti-aliasing lines
  move to 156 and 157 and seedds#1420's packaged-entry line to 158 (array 159).
- Host: hle_net.c and t_net_eventfd.c are upstream's (its eventfd poll waits
  out the whole timeout, which our 5d7920c fixed another way).
- Kept ours: dm/tools/romfs_image.py and its test, so the 26 GB RomFS image is
  not packed again.

dm/tools tests on Linux: same failures as on 875449e (Linux-only quirks),
plus test_nlg4 and test_dmsh_pool_arrays, which fail on upstream main too.
Not yet built on the Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants