Skip to content

Profile key pool_skip_retired: pool tables give a finalized texture's ID the dummy instead of making its texture again (Ocarina of Time out of GPU memory at a scene change) - #1393

Open
david-on-spring wants to merge 1 commit into
seedds:mainfrom
david-on-spring:pr/pool-skip-retired

Conversation

@david-on-spring

Copy link
Copy Markdown

What and why

Ocarina of Time (Switch 2, 0400486024DEC000, v131072; profile in #1299) runs out of GPU memory when Link leaves his house for Kokiri Forest. Command buffers fail with kIOGPUCommandBufferCallbackErrorOutOfMemory, with 16 GB of GPU memory allocated against Metal's recommended working set of 12.1 GB (16 GB Mac).

Cause: a pool table for handles traced to no slot (pool_gpu) resolves every registered texture ID, as a shader reading textures through handles may reach any of them. Ocarina of Time's texture pool has about 33,000 IDs. At a scene change the game finalizes the old scene's textures and leaves their IDs registered. mtl_tex_release drops their views, but the next table rebuild made a Metal texture for each of them again from its storage, and dropped it again at the end of the submission. At the house-to-forest change that was about 1,500 textures and 2.5 GB held only through retired IDs, plus as much again waiting in deferred release.

Fix: a new profile key, pool_skip_retired, off by default (DM_NVN_POOL_SKIP_RETIRED=1 / 0 overrides it). With the key on, a retired ID that holds no view gets the dummy texture in a pool table. Not affected:

  • an ID bound to a slot;
  • a retired ID that a draw in the same submission already resolved.

The table log line now counts the IDs it skipped.

It's a key rather than the default because on the console, a handle to a finalized texture still reads the texture's storage while that storage exists. A game that keeps sampling through such handles would read the dummy. If you'd rather make it the default, or drop retired IDs from the tables some other way, I'm happy to change it.

The profile line (pool_skip_retired on) can only go in #1299's game.tsv. I've put it in a comment there rather than stacking this PR on #1299. #1299 also splits this loop out into pool_entry, so the check moves there; the comment has those lines.

Reference entry: Finalized textures made again by pool tables. The symptom index is regenerated.

Checked

  • Tests:
    • make -C dm/gfx/tests GAME=0400380020056000 check_t_pool_retired (new, 11 checks): one table with a live ID and a retired ID, with the key and without. With the skip line disabled, the "retired ID gets the dummy" check fails.
    • check_t_cpu_tex (56 checks, retired descriptors) and check_t_pool_chunks (75) pass.
    • test_game_profile.py, test_env_switches.py and test_docs.py pass.
    • lint_c.py tidy on nvn_mtl.m and the new test: no finding above the baseline. My clang-tidy is LLVM 21, and the baseline was made with 23.
    • make -C dm/host GAME=0400380020056000 dmhost builds.
    • The rest of the Metal suite (make -C dm/gfx/tests run) was not run.
  • Game run: Ocarina of Time v131072, on our stack ([WIP] The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0, SDK 22.2.8) runs on the Mac through the title movie into Kokiri Forest #1299 with other open PRs and the other two fixes I'm sending), with the key on. 25 minutes in Kokiri Forest with no out-of-memory error, GPU memory peaking at about 12 GB, and frames down from 44 to 31 ms. This branch itself wasn't played, since main has no profile for the game.
  • Devices and OS: Apple M4 (16 GB), macOS 27.0.1.
  • regress.py: not run. The key is off by default, so no game on main changes behavior.

Docs

  • The finding is in its docs/reference/nvn-objects.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
  • The key is in the keys table (dm/games/README.md), and the switch is in docs/env_switches.tsv
  • No game files, keys or data extracted from a game in the change or the attachments

Drafted with Claude; reviewed by me.

🤖 Generated with Claude Code

… ID the dummy instead of making its texture again, which ran Ocarina of Time out of GPU memory at a scene change

A pool table for handles traced to no slot (pool_gpu) resolves every registered texture ID. At a scene change Ocarina of Time (0400486024DEC000, profile in seedds#1299) finalizes the old scene's textures and leaves their IDs registered: mtl_tex_release dropped their views, and the next table rebuild made a Metal texture for each of them again from its storage. At the house-to-forest change that was about 1,500 textures and 2.5 GB held only through retired IDs, as much again in deferred release, 16 GB of GPU memory allocated against Metal's 12.1 GB working set on a 16 GB Mac, and command buffers failed with kIOGPUCommandBufferCallbackErrorOutOfMemory.

With pool_skip_retired on (DM_NVN_POOL_SKIP_RETIRED=1 / 0 overrides it), a retired ID that holds no view gets the dummy texture in the table; the table log counts them. It is a key, off by default: on the console a handle to a finalized texture still reads its storage while that is there. With it, Ocarina of Time played 25 minutes in Kokiri Forest with no out-of-memory, GPU memory peaking at about 12 GB and frames from 44 to 31 ms.

t_pool_retired checks a live and a retired ID in one table, with the key and without; it fails without the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PucaVaz

PucaVaz commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Testing this now on a second machine: Apple M4, 24 GB, macOS 27.2, Ocarina of Time v131072, docked. It's on lunks's fix/gmem-pool-table (574a38f) with this PR's check adapted and DM_NVN_POOL_SKIP_RETIRED=1, plus the 8 GB alias region from your #1299 comment. The route is the Link's house autosave → Kokiri Forest. Without the key, that same route on this Mac went from about 10 GB to a 14.6 GB footprint (13.2 GB GPU allocated) within seconds of stepping outside, so the before/after should be clear. I'll post the numbers here.

Drafted with Claude; reviewed by me.

@PucaVaz

PucaVaz commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Second-machine confirmation: Apple M4, 24 GB, macOS 27.2.

Method: lunks's fix/gmem-pool-table (574a38f, #1299's profile) with this PR ported: pool_skip_retired() with DM_NVN_POOL_SKIP_RETIRED, and your line from #1299 in pool_entry. The table line also counts the skipped IDs. Only the host was rebuilt, and the chain stays 6406124273ec. Docked at 2560x1440, from the Link's house autosave, out the door into Kokiri Forest. Our run without the key had been stopped by our own 14 GB memory limit 132 s in.

Results:

  • At the door: at frame 3900, just past the door, GPU allocated is 11.4 GB with the key against 13.2 GB without it (resident 12.2 against 14.2 GB). The house is the same in both (9.4 GB).
  • In the forest: 18 minutes with the key, GPU allocated 10.1–11.9 GB, resident 11.0–13.1 GB, no out-of-memory, GPU error or abort. Swap did not grow.
  • Tables: the 33,024-ID 2D table gives 12,000–17,000 retired IDs the dummy. With the key off, 29,767 of its 33,024 IDs had a texture.
  • Images: no wrong or missing textures in the forest screenshots (cutscenes, dialogue, free roam, the pond).
  • Frames: p50 is about 50 ms leaving the house either way. With the key, the forest settled to 34 ms p50 (p95 42) in the last minutes. I have no steady forest run without it, so I can't confirm the 44 → 31 ms.

Caveats:

Drafted with Claude; reviewed by me.

gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
…bles give a finalized texture's ID the dummy instead of making its texture again (Ocarina of Time out of GPU memory at a scene change)

Its retired-ID check goes into pool_entry, which the table patch and the full build share here (the patch passes no counter); Ocarina of Time's profile turns the key on, as upstream f3ac5ea does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
@david-on-spring
david-on-spring marked this pull request as ready for review November 6, 2026 02:30
@PaulEins

PaulEins commented Nov 6, 2026

Copy link
Copy Markdown

Tested on a real device, and it fixes the stutter on entering Kokiri Village / Forest for me.

Setup: MacBook Pro, M4 Pro (14-core CPU / 20-core GPU), 48 GB, macOS 26.7.2, update-only NUP v131072, Mac app. Tree: stan69b's oot-clean + #1332, #1343, #1350, #1376, #1390, #1408, #1409, #1417, #1420, #1430 and current main. This PR didn't merge there (its pool walk is pool_entry now), so I ported the same check into pool_entry: a retired ID without a view gets the dummy, behind DM_NVN_POOL_SKIP_RETIRED.

Without it, entering Kokiri Village gave 3–5 s at roughly 10–17 fps. In that 300-frame window:

  • packaged texture first uploads went from 26,594 to 198,561 (about 172,000, ~570 per frame)
  • residency set: 514,901 additions / 157,668 MB, 1,228 commits taking 4,948 ms
  • finalized textures still referenced 512 (small table, 188905 put in total)
  • frame time p50 58.4 ms, of it submit+present 58.3 ms; GPU only 20.9 ms

A second, smaller burst came on the way into the forest (135,795 additions, 2,224 ms of commits). stan69b's B-6 fix was already in, so these were CPU-written (packaged) textures being made again on every walk, not new private ones.

With DM_NVN_POOL_SKIP_RETIRED=1: no such window over two runs (~15,000 frames). The worst 300-frame window is 36–38 fps, which is GPU-bound (24–27 ms). The dummy was handed out about 19 million times in one session, and I saw no missing or black textures in Kokiri Village, Kokiri Forest or Link's house, including after scene changes.

Drafted with Claude; reviewed by me.

gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
…ss over IDs of another type (Ocarina of Time's residency commits 12 times fewer)

Resolved against NXForge's pool table (seedds#1360's patching, seedds#1393's pool_skip_retired):
- pool_entry passes over an ID whose registration resolved to another texture type (treg_t.tt) after the gone check, so both the full walk and the patch path skip it
- the full build and the patch path take their buffers from the spares (pool_buf) and retire the replaced one (pool_buf_retire); pool_snap keeps its sub field
- the pool table log line keeps NXForge's why/patched/kept counts and adds made/reused/passed over
- t_int_float_view.m keeps main's checks; the Makefile has t_gmem_raw and t_pool_table
- t_pool_table marks the 2D table out of date before its second rebuild, since a patch on registrations alone would not walk the 3D ID

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 6, 2026
) replaces the pull request heads

Upstream landed Ocarina of Time in main (chain adec7d81d970: "holds 60 fps in
Kokiri Forest, 4 GB less") and closed the pull requests it folded in, with
reworded and partly different code. Mixing a pull request head with its landed
version hunk by hunk silently broke things (two GlobalToStorage tables, a
duplicate AF2ip, two mtk_pso definitions), so the rule here is one version per
feature:

- Pull requests upstream closed (seedds#1299, seedds#1300, seedds#1312, seedds#1343, seedds#1345, seedds#1353,
  seedds#1354, seedds#1359, seedds#1360, seedds#1373, seedds#1428, seedds#1430, seedds#1434): their hunks are taken out
  of our side before the three-way merge, so upstream's landed version stands
  alone. Gone with them: bindless pool table patching (reg_bump), the ghost
  table's per-submission hold, the growing raw pool table and its write-log
  sizing, seedds#1428's pool_buf spares (upstream's pool_table_take is the same
  idea), the ixc-based indirect mesh draw (upstream uses mdx), and their tests
  (t_pool_table, t_mesh_indirect, t_gmem_raw, finalized_bindless_pool,
  test_bnsh_nvn2, test_ndve).
- Open pull requests that upstream's landing supersedes: seedds#1394 (upstream's
  mtl_pool_addressed) and seedds#1393 pool_skip_retired (upstream's budgeted pool
  walk; the profile does not use it) are dropped with their tests and docs.
  seedds#1368's mesh pipeline prewarm is dropped too: upstream kept the background
  build in nvn_geometry.inc, and the prewarm's mtk_build / mtk_claim /
  g_warm_mps went with it.
- Kept, as upstream does not have them: NXForge's own features (GPU watchdog,
  present filters and anti-aliasing, aniso floor, quick settings, pipeline
  archive compaction, upload breakdown, memory report lines) and the still
  open pull requests seedds#1304, seedds#1305, seedds#1308, seedds#1320, seedds#1331, seedds#1332, seedds#1355, seedds#1356,
  seedds#1369, seedds#1376, seedds#1377, seedds#1379, seedds#1382, seedds#1390, seedds#1391, seedds#1392, seedds#1414, seedds#1420,
  seedds#1436, seedds#1439, seedds#1446, seedds#1453.
- Combined where both sides changed the same lines: GW_N 1 << 16 with seedds#1320's
  wide-write intervals (GW_PAGES = SBW_WPAGES, 128 slots), the CPU upload log
  with upstream's linear upload route, arch_open_end before the compaction
  limit, cbn_prog in geo_gs_one (seedds#1382).
- Translator: upstream's sources and binaries, plus seedds#1414's
  InstEmitAmpereMem diff and the LDSM SubgroupBarrier in
  InstEmitAmpereTensor.cs. The binaries and dmsh_chain.lock are upstream's
  until the Mac rebuilds them (dotnet publish, tool_lock --update-lock,
  dmsh_chain --header); dmsh_chain.h is regenerated for now.
- RED_ONCE keys: upstream took 153 and 154, so the present anti-aliasing lines
  move to 156 and 157 and seedds#1420's packaged-entry line to 158 (array 159).
- Host: hle_net.c and t_net_eventfd.c are upstream's (its eventfd poll waits
  out the whole timeout, which our 5d7920c fixed another way).
- Kept ours: dm/tools/romfs_image.py and its test, so the 26 GB RomFS image is
  not packed again.

dm/tools tests on Linux: same failures as on 875449e (Linux-only quirks),
plus test_nlg4 and test_dmsh_pool_arrays, which fail on upstream main too.
Not yet built on the Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
cfryerdev pushed a commit to cfryerdev/nxnative that referenced this pull request Nov 8, 2026
…ures (seedds#1404)" and "Retire obsolete NVN chunks after whole-pool mapping (seedds#1417)"

They tank Ocarina of Time's submit thread: on the same route and save,
the host with them ran 12.4 fps with the submit thread at 56-78 ms CPU
(73-78 ms wall), and without them 58.6-59.1 fps with the GPU at 12 ms.
seedds#1393's pool_skip_retired stays (it is not implicated, and it fixes the
scene-change OOM); seedds#1394's bitmap stays reverted in favour of upstream's
address table.
AmateurWizrd pushed a commit to AmateurWizrd/nxnative that referenced this pull request Nov 10, 2026
… ID the dummy

Net change of PR seedds#1393 at 3a534ed (seedds/nxnative), ported into pool_entry (this tree's walk); the log line keeps this tree's form. Off unless the key or DM_NVN_POOL_SKIP_RETIRED=1 is set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants