Repository navigation
WIP: DM_NVN_TEX_DEDUPE gives back the guest copy of uploaded textures (Ocarina of Time: about 1.9 GB less footprint) - #1386
Draft
david-on-spring wants to merge 7 commits into
Draft
david-on-spring wants to merge 7 commits into
david-on-spring wants to merge 7 commits into
Conversation
…een: eventfds served offline, no virtual audio inputs, the movie library's NDVE shader pack - hle_net.c serves nn::socket eventfds (a local counter, no network): gRPC in Nintendo's online library needs one for its poller and asserted at boot when every socket call failed. Poll / Select / Fcntl / Close handle them; errors set the socket code and the guest's errno. t_net_eventfd checks it. - hle_audio.c answers ListVirtualInputDevices with none: SDK 22's client connected to auddmg:u itself and aborted. - shader_split.py reads Movie/Ndve/*.dat (NDVE): raw NVN2 programs and the ones its NDSR record table names, the decoder's colour conversion among them; the prefilter lets the pack through. test_ndve checks it. - build_ipa_from_nsp.py surveys a dump folder with no profile from its own ExeFS and RomFS (a NUP still needs a profile). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ditional clears (render_cond), NVN2 task and mesh shaders in BNSH slots 7 and 8, the title movie at 60 fps, and the first 3D scene's residency - Profile 0400486024DEC000 with its note and games-table row. render_cond on: agl clears the copy of the prepass depth again under a condition the console skips, and run unconditionally that clear left the main opaque pass's EQUAL test failing everywhere, so the scene came out white behind the full-screen sky. - shader_split.py reads NVN2 BNSH program slots 7 (task) and 8 (mesh). The six NVN slots alone dropped every mesh-shaded model (the door of Link's house among them); OoT's map gains 491 mesh and 17 task shaders. test_bnsh_nvn2.py. - Raw global memory: the write log is a bitmap by memory pool entry, and the pool lookup takes a per-chunk hint, so the movie library's full-frame NDVE shaders no longer overflow the log and stall the GPU (title movie 47.6 ms to 16.67 ms median on an Apple M5). The chain moves to d1f10abd26a7 (with Star Fox's change from main). - A bindless pool walk no longer re-creates finalized textures within a submission (the first 3D scene ran out of Metal memory); an asynchronous play report send completes offline. - The pass dump names a bound compute program's shader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ptors on shared textures out of the ghost ring: Ocarina of Time's Kokiri Forest from a 46-49 ms frame to 16-21 ms - pool_gpu built the whole table of a handle-indexed texture pool (33,025 IDs in Ocarina of Time) whenever a registration or a storage change had happened, and the game's threads register descriptors while the replay draws: about 94,000 builds, 55 s of a 90 s run. Registrations (reg_bump) and noted storage changes (ser_note) are now logged by generation, and a table takes again only the IDs they name: kept when nothing changes, else copied and patched, so draws already encoded read their own. Other bumps, sparse textures, a resized pool or a ring overrun still build it whole. - About 12,000 of its IDs hold descriptors of finalized texture objects, each at the storage of a live texture. They went through the 512-entry ghost ring, about 10,000 puts and twice that in storage notes a frame, and their views were made again in every submission. A finalized serial over a texture the sharing table holds now resolves to it with no ghost entry, and a retired ID keeps its view of it across submissions until the texture leaves the table (alias_sweep). - The pool line counts builds by cause and tables patched or kept; the pass dump names a bound compute shader. t_upgrade_alias checks the finalized serial's resolution rather than its ghost entry. Submit went from 45 ms to 5-8 ms a frame; the GPU (14-21 ms) is now the limit. Resident memory holds at 13-14.3 GB over 160 s. Host and graphics suites pass; regress.py was not run (no prepared Mac builds on this machine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory or aborting Ocarina of Time (0400486024DEC000, v131072) now plays in Kokiri Forest on a 16 GB M4. A 25-minute run at profile defaults, walking from Link's house into the village, had no GPU out-of-memory error, hang or abort, at about 31 ms a frame. Fixes: - Pool tables give a retired texture ID without a view the dummy texture (profile key pool_skip_retired, on for this game; DM_NVN_POOL_SKIP_RETIRED overrides). Rebuilding the table remade 2.5 GB of finalized textures at a scene change, taking GPU allocation past Metal's working set. - The alias region is 8 GB and the reserve grows to match. The game's streaming allocator spreads about 5,100 regions up to +4.95 GB, and with 2.38 GB AllocateAddressRegion failed, an nn::diag abort 5-15 minutes outdoors. - The raw global memory table has 8,191 entries, since 4,095 hung the GPU. It leaves out pools without a buffer address (DM_NVN_GMEM_ALL lists them). The texture storage check no longer wires whole pools. Texture-only uploads use transient buffers. Unreferenced shared textures are swept (DM_NVN_ALIAS_SWEEP). - nvnCommandBufferSetComputeWorkloadPriority stub. PR 1299's duplicate audio hook is removed. DM_MOVIE_SKIP skips movies. - The GPU out-of-memory report and DM_CTL restop list resident textures by holder. Includes the open upstream PRs 1300, 1319, 1320, 1329, 1331, 1340, 1343, 1315, 1345, 1353, 1357, 1359 and 1368, merged locally. 1360 is not included: it builds on main's page-hash write log, and this base's entry-bitmap log is sized to the fixed table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A texture the CPU wrote is held twice: swizzled in its guest memory pool, and deswizzled in its private Metal texture. The GPU only ever reads the second. With DM_NVN_TEX_DEDUPE=1 (off by default), the host gives the first copy back. Once such a texture has gone DM_NVN_TEX_DEDUPE_AGE_MS (default 2000) without a write, its write-protected, unwritten host pages are released: - they are written to an unlinked file (DM_NVN_TEX_DEDUPE_DIR, else TMPDIR); - they are given back with MADV_FREE_REUSABLE; - they are made PROT_NONE. Any later access by the game or the host puts the same bytes back first, through the dirty-page tracker (nvn_dirty.c). So does a buffer over them going into the residency set (pool_res, gmem_map), a large pool's chunk buffer being made (big_new: Metal refuses a zero-copy buffer over PROT_NONE pages), or an upload reading them. Only pages of pools outside the residency set are released, and uploads read through a transient buffer. DM_NVN_TEX_BREAKDOWN=1 logs host texture memory by kind, with the guest storage under each. A new test, t_dirty_dedupe, covers release and restore, including a second thread faulting on a page being restored. Ocarina of Time on a 16 GB M4, from Link's house out into Kokiri Forest: - about 1.9 GB of guest pages released, from 4,669 textures; - peak footprint 11.5 GB (about 2 GB less in the house in earlier A/B runs); - the picture is unchanged, and GPU memory is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions
Bot
added
host
gnosticgeek
pushed a commit
to gnosticgeek/nxforge
that referenced
this pull request
Nov 5, 2026
From upstream f3ac5ea (the Ocarina integration that seedds/nxnative draft PR seedds#1386 sits on): the game's streaming allocator keeps about 5,100 address regions in the alias region, spread up to +4.95 GB, and with the profile's 2.38 GB VammManager::AllocateAddressRegion failed after 5-15 minutes outdoors (nn::diag abort). mem_alias goes to 8 GB and mem_reserve grows by the same 5.6 GB so the heap keeps its size. Profile values only; the rest of that commit (pool tables skipping retired IDs, a 8,191-entry raw global table instead of seedds#1360's growing one, translator changes) waits for its own upstream pull requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
gnosticgeek
pushed a commit
to gnosticgeek/nxforge
that referenced
this pull request
Nov 5, 2026
… one NXForge has (seedds#1386 on f3ac5ea) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
This was referenced Nov 6, 2026
Contributor
|
A second 16 GB M4 on this branch (Mac mini, macOS 27.2, 562c1f4, chain 84aede056dea), docked, with the system-side numbers. Four runs in the order off, on, on, off ( Over the 5 minutes after the load:
Both runs with the switch ended almost identically: 2,859 and 2,860 textures, 1,416 MB given back in each, pages put back 797, read back from the file 691, kept with a racing write 0, failed 0. In this scene swap hardly moved in any run, so this says nothing about swap. |
2 tasks done
2 of 3 tasks
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, work in progress. Asked for in #1347. Refs #1281.
Stacked: this branch sits on my local Ocarina of Time integration, because the prototype was built and tested there. Review only the top commit,
562c1f4(7 files, about +500 lines).The commit below it,
f3ac5ea, is #1299 + #1312 plus these other open PRs merged locally: #1300, #1315, #1319, #1320, #1329, #1331, #1340, #1343, #1345, #1353, #1357, #1359 and #1368. It also has my own fixes from #1347. Until those land, the diff against main shows all of them.What and why
A texture the CPU wrote is held twice:
The GPU only ever reads the Metal one. With
DM_NVN_TEX_DEDUPE=1(off by default), the host gives the guest copy back.Release. Once an uploaded texture has gone
DM_NVN_TEX_DEDUPE_AGE_MS(default 2000) without a write, its host pages that the dirty tracker still has write-protected and unwritten are:DM_NVN_TEX_DEDUPE_DIR, elseTMPDIR);MADV_FREE_REUSABLE, which takes them out ofphys_footprintat once;PROT_NONE.Restore. Any access by the game or the host faults and gets the same bytes back first (
nvn_dirty.c). So do these:pool_res,gmem_map);big_new), because Metal refuses a zero-copy buffer overPROT_NONEpages;Scope. Only pages of pools outside the residency set are released. Uploads read through a transient buffer, so they don't wire pools. With
DM_GUEST_FILE, the switch turns itself off.DM_NVN_TEX_BREAKDOWN=1logs host texture memory by kind, with the guest storage under each.Measured
Ocarina of Time (
0400486024DEC000v131072) on a 16 GB M4, from Link's house out into Kokiri Forest. Same host binary, switch off (A) and on (B), six runs of 8 minutes alternated A B A B A B:House and forest values are memstat medians (20 s before leaving the house; the last 5 minutes outdoors).
¹ B1 missed the front door under load and stayed in the house.
² Run under heavy CPU load from parallel shader builds, so compare A1 with B3 (quiet) and A2/A3 with B2.
Tests
t_dirty_dedupe: release, restore on read, on write and onnvn_dirty_restore. It also covers a second thread faulting on a page being restored; that race was found and fixed.t_dirty_race,t_dirty_retire_race,t_dirty_slotsandt_nvnpass,t_nvnalso with the switch on.Open
EFAULTinstead of the bytes. Nothing I know of does that.Drafted with Claude; reviewed by me.
🤖 Generated with Claude Code