Repository navigation
Conversation
…inds no longer leaves its CPU-written textures at zero Kokiri Forest's ground in Ocarina of Time was missing wherever only the terrain covers it: aCidcOde traced it in seedds#1326 to the ten 260x260 R32F height tiles, all zeros on the host with data in their storage, as tex_cpu_content refused any uncompressed texture in a pool with gpu_written set, and the terrain's 202 MB pool (flags 0x24) also holds textures the GPU writes. Each tile lay flat far below the player and the sky showed through. mtl_tex_gpu_written now records the storage range of each render target, clear, copy destination and image store, and an uncompressed texture is refused, or dropped by cpu_unc_drop, only where its own storage overlaps a range. This is aCidcOde's patch from seedds#1326, checked against this tree, with one change: ranges go with their pool (gpuw_drop). They never shrank in the patch, so a pool made later at those addresses would have counted as written, where pool_t.gpu_written starts at 0. t_cpu_tex: a texture beside a GPU write is taken and one over it is not, a write over its storage ends that, ranges merge and split, a pool that goes takes them along, no range means the whole pool. t_cpu_storage passes unchanged but for the name.
…inds no longer leaves its CPU-written textures at zero (upstream PR seedds#1390)
…es, so a pool that holds both kinds no longer leaves its CPU-written textures at zero Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g # Conflicts: # dm/gfx/tests/t_cpu_tex.m
|
Played on a 16 GB Mac mini M4, macOS 27.2, docked, v131072. Host only: david-on-spring's #1386 branch (562c1f4 on f3ac5ea, chain 84aede056dea) + #1396 + #1430 + #1367, run without and with this PR (16e86fd cherry-picked; the code applied cleanly, and The same save was put back before each run, at a spot in Kokiri Forest where the ground was missing. I didn't touch the controller after the load, and
At the same frames (1500, 2400, 3300), the two runs differ by a mean of 9.7–10.2 levels, with 11.1–11.6% of pixels off by more than 32. Within one run, shots 300 frames apart differ by 4.3–5.2 (2.0–2.5%). Frame time p50 was 20.4–20.7 ms in both runs. GPU p50 was 37.2–37.5 ms without and 36.1–36.2 ms with. Both logged only the title movie's AV1 RED and exited cleanly. Tests on this stack: |
…itch 2) playable at 60 fps through Dodongo's Cavern Resolved against this tree's own Ocarina work: - Kept this tree's raw pool table (grows past 4,096 entries, two-word hints, per-entry write logs); the PR's merged-run table (GMEM_RUN) and one-word hints are left out, with the ABI and translator to match. - Kept the pool table spares (seedds#1428), type cache, range-tracked GPU writes (seedds#1390), 1430's depth-read rule, the synchronous task / mesh pipeline build, and the item walk of NDVE packs. - Taken from the PR: ALIAS_IDLE 512 and targeted storage notes on takeover, packaged-texture recycling (pkg_tex_put), cached zero-copy upload sources (texture-only pools keep a per-batch buffer), retired CPU-written IDs without a view get no texture, mesh pipelines without a vertex function, image binding upgrades with pixel-format views, F2FP/I2FP and other translator fixes not already here (its second F2IP dropped), loop_guard and tg_fence, the GPU watchdog, the eventfd poll timeout, the profile (16 GiB alias region, mem_map_limit, tex_flush_upload, tex_dirty_upload) and its shader tables. - Tests adapted: t_alias_takeover (textures in a pool), t_cpu_tex (retired descriptors), t_res_guard (512-submission idle window), t_upgrade_alias (ghost entry), t_pool_chunks (contiguous pools in a growing table), t_present_alloc (layer sized to its pictures after upstream's crop change), t_hid (poolstat / texstat stubs). - Translator rebuilt from the merged source; chain fe655346ea1c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ack the guest copy of uploaded textures (opt-in); a page on playing Ocarina of Time. Commit 1 (GPU-written ranges) is already here from seedds#1390 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mtl_tex_gpu_written), tests follow pool_skip_retired, the buffer-address pool table and the new present / DM_CTL calls - nvn_mtl.m ndve_scale called mtl_pool_gpu_written, which seedds#1390 replaced with mtl_tex_gpu_written (a range): it did not compile - t_cpu_tex pins pool_skip_retired off: Ocarina of Time's profile turns it on, and its retired-descriptor checks want textures made again (t_pool_retired covers on) - t_pool_chunks: the 5,000-pool table check takes a buffer address in each pool, since seedds#1394 puts only those in the table - host tests: stubs for mtl_present_layer (t_vi_display), mtl_present_screen (t_nro, t_symbols) and DM_CTL poolstat / texstat (t_hid) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
t_cpu_tex.m: main's new checks (a finalized copy, a copy outliving its object) and this branch's gpu_written_ranges() both stay. Checked: check_t_cpu_tex (68 checks, 0 failed), check_t_cpu_storage (32), test_docs.py, with GAME=01007EF00011E000.
) replaces the pull request heads Upstream landed Ocarina of Time in main (chain adec7d81d970: "holds 60 fps in Kokiri Forest, 4 GB less") and closed the pull requests it folded in, with reworded and partly different code. Mixing a pull request head with its landed version hunk by hunk silently broke things (two GlobalToStorage tables, a duplicate AF2ip, two mtk_pso definitions), so the rule here is one version per feature: - Pull requests upstream closed (seedds#1299, seedds#1300, seedds#1312, seedds#1343, seedds#1345, seedds#1353, seedds#1354, seedds#1359, seedds#1360, seedds#1373, seedds#1428, seedds#1430, seedds#1434): their hunks are taken out of our side before the three-way merge, so upstream's landed version stands alone. Gone with them: bindless pool table patching (reg_bump), the ghost table's per-submission hold, the growing raw pool table and its write-log sizing, seedds#1428's pool_buf spares (upstream's pool_table_take is the same idea), the ixc-based indirect mesh draw (upstream uses mdx), and their tests (t_pool_table, t_mesh_indirect, t_gmem_raw, finalized_bindless_pool, test_bnsh_nvn2, test_ndve). - Open pull requests that upstream's landing supersedes: seedds#1394 (upstream's mtl_pool_addressed) and seedds#1393 pool_skip_retired (upstream's budgeted pool walk; the profile does not use it) are dropped with their tests and docs. seedds#1368's mesh pipeline prewarm is dropped too: upstream kept the background build in nvn_geometry.inc, and the prewarm's mtk_build / mtk_claim / g_warm_mps went with it. - Kept, as upstream does not have them: NXForge's own features (GPU watchdog, present filters and anti-aliasing, aniso floor, quick settings, pipeline archive compaction, upload breakdown, memory report lines) and the still open pull requests seedds#1304, seedds#1305, seedds#1308, seedds#1320, seedds#1331, seedds#1332, seedds#1355, seedds#1356, seedds#1369, seedds#1376, seedds#1377, seedds#1379, seedds#1382, seedds#1390, seedds#1391, seedds#1392, seedds#1414, seedds#1420, seedds#1436, seedds#1439, seedds#1446, seedds#1453. - Combined where both sides changed the same lines: GW_N 1 << 16 with seedds#1320's wide-write intervals (GW_PAGES = SBW_WPAGES, 128 slots), the CPU upload log with upstream's linear upload route, arch_open_end before the compaction limit, cbn_prog in geo_gs_one (seedds#1382). - Translator: upstream's sources and binaries, plus seedds#1414's InstEmitAmpereMem diff and the LDSM SubgroupBarrier in InstEmitAmpereTensor.cs. The binaries and dmsh_chain.lock are upstream's until the Mac rebuilds them (dotnet publish, tool_lock --update-lock, dmsh_chain --header); dmsh_chain.h is regenerated for now. - RED_ONCE keys: upstream took 153 and 154, so the present anti-aliasing lines move to 156 and 157 and seedds#1420's packaged-entry line to 158 (array 159). - Host: hle_net.c and t_net_eventfd.c are upstream's (its eventfd poll waits out the whole timeout, which our 5d7920c fixed another way). - Kept ours: dm/tools/romfs_image.py and its test, so the 26 GB RomFS image is not packed again. dm/tools tests on Linux: same failures as on 875449e (Linux-only quirks), plus test_nlg4 and test_dmsh_pool_arrays, which fail on upstream main too. Not yet built on the Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qteK29T18U8p1c9LsVY8g
…atedSize currentAllocatedSize counts each no-copy buffer in full however many cover the same pages: 50 over one 64 MB region add 3,200 MB. In Ocarina of Time's Kokiri Forest on a 24 GB Mac it read 17.3 GB within a minute or a few, with 6.5 GB in the residency set and the driver's In use system memory near 7.7 GB, so the guard stopped the game each time. With its stop raised past Metal's working set, the refused commit was followed by InnocentVictim command buffer errors. The guard now reads the set's own size, so an addition counts from its commit. A large burst is seen one commit later than before, and the timer sees the size at the last commit. Checked: t_res_guard, both runs. Ocarina of Time v131072 on an M6 Mac with 24 GB, on main with seedds#1357, seedds#1390, seedds#1391, seedds#1453 and seedds#1455: about 49 minutes and 170,400 frames through Kokiri Forest and the Great Deku Tree, currentAllocatedSize at about 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Not checked: other games, a smaller Mac.
# Conflicts: # dm/gfx/tests/t_cpu_tex.m
…er and shader chain, the branch's host services, movie clock, packing and Ocarina of Time additions Upstream reached about 60 fps in Kokiri Forest with its own versions of most of the branch's merged PRs, so where both changed the same code upstream's is kept: - The shader chain is upstream's (translator, lock, dmsh tools, Ryujinx sources), plus the branch's bracket-depth flattening (dmsh_depth.py, test_dmsh_depth.py); chain c732699d4fdb. Not carried: the LDSM subgroup fence (seedds#1376, needs a translator rebuild; upstream's HMMA/LDSM rework reports clean movies) and the shared raw-memory lookups (seedds#1405, superseded by upstream's last-entry lookups). - The renderer (dm/gfx/nvn, dm/gfx/msl) and its tests are upstream's. Not carried: seedds#1390-seedds#1394, seedds#1404, seedds#1408, seedds#1409, seedds#1417, seedds#1418, seedds#1420, seedds#1427, seedds#1320, seedds#1331, seedds#1332, seedds#1368, seedds#1285 renderer parts, pool_skip_retired, and the branch's warning fixes and prewarm progress there. They stay on oot-integration. - Ocarina of Time's profile is upstream's, with the branch's 12 movie compute rows in pso_manifest.tsv, PLAN.md and PR-AUDIT.md. - shader_split.py is upstream's plus the BAHS record check (test_split_bahs_records.py). - Host: upstream's EventFd, PlayReport send, audio lock and vibration answers; the branch's service sessions (hle_sf, hle_prepo), sync waits, AV1 movie clock and parallel decode, diag_abort frame fix and focus control are kept. - Scripts: shader.noindex, tmp.noindex, notes outside the shader key, bench and regress changes. Checked: the Mac host builds (GAME=0400486024DEC000); host unit tests pass; the Python tool tests pass on Python 3.13 except test_dmsh_pool_arrays, test_nro_pack, test_nsp_dump and test_nsp_incremental, which fail on upstream main as well. Not run: the full shader build on this chain, gameplay.
# Conflicts: # dm/gfx/nvn/hle_nvn.c
|
Superseded by e2b76d4 (from #1604), which keeps GPU-written storage as address ranges, as this PR does. It also protects each range's pages, so storage the CPU rewrites after a GPU write is taken from storage again. On Ocarina of Time's walk, 10 of 20 such reuses were terrain maps, which ranges kept until their pool goes would have left as zeros. Thanks to you and to aCidcOde: the reference entry credits #1326 for finding the pool rule's part. Closing this one, so that one range table lands. |
|
Still needed on current main ( Drafted with Claude; reviewed by me. |
…atedSize currentAllocatedSize counts each no-copy buffer in full however many cover the same pages: 50 over one 64 MB region add 3,200 MB. In Ocarina of Time's Kokiri Forest on a 24 GB Mac it read 17.3 GB within a minute or a few, with 6.5 GB in the residency set and the driver's In use system memory near 7.7 GB, so the guard stopped the game each time. With its stop raised past Metal's working set, the refused commit was followed by InnocentVictim command buffer errors. The guard now reads the set's own size, so an addition counts from its commit. A large burst is seen one commit later than before, and the timer sees the size at the last commit. Checked: t_res_guard, both runs. Ocarina of Time v131072 on an M6 Mac with 24 GB, on main with seedds#1357, seedds#1390, seedds#1391, seedds#1453 and seedds#1455: about 49 minutes and 170,400 frames through Kokiri Forest and the Great Deku Tree, currentAllocatedSize at about 20 GB, driver in use at most 9.5 GB, wired about 10 GB, no GPU errors. Not checked: other games, a smaller Mac.
…s first write stays resident while any of them holds it: releasing one took it out of the residency set (GPU page faults, issue seedds#1326) The serials of one storage share one packaged texture (g_pkg_share). At the GPU's first write over that storage, cpu_unc_drop gives each of them the package's texture itself (tex_put), so they all hold one texture that was in no shared set. The first holder released (tex_put's ghost-ring eviction, serial_put or serial_del, through obj_drop or defer_obj_release), or the package's own release (NvPkg dealloc's res_remove), took it out of the residency set while the other serials still drew with it, and a command buffer reading it faulted. pkg_hand_over enters the handed-over texture in g_alias_set and in g_handed, a list by the package's sharing key, so releasing one holder only drops a reference (alias_has). The package's release and its re-upload's spare leave a handed-over texture resident, mtl_alias_drop_pool retires it with its pool, and tex_upgrade moves its entry. It does not enter g_alias, so a new texture object over the same storage still gets a texture of its own. DM_UNC_SHARE=0 gives back the old hand-over. Ocarina of Time v131072, Mac mini M4 16 GB, macOS 27.2, on main b2d46b8 with seedds#1590, seedds#1557, seedds#1556, seedds#1575, seedds#1522, seedds#1543 and seedds#1390: page faults at 44, 29, 24 and 15.5 minutes, with the missing textures named by DM_RES_CHECK and a diagnostic of where each left the residency set, before two of them, as 260x260 terrain tiles handed over this way and released by the ghost-ring eviction. One binary: with the change, 107 minutes and no fault (86 hand-overs); with DM_UNC_SHARE=0, a fault at 37 minutes with the same kind of tiles named. t_cpu_tex --metal handover_shared: 97 checks, 0 failed; with DM_UNC_SHARE=0, 2 failed. The other GPU suites are unchanged. (cherry picked from commit 8cbc330)

What and why
Ocarina of Time (Switch 2,
0400486024DEC000, profile in #1299): in Kokiri Forest the ground is missing wherever only the terrain covers it, the sky shows through, and from above the stone path floats. @aCidcOde traced it in #1326: the ten 260x260 R32F height tiles read as all zeros on the host while their storage holds data (DM_NVN_TEXINFOshows 141,674 to 204,571 non-zero bytes), so each terrain patch is a flat plane far below the player.Cause:
tex_cpu_contentrefuses an uncompressed texture when its pool hasgpu_writtenset. The terrain's 202 MB pool (flags 0x24) also holds textures the GPU writes, so the whole pool counts as written. It is the Mario Kart World terrain case described abovetex_cpu_content, in a pool the GPU writes too.Change:
mtl_tex_gpu_writtenrecords the storage range of each render target, clear, copy destination and image store (sorted, merged ranges). An uncompressed texture is refused, or dropped bycpu_unc_drop, only where its own storage overlaps a range.mtl_pool_gpu_writtenstays, as "the whole pool".This is aCidcOde's patch from #1326 (credit: @aCidcOde), checked against this tree, with one change: ranges go with their pool (
gpuw_drop). In the patch they never shrank, so a pool made later at those addresses would have counted as written, wherepool_t.gpu_writtenstarts at 0.Checked
make -C dm/gfx/tests GAME=010015100B514000 ...):check_t_cpu_tex65 checks, 0 failed (new: a texture beside a GPU write is taken and one over it is not; a write over its storage ends that; ranges merge and split; a pool that goes takes them along; no range means the whole pool);check_t_cpu_storage32 checks, 0 failed (unchanged but for the function's name);test_docs.pyOK;docs_symptoms.py --checkcurrent;lint_c.py tidyonnvn_mtl.m,hle_nvn.c,t_cpu_tex.m: 0 pairs above the baseline.fafad0e6(stack-next, Ocarina of Time v131072 on an M5 Pro), whose commit message records no in-game measurement of its own.make -C dm/gfx/tests run, the host tests,regress.py. The change touches every game that records GPU writes, but only through when a CPU-written uncompressed texture is taken from its storage; Mario Kart World (0400C3F00006E000), whose note describes the earlier pool rule, was not run.Docs
docs/reference/nvn-objects.md("Uncompressed storage the CPU wrote, where the GPU has not written", renamed from "...in a pool the GPU has not written": the two links to it in the Mario Kart World note and the symptom index follow) andpython3 dm/tools/docs_symptoms.pywas run