Repository navigation
The Legend of Zelda: Ocarina of Time (Switch 2, 0400486024DEC000, v0 & v131072): playable at 60 fps through Dodongo's Cavern - #1444
ksolenk38ze wants to merge 2 commits into
Conversation
github-actions
Bot
added
host
|
Building this branch (e03bdc8) from my own v131072 NUP on an M4 MacBook with 24 GB to test it on a smaller Mac than the M5 Pro. I'll report how far it gets, plus footprint and GPU-allocated numbers, once the shader build finishes (about 2 hours). Drafted with Claude; reviewed by me. |
|
Test report, part 1: a fresh build of e03bdc8 from the v131072 NUP stops at the shader split: The 42,857 rows are the same; only the order differs. The profile's Drafted with Claude; reviewed by me. |
…itch 2) playable at 60 fps through Dodongo's Cavern Resolved against this tree's own Ocarina work: - Kept this tree's raw pool table (grows past 4,096 entries, two-word hints, per-entry write logs); the PR's merged-run table (GMEM_RUN) and one-word hints are left out, with the ABI and translator to match. - Kept the pool table spares (seedds#1428), type cache, range-tracked GPU writes (seedds#1390), 1430's depth-read rule, the synchronous task / mesh pipeline build, and the item walk of NDVE packs. - Taken from the PR: ALIAS_IDLE 512 and targeted storage notes on takeover, packaged-texture recycling (pkg_tex_put), cached zero-copy upload sources (texture-only pools keep a per-batch buffer), retired CPU-written IDs without a view get no texture, mesh pipelines without a vertex function, image binding upgrades with pixel-format views, F2FP/I2FP and other translator fixes not already here (its second F2IP dropped), loop_guard and tg_fence, the GPU watchdog, the eventfd poll timeout, the profile (16 GiB alias region, mem_map_limit, tex_flush_upload, tex_dirty_upload) and its shader tables. - Tests adapted: t_alias_takeover (textures in a pool), t_cpu_tex (retired descriptors), t_res_guard (512-submission idle window), t_upgrade_alias (ghost entry), t_pool_chunks (contiguous pools in a growing table), t_present_alloc (layer sized to its pictures after upstream's crop change), t_hid (poolstat / texstat stubs). - Translator rebuilt from the merged source; chain fe655346ea1c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n unset offset and dropped agl's shaders Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mmon match 68bbe98b3bfd9876 this tree's bahs_shaders leaves out (seedds#1444's tables had it) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ead of a host crash nvnMemoryPoolGetFlags, nvnMemoryPoolMap, nvnMemoryPoolGetSize, nvnBufferGetAddress, nvnBufferMap, nvnBufferGetSize, nvnBufferGetMemoryPool and nvnBufferGetMemoryOffset read through their argument without a check. When the game's allocator gives nvnBufferInitialize no pool (Ocarina of Time on entering Hyrule Field: `BufferInitialize: pool has no storage (pool 0x0 ...)`), the next getter call on the null pool crashed the host (SIGSEGV at 0x4 in h_nvnMemoryPoolGetFlags). They now answer 0 or NULL. This only moves the failure: what made the game's allocator return no pool is not known. From seedds#1444 (hle_nvn.c); the reference entry notes the case. Checked: make -C dm/host GAME=040074A01BF12000 builds; test_docs.py and docs_symptoms.py --check pass. Not played.
…it: the same rows, so the NSP script's table check passes (PR seedds#1444 had them in another order)
|
Thank you for this huge amount of work and the 94-minute run across so many areas. The base port has landed on main in b1d7c70 and 6121fd7, and mem_map_limit is on. I'm leaving this open because it covers areas I haven't played and has fixes I haven't checked, such as finalized texture IDs, copies into uploaded textures and hang guards. It would help to split what still reproduces on current main into small PRs. Final checks on the rebased tree (full shader build, walks, regression) are still running, so please report anything you see. |
|
Test report, part 1b: past the shader split, the build finished (42,851 of 42,854 shaders), but the Mac app came out without its pipeline archive: e03bdc8 takes |
…eedds#1444 added a second beside Star Fox's, and the host aborts at load on a duplicate)
|
Test report, part 2: e03bdc8 (with the split's
With both fixed (this branch's copy of the takeover removed, and c83c48f's |
e03bdc8 to
d2aff9f
Compare
|
Hi everyone, and thank you @seedds and @lunks for landing the incremental bindless pool patching ( No worries at all on the PR pace — we are continuing our playthrough right now from Dodongo's Cavern / Hyrule Field up to Adult Link, rebasing cleanly onto We have rebased 1. Indirect mesh draws & GPU page faults / hangs on room transitions (#1326, #1347, #1458)
2. Terrain / rock blackouts and 50.8 GB / 300-frame residency explosion (#1404, #1455)
3. Global memory table overflow (>256 contiguous 64 KB pools, related to #1326)
4. Uncompressed 260×260 / 68×68 terrain tiles in GPU-written pools
5. Blocky / pixelated lava rocks & King Dodongo heat-haze / lava pool shaders
6. Frame-time spikes on entering Kakariko Village, Hyrule Field, Skullwalltula drops & boss cutscenes (#1385)
|
…ds#1444 brought back its own beside upstream's, and the loader stops on a duplicate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Data from main for items 1 and 3. M1 Max, macOS 27.0.1, v131072 Item 3 is already covered on main. Item 1: the early page fault still happens on main, so it's a good test case for the retire hold. In every run on main (9 over the last day, including this one on 0f510fd), it faults once, roughly 15–65 s after the autosave loads: It's always segment 11 or 12. Things it isn't, each tested on main with the fault still there:
One correlation holds in all 9 runs. The line just before the fault (7–17 log lines earlier) is always the first time raw global stores reach one group of 6 pools that were all created at load: The pool age in that line matches the fault time in each run (15 s → early fault, 65 s → late fault). So the fault follows the first raw store into that pool group, not a room change. No uniform shadow lies in those pools, and Your item 1 fix ( |
|
@mbanusic thanks, that is a useful correlation: "9 of 9 with the same line just before it" is the strongest lead on the page fault so far. One request, and it is not proven that it helps: could you run #1427 on the same autosave? It changes when the raw global-memory pool table words are taken (at commit, per pool the host has learned), which is the part that line is about. One run is enough. What would tell us something: whether the fault still comes at segment 11/12, and whether the "stores wrote in N pool(s) the host had not learned" line still precedes it. If it faults the same way, that rules the table words out and we are back to the mechanism you were looking for. |
|
@lunks I ran #1427 once, on the same autosave. Same M1 Max, built as 0f510fd + #1427 (4d7f1d1), chain unchanged (cfcc3a82c78b), no It faults the same way, so the table words are ruled out.
So the fault still follows the first raw store into that group of 6 pools, even with every argument block's table word pointing at the current table at commit. |
|
Hi everyone! Quick mid-day update as we continue our full playthrough (just finished Hyrule Market / Hyrule Castle / Great Fairy and heading into Zora's River & Zora's Domain right now). First, to answer the points raised in the comments above: 1. Replies to @PucaVaz, @yoBoss283, @mbanusic & @lunks
2. Issues encountered after recent merges & how we fixed them todayWhile playing through Kokiri Forest, Great Deku Tree, Hyrule Field, Kakariko, Death Mountain, Dodongo's Cavern, Hyrule Market, and Hyrule Castle on the rebased codebase, we ran into several regressions and edge cases and fixed them along the way:
3. Next Steps (New PR tonight)We are continuing the playthrough right now through Zora's Domain, Lord Jabu-Jabu's Belly, and the transition to Adult Link. Tonight, once today's playtest is complete, we will open a brand-new, cleanly rebased PR containing all of these fixes in atomic commits and post the link right here in #1444 so everyone can follow along! |
|
@ksolenk38ze thanks for the branch. I tested The segment 11 page fault is still there. It came in the same place as before:
This run recovered: there was 1 failed command buffer, the game played on, and it exited cleanly. Runs on main have done the same about 8 times in 11, so recovering doesn't show an improvement. That's 13 of 13 runs with this fault, now including:
So retired textures being released early, and runaway indirect mesh counts, don't look like the cause of the segment 11 fault. It still lines up with the newly learned pools from the stores. Also: on this build the loading video plays with no artifacts. On main 0f510fd and with #1427 it had visible artifacts, so one of the branch's other changes (possibly the |
d2aff9f to
ad4ba5e
Compare
|
Second run on the same build, from a Deku Tree autosave instead of the ladder: about 8.5 minutes in, during a fight, a whole-frame command buffer page-faulted (fence 133248, frame 22202; gpuEvent: BIF0 page fault, On this branch, the new |
|
Thanks @mbanusic and @yoBoss283 for the detailed tests and feedback! We have just rebased both
|
ad4ba5e to
22d616f
Compare
779128b to
77fb043
Compare
… attribute layout, pulling-base archive retention, image-view upgrades and retired alias safety through the Forest Temple Picture: - dmsh_meshstage.py lays out all v_blocks of 32 vertices before p_blocks of 32 per-primitive attributes in mesh stage memory (and honors sph[0x35] bit 3 PointSize and ctl[0x5f] point/line topologies), so the Forest Temple paintings (Poe Sisters and Phantom Ganon, rendered by ee4ef51a4fddc28e and 83c0dbc51c5481e3 with max_v = 128, max_p = 64 and per-primitive brush-stroke colors) read p0 from the primitive block instead of vertex block 1's screen-space positions. - fb_prepare ends the render pass when a draw samples R.rt.depth after an earlier draw in the same pass wrote depth (g_fb_dwrite): translucent geometry that reads the attached depth buffer for soft-depth fade (the Zora's Sapphire pedestal, Jabu-Jabu's interior walls) flickered black blocks on TBDR without a pass boundary. - image_prepare keeps NVN_TEXTURE_FLAGS_IMAGE on g_treg[tid].t.d.flags and upgrades R32Uint format views over RGBA8Unorm_sRGB before the pass (tex_upgrade_pfv), and tex_upgrade keeps an NvImageHeap owner and rebinds g_by_serial, g_ghost and g_alias views (tex_upgrade_alias_rebind) so agl_ndsr_vr does not leave blocky tiles on volcanic rocks or dynamic-resolution cutscenes. - bcrt_attach redirects renders into RGB_DXT1 (0x42) and RGTC1_UNORM (0x49) targets (flags 0x8) to an RGBA8Unorm / R8Unorm shadow texture and encodes them back on rp_end (bc_encode_bc1, bc_encode_bc4) for sky and water reflection baking. - mtk_pad_mesh_fn (nvn_geometry.inc) and var_pairs (shader_split.py) pad missing locnN outputs in mesh shaders (3a6340a04bf09797_b, a203347bb7bdbd03_b) with float4(0, 0, 0, 1) when paired with a fragment shader that reads them. Frame rate, pipeline archives and memory: - ArchEpoch separates vd == 0 (pulling base) and vd != 0 (vertex-layout base) flush periods (ep->kind) so pulling bases always serialize into p%05d.mtl4a instead of %05d.mtl4a and survive pso_compact_dir compaction (pso_prebuild also probes PRE_PROBE pairs against g_pull_arch4 before trusting prebuilt.tsv); pso4_init places compacted shards >= 256 MB at the head of g_arch4 outside g_compact_files and validates _MTL4Archive entry sizes (arch_entries_ok). - shader_prefetch_rest and pso_predict_rest_ copy the OP_CALL return stack (cur.ret) so lookahead continues across sub-command-buffer returns; hybrid_get_ex caches the reusable pulling base (vd == 0) on an uncached miss, and mtk_pso builds task/mesh pipelines on QOS_CLASS_USER_INITIATED. - treg_tex and tex_mtl_ skip re-creating or ghosting an unread retired GPU texture when a live alias already holds its (pool, off) range (alias_overlaps), and alias_take_over indexes g_by_serial by (pool, page) and marks retired g_treg slots over taken storage gone. - pool_gpu's patch scratch arrays (ids, ser_list, vals) are static under g_rmu instead of taking 1.96 MB on the 1 MB dm_bigstack. - Profile 0400486024DEC000: 16 GB mem_alias (0x400000000) for multi-dungeon virtual pool churn, v0 build entry, and updated README note. Checked: test_dmsh_meshstage 28, t_cpu_tex --metal 68, t_upgrade_alias 14, t_feedback 194, t_lookahead_bindless 22, t_pool_notes 81, dm/host/tests, test_docs, test_env_switches, test_game_profile, test_dmsh_pool_arrays, all 0 failed.
…nd Phantom Ganon (40,154 entries)
77fb043 to
4e6d541
Compare
|
Opened #1590 as a clean follow-up rebased on |
… pass wrote it ends the pass first: TBDR read unstored depth, black blocks on Ocarina of Time's depth-softened surfaces (from PR seedds#1444's 349c8006)
What and why
This completes the profile and host/renderer fixes for The Legend of Zelda: Ocarina of Time (Switch 2 Edition, title ID
0400486024DEC000, base dumpv0and update 1.2.0v131072, Nintendo SDK 22.2.8, NVN2), building on the initial Kokiri Forest draft (#1299).On the Mac it boots, plays the 1920×1080 AV1 title movie at 60 fps, and runs continuously at ~60 fps (frame p50 16.29–16.73 ms, GPU p50 10.8–13.9 ms on Apple M5 Pro) across Kokiri Forest, the Great Deku Tree (including the Queen Gohma boss fight and Triforce cutscene), Hyrule Field, Market, Hyrule Castle, Kakariko Village, Death Mountain Trail, the Great Fairy Fountain, and Dodongo's Cavern (including the King Dodongo boss fight and Spiritual Stone cutscene) over a 274,851-frame (94-minute) continuous gameplay session with zero crashes.
Summary of Changes
1. Title screen, audio, movie & Kokiri Forest entry (from #1299)
eventfdcreationhle_net.cservesnn::socket::EventFd/EventFdRead/EventFdWrite(fds 512..575) acrossPoll,Select,CloseandFcntl(t_net_eventfd, 23 checks)auddmg:uabort (0xa99) & SDK 22audrenlocknn::audio::ListVirtualInputDevicesand moves theAudioRenderSystemupdate lock offsethle_audio.canswers with 0 virtual input devices;hle_audren.creads the SDK 22 update lock offset dynamically fromUpdateImplNDVE(zstd dictionary pack) and 3 inShader/agl_ndsr.pck(NDSRv0x102)shader_split.pyextracts bothndve_shadersandndsr_shaders; all 42,853 shaders inshader_map.tsvtranslate and compileprepo:uabort (0xe15) at first scene transitionPlayReport::Sendwaits on anAsyncContexthle_offline.ccompletes the guest SDKAsyncContext(Result)in caller storageaglclears a copy of prepass depth under a render condition skipped on consolerender_cond on#var0) task/mesh shaders had an unbounded2^32loop whenc0_u[8] == 0; 521 contiguous 64 KB pools overflowed the 256-entryGMEM_RUN_MAXtableshader_split.pyreads BNSH slots 7 & 8;loop_guardindmsh_build.pybounds loops at262,144iterations;nvnCommandBufferDrawMeshTasksIndirectis implemented withdpc_clamp;GMEM_RUN_MAXraised to 1024;BarrierExitsandtg_fencehandle divergent barriers and shared-memory fences2. Foliage, water, terrain, memory & scene transitions
F2FP(0x23e/0x83e, mode0x4ffMERGE_AB) andI2FP(0x243,sA != RZ) missing;SULDdead-code elimination dropped coordinate inputs when one component was unused;a5288d0a6c4971fd's 3D texture typed asTexture2DF2FPandI2FPinInstEmitAmpereAlu.cs; keptSULDcoordinate inputs live inInstEmitMemory.cs; recordedTexture3Dinshader_facts.tsv+0x5f == 1;shader_pairs.tsvonly paired(vs, ps), leaving 21 mesh shaders withoutvs_padpadding for unwritten fragment inputs (locn5,locn2..15)dmsh_meshstage.pyreads point topology at+0x5f == 1;shader_split.pyemits the 1,284(ms, ps)BNSH variation pairs intoshader_pairs.tsvOptimizeChangeBanc) & long-sessionAllocateAddressRegionfailuresmem_alias 0x98000000) exhausted virtual space; long sessions mapped/unmapped scene heaps past the 2.80 GiBapp_pool - heapbudget without triggeringsead::StandardAllocatorunmappingmem_alias 0x400000000,mem_reserve 0x668000000, andmem_map_limit on(0x10801ResultLimitReached);hle_nvn.cguardsh_nvnBuffer*/h_nvnMemoryPool*againstNULLpoolsnn::hid::IsOunceVibrationDeviceMounted(VibrationDeviceHandle)was unbridgedhle_hid.ctera's bordered260×260/68×68heightmap and blend-weight tiles shared a 2.43 GB pool with GPU render targets;attach()callednvn_pool_mark_gpu_writteninsidetex_mtl(t), andalias_take_over/ retired descriptors evicted live terrain textures fromg_ghosttex_flush_upload onandtex_dirty_upload on;g_in_rt_attachguardsattach();tex_cpu_contentrecognizes260×260and68×68bordered terrain tiles;ft_addexcludesNVN_MEMORY_POOL_FLAGS_VIRTUALpools (t_sparse_tiles)format 66(RGB_DXT1) andformat 73(RGTC1_UNORM) targets (COMPRESSIBLE)bcrt_attachredirects BC1/BC4 colour attachments toRGBA8Unorm/R8Unormshadow textures and encodes their mips viabc_encode_bc1/bc_encode_bc4onrp_endg_mrbufgrowthfb_preparescannedg_by_serialon every draw in bindless scenes and matched active attachments against themselvesfb_prepareskipsg_by_serialscans unlesss_same_pass > 0and excludes active attachments;mrbuf_putcapsg_mrbufat 64 buffers / 256 MB3. Dodongo's Cavern, King Dodongo & Kakariko / Market frame-pacing fixes
AGX::RenderContextcrash (renderCommandEncoderWithDescriptor:) & 1,621MTLCompilerServicecrashes[base newRenderPipelineDescriptorForSpecialization]on a mesh pipeline (base.meshFunction != nil) returned a descriptor withvertexFunction == nilandmeshFunction == nil; creatingMTL4LibraryFunctionDescriptoron background threads crashedMTLCompilerServicepso_build_exreturnsbasedirectly whend.meshFunction != niland colour attachments match (or compiles viaarch_capture);mesh_fnandpso_offline.pyskipMTL4LibraryFunctionDescriptorfor mesh shaderstex_upgrade/tex_upgrade_pfv) that was ansRGB/linear view (view.parentTexture == base) or backed by anNvImageHeaplost the heap owner, leftg_by_serial/g_ghost/g_aliasstale, or recreated anRGBA8Unorm_sRGBview over anR32Uintstorage upgrade (agl_ndsr_vr);image_preparedid not persistNVN_TEXTURE_FLAGS_IMAGEong_treg[tid].t.d.flagstex_upgradeunwrapsview.parentTexture, preserves theNvImageHeapowner viaMTLTextureDescriptor+newTextureWithDescriptor:offset:, updatesg_by_serial/g_ghost/g_aliasin place, and only recreates views when pixel formats only differ by sRGB;image_preparepersistsNVN_TEXTURE_FLAGS_IMAGEong_treg[tid]and upgradesneed_pfvbefore the first passpkg_upload_forced sparse-pool textures (pool->is_sparse,< 64 MB) onto the synchronous CPUnvn_deswizzle+up_flush()pathpkg_upload_copies gathered sparse pages (sp_gather) into a batch-retainedMTLBufferand takes the batched GPU deswizzle pathres_commit& 965kg_ghostputs in Hyrule Market and Kakariko Villageup_big_srccreated/destroyed a temporarynewBufferWithBytesNoCopyMTLBufferper upload batch;g_pkg_aliashad a 512-entry cap, deleted entries on hit (hm_del_id), and ignored!is_live(treg_retired) texturesup_big_srccaches 16 MB (BIG_S) chunks ing_up_big_src(up_drop_poolinvalidates on unmap);g_pkg_aliasis unbounded with range-overlap invalidation, retains hits across frames, and caches!is_livepackaged textures_mtl_present_screenint_nro/t_symbolsunder ASan / coverage2202d610added a call tomtl_present_screenindm/host/loader.c, whicht_nro.candt_symbols.cinclude directly without linkingnvn_mtl.mvoid mtl_present_screen(uint32_t w, uint32_t h)indm/host/tests/t_nro.canddm/host/tests/t_symbols.cValidation & Measurements
1. Continuous Gameplay Session (Apple M5 Pro, macOS 26.4, Metal 4)
MTLCompilerServicecrashes.p50): 16.29–16.73 ms (~60.0 fps)p50): 10.8–13.9 msres_commit): reduced from 6,439.5 ms / 300f (21.5 ms/f) in Market/Kakariko to < 0.5 ms/f2. Static Shader & Pipeline Interface Validation (
scripts/regress.py)0400486024DEC000): 42,853 / 42,853 shaders built (.metallib,.metal,.json), 0 failed (chain73b8c3e231ca).dm/gfx/tools/interface_check.py): 34,474 / 34,474(vertex/mesh, fragment)pairs checked,"bad": [](0 interface findings),"not_in_archive": 0.3. Test Suites & CI
make -C dm/host/tests GAME=0400486024DEC000 -j8 run asan tsan: 100% pass (includingt_net_eventfd,t_nro,t_symbolsunder ASan/UBSan/TSan).make -C dm/gfx/tests GAME=0400486024DEC000 run: 100% pass (all GPU and CPU tests, includingt_sparse_tiles,t_readback,t_geometry,t_reorder_chain,t_upgrade_alias,t_zskip,t_present_crop,t_tex_fit_root,t_cpu_tex,t_pool_chunks,check_test_dmsh_pull,check_test_tbdr_budget).python3 dm/tools/docs_symptoms.py --check: 0 issues (506 symptoms in docs/reference).Build check,Coverage,Host unit tests and CPU-only gfx tests,Sanitizers,Tool tests,Unsigned iOS app,clang-tidy (ratchet),ruff and pyright (ratchet),label) pass 100% green one03bdc8f.scripts/regress.py impact origin/main..HEADregress.py impact
origin/main..HEAD3 games use what the change touches, 1 only through a broad rule (every game of its API), 0 not affected, 142 unknown (no usage manifest); 68 files changed, 67 NVN / HLE entry points reached through the host's callers, 33 rules of dm/tools/impact_map.tsv
01007EF00011E0000100A3D008C5C0000100F2C0115B60000100535012974000Unknown (no usage.tsv): 142 profiles (--all lists them).
Docs
dm/games/0400486024DEC000/README.mdanddocs/reference/(services.md,audio.md,shaders.md,nvn-objects.md,rendering.md,frame-rate.md), verified withpython3 dm/tools/docs_symptoms.py --check.dmsh_chain.py --headerrun anddm/gfx/nvn/dmsh_chain.h+dm/gfx/tools/dmsh_chain.lockupdated (73b8c3e231ca).