[oyaboon]

What and why

About an hour into an Ocarina of Time (0400486024DEC000, v131072) session, the loads into Hyrule Field and the passages around it froze in runs of 200-670 ms frames for seconds, where the same loads were smooth in the first minutes. The [longframe] lines showed 100-300 first texture uploads a frame, and GPU allocated was at 9.8 GB against 3.9 GB early in the session. Sennbon-99 saw the same on an M4 Pro (24 GB, comment below): loads around Kakariko and Lake Hylia at about 5 fps, with 17,139 IDs marked (14.2 GB) after 41 minutes.

Cause: a texture ID's read mark (pool_seen) stays with the ID through every new texture (PR #2022) and was cleared only when the ID was emptied, so the marks grew with every area passed through. An hour in, the pool walk line counted 19,201 of 33,024 IDs as read (22.7 GB of textures). A walk makes a marked ID's texture whatever its room (pool_entry, under POOL_REQ_BYTES), so an area load, which streams thousands of textures into IDs that earlier areas had read, had them made as fast as the read budget allowed.

Change (host only, nvn_mtl.m; the shaders' dm_pool_mark and the chain are unchanged):

• The host keeps the size of the marked IDs' textures (seen_size, g_seen_kb_all).
• Below a quarter of Metal's working set (SEEN_CAP_DIV: about 2.7 GB on a 16 GB Mac, 9 GB on a 48 GB one) nothing changes from main.
• Above it, pool_seen_scan takes the GPU's marks back every 60 frames (an atomic exchange) and notes the frame each mark was last found in, and mtl_reg_tex keeps the mark for the ID's next texture only when the ID was read within the last DM_NVN_SEEN_KEEP frames (600 by default). The streamed replacement of a material on screen keeps it, as PR Ocarina of Time: large models no longer go black for one frame twice, 4–5 frames apart, when the game streams new textures into them (a texture ID a shader has read keeps its read mark when it gets another texture) #2022 needs; a texture streamed into an ID left behind holds its stand-in (pool_stand) until it is read.
• DM_NVN_SEEN_KEEP=0 restores the old behaviour. The pool stand-ins log line counts the marks kept and dropped, with the cap and the marked size.

Updated (rebased on 713ac08): the first version aged marks whatever their size. dpadr measured that on an M2 MacBook Air (16 GB): a scripted walk out of Kokiri Forest ran 70-110 ms frames with the default against 34 ms with DM_NVN_SEEN_KEEP=0. A mark makes the ID's next texture when it is registered, often hundreds of frames before its first read, so dropping marks of the area just left probably moved that work into the frame the next area appears. The cap keeps those hints while they are small and ages them only once they have piled up.

Reference entry: Area loads froze late in a session as read marks piled up; the PR #2022 entry and Ocarina's note link it.

Checked

• Tests: make -C dm/gfx/tests GAME=0400486024DEC000 run on 713ac08 plus this, on an Apple-silicon Mac: every suite passes. t_pool_seen_replace: 67 checks, 15 new (a read notes its frame and the ID counts its texture's size; below the cap the GPU's mark stays and an unread ID keeps its mark; above it the GPU's mark is taken back while the host's stays, a mark not read within the window goes with the next texture, an ID read again keeps its mark; the size goes with the marks), 50 with DM_NVN_SEEN_KEEP=0. t_pool_standin, t_pool_notes, t_pool_chunks, test_docs, env_switches.py --check pass.
• Games run: Ocarina of Time v131072, with the first form (no cap), two sessions of play on one Mac, not on the same route.
  • Without the change: runs of 200-670 ms frames at the area loads after the first hour, tens of 300-frame windows with a p95 above 40 ms.
  • With it, 78 minutes: one such window in 903; GPU allocated 7.2-9.6 GB from the 6th minute on, following the area rather than the time played; area loads are single frames of 102 and 155 ms (658 and 248 first uploads). 92,703 new textures kept their ID's mark, 46,614 dropped it.
  • Cost: stand-ins made rose to 1,235,328 in 6.19 s over the session, from about 128,000 without the change (23 µs of the submit thread a frame on average).
• Devices and OS: M5 Pro, 48 GB, macOS 26.6.2.
• regress.py check: not run. No other local game streams into a bindless pool this way, and the reach is the pool walk's read marks.
• Not run: any game with the cap (neither the long session nor the Kokiri walk), an A/B on one route, another game, lint_c.py (no LLVM clang-tidy here; CI runs it), iOS.
• Open: whether a quarter of the working set is the right cap; whether the 100-155 ms area-load frames that remain come from the stand-ins made in them or from the first uploads.

Docs

• The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• A chain tool changed: not applicable, no chain tool changed
• No game files, keys or data extracted from a game in the change or the attachments

[Sennbon-99]

Same symptom here on main b149a1e4 (chain 2afd39090b76), without this PR: MacBook Pro M4 Pro (16-core GPU, 24 GB), macOS 26.7.2. Settings: Dynamic resolution on, MetalFX present upscaling, fullscreen.

As an adult, the loads around Kakariko's graveyard, Lake Hylia and the Kakariko gate from Hyrule Field dropped to about 5 fps for seconds. In a 41-minute session, 6 windows of 300 frames ran under 20 fps (lowest 8.7 fps), with single frames up to 910 ms, at about 16, 31 and 47 minutes.

From the log:

• GPU is not the limit: GPU p50 4–15 ms in those windows. NVN submit+present p50 37–66 ms, and the translation line says 52–96 ms of CPU per frame for 920–3,420 draws (about 5 ms in smooth windows).
• Pool residency churn: at frame 111600, residency set had 6,801 commits for 606 ms and 23.1 GB added over 300 frames, 19.9 GB of it from the pool residency in gmem_map (nvn_mtl.m:10666). pool tables rebuilt 43.7 times a frame over memory pool changes. The game's OptimizeChangeBanc, ParallelDecompThread and Resource Memory threads were busy at the same time.
• Growth over the session: GPU allocated went from 3.3 GB at start to 12.1 GB at minute 31, resident 18.1 GB, with 4.1 of 5 GB of swap in use. The last pool walk counted 17,139 of 33,024 IDs read (14.2 GB).
• Unlike the PR description, the long frames had few first texture uploads (0–27 a frame, under 25 MB), so the time goes to the pool residency and the submit thread rather than to uploads.

I have not run this PR yet.

Edited: the build was b149a1e4 (chain 2afd39090b76), not f75ac06e as first written; the measurements are unchanged.

[dpadr]

Measured on Ocarina of Time v131072, an M2 MacBook Air (16 GB), handheld at 1440x810, MetalFX, hold off. The host is our cut8 build (main ccd834e plus this PR at head 4ec0c1c). The route is a scripted Kokiri Forest walk from a save, with back-to-back runs and the same keys.

• DM_NVN_SEEN_KEEP=600 (the PR's default): frame time p50 per 300 frames rises to 70–110 ms (GPU p50 44–60 ms) at frames 2100–2400. That is the walk out of the village, where the area streams.
• DM_NVN_SEEN_KEEP=0 (marks stay for good, as before the PR): 34 ms frames (GPU 31 ms), with no spikes through frame 3300.
• Reference: the same route on our previous host without the PR runs 30 ms frames (GPU 26.6 ms).
• Picture: the same with either setting.
• Open: what the expired marks cost at the transition, whether the stand-ins' later uploads or the walk's re-reads. We haven't split it yet.

We run with DM_NVN_SEEN_KEEP=0 for now.

[oyaboon]

Thanks for the runs. Rebased on 713ac08 and changed in b5ad456:

• @dpadr: marks now age only once the marked IDs' textures come to more than a quarter of Metal's working set (about 2.7 GB on a 16 GB Mac). Below that nothing is taken back or dropped, so your Kokiri walk should run as on main. My guess for the spike: a mark makes the ID's next texture at registration, hundreds of frames before its first read, and aging the village's marks moved that work into the frame the next area appeared. Could you rerun the walk with the default on this head? The pool stand-ins line now prints the cap and the marked size, so it shows whether aging started.
• @Sennbon-99: with 14.2 GB marked on a 24 GB Mac you are above the cap (about 4 GB), so aging applies there. Your residency churn in gmem_map may be a separate cost this does not touch; a run on this head would show how much of it goes with the marks.