[roandegraaf]

What and why

Walking into Hyrule Castle Town in Ocarina of Time stutters for a few seconds: frames of 30–64 ms while the town's textures load, with no pipeline being built.

Cause. Two things in the host's look at a texture taken from guest memory (tex_mtl_ in dm/gfx/nvn/nvn_mtl.m), which show where a game's loader thread fills a texture over several frames while the game already draws with it:

1. One written page sent every layer and level of the texture through the upload again, into another Metal texture. The game fills a 1800x1380x8 array (38.5 MB) piece by piece: 13–17 whole uploads in 6 frames.
2. The pages were asked at every use. The queue's thread replays a submission while the loader thread writes on, so each look found the few pages written since the last one and uploaded again. Measured with the new submission field of the DM_NVN_REUPLOADS line: most of the re-uploads of one frame are inside one submission (5 of the array, 8 of a 3072x2096 BC5 texture).

Fixing (1) alone made (2) worse, as a shorter upload lets more looks see a fresh write: 17–22 uploads in one submission and a frame of 148 ms.

Change. General, in the host:

• A re-upload reads only the images (a layer's level, or a 3D texture's level) with a page written since the last upload; the others are copied on the GPU from the texture it replaces (pkg_written, up_copy_add, up_flush). DM_NVN_PARTIAL_UPLOAD=0 reads whole.
• A submission looks at a texture's pages once. A write another thread makes during the replay is found at the next submission's first look, and nvn_dirty_touch at that mtl_sub_begin sends the caches that remember "nothing written since" by generation through their check again. Writes by the replaying thread itself (nvn_dirty_own_writes) and looks outside a submission stay exact.
• A texture holding copied images is read whole once any upload batch was dropped since its last whole upload (g_up_lost_gen).
• Found on the way: the recovery of a dropped upload batch was skipped for a texture with edge copies (pkg_edges_clean found those bytes unchanged and kept the unwritten texture). t_direct_upload passed only because its recovery block ran when the report arrived before the test looked; it now waits for it.
• The DM_NVN_REUPLOADS line names how much was read, the time, and the submission.

Reference entry: A texture written while it is drawn read whole at every use (added here), with its lines in the game note.

Open:

• A single-image texture (the 3072x2096 BC5 one) is still read whole, once per submission: nothing finer than an image is tracked.
• Sparse, linear and multisample textures take the full path.
• A re-upload while the spare is busy still allocates a texture and copies the unwritten images.
• DM_NVN_DIRTYCHECK=1 reports one dirty-page miss on the town route, with the old logic as well. Unexplained.

Checked

• Tests, on this branch (ced3612 plus the change), make -C dm/gfx/tests GAME=0400486024DEC000 check_<name>: t_partial_upload 3,585 checks, 0 failed; t_direct_upload 26,384 + 26,388; t_cpu_tex 98; t_alias_takeover 28; t_cpu_storage 135; t_sparse_tiles 4,253; t_view_swizzle 554; t_pool_chunks 151; t_upload_slots 305 ×2; t_nvn 298; t_clear_msaa 32 (the multisample zero fill of 01de04f, which shares the upload batch); t_readback 218 cases; t_dirty_race, t_dirty_retire_race, t_dirty_slots; all 0 failed. make -C dm/host GAME=0400486024DEC000 builds. python3 -m pytest -q dm/tools/tests/test_docs.py dm/tools/tests/test_env_switches.py: 7 passed.
• Broken variants of the change against t_partial_upload (on base 8db5dd4): no copies of the unwritten images → 1,216 image mismatches; no nvn_dirty_touch → 44 failed; no own-write condition → 88 failed; no in-submission condition → 22 failed.
• Games run: The Legend of Zelda: Ocarina of Time, 0400486024DEC000, v131072, chain cb2e91c138e3, on base 8db5dd4: headless, a recorded walk into Hyrule Castle Town replayed from a fixed save, 70 s, DM_FPS_HOLD=off, compared by the DM_NVN_REUPLOADS lines and the long-frame lines; 0 red lines. A default Hyrule Field route: 56.5–58.1 fps, 0 red lines, no re-uploads (with the old logic too). Not repeated in the game on ced3612: the change applied to it without conflicts, and the tests above ran on it.
• Devices and OS: Apple M4 Pro (20-core GPU), macOS 27.2. iOS not checked.
• No scripts/regress.py check --games report: Ocarina of Time is the only game built on this Mac. Every NVN game with textures taken from guest memory passes through this path.
• Measurements, the town route, the runs in which the array is rewritten (the route skips that phase in about one run in three):

	Before	After
Re-uploads a run	31–52	16–17
The 38.5 MB array	16–17 times, whole	7–9 times, 4.7–14.2 MB read each
Frames while it is written	30–64 ms	25 ms at most
Red lines	0	0

In windowed play before the change: frames of 26–57 ms with 75.8–230 MB re-uploaded in each.

Docs

• The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
• A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change
• No game files, keys or data extracted from a game in the change or the attachments

[roandegraaf]

Re-measured on ced3612 with its own shader libraries (chain bc8830d4ba5a).

The body's results hold on this base, with two differences: the largest partial read of the array was 19.0 MB (14.2 MB in the body), and with DM_NVN_DIRTYCHECK=1 the branch's runs did not follow the recorded walk into the town, so the check was not exercised there on the branch (details below).

Machine and settings: Apple M4 Pro (20-core GPU, 24 GB), macOS 27.2, profile 0400486024DEC000 v131072, headless, DM_FPS_HOLD=off, DM_NVN_REUPLOADS=1, audio off, hosts built from ced3612 and 213a281, the same rewritten modules, every run from a copy of the same warmed pipeline archive. The route is the recorded walk from the drawbridge into the town, 70 s; the town loads near frame 1,220. "After the load" below means after frame 1,300. Baseline and branch were interleaved; the baseline column has eight runs because the same baseline runs served two other comparisons that night.

	ced3612 (8 runs)	branch (3 runs)
Runs with the town's array phase	7 of 8	2 of 3
Re-uploads a run	37–57 with the array phase, 27 without	13–15 with the array phase, 53 without
The 1800x1380x8 array (38.5 MB), reads a run	11–19, each whole: 424–732 MB	7–9, each 4.7–19.0 MB: 66–76 MB
The 3072x2096 texture, re-uploads a run (first 3 baseline runs)	21–22	2
Longest frame carrying a re-upload, after the load	49.2–90.3 ms	20.5–23.6 ms
Frames over 30 ms after the load	3–6 a run	1–2 a run
Frame rate, last 1,200 frames	57.96–60.12 fps	58.42–58.91 fps
Game's GPU time	15.14–15.86 ms a frame	15.63–15.72 ms a frame
Other processes' GPU time	5.8–8.1 ms/s	7.0–11.4 ms/s
[RED] lines	0–1 a run	0

Notes on the table:

• One frame of 111–119 ms near frame 2,460–2,490 is in every run on both sides and carries no upload and no pipeline work; it is counted in "frames over 30 ms". Without it the branch has 0–1 such frames a run (30.6 and 31.8 ms), none with a re-upload.
• The branch's run without the array phase re-uploaded 45 small textures whole (341x341, 512x512 and 768x768, 15 times each) that no other run of either side touched; its longest frame with a re-upload was 20.5 ms.
• The red lines are texture type mismatch … slot 7, at most one a run, in 3 of the 8 baseline runs.
• Compared with the body: re-uploads 31–52 → 16–17 there, 27–57 → 13–15 here (53 in the run without the array phase); the array 16–17 whole reads → 7–9 reads of 4.7–14.2 MB there, 11–19 → 7–9 reads of 4.7–19.0 MB here; frames 30–64 ms → 25 ms at most there, 49–90 ms → 24 ms at most here.

DM_NVN_DIRTYCHECK=1 (the route, 70 s, three runs a side):

• Baseline: reached the town in 3 of 3 runs, at 28 fps, and logged one [RED] dirty-page miss: texture ser=… pool content changed but was not detected in each (serials 14611, 14677, 14686; the host prints the line for a run's first miss only).
• Branch: no dirty-page miss in 3 of 3 runs, but none of them reached the town: the recorded inputs left the character at a wall beside the drawbridge (about 1,300 draws at frame 1,300 where the other runs have 2,700–2,800), with no re-upload in the whole run and 60 fps. The inputs are replayed by frame number and the check changes how long the loads take, so the walk can go out of step; I did not find out why it does so only on the branch. The body's "one unexplained miss either way" is therefore confirmed for the baseline and not checked for the branch.

Pictures: one frame 44 s into each run (DM_CTL shot), mean absolute difference of 255: branch against baseline 20.1–20.3, two baseline runs against each other 19.9–20.4. The shot lands in the middle of the walk through a crowd, so this only says the branch differs from the baseline no more than a run does from itself; the pictures look right by eye on both sides.

Tests on the branch at ced3612, all 0 failed: t_partial_upload 3,585 checks, t_direct_upload 26,384 and 26,388, t_cpu_tex 98, t_alias_takeover 28, t_cpu_storage 135, t_sparse_tiles 4,253, t_view_swizzle 554, t_pool_chunks 151, t_upload_slots 305 (twice), t_nvn 298, t_clear_msaa 32, t_readback 217 and 218, t_dirty_race, t_dirty_retire_race and t_dirty_slots passed; test_docs.py, test_env_switches.py 7 passed.

Notes and what was not checked:

• GPU time by process was sampled from outside the host (the accelerator's user clients in the I/O registry) over 19 s inside the measured window.
• Each run compiled 1–28 pipeline bases while loading, on both sides.
• Not checked: DM_NVN_DIRTYCHECK=1 in the town on the branch (above), a real window, iOS, other games (no regress.py run).
• GitHub now reports a conflict with main for this branch; it was measured as one commit on ced3612.

[seedds]

Landed on main as d2556cf, with your code, test and measurements kept as they are and you as co-author. Thank you: the diagnosis (that fixing the whole read alone made the repeated looks worse) is what made this easy to judge.

Checked here on main plus the change (M6): t_partial_upload 3,585 checks, t_direct_upload 26,384 and 26,388, t_cpu_tex, t_upload_slots, t_dirty_race, t_dirty_slots, all 0 failed. regress.py check passes on Bastion and The Binding of Isaac: Rebirth with no new red line. Neither route makes a re-upload, and neither does Ocarina's Kokiri Forest walk, so no game run here reached the partial read: in a game it rests on your town runs.

Left open in the reference entry:

• The dirty-page miss with DM_NVN_DIRTYCHECK=1 in the town. Your runs predate 1e69c18 (a write missed while opening shared pool pages, PR Hash shared pool pages before opening them for CPU writes #2136), which may be that miss. If you can, one town run with the check on current main would settle it; there is no save near the town on the Mac this was checked on.
• The single-image textures, still read whole once a submission.