A GPU-written span's sliver inside a texture does not make it the GPU's: Ocarina of Time's terrain tiles no longer vanish or show another tile's surface (#1566)

dm/gfx/nvn/nvn_mtl.m

@@ -3904,11 +3904,26 @@ void mtl_tex_gpu_written(const tex_t *t) {
     }
     pthread_mutex_unlock(&g_gws_mu);
 }
+// A span whose overlap with [lo, hi) holds no whole page of it, and which does not cover it, is a sliver of something beside it, and does not decide: textures are 512-byte aligned,
+// so a GPU-drawn one may end in the page a CPU-filled one begins in; a span outlives the texture it was recorded for, so the game may hand part of that storage to a new texture; and
+// a GPU write over part of a span leaves the parts outside, which can be smaller than a page (gws_set). gws_cpu_since cannot answer for a sliver (no whole page to ask the dirty
+// tracker about), and taking its silence as the GPU's made the texture behind it a GPU texture: Ocarina of Time's CPU-filled 260x260 height (R32F) and hole (RGBA8) maps came up as
+// zeros when the span of a 260x260 RG8 normal map (laid out at 214,528 bytes) reached 1,536 or 13,824 bytes into them, or a 2,560-byte remnant of an old span lay inside them, and
+// whole terrain tiles vanished or showed another tile's surface (GitHub issue #1566). A span that covers the whole texture keeps the GPU's: a texture smaller than a page inside storage
+// the GPU wrote, whose neighbours' CPU writes in the same page do not count for it either
+static int gws_sliver(const gws_span *s, uintptr_t lo, uintptr_t hi) {
+    if (s->lo <= lo && s->hi >= hi) {
+        return 0;
+    }
+    uintptr_t a, e;
+    whole_pages(MAX(lo, s->lo), MIN(hi, s->hi), &a, &e);
+    return e <= a;
+}
 static int gws_owned(uintptr_t lo, uintptr_t hi) {   // the GPU wrote some of [lo, hi) and the CPU has not rewritten it since: the storage is not the content
     int owned = 0;
     pthread_mutex_lock(&g_gws_mu);
     for (size_t i = gws_find(lo); i < g_ngws && g_gws[i].lo < hi && !owned; i++) {
-        owned = !gws_cpu_since(&g_gws[i], lo, hi);
+        owned = !gws_sliver(&g_gws[i], lo, hi) && !gws_cpu_since(&g_gws[i], lo, hi);
     }
     pthread_mutex_unlock(&g_gws_mu);
     return owned;

dm/gfx/tests/t_cpu_storage.m

@@ -118,6 +118,41 @@ int main(void) {
         mtl_tex_release(&upper);
         teardown(&f);
 
+        // A span's sliver inside a texture (gws_sliver): less than a page of it, not covering it. Ocarina of Time's terrain tiles, GitHub issue #1566
+        setup(&f, 7040, CACHED, NVN_FORMAT_RGBA8, 0);
+        tex_t drawn = texture_at(&f, 7041, 0x40000, 68, NVN_FORMAT_RGBA8, 0);   // 40,960 bytes: it ends 8 KB into its third page
+        uintptr_t pg = (uintptr_t)getpagesize(), drawn_end = (uintptr_t)drawn.mem + drawn.L.size, shared = drawn_end & ~(pg - 1);
+        CHECK("fixture: the drawn texture ends inside a page", drawn_end != shared && drawn_end - shared < pg);
+        tex_t filled = texture_at(&f, 7042, shared - (uintptr_t)f.mem, 68, NVN_FORMAT_RGBA8, 0);   // begins in that page: the drawn texture's span reaches 8 KB into it
+        mtl_tex_gpu_written(&drawn);
+        dm_guest_write_prepare(filled.mem, filled.L.size); fill(&filled, filled.mem, 40);
+        CHECK("a neighbour's span reaching less than a page into a texture does not make it the GPU's", !gws_owned((uintptr_t)filled.mem, (uintptr_t)filled.mem + filled.L.size));
+        CHECK("and the texture is taken from its storage", tex_cpu_content(&filled));
+        CHECK("while the drawn texture stays the GPU's", gws_owned((uintptr_t)drawn.mem, drawn_end));
+        mtl_tex_release(&filled); mtl_tex_release(&drawn);
+        teardown(&f);
+
+        setup(&f, 7043, CACHED, NVN_FORMAT_RGBA8, 0);   // a remnant of an old span between two newer ones (gws_set keeps the parts outside a new span), inside a texture the CPU then filled
+        large = texture_at(&f, 7044, 0x40000, LARGE, NVN_FORMAT_RGBA8, 0);
+        uintptr_t lo = (uintptr_t)large.mem, hi = lo + large.L.size, cut = lo + large.L.size / 2;
+        mtl_tex_gpu_written(&large);
+        pthread_mutex_lock(&g_gws_mu); gws_set(lo, cut); gws_set(cut + 2560, hi); pthread_mutex_unlock(&g_gws_mu);
+        CHECK("fixture: three spans, the middle one 2,560 bytes", g_ngws >= 3 && g_gws[gws_find(cut)].lo == cut && g_gws[gws_find(cut)].hi == cut + 2560);
+        dm_guest_write_prepare(large.mem, large.L.size); fill(&large, large.mem, 40);
+        CHECK("a sub-page remnant of an old span inside a texture the CPU rewrote does not keep it the GPU's", !gws_owned(lo, hi));
+        CHECK("and the texture is taken from its storage", tex_cpu_content(&large));
+        mtl_tex_release(&large);
+        teardown(&f);
+
+        setup(&f, 7045, CACHED, NVN_FORMAT_RGBA8, 0);   // a texture smaller than a page over storage the GPU wrote stays the GPU's: the span covers it, and a neighbour's CPU write in its page is not its
+        tex_t beside = texture_at(&f, 7046, f.t.L.size, SIZE, NVN_FORMAT_RGBA8, 0);
+        mtl_tex_gpu_written(&f.t);
+        dm_guest_write_prepare(beside.mem, beside.L.size); fill(&beside, beside.mem, 40);
+        CHECK("a texture smaller than a page over storage the GPU wrote stays the GPU's", gws_owned((uintptr_t)f.t.mem, (uintptr_t)f.t.mem + f.t.L.size));
+        CHECK("though a neighbour in its page is CPU-written", !tex_cpu_content(&f.t) && tex_cpu_content(&beside));
+        mtl_tex_release(&beside);
+        teardown(&f);
+
         // nvn_ft_poll's question of a registered texture the CPU wrote after its first use (mtl_tex_cpu_rewritten)
         setup(&f, 7030, CACHED, NVN_FORMAT_RGBA8, 0);
         large = texture_at(&f, 7031, 0x40000, LARGE, NVN_FORMAT_RGBA8, 0);

docs/agents/symptoms.md

@@ -576,6 +576,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - a movie plays black (Age of Calamity: its intro movie under the "+ Skip" prompt). → [Mapped-range flushes taken as uploads (`tex_flush_upload`)](../reference/nvn-objects.md#mapped-range-flushes-taken-as-uploads-tex_flush_upload)
 - open-world ground is one flat colour on a flat plane with blocky seams, or water and sand are missing, while course meshes, rocks and the kart look right. `DM_NVN_TEXINFO=WxH` shows uncompressed textures with `swizzled=0`, no packaged layout and storage holding data in a `CPU_CACHED` pool. → [Uncompressed storage the CPU wrote, in a pool the GPU has not written](../reference/nvn-objects.md#uncompressed-storage-the-cpu-wrote-in-a-pool-the-gpu-has-not-written)
 - after some walking, the ground opens in holes (black, or the water and sky below showing through) and the hole spreads to areas loaded later, including ones that were fine before; saving and loading the game makes it right again for a while. A diagnostic line at the GPU texture's creation shows uncompressed terrain textures in a `CPU_CACHED | GPU_CACHED` pool (flags 0x24) whose storage holds data, not flushed, made as GPU textures of zeros. → [Uncompressed storage beside storage the GPU writes](../reference/nvn-objects.md#uncompressed-storage-beside-storage-the-gpu-writes)
+- after some walking, a whole terrain tile is missing (the ground vanishes, or the tile sits flat and low) or shows another surface with a hard square edge (a square of grass across a cobbled path, a grass patch on a mountain trail); a reload makes it right for a while. The host's copy of the tile's 260x260 R32F height map or RGBA8 hole map is all zeros while the game's memory holds data (`DM_NVN_TEXINFO=260x260` shows storage non-zero at first use; a `texraw` dump of the host texture is zero). → [A span's sliver inside a texture](../reference/nvn-objects.md#a-spans-sliver-inside-a-texture)
 - magenta surfaces (Metal decodes an all-zero ASTC block as the error colour (1, 0, 1)); a zero BC block decodes black, which looks like missing light; black screen while the UI (or a dialog box) draws in the right places under `HILITE` or in the pass dump; `packaged textures: first uploads 0`. → [CPU writes with no flush, caught by page protection (`tex_dirty_upload`)](../reference/nvn-objects.md#cpu-writes-with-no-flush-caught-by-page-protection-tex_dirty_upload)
 - a texture the game builds from CPU-filled textures read back with `nvnTextureReadTexels` comes out zero. → [`ReadTexels` from guest memory and textures written after first use](../reference/nvn-objects.md#readtexels-from-guest-memory-and-textures-written-after-first-use)
 - with `tex_dirty_upload`, a texture the CPU fills again after its first use keeps its old content: Ocarina of Time's ground opened flat blue holes with grass and rocks floating over them (Kokiri Forest, the village floor, the Deku Tree) after an hour of play, as terrain tile slots were reused; no red line, and the frame's `texture uploads … re-uploads` stays at 0 while the pool's pages change. → [Textures made for GPU use kept from their CPU rewrites](../reference/nvn-objects.md#textures-made-for-gpu-use-kept-from-their-cpu-rewrites)

docs/reference/nvn-objects.md

@@ -541,6 +541,14 @@ Games read this count as a position in their own command memory and size their h
 - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) (the pool rule's part first by aCidcOde, GitHub issue #1326): its terrain tiles' height maps (260x260 R32F) and blend and hole maps (260x260 RGBA8, one and two layers) share one pool with the tiles' normal maps (260x260 RG8, drawn by the GPU); after the first normal map was drawn, every tile streamed in came out as zeros, and the ground opened while walking out of a cave and back into it.
 - **Check:** `make -C dm/gfx/tests GAME=<id> check_t_cpu_storage`: a texture beside a GPU-written texture in the same pool is taken from its storage, one over storage the GPU wrote and the CPU then rewrote is taken until the GPU writes there again, a GPU write over another part of that storage leaves the rewrite seen, and storage the CPU cannot write gets no range.
 
+### A span's sliver inside a texture
+
+- **Symptom:** after some walking, a whole terrain tile is missing (the ground vanishes, or the tile sits flat and low) or shows another surface with a hard square edge (a square of grass across a cobbled path, a grass patch on a mountain trail); a reload makes it right for a while. The host's copy of the tile's 260x260 R32F height map or RGBA8 hole map is all zeros while the game's memory holds data (`DM_NVN_TEXINFO=260x260` shows storage non-zero at first use; a `texraw` dump of the host texture is zero).
+- **Cause:** [the previous entry](#uncompressed-storage-beside-storage-the-gpu-writes)'s question, whether the GPU wrote a texture's storage and the CPU has not rewritten it since, is asked of every span that overlaps the texture through its whole pages (`gws_cpu_since`): a span's first and last pages are shared with whatever lies beside it. An overlap that holds no whole page of the texture cannot be answered that way, and its silence was taken as the GPU's. Two slivers do that to Ocarina of Time's tile maps: the span of a 260x260 RG8 normal map (laid out at 214,528 bytes) reaching 1,536 or 13,824 bytes past the texture's start, since textures are 512-byte aligned and a span outlives the texture it was recorded for, so the game hands part of that storage to the next tile's map; and a 2,560-byte remnant of an old span left between two newer ones (`gws_set` keeps the parts outside a new span). Either made the map a GPU texture of zeros at its first use.
+- **Fix:** every game. `gws_sliver`: a span whose overlap with the texture holds no whole page of it, and which does not cover the texture, does not decide in `gws_owned`. A span that covers the whole texture keeps the GPU's, so a texture smaller than a page over storage the GPU wrote stays a GPU texture, and a neighbour's CPU write in the same page is not its.
+- **Check:** `make -C dm/gfx/tests GAME=<id> check_t_cpu_storage`: a neighbour's span reaching less than a page into a texture the CPU filled leaves it taken from its storage while the drawn neighbour stays the GPU's; a sub-page remnant of an old span inside a texture the CPU rewrote does not keep it the GPU's; a texture smaller than a page over storage the GPU wrote stays the GPU's though a neighbour in its page is CPU-written.
+- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) on the Mac (M4 Pro, macOS 26.6; GitHub issue #1566's reports from Death Mountain Trail and Kakariko are the same class): a first-use trace of every 260x260 texture logged 12 of 620 CPU-filled maps resolved as GPU textures in 26 minutes of play, all with a sliver as the only unanswered span, and whole tiles missing in Hyrule Field, a grass square across the path from Hyrule Castle Town, a grass patch on Death Mountain. With the rule, 1,498 of 1,510 tile maps in the next 26 minutes were taken from their storage (the other 12 were still zeros at first use, filled later), 162 of them past a sliver that would have made them GPU textures, and none came up as zeros. PR #1876 found the remnant case independently and keeps a split span's protected pages; the edge case, 10 of the 12, is not covered by that.
+
 ### CPU writes with no flush, caught by page protection (`tex_dirty_upload`)
 
 - **Symptom:** magenta surfaces (Metal decodes an all-zero ASTC block as the error colour (1, 0, 1)); a zero BC block decodes black, which looks like missing light; black screen while the UI (or a dialog box) draws in the right places under `HILITE` or in the pass dump; `packaged textures: first uploads 0`.