From 2261a388d9f5e0ae22f42dca8463f565eb5e20da Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 09:20:38 +0100 Subject: [PATCH 01/40] File pages under a buffer joining the residency set are wired by mlock before the commit, so the commit no longer pages the guest's files in from the GPU driver's path, where the M1 iPad froze for good With the guest's memory in files (the iOS app's default when the game's pool is larger than its memory), Ocarina of Time on an M1 iPad (8 GB) froze with sound and no picture, once in Link's house and once at the title movie. A sysdiagnose spindump during the freeze: the encode thread in res_commit <- gmem_map, IOGPUFamily -> the VM -> apfs pagein, blocked by an APFS rwlock owned by a launchd background-I/O thread that stopped 4 ms after it and never ran again. Ordinary page-ins of the same files (700,000 in a run) never hung. file_wire (nvn_mtl.m): a shared buffer whose pages are file-backed (vm_region_recurse_64: external pager) is mlocked before it joins the set, and the mlocks are given back right after the commit, which holds the pages for the GPU from then on. Anonymous memory and Metal's own buffers are untouched, so nothing changes with the guest anonymous (every Mac run unless DM_GUEST_FILE=1). Counted in the 300-frame residency line. t_file_wire checks it (failed before the fix, passes after). iPad, the same route that froze twice: alive past frame 12,900, 20 fps in the house (GPU 34-42 ms), headroom 5.3 GB, commits under 2 ms per 300 frames. Mac (M4 Air 16 GB, idle) with DM_GUEST_FILE=1, a route that stalled at 7-37 s a frame before: played through, frame time p50 16.7 ms, 3.9 GB wired at most. memory.md "The residency commit froze over the guest's files"; Ocarina of Time's note. Rebased onto main 0c078130, whose residency changes are queued and applied by the commit outside g_res_mu (0c078130): res_add_at wires a buffer's file pages before it queues the change, and res_commit takes the wired ranges together with the queue, so each commit gives back after itself exactly the ranges of the buffers it made resident; a range wired for a change queued after it waits for that change's own commit. --- dm/games/0400486024DEC000/README.md | 1 + dm/gfx/nvn/nvn_mtl.m | 40 ++++++++++++++ dm/gfx/tests/Makefile | 2 + dm/gfx/tests/t_file_wire.m | 84 +++++++++++++++++++++++++++++ docs/agents/symptoms.md | 1 + docs/reference/memory.md | 8 +++ 6 files changed, 136 insertions(+) create mode 100644 dm/gfx/tests/t_file_wire.m diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 1487d7b177..ade7e33748 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -139,6 +139,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [A NaN colour under a zero blend weight](../../../docs/reference/shaders.md#a-nan-colour-under-a-zero-blend-weight): resolved, white clouds over the ground mist between Kokiri Forest and Hyrule Field and over the ferns at the camera in Kokiri Forest ([issue #1839](https://github.com/seedds/nxnative/issues/1839)). The mist shader `5ad65c0eefd09c7f` writes NaN at alpha 0 where its layer has density 0; NVIDIA's blender, with the option NVN's driver sets, adds nothing there, Apple's stored the NaN. - [A first start builds every base of the prewarm list before the first frame](../../../docs/reference/rendering.md#a-first-start-builds-every-base-of-the-prewarm-list-before-the-first-frame): the 450-475 s first-start prepare on an M5 Max; the Mac app now bundles the list's pipelines (`--prebake`, on by default) and it took 4.3 s. - [The 60 Hz link at 40 fps in fullscreen on a ProMotion screen](../../../docs/reference/frame-rate.md#the-60-hz-link-at-40-fps-in-fullscreen-on-a-promotion-screen): 40 fps in fullscreen on an M5 Pro MacBook Pro with the GPU at 10 ms of the 16.7, every present one 120 Hz refresh late and every other display-link tick skipped; the screen's fixed 60 Hz mode held 60.0 fps there and fullscreen now switches to it, with the 120 Hz link (59 fps) for screens with no such mode. +- [The residency commit froze over the guest's files](../../../docs/reference/memory.md#the-residency-commit-froze-over-the-guests-files): resolved, the M1 iPad's freeze with sound and no picture. - [The shader package and the shader map come from different splits](../../../docs/reference/packing.md#the-shader-package-and-the-shader-map-come-from-different-splits): resolved, a black screen with sound in a build through NX2Apple ([issue #1841](https://github.com/seedds/nxnative/issues/1841)); the present pass was translated from a terrain shader's record. Records now carry their keys, and the shader build finds them by key. - [Native mesh pipelines compiled at their first draw](../../../docs/reference/shaders.md#native-mesh-pipelines-compiled-at-their-first-draw): resolved. The foliage, shadows and enemies drawn with task and mesh shaders stalled at their first draw on a cold Metal cache (59.5 s of builds at the Kokiri Forest entry on an M5 Max, one frame of 42.9 s), as nothing recorded or kept their pipelines; `pso_manifest.tsv` now carries 164 `M` rows (Kokiri Forest, Hyrule Field, Castle Town, Hyrule Castle, and 41 from later play, Death Mountain Trail among it), and with the prepare step they are built before the first frame. Pipelines outside the list still stall once, then go into `pso_user.tsv`. With the first 123 alone, a cold first start (empty Metal cache and pipeline archive, M4 Pro) on a Death Mountain Trail save built 9 and 10 at their first draw in two runs, 2.7 and 4.2 s, frames of 1,775 and 1,657 ms; with all 164, none, worst frame 25.6 and 18.5 ms, the prepare step 29.6 and 29.3 s → 32.1 and 31.0 s. - [Early fragment tests from the control section](../../../docs/reference/shaders.md#early-fragment-tests-from-the-control-section): 746 of its 22,352 pixel shaders set the NVN2 flag (control +0x5b), none of them writes depth; before, no NVN2 shader got the attribute. diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index a66eaad09e..830f55d1e2 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -285,6 +285,41 @@ static void ws_report(uint64_t frame, uint64_t frames) { // Residency churn by call site (the power line every 300 frames, mtl_res_stats): each commit with changes is a kernel call whose cost grows with what was added or removed enum { RS_SITES = 16 }; static struct { int line; uint64_t n, bytes; } g_rs_site[RS_SITES]; static uint64_t g_rs_rm, g_rs_commits, g_rs_commit_ns; #define res_add(r) res_add_at((r), __LINE__) +// File pages under a shared buffer that joins the set are wired through the CPU's fault path (mlock) before the commit, and given back after it, when the commit holds them for the GPU. +// The commit wires an added buffer's pages itself, and pages a file in from the GPU driver's wiring path (IOGPUResourceGroupUpdateResources → the VM → APFS pagein). With the guest's +// memory in files (DM_GUEST_FILE) on the M1 iPad, that page-in waited for good on an APFS lock held for writing by a launchd thread (its background I/O queue), which itself had stopped +// 4 ms after it and never ran again: the game froze with sound and no picture, the encode thread in res_commit, guest threads that touched the files stuck in their faults (a sysdiagnose +// spindump, 2026-11-08). Hundreds of thousands of ordinary page-ins of the same files never hung, so the pages are brought in and held that way first, and the commit finds them resident. +// What launchd's thread waited for is not known (the kernel had no symbols): memory.md, "The residency commit froze over the guest's files". Anonymous memory and Metal's own buffers are not +// file-backed and are left alone. Under g_res_mu +typedef struct { void *p; size_t n; } fwire_t; static fwire_t *g_fwire; static size_t g_nfwire, g_cfwire; static uint64_t g_fwire_n, g_fwire_bytes, g_fwire_ns, g_fwire_fail; +static int range_file_backed(const void *p, size_t n) { + vm_address_t a = (vm_address_t)p, e = a + n; natural_t depth = 0; // vm_region_recurse_64: mach_vm.h is not in the iOS SDK + while (a < e) { + vm_address_t r = a; vm_size_t sz = 0; vm_region_submap_info_data_64_t info; mach_msg_type_number_t c = VM_REGION_SUBMAP_INFO_COUNT_64; + if (vm_region_recurse_64(mach_task_self(), &r, &sz, &depth, (vm_region_recurse_info_t)&info, &c) || r >= e) return 0; + if (info.is_submap) { depth++; continue; } + if (info.external_pager) return 1; + a = r + sz; + } + return 0; +} +static void file_wire(id r) { + if (![(id)r conformsToProtocol:@protocol(MTLBuffer)]) return; + id b = (id)r; + if (b.storageMode != MTLStorageModeShared || !b.length) return; + uintptr_t pg = (uintptr_t)getpagesize(), lo = (uintptr_t)b.contents & ~(pg - 1), hi = ((uintptr_t)b.contents + b.length + pg - 1) & ~(pg - 1); + if (!range_file_backed((void *)lo, hi - lo)) return; + uint64_t t0 = clock_gettime_nsec_np(CLOCK_UPTIME_RAW); + if (mlock((void *)lo, hi - lo)) { // the commit pages them in itself, as before + if (!g_fwire_fail++) MLOG("[RED] residency: file pages under a buffer of %.1f MB could not be wired before the commit (mlock errno %d); the commit pages them in itself", (hi - lo) / 1048576.0, errno); + return; + } + g_fwire_ns += clock_gettime_nsec_np(CLOCK_UPTIME_RAW) - t0; g_fwire_n++; g_fwire_bytes += hi - lo; + if (g_nfwire == g_cfwire) { g_cfwire = g_cfwire ? 2 * g_cfwire : 64; g_fwire = realloc_or_die(g_fwire, g_cfwire * sizeof *g_fwire, "the wired file ranges"); } + g_fwire[g_nfwire].p = (void *)lo; g_fwire[g_nfwire++].n = hi - lo; +} +static void file_unwire(fwire_t *fw, size_t n) { for (size_t i = 0; i < n; i++) munlock(fw[i].p, fw[i].n); free(fw); } // a range the game has mapped anew meanwhile fails, harmlessly // Both run on guest threads too (pool_res from mtl_addr, uploads), whose implicit autorelease pools never drain; containsObject: / removeObject: on a weak NSHashTable leave the object // autoreleased in the calling thread's pool (measured in a standalone test: the object outlived its last strong reference until the thread ended), so each has a pool of its own static void res_q_put(id r, int add) { // Caller holds g_res_mu @@ -299,6 +334,7 @@ static void res_add_at(id allocation, int line) { @autoreleasepoo if (!g_res_objects) g_res_objects = [NSHashTable weakObjectsHashTable]; if ([g_res_objects containsObject:r]) { pthread_mutex_unlock(&g_res_mu); return; } // already resident (a shared texture resolved again): marking the set dirty would cost a commit that changes nothing { int k = 0; while (k < RS_SITES - 1 && g_rs_site[k].line && g_rs_site[k].line != line) k++; g_rs_site[k].line = line; g_rs_site[k].n++; g_rs_site[k].bytes += r.allocatedSize; } + file_wire(r); // before its change is queued: the commit that applies the change takes this range with it and gives it back after (res_commit) res_q_put(r, 1); [g_res_objects addObject:r]; pthread_mutex_unlock(&g_res_mu); @@ -454,6 +490,7 @@ static void res_commit(void) { @autoreleasepool { pthread_mutex_lock(&g_res_commit_mu); pthread_mutex_lock(&g_res_mu); res_op *q = g_res_q; size_t n = g_res_qn; g_res_q = NULL; g_res_qn = g_res_qcap = 0; + fwire_t *fw = g_fwire; size_t nfw = g_nfwire; g_fwire = NULL; g_nfwire = g_cfwire = 0; // the file pages wired for exactly these changes pthread_mutex_unlock(&g_res_mu); if (n) { uint64_t t0 = clock_gettime_nsec_np(CLOCK_UPTIME_RAW); @@ -462,6 +499,7 @@ static void res_commit(void) { @autoreleasepool { uint64_t dt = clock_gettime_nsec_np(CLOCK_UPTIME_RAW) - t0; pthread_mutex_lock(&g_res_mu); g_rs_commits++; g_rs_commit_ns += dt; pthread_mutex_unlock(&g_res_mu); } + file_unwire(fw, nfw); // after the commit that made those buffers resident; under g_res_commit_mu, so a later commit cannot run before them pthread_mutex_unlock(&g_res_commit_mu); for (size_t i = 0; i < n; i++) CFRelease(q[i].o); free(q); @@ -508,6 +546,8 @@ void mtl_res_stats(char *out, size_t n) { for (int i = 0; i < 4 && o < n; i++) { int best = -1; for (int k = 0; k < RS_SITES; k++) if (g_rs_site[k].n && (best < 0 || g_rs_site[k].bytes > g_rs_site[best].bytes)) best = k; if (best < 0) break; o += snprintf(out + o, n - o, "%sline %d: %llu, %.0f MB", i ? ", " : "", g_rs_site[best].line, (unsigned long long)g_rs_site[best].n, g_rs_site[best].bytes / 1048576.0); g_rs_site[best].n = 0; } if (o < n) snprintf(out + o, n - o, "), removed %llu, %lu allocations in the set (%.0f MB)", (unsigned long long)g_rs_rm, set_n, set_mb); + if (g_fwire_n + g_fwire_fail) { o = strlen(out); if (o < n) snprintf(out + o, n - o, ", file-backed buffers wired before their commit %llu, %.0f MB, %.1f ms (refused %llu)", (unsigned long long)g_fwire_n, g_fwire_bytes / 1048576.0, g_fwire_ns / 1e6, (unsigned long long)g_fwire_fail); } + g_fwire_n = g_fwire_bytes = g_fwire_ns = 0; memset(g_rs_site, 0, sizeof g_rs_site); g_rs_rm = g_rs_commits = g_rs_commit_ns = 0; pthread_mutex_unlock(&g_res_mu); } diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 9f557c1db8..251e26d299 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -166,6 +166,8 @@ check_t_tex_fit_root: t_tex_fit_root MTL_DEBUG_LAYER=1 ./t_tex_fit_root t_addr_known: t_addr_known.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_addr_known.m $(MTL_T_SRC) $(FW) +t_file_wire: t_file_wire.m $(MTL_T_DEP) + clang $(CFLAGS) -fobjc-arc -o $@ t_file_wire.m $(MTL_T_SRC) $(FW) t_pool_chunks: t_pool_chunks.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_pool_chunks.m $(MTL_T_SRC) $(FW) t_pool_retire: t_pool_retire.m $(MTL_T_DEP) diff --git a/dm/gfx/tests/t_file_wire.m b/dm/gfx/tests/t_file_wire.m new file mode 100644 index 0000000000..80ccc9a5e3 --- /dev/null +++ b/dm/gfx/tests/t_file_wire.m @@ -0,0 +1,84 @@ +// A shared buffer over file-backed pages joins the residency set with its pages already resident, wired through the CPU's fault path (mlock) before the commit, and that wiring +// is given back once the commit has made the pages resident for the GPU. With the guest's memory in files (DM_GUEST_FILE) the commit itself paged the file in from the GPU +// driver's wiring path, and on the M1 iPad that page-in waited for good on an APFS lock held by a launchd thread, itself waiting: the game froze with sound and no picture +// (docs/reference/memory.md, "The residency commit froze over the guest's files"). Anonymous memory is not touched +#include "../nvn/nvn_mtl.m" +#include +void *front_metal_layer(void) { return NULL; } +module_t *module_of(const void *p) { (void)p; return NULL; } +module_t *g_mod; int g_nmod; +void svc_thread_dump(void) {} +uint64_t now_ns(void) { return bp_now(); } +static int checks, failed; +#define CHECK(label, c) do { checks++; if (!(c)) { failed++; fprintf(stderr, "FAIL: %s (line %d)\n", label, __LINE__); } } while (0) +// The smallest and largest user wiring count over [p, p + n) +static void wired(const void *p, size_t n, unsigned *lo, unsigned *hi) { + mach_vm_address_t a = (mach_vm_address_t)p, e = a + n; *lo = ~0u; *hi = 0; + while (a < e) { + mach_vm_size_t sz = 0; natural_t depth = 0; vm_region_submap_info_data_64_t info; mach_msg_type_number_t c = VM_REGION_SUBMAP_INFO_COUNT_64; + if (mach_vm_region_recurse(mach_task_self(), &a, &sz, &depth, (vm_region_recurse_info_t)&info, &c) || a >= e) break; + if (info.user_wired_count < *lo) *lo = info.user_wired_count; + if (info.user_wired_count > *hi) *hi = info.user_wired_count; + a += sz; + } +} +static int all_resident(const void *p, size_t n) { + size_t pg = getpagesize(), np = n / pg; char *v = malloc(np); int ok = !mincore(p, n, v); + for (size_t i = 0; ok && i < np; i++) ok = v[i] & MINCORE_INCORE; + free(v); return ok; +} +static int in_set(id b) { pthread_mutex_lock(&g_res_mu); int r = [g_res_objects containsObject:b]; pthread_mutex_unlock(&g_res_mu); return r; } +int main(void) { @autoreleasepool { + mtl_init(); + size_t pg = getpagesize(), n = 256 * pg; + char path[] = "/tmp/t_file_wire.XXXXXX"; int fd = mkstemp(path); if (fd < 0) abort(); unlink(path); + if (ftruncate(fd, (off_t)n)) abort(); + uint8_t *m = mmap(NULL, n, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); if (m == MAP_FAILED) abort(); + memset(m, 0x5a, n / 2); // half written, half holes + msync(m, n, MS_SYNC); msync(m, n, MS_INVALIDATE); // written out and dropped: the next touch is a page-in + mprotect(m + 16 * pg, 16 * pg, PROT_READ); // as the write tracking leaves a texture's pages + unsigned lo, hi; + + id b = [g_dev newBufferWithBytesNoCopy:m length:n options:MTLResourceStorageModeShared deallocator:nil]; + res_add(b); + wired(m, n, &lo, &hi); + CHECK("the file pages are user-wired once res_add returns, before the commit", lo == 1 && hi == 1); + CHECK("and resident: the commit has nothing to page in", all_resident(m, n)); + CHECK("the buffer is in the set", in_set(b)); + res_commit(); + wired(m, n, &lo, &hi); + CHECK("the commit gives the user wiring back", lo == 0 && hi == 0); + CHECK("the bytes are the file's", m[0] == 0x5a && m[n - 1] == 0); + + // Wiring someone else holds over the range is left as it was (mlock counts) + mlock(m, 8 * pg); + id b2 = [g_dev newBufferWithBytesNoCopy:m length:n options:MTLResourceStorageModeShared deallocator:nil]; + res_add(b2); res_commit(); + wired(m, 8 * pg, &lo, &hi); + CHECK("an mlock taken outside stays", lo == 1 && hi == 1); + munlock(m, 8 * pg); + + // Added twice before one commit: wired once, given back once + id b3 = [g_dev newBufferWithBytesNoCopy:m + 128 * pg length:64 * pg options:MTLResourceStorageModeShared deallocator:nil]; + res_add(b3); res_add(b3); + wired(m + 128 * pg, 64 * pg, &lo, &hi); + CHECK("a second res_add of a buffer in the set wires nothing more", lo == 1 && hi == 1); + res_commit(); + wired(m + 128 * pg, 64 * pg, &lo, &hi); + CHECK("and the commit leaves nothing wired", lo == 0 && hi == 0); + + // Anonymous memory: no wiring at all + uint8_t *a = mmap(NULL, n, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0); if (a == MAP_FAILED) abort(); + id ab = [g_dev newBufferWithBytesNoCopy:a length:n options:MTLResourceStorageModeShared deallocator:nil]; + res_add(ab); + wired(a, n, &lo, &hi); + CHECK("an anonymous range is not wired by res_add", hi == 0); + res_commit(); + id own = [g_dev newBufferWithLength:n options:MTLResourceStorageModeShared]; + res_add(own); res_commit(); + CHECK("a buffer Metal allocated goes in as before", in_set(own)); + + res_remove(b); res_remove(b2); res_remove(b3); res_remove(ab); res_remove(own); res_commit(); + printf("t_file_wire: %d checks, %d failed\n", checks, failed); + return failed != 0; +} } diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index fbc4c1a393..e42abfb307 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -187,6 +187,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - with a profile that has `tex_dirty_upload`, a texture in a sparse pool that the game draws into or writes as an image logs `CPU-written texture ser=…` at its first use, and takes its content from its storage. `t_sparse_tiles` crashed at `and unmapped, not drawn: its heap tile goes back` when built with such a profile, and passed with one without the key. → [A texture in a sparse pool came out CPU-written at its first use](../reference/memory.md#a-texture-in-a-sparse-pool-came-out-cpu-written-at-its-first-use) - on an iPhone the app is ended by the system (a jetsam event: no crash report, no `[dm] CRASH` line) once its footprint reaches the limit: about 6 GB on a 12 GB iPhone with the increased memory limit (6,143 MiB measured in LiveContainer), 3.3 GB without it. The log stops mid-frame. → [iOS ends the app once the guest's touched pages reach its memory limit](../reference/memory.md#ios-ends-the-app-once-the-guests-touched-pages-reach-its-memory-limit) - with the guest in files on an iPad, `[RED] [mem] guest file: 5.0 GB free on its disk, under 5.0: stopping the game` after a few minutes of play. The files held 6.6 GB on disk (`allocated on disk` in the `[mem] guest file` line) while the heap was 6.2 GB and the alias region had 2.5 GB mapped, though mapped up to +3.8 GB. The disk's free space went from 12.0 to 5.0 GB, almost all of it the files. → [Guest files keep the blocks of memory the game gave back](../reference/memory.md#guest-files-keep-the-blocks-of-memory-the-game-gave-back) +- with the guest's memory in files, the game stops presenting for good while sound plays on: a black screen, or the last picture held. A `DM_CTL` `dump` shows the encode thread in the kernel under `res_commit ← gmem_map` (`iokit_user_client_trap` or `IOGPUResourceGroupUpdateResources`), the main fence completed, and at times guest threads at the same guest instruction, loads and stores, dump after dump. On the M1 iPad, Ocarina of Time froze at the title movie (frame 1614) in one run and walking in Link's house (frame ~4755) in another, with 7.6 GB of headroom and 1.6 GB resident: not memory pressure. The app cannot be started again until the device restarts: `devicectl device process launch --terminate-existing` hangs, the process gone from the list. → [The residency commit froze over the guest's files](../reference/memory.md#the-residency-commit-froze-over-the-guests-files) - a Switch 2 game goes past Metal's recommended working set while its first 3D scene loads: `Insufficient Memory` command buffer errors, then a hung Mac. Ocarina of Time on a 16 GB Mac: 12 GB and more wired, once a kernel panic in `IOGPUGroupMemory.cpp` at 14.7 GB wired. → [Memory pools nothing gave an address to stay out of the raw pool table](../reference/memory.md#memory-pools-nothing-gave-an-address-to-stay-out-of-the-raw-pool-table) - a game whose shaders read texture handles from memory wires far past the working set while a scene loads, with `Insufficient Memory` errors or a panic, and stutters when new textures stream in. With a limit on how many it makes, surfaces in shadow go near black instead. → [Shaders report which textures they read through a pool](../reference/memory.md#shaders-report-which-textures-they-read-through-a-pool) - memory pools that hold only shader code are in the residency set, though no shader reads them. In The Great Ace Attorney Chronicles, three shared buffers, about 5 MB: chunks of its 1,000 MB pool (`DM_CTL` `restop`: shared buffers 241 MB at 30 s, 236 MB with the fix). → [Pools that hold shader code made resident to key the shaders](../reference/memory.md#pools-that-hold-shader-code-made-resident-to-key-the-shaders) diff --git a/docs/reference/memory.md b/docs/reference/memory.md index b5dc732908..00a0bfecbc 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -553,6 +553,14 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Open, a GPU hang reported elsewhere:** S2Arm, another project running Switch games on Apple hardware, saw Metal report `kIOGPUCommandBufferCallbackErrorHang` in 6 of 6 runs of Ocarina of Time on an M1 Pro (16 GB, macOS 27.0.1) when it punched its guest files during play, either each unmapped range in one call or 16 KB at a time from a background thread ([S2Arm #258](https://github.com/jhaizhou-ops/S2Arm/issues/258), [#305](https://github.com/jhaizhou-ops/S2Arm/pull/305)). The punching thread sat in `fcntl(F_PUNCHHOLE)` while Metal's submission thread waited in `IOGPUCommandQueueSubmitCommandBuffers`, and a 1-s pause after each large unmap made it go away; it now never punches. Not reproduced here (M4 Pro, 48 GB, macOS 27.2, Ocarina of Time idle on the Kokiri Forest deck save, 8 minutes a run). A probe wrote, fsynced and punched 16 MB of alias-region pages the game had not mapped, in the files it uses, every cycle: 36.5 GB in 2.39 million 16 KB punches in one run and 49 GB in 140,000 whole-run punches in another, with no command buffer error. Four runs with the sweep itself had none either, though the game gave memory back only while loading. Under the map lock the probe's 16 KB punches took up to 784 ms a cycle and whole runs up to 53 ms; the sweep punches runs (16–80 ms a sweep, once 162 ms for one 0.7 MB punch). Frames over 1 s came with and without punching. What may differ: the machine and the OS, and that S2Arm's hanging versions punched ranges the moment they were unmapped, where the sweep punches only pages given back a whole interval earlier (the heap's are punched at once). Not run on an iPad or an M1-class Mac; `DM_GUEST_FILE_PUNCH=0` turns punching off. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) on an iPad Pro (M5, iOS 27, 12.5 GB of guest files). The stop rule and the sweep have not run on an iPad yet. The GPU hang above was reported by S2Arm. +### The residency commit froze over the guest's files + +- **Symptom:** with the guest's memory in files, the game stops presenting for good while sound plays on: a black screen, or the last picture held. A `DM_CTL` `dump` shows the encode thread in the kernel under `res_commit ← gmem_map` (`iokit_user_client_trap` or `IOGPUResourceGroupUpdateResources`), the main fence completed, and at times guest threads at the same guest instruction, loads and stores, dump after dump. On the M1 iPad, Ocarina of Time froze at the title movie (frame 1614) in one run and walking in Link's house (frame ~4755) in another, with 7.6 GB of headroom and 1.6 GB resident: not memory pressure. The app cannot be started again until the device restarts: `devicectl device process launch --terminate-existing` hangs, the process gone from the list. +- **Cause:** a residency commit wires the pages of each buffer it adds, and pages a file in from the GPU driver's own wiring path. A sysdiagnose spindump during the freeze: the commit's thread runs `IOGPUFamily` → the VM → `apfs` pagein → `blocked by krwlock for reading owned by launchd [1]` (a thread of launchd's background I/O queue, base priority 4, boosted), and that launchd thread stopped 4 ms after it, inside APFS, and never ran again. What it waits for is not recorded, as for a lock held for reading, and the kernel had no symbols; that it stopped right after the commit's thread and stays stopped points to something the wiring holds. Ordinary page-ins of the same files (700,000 in a run) never hung, nor did guest threads that faulted only after the commit was stuck: they queued behind the APFS lock. Raw pool table pools are added in groups, a large pool as one whole-pool buffer (`gmem_map`), so a commit can page in hundreds of MB of file at once. +- **Fix:** general (`nvn_mtl.m` `file_wire`): a shared buffer whose pages are file-backed (`vm_region_recurse_64` reports an external pager) is wired with `mlock` before it joins the set, so the pages come in through the ordinary fault path and the commit finds them resident; the `mlock`s are given back right after the commit, which holds the pages for the GPU from then on. Anonymous memory and Metal's own buffers are not touched, so nothing changes with the guest anonymous (every Mac run by default). The residency line every 300 frames counts them: `file-backed buffers wired before their commit , , (refused )`; a refused `mlock` logs a red line once and leaves the commit to page them in as before. `t_file_wire` checks it. +- **Result on Ocarina of Time,** the M1 iPad, the same route that froze twice, driven over `DM_CTL` (from a save by Kokiri Forest into Link's house, camera turns and a walk): alive past frame 12,900, no red line; 20 fps in the house (the frame step's 50 ms, GPU 34–42 ms), headroom 5.3 GB, residency commits under 2 ms per 300 frames. The fix wired 26 to 3,729 buffers per 300 frames ahead of their commits, 2.1 GB in 6.7 s while the house loaded. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#memory) (v131072, 2026-11-08). On a Mac the guest is in files only with `DM_GUEST_FILE=1`; there the same commit showed in samples of a stall that lasted 7–37 s a frame (`IOGPUResourceGroupUpdateResources` under `_commitAddedAllocations`), and with this fix the same route with the files (16 GB M4 Air, idle) played through: 300-frame frame time p50 16.7 ms from the title to the walk's end (19,800 frames), no hitch dump, 3.9 GB wired at most. + ### Memory pools nothing gave an address to stay out of the raw pool table - **Symptom:** a Switch 2 game goes past Metal's recommended working set while its first 3D scene loads: `Insufficient Memory` command buffer errors, then a hung Mac. Ocarina of Time on a 16 GB Mac: 12 GB and more wired, once a kernel panic in `IOGPUGroupMemory.cpp` at 14.7 GB wired. From bfa12144623456d6801bcbaab8131264ea4fcdff Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 09:27:14 +0100 Subject: [PATCH 02/40] Where Metal's working set is under 8 GB, textures in GPU-only memory share bytes in a placement heap per 32 MB span, as on NVN: Ocarina of Time no longer runs out of GPU memory going outdoors on an M1 iPad With the render targets' recycled heaps (rh_texture, 7f502220) every live description in a GPU-only pool is a Metal texture of its own. On the M1 iPad (8 GB, recommended working set 5,461 MB) that reached 5,184 MB resident at the outdoor save load (544 heaps of 1,384 MB over 1,455 MB of textures), and every command buffer then failed with kIOGPUCommandBufferCallbackErrorOutOfMemory. window_texture (nvn_mtl.m): a texture whose storage lies in a CPU_NO_ACCESS pool (not sparse, not linear, one sample) is placed in a heap standing for a 32 MB span of guest addresses, at its NVN offset scaled by 9/8 and rounded to Metal's alignment, if its Metal size ends within its scaled NVN range, so textures overlap here where they overlap on NVN. A span's heap is made only once it saves memory; a texture across a span's end tries the half-offset spans; covered bytes are inherited, new ones zero-filled. Placed textures have no GPU-optimized contents: with lossless compression, another format's writes over the same bytes hung the iPad's GPU (kIOGPUCommandBufferCallbackErrorHang by Link's house). Anything that does not fit takes the usual path (image_texture, then rh_texture). On only where recommendedMaxWorkingSetSize is under 8 GB (window_wants): sharing a target pool's bytes corrupted the picture on the Mac in PR #1453 (the game keeps live targets in the part of the full-size scene target its current resolution leaves unused, which only NVN's layout makes safe), so larger devices keep each target's own layout. 8 GB devices have 5.3-5.5 GB of working set, a 16 GB M4 Mac 12,124 MB; the separate heaps needed 5.2 GB at the outdoor save load, sharing 4.3 GB. DM_NVN_WINDOW_ALIAS=0/1 forces it, and a log line says which way it went. The gfx test Makefile runs every test with it off unless the test turns it on, so an 8 GB Mac runs them as a 16 GB one does. Also for reading the set (res_top: the first GPU out-of-memory error, DM_CTL restop, and once when the set passes three quarters of the working set): the set by the source line that added each object, the resident pools by raw table / binding, by who gave an address in them (hle_nvn.c passes the kind: mtl_pool_addressed_by) and by pool flags (mtl_pool_tag), the GPU-made textures by pool, and the spans' counts. Tests: t_window_alias (new; 25 checks with the spans on, direct and deferred encoding, Metal validation; 8 fail with them off, the separate allocations), including the 8 GB rule. t_image_alias, t_rt_recycle, t_alias_takeover, t_upgrade_alias, t_copy_alias, t_rt_lifetime, t_pool_chunks, t_addr_known, t_file_wire pass. memory.md "Textures in GPU-only memory took an allocation per description"; rendering.md's recycled-heap entry links it; env_switches.tsv. Rebased onto main's pool and render-target structure: the virtual pool's runs are tagged after main's one-step mtl_pool_replace (vrun_set); the new pools' tag is set after pool_new_put; the storage range for res_top is kept on the texture before own_alias_publish (nvn_owner.m); a span's heap is an NvImageHeap with window set, which main's family-heap search (own_alias_at_locked) skips and which is freed with its last texture rather than waiting in rh_wait's free list. rendering.md keeps main's Fix and Check wording and adds the window heap. Rebased onto main 0c078130: res_add_at records the adding line in g_res_line beside main's queued change (res_q_put), and sets no g_res_dirty, which the queue replaced. --- dm/gfx/nvn/hle_nvn.c | 11 +- dm/gfx/nvn/nvn_mtl.h | 2 + dm/gfx/nvn/nvn_mtl.m | 218 ++++++++++++++++++++++++++++++++-- dm/gfx/tests/Makefile | 10 ++ dm/gfx/tests/t_window_alias.m | 104 ++++++++++++++++ docs/agents/symptoms.md | 1 + docs/env_switches.tsv | 1 + docs/reference/memory.md | 11 ++ docs/reference/rendering.md | 2 +- 9 files changed, 347 insertions(+), 13 deletions(-) create mode 100644 dm/gfx/tests/t_window_alias.m diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index a4d838a0d1..a4382731fc 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -260,7 +260,7 @@ NVN(nvnMemoryPoolInitialize, NVNboolean, NVNmemoryPool *p, const NVNmemoryPoolBu const pool_builder_t *s = (const pool_builder_t *)b; pool_t *o = (pool_t *)p; o->magic = MAGIC('P', 'O', 'O', 'L'); o->flags = s->flags; o->mem = s->mem; o->size = s->size; o->vp = NULL; o->vmode = 0; if (!o->mem && (o->flags & 0x200)) { uint8_t *vr = vpool_reserve(o); if (vr) o->mem = vr; } // a virtual pool's own address range until its first mapping decides (see MapVirtual) - else if (o->mem) { mtl_pool_add(o->mem, o->size); nvn_dirty_add_range(o->mem, o->size); } + else if (o->mem) { mtl_pool_add(o->mem, o->size); mtl_pool_tag(o->mem, (uint32_t)o->flags); nvn_dirty_add_range(o->mem, o->size); } if (o->mem && !(o->flags & 0x200)) dm_device_share(o->mem, o->size, 1); // the GPU maps the storage: the host refuses to unmap it until Finalize, as Horizon does (hle_svc.c) if (!o->mem && !(o->flags & 0x200)) LOG("[RED] MemoryPoolInitialize: non-virtual pool has no storage flags=%#x size=%#llx", o->flags, o->size); if (getenv("DM_NVN_VLOG")) { LOG("[nvn] MemoryPool %p mem=%p size=%#llx flags=%#x", (void *)p, (void *)o->mem, (unsigned long long)o->size, o->flags); void dm_backtrace(uint64_t fp); dm_backtrace((uint64_t)__builtin_frame_address(0)); } @@ -283,7 +283,7 @@ NVN(nvnMemoryPoolFlushMappedRange, void, const NVNmemoryPool *p, ptrdiff_t off, ft_flush(o->mem + off, n); } // A GPU address exists only once the driver has given one: here, by a buffer (nvnBufferInitialize) or by a texture (nvnTextureGetTextureAddress). Raw global memory can reach only what these name (mtl_pool_addressed) -static void pool_mark_addressed(const pool_t *o) { static __thread const uint8_t *last; if (o->mem && o->mem != last) { last = o->mem; mtl_pool_addressed(o->mem, o->size); } } +static void pool_mark_addressed(const pool_t *o) { static __thread const uint8_t *last; if (o->mem && o->mem != last) { last = o->mem; mtl_pool_addressed_by(o->mem, o->size, 0); } } NVN(nvnMemoryPoolGetBufferAddress, NVNbufferAddress, const NVNmemoryPool *p) { pool_mark_addressed((const pool_t *)p); return (NVNbufferAddress)((const pool_t *)p)->mem; } // ---------- Descriptor pools ---------- @@ -626,7 +626,7 @@ NVN(nvnTextureGetSamples, int, const NVNtexture *t) { return CTX(t)->d.samples; NVN(nvnTextureGetMemoryPool, NVNmemoryPool *, const NVNtexture *t) { return (NVNmemoryPool *)CTX(t)->pool; } NVN(nvnTextureGetMemoryOffset, ptrdiff_t, const NVNtexture *t) { return (ptrdiff_t)CTX(t)->off; } // The GPU address of the texture's storage (DRAGON QUEST XI S, Unreal Engine 4.18, asks for it at start): pool address + offset, the same address space as nvnMemoryPoolGetBufferAddress -NVN(nvnTextureGetTextureAddress, NVNtextureAddress, const NVNtexture *t) { const tex_t *o = CTX(t); if (o->pool) mtl_pool_addressed(o->pool->mem + o->off, o->L.size ? o->L.size : 1); return o->pool ? (NVNtextureAddress)(o->pool->mem + o->off) : 0; } +NVN(nvnTextureGetTextureAddress, NVNtextureAddress, const NVNtexture *t) { const tex_t *o = CTX(t); if (o->pool) mtl_pool_addressed_by(o->pool->mem + o->off, o->L.size ? o->L.size : 1, 2); return o->pool ? (NVNtextureAddress)(o->pool->mem + o->off) : 0; } // The size of a texture's ZCull save area (SaveZCullData / RestoreZCullData). The host does no ZCull and never touches the area, but games allocate it from their own pools, and NVN // does not report 0. Cuphead (Unity 2017.4) allocates one per depth target (main+0x1948060) from its pool allocator, which gives a zero-size request the offset of a live block; its // map of allocations then loses that block (the zero-size one's size overwrites it, and freeing the zero-size one deletes it), and the block's own free later pushes a garbage-sized @@ -2602,7 +2602,7 @@ NVN(nvnBufferInitialize, NVNboolean, NVNbuffer *buf, const NVNbufferBuilder *b) // compressing encoder (when the pause menu opens, and near a save commit): the buffer's one use is a texture-to-buffer copy of the picture's own extent, which lies inside the pool. Not an error by itself: one line. if ((uint64_t)bb->off + bb->size > bb->pool->size) ONCE_LOG("BufferInitialize: a buffer reaches past the end of its pool (offset %#llx size %#llx pool %#llx); only an access that leaves the pool is an error, and it is reported where it happens", (unsigned long long)bb->off, (unsigned long long)bb->size, (unsigned long long)bb->pool->size); *(buf_t *)buf = (buf_t){.magic = MAGIC('B', 'U', 'F', 0), .pool = bb->pool, .off = bb->off, .size = bb->size}; - pool_mark_addressed(bb->pool); + pool_mark_addressed(bb->pool); mtl_pool_addressed_by(bb->pool->mem + bb->off, bb->size ? bb->size : 1, 1); OLOG("BufferInitialize %p pool=%p off=%#llx size=%#llx", (void *)buf, (const void *)bb->pool, (unsigned long long)bb->off, (unsigned long long)bb->size); return 1; } @@ -2715,6 +2715,7 @@ static void vrun_set(struct vpool *v, uint64_t lo, uint64_t hi, int mapped) { } if (mapped) { nl[nn++] = (struct vrun){mlo, mhi}; add[na] = v->base + mlo; asz[na++] = mhi - mlo; } if (nd || na) mtl_pool_replace(del, nd, add, asz, na); + for (int i = 0; i < na; i++) mtl_pool_tag(add[i], (uint32_t)v->pool->flags | 0x80000000u); // res_top: a virtual pool's mapped run free(del); for (int a = 1; a < nn; a++) for (int b = a; b > 0 && nl[b].lo < nl[b - 1].lo; b--) { struct vrun t = nl[b]; nl[b] = nl[b - 1]; nl[b - 1] = t; } if (nn > v->nrun && nn >= 1024 && !(nn & (nn - 1))) LOG("[nvn] virtual pool %p: %d separate mapped runs", (const void *)v->pool, nn); @@ -2843,7 +2844,7 @@ NVN(nvnMemoryPoolMapVirtual, NVNboolean, NVNmemoryPool *p, int n, const NVNmappi if (o->mem && o->mem != base) { LOG("[RED] MapVirtual: non-contiguous mapping (true sparse) not implemented: virtual pool %p already maps to %p, this request wants %p", (void *)o, (void *)o->mem, (void *)base); return 0; } if (!o->mem) LOG("[nvn] MapVirtual: the storage of virtual pool %p (%#llx bytes) points at physical pool %p + %#llx", (void *)o, (unsigned long long)o->size, (const void *)ph, (unsigned long long)(r[i].poff - r[i].voff)); o->mem = base; - mtl_pool_addressed(o->mem, o->size); // an address may have been given out before it had storage + mtl_pool_addressed_by(o->mem, o->size, 3); // an address may have been given out before it had storage } return 1; } diff --git a/dm/gfx/nvn/nvn_mtl.h b/dm/gfx/nvn/nvn_mtl.h index fb9fb002d7..063f938dc2 100644 --- a/dm/gfx/nvn/nvn_mtl.h +++ b/dm/gfx/nvn/nvn_mtl.h @@ -15,6 +15,8 @@ void bf_want_note(const char *key); // Memory pool: [mem, mem+size) is expanded outward to page boundaries to build a zero-copy buffer; look up the buffer and offset by GPU address (= CPU address) void mtl_pool_add(const void *mem, uint64_t size); void mtl_pool_addressed(const void *mem, uint64_t size); // the game holds a GPU address in this range, so raw global memory may reach it +void mtl_pool_addressed_by(const void *mem, uint64_t size, int kind); // the same, with who gave it out (0 pool, 1 buffer: its range, the table takes the whole pool, 2 texture, 3 virtual mapping), for res_top +void mtl_pool_tag(const void *mem, uint32_t tag); // res_top: the pool's flags (bit 31: a mapped run of a virtual pool) void mtl_pool_del(const void *mem); void mtl_pool_replace(const void *const *del, int ndel, const void *const *add, const uint64_t *size, int nadd); // in one step for every lookup (a sparse pool's runs, hle_nvn.c vrun_set) void dm_gfx_addr_note(uint64_t a, char *note, size_t n); // what the graphics layer knows of an address (the crash dump, red lines) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 830f55d1e2..b16987aebe 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -280,6 +280,8 @@ static void ws_report(uint64_t frame, uint64_t frames) { } #include "nvn_rec.h" // deferred encoding (DM_NVN_ENC_THREAD) static NSHashTable *g_res_objects; +static char g_tex_range_key; // res_top: a GPU-made texture's storage (guest address, NVN size, pool flags) +static NSMapTable *g_res_line; // res_top: the source line that put each object in the set static _Atomic unsigned g_image_heap_n, g_big_n; // g_big_n: large pools in chunks, whose buffers overlap (BIG_MIN) static inline MTL4VisibilityOptions resource_vis(void) { return MTL4VisibilityOptionDevice | (g_image_heap_n || g_big_n ? MTL4VisibilityOptionResourceAlias : 0); } // Residency churn by call site (the power line every 300 frames, mtl_res_stats): each commit with changes is a kernel call whose cost grows with what was added or removed @@ -337,6 +339,8 @@ static void res_add_at(id allocation, int line) { @autoreleasepoo file_wire(r); // before its change is queued: the commit that applies the change takes this range with it and gives it back after (res_commit) res_q_put(r, 1); [g_res_objects addObject:r]; + if (!g_res_line) g_res_line = [NSMapTable weakToStrongObjectsMapTable]; + [g_res_line setObject:@(line) forKey:r]; pthread_mutex_unlock(&g_res_mu); } } static void res_remove(id r) { @autoreleasepool { @@ -388,9 +392,11 @@ @interface NvRecycledHeap : NSObject { @public id heap; } @end @implementation NvRecycledHeap - (void)dealloc { rh_wait(heap); } @end -@interface NvImageHeap : NSObject { @public id heap; } @end // a resolution family's heap (image_texture) +// A resolution family's heap (image_texture), or with window set a heap over a span of GPU-only memory (window_texture), whose textures sit at their own offsets: a span's +// heap goes with its last texture instead of waiting in the recycled list, where its 36 MB would serve only a texture at least two thirds as large +@interface NvImageHeap : NSObject { @public id heap; int window; } @end @implementation NvImageHeap -- (void)dealloc { rh_wait(heap); atomic_fetch_sub(&g_image_heap_n, 1); } +- (void)dealloc { if (window) res_remove(heap); else rh_wait(heap); atomic_fetch_sub(&g_image_heap_n, 1); } @end static char g_image_heap_owner_key, g_rh_owner_key; // A placement heap of at least `size` bytes: the smallest waiting one up to half as large again, or a new one an eighth larger (its exact size with the switch off). nil if Metal @@ -507,6 +513,7 @@ static void res_commit(void) { @autoreleasepool { static int res_pending(void) { pthread_mutex_lock(&g_res_mu); int n = g_res_qn != 0; pthread_mutex_unlock(&g_res_mu); return n; } // What the residency set holds: totals by kind and the largest objects. At the first GPU out-of-memory error (the error names none of them), and for DM_CTL "restop" (hle_hid.c). The set holds // the game's pools (shared buffers over guest memory), textures, heaps and staging alike +static void res_top_pools(const char *why); static void res_top(const char *why) { @autoreleasepool { enum { N = 24 }; id top[N] = {nil}; uint64_t total = 0, n = 0, kind[4] = {0}, count[4] = {0}; pthread_mutex_lock(&g_res_mu); @@ -518,6 +525,18 @@ static void res_top(const char *why) { @autoreleasepool { MLOG("%s: the residency set holds %llu objects, %llu MB (Metal's recommended working set %llu MB, GPU allocated %llu MB): textures %llu, %llu MB; heaps %llu, %llu MB; shared buffers %llu, %llu MB; private buffers %llu, %llu MB; the largest:", why, (unsigned long long)n, (unsigned long long)(total >> 20), (unsigned long long)(g_dev.recommendedMaxWorkingSetSize >> 20), (unsigned long long)(g_dev.currentAllocatedSize >> 20), (unsigned long long)count[0], (unsigned long long)(kind[0] >> 20), (unsigned long long)count[1], (unsigned long long)(kind[1] >> 20), (unsigned long long)count[2], (unsigned long long)(kind[2] >> 20), (unsigned long long)count[3], (unsigned long long)(kind[3] >> 20)); + { // by the source line that added each (res_add), textures and buffers apart + enum { NL = 24 }; struct { int line, tex; uint64_t n, b; } bl[NL]; int nl = 0; + pthread_mutex_lock(&g_res_mu); + for (id o in g_res_objects.allObjects) { int line = [[g_res_line objectForKey:o] intValue], tx = [(id)o conformsToProtocol:@protocol(MTLTexture)]; + int i = 0; while (i < nl && (bl[i].line != line || bl[i].tex != tx)) i++; if (i == nl && nl < NL) { bl[nl].line = line; bl[nl].tex = tx; bl[nl].n = bl[nl].b = 0; nl++; } + if (i < NL) { bl[i].n++; bl[i].b += o.allocatedSize; } } + pthread_mutex_unlock(&g_res_mu); + char line[1400] = ""; size_t lo = 0; + for (int i = 0; i < nl && lo < sizeof line; i++) lo += snprintf(line + lo, sizeof line - lo, "%sline %d %s %llu, %llu MB", i ? "; " : "", bl[i].line, bl[i].tex ? "textures" : "other", (unsigned long long)bl[i].n, (unsigned long long)(bl[i].b >> 20)); + MLOG("%s: residency by the line that added it: %s", why, line); + } + res_top_pools(why); for (int k = 0; k < N && top[k]; k++) { id o = top[k]; if ([o conformsToProtocol:@protocol(MTLTexture)]) { id t = o; MLOG(" texture %lux%lux%lu format %lu levels %lu%s: %llu MB", (unsigned long)t.width, (unsigned long)t.height, (unsigned long)MAX(t.depth, t.arrayLength), (unsigned long)t.pixelFormat, (unsigned long)t.mipmapLevelCount, t.isSparse ? " (sparse)" : "", (unsigned long long)(t.allocatedSize >> 20)); } else if ([o conformsToProtocol:@protocol(MTLHeap)]) MLOG(" heap: %llu MB", (unsigned long long)(((id)o).size >> 20)); @@ -549,7 +568,11 @@ void mtl_res_stats(char *out, size_t n) { if (g_fwire_n + g_fwire_fail) { o = strlen(out); if (o < n) snprintf(out + o, n - o, ", file-backed buffers wired before their commit %llu, %.0f MB, %.1f ms (refused %llu)", (unsigned long long)g_fwire_n, g_fwire_bytes / 1048576.0, g_fwire_ns / 1e6, (unsigned long long)g_fwire_fail); } g_fwire_n = g_fwire_bytes = g_fwire_ns = 0; memset(g_rs_site, 0, sizeof g_rs_site); g_rs_rm = g_rs_commits = g_rs_commit_ns = 0; + uint64_t held = g_res ? g_res.allocatedSize : 0; pthread_mutex_unlock(&g_res_mu); + // What the set is made of, once, when it first passes three quarters of Metal's working set: a device that later fails with GPU out-of-memory has said it before (the M1 iPad's + // 5,461 MB is passed going outside in Ocarina of Time), and one that does not still tells what to cut + static int told; if (!told && g_dev && held > g_dev.recommendedMaxWorkingSetSize / 4 * 3) { told = 1; res_top("[mem] the residency set passed three quarters of Metal's working set"); } } @interface NvDeferredRelease : NSObject { @public id object; uint64_t fence; int keep_resident; } @end @@ -670,7 +693,7 @@ static void want_note(want_list *w, const char *key) { void bf_want_note(const char *key) { want_note(&g_bf_want, key); } // ---------- Memory pools ---------- -typedef struct { uintptr_t lo, hi; uintptr_t base; __unsafe_unretained id buf; uint64_t gpu; const void *key; int res; uint64_t born; uintptr_t end; uint64_t unk; uint32_t n_tex_reads, n_other; } pool_ent; // res: already in the residency set (on demand, see pool_res); born: when it was added (ns); end, unk: a large pool's chunk (see BIG_MIN); n_tex_reads / n_other: lookups by texture uploads / by anything else (mem_census) +typedef struct { uintptr_t lo, hi; uintptr_t base; __unsafe_unretained id buf; uint64_t gpu; const void *key; int res; uint64_t born; uintptr_t end; uint64_t unk; uint32_t n_tex_reads, n_other; uint32_t tag; } pool_ent; // res: already in the residency set (on demand, see pool_res); born: when it was added (ns); end, unk: a large pool's chunk (see BIG_MIN); n_tex_reads / n_other: lookups by texture uploads / by anything else (mem_census) static pool_ent *g_pools; static int g_npool, g_cap; static int g_pidx_ok, g_plast = -1; // whether the address→pool index is valid, the last hit (pool_find) static NSMutableArray *g_keep; // Holds the buffer references (pool_ent has non-owning pointers) @@ -890,6 +913,26 @@ void mtl_pool_addressed(const void *mem, uint64_t size) { } pthread_mutex_unlock(&g_mu); } +// Diagnostic (res_top): the addressed ranges again, by who gave the address out: 0 a pool's address (nvnMemoryPoolGetBufferAddress), 1 a buffer (its own range, though the +// table above takes its whole pool), 2 a texture (nvnTextureGetTextureAddress), 3 a virtual pool's mapping. Under g_mu +typedef struct { struct { uintptr_t lo, hi; } *r; int n, cap; } rng_tab; +static rng_tab g_addrk[4]; +static void rng_put(rng_tab *t, uintptr_t lo, uintptr_t hi) { + int a = 0, b = t->n; while (a < b) { int m = (a + b) / 2; if (t->r[m].hi < lo) a = m + 1; else b = m; } + int j = a; while (j < t->n && t->r[j].lo <= hi) { lo = MIN(lo, t->r[j].lo); hi = MAX(hi, t->r[j].hi); j++; } + if (j == a && t->n == t->cap) { t->cap = t->cap ? t->cap * 2 : 256; t->r = realloc_or_die(t->r, (size_t)t->cap * sizeof *t->r, "the addressed range table by kind"); } + if (j != a + 1) memmove(t->r + a + 1, t->r + j, (size_t)(t->n - j) * sizeof *t->r); + t->n += 1 - (j - a); t->r[a].lo = lo; t->r[a].hi = hi; +} +static uint64_t rng_overlap(const rng_tab *t, uintptr_t lo, uintptr_t hi) { + int a = 0, b = t->n; while (a < b) { int m = (a + b) / 2; if (t->r[m].hi <= lo) a = m + 1; else b = m; } + uint64_t z = 0; for (int i = a; i < t->n && t->r[i].lo < hi; i++) z += MIN(hi, t->r[i].hi) - MAX(lo, t->r[i].lo); + return z; +} +void mtl_pool_addressed_by(const void *mem, uint64_t size, int kind) { + if (kind != 1) mtl_pool_addressed(mem, size); + pthread_mutex_lock(&g_mu); if (kind >= 0 && kind < 4) rng_put(&g_addrk[kind], (uintptr_t)mem, (uintptr_t)mem + size); pthread_mutex_unlock(&g_mu); +} // On-demand residency of memory pools: putting a zero-copy buffer into the residency set makes iOS read the whole pool into memory and pin it as wired, even a pool the game never uses // (they measured on a 17 Pro Max a 512 MB render-target pool addressed 0 times in a run, with about 530 MB more device wired memory; a 4 GB A14 was ended by the system within seconds of launch because of it). // So a pool is not put into the residency set when created; it is put in, and the residency set committed on the spot, the first time the GPU gets its address through mtl_addr / mtl_gpu_addr -- these are the only two ways the GPU gets pool addresses, @@ -905,10 +948,64 @@ static void pool_res(int i) { // pools' buffers do at shared pages; GPU addresses are not contiguous across them, so a lookup must say how far it reads. One that does not know (length 1: an indirect draw's index buffer, a raw storage // address) gets BIG_UNKNOWN past its address. Raw global memory, whose pool table the shader walks by itself, takes the whole pool: big_full enum { BIG_MIN = 512u << 20, BIG_S = 16u << 20, BIG_UNKNOWN = 64u << 20, BIG_MAX = 32 }; -typedef struct { uintptr_t lo, hi, a, b; const void *mem; int *chunk, full; NSMutableArray *old; } big_pool; // chunk: stride → g_pools index + 1; full: the whole pool's entry + 1; old: buffers a longer chunk replaced, alive with the pool +typedef struct { uintptr_t lo, hi, a, b; const void *mem; int *chunk, full; NSMutableArray *old; uint32_t tag; } big_pool; // chunk: stride → g_pools index + 1; full: the whole pool's entry + 1; old: buffers a longer chunk replaced, alive with the pool static big_pool g_big[BIG_MAX]; static int g_nbig; static id pool_buf(uintptr_t a, uintptr_t b); // a pool's zero-copy buffer over [a, b), made over writable pages (below) static int big_of(uint64_t addr) { for (int i = 0; i < g_nbig; i++) if (addr >= g_big[i].lo && addr < g_big[i].hi) return i; return -1; } +// Diagnostic (res_top): the pool's flags, or a virtual pool's for one of its mapped runs (bit 31) +void mtl_pool_tag(const void *mem, uint32_t tag) { + pthread_mutex_lock(&g_mu); + for (int i = 0; i < g_npool; i++) if (g_pools[i].key == mem) g_pools[i].tag = tag; + for (int k = 0; k < g_nbig; k++) if (g_big[k].mem == mem) g_big[k].tag = tag; + pthread_mutex_unlock(&g_mu); +} +static _Atomic uint64_t g_win_n, g_win_bytes, g_win_fit[4], g_win_fit_b[4]; // textures placed / bytes; not placed by reason (0 no size, 1 Metal larger than its scaled NVN range, 2 past the window, 3 heap refused) +static void res_top_pools(const char *why) { + { // the memory pools in it: in the raw pool table or not, looked up by anything but texture uploads or not, by who gave an address in them, by pool flags + uint64_t cn[4] = {0}, cb[4] = {0}, mn[16] = {0}, mb[16] = {0}, mcov[16][4] = {{0}}, vn = 0, vb = 0, rn = 0, rb = 0; struct { uint32_t tag; uint64_t n, b, ln, lb; } tg[12]; int ntg = 0; + pthread_mutex_lock(&g_mu); + for (int i = 0; i < g_npool; i++) { pool_ent *p = &g_pools[i]; if (!p->res) continue; + uint64_t z = p->hi - p->lo; int raw = addr_meets(p->lo, p->hi), c = raw << 1 | (p->n_other != 0); cn[c]++; cb[c] += z; rn++; rb += z; + int m = 0; uint64_t cov[4]; for (int k = 0; k < 4; k++) { cov[k] = rng_overlap(&g_addrk[k], p->lo, p->hi); if (cov[k]) m |= 1 << k; } + mn[m]++; mb[m] += z; for (int k = 0; k < 4; k++) mcov[m][k] += cov[k]; + if (p->tag >> 31) { vn++; vb += z; } + int j = 0; while (j < ntg && tg[j].tag != p->tag) j++; if (j == ntg && ntg < 12) { tg[ntg].tag = p->tag; tg[ntg].n = tg[ntg].b = tg[ntg].ln = tg[ntg].lb = 0; ntg++; } if (j < 12) { tg[j].n++; tg[j].b += z; if (p->n_other) { tg[j].ln++; tg[j].lb += z; } } } + uint64_t ak[4] = {0}; for (int k = 0; k < 4; k++) for (int i = 0; i < g_addrk[k].n; i++) ak[k] += g_addrk[k].r[i].hi - g_addrk[k].r[i].lo; + pthread_mutex_unlock(&g_mu); + MLOG("%s: resident memory pools %llu, %llu MB: in the raw pool table and looked up otherwise %llu, %llu MB; in the table only %llu, %llu MB; not in it, looked up %llu, %llu MB; neither %llu, %llu MB; mapped runs of virtual pools %llu, %llu MB; addressed in all (MB): pools %llu, buffers %llu, textures %llu, virtual mappings %llu", why, + (unsigned long long)rn, (unsigned long long)(rb >> 20), (unsigned long long)cn[3], (unsigned long long)(cb[3] >> 20), (unsigned long long)cn[2], (unsigned long long)(cb[2] >> 20), (unsigned long long)cn[1], (unsigned long long)(cb[1] >> 20), (unsigned long long)cn[0], (unsigned long long)(cb[0] >> 20), + (unsigned long long)vn, (unsigned long long)(vb >> 20), (unsigned long long)(ak[0] >> 20), (unsigned long long)(ak[1] >> 20), (unsigned long long)(ak[2] >> 20), (unsigned long long)(ak[3] >> 20)); + char line[1400] = ""; size_t lo = 0; + for (int m = 0; m < 16 && lo < sizeof line; m++) if (mn[m]) lo += snprintf(line + lo, sizeof line - lo, "%s[%s%s%s%s] %llu, %llu MB (buffers' ranges %llu MB, textures' %llu MB)", lo ? "; " : "", m & 1 ? "P" : "", m & 2 ? "B" : "", m & 4 ? "T" : "", m & 8 ? "V" : "", (unsigned long long)mn[m], (unsigned long long)(mb[m] >> 20), (unsigned long long)(mcov[m][1] >> 20), (unsigned long long)(mcov[m][2] >> 20)); + MLOG("%s: resident memory pools by who gave an address in them (P pool, B buffer, T texture, V virtual mapping): %s", why, line); + { // GPU-made textures in the set: by the memory pool holding their storage, NVN bytes summed against the union of their storage (what aliasing as on NVN would keep) + enum { NQ = 48 }; struct { const void *key; uint32_t flags; uint64_t n, metal, nvn, uni, psize; NSMutableArray *r; } q[NQ]; int nq = 0; + NSArray *objs; pthread_mutex_lock(&g_res_mu); objs = g_res_objects.allObjects; pthread_mutex_unlock(&g_res_mu); + pthread_mutex_lock(&g_mu); + for (id o in objs) { if (![o conformsToProtocol:@protocol(MTLTexture)]) continue; NSArray *rg = objc_getAssociatedObject(o, &g_tex_range_key); if (!rg) continue; + uint64_t a = [rg[0] unsignedLongLongValue], z = [rg[1] unsignedLongLongValue]; uint32_t fl = [rg[2] unsignedIntValue]; + const void *key = NULL; uint64_t ps = 0; int b = big_of(a); + if (b >= 0) { key = g_big[b].mem; ps = g_big[b].hi - g_big[b].lo; } else for (int i = 0; i < g_npool; i++) if (a >= g_pools[i].lo && a < g_pools[i].hi) { key = g_pools[i].key; ps = g_pools[i].hi - g_pools[i].lo; break; } + int j = 0; while (j < nq && (q[j].key != key || q[j].flags != fl)) j++; + if (j == nq && nq < NQ) { q[nq].key = key; q[nq].flags = fl; q[nq].n = q[nq].metal = q[nq].nvn = q[nq].uni = 0; q[nq].psize = ps; q[nq].r = [NSMutableArray array]; nq++; } + if (j < NQ) { q[j].n++; q[j].metal += ((id)o).allocatedSize; q[j].nvn += z; [q[j].r addObject:@[@(a), @(a + z)]]; } } + pthread_mutex_unlock(&g_mu); + for (int a = 0; a < nq; a++) for (int b = a + 1; b < nq; b++) if (q[b].metal > q[a].metal) { __typeof__(q[0]) x = q[a]; q[a] = q[b]; q[b] = x; } + lo = 0; + for (int j = 0; j < nq && j < 12 && lo < sizeof line; j++) { + [q[j].r sortUsingComparator:^NSComparisonResult(NSArray *x, NSArray *y) { return [x[0] compare:y[0]]; }]; + uint64_t end = 0; for (NSArray *r in q[j].r) { uint64_t a = [r[0] unsignedLongLongValue], e = [r[1] unsignedLongLongValue]; if (e <= end) continue; q[j].uni += e - MAX(a, end); end = e; } + lo += snprintf(line + lo, sizeof line - lo, "%spool %p (%llu MB) texture pool flags %#x: %llu textures, Metal %llu MB, NVN %llu MB, union %llu MB", j ? "; " : "", q[j].key, (unsigned long long)(q[j].psize >> 20), q[j].flags, + (unsigned long long)q[j].n, (unsigned long long)(q[j].metal >> 20), (unsigned long long)(q[j].nvn >> 20), (unsigned long long)(q[j].uni >> 20)); } + MLOG("%s: GPU-made textures in the set by the pool that holds their storage: %s", why, line); + MLOG("%s: GPU-only memory windows (window_texture): %llu textures placed, %llu MB; kept apart: no size %llu (%llu MB), Metal larger than 9/8 of NVN %llu (%llu MB), past the window %llu (%llu MB), heap refused %llu", why, + (unsigned long long)g_win_n, (unsigned long long)(g_win_bytes >> 20), (unsigned long long)g_win_fit[0], (unsigned long long)(g_win_fit_b[0] >> 20), (unsigned long long)g_win_fit[1], (unsigned long long)(g_win_fit_b[1] >> 20), + (unsigned long long)g_win_fit[2], (unsigned long long)(g_win_fit_b[2] >> 20), (unsigned long long)g_win_fit[3]); + } + lo = 0; for (int j = 0; j < ntg && lo < sizeof line; j++) lo += snprintf(line + lo, sizeof line - lo, "%s%#x: %llu, %llu MB (looked up otherwise %llu, %llu MB)", j ? "; " : "", tg[j].tag, (unsigned long long)tg[j].n, (unsigned long long)(tg[j].b >> 20), (unsigned long long)tg[j].ln, (unsigned long long)(tg[j].lb >> 20)); + MLOG("%s: resident memory pools by flags (bit 31: a virtual pool's mapped run): %s", why, line); + } +} // A pool to put into the table: its zero-copy buffer is made first, outside g_mu (pool_new_prep; it may retry for seconds), and the entry goes in under g_mu (pool_new_put), so a change // of several pools can be one step for every lookup (mtl_pool_replace) typedef struct { const void *mem; uint64_t size; uintptr_t lo, hi, a, b; __strong id buf; } pool_new; // buf nil: in chunks (a large pool) @@ -1037,7 +1134,7 @@ static int big_new(big_pool *b, uintptr_t lo, uintptr_t hi, uintptr_t end, uint6 if (!buf) { MLOG("[RED] memory pool %p: the zero-copy buffer over %#lx..%#lx was refused", b->mem, (unsigned long)lo, (unsigned long)end); return -1; } if (g_npool == g_cap) { g_cap = g_cap ? g_cap * 2 : 64; g_pools = realloc_or_die(g_pools, g_cap * sizeof *g_pools, "the memory pool table"); } g_pidx_ok = 0; atomic_fetch_add(&g_pool_gen, 1); - g_pools[g_npool++] = (pool_ent){lo < b->lo ? b->lo : lo, hi > b->hi ? b->hi : hi, lo, buf, buf.gpuAddress, b->mem, 0, clock_gettime_nsec_np(CLOCK_MONOTONIC), end > b->hi ? b->hi : end, unk}; + g_pools[g_npool++] = (pool_ent){lo < b->lo ? b->lo : lo, hi > b->hi ? b->hi : hi, lo, buf, buf.gpuAddress, b->mem, 0, clock_gettime_nsec_np(CLOCK_MONOTONIC), end > b->hi ? b->hi : end, unk, 0, 0, b->tag}; [g_keep addObject:buf]; return g_npool - 1; } @@ -4341,6 +4438,109 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { if (r != (unsigned)pk->vbase) { static int nl; if (nl++ < 6) MLOG("sparse texture ser=%llu %ux%u: sampled from level %u of %u (finer levels not mapped)", (unsigned long long)t->serial, t->d.width, t->d.height, r, n); } pk->vbase = (int)r; } +// GPU-only memory as on NVN: textures whose storage lies in a CPU_NO_ACCESS pool are placed in a placement heap that stands for an aligned span (window) of guest addresses, each at +// its own address's offset in it, so textures that share bytes on NVN share them here and the GPU memory they take is bounded by the spans used, not by the descriptions made. +// Ocarina of Time keeps its render targets in virtual pools (flags 0x2a1) over one 305 MB physical pool and describes the same bytes as targets of other sizes and formats from +// pass to pass and as its dynamic resolution moves: walking in Link's house, 400 Metal textures of 793 MB stood for 89 MB of NVN storage, and on an 8 GB iPad the residency set +// passed the GPU's working set going outside (docs/reference/memory.md). A texture goes there only if its Metal size fits within its NVN size and its offset meets Metal's alignment, +// so textures that do not overlap on NVN do not overlap here; any other keeps its own allocation (image_texture, then a texture of its own). A window heap lives while one of its +// textures does, and is made only once it saves memory: when a texture shares bytes with one made apart over its span, or those pass its size. Its memory starts as zeros: a texture over bytes no earlier texture of the window covered is zero-filled as before; one over covered bytes inherits them, as on NVN, +// and is not filled, as a fill would wipe a live alias in the earlier upload batch +enum { WIN_SHIFT = 25, WIN_SIZE = 1u << WIN_SHIFT }; // 32 MB spans +// Metal's size for a texture can pass NVN's (an RGBA8 1024x1024 target: 4,227,072 bytes against 4,194,304, its compression metadata), so the heap is the span scaled by 9/8 and a +// texture sits at its NVN offset scaled the same way (rounded up to Metal's alignment): it fits if it ends within its scaled NVN range +#define WIN_SCALE(x) ((uint64_t)(x) * 9 / 8) +#define WIN_HEAP ((NSUInteger)WIN_SCALE(WIN_SIZE)) +static NSMapTable *g_win; // window index → NvImageHeap (weak), under g_win_mu +static pthread_mutex_t g_win_mu = PTHREAD_MUTEX_INITIALIZER; +static char g_win_cov_key; // the owner's covered ranges (NSMutableData of rng_tab entries; under g_win_mu) +// On only where Metal's working set is too small for the render targets' separate heaps (rh_texture), which keep each texture's own layout and are the default elsewhere: a game +// that packs live targets into the part of a full-size target its current resolution leaves unused relies on NVN's layout (PR #1453), so sharing is the small device's choice, not +// everyone's. The numbers, Ocarina of Time on the M1 iPad (8 GB, working set 5,461 MB): with the separate heaps the residency set reached 5,184 MB at the outdoor save load (544 +// heaps, 1,384 MB, over 1,455 MB of textures) and every command buffer then failed out of memory; sharing held 4.3 GB outdoors for 30 minutes. So the separate heaps need about +// 5.2 GB plus what play outdoors adds (the set grew by up to 950 MB over a two-minute walk before sharing): under 8 GB of working set (8 GB devices: 5.3-5.5 GB) sharing is on; +// a 16 GB Mac (12,124 MB on an M4) keeps the separate heaps. DM_NVN_WINDOW_ALIAS=0 / 1 forces it off / on +enum { WIN_AUTO_BELOW_MB = 8192 }; +static int window_wants(uint64_t working_set) { return working_set && working_set < ((uint64_t)WIN_AUTO_BELOW_MB << 20); } +static int window_on(void) { + static int v = -1; + if (v < 0) { + const char *e = getenv("DM_NVN_WINDOW_ALIAS"); uint64_t ws = g_dev ? g_dev.recommendedMaxWorkingSetSize : 0; + v = e && *e ? atoi(e) > 0 : window_wants(ws); + MLOG("textures in GPU-only memory %s (window_texture; Metal's working set %llu MB, %s)", v ? "share bytes in a heap per 32 MB span" : "keep allocations of their own", + (unsigned long long)(ws >> 20), e && *e ? "DM_NVN_WINDOW_ALIAS" : v ? "under 8192 MB" : "8192 MB or more"); + } + return v; +} +static int window_ok(const tex_t *t) { + return t && t->pool && t->pool->mem && t->mem && (t->pool->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS) && !nvn_sparse_pool(t->pool) && !t->has_layout && !t->d.stride + && !(t->d.flags & IMAGE_NOT_SHARED) && t->d.samples <= 1 && t->L.size && window_on(); +} +// Eligible textures made apart while a span has no heap: their bytes, by span, so a heap is made only once it saves memory (a span with a few small targets keeps them as they are) +static NSMutableDictionary *g_win_live; // under g_win_mu +static NSMutableDictionary *g_win_apart; // span → its textures made apart (weak NvWinLive), under g_win_mu +@interface NvWinLive : NSObject { @public uint64_t win, size; uintptr_t lo, hi; } @end +@implementation NvWinLive +- (void)dealloc { pthread_mutex_lock(&g_win_mu); NSNumber *k = @(win); uint64_t v = [g_win_live[k] unsignedLongLongValue]; v = v > size ? v - size : 0; if (v) g_win_live[k] = @(v); else [g_win_live removeObjectForKey:k]; pthread_mutex_unlock(&g_win_mu); } +@end +static char g_win_live_key; +static void window_note(id mt, const tex_t *t, uint64_t win, uint64_t size) { // mt was made apart over span win: count it until it goes + if (!mt || !size) return; + NvWinLive *l = [NvWinLive new]; l->win = win; l->size = size; l->lo = (uintptr_t)t->mem; l->hi = l->lo + t->L.size; + pthread_mutex_lock(&g_win_mu); if (!g_win_live) { g_win_live = [NSMutableDictionary dictionary]; g_win_apart = [NSMutableDictionary dictionary]; } + g_win_live[@(win)] = @([g_win_live[@(win)] unsignedLongLongValue] + size); + NSHashTable *ap = g_win_apart[@(win)]; if (!ap) g_win_apart[@(win)] = ap = [NSHashTable weakObjectsHashTable]; [ap addObject:l]; + pthread_mutex_unlock(&g_win_mu); + objc_setAssociatedObject(mt, &g_win_live_key, l, OBJC_ASSOCIATION_RETAIN_NONATOMIC); +} +// → the texture placed in its span's heap, or nil: then *note_size, if set, is what the caller notes (window_note) for the texture it makes apart +static id window_texture(const tex_t *t, MTLTextureDescriptor *gd, int *fill, uint64_t *note_win, uint64_t *note_size) { + *fill = 1; *note_size = 0; + if (!window_ok(t) || gd.storageMode != MTLStorageModePrivate) return nil; + // Without GPU-optimized contents. Textures that overlap here read each other's bytes, which on NVN mean the same thing (one block-linear layout) and in Metal do not: another + // description's writes leave a lossless-compressed texture's metadata invalid, and reading it is undefined. On the M1 iPad Ocarina of Time's GPU hung that way + // (kIOGPUCommandBufferCallbackErrorHang, a white flash, then GPU access revoked) turning the camera by Link's house, where its passes load targets that another format + // wrote over the same bytes in the same frame. Plain contents read as wrong values, as NVN's would after another format, but nothing the GPU can fail on + MTLTextureDescriptor *d = [gd copy]; d.allowGPUOptimizedContents = NO; + MTLSizeAndAlign need = [g_dev heapTextureSizeAndAlignWithDescriptor:d]; + // A span is keyed by its index times 2; one across a span's end tries the spans offset by half a span (odd keys). Textures in different heaps do not share bytes, as before + uintptr_t a = (uintptr_t)t->mem, rel = 0; uint64_t win = 0, lo = 0; int why = 0; + for (int half = 0; half < 2; half++) { + uintptr_t b = a - (half ? WIN_SIZE / 2 : 0); rel = b & (WIN_SIZE - 1); win = (uint64_t)(b >> WIN_SHIFT) << 1 | (uint64_t)half; + lo = need.align ? (WIN_SCALE(rel) + need.align - 1) / need.align * need.align : 0; // its NVN range scaled: containment and overlaps as on NVN + why = !need.size || !need.align ? 0 : lo + need.size > WIN_SCALE(rel + t->L.size) ? 1 : lo + need.size > WIN_HEAP ? 2 : -1; + if (why != 2) break; + } + if (why >= 0) { atomic_fetch_add(&g_win_fit[why], 1); atomic_fetch_add(&g_win_fit_b[why], need.size); return nil; } + pthread_mutex_lock(&g_win_mu); + if (!g_win) g_win = [NSMapTable strongToWeakObjectsMapTable]; + NvImageHeap *owner = [g_win objectForKey:@(win)]; + if (!owner && [g_win_live[@(win)] unsignedLongLongValue] + need.size <= WIN_HEAP) { // apart, until a heap would save memory: the span's live textures pass its size, or this one shares bytes with one of them + int shares = 0; for (NvWinLive *l in g_win_apart[@(win)]) if (l->lo < a + t->L.size && a < l->hi) { shares = 1; break; } + if (!shares) { pthread_mutex_unlock(&g_win_mu); *note_win = win; *note_size = need.size; return nil; } + } + if (!owner) { + MTLHeapDescriptor *hd = [MTLHeapDescriptor new]; hd.type = MTLHeapTypePlacement; hd.size = WIN_HEAP; + hd.storageMode = MTLStorageModePrivate; hd.cpuCacheMode = d.cpuCacheMode; hd.hazardTrackingMode = d.hazardTrackingMode; + id heap = [g_dev newHeapWithDescriptor:hd]; + if (!heap) { pthread_mutex_unlock(&g_win_mu); atomic_fetch_add(&g_win_fit[3], 1); atomic_fetch_add(&g_win_fit_b[3], need.size); return nil; } + owner = [NvImageHeap new]; owner->heap = heap; owner->window = 1; atomic_fetch_add(&g_image_heap_n, 1); + objc_setAssociatedObject(owner, &g_win_cov_key, [NSMutableData data], OBJC_ASSOCIATION_RETAIN_NONATOMIC); + [g_win setObject:owner forKey:@(win)]; + static int nl; if (nl++ < 8) MLOG("GPU-only memory %#llx..%#llx: textures there share a %lu MB placement heap at their own offsets (window_texture)", (unsigned long long)(((win >> 1) << WIN_SHIFT) + (win & 1) * WIN_SIZE / 2), (unsigned long long)(((win >> 1) << WIN_SHIFT) + (win & 1) * WIN_SIZE / 2 + WIN_SIZE), (unsigned long)(WIN_HEAP >> 20)); + } + id mt = [owner->heap newTextureWithDescriptor:d offset:lo]; + if (mt) { // covered before: inherit the bytes; else zero-fill (fill stays 1) + NSMutableData *cov = objc_getAssociatedObject(owner, &g_win_cov_key); rng_tab tab = {cov.mutableBytes, (int)(cov.length / sizeof *tab.r), (int)(cov.length / sizeof *tab.r)}; + if (rng_overlap(&tab, lo, lo + need.size)) *fill = 0; + rng_tab grown = {NULL, 0, 0}; for (int i = 0; i < tab.n; i++) rng_put(&grown, tab.r[i].lo, tab.r[i].hi); rng_put(&grown, lo, lo + need.size); + [cov setLength:0]; if (grown.n) [cov appendBytes:grown.r length:(size_t)grown.n * sizeof *grown.r]; free(grown.r); + objc_setAssociatedObject(mt, &g_image_heap_owner_key, owner, OBJC_ASSOCIATION_RETAIN_NONATOMIC); + atomic_fetch_add(&g_win_n, 1); atomic_fetch_add(&g_win_bytes, need.size); + } + pthread_mutex_unlock(&g_win_mu); + return mt; +} static uint64_t g_texinfo_ser[8]; static int g_texinfo_n; // Distinct compressible image descriptions at one NVN address alias physical memory, even while all their descriptor objects are live (LM3's resolution families). // Single-description storage keeps the ordinary allocation. Families share an exact-sized placement heap, found through the existing alias table rather than a pool-lifetime owner. @@ -4363,7 +4563,7 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { NvImageHeap *o = objc_getAssociatedObject(other, &g_image_heap_owner_key); // Metal omits ShaderAtomic from usage; tex_desc gives every writable R32 integer image that capability. MTLTextureUsage usage = other.usage | (tex_atomic_capable(other) ? MTLTextureUsageShaderAtomic : 0); - if (o && o->heap.size >= need.size && usage == d.usage && (!owner || o->heap.size < owner->heap.size)) owner = o; + if (o && !o->window && o->heap.size >= need.size && usage == d.usage && (!owner || o->heap.size < owner->heap.size)) owner = o; }); if (!owner) for (uint32_t i = 0; i < g_image_desc.cap; i++) if (g_image_desc.vals[i]) { const tex_t *other = g_image_desc.vals[i]; @@ -4586,9 +4786,12 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { if (!mt) { MTLTextureDescriptor *d = tex_desc(t, 0); if (!d) return nil; - mt = image_texture(t, d); + int wfill = 1; uint64_t wwin = 0, wsize = 0; mt = window_texture(t, d, &wfill, &wwin, &wsize); // on a small device only (window_on) + if (mt) { res_add(mt); if (wfill && !tex_init_fill(t, mt)) tex_zero_fill(mt); } + if (!mt) mt = image_texture(t, d); if (!mt) { mt = rh_texture(t, d); if (!mt) mt = [g_dev newTextureWithDescriptor:d]; if (mt) { res_add(mt); if (!tex_init_fill(t, mt)) tex_zero_fill(mt); } } + if (mt && wsize) window_note(mt, t, wwin, wsize); if (!mt) { RED_ONCE(2, "failed to create MTLTexture %ux%u format %#x target %d", t->d.width, t->d.height, t->d.format, t->d.target); return nil; } } if (mt.allocatedSize >= (256u << 20)) { static int nb; if (nb++ < 20) // the textures that take most of the GPU's memory (a GPU out-of-memory error says nothing of which) @@ -4597,6 +4800,7 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { if (g_allocprof > 0) { static int n; if (n++ < 30 || n % 500 == 0) MLOG("new texture (%s) #%d ser=%llu %ux%ux%u format %#x target %d levels %u storage %p", __func__, n, (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, (void *)t->mem); } if (getenv("DM_NVN_TEXCPU") && t->pool && !(t->pool->flags & NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS)) MLOG("non-swizzled texture on a CPU-visible pool ser=%llu %ux%ux%u format %#x target %d levels %u texture flags %#x pool flags %#x storage %p", (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, t->d.flags, t->pool->flags, (void *)t->mem); res_add(mt); + if (t->pool && t->mem) objc_setAssociatedObject(mt, &g_tex_range_key, @[@((uint64_t)(uintptr_t)t->mem), @(t->L.size), @(t->pool->flags)], OBJC_ASSOCIATION_RETAIN_NONATOMIC); // res_top: its storage // The takeover of the storage, the entry and the serial's publication in one hold (nvn_owner.m). A texture another thread registered under the key since the lookup above is the // one the storage's serials share: this one is retired id got = own_alias_publish(t->serial, &ak, mt); diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 251e26d299..697f329682 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -25,6 +25,9 @@ host_msl_force: FW=-framework Metal -framework MetalFX -framework Foundation -framework QuartzCore -framework ImageIO -framework CoreGraphics -framework UniformTypeIdentifiers # Host crash.c is linked in as well: the write-protect unit tests go through the same page-fault hook as production # Tests that #include nvn_mtl.m link the rest of the NVN layer beside it (hle_nvn.c: memory pools, sparse pools, serials) and the host's large stack, with weak stubs of what they need of the host (t_host_stubs.c) +# Textures in GPU-only memory sharing a heap per span (nvn_mtl.m window_on) turn on by themselves where Metal's working set is under 8 GB: the tests run as on a larger Mac +# wherever they run, and t_window_alias turns the spans on itself +export DM_NVN_WINDOW_ALIAS ?= 0 MTL_T_DEP=../nvn/hle_nvn.c ../../host/crash.c ../../host/hle_bigstack.c t_host_stubs.c $(GFX) $(GFX_H) MTL_T_SRC=../nvn/hle_nvn.c ../../host/crash.c ../../host/hle_bigstack.c t_host_stubs.c $(filter-out ../nvn/nvn_mtl.m,$(GFX)) # The run list: every t_*.c and t_*.m but those in NOT_RUN. `make run` builds them all (in parallel with -j), then runs them one after another, each through its check_ @@ -503,9 +506,16 @@ check_t_partial_upload: RUN_ENV = MTL_DEBUG_LAYER=1 t_srgb_alias: t_srgb_alias.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_srgb_alias.m $(MTL_T_SRC) $(FW) check_t_srgb_alias: RUN_ENV = MTL_DEBUG_LAYER=1 +t_window_alias: t_window_alias.m $(MTL_T_DEP) + clang $(CFLAGS) -fobjc-arc -o $@ t_window_alias.m $(MTL_T_SRC) $(FW) +.PHONY: check_t_window_alias +check_t_window_alias: t_window_alias + DM_NVN_WINDOW_ALIAS=1 DM_NVN_RT_RECYCLE=0 DM_NVN_ENC_THREAD=0 MTL_DEBUG_LAYER=1 ./t_window_alias + DM_NVN_WINDOW_ALIAS=1 DM_NVN_RT_RECYCLE=0 DM_NVN_ENC_THREAD=1 MTL_DEBUG_LAYER=1 ./t_window_alias t_image_alias: t_image_alias.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_image_alias.m $(MTL_T_SRC) $(FW) .PHONY: check_t_image_alias +# t_image_alias checks the same-offset families (image_texture) a texture falls back to when its span's heap cannot take it: with the spans off, as every test runs (DM_NVN_WINDOW_ALIAS above) check_t_image_alias: t_image_alias DM_NVN_ENC_THREAD=0 MTL_DEBUG_LAYER=1 ./t_image_alias DM_NVN_ENC_THREAD=1 MTL_DEBUG_LAYER=1 ./t_image_alias diff --git a/dm/gfx/tests/t_window_alias.m b/dm/gfx/tests/t_window_alias.m new file mode 100644 index 0000000000..39dd558275 --- /dev/null +++ b/dm/gfx/tests/t_window_alias.m @@ -0,0 +1,104 @@ +// Textures in GPU-only memory share bytes as on NVN: descriptions of other sizes and formats over overlapping storage of a CPU_NO_ACCESS pool are placed in one placement heap per +// 32 MB span of guest addresses, each at its own address's offset scaled by 9/8 (window_texture). Ocarina of Time: 400 Metal textures of 793 MB stood for 89 MB of NVN storage +// (docs/reference/memory.md). A span gets its heap only once the textures made apart over it pass the heap's size. The spans are on by themselves only where Metal's working set +// is under 8 GB (window_wants); the check target turns them on with DM_NVN_WINDOW_ALIAS=1. Run with DM_NVN_WINDOW_ALIAS=0 it shows the old behaviour, every description an +// allocation of its own, and fails +#include "../nvn/nvn_mtl.m" +void *front_metal_layer(void) { return NULL; } +module_t *module_of(const void *p) { (void)p; return NULL; } +module_t *g_mod; int g_nmod; +void svc_thread_dump(void) {} +uint64_t now_ns(void) { return bp_now(); } +static int checks, failed; +#define CHECK(c, label) do { checks++; if (!(c)) { failed++; fprintf(stderr, "FAIL: %s (line %d)\n", label, __LINE__); } } while (0) + +static tex_t target(pool_t *pool, uint64_t serial, uint64_t off, unsigned w, unsigned h, int format) { + tex_t t = {.magic=MAGIC('T','E','X',0), .serial=serial, .pool=pool, .mem=pool->mem+off, .off=(int64_t)off}; + t.d = (nvn_texdesc){.target=NVN_TEXTURE_TARGET_2D, .format=format, .flags=NVN_TEXTURE_FLAGS_COMPRESSIBLE, .width=w, .height=h, .depth=1, .levels=1, .samples=1}; + nvn_tex_layout(&t.d, &t.L); return t; +} +static id read_image(id t) { + NSUInteger row=t.width*4, bytes=row*t.height; + id b=[g_dev newBufferWithLength:bytes options:MTLResourceStorageModeShared]; res_add(b); + enc_b_copy_texture_buffer(blit(), t, 0, 0, MTLOriginMake(0,0,0), MTLSizeMake(t.width,t.height,1), b, 0, row, bytes); + mtl_sub_commit(); mtl_drain(); CHECK(!mtl_last_commit_error(), "image read completes without GPU error"); + res_remove(b); mtl_sub_begin(); return b; +} +static void fill(id t, uint32_t seed) { + NSUInteger bytes=t.width*t.height*4; + id b=[g_dev newBufferWithLength:bytes options:MTLResourceStorageModeShared]; + uint32_t *p=b.contents; for (NSUInteger i=0;i t, uint32_t seed) { + id b=read_image(t); uint32_t *p=b.contents; int ok=1; + for (NSUInteger i=0;i t) { id b=read_image(t); uint32_t *p=b.contents; int ok=1; for (NSUInteger i=0;i weak_heap; + if (posix_memalign(&mem, WIN_SIZE, 2*WIN_SIZE) || posix_memalign(&cmem, 16384, 16<<20)) abort(); + memset(mem, 0, 2*WIN_SIZE); memset(cmem, 0, 16<<20); + @autoreleasepool { + mtl_sub_begin(); + pool_t gpu={.magic=MAGIC('P','O','O','L'), .flags=NVN_MEMORY_POOL_FLAGS_CPU_NO_ACCESS|NVN_MEMORY_POOL_FLAGS_GPU_CACHED|NVN_MEMORY_POOL_FLAGS_COMPRESSIBLE, .mem=mem, .size=2*WIN_SIZE}; + pool_t cpu={.magic=MAGIC('P','O','O','L'), .flags=NVN_MEMORY_POOL_FLAGS_CPU_CACHED|NVN_MEMORY_POOL_FLAGS_GPU_CACHED|NVN_MEMORY_POOL_FLAGS_COMPRESSIBLE, .mem=cmem, .size=16<<20}; + // 48 descriptions over the first 8 MB of the window, of other sizes and formats, as a game's passes and its dynamic resolution make them + enum { N=48 }; tex_t a[N]; id m[N]; + uint64_t before=g_dev.currentAllocatedSize; + for (int i=0;i>20), (unsigned long long)(delta>>20)); + CHECK(!m[0].heap, "the first texture of a span keeps its own allocation: a heap would not save memory yet"); + int first=-1; for (int i=0;iwindow) first=i; } // before it, same-offset families may share heaps of their own (image_texture) + int one=first>0; for (int i=first>0?first:N;i0; for (int i=first>0?first:N;i0; for (int i=first>0?first:N;i0 && [g_res_objects containsObject:m[first].heap] && ![g_res_objects containsObject:m[N-1]], "the heap is resident, its textures through it"); + // Bytes are shared as on NVN: a texture over storage another wrote reads it; storage that does not overlap is untouched + tex_t x=target(&gpu, 971000, 16u<<20, 256, 256, NVN_FORMAT_RGBA8), y=target(&gpu, 971001, 16u<<20, 256, 256, NVN_FORMAT_RGBA8UI), z=target(&gpu, 971002, 20u<<20, 256, 256, NVN_FORMAT_RGBA8); + mtl_tex_init(&x); mtl_tex_init(&z); id tx=tex_mtl(&x), tz=tex_mtl(&z); + CHECK(tx.heap && tx.heap==m[N-1].heap && tz.heap==tx.heap, "later textures of the span join its heap"); + CHECK(zero(tx) && zero(tz), "storage no texture covered before starts as zeros"); + fill(tx, 0x5a5a1234); fill(tz, 0x0f0f9876); + mtl_tex_init(&y); id ty=tex_mtl(&y); + CHECK(ty.heap==tx.heap && ty.heapOffset==tx.heapOffset && ty!=tx, "another format over the same storage is another texture over the same bytes"); + CHECK(holds(ty, 0x5a5a1234), "a description made over written storage inherits its bytes (no zero fill over a live alias)"); + CHECK(holds(tz, 0x0f0f9876) && holds(tx, 0x5a5a1234), "storage that does not overlap keeps its own bytes"); + // The second span has no heap for one texture; CPU-visible pools, linear textures and textures past their span's end keep allocations of their own + tex_t w=target(&gpu, 971010, WIN_SIZE+(1u<<20), 512, 512, NVN_FORMAT_RGBA8); mtl_tex_init(&w); + CHECK(!tex_mtl(&w).heap, "a lone texture in another span keeps its own allocation (no heap is made for it)"); + tex_t c=target(&cpu, 971020, 0, 512, 512, NVN_FORMAT_RGBA8); mtl_tex_init(&c); + CHECK(!tex_mtl(&c).heap, "a CPU-visible pool's texture keeps its own allocation"); + tex_t l=target(&gpu, 971030, 24u<<20, 128, 64, NVN_FORMAT_RGBA8); l.d.flags|=NVN_TEXTURE_FLAGS_LINEAR; nvn_tex_layout(&l.d, &l.L); mtl_tex_init(&l); + CHECK(!tex_mtl(&l).heap || tex_mtl(&l).heap!=tx.heap, "a linear texture stays out of the span's heap"); + // One across a span's end goes to the spans offset by half a span, made once they save memory too: two overlapping descriptions there share the second span's heap + tex_t e=target(&gpu, 971040, WIN_SIZE-(64u<<10), 1024, 1024, NVN_FORMAT_RGBA8), e2=target(&gpu, 971041, WIN_SIZE-(64u<<10), 1000, 1000, NVN_FORMAT_R32F); mtl_tex_init(&e); mtl_tex_init(&e2); + id te=tex_mtl(&e), te2=tex_mtl(&e2); + CHECK(te.heap!=tx.heap && te2.heap && te2.heap!=tx.heap, "a texture across a span's end does not go to that span's heap"); + CHECK(te2.heap && (te.heap==te2.heap || !te.heap), "overlapping textures across the span's end share the offset span's heap"); + te=te2=nil; + // The heap lives while one of its textures does + weak_heap=tx.heap; + for (int i=0;i` also writes the raw storage of the first one per format and size DM_NVN_ZSKIP switch `0` turns off the zero-source skip, which drops a draw whose fragment shader is proven to write 0 over targets already 0 (on by default, only with `DM_NVN_TBDR` on) diff --git a/docs/reference/memory.md b/docs/reference/memory.md index 00a0bfecbc..a91ef98648 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -587,6 +587,17 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Where it does not help:** a game whose shaders read raw global memory has every pool it took an address in made resident by the raw pool table (`gmem_map`), shader code pools included. Ocarina of Time is one: on the village walk it kept 4,276 pools resident before the fix and 4,277 after, 2,324 on both at the title. The 222 pools and 334 MB first reported there came from a build with other changes. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) (reported), [The Great Ace Attorney Chronicles](../../dm/games/010036E00FB20000/README.md) (measured). +### Textures in GPU-only memory took an allocation per description + +- **Symptom:** on a small device the residency set grows past Metal's working set and every command buffer fails with `kIOGPUCommandBufferCallbackErrorOutOfMemory`, then `FATAL the GPU is out of memory`. On the M1 iPad (8 GB, working set 5,461 MB) Ocarina of Time died going outside from Link's house, the set at 5,147 MB, 2,780 MB of it textures; with the render targets' recycled heaps ([rendering.md](rendering.md#render-targets-that-move-with-the-dynamic-resolution-get-new-metal-memory-each-time)) it died at the outdoor save load, the set at 5,184 MB (544 heaps of 1,384 MB over 1,455 MB of textures). `res_top`'s line `GPU-made textures in the set by the pool that holds their storage` shows Metal bytes far above the `union` of their NVN storage. +- **Cause:** a game describes the same bytes of a GPU-only pool (`CPU_NO_ACCESS`) as targets of other sizes and formats from pass to pass and as its dynamic resolution moves. On NVN that costs nothing: they share the pool's memory. Here each description at each address is a Metal texture of its own (the recycled heaps reuse a freed texture's memory, but live descriptions still take one each). Ocarina of Time keeps its render targets in virtual pools (flags 0x2a1) over one 305 MB physical pool: walking in Link's house on a 16 GB Mac, 400 Metal textures of 793 MB stood for 89 MB of NVN storage, and before the recycled heaps the set grew by 365-950 MB over a two-minute walk. +- **Fix:** general, for small devices (`nvn_mtl.m` `window_texture`): a texture whose storage lies in a `CPU_NO_ACCESS` pool (not sparse, not linear, one sample) goes into a placement heap that stands for a 32 MB span of guest addresses, at its NVN offset in the span scaled by 9/8 and rounded up to Metal's alignment, if it ends within its scaled NVN range: textures that overlap on NVN overlap here, and those that do not, do not. The scale covers Metal sizes above NVN's (an RGBA8 1024x1024 target: 4,227,072 bytes against 4,194,304). A span's heap (36 MB) is made only once it saves memory: when a texture shares bytes with one made apart over the span, or those made apart pass the heap's size. A texture across a span's end tries the spans offset by half a span. Bytes no earlier texture of the heap covered are zero-filled; a texture over covered bytes inherits them, as on NVN. Placed textures have no GPU-optimized contents (`allowGPUOptimizedContents` off): a lossless-compressed texture whose metadata another description overwrote is undefined to read, and with compression the M1 iPad's GPU hung by Link's house (`kIOGPUCommandBufferCallbackErrorHang`, a white flash, then `GPU access revoked`). Anything that does not fit takes the usual path (`image_texture`, then `rh_texture`). +- **When it is on:** only where Metal's `recommendedMaxWorkingSetSize` is under 8 GB (`window_wants`), with a log line at the first texture saying which way it went; `DM_NVN_WINDOW_ALIAS=0` / `1` forces it. Sharing is not free of risk: a game may keep live targets in the part of a full-size target that its current resolution leaves unused, which only NVN's layout makes safe, and a heap mirroring Ocarina of Time's target pool on the Mac showed cyan blocks in the fog for that reason ([PR #1453](https://github.com/seedds/nxnative/pull/1453), where compression off and the offset scale were ruled out as the cause). So larger devices keep each target's own layout in its recycled heap, and sharing is the small device's choice, where the separate heaps do not fit: they took 5.2 GB at Ocarina of Time's outdoor save load, sharing 4.3 GB. The 8 GB line leaves room for what play adds to the separate heaps' 5.2 GB; 8 GB devices have 5.3-5.5 GB of working set, a 16 GB M4 Mac 12,124 MB. The `dm/gfx/tests` Makefile runs every test with it off unless the test turns it on. +- **Result on Ocarina of Time** on a 16 GB Mac (same route, 90 s in the house then a walk, before the recycled heaps): the residency set at the walk's end 5,199 → 4,426 MB (textures 2,787 → 1,829 MB, heaps 69 → 254 MB), flat over the walk; `phys_footprint` 9,461 → 8,696 MB. 2,550 descriptions were placed; 6 (545 MB) did not fit a span, among them a 12288x12288 Depth16 target of 296 MB in a 288 MB pool of its own (flags 0xa1), a real allocation on the console too. On the M1 iPad, see the game note. +- **Reading the set:** `res_top` (the first GPU out-of-memory error, `DM_CTL` `restop`, and once when the set first passes three quarters of Metal's working set) also gives the set by the source line that added each object, the resident memory pools (in the raw pool table or not, looked up by a binding or not, by who gave an address in them: pool, buffer, texture or virtual mapping, and by pool flags) and the spans' counts. On Ocarina of Time's walk the 4,285 resident pools (1,967 MB) are all in the raw pool table: 4,058 hold a buffer and the game took every pool's address (1,143 MB), 226 have only a pool address (519 MB; 225 of them, 317 MB, shader code, flags 0x64), one is the physical pool behind the targets (305 MB). Texture addresses reach 2 MB. 3,628 of the pools (1,220 MB) are looked up by nothing but the table, but whether a raw-memory shader reads them is known only on the GPU, so they stay resident. +- **Check:** `make -C dm/gfx/tests GAME= check_t_window_alias` (with the spans on, direct and deferred encoding, Metal validation): the 8 GB rule, a heap only once it saves memory, every later description in one heap at its scaled offset, no GPU-optimized contents, bytes shared and inherited as on NVN, storage apart kept apart, CPU-visible, linear and lone textures kept out, a texture across a span's end, and the heap leaving the set with its last texture. With `DM_NVN_WINDOW_ALIAS=0` it fails, showing the separate allocations. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#memory). + ### Repeated GPU out-of-memory failures end the game - **Symptom:** the Mac grows unresponsive for hours, then panics, with the host still submitting work that fails. diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index 0d24f113b6..7a89a4539b 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -422,7 +422,7 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Symptom:** while a game's dynamic resolution moves, the residency set's `added` count in the `residency set up to frame` lines stays at hundreds of MB per 300 frames from the shared-texture site, and frames stutter. On a 16 GB Mac, wired memory and swap follow, with p95 frame times near 100 ms. Ocarina of Time after 04419bc2, which freed its dynamic resolution from its floor (GitHub issue #1831). - **Cause:** the engine keeps its render targets in a GPU-only pool and gives them new offsets and sizes as its resolution changes. Each `(pool, offset, description)` is a Metal texture of its own (the sharing table), and the storage takeover (`alias_take_over`) lets as many go. On an M6, 3,468 textures were made in about 1,000 frames of Ocarina's village: 97% were in its 305 MB virtual pool, at 1,965 offsets and 289 sizes, and they added 450-575 MB to the residency set every 300 frames. The live total stayed bounded at about 2.1 GB: it is churn, not a leak. Metal does not lay a texture out as NVN does, so these targets cannot share the pool's memory as on the console. A placement heap that mirrors the pool (PR #1453) and a reclaim of targets dead by overwrite (PR #1588) both corrupted the picture: the game keeps live targets in the part of the full-size scene target that its current resolution leaves unused. -- **Fix:** general, `rh_heap`, `rh_place` and `rh_texture` in `nvn_mtl.m`. Such a target (a texture in a `CPU_NO_ACCESS` pool that the GPU can render or write to, so not of a block-compressed format, private storage, not display, video, linear or sparse) is made in a placement heap of its own, an eighth larger than the texture. When its texture is freed, which happens after the submissions that used it complete (`defer_drain`), the heap stays resident in a free list. The smallest waiting heap that fits, up to half as large again, takes the next texture. Metal's allocation sizes come in steps of about 10% near 2 MB (#1583's `rt_sizes`: 855 resolutions at 2048x1152 fall into 21), so the eighth of headroom lets a target that grows by one step fit the heap it freed the frame before. The same free list serves the heaps of resolution families (`image_texture`, the entry above: a new family heap is taken from it, and a family heap whose last texture goes waits in it) and same-pass feedback snapshots (`fb_snap`), which are made again at each new size of their attachment. The new texture is filled as a new allocation is (`tex_init_fill` or zeros), so nothing it reads depends on the reuse. Its memory was the old texture's only up to that texture's completed last use, so no alias flush is added. Heaps that wait 3 s, and the oldest past 256 MB waiting, leave the residency set when a submission begins (`rh_trim`). The memory line counts the heaps made, reused, freed and waiting. `DM_NVN_RT_RECYCLE=0` turns it off (each texture its own allocation, family heaps exact and freed with their last texture), and `ab` alternates it with the `DM_AB` phases. This does not lower the steady total of the live targets, which stays open, and it does not remove the rest of a resize's cost: each new size is still a new Metal texture, filled, and the takeover of the old one. +- **Fix:** general, `rh_heap`, `rh_place` and `rh_texture` in `nvn_mtl.m`. Such a target (a texture in a `CPU_NO_ACCESS` pool that the GPU can render or write to, so not of a block-compressed format, private storage, not display, video, linear or sparse) is made in a placement heap of its own, an eighth larger than the texture. When its texture is freed, which happens after the submissions that used it complete (`defer_drain`), the heap stays resident in a free list. The smallest waiting heap that fits, up to half as large again, takes the next texture. Metal's allocation sizes come in steps of about 10% near 2 MB (#1583's `rt_sizes`: 855 resolutions at 2048x1152 fall into 21), so the eighth of headroom lets a target that grows by one step fit the heap it freed the frame before. The same free list serves the heaps of resolution families (`image_texture`, the entry above: a new family heap is taken from it, and a family heap whose last texture goes waits in it) and same-pass feedback snapshots (`fb_snap`), which are made again at each new size of their attachment. The new texture is filled as a new allocation is (`tex_init_fill` or zeros), so nothing it reads depends on the reuse. Its memory was the old texture's only up to that texture's completed last use, so no alias flush is added. Heaps that wait 3 s, and the oldest past 256 MB waiting, leave the residency set when a submission begins (`rh_trim`). The memory line counts the heaps made, reused, freed and waiting. `DM_NVN_RT_RECYCLE=0` turns it off (each texture its own allocation, family heaps exact and freed with their last texture), and `ab` alternates it with the `DM_AB` phases. This does not lower the steady total of the live targets, which stays open, and it does not remove the rest of a resize's cost: each new size is still a new Metal texture, filled, and the takeover of the old one. Where Metal's working set is under 8 GB the separate heaps do not fit Ocarina of Time outdoors, and textures in GPU-only memory share bytes in a heap per 32 MB span first ([memory.md](memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description)); such a span's heap is not a family heap: it is freed with its last texture instead of waiting in the free list. - **Check:** `make -C dm/gfx/tests GAME= check_t_rt_recycle`: placement with an eighth of headroom, the heap kept resident while it waits, reuse at a smaller size with zeroed contents and at a size one of Metal's steps larger (a texture without `COMPRESSIBLE`), a new heap for a size the waiting ones cannot hold, CPU-visible pools and block-compressed textures kept apart, the trim, a feedback snapshot in a recycled heap, and the switch off. `check_t_image_alias` checks that a family heap holds its largest size and waits, resident, after its last texture. It runs with direct and deferred encoding under Metal validation. In the game: Ocarina of Time on an M6 (16 GB), the village walk with dynamic resolution on and `DM_NVN_RT_RECYCLE=ab DM_AB=300`, one run. Over the phases while its resolution moved, the shared-texture site added 455, 457 and 521 MB per 300 frames off, and 317, 37 and 27 MB on. Frame p90 was 33.3 and 49.8 ms off, 16.8 and 17.6 ms on. Phases after the resolution settled measured the same either way. Screenshots of the village were clean, with no GPU errors or red lines. Measured with the shaders of chain f9f01e4f9994, which were on disk. Mario Kart World on an M6 (16 GB), its race with two outside GPU loads so that its dynamic resolution moved every frame, one run with the free list as it was before (exact heaps taken up to an eighth larger, only `COMPRESSIBLE` targets, family heaps and snapshots allocated apart) and as above alternating every 600 frames: per 300 frames, 970 residency commits (137 ms) and 1.8 GB added before, 272 commits (26 ms) and 0.34 GB after; the replay's CPU time a frame (encoding and commits) 6.94 ms before, 6.36 ms after. Frame time (p50 16.7 ms, p95 about 24 ms) and GPU time (13.2-13.4 ms) were the same: on the M6 the gain is CPU headroom. Measured on main a77c0ee8 with the shaders of chain cb2e91c138e3. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#memory) (Switch 2, `0400486024DEC000`, v131072), [#1831](https://github.com/seedds/nxnative/issues/1831). [Mario Kart World](../../dm/games/0400C3F00006E000/README.md#frame-rate) (Switch 2, `0400C3F00006E000`): its dynamic resolution moves since it reads real GPU times ([#1823](https://github.com/seedds/nxnative/issues/1823)). From 3db7eb538db078e8eda6213f8468cec4db9ff992 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 12:02:25 +0100 Subject: [PATCH 03/40] The power line names the threads that make clock waits, with their predicted sleeps, the waits a sleep ended too late for, how late the sleeps woke and the period: Ocarina of Time makes none, so its slow motion on the M1 iPad is GPU time Before, the power line counted clock waits only for its two top yielders, so a clock wait on any other thread was invisible. On an M1 iPad, Ocarina of Time ran at game speed 0.62 in one run and 0.99 in another with DM_YIELD_CLOCK=0, which pointed at 8bdfcb73's predicted sleep. dm_clock_top (hle_thread.c), printed by the power report every 300 frames (hle_nvn.c) as "clock waits up to frame ": the two threads with most clock-wait yields since the last report, their predicted sleeps, how many of those waits had already ended by the first poll after the sleep (slept through), how late the sleeps woke (mean / max) and the learned period. No clock wait, no line. No behaviour changes: counters only. On the M1 iPad, 12 minutes from the title through the Deku Tree (upstream main + this branch, no switches but the device's own): no line at all, so no thread of the game makes a clock wait and the predicted sleep never runs. Its slow motion there is the GPU: views that take 54-67 ms of GPU a frame cannot fit the frame step's longest hold, 3 vsyncs (game speed 0.61, then 0.53), where lighter views hold 20 fps at 34-36 ms. t_clockwait checks the clause (the thread, 40 predicted sleeps, the waits slept through, a 16.68 ms period, and nothing once no clock wait came); t_pollwait, t_preempt, t_condvar, t_condvar_old, t_sync_wait, t_system_event and t_notify_sched pass. threads.md "Clock waits sleep to their predicted end". --- dm/gfx/nvn/hle_nvn.c | 4 +++- dm/host/hle_thread.c | 23 +++++++++++++++++++++-- dm/host/tests/t_clockwait.c | 11 +++++++++++ docs/reference/threads.md | 2 +- 4 files changed, 36 insertions(+), 4 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index a4382731fc..89ae057794 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -1836,7 +1836,8 @@ static int link_wait(const win_t *o, uint64_t timeout_ns); __attribute__((weak)) void dm_main_yield_stats(uint64_t *n, uint64_t *ns) { *n = 0; *ns = 0; } // The host's hle_thread.c provides the ground truth; 0 in graphics unit tests __attribute__((weak)) uint32_t dm_guest_work_qos(unsigned *count) { if (count) *count = 0; return QOS_CLASS_UNSPECIFIED; } int mtl_probe_ramp_on(void); // nvn_mtl.m: DM_FSTEP_PROBE_RAMP (default on; ab with DM_AB) -__attribute__((weak)) void dm_yield_top(char *out, size_t n) { if (n) snprintf(out, n, "-"); } // hle_thread.c: the threads that yielded most since the last call; none in graphics unit tests +__attribute__((weak)) void dm_yield_top(char *out, size_t n) { if (n) snprintf(out, n, "-"); } +__attribute__((weak)) void dm_clock_top(char *out, size_t n) { if (n) out[0] = 0; } // hle_thread.c: the threads that made most clock waits since the last call // hle_thread.c: the threads that yielded most since the last call; none in graphics unit tests __attribute__((weak)) void dm_thread_cpu(char *out, size_t n, double secs) { (void)secs; if (n) snprintf(out, n, "-"); } // hle_thread.c: guest-thread CPU by Switch priority band since the last call void mtl_res_stats(char *out, size_t n); void mtl_barrier_stats(uint64_t *skip, uint64_t *full); @@ -1986,6 +1987,7 @@ static void power_line(uint64_t frame) { if (dg + dc + du + dn != pdg + pdc + pdu + pdn) LOG("[nvn] compute dispatches up to frame %llu: %llu after a barrier of the game's, %llu after a copy's range, %llu in the full order (a binding with no range), %llu with no barrier (DM_NVN_DISPATCH_BARRIER)", frame, (unsigned long long)(dg - pdg), (unsigned long long)(dc - pdc), (unsigned long long)(du - pdu), (unsigned long long)(dn - pdn)); pdg = dg; pdc = dc; pdu = du; pdn = dn; + { char ck[400]; dm_clock_top(ck, sizeof ck); if (ck[0]) LOG("[nvn] clock waits up to frame %llu: %s", frame, ck); } char rs[400]; mtl_res_stats(rs, sizeof rs); static uint64_t pbs, pbf; uint64_t bs, bf; mtl_barrier_stats(&bs, &bf); uint64_t mtl_cut_count(void), mtl_copy_cut_count(void), mtl_blit_splits(void); void mtl_ubd_stats(uint64_t *, uint64_t *); static uint64_t pcut, pccut, pbsp, pud, puc; uint64_t cut = mtl_cut_count(), ccut = mtl_copy_cut_count(), bsp = mtl_blit_splits(), ud, uc; mtl_ubd_stats(&ud, &uc); void mtl_ubp_stats(uint64_t *, uint64_t *, uint64_t *); static uint64_t ppm, ppr, ppf; uint64_t pm, pr, pf_; mtl_ubp_stats(&pm, &pr, &pf_); diff --git a/dm/host/hle_thread.c b/dm/host/hle_thread.c index f67b76581e..0684a0a46d 100644 --- a/dm/host/hle_thread.c +++ b/dm/host/hle_thread.c @@ -74,13 +74,14 @@ typedef struct thr { int closed, sync_refs; // Reclaim only after handle close, thread exit and the final synchronization wait reference uint64_t nsvc; // SVC count (for the state dump) uint64_t nyield, nyield_seen, nbackoff, nbackoff_seen; // Yields (SleepThread 0/-1/-2) and those the back-off turned into short sleeps, with both counts at the last power report (dm_yield_top) - uint64_t nclkyield, nclkyield_seen; // Yields of clock waits (thr_yield), and their count at the last power report + uint64_t nclkyield, nclkyield_seen, nclkyield_rep; // Yields of clock waits (thr_yield), and their count at the last power report uint64_t nspin, nspin_seen, spin_site; const char *spin_mod; // Spin wait passes (dm_spin_wait; those that yield count in nyield too) and where the last one was uint64_t srun_svc, srun_t0, spin_ret; // Current spin wait: nsvc at its passes, when it began, when the last pass returned uint64_t cpu_seen_us; // CPU time (µs) at the last power report (dm_thread_cpu) uint64_t yrun_svc, yrun_t0, yrun_sig, yrun_sig2, poll_ret; // Current run of yields: nsvc of the last yield, when the run began, and the caller's x19–x28 at the last two yields (a run lasts while no other SVC comes between two yields and those registers match one of the last two, thr_yield); when the last yield or short timed-out wait returned (poll_wait_begin) uint64_t yrun_clk; // dm_clock_reads at the last yield: a pass that read the clock since makes the yield a clock wait (thr_yield) uint64_t cw_sig, cw_run, cw_last, cw_end, cw_per, cw_per2, cw_wake, ncwpred, ncwpred_seen; // The clock wait followed (clock_wait): its x19–x28, the run it was last seen in, when its last yield returned, + uint64_t ncwfirst, cw_late_sum, cw_late_max, ncwfirst_seen, ncwpred_rep, cw_late_seen; // predicted sleeps whose wait had ended at the first poll after them, how late those sleeps woke (sum, max), at the last report (dm_clock_top) int cw_ok, cw_pred, cw_after, cw_trip; // when its last wait ended, the last two periods between ends, when the predicted sleep ended, predicted sleeps (and at the last power report); periods in a row that matched, a predicted sleep in this run, yields after it, waits in a row that ended at the first poll after it mach_port_t mport; // Host thread port (to get CPU time) _Atomic int qos_pending; // SetThreadPriority from another thread changed prio: the thread applies the new class at its next SVC (only a thread can set its own QoS) @@ -862,6 +863,7 @@ static void clock_wait(thr_t *me, uint64_t sig, uint64_t now, int predict) { if (me->cw_sig && now - me->cw_last < 1000000000ull) { yield_for(me, now - me->yrun_t0, 1); return; } // another clock wait: the followed loop is still in use me->cw_sig = sig; me->cw_end = me->cw_per = me->cw_per2 = 0; me->cw_ok = me->cw_pred = me->cw_trip = 0; } else { + if (me->cw_pred && !me->cw_after) me->ncwfirst++; if (me->cw_pred && !me->cw_after) { // the last wait ended at the first poll after its predicted sleep: maybe late // once: the period learned from waits the back-off ended carries their lateness (up to 250 µs, and nanosleep's quarter): predict it shorter; // twice in a row: the period changed, learn it again @@ -880,7 +882,9 @@ static void clock_wait(thr_t *me, uint64_t sig, uint64_t now, int predict) { uint64_t per = me->cw_per2 && me->cw_per2 < me->cw_per ? me->cw_per2 : me->cw_per, dl = me->cw_end + per; // the shorter of the last two periods if (dl > now + 2 * CW_MARGIN) { me->nbackoff++; me->ncwpred++; me->sleeping = 1; sleep_precise(dl - CW_MARGIN); me->sleeping = 0; - me->cw_pred = 1; me->cw_wake = now_ns(); return; + me->cw_pred = 1; me->cw_wake = now_ns(); + { uint64_t late = me->cw_wake > dl - CW_MARGIN ? me->cw_wake - (dl - CW_MARGIN) : 0; me->cw_late_sum += late; if (late > me->cw_late_max) me->cw_late_max = late; } + return; } } } else if (me->cw_pred) { // polling after the predicted sleep @@ -974,6 +978,21 @@ void dm_yield_top(char *out, size_t n) { pthread_mutex_unlock(&G); if (!o && n) snprintf(out, n, "none"); } +// The threads that made most clock waits since the last call (the power line every 300 frames, hle_nvn.c), at most two: "handle name: N yields, M predicted sleeps (ended at the first +// poll after it K; the sleeps woke late by mean / max µs), period P ms". A wait that had ended by the first poll after its predicted sleep was slept through: it ended before the sleep did +void dm_clock_top(char *out, size_t n) { + thr_t *top[2] = {0}; uint64_t c[2] = {0}; + pthread_mutex_lock(&G); + for (thr_t *t = g_all; t; t = t->all) { uint64_t k = t->nclkyield - t->nclkyield_rep; if (k > c[0]) { top[1] = top[0]; c[1] = c[0]; top[0] = t; c[0] = k; } else if (k > c[1]) { top[1] = t; c[1] = k; } } + size_t o = 0; char nb[32]; if (n) out[0] = 0; + for (int i = 0; i < 2 && top[i] && o < n; i++) { thr_t *t = top[i]; + uint64_t pr = t->ncwpred - t->ncwpred_rep, fi = t->ncwfirst - t->ncwfirst_seen, ls = t->cw_late_sum - t->cw_late_seen; + o += snprintf(out + o, n - o, "%s%#x '%s' %llu yields, %llu predicted sleeps (wait over at the first poll after it %llu; woke late by %.0f / %.0f us), period %.2f ms", i ? ", " : "", t->handle, thr_name(t, nb, sizeof nb), + (unsigned long long)c[i], (unsigned long long)pr, (unsigned long long)fi, pr ? ls / 1e3 / (double)pr : 0.0, t->cw_late_max / 1e3, t->cw_per / 1e6); + } + for (thr_t *t = g_all; t; t = t->all) { t->nclkyield_rep = t->nclkyield; t->ncwpred_rep = t->ncwpred; t->ncwfirst_seen = t->ncwfirst; t->cw_late_seen = t->cw_late_sum; t->cw_late_max = 0; } + pthread_mutex_unlock(&G); +} // CPU time of the guest threads since the last call, by Switch priority band (<= 44: the main thread's level and above; 45-59; >= 60) and the three busiest threads with their priority. // For the power line (hle_nvn.c power_line): it shows which threads a priority-to-QoS mapping would move and how much CPU they use. secs = the window length void dm_thread_cpu(char *out, size_t n, double secs) { diff --git a/dm/host/tests/t_clockwait.c b/dm/host/tests/t_clockwait.c index aa628cb88f..2d11bf6b19 100644 --- a/dm/host/tests/t_clockwait.c +++ b/dm/host/tests/t_clockwait.c @@ -71,5 +71,16 @@ int main(void) { assert(late_frames == 2); assert(late_max < 2000); puts("PASS: a shorter period costs two late waits, then the waits end on time again"); + + // The power line's clause (dm_clock_top) names the thread, its predicted sleeps, the waits a sleep ended too late for (the two above) and the period; a second call, with no clock + // wait since, gives nothing: a game with no clock waits prints no clause + char ck[400]; dm_clock_top(ck, sizeof ck); printf(" clause: %s\n", ck); + unsigned long long y = 0, pr = 0, first = 0; double per = 0; + const char *q = strchr(ck, '\''); q = q ? strchr(q + 1, '\'') : NULL; + assert(q && sscanf(q, "' %llu yields, %llu predicted sleeps (wait over at the first poll after it %llu;", &y, &pr, &first) == 3); + assert(strstr(ck, "period ") && sscanf(strstr(ck, "period ") + 7, "%lf", &per) == 1); + assert(y > 0 && pr >= 40 && first >= 2 && first <= 4 && per > 16.0 && per < 17.0); + dm_clock_top(ck, sizeof ck); assert(!ck[0]); + puts("PASS: the power line names the thread that made clock waits, and says nothing once none did"); return 0; } diff --git a/docs/reference/threads.md b/docs/reference/threads.md index a267c4cabc..99e0298248 100644 --- a/docs/reference/threads.md +++ b/docs/reference/threads.md @@ -251,7 +251,7 @@ The yield back-off (`DM_YIELD_BACKOFF`) and the rewriter's spin waits (`patch_ns - **Symptom:** a game's frame limiter that yields and reads the clock holds 59.4–59.5 fps instead of 60 (frame time p50 about 16.8 ms instead of 16.68; the HUD rounds it to 60), and uses more CPU energy than the same game sleeping: the power line's top yielder is the main thread with thousands of yields a frame. - **Cause:** the yield back-off ([development.md](../development.md#power), "Yield loops") lets a run of yields spin for its first 2 ms, which protects handoffs between threads, and then sleeps an eighth of the wait so far, up to 250 µs. A limiter waits for a time, not for another thread, so the spin is pure cost (about 9,700 plain yields a frame on a performance core in Three Houses), and since the back-off cannot know the deadline, each wait ends up to 250 µs late, plus a quarter for `nanosleep`'s slack ([Timer slack in guest sleeps](#timer-slack-in-guest-sleeps-sleep_precise)). A shorter cap trades one for the other: 100 µs gave 16.73 ms at 0.6–1.0 W. -- **Fix:** every guest read of the system counter counts on its thread (`dm_clock_reads`, `guest_clock_read_wait` in `hle_os.c`, through the `cntpct` gate and `svc GetSystemTick`). A run of yields whose passes read the clock is a clock wait (`thr_yield`, `clock_wait` in `hle_thread.c`): it backs off from its start, with no 2 ms of plain yields. A clock wait that ends at a steady period also knows its deadline, its last end plus the period: the loop followed (one per thread, by its x19–x28) learns the period from the times its waits end, and after three periods in a row within 250 µs of each other a wait sleeps on time (`sleep_precise`) to 150 µs before the predicted end, then polls with plain yields. A wait that ends at the first poll after such a sleep may have ended late: the first time, the period is taken 300 µs shorter (one learned from waits the back-off ended carries their lateness); the second time in a row, the period changed (a game going from 30 to 60) and is learned again, so such a change costs two late waits. A prediction that comes early (the period got longer) costs only polls and back-off sleeps. `DM_YIELD_CLOCK=1` keeps the back-off from the start without the predicted sleep, `=0` makes clock waits ordinary yields; the power line's top yielders show `clock waits N, M of them slept to a predicted end`. +- **Fix:** every guest read of the system counter counts on its thread (`dm_clock_reads`, `guest_clock_read_wait` in `hle_os.c`, through the `cntpct` gate and `svc GetSystemTick`). A run of yields whose passes read the clock is a clock wait (`thr_yield`, `clock_wait` in `hle_thread.c`): it backs off from its start, with no 2 ms of plain yields. A clock wait that ends at a steady period also knows its deadline, its last end plus the period: the loop followed (one per thread, by its x19–x28) learns the period from the times its waits end, and after three periods in a row within 250 µs of each other a wait sleeps on time (`sleep_precise`) to 150 µs before the predicted end, then polls with plain yields. A wait that ends at the first poll after such a sleep may have ended late: the first time, the period is taken 300 µs shorter (one learned from waits the back-off ended carries their lateness); the second time in a row, the period changed (a game going from 30 to 60) and is learned again, so such a change costs two late waits. A prediction that comes early (the period got longer) costs only polls and back-off sleeps. `DM_YIELD_CLOCK=1` keeps the back-off from the start without the predicted sleep, `=0` makes clock waits ordinary yields; the power line's top yielders show `clock waits N, M of them slept to a predicted end`. A line of its own every 300 frames, `clock waits up to frame `, names the two threads that made most clock waits, with their predicted sleeps, how many waits had already ended by the first poll after the sleep (slept through), how late the sleeps woke (mean / max) and the period learned; no thread making clock waits, no line. Ocarina of Time makes none: on the M1 iPad, 12 minutes from the title through the Deku Tree printed no such line, so its slow motion there is GPU time, not this rule (its note). - **Found on:** [Fire Emblem: Three Houses](../../dm/games/010055D009F78000/README.md#frame-rate), whose frame limiter (`main+0x5be430`) had a per-build cut that made it sleep; the cut is removed. On the Mac in a window, 100 s from the boot to the title: without the cut, the old back-off 59.4–59.5 fps, frame p50 16.80–16.85 ms, 0.28–0.79 W of CPU; backing off from the start alone 59.4–59.5 fps, 0.19–0.35 W; with the predicted sleep 59.8–60.0 fps, 16.67–16.81 ms in most windows (up to 17.06), 0.09–0.16 W, about 1,330 wakeups a second, 294–300 of every 300 waits slept to their predicted end. With the cut it was 59.9 fps, 16.68 ms, 0.11–0.49 W. - **Check:** `make -C dm/host/tests GAME= build/t_clockwait && dm/host/tests/build/t_clockwait`, on a test clock that only sleeps and yields advance: a yield loop that reads the clock backs off from its start and one that does not after 2 ms; a limiter at a steady period sleeps to its predicted end and ends within a few yields of its deadline; a longer period never makes a wait later than the back-off's own lateness; a shorter period costs two late waits, then the waits end on time again. From aa82d4cf81ecb6a5e2f7995a82d58fc9debd9059 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 13:30:15 +0100 Subject: [PATCH 04/40] Ocarina of Time's note: the M1 iPad holds 20 fps at the Great Deku Tree's entrance with the guest in files and a 3.2 GB residency set; heavier views are GPU-bound "For an iPhone or iPad build" under Memory: the device (iPad Pro 11-inch 3rd generation, M1, 8 GB, iPadOS 27.0.1), the memory limit and headroom (7.97 GB of room against the 9 GB pool, so the guest in files; footprint 2.2-2.3 GB, headroom 5.87 GB at its lowest; working set 5,461 MB, residency set 3.2 GB with the GPU-only spans on), fps and GPU time (20.0 fps in every window of two 32-minute runs at the Deku Tree's entrance, GPU 34-35 ms, game speed 0.97-1.00; heavier views 54-67 ms of GPU, 11-12 fps and game speed 0.53-0.61; the title and game over screens 61 and 66-74 ms), no clock waits, no switch needed, and what is open: the heavy views' GPU time, a GPU hang on an earlier build in the draws of PR #1842's M1 hangs, one kernel data abort, and PR #1906 for an iOS build on 7ffa63cc. Found on: "Textures in GPU-only memory took an allocation per description"; memory.md's two entries link the new section. The Open list's "no iOS run" line points at it. --- dm/games/0400486024DEC000/README.md | 11 ++++++++++- docs/reference/memory.md | 6 +++--- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index ade7e33748..14505dcfd6 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -80,6 +80,14 @@ Measured on an M6 Mac with 16 GB, headless, on one walk with camera turns (`walk - Why so much: the game's own memory is about the console's 9 GB application pool, and on the Mac its textures also exist as Metal textures, as Metal cannot sample the guest's block-linear storage. - Resolved in part ([#1831](https://github.com/seedds/nxnative/issues/1831)): render targets that move in the game's 305 MB pool and change size with its dynamic resolution still get a new Metal texture each time. Their memory is now recycled ([Render targets that move with the dynamic resolution](../../../docs/reference/rendering.md#render-targets-that-move-with-the-dynamic-resolution-get-new-metal-memory-each-time)). Since 04419bc2 the resolution moves, and on a 16 GB MacBook Air M4 the game stuttered (p95 99 ms) with GPU allocation near 15 GB. On an M6, while the resolution moved in the village, the target site added 455-521 MB to the residency set per 300 frames without recycling and 27-37 MB with it once heaps were waiting, and frame p90 went from 33-50 ms to 17-18 ms. Open: the live targets' steady total, about 2.1 GB in the village. They cannot share the pool's memory as on the console: the game keeps live targets in the part of the full-size scene target its resolution leaves unused, which only NVN's layout allows (PR #1453). [A report of a GPU driver panic](https://github.com/seedds/nxnative/pull/1299) lines up with this churn. The reporter's Mac has not been tested. +### For an iPhone or iPad build + +Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13,4`, 8 GB, iPadOS 27.0.1), docked, at 20 fps where its views are light. Built from upstream `main` 7ffa63cc with this branch, chain `dedc9391c30b`, the device's `env.txt` holding only `DM_LAUNCHER=0`, `DM_NVN_GPUPROF=every:600` and `DM_GUEST_FILE_MIN_FREE=2G` (none needed for play). Two 32-minute scripted runs from a save at the Great Deku Tree's entrance (camera turns and short steps, "Try Again" after a death). +- **Memory.** iOS gives the app 7.97 GB of room against the game's 9.00 GB pool, so the guest goes into files by itself ([the guest in files](../../../docs/reference/memory.md#ios-ends-the-app-once-the-guests-touched-pages-reach-its-memory-limit)): footprint 2.2-2.3 GB, headroom to the memory limit 5.87 GB at its lowest, 4.4 GB of guest pages on disk. Metal's working set is 5,461 MB: [textures in GPU-only memory share bytes](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description) there by themselves, and the residency set holds 3.2 GB. With each render target in a recycled heap of its own, the set reached 5,184 MB at an outdoor save load and every command buffer failed out of memory. +- **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. +- **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. +- Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needs [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` is linked otherwise). + ## Saves The game autosaves into `slot_00` and `slot_01`, with `option.sav` and `account.sav`. The Load screen lists them and reloading works. A save's thumbnail is a 543x306 JPEG in `caption.sav`, made from a copy of a texture to a buffer. A manual save (Menu, System, Save) on the current build has the picture (slot 02, mean colour 138 / 123 / 86). The autosaves in the work folder's saves, made on older chains, hold a black one. Open: an autosave made on the current build has not been looked at. @@ -112,7 +120,7 @@ From an empty save: at about 45 s `A` (title), `A` (New Game), then the movie. F - The "Navi" name label in the Great Deku Tree scene was missing on an earlier chain and has not been checked since. Not played yet: the Deku Tree dungeon, combat, later areas. Sound opens with no red line; movies and speech were not listened to. Physics issues reported in PR 1299 were not seen on this route (walking, the ladder, the camera). - Resolved: dotted horizontal rows across the picture in play, in fog and light (GitHub issue 1537): the agl_ndsr_vr networks' HMMA calls outside a chain overwrote an accumulator a chain held ([An HMMA outside a chain overwrote the chain's accumulator](../../../docs/reference/shaders.md#an-hmma-outside-a-chain-overwrote-the-chains-accumulator)), since chain 23a1ea1db092; found by bisecting chains on the Kokiri Forest autosave (0f510fd5's chain clean, bd9f422d's not). Under the game's upscaler they had also grown into white blotches: [LDSM reads rows other lanes stored](../../../docs/reference/shaders.md#ldsm-reads-rows-other-lanes-stored). - Open: GPU hangs in play ([#1326](https://github.com/seedds/nxnative/issues/1326), [#1605](https://github.com/seedds/nxnative/issues/1605)), repeatable in a Kokiri Forest conversation and the Impa escort on an M2 Pro and gone with the pipelines of task `a5288d0a6c4971fd` dropped ([#2011](https://github.com/seedds/nxnative/issues/2011)). They are not [threads returning before a barrier](../../../docs/reference/shaders.md#compute-threads-that-return-before-a-barrier): that task, the culling tasks `47f9b38490dcf9a2` and `99280b7b8aae78df` and mesh `3439aaa071d868ae` have no barrier in their bodies (none of the game's 520 task and mesh shaders has one). The lead in #2011 is pixel shader `bd09ab5fc4fd7680`'s light loop, which runs `min(count, 1000) + 1` times per pixel with per-tile counts the GPU reads. -- No iOS shader build or iPhone run has been done. Its memory use makes an iPhone unlikely as it stands. +- Not run on an iPhone yet; the M1 iPad: [For an iPhone or iPad build](#for-an-iphone-or-ipad-build). ## Unused border corrections @@ -139,6 +147,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [A NaN colour under a zero blend weight](../../../docs/reference/shaders.md#a-nan-colour-under-a-zero-blend-weight): resolved, white clouds over the ground mist between Kokiri Forest and Hyrule Field and over the ferns at the camera in Kokiri Forest ([issue #1839](https://github.com/seedds/nxnative/issues/1839)). The mist shader `5ad65c0eefd09c7f` writes NaN at alpha 0 where its layer has density 0; NVIDIA's blender, with the option NVN's driver sets, adds nothing there, Apple's stored the NaN. - [A first start builds every base of the prewarm list before the first frame](../../../docs/reference/rendering.md#a-first-start-builds-every-base-of-the-prewarm-list-before-the-first-frame): the 450-475 s first-start prepare on an M5 Max; the Mac app now bundles the list's pipelines (`--prebake`, on by default) and it took 4.3 s. - [The 60 Hz link at 40 fps in fullscreen on a ProMotion screen](../../../docs/reference/frame-rate.md#the-60-hz-link-at-40-fps-in-fullscreen-on-a-promotion-screen): 40 fps in fullscreen on an M5 Pro MacBook Pro with the GPU at 10 ms of the 16.7, every present one 120 Hz refresh late and every other display-link tick skipped; the screen's fixed 60 Hz mode held 60.0 fps there and fullscreen now switches to it, with the 120 Hz link (59 fps) for screens with no such mode. +- [Textures in GPU-only memory took an allocation per description](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description): the M1 iPad's GPU out-of-memory failure at the outdoor save load. - [The residency commit froze over the guest's files](../../../docs/reference/memory.md#the-residency-commit-froze-over-the-guests-files): resolved, the M1 iPad's freeze with sound and no picture. - [The shader package and the shader map come from different splits](../../../docs/reference/packing.md#the-shader-package-and-the-shader-map-come-from-different-splits): resolved, a black screen with sound in a build through NX2Apple ([issue #1841](https://github.com/seedds/nxnative/issues/1841)); the present pass was translated from a terrain shader's record. Records now carry their keys, and the shader build finds them by key. - [Native mesh pipelines compiled at their first draw](../../../docs/reference/shaders.md#native-mesh-pipelines-compiled-at-their-first-draw): resolved. The foliage, shadows and enemies drawn with task and mesh shaders stalled at their first draw on a cold Metal cache (59.5 s of builds at the Kokiri Forest entry on an M5 Max, one frame of 42.9 s), as nothing recorded or kept their pipelines; `pso_manifest.tsv` now carries 164 `M` rows (Kokiri Forest, Hyrule Field, Castle Town, Hyrule Castle, and 41 from later play, Death Mountain Trail among it), and with the prepare step they are built before the first frame. Pipelines outside the list still stall once, then go into `pso_user.tsv`. With the first 123 alone, a cold first start (empty Metal cache and pipeline archive, M4 Pro) on a Death Mountain Trail save built 9 and 10 at their first draw in two runs, 2.7 and 4.2 s, frames of 1,775 and 1,657 ms; with all 164, none, worst frame 25.6 and 18.5 ms, the prepare step 29.6 and 29.3 s → 32.1 and 31.0 s. diff --git a/docs/reference/memory.md b/docs/reference/memory.md index a91ef98648..a6c51ce060 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -559,7 +559,7 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Cause:** a residency commit wires the pages of each buffer it adds, and pages a file in from the GPU driver's own wiring path. A sysdiagnose spindump during the freeze: the commit's thread runs `IOGPUFamily` → the VM → `apfs` pagein → `blocked by krwlock for reading owned by launchd [1]` (a thread of launchd's background I/O queue, base priority 4, boosted), and that launchd thread stopped 4 ms after it, inside APFS, and never ran again. What it waits for is not recorded, as for a lock held for reading, and the kernel had no symbols; that it stopped right after the commit's thread and stays stopped points to something the wiring holds. Ordinary page-ins of the same files (700,000 in a run) never hung, nor did guest threads that faulted only after the commit was stuck: they queued behind the APFS lock. Raw pool table pools are added in groups, a large pool as one whole-pool buffer (`gmem_map`), so a commit can page in hundreds of MB of file at once. - **Fix:** general (`nvn_mtl.m` `file_wire`): a shared buffer whose pages are file-backed (`vm_region_recurse_64` reports an external pager) is wired with `mlock` before it joins the set, so the pages come in through the ordinary fault path and the commit finds them resident; the `mlock`s are given back right after the commit, which holds the pages for the GPU from then on. Anonymous memory and Metal's own buffers are not touched, so nothing changes with the guest anonymous (every Mac run by default). The residency line every 300 frames counts them: `file-backed buffers wired before their commit , , (refused )`; a refused `mlock` logs a red line once and leaves the commit to page them in as before. `t_file_wire` checks it. - **Result on Ocarina of Time,** the M1 iPad, the same route that froze twice, driven over `DM_CTL` (from a save by Kokiri Forest into Link's house, camera turns and a walk): alive past frame 12,900, no red line; 20 fps in the house (the frame step's 50 ms, GPU 34–42 ms), headroom 5.3 GB, residency commits under 2 ms per 300 frames. The fix wired 26 to 3,729 buffers per 300 frames ahead of their commits, 2.1 GB in 6.7 s while the house loaded. -- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#memory) (v131072, 2026-11-08). On a Mac the guest is in files only with `DM_GUEST_FILE=1`; there the same commit showed in samples of a stall that lasted 7–37 s a frame (`IOGPUResourceGroupUpdateResources` under `_commitAddedAllocations`), and with this fix the same route with the files (16 GB M4 Air, idle) played through: 300-frame frame time p50 16.7 ms from the title to the walk's end (19,800 frames), no hitch dump, 3.9 GB wired at most. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#for-an-iphone-or-ipad-build) (v131072, 2026-11-08). On a Mac the guest is in files only with `DM_GUEST_FILE=1`; there the same commit showed in samples of a stall that lasted 7–37 s a frame (`IOGPUResourceGroupUpdateResources` under `_commitAddedAllocations`), and with this fix the same route with the files (16 GB M4 Air, idle) played through: 300-frame frame time p50 16.7 ms from the title to the walk's end (19,800 frames), no hitch dump, 3.9 GB wired at most. ### Memory pools nothing gave an address to stay out of the raw pool table @@ -593,10 +593,10 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Cause:** a game describes the same bytes of a GPU-only pool (`CPU_NO_ACCESS`) as targets of other sizes and formats from pass to pass and as its dynamic resolution moves. On NVN that costs nothing: they share the pool's memory. Here each description at each address is a Metal texture of its own (the recycled heaps reuse a freed texture's memory, but live descriptions still take one each). Ocarina of Time keeps its render targets in virtual pools (flags 0x2a1) over one 305 MB physical pool: walking in Link's house on a 16 GB Mac, 400 Metal textures of 793 MB stood for 89 MB of NVN storage, and before the recycled heaps the set grew by 365-950 MB over a two-minute walk. - **Fix:** general, for small devices (`nvn_mtl.m` `window_texture`): a texture whose storage lies in a `CPU_NO_ACCESS` pool (not sparse, not linear, one sample) goes into a placement heap that stands for a 32 MB span of guest addresses, at its NVN offset in the span scaled by 9/8 and rounded up to Metal's alignment, if it ends within its scaled NVN range: textures that overlap on NVN overlap here, and those that do not, do not. The scale covers Metal sizes above NVN's (an RGBA8 1024x1024 target: 4,227,072 bytes against 4,194,304). A span's heap (36 MB) is made only once it saves memory: when a texture shares bytes with one made apart over the span, or those made apart pass the heap's size. A texture across a span's end tries the spans offset by half a span. Bytes no earlier texture of the heap covered are zero-filled; a texture over covered bytes inherits them, as on NVN. Placed textures have no GPU-optimized contents (`allowGPUOptimizedContents` off): a lossless-compressed texture whose metadata another description overwrote is undefined to read, and with compression the M1 iPad's GPU hung by Link's house (`kIOGPUCommandBufferCallbackErrorHang`, a white flash, then `GPU access revoked`). Anything that does not fit takes the usual path (`image_texture`, then `rh_texture`). - **When it is on:** only where Metal's `recommendedMaxWorkingSetSize` is under 8 GB (`window_wants`), with a log line at the first texture saying which way it went; `DM_NVN_WINDOW_ALIAS=0` / `1` forces it. Sharing is not free of risk: a game may keep live targets in the part of a full-size target that its current resolution leaves unused, which only NVN's layout makes safe, and a heap mirroring Ocarina of Time's target pool on the Mac showed cyan blocks in the fog for that reason ([PR #1453](https://github.com/seedds/nxnative/pull/1453), where compression off and the offset scale were ruled out as the cause). So larger devices keep each target's own layout in its recycled heap, and sharing is the small device's choice, where the separate heaps do not fit: they took 5.2 GB at Ocarina of Time's outdoor save load, sharing 4.3 GB. The 8 GB line leaves room for what play adds to the separate heaps' 5.2 GB; 8 GB devices have 5.3-5.5 GB of working set, a 16 GB M4 Mac 12,124 MB. The `dm/gfx/tests` Makefile runs every test with it off unless the test turns it on. -- **Result on Ocarina of Time** on a 16 GB Mac (same route, 90 s in the house then a walk, before the recycled heaps): the residency set at the walk's end 5,199 → 4,426 MB (textures 2,787 → 1,829 MB, heaps 69 → 254 MB), flat over the walk; `phys_footprint` 9,461 → 8,696 MB. 2,550 descriptions were placed; 6 (545 MB) did not fit a span, among them a 12288x12288 Depth16 target of 296 MB in a 288 MB pool of its own (flags 0xa1), a real allocation on the console too. On the M1 iPad, see the game note. +- **Result on Ocarina of Time** on a 16 GB Mac (same route, 90 s in the house then a walk, before the recycled heaps): the residency set at the walk's end 5,199 → 4,426 MB (textures 2,787 → 1,829 MB, heaps 69 → 254 MB), flat over the walk; `phys_footprint` 9,461 → 8,696 MB. 2,550 descriptions were placed; 6 (545 MB) did not fit a span, among them a 12288x12288 Depth16 target of 296 MB in a 288 MB pool of its own (flags 0xa1), a real allocation on the console too. On the M1 iPad, with it on: the residency set at 3.2 GB in the Great Deku Tree, two 32-minute runs with no GPU error, the picture clean in their screenshots ([the game note](../../dm/games/0400486024DEC000/README.md#for-an-iphone-or-ipad-build)). - **Reading the set:** `res_top` (the first GPU out-of-memory error, `DM_CTL` `restop`, and once when the set first passes three quarters of Metal's working set) also gives the set by the source line that added each object, the resident memory pools (in the raw pool table or not, looked up by a binding or not, by who gave an address in them: pool, buffer, texture or virtual mapping, and by pool flags) and the spans' counts. On Ocarina of Time's walk the 4,285 resident pools (1,967 MB) are all in the raw pool table: 4,058 hold a buffer and the game took every pool's address (1,143 MB), 226 have only a pool address (519 MB; 225 of them, 317 MB, shader code, flags 0x64), one is the physical pool behind the targets (305 MB). Texture addresses reach 2 MB. 3,628 of the pools (1,220 MB) are looked up by nothing but the table, but whether a raw-memory shader reads them is known only on the GPU, so they stay resident. - **Check:** `make -C dm/gfx/tests GAME= check_t_window_alias` (with the spans on, direct and deferred encoding, Metal validation): the 8 GB rule, a heap only once it saves memory, every later description in one heap at its scaled offset, no GPU-optimized contents, bytes shared and inherited as on NVN, storage apart kept apart, CPU-visible, linear and lone textures kept out, a texture across a span's end, and the heap leaving the set with its last texture. With `DM_NVN_WINDOW_ALIAS=0` it fails, showing the separate allocations. -- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#memory). +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#for-an-iphone-or-ipad-build). ### Repeated GPU out-of-memory failures end the game From e0209b590d8ef02af225bcfaa3f13eaa08447966 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 18:05:26 +0100 Subject: [PATCH 05/40] The present takes the shape of the game's crop: a crop whose width / height matches the shape the host asked for gives the picture that shape instead of 16:9 A game that renders a wider or taller picture into larger window textures and crops the window to it (Ocarina of Time: a 2536x1152 crop of 2560x1440 textures at 2.2:1) was presented as a 16:9 picture. present_shape / present_fit (nvn_mtl.m) now take the crop's own ratio when it matches the screen shape mtl_present_screen set (screen_aspect any), and keep 16:9 for a crop of any other shape. t_present_crop checks a wider crop, a taller crop, a crop of another shape and a 5:4 crop filling a 5:4 canvas on the classic and Metal 4 presents: 120 / 120. --- dm/gfx/nvn/nvn_mtl.m | 18 +++++++++++------- dm/gfx/tests/t_present_crop.m | 28 +++++++++++++++++++++++----- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index b16987aebe..ad3b1bf4a7 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -16403,17 +16403,21 @@ static MTLViewport present_viewport(double pw, double ph, double dw, double dh) } // The shape the picture is shown in, whatever part of the texture it comes from. The Switch's compositor stretches a buffer's crop to the layer, so a crop only picks texels: Super // Mario Galaxy 2 draws its storybook cutscenes anamorphically into 1200x960 of a 1920x1080 swapchain (800x640 handheld). The layer fills the display, 16:9, unless the game sized it -// (vi::CreateLayer with a size, hle_vi.c). A screen taller than 16:9 (DM_SCREEN_ASPECT, loader.c) counts only for a texture of its shape: only builds with screen-shape stubs render it. +// (vi::CreateLayer with a size, hle_vi.c). A screen of another shape than 16:9 (DM_SCREEN_ASPECT, loader.c) counts only for a crop or a texture of its shape: only builds with +// screen-shape stubs render it. Breath of the Wild presents a whole texture of the taller shape; Ocarina of Time a crop of that shape inside larger window textures (the game's +// window keeps its texture size and moves its crop, as it does between docked and handheld). static _Atomic uint64_t g_layer_shape, g_screen_shape; // w | h << 32, 0 = none void mtl_present_layer(uint32_t w, uint32_t h) { atomic_store(&g_layer_shape, (uint64_t)w | (uint64_t)h << 32); } void mtl_present_screen(uint32_t w, uint32_t h) { atomic_store(&g_screen_shape, (uint64_t)w | (uint64_t)h << 32); } -static void present_shape(id src, double *w, double *h) { +static void present_shape(id src, const int crop[4], double *w, double *h) { uint64_t v = atomic_load(&g_screen_shape); - if (v && fabs(((double)src.width / src.height) / ((double)(uint32_t)v / (double)(v >> 32)) - 1) < 0.01) { *w = src.width; *h = src.height; return; } + double sr = v ? (double)(uint32_t)v / (double)(v >> 32) : 0; + if (v && crop && crop[2] > 0 && crop[3] > 0 && fabs(((double)crop[2] / crop[3]) / sr - 1) < 0.01) { *w = crop[2]; *h = crop[3]; return; } + if (v && fabs(((double)src.width / src.height) / sr - 1) < 0.01) { *w = src.width; *h = src.height; return; } v = atomic_load(&g_layer_shape); *w = v ? (double)(uint32_t)v : 1920; *h = v ? (double)(v >> 32) : 1080; } -static MTLViewport present_fit(id src, double dw, double dh) { double w, h; present_shape(src, &w, &h); return present_viewport(w, h, dw, dh); } +static MTLViewport present_fit(id src, const int crop[4], double dw, double dh) { double w, h; present_shape(src, crop, &w, &h); return present_viewport(w, h, dw, dh); } // The valid source rectangle of a present (x, y, width, height; the crop of nvnWindowSetCrop, zero = the whole texture); 0 when it lies outside the texture static int present_crop_rect(id src, const int crop[4], NSUInteger r[4]) { r[0] = 0; r[1] = 0; r[2] = src.width; r[3] = src.height; @@ -16434,7 +16438,7 @@ static int present_encode(id cb, id src, id cb, id src, id tex = present_raw_src(src), up = fx ? fx_encode(fx, cb, tex, r, vp, dst.pixelFormat, fx == &g_fx_present) : nil; if (up) { tex = up; r[0] = r[1] = 0; r[2] = up.width; r[3] = up.height; } // the scaled picture, the viewport's size: drawn 1:1 @@ -16761,7 +16765,7 @@ static int present_now(const tex_t *t, double period, uint64_t frame, uint64_t t if (!src) { RED_ONCE(8, "present texture could not get an MTLTexture"); return 0; } // The picture's shape for the front end's layout (front.h front_picture_shape): reported when its width / height changes by more than 0.1 %, not on a size change of the same // shape (1280x720 handheld, 1920x1080 docked) - { static double last; double w, h; present_shape(src, &w, &h); double a = w / h; + { static double last; double w, h; present_shape(src, crop, &w, &h); double a = w / h; if (fabs(a - last) > last * 0.001) { last = a; MLOG("present: picture shape %.0fx%.0f (%.3f:1) for the front end's layout", w, h, a); front_picture_shape((int)w, (int)h); } } // A texture made on first resolution joins the residency set (res_add), which a submission commits; a window texture presented before any submission has used it was still // outside the committed set when the display command buffer sampled it. Super Smash Bros. Ultimate presents its window texture in its first frames before drawing into it: diff --git a/dm/gfx/tests/t_present_crop.m b/dm/gfx/tests/t_present_crop.m index ee10e11145..b13edd0170 100644 --- a/dm/gfx/tests/t_present_crop.m +++ b/dm/gfx/tests/t_present_crop.m @@ -253,7 +253,7 @@ int main(void) { @autoreleasepool { for(int y=0;y=crop54[2]||y>=crop54[3]],4); [sw replaceRegion:MTLRegionMake2D(0,0,TW,TH) mipmapLevel:0 withBytes:sp bytesPerRow:TW*4];free(sp);res_add(sw);res_commit(); double sa,sb; - mtl_present_layer(0,0);present_shape(sw,&sa,&sb);EXPECT("no sized layer: the display's width",1920,(int)sa);EXPECT("no sized layer: the display's height",1080,(int)sb); + mtl_present_layer(0,0);present_shape(sw,NULL,&sa,&sb);EXPECT("no sized layer: the display's width",1920,(int)sa);EXPECT("no sized layer: the display's height",1080,(int)sb); for(int m4=0;m4<2;m4++) { int outH[2]={H,40}; // 16:9 fills; a 16:10 canvas keeps the bars above and below for(int k=0;k<2;k++) { @@ -267,12 +267,30 @@ int main(void) { @autoreleasepool { EXPECT(k?"5:4 crop on a 16:10 canvas: 16:9 of red, black above and below":"5:4 crop on a 16:9 canvas: all red, no side bars",0,bad);free(o);res_remove(ot); } } - mtl_present_layer(4,3);present_shape(sw,&sa,&sb);EXPECT("a sized layer gives its own shape",4*(int)sb,3*(int)sa);mtl_present_layer(0,0); + mtl_present_layer(4,3);present_shape(sw,NULL,&sa,&sb);EXPECT("a sized layer gives its own shape",4*(int)sb,3*(int)sa);mtl_present_layer(0,0); MTLTextureDescriptor *td2=[MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA8Unorm width:1920 height:1344 mipmapped:NO];td2.storageMode=MTLStorageModePrivate;td2.usage=MTLTextureUsageShaderRead; id tall=[g_dev newTextureWithDescriptor:td2]; - present_shape(tall,&sa,&sb);EXPECT("a taller texture without DM_SCREEN_ASPECT: 16:9",1080,(int)sb); - mtl_present_screen(1920,1344);present_shape(tall,&sa,&sb);EXPECT("DM_SCREEN_ASPECT: a texture that follows it keeps the taller shape",1344,(int)sb); - present_shape(sw,&sa,&sb);EXPECT("DM_SCREEN_ASPECT: a 16:9 texture of a build without stubs stays 16:9",1080,(int)sb);mtl_present_screen(0,0); + present_shape(tall,NULL,&sa,&sb);EXPECT("a taller texture without DM_SCREEN_ASPECT: 16:9",1080,(int)sb); + mtl_present_screen(1920,1344);present_shape(tall,NULL,&sa,&sb);EXPECT("DM_SCREEN_ASPECT: a texture that follows it keeps the taller shape",1344,(int)sb); + present_shape(sw,NULL,&sa,&sb);EXPECT("DM_SCREEN_ASPECT: a 16:9 texture of a build without stubs stays 16:9",1080,(int)sb);mtl_present_screen(0,0); + // A crop of the screen's shape inside larger window textures (Ocarina of Time keeps its window textures and moves its crop): the picture takes the crop's shape, wider or + // taller than 16:9; a crop of another shape, or no screen shape set, keeps 16:9 (the crop only picks texels) + { const int wide[4]={0,0,56,24}; + mtl_present_screen(7,3);present_shape(sw,wide,&sa,&sb);EXPECT("a wider screen: a crop of its shape, width",56,(int)sa);EXPECT("a wider screen: a crop of its shape, height",24,(int)sb); + present_shape(sw,crop54,&sa,&sb);EXPECT("a wider screen: a crop of another shape stays 16:9",1080,(int)sb); + mtl_present_screen(5,4);present_shape(sw,crop54,&sa,&sb);EXPECT("a taller screen: a crop of its shape, width",40,(int)sa);EXPECT("a taller screen: a crop of its shape, height",32,(int)sb); + mtl_present_screen(0,0);present_shape(sw,wide,&sa,&sb);EXPECT("no screen shape: a wide crop stays 16:9",1080,(int)sb); + // drawn: the 5:4 crop under a 5:4 screen fills a 5:4 canvas (all red, no bars), on the classic and the Metal 4 present + mtl_present_screen(5,4); + for(int m4=0;m4<2;m4++){ + MTLTextureDescriptor *od=[MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA8Unorm width:50 height:40 mipmapped:NO];od.storageMode=MTLStorageModeShared;od.usage=MTLTextureUsageRenderTarget; + id ot=[g_dev newTextureWithDescriptor:od];res_add(ot);res_commit(); + if(m4)present_offscreen_wait(sw,crop54,ot);else{id cb=[g_q commandBuffer];present_encode(cb,sw,ot,crop54);[cb commit];[cb waitUntilCompleted];} + uint8_t *o=malloc(50*40*4);[ot getBytes:o bytesPerRow:50*4 fromRegion:MTLRegionMake2D(0,0,50,40) mipmapLevel:0];int bad=0; + for(int i=0;i<50*40;i++)bad+=memcmp(o+4*i,colors[0],4)!=0;free(o);res_remove(ot); + EXPECT(m4?"Metal 4: a crop of the screen's shape fills a canvas of that shape":"classic: a crop of the screen's shape fills a canvas of that shape",0,bad); + } + mtl_present_screen(0,0); } // The front end's layout and the mouse follow the same shape, not the crop's (front_picture_shape): a real present of the 5:4 crop tex_t st={.magic=MAGIC('T', 'E', 'X', 0), .serial =10002};st.d.width=TW;st.d.height=TH;st.d.depth=st.d.levels=1;st.d.format=NVN_FORMAT_RGBA8;st.d.target=NVN_TEXTURE_TARGET_2D;serial_put(st.serial,sw); win_t w3={.ntex=1};w3.tex[0]=(NVNtexture *)&st;set((NVNwindow *)&w3,crop54[0],crop54[1],crop54[2],crop54[3]); From cf623eed2fb4e229ec634839ad942f8fc964beba Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 18:05:26 +0100 Subject: [PATCH 06/40] screen_aspect any: a profile's stubs follow any window or screen shape while the game runs, from 0.25 to 6, portrait included game.py: screen_aspect off / on / any (DM_GAME_SCREEN_ASPECT 0 / 1 / 2); on (Breath of the Wild, Animal Crossing: taller than 16:9, set at the start) is unchanged. loader.c (screen_any_apply / screen_any_write / dm_screen_view_changed): for an any profile, DM_SCREEN_ASPECT unset or auto follows the front end's view, 1.777 or 0 is 16:9, a number fixes the shape. It writes the state page: +0xa0 the ratio (0 = 16:9), +0xa4 / +0xa8 the width and height factors, +0xac a change count, +0xc4 / +0xc8 the camera's x / y multipliers (wider than 16:9 keeps the vertical angle, between 1:1 and 16:9 the horizontal one; upright takes the scale of the same screen turned landscape), tells the present the shape, and on a change pushes the display-change notifications (hle_oe.c dm_display_changed) so the game lays out again. front_mac.m reports the window's shape 300 ms after a resize settles; front_ios.m the app window's in landscape (upright stays 16:9 for now). settings.tsv: the Picture shape setting for any profiles (Match window / Match screen, or 16:9), auto by default. Front-ends: new: front_view_aspect() (front.h) answers the game view's width / height and dm_screen_view_changed(aspect) reports a settled resize; a profile with screen_aspect any (Ocarina of Time) follows them under DM_SCREEN_ASPECT auto, and renders 16:9 until a front end provides them (the loader's weak front_view_aspect returns 0). --- dm/games/game.py | 2 +- dm/games/settings.tsv | 3 ++- dm/host/front.h | 6 +++++ dm/host/front_mac.m | 10 +++++++ dm/host/hle_oe.c | 7 +++++ dm/host/loader.c | 53 +++++++++++++++++++++++++++++++++++++- dm/host/tests/t_nro.c | 1 + dm/host/tests/t_symbols.c | 1 + dm/ios/Sources/front_ios.m | 15 ++++++++++- 9 files changed, 94 insertions(+), 4 deletions(-) diff --git a/dm/games/game.py b/dm/games/game.py index b7c198fd62..1699336e06 100644 --- a/dm/games/game.py +++ b/dm/games/game.py @@ -236,7 +236,7 @@ def header_one(g): f'#define DM_GAME_LAUNCH_PARAM {q(g.get("launch_param", ""))}', # what oe::TryPopLaunchParameter hands over once (hle_oe.c; a program another program of the application launches: Super Mario 3D All-Stars) f'#define DM_GAME_MEM_MAP_LIMIT {int(mml == "on")}', f'#define DM_GAME_STDALLOC_CAP {int(g.get("stdalloc_cap", "0"), 0):#x}ull', # StandardAllocator's address space with virtual address memory (hle_svc.c); 0 = what the alias region has room for - f'#define DM_GAME_SCREEN_ASPECT {("off", "on").index(g.get("screen_aspect", "off"))}', # stubs that follow a screen taller than 16:9 (DM_SCREEN_ASPECT): off / on + f'#define DM_GAME_SCREEN_ASPECT {("off", "on", "any").index(g.get("screen_aspect", "off"))}', # stubs that follow the screen's shape (DM_SCREEN_ASPECT): off / on (taller than 16:9, set at the start) / any (any shape from 0.25 to 6, followed while the game runs) f'#define DM_GAME_DISPLAY_VERSIONS {vers}', f'#define DM_GAME_PRESENT_DISPLAY {int(pi == "display")}', f'#define DM_GAME_CORE_PREEMPT {int(cp == "on")}', diff --git a/dm/games/settings.tsv b/dm/games/settings.tsv index 15f4824107..10ceb8f17b 100644 --- a/dm/games/settings.tsv +++ b/dm/games/settings.tsv @@ -30,7 +30,8 @@ DM_RUMBLE_GAIN range 0:3:0.1 1.0 always Controls Rumble strength needs=DM_RUMBL DM_RUMBLE_MIN range 0:0.9:0.05 0 always Controls Rumble cutoff DM_PAD_PRESS_TO_JOIN toggle - 0 always Controls Press a button to join DM_SW2 toggle - 0 platform:mac Controls Switch 2 controllers Plays Joy-Con 2 and Pro Controller 2 over Bluetooth: hold Sync on one during the game to pair it. macOS asks once for Bluetooth access. -DM_SCREEN_ASPECT choice auto=Match screen?platform:ios,1.43=iPhone Duo screen (1.43:1),0=16:9 (Switch) auto|0 DM_GAME_SCREEN_ASPECT!=0 Game Picture shape Renders about a quarter more pixels: a little slower and warmer. +DM_SCREEN_ASPECT choice auto=Match screen?platform:ios,1.43=iPhone Duo screen (1.43:1),0=16:9 (Switch) auto|0 DM_GAME_SCREEN_ASPECT==1 Game Picture shape Renders about a quarter more pixels: a little slower and warmer. +DM_SCREEN_ASPECT choice auto=Match screen?platform:ios,auto=Match window?platform:mac,1.777=16:9 (Switch) auto DM_GAME_SCREEN_ASPECT==2 Game Picture shape Match fills the screen or the window, wider or taller, with no bars: the picture shows more at the sides or above and below. 16:9 shows the Switch picture. DM_PRESENT_SNAP toggle - DM_GAME_PRESENT_SNAP gfxapi:nvn Performance Frame buffering Up to one frame more display latency. DM_PSO_MTL4 toggle - DM_GAME_PSO_MTL4|1 gfxapi:nvn Performance Metal 4 pipelines Keeps a pipeline archive; uses more memory. DM_PSO_PREBUILD toggle - 1 gfxapi:nvn Performance Pipeline prebuild The first start waits a few minutes for it, once. needs=DM_PSO_MTL4=1 diff --git a/dm/host/front.h b/dm/host/front.h index b657e6c1c0..392bf35540 100644 --- a/dm/host/front.h +++ b/dm/host/front.h @@ -27,6 +27,12 @@ void front_screen_match(int fps); // lays the picture out in that shape instead of 16:9, so a profile with screen_aspect under DM_SCREEN_ASPECT (1920x1344) fills a 1.43:1 screen and touches map onto the picture // as drawn. Before the first call the picture is 16:9. Called from the present thread; the front end switches to its main thread and does not block void front_picture_shape(int w, int h); +// The game view's width / height (front_mac.m: the window's content view; front_ios.m: the app window), 0 when there is no view (a run with no window). Any thread. A profile with +// screen_aspect any renders this shape under DM_SCREEN_ASPECT auto (loader.c) +double front_view_aspect(void); +// The front end's game view changed shape (loader.c): a profile with screen_aspect any under DM_SCREEN_ASPECT auto renders the new shape. Called from the main thread once a resize +// has settled, not for every step of a live resize +void dm_screen_view_changed(double aspect); // Progress screen of the first-start prepare step (nvn_mtl.m warm_prepare: "Preparing shaders" while the pipelines of the prewarm lists are built before the game's first frame): // title and done of total shown over the game's view (a dark screen, the title, a bar and a count), updated on each call; title NULL hides it. Any thread; the front end switches to its main // thread and does not block. Nothing is shown without a window diff --git a/dm/host/front_mac.m b/dm/host/front_mac.m index 278bf27240..fb31b71ff5 100644 --- a/dm/host/front_mac.m +++ b/dm/host/front_mac.m @@ -620,6 +620,15 @@ static void pads_start(void) { // The picture's width / height (front.h front_picture_shape; 16:9 until the first present), which the present fits into the view static _Atomic double g_picture_aspect = 16.0 / 9.0; void front_picture_shape(int w, int h) { if (w > 0 && h > 0) g_picture_aspect = (double)w / h; } +// The game view's width / height (front.h front_view_aspect; 0 before the window), and the shape changes handed to the host once a resize has settled (dm_screen_view_changed): +// a live resize steps through many sizes, and each new shape makes a profile with screen_aspect any lay its picture out again +static _Atomic double g_view_aspect; static uint64_t g_view_change; // g_view_change: main thread +double front_view_aspect(void) { return g_view_aspect; } +static void view_shape(NSSize px) { + double a = px.width / px.height; if (a == g_view_aspect) return; + g_view_aspect = a; uint64_t n = ++g_view_change; + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 300 * NSEC_PER_MSEC), dispatch_get_main_queue(), ^{ if (n == g_view_change) dm_screen_view_changed(a); }); +} @interface DmGameView : NSView @end @implementation DmGameView - (void)mouseDown:(NSEvent *)e { [self mouseDragged:e]; } @@ -641,6 +650,7 @@ - (void)fitDrawable { l.contentsScale = self.window.backingScaleFactor; NSSize px = [self convertSizeToBacking:self.bounds.size]; if (px.width < 1 || px.height < 1 || CGSizeEqualToSize(l.drawableSize, px)) return; l.drawableSize = px; LOG("front end: drawable %.0fx%.0f", px.width, px.height); + view_shape(px); } - (void)setFrameSize:(NSSize)s { [super setFrameSize:s]; [self fitDrawable]; } - (void)viewDidChangeBackingProperties { [super viewDidChangeBackingProperties]; [self fitDrawable]; } diff --git a/dm/host/hle_oe.c b/dm/host/hle_oe.c index b5264cf26a..d4a57ad1bf 100644 --- a/dm/host/hle_oe.c +++ b/dm/host/hle_oe.c @@ -64,6 +64,13 @@ void dm_set_opmode(int m) { // called by the front end (overheat downgrade etc if (g_ntf_opmode) mq_push(MSG_OPMODE); if (g_ntf_perfmode) mq_push(MSG_PERFMODE); } +// The picture's shape changed (loader.c, a profile with screen_aspect any following its window): the notifications a console sends when the display changes, docked to handheld, +// with the mode unchanged, so the game queries its display again and lays its picture out for the new shape as it does for a mode change (Ocarina of Time resizes its targets then) +void dm_display_changed(void) { + LOG("oe: display changed (picture shape): operation mode notification%s", g_ntf_opmode ? "" : " not enabled by the game"); + if (g_ntf_opmode) mq_push(MSG_OPMODE); + if (g_ntf_perfmode) mq_push(MSG_PERFMODE); +} // FocusState: 1 InFocus, 2 OutOfFocus, 3 Background. The front end calls dm_focus_set (iOS lifecycle / Mac window losing focus) static _Atomic int g_focus = 1; static _Atomic uint64_t g_focus_in_ns; // the front end's state; time of its most recent entry into InFocus // Focus handling mode the game set (-1: none). The SDK's oe::SetFocusHandlingMode (SDK 18 sdk+0x3c5770) turns it into am's SetFocusHandlingMode(notify, ?, suspend) and diff --git a/dm/host/loader.c b/dm/host/loader.c index ce63ce79cd..9d6921a537 100644 --- a/dm/host/loader.c +++ b/dm/host/loader.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include "hle.h" @@ -314,6 +315,55 @@ static void install_fps_state(module_t *m) { dm_fps_state = m->base + m->fps_state_off; g_fps_mod = m->name; dm_fps_step_max = (int)m->step_max; if (m->render_scale_max > 1) g_render_scale_max = (float)m->render_scale_max; } +// Screen shape for a profile whose stubs take any shape (game.tsv screen_aspect any, DM_GAME_SCREEN_ASPECT 2: Ocarina of Time's patches.py), wider or taller than 16:9, and follow it +// while the game runs. State +0xa0 float the screen's width / height (0 = 16:9: the stubs run the game's own code), +0xa4 float the width factor (the ratio over 16:9, at least 1), +// +0xa8 float the height factor (16:9 over the ratio, at least 1), +0xac u32 a count bumped at each change, +0xc4 / +0xc8 floats the camera's x / y multipliers (below). A wider screen keeps the game's height and vertical angle and shows more +// at the sides; a taller one keeps its width and horizontal angle and shows more above and below, whatever the shape, as a PC game in a window does: a portrait window too (the +// profile's stubs keep the picture's pixel count in bounds and grow the window textures). The bounds below only keep a degenerate window (a few pixels high) out +#define SCREEN_MIN 0.25 +#define SCREEN_MAX 6.0 +static int g_screen_follow; // DM_SCREEN_ASPECT auto (the default): the front end's view gives the shape, now and when it changes +static double g_screen_ratio = -1; // what the state page holds (0 = 16:9), -1 = not written yet +__attribute__((weak)) double front_view_aspect(void) { return 0; } // front.h: the game view's width / height, 0 = no view (a run with no window); front_mac.m, front_ios.m +static pthread_mutex_t g_screen_mu = PTHREAD_MUTEX_INITIALIZER; // the game's start (main.c) and the front end's main thread both write the shape +static void screen_any_write_(double a, const char *why); +static void screen_any_write(double a, const char *why) { pthread_mutex_lock(&g_screen_mu); screen_any_write_(a, why); pthread_mutex_unlock(&g_screen_mu); } +static void screen_any_write_(double a, const char *why) { + if (!dm_fps_state) return; + double r = a < SCREEN_MIN ? SCREEN_MIN : a > SCREEN_MAX ? SCREEN_MAX : a; + if (fabs(r / (16.0 / 9.0) - 1) < 0.005) r = 0; // 16:9 within half a percent: the game's own picture + if (r == g_screen_ratio) return; + float f[3] = {(float)r, r ? (float)fmax(r * 9 / 16, 1) : 1, r ? (float)fmax(16 / (9 * r), 1) : 1}; + memcpy(dm_fps_state + 0xa0, f, sizeof f); + // +0xc4 / +0xc8: the camera's x / y multipliers (the projection's rows). Landscape: wider than 16:9 keeps the vertical angle (x × 16:9 / r), narrower keeps the horizontal one + // (y × r / 16:9). Portrait takes the scale of the same screen turned landscape, its width getting what that one's height gets: a phone or a window turned keeps the + // characters' size and shows another part of the world. The size stub copies them with the size it sets (patches.py screen_shape) + { double l = r >= 1 ? r : 1 / r, xl = l >= 16.0 / 9 ? 16.0 / 9 / l : 1, yl = l >= 16.0 / 9 ? 1 : l * 9 / 16; + float c[2] = {r == 0 ? 1 : r >= 1 ? (float)xl : (float)(yl * 16 / 9), r == 0 ? 1 : r >= 1 ? (float)yl : (float)(xl * 9 / 16)}; + memcpy(dm_fps_state + 0xc4, c, sizeof c); } + uint32_t gen; memcpy(&gen, dm_fps_state + 0xac, 4); gen++; memcpy(dm_fps_state + 0xac, &gen, 4); + extern void mtl_present_screen(uint32_t w, uint32_t h); mtl_present_screen(r ? (uint32_t)(r * 10000 + 0.5) : 0, r ? 10000 : 0); // the picture's shape for a crop of that shape (nvn_mtl.m present_shape) + int first = g_screen_ratio < 0; g_screen_ratio = r; + if (r) LOG("%s: screen shape %.3f (%s%s): %s, picture %.3f x %.3f of 16:9", g_fps_mod, r, why, a != r && a > 0 ? ", held to 0.25..6" : "", r > 16.0 / 9 ? "more at the sides" : "more above and below", f[1], f[2]); + else LOG("%s: screen shape 16:9 (%s)", g_fps_mod, why); + if (!first) { void dm_display_changed(void); dm_display_changed(); } // the game lays its picture out again (hle_oe.c) +} +static void screen_any_apply(const char *as) { + if (!as || !*as || !strcmp(as, "auto")) { // the default: follow the view (16:9 until there is one) + g_screen_follow = 1; double v = front_view_aspect(); screen_any_write(v > 0 ? v : 16.0 / 9.0, v > 0 ? "the window's shape" : "no window yet"); + return; + } + char *end; double a = strtod(as, &end); + if (!strcmp(as, "0")) a = 16.0 / 9.0; + else if (end == as || *end || !(a >= SCREEN_MIN && a <= SCREEN_MAX)) { LOG("[RED] DM_SCREEN_ASPECT=%s: auto, 0 or a width / height (0.25 to 6); keeping 16:9", as); a = 16.0 / 9.0; } + screen_any_write(a, "DM_SCREEN_ASPECT"); +} +// front.h: the front end's game view changed shape (a window resized, full screen, a rotated or resized iPad window). Followed only under DM_SCREEN_ASPECT auto by a profile with +// screen_aspect any; a fixed DM_SCREEN_ASPECT keeps its shape +void dm_screen_view_changed(double aspect) { + if (DM_GAME_SCREEN_ASPECT != 2 || !g_screen_follow || g_screen_ratio < 0 || !(aspect > 0)) return; + screen_any_write(aspect, "the window's shape"); +} // The settings in the state page, written when the game starts (main.c, after host loading): on iOS the loading runs at app launch, before the launcher has set DM_FPS_TARGET and // DM_RENDER_SCALE (Launcher.swift), and nothing reads the page before the game runs void dm_fps_state_apply(void) { @@ -336,7 +386,8 @@ void dm_fps_state_apply(void) { // (Breath of the Wild 1.9.0's patches.py) render that shape: the output keeps its width (1920) and takes the height that gives the ratio, a multiple of 48 so the // 2/3-size layout space (1280 wide) stays whole: 1.43 → 1920x1344 and 1280x896. State +0x48 u32 output height, +0x4c u32 layout height, +0x50 its float; 0 = 16:9 const char *as = getenv("DM_SCREEN_ASPECT"); - if (as && *as && strcmp(as, "0") && strcmp(as, "auto")) { // 0: 16:9; auto: the iOS front end resolves it to the screen's shape (a Mac window has none) + if (DM_GAME_SCREEN_ASPECT == 2) screen_any_apply(as); // a profile whose stubs take any shape (screen_aspect any): below + else if (as && *as && strcmp(as, "0") && strcmp(as, "auto")) { // 0: 16:9; auto: the iOS front end resolves it to the screen's shape (a Mac window has none) char *end; float a = strtof(as, &end); if (end == as || *end || !(a >= 1.3f && a < 1.77f)) LOG("[RED] DM_SCREEN_ASPECT=%s: a width / height from 1.3 to 1.77; keeping 16:9", as); else { diff --git a/dm/host/tests/t_nro.c b/dm/host/tests/t_nro.c index c34d74d37c..1be29a604c 100644 --- a/dm/host/tests/t_nro.c +++ b/dm/host/tests/t_nro.c @@ -7,6 +7,7 @@ void dm_thread_service_connect(void) {} // threads started so far have nothing #include #include void mtl_present_screen(uint32_t w, uint32_t h) { (void)w; (void)h; } // loader.c: a screen taller than 16:9; no renderer here +void dm_display_changed(void) {} // loader.c: a profile with screen_aspect any changed shape (hle_oe.c pushes the display notifications); no game here struct dm_flags g_flag; uint64_t now_ns(void) { return 0; } diff --git a/dm/host/tests/t_symbols.c b/dm/host/tests/t_symbols.c index 7aa8dd1d2f..bd88fee595 100644 --- a/dm/host/tests/t_symbols.c +++ b/dm/host/tests/t_symbols.c @@ -21,6 +21,7 @@ static uint8_t *test_section(const struct mach_header_64 *mh, const char *seg, c #include "../hle.c" #include "../loader.c" void mtl_present_screen(uint32_t w, uint32_t h) { (void)w; (void)h; } // loader.c: a screen taller than 16:9; no renderer here +void dm_display_changed(void) {} // loader.c: a profile with screen_aspect any changed shape (hle_oe.c pushes the display notifications); no game here #undef getsectiondata struct dm_flags g_flag; diff --git a/dm/ios/Sources/front_ios.m b/dm/ios/Sources/front_ios.m index 1495d647e0..5bfce204a1 100644 --- a/dm/ios/Sources/front_ios.m +++ b/dm/ios/Sources/front_ios.m @@ -96,6 +96,16 @@ - (void)showFPS:(BOOL)on; @end static DmRootVC *g_root; static CGFloat g_picture_aspect = DM_PICTURE_16_9; // the presented picture's width / height (front_picture_shape); main thread +// The game view's shape for a profile with screen_aspect any (front.h front_view_aspect, dm_screen_view_changed): the app window's width / height in landscape (an iPad's 1.43, an +// iPhone's 2.17, a window under Stage Manager), 16:9 in portrait, where the picture sits above the pad. A change is handed to the host once the layout has settled for 300 ms +static _Atomic double g_view_aspect; static uint64_t g_view_change; // g_view_change: main thread +double front_view_aspect(void) { return g_view_aspect; } +static void view_shape(CGSize sz, int notify) { + double a = sz.width > sz.height && sz.height > 0 ? sz.width / sz.height : 16.0 / 9.0; if (a == g_view_aspect) return; + g_view_aspect = a; if (!notify) return; + uint64_t n = ++g_view_change; + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 300 * NSEC_PER_MSEC), dispatch_get_main_queue(), ^{ if (n == g_view_change) dm_screen_view_changed(a); }); +} static void launcher_done(void) { [g_root startGame]; } @implementation DmRootVC - (void)loadView { self.view = [DmRootView new]; } @@ -154,7 +164,9 @@ - (void)startGame { // takes 1.3 to 1.77), else 16:9. The window's, not the screen's: on an iPad under Stage Manager or Split View the app does not fill the screen. Resolved here, before the // host writes the state page (main.c dm_fps_state_apply) const char *a = getenv("DM_SCREEN_ASPECT"); - if (a && !strcmp(a, "auto")) { + if (DM_GAME_SCREEN_ASPECT == 2) { // a profile whose stubs take any shape (screen_aspect any) follows the window itself, wider or taller (loader.c, front_view_aspect) + CGSize sz = self.view.window ? self.view.window.bounds.size : UIScreen.mainScreen.bounds.size; view_shape(sz, 0); + } else if (a && !strcmp(a, "auto")) { CGSize sz = self.view.window ? self.view.window.bounds.size : UIScreen.mainScreen.bounds.size; double r = MAX(sz.width, sz.height) / MAX(MIN(sz.width, sz.height), 1); char v[16]; if (r >= 1.3 && r < 1.77) snprintf(v, sizeof v, "%.3f", r); else strcpy(v, "0"); setenv("DM_SCREEN_ASPECT", v, 1); LOG("front end: picture shape auto: window %.0fx%.0f points (%.3f:1) → DM_SCREEN_ASPECT=%s", sz.width, sz.height, r, v); @@ -182,6 +194,7 @@ - (void)startGame { - (void)viewDidLayoutSubviews { [super viewDidLayoutSubviews]; CGRect b = self.view.bounds; if (!self.game) return; + view_shape(b.size, 1); CGRect division = CGRectNull; // an iPhone Duo's fold, as UIKit reports it (iOS 27.1) NSMutableArray *regionLog = [NSMutableArray array]; #if __has_include() From 021300a7ab82dd5446627233bc77236bd482d083 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 18:05:26 +0100 Subject: [PATCH 07/40] Probe diagnostics for values the game caches: float arguments, a scan for matrices and float pairs in guest memory, and a write watch armed from a probe's register DM_PROBE_FP prints s0-s7 on probe hits. DM_PROBE_FINDMAT=[,] scans the guest heap for perspective matrices and arms the write watch on the n-th at 16:9; DM_PROBE_FINDF=,,[,[,]] scans the heap and the alias region's mapped parts for two floats and arms it on one. DM_PROBE_WATCH=@xN+[+] arms the watch on that register plus the offset at the first printed probe hit, for objects whose addresses move between runs (Ocarina of Time's layouts in the alias region); DM_PROBE_XREG=N lets DM_PROBE_X2 dump xN's memory. The watch carries out CAS and single-lane ST1 itself. Found Ocarina of Time's camera copy and HUD screen size. --- dm/host/probe.c | 169 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 164 insertions(+), 5 deletions(-) diff --git a/dm/host/probe.c b/dm/host/probe.c index 745c78b17f..138b4b8366 100644 --- a/dm/host/probe.c +++ b/dm/host/probe.c @@ -16,6 +16,7 @@ #include #include #include +#include void crash_set_trap_hook(int (*fn)(ucontext_t *uc)); #if !TARGET_OS_IPHONE // Syntax module:offset[@x0 offset]. Put a brk at the guest address; on hit print x0..x3 (and the readable strings they point to) and a backtrace, then execute the original instruction from a trampoline slot and jump back. @@ -107,6 +108,7 @@ static void probe_dump_once(void) { fwrite(m->base + strtoull(o, 0, 16), 1, strtoull(n, 0, 16), fp); fclose(fp); LOG("probe dump %s+%s length %s → %s", mod, o, n, f ? f : def); } +static void watch_from_hit(ucontext_t *uc); // DM_PROBE_WATCH=@xN+: armed from the first printed probe hit (below) static int probe_hit(ucontext_t *uc) { // Registered by probe_install with the SIGTRAP branch of crash.c uint64_t pc = uc->uc_mcontext->__ss.__pc; for (int i = 0; i < g_nprobe; i++) if (g_probe[i].at == pc) { @@ -126,11 +128,13 @@ static int probe_hit(ucontext_t *uc) { // Registered by probe_install with the if (!strstr(nm, g_hw.want)) { uc->uc_mcontext->__ss.__pc = (uint64_t)g_probe[i].slot; return 1; } } probe_dump_once(); + watch_from_hit(uc); flockfile(stderr); // One block per hit when threads hit at once { extern uint64_t g_t0_ns; uint64_t now_ns(void); uint32_t thr_cur_handle(void); ++g_probe[i].hits; fprintf(stderr, "[dm] PROBE %s #%u [%.4fs] F=%llu T=%x", g_probe[i].name, g_probe_every ? g_probe[i].seen : g_probe[i].hits, (now_ns() - g_t0_ns) / 1e9, (unsigned long long)frame, thr_cur_handle()); } for (int r = 0; r < 8; r++) { uint64_t v = uc->uc_mcontext->__ss.__x[r]; fprintf(stderr, " x%d=%#llx", r, v); probe_str(v); } for (int r = 19; r < 24; r++) fprintf(stderr, " x%d=%#llx", r, uc->uc_mcontext->__ss.__x[r]); if (getenv("DM_PROBE_ALL")) for (int r = 8; r < 19; r++) fprintf(stderr, " x%d=%#llx", r, uc->uc_mcontext->__ss.__x[r]); // also print x8–x18 + if (getenv("DM_PROBE_FP")) for (int r = 0; r < 8; r++) { float f; memcpy(&f, &uc->uc_mcontext->__ns.__v[r], 4); fprintf(stderr, " s%d=%g", r, f); } // also print s0–s7 (float arguments) fprintf(stderr, " lr=%#llx", uc->uc_mcontext->__ss.__lr); { uint64_t x8 = uc->uc_mcontext->__ss.__x[8]; Dl_info di; fprintf(stderr, " x8=%#llx", x8); // x8: indirect call targets are often here (probe placed before blr x8) if (dladdr((void *)x8, &di) && di.dli_sname) fprintf(stderr, " (%s+%#lx)", di.dli_sname, (unsigned long)(x8 - (uint64_t)di.dli_saddr)); } @@ -145,9 +149,9 @@ static int probe_hit(ucontext_t *uc) { // Registered by probe_install with the fputc('=', stderr); for (int k = 0; k < 16; k++) fprintf(stderr, "%02x", b[k]); } } fputc('\n', stderr); - { static long xd = -2; if (xd == -2) { const char *e = getenv("DM_PROBE_X2"); xd = e ? strtol(e, 0, 16) : -1; } // DM_PROBE_X2=bytes: also print the memory x2 points to (to read object fields) - if (xd > 0) { uint64_t at = uc->uc_mcontext->__ss.__x[2]; uint8_t b[512]; vm_size_t got = 0; long n = xd > 512 ? 512 : xd; - if (!vm_read_overwrite(mach_task_self(), at, n, (vm_address_t)b, &got)) { fprintf(stderr, "[dm] [x2]:"); for (long k = 0; k < (long)got; k += 8) { uint64_t v; memcpy(&v, b + k, 8); fprintf(stderr, "%s%llx", k % 32 ? " " : "\n[dm] ", (unsigned long long)v); } fputc('\n', stderr); } } } + { static long xd = -2, xr = 2; if (xd == -2) { const char *e = getenv("DM_PROBE_X2"); xd = e ? strtol(e, 0, 16) : -1; e = getenv("DM_PROBE_XREG"); if (e) xr = strtol(e, 0, 10) & 31; } // DM_PROBE_X2=bytes: also print the memory x2 points to (to read object fields); DM_PROBE_XREG=N: xN's instead + if (xd > 0) { uint64_t at = uc->uc_mcontext->__ss.__x[xr]; uint8_t b[512]; vm_size_t got = 0; long n = xd > 512 ? 512 : xd; + if (!vm_read_overwrite(mach_task_self(), at, n, (vm_address_t)b, &got)) { fprintf(stderr, "[dm] [x%ld]:", xr); for (long k = 0; k < (long)got; k += 8) { uint64_t v; memcpy(&v, b + k, 8); fprintf(stderr, "%s%llx", k % 32 ? " " : "\n[dm] ", (unsigned long long)v); } fputc('\n', stderr); } } } if (g_probe[i].hreg >= 0) probe_heapwalk(uc->uc_mcontext->__ss.__x[g_probe[i].hreg]); dm_backtrace(uc->uc_mcontext->__ss.__fp); funlockfile(stderr); @@ -184,6 +188,7 @@ static void probe_add(module_t *m, uint64_t off, long dump, probe_fn fn) { // (a store in that millisecond goes unseen). STEINS;GATE RE:BOOT: the engine object's page takes bursts of stores every frame, and a watch that unprotected the page for each fault missed // the one that matters; this one found it at once (main+0x389e90, a setter its start-up code calls with 0: the game renders 1280x720 docked by its own choice) static uint64_t g_wat, g_wlen, g_wpg, g_wpsz; static uint8_t *g_walias; static _Atomic int g_warmed; +static unsigned g_wcap; // printed watch hits left, 0 = no cap (DM_PROBE_FINDMAT) static void watch_where(const char *tag, uint64_t a) { module_t *m = module_of((void *)a); Dl_info di; if (m) fprintf(stderr, " %s=%s+%#llx", tag, m->name, (unsigned long long)(a - (uint64_t)m->base)); @@ -199,7 +204,16 @@ static int watch_decode(ucontext_t *uc, uint32_t in, uint64_t *ea, unsigned *len unsigned rt = in & 31, rn = (in >> 5) & 31, v = (in >> 26) & 1; *wb = -1; if ((in & 0xffffffe0) == 0xd50b7420) { // DC ZVA: zero the 64-byte block *ea = XT(rt) & ~63ull; *len = 64; memset(data, 0, 64); return 1; } - if ((in & 0x3a000000) == 0x28000000) { // STP / STNP: opc 101 V 0 type L imm7 + if ((in & 0xbfff0000) == 0x0d000000 && !(((in >> 13) & 7) & 1)) { // ST1 (single structure, no offset): one lane of Vt; opcode 000 / 010 / 100 = byte / half / word or double + unsigned op = (in >> 13) & 7, q = (in >> 30) & 1, s = (in >> 12) & 1, size = (in >> 10) & 3, sz, idx; + if (op == 0) { sz = 1; idx = q << 3 | s << 2 | size; } + else if (op == 2) { if (size & 1) return 0; sz = 2; idx = q << 2 | s << 1 | size >> 1; } + else if (size == 0) { sz = 4; idx = q << 1 | s; } + else if (size == 1 && !s) { sz = 8; idx = q; } + else return 0; + *ea = XN(rn); *len = sz; memcpy(data, (uint8_t *)&ns->__v[rt] + idx * sz, sz); return 1; + } + if ((in & 0x3a000000) == 0x28000000) { // STP / STNP: opc 101 V 0 type L imm7 if (in & (1u << 22)) return 0; // LDP unsigned opc = in >> 30, type = (in >> 23) & 3, sz = v ? 4u << opc : (opc & 2 ? 8 : 4); if ((!v && opc == 1) || opc == 3) return 0; // STGP / LDPSW, reserved @@ -247,6 +261,8 @@ static int watch_fault(void *addr, int is_write, void *ucv) { } if (wb >= 0) { if (wb == 31) uc->uc_mcontext->__ss.__sp = wbv; else if (wb == 29) uc->uc_mcontext->__ss.__fp = wbv; else if (wb == 30) uc->uc_mcontext->__ss.__lr = wbv; else uc->uc_mcontext->__ss.__x[wb] = wbv; } uc->uc_mcontext->__ss.__pc = pc + 4; + if (hit && g_wcap == 1) hit = 0; // DM_PROBE_FINDMAT's cap reached + else if (hit && g_wcap) g_wcap--; if (hit) { flockfile(stderr); { extern uint64_t g_t0_ns; uint64_t now_ns(void); uint32_t thr_cur_handle(void); @@ -257,6 +273,36 @@ static int watch_fault(void *addr, int is_write, void *ucv) { } return 1; } + // CAS / CASA / CASL / CASAL (one register, any size): carried out atomically through the alias, so a page busy with them stays protected. Ocarina of Time: lock words beside + // the camera's projection matrix took CAS at every frame, and the 1 ms the page was left unprotected for each one hid the projection's own store + { uint32_t in = *(uint32_t *)pc; + if (g_walias && (in & 0x3fa07c00) == 0x08a07c00) { + _STRUCT_ARM_THREAD_STATE64 *ss = &uc->uc_mcontext->__ss; unsigned rs = (in >> 16) & 31, rn = (in >> 5) & 31, rt = in & 31, sz = 1u << (in >> 30); + uint64_t ea = rn == 31 ? ss->__sp : rn == 29 ? ss->__fp : rn == 30 ? ss->__lr : ss->__x[rn]; + #define RX(r) ((r) == 31 ? 0 : (r) == 29 ? ss->__fp : (r) == 30 ? ss->__lr : ss->__x[r]) + uint64_t exp = RX(rs), nv = RX(rt), was = 0; + #undef RX + if (ea >= g_wpg && ea + sz <= g_wpg + g_wpsz && !(ea & (sz - 1))) { + int hit = ea < g_wat + g_wlen && ea + sz > g_wat; uint64_t old = hit ? watch_val() : 0; void *p = g_walias + (ea - g_wpg); + switch (sz) { + case 1: { uint8_t e = (uint8_t)exp; __atomic_compare_exchange_n((uint8_t *)p, &e, (uint8_t)nv, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); was = e; break; } + case 2: { uint16_t e = (uint16_t)exp; __atomic_compare_exchange_n((uint16_t *)p, &e, (uint16_t)nv, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); was = e; break; } + case 4: { uint32_t e = (uint32_t)exp; __atomic_compare_exchange_n((uint32_t *)p, &e, (uint32_t)nv, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); was = e; break; } + default: { uint64_t e = exp; __atomic_compare_exchange_n((uint64_t *)p, &e, nv, 0, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST); was = e; break; } + } + if (rs != 31) { if (rs == 29) ss->__fp = was; else if (rs == 30) ss->__lr = was; else ss->__x[rs] = was; } // Ws is zero-extended: was already is + ss->__pc = pc + 4; + if (hit && watch_val() != old) { + flockfile(stderr); + { extern uint64_t g_t0_ns; uint64_t now_ns(void); uint32_t thr_cur_handle(void); + fprintf(stderr, "[dm] WATCH %#llx [%.4fs] T=%x CAS of %u bytes at %#llx: %#llx → %#llx", (unsigned long long)g_wat, (now_ns() - g_t0_ns) / 1e9, thr_cur_handle(), sz, (unsigned long long)ea, (unsigned long long)old, (unsigned long long)watch_val()); } + watch_where("pc", pc); watch_where("lr", ss->__lr); fputc('\n', stderr); + dm_backtrace(ss->__fp); + funlockfile(stderr); + } + return 1; + } + } } // Exclusive and atomic stores (STXR / STLXR / STLR / STXP, CAS, SWP, LD) address [Xn] with no offset: one that reaches the watched bytes is reported like a store, every time // (its new value lands when it runs on the unprotected page). Tears of the Kingdom's memory-shortage flag is set by such a store, and the page it lies on takes many of them { uint32_t in = *(uint32_t *)pc; int ex = (in & 0x3f000000) == 0x08000000, at = (in & 0x3f200c00) == 0x38200000; // exclusive class (CAS / CASP among them); LD / SWP @@ -281,6 +327,16 @@ static int watch_fault(void *addr, int is_write, void *ucv) { return 1; } static uint64_t g_wrel; static int g_wheap; // heap+: the address is that far past the guest heap's base, resolved once the game opens its heap +// @xN+: the address is that register plus the offset at the first printed probe hit (an object the game allocates where addresses move from run to run, as Ocarina of Time's +// layout resources and panes in the alias region do); the watch arms itself on it as on any other address +static int g_wfrom = -1; static uint64_t g_wfoff; +static void watch_from_hit(ucontext_t *uc) { + if (g_wfrom < 0 || g_wat) return; + uint64_t at = uc->uc_mcontext->__ss.__x[g_wfrom] + g_wfoff; + if ((at & (g_wpsz - 1)) + g_wlen > g_wpsz) { LOG("WATCH: [RED] %#llx + %#llx crosses a host page: not armed", (unsigned long long)at, (unsigned long long)g_wlen); g_wfrom = -1; return; } + g_wpg = at & ~(g_wpsz - 1); g_wat = at; + LOG("write watch from x%d + %#llx = %#llx", g_wfrom, (unsigned long long)g_wfoff, (unsigned long long)at); +} static void *watch_arm(void *p) { (void)p; for (;;) { @@ -352,14 +408,117 @@ static void *poll_run(void *p) { } return NULL; } +// DM_PROBE_FINDMAT=[,]: who builds the camera's projection. That many seconds after start a host thread scans the guest heap for perspective matrices (4x4 floats, +// row or column major, the w row or column 0,0,±1,0, off-diagonal zeros, x and y scales positive) and prints each with its y / x scale ratio (the projection's aspect), then arms the +// write watch above on the n-th (default 0) whose ratio is 16:9 within 0.5 %, its 64 bytes, for 40 printed stores. Heap addresses move between runs, so the watch is armed in the same run. +// Ocarina of Time: the game renders a wider crop but projects at 16:9, and neither sead::PerspectiveProjection nor tanf runs in play +static void *findmat_run(void *p) { + double secs = ((double *)p)[0]; int pick = (int)((double *)p)[1]; extern uint64_t g_t0_ns; uint64_t now_ns(void); + while ((now_ns() - g_t0_ns) / 1e9 < secs) usleep(100000); + uint64_t hb = 0; size_t hl = 0; dm_guest_heap_range(&hb, &hl); + if (!hb || !hl) { LOG("FINDMAT: [RED] no guest heap"); return NULL; } + LOG("FINDMAT: scanning heap %#llx + %#zx", (unsigned long long)hb, hl); + enum { CH = 1 << 20 }; static float buf[CH / 4 + 16]; int found = 0, n16 = 0; uint64_t pickat = 0; + for (uint64_t a = hb; a < hb + hl; a += CH) { + vm_size_t got = 0; size_t want = a + CH + 64 <= hb + hl ? CH + 64 : hb + hl - a; + if (vm_read_overwrite(mach_task_self(), a, want, (vm_address_t)buf, &got) || got < 64) continue; + for (size_t i = 0; i + 16 <= got / 4 && i < CH / 4; i++) { + const float *f = buf + i; + if (!(f[0] > 0.05f && f[0] < 50 && f[5] > 0.05f && f[5] < 50) || f[1] != 0 || f[4] != 0 || f[3] != 0 || f[7] != 0 || f[15] != 0) continue; + int rowm = (f[14] == -1 || f[14] == 1) && f[12] == 0 && f[13] == 0 && f[8] == 0 && f[9] == 0 && f[11] != 0; // sead::Matrix44 rows: w row 0,0,±1,0 + int colm = (f[11] == -1 || f[11] == 1) && f[2] == 0 && f[6] == 0 && f[12] == 0 && f[13] == 0 && f[14] != 0; // column major: w column + if (!rowm && !colm) continue; + float r = f[5] / f[0]; int is169 = fabsf(r / (16.0f / 9) - 1) < 0.005f; + uint64_t at = a + i * 4; + if (found++ < 200) LOG("FINDMAT %#llx (heap+%#llx) %s ratio %.4f%s x %g y %g z %g %g", (unsigned long long)at, (unsigned long long)(at - hb), rowm ? "rows" : "cols", + r, is169 ? " 16:9" : "", f[0], f[5], f[10], rowm ? f[11] : f[14]); + if (is169 && n16++ == pick) pickat = at; + } + } + LOG("FINDMAT: %d matrices, %d at 16:9", found, n16); + if (!pickat) return NULL; + g_wpsz = (uint64_t)getpagesize(); g_wlen = 64; g_wcap = 41; + if ((pickat & (g_wpsz - 1)) + g_wlen > g_wpsz) g_wlen = g_wpsz - (pickat & (g_wpsz - 1)); + g_wpg = pickat & ~(g_wpsz - 1); g_wat = pickat; + LOG("FINDMAT: watching %#llx + %#llx", (unsigned long long)g_wat, (unsigned long long)g_wlen); + return NULL; +} +// DM_PROBE_FINDF=,,[,[,]]: where a value the game computes lives. That many seconds after start a host thread scans the opened guest heap and the alias region +// for the float a with the float b bytes after it (4 by default; 0 = both 4 and 16), prints each place (up to 100), and arms the write watch above on the n-th (default 0), both floats, for 40 +// printed stores. Ocarina of Time: the HUD's corner groups sit at (-1920, 1078) in its 3840x2160 layout space; their world matrices hold that translation +static void *findf_run(void *p) { + double *arg = p; double secs = arg[0]; float fa = (float)arg[1], fb = (float)arg[2]; int pick = (int)arg[4]; extern uint64_t g_t0_ns; uint64_t now_ns(void); + int gaps[2] = { (int)arg[3], (int)arg[3] }; if (!gaps[0]) { gaps[0] = 4; gaps[1] = 16; } // gap 0: both 4 (a vector, a column-major matrix) and 16 (a row-major 3x4 or 4x4) + void svc_alias_range(uint8_t **base, size_t *size); + while ((now_ns() - g_t0_ns) / 1e9 < secs) usleep(100000); + uint64_t hb = 0; size_t hl = 0; dm_guest_heap_range(&hb, &hl); uint8_t *ab = NULL; size_t al = 0; svc_alias_range(&ab, &al); + struct { uint64_t b; size_t n; const char *nm; } reg[2] = { { hb, hl, "heap" }, { (uint64_t)(uintptr_t)ab, al, "alias" } }; + enum { CH = 1 << 20 }; static uint8_t buf[CH + 64]; int found = 0; uint64_t pickat = 0; int pickgap = 4; + for (int r = 0; r < 2; r++) { + if (!reg[r].b || !reg[r].n) continue; + LOG("FINDF: scanning %s %#llx + %#zx for %g, %g %d or %d bytes after", reg[r].nm, (unsigned long long)reg[r].b, reg[r].n, fa, fb, gaps[0], gaps[1]); + uint64_t end = reg[r].b + reg[r].n; + for (mach_vm_address_t ra = reg[r].b; ra < end; ) { // the mapped, readable parts only (the alias region is 64 GB, mostly unmapped) + mach_vm_size_t rs = 0; vm_region_basic_info_data_64_t bi; mach_msg_type_number_t cnt = VM_REGION_BASIC_INFO_COUNT_64; mach_port_t obj; + if (mach_vm_region(mach_task_self(), &ra, &rs, VM_REGION_BASIC_INFO_64, (vm_region_info_t)&bi, &cnt, &obj) || ra >= end) break; + uint64_t re = ra + rs < end ? ra + rs : end; + if (bi.protection & VM_PROT_READ) for (uint64_t a = ra; a < re; a += CH) { + vm_size_t got = 0; size_t want = a + CH + 64 <= re ? CH + 64 : re - a; + if (vm_read_overwrite(mach_task_self(), a, want, (vm_address_t)buf, &got) || got < 24) continue; + for (size_t i = 0; i + 20 <= got && i < CH; i += 4) { + float x, y; memcpy(&x, buf + i, 4); if (x != fa) continue; + for (int g = 0; g < 2; g++) { + if (g && gaps[1] == gaps[0]) break; + memcpy(&y, buf + i + gaps[g], 4); if (y != fb) continue; + uint64_t at = a + i; + if (found < 100) { float c[12] = { 0 }; size_t from = i >= 16 ? i - 16 : 0; memcpy(c, buf + from, from + 48 <= got ? 48 : got - from); + LOG("FINDF %d %#llx (%s+%#llx) gap %d around: %g %g %g %g | %g %g %g %g | %g %g %g %g", found, (unsigned long long)at, reg[r].nm, (unsigned long long)(at - reg[r].b), + gaps[g], c[0], c[1], c[2], c[3], c[4], c[5], c[6], c[7], c[8], c[9], c[10], c[11]); } + if (found++ == pick) { pickat = at; pickgap = gaps[g]; } + } + } + } + ra = re; + } + } + LOG("FINDF: %d places", found); + if (!pickat) return NULL; + g_wpsz = (uint64_t)getpagesize(); g_wlen = (uint64_t)pickgap + 4; g_wcap = 41; + if ((pickat & (g_wpsz - 1)) + g_wlen > g_wpsz) g_wlen = g_wpsz - (pickat & (g_wpsz - 1)); + g_wpg = pickat & ~(g_wpsz - 1); g_wat = pickat; + LOG("FINDF: watching %#llx + %#llx", (unsigned long long)g_wat, (unsigned long long)g_wlen); + return NULL; +} static void watch_install(void) { + { const char *e = getenv("DM_PROBE_FINDMAT"); + if (e) { static double arg[2]; char *end; arg[0] = strtod(e, &end); arg[1] = *end == ',' ? strtod(end + 1, 0) : 0; + dm_fault_hook_register(watch_fault); + pthread_t t; pthread_create(&t, NULL, findmat_run, arg); pthread_detach(t); + pthread_create(&t, NULL, watch_arm, NULL); pthread_detach(t); } } + { const char *e = getenv("DM_PROBE_FINDF"); + if (e) { static double arg[5]; arg[3] = 4; char *end = (char *)e; + for (int k = 0; k < 5 && *end; k++) { arg[k] = strtod(end, &end); if (*end == ',') end++; else break; } + dm_fault_hook_register(watch_fault); + pthread_t t; pthread_create(&t, NULL, findf_run, arg); pthread_detach(t); + pthread_create(&t, NULL, watch_arm, NULL); pthread_detach(t); } } { const char *e = getenv("DM_PROBE_POLL"); if (e) { char *end; g_pheap = !strncmp(e, "heap+", 5); if (g_pheap) e += 5; g_palias = !strncmp(e, "alias+", 6); if (g_palias) e += 6; g_poll = strtoull(e, &end, 16); g_plen = *end == '+' ? strtoull(end + 1, &end, 16) : 4; if (*end == '=') { g_pwant = 1; g_pval = strtoull(end + 1, 0, 16); } if ((!g_poll && !g_pheap && !g_palias) || !g_plen || g_plen > 8) die("DM_PROBE_POLL format: [heap+|alias+]
[+][=] (hex)"); pthread_t t; pthread_create(&t, NULL, poll_run, NULL); pthread_detach(t); } } const char *e = getenv("DM_PROBE_WATCH"); if (!e) return; - char *end; g_wheap = !strncmp(e, "heap+", 5); + char *end; + if (e[0] == '@' && e[1] == 'x') { // @xN+[+]: armed at the first printed probe hit + g_wfrom = (int)strtol(e + 2, &end, 10) & 31; g_wfoff = *end == '+' ? strtoull(end + 1, &end, 16) : 0; g_wlen = *end == '+' ? strtoull(end + 1, 0, 16) : 8; + if (!g_wlen) die("DM_PROBE_WATCH format: @xN+[+] (hex)"); + g_wpsz = (uint64_t)getpagesize(); + dm_fault_hook_register(watch_fault); + pthread_t t; pthread_create(&t, NULL, watch_arm, NULL); pthread_detach(t); + g_wcap = 41; // a per-frame writer: 40 printed stores + LOG("write watch on x%d + %#llx + %#llx at the first probe hit", g_wfrom, (unsigned long long)g_wfoff, (unsigned long long)g_wlen); + return; + } + g_wheap = !strncmp(e, "heap+", 5); uint64_t at = strtoull(e + (g_wheap ? 5 : 0), &end, 16); g_wlen = *end == '+' ? strtoull(end + 1, 0, 16) : 8; if ((!at && !g_wheap) || !g_wlen) die("DM_PROBE_WATCH format: [heap+]
[+] (hex)"); g_wpsz = (uint64_t)getpagesize(); From d97bed7bc31a124486d2c89b762def0ea2ebcd56 Mon Sep 17 00:00:00 2001 From: seedds Date: Mon, 9 Nov 2026 18:05:55 +0100 Subject: [PATCH 08/40] Ocarina of Time fills any window or screen shape, live while playing: its output size, window textures and camera projection follow the shape, with no bars and no stretch screen_aspect any, and four stubs as function-matched others of the frame-step rule (patches.py, found on v131072 main): - output size (main+0x1087f40, the display settings' size setter): the requested size times the shape's factors, capped at 1.5 times the pixels, each side a multiple of 8; the crop, the scene's targets and the dynamic-resolution sizes follow. It copies the camera multipliers the size was made with to state +0xb8 / +0xbc. - window crop (main+0xe9a860, setCrop): a crop beyond the 2560x1440 window textures grows them; the next acquire re-creates them. - camera copy (main+0x33a8c0 +0x2ec): the camera named "Main" hands its cached sead::PerspectiveProjection matrix here each frame; rows x / y times the multipliers, the aspect beside it, in memory and in the registers the frustum code reads after. set(), tanf and the 16/9 literals do not run in play: found with DM_PROBE_FINDMAT and the write watch. - display update (main+0x6502e0 +0x18): flag bit 17 set when the loader's change count moved, so the game recomputes its size and a resized window takes effect. Wider than 16:9 keeps the vertical angle, between 1:1 and 16:9 the horizontal one, upright the scale of the same screen turned landscape. test_oot_aspect.py runs the stubs with unicorn (12 tests). Checked on the Mac at six window shapes resized live (iPhone and iPad upright and landscape, 1700x478, 16:9). The HUD is the next commit. --- dm/games/0400486024DEC000/game.tsv | 3 + dm/games/0400486024DEC000/patches.py | 106 ++++++++++- dm/tools/tests/test_oot_aspect.py | 264 +++++++++++++++++++++++++++ 3 files changed, 372 insertions(+), 1 deletion(-) create mode 100644 dm/tools/tests/test_oot_aspect.py diff --git a/dm/games/0400486024DEC000/game.tsv b/dm/games/0400486024DEC000/game.tsv index 45ccc5ed9e..a0bbb2b70e 100644 --- a/dm/games/0400486024DEC000/game.tsv +++ b/dm/games/0400486024DEC000/game.tsv @@ -42,6 +42,9 @@ shader_note Ocarina of Time shaders fps_target 60 # ---- Graphics ---- +# Screen shape (patches.py, README "Screen shape"): the picture takes the window's or the screen's shape, any from 0.25 to 6, portrait included, and follows it while +# the game runs (DM_SCREEN_ASPECT, auto by default): wider shows more at the sides, taller more above and below; the HUD keeps its size +screen_aspect any draw_barriers game # As Tears of the Kingdom: the terrain's height, blend and hole maps are written by the CPU into textures after they are created. Without these keys they stay zero: holes in the ground and the wrong materials tex_flush_upload on diff --git a/dm/games/0400486024DEC000/patches.py b/dm/games/0400486024DEC000/patches.py index cade577a2e..c1fe858d19 100644 --- a/dm/games/0400486024DEC000/patches.py +++ b/dm/games/0400486024DEC000/patches.py @@ -32,6 +32,7 @@ def frame_step(ctx): emit(0x1e201828); emit(B(here(), site + 4)) # orig: fdiv s8,s1,s0; back fs_put(site, [B(site, start)], 'frame ratio: the host\'s step × base rate / 60 instead of base rate / the game\'s frame rate (state page target fps; 0 = the game\'s ratio)') dynamic_resolution(ctx) + screen_shape(ctx) # The game's dynamic resolution (game__gfx__DynamicResolutionParam: LoadThreshold, ReduceCounterMax, IncreaseCounterMax and a list of levels, each a ResolutionReduceScale and a # Priority; on v131072 levels 0..9 at 0.75..0.333 of 2048x1152, 1536x864 to 683x384). Its update (main+0x805aa0) reads the frame's GPU load (about 0.2 ms a unit) into +0x8, subtracts @@ -71,10 +72,110 @@ def dynamic_resolution(ctx): fs_put(site, [B(site, start)], 'dynamic resolution level: 0 (full resolution), or with state +0x40 (DM_DYNAMIC_RESOLUTION) a climb on the load measured at each level') ctx.capabilities.add('DYNAMIC_RESOLUTION') +# Screen shape (game.tsv screen_aspect any; the loader writes state +0xa0 the ratio, 0 = 16:9, +0xa4 the width factor, +0xa8 the height factor: loader.c screen_any_write). +# The game's output size, the picture its upscaler writes and the window's crop (2048x1152 docked, 1536x864 or 1920x1080 handheld on v131072), is set in one place: the setter +# main+0x1087f40 (display settings x0, x1 -> {w, h}) stores it at +0x10, raises bit 21 of +0x8 when it changed (main+0x6502e0 then hands it to the window's setCrop, main+0xe9a860, +# which nvnWindowSetCrop takes at each present) and classifies it (+0x18: 640x480 .. 3840x2160, 7 for any other size, as 2048x1152 already is). Its two callers are the settings' +# init (main+0x1087b70, the size from its config struct) and the mode change (main+0x6502e0: a level's size × its scale, capped by a table). The dynamic resolution scales from the +# output height by one factor for both axes (main+0x31d3f64..), so its sizes follow the output's shape. The stub scales the requested size by the factors (the width for a wider +# screen, the height for a taller one); when that is more than 1.5 times the requested size's pixels (beyond 2.67:1 or 1.19:1, a portrait window) it scales both sides down to +# 1.5 times, so the pixel count stays in bounds whatever the shape. Each side is rounded to a multiple of 8, into state +0xb0 (scratch, never the caller's struct: the init passes +# its config), and the setter runs on it. It also copies the camera's multipliers for that shape (state +0xc4 / +0xc8, from the loader) to +0xb8 / +0xbc (0 at 16:9): the camera +# (camera_projection) follows the size the game renders at, not the shape the host asks for, as the game sets its size only at its init, at a mode change and when shape_change +# asks it to. A leaf: x9 / x10 / x16 / s16..s19 are free (the setter loads w8..w11 before it reads them; s16..s19 are caller-saved), x1 an argument it only reads +size_stub = ( # size_stub.s, assembled with clang; x16 the state page (the ADRP before it) + 0xbd40a210, 0x1e202208, 0x54000061, 0xf9005e1f, 0x1400001b, # ratio (+0xa0); 0 (16:9): no camera multipliers (+0xb8 = 0), the setter as it is + 0x29402829, 0x1e230132, 0x1e230153, 0x2d58c610, 0x2d174610, 0x2d54c610, # w, h; the camera's multipliers to +0xb8 / +0xbc; the size factors + 0x1e300a50, 0x1e310a71, # w × width factor, h × height factor + 0x1e330a52, 0x1e2f1013, 0x1e330a52, 0x1e310a13, 0x1e322260, 0x540000ad, # more than 1.5 × the requested pixels? + 0x1e331a52, 0x1e21c252, 0x1e320a10, 0x1e320a31, # both sides × sqrt(limit / pixels) + 0x1e390209, 0x11001129, 0x121d7129, 0x1e39022a, 0x1100114a, 0x121d714a, # each to a multiple of 8 + 0x29162a09, 0x9102c201) # into state +0xb0; x1 -> it +def screen_shape(ctx): + here, emit, fs_site, fs_put, FS_STATE_OFF = ctx.here, ctx.emit, ctx.site, ctx.put, ctx.state + site = ctx.others.get('output size') + if site is None: return # not found in this build (a red line from the locator): 16:9 + fs_site(site, '08244229') # ldp w8,w9,[x0,#0x10] + start = here(); emit(ADRP(16, here(), FS_STATE_OFF)) + for w in size_stub: emit(w) + emit(0x29422408); emit(B(here(), site + 4)) # ldp w8,w9,[x0,#0x10]; back + fs_put(site, [B(site, start)], 'output size: the requested size times the screen shape\'s width and height factors (state +0xa4 / +0xa8; +0xa0 0 = 16:9, as it is)') + window_size(ctx) + camera_projection(ctx) + +# The window textures. The game creates its three window textures at 2560x1440 (the window object's +0x8 / +0xc floats; defaults main+0xecbc80), and a crop larger than them +# (2048x1576 at 1.3:1, 2688x1152 at 21:9) cannot be presented. The game re-creates its window and textures from +0x8 / +0xc when +0xc8 is 1: main+0x527fc0, which the +# per-frame texture acquire (main+0x527f20, main+0x346c9b0) tail-calls and which returns at once while +0xc8 is 0. The stub at the entry of setCrop (main+0xe9a860, a leaf: +# win x0, x w1, y w2, w w3, h w4) raises +0x8 / +0xc to hold the crop when it does not fit and sets +0xc8, so the next frame's acquire re-creates the textures at that size. It +# only grows them: a later, smaller crop is presented from the larger textures without another re-creation +crop_stub = ( # crop.s, assembled with clang + 0x2d414410, 0x0b030029, 0x0b04004a, 0x1e230132, 0x1e230153, # the window's size; the crop's right and bottom edges + 0x1e306a52, 0x1e316a73, 0x1e302240, 0x54000061, 0x1e312260, 0x54000080, # the larger of each; both the same: as it is + 0x2d014c12, 0x52800029, 0x39032009) # the window's new size; +0xc8 = 1: re-create at the next acquire +def window_size(ctx): + here, emit, fs_site, fs_put = ctx.here, ctx.emit, ctx.site, ctx.put + site = ctx.others.get('window crop') + if site is None: return # not found in this build: crops beyond 2560x1440 are not presented + fs_site(site, '28008052') # mov w8,#1 + start = here() + for w in crop_stub: emit(w) + emit(0x52800028); emit(B(here(), site + 4)) # mov w8,#1; back + fs_put(site, [B(site, start)], 'window crop: the window textures grow to hold a crop beyond their size (+0x8 / +0xc, re-created at the next acquire through +0xc8)') + +# The camera's projection. Each frame the game finds the camera named "Main" and hands its view and its sead::PerspectiveProjection (x2: near +0x98, far +0x9c, fovy +0xa0, +# aspect +0xb0, the cached matrix at +0xc) to main+0x33a8c0, which copies the projection's matrix into the render camera (x19 +0x148, row major; main+0x33ab7c..0x33abac) and +# stores near / far / fovy / aspect beside it (+0x1c8..+0x1d4), then derives the frustum from the rows it holds in v4..v18. The projection's aspect is 16:9 whatever the output's +# shape (sead::PerspectiveProjection::set, tanf and the 16/9 literals do not run in play: found with DM_PROBE_FINDMAT and a write watch on the render camera's matrix). After +# the last row is stored the stub multiplies row 0 (x) and row 1 (y) by the camera's multipliers, in memory and in v4 / v5, and the aspect at +0x1d4 by y's over x's. The loader +# chooses them (loader.c screen_any_write): landscape wider than 16:9 keeps the vertical angle and shows more at the sides; landscape between 1:1 and 16:9 keeps the horizontal +# angle and shows more above and below; portrait takes the scale of the same screen turned landscape (its width gets what that one's height gets), so turning a phone or a window +# keeps the characters' size and shows another part of the world. They are those the output size was made with (state +0xb8 / +0xbc, copied by size_stub; 0 = 16:9), so the +# projection always matches the picture the game renders. The fcmp at main+0x33ab78 +# is still live (b.eq at +0x33abb0): the stub tests with cbz and keeps the flags. w9 (reloaded at +0x33abbc), v6 and v29 (written before they are read) are free +proj_stub = ( # proj.s, assembled with clang; x16 the state page (the ADRP before it) + 0xb940ba09, 0x340001c9, # the camera's x multiplier (+0xb8); 0 (16:9): as it is + 0xbd40ba06, 0xbd41d67d, 0x1e261bbd, 0xbd40be06, 0x1e260bbd, 0xbd01d67d, # aspect ÷ x multiplier × y multiplier + 0x4e0404c6, 0x6e26dca5, 0x3d800685, # row 1 × y multiplier + 0xbd40ba06, 0x4e0404c6, 0x6e26dc84, 0x3d800284) # row 0 × x multiplier +def camera_projection(ctx): + here, emit, fs_site, fs_put, FS_STATE_OFF = ctx.here, ctx.emit, ctx.site, ctx.put, ctx.state + fn = ctx.others.get('camera copy') + if fn is None: return # not found in this build (a red line from the locator): the projection stays 16:9 + site = fn + 0x2ec + fs_site(site, '920e803d') # str q18,[x20,#0x30]: the matrix's last row + start = here(); emit(0x3d800e92); emit(ADRP(16, here(), FS_STATE_OFF)) + for w in proj_stub: emit(w) + emit(B(here(), site + 4)) + fs_put(site, [B(site, start)], 'camera projection: matrix rows x / y times the camera multipliers the output size was made with, the aspect beside it (state +0xb8 / +0xbc; 0 = 16:9)') + shape_change(ctx) + +# A shape asked for while playing (a window resized): the display settings' update main+0x6502e0 (x0 the settings, the same object as the size setter's) recomputes the output size +# and hands it to the size setter (main+0x650600) when bit 17 (0x20000) of its flags (+0x8) is set, and clears the bit after; the mode change path sets it (main+0x6506a4). The +# stub, after the update loads its flags (ldp w11,w9,[x0,#8] at +0x18), compares the loader's change count (state +0xac) with the count last applied (+0xc0) and, when they +# differ, records it and sets bit 17 in w11, which the update stores back (main+0x650348) and tests (main+0x65036c): the size, and with it the crop, the targets and the camera +# (camera_projection), follow the new shape at the next update. w10 and w12 are written right after (+0x1c, +0x20); the flags are set again by the tst at +0x34 +trig_stub = ( # trig.s, assembled with clang; x16 the state page (the ADRP before it) + 0xb940ae0a, 0xb940c20c, 0x6b0c015f, 0x54000060, # the change count (+0xac) against the one applied (+0xc0); the same: as it is + 0xb900c20a, 0x320f016b) # recorded; flags bit 17: recompute the output size +def shape_change(ctx): + here, emit, fs_site, fs_put, FS_STATE_OFF = ctx.here, ctx.emit, ctx.site, ctx.put, ctx.state + fn = ctx.others.get('display update') + if fn is None: return # not found in this build: the shape follows only at start and at a mode change + site = fn + 0x18 + fs_site(site, '0b244129') # ldp w11,w9,[x0,#8] + start = here(); emit(0x2941240b); emit(ADRP(16, here(), FS_STATE_OFF)) + for w in trig_stub: emit(w) + emit(B(here(), site + 4)) + fs_put(site, [B(site, start)], 'screen shape change: the display update recomputes the output size when the loader\'s change count (state +0xac) moved since the last (+0xc0)') + MATCH_FRAME_STEP = {'main': [ {'name': 'frame ratio from the host\'s step', 'size': 0x9b4, 'fingerprints': ['6ae2aff5344d14568e4dbaf93455e2d8278384bf892ed6c6eca87fef98fbcd1d'], 'stub': frame_step, # v131072 main+0x6e6be0 'others': {'dynamic resolution': {'size': 0x294, 'fingerprints': ['7af6ceb0aedf8e0c17dc08ce17fcd6a7cba6d394a0d09e3b504c6d668dc418e7']}, # v131072 main+0x8065e0 - 'dynamic resolution update': {'size': 0xb24, 'fingerprints': ['5a423697838976baf286cffa5e2ceef0d4e037abbcd9d371e90e9e702d61b3d4']}}}, # main+0x805aa0 + 'dynamic resolution update': {'size': 0xb24, 'fingerprints': ['5a423697838976baf286cffa5e2ceef0d4e037abbcd9d371e90e9e702d61b3d4']}, # main+0x805aa0 + 'output size': {'size': 0xd8, 'fingerprints': ['9e24f74affcfca6fb386818db0e5a9d4c9e3c57aee7b35c5afff3abed93a58a4']}, # main+0x1087f40 + 'camera copy': {'size': 0x1648, 'fingerprints': ['66a50abf2237dee694c40d6a2371071d62e91379d40b643869d680f597bfab02']}, # main+0x33a8c0 + 'display update': {'size': 0x554, 'fingerprints': ['c0cab0dbf6f0644635315a356f8fd23a4701515c4ab4775d52191835b1a12e3c']}, # main+0x6502e0 + 'window crop': {'size': 0x14, 'fingerprints': ['a6116ba396321af2dacdfc1b236bbf1103285a96b92db1c2c6cc3f1aed190f6d']}}}, # main+0xe9a860 ]} BINDING = [ @@ -84,4 +185,7 @@ def dynamic_resolution(ctx): {'what': 'dynamic resolution: full size, or a climb on the load measured at each level', 'now': 'game', 'goal': 'game', 'purpose': 'rendering', 'why': "the game's own dynamic-resolution level function and update, found by their functions' fingerprints on any build that keeps both; switchable (DM_DYNAMIC_RESOLUTION)", 'checked': 'v131072 main: one function with each fingerprint (the level function a leaf, x2..x11 / x16 / x17 / s0..s2 free), its callers, the update and the controller fields read by disassembly, the fields read in play (2026-11-08); the climb run natively against a model of the update and in play to full size (2026-11-09)'}, + {'what': 'screen shape: the output size, the window textures, the camera projection and a live shape change (screen_aspect any)', 'now': 'game', 'goal': 'game', 'purpose': 'rendering', + 'why': "the game's own size setter, setCrop, camera copy and display update, found by their functions' fingerprints on any build that keeps them; switchable (DM_SCREEN_ASPECT)", + 'checked': 'v131072 main: one function with each fingerprint, the sites and their free registers read by disassembly, the camera matrix and its writer found in play with DM_PROBE_FINDMAT and the write watch, the stubs run natively (test_oot_aspect.py) and on the Mac at six window shapes resized live (2026-11-09)'}, ] diff --git a/dm/tools/tests/test_oot_aspect.py b/dm/tools/tests/test_oot_aspect.py new file mode 100644 index 0000000000..5f37708933 --- /dev/null +++ b/dm/tools/tests/test_oot_aspect.py @@ -0,0 +1,264 @@ +"""Execute Ocarina of Time's emitted screen-shape stubs on ARM64, without game files. + +Optional dependency: unicorn. On macOS its JIT needs execution outside the sandbox. +""" +import importlib +import importlib.util +from pathlib import Path +import struct +import sys +import types +from typing import Any +import unittest + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +uc: Any = None # unicorn and its ARM64 constants when installed (typed Any: the tests are skipped without it) +arm: Any = None +try: + uc, arm = importlib.import_module('unicorn'), importlib.import_module('unicorn.arm64_const') +except ImportError: + pass + +spec = importlib.util.spec_from_file_location('oot_patches', Path(__file__).resolve().parents[2] / 'games/0400486024DEC000/patches.py') +assert spec is not None and spec.loader is not None +patch = importlib.util.module_from_spec(spec); spec.loader.exec_module(patch) + + +def camera(r): + """The camera's x / y multipliers the loader writes for a shape (loader.c screen_any_write)""" + if not r: return (1, 1) + l = r if r >= 1 else 1 / r + xl, yl = (16 / 9 / l, 1) if l >= 16 / 9 else (1, l * 9 / 16) + return (xl, yl) if r >= 1 else (yl * 16 / 9, xl * 9 / 16) + + +@unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') +class OutputSize(unittest.TestCase): + # The setter's first instruction (ldp w8,w9,[x0,#0x10]) at SITE in a code area; the stub branches back to SITE + 4 + base, area, state, obj, req = 0x4000000, 0x4100000, 0x4200000, 0x6000000, 0x6001000 + SITE = 0x1000 + + def setUp(self): + self.u = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + for a, n in ((self.base, 0x100000), (self.area, 0x10000), (self.state, 0x4000), (self.obj, 0x2000)): + self.u.mem_map(a, n) + self.u.mem_write(self.base + self.SITE, struct.pack('<2I', 0x29422408, 0xd503201f)) # ldp w8,w9,[x0,#0x10]; nop + self.cursor = self.area - self.base + self.ctx = types.SimpleNamespace(here=lambda: self.cursor, emit=self.emit, site=lambda off, orig: None, put=self.put, + state=self.state - self.base, others={'output size': self.SITE}) + patch.screen_shape(self.ctx) + self.u.mem_write(self.obj + 0x10, struct.pack('<2I', 1111, 2222)) # the size the settings hold now + + def emit(self, w): + self.u.mem_write(self.base + self.cursor, struct.pack(' {w, h}; the size it goes on with (x1's pair) and the registers its first instruction loads""" + self.u.mem_write(self.req, struct.pack('<2I', w, h)) + self.u.reg_write(arm.UC_ARM64_REG_X0, self.obj); self.u.reg_write(arm.UC_ARM64_REG_X1, self.req) + end = self.base + self.SITE + 4 + stop = self.u.hook_add(uc.UC_HOOK_CODE, lambda u, a, n, d: u.emu_stop(), begin=end, end=end) + try: self.u.emu_start(self.base + self.SITE, end, count=100) + finally: self.u.hook_del(stop) + self.assertEqual(self.u.reg_read(arm.UC_ARM64_REG_PC), end) + self.assertEqual((self.u.reg_read(arm.UC_ARM64_REG_W8), self.u.reg_read(arm.UC_ARM64_REG_W9)), (1111, 2222)) # the original first instruction ran + x1 = self.u.reg_read(arm.UC_ARM64_REG_X1) + self.assertEqual(struct.unpack('<2I', self.u.mem_read(self.req, 8)), (w, h)) # the caller's struct (the init's config) is never written + return x1, struct.unpack('<2I', self.u.mem_read(x1, 8)) + + def factors(self): return struct.unpack('<2f', self.u.mem_read(self.state + 0xb8, 8)) + + def test_16_9_is_the_setter_as_it_is(self): + self.u.mem_write(self.state + 0xb8, struct.pack('<2f', 1.2375, 1)) # from an earlier, wider size + self.shape(0) + self.assertEqual(self.call(2048, 1152), (self.req, (2048, 1152))) + self.assertEqual(self.factors(), (0, 0)) # the camera: 16:9 + + def test_the_camera_multipliers_go_with_the_size(self): + for ratio in (2.2, 1.43, 0.46): + self.shape(ratio); self.call(2048, 1152) + self.assertEqual(self.factors(), tuple(struct.unpack('<2f', struct.pack('<2f', *camera(ratio))))) + + def test_wider_keeps_the_height(self): + for ratio, docked, handheld in ((2.2, (2536, 1152), (1904, 864)), (2.4, (2768, 1152), (2072, 864)), (21 / 9, (2688, 1152), (2016, 864))): + self.shape(ratio) + self.assertEqual(self.call(2048, 1152), (self.state + 0xb0, docked), ratio) + self.assertEqual(self.call(1536, 864), (self.state + 0xb0, handheld), ratio) + + def test_taller_keeps_the_width(self): + for ratio, docked, handheld in ((1.43, (2048, 1432), (1920, 1344)), (1.3, (2048, 1576), (1920, 1480)), (4 / 3, (2048, 1536), (1920, 1440))): + self.shape(ratio) + self.assertEqual(self.call(2048, 1152), (self.state + 0xb0, docked), ratio) + self.assertEqual(self.call(1920, 1080), (self.state + 0xb0, handheld), ratio) + + def test_any_shape_keeps_the_pixels_in_bounds(self): + for ratio in (1179 / 2556, 0.75, 1.0, 3.0, 5.0): # an iPhone held upright, a portrait window, a square, ultrawide beyond 21:9 + self.shape(ratio) + _, (w, h) = self.call(2048, 1152) + self.assertLessEqual(w * h, 1.5 * 2048 * 1152 * 1.01, ratio) + self.assertGreater(w * h, 1.5 * 2048 * 1152 * 0.97, ratio) # scaled down only as far as the bound + self.assertLess(abs(w / h / ratio - 1), 0.01, ratio) + self.assertEqual((w % 8, h % 8), (0, 0)) + + def test_the_shape_holds_to_a_multiple_of_8(self): + for ratio in (1.3, 1.43, 1.6, 2.0, 2.2, 2.4): + self.shape(ratio) + _, (w, h) = self.call(2048, 1152) + self.assertEqual((w % 8, h % 8), (0, 0)) + self.assertLess(abs(w / h / ratio - 1), 0.004, ratio) + + +@unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') +class CameraProjection(unittest.TestCase): + # The camera copy's last matrix store (str q18,[x20,#0x30]) at FN + 0x2ec; the stub branches back after it. x19 the render camera, x20 its matrix (+0x148) + base, area, state, cam = 0x4000000, 0x4100000, 0x4200000, 0x6000000 + FN = 0x1000; SITE = FN + 0x2ec + M = (1.20628, 0, 0, 0, 0, 2.14451, 0, 0, 0, 0, -1.00001, -0.200001, 0, 0, -1, 0) # the game's 16:9 projection (fovy 50°, near 0.1) + + def setUp(self): + self.u = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + for a, n in ((self.base, 0x100000), (self.area, 0x10000), (self.state, 0x4000), (self.cam, 0x1000)): + self.u.mem_map(a, n) + self.u.mem_write(self.base + self.SITE, struct.pack('<2I', 0x3d800e92, 0xd503201f)) # str q18,[x20,#0x30]; nop + self.cursor = self.area - self.base + ctx = types.SimpleNamespace(here=lambda: self.cursor, emit=self.emit, site=lambda off, orig: None, put=self.put, state=self.state - self.base, others={'camera copy': self.FN}) + patch.camera_projection(ctx) + + emit, put = OutputSize.emit, OutputSize.put + + def shape(self, ratio, asked=None): + """The factors the output size was made with (state +0xb8 / +0xbc, 0 = 16:9) and, apart, the shape the host asks for now (+0xa0..: the camera ignores it)""" + f = lambda r: (r, max(r * 9 / 16, 1), max(16 / (9 * r), 1)) if r else (0, 1, 1) + self.u.mem_write(self.state + 0xa0, struct.pack('<3f', *f(ratio if asked is None else asked))) + self.u.mem_write(self.state + 0xb8, struct.pack('<2f', *camera(ratio)) if ratio else bytes(8)) + + def run_copy(self): + """The matrix as the copy leaves it (rows 0..2 stored, row 3 in q18, rows 0 / 1 also in v4 / v5), aspect 16:9 at +0x1d4; the matrix, v4, v5 and the aspect after""" + m = struct.pack('<16f', *self.M) + self.u.mem_write(self.cam + 0x148, m[:48] + bytes(16)); self.u.mem_write(self.cam + 0x1d4, struct.pack(' 16 / 9: self.assertAlmostEqual(m[5], self.M[5], 5) # wider: the vertical angle kept + elif ratio >= 1: self.assertAlmostEqual(m[0], self.M[0], 5) # taller landscape: the horizontal angle kept + elif 1 / ratio >= 16 / 9: self.assertAlmostEqual(m[0], self.M[5], 4) # a phone upright: its width gets landscape's vertical angle + else: self.assertAlmostEqual(m[5], self.M[0], 4) # an iPad upright: its height gets landscape's horizontal angle + for a, b in zip(m[8:], self.M[8:]): self.assertAlmostEqual(a, b, 5) # z and w rows untouched + for a, b in zip(v4 + v5, m[:8]): self.assertAlmostEqual(a, b, 6) # the frustum code after reads the same rows + self.assertAlmostEqual(aspect, ratio, 3, ratio) + + +@unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') +class ShapeChange(unittest.TestCase): + # The display update's flag load (ldp w11,w9,[x0,#8]) at FN + 0x18; the stub branches back after it + base, area, state, obj = 0x4000000, 0x4100000, 0x4200000, 0x6000000 + FN = 0x1000; SITE = FN + 0x18 + + def setUp(self): + self.u = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + for a, n in ((self.base, 0x100000), (self.area, 0x10000), (self.state, 0x4000), (self.obj, 0x1000)): + self.u.mem_map(a, n) + self.u.mem_write(self.base + self.SITE, struct.pack('<2I', 0x2941240b, 0xd503201f)) + self.cursor = self.area - self.base + patch.shape_change(types.SimpleNamespace(here=lambda: self.cursor, emit=self.emit, site=lambda off, orig: None, put=self.put, state=self.state - self.base, others={'display update': self.FN})) + self.u.mem_write(self.obj + 8, struct.pack('<2I', 0x1004, 0x2)) # flags, the mode word + + emit, put = OutputSize.emit, OutputSize.put + + def update(self): + self.u.reg_write(arm.UC_ARM64_REG_X0, self.obj) + end = self.base + self.SITE + 4 + stop = self.u.hook_add(uc.UC_HOOK_CODE, lambda u, a, n, d: u.emu_stop(), begin=end, end=end) + try: self.u.emu_start(self.base + self.SITE, end, count=50) + finally: self.u.hook_del(stop) + self.assertEqual(self.u.reg_read(arm.UC_ARM64_REG_W9), 0x2) + return self.u.reg_read(arm.UC_ARM64_REG_W11) + + def test_a_new_shape_recomputes_the_size_once(self): + self.u.mem_write(self.state + 0xac, struct.pack(' Date: Mon, 9 Nov 2026 18:05:55 +0100 Subject: [PATCH 09/40] Ocarina of Time's HUD takes the output's shape at its own pixel size instead of a 16:9 layout stretched over the picture Every eui layout file holds a screen size of 3840x2160 (LayoutResData +0x200, file offset 0x2f8). A screen's init (main+0x773b90) builds its orthographic projection from it once and keeps it in the screen's uniform block, which the draw copies to the frame's ring each frame (main+0x2d2ba0). Two stubs, function-matched others of the frame-step rule: - hud upload (main+0x2d2ba0 +0x4c, after the memcpy): the copy's x / y rows times the HUD's multipliers and the size divided by them, each frame, so a resize takes effect at the next one; a capture's block (its own size beside the screen's) is left as it is. - layout size (main+0x583fd0, the getter): the size the HUD draws with for the game's conversions between the layout space and the screen; the screen's init still gets the file's. The HUD's multipliers are the camera's divided by the largest of them and 1: in landscape the HUD keeps the size it has at 16:9 of the same height or width; upright the whole 16:9 space fits the width, smaller, nothing cut off. test_oot_aspect.py: 6 more tests (18). Checked on the Mac at six shapes resized live, the pause menu, the "A Open" prompt and the Kokiri Forest cutscene with its location card. Left open (game note, Screen shape): eui places panes about the screen's centre and does not anchor them to its edges, so the HUD keeps its 16:9 arrangement in the middle (in wide shapes the hearts sit at the 16:9 band's edge, upright the HUD sits around mid-height), and full-screen panes (fades, the menu backdrop, 3840x2160) cover the 16:9 middle only. Moving the edge groups (60x80 nulls at the space's edges) and stretching those panes needs eui's per-frame pane transform, not found yet. --- dm/games/0400486024DEC000/patches.py | 54 +++++++++++- dm/tools/tests/test_oot_aspect.py | 123 +++++++++++++++++++++++++++ 2 files changed, 175 insertions(+), 2 deletions(-) diff --git a/dm/games/0400486024DEC000/patches.py b/dm/games/0400486024DEC000/patches.py index c1fe858d19..8c531c0760 100644 --- a/dm/games/0400486024DEC000/patches.py +++ b/dm/games/0400486024DEC000/patches.py @@ -1,6 +1,6 @@ # The Legend of Zelda: Ocarina of Time (0400486024DEC000): the rewriter's patches of this game's own code. dm/tools/patch_nso.py loads every profile's patches.py; # MATCH_FRAME_STEP as described there ("Registered patches"). Patches of Nintendo SDK module builds are in dm/tools/sdk_patches.py -from nso_asm import ADRP, B +from nso_asm import ADDI, ADRP, B # The game's frame ratio. Once a frame the function at main+0x6e6be0 (v131072) divides its base rate (60.0, or 30.0 while bit 0 of [x21+0x24f] is set) by the int frame rate at # [x21+0x2c8] (fdiv s8,s1,s0 at +0xe4) and hands the result to the time object (main+0x3b4e300: ratio at +0x10, 1 / ratio at +0x14, round(60 / ratio) at +0x1c), which the game @@ -102,6 +102,7 @@ def screen_shape(ctx): fs_put(site, [B(site, start)], 'output size: the requested size times the screen shape\'s width and height factors (state +0xa4 / +0xa8; +0xa0 0 = 16:9, as it is)') window_size(ctx) camera_projection(ctx) + hud(ctx) # The window textures. The game creates its three window textures at 2560x1440 (the window object's +0x8 / +0xc floats; defaults main+0xecbc80), and a crop larger than them # (2048x1576 at 1.3:1, 2688x1152 at 21:9) cannot be presented. The game re-creates its window and textures from +0x8 / +0xc when +0xc8 is 1: main+0x527fc0, which the @@ -168,6 +169,49 @@ def shape_change(ctx): emit(B(here(), site + 4)) fs_put(site, [B(site, start)], 'screen shape change: the display update recomputes the output size when the loader\'s change count (state +0xac) moved since the last (+0xc0)') +# The HUD and menus (eui, Nintendo's newer UI: layouts UI/Layout2Archive/*.blarc.zs). Every layout file holds its screen size, 3840x2160 (LayoutResData +0x200, file offset 0x2f8), and +# the game reads it in place through one getter (main+0x583fd0: [x0] + 0x200). A screen's init (main+0x773b90, a virtual) builds its orthographic projection from it (+-w/2, +# +-h/2, main+0x773f28) and keeps it in the screen's uniform block (+0x1b8: view-projection +8 and its y-flipped copy +0x48, column major, the size twice at +0x88, main+0x7741e0); +# each frame the screen's draw copies the block's 0x90 bytes into the frame's ring (main+0x2d2ba0, its memcpy at +0x48), so the HUD drew in a 16:9 space stretched over any +# output. The stub after that memcpy scales the copy's x / y rows by the HUD's multipliers and divides the size by them: the space takes the output's shape at the HUD's pixel +# size, at the next frame after a shape change and without touching the screen's block. A capture's block (main+0x154b660) holds its texture size beside the screen's and +# is left as it is. The HUD's multipliers are the camera's (state +0xb8 / +0xbc, those the output size was made with) divided by the largest of them and 1: in landscape the +# camera's, so the HUD keeps the size it has at 16:9 of the same height (wider) or width (taller); upright, the 16:9 space still fits the width, so nothing is cut off. The +# layouts place everything about the screen's centre (eui does not anchor panes to the screen's edges: a layout made 4752 wide in a copy of its file drew round hearts at +# their 16:9 place), so the HUD keeps its 16:9 arrangement in the middle; moving the edge groups (60x80 nulls at the space's edges, as PlayerStatus_00's N_ScaleL_00 at +# (-1920, 1078)) and stretching full-screen panes (3840x2160 at the centre: fades, menu backgrounds) is open (game note, Screen shape) +hud_upload = ( # hud.s, assembled with clang; x0 the copy (memcpy's result), x20 the block, x16 the state page (the ADRP before it) + 0xa948aa89, 0xeb0a013f, 0x54000441, 0xb4000429, 0xb940ba09, 0x340003e9, # the size twice (a screen's own projection) and a shape other than 16:9 + 0x2d574610, 0x1e316a12, 0x1e2e1013, 0x1e336a52, 0x1e321a10, 0x1e321a31, # the camera's multipliers / max(mx, my, 1) + 0x4f03f614, 0x6e040614, 0x6e0c0634, # (mx', my', 1, 1) + 0xad405815, 0x6e34deb5, 0x6e34ded6, 0xad005815, 0xad415815, 0x6e34deb5, 0x6e34ded6, 0xad015815, # both matrices' columns: rows x / y scaled + 0xad425815, 0x6e34deb5, 0x6e34ded6, 0xad025815, 0xad435815, 0x6e34deb5, 0x6e34ded6, 0xad035815, + 0x2d504c12, 0x1e301a52, 0x1e311a73, 0x2d104c12, 0x2d114c12) # the size / the multipliers, both copies +# The game's other conversions between the layout space and the screen (a pane's rect in screen coordinates, a 3D position's place on the HUD) read the same getter: the +# stub at its entry returns the size the HUD draws with (state +0xd0, scratch), except to the screen's init (its two calls, main+0x773ee0 / +0x773f24), whose projection the +# upload stub widens each frame. A leaf: x8..x10, x16 and s16..s21 are free +layout_size = ( # lsize.s, assembled with clang; x16 the state page, x10 the init's first return address (the ADRP / ADD before it) + 0xf9400008, 0x91080100, 0xb940ba09, 0x34000229, # the layout's size; 16:9: as it is + 0xeb0a03df, 0x540001e0, 0x9101114a, 0xeb0a03df, 0x54000180, # the screen's init: as it is + 0x2d404410, 0x2d574e12, 0x1e336a54, 0x1e2e1015, 0x1e356a94, 0x1e341a52, 0x1e341a73, # the HUD's multipliers + 0x1e321a10, 0x1e331a31, 0x91034200, 0x2d004410, 0xd65f03c0) # the size / them, into state +0xd0; x0 -> it +def hud(ctx): + here, emit, fs_site, fs_put, FS_STATE_OFF = ctx.here, ctx.emit, ctx.site, ctx.put, ctx.state + fn = ctx.others.get('hud upload') + if fn is not None: # not found in this build (a red line from the locator): the HUD stays 16:9, stretched over the output + site = fn + 0x4c + fs_site(site, 'e00315aa') # mov x0,x21 (after the memcpy) + start = here(); emit(ADRP(16, here(), FS_STATE_OFF)) + for w in hud_upload: emit(w) + emit(0xaa1503e0); emit(B(here(), site + 4)) # mov x0,x21; back + fs_put(site, [B(site, start)], 'HUD projection: the screen\'s uniform block copy takes the output\'s shape at the HUD\'s pixel size (state +0xb8 / +0xbc; 0 = 16:9)') + fn, init = ctx.others.get('layout size'), ctx.others.get('screen init') + if fn is None or init is None: return # the game's own layout size for its conversions + fs_site(fn, '080040f9') # ldr x8,[x0] + start = here(); emit(ADRP(16, here(), FS_STATE_OFF)); emit(ADRP(10, here(), init + 0x354)); emit(ADDI(10, 10, (init + 0x354) & 0xfff)) + for w in layout_size: emit(w) + fs_put(fn, [B(fn, start)], 'layout size: the size the HUD draws with for the game\'s conversions to and from the screen (state +0xd0), the screen\'s init as it is') + MATCH_FRAME_STEP = {'main': [ {'name': 'frame ratio from the host\'s step', 'size': 0x9b4, 'fingerprints': ['6ae2aff5344d14568e4dbaf93455e2d8278384bf892ed6c6eca87fef98fbcd1d'], 'stub': frame_step, # v131072 main+0x6e6be0 'others': {'dynamic resolution': {'size': 0x294, 'fingerprints': ['7af6ceb0aedf8e0c17dc08ce17fcd6a7cba6d394a0d09e3b504c6d668dc418e7']}, # v131072 main+0x8065e0 @@ -175,7 +219,10 @@ def shape_change(ctx): 'output size': {'size': 0xd8, 'fingerprints': ['9e24f74affcfca6fb386818db0e5a9d4c9e3c57aee7b35c5afff3abed93a58a4']}, # main+0x1087f40 'camera copy': {'size': 0x1648, 'fingerprints': ['66a50abf2237dee694c40d6a2371071d62e91379d40b643869d680f597bfab02']}, # main+0x33a8c0 'display update': {'size': 0x554, 'fingerprints': ['c0cab0dbf6f0644635315a356f8fd23a4701515c4ab4775d52191835b1a12e3c']}, # main+0x6502e0 - 'window crop': {'size': 0x14, 'fingerprints': ['a6116ba396321af2dacdfc1b236bbf1103285a96b92db1c2c6cc3f1aed190f6d']}}}, # main+0xe9a860 + 'window crop': {'size': 0x14, 'fingerprints': ['a6116ba396321af2dacdfc1b236bbf1103285a96b92db1c2c6cc3f1aed190f6d']}, # main+0xe9a860 + 'hud upload': {'size': 0x70, 'fingerprints': ['7cf489e6e9cc18b56c67f82ba4ca22e55abbd592d21309873d6803d19e0e7872']}, # main+0x2d2ba0 + 'screen init': {'size': 0x62c, 'fingerprints': ['103de33fb929cda858c2fbd57b726e6e1bca705bad66de872701819fbf5c6089']}, # main+0x773b90 + 'layout size': {'size': 0xc, 'fingerprints': ['d451096b59a2483a7e026bd2764333eecd505ea3aad2b3f8d902aa20ba5b02f7']}}}, # main+0x583fd0 ]} BINDING = [ @@ -188,4 +235,7 @@ def shape_change(ctx): {'what': 'screen shape: the output size, the window textures, the camera projection and a live shape change (screen_aspect any)', 'now': 'game', 'goal': 'game', 'purpose': 'rendering', 'why': "the game's own size setter, setCrop, camera copy and display update, found by their functions' fingerprints on any build that keeps them; switchable (DM_SCREEN_ASPECT)", 'checked': 'v131072 main: one function with each fingerprint, the sites and their free registers read by disassembly, the camera matrix and its writer found in play with DM_PROBE_FINDMAT and the write watch, the stubs run natively (test_oot_aspect.py) and on the Mac at six window shapes resized live (2026-11-09)'}, + {'what': "HUD: the screens' orthographic projection and the layout size take the output's shape", 'now': 'game', 'goal': 'game', 'purpose': 'rendering', + 'why': "the screen draw's uniform block copy and the layout-size getter, found by their functions' fingerprints with the screen init beside them", + 'checked': 'v131072 main: one function with each fingerprint, the block layout and its per-frame copy read by disassembly, the 3840x2160 size read in play with probes and changed in a copy of a layout file, the stubs run natively (test_oot_aspect.py) and on the Mac at six shapes, the pause menu and a cutscene (2026-11-09)'}, ] diff --git a/dm/tools/tests/test_oot_aspect.py b/dm/tools/tests/test_oot_aspect.py index 5f37708933..f49432cdab 100644 --- a/dm/tools/tests/test_oot_aspect.py +++ b/dm/tools/tests/test_oot_aspect.py @@ -260,5 +260,128 @@ def test_a_larger_crop_grows_the_textures(self): self.assertEqual(self.crop(2048, 1152), ((2688, 1576), 0)) # back to 16:9: no re-creation +def hud_space(r): + """The HUD's layout space for a shape: the camera's view of 3840x2160, grown to hold all of it upright""" + mx, my = camera(r); k = max(mx, my, 1) + return 3840 * k / mx, 2160 * k / my + + +def applied(u, state, r): + """The camera multipliers the output size was made with (state +0xb8 / +0xbc; 0 = 16:9), which the HUD's stubs read""" + u.mem_write(state + 0xb8, struct.pack('<2f', *camera(r)) if r else bytes(8)) + + +@unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') +class HudUpload(unittest.TestCase): + # The screen draw's uniform block copy: memcpy's return (x0 = the copy) at FN + 0x4c (mov x0,x21); x20 the block, x21 the ring object + base, area, state, blk, ring = 0x4000000, 0x4100000, 0x4200000, 0x6000000, 0x6100000 + FN = 0x1000; SITE = FN + 0x4c + + def setUp(self): + self.u = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + for a, n in ((self.base, 0x100000), (self.area, 0x10000), (self.state, 0x4000), (self.blk, 0x1000), (self.ring, 0x1000)): + self.u.mem_map(a, n) + self.u.mem_write(self.base + self.SITE, struct.pack('<2I', 0xaa1503e0, 0xd503201f)) # mov x0,x21; nop + self.cursor = self.area - self.base + patch.hud(types.SimpleNamespace(here=lambda: self.cursor, emit=self.emit, site=lambda off, orig: None, put=self.put, state=self.state - self.base, others={'hud upload': self.FN})) + + emit, put = OutputSize.emit, OutputSize.put + + @staticmethod + def block(w=3840.0, h=2160.0, other=None): + """A screen's block as main+0x773b90 leaves it: its orthographic view-projection, column major (rows x, y, z, w), the y-flipped copy, the size twice""" + m = (2 / w, 0, 0, 0, 0, 2 / h, 0, 0, 0, 0, -0.5, 0, 0, 0, 0.5, 1) # columns: (x, y, z, w) of each + f = (m[0], -m[1], m[2], m[3], m[4], -m[5], m[6], m[7], m[8], -m[9], m[10], m[11], m[12], -m[13], m[14], m[15]) + return struct.pack('<16f16f4f', *m, *f, w, h, *(other or (w, h))) + + def upload(self, blk): + """Run the stub on a copy of blk's 0x90 bytes; the copy after, as floats, and the block (never written)""" + self.u.mem_write(self.blk, bytes(8) + blk); self.u.mem_write(self.ring, blk[:0x90]) + for r, v in ((arm.UC_ARM64_REG_X0, self.ring), (arm.UC_ARM64_REG_X20, self.blk), (arm.UC_ARM64_REG_X21, 0x1234)): self.u.reg_write(r, v) + end = self.base + self.SITE + 4 + stop = self.u.hook_add(uc.UC_HOOK_CODE, lambda u, a, n, d: u.emu_stop(), begin=end, end=end) + try: self.u.emu_start(self.base + self.SITE, end, count=100) + finally: self.u.hook_del(stop) + self.assertEqual(self.u.reg_read(arm.UC_ARM64_REG_PC), end) + self.assertEqual(self.u.reg_read(arm.UC_ARM64_REG_X0), 0x1234) # the original instruction ran + self.assertEqual(bytes(self.u.mem_read(self.blk + 8, len(blk))), blk) + return struct.unpack('<36f', self.u.mem_read(self.ring, 0x90)) + + def shape(self, r): applied(self.u, self.state, r) + + def test_16_9_is_the_copy_as_it_is(self): + self.shape(0) + self.assertEqual(self.upload(self.block()), struct.unpack('<36f', self.block()[:0x90])) + + def test_a_capture_is_left_as_it_is(self): + self.shape(2.2) + b = self.block(other=(1984.0, 1184.0)) # main+0x154b660: the capture's own size beside the screen's + self.assertEqual(self.upload(b), struct.unpack('<36f', b[:0x90])) + + def test_shapes(self): + for r in (2.2, 2.4, 1.43, 1.3, 1.0, 0.7, 1179 / 2556, 9 / 16): + self.shape(r) + c = self.upload(self.block()) + w, h = c[32], c[33] + self.assertAlmostEqual(w / h, r, 3, r) # the HUD's space has the output's shape + self.assertEqual((c[34], c[35]), (w, h)) + ew, eh = hud_space(r) + self.assertAlmostEqual(w, ew, 0, r); self.assertAlmostEqual(h, eh, 0, r) + self.assertGreaterEqual(w, 3839.9, r); self.assertGreaterEqual(h, 2159.9, r) # all of the 16:9 space in view + if r >= 1: self.assertTrue(abs(w - 3840) < 0.5 or abs(h - 2160) < 0.5, r) # landscape: the HUD keeps its size + self.assertAlmostEqual(c[0], 2 / w, 8, r); self.assertAlmostEqual(c[5], 2 / h, 8, r) + self.assertAlmostEqual(c[16], 2 / w, 8, r); self.assertAlmostEqual(c[21], -2 / h, 8, r) # the y-flipped copy too + for i in (2, 3, 10, 11, 14, 15, 18, 19, 26, 27, 30, 31): # z and w untouched + self.assertEqual(c[i], struct.unpack('<36f', self.block()[:0x90])[i], (r, i)) + + +@unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') +class LayoutSize(unittest.TestCase): + # The getter (ldr x8,[x0]; add x0,x8,#0x200; ret) at FN; the screen's init at INIT calls it at +0x350 and +0x394 + base, area, state, obj, res = 0x4000000, 0x4100000, 0x4200000, 0x6000000, 0x6100000 + FN, INIT, CALLER = 0x1000, 0x8000, 0x9000 + + def setUp(self): + self.u = uc.Uc(uc.UC_ARCH_ARM64, uc.UC_MODE_ARM) + for a, n in ((self.base, 0x100000), (self.area, 0x10000), (self.state, 0x4000), (self.obj, 0x1000), (self.res, 0x1000)): + self.u.mem_map(a, n) + self.u.mem_write(self.base + self.FN, struct.pack('<3I', 0xf9400008, 0x91080100, 0xd65f03c0)) + self.cursor = self.area - self.base + patch.hud(types.SimpleNamespace(here=lambda: self.cursor, emit=self.emit, site=lambda off, orig: None, put=self.put, state=self.state - self.base, + others={'layout size': self.FN, 'screen init': self.INIT})) + self.u.mem_write(self.obj, struct.pack(' Date: Mon, 9 Nov 2026 18:06:06 +0100 Subject: [PATCH 10/40] Docs for any screen shape: Ocarina of Time's Screen shape section, the screen_aspect key, DM_SCREEN_ASPECT for any profiles, the probe switches, and a reference entry for a camera that caches its projection - Game note: a Screen shape section (the six stubs, the rule for wider, taller and upright, the HUD, what is open: the HUD's edge groups and full-screen panes, upright on iOS, the vertical angle upright) and its line under Found on this game. - docs/reference/rendering.md: a screen-shape stub on the projection setter changes nothing, the camera caches its matrix (find the per-frame reader with DM_PROBE_FINDMAT, DM_PROBE_FINDF and the watch armed from a register); the symptom index regenerated. - dm/games/README.md: the screen_aspect key (off / on / any). docs/env_switches.tsv: DM_SCREEN_ASPECT's any range and auto, DM_PROBE_FINDMAT, DM_PROBE_FINDF, DM_PROBE_XREG and DM_PROBE_WATCH's heap+ and @xN forms. docs/settings.md: Picture shape for any profiles. - docs/agents/game-patches.md regenerated: Ocarina of Time's screen-shape and HUD stubs are function-matched (BINDING rows in its patches.py), and Animal Crossing's row names them as the model. docs_patches.py's dry run puts a rule's other functions 4 MB apart instead of 16 MB: with 16 MB the ninth (Ocarina's layout size) landed out of a branch's reach of the patch area. --- dm/games/01006F8002326000/patches.py | 2 +- dm/games/0400486024DEC000/README.md | 21 +++++++++++++++++++++ dm/games/README.md | 1 + dm/tools/docs_patches.py | 8 ++++---- docs/agents/game-patches.md | 10 ++++++---- docs/agents/symptoms.md | 1 + docs/env_switches.tsv | 8 ++++++-- docs/reference/rendering.md | 7 +++++++ docs/settings.md | 2 +- 9 files changed, 48 insertions(+), 12 deletions(-) diff --git a/dm/games/01006F8002326000/patches.py b/dm/games/01006F8002326000/patches.py index a4892787cc..56c0886327 100644 --- a/dm/games/01006F8002326000/patches.py +++ b/dm/games/01006F8002326000/patches.py @@ -233,5 +233,5 @@ def screen_shape(ctx): BINDING = [ {'what': "screen shape: the sizes, the view, the layouts' projection and their edge widgets (DM_SCREEN_ASPECT)", 'now': 'build', 'goal': 'game', 'purpose': 'rendering', 'why': "the output size is the host's, but where the game takes its sizes, its cameras and its layouts are this build's code", - 'blocked': 'the stubs have no function-matched form yet'}, + 'blocked': "the stubs have no function-matched form yet (Ocarina of Time's screen-shape stubs, function-matched others of its frame-step rule, are the model)"}, ] diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 14505dcfd6..2c0bad41a7 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -14,6 +14,7 @@ A Switch 2 game: update v131072, SDK 22.2.8 (`Ounce`). Nintendo EPD's engine (se | `tex_flush_upload on`, `tex_dirty_upload on` | `game.tsv` | as Tears of the Kingdom: the CPU writes the terrain tiles' height (R32F), blend and hole (RGBA8) maps into textures after creating them. Without the keys those textures stay zero: flat blue holes in the ground around the path with grass floating in them, and grey stone where the console shows dirt | | `shader_groups`, `shader_note` | `game.tsv` | shaders grouped as `/Shader/`, `/Effect/`, `/Lib/`, `/UI/` | | `pad_layout full` | `game.tsv` | a 3D field and both sticks | +| `screen_aspect any`, screen-shape stubs | `game.tsv`, [`patches.py`](patches.py) (frame step rule, `others`) | the picture and the HUD take the window's or the screen's shape ([Screen shape](#screen-shape)) | Render conditions, which the host evaluates for every game ([Conditional rendering evaluated at replay](../../../docs/reference/rendering.md#conditional-rendering-evaluated-at-replay); until 2026-11-12 the profile key `render_cond`): agl clears a copy of the prepass depth again under a render condition the console skips. Run unconditionally, the opaque pass's `EQUAL` test fails everywhere ([PR 1299](https://github.com/seedds/nxnative/pull/1299)'s finding). @@ -88,6 +89,25 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, - **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. - Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needs [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` is linked otherwise). +## Screen shape + +Current. With `screen_aspect any` (launcher *Picture shape*, `DM_SCREEN_ASPECT`, `auto` by default) the game renders in the shape of the Mac window, live while playing, any shape from 0.25 to 6 (`loader.c` `SCREEN_MIN` / `SCREEN_MAX`, only to keep out a window a few pixels high). On iOS it follows the screen in landscape; upright it stays 16:9 (`front_ios.m`). The stubs are function-matched `others` of the frame-step rule in [`patches.py`](patches.py), found on v131072 `main` (build `e0f3ccfa…`): + +| Label | Function | What the stub does | +|---|---|---| +| `output size` | `main+0x1087f40`, the display settings' size setter | the requested size × the shape's factors (state +0xa4 / +0xa8), capped at 1.5× the pixels, each side a multiple of 8; copies the camera multipliers asked (+0xc4 / +0xc8) to the applied ones (+0xb8 / +0xbc). The crop, the scene's targets and the dynamic-resolution sizes follow | +| `window crop` | `main+0xe9a860`, setCrop | a crop beyond the window textures (2560x1440 at start) grows them; the next acquire re-creates them (`main+0x527fc0`) | +| `camera copy` | `main+0x33a8c0` (+0x2ec) | each frame the camera named "Main" hands its cached `sead::PerspectiveProjection` matrix here; rows x / y × the applied multipliers, the aspect beside it, in memory and in the registers the frustum code reads after | +| `display update` | `main+0x6502e0` (+0x18) | sets flag bit 17 when the loader's change count (+0xac) moved since the one applied (+0xc0): the game recomputes its size, so a resized window takes effect while playing | +| `hud upload` | `main+0x2d2ba0` (+0x4c) | the screen draw's copy of its uniform block (both orthographic matrices and the layout size): rows x / y × the HUD's multipliers, the size ÷ them, each frame | +| `layout size` | `main+0x583fd0`, the getter of a layout file's screen size | the size the HUD draws with, for the game's conversions between the layout space and the screen; the screen's init (`screen init`, `main+0x773b90`) still gets the file's | + +- **The rule.** Landscape wider than 16:9 keeps the vertical angle and shows more at the sides; between 1:1 and 16:9 it keeps the horizontal angle and shows more above and below. Upright takes the scale of the same screen turned landscape: its width gets what that screen's height gets, so turning a phone or a window keeps the characters' size and shows another part of the world. The output keeps the requested height (wider) or width (taller), scaled down to 1.5× the requested pixels past that (a phone upright renders 1280x2768). +- **Why the first attempts changed nothing:** the projection is cached ([a camera that caches its matrix](../../../docs/reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix)): `PerspectiveProjection::set`, `tanf` and the 16/9 literals do not run in play. Found with `DM_PROBE_FINDMAT` and the write watch on the render camera's matrix. +- **The HUD** (eui: layouts `UI/Layout2Archive/*.blarc.zs`). Every layout file holds a screen size of 3840x2160 (LayoutResData +0x200, file offset 0x2f8); each screen builds its orthographic projection from it once and copies it to the GPU each frame, so the HUD was drawn in a 16:9 space stretched over the output. The `hud upload` stub gives that space the output's shape at the HUD's pixel size: its multipliers are the camera's divided by the largest of them and 1, so in landscape the HUD keeps the size it has at 16:9 of the same height or width, and upright the whole 16:9 space still fits the width (smaller, nothing cut off). Checked on the Mac with live resizes (iPhone and iPad shapes upright and landscape, 1700x478, 16:9), the pause menu, the "A Open" prompt and the Kokiri Forest cutscene with its location card at 2.2. +- **Open: the HUD's edges.** eui places every pane about the screen's centre and does not anchor to its edges (a copy of PlayerStatus_00 made 4752 wide drew round hearts at their 16:9 place), so the HUD keeps its 16:9 arrangement in the middle: in wide shapes the hearts are at the 16:9 band's left edge, not the window's; upright the HUD sits around mid-height. The layouts mark their edge groups with 60x80 null panes at the space's edges (PlayerStatus_00's `N_ScaleL_00` at (-1920, 1078), `N_ScaleR_00` at (1920, 1078); MenuOverlay_00's at (0, ±1000)), and full-screen panes are exactly 3840x2160 at the centre (EventFade_00, LoadingFade_00, MenuBG_00): moving the first to the screen's edges and stretching the second needs eui's per-frame pane transform, not found yet. A pane instance (0xb0 bytes in the layout's array, layout +0x100) holds size +0x58, scale +0x60, rotation +0x68, translation +0x6c, a local rect +0x84..+0x90 and flags +0xa8, but no world matrix and no link to its record in the file. Until then the pause menu's backdrop and fades cover the 16:9 middle only, with the scene at the sides. +- **Open:** upright on iOS (16:9 for now); upright, a phone sees about 90° vertically, so parts of the world the game never shows (above Link's house) appear unbuilt: a cap on the vertical angle was offered, not decided. The title movie and other videos stay 16:9 with bars. + ## Saves The game autosaves into `slot_00` and `slot_01`, with `option.sav` and `account.sav`. The Load screen lists them and reloading works. A save's thumbnail is a 543x306 JPEG in `caption.sav`, made from a copy of a texture to a buffer. A manual save (Menu, System, Save) on the current build has the picture (slot 02, mean colour 138 / 123 / 86). The autosaves in the work folder's saves, made on older chains, hold a black one. Open: an autosave made on the current build has not been looked at. @@ -147,6 +167,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [A NaN colour under a zero blend weight](../../../docs/reference/shaders.md#a-nan-colour-under-a-zero-blend-weight): resolved, white clouds over the ground mist between Kokiri Forest and Hyrule Field and over the ferns at the camera in Kokiri Forest ([issue #1839](https://github.com/seedds/nxnative/issues/1839)). The mist shader `5ad65c0eefd09c7f` writes NaN at alpha 0 where its layer has density 0; NVIDIA's blender, with the option NVN's driver sets, adds nothing there, Apple's stored the NaN. - [A first start builds every base of the prewarm list before the first frame](../../../docs/reference/rendering.md#a-first-start-builds-every-base-of-the-prewarm-list-before-the-first-frame): the 450-475 s first-start prepare on an M5 Max; the Mac app now bundles the list's pipelines (`--prebake`, on by default) and it took 4.3 s. - [The 60 Hz link at 40 fps in fullscreen on a ProMotion screen](../../../docs/reference/frame-rate.md#the-60-hz-link-at-40-fps-in-fullscreen-on-a-promotion-screen): 40 fps in fullscreen on an M5 Pro MacBook Pro with the GPU at 10 ms of the 16.7, every present one 120 Hz refresh late and every other display-link tick skipped; the screen's fixed 60 Hz mode held 60.0 fps there and fullscreen now switches to it, with the 120 Hz link (59 fps) for screens with no such mode. +- [A screen-shape stub on the projection setter changes nothing: the camera caches its matrix](../../../docs/reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix): the camera copy and the HUD's uniform block copy ([Screen shape](#screen-shape)). - [Textures in GPU-only memory took an allocation per description](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description): the M1 iPad's GPU out-of-memory failure at the outdoor save load. - [The residency commit froze over the guest's files](../../../docs/reference/memory.md#the-residency-commit-froze-over-the-guests-files): resolved, the M1 iPad's freeze with sound and no picture. - [The shader package and the shader map come from different splits](../../../docs/reference/packing.md#the-shader-package-and-the-shader-map-come-from-different-splits): resolved, a black screen with sound in a build through NX2Apple ([issue #1841](https://github.com/seedds/nxnative/issues/1841)); the present pass was translated from a terrain shader's record. Records now carry their keys, and the shader build finds them by key. diff --git a/dm/games/README.md b/dm/games/README.md index 4888adcaef..847b70770d 100644 --- a/dm/games/README.md +++ b/dm/games/README.md @@ -247,6 +247,7 @@ One row per profile; the notes are each profile's `README.md`. | `device_api` | Optional, decided by the SDK version without it: the NVN API version `major.minor` that `nvnDeviceGetInteger` reports, 55.15 for SDK 11 and later (what Mario requires) and 53.313 for SDK 10 and older (`hle_nvn.c` `nvn_api`). Games check it and skip creating their device on a mismatch: Super Mario 3D World + Bowser's Fury (SDK 10) requires major 53 and minor ≥ 313. No profile sets it. `DM_NVN_API` overrides. Not part of the shader chain version (only `shader_*` keys are) | | `depth_mode` | `gl` (the default, NVN's near = −w) or `zero` (near = 0): the depth mode the game sets with `nvnDeviceSetDepthMode`, for the pipeline bases made before the game runs: the [pipeline archive made offline](../../docs/development.md#the-pipeline-archive-made-offline) for the iPhone (`main0` or `main0_z` vertex entries; built for a game with `pso_mtl4 on` or with `DM_PSO_GPUS`) and the NSP script's Mac prebuild (`DM_PSO_PREBUILD_DEPTH`). The host itself follows the mode the game sets; a wrong value only costs the device its own prebuild. The heap and address region checks of a provisional or derived profile set it when their boot reaches the game's call; without that call they leave it, as no call means unknown, not GL. Keep the key when the profile moves into `dm/games`: it is no shader input. Great Ace Attorney Chronicles, 13 Sentinels (both releases), Big Brain Academy, Link's Awakening, Engage, ASTRAL CHAIN, Luigi's Mansion 3 and Octopath Traveler 0 set `zero` | | `render_base` | The 3D views' base size, docked then handheld (`1600x900 1280x720`): what `DM_RENDER_SCALE` scales. `settings.py` writes it into the render scale's row of `launcher.json` as `base`, so a front end can show the setting in pixels. Only for a game with a render scale stub, and only a size its note documents. The slider's top is the stub's largest scale, which the rewriter writes as `RENDER_SCALE` in the fixups table (1.0, or 3.0 for Hades II, whose stub also raises its 1280x720) | +| `screen_aspect` | `off` (the default), `on` or `any`: the profile's stubs render the game in the screen's shape (`DM_SCREEN_ASPECT`, launcher setting *Picture shape*; `game.py` `DM_GAME_SCREEN_ASPECT` 0 / 1 / 2). `on` (Breath of the Wild, Animal Crossing: New Horizons): taller than 16:9 only, 1.3 to 1.77, set at the start; the host gives the game a taller output and layout size (`loader.c`, state +0x48..+0x9c). `any` (Ocarina of Time): any shape from 0.25 to 6, portrait included, followed while the game runs: the loader writes the ratio, size factors, a change count and the camera's multipliers (state +0xa0..+0xc8, `loader.c` `screen_any_write`) and the stubs scale the game's own output size, camera and HUD; the present shows the game's crop in its own shape | | `op_mode` | The operation mode the game sees at start: `console` (the default, docked) or `handheld`. Super Mario Galaxy and Captain Toad: Treasure Tracker use `handheld`: docked, their pointer is the Joy-Con gyro, which the host does not have; handheld, it is the touch screen, which the iPhone front end feeds with the touches no gamepad control takes (in the 3D layout, outside the right third of the screen, which is the right stick; `DmGamepad.m`). Handheld also renders at 720p. `DM_OPMODE=0` / `1` overrides, also from `Documents/env.txt` | | `perf_mode` | `follow` (the default): the performance mode follows the operation mode (Boost docked, Normal handheld); `boost`: Boost whatever the operation mode (`hle_oe.c`), a combination a Switch 1 never reports, so a profile choice. Rhythm Heaven Groove plays handheld with Boost: docked means a TV to the game, which then asks for a latency calibration, and only Boost gives its 1920x1080 draw size (handheld Normal draws 1280x720) | | `pad_layout` | The iPhone virtual gamepad's layout on first launch: `simple` (the default) = 2D (left stick, A/B/X/Y, shoulders, − / +), `full` = 3D (adds a D-pad, a right stick and L3 / R3; the right stick is not drawn: a touch landing in the right third of the screen on no control holds it, centred where it lands), `touch` = Tap (no control but the layout button: every touch goes to the game's touch screen; needs `op_mode handheld`, Balatro). The layout button at the top centre goes to the next layout in game (Tap only in handheld mode) and the app remembers the player's choice (`DmGamepad.m`) | diff --git a/dm/tools/docs_patches.py b/dm/tools/docs_patches.py index d164f274b6..dca82546fc 100644 --- a/dm/tools/docs_patches.py +++ b/dm/tools/docs_patches.py @@ -173,18 +173,18 @@ def build_rows(gs): # {build ID: [(game, display version, module)]} return out -OTHER_BASE = 0x8000000 # where the dry run puts a MATCH_FRAME_STEP rule's other functions, 16 MB apart, so their sites read as offsets in them; its patch area starts at 0x6000000, - # within a b's reach (128 MB) of a main's sites below it and of up to six other functions above +OTHER_BASE = 0x8000000 # where the dry run puts a MATCH_FRAME_STEP rule's other functions, OTHER_SPAN apart, so their sites read as offsets in them; its patch area starts at +OTHER_SPAN = 0x400000 # 0x6000000, within a b's reach (128 MB) of a main's sites below it and of up to 24 other functions above (16 MB apart, Ocarina of Time's ninth was out of reach) def frame_step_sites(fn, base=0, others=(), found=None): # (offset, reason) of each site a FRAME_STEP function (or a MATCH_FRAME_STEP stub, its function at base) writes, whether it marks a pos, puts = [0x6000000], [] # render-scale stub, the sites in each of the rule's other functions ({label: [(offset, reason)]}) and the capabilities it declares def emit(w): pos[0] += 4 - at = {label: OTHER_BASE + i * 0x1000000 for i, label in enumerate(others)} + at = {label: OTHER_BASE + i * OTHER_SPAN for i, label in enumerate(others)} ctx = types.SimpleNamespace(here=lambda: pos[0], emit=emit, site=lambda off, orig: None, put=lambda off, words, why: puts.append((off, why)), state=0x9000000, render_scale=0, step_max=0, base=base, others=at, found=found, capabilities=set(), address=lambda off: 0) fn(ctx) - in_others = {label: sorted((off - a, why) for off, why in puts if a <= off < a + 0x1000000) for label, a in at.items()} + in_others = {label: sorted((off - a, why) for off, why in puts if a <= off < a + OTHER_SPAN) for label, a in at.items()} return sorted((off, why) for off, why in puts if off < OTHER_BASE), ctx.render_scale, in_others, sorted(ctx.capabilities) diff --git a/docs/agents/game-patches.md b/docs/agents/game-patches.md index 2a2b3a199d..a229518fe6 100644 --- a/docs/agents/game-patches.md +++ b/docs/agents/game-patches.md @@ -43,7 +43,7 @@ Every patch bound tighter than it could be: the open gaps, then the blocked, kep | `build` → `engine` | RE Engine's frame limit at a fixed pointer | frame rate | blocked: one RE Engine game here | judgement | 1 | [Monster Hunter Rise](../../dm/games/0100B04011742000/README.md) | engine code other RE Engine games ship | | `build` → `engine` | Unity's constant-buffer table locked on the shared device | race | blocked: the stubs have no function-matched form yet, and no other Unity 2018.4 main on the drive shares these functions (Ori and the Blind Forest 1.0.2: no fingerprint match) | checked: probes of both threads inserting the same key and of the table API's callers by thread; Blind Forest main compared with nso_fnmap | 1 | [Ori and the Will of the Wisps](../../dm/games/01008DD013200000/README.md) | Unity 2018.4's threaded GfxDevice: the main thread registers layouts on the real device while the render thread uses the same table; any Unity 2018.4 build with this code has the race | | `build` → `game` | 60 fps frame step on 1.3.0 (the same stub at its known addresses) | frame rate | blocked: 1.3.0's main is on neither drive to fingerprint (v458752 is the update on them) | judgement | 1 | [Hyrule Warriors: Age of Calamity](../../dm/games/01002B00111A2000/README.md) | the stub above, placed at 1.3.0's addresses: the matched rule would serve it once its functions are fingerprinted | -| `build` → `game` | screen shape: the sizes, the view, the layouts' projection and their edge widgets (DM_SCREEN_ASPECT) | rendering | blocked: the stubs have no function-matched form yet | judgement | 1 | [Animal Crossing: New Horizons](../../dm/games/01006F8002326000/README.md) | the output size is the host's, but where the game takes its sizes, its cameras and its layouts are this build's code | +| `build` → `game` | screen shape: the sizes, the view, the layouts' projection and their edge widgets (DM_SCREEN_ASPECT) | rendering | blocked: the stubs have no function-matched form yet (Ocarina of Time's screen-shape stubs, function-matched others of its frame-step rule, are the model) | judgement | 1 | [Animal Crossing: New Horizons](../../dm/games/01006F8002326000/README.md) | the output size is the host's, but where the game takes its sizes, its cameras and its layouts are this build's code | | `build` → `game` | 60 fps intervals and dynamic-resolution constants | frame rate | kept: 1.0.3 (v131072) is the newest update of both releases and likely their last: a function match would serve no other build; the rewriter would also need matched rules that write fixups and declare capabilities, a new mechanism (2026-11-09) | checked: both releases' mains, where the host's rows for them were found; the global main's sites mapped into ASTRAL CHAIN and Bayonetta 1-3 (2026-11-09) | 2 | [NieR:Automata The End of YoRHa Edition (010056B015FE8000)](../../dm/games/010056B015FE8000/README.md); [NieR:Automata The End of YoRHa Edition (0100B8E016F76000)](../../dm/games/0100B8E016F76000/README.md) | the game's own constants; the Asian release's function is the global one's 0x13c0 further on, so one function-matched rule could serve both releases. No PlatinumGames sibling shares the code: none of its sites has an exact match in ASTRAL CHAIN or Bayonetta 1-3 (nso_fnmap, 2026-11-09) | | `build` → `game` | NieR's frame interval and dynamic-resolution floor at fixed addresses | frame rate | kept: 1.0.3 (v131072) is the newest update of both releases and likely their last: a function match would serve no other build; the rewriter would also need matched rules that write fixups and declare capabilities, a new mechanism (2026-11-09) | checked: both releases' mains, where these rows were found; the global main's sites mapped into ASTRAL CHAIN and Bayonetta 1-3 (2026-11-09) | 2 | [NieR:Automata The End of YoRHa Edition (010056B015FE8000)](../../dm/games/010056B015FE8000/README.md); [NieR:Automata The End of YoRHa Edition (0100B8E016F76000)](../../dm/games/0100B8E016F76000/README.md) | the game's own display object; the Asian release's function is the global one's 0x13c0 further on, so the rewriter's function match could give the host both releases' addresses. No PlatinumGames sibling shares the code: none of its sites has an exact match in ASTRAL CHAIN or Bayonetta 1-3 (nso_fnmap, 2026-11-09) | | `game` → `host` | one vertex shader's visibility counter (VertexCounterLowering) | shaders | kept: one shader in one game, already handled; a general lowering is a chain change (every game's shaders rebuilt) for no measured gain: revisit when a vertex shader with subgroup operations fails in another game | checked: the shader-build logs of the 51 game builds on the game drive: no shader failed to translate or compile, none mentions a subgroup operation (2026-11-09) | — | [Program.cs:266](../../dm/gfx/tools/dmsh_xlate/Program.cs#L266) | a general lowering of subgroup ballots in vertex shaders (subgroups of one invocation: ballots, votes, lane ids and counts, refusing shuffles that exchange data between lanes) would cover any shader that counts that way | @@ -80,7 +80,7 @@ Every patch bound tighter than it could be: the open gaps, then the blocked, kep | [Taiko Risshiden V DX](#taiko-risshiden-v-dx) | 1 | 0 | 0 | 2 | 0 | `game` | | [The Legend of Zelda: Breath of the Wild](#the-legend-of-zelda-breath-of-the-wild) | 7 | 0 | 43 | 0 | 0 | `build` → `game` ×2 | | [The Legend of Zelda: Link's Awakening](#the-legend-of-zelda-links-awakening) | 1 | 0 | 0 | 0 | 0 | `build` → `game` | -| [The Legend of Zelda: Ocarina of Time](#the-legend-of-zelda-ocarina-of-time) | 0 | 0 | 2 | 0 | 0 | `game` ×2 | +| [The Legend of Zelda: Ocarina of Time](#the-legend-of-zelda-ocarina-of-time) | 0 | 0 | 8 | 0 | 0 | `game` ×4 | | [The Legend of Zelda: Tears of the Kingdom](#the-legend-of-zelda-tears-of-the-kingdom) | 0 | 0 | 9 | 0 | 0 | `game` | | [Xenoblade Chronicles 3](#xenoblade-chronicles-3) | 6 | 0 | 0 | 0 | 0 | `build` → `game` | @@ -90,7 +90,7 @@ Every patch bound tighter than it could be: the open gaps, then the blocked, kep [Animal Crossing: New Horizons](../../dm/games/01006F8002326000/README.md), `01006F8002326000`, [patches.py](../../dm/games/01006F8002326000/patches.py). -- **`build` → `game`**, rendering, blocked: the stubs have no function-matched form yet, judgement: screen shape: the sizes, the view, the layouts' projection and their edge widgets (DM_SCREEN_ASPECT). The output size is the host's, but where the game takes its sizes, its cameras and its layouts are this build's code +- **`build` → `game`**, rendering, blocked: the stubs have no function-matched form yet (Ocarina of Time's screen-shape stubs, function-matched others of its frame-step rule, are the model), judgement: screen shape: the sizes, the view, the layouts' projection and their edge widgets (DM_SCREEN_ASPECT). The output size is the host's, but where the game takes its sizes, its cameras and its layouts are this build's code - **3.0.3 `main`** `ff1d1c05…` ([patches.py:228](../../dm/games/01006F8002326000/patches.py#L228))
14 frame-step sites @@ -580,8 +580,10 @@ Every patch bound tighter than it could be: the open gaps, then the blocked, kep - **`game`**, frame rate, at goal, checked: v131072 main: one function with this fingerprint, the site and its free registers read by disassembly (2026-11-08): frame ratio from the host's step. The game's own frame-ratio division, found by its function's fingerprint on any build that keeps the function - **`game`**, rendering, at goal, checked: v131072 main: one function with each fingerprint (the level function a leaf, x2..x11 / x16 / x17 / s0..s2 free), its callers, the update and the controller fields read by disassembly, the fields read in play (2026-11-08); the climb run natively against a model of the update and in play to full size (2026-11-09): dynamic resolution: full size, or a climb on the load measured at each level. The game's own dynamic-resolution level function and update, found by their functions' fingerprints on any build that keeps both; switchable (DM_DYNAMIC_RESOLUTION) +- **`game`**, rendering, at goal, checked: v131072 main: one function with each fingerprint, the sites and their free registers read by disassembly, the camera matrix and its writer found in play with DM_PROBE_FINDMAT and the write watch, the stubs run natively (test_oot_aspect.py) and on the Mac at six window shapes resized live (2026-11-09): screen shape: the output size, the window textures, the camera projection and a live shape change (screen_aspect any). The game's own size setter, setCrop, camera copy and display update, found by their functions' fingerprints on any build that keeps them; switchable (DM_SCREEN_ASPECT) +- **`game`**, rendering, at goal, checked: v131072 main: one function with each fingerprint, the block layout and its per-frame copy read by disassembly, the 3840x2160 size read in play with probes and changed in a copy of a layout file, the stubs run natively (test_oot_aspect.py) and on the Mac at six shapes, the pause menu and a cutscene (2026-11-09): HUD: the screens' orthographic projection and the layout size take the output's shape. The screen draw's uniform block copy and the layout-size getter, found by their functions' fingerprints with the screen init beside them -- **function-matched frame step** (`main` of builds without FRAME_STEP): frame ratio from the host's step: a function of 0x9b4 bytes, `+0xe4` in it; dynamic resolution, a function of 0x294 bytes, `+0x0` in it; dynamic resolution update, a function of 0xb24 bytes, in it; launcher capabilities: DYNAMIC_RESOLUTION +- **function-matched frame step** (`main` of builds without FRAME_STEP): frame ratio from the host's step: a function of 0x9b4 bytes, `+0xe4` in it; dynamic resolution, a function of 0x294 bytes, `+0x0` in it; dynamic resolution update, a function of 0xb24 bytes, in it; output size, a function of 0xd8 bytes, `+0x0` in it; camera copy, a function of 0x1648 bytes, `+0x2ec` in it; display update, a function of 0x554 bytes, `+0x18` in it; window crop, a function of 0x14 bytes, `+0x0` in it; hud upload, a function of 0x70 bytes, `+0x4c` in it; screen init, a function of 0x62c bytes, in it; layout size, a function of 0xc bytes, `+0x0` in it; launcher capabilities: DYNAMIC_RESOLUTION ### The Legend of Zelda: Tears of the Kingdom diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 99e499a814..b12895b303 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -915,6 +915,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - read-only depth/stencil attachments are written back after every pass, and the periodic unwritten-store savings counter remains zero with the default configuration. → [Unwritten attachment stores without lookahead](../reference/rendering.md#unwritten-attachment-stores-without-lookahead) - portrait gameplay overlaps the touch controls, touch coordinates land outside the game picture, or a TV's picture renders at 4K again after a layout pass. → [iOS layouts: portrait, a half-folded Duo, iPad windows](../reference/rendering.md#ios-layouts-portrait-a-half-folded-duo-ipad-windows) - with *Picture shape* at 1.43:1 (`DM_SCREEN_ASPECT`) the game renders 1920x1344, but on an iPhone Duo in Fit (the default) the picture comes out smaller than the 16:9 one, with black bars on all four sides; it fills the screen only in Fill. On the Mac a click lands off the point under the pointer (it was mapped on a 16:9 rectangle of the window). → [The picture laid out in the shape the game presents](../reference/rendering.md#the-picture-laid-out-in-the-shape-the-game-presents) +- with a wider or taller output (`DM_SCREEN_ASPECT`) the picture is stretched although a stub changes the aspect `sead::PerspectiveProjection::set` receives, or the 16:9 constants the code holds; the HUD stays stretched over the output too. → [A screen-shape stub on the projection setter changes nothing: the camera caches its matrix](../reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix) ## Shaders diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 73f7379942..2168b73670 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -280,6 +280,9 @@ DM_PROBE diagnostic `=module:offset,…` (Mac only): puts a `brk` at each guest DM_PROBE_ALL diagnostic Set: probe hits (`DM_PROBE`) also print x8–x18 DM_PROBE_DUMP diagnostic `=module:offset:length[:file]`: on the first hit of any probe, writes that range of guest memory to a file (default `run//probe_dump.bin`) DM_PROBE_EVERY diagnostic `=N`: every guest probe hit inside `DM_PROBE_FRAMES` counts, every N-th prints (its `#` is then the true hit count, for rates); `DM_PROBE_LIMIT` caps the printed ones +DM_PROBE_FINDF diagnostic `=,,[,[,]]` (Mac): that many seconds after start scans the opened guest heap and the alias region's mapped parts for the float a with the float b `` bytes after it (0: both 4 and 16), prints each place with the floats around it, and arms the write watch on the n-th, for 40 printed stores +DM_PROBE_FINDMAT diagnostic `=[,]` (Mac): that many seconds after start scans the guest heap for perspective matrices (row or column major), prints each with its aspect, and arms the write watch on the n-th at 16:9, for 40 printed stores: who builds a camera's projection when it is cached +DM_PROBE_FP diagnostic Set: probe hits (`DM_PROBE`) also print s0–s7 as floats (the float arguments of the probed function) DM_PROBE_FRAMES diagnostic Presented frame or inclusive `start-end` range in which guest probes print observations DM_PROBE_HEAP diagnostic `=,,,,`: hex offsets of the sead heap layout the probes' `!xN` heap walk uses (default Mario's `0,28,30,b0,c8`) DM_PROBE_HEAPFRAME diagnostic `=` (hex): the probes' heap walk treats heaps of that class as frame heaps (head +0xe0, tail +0xe8) @@ -288,8 +291,9 @@ DM_PROBE_HEAPRAW diagnostic Set: the probes' heap walk also prints the heap obje DM_PROBE_HEAPWORDS diagnostic Set: the probes' heap walk also prints each used block's first two words DM_PROBE_LIMIT diagnostic Maximum printed hits per guest probe; 0 or absent leaves the count unlimited DM_PROBE_POLL diagnostic `=
[+]` (hex, 4 bytes by default, up to 8): a host thread spins on the bytes and on each change snapshots every thread's pc, lr and frames, for writes the watch misses -DM_PROBE_WATCH diagnostic `=
[+]` (hex, 8 bytes by default): write-protects that host page and reports who writes these bytes, without a debugger +DM_PROBE_WATCH diagnostic `=
[+]` (hex, 8 bytes by default): write-protects that host page and reports who writes these bytes, without a debugger; `heap+` from the guest heap's base; `@xN+[+]` that register plus the offset at the first printed probe hit (objects whose addresses move from run to run) DM_PROBE_X2 diagnostic `=` (hex): probe hits also dump that many bytes of the memory x2 points to +DM_PROBE_XREG diagnostic `=N`: `DM_PROBE_X2` dumps the memory xN points to instead of x2's DM_PROGRAM switch `=`: the program of a game of several programs that a host over their profiles runs (default 0; `main.c` `dm_game_cur`) DM_PROGRAMS path Table of a game's programs on the Mac (`programs.tsv`: index, host, modules, run folder, environment): `oe::ExecuteProgram` starts the named one in this process's place; the NSP script's run command and the Mac app set it DM_PROGRAM_ID switch `=`: the program ID the host reports from `svcGetInfo(ProgramId)` (default: the profile's `program_id`) @@ -345,7 +349,7 @@ DM_SAVED_CHOICES switch `0` ignores the player's saved choices (launcher. i DM_SAVE_SYNC player Cloud saves: the folder whose `/save.dmsync` the save is synced through, `icloud` for iCloud Drive/nxnative Saves (the Mac); `0` or unset, off. The Mac launcher sets the folder chosen in its settings window's Cloud saves, the iPhone app the one picked in its launcher (dm/host/save_sync.h); regress.py, bench.py and the NSP script's own host runs set `0` DM_SAVE_SYNC_CHOICE switch Cloud saves: set by a front end that asks the player when the cloud's save and this device's both changed (any value: a save that changed on both sides is replaced on neither without a choice); `ask`: no choice yet, `cloud:<hash>` brings in the cloud's save with that hash, `local:<hash>` sends this device's over it (`local:none`: the cloud had none), `offline`: the cloud could not be read before the start, so the host does not wait for it; unset, the newer save wins (dm/host/save_sync.c, docs/frontends.md) DM_SCHEME_NAME switch Set by `build_ipa.sh` itself for xcodegen's `project.yml`: the Xcode scheme name, `DmIOS`, or `<display name> (<title ID>)` for a project prepared with `--prepare-xcode` -DM_SCREEN_ASPECT player `=<width/height>` (1.3 to 1.77, e.g. `1.43` for the iPhone Duo) renders a game whose profile has `screen_aspect on` in that shape: output 1920 wide, 1280-wide layout space, the HUD at the taller edges and 16:9's sides in view; `auto` (iOS) the app window's own shape when it is taller than 16:9, else 16:9; `0` or unset is 16:9; launcher setting +DM_SCREEN_ASPECT player `=<width/height>` renders a game whose profile has `screen_aspect` in that shape. `on` profiles (taller than 16:9, set at the start): 1.3 to 1.77, e.g. `1.43` for the iPhone Duo; output 1920 wide, 1280-wide layout space, the HUD at the taller edges and 16:9's sides in view; `auto` (iOS) the app window's own shape when it is taller than 16:9, else 16:9; `0` or unset is 16:9. `any` profiles (followed while the game runs): 0.25 to 6, portrait included; `auto` or unset follows the Mac window, or the iOS screen in landscape (upright on iOS it stays 16:9), `1.777` or `0` is 16:9, a number fixes the shape. Launcher setting DM_SCREEN_ASPECT_SITES diagnostic `=<mask>` with DM_SCREEN_ASPECT: which of the profile's layout-size conversion stubs follow the taller layout, one bit each (default all) DM_SCREEN_FOV diagnostic `off` with DM_SCREEN_ASPECT: the scene keeps the game's vertical angle (narrower sides) instead of 16:9's sides with more above and below DM_SCREEN_HUD diagnostic `=<edge>,<listed>,<full width>,<full height>[,<tallest>]` with DM_SCREEN_ASPECT: the layout rules' distances and sizes in the 1280x720 space (default 280,100,1270,710,1300) diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index 7a89a4539b..156968ae10 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -1862,3 +1862,10 @@ Digimon Story Cyber Sleuth: Complete Edition (Media.Vision's MVGL engine, SDK 6. - **Fix:** the present reports the shape of the frame it draws ([the layer's](#a-crop-only-picks-texels-the-picture-has-the-layers-shape)) whenever its width / height changes (`nvn_mtl.m` `present_now` → `front.h` `front_picture_shape`; every graphics API presents there). The iOS front end lays the game's view out in that shape (`DmLayout.h` takes it as a parameter: the overlay, the split, Fill and the TV alike), so a 1.43:1 picture fills a 1.43:1 screen in Fit and touches are normalized on the picture as drawn; the Mac maps the mouse on the picture's own rectangle. A 16:9 game reports 16:9 and keeps its layout, and a size change of the same shape (1280x720 handheld, 1920x1080 docked) is not reported. - **Check:** the [Duo harness](../../dm/ios/tests/duo/README.md) tests the taller picture in Fit, in Fill and on a wider screen, and the production root view following a reported shape and returning to 16:9. Not yet run on a device. - **Found on:** [Breath of the Wild](../../dm/games/01007EF00011E000/README.md#screen-shape-dm_screen_aspect-current) and [Animal Crossing: New Horizons](../../dm/games/01006F8002326000/README.md#screen-shape-dm_screen_aspect-current) with `DM_SCREEN_ASPECT=1.43`. + +### A screen-shape stub on the projection setter changes nothing: the camera caches its matrix + +- **Symptom:** with a wider or taller output (`DM_SCREEN_ASPECT`) the picture is stretched although a stub changes the aspect `sead::PerspectiveProjection::set` receives, or the 16:9 constants the code holds; the HUD stays stretched over the output too. +- **Cause:** the projection is computed once and cached: the camera's matrix lives inside the projection object, and in play the game only copies it into the render camera each frame (`set`, `tanf` and the 16/9 literals do not run). The HUD's orthographic projection is built once per screen from the layout file's own screen size and kept in the screen's uniform block, which each frame's draw only copies. +- **Fix:** find the per-frame reader, not the builder, and change what it hands on: `DM_PROBE_FINDMAT=<seconds>` scans the guest heap for perspective matrices and arms the write watch (`DM_PROBE_WATCH`) on one at 16:9, whose writer is the per-frame copy; `DM_PROBE_FINDF` does the same for any pair of floats (a pane's position), and `DM_PROBE_WATCH=@xN+<offset>` arms the watch on an object a probe reaches, for objects whose addresses move from run to run. The stub then scales the copied rows by the shape's multipliers each frame (idempotent, so a live resize takes effect at the next frame) and leaves the cached object alone. +- **Found on:** [Ocarina of Time](../../dm/games/0400486024DEC000/README.md#screen-shape): the camera copy (`main+0x33a8c0`) and the HUD's uniform block copy (`main+0x2d2ba0`). diff --git a/docs/settings.md b/docs/settings.md index 95ef082f48..e2e3db055b 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -272,7 +272,7 @@ A line like `DM_PRESENT_SNAP=0` in `Documents/env.txt` (in the Files app, under | Display mode (Mac) | `DM_FULLSCREEN` | `1` fullscreen, `0` a window | | Cloud saves folder (Mac) | `DM_SAVE_SYNC` | a folder; `icloud`: iCloud Drive › nxnative Saves; `0`: off | | Picture (iPhone and iPad) | `DM_PICTURE_FIT` | `fit` 16:9 with black bars, `fill` covers the screen, cropped | -| Picture shape | `DM_SCREEN_ASPECT` | `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9 | +| Picture shape | `DM_SCREEN_ASPECT` | Profiles with `screen_aspect on`: `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9. Profiles with `screen_aspect any` (Ocarina of Time): `auto` (the default) follows the Mac window while playing, any shape, or the iOS screen in landscape; `1.777` 16:9 | | Frame-rate counter | `DM_SHOW_FPS` | `1`, `0` | | Touch feedback | `DM_TOUCH_HAPTICS` | `1`, `0` | | Game rumble | `DM_RUMBLE` | `1`, `0` | From 6f4c4276a763406b0b3dfb2a3df28baea8aa374f Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Mon, 9 Nov 2026 18:17:23 +0100 Subject: [PATCH 11/40] file_wire keeps its list when growing it fails: the range it cannot record is unwired at once instead of the list being lost and written through a null pointer clang-tidy (bugprone-suspicious-realloc-usage) on 29fa5b0e's g_fwire = realloc(g_fwire, ...). t_file_wire: 10 / 10. --- dm/gfx/nvn/nvn_mtl.m | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index ad3b1bf4a7..81faa7ca41 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -318,7 +318,11 @@ static void file_wire(id<MTLAllocation> r) { return; } g_fwire_ns += clock_gettime_nsec_np(CLOCK_UPTIME_RAW) - t0; g_fwire_n++; g_fwire_bytes += hi - lo; - if (g_nfwire == g_cfwire) { g_cfwire = g_cfwire ? 2 * g_cfwire : 64; g_fwire = realloc_or_die(g_fwire, g_cfwire * sizeof *g_fwire, "the wired file ranges"); } + if (g_nfwire == g_cfwire) { // the list file_unwire undoes; a range it cannot hold is unwired at once (the commit pages it in itself, as before) + size_t n = g_cfwire ? 2 * g_cfwire : 64; void *grown = realloc(g_fwire, n * sizeof *g_fwire); + if (!grown) { munlock((void *)lo, hi - lo); return; } + g_fwire = grown; g_cfwire = n; + } g_fwire[g_nfwire].p = (void *)lo; g_fwire[g_nfwire++].n = hi - lo; } static void file_unwire(fwire_t *fw, size_t n) { for (size_t i = 0; i < n; i++) munlock(fw[i].p, fw[i].n); free(fw); } // a range the game has mapped anew meanwhile fails, harmlessly From 1c92e400afa739dd9bc5a9b93acf6e04a034624e Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Mon, 9 Nov 2026 23:29:56 +0100 Subject: [PATCH 12/40] A shader-code pool's own address keeps it out of the raw pool table: the GPU never reads shader code, so nothing makes those pools resident and wires their pages for the table's sake (Ocarina of Time on the iPhone: 223 pools, 317 MB) nvnMemoryPoolGetBufferAddress on a pool with NVN_MEMORY_POOL_FLAGS_SHADER_CODE (not virtual) reports kind 4, which nvn_mtl.m records for res_top but leaves out of the addressed ranges. A buffer made in such a pool still puts it in (nvnBufferInitialize reports kind 0 and 1). On the 8 GB iPhone the wired pools left too little room for the guest's file pages, which were read back 0.2-0.4 GB every 10 s while playing. t_addr_known checks it. --- dm/gfx/nvn/hle_nvn.c | 10 +++++++++- dm/gfx/nvn/nvn_mtl.h | 2 +- dm/gfx/nvn/nvn_mtl.m | 12 ++++++++++-- dm/gfx/tests/t_addr_known.m | 18 ++++++++++++++++++ 4 files changed, 38 insertions(+), 4 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index 89ae057794..af48ef7b11 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -284,7 +284,15 @@ NVN(nvnMemoryPoolFlushMappedRange, void, const NVNmemoryPool *p, ptrdiff_t off, } // A GPU address exists only once the driver has given one: here, by a buffer (nvnBufferInitialize) or by a texture (nvnTextureGetTextureAddress). Raw global memory can reach only what these name (mtl_pool_addressed) static void pool_mark_addressed(const pool_t *o) { static __thread const uint8_t *last; if (o->mem && o->mem != last) { last = o->mem; mtl_pool_addressed_by(o->mem, o->size, 0); } } -NVN(nvnMemoryPoolGetBufferAddress, NVNbufferAddress, const NVNmemoryPool *p) { pool_mark_addressed((const pool_t *)p); return (NVNbufferAddress)((const pool_t *)p)->mem; } +// A shader-code pool's own address (NVN_MEMORY_POOL_FLAGS_SHADER_CODE, 0x40; not virtual) is asked for to place programs: nvnProgramSetShaders takes code addresses. The GPU never reads that code, +// as the shaders run translated and their constants are copied into the host's buffers, so it does not put the pool into the raw pool table (kind 4), where it would be resident and its pages wired. +// Ocarina of Time on the iPhone: 223 such pools, 317 MB, wired for nothing on an 8 GB device that pages the guest's files out. A buffer made in the pool still puts it in (nvnBufferInitialize) +static void code_pool_addressed(const pool_t *o) { static __thread const uint8_t *last; if (o->mem && o->mem != last) { last = o->mem; mtl_pool_addressed_by(o->mem, o->size, 4); } } +NVN(nvnMemoryPoolGetBufferAddress, NVNbufferAddress, const NVNmemoryPool *p) { + const pool_t *o = (const pool_t *)p; + if ((o->flags & 0x240) == 0x40) code_pool_addressed(o); else pool_mark_addressed(o); + return (NVNbufferAddress)o->mem; +} // ---------- Descriptor pools ---------- // One ID space per pool. In NVN a texture or sampler ID indexes the pool the command buffer has bound (SetTexturePool / SetSamplerPool), and two pools may use the same IDs. The replay keeps diff --git a/dm/gfx/nvn/nvn_mtl.h b/dm/gfx/nvn/nvn_mtl.h index 063f938dc2..2738426a72 100644 --- a/dm/gfx/nvn/nvn_mtl.h +++ b/dm/gfx/nvn/nvn_mtl.h @@ -15,7 +15,7 @@ void bf_want_note(const char *key); // Memory pool: [mem, mem+size) is expanded outward to page boundaries to build a zero-copy buffer; look up the buffer and offset by GPU address (= CPU address) void mtl_pool_add(const void *mem, uint64_t size); void mtl_pool_addressed(const void *mem, uint64_t size); // the game holds a GPU address in this range, so raw global memory may reach it -void mtl_pool_addressed_by(const void *mem, uint64_t size, int kind); // the same, with who gave it out (0 pool, 1 buffer: its range, the table takes the whole pool, 2 texture, 3 virtual mapping), for res_top +void mtl_pool_addressed_by(const void *mem, uint64_t size, int kind); // the same, with who gave it out (0 pool, 1 buffer: its range, the table takes the whole pool, 2 texture, 3 virtual mapping), for res_top; 4: a shader-code pool's own address, which keeps it out of the table void mtl_pool_tag(const void *mem, uint32_t tag); // res_top: the pool's flags (bit 31: a mapped run of a virtual pool) void mtl_pool_del(const void *mem); void mtl_pool_replace(const void *const *del, int ndel, const void *const *add, const uint64_t *size, int nadd); // in one step for every lookup (a sparse pool's runs, hle_nvn.c vrun_set) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 81faa7ca41..1942e8b467 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -933,7 +933,13 @@ static uint64_t rng_overlap(const rng_tab *t, uintptr_t lo, uintptr_t hi) { uint64_t z = 0; for (int i = a; i < t->n && t->r[i].lo < hi; i++) z += MIN(hi, t->r[i].hi) - MAX(lo, t->r[i].lo); return z; } +static rng_tab g_addr_code; // shader-code pools whose own address was asked for (kind 4, hle_nvn.c nvnMemoryPoolGetBufferAddress), kept out of the table. Under g_mu void mtl_pool_addressed_by(const void *mem, uint64_t size, int kind) { + if (kind == 4) { + pthread_mutex_lock(&g_mu); int first = !g_addr_code.n; rng_put(&g_addr_code, (uintptr_t)mem, (uintptr_t)mem + size); pthread_mutex_unlock(&g_mu); + if (first) MLOG("raw global memory: shader-code pools whose address the game asks for stay out of the pool table, so nothing makes them resident for its sake (first %p, %llu KB)", mem, (unsigned long long)(size >> 10)); + return; + } if (kind != 1) mtl_pool_addressed(mem, size); pthread_mutex_lock(&g_mu); if (kind >= 0 && kind < 4) rng_put(&g_addrk[kind], (uintptr_t)mem, (uintptr_t)mem + size); pthread_mutex_unlock(&g_mu); } @@ -975,10 +981,12 @@ static void res_top_pools(const char *why) { if (p->tag >> 31) { vn++; vb += z; } int j = 0; while (j < ntg && tg[j].tag != p->tag) j++; if (j == ntg && ntg < 12) { tg[ntg].tag = p->tag; tg[ntg].n = tg[ntg].b = tg[ntg].ln = tg[ntg].lb = 0; ntg++; } if (j < 12) { tg[j].n++; tg[j].b += z; if (p->n_other) { tg[j].ln++; tg[j].lb += z; } } } uint64_t ak[4] = {0}; for (int k = 0; k < 4; k++) for (int i = 0; i < g_addrk[k].n; i++) ak[k] += g_addrk[k].r[i].hi - g_addrk[k].r[i].lo; + uint64_t code_n = 0, code_b = 0; for (int i = 0; i < g_addr_code.n; i++) if (!addr_meets(g_addr_code.r[i].lo, g_addr_code.r[i].hi)) { code_n++; code_b += g_addr_code.r[i].hi - g_addr_code.r[i].lo; } pthread_mutex_unlock(&g_mu); - MLOG("%s: resident memory pools %llu, %llu MB: in the raw pool table and looked up otherwise %llu, %llu MB; in the table only %llu, %llu MB; not in it, looked up %llu, %llu MB; neither %llu, %llu MB; mapped runs of virtual pools %llu, %llu MB; addressed in all (MB): pools %llu, buffers %llu, textures %llu, virtual mappings %llu", why, + MLOG("%s: resident memory pools %llu, %llu MB: in the raw pool table and looked up otherwise %llu, %llu MB; in the table only %llu, %llu MB; not in it, looked up %llu, %llu MB; neither %llu, %llu MB; mapped runs of virtual pools %llu, %llu MB; addressed in all (MB): pools %llu, buffers %llu, textures %llu, virtual mappings %llu; shader-code ranges kept out of the table %llu, %llu MB", why, (unsigned long long)rn, (unsigned long long)(rb >> 20), (unsigned long long)cn[3], (unsigned long long)(cb[3] >> 20), (unsigned long long)cn[2], (unsigned long long)(cb[2] >> 20), (unsigned long long)cn[1], (unsigned long long)(cb[1] >> 20), (unsigned long long)cn[0], (unsigned long long)(cb[0] >> 20), - (unsigned long long)vn, (unsigned long long)(vb >> 20), (unsigned long long)(ak[0] >> 20), (unsigned long long)(ak[1] >> 20), (unsigned long long)(ak[2] >> 20), (unsigned long long)(ak[3] >> 20)); + (unsigned long long)vn, (unsigned long long)(vb >> 20), (unsigned long long)(ak[0] >> 20), (unsigned long long)(ak[1] >> 20), (unsigned long long)(ak[2] >> 20), (unsigned long long)(ak[3] >> 20), + (unsigned long long)code_n, (unsigned long long)(code_b >> 20)); char line[1400] = ""; size_t lo = 0; for (int m = 0; m < 16 && lo < sizeof line; m++) if (mn[m]) lo += snprintf(line + lo, sizeof line - lo, "%s[%s%s%s%s] %llu, %llu MB (buffers' ranges %llu MB, textures' %llu MB)", lo ? "; " : "", m & 1 ? "P" : "", m & 2 ? "B" : "", m & 4 ? "T" : "", m & 8 ? "V" : "", (unsigned long long)mn[m], (unsigned long long)(mb[m] >> 20), (unsigned long long)(mcov[m][1] >> 20), (unsigned long long)(mcov[m][2] >> 20)); MLOG("%s: resident memory pools by who gave an address in them (P pool, B buffer, T texture, V virtual mapping): %s", why, line); diff --git a/dm/gfx/tests/t_addr_known.m b/dm/gfx/tests/t_addr_known.m index 15994d41b7..23f1e704a4 100644 --- a/dm/gfx/tests/t_addr_known.m +++ b/dm/gfx/tests/t_addr_known.m @@ -41,6 +41,24 @@ int main(void) { @autoreleasepool { CHECK("asking makes no chunk and nothing resident", g_npool == np0 && g_pool_res_n == rb0); CHECK("mtl_addr makes a chunk for the same range", mtl_addr(bb + (100u << 20), 4096, &mb, &mo) && g_npool == np0 + 1); mtl_pool_del(big); + + // The raw pool table (gmem_map) and shader-code pools: hle_nvn.c nvnMemoryPoolGetBufferAddress reports a shader-code pool's own address as kind 4, which keeps the pool out of + // the table, so nothing makes it resident and wires its pages for the table's sake; a buffer made in it (kind 0, then 1) puts it in, as does any other pool's address + uint8_t *code = mmap(NULL, n, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0), *data = mmap(NULL, n, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0); + if (code == MAP_FAILED || data == MAP_FAILED) abort(); + uint64_t cb = (uint64_t)(uintptr_t)code, db = (uint64_t)(uintptr_t)data; + mtl_pool_add(code, n); mtl_pool_tag(code, 0x64); mtl_pool_add(data, n); mtl_pool_tag(data, 0x24); + mtl_pool_addressed_by(code, n, 4); mtl_pool_addressed_by(data, n, 0); + #define IN_TABLE(b) ({ id<MTLBuffer> t_ = gmem_map(); const uint8_t *e_ = (const uint8_t *)t_.contents; int in_ = 0; \ + for (int k = 1; k < DMSH_GMEM_MAP_ENTRIES && !in_; k++) { uint64_t r_[2]; memcpy(r_, e_ + (size_t)k * DMSH_GMEM_MAP_ENTRY, sizeof r_); in_ = r_[0] <= (b) && (b) < r_[1]; } in_; }) + CHECK("a shader-code pool's own address keeps it out of the raw pool table", !IN_TABLE(cb + 4096)); + CHECK("so the table makes it not resident", !pool_resident(cb)); + CHECK("another pool's address puts it in the table", IN_TABLE(db + 4096)); + CHECK("and makes it resident", pool_resident(db)); + mtl_pool_addressed_by(code + 8192, 4096, 0); mtl_pool_addressed_by(code + 8192, 4096, 1); // what nvnBufferInitialize reports for a buffer in the code pool + CHECK("a buffer in the shader-code pool puts it in the table", IN_TABLE(cb + 4096)); + CHECK("and makes it resident", pool_resident(cb)); + mtl_pool_del(code); mtl_pool_del(data); printf("t_addr_known: %d checks, %d failed\n", checks, failed); return failed != 0; } } From 66867a7864ccbfc94a32d3f9cc66b547b3f8d274 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Fri, 6 Nov 2026 14:59:47 +0100 Subject: [PATCH 13/40] OoT3 2b: the frame step calls the GPU slowed only against a reference that measured the same work, so a loading screen's GPU time no longer starts probes of 60 fps nvn_fstep_gpu_slowed compares the GPU time at a longer hold with a running mean of the frames at kmin; 1.3 times as much is taken for a clock that fell with the hold and starts a probe back to kmin. In Ocarina of Time on an M4 Air the only long run at hold 1 was the title and the loading screen (GPU 0.2 vsyncs, 2-60 draws). Gameplay (700-900 draws, 1.3 vsyncs at hold 1, 1.4 at hold 2) read as a GPU slowed 6 times: 32 probes of hold 1 in a 120 s walk, all failed, each failure's late frames stepping on to hold 3. nvn_fgpu keeps the draws a frame encodes beside each reference (dref_at) and at the longer hold (dcur); fstep_tick takes them from mtl_stats. The GPU counts as slowed only while those are within FSTEP_GPU_WORK (1.5) of each other; without draw counts (0) it is as before. t_fstep (81 checks, 0 failed): a loading-screen reference (30 draws) against gameplay at 900 is not slowed and makes no probe, where the same run without draw counts makes 8 failed probes; the same draws at both holds still give Breath of the Wild's one probe that holds. Measured on the M4 Air, Link's house (120 s walk): probes 32 (all failed) -> 7 (all at the title and the load), frame max 122 -> 73 ms, p95 51.1 -> 51.8 ms, but 27.2 -> 24.2 fps: the failed probes' late frames had stepped down to hold 2 now and then, and the hold-3 stretches left come from GPU time at hold 2 of p90 33 ms against 33.3. Kokiri Forest: 19.0 fps, every frame at hold 3, GPU span 42.4 ms, game speed 0.95; with DM_FPS_TARGET=30 21.0 fps at hold 2, game speed 0.92-0.94. Not run: regress.py (the owner's rule for this branch); BotW and TotK, which use the same controller, were not measured. --- dm/gfx/nvn/hle_nvn.c | 4 +++- dm/gfx/nvn/nvn_fstep.h | 31 +++++++++++++++++++++++-------- dm/gfx/tests/t_fstep.c | 34 +++++++++++++++++++++++++++++----- 3 files changed, 55 insertions(+), 14 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index af48ef7b11..9e7e38ee0d 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2372,6 +2372,8 @@ static void fstep_tick(const win_t *o, int c) { uint64_t st = g_fs_start ? g_fs_start : now_ns(), pw = g_fs_wait; g_fs_start = g_fs_wait = 0; double shown = prev ? (st - prev) / v : 0, gpu = 0; prev = st; for (uint64_t f = g_frame; f-- > 0 && g_frame - f <= 8;) { double g = mtl_gpu_ms(f); if (g >= 0) { gpu = g * 1e6 / v; break; } } + static uint64_t pdraw; uint64_t npass, ndraw; mtl_stats(&npass, &ndraw); // the draws encoded since the last tick: what the GPU reference measured (nvn_fstep.h FSTEP_GPU_WORK) + double draws = pdraw && ndraw > pdraw ? (double)(ndraw - pdraw) : 0; pdraw = ndraw; double cpu = shown - pw / v, cost = cpu > gpu ? cpu : gpu; int kmin = fstep_hz() / c, kmax = fps_high() ? 2 : !fstep_hold() ? kmin : kmin >= 2 ? 2 : FSTEP_MAX, k0 = g_fs.k, // at 120 fps a late frame must step down whatever fps_hold says: the game steps by the target kmin0 = g_fs.kmin, probe0 = g_fs.probe; @@ -2380,7 +2382,7 @@ static void fstep_tick(const win_t *o, int c) { // A screen whose rate is not a multiple of 60 (144, 165 Hz) shows the link's 1/60 s ticks on their nearest refreshes (nvn_mtl.m link_range), so a frame on time can show for one // refresh more than its hold: a cheap 50 ms frame shows 54.5 ms at 165 Hz, which counted as late kept the hold at 20 fps for good int hz = front_screen_max_fps(); g_fs.shown_slack = hz > fstep_hz() && hz % fstep_hz() && mtl_link_active() ? (double)fstep_hz() / hz : 0; - int cslow = fstep_cpu_slowed(o->presents), gslow = nvn_fstep_gpu_slowed(&g_fg, gpu, k0 ? k0 : kmin, kmin, probe0) && dm_thermal_now() <= 1 && !dm_low_power_now(); // same gates as the CPU's + int cslow = fstep_cpu_slowed(o->presents), gslow = nvn_fstep_gpu_slowed(&g_fg, gpu, draws, k0 ? k0 : kmin, kmin, probe0) && dm_thermal_now() <= 1 && !dm_low_power_now(); // same gates as the CPU's // Present snapshots (nvn_mtl.m snap_take): the game runs up to a frame ahead of the screen, so a frame whose work runs long is absorbed instead of shown late, and its cost is // no evidence against the hold (Breath of the Wild in Zora's Domain stepped to 30 at a GPU time of 0.75 vsyncs: the frame's CPU time held its waits on its own GPU work). // Lateness is then what the screen shows: the on-screen interval of the frame 8 presents back against its hold (pres_expect_ms), late by a quarter or dropped diff --git a/dm/gfx/nvn/nvn_fstep.h b/dm/gfx/nvn/nvn_fstep.h index 404fc7fdf9..dacb3218db 100644 --- a/dm/gfx/nvn/nvn_fstep.h +++ b/dm/gfx/nvn/nvn_fstep.h @@ -161,18 +161,33 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km // FSTEP_GPU_SLOW times its time at kmin and it alone does not fit k − 1 (a GPU at 3 ms of 17, 6 times its time on a loading screen, is not what holds a frame back). // The reference is not updated during a probe's trial (its frames run while the clock is still rising), so a failed probe leaves it as it was. It is kept for each kmin: one // learned at a 30 fps cap is the GPU time at the clock that cap left (lowered for 30 fps, and under the thermal cap for the heat), and taken as hold 1's after the cap rose it -// made the same work at hold 2 look not slowed +// made the same work at hold 2 look not slowed. +// The comparison holds only for the same work: a clock that fell shows as the same draws taking longer. The draws a frame encodes are kept beside each reference (dref_at) and at +// the longer hold (dcur); when both are known, the GPU counts as slowed only while they are within FSTEP_GPU_WORK of each other. Ocarina of Time on an M4 Air ran its title and +// loading screen at hold 1 (GPU 0.2 vsyncs, 2-60 draws), then gameplay at hold 2 (1.4 vsyncs, 700-900 draws, 1.3 at hold 1): against the loading screen's reference the GPU +// read 6 times slowed, and the host probed hold 1 every few seconds (32 probes in 66 s, all failed), each failure's late frames stepping it on to hold 3 (20 fps) #define FSTEP_GPU_SLOW 1.3 // GPU time at the longer hold against its time at kmin #define FSTEP_GPU_N 30 // frames at the longer hold before the comparison counts -typedef struct { double ref, cur; int n, k; double ref_at[FSTEP_MAX + 1]; } nvn_fgpu; // GPU time at kmin (running mean; of every kmin in ref_at); at the current hold k and its frame count -static inline int nvn_fgpu_slow(const nvn_fgpu *g) { return g->n >= FSTEP_GPU_N && g->ref > 0 && g->cur >= FSTEP_GPU_SLOW * g->ref && g->cur > FSTEP_FIT * (g->k - 1); } -// Once per frame, with the frame's GPU time (vsyncs; 0 = none finished yet) and the hold and probe state (nvn_fstep.probe) it ran under -static inline int nvn_fstep_gpu_slowed(nvn_fgpu *g, double gpu, int k, int kmin, int probe) { +#define FSTEP_GPU_WORK 1.5 // draws at the longer hold against the reference's, either way, that still count as the same work (one scene's views vary by about 1.3) +typedef struct { double ref, cur; int n, k; double ref_at[FSTEP_MAX + 1]; double dref, dcur, dref_at[FSTEP_MAX + 1]; } nvn_fgpu; // GPU time at kmin (running mean; of every kmin + // in ref_at); at the current hold k and its frame count; the draws a frame encodes beside each (0 = not known) +static inline int nvn_fgpu_same_work(const nvn_fgpu *g) { return g->dref <= 0 || g->dcur <= 0 || (g->dcur <= FSTEP_GPU_WORK * g->dref && g->dref <= FSTEP_GPU_WORK * g->dcur); } +static inline int nvn_fgpu_slow(const nvn_fgpu *g) { return g->n >= FSTEP_GPU_N && g->ref > 0 && g->cur >= FSTEP_GPU_SLOW * g->ref && g->cur > FSTEP_FIT * (g->k - 1) && nvn_fgpu_same_work(g); } +// Once per frame, with the frame's GPU time (vsyncs; 0 = none finished yet), its draws (0 = not counted) and the hold and probe state (nvn_fstep.probe) it ran under +static inline int nvn_fstep_gpu_slowed(nvn_fgpu *g, double gpu, double draws, int k, int kmin, int probe) { if (kmin < 1 || kmin > FSTEP_MAX) return 0; // no reference slot for it (hle_nvn.c passes 1 or 2) - g->ref = g->ref_at[kmin]; + g->ref = g->ref_at[kmin]; g->dref = g->dref_at[kmin]; if (gpu <= 0) return nvn_fgpu_slow(g); - if (k <= kmin) { if (probe != 1) g->ref = g->ref_at[kmin] = g->ref > 0 ? g->ref + (gpu - g->ref) / 32 : gpu; g->n = 0; g->k = k; return 0; } + if (k <= kmin) { + if (probe != 1) { + g->ref = g->ref_at[kmin] = g->ref > 0 ? g->ref + (gpu - g->ref) / 32 : gpu; + if (draws > 0) g->dref = g->dref_at[kmin] = g->dref > 0 ? g->dref + (draws - g->dref) / 32 : draws; + } + g->n = 0; g->k = k; return 0; + } if (k != g->k) { g->k = k; g->n = 0; } - g->cur = g->n ? g->cur + (gpu - g->cur) / 8 : gpu; g->n++; + g->cur = g->n ? g->cur + (gpu - g->cur) / 8 : gpu; + if (draws > 0) g->dcur = g->n && g->dcur > 0 ? g->dcur + (draws - g->dcur) / 8 : draws; + g->n++; return nvn_fgpu_slow(g); } diff --git a/dm/gfx/tests/t_fstep.c b/dm/gfx/tests/t_fstep.c index eb42729014..634a426b6c 100644 --- a/dm/gfx/tests/t_fstep.c +++ b/dm/gfx/tests/t_fstep.c @@ -210,7 +210,7 @@ int main(void) { for (int i = 0; i < 2000; i++) { double w = i < 600 ? 0.86 : i < 603 ? 1.25 : m.s.k >= 2 ? 1.5 : at1++ < 20 ? 1.5 : 0.86; stepped |= i >= 603 && m.s.k == 2; - m.slow = nvn_fstep_gpu_slowed(&g, w, m.s.k ? m.s.k : 1, 1, m.s.probe); + m.slow = nvn_fstep_gpu_slowed(&g, w, 0, m.s.k ? m.s.k : 1, 1, m.s.probe); run(&m, 1, w, 1, 3, 0); } snprintf(b, sizeof b, "GPU clock follows the hold (BotW): one probe back to hold 1 that holds (k %d, probes %llu, failed %llu)", m.s.k, (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); @@ -220,19 +220,43 @@ int main(void) { } { // a heavier scene at the same clock: 0.86 at hold 1, then 1.12 (19 ms) for good, which is 1.3 times less than enough to call the GPU slowed sim m = {0}; nvn_fgpu g = {0}; - for (int i = 0; i < 2000; i++) { double w = i < 600 ? 0.86 : 1.12; m.slow = nvn_fstep_gpu_slowed(&g, w, m.s.k ? m.s.k : 1, 1, m.s.probe); run(&m, 1, w, 1, 3, 0); } + for (int i = 0; i < 2000; i++) { double w = i < 600 ? 0.86 : 1.12; m.slow = nvn_fstep_gpu_slowed(&g, w, 0, m.s.k ? m.s.k : 1, 1, m.s.probe); run(&m, 1, w, 1, 3, 0); } check(m.s.k == 2 && m.s.probes == 0, "heavier scene, GPU 0.86 → 1.12 at hold 2: not slowed, no probe"); } { // a loading screen: the GPU at 0.03, then lag on the CPU steps to 2, where the GPU takes 0.17 (5.7 times as much but 3 ms): the CPU holds the frame back, not the GPU sim m = {0}; nvn_fgpu g = {0}; int slowed = 0; - for (int i = 0; i < 1200; i++) { double gp = i < 600 || m.s.k == 1 ? 0.03 : 0.17, w = i < 600 ? 0.5 : 1.3; m.slow = nvn_fstep_gpu_slowed(&g, gp, m.s.k ? m.s.k : 1, 1, m.s.probe); slowed |= m.slow; run(&m, 1, w, 1, 3, 0); } + for (int i = 0; i < 1200; i++) { double gp = i < 600 || m.s.k == 1 ? 0.03 : 0.17, w = i < 600 ? 0.5 : 1.3; m.slow = nvn_fstep_gpu_slowed(&g, gp, 0, m.s.k ? m.s.k : 1, 1, m.s.probe); slowed |= m.slow; run(&m, 1, w, 1, 3, 0); } check(m.s.k == 2 && !slowed && m.s.probes == 0, "GPU 0.03 → 0.17 under CPU lag: a GPU at 3 ms is not called slowed, no probe"); } + { // Ocarina of Time on an M4 Air: the title and the loading screen run at hold 1 for 20 s (GPU 0.2 vsyncs, 30 draws); gameplay has 900 draws, 1.3 vsyncs at hold 1 and 1.4 at + // hold 2, where the clock is lower. The hold-1 reference measured other work: the GPU at hold 2 is not slowed, and no probe goes back to hold 1, which cannot hold + for (int known = 1; known >= 0; known--) { + sim m = {0}; nvn_fgpu g = {0}; int slowed = 0; + for (int i = 0; i < 3000; i++) { + int play = i >= 1200, k = m.s.k ? m.s.k : 1; double w = !play ? 0.2 : k >= 2 ? 1.4 : 1.3; + m.slow = nvn_fstep_gpu_slowed(&g, w, known ? (play ? 900 : 30) : 0, k, 1, m.s.probe); slowed |= play && m.slow; + run(&m, 1, play ? w : 0.5, 1, 3, 0); + } + if (known) { snprintf(b, sizeof b, "GPU reference from a loading screen (30 draws), gameplay at 900: not slowed, no probe (k %d, probes %llu, failed %llu)", m.s.k, (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(m.s.k == 2 && !slowed && m.s.probes == 0, b); } + else { snprintf(b, sizeof b, " without draw counts the same run is called slowed and probes (probes %llu, failed %llu)", (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(slowed && m.s.probes > 0, b); } + } + sim m = {0}; nvn_fgpu g = {0}; int at1 = 0, stepped = 0; // the same work at both holds is still a clock that fell (BotW above, with its draw counts) + for (int i = 0; i < 2000; i++) { + double w = i < 600 ? 0.86 : i < 603 ? 1.25 : m.s.k >= 2 ? 1.5 : at1++ < 20 ? 1.5 : 0.86; + stepped |= i >= 603 && m.s.k == 2; + m.slow = nvn_fstep_gpu_slowed(&g, w, i % 3 ? 4900 : 4700, m.s.k ? m.s.k : 1, 1, m.s.probe); + run(&m, 1, w, 1, 3, 0); + } + snprintf(b, sizeof b, " the same draws at both holds (4,700-4,900): one probe back to hold 1 that holds (k %d, probes %llu)", m.s.k, (unsigned long long)m.s.probes); + check(stepped && m.s.k == 1 && m.s.probes == 1 && m.s.failed == 0, b); + } { // the GPU stays slow at hold 1 too: each probe fails, the reference stays the hold-1 time from before, and the probes spread out sim m = {0}; nvn_fgpu g = {0}; double last = 0, gaps[4]; int ng = 0; uint64_t probes = 0; for (int i = 0; i < 6000 && ng < 4; i++) { double w = i < 600 ? 0.86 : i < 603 ? 1.25 : 1.5; - m.slow = nvn_fstep_gpu_slowed(&g, w, m.s.k ? m.s.k : 1, 1, m.s.probe); + m.slow = nvn_fstep_gpu_slowed(&g, w, 0, m.s.k ? m.s.k : 1, 1, m.s.probe); run(&m, 1, w, 1, 3, 0); if (m.s.probes != probes) { probes = m.s.probes; gaps[ng++] = m.real - last; last = m.real; } } @@ -328,7 +352,7 @@ int main(void) { int kmin = i >= 600 && i < rise ? 2 : 1, kmax = kmin == 2 ? 2 : 3, k = m.s.k ? m.s.k : kmin; since1 = k == 1 ? since1 + 1 : 0; double w = k >= 2 ? 1.5 : i < rise ? 0.86 : c == 2 ? 1.3 : since1 <= 10 || (c == 1 && i < rise + 60) ? 1.5 : 0.86; - m.slow = nvn_fstep_gpu_slowed(&g, w, k, kmin, m.s.probe); + m.slow = nvn_fstep_gpu_slowed(&g, w, 0, k, kmin, m.s.probe); run(&m, 1, w, kmin, kmax, 0); if (i == rise - 1) ups = m.s.ups; if (i >= n - 600) at1 += m.s.k == 1; From fb9f13bec88258fc0c01d25c1ede401ed904b11e Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Sat, 7 Nov 2026 10:55:52 +0100 Subject: [PATCH 14/40] OoT5 A-4: a movie the player reads one picture per game frame follows its own clock, so Ocarina of Time's title movie no longer plays in slow motion under the frame step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner saw the scene behind the title in slow motion, as the game was before the frame step. That scene is a 60 fps movie (Event/Movie/Dm_Title_*.webm, AV1 through nn::codec), and the game's player reads one picture from the decoder each game frame, never skipping: right on the console, where the title runs at 60; under the host's frame step the title runs at 20-30 fps. Measured with a new DM_MOVIE_TRACE trace of nn::codec packets and picture reads (title, 100 s, M4 Air): one read per game frame, times 16-17 ms apart, movie speed 0.50 at a 30 cap and 0.36 at the 60 cap (the title at 21.8 fps). The decoder now answers for such a player (cv_due in hle_codec_video.c): when a released buffer takes a picture, held pictures behind the movie's clock are dropped and the one nearest the clock goes in. The clock is a picture's time when put in plus host time since; it starts again at the first picture, after 200 ms with none put in (paused), after a reset and when the stream's time goes back. A dropped picture signals the submission-ready event as a packet taken does. Only streams whose pictures step 4-100 ms read as nanoseconds are clocked: the unit is the client's, and any other is left as before. A player that keeps up drops nothing. The frame step's movie rule (hold 30 for 30 fps movies, raised by movie::Decoder only) is unchanged. Test first: t_codec_video's movie clock case (written failing: at 30 reads a second no picture was skipped): 60 reads/s lose none, 30 reads/s step two pictures each read (299 of 299) and show 10 s of movie in 10 s, 20 reads/s 15 s in 15 s, a µs-timed stream drops none, a 500 ms pause gives no catch-up. "nearest the clock" rather than "last not after it": on the stream's millisecond-rounded times the latter stepped 1 and 3 pictures by turns (199 skips). t_codec_video all passed; test_env_switches and test_docs OK. In the game (dmhost_main_mclock, title, 100 s each): speed 1.00 at the 30 cap (30.0 reads/s, steps 33 / 34 ms) and at the 60 cap. Not run on the other nn::codec games (Donkey Kong Bananza, Fire Emblem Fortune's Weave, Mario Tennis Fever): no regress.py on this branch (the owner's rule for it); their players, read at their movies' rate, drop nothing by the test's 60 reads/s case. Docs: docs/reference/movies.md entry, the game note (resolved, and Found on this game), DM_MOVIE_TRACE's line in docs/env_switches.tsv, symptoms index. Rebased onto main's decoder thread (cv_worker): the clock runs in fill(), where a released buffer takes a picture, on the guest's release and on the decoder's thread alike; fill returns the pictures it dropped, and they reach ready() as a packet taken (the release, and the packet's end through cv_out.dropped). A Stop / Flush restarts the clock with the epoch. t_codec_video's paced player waits for the decoder's thread (settle) before each read, as main's checks do, and its clock is atomic since that thread reads it; with cv_due disabled it fails again (skips). --- dm/host/hle_codec_video.c | 53 +++++++++++++++++++++++++++++------ dm/host/tests/t_codec_video.c | 38 ++++++++++++++++++++++++- docs/agents/symptoms.md | 1 + docs/env_switches.tsv | 2 +- docs/reference/movies.md | 8 ++++++ 5 files changed, 91 insertions(+), 11 deletions(-) diff --git a/dm/host/hle_codec_video.c b/dm/host/hle_codec_video.c index 47120c47e7..bf69cec7dd 100644 --- a/dm/host/hle_codec_video.c +++ b/dm/host/hle_codec_video.c @@ -60,6 +60,7 @@ typedef struct { uint32_t epoch; // counts Stop / Flush: the pictures of a packet taken before one are dropped struct { uint8_t *data; uint64_t size; int64_t ts; uint8_t eos; } q[CV_HOLD]; uint32_t qhead, qcount; // the packets taken and not yet decoded, in order (copies) cv_times st, all; // decode times of the current run of CV_STAT packets and of the runs before it + int64_t clk_ts, fill_ts; uint64_t clk_ns, fill_ns, dropped; // the movie clock (cv_due): a picture's time and the host time it went into a buffer; the last picture put in and when } cv_dec; static cv_dec *g_cv[16]; @@ -112,13 +113,41 @@ static void pool_native_write(cv_pool *p, uint32_t i, const vpx_image_t *im) { free(tmp); } static int cv_pending(const cv_dec *d) { return d->nheld + (int)d->qcount + d->busy; } // what the decoder holds for the client: packets not yet decoded and pictures no released buffer has taken (decoder lock held) +// The movie's clock for a player that takes one picture per game frame. Ocarina of Time's title player reads one picture each frame and never skips: on the console the title runs +// at 60, its movies' rate, but under the host's frame step it runs at 20-30 fps and the 60 fps movie played at a half to a third of its speed. The cause is the game's player, so +// the decoder answers for it: when a released buffer takes a picture, the held pictures behind the movie's clock are dropped and the one nearest it goes in, as a console that +// keeps the movie's rate would have shown it. The clock is the time of a picture put in plus the host time since; it starts again (no catch-up) at the first picture, after +// 200 ms with none put in (the player paused) and when the stream's time goes back (a new stream or a loop). A player that keeps up never loses a picture: the next one is +// always due one picture later. Times are the client's own unit: only a stream whose pictures step by 4-100 ms read as nanoseconds (OoT's WebM times, milliseconds in ns) is +// clocked; any other unit is left as it was. Called with the decoder locked; returns the pictures dropped +#define CV_CLK_GAP 200000000ull +static int cv_due(cv_dec *d) { + uint64_t now = now_ns(); int n = 0; + if (!d->clk_ns || now - d->fill_ns > CV_CLK_GAP || d->held_ts[0] < d->fill_ts) { d->clk_ts = d->held_ts[0]; d->clk_ns = now; } + else { + int64_t due = d->clk_ts + (int64_t)(now - d->clk_ns); + while (d->nheld > 1 && !d->held_eos[0] && d->held_ts[1] > d->held_ts[0] && d->held_ts[1] - d->held_ts[0] >= 4000000 && d->held_ts[1] - d->held_ts[0] <= 100000000 && + d->held_ts[1] <= due + (d->held_ts[1] - d->held_ts[0]) / 2) { // the picture nearest the clock: "the last one not after it" flips between one and three pictures on rounded times + vpx_img_free(d->held[0]); + memmove(d->held, d->held + 1, sizeof d->held[0] * (CV_HOLD - 1)); memmove(d->held_ts, d->held_ts + 1, sizeof d->held_ts[0] * (CV_HOLD - 1)); + memmove(d->held_eos, d->held_eos + 1, CV_HOLD - 1); d->nheld--; n++; + } + } + d->fill_ts = d->held_ts[0]; d->fill_ns = now; + if (n && !d->dropped) LOG("[codec] %s decoder %u: the player reads slower than the movie's rate: pictures behind the movie's clock are dropped", k_cv_name[d->codec], d->id); + d->dropped += n; + if (n && d->dropped % 600 < (uint64_t)n) LOG("[codec] %s decoder %u: %llu pictures dropped to the movie's clock, %llu put in", k_cv_name[d->codec], d->id, (unsigned long long)d->dropped, (unsigned long long)d->pictures); + return n; +} // Pictures and end of stream into the released buffers, in order (decoder lock held). decoded: the decoder's thread calls, with the pictures of the packet it has just decoded: -// a buffer they go into was waiting for them -static void fill(cv_dec *d, int decoded) { - int had = d->nheld; +// a buffer they go into was waiting for them. Returns the pictures dropped to the movie's clock (cv_due): they make room for a packet as one taken does, so a client that sends +// one packet per signal gets the signal (ready's took) +static int fill(cv_dec *d, int decoded) { + int had = d->nheld, dropped = 0; while (d->pcount && (d->nheld || d->eos_pending)) { uint32_t i = d->pend[d->phead]; d->phead = (d->phead + 1) % CV_BUFS; d->pcount--; typeof(d->buf[0]) *b = &d->buf[i]; + if (d->nheld) dropped += cv_due(d); if (d->nheld) { vpx_image_t *im = d->held[0]; int64_t ts = d->held_ts[0]; uint8_t eos = d->held_eos[0]; memmove(d->held, d->held + 1, sizeof d->held[0] * (CV_HOLD - 1)); memmove(d->held_ts, d->held_ts + 1, sizeof d->held_ts[0] * (CV_HOLD - 1)); @@ -133,6 +162,7 @@ static void fill(cv_dec *d, int decoded) { spt_fire(d, i); } if (had >= CV_HOLD && d->nheld < CV_HOLD) pthread_cond_signal(&d->work); // the decoder's thread waited for room for its next packet's pictures + return dropped; } // The event says a packet can come: it rises when one was taken (took) or room came back (before: cv_pending before the change), not at every release (decoder lock held). // Mario Tennis Fever's player counts each signal as one packet it may send: raised at every release, its count grew while a seek held back the samples (a skipped cutscene), @@ -355,7 +385,7 @@ static Result cv_dec_reset(void *t) { for (uint32_t k = 0; k < d->qcount; k++) free(d->q[(d->qhead + k) % CV_HOLD].data); d->qhead = d->qcount = 0; for (int i = 0; i < d->nheld; i++) vpx_img_free(d->held[i]); - d->nheld = 0; d->eos_pending = 0; + d->nheld = 0; d->eos_pending = 0; d->clk_ns = 0; // the movie clock starts again at the next picture (cv_due) d->epoch++; d->reopen = 1; cv_stat_sum(d); pthread_cond_signal(&d->work); // it may have waited for room @@ -393,7 +423,7 @@ static vpx_image_t *av1_to_i420(const mav1_picture *p) { // the decoder's NV12 return c; } // A packet's pictures on their way from the codec to the pictures waiting (held). The packet's last one is kept back until the packet ends, as it carries an end of stream -typedef struct { cv_dec *d; uint32_t epoch; vpx_image_t *last; int64_t last_ts; int stop; uint64_t waited; } cv_out; // stop: a Stop / Flush or Finalize came; waited: time spent waiting for room +typedef struct { cv_dec *d; uint32_t epoch; vpx_image_t *last; int64_t last_ts; int stop; uint64_t waited; int dropped; } cv_out; // stop: a Stop / Flush or Finalize came; waited: time spent waiting for room; dropped: pictures dropped to the movie's clock (fill) // Room for one more waiting picture (decoder lock held, on the decoder's thread): a release makes some (fill). 0 when a Stop / Flush or Finalize came instead static int cv_room(cv_dec *d, cv_out *o) { if (d->nheld >= CV_HOLD && !d->quit && o->epoch == d->epoch) { @@ -413,7 +443,7 @@ static void cv_put(cv_out *o, vpx_image_t *c, int64_t ts) { if (o->last) { cv_dec *d = o->d; pthread_mutex_lock(&d->mu); - if (cv_room(d, o)) { cv_hold(d, o->last, o->last_ts, 0); fill(d, 1); } else { vpx_img_free(o->last); o->stop = 1; } + if (cv_room(d, o)) { cv_hold(d, o->last, o->last_ts, 0); o->dropped += fill(d, 1); } else { vpx_img_free(o->last); o->stop = 1; } pthread_mutex_unlock(&d->mu); } o->last = c; o->last_ts = ts; @@ -471,9 +501,9 @@ static void *cv_worker(void *arg) { if (d->quit || o.epoch != d->epoch) { if (o.last) vpx_img_free(o.last); if (d->quit) break; } // Finalize, or a Stop / Flush since the packet was taken: its pictures are dropped else { if (o.last) cv_hold(d, o.last, o.last_ts, p.eos); else if (p.eos) d->eos_pending = 1; - fill(d, 1); + o.dropped += fill(d, 1); } - ready(d, before, 0); + ready(d, before, o.dropped); cv_stat(d, dt); } pthread_mutex_unlock(&d->mu); @@ -483,8 +513,12 @@ static void *cv_worker(void *arg) { // A refusal (CV_BUSY, nn::codec 3603) is the decoder's defined answer, not a fault: NintendoSDK_movie's detail::SubmitDecoderInputBuffer turns exactly that result into its // own retry code 0x7a, which VideoDecoder::DecodeAsync hands back to the player, and aborts on any other failure (Ocarina of Time's subsdk0 +0x10fd20, +0x103840). So it is // logged plainly +// DM_MOVIE_TRACE (as hle_movie.c): each packet the client submits and each picture's properties it reads, with the picture's time and the host clock, to tell a player that follows its +// movie's clock from one that takes a picture per game frame +static int cv_trace(void) { static int on = -1; if (on < 0) on = getenv("DM_MOVIE_TRACE") != NULL; return on; } static Result cv_dec_submit(uint64_t *out, void *t, const void *data, uint64_t size, int64_t ts, bool eos) { cv_dec *d = cv_of(t); if (!d) return CV_R(2004); + if (cv_trace()) LOG("[codec] %s submit time %lld size %llu%s at %.3f ms", k_cv_name[d->codec], (long long)ts, (unsigned long long)size, eos ? " eos" : "", now_ns() / 1e6); dm_host_mutex_lock(&d->mu); uint8_t *copy = NULL; if (cv_pending(d) >= CV_HOLD || (size && !(copy = malloc((size_t)size)))) { @@ -517,7 +551,7 @@ static void cv_dec_rel(void *t, uint32_t i, cv_sync sync) { if (d->pcount < CV_BUFS) { d->pend[(d->phead + d->pcount) % CV_BUFS] = i; d->pcount++; } if (d->fcount < 8) { d->fifo[(d->fhead + d->fcount) % 8] = i; d->fcount++; } else LOG("[RED] nn::codec: acquirable queue full at release %u", i); int before = cv_pending(d); - fill(d, 0); ready(d, before, 0); + int dropped = fill(d, 0); ready(d, before, dropped); pthread_mutex_unlock(&d->mu); } static Result cv_dec_props(uint16_t *w, uint16_t *h, uint16_t *cw, uint16_t *ch, int64_t *ts, bool *eos, void *t, uint32_t i) { @@ -526,6 +560,7 @@ static Result cv_dec_props(uint16_t *w, uint16_t *h, uint16_t *cw, uint16_t *ch, int ok = d->buf[i].filled; if (ok) { *w = *cw = d->buf[i].w; *h = *ch = d->buf[i].h; *ts = d->buf[i].ts; *eos = d->buf[i].eos != 0; } pthread_mutex_unlock(&d->mu); + if (ok && cv_trace()) LOG("[codec] %s picture index %u time %lld read at %.3f ms", k_cv_name[d->codec], i, (long long)*ts, now_ns() / 1e6); return ok ? 0 : CV_NONE; } static Result cv_dec_encrypted(void) { static int once; if (!once++) LOG("[RED] nn::codec: an encrypted input buffer: protected streams are not served"); return CV_R(2004); } diff --git a/dm/host/tests/t_codec_video.c b/dm/host/tests/t_codec_video.c index 7c581d90cb..4b3552ee97 100644 --- a/dm/host/tests/t_codec_video.c +++ b/dm/host/tests/t_codec_video.c @@ -13,7 +13,8 @@ #include <assert.h> #include <unistd.h> -uint64_t now_ns(void) { return 1; } +static _Atomic uint64_t g_clock = 1; // the host clock the decoder sees (paced(): pictures read at a game's frame rate); the decoder's thread reads it too +uint64_t now_ns(void) { return g_clock; } __thread _Atomic int dm_host_work; // hle_thread.c void dm_host_helper_qos(qos_class_t q) { (void)q; } // hle_thread.c: the decoder's thread keeps the default class in the unit test static struct { void *ev; int bound, set; pthread_t by; } g_ev[16]; // by: the thread that signalled it last @@ -59,7 +60,42 @@ static const uint8_t vp9[] = { 0xff,0xff,0xff,0xf2,0x2a,0xc0,0x00, }; +// A player that reads one picture per game frame (Ocarina of Time's title movie, 60 fps, through the AV1 decoder): with two buffers, packets topped up as the decoder takes them, +// one picture read every read_ns of host clock; times step 16 / 17 ms (the stream's milliseconds, in ns) or, with unit 1000, the same in µs. Returns the movie time the last +// of n reads showed and counts the reads whose picture was not the one after the previous (skips) +static int64_t paced(uint64_t read_ns, int n, int64_t unit, uint64_t pause_at, int *skips) { + _Alignas(16) static uint8_t dec[0x400], pool[2][0x40], mem[2][16 * 16 * 4]; uint64_t size, align, id; uint32_t idx; cv_sync sync; uint16_t w, h, cw, ch; int64_t ts = 0, prev = -1; bool eos; + cv_pref pref = h_cv_pref(16, 16, 3, 0); assert(!h_cv_poolsize(&size, &align, &pref, 1)); + for (int i = 0; i < 2; i++) assert(!h_cv_pool(pool[i], mem[i], size, &pref, 1, 0)); + assert(!h_cv_vp9init2(dec, 0x10000, 16, 16, 1, 1, 3, NULL, 0)); cv_dec *d = cv_of(dec); + for (uint32_t i = 0; i < 2; i++) { assert(!h_cv_vp9reg(dec, i, pool[i], 0)); assert(!h_cv_vp9acq(&idx, &sync, dec)); } + int sent = 0; *skips = 0; g_clock = 1000000000ull; + for (int r = 0; r < n; r++) { + for (;;) { // the decoder full: CV_HOLD pictures waiting, every packet decoded (settle), so the release chooses among them + pthread_mutex_lock(&d->mu); int full = cv_pending(d) >= CV_HOLD; pthread_mutex_unlock(&d->mu); if (full) break; + int64_t t = (sent * 50 + 1) / 3; assert(!h_cv_vp9submit(&id, dec, vp9, sizeof vp9, t * unit * 1000, false)); sent++; // t: 16 or 17 ms + } + settle(dec); + uint32_t b = (uint32_t)(r & 1); h_cv_vp9rel(dec, b, sync); assert(!h_cv_vp9acq(&idx, &sync, dec) && idx == b); + assert(!h_cv_vp9props(&w, &h, &cw, &ch, &ts, &eos, dec, b)); + if (prev >= 0 && ts - prev > 17 * unit * 1000) (*skips)++; + prev = ts; g_clock += read_ns; if (pause_at && r == (int)pause_at) g_clock += 500000000ull; + } + assert(!h_cv_vp9fin(dec)); for (int i = 0; i < 2; i++) h_cv_poolfin(pool[i]); + g_clock = 1; return ts; +} + int main(void) { + { // the movie follows its clock when the player reads slower than the movie's rate, and loses nothing when it keeps up + int skips; int64_t t; + t = paced(16666667, 300, 1000, 0, &skips); assert(skips == 0); assert(t >= 4900000000ll && t <= 5000000000ll); // 60 reads a second: every picture, 5 s of movie in 5 s + t = paced(33333333, 300, 1000, 0, &skips); assert(skips >= 290); assert(t >= 9700000000ll && t <= 10000000000ll); // 30 a second: every other picture, 10 s of movie in 10 s + t = paced(50000000, 300, 1000, 0, &skips); assert(t >= 14500000000ll && t <= 15000000000ll); // 20 a second (hold 3): 15 s in 15 s + t = paced(33333333, 300, 1, 0, &skips); assert(skips == 0); // times in µs: the unit is not known for certain, nothing is dropped + t = paced(33333333, 300, 1000, 150, &skips); assert(t >= 9700000000ll && t <= 10000000000ll); // a 500 ms pause: the clock starts again from the picture after it, no catch-up + printf("t_codec_video: the movie clock passed\n"); + } + _Alignas(16) static uint8_t dec[0x400], pool[3][0x40], spt[0x80], mem[16 * 16 * 4]; uint64_t size, align, id; uint32_t idx; cv_sync sync; uint16_t w, h, cw, ch; int64_t ts; bool eos; diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index b12895b303..2bd89954ca 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -449,6 +449,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - a movie stutters in small regular jerks (a picture stays for two or three frames, then the movie jumps on) while the frame rate reads a steady 60 and the frame times are flat: the game's frames are on time, the movie's pictures are not. Worst where the host decodes in software. → [`nn::codec` packets decoded inside the submission](../reference/movies.md#nncodec-packets-decoded-inside-the-submission) - on a chip without a hardware AV1 decoder (M1, M2), a 1080p 60 fps AV1 movie shows a picture twice now and then in its heavy parts, with the decoder already on its own thread, and the log has `[codec] AV1 decoder 1: the decoder is behind the movie: of the last 600 packets decoded in … ms each, … ms the longest; … pictures came after their buffer was released`. → [dav1d decoded one frame at a time](../reference/movies.md#dav1d-decoded-one-frame-at-a-time) - a crash in `nn::codec::video::InitializeAv1DecoderByTransferMemory` when a movie starts. → [AV1 movies through SDK 22's decoder](../reference/movies.md#av1-movies-through-sdk-22s-decoder) +- under the frame step a movie plays in slow motion while the game's world runs at the right speed: Ocarina of Time's title (the scene behind it is a 60 fps movie, `Event/Movie/Dm_Title_*.webm`) at about half speed at a 30 fps cap. `DM_MOVIE_TRACE=1` shows `[codec] … picture … time … read at` lines one per game frame with times 16–17 ms apart. → [A movie read one picture per game frame follows its own clock](../reference/movies.md#a-movie-read-one-picture-per-game-frame-follows-its-own-clock) - an abort in `nn::codec` (the SDK's own decoder client, which opens a service the host does not have) when a movie starts in a game built with SDK 19 for the Switch 2. → [SDK 19's decoder forms](../reference/movies.md#sdk-19s-decoder-forms) ## Audio diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 2168b73670..2b418b0f59 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -115,7 +115,7 @@ DM_MOTION_AXES switch `=<map>` such as `x,-z,y`: maps Apple's controller axes to DM_MOTION_GAIN switch `=<gain>`: how fast the orientation is pulled toward the measured gravity, in rad/s per unit error (default 0.5; `0` integrates the gyro alone) DM_MOTION_HID switch On the Mac, `0` disables the passive Switch Pro Controller HID motion fallback (`motion_pro_mac.h`); enabled by default when exactly one physical Pro and one GameController Pro can use the same player slot; usable GameController motion takes priority DM_MOTION_LOG diagnostic `=1` logs player 1's mapped motion sample once a second per source (phone / GameController / controller HID; acceleration in G, gyro in revolutions/s), for checking the axes: flat and still reads acc 0 0 -1 -DM_MOVIE_TRACE diagnostic When set, the movie decoder logs every call with its thread and decoder (by default the first 12 calls, then every 2^k-th) +DM_MOVIE_TRACE diagnostic When set, the movie decoder logs every call with its thread and decoder (by default the first 12 calls, then every 2^k-th), and the nn::codec video decoders each packet and picture read with its time and the host clock DM_MSL_DIR path Parent folder of the shader libraries: NVN loads from `b-<chain version>/` or the one file `b-<chain version>.img` beneath it, Vulkan from `vk/`, GL from `gl/`; the app and the Mac launcher set it DM_NDVE_ENGINE switch `0` runs nn::ndve's movie network (Ocarina of Time's title and opening movies) as the game's 12 translated dispatches after its colour conversion instead of the host's kernels (dm/gfx/msl/ndve_engine.metal, on by default: the same output, about half the frame's GPU time on an M4 Pro); `ab` alternates with DM_AB; `check` runs the game's dispatches and the host's network beside them and logs every 300 frames how many texels of the two differ. The host's kernels take only the model they were written for, bound by its shaders' keys and its weights' layout (docs/reference/shaders.md, The NDVE movie network by the host) DM_NICK switch The account nickname the game sees (default `Player`) diff --git a/docs/reference/movies.md b/docs/reference/movies.md index b3565e50ba..ea67cdbddd 100644 --- a/docs/reference/movies.md +++ b/docs/reference/movies.md @@ -353,6 +353,14 @@ A decoder hands out as many outputs as it has slots, and while every slot is tak - **Check:** `make -C dm/host/tests GAME=<id> run` runs `t_codec_video` (AddressSanitizer and UBSan), which covers the AV1 and SDK 22 forms, native images and the av1C record, and decodes a 64x64 AV1 key frame (SVT-AV1) on each backend the Mac has, read back as NV12. - **Found on:** [Fire Emblem Fortune's Weave](../../dm/games/04007EF01F8AE000/README.md): 82 movies in WebM at 2560x1440; the opening's 3,301 pictures play. On #957's branch, VideoToolbox and dav1d gave the same pictures, bit for bit, over the opening's first 600 (about 450 pictures a second each on an M3 Pro), and a tester's M1 Pro played the first two movies through dav1d, smooth and in sync ([seedds/nxnative#927](https://github.com/seedds/nxnative/issues/927)). +### A movie read one picture per game frame follows its own clock + +- **Symptom:** under the frame step a movie plays in slow motion while the game's world runs at the right speed: Ocarina of Time's title (the scene behind it is a 60 fps movie, `Event/Movie/Dm_Title_*.webm`) at about half speed at a 30 fps cap. `DM_MOVIE_TRACE=1` shows `[codec] … picture … time … read at` lines one per game frame with times 16–17 ms apart. +- **Cause:** the game's player reads one picture from the `nn::codec` decoder each game frame and never skips one. On the console its title runs at 60 fps, the movie's rate, so that is right there; the host's frame step runs it at 20–30 fps, and the movie advanced 16.7 ms per game frame: speed 0.50 at a 30 cap, 0.36 at the 60 cap where the title ran 21.8 fps (title runs of 100 s, M4 Air). The [frame step's movie rule](frame-rate.md#movies-hold-30-fps-under-the-frame-step) does not apply: it holds 30 fps for 30 fps movies, and only `movie::Decoder` raises its signal. +- **Fix:** the decoder answers for the player (`cv_due`, `hle_codec_video.c`): when a released buffer takes a picture, the held pictures behind the movie's clock are dropped and the one nearest it goes in. The clock is a picture's time when it went into a buffer plus the host time since; it starts again at the first picture, after 200 ms with no picture put in (the player paused), after a reset and when the stream's time goes back. A player that reads at the movie's rate drops nothing. A dropped picture signals the submission-ready event as a packet taken does ([the event counted as packets](#the-submission-ready-event-counted-as-packets)). Times are the client's unit: only a stream whose pictures step by 4–100 ms read as nanoseconds is clocked (OoT's are WebM milliseconds in nanoseconds); any other is left as before. The log has `the player reads slower than the movie's rate` once and the count dropped every 600. `DM_MOVIE_TRACE` also logs the `nn::codec` decoders' packets and picture reads. +- **Check:** `t_codec_video` (the movie clock case: two buffers, packets kept topped up, pictures read on a test clock): 60 reads a second lose no picture; 30 a second step two pictures every read and show 10 s of movie in 10 s; 20 a second 15 s in 15 s; a stream timed in µs drops nothing; a 500 ms pause starts the clock again with no catch-up. In the game (Ocarina of Time's title, 100 s, M4 Air): speed 1.00 at the 30 and 60 caps, 30.0 reads a second stepping 33 / 34 ms at 30. Not run on the other `nn::codec` games (Donkey Kong Bananza, Fire Emblem Fortune's Weave, Mario Tennis Fever), whose players were not seen reading slower than their movies. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#frame-rate) (v131072). + ### SDK 19's decoder forms - **Symptom:** an abort in `nn::codec` (the SDK's own decoder client, which opens a service the host does not have) when a movie starts in a game built with SDK 19 for the Switch 2. From c855a8ceb3db2f4c962f57ec1357ddca4bc4bbf4 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Mon, 9 Nov 2026 15:11:41 +0100 Subject: [PATCH 15/40] The frame step's GPU reference also matches compute dispatches, so Ocarina of Time's title no longer probes hold 1 against the boot screens' reference 787e00d5 counts the GPU as slowed only while the draws at the longer hold match the reference's. Draws do not see compute: on the M1 iPad the title spends 61 ms on its movie network's 14 dispatches with 20-80 draws, while the hold-1 reference came from the boot screens (0.4 ms, no dispatch). The title read as a GPU slowed 9.9 times and probed hold 1 every 3-6 s (4 in its first 350 presents), each probe about 33 frames at hold 1, none held. nvn_fgpu now keeps the dispatches a frame encodes (+1, so none is a count) beside the draws, and both must match; mtl_dispatches() gives the count (dispatch_enc). t_fstep 94 checks, 0 failed: the title case does not probe with dispatch counts and probes 17 times without; the same draws and dispatches at both holds still give BotW's one probe that holds. --- dm/gfx/nvn/hle_nvn.c | 4 +++- dm/gfx/nvn/nvn_fstep.h | 23 ++++++++++++++++------- dm/gfx/nvn/nvn_mtl.m | 5 +++-- dm/gfx/nvn/nvn_obj.h | 1 + dm/gfx/tests/t_fstep.c | 24 ++++++++++++++++++++++++ 5 files changed, 47 insertions(+), 10 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index 9e7e38ee0d..f5eaa5372e 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2374,6 +2374,8 @@ static void fstep_tick(const win_t *o, int c) { for (uint64_t f = g_frame; f-- > 0 && g_frame - f <= 8;) { double g = mtl_gpu_ms(f); if (g >= 0) { gpu = g * 1e6 / v; break; } } static uint64_t pdraw; uint64_t npass, ndraw; mtl_stats(&npass, &ndraw); // the draws encoded since the last tick: what the GPU reference measured (nvn_fstep.h FSTEP_GPU_WORK) double draws = pdraw && ndraw > pdraw ? (double)(ndraw - pdraw) : 0; pdraw = ndraw; + static uint64_t pdisp; static int pdisp_ok; uint64_t ndisp = mtl_dispatches(); // and the dispatches (none is a count too) + double disps = pdisp_ok ? (double)(ndisp - pdisp) : -1; pdisp = ndisp; pdisp_ok = 1; double cpu = shown - pw / v, cost = cpu > gpu ? cpu : gpu; int kmin = fstep_hz() / c, kmax = fps_high() ? 2 : !fstep_hold() ? kmin : kmin >= 2 ? 2 : FSTEP_MAX, k0 = g_fs.k, // at 120 fps a late frame must step down whatever fps_hold says: the game steps by the target kmin0 = g_fs.kmin, probe0 = g_fs.probe; @@ -2382,7 +2384,7 @@ static void fstep_tick(const win_t *o, int c) { // A screen whose rate is not a multiple of 60 (144, 165 Hz) shows the link's 1/60 s ticks on their nearest refreshes (nvn_mtl.m link_range), so a frame on time can show for one // refresh more than its hold: a cheap 50 ms frame shows 54.5 ms at 165 Hz, which counted as late kept the hold at 20 fps for good int hz = front_screen_max_fps(); g_fs.shown_slack = hz > fstep_hz() && hz % fstep_hz() && mtl_link_active() ? (double)fstep_hz() / hz : 0; - int cslow = fstep_cpu_slowed(o->presents), gslow = nvn_fstep_gpu_slowed(&g_fg, gpu, draws, k0 ? k0 : kmin, kmin, probe0) && dm_thermal_now() <= 1 && !dm_low_power_now(); // same gates as the CPU's + int cslow = fstep_cpu_slowed(o->presents), gslow = nvn_fstep_gpu_slowed2(&g_fg, gpu, draws, disps, k0 ? k0 : kmin, kmin, probe0) && dm_thermal_now() <= 1 && !dm_low_power_now(); // same gates as the CPU's // Present snapshots (nvn_mtl.m snap_take): the game runs up to a frame ahead of the screen, so a frame whose work runs long is absorbed instead of shown late, and its cost is // no evidence against the hold (Breath of the Wild in Zora's Domain stepped to 30 at a GPU time of 0.75 vsyncs: the frame's CPU time held its waits on its own GPU work). // Lateness is then what the screen shows: the on-screen interval of the frame 8 presents back against its hold (pres_expect_ms), late by a quarter or dropped diff --git a/dm/gfx/nvn/nvn_fstep.h b/dm/gfx/nvn/nvn_fstep.h index dacb3218db..892f657450 100644 --- a/dm/gfx/nvn/nvn_fstep.h +++ b/dm/gfx/nvn/nvn_fstep.h @@ -165,29 +165,38 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km // The comparison holds only for the same work: a clock that fell shows as the same draws taking longer. The draws a frame encodes are kept beside each reference (dref_at) and at // the longer hold (dcur); when both are known, the GPU counts as slowed only while they are within FSTEP_GPU_WORK of each other. Ocarina of Time on an M4 Air ran its title and // loading screen at hold 1 (GPU 0.2 vsyncs, 2-60 draws), then gameplay at hold 2 (1.4 vsyncs, 700-900 draws, 1.3 at hold 1): against the loading screen's reference the GPU -// read 6 times slowed, and the host probed hold 1 every few seconds (32 probes in 66 s, all failed), each failure's late frames stepping it on to hold 3 (20 fps) +// read 6 times slowed, and the host probed hold 1 every few seconds (32 probes in 66 s, all failed), each failure's late frames stepping it on to hold 3 (20 fps). +// Compute dispatches are work too, which draws do not see: Ocarina of Time's title spends 61 ms of an M1 iPad's GPU on its movie network's 14 dispatches with 20-80 draws, and its +// hold-1 reference came from the boot screens (0.4 ms, 22 draws, no dispatch), so the title read as a GPU slowed 9.9 times and probed hold 1 every 3-6 s, none held. The +// dispatches a frame encodes are kept the same way (cref_at, ccur; counted as dispatches + 1, so a frame with none is known), and both must match #define FSTEP_GPU_SLOW 1.3 // GPU time at the longer hold against its time at kmin #define FSTEP_GPU_N 30 // frames at the longer hold before the comparison counts #define FSTEP_GPU_WORK 1.5 // draws at the longer hold against the reference's, either way, that still count as the same work (one scene's views vary by about 1.3) -typedef struct { double ref, cur; int n, k; double ref_at[FSTEP_MAX + 1]; double dref, dcur, dref_at[FSTEP_MAX + 1]; } nvn_fgpu; // GPU time at kmin (running mean; of every kmin - // in ref_at); at the current hold k and its frame count; the draws a frame encodes beside each (0 = not known) -static inline int nvn_fgpu_same_work(const nvn_fgpu *g) { return g->dref <= 0 || g->dcur <= 0 || (g->dcur <= FSTEP_GPU_WORK * g->dref && g->dref <= FSTEP_GPU_WORK * g->dcur); } +typedef struct { double ref, cur; int n, k; double ref_at[FSTEP_MAX + 1]; double dref, dcur, dref_at[FSTEP_MAX + 1]; double cref, ccur, cref_at[FSTEP_MAX + 1]; } nvn_fgpu; // GPU time at kmin + // (running mean; of every kmin in ref_at); at the current hold k and its frame count; the draws and the dispatches + 1 a frame encodes beside each (0 = not known) +static inline int nvn_fgpu_near(double a, double b) { return a <= 0 || b <= 0 || (b <= FSTEP_GPU_WORK * a && a <= FSTEP_GPU_WORK * b); } +static inline int nvn_fgpu_same_work(const nvn_fgpu *g) { return nvn_fgpu_near(g->dref, g->dcur) && nvn_fgpu_near(g->cref, g->ccur); } static inline int nvn_fgpu_slow(const nvn_fgpu *g) { return g->n >= FSTEP_GPU_N && g->ref > 0 && g->cur >= FSTEP_GPU_SLOW * g->ref && g->cur > FSTEP_FIT * (g->k - 1) && nvn_fgpu_same_work(g); } -// Once per frame, with the frame's GPU time (vsyncs; 0 = none finished yet), its draws (0 = not counted) and the hold and probe state (nvn_fstep.probe) it ran under -static inline int nvn_fstep_gpu_slowed(nvn_fgpu *g, double gpu, double draws, int k, int kmin, int probe) { +// Once per frame, with the frame's GPU time (vsyncs; 0 = none finished yet), its draws (0 = not counted), its dispatches (< 0 = not counted) and the hold and probe state +// (nvn_fstep.probe) it ran under +static inline int nvn_fstep_gpu_slowed2(nvn_fgpu *g, double gpu, double draws, double disps, int k, int kmin, int probe) { if (kmin < 1 || kmin > FSTEP_MAX) return 0; // no reference slot for it (hle_nvn.c passes 1 or 2) - g->ref = g->ref_at[kmin]; g->dref = g->dref_at[kmin]; + double cw = disps >= 0 ? disps + 1 : 0; + g->ref = g->ref_at[kmin]; g->dref = g->dref_at[kmin]; g->cref = g->cref_at[kmin]; if (gpu <= 0) return nvn_fgpu_slow(g); if (k <= kmin) { if (probe != 1) { g->ref = g->ref_at[kmin] = g->ref > 0 ? g->ref + (gpu - g->ref) / 32 : gpu; if (draws > 0) g->dref = g->dref_at[kmin] = g->dref > 0 ? g->dref + (draws - g->dref) / 32 : draws; + if (cw > 0) g->cref = g->cref_at[kmin] = g->cref > 0 ? g->cref + (cw - g->cref) / 32 : cw; } g->n = 0; g->k = k; return 0; } if (k != g->k) { g->k = k; g->n = 0; } g->cur = g->n ? g->cur + (gpu - g->cur) / 8 : gpu; if (draws > 0) g->dcur = g->n && g->dcur > 0 ? g->dcur + (draws - g->dcur) / 8 : draws; + if (cw > 0) g->ccur = g->n && g->ccur > 0 ? g->ccur + (cw - g->ccur) / 8 : cw; g->n++; return nvn_fgpu_slow(g); } +static inline int nvn_fstep_gpu_slowed(nvn_fgpu *g, double gpu, double draws, int k, int kmin, int probe) { return nvn_fstep_gpu_slowed2(g, gpu, draws, -1, k, kmin, probe); } // dispatches not counted diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 1942e8b467..43b01223e1 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -1737,7 +1737,7 @@ static inline void m4_end(id e) { static const void *g_gp_e; // the encoder g_gp_cur belongs to: m4_end also ends the upload / diagnostics / present encoders, which may end while a profiled pass is open (#531) static _Atomic int g_gp_busy; // a periodic profile's report has not been printed yet: the heap is still being read static uint64_t g_bw_load, g_bw_store, g_gp_bw_l0, g_gp_bw_s0; // the bandwidth ledger (defined with the pass statistics below) and its values at the last render pass end -static uint64_t g_stat_pass, g_stat_draw; static _Atomic uint64_t g_cur_frame; static uint32_t g_seg; static uint32_t g_gp_copies, g_gp_disp, g_gp_ubd; // copies / dispatches encoded (for the compute encoders' labels) +static uint64_t g_stat_pass, g_stat_draw, g_stat_disp; static _Atomic uint64_t g_cur_frame; static uint32_t g_seg; static uint32_t g_gp_copies, g_gp_disp, g_gp_ubd; // copies / dispatches encoded (for the compute encoders' labels) static int gp_active(void) { if (__builtin_expect(g_gp_f0 == -1, 1)) return 0; if (g_gp_f0 == -2) { const char *e = getenv("DM_NVN_GPUPROF"); @@ -14788,7 +14788,7 @@ static int dispatch_own(void) { return v; } static id<MTL4ComputeCommandEncoder> dispatch_enc(void) { - id<MTL4ComputeCommandEncoder> e = blit_open(); g_gp_disp++; + id<MTL4ComputeCommandEncoder> e = blit_open(); g_gp_disp++; g_stat_disp++; if (g_blit_ops) blit_barrier(); g_bw_lo[0] = 0; g_bw_hi[0] = ~0ull; g_bw_n = 1; g_br_n = 0; g_blit_ops = 1; // recorded as "everything written": the next copy or dispatch gets a barrier if (!g_blit_ct) { [e setArgumentTable:g_ct]; g_blit_ct = 1; } @@ -17336,6 +17336,7 @@ void mtl_present_wait(uint64_t tk) { // Wait until all queued presents have been processed (used by mtl_finish = nvnQueueFinish etc.): goes through mtl_present_wait, so there is still a fallback when the display link stops (real device: the game called QueueFinish once while the display link was stopped, // and this used to wait here without a timeout; nobody took the 3 queued frames and the game stalled; those two runs on Mac did not hit this call) void mtl_present_flush(void) { pthread_mutex_lock(&g_pq_mu); uint64_t h = g_pq_in; pthread_mutex_unlock(&g_pq_mu); mtl_present_wait(h); } +uint64_t mtl_dispatches(void) { return g_stat_disp; } // compute dispatches encoded so far, the game's and the host's (the frame step's work measure, nvn_fstep.h) void mtl_stats(uint64_t *passes, uint64_t *draws) { *passes = g_stat_pass; *draws = g_stat_draw; cp_report(); } void mtl_args_stats(uint64_t *enc, uint64_t *reuse, uint64_t *patch) { *enc = g_stat_args_enc; *reuse = g_stat_args_reuse; *patch = g_stat_args_patch; } void mtl_vbuf_stats(uint64_t *resolve, uint64_t *reuse, uint64_t *set, uint64_t *skip, uint64_t *copies, uint64_t *copy_bytes) { diff --git a/dm/gfx/nvn/nvn_obj.h b/dm/gfx/nvn/nvn_obj.h index b3bcc47479..10aa593b4a 100644 --- a/dm/gfx/nvn/nvn_obj.h +++ b/dm/gfx/nvn/nvn_obj.h @@ -295,6 +295,7 @@ void mtl_sub_index(const uint64_t *h, int n); // Start of submission: build th void xv_test_counts(int *dc, int *viol); int mtl_gpu_span(uint64_t f, uint64_t *t0, uint64_t *t1); // The frame's earliest GPUStart / latest GPUEnd (mach absolute time, ns); returns 0 if not all complete or there is no GPU time void mtl_stats(uint64_t *passes, uint64_t *draws); +uint64_t mtl_dispatches(void); void mtl_args_stats(uint64_t *enc, uint64_t *reuse, uint64_t *patch); void mtl_use_stats(uint64_t *call, uint64_t *skip); uint64_t mtl_ds_omit(void); // Cumulative number of passes with no depth attachment because the lookahead proved depth-stencil unused diff --git a/dm/gfx/tests/t_fstep.c b/dm/gfx/tests/t_fstep.c index 634a426b6c..d161c435e0 100644 --- a/dm/gfx/tests/t_fstep.c +++ b/dm/gfx/tests/t_fstep.c @@ -242,6 +242,30 @@ int main(void) { else { snprintf(b, sizeof b, " without draw counts the same run is called slowed and probes (probes %llu, failed %llu)", (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); check(slowed && m.s.probes > 0, b); } } + { // Ocarina of Time's title on an M1 iPad: the boot screens set the hold-1 reference (GPU 0.03 vsyncs, 50 draws, no dispatch), then the title's movie network adds + // 14 dispatches and 3.6 vsyncs at the same draws. Counted dispatches tell the work apart: not slowed, no probe; without them it probes hold 1 again and again + for (int known = 1; known >= 0; known--) { + sim m = {0}; nvn_fgpu g = {0}; int slowed = 0; + for (int i = 0; i < 3000; i++) { + int title = i >= 300, k = m.s.k ? m.s.k : 1; double w = title ? 3.6 : 0.03; + m.slow = nvn_fstep_gpu_slowed2(&g, w, 50, known ? (title ? 14 : 0) : -1, k, 1, m.s.probe); slowed |= title && m.slow; + run(&m, 1, title ? w : 0.5, 1, 3, 0); + } + if (known) { snprintf(b, sizeof b, "title's dispatches against a boot-screen reference with none: not slowed, no probe (k %d, probes %llu)", m.s.k, (unsigned long long)m.s.probes); + check(m.s.k == 3 && !slowed && m.s.probes == 0, b); } + else { snprintf(b, sizeof b, " without dispatch counts the same run is called slowed and probes (probes %llu, failed %llu)", (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(slowed && m.s.probes > 0, b); } + } + sim m = {0}; nvn_fgpu g = {0}; int at1 = 0, stepped = 0; // the same draws and dispatches at both holds: still a clock that fell (BotW's run below, with 14 dispatches) + for (int i = 0; i < 2000; i++) { + double w = i < 600 ? 0.86 : i < 603 ? 1.25 : m.s.k >= 2 ? 1.5 : at1++ < 20 ? 1.5 : 0.86; + stepped |= i >= 603 && m.s.k == 2; + m.slow = nvn_fstep_gpu_slowed2(&g, w, i % 3 ? 4900 : 4700, 14, m.s.k ? m.s.k : 1, 1, m.s.probe); + run(&m, 1, w, 1, 3, 0); + } + snprintf(b, sizeof b, " the same draws and dispatches at both holds: one probe back to hold 1 that holds (k %d, probes %llu)", m.s.k, (unsigned long long)m.s.probes); + check(stepped && m.s.k == 1 && m.s.probes == 1 && m.s.failed == 0, b); + } sim m = {0}; nvn_fgpu g = {0}; int at1 = 0, stepped = 0; // the same work at both holds is still a clock that fell (BotW above, with its draw counts) for (int i = 0; i < 2000; i++) { double w = i < 600 ? 0.86 : i < 603 ? 1.25 : m.s.k >= 2 ? 1.5 : at1++ < 20 ? 1.5 : 0.86; From d845268c35f8a4cf007fe05ff09eed49fd15092a Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Mon, 9 Nov 2026 23:47:48 +0100 Subject: [PATCH 16/40] Docs: Ocarina of Time plays on an iPhone 17 (27-30 fps in Kokiri Forest), and a reference entry for the shader-code pools the raw pool table wired --- dm/games/0400486024DEC000/README.md | 6 ++++-- docs/agents/symptoms.md | 1 + docs/reference/memory.md | 10 +++++++++- 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 2c0bad41a7..be3c551eac 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -86,8 +86,9 @@ Measured on an M6 Mac with 16 GB, headless, on one walk with camera turns (`walk Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13,4`, 8 GB, iPadOS 27.0.1), docked, at 20 fps where its views are light. Built from upstream `main` 7ffa63cc with this branch, chain `dedc9391c30b`, the device's `env.txt` holding only `DM_LAUNCHER=0`, `DM_NVN_GPUPROF=every:600` and `DM_GUEST_FILE_MIN_FREE=2G` (none needed for play). Two 32-minute scripted runs from a save at the Great Deku Tree's entrance (camera turns and short steps, "Try Again" after a death). - **Memory.** iOS gives the app 7.97 GB of room against the game's 9.00 GB pool, so the guest goes into files by itself ([the guest in files](../../../docs/reference/memory.md#ios-ends-the-app-once-the-guests-touched-pages-reach-its-memory-limit)): footprint 2.2-2.3 GB, headroom to the memory limit 5.87 GB at its lowest, 4.4 GB of guest pages on disk. Metal's working set is 5,461 MB: [textures in GPU-only memory share bytes](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description) there by themselves, and the residency set holds 3.2 GB. With each render target in a recycled heap of its own, the set reached 5,184 MB at an outdoor save load and every command buffer failed out of memory. - **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. +- **iPhone** (current, 2026-11-09): plays on an iPhone 17 (`iPhone18,3`, 8 GB, A19, iOS 26.6.1) from the owner's save in Kokiri Forest, with the launcher's 30 fps cap and Dynamic resolution on. Built from upstream `main` 04811a63 with this branch and, for testing, PRs #1842, #1901, #1942 and #1751, chain `39eb2f781d07`. iOS gives the app 5.96 GB of room (the iPad 7.97), Metal's working set is 5,461 MB. Before [the pool table left shader-code pools out](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) it stuttered too much to play: the guest's file pages were read back all the time, 14-17 fps. Now 27 fps standing, 27-30 walking, frame p50 36.5 / p95 40 ms, GPU p50 32 ms, the guest file's page-ins at 100-400 every 10 s. Frames just over the cap's 33.3 ms are late: about 30 of 300 are dropped, a light stutter walking. The title, with its movie on its own clock, holds 30 fps. An earlier build without these fixes ended in a kernel panic after the save loaded (`wait for termination failed … AppleCentauriManager`). - **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. -- Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needs [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` is linked otherwise). +- Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needed [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` was linked otherwise); main has it since 9983d659. ## Screen shape @@ -140,7 +141,7 @@ From an empty save: at about 45 s `A` (title), `A` (New Game), then the movie. F - The "Navi" name label in the Great Deku Tree scene was missing on an earlier chain and has not been checked since. Not played yet: the Deku Tree dungeon, combat, later areas. Sound opens with no red line; movies and speech were not listened to. Physics issues reported in PR 1299 were not seen on this route (walking, the ladder, the camera). - Resolved: dotted horizontal rows across the picture in play, in fog and light (GitHub issue 1537): the agl_ndsr_vr networks' HMMA calls outside a chain overwrote an accumulator a chain held ([An HMMA outside a chain overwrote the chain's accumulator](../../../docs/reference/shaders.md#an-hmma-outside-a-chain-overwrote-the-chains-accumulator)), since chain 23a1ea1db092; found by bisecting chains on the Kokiri Forest autosave (0f510fd5's chain clean, bd9f422d's not). Under the game's upscaler they had also grown into white blotches: [LDSM reads rows other lanes stored](../../../docs/reference/shaders.md#ldsm-reads-rows-other-lanes-stored). - Open: GPU hangs in play ([#1326](https://github.com/seedds/nxnative/issues/1326), [#1605](https://github.com/seedds/nxnative/issues/1605)), repeatable in a Kokiri Forest conversation and the Impa escort on an M2 Pro and gone with the pipelines of task `a5288d0a6c4971fd` dropped ([#2011](https://github.com/seedds/nxnative/issues/2011)). They are not [threads returning before a barrier](../../../docs/reference/shaders.md#compute-threads-that-return-before-a-barrier): that task, the culling tasks `47f9b38490dcf9a2` and `99280b7b8aae78df` and mesh `3439aaa071d868ae` have no barrier in their bodies (none of the game's 520 task and mesh shaders has one). The lead in #2011 is pixel shader `bd09ab5fc4fd7680`'s light loop, which runs `min(count, 1000) + 1` times per pixel with per-tile counts the GPU reads. -- Not run on an iPhone yet; the M1 iPad: [For an iPhone or iPad build](#for-an-iphone-or-ipad-build). +- On an iPhone 17 and an M1 iPad: [For an iPhone or iPad build](#for-an-iphone-or-ipad-build). Open on the iPhone: frames a little over the 30 fps cap's 33.3 ms (GPU p50 32 ms in Kokiri Forest) make a light stutter. ## Unused border corrections @@ -239,6 +240,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [Shaders report which textures they read through a pool](../../../docs/reference/memory.md#shaders-report-which-textures-they-read-through-a-pool) - [Repeated GPU out-of-memory failures end the game](../../../docs/reference/memory.md#repeated-gpu-out-of-memory-failures-end-the-game) - [Pools that hold shader code made resident to key the shaders](../../../docs/reference/memory.md#pools-that-hold-shader-code-made-resident-to-key-the-shaders): first reported here (222 pools, 334 MB on a build with other changes); on main the raw pool table makes those pools resident anyway (4,276 resident pools before the fix, 4,277 after, on the village walk). +- [Shader-code pools wired by the raw pool table](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table): the iPhone 17 read its guest file back all the time in Kokiri Forest (14-17 fps); with the 223 shader-code pools (317 MB) out of the table, 27-30 fps. - [Guest files keep the blocks of memory the game gave back](../../../docs/reference/memory.md#guest-files-keep-the-blocks-of-memory-the-game-gave-back): the iPad stopped at 5.0 GB free with 6.6 GB of guest files. - [A GPU whose access macOS revoked stops the game](../../../docs/reference/rendering.md#a-gpu-whose-access-macos-revoked-stops-the-game): after a GPU hang or page faults (issue #1326), nothing was drawn for about 70 s before the game aborted, and in full screen WindowServer's watchdog killed it. - [A texture ID with no texture kept its cached view](../../../docs/reference/rendering.md#a-texture-id-with-no-texture-kept-its-cached-view): the page faults at f3,677 and f4,851 with the render-target reclaim on. diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 2bd89954ca..aa3337a0eb 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -191,6 +191,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - a Switch 2 game goes past Metal's recommended working set while its first 3D scene loads: `Insufficient Memory` command buffer errors, then a hung Mac. Ocarina of Time on a 16 GB Mac: 12 GB and more wired, once a kernel panic in `IOGPUGroupMemory.cpp` at 14.7 GB wired. → [Memory pools nothing gave an address to stay out of the raw pool table](../reference/memory.md#memory-pools-nothing-gave-an-address-to-stay-out-of-the-raw-pool-table) - a game whose shaders read texture handles from memory wires far past the working set while a scene loads, with `Insufficient Memory` errors or a panic, and stutters when new textures stream in. With a limit on how many it makes, surfaces in shadow go near black instead. → [Shaders report which textures they read through a pool](../reference/memory.md#shaders-report-which-textures-they-read-through-a-pool) - memory pools that hold only shader code are in the residency set, though no shader reads them. In The Great Ace Attorney Chronicles, three shared buffers, about 5 MB: chunks of its 1,000 MB pool (`DM_CTL` `restop`: shared buffers 241 MB at 30 s, 236 MB with the fix). → [Pools that hold shader code made resident to key the shaders](../reference/memory.md#pools-that-hold-shader-code-made-resident-to-key-the-shaders) +- on an 8 GB iPhone a game stutters badly in play although its GPU time fits: the `[mem]` lines show the process reading its guest file back all the time (Ocarina of Time in Kokiri Forest on an iPhone 17: 6,000-15,600 page-ins and 0.2-0.4 GB every 10 s, 13.6 GB read in all), whole frames of 117 ms on average with a GPU time of 27 ms, 14-17 fps, and 122 of 300 frames dropped. The app's own footprint stays far under its limit (headroom 2.9 GB): the device as a whole is short of memory. → [Shader-code pools wired by the raw pool table](../reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) - on a small device the residency set grows past Metal's working set and every command buffer fails with `kIOGPUCommandBufferCallbackErrorOutOfMemory`, then `FATAL the GPU is out of memory`. On the M1 iPad (8 GB, working set 5,461 MB) Ocarina of Time died going outside from Link's house, the set at 5,147 MB, 2,780 MB of it textures; with the render targets' recycled heaps (rendering.md) it died at the outdoor save load, the set at 5,184 MB (544 heaps of 1,384 MB over 1,455 MB of textures). `res_top`'s line `GPU-made textures in the set by the pool that holds their storage` shows Metal bytes far above the `union` of their NVN storage. → [Textures in GPU-only memory took an allocation per description](../reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description) - the Mac grows unresponsive for hours, then panics, with the host still submitting work that fails. → [Repeated GPU out-of-memory failures end the game](../reference/memory.md#repeated-gpu-out-of-memory-failures-end-the-game) - a scene load logs `upload command buffer execution error: … Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)` right after a `large texture … format 0x32 … 296 MB` line, then `upload command buffer dropped`, then the game's own command buffers fail one after another, up to the 32-failures stop (Repeated GPU out-of-memory failures end the game), with most of the GPU's working set free. → [A clear pass on a large depth texture fails with `Insufficient Memory`](../reference/memory.md#a-clear-pass-on-a-large-depth-texture-fails-with-insufficient-memory) diff --git a/docs/reference/memory.md b/docs/reference/memory.md index a6c51ce060..6ca466c8a6 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -584,9 +584,17 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Symptom:** memory pools that hold only shader code are in the residency set, though no shader reads them. In The Great Ace Attorney Chronicles, three shared buffers, about 5 MB: chunks of its 1,000 MB pool (`DM_CTL` `restop`: shared buffers 241 MB at 30 s, 236 MB with the fix). - **Cause:** `keys_hash` (`hle_nvn.c`) checks that a shader's code lies in a pool before it hashes it, and it asked `mtl_addr`, which also makes the pool resident (and, in a pool in chunks, makes the chunk that holds the code). The GPU never reads that code: the shaders run translated, and a shader's c1 constants are copied into the host's own buffers. - **Fix:** general: `keys_hash` asks `mtl_addr_known` (`nvn_mtl.m`), the pool lookup with no side effect that texture uploads already used: it makes no pool resident, and in a pool in chunks it makes no chunk (it answers from the pool's range). A pool the GPU does read through a binding or an address is made resident there, as before. Keys are unchanged. `t_addr_known` checks it. -- **Where it does not help:** a game whose shaders read raw global memory has every pool it took an address in made resident by the raw pool table (`gmem_map`), shader code pools included. Ocarina of Time is one: on the village walk it kept 4,276 pools resident before the fix and 4,277 after, 2,324 on both at the title. The 222 pools and 334 MB first reported there came from a build with other changes. +- **Where it does not help:** a game whose shaders read raw global memory has every pool it took an address in made resident by the raw pool table (`gmem_map`), shader code pools included, until [the pool table left shader-code pools out](#shader-code-pools-wired-by-the-raw-pool-table). Ocarina of Time is one: on the village walk it kept 4,276 pools resident before the fix and 4,277 after, 2,324 on both at the title. The 222 pools and 334 MB first reported there came from a build with other changes. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) (reported), [The Great Ace Attorney Chronicles](../../dm/games/010036E00FB20000/README.md) (measured). + +### Shader-code pools wired by the raw pool table + +- **Symptom:** on an 8 GB iPhone a game stutters badly in play although its GPU time fits: the `[mem]` lines show the process reading its guest file back all the time (Ocarina of Time in Kokiri Forest on an iPhone 17: 6,000-15,600 page-ins and 0.2-0.4 GB every 10 s, 13.6 GB read in all), whole frames of 117 ms on average with a GPU time of 27 ms, 14-17 fps, and 122 of 300 frames dropped. The app's own footprint stays far under its limit (headroom 2.9 GB): the device as a whole is short of memory. +- **Cause:** the raw pool table (`gmem_map`) lists every pool something gave a GPU address in and makes each resident, which wires its pages. A game asks for each shader-code pool's address (`nvnMemoryPoolGetBufferAddress`) to place its programs, so all of them were wired, though the GPU never reads shader code here: the shaders run translated and their constants are copied into the host's buffers. On Ocarina of Time `res_top` counted 223 pools of flags 0x64 (`SHADER_CODE`, CPU and GPU cached), 317 MB, in the table only. On an 8 GB device those wired pages left too little room for the guest's own file pages, which the system evicted and the game read back. +- **Fix:** general (`hle_nvn.c` `nvnMemoryPoolGetBufferAddress`, `nvn_mtl.m` `mtl_pool_addressed_by`): a shader-code pool's own address (flag 0x40, not virtual) is reported as kind 4, which the table leaves out; the memory report counts those ranges (`shader-code ranges kept out of the table`). A buffer made in such a pool still puts the pool in (`nvnBufferInitialize`), so data the game keeps there stays reachable. `t_addr_known` checks it. +- **Result on Ocarina of Time,** iPhone 17, Kokiri Forest, the owner's play: page-ins 100-400 every 10 s (from 6,000-15,600), the guest file's dirty pages in memory 1.70 GB (from 2.02), resident pools 4,041 (from 4,261), 27 fps standing and 27-30 walking (from 14-17), frame p50 36.5 ms, GPU p50 32 ms, about 30 of 300 frames dropped under the launcher's 30 fps cap. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#for-an-iphone-or-ipad-build). ### Textures in GPU-only memory took an allocation per description - **Symptom:** on a small device the residency set grows past Metal's working set and every command buffer fails with `kIOGPUCommandBufferCallbackErrorOutOfMemory`, then `FATAL the GPU is out of memory`. On the M1 iPad (8 GB, working set 5,461 MB) Ocarina of Time died going outside from Link's house, the set at 5,147 MB, 2,780 MB of it textures; with the render targets' recycled heaps ([rendering.md](rendering.md#render-targets-that-move-with-the-dynamic-resolution-get-new-metal-memory-each-time)) it died at the outdoor save load, the set at 5,184 MB (544 heaps of 1,384 MB over 1,455 MB of textures). `res_top`'s line `GPU-made textures in the set by the pool that holds their storage` shows Metal bytes far above the `union` of their NVN storage. From 0e303a66e25a02ac3897cea41a1b7a6757ec92d3 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:18:38 +0100 Subject: [PATCH 17/40] iOS: a Rotation setting (Automatic, Landscape, Portrait) on the settings screen and in the in-game menu locks the orientation, and lets an iPhone be played upright DM_ROTATION, a launcher and in-game menu setting (dm/games/settings.tsv, platform:ios). The phone's Info.plist lists only the two landscapes, so an iPhone could never turn upright: the app delegate (main.m) now answers application:supportedInterfaceOrientationsForWindow: with front_ios_app_orientations (front_ios.m), which replaces that list. Under auto, the default, it returns the list itself (the phone's two landscapes, the iPad's four), so nothing changes for anyone who leaves it. landscape locks the two landscapes; portrait upright (and upside down where the device has it). The root view controller and the launcher's DmHostingController allow every orientation under auto and the locked ones otherwise (dm_rotation_vc_mask; the pure mask is DmLayout.h dm_rotation_mask). A pick calls setNeedsUpdateOfSupportedInterfaceOrientations on every view controller and, when the window's orientation is no longer allowed, requestGeometryUpdateWithPreferences, so the screen turns at once: on the settings screen as soon as it is picked (Launcher.swift onChange), in the in-game menu (dm_setting_live), and at the start for a saved pick, DM_LAUNCHER=0 or an env file. Upright, the settings screen puts the game's name and Play above the settings instead of in a third of the width. Duo harness: the fixture defines DM_GAME_SCREEN_ASPECT and the stubs add dm_screen_view_changed, dm_players_reported and front_web_touch, which front_ios.m and DmGamepad.m call since screen_aspect any and the controller applet's prompt: the harness did not compile or link without them. Its app delegate answers as main.m's. New test: the masks for each value, and the in-game menu offering Rotation. test_launcher_settings: DM_ROTATION joins the iOS-only settings (the test listed them; updated deliberately). Checked: the Duo harness on the iPhone 17 Simulator, 41 tests, 0 failures (1 skipped, iOS 27.1 only), including the rotation test of the next commit, which turns the Simulator's landscape-only iPhone upright; test_launcher_settings, test_orientation_policy, test_docs. Front-ends: new: DM_ROTATION (Rotation). An iOS host app with its own app delegate answers application:supportedInterfaceOrientationsForWindow: with front_ios_app_orientations() (DmRotation.h) for Portrait to turn a phone upright --- dm/games/settings.tsv | 1 + dm/ios/Sources/DmHostingController.swift | 4 +- dm/ios/Sources/DmIOS-Bridging-Header.h | 3 +- dm/ios/Sources/DmLayout.h | 9 ++ dm/ios/Sources/DmRotation.h | 14 +++ dm/ios/Sources/Launcher.swift | 98 +++++++++++-------- dm/ios/Sources/front_ios.m | 38 ++++++- dm/ios/Sources/main.m | 3 + dm/ios/tests/duo/README.md | 2 +- dm/ios/tests/duo/Sources/DuoHarnessApp.m | 5 + dm/ios/tests/duo/Sources/DuoHarnessRuntime.h | 1 + dm/ios/tests/duo/Sources/DuoRuntimeStubs.m | 5 + .../tests/duo/Sources/Fixtures/game_profile.h | 1 + dm/ios/tests/duo/Tests/DuoHarnessTests.m | 27 +++++ dm/tools/tests/test_launcher_settings.py | 2 +- docs/env_switches.tsv | 1 + docs/settings.md | 9 +- 17 files changed, 175 insertions(+), 48 deletions(-) create mode 100644 dm/ios/Sources/DmRotation.h diff --git a/dm/games/settings.tsv b/dm/games/settings.tsv index 10ceb8f17b..5e7689732c 100644 --- a/dm/games/settings.tsv +++ b/dm/games/settings.tsv @@ -23,6 +23,7 @@ DM_UPSCALER choice game=Game's own,metalfx=MetalFX game DM_GAME_UPSCALER_N!=0 Ga DM_LANG choice languages build:DM_LANG languages>1 Game Game language level=basic DM_FULLSCREEN choice 0=Window,1=Fullscreen 0 platform:mac Game Display mode level=basic DM_PICTURE_FIT choice fit=Fit,fill=Fill fit platform:ios Game Picture Fill crops the game's 16:9 picture to cover the screen. Health bars, subtitles or maps near the edges can be cut off. A full-screen picture without cropping needs a patch for each game. +DM_ROTATION choice auto=Automatic,landscape=Landscape,portrait=Portrait auto platform:ios Game Rotation DM_SHOW_FPS toggle - build:DM_SHOW_FPS platform:ios Game Frame-rate counter DM_TOUCH_HAPTICS toggle - 1 platform:ios Controls Touch feedback DM_RUMBLE toggle - 1 always Controls Game rumble level=basic diff --git a/dm/ios/Sources/DmHostingController.swift b/dm/ios/Sources/DmHostingController.swift index 14eee41774..36e4f969cb 100644 --- a/dm/ios/Sources/DmHostingController.swift +++ b/dm/ios/Sources/DmHostingController.swift @@ -1,7 +1,7 @@ import SwiftUI import UIKit -// Full-screen SwiftUI presentations must preserve the game's four-orientation policy. +// Full-screen SwiftUI presentations (the launcher) follow the game's orientation policy: every orientation, so the app's list decides, unless Rotation locks it (front_ios.m dm_rotation_vc_mask). final class DmHostingController<Content: View>: UIHostingController<Content> { - override var supportedInterfaceOrientations: UIInterfaceOrientationMask { .all } + override var supportedInterfaceOrientations: UIInterfaceOrientationMask { dm_rotation_vc_mask() } } diff --git a/dm/ios/Sources/DmIOS-Bridging-Header.h b/dm/ios/Sources/DmIOS-Bridging-Header.h index 2c4aad68d3..db7a7ae894 100644 --- a/dm/ios/Sources/DmIOS-Bridging-Header.h +++ b/dm/ios/Sources/DmIOS-Bridging-Header.h @@ -1,4 +1,5 @@ -// The C and Objective-C functions Launcher.swift and GameMenu.swift call (CloudSaves.h, DmGameMenu.h, save_sync.h's report before Play). project.yml points SWIFT_OBJC_BRIDGING_HEADER here. +// The C and Objective-C functions Launcher.swift, GameMenu.swift and DmHostingController.swift call (CloudSaves.h, DmGameMenu.h, DmRotation.h, save_sync.h's report before Play). project.yml points SWIFT_OBJC_BRIDGING_HEADER here. #import "CloudSaves.h" #import "DmGameMenu.h" +#import "DmRotation.h" #include "save_sync.h" diff --git a/dm/ios/Sources/DmLayout.h b/dm/ios/Sources/DmLayout.h index bb3631f749..8b1ac90ace 100644 --- a/dm/ios/Sources/DmLayout.h +++ b/dm/ios/Sources/DmLayout.h @@ -7,6 +7,15 @@ NS_ASSUME_NONNULL_BEGIN typedef struct { CGRect game, controls; BOOL separated, topSafeOnly; } DmLayout; +// Rotation (DM_ROTATION, a launcher and in-game menu setting): auto, landscape or portrait. auto gives `automatic`: every orientation for the view controllers, so the app's own list +// decides, and for the app (front_ios_app_orientations) its Info.plist list for the device: the phone's two landscapes, the iPad's four. landscape locks the two landscapes, portrait +// upright (upside down too where the device has it: an iPad, not a Face ID iPhone) +static inline UIInterfaceOrientationMask dm_rotation_mask(const char *_Nullable value, UIInterfaceOrientationMask automatic) { + if (value && !strcmp(value, "landscape")) return UIInterfaceOrientationMaskLandscape; + if (value && !strcmp(value, "portrait")) return UIInterfaceOrientationMaskPortrait | UIInterfaceOrientationMaskPortraitUpsideDown; + return automatic; +} + static inline UIEdgeInsets dm_game_safe_insets(CGRect bounds, UIEdgeInsets insets, BOOL duo) { return duo && bounds.size.height > bounds.size.width ? UIEdgeInsetsMake(insets.top, 0, 0, 0) : insets; } diff --git a/dm/ios/Sources/DmRotation.h b/dm/ios/Sources/DmRotation.h new file mode 100644 index 0000000000..c61c4330e6 --- /dev/null +++ b/dm/ios/Sources/DmRotation.h @@ -0,0 +1,14 @@ +// Rotation (DM_ROTATION, a launcher and in-game menu setting: auto, landscape, portrait; DmLayout.h dm_rotation_mask): front_ios.m keeps it, the root view controller, the launcher's +// hosting controller (DmHostingController.swift) and the app delegate (main.m) ask it which orientations they allow. +#pragma once +#import <UIKit/UIKit.h> + +#ifdef __cplusplus +extern "C" { +#endif +UIInterfaceOrientationMask dm_rotation_vc_mask(void); // a full-screen view controller's: every orientation under auto (the app's list decides) +UIInterfaceOrientationMask front_ios_app_orientations(void); // the app's (application:supportedInterfaceOrientationsForWindow:): under auto its Info.plist list for this device +void dm_rotation_set(const char *_Nullable value); // main thread: DM_ROTATION = value (NULL keeps the environment's), and the windows turn to an orientation it allows +#ifdef __cplusplus +} +#endif diff --git a/dm/ios/Sources/Launcher.swift b/dm/ios/Sources/Launcher.swift index 0f9d8b6639..717a294df9 100644 --- a/dm/ios/Sources/Launcher.swift +++ b/dm/ios/Sources/Launcher.swift @@ -103,48 +103,19 @@ private struct LauncherView: View { var body: some View { GeometryReader { geometry in - HStack(spacing: 0) { - VStack(spacing: 16) { - if let icon = appIcon() { - Image(uiImage: icon).resizable().frame(width: 96, height: 96) - .clipShape(RoundedRectangle(cornerRadius: 22)) - .accessibilityHidden(true) - } - Text((info("CFBundleDisplayName") as? String) ?? "Game") - .font(.title.weight(.bold)).multilineTextAlignment(.center) - Text("Version \((info("CFBundleShortVersionString") as? String) ?? "")") - .font(.callout).foregroundStyle(.secondary) - playButton + // Landscape: the game's name and Play on the left third, the settings beside them; upright (Rotation, an iPad turned), the name and Play above the settings + if geometry.size.width >= geometry.size.height { + HStack(spacing: 0) { + header(icon: 96) + .padding(16) + .frame(width: geometry.size.width / 3, height: geometry.size.height) + settingsList.frame(width: geometry.size.width * 2 / 3) } - .padding(16) - .frame(width: geometry.size.width / 3, height: geometry.size.height) - List { - ForEach(m.sections, id: \.self) { section in - let rows = m.rows(in: section) - if !rows.isEmpty { - Section(section) { - ForEach(rows) { SettingRow(s: $0, m: m) } - } - } - } - Section { - Button("Reset to tested defaults") { m.reset() } - .disabled(!m.changed) - } footer: { - Text("Brings back all settings this game was tested with.") - } - CloudSavesSection() - Section { - HStack { - Text("Memory available") - Spacer() - Text(ByteCountFormatter.string(fromByteCount: Int64(memory), countStyle: .memory)).monospacedDigit().foregroundStyle(.secondary) - } - } + } else { + VStack(spacing: 0) { + header(icon: 64).padding(.horizontal, 16).padding(.vertical, 12) + settingsList } - .listStyle(.insetGrouped) - .contentMargins(.horizontal, 12, for: .scrollContent) - .frame(width: geometry.size.width * 2 / 3) } } .preferredColorScheme(.dark) @@ -164,6 +135,53 @@ private struct LauncherView: View { try? await Task.sleep(for: .seconds(2)) } } + // Rotation turns the screen as soon as it is picked, before Play (front_ios.m dm_rotation_set); the saved pick at once when the screen shows + .onAppear { if let v = m.values["DM_ROTATION"] { dm_rotation_set(v) } } + .onChange(of: m.values["DM_ROTATION"]) { _, v in if let v { dm_rotation_set(v) } } + } + + private func header(icon size: CGFloat) -> some View { + VStack(spacing: size > 64 ? 16 : 8) { + if let icon = appIcon() { + Image(uiImage: icon).resizable().frame(width: size, height: size) + .clipShape(RoundedRectangle(cornerRadius: size * 22 / 96)) + .accessibilityHidden(true) + } + Text((info("CFBundleDisplayName") as? String) ?? "Game") + .font(.title.weight(.bold)).multilineTextAlignment(.center) + Text("Version \((info("CFBundleShortVersionString") as? String) ?? "")") + .font(.callout).foregroundStyle(.secondary) + playButton + } + } + + private var settingsList: some View { + List { + ForEach(m.sections, id: \.self) { section in + let rows = m.rows(in: section) + if !rows.isEmpty { + Section(section) { + ForEach(rows) { SettingRow(s: $0, m: m) } + } + } + } + Section { + Button("Reset to tested defaults") { m.reset() } + .disabled(!m.changed) + } footer: { + Text("Brings back all settings this game was tested with.") + } + CloudSavesSection() + Section { + HStack { + Text("Memory available") + Spacer() + Text(ByteCountFormatter.string(fromByteCount: Int64(memory), countStyle: .memory)).monospacedDigit().foregroundStyle(.secondary) + } + } + } + .listStyle(.insetGrouped) + .contentMargins(.horizontal, 12, for: .scrollContent) } @ViewBuilder var playButton: some View { diff --git a/dm/ios/Sources/front_ios.m b/dm/ios/Sources/front_ios.m index 5bfce204a1..098368c118 100644 --- a/dm/ios/Sources/front_ios.m +++ b/dm/ios/Sources/front_ios.m @@ -18,6 +18,7 @@ void dm_gamepad_active(void); #import "DmLayout.h" #import "DmGameMenu.h" +#import "DmRotation.h" void front_ios_boot(void); int front_ios_boot_ok(void); @@ -172,6 +173,7 @@ - (void)startGame { setenv("DM_SCREEN_ASPECT", v, 1); LOG("front end: picture shape auto: window %.0fx%.0f points (%.3f:1) → DM_SCREEN_ASPECT=%s", sz.width, sz.height, r, v); } } + dm_rotation_set(NULL); // the launcher's pick (Play put it in the environment), a saved one with DM_LAUNCHER=0, or an env file's self.fill = dm_picture_fill(getenv("DM_PICTURE_FIT")); if (self.fill) LOG("front end: picture Fill (DM_PICTURE_FIT=fill), the 16:9 picture covers the screen in landscape and its edges are cropped"); self.view.clipsToBounds = YES; // with Fill the game view is larger than the screen @@ -242,10 +244,43 @@ - (void)viewDidAppear:(BOOL)a { - (BOOL)prefersHomeIndicatorAutoHidden { return YES; } - (UIRectEdge)preferredScreenEdgesDeferringSystemGestures { return UIRectEdgeAll; } // Edge gestures need two swipes, to avoid accidental exits - (BOOL)prefersStatusBarHidden { return YES; } -- (UIInterfaceOrientationMask)supportedInterfaceOrientations { return UIInterfaceOrientationMaskAll; } +- (UIInterfaceOrientationMask)supportedInterfaceOrientations { return dm_rotation_vc_mask(); } @end UIViewController *front_ios_root(void) { return [DmRootVC new]; } +// Rotation (DM_ROTATION; DmRotation.h, DmLayout.h dm_rotation_mask). The phone's Info.plist lists only the two landscapes (the Duo's inner display follows its bounds) and the iPad's +// all four; the app delegate (main.m) answers with this, which replaces that list, so portrait can be chosen on a phone too while auto keeps each device's list +UIInterfaceOrientationMask dm_rotation_vc_mask(void) { return dm_rotation_mask(getenv("DM_ROTATION"), UIInterfaceOrientationMaskAll); } +static UIInterfaceOrientationMask plist_orientations(void) { + NSArray *list = nil; + if (UIDevice.currentDevice.userInterfaceIdiom == UIUserInterfaceIdiomPad) list = [NSBundle.mainBundle objectForInfoDictionaryKey:@"UISupportedInterfaceOrientations~ipad"]; + if (![list isKindOfClass:NSArray.class]) list = [NSBundle.mainBundle objectForInfoDictionaryKey:@"UISupportedInterfaceOrientations"]; + UIInterfaceOrientationMask m = 0; + if ([list isKindOfClass:NSArray.class]) for (id o in list) { + if ([o isEqual:@"UIInterfaceOrientationPortrait"]) m |= UIInterfaceOrientationMaskPortrait; + else if ([o isEqual:@"UIInterfaceOrientationPortraitUpsideDown"]) m |= UIInterfaceOrientationMaskPortraitUpsideDown; + else if ([o isEqual:@"UIInterfaceOrientationLandscapeLeft"]) m |= UIInterfaceOrientationMaskLandscapeLeft; + else if ([o isEqual:@"UIInterfaceOrientationLandscapeRight"]) m |= UIInterfaceOrientationMaskLandscapeRight; + } + return m ? m : UIInterfaceOrientationMaskAll; +} +UIInterfaceOrientationMask front_ios_app_orientations(void) { return dm_rotation_mask(getenv("DM_ROTATION"), plist_orientations()); } +void dm_rotation_set(const char *value) { + if (value) setenv("DM_ROTATION", value, 1); + const char *now = getenv("DM_ROTATION"); if (!now || !*now) now = "auto"; + static char was[16]; if (strcmp(was, now)) { snprintf(was, sizeof was, "%s", now); LOG("front end: rotation %s (DM_ROTATION)", now); } + UIInterfaceOrientationMask m = front_ios_app_orientations() & dm_rotation_vc_mask(); + for (UIScene *sc in UIApplication.sharedApplication.connectedScenes) { + if (![sc isKindOfClass:UIWindowScene.class] || ![sc.session.role isEqualToString:UIWindowSceneSessionRoleApplication]) continue; // not a TV's + UIWindowScene *ws = (UIWindowScene *)sc; + for (UIWindow *w in ws.windows) for (UIViewController *vc = w.rootViewController; vc; vc = vc.presentedViewController) [vc setNeedsUpdateOfSupportedInterfaceOrientations]; + // UIKit turns a window whose orientation the new mask leaves out only when asked: a lock chosen while upright turns the screen at once + if (m && !(m & (1u << ws.effectiveGeometry.interfaceOrientation))) + [ws requestGeometryUpdateWithPreferences:[[UIWindowSceneGeometryPreferencesIOS alloc] initWithInterfaceOrientations:m] errorHandler:^(NSError *e) { + LOG("front end: rotation %s: the window did not turn (%s)", now, e.localizedDescription.UTF8String); }]; + } +} + // A TV (cable or AirPlay): iOS gives the app the TV as a scene of its own instead of mirroring the phone. The game's view moves there, 16:9 fills the TV, and the game is // docked as a Switch on its dock (1080p, the operation mode change notification); the phone keeps the pad. Unplugging puts the view back and the mode the game had before @interface DmTVVC : UIViewController @@ -298,6 +333,7 @@ bool dm_setting_live(const char *env, const char *value) { dm_set_opmode(m); dm_gamepad_opmode_changed(); LOG("front end: console mode %s (in-game menu)", m ? "docked" : "handheld"); return true; } + if (!strcmp(env, "DM_ROTATION")) { if (value) dm_rotation_set(value); return true; } if (!strcmp(env, "DM_SHOW_FPS")) { if (value) [g_root showFPS:atoi(value) != 0]; return true; } if (!strcmp(env, "DM_TOUCH_HAPTICS")) { if (value) dm_gamepad_touch_feedback(atoi(value) != 0); return true; } if (!strcmp(env, "DM_RUMBLE")) { if (value) { dm_haptics_set_enabled(atoi(value) != 0); LOG("front end: game rumble %s (in-game menu)", atoi(value) ? "on" : "off"); } return true; } diff --git a/dm/ios/Sources/main.m b/dm/ios/Sources/main.m index 7cd198ec54..19c983ab9d 100644 --- a/dm/ios/Sources/main.m +++ b/dm/ios/Sources/main.m @@ -21,6 +21,7 @@ int dm_main(int argc, char **argv); // dm/host/main.c compiled in with -Dmain=dm_main UIViewController *front_ios_root(void); +UIInterfaceOrientationMask front_ios_app_orientations(void); // front_ios.m: the orientations Rotation allows (DM_ROTATION) void front_ios_focus_event(int ev); static char *g_argv[4]; static int g_argc; @@ -246,6 +247,8 @@ @implementation App - (UISceneConfiguration *)application:(UIApplication *)a configurationForConnectingSceneSession:(UISceneSession *)s options:(UISceneConnectionOptions *)o { return [[UISceneConfiguration alloc] initWithName:[s.role isEqualToString:UIWindowSceneSessionRoleExternalDisplayNonInteractive] ? @"TV" : @"Default" sessionRole:s.role]; // a TV (front_ios.m DmTVSceneDelegate) } +// Replaces the Info.plist orientation list: Rotation (DM_ROTATION) can turn a phone upright, auto keeps the list (front_ios.m front_ios_app_orientations) +- (UIInterfaceOrientationMask)application:(UIApplication *)a supportedInterfaceOrientationsForWindow:(UIWindow *)w { return front_ios_app_orientations(); } @end // Scene lifecycle: create the window + hand focus events to front_ios.m diff --git a/dm/ios/tests/duo/README.md b/dm/ios/tests/duo/README.md index 782fd74c68..606bdb10e9 100644 --- a/dm/ios/tests/duo/README.md +++ b/dm/ios/tests/duo/README.md @@ -2,7 +2,7 @@ This XCTest app compiles the production iOS gamepad, layout and hosting-controller code with inert host-runtime stubs. Its fixture `game_profile.h` supplies no game data, so the harness does not generate or require a game profile or extracted files. -The Simulator unit tests cover portrait and landscape layout, safe areas, valid and invalid division regions, 16:9 fitting and the Fill picture, touch hit ownership (the menu button against − / +), multitouch and video coordinates, pose-change input release, the in-game menu's layout pick and its live settings (picture, console mode, rumble, frame-rate counter) on the production root view controller, the drawable's pixel cap, controller-to-player slots, stale controller callbacks, and preservation of a seeded controller-motion sample across layout changes. The optional virtual-controller case runs when Simulator exposes one. They use synthetic touches, controller snapshots, hinge statuses and division callbacks; they do not exercise UIKit's own hit-testing of real touches, the menu's presentation or system touch cancellation. +The Simulator unit tests cover portrait and landscape layout, safe areas, valid and invalid division regions, 16:9 fitting and the Fill picture, touch hit ownership (the menu button against − / +), multitouch and video coordinates, pose-change input release, the in-game menu's layout pick and its live settings (picture, console mode, rumble, frame-rate counter) on the production root view controller, the drawable's pixel cap, the Rotation setting's orientation masks with a real turn of the Simulator's window (the harness's app delegate answers as the production one, main.m), the picture area's shape handed to the host, controller-to-player slots, stale controller callbacks, and preservation of a seeded controller-motion sample across layout changes. The optional virtual-controller case runs when Simulator exposes one. They use synthetic touches, controller snapshots, hinge statuses and division callbacks; they do not exercise UIKit's own hit-testing of real touches, the menu's presentation or system touch cancellation. The harness uses synthetic host state and does not validate an installed game, physical hinge sensors or device motion. diff --git a/dm/ios/tests/duo/Sources/DuoHarnessApp.m b/dm/ios/tests/duo/Sources/DuoHarnessApp.m index e9b168d263..bccd185d44 100644 --- a/dm/ios/tests/duo/Sources/DuoHarnessApp.m +++ b/dm/ios/tests/duo/Sources/DuoHarnessApp.m @@ -1,6 +1,7 @@ #import <UIKit/UIKit.h> #import "DmLayout.h" #import "DuoHarnessRuntime.h" +#import "DmRotation.h" #include <stdlib.h> #include <string.h> #if __has_include(<UIKit/UIHingeInteraction.h>) @@ -61,6 +62,10 @@ - (UISceneConfiguration *)application:(UIApplication *)application (void)application; (void)options; return [[UISceneConfiguration alloc] initWithName:@"Default" sessionRole:session.role]; } +// As the production app delegate (main.m): Rotation replaces the Info.plist list, so a phone can turn upright +- (UIInterfaceOrientationMask)application:(UIApplication *)application supportedInterfaceOrientationsForWindow:(UIWindow *)window { + (void)application; (void)window; return front_ios_app_orientations(); +} @end @implementation DuoHarnessSceneDelegate diff --git a/dm/ios/tests/duo/Sources/DuoHarnessRuntime.h b/dm/ios/tests/duo/Sources/DuoHarnessRuntime.h index 9d0d779c6f..f3def7278b 100644 --- a/dm/ios/tests/duo/Sources/DuoHarnessRuntime.h +++ b/dm/ios/tests/duo/Sources/DuoHarnessRuntime.h @@ -32,6 +32,7 @@ typedef struct { } DuoHarnessState; DuoHarnessInput duo_harness_input(void); +double duo_harness_view_shape(void); // the last shape front_ios.m handed the host (dm_screen_view_changed), 0 = none yet int duo_harness_rumble(void); // the in-game menu's rumble switch (dm_haptics_set_enabled) void dm_set_opmode(int mode); int dm_opmode(void); diff --git a/dm/ios/tests/duo/Sources/DuoRuntimeStubs.m b/dm/ios/tests/duo/Sources/DuoRuntimeStubs.m index 84efd5dfd9..64476115d6 100644 --- a/dm/ios/tests/duo/Sources/DuoRuntimeStubs.m +++ b/dm/ios/tests/duo/Sources/DuoRuntimeStubs.m @@ -94,6 +94,8 @@ void dm_motion_clear(int player) { } uint32_t dm_player_npad(int player) { return player == 0 ? 0x20 : (uint32_t)player; } +int dm_players_reported(void) { return 1; } // hle_hid.c: the players the game sees (the controller applet's prompt) +int front_web_touch(void) { return 0; } // web_apple.m: no offline page is shown static int g_opmode; // handheld, until a test or the in-game menu docks it int dm_opmode(void) { return g_opmode; } @@ -104,6 +106,9 @@ void dm_motion_clear(int player) { void dm_haptics_apple_init(void) {} void dm_haptics_apple_controller(int player, GCController *controller) { (void)player; (void)controller; } +static double g_view_shape; +void dm_screen_view_changed(double aspect) { g_view_shape = aspect; } // loader.c: the shape a screen_aspect any profile renders +double duo_harness_view_shape(void) { return g_view_shape; } int front_ios_boot_ok(void) { return 1; } void front_ios_boot(void) {} bool dm_show_launcher(void *viewController, void (*done)(void)) { (void)viewController; (void)done; return false; } diff --git a/dm/ios/tests/duo/Sources/Fixtures/game_profile.h b/dm/ios/tests/duo/Sources/Fixtures/game_profile.h index cf8eb6920c..71d9f0810d 100644 --- a/dm/ios/tests/duo/Sources/Fixtures/game_profile.h +++ b/dm/ios/tests/duo/Sources/Fixtures/game_profile.h @@ -2,3 +2,4 @@ // Synthetic defaults for the production virtual gamepad; no extracted game data. #define DM_GAME_PAD_LAYOUT 1 +#define DM_GAME_SCREEN_ASPECT 2 // screen_aspect any: the front end reports its picture area's shape (front_view_aspect, dm_screen_view_changed) diff --git a/dm/ios/tests/duo/Tests/DuoHarnessTests.m b/dm/ios/tests/duo/Tests/DuoHarnessTests.m index 641d8f50f2..e66a7cd96a 100644 --- a/dm/ios/tests/duo/Tests/DuoHarnessTests.m +++ b/dm/ios/tests/duo/Tests/DuoHarnessTests.m @@ -12,6 +12,7 @@ #import "DmLayout.h" #import "DmGameMenu.h" #import "DuoHarnessRuntime.h" +#import "DmRotation.h" #include "dm_input.h" #include <math.h> #include <stdio.h> @@ -1362,4 +1363,30 @@ - (void)testControllersKeepIndependentPlayersAndLayoutKeepsMotionState { } +// Rotation (DM_ROTATION): the view controllers' mask and the app's, which replaces the Info.plist list (main.m), so a phone can be locked upright +- (void)testRotationSettingSetsTheOrientationMasks { + const UIInterfaceOrientationMask all = UIInterfaceOrientationMaskAll, land = UIInterfaceOrientationMaskLandscape, + upright = UIInterfaceOrientationMaskPortrait | UIInterfaceOrientationMaskPortraitUpsideDown; + XCTAssertEqual(dm_rotation_mask(NULL, all), all); + XCTAssertEqual(dm_rotation_mask("auto", land), land, @"auto keeps the device's own list"); + XCTAssertEqual(dm_rotation_mask("landscape", all), land); + XCTAssertEqual(dm_rotation_mask("portrait", land), upright, @"portrait is allowed even where the Info.plist lists only landscape"); + XCTAssertEqual(dm_rotation_mask("sideways", land), land, @"an unknown value is auto"); + + UIViewController *root = [self harnessRootViewController]; + XCTAssertNotNil(root); + [root.view layoutIfNeeded]; + unsetenv("DM_ROTATION"); + BOOL pad = UIDevice.currentDevice.userInterfaceIdiom == UIUserInterfaceIdiomPad; + XCTAssertEqual(root.supportedInterfaceOrientations, all); + XCTAssertEqual(front_ios_app_orientations(), pad ? all : land, @"auto: the Info.plist list for this device"); + XCTAssertTrue(dm_setting_live("DM_ROTATION", NULL), @"the in-game menu offers Rotation"); + dm_setting_live("DM_ROTATION", "landscape"); + XCTAssertEqual(root.supportedInterfaceOrientations, land); + XCTAssertEqual(front_ios_app_orientations(), land); + dm_setting_live("DM_ROTATION", "auto"); + XCTAssertEqual(root.supportedInterfaceOrientations, all); + XCTAssertEqual(front_ios_app_orientations(), pad ? all : land); + unsetenv("DM_ROTATION"); +} @end diff --git a/dm/tools/tests/test_launcher_settings.py b/dm/tools/tests/test_launcher_settings.py index db5bfb597f..8d50cd974a 100644 --- a/dm/tools/tests/test_launcher_settings.py +++ b/dm/tools/tests/test_launcher_settings.py @@ -75,7 +75,7 @@ def test_hold_frames_takes_on_and_off(self): # the host reads DM_FPS_HOLD as o def test_iphone_switches_only_on_the_iphone(self): # the frame-rate counter and touch feedback are the iOS front end's (front_ios.m, DmGamepad.m) ios, mac = build(BOTW, 'nvn'), build(BOTW, 'nvn', 'mac') self.assertEqual(build(BOTW, 'nvn', 'ios').keys(), ios.keys()) # ios when unset - self.assertEqual(set(ios) - set(mac), {'DM_SHOW_FPS', 'DM_TOUCH_HAPTICS', 'DM_PICTURE_FIT'}) + self.assertEqual(set(ios) - set(mac), {'DM_SHOW_FPS', 'DM_TOUCH_HAPTICS', 'DM_PICTURE_FIT', 'DM_ROTATION'}) self.assertIn('DM_RUMBLE', mac); self.assertIn('DM_METAL_HUD', mac) self.assertEqual(set(mac) - set(ios), {'DM_FULLSCREEN', 'DM_SW2'}) # the Mac window's Display mode (front_mac.m reads 1 or 0); Switch 2 controllers (pad_switch2_apple.m, wired on the Mac only) s = mac['DM_FULLSCREEN'] diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 2b418b0f59..97535cc72b 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -338,6 +338,7 @@ DM_REWRITE_J switch `=<positive integer>`: worker processes per module for `patc DM_ROMFS path The game's RomFS folder, mounted read-only as `rom` (default `$DM_ROOT/romfs`); the Mac launcher, the iOS app and the NSP script set it DM_ROMFS_OVERLAY path `=<dir>[:<dir>…]`: RomFS reads look in these folders first, in order, and fall back to `DM_ROMFS` (for a modified copy of some files) DM_ROOT path Workspace root (default `~/dm-work`): the host's and tools' default paths (RomFS, temporary storage, shader libraries under `build/cache/metal_shader_cache`, probe dumps) are under it +DM_ROTATION player `landscape` locks the iPhone or iPad app to the two landscape orientations, `portrait` upright (the picture above the touch controls; a `screen_aspect any` profile renders the shape of that area); `auto` or unset turns as the device's Info.plist list allows (an iPhone's two landscapes, an iPad's four). Applied at once (`front_ios.m` `dm_rotation_set`). Launcher and in-game menu setting DM_RT diagnostic `0` leaves the thread that acquires the window on its QoS only, with no Mach time-constraint policy declaring the frame deadline (on by default; for power A/B) DM_RT_COMP switch `=<ms>` fixes the computation the present thread's time-constraint policy declares; default `auto` re-declares it every 120 frames from that thread's per-frame CPU time (p90 and peak) DM_RUMBLE player `0` turns game rumble off (on by default); launcher setting diff --git a/docs/settings.md b/docs/settings.md index e2e3db055b..9c6f4cd9d1 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -3,7 +3,7 @@ The screen before the game starts shows the settings the game supports. A setting that only matters while another is on appears under it and disappears when that one is turned off. Tap **Play** to start; the app remembers your choices for the next start. All settings are on one scrollable screen, under **Game**, **Controls** and **Performance**. Tap a menu or move a slider there; no settings page needs opening. **Upscaling** is in **Game**, beside **Console mode**. A Mac app shows the same settings in a [window of its own](#the-mac-settings-window). - [Tested defaults](#tested-defaults) -- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game's upscaler](#games-upscaler), [Game language](#game-language), [Picture](#picture), [Frame-rate counter](#frame-rate-counter) +- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game's upscaler](#games-upscaler), [Game language](#game-language), [Picture](#picture), [Rotation](#rotation), [Frame-rate counter](#frame-rate-counter) - [Controls](#controls): [Touch feedback](#touch-feedback), [Game rumble](#game-rumble), [Rumble strength](#rumble-strength), [Rumble cutoff](#rumble-cutoff), [Press a button to join](#press-a-button-to-join) - [Performance](#performance): [Frame buffering](#frame-buffering), [Metal 4 pipelines](#metal-4-pipelines), [Pipeline prebuild](#pipeline-prebuild), [Encode thread](#encode-thread), [Hold frames when late](#hold-frames-when-late), [Performance HUD](#performance-hud), [Reset to tested defaults](#reset-to-tested-defaults) - [The Mac settings window](#the-mac-settings-window): [Display mode](#display-mode) @@ -89,6 +89,10 @@ iPhone and iPad. **Fit**, the default, shows the game's picture (16:9, or the ta Games whose profile supports it (Breath of the Wild, Animal Crossing: New Horizons). **iPhone Duo screen (1.43:1)** renders a taller picture of the Duo's own shape: the sides of 16:9 with more above and below, and the HUD at the screen's edges. **16:9** is the Switch's picture. The taller picture has about a quarter more pixels, so it runs a little slower and warmer. +### Rotation + +iPhone and iPad. **Automatic**, the default, turns with the device as before: landscape either way on an iPhone, any way on an iPad. **Landscape** stays in landscape, either way up. **Portrait** stays upright; the picture sits above the touch controls (with a controller connected it takes the whole screen), and a game that renders any shape, such as Ocarina of Time, fills that space instead of showing a 16:9 picture with bars. The screen turns as soon as you pick it, on the settings screen or in the in-game menu. + ### Frame-rate counter Shows the frames per second at the top left. @@ -251,7 +255,7 @@ iCloud Drive takes a moment to carry a file from one device to the other. Quit t ## In-game menu -On the iPhone and iPad the gear at the top centre of the touch controls opens a menu while the game runs; it stays when a controller hides the other controls. It holds **Console mode**, **Picture** and the **Frame-rate counter**, then **Controller layout** (**2D**, **3D**, and **Tap**, no controls and every touch to the game, in handheld mode), **Touch feedback** and **Game rumble**: the settings that take effect at once. A choice opens to show its options, the current one ticked. The menu's text keeps one size whatever the system text size, so it stays short over the game. A change is remembered as on the settings screen, which shows it at the next start; a setting `Documents/env.txt` fixes is shown, not changed. The other settings need the game to start again, so they are only on the settings screen: close the app fully and open it again. +On the iPhone and iPad the gear at the top centre of the touch controls opens a menu while the game runs; it stays when a controller hides the other controls. It holds **Console mode**, **Picture**, **Rotation** and the **Frame-rate counter**, then **Controller layout** (**2D**, **3D**, and **Tap**, no controls and every touch to the game, in handheld mode), **Touch feedback** and **Game rumble**: the settings that take effect at once. A choice opens to show its options, the current one ticked. The menu's text keeps one size whatever the system text size, so it stays short over the game. A change is remembered as on the settings screen, which shows it at the next start; a setting `Documents/env.txt` fixes is shown, not changed. The other settings need the game to start again, so they are only on the settings screen: close the app fully and open it again. Console mode changes as on a Switch put in or taken out of its dock. While a TV shows the game it stays **Docked**, and the mode you pick applies when the TV is disconnected. **Docked** removes the Tap layout, as a docked Switch reads no touch screen. @@ -273,6 +277,7 @@ A line like `DM_PRESENT_SNAP=0` in `Documents/env.txt` (in the Files app, under | Cloud saves folder (Mac) | `DM_SAVE_SYNC` | a folder; `icloud`: iCloud Drive › nxnative Saves; `0`: off | | Picture (iPhone and iPad) | `DM_PICTURE_FIT` | `fit` 16:9 with black bars, `fill` covers the screen, cropped | | Picture shape | `DM_SCREEN_ASPECT` | Profiles with `screen_aspect on`: `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9. Profiles with `screen_aspect any` (Ocarina of Time): `auto` (the default) follows the Mac window while playing, any shape, or the iOS screen in landscape; `1.777` 16:9 | +| Rotation (iPhone and iPad) | `DM_ROTATION` | `auto` as the device turns, `landscape`, `portrait` | | Frame-rate counter | `DM_SHOW_FPS` | `1`, `0` | | Touch feedback | `DM_TOUCH_HAPTICS` | `1`, `0` | | Game rumble | `DM_RUMBLE` | `1`, `0` | From e5bc3421c5e68920a85797f58c6741f6c570a45e Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:18:49 +0100 Subject: [PATCH 18/40] iOS: upright, a game that renders any shape fills the space above the touch controls instead of a 16:9 picture with bars, and with a controller the whole screen A profile with screen_aspect any renders the shape the front end reports (front_view_aspect, dm_screen_view_changed). front_ios.m reported the app window's shape in landscape and 16:9 upright, so upright the game rendered 16:9 and the front end fitted it, with bars, into the part above the controls. DmLayout now carries `area`, where the picture may go: the view in the overlay layout, the part above the controls in a split. viewDidLayoutSubviews reports that area's shape (on a TV the TV's), so upright the game renders the space above the controls: 402x375 points (1.07:1) on an iPhone 17 instead of a 402x226 picture. Landscape reports the whole view, as before. Profiles without screen_aspect any keep the 16:9 picture above the controls. With a controller connected the touch controls hide (DmGamepad.m, now dm_gamepad_off, which lays the view out again when that changes): upright with no fold to split at, the picture's area is then the whole safe area (DmLayout.h dm_layout_controls_hidden); a half-folded Duo keeps its split, landscape is unchanged. Fill applies only to a landscape view, so a 16:9 picture upright with the controls hidden stays fitted instead of losing most of its width. Tests (Duo harness): the area in landscape and upright, the hidden-controls layout and Fill upright, the shape the production root reports, and a real turn of the Simulator's iPhone to Portrait and back: the shape handed to the host is the area's, and once the game presents it the picture fills the area (402x375). Checked: the Duo harness on the iPhone 17 Simulator, 41 tests, 0 failures (1 skipped, iOS 27.1 only). --- dm/ios/Sources/DmGamepad.m | 8 ++- dm/ios/Sources/DmLayout.h | 24 +++++-- dm/ios/Sources/front_ios.m | 18 +++-- dm/ios/tests/duo/Tests/DuoHarnessTests.m | 87 ++++++++++++++++++++++++ docs/env_switches.tsv | 2 +- docs/settings.md | 2 +- 6 files changed, 125 insertions(+), 16 deletions(-) diff --git a/dm/ios/Sources/DmGamepad.m b/dm/ios/Sources/DmGamepad.m index 1f2040e7a1..f3b1eaabf1 100644 --- a/dm/ios/Sources/DmGamepad.m +++ b/dm/ios/Sources/DmGamepad.m @@ -136,6 +136,7 @@ static void phone_motion_start(UIView *body) { @interface DmGamepad : UIView - (void)arrange:(DmLayout)layout; - (BOOL)menuTakes:(CGPoint)p; +- (BOOL)padOff; @end // The menu button: a UIButton whose menu is the in-game menu (UIKit draws it and takes the touches while it is open). Its hit circle is larger than the drawn one, and a key nearer // to the touch wins (− / + sit beside it), so it asks the pad @@ -643,11 +644,15 @@ static BOOL pads_slotted(GCController *gone) { dispatch_async(dispatch_get_main_queue(), ^{ cp_poll(); }); // a controller came or went while the controller applet's prompt is up return waiting || pad_slot_mask(&g_slots) != 0; // a controller waiting for its first press hides the touch gamepad too } +- (BOOL)padOff { return _padOff; } - (void)padsChanged:(NSNotification *)note { GCController *c = note.object, *gone = [note.name isEqual:GCControllerDidDisconnectNotification] ? c : nil; if (c && (!c.extendedGamepad || g_merge)) LOG("controller %s: %s%s", gone ? "disconnected" : "connected", pad_name(c), c.extendedGamepad ? "" : " (no extended gamepad profile: not used)"); BOOL any = g_merge ? pads_merged(gone) : pads_slotted(gone); - dispatch_async(dispatch_get_main_queue(), ^{ self->_padOff = any; self->_controls.hidden = any; [self reset]; }); // the touch gamepad hides while any controller is in use + dispatch_async(dispatch_get_main_queue(), ^{ // the touch gamepad hides while any controller is in use; upright the picture then takes the whole screen (front_ios.m, DmLayout.h dm_layout_controls_hidden) + BOOL was = self->_padOff; self->_padOff = any; self->_controls.hidden = any; [self reset]; + if (was != any) [self.superview setNeedsLayout]; + }); } @end @@ -745,6 +750,7 @@ int front_ctrl_support(int min, int max) { UIView *dm_gamepad_view(CGRect frame) { DmGamepad *v = [[DmGamepad alloc] initWithFrame:frame]; g_view = v; return v; } void dm_gamepad_arrange(UIView *view, DmLayout layout) { [(DmGamepad *)view arrange:layout]; } void dm_gamepad_opmode_changed(void) { [g_view opmodeChanged]; } +BOOL dm_gamepad_off(UIView *view) { return [view isKindOfClass:DmGamepad.class] && [(DmGamepad *)view padOff]; } void dm_gamepad_touch_feedback(bool on) { [g_view setTouchFeedback:on]; } int dm_pad_layout(void) { return g_view ? [g_view padLayout] : 0; } bool dm_pad_layout_offered(int layout) { return layout == 0 || layout == 1 || (layout == 2 && dm_opmode() == 0); } diff --git a/dm/ios/Sources/DmLayout.h b/dm/ios/Sources/DmLayout.h index 8b1ac90ace..73e0c1bb18 100644 --- a/dm/ios/Sources/DmLayout.h +++ b/dm/ios/Sources/DmLayout.h @@ -5,7 +5,9 @@ NS_ASSUME_NONNULL_BEGIN -typedef struct { CGRect game, controls; BOOL separated, topSafeOnly; } DmLayout; +// game: the picture's rectangle; controls: the touch controls'; area: where the picture may go (the view in the overlay layout, the part above the controls in a split), whose shape a +// profile with screen_aspect any renders (front_ios.m view_shape, front.h front_view_aspect) +typedef struct { CGRect game, controls; BOOL separated, topSafeOnly; CGRect area; } DmLayout; // Rotation (DM_ROTATION, a launcher and in-game menu setting): auto, landscape or portrait. auto gives `automatic`: every orientation for the view controllers, so the app's own list // decides, and for the app (front_ios_app_orientations) its Info.plist list for the device: the phone's two landscapes, the iPad's four. landscape locks the two landscapes, portrait @@ -55,7 +57,7 @@ static inline CGRect dm_horizontal_division(CGRect b, CGRect division) { } static inline DmLayout dm_layout(CGRect b, UIEdgeInsets insets, BOOL partiallyOpen, CGRect division, CGFloat aspect) { - DmLayout l = { dm_aspect_fit(b, aspect), b, NO, NO }; + DmLayout l = { dm_aspect_fit(b, aspect), b, NO, NO, b }; if (CGRectIsEmpty(b)) return l; division = dm_horizontal_division(b, division); BOOL portrait = b.size.height > b.size.width; @@ -67,17 +69,26 @@ static inline DmLayout dm_layout(CGRect b, UIEdgeInsets insets, BOOL partiallyOp CGRect lower = CGRectMake(CGRectGetMinX(b), CGRectGetMaxY(division), b.size.width, CGRectGetMaxY(b) - CGRectGetMaxY(division)); if (upper.size.height <= 0 || CGRectGetMaxY(safe) <= CGRectGetMinY(lower)) return l; - l.game = dm_aspect_fit(upper, aspect); l.controls = lower; l.separated = YES; + l.game = dm_aspect_fit(upper, aspect); l.controls = lower; l.separated = YES; l.area = upper; return l; } +// The touch controls are hidden (a controller is connected, DmGamepad.m dm_gamepad_off): upright with no fold to split at, the picture takes the whole screen inside its safe area +// instead of the half above controls that are not there. Landscape, and a half-folded Duo's split at its fold, stay as dm_layout makes them +static inline DmLayout dm_layout_controls_hidden(DmLayout l, CGRect b, UIEdgeInsets insets, BOOL partiallyOpen, CGRect division, CGFloat aspect) { + if (CGRectIsEmpty(b) || b.size.height <= b.size.width || partiallyOpen || !CGRectIsNull(dm_horizontal_division(b, division))) return l; + CGRect safe = UIEdgeInsetsInsetRect(b, insets); + if (CGRectIsEmpty(safe)) return l; + return (DmLayout){ dm_aspect_fit(safe, aspect), b, NO, NO, safe }; +} + // The picture is on a TV (front_ios.m DmTVSceneDelegate): the controls take the whole view and no touch goes to the game's touch screen (the game is docked there). -static inline DmLayout dm_layout_pad_only(CGRect b) { return (DmLayout){ CGRectZero, b, NO, NO }; } +static inline DmLayout dm_layout_pad_only(CGRect b) { return (DmLayout){ CGRectZero, b, NO, NO, CGRectZero }; } // Fill covers the full view only in the overlay layout (landscape). A portrait or tabletop split keeps the picture fitted in its -// area above the controls, where cropping a small picture would cost more than the bars. +// area above the controls, where cropping a small picture would cost more than the bars, and so does an upright screen with the controls hidden (a 16:9 picture would lose most of its width) static inline DmLayout dm_layout_picture(DmLayout l, CGRect b, BOOL fill, CGFloat aspect) { - if (fill && !l.separated && !CGRectIsEmpty(b)) l.game = dm_aspect_fill(b, aspect); + if (fill && !l.separated && !CGRectIsEmpty(b) && b.size.width >= b.size.height) l.game = dm_aspect_fill(b, aspect); return l; } @@ -93,5 +104,6 @@ UIView *dm_gamepad_view(CGRect frame); void dm_gamepad_arrange(UIView *view, DmLayout layout); void dm_gamepad_reset(void); void dm_gamepad_opmode_changed(void); // the console mode changed (the in-game menu, a TV): the Tap layout exists only in handheld mode +BOOL dm_gamepad_off(UIView *view); // the touch controls are hidden while a controller is in use (the view stays, for game touches) NS_ASSUME_NONNULL_END diff --git a/dm/ios/Sources/front_ios.m b/dm/ios/Sources/front_ios.m index 098368c118..05fd60b7e2 100644 --- a/dm/ios/Sources/front_ios.m +++ b/dm/ios/Sources/front_ios.m @@ -97,12 +97,14 @@ - (void)showFPS:(BOOL)on; @end static DmRootVC *g_root; static CGFloat g_picture_aspect = DM_PICTURE_16_9; // the presented picture's width / height (front_picture_shape); main thread -// The game view's shape for a profile with screen_aspect any (front.h front_view_aspect, dm_screen_view_changed): the app window's width / height in landscape (an iPad's 1.43, an -// iPhone's 2.17, a window under Stage Manager), 16:9 in portrait, where the picture sits above the pad. A change is handed to the host once the layout has settled for 300 ms +// The game view's shape for a profile with screen_aspect any (front.h front_view_aspect, dm_screen_view_changed): the shape of the picture's area (DmLayout.h DmLayout.area), so +// the game renders the space it is shown in with no bars: in landscape the app window (an iPad's 1.43, an iPhone's 2.17, a window under Stage Manager), upright the part above the +// touch controls (an iPhone 17's 402x375 points, 1.07) or, with a controller, the whole safe area; on a TV the TV's. A change is handed to the host once the layout has settled for 300 ms static _Atomic double g_view_aspect; static uint64_t g_view_change; // g_view_change: main thread double front_view_aspect(void) { return g_view_aspect; } static void view_shape(CGSize sz, int notify) { - double a = sz.width > sz.height && sz.height > 0 ? sz.width / sz.height : 16.0 / 9.0; if (a == g_view_aspect) return; + if (sz.width <= 0 || sz.height <= 0) return; + double a = sz.width / sz.height; if (a == g_view_aspect) return; g_view_aspect = a; if (!notify) return; uint64_t n = ++g_view_change; dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 300 * NSEC_PER_MSEC), dispatch_get_main_queue(), ^{ if (n == g_view_change) dm_screen_view_changed(a); }); @@ -165,8 +167,8 @@ - (void)startGame { // takes 1.3 to 1.77), else 16:9. The window's, not the screen's: on an iPad under Stage Manager or Split View the app does not fill the screen. Resolved here, before the // host writes the state page (main.c dm_fps_state_apply) const char *a = getenv("DM_SCREEN_ASPECT"); - if (DM_GAME_SCREEN_ASPECT == 2) { // a profile whose stubs take any shape (screen_aspect any) follows the window itself, wider or taller (loader.c, front_view_aspect) - CGSize sz = self.view.window ? self.view.window.bounds.size : UIScreen.mainScreen.bounds.size; view_shape(sz, 0); + if (DM_GAME_SCREEN_ASPECT == 2) { // a profile whose stubs take any shape (screen_aspect any) follows the picture's area, wider or taller (loader.c, front_view_aspect): + // the layout pass below sets it (viewDidLayoutSubviews), before the host starts } else if (a && !strcmp(a, "auto")) { CGSize sz = self.view.window ? self.view.window.bounds.size : UIScreen.mainScreen.bounds.size; double r = MAX(sz.width, sz.height) / MAX(MIN(sz.width, sz.height), 1); char v[16]; if (r >= 1.3 && r < 1.77) snprintf(v, sizeof v, "%.3f", r); else strcpy(v, "0"); @@ -196,7 +198,6 @@ - (void)startGame { - (void)viewDidLayoutSubviews { [super viewDidLayoutSubviews]; CGRect b = self.view.bounds; if (!self.game) return; - view_shape(b.size, 1); CGRect division = CGRectNull; // an iPhone Duo's fold, as UIKit reports it (iOS 27.1) NSMutableArray<NSString *> *regionLog = [NSMutableArray array]; #if __has_include(<UIKit/UIHingeInteraction.h>) @@ -211,7 +212,10 @@ - (void)viewDidLayoutSubviews { BOOL duoPortrait = b.size.height > b.size.width && (self.hingeStatus != 0 || !CGRectIsNull(division)); UIEdgeInsets gameInsets = dm_game_safe_insets(b, self.view.safeAreaInsets, duoPortrait); BOOL onTV = self.game.superview != self.view; // the TV's view lays the picture out; the phone is all pad - DmLayout layout = onTV ? dm_layout_pad_only(b) : dm_layout_picture(dm_layout(b, gameInsets, self.partiallyOpen, division, g_picture_aspect), b, self.fill, g_picture_aspect); + DmLayout layout = dm_layout(b, gameInsets, self.partiallyOpen, division, g_picture_aspect); + if (dm_gamepad_off(self.pad)) layout = dm_layout_controls_hidden(layout, b, gameInsets, self.partiallyOpen, division, g_picture_aspect); + layout = onTV ? dm_layout_pad_only(b) : dm_layout_picture(layout, b, self.fill, g_picture_aspect); + view_shape(onTV ? self.game.superview.bounds.size : layout.area.size, 1); if (!onTV) self.game.frame = layout.game; layout.topSafeOnly = duoPortrait; NSString *line = [NSString stringWithFormat:@"%@, hinge %ld%s, divisions %@, picture %@, controls %@", onTV ? @"pad only (picture on the TV)" : layout.separated ? @"split" : self.fill ? @"overlay, Fill" : @"overlay", diff --git a/dm/ios/tests/duo/Tests/DuoHarnessTests.m b/dm/ios/tests/duo/Tests/DuoHarnessTests.m index e66a7cd96a..339f775660 100644 --- a/dm/ios/tests/duo/Tests/DuoHarnessTests.m +++ b/dm/ios/tests/duo/Tests/DuoHarnessTests.m @@ -1389,4 +1389,91 @@ - (void)testRotationSettingSetsTheOrientationMasks { XCTAssertEqual(front_ios_app_orientations(), pad ? all : land); unsetenv("DM_ROTATION"); } + +- (void)waitForRoot:(UIViewController *)root portrait:(BOOL)portrait { + NSPredicate *p = [NSPredicate predicateWithBlock:^BOOL(UIViewController *vc, NSDictionary *bindings) { + (void)bindings; CGSize sz = vc.view.window.bounds.size; return portrait ? sz.height > sz.width : sz.width > sz.height; + }]; + XCTNSPredicateExpectation *e = [[XCTNSPredicateExpectation alloc] initWithPredicate:p object:root]; + XCTAssertEqual([XCTWaiter waitForExpectations:@[e] timeout:8.0], XCTWaiterResultCompleted, @"the window turns %s", portrait ? "upright" : "to landscape"); + [root.view layoutIfNeeded]; +} + +// The pick turns the window at once, a phone included (its Info.plist lists only landscape); upright the picture area is above the controls, and a screen_aspect any profile is +// handed that area's shape, not 16:9 (front_ios.m view_shape) +- (void)testRotationTurnsTheWindowAndTheGameTakesThePictureAreasShape { + UIViewController *root = [self harnessRootViewController]; + XCTAssertNotNil(root); + UIView *game = [root valueForKey:@"game"]; + dm_setting_live("DM_ROTATION", "portrait"); + [self waitForRoot:root portrait:YES]; + XCTAssertEqual(root.supportedInterfaceOrientations, UIInterfaceOrientationMaskPortrait | UIInterfaceOrientationMaskPortraitUpsideDown); + CGRect b = root.view.bounds; + DmLayout l = dm_layout(b, root.view.safeAreaInsets, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(l.separated); + XCTAssertEqualWithAccuracy(front_view_aspect(), l.area.size.width / l.area.size.height, 1e-3, @"the area above the controls: %@", NSStringFromCGRect(l.area)); + XCTAssertLessThan(front_view_aspect(), 16.0 / 9.0 - 0.1, @"not 16:9"); + // the game presents that shape (its sides rounded to 8 pixels): the picture fills the area + CGFloat a = l.area.size.width / l.area.size.height; + front_picture_shape((int)(a * 10000), 10000); + XCTestExpectation *applied = [self expectationWithDescription:@"shape applied"]; + dispatch_async(dispatch_get_main_queue(), ^{ [applied fulfill]; }); + [self waitForExpectations:@[applied] timeout:2]; + [root.view setNeedsLayout]; [root.view layoutIfNeeded]; + XCTAssertEqualWithAccuracy(game.frame.size.width, l.area.size.width, 1.0, @"%@ in %@", NSStringFromCGRect(game.frame), NSStringFromCGRect(l.area)); + XCTAssertEqualWithAccuracy(game.frame.size.height, l.area.size.height, 1.0); + front_picture_shape(1920, 1080); + dm_setting_live("DM_ROTATION", "landscape"); + [self waitForRoot:root portrait:NO]; + XCTAssertEqualWithAccuracy(front_view_aspect(), root.view.bounds.size.width / root.view.bounds.size.height, 1e-3, @"landscape: the whole view"); + dm_setting_live("DM_ROTATION", "auto"); + unsetenv("DM_ROTATION"); + XCTestExpectation *restored = [self expectationWithDescription:@"16:9 restored"]; + dispatch_async(dispatch_get_main_queue(), ^{ [restored fulfill]; }); + [self waitForExpectations:@[restored] timeout:2]; +} + +- (void)testPictureAreaIsTheViewOrTheSpaceAboveTheControls { + CGRect land = CGRectMake(0, 0, 874, 402), up = CGRectMake(0, 0, 402, 874); + UIEdgeInsets landInsets = UIEdgeInsetsMake(0, 62, 21, 62), upInsets = UIEdgeInsetsMake(62, 0, 34, 0); + DmLayout l = dm_layout(land, landInsets, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(CGRectEqualToRect(l.area, land), @"landscape overlay: the whole view, %@", NSStringFromCGRect(l.area)); + l = dm_layout(up, upInsets, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(l.separated); + XCTAssertTrue(CGRectEqualToRect(l.area, CGRectMake(0, 62, 402, 375)), @"upright: above the midpoint, inside the safe area, %@", NSStringFromCGRect(l.area)); + CGFloat a = l.area.size.width / l.area.size.height; + DmLayout own = dm_layout(up, upInsets, NO, CGRectNull, a); + XCTAssertEqualWithAccuracy(own.game.size.width, 402, 1); XCTAssertEqualWithAccuracy(own.game.size.height, 375, 1); + XCTAssertTrue(CGRectEqualToRect(dm_layout_pad_only(up).area, CGRectZero)); +} + +// A controller hides the touch controls: upright the picture takes the whole safe area; landscape and a half-folded Duo's split stay +- (void)testHiddenControlsGiveAnUprightPictureTheWholeScreen { + CGRect land = CGRectMake(0, 0, 874, 402), up = CGRectMake(0, 0, 402, 874); + UIEdgeInsets upInsets = UIEdgeInsetsMake(62, 0, 34, 0); + DmLayout l = dm_layout_controls_hidden(dm_layout(up, upInsets, NO, CGRectNull, DM_PICTURE_16_9), up, upInsets, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertFalse(l.separated); + XCTAssertTrue(CGRectEqualToRect(l.area, CGRectMake(0, 62, 402, 778)), @"%@", NSStringFromCGRect(l.area)); + XCTAssertTrue(CGRectEqualToRect(l.controls, up)); + XCTAssertEqualWithAccuracy(l.game.size.width, 402, 1, @"16:9 fits the width"); + DmLayout filled = dm_layout_picture(l, up, YES, DM_PICTURE_16_9); + XCTAssertTrue(CGRectEqualToRect(filled.game, l.game), @"Fill stays fitted upright: %@", NSStringFromCGRect(filled.game)); + DmLayout over = dm_layout(land, UIEdgeInsetsZero, NO, CGRectNull, DM_PICTURE_16_9); + DmLayout same = dm_layout_controls_hidden(over, land, UIEdgeInsetsZero, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(CGRectEqualToRect(same.game, over.game) && CGRectEqualToRect(same.area, over.area), @"landscape as it was"); + DmLayout half = dm_layout(up, upInsets, YES, CGRectNull, DM_PICTURE_16_9); + DmLayout halfHidden = dm_layout_controls_hidden(half, up, upInsets, YES, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(halfHidden.separated && CGRectEqualToRect(halfHidden.area, half.area), @"a half-folded Duo keeps its split"); +} + +// In landscape the game is handed the whole view's shape, as before +- (void)testProductionRootReportsThePictureAreaShape { + UIViewController *root = [self harnessRootViewController]; + XCTAssertNotNil(root); + [root.view setNeedsLayout]; [root.view layoutIfNeeded]; + CGRect b = root.view.bounds; + BOOL duo = [[root valueForKey:@"hingeStatus"] integerValue] != 0; + DmLayout l = dm_layout(b, dm_game_safe_insets(b, root.view.safeAreaInsets, duo), [[root valueForKey:@"partiallyOpen"] boolValue], CGRectNull, DM_PICTURE_16_9); + XCTAssertEqualWithAccuracy(front_view_aspect(), l.area.size.width / l.area.size.height, 1e-3, @"area %@ in %@", NSStringFromCGRect(l.area), NSStringFromCGRect(b)); +} @end diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 97535cc72b..8eb66f4ac2 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -350,7 +350,7 @@ DM_SAVED_CHOICES switch `0` ignores the player's saved choices (launcher.<VAR> i DM_SAVE_SYNC player Cloud saves: the folder whose `<title ID>/save.dmsync` the save is synced through, `icloud` for iCloud Drive/nxnative Saves (the Mac); `0` or unset, off. The Mac launcher sets the folder chosen in its settings window's Cloud saves, the iPhone app the one picked in its launcher (dm/host/save_sync.h); regress.py, bench.py and the NSP script's own host runs set `0` DM_SAVE_SYNC_CHOICE switch Cloud saves: set by a front end that asks the player when the cloud's save and this device's both changed (any value: a save that changed on both sides is replaced on neither without a choice); `ask`: no choice yet, `cloud:<hash>` brings in the cloud's save with that hash, `local:<hash>` sends this device's over it (`local:none`: the cloud had none), `offline`: the cloud could not be read before the start, so the host does not wait for it; unset, the newer save wins (dm/host/save_sync.c, docs/frontends.md) DM_SCHEME_NAME switch Set by `build_ipa.sh` itself for xcodegen's `project.yml`: the Xcode scheme name, `DmIOS`, or `<display name> (<title ID>)` for a project prepared with `--prepare-xcode` -DM_SCREEN_ASPECT player `=<width/height>` renders a game whose profile has `screen_aspect` in that shape. `on` profiles (taller than 16:9, set at the start): 1.3 to 1.77, e.g. `1.43` for the iPhone Duo; output 1920 wide, 1280-wide layout space, the HUD at the taller edges and 16:9's sides in view; `auto` (iOS) the app window's own shape when it is taller than 16:9, else 16:9; `0` or unset is 16:9. `any` profiles (followed while the game runs): 0.25 to 6, portrait included; `auto` or unset follows the Mac window, or the iOS screen in landscape (upright on iOS it stays 16:9), `1.777` or `0` is 16:9, a number fixes the shape. Launcher setting +DM_SCREEN_ASPECT player `=<width/height>` renders a game whose profile has `screen_aspect` in that shape. `on` profiles (taller than 16:9, set at the start): 1.3 to 1.77, e.g. `1.43` for the iPhone Duo; output 1920 wide, 1280-wide layout space, the HUD at the taller edges and 16:9's sides in view; `auto` (iOS) the app window's own shape when it is taller than 16:9, else 16:9; `0` or unset is 16:9. `any` profiles (followed while the game runs): 0.25 to 6, portrait included; `auto` or unset follows the Mac window, or on iOS the picture's area (the screen in landscape; upright the space above the touch controls, or the whole screen with a controller), `1.777` or `0` is 16:9, a number fixes the shape. Launcher setting DM_SCREEN_ASPECT_SITES diagnostic `=<mask>` with DM_SCREEN_ASPECT: which of the profile's layout-size conversion stubs follow the taller layout, one bit each (default all) DM_SCREEN_FOV diagnostic `off` with DM_SCREEN_ASPECT: the scene keeps the game's vertical angle (narrower sides) instead of 16:9's sides with more above and below DM_SCREEN_HUD diagnostic `=<edge>,<listed>,<full width>,<full height>[,<tallest>]` with DM_SCREEN_ASPECT: the layout rules' distances and sizes in the 1280x720 space (default 280,100,1270,710,1300) diff --git a/docs/settings.md b/docs/settings.md index 9c6f4cd9d1..c521282e2c 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -276,7 +276,7 @@ A line like `DM_PRESENT_SNAP=0` in `Documents/env.txt` (in the Files app, under | Display mode (Mac) | `DM_FULLSCREEN` | `1` fullscreen, `0` a window | | Cloud saves folder (Mac) | `DM_SAVE_SYNC` | a folder; `icloud`: iCloud Drive › nxnative Saves; `0`: off | | Picture (iPhone and iPad) | `DM_PICTURE_FIT` | `fit` 16:9 with black bars, `fill` covers the screen, cropped | -| Picture shape | `DM_SCREEN_ASPECT` | Profiles with `screen_aspect on`: `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9. Profiles with `screen_aspect any` (Ocarina of Time): `auto` (the default) follows the Mac window while playing, any shape, or the iOS screen in landscape; `1.777` 16:9 | +| Picture shape | `DM_SCREEN_ASPECT` | Profiles with `screen_aspect on`: `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9. Profiles with `screen_aspect any` (Ocarina of Time): `auto` (the default) follows the Mac window while playing, any shape, or on iOS the space the picture has (the screen in landscape, upright the part above the touch controls); `1.777` 16:9 | | Rotation (iPhone and iPad) | `DM_ROTATION` | `auto` as the device turns, `landscape`, `portrait` | | Frame-rate counter | `DM_SHOW_FPS` | `1`, `0` | | Touch feedback | `DM_TOUCH_HAPTICS` | `1`, `0` | From 7bfa42617eb3e8b7c462796a49c4836321eadd3d Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:19:00 +0100 Subject: [PATCH 19/40] screen_aspect any: upright, the vertical angle is held to a square picture's (DM_SCREEN_UPRIGHT_FOV), so a phone upright no longer looks up into parts of the world the game never builds Upright, the camera rule takes the scale of the same screen turned landscape, its width getting landscape's vertical angle; a phone upright (0.46:1) then sees about 90 degrees vertically (Ocarina's fovy is 50 degrees at 16:9), and above Link's house the world is not built. loader.c now raises the camera's y multiplier (state +0xc4 / +0xc8, which the size stub copies with the size) to that of a picture of the cap's shape when it is smaller, and x with it, so the projection keeps the screen's aspect and the width narrows instead: by default the cap is a square (79 degrees). DM_SCREEN_UPRIGHT_FOV moves it (1.777: the game's own vertical angle) or turns it off (0); landscape is never capped. The HUD's multipliers are the camera's divided by the largest of them, so the HUD does not change. No stub changes: the multipliers were already the loader's (screen_camera, screen_upright_cap; the log line names them). test_oot_aspect: camera() mirrors the cap (checked against loader.c's code compiled on its own for 0, 1 and 1.777 and six shapes: equal to 1e-5). test_shapes encoded the uncapped rule for a phone upright; it now expects every upright shape to show a square's height (updated deliberately); the uncapped rule moved to test_upright_without_the_cap, and test_the_upright_cap_holds_the_vertical_angle checks the cap at 1, 0.75 and 16:9, the shape kept, and landscape untouched. Checked: test_oot_aspect (20), loader.c syntax-checked with -Wall -Wextra. Not yet looked at above Link's house or in the Kokiri Forest cutscene upright. --- dm/host/loader.c | 38 ++++++++++++++++++++++----- dm/tools/tests/test_oot_aspect.py | 43 ++++++++++++++++++++++++++----- docs/env_switches.tsv | 1 + 3 files changed, 68 insertions(+), 14 deletions(-) diff --git a/dm/host/loader.c b/dm/host/loader.c index 9d6921a537..c1b481ba81 100644 --- a/dm/host/loader.c +++ b/dm/host/loader.c @@ -328,6 +328,28 @@ __attribute__((weak)) double front_view_aspect(void) { return 0; } // front.h: static pthread_mutex_t g_screen_mu = PTHREAD_MUTEX_INITIALIZER; // the game's start (main.c) and the front end's main thread both write the shape static void screen_any_write_(double a, const char *why); static void screen_any_write(double a, const char *why) { pthread_mutex_lock(&g_screen_mu); screen_any_write_(a, why); pthread_mutex_unlock(&g_screen_mu); } +// The camera's x / y multipliers for a shape (state +0xc4 / +0xc8; 0 = 16:9). Landscape: wider than 16:9 keeps the vertical angle (x × 16:9 / r), narrower keeps the horizontal one +// (y × r / 16:9). Portrait takes the scale of the same screen turned landscape, its width getting what that one's height gets: a phone or a window turned keeps the characters' +// size and shows another part of the world. Their ratio y / x is always r × 9 / 16, the projection's aspect over 16:9's +static void screen_camera(double r, float c[2]) { + double l = r >= 1 ? r : 1 / r, xl = l >= 16.0 / 9 ? 16.0 / 9 / l : 1, yl = l >= 16.0 / 9 ? 1 : l * 9 / 16; + c[0] = r == 0 ? 1 : r >= 1 ? (float)xl : (float)(yl * 16 / 9); c[1] = r == 0 ? 1 : r >= 1 ? (float)yl : (float)(xl * 9 / 16); +} +// Upright, the vertical angle is capped (DM_SCREEN_UPRIGHT_FOV, read once): an upright screen shows no more height than a picture of that shape (default 1, a square) would, its +// width narrowing instead. A phone upright with the rule above sees about 90° vertically (Ocarina's fovy 50° at 16:9, 79° at 1:1), past what the game ever shows: above Link's +// house the world is not built. 0 turns the cap off; another shape from 0.25 to 6 moves it (1.777: the game's own vertical angle; under 1 only screens taller than it are held). +// Landscape is never capped. The y multiplier is raised to the cap's and x follows it, so the projection keeps the screen's aspect +static double g_upright_cap = -1; +static double screen_upright_cap(void) { + if (g_upright_cap >= 0) return g_upright_cap; + const char *e = getenv("DM_SCREEN_UPRIGHT_FOV"); double v = 1; + if (e && *e) { + char *end; v = strtod(e, &end); + if (end == e || *end || !(v == 0 || (v >= SCREEN_MIN && v <= SCREEN_MAX))) { LOG("[RED] DM_SCREEN_UPRIGHT_FOV=%s: 0 (off) or a width / height from 0.25 to 6; keeping 1", e); v = 1; } + else LOG("%s: upright vertical angle %s (DM_SCREEN_UPRIGHT_FOV)", g_fps_mod, v ? "held to that shape's" : "not capped"); + } + return g_upright_cap = v; +} static void screen_any_write_(double a, const char *why) { if (!dm_fps_state) return; double r = a < SCREEN_MIN ? SCREEN_MIN : a > SCREEN_MAX ? SCREEN_MAX : a; @@ -335,16 +357,18 @@ static void screen_any_write_(double a, const char *why) { if (r == g_screen_ratio) return; float f[3] = {(float)r, r ? (float)fmax(r * 9 / 16, 1) : 1, r ? (float)fmax(16 / (9 * r), 1) : 1}; memcpy(dm_fps_state + 0xa0, f, sizeof f); - // +0xc4 / +0xc8: the camera's x / y multipliers (the projection's rows). Landscape: wider than 16:9 keeps the vertical angle (x × 16:9 / r), narrower keeps the horizontal one - // (y × r / 16:9). Portrait takes the scale of the same screen turned landscape, its width getting what that one's height gets: a phone or a window turned keeps the - // characters' size and shows another part of the world. The size stub copies them with the size it sets (patches.py screen_shape) - { double l = r >= 1 ? r : 1 / r, xl = l >= 16.0 / 9 ? 16.0 / 9 / l : 1, yl = l >= 16.0 / 9 ? 1 : l * 9 / 16; - float c[2] = {r == 0 ? 1 : r >= 1 ? (float)xl : (float)(yl * 16 / 9), r == 0 ? 1 : r >= 1 ? (float)yl : (float)(xl * 9 / 16)}; - memcpy(dm_fps_state + 0xc4, c, sizeof c); } + // +0xc4 / +0xc8: the camera's x / y multipliers (the projection's rows; screen_camera), upright capped (screen_upright_cap). The size stub copies them with the size it sets + // (patches.py screen_shape), and the HUD's follow from them + float c[2]; screen_camera(r, c); double cap = screen_upright_cap(); int capped = 0; + { float k[2]; + if (r && cap && r < 1 && r < cap) { screen_camera(cap, k); if (c[1] < k[1]) { c[1] = k[1]; c[0] = (float)(k[1] * 16 / (9 * r)); capped = 1; } } } + memcpy(dm_fps_state + 0xc4, c, sizeof c); uint32_t gen; memcpy(&gen, dm_fps_state + 0xac, 4); gen++; memcpy(dm_fps_state + 0xac, &gen, 4); extern void mtl_present_screen(uint32_t w, uint32_t h); mtl_present_screen(r ? (uint32_t)(r * 10000 + 0.5) : 0, r ? 10000 : 0); // the picture's shape for a crop of that shape (nvn_mtl.m present_shape) int first = g_screen_ratio < 0; g_screen_ratio = r; - if (r) LOG("%s: screen shape %.3f (%s%s): %s, picture %.3f x %.3f of 16:9", g_fps_mod, r, why, a != r && a > 0 ? ", held to 0.25..6" : "", r > 16.0 / 9 ? "more at the sides" : "more above and below", f[1], f[2]); + if (r) LOG("%s: screen shape %.3f (%s%s): %s, picture %.3f x %.3f of 16:9, camera x %.3f y %.3f%s", g_fps_mod, r, why, a != r && a > 0 ? ", held to 0.25..6" : "", + r > 16.0 / 9 ? "more at the sides" : "more above and below", f[1], f[2], c[0], c[1], capped ? ", upright cap" : ""); + if (r && capped) LOG("%s: upright: vertical angle held to a %.2f picture's (DM_SCREEN_UPRIGHT_FOV)", g_fps_mod, cap); else LOG("%s: screen shape 16:9 (%s)", g_fps_mod, why); if (!first) { void dm_display_changed(void); dm_display_changed(); } // the game lays its picture out again (hle_oe.c) } diff --git a/dm/tools/tests/test_oot_aspect.py b/dm/tools/tests/test_oot_aspect.py index f49432cdab..a67c5186ca 100644 --- a/dm/tools/tests/test_oot_aspect.py +++ b/dm/tools/tests/test_oot_aspect.py @@ -24,12 +24,17 @@ patch = importlib.util.module_from_spec(spec); spec.loader.exec_module(patch) -def camera(r): - """The camera's x / y multipliers the loader writes for a shape (loader.c screen_any_write)""" +def camera(r, cap=1): + """The camera's x / y multipliers the loader writes for a shape (loader.c screen_camera), upright held to the vertical angle of a picture of shape cap (screen_upright_cap, + DM_SCREEN_UPRIGHT_FOV: 1 by default, 0 off)""" if not r: return (1, 1) l = r if r >= 1 else 1 / r xl, yl = (16 / 9 / l, 1) if l >= 16 / 9 else (1, l * 9 / 16) - return (xl, yl) if r >= 1 else (yl * 16 / 9, xl * 9 / 16) + x, y = (xl, yl) if r >= 1 else (yl * 16 / 9, xl * 9 / 16) + if cap and r < 1 and r < cap: + ky = camera(cap, 0)[1] + if y < ky: x, y = ky * 16 / (9 * r), ky + return x, y @unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') @@ -134,11 +139,11 @@ def setUp(self): emit, put = OutputSize.emit, OutputSize.put - def shape(self, ratio, asked=None): + def shape(self, ratio, asked=None, cap=1): """The factors the output size was made with (state +0xb8 / +0xbc, 0 = 16:9) and, apart, the shape the host asks for now (+0xa0..: the camera ignores it)""" f = lambda r: (r, max(r * 9 / 16, 1), max(16 / (9 * r), 1)) if r else (0, 1, 1) self.u.mem_write(self.state + 0xa0, struct.pack('<3f', *f(ratio if asked is None else asked))) - self.u.mem_write(self.state + 0xb8, struct.pack('<2f', *camera(ratio)) if ratio else bytes(8)) + self.u.mem_write(self.state + 0xb8, struct.pack('<2f', *camera(ratio, cap)) if ratio else bytes(8)) def run_copy(self): """The matrix as the copy leaves it (rows 0..2 stored, row 3 in q18, rows 0 / 1 also in v4 / v5), aspect 16:9 at +0x1d4; the matrix, v4, v5 and the aspect after""" @@ -178,13 +183,37 @@ def test_shapes(self): self.assertAlmostEqual(m[5] / m[0], ratio, 3, ratio) # the projection's aspect is the screen's if ratio > 16 / 9: self.assertAlmostEqual(m[5], self.M[5], 5) # wider: the vertical angle kept elif ratio >= 1: self.assertAlmostEqual(m[0], self.M[0], 5) # taller landscape: the horizontal angle kept - elif 1 / ratio >= 16 / 9: self.assertAlmostEqual(m[0], self.M[5], 4) # a phone upright: its width gets landscape's vertical angle - else: self.assertAlmostEqual(m[5], self.M[0], 4) # an iPad upright: its height gets landscape's horizontal angle + else: self.assertAlmostEqual(m[5], self.M[0], 4) # upright: no more height than a square, whose height gets landscape's horizontal angle for a, b in zip(m[8:], self.M[8:]): self.assertAlmostEqual(a, b, 5) # z and w rows untouched for a, b in zip(v4 + v5, m[:8]): self.assertAlmostEqual(a, b, 6) # the frustum code after reads the same rows self.assertAlmostEqual(aspect, ratio, 3, ratio) + def test_upright_without_the_cap(self): + # DM_SCREEN_UPRIGHT_FOV=0: upright takes the scale of the same screen turned landscape (a phone sees about 90 degrees vertically) + for ratio in (0.7, 1179 / 2556, 9 / 16): + self.shape(ratio, cap=0) + m = self.run_copy()[0] + self.assertAlmostEqual(m[5] / m[0], ratio, 3, ratio) + if 1 / ratio >= 16 / 9: self.assertAlmostEqual(m[0], self.M[5], 4) # a phone upright: its width gets landscape's vertical angle + else: self.assertAlmostEqual(m[5], self.M[0], 4) # an iPad upright: its height gets landscape's horizontal angle + + def test_the_upright_cap_holds_the_vertical_angle(self): + tan = lambda m: 1 / m[5] # tan of half the vertical angle + for cap in (1, 0.75, 16 / 9): + self.shape(cap, cap=0); held = tan(self.run_copy()[0]) # the cap's own picture + for ratio in (min(cap, 1) - 0.05, 0.6, 1179 / 2556, 0.3): + self.shape(ratio, cap=cap); m = self.run_copy()[0] + self.assertAlmostEqual(tan(m), held, 4, (cap, ratio)) # no more height than the cap's picture + self.assertAlmostEqual(m[5] / m[0], ratio, 3, (cap, ratio)) # the picture keeps its own shape: the width narrows + self.assertAlmostEqual(tan(self.M), 1 / 2.14451, 5) # 16:9: fovy 50 degrees + self.shape(1.0, cap=0); square = tan(self.run_copy()[0]) + self.shape(1179 / 2556, cap=0); phone = tan(self.run_copy()[0]) + self.assertGreater(phone, 1.2 * square) # a phone upright saw about 90 degrees, a square 79 + self.shape(1.3, cap=16 / 9); m = self.run_copy()[0] + self.assertAlmostEqual(m[0], self.M[0], 5) # landscape is never capped + + @unittest.skipIf(uc is None, 'optional ARM64 execution dependency: unicorn') class ShapeChange(unittest.TestCase): # The display update's flag load (ldp w11,w9,[x0,#8]) at FN + 0x18; the stub branches back after it diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 8eb66f4ac2..a23d715c88 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -355,6 +355,7 @@ DM_SCREEN_ASPECT_SITES diagnostic `=<mask>` with DM_SCREEN_ASPECT: which of the DM_SCREEN_FOV diagnostic `off` with DM_SCREEN_ASPECT: the scene keeps the game's vertical angle (narrower sides) instead of 16:9's sides with more above and below DM_SCREEN_HUD diagnostic `=<edge>,<listed>,<full width>,<full height>[,<tallest>]` with DM_SCREEN_ASPECT: the layout rules' distances and sizes in the 1280x720 space (default 280,100,1270,710,1300) DM_SCREEN_MATCH switch `0` keeps the screen's refresh rate in fullscreen; by default a Mac screen whose rate is not a multiple of the game's frame rate switches to one that is (144 → 120 Hz), and a variable-refresh screen to a fixed-rate mode at a multiple (ProMotion 120 → 60 Hz at 60 and 30 fps) +DM_SCREEN_UPRIGHT_FOV diagnostic `=<width/height>` for a `screen_aspect any` profile: an upright picture shows no more height than a picture of that shape (default `1`, a square; `1.777` the game's own vertical angle; under 1 only screens taller than it are held), its width narrowing instead; `0` turns the cap off (a phone upright then sees about 90° vertically). Landscape is never capped (`loader.c` `screen_upright_cap`) DM_SF_TRACE diagnostic `=1` logs every request on the game's own HIPC client sessions that the host answers (`hle_sf.c`) DM_SHADER_FILES switch `1` packs an NVN game's shader library into the unsigned IPA of `build_ipa.sh` as its separate files, `msl/b-<chain>/`, not as the one file `b-<chain>.img`, for install tools that cannot copy a large file (AltServer: 2 GiB) (`pack_ipa.py --shader-files`; off by default; `--shader-files` of the NSP script sets this) DM_SHAKE_G switch `=<G>`: on iOS, the acceleration beyond gravity above which a flick of the phone shakes the controller's six-axis sensor (default 1.5) From c4aae7a4a08b5e22d18eb486e7cbf3e917a0c00a Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 01:09:10 +0100 Subject: [PATCH 20/40] iOS: Portrait layout, Full screen by default: upright, a game that renders any shape fills the whole screen with the touch controls drawn over it, as in landscape; Picture above controls keeps the split The owner asked for the picture to fill the whole screen upright with the controls over it, as in landscape, and to keep the split of the previous commit as a choice. DM_PORTRAIT_LAYOUT (launcher and in-game menu): full (the default) or split. Full screen (DmLayout.h dm_layout_full_picture) keeps the split's control arrangement in the lower half, so the controls are where the portrait layout puts them, and lays the picture over the whole view under them; the picture's area is the whole view, so the game renders the screen's shape (0.46:1 on an iPhone) and the upright cap of the previous commit holds its vertical angle to a square's. Touches above the controls go to the game's touch screen as in the split. It needs a picture that takes the screen's shape, so it applies only to a profile with screen_aspect any following the view (DM_SCREEN_ASPECT auto, front_ios.m portrait_full); other games keep their 16:9 picture above the controls, where centred on an upright screen it would sit under them, and a half-folded Duo splits at its fold. With a controller Full screen stays as it is (the controls hide). The row is shown only for a screen_aspect any profile on iOS: settings.py's shown column now takes <cond>&<cond>, as the options column already did. Tests: Duo harness, the full-screen layout (area the whole view, the controls' rectangle kept and inside the picture, landscape and a half-folded Duo unchanged) and the production root turned upright in each layout; test_launcher_settings, the row for Ocarina on iOS only. Checked: the Duo harness on the iPhone 17 Simulator, 42 tests, 0 failures (1 skipped, iOS 27.1 only); test_launcher_settings, test_docs. Front-ends: new: DM_PORTRAIT_LAYOUT (Portrait layout), and settings.tsv's shown column takes <cond>&<cond> --- dm/games/settings.py | 2 +- dm/games/settings.tsv | 3 ++- dm/ios/Sources/DmLayout.h | 9 +++++++ dm/ios/Sources/front_ios.m | 12 +++++++-- dm/ios/tests/duo/Tests/DuoHarnessTests.m | 33 ++++++++++++++++++++++++ dm/tools/tests/test_launcher_settings.py | 6 +++++ docs/env_switches.tsv | 1 + docs/settings.md | 11 +++++--- 8 files changed, 70 insertions(+), 7 deletions(-) diff --git a/dm/games/settings.py b/dm/games/settings.py index 506134421a..2351629f78 100644 --- a/dm/games/settings.py +++ b/dm/games/settings.py @@ -110,7 +110,7 @@ def offered(v, n): # value=label?cond&cond: an option only some games or platf if need not in table: sys.exit(f'settings.tsv:{n}: needs names a variable no row sets: {m["needs"]}') can = table[need] if can is not None and want not in can: sys.exit(f'settings.tsv:{n}: needs a value {need} never takes: {want} (it takes {", ".join(sorted(can))})') - if not shown(cond, n): continue + if not all(shown(c, n) for c in cond.split('&')): continue # every condition of <cond>&<cond> s = {'env': var, 'kind': kind, 'section': section, 'label': label, 'note': note, 'default': value(default, n), **m} if var == 'DM_RENDER_SCALE' and base: s['base'] = base # the render scale in pixels, for the console mode chosen if kind == 'choice': diff --git a/dm/games/settings.tsv b/dm/games/settings.tsv index 5e7689732c..1e8151fd5b 100644 --- a/dm/games/settings.tsv +++ b/dm/games/settings.tsv @@ -8,7 +8,7 @@ # default a value (a toggle's: 1 on, 0 off); DM_GAME_<NAME>: the host's own default from the game profile (game_profile.h, game.py); build:<VAR>: the IPA build's setting (build_ipa.sh) # <ios>|<mac> selects the platform's default before resolving a profile macro or build variable # shown always; DM_GAME_<NAME>!=<value> or ==<value>; fixups:<KEY>: main's rewriter table has that key (a stub in the game's build, patch_nso.py); languages>1 (more than one to pick, so also when the control data lists none); gfxapi:<API>: the packer's GFXAPI; -# platform:<ios | mac>: the app's platform (SETTINGS_PLATFORM, ios when unset), for a switch only that platform's front end reads +# platform:<ios | mac>: the app's platform (SETTINGS_PLATFORM, ios when unset), for a switch only that platform's front end reads; <cond>&<cond>: all of them # section Game, Controls or Performance: a heading on the single settings screen. Rows appear in table order within each section # Reset to tested defaults clears all launcher choices; values fixed by an env file stay # label, note the setting's name and the line under it (may be empty; only for a setting with a cost: latency, memory, heat) @@ -24,6 +24,7 @@ DM_LANG choice languages build:DM_LANG languages>1 Game Game language level=bas DM_FULLSCREEN choice 0=Window,1=Fullscreen 0 platform:mac Game Display mode level=basic DM_PICTURE_FIT choice fit=Fit,fill=Fill fit platform:ios Game Picture Fill crops the game's 16:9 picture to cover the screen. Health bars, subtitles or maps near the edges can be cut off. A full-screen picture without cropping needs a patch for each game. DM_ROTATION choice auto=Automatic,landscape=Landscape,portrait=Portrait auto platform:ios Game Rotation +DM_PORTRAIT_LAYOUT choice full=Full screen,split=Picture above controls full DM_GAME_SCREEN_ASPECT==2&platform:ios Game Portrait layout DM_SHOW_FPS toggle - build:DM_SHOW_FPS platform:ios Game Frame-rate counter DM_TOUCH_HAPTICS toggle - 1 platform:ios Controls Touch feedback DM_RUMBLE toggle - 1 always Controls Game rumble level=basic diff --git a/dm/ios/Sources/DmLayout.h b/dm/ios/Sources/DmLayout.h index 73e0c1bb18..62431d4646 100644 --- a/dm/ios/Sources/DmLayout.h +++ b/dm/ios/Sources/DmLayout.h @@ -73,6 +73,15 @@ static inline DmLayout dm_layout(CGRect b, UIEdgeInsets insets, BOOL partiallyOp return l; } +// Portrait layout Full screen (DM_PORTRAIT_LAYOUT=full, the default, for a profile whose picture takes any shape: front_ios.m portrait_full): upright with no fold to split at, +// the picture covers the whole view and the touch controls, still arranged in the lower half as in the split, are drawn over it, as they are over the picture in landscape. +// Touches above the controls go to the game's touch screen as in the split. A half-folded Duo keeps its split at the fold +static inline DmLayout dm_layout_full_picture(DmLayout l, CGRect b, BOOL partiallyOpen, CGRect division, CGFloat aspect) { + if (!l.separated || CGRectIsEmpty(b) || b.size.height <= b.size.width || partiallyOpen || !CGRectIsNull(dm_horizontal_division(b, division))) return l; + l.game = dm_aspect_fit(b, aspect); l.area = b; + return l; +} + // The touch controls are hidden (a controller is connected, DmGamepad.m dm_gamepad_off): upright with no fold to split at, the picture takes the whole screen inside its safe area // instead of the half above controls that are not there. Landscape, and a half-folded Duo's split at its fold, stay as dm_layout makes them static inline DmLayout dm_layout_controls_hidden(DmLayout l, CGRect b, UIEdgeInsets insets, BOOL partiallyOpen, CGRect division, CGFloat aspect) { diff --git a/dm/ios/Sources/front_ios.m b/dm/ios/Sources/front_ios.m index 05fd60b7e2..2474b959f8 100644 --- a/dm/ios/Sources/front_ios.m +++ b/dm/ios/Sources/front_ios.m @@ -109,6 +109,12 @@ static void view_shape(CGSize sz, int notify) { uint64_t n = ++g_view_change; dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 300 * NSEC_PER_MSEC), dispatch_get_main_queue(), ^{ if (n == g_view_change) dm_screen_view_changed(a); }); } +// Portrait layout (DM_PORTRAIT_LAYOUT): full (the default) or split. Full screen needs a picture that takes the screen's shape: a profile with screen_aspect any following the view +// (DM_SCREEN_ASPECT auto); any other picture stays above the controls, where a 16:9 one centred on an upright screen would sit under them +static BOOL portrait_full(void) { + const char *a = getenv("DM_SCREEN_ASPECT"), *p = getenv("DM_PORTRAIT_LAYOUT"); + return DM_GAME_SCREEN_ASPECT == 2 && (!a || !*a || !strcmp(a, "auto")) && !(p && !strcmp(p, "split")); +} static void launcher_done(void) { [g_root startGame]; } @implementation DmRootVC - (void)loadView { self.view = [DmRootView new]; } @@ -213,12 +219,13 @@ - (void)viewDidLayoutSubviews { UIEdgeInsets gameInsets = dm_game_safe_insets(b, self.view.safeAreaInsets, duoPortrait); BOOL onTV = self.game.superview != self.view; // the TV's view lays the picture out; the phone is all pad DmLayout layout = dm_layout(b, gameInsets, self.partiallyOpen, division, g_picture_aspect); - if (dm_gamepad_off(self.pad)) layout = dm_layout_controls_hidden(layout, b, gameInsets, self.partiallyOpen, division, g_picture_aspect); + if (portrait_full()) layout = dm_layout_full_picture(layout, b, self.partiallyOpen, division, g_picture_aspect); + else if (dm_gamepad_off(self.pad)) layout = dm_layout_controls_hidden(layout, b, gameInsets, self.partiallyOpen, division, g_picture_aspect); layout = onTV ? dm_layout_pad_only(b) : dm_layout_picture(layout, b, self.fill, g_picture_aspect); view_shape(onTV ? self.game.superview.bounds.size : layout.area.size, 1); if (!onTV) self.game.frame = layout.game; layout.topSafeOnly = duoPortrait; - NSString *line = [NSString stringWithFormat:@"%@, hinge %ld%s, divisions %@, picture %@, controls %@", onTV ? @"pad only (picture on the TV)" : layout.separated ? @"split" : self.fill ? @"overlay, Fill" : @"overlay", + NSString *line = [NSString stringWithFormat:@"%@, hinge %ld%s, divisions %@, picture %@, controls %@", onTV ? @"pad only (picture on the TV)" : layout.separated ? (CGRectIntersectsRect(layout.game, layout.controls) ? @"portrait full screen, controls over the picture" : @"split") : self.fill ? @"overlay, Fill" : @"overlay", (long)self.hingeStatus, self.partiallyOpen ? " (half open)" : "", regionLog, NSStringFromCGRect(layout.game), NSStringFromCGRect(layout.controls)]; if (![line isEqualToString:self.layoutLog]) { self.layoutLog = line; LOG("front end: layout %s", line.UTF8String); } self.pad.frame = b; @@ -338,6 +345,7 @@ bool dm_setting_live(const char *env, const char *value) { return true; } if (!strcmp(env, "DM_ROTATION")) { if (value) dm_rotation_set(value); return true; } + if (!strcmp(env, "DM_PORTRAIT_LAYOUT")) { if (value) { [g_root.view setNeedsLayout]; LOG("front end: portrait layout %s (in-game menu)", value); } return true; } if (!strcmp(env, "DM_SHOW_FPS")) { if (value) [g_root showFPS:atoi(value) != 0]; return true; } if (!strcmp(env, "DM_TOUCH_HAPTICS")) { if (value) dm_gamepad_touch_feedback(atoi(value) != 0); return true; } if (!strcmp(env, "DM_RUMBLE")) { if (value) { dm_haptics_set_enabled(atoi(value) != 0); LOG("front end: game rumble %s (in-game menu)", atoi(value) ? "on" : "off"); } return true; } diff --git a/dm/ios/tests/duo/Tests/DuoHarnessTests.m b/dm/ios/tests/duo/Tests/DuoHarnessTests.m index 339f775660..2c7a5aa57d 100644 --- a/dm/ios/tests/duo/Tests/DuoHarnessTests.m +++ b/dm/ios/tests/duo/Tests/DuoHarnessTests.m @@ -1405,6 +1405,7 @@ - (void)testRotationTurnsTheWindowAndTheGameTakesThePictureAreasShape { UIViewController *root = [self harnessRootViewController]; XCTAssertNotNil(root); UIView *game = [root valueForKey:@"game"]; + setenv("DM_PORTRAIT_LAYOUT", "split", 1); // Picture above controls; Full screen below dm_setting_live("DM_ROTATION", "portrait"); [self waitForRoot:root portrait:YES]; XCTAssertEqual(root.supportedInterfaceOrientations, UIInterfaceOrientationMaskPortrait | UIInterfaceOrientationMaskPortraitUpsideDown); @@ -1422,6 +1423,19 @@ - (void)testRotationTurnsTheWindowAndTheGameTakesThePictureAreasShape { [root.view setNeedsLayout]; [root.view layoutIfNeeded]; XCTAssertEqualWithAccuracy(game.frame.size.width, l.area.size.width, 1.0, @"%@ in %@", NSStringFromCGRect(game.frame), NSStringFromCGRect(l.area)); XCTAssertEqualWithAccuracy(game.frame.size.height, l.area.size.height, 1.0); + // Full screen (the default): the picture covers the view, the controls stay in the lower half, over it + setenv("DM_PORTRAIT_LAYOUT", "full", 1); XCTAssertTrue(dm_setting_live("DM_PORTRAIT_LAYOUT", "full")); + [root.view layoutIfNeeded]; + XCTAssertEqualWithAccuracy(front_view_aspect(), b.size.width / b.size.height, 1e-3, @"the whole view's shape"); + front_picture_shape((int)(b.size.width / b.size.height * 10000), 10000); + XCTestExpectation *full = [self expectationWithDescription:@"full shape applied"]; + dispatch_async(dispatch_get_main_queue(), ^{ [full fulfill]; }); + [self waitForExpectations:@[full] timeout:2]; + [root.view setNeedsLayout]; [root.view layoutIfNeeded]; + XCTAssertEqualWithAccuracy(game.frame.size.width, b.size.width, 1.0, @"%@", NSStringFromCGRect(game.frame)); + XCTAssertEqualWithAccuracy(game.frame.size.height, b.size.height, 1.0); + XCTAssertTrue(CGRectGetMaxY(game.frame) > CGRectGetMidY(b) + 100, @"the picture runs under the controls"); + unsetenv("DM_PORTRAIT_LAYOUT"); front_picture_shape(1920, 1080); dm_setting_live("DM_ROTATION", "landscape"); [self waitForRoot:root portrait:NO]; @@ -1447,6 +1461,25 @@ - (void)testPictureAreaIsTheViewOrTheSpaceAboveTheControls { XCTAssertTrue(CGRectEqualToRect(dm_layout_pad_only(up).area, CGRectZero)); } +// Portrait layout Full screen: the picture covers the view under the controls, which keep their lower-half rectangle; landscape and a half-folded Duo stay +- (void)testFullScreenPortraitPutsTheControlsOverThePicture { + CGRect land = CGRectMake(0, 0, 874, 402), up = CGRectMake(0, 0, 402, 874); + UIEdgeInsets upInsets = UIEdgeInsetsMake(62, 0, 34, 0); + CGFloat a = up.size.width / up.size.height; + DmLayout split = dm_layout(up, upInsets, NO, CGRectNull, a); + DmLayout l = dm_layout_full_picture(split, up, NO, CGRectNull, a); + XCTAssertTrue(CGRectEqualToRect(l.area, up), @"%@", NSStringFromCGRect(l.area)); + XCTAssertEqualWithAccuracy(l.game.size.width, 402, 1); XCTAssertEqualWithAccuracy(l.game.size.height, 874, 1); + XCTAssertTrue(l.separated && CGRectEqualToRect(l.controls, split.controls), @"the controls keep their portrait arrangement"); + XCTAssertTrue(CGRectContainsRect(l.game, l.controls), @"and are drawn over the picture"); + DmLayout over = dm_layout(land, UIEdgeInsetsZero, NO, CGRectNull, DM_PICTURE_16_9); + DmLayout same = dm_layout_full_picture(over, land, NO, CGRectNull, DM_PICTURE_16_9); + XCTAssertTrue(CGRectEqualToRect(same.game, over.game) && CGRectEqualToRect(same.area, over.area), @"landscape as it was"); + DmLayout half = dm_layout(up, upInsets, YES, CGRectNull, a); + DmLayout halfFull = dm_layout_full_picture(half, up, YES, CGRectNull, a); + XCTAssertTrue(CGRectEqualToRect(halfFull.game, half.game) && CGRectEqualToRect(halfFull.area, half.area), @"a half-folded Duo keeps its split"); +} + // A controller hides the touch controls: upright the picture takes the whole safe area; landscape and a half-folded Duo's split stay - (void)testHiddenControlsGiveAnUprightPictureTheWholeScreen { CGRect land = CGRectMake(0, 0, 874, 402), up = CGRectMake(0, 0, 402, 874); diff --git a/dm/tools/tests/test_launcher_settings.py b/dm/tools/tests/test_launcher_settings.py index 8d50cd974a..3262642adf 100644 --- a/dm/tools/tests/test_launcher_settings.py +++ b/dm/tools/tests/test_launcher_settings.py @@ -72,6 +72,12 @@ def test_hold_frames_takes_on_and_off(self): # the host reads DM_FPS_HOLD as o s = build(BOTW, 'nvn')['DM_SHOW_FPS'] self.assertEqual((s['on'], s['off']), ('1', '0')) + def test_portrait_layout_only_for_a_picture_of_any_shape_on_the_iphone(self): # a shown condition of two parts (<cond>&<cond>): screen_aspect any, and iOS + oot = '0400486024DEC000' # Ocarina of Time: screen_aspect any + self.assertEqual(build(oot, 'nvn')['DM_PORTRAIT_LAYOUT']['default'], 'full') + self.assertNotIn('DM_PORTRAIT_LAYOUT', build(oot, 'nvn', 'mac')) + self.assertNotIn('DM_PORTRAIT_LAYOUT', build(BOTW, 'nvn')) # screen_aspect on: a 16:9 or taller picture stays above the controls + def test_iphone_switches_only_on_the_iphone(self): # the frame-rate counter and touch feedback are the iOS front end's (front_ios.m, DmGamepad.m) ios, mac = build(BOTW, 'nvn'), build(BOTW, 'nvn', 'mac') self.assertEqual(build(BOTW, 'nvn', 'ios').keys(), ios.keys()) # ios when unset diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index a23d715c88..33067c55da 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -266,6 +266,7 @@ DM_PAD_PRESS_TO_JOIN player `1`: a controller is no player until one of its butt DM_PATCH_FRONT switch `=<bytes>`: the size of the patch area right before the module base for a long text (`BIG_TEXT`); set only by the rewriter's own rerun with the size it needs DM_PHONE_MOTION switch On iOS, whether the phone's own device motion drives player 1's six-axis sensor: `auto` (default) while no controller supplies fresh motion, `1` phone only, `0` never; each source keeps its own orientation DM_PICTURE_FIT player `fill` scales the 16:9 picture on the iPhone and iPad until it covers the screen in landscape and crops the edges (a portrait or tabletop split and a TV stay fitted); `fit` or unset, the largest 16:9 rectangle with black bars. Launcher and in-game menu setting +DM_PORTRAIT_LAYOUT player Upright on iOS, for a `screen_aspect any` profile following the view (`DM_SCREEN_ASPECT` auto): `full` (the default) the picture covers the whole view and the touch controls, arranged in the lower half, are drawn over it; `split` the picture takes the part above the controls. Other profiles always keep their picture above the controls; a half-folded iPhone Duo splits at its fold. Applied at once (`front_ios.m` `portrait_full`, `DmLayout.h` `dm_layout_full_picture`). Launcher and in-game menu setting DM_PREEMPT switch `0` / `1` overrides the profile's `core_preempt`: also check already-runnable higher-priority peers on the caller's core after every thread SVC DM_PRELOAD diagnostic Set: on the Mac, runs the host's early background loading (`dm_host_preload_start`) as the iOS app does, to exercise that path DM_PRESENT_FIFO switch `0` makes the display link take the newest queued frame each tick, as before; by default it shows frames in order, at most one tick behind diff --git a/docs/settings.md b/docs/settings.md index c521282e2c..f309c6ba12 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -3,7 +3,7 @@ The screen before the game starts shows the settings the game supports. A setting that only matters while another is on appears under it and disappears when that one is turned off. Tap **Play** to start; the app remembers your choices for the next start. All settings are on one scrollable screen, under **Game**, **Controls** and **Performance**. Tap a menu or move a slider there; no settings page needs opening. **Upscaling** is in **Game**, beside **Console mode**. A Mac app shows the same settings in a [window of its own](#the-mac-settings-window). - [Tested defaults](#tested-defaults) -- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game's upscaler](#games-upscaler), [Game language](#game-language), [Picture](#picture), [Rotation](#rotation), [Frame-rate counter](#frame-rate-counter) +- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game's upscaler](#games-upscaler), [Game language](#game-language), [Picture](#picture), [Rotation](#rotation), [Portrait layout](#portrait-layout), [Frame-rate counter](#frame-rate-counter) - [Controls](#controls): [Touch feedback](#touch-feedback), [Game rumble](#game-rumble), [Rumble strength](#rumble-strength), [Rumble cutoff](#rumble-cutoff), [Press a button to join](#press-a-button-to-join) - [Performance](#performance): [Frame buffering](#frame-buffering), [Metal 4 pipelines](#metal-4-pipelines), [Pipeline prebuild](#pipeline-prebuild), [Encode thread](#encode-thread), [Hold frames when late](#hold-frames-when-late), [Performance HUD](#performance-hud), [Reset to tested defaults](#reset-to-tested-defaults) - [The Mac settings window](#the-mac-settings-window): [Display mode](#display-mode) @@ -91,7 +91,11 @@ Games whose profile supports it (Breath of the Wild, Animal Crossing: New Horizo ### Rotation -iPhone and iPad. **Automatic**, the default, turns with the device as before: landscape either way on an iPhone, any way on an iPad. **Landscape** stays in landscape, either way up. **Portrait** stays upright; the picture sits above the touch controls (with a controller connected it takes the whole screen), and a game that renders any shape, such as Ocarina of Time, fills that space instead of showing a 16:9 picture with bars. The screen turns as soon as you pick it, on the settings screen or in the in-game menu. +iPhone and iPad. **Automatic**, the default, turns with the device as before: landscape either way on an iPhone, any way on an iPad. **Landscape** stays in landscape, either way up. **Portrait** stays upright. A game that renders any shape, such as Ocarina of Time, fills the whole screen with the touch controls over it, as in landscape, or the space above them ([Portrait layout](#portrait-layout)); any other game shows its 16:9 picture above the controls (with a controller connected, as large as the screen allows). The screen turns as soon as you pick it, on the settings screen or in the in-game menu. + +### Portrait layout + +Games that render any shape (Ocarina of Time), upright on an iPhone or iPad. **Full screen**, the default, fills the whole screen with the game and draws the touch controls over the lower half of it, as they are over the picture in landscape. **Picture above controls** keeps the game in the upper half and the controls below it, with nothing under them. Either way the game renders exactly the space it has, with no bars; upright it shows no more height than a square picture would, so the world above stays as the game builds it. ### Frame-rate counter @@ -255,7 +259,7 @@ iCloud Drive takes a moment to carry a file from one device to the other. Quit t ## In-game menu -On the iPhone and iPad the gear at the top centre of the touch controls opens a menu while the game runs; it stays when a controller hides the other controls. It holds **Console mode**, **Picture**, **Rotation** and the **Frame-rate counter**, then **Controller layout** (**2D**, **3D**, and **Tap**, no controls and every touch to the game, in handheld mode), **Touch feedback** and **Game rumble**: the settings that take effect at once. A choice opens to show its options, the current one ticked. The menu's text keeps one size whatever the system text size, so it stays short over the game. A change is remembered as on the settings screen, which shows it at the next start; a setting `Documents/env.txt` fixes is shown, not changed. The other settings need the game to start again, so they are only on the settings screen: close the app fully and open it again. +On the iPhone and iPad the gear at the top centre of the touch controls opens a menu while the game runs; it stays when a controller hides the other controls. It holds **Console mode**, **Picture**, **Rotation**, **Portrait layout** (games that render any shape) and the **Frame-rate counter**, then **Controller layout** (**2D**, **3D**, and **Tap**, no controls and every touch to the game, in handheld mode), **Touch feedback** and **Game rumble**: the settings that take effect at once. A choice opens to show its options, the current one ticked. The menu's text keeps one size whatever the system text size, so it stays short over the game. A change is remembered as on the settings screen, which shows it at the next start; a setting `Documents/env.txt` fixes is shown, not changed. The other settings need the game to start again, so they are only on the settings screen: close the app fully and open it again. Console mode changes as on a Switch put in or taken out of its dock. While a TV shows the game it stays **Docked**, and the mode you pick applies when the TV is disconnected. **Docked** removes the Tap layout, as a docked Switch reads no touch screen. @@ -278,6 +282,7 @@ A line like `DM_PRESENT_SNAP=0` in `Documents/env.txt` (in the Files app, under | Picture (iPhone and iPad) | `DM_PICTURE_FIT` | `fit` 16:9 with black bars, `fill` covers the screen, cropped | | Picture shape | `DM_SCREEN_ASPECT` | Profiles with `screen_aspect on`: `auto` the app window's own shape where it is taller than 16:9 (iPad 4:3, iPhone Duo 1.43:1), else 16:9 (iOS only); `1.43` the iPhone Duo's shape, `0` 16:9. Profiles with `screen_aspect any` (Ocarina of Time): `auto` (the default) follows the Mac window while playing, any shape, or on iOS the space the picture has (the screen in landscape, upright the part above the touch controls); `1.777` 16:9 | | Rotation (iPhone and iPad) | `DM_ROTATION` | `auto` as the device turns, `landscape`, `portrait` | +| Portrait layout (iPhone and iPad) | `DM_PORTRAIT_LAYOUT` | `full` the game fills the screen under the controls, `split` the game above the controls | | Frame-rate counter | `DM_SHOW_FPS` | `1`, `0` | | Touch feedback | `DM_TOUCH_HAPTICS` | `1`, `0` | | Game rumble | `DM_RUMBLE` | `1`, `0` | From e0f33bd31d6fe29637c3777e74e14a4e13f2a536 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 02:23:18 +0100 Subject: [PATCH 21/40] screen_aspect any: the shape line no longer adds "screen shape 16:9" after every uncapped shape 55b30f60 put the upright cap's line before the 16:9 line's else, so the else followed the cap test instead of the shape test: an iPhone run logged "screen shape 1.072 ... camera x 1.000 y 0.603" then "screen shape 16:9" although the state page held 1.072. Only the log was wrong. --- dm/host/loader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dm/host/loader.c b/dm/host/loader.c index c1b481ba81..d8251aa316 100644 --- a/dm/host/loader.c +++ b/dm/host/loader.c @@ -368,8 +368,8 @@ static void screen_any_write_(double a, const char *why) { int first = g_screen_ratio < 0; g_screen_ratio = r; if (r) LOG("%s: screen shape %.3f (%s%s): %s, picture %.3f x %.3f of 16:9, camera x %.3f y %.3f%s", g_fps_mod, r, why, a != r && a > 0 ? ", held to 0.25..6" : "", r > 16.0 / 9 ? "more at the sides" : "more above and below", f[1], f[2], c[0], c[1], capped ? ", upright cap" : ""); - if (r && capped) LOG("%s: upright: vertical angle held to a %.2f picture's (DM_SCREEN_UPRIGHT_FOV)", g_fps_mod, cap); else LOG("%s: screen shape 16:9 (%s)", g_fps_mod, why); + if (capped) LOG("%s: upright: vertical angle held to a %.2f picture's (DM_SCREEN_UPRIGHT_FOV)", g_fps_mod, cap); if (!first) { void dm_display_changed(void); dm_display_changed(); } // the game lays its picture out again (hle_oe.c) } static void screen_any_apply(const char *as) { From 2e35fa7dcbe65f9c9182fea2be0fbf5bbbb28770 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 02:49:26 +0100 Subject: [PATCH 22/40] Docs: portrait play on the iPhone (Rotation, Portrait layout, the upright cap), measured on an iPhone 17 with Ocarina of Time The game note's Screen shape section follows the picture area upright (Full screen 0.46:1, capped; Picture above controls 1.07:1), records the upright cap and the HUD upright, and the six iPhone runs: every shape in proportion, the same frame rate upright and in landscape, and the one-frame present crop red line while the window textures grow. The iOS layouts reference entry gains the symptoms, the fix, the harness checks and the iPhone test; the symptom index is regenerated. --- dm/games/0400486024DEC000/README.md | 9 ++++++--- docs/agents/symptoms.md | 2 +- docs/reference/rendering.md | 8 ++++---- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index be3c551eac..01787d85fd 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -92,7 +92,7 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, ## Screen shape -Current. With `screen_aspect any` (launcher *Picture shape*, `DM_SCREEN_ASPECT`, `auto` by default) the game renders in the shape of the Mac window, live while playing, any shape from 0.25 to 6 (`loader.c` `SCREEN_MIN` / `SCREEN_MAX`, only to keep out a window a few pixels high). On iOS it follows the screen in landscape; upright it stays 16:9 (`front_ios.m`). The stubs are function-matched `others` of the frame-step rule in [`patches.py`](patches.py), found on v131072 `main` (build `e0f3ccfa…`): +Current. With `screen_aspect any` (launcher *Picture shape*, `DM_SCREEN_ASPECT`, `auto` by default) the game renders in the shape of the Mac window, live while playing, any shape from 0.25 to 6 (`loader.c` `SCREEN_MIN` / `SCREEN_MAX`, only to keep out a window a few pixels high). On iOS it follows the space the picture has (`front_ios.m`, [iOS layouts](../../../docs/reference/rendering.md#ios-layouts-portrait-a-half-folded-duo-ipad-windows)): the screen in landscape; upright ([Rotation](../../../docs/settings.md#rotation) Portrait on an iPhone, or an iPad turned) the whole screen with the touch controls over its lower half ([Portrait layout](../../../docs/settings.md#portrait-layout) Full screen, the default; 0.46:1 on an iPhone 17), or the part above the controls (Picture above controls; 402x375 points, 1.07:1). The stubs are function-matched `others` of the frame-step rule in [`patches.py`](patches.py), found on v131072 `main` (build `e0f3ccfa…`): | Label | Function | What the stub does | |---|---|---| @@ -103,11 +103,14 @@ Current. With `screen_aspect any` (launcher *Picture shape*, `DM_SCREEN_ASPECT`, | `hud upload` | `main+0x2d2ba0` (+0x4c) | the screen draw's copy of its uniform block (both orthographic matrices and the layout size): rows x / y × the HUD's multipliers, the size ÷ them, each frame | | `layout size` | `main+0x583fd0`, the getter of a layout file's screen size | the size the HUD draws with, for the game's conversions between the layout space and the screen; the screen's init (`screen init`, `main+0x773b90`) still gets the file's | -- **The rule.** Landscape wider than 16:9 keeps the vertical angle and shows more at the sides; between 1:1 and 16:9 it keeps the horizontal angle and shows more above and below. Upright takes the scale of the same screen turned landscape: its width gets what that screen's height gets, so turning a phone or a window keeps the characters' size and shows another part of the world. The output keeps the requested height (wider) or width (taller), scaled down to 1.5× the requested pixels past that (a phone upright renders 1280x2768). +- **The rule.** Landscape wider than 16:9 keeps the vertical angle and shows more at the sides; between 1:1 and 16:9 it keeps the horizontal angle and shows more above and below. Upright takes the scale of the same screen turned landscape: its width gets what that screen's height gets, so turning a phone or a window keeps the characters' size and shows another part of the world, up to the upright cap below. The output keeps the requested height (wider) or width (taller), scaled down to 1.5× the requested pixels past that (a phone upright renders 1280x2768). - **Why the first attempts changed nothing:** the projection is cached ([a camera that caches its matrix](../../../docs/reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix)): `PerspectiveProjection::set`, `tanf` and the 16/9 literals do not run in play. Found with `DM_PROBE_FINDMAT` and the write watch on the render camera's matrix. - **The HUD** (eui: layouts `UI/Layout2Archive/*.blarc.zs`). Every layout file holds a screen size of 3840x2160 (LayoutResData +0x200, file offset 0x2f8); each screen builds its orthographic projection from it once and copies it to the GPU each frame, so the HUD was drawn in a 16:9 space stretched over the output. The `hud upload` stub gives that space the output's shape at the HUD's pixel size: its multipliers are the camera's divided by the largest of them and 1, so in landscape the HUD keeps the size it has at 16:9 of the same height or width, and upright the whole 16:9 space still fits the width (smaller, nothing cut off). Checked on the Mac with live resizes (iPhone and iPad shapes upright and landscape, 1700x478, 16:9), the pause menu, the "A Open" prompt and the Kokiri Forest cutscene with its location card at 2.2. - **Open: the HUD's edges.** eui places every pane about the screen's centre and does not anchor to its edges (a copy of PlayerStatus_00 made 4752 wide drew round hearts at their 16:9 place), so the HUD keeps its 16:9 arrangement in the middle: in wide shapes the hearts are at the 16:9 band's left edge, not the window's; upright the HUD sits around mid-height. The layouts mark their edge groups with 60x80 null panes at the space's edges (PlayerStatus_00's `N_ScaleL_00` at (-1920, 1078), `N_ScaleR_00` at (1920, 1078); MenuOverlay_00's at (0, ±1000)), and full-screen panes are exactly 3840x2160 at the centre (EventFade_00, LoadingFade_00, MenuBG_00): moving the first to the screen's edges and stretching the second needs eui's per-frame pane transform, not found yet. A pane instance (0xb0 bytes in the layout's array, layout +0x100) holds size +0x58, scale +0x60, rotation +0x68, translation +0x6c, a local rect +0x84..+0x90 and flags +0xa8, but no world matrix and no link to its record in the file. Until then the pause menu's backdrop and fades cover the 16:9 middle only, with the scene at the sides. -- **Open:** upright on iOS (16:9 for now); upright, a phone sees about 90° vertically, so parts of the world the game never shows (above Link's house) appear unbuilt: a cap on the vertical angle was offered, not decided. The title movie and other videos stay 16:9 with bars. +- **The upright cap** (current). Upright, the rule alone gave a phone (0.46:1) about 90° vertically against the game's 50° at 16:9, so parts of the world the game never shows (above Link's house) appeared unbuilt. The loader now holds an upright picture to the vertical angle of a square (79°), the width narrowing instead (`loader.c` `screen_upright_cap`; `DM_SCREEN_UPRIGHT_FOV`: `0` off, `1.777` the game's own angle). Portrait layout Full screen (0.46:1 on an iPhone) and a tall Mac window are capped; the picture above an iPhone's controls (1.07:1) is landscape and is not. The HUD's multipliers do not change with it. +- **The HUD upright** (current, nothing changed). The 16:9 HUD space fits the picture's width and sits at mid-height: full screen on an iPhone a 402x226-point band at mid-height, about where the controls' top edge is; above the controls one in the 402x375 picture. The `hud upload` stub can only scale that space as a whole, and larger would cut off its sides; moving the hearts and buttons to the picture's edges is the open item above. +- **Upright on the iPhone 17** (current, 2026-11-10; 3-minute runs to Kokiri Forest, `$W/ipad/device/pD_portrait_full`, `pE_portrait_split`, `pF_landscape`, and `pA`..`pC` before Portrait layout existed). Rotation Portrait turned the phone upright at launch (`front end: rotation portrait`). Portrait layout Full screen (`layout portrait full screen, controls over the picture`) rendered the whole screen's 0.46:1 as 1200x2600 (camera x 2.174 y 0.562, `upright cap`): Link in proportion and the trees above him fully built. Picture above controls rendered 1.072:1 (a 1824x1704 crop, camera x 1.000 y 0.603, not capped). Rotation Landscape gave 2352x1080 (2.178:1). Each run was in proportion, with the hearts at the left of the 16:9 band. The frame rate was the same upright and in landscape (13-21 fps at thermal state 2 in all six), so the shape costs nothing measurable. The screenshots are the game's texture: the touch controls over the full-screen picture are seen only on the phone. A first upright shape logs one `[RED] present crop out of bounds` (1824x1704 against 2560x1440): `window crop` grows the textures at setCrop, they are re-created at the next acquire, and the present between them is dropped. A single frame, so left as it is. +- **Open:** the title movie and other videos stay 16:9 with bars. ## Saves diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index aa3337a0eb..853ea01504 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -915,7 +915,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - a program whose shaders were deleted after `glAttachShader` and before `glLinkProgram` logs `program N: a shader is missing from the library, its draws are skipped`, or links a shader of the wrong stage; draws after `glDeleteProgram` of the current program log `draw with no usable program (0)`. → [GL shaders and programs deleted while in use](../reference/rendering.md#gl-shaders-and-programs-deleted-while-in-use) - an image-only fragment draw's writes disappear from the subresource also bound as its color attachment when the pass ends; other image layers retain their writes. → [Image stores into an attachment of their own pass](../reference/rendering.md#image-stores-into-an-attachment-of-their-own-pass) - read-only depth/stencil attachments are written back after every pass, and the periodic unwritten-store savings counter remains zero with the default configuration. → [Unwritten attachment stores without lookahead](../reference/rendering.md#unwritten-attachment-stores-without-lookahead) -- portrait gameplay overlaps the touch controls, touch coordinates land outside the game picture, or a TV's picture renders at 4K again after a layout pass. → [iOS layouts: portrait, a half-folded Duo, iPad windows](../reference/rendering.md#ios-layouts-portrait-a-half-folded-duo-ipad-windows) +- portrait gameplay overlaps the touch controls, touch coordinates land outside the game picture, or a TV's picture renders at 4K again after a layout pass; an iPhone cannot be played upright; upright, a game that renders any shape shows a small 16:9 picture with bars above the controls. → [iOS layouts: portrait, a half-folded Duo, iPad windows](../reference/rendering.md#ios-layouts-portrait-a-half-folded-duo-ipad-windows) - with *Picture shape* at 1.43:1 (`DM_SCREEN_ASPECT`) the game renders 1920x1344, but on an iPhone Duo in Fit (the default) the picture comes out smaller than the 16:9 one, with black bars on all four sides; it fills the screen only in Fill. On the Mac a click lands off the point under the pointer (it was mapped on a 16:9 rectangle of the window). → [The picture laid out in the shape the game presents](../reference/rendering.md#the-picture-laid-out-in-the-shape-the-game-presents) - with a wider or taller output (`DM_SCREEN_ASPECT`) the picture is stretched although a stub changes the aspect `sead::PerspectiveProjection::set` receives, or the 16:9 constants the code holds; the HUD stays stretched over the output too. → [A screen-shape stub on the projection setter changes nothing: the camera caches its matrix](../reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix) diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index 156968ae10..a6b2303766 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -1849,11 +1849,11 @@ Digimon Story Cyber Sleuth: Complete Edition (Media.Vision's MVGL engine, SDK 6. ### iOS layouts: portrait, a half-folded Duo, iPad windows -- **Symptom:** portrait gameplay overlaps the touch controls, touch coordinates land outside the game picture, or a TV's picture renders at 4K again after a layout pass. +- **Symptom:** portrait gameplay overlaps the touch controls, touch coordinates land outside the game picture, or a TV's picture renders at 4K again after a layout pass; an iPhone cannot be played upright; upright, a game that renders any shape shows a small 16:9 picture with bars above the controls. - **Cause:** the front end assumed a landscape screen with the controls over the whole view, so its picture and touch geometry did not follow a portrait screen (an iPad, an iPhone Duo's inner display) or a Duo's fold. On iPadOS the deprecated `UIRequiresFullScreen` kept the game out of resizable windows, and a native rotation A/B on the iPhone Duo iOS 27.1 simulator showed that it also blocked upside-down portrait. -- **Fix:** `dm/ios/Sources/DmLayout.h` computes the picture and controls rectangles from the view's bounds, its safe area and UIKit's division region (`UIHingeInteraction`, iOS 27.1); front_ios.m lays the views out from them and DmGamepad.m its controls. Landscape keeps the overlay unless the hinge is half open over a horizontal division; portrait puts a fitted 16:9 picture above the controls, at the division or the midpoint. On a Duo in portrait only the top safe-area inset applies. Game touches map through the picture rectangle (with [Fill](../settings.md#picture), larger than the view), and a layout change releases held inputs. The phone's orientation list stays landscape (the Duo's inner display follows its bounds), the iPad's has all four and no `UIRequiresFullScreen`. While a TV shows the picture the phone is all controls (`dm_layout_pad_only`). The drawable's scale is set in one place, `DmGameView layoutSubviews`: the screen's pixels, at most `maxPixelWidth` across, which the TV sets to 1920, so a later layout pass cannot undo the TV's 1080p cap. -- **Check:** the [Duo harness](../../dm/ios/tests/duo/README.md) tests layout, hit targets, touches, the in-game menu's live settings and the drawable cap in the Simulator with synthetic host state. A physical Duo was tested by the PR #683 author on their fork; full-game tests on a physical iPad and a Duo with this repository's build are open. -- **Found on:** [Pokémon Brilliant Diamond](../../dm/games/0100000011D90000/README.md). +- **Fix:** `dm/ios/Sources/DmLayout.h` computes the picture and controls rectangles from the view's bounds, its safe area and UIKit's division region (`UIHingeInteraction`, iOS 27.1); front_ios.m lays the views out from them and DmGamepad.m its controls. Landscape keeps the overlay unless the hinge is half open over a horizontal division; portrait puts a fitted 16:9 picture above the controls, at the division or the midpoint. On a Duo in portrait only the top safe-area inset applies. Game touches map through the picture rectangle (with [Fill](../settings.md#picture), larger than the view), and a layout change releases held inputs. The phone's Info.plist orientation list stays landscape (the Duo's inner display follows its bounds), the iPad's has all four and no `UIRequiresFullScreen`. [Rotation](../settings.md#rotation) (`DM_ROTATION`: `auto`, `landscape`, `portrait`) is answered by the app delegate's `application:supportedInterfaceOrientationsForWindow:` (main.m, front_ios.m `front_ios_app_orientations`), which replaces that list, so a phone can be locked upright; `auto` returns the list itself, so nothing changes by default. The root view controller and the launcher's `DmHostingController` allow every orientation under `auto` and the locked ones otherwise (`dm_rotation_vc_mask`); a pick on the settings screen or in the in-game menu calls `setNeedsUpdateOfSupportedInterfaceOrientations` on every view controller and, when the window's orientation is no longer allowed, `requestGeometryUpdate`, which turns it at once. `DmLayout.area` is where the picture may go: the view in the overlay layout, the part above the controls in a split, the whole safe area upright while a controller hides the controls (`dm_layout_controls_hidden`; a half-folded Duo keeps its split). A profile with `screen_aspect any` is handed that area's shape (`front_view_aspect`, `dm_screen_view_changed`; on a TV the TV's). Upright it follows [Portrait layout](../settings.md#portrait-layout) (`DM_PORTRAIT_LAYOUT`): `full`, the default, makes the area the whole view and keeps the controls in the lower half, drawn over the picture (`dm_layout_full_picture`), so the game renders the screen's shape (0.46:1 on an iPhone); `split` renders the space above the controls (402x375 points, 1.07:1, on an iPhone 17). Other profiles keep the 16:9 picture fitted above the controls. Fill applies only to a landscape view. While a TV shows the picture the phone is all controls (`dm_layout_pad_only`). The drawable's scale is set in one place, `DmGameView layoutSubviews`: the screen's pixels, at most `maxPixelWidth` across, which the TV sets to 1920, so a later layout pass cannot undo the TV's 1080p cap. +- **Check:** the [Duo harness](../../dm/ios/tests/duo/README.md) tests layout, hit targets, touches, the in-game menu's live settings, the drawable cap, the Rotation masks with a real turn of the Simulator's window, and the picture area's shape handed to the host, in the Simulator with synthetic host state. A physical Duo was tested by the PR #683 author on their fork; Portrait play (Rotation, both Portrait layouts, the upright cap) ran on an iPhone 17 with Ocarina of Time ([game note](../../dm/games/0400486024DEC000/README.md#screen-shape)); full-game tests on a physical iPad and a Duo with this repository's build are open. +- **Found on:** [Pokémon Brilliant Diamond](../../dm/games/0100000011D90000/README.md); portrait play and the picture area's shape on [Ocarina of Time](../../dm/games/0400486024DEC000/README.md#screen-shape). ### The picture laid out in the shape the game presents From adb643108d5a68c5fdab64a52a403b0e9d8dace1 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:32:07 +0100 Subject: [PATCH 23/40] OoT-iPhone A-2: a submission's end hands its records to the encode thread and the replay goes on, so the GPU no longer idles before each submission With Metal encoding on its own thread (the iOS default), mtl_sub_commit waited for the encode thread to finish the submission (rec_drain). Ocarina of Time makes about 30 submissions a frame (the queue scheduler cuts one at every event signal), each under the 64-draw segment size, so the GPU got each one only after the replay had waited for the encode thread and then translated it: on the iPhone 17 frame 3600 spanned 34.9 ms for 20.2 ms of GPU work, 13.6 ms of the gaps before a submission's first encoder, and the replay waited for the encode thread 36 times a frame (10 ms). Now the submission's records go to the encode thread with a note of its fence (RO_NOTE on g_sub_cmt) and the replay goes on. What follows on the queue is a record too; a CPU wait for the GPU waits for the fence's signal behind the commit; graphics suspend waits for the last fence. The display command buffer of a present without a snapshot, the one direct commit that reads the game's work, waits for the note of the submissions before it (sub_commit_wait). DM_NVN_SUB_DRAIN=1 brings the old wait back, ab alternates with DM_AB. t_sub_overlap holds the encode thread inside a submission and checks the submission returns, the note and a present's wait hold until the commit (3 of 8 checks fail with the old wait). With DM_NVN_ENC_THREAD=1: t_reorder_chain, t_defer_chain, t_copy_alias, t_unwritten_store, t_alias_takeover, t_rt_range, t_clear_tex, t_clear_ds, t_upgrade_alias, t_dispatch_overlap, t_perf_order, t_present_crop and check_t_present_snap pass; t_feedback's 3 segment-timestamp checks fail the same with DM_NVN_SUB_DRAIN=1 (the encode thread writes no segment timestamps, frame-rate.md), so not from this change. Left: the iPhone A/B (DM_NVN_SUB_DRAIN=ab) and the regress games. --- dm/gfx/nvn/nvn_mtl.m | 20 +++++++++++---- dm/gfx/tests/Makefile | 2 ++ dm/gfx/tests/t_sub_overlap.m | 49 ++++++++++++++++++++++++++++++++++++ docs/agents/symptoms.md | 1 + docs/reference/rendering.md | 8 ++++++ 5 files changed, 75 insertions(+), 5 deletions(-) create mode 100644 dm/gfx/tests/t_sub_overlap.m diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 43b01223e1..f97213c4a5 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -278,6 +278,7 @@ static void ws_report(uint64_t frame, uint64_t frames) { if (n && o < sizeof line - 160) o += snprintf(line + o, sizeof line - o, " %s %.2f/frame (blocked %.2f, %.3f ms);", ws_name[i], (double)n / frames, (double)b / frames, ns / 1e6 / frames); } MLOG("[waits] up to frame %llu, per frame:%s", (unsigned long long)frame, o ? line : " none"); } +static _Atomic uint64_t g_sub_cmt; // the last submission whose commit the encode thread has made (its fence; with no encode thread, the last committed: mtl_sub_commit) #include "nvn_rec.h" // deferred encoding (DM_NVN_ENC_THREAD) static NSHashTable *g_res_objects; static char g_tex_range_key; // res_top: a GPU-made texture's storage (guest address, NVN size, pool flags) @@ -2670,7 +2671,14 @@ uint64_t mtl_sub_commit(void) { [g_cb endCommandBuffer]; rec_commit(g_cb, o); g_cbgen++; // the stores it holds come before any later upload batch (gsb_held) if (!lost_test) rec_signal(g_ev, fence); - rec_drain(); + // With the encode thread the records go to it and the replay goes on with the next submission: what comes after it on the queue is a record too (the next submission's commits, + // upload batches, snapshot copies), a CPU wait for the GPU waits for the fence's signal behind this commit, and a present of the game's own texture waits for this note + // (sub_commit_wait). The replay used to wait for the encode thread here: Ocarina of Time makes about 30 submissions a frame (one at each event signal), and on the iPhone the GPU + // sat idle before most of them while the replay waited and then translated the next one (13.6 of 14.8 ms of gaps in a 35 ms frame; docs/reference/rendering.md) + // DM_NVN_SUB_DRAIN=1 waits for the encode thread here as before (ab: alternates with the DM_AB phases, for measurement) + static int drain = -1; if (drain < 0) drain = sw_mode("DM_NVN_SUB_DRAIN", 0); + if (rec_on() && g_rb_started) { rec_t *r = rec_put(RO_NOTE, 0); r->a[0] = (uint64_t)(uintptr_t)&g_sub_cmt; r->a[1] = fence; if (drain && sw_on(drain)) rec_drain(); else rec_publish(); } + else atomic_store(&g_sub_cmt, fence); g_alloc_fence[g_alloc_cur] = fence; g_cb = nil; atomic_store(&g_sub_on, 0); // after ++g_fence and the last encoding: a buffer retired from now on waits for this fence at most @@ -2680,6 +2688,8 @@ uint64_t mtl_sub_commit(void) { return fence; } +// A present of the game's own texture waits for the submissions before it (g_sub_cmt), as the display command buffer goes to the queue directly and must come after the work that drew the picture +static void sub_commit_wait(uint64_t fence) { if (fence && rec_on()) rec_note_wait(&g_sub_cmt, fence); } uint64_t mtl_completed_fence(void) { return atomic_load_explicit(&g_done,memory_order_acquire); } uint64_t mtl_last_fence(void) { return g_fence; } int mtl_last_commit_error(void) { return atomic_load(&g_last_commit_error); } @@ -16892,7 +16902,7 @@ static int present_now(const tex_t *t, double period, uint64_t frame, uint64_t t // Swapchain texture return: before AcquireTexture gets a texture it waits for the ticket of that texture's previous present to be processed (mtl_present_wait), after which the commands the game draws into it // are necessarily ordered after that blit on the GPU queue. #define PQ 8 -typedef struct { tex_t t; double period; uint64_t frame; int crop[4]; double t_in; uint64_t ev_val, snap_v; } pq_item; // snap_v: the snapshot copy this present shows (snap_commit_wait) // t_in: the time the game handed this frame into the present queue (CACurrentMediaTime); ev_val: see g_ext_ev +typedef struct { tex_t t; double period; uint64_t frame; int crop[4]; double t_in; uint64_t ev_val, snap_v, sub_v; } pq_item; // snap_v: the snapshot copy this present shows (snap_commit_wait); sub_v: the submissions a present of the game's own texture follows (sub_commit_wait) // t_in: the time the game handed this frame into the present queue (CACurrentMediaTime); ev_val: see g_ext_ev // External producer (the Vulkan layer, dm/gfx/vk) renders on its own queue and signals this event when a frame's rendering is done: the present of that frame waits for the value on g_q4 static id<MTLSharedEvent> g_ext_ev; static void ext_wait(uint64_t v) { if (v && g_ext_ev) [g_q4 waitForEvent:g_ext_ev value:v]; } @@ -16964,7 +16974,7 @@ - (void)metalDisplayLink:(CAMetalDisplayLink *)link needsUpdate:(CAMetalDisplayL pthread_cond_broadcast(&g_pq_cv); pthread_mutex_unlock(&g_pq_mu); int submitted; - ext_wait(it.ev_val); snap_commit_wait(it.snap_v); + ext_wait(it.ev_val); snap_commit_wait(it.snap_v); sub_commit_wait(it.sub_v); @autoreleasepool { submitted = present_now(&it.t, it.period, it.frame, ticket, it.crop, update.drawable, update.targetPresentationTimestamp); mtl_tex_unpin(it.t.serial); } g_link_hold_until = submitted && it.period > 0.025 ? update.targetPresentationTimestamp + it.period : 0; if (!submitted) { @@ -17104,7 +17114,7 @@ static void present_drop_for_suspend(void) { uint64_t ticket = ++g_pq_read; g_classic_busy = 1; pthread_mutex_unlock(&g_pq_mu); int submitted; - ext_wait(it.ev_val); snap_commit_wait(it.snap_v); + ext_wait(it.ev_val); snap_commit_wait(it.snap_v); sub_commit_wait(it.sub_v); @autoreleasepool { submitted = present_now(&it.t, it.period, it.frame, ticket, it.crop, nil, 0); mtl_tex_unpin(it.t.serial); } pthread_mutex_lock(&g_pq_mu); g_classic_busy = 0; pthread_cond_broadcast(&g_pq_cv); pthread_mutex_unlock(&g_pq_mu); if (!submitted) { @@ -17275,7 +17285,7 @@ static uint64_t present_enqueue(const tex_t *t, double period, const int crop[4] mtl_tex_pin(t->serial); pthread_mutex_lock(&g_pq_mu); while (g_pq_in - g_pq_done >= PQ) pthread_cond_wait(&g_pq_cv, &g_pq_mu); // Wait only when the queue is full; normally AcquireTexture blocks first - pq_item *it = &g_pq[g_pq_in % PQ]; *it = (pq_item){*t, period, atomic_load(&g_cur_frame), {0}, CACurrentMediaTime(), ev_val, ss ? g_snap[slot].ev : 0}; + pq_item *it = &g_pq[g_pq_in % PQ]; *it = (pq_item){*t, period, atomic_load(&g_cur_frame), {0}, CACurrentMediaTime(), ev_val, ss ? g_snap[slot].ev : 0, !ss && !ev_val && on_sub_thread() ? g_fence : 0}; if (crop) memcpy(it->crop, crop, sizeof it->crop); // The current frame's state is enqueued with the ticket number, rather than reading the mutable window late on the present thread uint64_t tk = ++g_pq_in; if (ss) { g_snap[slot].ticket = tk; atomic_fetch_add(&g_snap_next, 1); if (copy) *copy = g_snap[slot].ev; } diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 697f329682..21dd1f6b70 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -242,6 +242,8 @@ t_ub: t_ub.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_ub.m $(MTL_T_SRC) $(FW) t_gpuprof: t_gpuprof.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_gpuprof.m $(MTL_T_SRC) $(FW) +t_sub_overlap: t_sub_overlap.m $(MTL_T_DEP) + clang $(CFLAGS) -fobjc-arc -o $@ t_sub_overlap.m $(MTL_T_SRC) $(FW) t_dispatch_own: t_dispatch_own.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_dispatch_own.m $(MTL_T_SRC) $(FW) .PHONY: check_t_dispatch_own diff --git a/dm/gfx/tests/t_sub_overlap.m b/dm/gfx/tests/t_sub_overlap.m new file mode 100644 index 0000000000..646736b326 --- /dev/null +++ b/dm/gfx/tests/t_sub_overlap.m @@ -0,0 +1,49 @@ +// With Metal encoding on its own thread (DM_NVN_ENC_THREAD), a submission's end hands its records to the encode thread and returns: the replay translates the next submission while +// the encode thread finishes this one. It used to wait for the encode thread (rec_drain) at the end of every submission, and Ocarina of Time makes about 30 a frame (one at each event +// signal), so on the iPhone the GPU sat idle before most of them (13.6 of 14.8 ms of gaps in a 35 ms frame). What needs the work committed waits for the submission's note +// (g_sub_cmt): a present of the game's own texture (no snapshot), before its display command buffer. The encode thread is held inside a submission by a queue block here. +#include "../nvn/nvn_mtl.m" +void *front_metal_layer(void) { return NULL; } +module_t *module_of(const void *p) { (void)p; return NULL; } +module_t *g_mod; int g_nmod; +void svc_thread_dump(void) {} +uint64_t now_ns(void) { return bp_now(); } +static int checks, failed; +#define CHECK(name, ok) do { int pass = !!(ok); checks++; failed += !pass; printf("%s: %s\n", name, pass ? "PASS" : "FAIL"); } while (0) + +static dispatch_semaphore_t g_gate, g_sub_done, g_wait_done; +static _Atomic uint64_t g_fence_out; +static void *submit_thread(void *arg) { // one submission whose encoding the gate holds + (void)arg; @autoreleasepool { + state_reset(); memset(&R, 0, sizeof R); mtl_sub_begin(); + dispatch_semaphore_t gate = g_gate; rec_queue_block(^{ dispatch_semaphore_wait(gate, DISPATCH_TIME_FOREVER); }); + atomic_store(&g_fence_out, mtl_sub_commit()); dispatch_semaphore_signal(g_sub_done); + } return NULL; +} +static void *present_wait_thread(void *arg) { sub_commit_wait((uint64_t)(uintptr_t)arg); dispatch_semaphore_signal(g_wait_done); return NULL; } + +int main(void) { @autoreleasepool { + alarm(20); setbuf(stdout, NULL); + setenv("DM_NVN_ENC_THREAD", "1", 1); setenv("DM_PRESENT_SNAP", "0", 1); + mtl_init(); + state_reset(); memset(&R, 0, sizeof R); mtl_sub_begin(); uint64_t f0 = mtl_sub_commit(); + CHECK("a submission completes on the GPU", mtl_wait(f0, 2000000000ull)); + CHECK("its commit note is passed", atomic_load(&g_sub_cmt) >= f0); + + g_gate = dispatch_semaphore_create(0); g_sub_done = dispatch_semaphore_create(0); g_wait_done = dispatch_semaphore_create(0); + pthread_t th; pthread_create(&th, NULL, submit_thread, NULL); + long returned = dispatch_semaphore_wait(g_sub_done, dispatch_time(DISPATCH_TIME_NOW, 500 * NSEC_PER_MSEC)) == 0; + CHECK("the submission returns while the encode thread is still encoding it", returned); + uint64_t f1 = returned ? atomic_load(&g_fence_out) : f0 + 1; + CHECK("its commit note is not passed before the encode thread commits it", atomic_load(&g_sub_cmt) < f1); + pthread_t pw; pthread_create(&pw, NULL, present_wait_thread, (void *)(uintptr_t)f1); + CHECK("a present of the game's texture waits for that commit", dispatch_semaphore_wait(g_wait_done, dispatch_time(DISPATCH_TIME_NOW, 200 * NSEC_PER_MSEC)) != 0); + dispatch_semaphore_signal(g_gate); + if (!returned) dispatch_semaphore_wait(g_sub_done, DISPATCH_TIME_FOREVER); + pthread_join(th, NULL); + CHECK("the present goes on once the encode thread has committed it", dispatch_semaphore_wait(g_wait_done, dispatch_time(DISPATCH_TIME_NOW, 2000 * NSEC_PER_MSEC)) == 0); + pthread_join(pw, NULL); + CHECK("the held submission completes on the GPU", mtl_wait(atomic_load(&g_fence_out), 2000000000ull)); + CHECK("its commit note is passed", atomic_load(&g_sub_cmt) >= atomic_load(&g_fence_out)); + printf("t_sub_overlap: %d checks, %d failed\n", checks, failed); return failed != 0; +} } diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 853ea01504..a3527419a7 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -846,6 +846,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - `[RED] texture→texture copy case not implemented: 1920x1080x1→1920x1080x1 flags 0 format 260→252 3D 0→0`. → [Depth plane copies between depth formats](../reference/rendering.md#depth-plane-copies-between-depth-formats) - `FATAL deferred encoding: DmRecEnc has no stand-in for -copyFromTexture:sourceSlice:sourceLevel:sourceOrigin:sourceSize:toBuffer:destinationOffset:destinationBytesPerRow:destinationBytesPerImage:options:` with Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the default on iOS), at the first depth plane copy. → [Depth plane copies on the encode thread](../reference/rendering.md#depth-plane-copies-on-the-encode-thread) - `FATAL deferred encoding: DmRecEnc has no stand-in for -drawMeshThreadgroupsWithIndirectBuffer:threadsPerObjectThreadgroup:threadsPerMeshThreadgroup:`, then `CRASH sig=6` under `mtk_encode`, with Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the default on iOS), at the first indirect task / mesh draw. → [Indirect task / mesh draws on the encode thread](../reference/rendering.md#indirect-task--mesh-draws-on-the-encode-thread) +- with Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the default on iOS), a frame's GPU span is far above its GPU work, and the gaps sit before the first encoder of a submission: `DM_NVN_GPUPROF` with its CPU timeline shows the GPU done with everything committed while the next submission is still being encoded. The wait ledger (`DM_NVN_WAITS=1`) shows `encode thread drain` blocked about once per submission. → [The GPU idles before each submission while the replay waits for the encode thread](../reference/rendering.md#the-gpu-idles-before-each-submission-while-the-replay-waits-for-the-encode-thread) - a GPU page fault (`kIOGPUCommandBufferCallbackErrorPageFault`) 30 to 40 s after a save loads, then the game hangs or stops. `DM_NVN_SKIP_MESH=1` makes it go away. → [Mesh draws with no task shader above 1024 workgroups](../reference/rendering.md#mesh-draws-with-no-task-shader-above-1024-workgroups) - a scene opens near white and fades to its normal colours over several seconds, at each scene change, while the settled picture is right. → [Pending clears land before a compute dispatch](../reference/rendering.md#pending-clears-land-before-a-compute-dispatch) - a regular lattice of dots over the ground, and the light wrong: three of four layers of an interleaved ambient occlusion array stay empty, while the draws and their bindings look right. → [`[[primitive_id]]` in a specialized Metal 4 pipeline](../reference/rendering.md#primitive_id-in-a-specialized-metal-4-pipeline) diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index a6b2303766..9d61f44279 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -1207,6 +1207,14 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Fix:** `nvn_rec.h` records the indirect draw (`RO_R_MESH_IND`: the argument buffer's GPU address and the threads per object and mesh threadgroup, as `RO_R_MESH` keeps them) and replays it. `t_rec_draws` checks that the recorder has a stand-in for every draw the NVN layer encodes. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md): the iOS app aborted when a save loaded; on the Mac with the encode thread on, 28 s after the title ([issue #1265](https://github.com/seedds/nxnative/issues/1265)). +### The GPU idles before each submission while the replay waits for the encode thread + +- **Symptom:** with Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the default on iOS), a frame's GPU span is far above its GPU work, and the gaps sit before the first encoder of a submission: `DM_NVN_GPUPROF` with its CPU timeline shows the GPU done with everything committed while the next submission is still being encoded. The wait ledger (`DM_NVN_WAITS=1`) shows `encode thread drain` blocked about once per submission. +- **Cause:** `mtl_sub_commit` waited for the encode thread to finish the submission (`rec_drain`) before returning, so the replay never translated the next submission while the encode thread encoded the last one, and the GPU got each submission only once both had finished it. A game that splits its frame at every event signal makes many submissions (each `nvnQueueSubmitCommands` is cut into one per wait, signal and fence by the queue scheduler, `nq_flatten`), and a submission under `cut_draws()` draws has no [segment](frame-rate.md#submissions-handed-to-the-gpu-in-segments) to start the GPU early. +- **Fix:** the submission's records go to the encode thread with a note of its fence (`RO_NOTE` on `g_sub_cmt`), and the replay goes on. Everything else that goes to the queue after it is a record too (the next submission's commits, upload batches, snapshot copies, sparse mappings), a CPU wait for the GPU waits for the fence's signal, which comes after the commit, and graphics suspend waits for the last fence. The one direct commit that reads the game's work is the display command buffer of a present without a snapshot: it waits for the note of the submissions before it (`sub_commit_wait`, `pq_item.sub_v`). `DM_NVN_SUB_DRAIN=1` brings the wait back (`ab` alternates with `DM_AB`). +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) on the iPhone 17 (Kokiri Forest, 2026-11-10): about 30 submissions a frame; frame 3600 spanned 34.9 ms for 20.2 ms of GPU work, and 13.6 of its 14.75 ms of gaps came before a submission's first encoder; the replay thread waited for the encode thread 36 times a frame, 10 ms in all. +- **Check:** `make -C dm/gfx/tests GAME=<id> t_sub_overlap && dm/gfx/tests/t_sub_overlap` holds the encode thread inside a submission (a queue block) and checks that the submission returns, that its note waits for the encode thread's commit, and that a present's wait holds until then (3 of 8 checks fail with the old wait). `check_t_present_crop` and `check_t_present_snap` run presents with and without the encode thread and snapshots. + ### Mesh draws with no task shader above 1024 workgroups - **Symptom:** a GPU page fault (`kIOGPUCommandBufferCallbackErrorPageFault`) 30 to 40 s after a save loads, then the game hangs or [stops](#a-gpu-whose-access-macos-revoked-stops-the-game). `DM_NVN_SKIP_MESH=1` makes it go away. From e73aaac1e53a81db2aa85c4e432243e5c952f39d Mon Sep 17 00:00:00 2001 From: uncraftablexd <352372032+uncraftablexd@users.noreply.github.com> Date: Mon, 9 Nov 2026 04:02:14 +0000 Subject: [PATCH 24/40] Frame step: late frames with the GPU idle do not step up, and a longer hold the GPU does not explain is probed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On a warm start Ocarina of Time stepped hold 1 → 2 → 3 at presents 9 and 12 (cost 1.92 and 5.57 vsyncs, GPU 0.02 and 0.01) on a progress screen while the pipeline archive was written. The game waits for its frames outside the window's acquire, so its cost at hold 3 was about the hold itself (54 ms, GPU 12.5 ms): the fit trial waited and no probe fired, and it stayed at 18 fps for about 1,470 presents. - nvn_fstep.h: the caller reports each frame's GPU time (gpu_on, gpu). A late frame with the GPU under FSTEP_GPU_IDLE (0.05 of a frame) neither counts toward the 3 late frames of a step up nor ends their run; FSTEP_RUN late frames in a row still step up - nvn_fstep.h: FSTEP_RUN frames in a row at a hold above kmin whose GPU time fits k − 1 and that waited for nothing in the host's pacing start a probe (nvn_fstep_unpaced), with their own hold-off (uhold) doubled after each failure up to 60 s and back to 1 s when kmin holds - hle_nvn.c fstep_tick: passes the GPU time; the step log names the idle-GPU step up and the new probe, and its appends clamp their length - t_fstep: idle-GPU late frames, the unpaced probe (also with the GPU clock falling at hold 3), CPU-bound games at hold 2, 2.95 and 3.2 vsyncs, a CPU-heavy scene with the GPU idle (5 of 88 checks fail without the change) - docs/reference/frame-rate.md: new entry; symptom index; Ocarina of Time's Found-on line Rebased onto main's average rule (1e3fd031): idle-GPU late frames and FSTEP_RUN late frames in a row both still need the last FSTEP_RUN frames at the hold to fall short of it on average (short_of_k), the new hold is the cheapest of the three late frames' cost, and a fixed step is held within the stub's smax. t_fstep: main's 86 checks and this commit's, 102 in all, pass. --- dm/games/0400486024DEC000/README.md | 1 + dm/gfx/nvn/hle_nvn.c | 15 ++++--- dm/gfx/nvn/nvn_fstep.h | 52 ++++++++++++++++++----- dm/gfx/tests/t_fstep.c | 65 +++++++++++++++++++++++++++++ docs/agents/symptoms.md | 1 + docs/reference/frame-rate.md | 9 ++++ 6 files changed, 127 insertions(+), 16 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 01787d85fd..225265eedd 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -194,6 +194,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [A streamed texture's first read drew its surface dark for one frame](../../../docs/reference/rendering.md#a-streamed-textures-first-read-drew-its-surface-dark-for-one-frame): resolved, the cliff beside the castle town wall in Hyrule Field darker for one frame as the game streamed it a sharper texture set ([issue #1969](https://github.com/seedds/nxnative/issues/1969), PR #2075); the first read now samples the texture's own colours, coarse, for that frame. Seen also in Kakariko Village and its graveyard on a MacBook Air M5 (three flashes in 8 minutes before, none in 4 after, by eye). - [Vertex shaders with layer entry points left out of the prebuild](../../../docs/reference/shaders.md#vertex-shaders-with-layer-entry-points-left-out-of-the-prebuild): resolved, a 230 ms frame at a camera turn in Hyrule Field. The effect vertex shaders (no stage inputs, layer entry points of their own) had no base in the bundled archive and were compiled at their first draw; with the fix (measured with PR #2065's form of it, the same bases as the reflection's `layer_out` gives) the archive holds 33,365 bases instead of 30,690, and the worst frame on the `field-run` route is 39 ms (M6). - [Unused border corrections retain sample registers](../../../docs/reference/rendering.md#unused-border-corrections-retain-sample-registers): about 1.5 ms of GPU a frame at 1080p ([measurements](#unused-border-corrections)). +- [A hold the GPU does not explain after a warm start's progress screen](../../../docs/reference/frame-rate.md#a-hold-the-gpu-does-not-explain-after-a-warm-starts-progress-screen): hold 1 → 3 on a progress screen's idle-GPU late frames after a warm start, then about 1,470 presents at 18 fps. - [Position invariance across shaders](../../../docs/reference/shaders.md#position-invariance-across-shaders): white shapes on plants and bushes for single frames in Kokiri Forest, from the foliage's mesh pre-pass and mesh colour pass rounding the position apart. - [gl_FragCoord.z against the depth stored for its own surface](../../../docs/reference/shaders.md#gl_fragcoordz-against-the-depth-stored-for-its-own-surface): the dark specks flickering in the ground mist (long-standing). The mist's own depth test (`5ad65c0eefd09c7f`, against its depth-only pass `4757b5b373a5406e`) rejected its own surface where Apple's GPU stored the depth one ulp under the fragment's z. Bisected by dropping fragment shaders: after mesh draws took the drop too (`skip_fs_draw`), the specks went with the mist or its depth-only pass, and when that pass wrote `[[position]].z` as depth. Resolved by the rewrite: no specks in play with its 39 rewritten shaders as an overlay (the player's check, 2026-11-09); a full library build on the new chain has not been run. diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index f5eaa5372e..d8566e8f77 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2370,8 +2370,8 @@ static void fstep_tick(const win_t *o, int c) { static uint64_t prev, t0, v0, fixed_sum, held[FSTEP_MAX + 1]; static int nlog; const double v = 1e9 / fstep_hz(); uint64_t st = g_fs_start ? g_fs_start : now_ns(), pw = g_fs_wait; g_fs_start = g_fs_wait = 0; - double shown = prev ? (st - prev) / v : 0, gpu = 0; prev = st; - for (uint64_t f = g_frame; f-- > 0 && g_frame - f <= 8;) { double g = mtl_gpu_ms(f); if (g >= 0) { gpu = g * 1e6 / v; break; } } + double shown = prev ? (st - prev) / v : 0, gpu = 0; prev = st; int gpu_on = 0; + for (uint64_t f = g_frame; f-- > 0 && g_frame - f <= 8;) { double g = mtl_gpu_ms(f); if (g >= 0) { gpu = g * 1e6 / v; gpu_on = 1; break; } } static uint64_t pdraw; uint64_t npass, ndraw; mtl_stats(&npass, &ndraw); // the draws encoded since the last tick: what the GPU reference measured (nvn_fstep.h FSTEP_GPU_WORK) double draws = pdraw && ndraw > pdraw ? (double)(ndraw - pdraw) : 0; pdraw = ndraw; static uint64_t pdisp; static int pdisp_ok; uint64_t ndisp = mtl_dispatches(); // and the dispatches (none is a count too) @@ -2404,6 +2404,7 @@ static void fstep_tick(const win_t *o, int c) { g_fs.per_vsync = fstep_hz() / 60; g_fs.smax = dm_fps_step_max; // the stub's largest step (Mario Kart World: 2): frames held longer than the game steps would play in slow motion g_fs.no_payback = fps_high(); // at 120 fps the game steps by its target, not by the step: a late frame is slow motion, so three raise the hold whatever the average + g_fs.gpu_on = gpu_on; g_fs.gpu = gpu; // an idle GPU's late frames do not step up; a hold the GPU does not explain is probed (nvn_fstep.h) int n = nvn_fstep_next(&g_fs, shown, cost, kmin, kmax, fixed, cslow || gslow); if (dm_fps_state) __atomic_store_n((uint32_t *)(dm_fps_state + 0x20), (uint32_t)n, __ATOMIC_RELAXED); atomic_store(&g_fs_k, g_fs.k); @@ -2412,13 +2413,17 @@ static void fstep_tick(const win_t *o, int c) { if (fixed != fixed_was && nfix < 40) { nfix++; if (fixed) LOG("[nvn] frame step: the game fixes its own step at %d vsyncs at present %llu", fixed, (unsigned long long)o->presents); else LOG("[nvn] frame step: the game hands the step back to the host at present %llu", (unsigned long long)o->presents); } fixed_was = fixed; if (k0 && g_fs.k != k0 && nlog < 40) { - char why[160], held[48] = ""; int w = snprintf(why, sizeof why, "probe"); + char why[256], held[48] = ""; int w = snprintf(why, sizeof why, "probe"); if (cslow) w += snprintf(why + w, sizeof why - w, ", CPU %d%% on performance cores", g_fs_pshare); + if (w >= (int)sizeof why) w = sizeof why - 1; if (gslow) w += snprintf(why + w, sizeof why - w, ", GPU %.2f× its time at hold %d", g_fg.cur / g_fg.ref, kmin); - if (nvn_fstep_same(&g_fs)) snprintf(why + w, sizeof why - w, ", cost %.2f× the cost that held hold %d", g_fs.ccur / g_fs.cref, kmin); + if (w >= (int)sizeof why) w = sizeof why - 1; + if (nvn_fstep_same(&g_fs)) w += snprintf(why + w, sizeof why - w, ", cost %.2f× the cost that held hold %d", g_fs.ccur / g_fs.cref, kmin); + if (w >= (int)sizeof why) w = sizeof why - 1; // snprintf returns the length it would have written + if (g_fs.uprobe) snprintf(why + w, sizeof why - w, ", GPU fits a shorter hold, frames not waiting in the pacing"); if (g_fs.k > k0 && g_fs.cref > 0) snprintf(held, sizeof held, "; hold %d held at %.2f", kmin, g_fs.cref); // what a probe compares the longer hold's cost with LOG("[nvn] frame step: hold %d → %d %s at present %llu (%s; cost %.2f, GPU %.2f%s)%s", k0, g_fs.k, fps_high() ? "ticks of 1/120 s" : "vsyncs", (unsigned long long)o->presents, - fixed ? "the game's own step" : kmin0 > FSTEP_MAX ? "the game's own step ended" : kmin != kmin0 ? "the cap changed" : g_fs.k > k0 ? "3 late frames" : g_fs.probes != probes0 ? why : "trial", cost, gpu, held, ++nlog == 40 ? "; further changes only counted below" : ""); + fixed ? "the game's own step" : kmin0 > FSTEP_MAX ? "the game's own step ended" : kmin != kmin0 ? "the cap changed" : g_fs.k > k0 ? (g_fs.up_idle ? "30 late frames, some with the GPU idle" : "3 late frames") : g_fs.probes != probes0 ? why : "trial", cost, gpu, held, ++nlog == 40 ? "; further changes only counted below" : ""); } if (probe0 == 1 && g_fs.probe != 1 && !fixed && nlog < 40) // whether the controller raised the clock for the shorter deadline LOG("[nvn] frame step: the probe %s at present %llu (CPU %d%% on performance cores)%s", g_fs.probe == 2 ? "held" : "failed", (unsigned long long)o->presents, g_fs_pshare, ++nlog == 40 ? "; further changes only counted below" : ""); diff --git a/dm/gfx/nvn/nvn_fstep.h b/dm/gfx/nvn/nvn_fstep.h index 892f657450..fcbae7edef 100644 --- a/dm/gfx/nvn/nvn_fstep.h +++ b/dm/gfx/nvn/nvn_fstep.h @@ -25,6 +25,15 @@ // rise set k to kmin with no grace, so three frames late while the clock rose stepped up again, with cref dropped at the cap change and a GPU reference taken at the 30 fps // cap: neither the fit trial (a GPU-bound cost near FSTEP_FIT) nor a probe brought kmin back (t_fstep: the movie and thermal cap cases). A cref kept from before the cap // starts a probe only when the cost at the longer hold is within FSTEP_SAME of it, and a failed probe doubles the hold-off as any other does. +// The caller may report each frame's GPU time (gpu_on, gpu). A late frame whose GPU was idle (under FSTEP_GPU_IDLE: a loading or progress screen, the first frames after a +// pipeline prepare) does not count toward the 3 late frames of a step up: nothing is being drawn, so the lateness is the CPU's startup or load work, which the debt rule +// absorbs, not what the scene costs (FSTEP_RUN late frames in a row step up whatever the GPU). Ocarina of Time stepped 1 → 2 → 3 at presents 9 and 12 of a warm start (cost 1.92 and 5.57 vsyncs, GPU 0.02 and 0.01) while a 118 s archive write ran. +// And a longer hold that the GPU does not explain is probed: FSTEP_RUN frames in a row at a hold above kmin whose GPU time fits k − 1 and that waited for nothing in the +// host's pacing (cost within FSTEP_LATE of the time shown) start a probe after the hold-off. Such a game waits for its frames outside the window's acquire (Ocarina of +// Time: no pacing wait at any hold, its frame time all counted as work), so its cost is the hold itself whatever the hold is, never fits k − 1, and the fit trial never comes: +// it stayed at hold 3 (18 fps, work 50 ms, GPU 12.5 ms) for about 1,470 presents where hold 1 held at 16.6 ms before. A game whose CPU really needs the longer hold fails the +// probe; these probes have their own hold-off (uhold), doubled after each one fails up to FSTEP_UHOLD_MAX (60 s), so such a game pays one failed probe a minute at most +// and not one every 8 s. Late frames with the GPU idle still step up after FSTEP_RUN of them in a row: a CPU-heavy scene that draws almost nothing is not a load. #pragma once #include <math.h> #include <stdint.h> @@ -46,6 +55,8 @@ // second half cost at most 1 − FSTEP_RAMP of its first half, and had a late frame (a window on time is judged as a trial), starts another, up to FSTEP_PROBE_MAX frames in all #define FSTEP_RAMP 0.05 // the fall in cost from one half of a grace window to the next that shows the clock still rising (the per-frame noise of a 25 ms frame is 4%, of 15 frames 1%) #define FSTEP_PROBE_MAX 180 // grace frames a probe may take in all (3 s at 60 fps): a clock that keeps rising without reaching the hold is not waited for longer +#define FSTEP_GPU_IDLE 0.05 // GPU time (in the frame's unit) under which nothing is being drawn: 0.8 ms at 60 fps (Ocarina of Time's progress screen 0.2–0.3 ms, its scenes 12 ms) +#define FSTEP_UHOLD_MAX 3600.0 // the longest hold-off before a probe of a hold the GPU does not explain (nvn_fstep_unpaced), doubled after each such probe fails: 60 s typedef struct { int k, step; // vsyncs each frame stays on screen; the step handed to the game for the next frame (k, or k + 1 while paying back) @@ -63,6 +74,11 @@ typedef struct { double run, crun, cref; // vsyncs at kmin in a row; the cost of its frames on time (running mean; not during a probe's trial, whose frames run while the clock rises); // crun when the last run of FSTEP_HOLD_MAX or more ended in a step up or a cap change (0 = none yet) double ccur; int ck, cn; // the cost of the frames on time at the longer hold ck (running mean) and their count + int gpu_on; double gpu; // set by the caller before each call, then cleared: 1 = gpu is the frame's GPU time (the latest finished frame's, in the unit of shown); 0 = not reported + int urun; // frames in a row at a hold above kmin whose GPU time fits k − 1 and that waited for nothing in the host's pacing (nvn_fstep_unpaced) + int ilate_run, uprobe; // late frames in a row, those with the GPU idle included; 1 = the probe running was started by nvn_fstep_unpaced alone + int up_idle; // 1 = the last step up came from FSTEP_RUN late frames with the GPU idle among them (for the caller's log) + double uhold; // vsyncs since the last step up before such a probe: 1 s, doubled after each one fails up to FSTEP_UHOLD_MAX, back to 1 s when kmin holds int ckmin; // the kmin run, crun and cref belong to double cref_at[FSTEP_MAX + 1]; // cref of each kmin, kept while another kmin is in force double over[FSTEP_RUN], over_sum; int over_i, over_k; // the last FSTEP_RUN frames at the hold over_k: shown − k of each (clamped), their sum; the next slot @@ -74,22 +90,27 @@ typedef struct { // Whether the cost at the longer hold is the cost that held kmin (a probe without a slowed report) static inline int nvn_fstep_same(const nvn_fstep *s) { return s->cref > 0 && s->cn >= FSTEP_RUN && s->ccur <= FSTEP_SAME * s->cref; } +// Whether the longer hold is one the GPU does not explain while the frames' time is all the game's own (a probe without a slowed report) +static inline int nvn_fstep_unpaced(const nvn_fstep *s) { return s->urun >= FSTEP_RUN; } // Once per frame. shown: real time since the previous frame's start, in vsyncs; cost: the frame's work, max(CPU, GPU), in vsyncs (a GPU that cannot keep up makes frames late too); kmin: 1 at a 60 fps cap, 2 at 30; // kmax: the longest hold (FSTEP_MAX; 2 at a 30 fps cap, where the display link runs at 30 Hz and holds only even counts); fixed: a step the game has set itself (BotW's // VFR override; 0 = none), which the host then holds as it is; slow: the CPU runs below its speed for reasons other than heat, so a cost above k − 1 is no evidence // against kmin (probes). Returns the step for the next frame; s->k is how long that frame stays on screen static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int kmin, int kmax, int fixed, int slow) { - const double hold0 = FSTEP_HOLD0 * (s->per_vsync ? s->per_vsync : 1), hold_max = FSTEP_HOLD_MAX * (s->per_vsync ? s->per_vsync : 1); + const double hold0 = FSTEP_HOLD0 * (s->per_vsync ? s->per_vsync : 1), hold_max = FSTEP_HOLD_MAX * (s->per_vsync ? s->per_vsync : 1), + uhold_max = FSTEP_UHOLD_MAX * (s->per_vsync ? s->per_vsync : 1); if (kmin < 1) kmin = 1; const int smax = s->smax > 0 && s->smax < FSTEP_MAX ? s->smax : FSTEP_MAX; if (kmax > smax) kmax = smax; if (kmin > smax) kmin = smax; if (kmax < kmin) kmax = kmin; if (!s->hold) s->hold = hold0; + if (!s->uhold) s->uhold = hold0; + int gpu_on = s->gpu_on; double gpu = s->gpu; s->gpu_on = 0; s->gpu = 0; // cleared on every return if (fixed > 0) { // the game's own step: speed is right by construction, nothing to pay back s->k = s->step = fixed < smax ? fixed : smax; - s->debt = 0; s->late_run = s->fit_run = s->trial = s->probe = s->grace = 0; s->run = 0; + s->debt = 0; s->late_run = s->ilate_run = s->fit_run = s->trial = s->probe = s->grace = 0; s->run = 0; s->kmin = FSTEP_MAX + 1; // when it ends, k starts again from kmin return s->step; } @@ -105,18 +126,23 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km s->since += shown; int cap_probe = kmin < s->kmin && s->kmin <= FSTEP_MAX && s->k > kmin; if (cap_probe) { // the cap rose: a probe at once, whose grace covers the clocks still set for the longer hold - s->k = kmin; s->probes++; s->trial = FSTEP_RUN; s->probe = 1; s->grace = FSTEP_GRACE; s->fit_run = s->late_run = 0; s->pa = s->pb = 0; s->plate = s->ptotal = 0; + s->k = kmin; s->probes++; s->trial = FSTEP_RUN; s->probe = 1; s->uprobe = 0; s->grace = FSTEP_GRACE; s->fit_run = s->late_run = s->ilate_run = 0; s->pa = s->pb = 0; s->plate = s->ptotal = 0; } else if (kmin < s->kmin || s->k < kmin) s->k = kmin; // the game's own step ended, or the cap fell if (s->k > kmax) s->k = kmax; s->kmin = kmin; int scr = s->screen; s->screen = 0; + int idle = gpu_on && gpu < FSTEP_GPU_IDLE; // nothing drawn: a late frame is load or startup work, not the scene's cost int late = scr ? scr == 2 : shown > held + FSTEP_LATE + s->shown_slack || cost > held + FSTEP_LATE, grace = s->grace > 0; if (grace && !cap_probe) { // the completed frame at the old cap does not consume the new hold's grace if (s->grace > FSTEP_GRACE / 2) s->pa += cost; else s->pb += cost; s->plate += late; s->ptotal++; if (!--s->grace && s->ramp && s->plate && s->ptotal + FSTEP_GRACE <= FSTEP_PROBE_MAX && s->pb <= (1 - FSTEP_RAMP) * s->pa) { s->grace = FSTEP_GRACE; s->pa = s->pb = 0; s->plate = 0; } // still getting cheaper: another window } - if (late) { s->late++; s->late_run = grace ? 0 : s->late_run + 1; s->fit_run = 0; } else s->late_run = 0; + // A late frame with the GPU idle neither counts toward the 3 nor ends their run; FSTEP_RUN late frames in a row step up whatever the GPU (a CPU-heavy scene, not a load) + if (late) { s->late++; s->late_run = grace ? 0 : s->late_run + !idle; s->ilate_run = grace ? 0 : s->ilate_run + 1; s->fit_run = 0; } else s->late_run = s->ilate_run = 0; + // The frame waited for nothing in the host's pacing while held longer than kmin, and its GPU time fits a shorter hold: the hold is not what the GPU needs (nvn_fstep_unpaced) + // (k − 1, not kmin: the GPU clock falls at a longer hold, so a GPU that fits kmin there may read above it; a GPU that really needs the hold fails the probe) + s->urun = held > kmin && s->k == held && gpu_on && gpu <= FSTEP_FIT * (held - 1) && cost >= shown - FSTEP_LATE ? s->urun + 1 : 0; // The cost of the frames on time (a hitch is not what the scene costs): at kmin, and at the longer hold if (kmin != s->ckmin) { // another cap: its run starts over, and what held it before comes back if (s->ckmin >= 1 && s->ckmin <= FSTEP_MAX) { @@ -136,20 +162,24 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km } // A rate below k over the last FSTEP_RUN frames (the slots not filled yet since the hold changed count as on time); the screen's verdict (present snapshots) is its own int short_of_k = scr || s->no_payback || s->over_sum > FSTEP_LATE * FSTEP_RUN; - if (s->late_run >= 3 && s->k < kmax && short_of_k) { // step up to the cost of the cheapest of the three late frames, at least one vsync + if ((s->late_run >= 3 || s->ilate_run >= FSTEP_RUN) && s->k < kmax && short_of_k) { // step up to the cost of the cheapest of the three late frames, at least one vsync double c3 = s->last_cost[0] < s->last_cost[1] ? s->last_cost[0] : s->last_cost[1]; if (s->last_cost[2] < c3) c3 = s->last_cost[2]; int want = (int)ceil(c3 - 0.05); if (want < s->k + 1) want = s->k + 1; if (want > kmax) want = kmax; - if (s->trial || (s->probe == 2 && s->since < hold_max)) { s->failed++; s->trial = 0; s->hold = s->hold * 2 < hold_max ? s->hold * 2 : hold_max; } - else if (s->probe == 2) s->hold = hold0; // the probe's run at kmin lasted - if (held == kmin && s->run >= hold_max) s->cref = s->crun; // kmin held 8 s at this cost - s->k = want; s->ups++; s->late_run = s->fit_run = 0; s->since = 0; s->probe = 0; + if (s->trial || (s->probe == 2 && s->since < hold_max)) { + s->failed++; s->trial = 0; s->hold = s->hold * 2 < hold_max ? s->hold * 2 : hold_max; + if (s->uprobe) s->uhold = s->uhold * 2 < uhold_max ? s->uhold * 2 : uhold_max; // a CPU that needs the hold: probed ever more rarely + } else if (s->probe == 2) s->hold = s->uhold = hold0; // the probe's run at kmin lasted + if (held == kmin && s->run >= hold_max) { s->cref = s->crun; s->uhold = hold0; } // kmin held 8 s at this cost + s->up_idle = s->late_run < 3; + s->k = want; s->ups++; s->late_run = s->ilate_run = s->fit_run = s->urun = 0; s->since = 0; s->probe = s->uprobe = 0; } else if (s->trial && !grace && !--s->trial) { if (s->probe) { s->probe = 2; s->since = 0; } else s->hold = hold0; } // the trial held (a probe's run at kmin is judged at the next step up) if (!late) s->fit_run = s->k > kmin && cost <= FSTEP_FIT * (s->k - 1) ? s->fit_run + 1 : 0; if (s->fit_run >= FSTEP_RUN && s->since >= s->hold && !s->trial) { s->k--; s->downs++; s->trial = FSTEP_RUN; s->fit_run = 0; } - else if ((slow || nvn_fstep_same(s)) && s->k > kmin && s->since >= s->hold && !s->trial) { // probe: straight to kmin, so the controller sees the whole deadline - s->k = kmin; s->downs++; s->probes++; s->trial = FSTEP_RUN; s->probe = 1; s->grace = FSTEP_GRACE; s->fit_run = s->late_run = 0; s->pa = s->pb = 0; s->plate = s->ptotal = 0; + else if ((slow || nvn_fstep_same(s) || (nvn_fstep_unpaced(s) && s->since >= s->uhold)) && s->k > kmin && s->since >= s->hold && !s->trial) { // probe: straight to kmin, so the controller sees the whole deadline + s->uprobe = !slow && !nvn_fstep_same(s); + s->k = kmin; s->downs++; s->probes++; s->trial = FSTEP_RUN; s->probe = 1; s->grace = FSTEP_GRACE; s->fit_run = s->late_run = s->ilate_run = 0; s->pa = s->pb = 0; s->plate = s->ptotal = 0; // urun ends with the next frame at kmin (kept for the caller's log) } s->step = s->k + (s->debt >= 1 && s->k < smax); s->paid += s->step > s->k; diff --git a/dm/gfx/tests/t_fstep.c b/dm/gfx/tests/t_fstep.c index d161c435e0..2fd019d97a 100644 --- a/dm/gfx/tests/t_fstep.c +++ b/dm/gfx/tests/t_fstep.c @@ -21,6 +21,17 @@ static void run(sim *m, int n, double w, int kmin, int kmax, int fixed) { // A hitch that steps hold 1 up: six frames of 1.6 vsyncs (pipeline compiles), 3.6 vsyncs behind in the last 30 frames, more than FSTEP_LATE a frame on average. Three such // frames among frames on time are paid back by the step and keep hold 1 (below) static void hitch(sim *m) { run(m, 6, 1.6, 1, 3, 0); } +// Frames with the GPU time reported (nvn_fstep gpu_on): CPU work cpu and GPU time gpu (vsyncs). paced = 1: the frame waits out its hold in the host's pacing, so its cost is +// max(cpu, gpu); paced = 0: the game waits for its frames elsewhere (Ocarina of Time), so the whole time shown is its cost +static void rung(sim *m, int n, double cpu, double gpu, int paced, int kmin, int kmax) { + for (int i = 0; i < n; i++) { + int k = m->s.k ? m->s.k : kmin, step = m->next ? m->next : kmin; + double w = cpu > gpu ? cpu : gpu, shown = w > k ? w : k; + m->real += shown; m->game += step; + m->s.gpu_on = 1; m->s.gpu = gpu; + m->next = nvn_fstep_next(&m->s, shown, paced ? w : shown, kmin, kmax, 0, m->slow); + } +} int main(void) { { // 120 fps (hle_nvn.c fps_high): kmin 1, kmax 2, in ticks of 1/120 s. The target follows the hold, so a hold above what frames take is the game at the right speed @@ -443,6 +454,60 @@ int main(void) { check(first_len == want[c] && m.s.k == 2 && m.s.failed == 1, b); } } + { // Ocarina of Time on a warm start (2026-11-08, run hu_fog): the first frames after a 162 s pipeline prepare ran 2–7 vsyncs with the GPU idle (0.01–0.02 vsyncs) while an + // archive write ran, and stepped the hold 1 → 2 → 3; its frames wait outside the host's pacing, so their cost is the hold and it stayed at 3 (20 fps) for 1,470 presents + sim m = {0}; rung(&m, 8, 0.4, 0.01, 0, 1, 3); + static const double burst[] = {1.9, 4.5, 7.0, 5.6, 4.6, 6.6, 5.4, 4.8, 7.0}; + for (int i = 0; i < 9; i++) rung(&m, 1, burst[i], 0.01, 0, 1, 3); + snprintf(b, sizeof b, "late frames with the GPU idle (a progress screen after a prepare) do not step up (k %d, ups %llu)", m.s.k, (unsigned long long)m.s.ups); + check(m.s.k == 1 && m.s.ups == 0, b); + rung(&m, 600, 0.8, 0.75, 0, 1, 3); + check(m.s.k == 1 && m.s.ups == 0, " then the scene (GPU 0.75, frames not waiting in the pacing) holds hold 1"); + } + { // the same game stepped to 3 by a real hitch (three frames of 3.5 vsyncs with the GPU busy): at hold 3 its cost is the hold, the GPU 0.75 fits hold 1, so a probe + // brings hold 1 back within 2 s instead of never (no fit trial: the cost never fits k − 1) + sim m = {0}; rung(&m, 600, 0.8, 0.75, 0, 1, 3); rung(&m, 3, 3.5, 0.75, 0, 1, 3); + check(m.s.k == 3, "self-paced game, a hitch with the GPU busy: hold 3"); + double t0 = m.real, back = -1; + for (int i = 0; i < 1200 && back < 0; i++) { rung(&m, 1, 0.8, 0.75, 0, 1, 3); if (m.s.k == 1) back = m.real - t0; } + rung(&m, 1200, 0.8, 0.75, 0, 1, 3); + snprintf(b, sizeof b, " its hold the GPU does not explain is probed: hold 1 after %.0f vsyncs, and it stays (k %d, probes %llu, failed %llu)", back, m.s.k, (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(back > 0 && back <= 120 && m.s.k == 1 && m.s.probes == 1 && m.s.failed == 0, b); + } + { // a CPU-bound game (1.4 vsyncs of CPU, GPU 0.3) waits out hold 2 in the pacing: its cost explains the hold, so no probe + sim m = {0}; rung(&m, 60, 0.5, 0.3, 1, 1, 3); rung(&m, 1200, 1.4, 0.3, 1, 1, 3); + snprintf(b, sizeof b, "CPU-bound game paced at hold 2 (cost 1.4, GPU 0.3): stays at 2, no probes (k %d, probes %llu)", m.s.k, (unsigned long long)m.s.probes); + check(m.s.k == 2 && m.s.probes == 0, b); + } + { // a CPU-bound game at 3.2 vsyncs (late even at hold 3, so no pacing wait), GPU 0.3: the probes fail and spread out 1 → 2 → 4 → 8 s + sim m = {0}; rung(&m, 60, 0.5, 0.3, 1, 1, 3); rung(&m, 30, 3.2, 0.3, 1, 1, 3); + double last = m.real, gaps[4] = {-1, -1, -1, -1}; int ng = 0; uint64_t probes = m.s.probes; + for (int i = 0; i < 6000 && ng < 4; i++) { rung(&m, 1, 3.2, 0.3, 1, 1, 3); if (m.s.probes != probes) { probes = m.s.probes; gaps[ng++] = m.real - last; last = m.real; } } + snprintf(b, sizeof b, "CPU-bound game at 3.2 vsyncs: probes %.0f, %.0f, %.0f, %.0f vsyncs apart; %llu failed", gaps[0], gaps[1], gaps[2], gaps[3], (unsigned long long)m.s.failed); + check(ng == 4 && gaps[1] >= 120 && gaps[2] >= 240 && gaps[3] >= 480 && m.s.failed >= 3, b); + } + { // the same self-paced game, but its GPU clock falls at hold 3 (GPU 0.75 at hold 1, 1.0 at hold 3): GPU time that fits k − 1 still starts the probe + sim m = {0}; rung(&m, 600, 0.8, 0.75, 0, 1, 3); rung(&m, 3, 3.5, 1.0, 0, 1, 3); + for (int i = 0; i < 2400; i++) rung(&m, 1, 0.8, m.s.k > 1 ? 1.0 : 0.75, 0, 1, 3); + snprintf(b, sizeof b, "self-paced game whose GPU clock falls at the longer hold: probed back to hold 1 and stays (k %d, probes %llu, failed %llu)", m.s.k, (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(m.s.k == 1 && m.s.probes == 1 && m.s.failed == 0, b); + } + { // a CPU-heavy scene that draws almost nothing (CPU 1.5 vsyncs, GPU 0.01, paced): its late frames are not a load, so FSTEP_RUN of them in a row still step up to 2 + sim m = {0}; rung(&m, 60, 0.5, 0.01, 1, 1, 3); rung(&m, 3000, 1.5, 0.01, 1, 1, 3); + snprintf(b, sizeof b, "CPU-heavy scene with the GPU idle: steps up once and holds 2 (k %d, ups %llu, late %llu)", m.s.k, (unsigned long long)m.s.ups, (unsigned long long)m.s.late); + check(m.s.k == 2 && m.s.ups == 1 && m.s.late <= 40, b); + } + { // a self-paced game whose CPU needs hold 2 (CPU 1.5, GPU 0.3; 20,000 frames, about 11 minutes): the unpaced probes fail and their hold-off grows to 60 s + sim m = {0}; rung(&m, 60, 0.5, 0.3, 0, 1, 3); rung(&m, 20000, 1.5, 0.3, 0, 1, 3); + snprintf(b, sizeof b, "self-paced game that needs hold 2: at most one probe a minute after the first few (k %d, probes %llu, failed %llu)", m.s.k, (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check(m.s.k == 2 && m.s.probes <= 16 && m.s.failed == m.s.probes, b); + } + { // a host-paced CPU-bound game at 2.95 vsyncs (pacing wait under FSTEP_LATE at hold 3): 20,000 frames, about 17 minutes: probes at most once a minute after the first few, the game within 1.5% of real time (one failed probe every 8 s: 5.5% slow) + sim m = {0}; rung(&m, 60, 0.5, 0.3, 1, 1, 3); rung(&m, 30, 2.95, 0.3, 1, 1, 3); + double r0 = m.real, g0 = m.game; rung(&m, 20000, 2.95, 0.3, 1, 1, 3); + snprintf(b, sizeof b, "host-paced game at 2.95 vsyncs: game/real %.3f (probes %llu, failed %llu)", (m.game - g0) / (m.real - r0), (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); + check((m.game - g0) / (m.real - r0) >= 0.985 && m.s.probes <= 24, b); + } printf("== %d checks, %d failed ==\n", checks, fail); return fail != 0; } diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index a3527419a7..b94c109564 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -516,6 +516,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - in a game the host steps at 60 fps (`FRAME_STEP`), a prerendered movie plays at twice its speed, its sound falls behind, and the frame counter shows 60 during it. The log has `[movie]` lines with pictures 33 ms apart (`time 0 / 33000 / 67000 µs`) and no `frame step cap 60 → 30` at the movie. → [Movies hold 30 fps under the frame step](../reference/frame-rate.md#movies-hold-30-fps-under-the-frame-step) - after a 30 fps cap ends (a prerendered movie, the thermal cap), a GPU-bound game stays at 30 fps in a scene that held 60 before the cap; the log shows `frame step cap 30 → 60 fps` and a few frames later `frame step: hold 1 → 2 vsyncs at present … (3 late frames; …)`, with no probe after it. → [The frame step after a 30 fps cap ends](../reference/frame-rate.md#the-frame-step-after-a-30-fps-cap-ends) - on a Mac, a GPU-bound game at 60 fps falls to 30 or 20 fps when the scene gets heavier and stays there until a menu or a lighter scene, while the GPU clock sits at its floor (338–400 MHz on an M5 Pro) and the GPU is powered 60–75% of the time; the log shows probes back to hold 1 that fail after about 33 frames (`frame step: hold 2 → 1 vsyncs … (probe, GPU 1.67× its time at hold 1)` then `the probe failed`), and the host's profiler (`@gpu_mhz`) shows the clock rising during each probe and falling back after it. → [A probe's grace lasts while its frames keep getting cheaper](../reference/frame-rate.md#a-probes-grace-lasts-while-its-frames-keep-getting-cheaper) +- a game that held 60 fps steps to hold 3 (20 fps) within its first frames after a warm start and stays there for over a minute; the log shows `frame step: hold 1 → 2 vsyncs at present 9 (3 late frames; cost 1.92, GPU 0.02 …)` and `hold 2 → 3 … at present 12 (… cost 5.57, GPU 0.01 …)`, and the next change only much later (Ocarina of Time: a trial at present 1480, 18 fps measured in between). → [A hold the GPU does not explain after a warm start's progress screen](../reference/frame-rate.md#a-hold-the-gpu-does-not-explain-after-a-warm-starts-progress-screen) - a game whose profile has a frame step (`FRAME_STEP` in its `patches.py`) runs at double speed when packed from another update; the log has no `frame step in whole vsyncs` line, and the frame counter shows 60. → [A derived profile drops the frame step its build lacks](../reference/frame-rate.md#a-derived-profile-drops-the-frame-step-its-build-lacks) - a fixed-step game runs at double speed (or slow motion at 30) after `DM_FPS_TARGET` is set, in `Documents/env.txt` or the launcher, on a build its profile's frame step stubs and limiter rows do not know. → [A present cap applies only with a frame step or an engine limit the host writes](../reference/frame-rate.md#a-present-cap-applies-only-with-a-frame-step-or-an-engine-limit-the-host-writes) - a game holds 30 fps with spare CPU and GPU time; it asks for present interval 2 once (`the game sets present interval 1 → 2`) and nothing else limits it. With `present_interval display` it runs at 60, but `fps_target` logs `present cap … not applied` and the player's **Frame rate** (30 or 60) does nothing: frames stay at one a display tick. → [A present interval that is the game's only limit, in display mode](../reference/frame-rate.md#a-present-interval-that-is-the-games-only-limit-in-display-mode) diff --git a/docs/reference/frame-rate.md b/docs/reference/frame-rate.md index 69f0b16947..d7b1124f48 100644 --- a/docs/reference/frame-rate.md +++ b/docs/reference/frame-rate.md @@ -94,6 +94,15 @@ Frame pacing: the host's frame step, engine frame limiters, present intervals an - **Note:** what shortens the serial chain itself, so the GPU's occupancy at a GPU-bound hold is higher and the clock rises faster: Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the profile's `enc_thread on`; the submit thread's translation 6.5 → about 4 ms a frame, measured on an M4 Pro and for Need for Speed), and a smaller first segment. Neither can start the GPU before the game's own work after its wait, and neither was measured on this symptom yet. +### A hold the GPU does not explain after a warm start's progress screen + +- **Symptom:** a game that held 60 fps steps to hold 3 (20 fps) within its first frames after a warm start and stays there for over a minute; the log shows `frame step: hold 1 → 2 vsyncs at present 9 (3 late frames; cost 1.92, GPU 0.02 …)` and `hold 2 → 3 … at present 12 (… cost 5.57, GPU 0.01 …)`, and the next change only much later (Ocarina of Time: a trial at present 1480, 18 fps measured in between). +- **Cause:** the first frames after a pipeline prepare ran 2–7 vsyncs late on the CPU (a progress screen while the pipeline archive was written) with the GPU idle, and three of them stepped the hold up. A game that waits for its frames outside the window's acquire (Ocarina of Time) shows no wait in the host's pacing, so its cost at a longer hold is about the hold itself (frame time and frame work both p50 54 ms at hold 3, GPU 12.5 ms, CPU 29% of one core). That cost does not fit k − 1, so the fit trial waits until the frames happen to get cheaper, and with no slowed report and no `cref` no probe fires. +- **Fix:** `nvn_fstep.h`: the caller reports each frame's GPU time (`gpu_on`, `gpu`; `fstep_tick` in `hle_nvn.c` passes the latest finished frame's). A late frame whose GPU was under `FSTEP_GPU_IDLE` (0.05 of a frame, 0.8 ms at 60 fps) does not count toward the 3 late frames and does not end their run; `FSTEP_RUN` late frames in a row still step up, so a CPU-heavy scene that draws almost nothing is not taken for a load. And `FSTEP_RUN` frames in a row at a hold above kmin whose GPU time fits k − 1 (`FSTEP_FIT`; not kmin, since the GPU clock falls at a longer hold) and that waited for nothing in the host's pacing (cost within `FSTEP_LATE` of the time shown) start a probe (`nvn_fstep_unpaced`, logged as `GPU fits a shorter hold, frames not waiting in the pacing`). A step up from the 30 late frames is logged as `30 late frames, some with the GPU idle`. These probes have their own hold-off (`uhold`): 1 s, doubled after each one fails up to `FSTEP_UHOLD_MAX` (60 s), back to 1 s when kmin holds, so a game whose CPU needs the longer hold pays at most one failed probe a minute. +- **Check:** `make -C dm/gfx/tests GAME=0400486024DEC000 t_fstep && dm/gfx/tests/t_fstep`: late frames with the GPU idle do not step up; a self-paced game stepped to 3 by a hitch with the GPU busy is probed back to hold 1 within 2 s and stays, also when its GPU clock falls at hold 3; a CPU-bound game at 3.2 vsyncs pays failed probes at growing gaps; a CPU-bound game paced at hold 2 is not probed; a CPU-heavy scene with the GPU idle still steps up once; self-paced and host-paced CPU-bound games are probed at most once a minute after the first few. The idle-GPU, hitch, clock-fall and 3.2-vsync cases fail without the change (5 of 88 checks); the rest guard against the change's own costs. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) (Mac, warm start after a 162 s pipeline prepare). +- **Note:** the GPU time is the latest finished frame's, up to 8 frames old, so the first late frames of a heavy scene right after a progress screen can still read as idle (they count once `FSTEP_RUN` are late in a row), and a progress screen late for more than `FSTEP_RUN` frames in a row still steps up. `uhold` only resets when kmin holds, so after a CPU-heavy stretch a lighter scene can wait up to 60 s for its probe. The acquire's wait for the present thread (`nvn_queue_present_wait`) is not in the pacing wait, so frames held there count as not waiting. Measuring the game's own frame wait and taking it out of the cost would let the fit trial work for such games without the probe; not done here. + ### A derived profile drops the frame step its build lacks - **Symptom:** a game whose profile has a frame step (`FRAME_STEP` in its `patches.py`) runs at double speed when packed from another update; the log has no `frame step in whole vsyncs` line, and the frame counter shows 60. From 1a273e81567a76a85cf04d0870c95c2dbfc86409 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:05:08 +0100 Subject: [PATCH 25/40] OoT-iPhone B-2: frames late at a 30 fps cap's hold 2 are shown at the next vsync (60 Hz link), not on the 30 Hz link's next tick At a 30 fps cap the frame step holds at most 2 vsyncs, and the display link runs at 30 Hz, so a frame that misses its tick waits for the next one: 66.7 ms on screen. Ocarina of Time on an iPhone at the launcher's 30 cap (frames 36.5 ms, GPU 32 ms) left 33 of each ~333 ticks empty: 33 intervals of 66.7 ms in each 300 frames, the rest 33.3, 27 fps (log iphone5/log_walk.txt). nvn_fstep.h: the late frames that would step the hold up (3 in a row, or FSTEP_RUN with the GPU idle) set fine when the hold is already at kmax; it ends after FSTEP_HOLD_MAX vsyncs with no late frame, when the hold may rise again and while the game fixes its own step. hle_nvn.c: nvn_game_fps reports 60 while fine at a 30 cap, so link_sync runs the 60 Hz link, which holds each frame its two vsyncs (g_link_hold_until) and shows a late one at the next vsync: 33.3 / 50 ms, never faster than 30. The switch is logged both ways; the frame-interval line keeps its 33.3 ms target, and the on-screen line counts intervals at the hold / one vsync longer / two or more. t_fstep: 13 new checks (106, all pass); t_nvn 277 and t_thermal 21 pass. --- dm/gfx/nvn/hle_nvn.c | 22 ++++++++++++++++------ dm/gfx/nvn/nvn_fstep.h | 14 ++++++++++++-- dm/gfx/tests/t_fstep.c | 23 +++++++++++++++++++++++ 3 files changed, 51 insertions(+), 8 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index d8566e8f77..cf208995d0 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -1943,6 +1943,7 @@ static double g_pres_exp[TWIN], g_pres_exp_last; // how long that frame should // The hold (60 Hz vsyncs) each present was given, by present number. On-screen intervals are read 8 presents late, so each is judged against the holds of its own presents: // with the frame step at 2 vsyncs every frame stays 33.3 ms, which is not a dropped frame static uint8_t g_hold_ring[16]; +static _Atomic int g_fs_fine; // nvn_fstep.fine (fstep_tick): frames late at hold 2, the longest a 30 fps cap allows, each shown at the next vsync after its hold (nvn_game_fps) static int fstep_hz(void); static double pres_expect_ms(uint64_t n) { // the interval between presents n - 1 and n shows frame n - 1: the display link holds it for its own present's hold, the Mac pacer spaces present n by n's, so the larger of the two int k = g_hold_ring[n % 16] > g_hold_ring[(n - 1) % 16] ? g_hold_ring[n % 16] : g_hold_ring[(n - 1) % 16]; @@ -2045,15 +2046,16 @@ static void tsummary(uint64_t frame) { } LOG("[nvn] timing up to frame %llu:%s", frame, line); { void mtl_pso_line(char *, size_t); char pl[1024]; mtl_pso_line(pl, sizeof pl); LOG("[nvn] pipelines up to frame %llu: %s", frame, pl); } - double tgt = 1000.0 / mtl_fps_target(); // milliseconds one frame should take: 16.7 at 60 fps, 33.3 when the game itself asks for 30 (the over-time / dropped-frame counts below use it) + double tgt = 1000.0 / (atomic_load(&g_fs_fine) ? 30 : mtl_fps_target()); // milliseconds one frame should take: 16.7 at 60 fps, 33.3 when the game itself asks for 30 or at a 30 cap whose late frames the 60 Hz link shows (the over-time / dropped-frame counts below use it) { unsigned n = g_tn[0] < TWIN ? g_tn[0] : TWIN, over = 0; double v[TWIN]; // frame interval gets its own line (M1 baseline): number of frames longer than one frame period in this window and p99 if (n) { memcpy(v, g_tw[0], n * sizeof *v); qsort(v, n, sizeof *v, dcmp); for (unsigned i = 0; i < n; i++) over += v[i] > tgt; LOG("[nvn] frame interval up to frame %llu: >%.1fms %u / %u frames, p99 %.2f ms", frame, tgt, over, n, v[(n * 99) / 100 < n ? (n * 99) / 100 : n - 1]); } } - { unsigned n = g_presn < TWIN ? g_presn : TWIN, k = 0, drop = 0, discard = 0; double v[TWIN]; // on-screen present: dropped frame = on screen longer than its hold × 1.5 (pres_expect_ms) or discarded - for (unsigned i = 0; i < n; i++) { double x = g_pres_iv[i]; if (x < 0) { drop++; discard++; } else if (x > 0) { v[k++] = x; drop += x > g_pres_exp[i] * 1.5; } } + { unsigned n = g_presn < TWIN ? g_presn : TWIN, k = 0, drop = 0, discard = 0, over[3] = {0}; double v[TWIN], vs = 1000.0 / fstep_hz(); // on-screen present: dropped frame = on screen longer than its hold × 1.5 (pres_expect_ms) or discarded + // over: intervals at their hold, one vsync longer and two or more longer (at hold 2: 33.3 / 50 / 66.7 ms and more; a frame late at hold 2 shows 50 ms on the 60 Hz link, 66.7 on the 30 Hz one) + for (unsigned i = 0; i < n; i++) { double x = g_pres_iv[i]; if (x < 0) { drop++; discard++; } else if (x > 0) { v[k++] = x; drop += x > g_pres_exp[i] * 1.5; over[x < g_pres_exp[i] + 0.5 * vs ? 0 : x < g_pres_exp[i] + 1.5 * vs ? 1 : 2]++; } } task_vm_info_data_t vi; mach_msg_type_number_t vc = TASK_VM_INFO_COUNT; double fp = task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&vi, &vc) == KERN_SUCCESS ? vi.phys_footprint / 1048576.0 : -1; double br = dm_brightness_now(); // footprint / brightness are carried on the on-screen present row: does not depend on the tick-entry observation gate (camp runs have it too) - if (k) { qsort(v, k, sizeof *v, dcmp); LOG("[nvn] on-screen up to frame %llu: %u frames, dropped %u (discarded %u), p99 %.2f ms; footprint %.0f MiB; brightness %.2f; thermal %d", frame, k + discard, drop, discard, v[(k * 99) / 100 < k ? (k * 99) / 100 : k - 1], fp, br, dm_thermal_now()); } + if (k) { qsort(v, k, sizeof *v, dcmp); LOG("[nvn] on-screen up to frame %llu: %u frames, dropped %u (discarded %u), p99 %.2f ms; intervals at the hold / one vsync longer / two or more %u / %u / %u; footprint %.0f MiB; brightness %.2f; thermal %d", frame, k + discard, drop, discard, v[(k * 99) / 100 < k ? (k * 99) / 100 : k - 1], over[0], over[1], over[2], fp, br, dm_thermal_now()); } g_presn = 0; } power_line(frame); { static uint64_t prev[6], pf; uint64_t v[6]; mtl_vbuf_stats(&v[0], &v[1], &v[2], &v[3], &v[4], &v[5]); @@ -2264,13 +2266,15 @@ static int fps_present_cap(void) { static nvn_fstep g_fs; // present thread only static nvn_fgpu g_fg; // the GPU time at kmin and at the current hold (nvn_fstep_gpu_slowed) static _Atomic int g_fs_k; // the hold presentation follows (0 = not decided yet) -// The display link runs at 30 Hz at a 30 cap, except while the game fixes its own step at 1 (BotW's VFR override) +// The display link runs at 30 Hz at a 30 cap, except while the game fixes its own step at 1 (BotW's VFR override) and while frames are late at hold 2 (nvn_fstep.h fine): the 30 Hz link +// shows a late frame only on its next tick, 66.7 ms after the one before, where the 60 Hz link, which holds each frame its two vsyncs (g_link_hold_until, nvn_mtl.m), shows it at the next +// vsync, 50 ms. Ocarina of Time on an iPhone at the launcher's 30 fps cap (frames 36.5 ms) got 33 intervals of 66.7 ms in each 300 on the 30 Hz link int nvn_fps_high_now(void) { return atomic_load(&g_fs_k) == 1 ? 120 : 60; } // hle_fpslock.c: the target of the next game frame, from the hold in force // The rate of vi::GetDisplayVsyncEvent (hle_sdk16.c, nvn_mtl.m mtl_link_wait_vsync): the Switch's 60 Hz, or 120 while 120 fps is on and its hold is one tick of 1/120 s. A game that // waits for the next vblank each frame is held to the event's rate whatever the present cap: Unity's NVN device waits until its vblank time (set from the event) passes // (The Rogue Prince of Persia: main+0x14f65f0), so at 120 every frame came two ticks late and the hold fell back to 60. The hold's trial at 120 must already have the event at 120 int nvn_vsync_hz(void) { return fps_high() && atomic_load(&g_fs_k) == 1 && atomic_load(&g_high_cap) == 120 ? 120 : 60; } -int nvn_game_fps(void) { int c = fps_present_cap(); if (c) return c == 30 && atomic_load(&g_fs_k) == 1 && fstep_on() ? 60 : c; uint64_t s = atomic_load(&g_itv2_since); return g_hz > 0 && !g_display_mode && s && now_ns() >= s + 1000000000ull ? 30 : 60; } +int nvn_game_fps(void) { int c = fps_present_cap(); if (c) return c == 30 && (atomic_load(&g_fs_k) == 1 || atomic_load(&g_fs_fine)) && fstep_on() ? 60 : c; uint64_t s = atomic_load(&g_itv2_since); return g_hz > 0 && !g_display_mode && s && now_ns() >= s + 1000000000ull ? 30 : 60; } static void present_cfg(const win_t *o) { if (g_hz >= 0) return; // The interval mode comes from the game profile (present_interval, default game): whether frames follow the present interval the game sets depends on whether the game's logic counts frames, which differs per game. Environment variables override it, and can override the fixed 60 Hz. @@ -2408,6 +2412,12 @@ static void fstep_tick(const win_t *o, int c) { int n = nvn_fstep_next(&g_fs, shown, cost, kmin, kmax, fixed, cslow || gslow); if (dm_fps_state) __atomic_store_n((uint32_t *)(dm_fps_state + 0x20), (uint32_t)n, __ATOMIC_RELAXED); atomic_store(&g_fs_k, g_fs.k); + int fine = g_fs.fine && c == 30 && !fps_high(), fine0 = atomic_exchange(&g_fs_fine, fine); // only a 30 fps cap's link runs slower than the vsyncs + static int nfine; + if (fine != fine0 && nfine < 40) { + if (fine) LOG("[nvn] frame step: frames late at hold %d, the longest the 30 fps cap allows (cost %.2f, GPU %.2f vsyncs): each is shown at the next vsync (a 60 Hz display link: 50 ms) instead of a 30 Hz link's next tick (66.7 ms), at present %llu%s", g_fs.k, cost, gpu, (unsigned long long)o->presents, ++nfine == 40 ? "; further changes only counted below" : ""); + else LOG("[nvn] frame step: %s: the 30 fps cap's own 30 Hz display link again at present %llu%s", c != 30 ? "the cap is no longer 30 fps" : fixed ? "the game fixes its own step" : g_fs.k < kmax ? "the hold may rise again" : "no late frame for 8 s", (unsigned long long)o->presents, ++nfine == 40 ? "; further changes only counted below" : ""); + } held[g_fs.k]++; if (fixed) fixed_sum += (uint64_t)n; // the stubs count only the host's steps static int fixed_was, nfix; if (fixed != fixed_was && nfix < 40) { nfix++; if (fixed) LOG("[nvn] frame step: the game fixes its own step at %d vsyncs at present %llu", fixed, (unsigned long long)o->presents); else LOG("[nvn] frame step: the game hands the step back to the host at present %llu", (unsigned long long)o->presents); } diff --git a/dm/gfx/nvn/nvn_fstep.h b/dm/gfx/nvn/nvn_fstep.h index fcbae7edef..74e0768c3f 100644 --- a/dm/gfx/nvn/nvn_fstep.h +++ b/dm/gfx/nvn/nvn_fstep.h @@ -34,6 +34,12 @@ // it stayed at hold 3 (18 fps, work 50 ms, GPU 12.5 ms) for about 1,470 presents where hold 1 held at 16.6 ms before. A game whose CPU really needs the longer hold fails the // probe; these probes have their own hold-off (uhold), doubled after each one fails up to FSTEP_UHOLD_MAX (60 s), so such a game pays one failed probe a minute at most // and not one every 8 s. Late frames with the GPU idle still step up after FSTEP_RUN of them in a row: a CPU-heavy scene that draws almost nothing is not a load. +// Late frames at the longest hold allowed (kmax: 2 at a 30 fps cap, or kmin with fps_hold off) cannot step up. The caller's display link at the cap's rate (30 Hz) can then only +// show a late frame on its next tick, 66.7 ms after the one before: Ocarina of Time on an iPhone at the launcher's 30 fps cap (frames 36.5 ms, GPU 32 ms) left one tick in 9 empty, +// 33 intervals of 66.7 ms in each 300 and the rest 33.3, at 27 fps. So the same late frames that would step up (3 in a row, or FSTEP_RUN with the GPU idle among them) set fine: +// the caller then shows each frame at the next vsync after its hold instead (a 60 Hz link that holds each frame its k vsyncs), 33.3 or 50 ms and never 66.7, still never faster +// than the cap. fine ends after FSTEP_HOLD_MAX vsyncs with no such late frame (a scene that fits again; a late frame now and then does not flap the link), when the hold may rise +// again (the cap rose) and while the game fixes its own step. #pragma once #include <math.h> #include <stdint.h> @@ -85,7 +91,8 @@ typedef struct { double last_cost[3]; // the last three frames' costs, oldest first int smax; // set by the caller: the largest step the game takes (its stub's STEP_MAX; 0 = FSTEP_MAX) int no_payback; // set by the caller: 1 = the game does not take the step (120 fps: it steps by its target), so three late frames raise the hold whatever the average - uint64_t late, paid, ups, downs, failed, probes; double dropped; // statistics for the log + int fine; double fine_ok; // 1 = frames are late at kmax, shown at the next vsync after their hold; vsyncs since the last late frame while fine (FSTEP_HOLD_MAX ends it) + uint64_t late, paid, ups, downs, failed, probes, fines; double dropped; // statistics for the log (fines: times fine was set) } nvn_fstep; // Whether the cost at the longer hold is the cost that held kmin (a probe without a slowed report) @@ -110,7 +117,7 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km int gpu_on = s->gpu_on; double gpu = s->gpu; s->gpu_on = 0; s->gpu = 0; // cleared on every return if (fixed > 0) { // the game's own step: speed is right by construction, nothing to pay back s->k = s->step = fixed < smax ? fixed : smax; - s->debt = 0; s->late_run = s->ilate_run = s->fit_run = s->trial = s->probe = s->grace = 0; s->run = 0; + s->debt = 0; s->late_run = s->ilate_run = s->fit_run = s->trial = s->probe = s->grace = 0; s->run = 0; s->fine = 0; s->kmin = FSTEP_MAX + 1; // when it ends, k starts again from kmin return s->step; } @@ -175,6 +182,9 @@ static inline int nvn_fstep_next(nvn_fstep *s, double shown, double cost, int km s->up_idle = s->late_run < 3; s->k = want; s->ups++; s->late_run = s->ilate_run = s->fit_run = s->urun = 0; s->since = 0; s->probe = s->uprobe = 0; } else if (s->trial && !grace && !--s->trial) { if (s->probe) { s->probe = 2; s->since = 0; } else s->hold = hold0; } // the trial held (a probe's run at kmin is judged at the next step up) + // Late at the longest hold allowed: shown at the next vsync, as the hold cannot rise (fine above) + if (s->k >= kmax && (s->late_run >= 3 || s->ilate_run >= FSTEP_RUN)) { if (!s->fine) s->fines++; s->fine = 1; s->fine_ok = 0; } + else if (s->fine) { s->fine_ok = late && !idle ? 0 : s->fine_ok + shown; if (s->k < kmax || s->fine_ok >= hold_max) s->fine = 0; } if (!late) s->fit_run = s->k > kmin && cost <= FSTEP_FIT * (s->k - 1) ? s->fit_run + 1 : 0; if (s->fit_run >= FSTEP_RUN && s->since >= s->hold && !s->trial) { s->k--; s->downs++; s->trial = FSTEP_RUN; s->fit_run = 0; } else if ((slow || nvn_fstep_same(s) || (nvn_fstep_unpaced(s) && s->since >= s->uhold)) && s->k > kmin && s->since >= s->hold && !s->trial) { // probe: straight to kmin, so the controller sees the whole deadline diff --git a/dm/gfx/tests/t_fstep.c b/dm/gfx/tests/t_fstep.c index 2fd019d97a..062276c839 100644 --- a/dm/gfx/tests/t_fstep.c +++ b/dm/gfx/tests/t_fstep.c @@ -508,6 +508,29 @@ int main(void) { snprintf(b, sizeof b, "host-paced game at 2.95 vsyncs: game/real %.3f (probes %llu, failed %llu)", (m.game - g0) / (m.real - r0), (unsigned long long)m.s.probes, (unsigned long long)m.s.failed); check((m.game - g0) / (m.real - r0) >= 0.985 && m.s.probes <= 24, b); } + { // a 30 fps cap (kmin = kmax = 2) whose frames run late (Ocarina of Time on an iPhone: 36.5 ms, GPU 32 ms): the hold cannot rise, so the frame step asks the caller to show each late + // frame at the next vsync (nvn_fstep.fine: the 60 Hz display link) instead of the next tick of the cap's 30 Hz link, and returns to it once the frames fit for FSTEP_HOLD_MAX + sim m = {0}; rung(&m, 120, 1.2, 1.0, 1, 2, 2); + check(m.s.k == 2 && !m.s.fine, "30 fps cap, frames that fit hold 2: the cap's own link"); + rung(&m, 2, 2.19, 1.92, 1, 2, 2); check(!m.s.fine, " two late frames are not yet a scene that does not fit"); + rung(&m, 1, 2.19, 1.92, 1, 2, 2); check(m.s.fine && m.s.k == 2 && m.s.fines == 1, " the third: late frames show at the next vsync, the hold stays 2"); + double r0 = m.real, g0 = m.game; rung(&m, 1800, 2.19, 1.92, 1, 2, 2); + snprintf(b, sizeof b, " 1,800 late frames: stays at the next vsync, game time %.3f of real time (switches %llu)", (m.game - g0) / (m.real - r0), (unsigned long long)m.s.fines); + check(m.s.fine && m.s.fines == 1 && (m.game - g0) / (m.real - r0) > 0.99, b); + rung(&m, 200, 1.2, 1.0, 1, 2, 2); check(m.s.fine, " frames that fit for 400 vsyncs keep it (no flapping)"); + rung(&m, 60, 1.2, 1.0, 1, 2, 2); check(!m.s.fine, " after 8 s of frames that fit, the cap's own link again"); + for (int i = 0; i < 20; i++) { rung(&m, 1, 2.3, 2.0, 1, 2, 2); rung(&m, 49, 1.2, 1.0, 1, 2, 2); } + check(!m.s.fine && m.s.fines == 1, " a single late frame every 50 does not switch"); + sim i = {0}; rung(&i, 120, 1.2, 1.0, 1, 2, 2); rung(&i, 20, 2.5, 0.01, 1, 2, 2); + check(!i.s.fine, "30 fps cap, 20 late frames with the GPU idle (a load): the cap's own link"); + rung(&i, 20, 2.5, 0.01, 1, 2, 2); check(i.s.fine, " 30 in a row are a CPU-heavy scene, as for a step up"); + sim u = {0}; rung(&u, 120, 0.5, 0.4, 1, 1, 3); rung(&u, 600, 2.19, 1.92, 1, 1, 3); + check(u.s.k == 3 && !u.s.fine, "60 fps cap, 36.5 ms frames: the hold rises to 3, no fine pacing needed"); + sim f = {0}; rung(&f, 120, 1.2, 1.0, 1, 2, 2); rung(&f, 10, 2.19, 1.92, 1, 2, 2); run(&f, 1, 2.19, 2, 2, 2); + check(!f.s.fine, "the game fixing its own step ends it"); + sim c = {0}; rung(&c, 120, 1.2, 1.0, 1, 2, 2); rung(&c, 10, 2.19, 1.92, 1, 2, 2); rung(&c, 1, 0.8, 0.7, 1, 1, 3); + check(!c.s.fine, "the cap rising (the hold may rise again) ends it"); + } printf("== %d checks, %d failed ==\n", checks, fail); return fail != 0; } From 99a52c87df5166cc0f53f9c0de2afcaf4791a0b4 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:06:16 +0100 Subject: [PATCH 26/40] OoT-iPhone B-3: the frame step's log lines name the cap, the longest hold it allows and why the hold may go no higher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A log whose frame step sat at hold 2 under a 30 fps cap (iPhone, held 1 / 2 / 3 vsyncs 0 / 1800 / 0 with 80% of frames late) did not say that hold 2 was the ceiling, nor why. fstep_kmax() now gives fstep_tick its kmax and the reason: the 30 fps cap's 30 Hz display link, where a hold of 3 would take 66.7 ms (late frames go to the next vsync of a 60 Hz link instead, B-2); fps_hold off; 120 fps; or the games' own interval logic (1-3) at a 60 cap. The cap-change line ("frame step cap 0 → 30 fps at present 1: hold at most 2 vsyncs, ...") and the periodic game-time line carry it, the latter with whether late frames are shown at the next vsync now and how often that was switched on. t_nvn 277 and t_thermal 21 pass. --- dm/gfx/nvn/hle_nvn.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index cf208995d0..e20ba48cfa 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2368,6 +2368,15 @@ static int fstep_hold(void) { } return h; } +// The longest hold the frame step may give at cap c (nvn_fstep_next's kmax), and why it may go no higher, for the log: a 30 fps cap's display link runs at 30 Hz, where a hold of 3 +// would take two ticks (66.7 ms), so it holds 2 and late frames are shown at the next vsync instead (nvn_fstep.h fine) +static int fstep_kmax(int c, const char **why) { + int kmin = fstep_hz() / c; + if (fps_high()) { *why = "120 fps: the game steps by its target, so a late frame steps down to 60 and no further"; return 2; } + if (!fstep_hold()) { *why = "fps_hold off: frames are not held longer than the cap's vsyncs"; return kmin; } + if (kmin >= 2) { *why = "the 30 fps cap's display link runs at 30 Hz, where a hold of 3 would take 66.7 ms; late frames are shown at the next vsync of a 60 Hz link instead"; return 2; } + *why = "the games' own interval logic uses 1-3"; return FSTEP_MAX; +} // Once per present, before this frame's period is taken: the time since the previous frame's start (shown) and the frame's cost decide this frame's hold and the next frame's step. // Cost = max(CPU: the time between the frames' starts without the pacing wait, GPU: the latest finished frame's GPU time), in vsyncs static void fstep_tick(const win_t *o, int c) { @@ -2381,8 +2390,8 @@ static void fstep_tick(const win_t *o, int c) { static uint64_t pdisp; static int pdisp_ok; uint64_t ndisp = mtl_dispatches(); // and the dispatches (none is a count too) double disps = pdisp_ok ? (double)(ndisp - pdisp) : -1; pdisp = ndisp; pdisp_ok = 1; double cpu = shown - pw / v, cost = cpu > gpu ? cpu : gpu; - int kmin = fstep_hz() / c, kmax = fps_high() ? 2 : !fstep_hold() ? kmin : kmin >= 2 ? 2 : FSTEP_MAX, k0 = g_fs.k, // at 120 fps a late frame must step down whatever fps_hold says: the game steps by the target - kmin0 = g_fs.kmin, probe0 = g_fs.probe; + const char *kwhy; int kmin = fstep_hz() / c, kmax = fstep_kmax(c, &kwhy), k0 = g_fs.k; // at 120 fps a late frame must step down whatever fps_hold says: the game steps by the target + int kmin0 = g_fs.kmin, probe0 = g_fs.probe; int fixed = dm_fps_state ? (int)__atomic_load_n((uint32_t *)(dm_fps_state + 0x24), __ATOMIC_RELAXED) : 0; uint64_t probes0 = g_fs.probes, failed0 = g_fs.failed; // A screen whose rate is not a multiple of 60 (144, 165 Hz) shows the link's 1/60 s ticks on their nearest refreshes (nvn_mtl.m link_range), so a frame on time can show for one @@ -2443,9 +2452,10 @@ static void fstep_tick(const win_t *o, int c) { if (o->presents % 1800 == 0) { LOG("[nvn] 120 fps: held 1 / 2 ticks %llu / %llu frames, %llu late, %llu up / %llu down (%llu probes, %llu failed trials)", (unsigned long long)held[1], (unsigned long long)held[2], (unsigned long long)g_fs.late, (unsigned long long)g_fs.ups, (unsigned long long)g_fs.downs, (unsigned long long)g_fs.probes, (unsigned long long)g_fs.failed); memset(held, 0, sizeof held); } } else if (o->presents % 1800 == 0) { // game time against real time (1.000 = normal speed; lower when a device cannot hold 20 fps or after long stalls) uint64_t vs = __atomic_load_n((uint64_t *)(dm_fps_state + 0x30), __ATOMIC_RELAXED) + fixed_sum, t = now_ns(); - if (t0) LOG("[nvn] frame step: game time %.2f s in %.2f s (%.3f of real time); held 1 / 2 / 3 vsyncs %llu / %llu / %llu frames, %llu late, %llu vsyncs paid back, %.0f dropped, %llu up / %llu down (%llu probes, %llu failed trials)", + char fn[96] = ""; if (kmax == 2 && kmin == 2) snprintf(fn, sizeof fn, ", now %s (%llu times so far)", g_fs.fine ? "at the next vsync" : "on the 30 Hz link", (unsigned long long)g_fs.fines); + if (t0) LOG("[nvn] frame step: game time %.2f s in %.2f s (%.3f of real time); held 1 / 2 / 3 vsyncs %llu / %llu / %llu frames, %llu late, %llu vsyncs paid back, %.0f dropped, %llu up / %llu down (%llu probes, %llu failed trials); cap %d fps, hold at most %d vsyncs: %s%s", (vs - v0) / 60.0, (t - t0) / 1e9, (vs - v0) / 60.0 / ((t - t0) / 1e9), (unsigned long long)held[1], (unsigned long long)held[2], (unsigned long long)held[3], - (unsigned long long)g_fs.late, (unsigned long long)g_fs.paid, g_fs.dropped, (unsigned long long)g_fs.ups, (unsigned long long)g_fs.downs, (unsigned long long)g_fs.probes, (unsigned long long)g_fs.failed); + (unsigned long long)g_fs.late, (unsigned long long)g_fs.paid, g_fs.dropped, (unsigned long long)g_fs.ups, (unsigned long long)g_fs.downs, (unsigned long long)g_fs.probes, (unsigned long long)g_fs.failed, c, kmax, kwhy, fn); t0 = t; v0 = vs; memset(held, 0, sizeof held); } } @@ -2469,7 +2479,8 @@ NVN(nvnQueuePresentTexture, void, NVNqueue *q, NVNwindow *w, int idx) { if (fps_high()) high_cap_tick(); if (dm_fps_state) __atomic_store_n((uint32_t *)(dm_fps_state + 4), (uint32_t)dm_movie_playing(), __ATOMIC_RELAXED); // the stubs' copy of the movie state (hle.h) int c = fps_step_cap(); - if (dm_fps_state) { static int was; if (c != was) { LOG("[nvn] frame step cap %d → %d fps at present %llu%s", was, c, (unsigned long long)o->presents, c == 30 && was == 60 ? (dm_movie_playing() ? " (movie playing)" : thermal_capped() ? " (thermal state serious)" : "") : ""); was = c; } } + if (dm_fps_state) { static int was; if (c != was) { char kw[256] = ""; const char *kwhy; if (c) snprintf(kw, sizeof kw, ": hold at most %d vsyncs, %s", fstep_kmax(c, &kwhy), kwhy); // the cap and why the hold may not rise above it, so a log shows both at once + LOG("[nvn] frame step cap %d → %d fps at present %llu%s%s", was, c, (unsigned long long)o->presents, c == 30 && was == 60 ? (dm_movie_playing() ? " (movie playing)" : thermal_capped() ? " (thermal state serious)" : "") : "", kw); was = c; } } if (c) fstep_tick(o, c); dm_fps_lock_tick(); // after the frame step: at 120 fps the target follows the hold just decided static uint64_t prev_in, prev_out, prev_pass, prev_draw; From 4850da5a127165738f1eb6a956fec7a5bec63325 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:50:00 +0100 Subject: [PATCH 27/40] OoT-iPhone B-2: the 60 Hz link that shows a 30 fps cap's late frames runs at frame latency 2, so a ProMotion screen shows them at 33 / 50 ms On an M1 iPad (120 Hz ProMotion) the first B-2 build switched to the 60 Hz link at frame latency 1, which calls back 16.5 ms before the target time. It took a frame whose GPU work was still running (frame 1951: GPU end 3 ms after its target), the present landed one 120 Hz refresh late, and the link skipped its next tick. At the title (40 ms frames) the acquire, which counts ticks, then held every frame 50 ms (25 -> 20 fps). In gameplay (36 ms frames) the screen showed frames on the 120 Hz grid, 33 / 42 ms (41 / 54 %), instead of 33 / 50. link_latency (nvn_mtl.m, agreed with session A, who owns the file) returns 2 while hle_nvn.c's new nvn_fstep_fine() reports the fine mode: the same 33 ms lead the 30 Hz link has at latency 1, so the input lag is the cap's own link's. link_sync already applies a latency change each frame and logs it. M1 iPad, Deku Tree spot, DM_FPS_TARGET=30, dynamic resolution on, 9,000 frames of camera turns each: before (30 Hz link): 27.9 fps, on screen 33 / 42 / 67 ms 88.6 / 3.8 / 7.5 %, 22.8 of 66.7 ms per 300, p99 66.7 B-2, latency 1: 27.2 fps, 33 / 42 / 50 ms 40.8 / 53.7 / 5.3 %, 0.2 per 300, p99 50; title held at 20 fps B-2, latency 2 (this): 27.7 fps, 33 / 42 / 50 ms 81.0 / 2.3 / 16.7 %, 0.2 per 300, p99 50; title 25 fps Game time 1.000 of real time in all three. present_rescue 101 checks, t_nvn, t_thermal pass. --- dm/gfx/nvn/hle_nvn.c | 1 + dm/gfx/nvn/nvn_mtl.m | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index e20ba48cfa..d8eb53b17c 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2274,6 +2274,7 @@ int nvn_fps_high_now(void) { return atomic_load(&g_fs_k) == 1 ? 120 : 60; } // // waits for the next vblank each frame is held to the event's rate whatever the present cap: Unity's NVN device waits until its vblank time (set from the event) passes // (The Rogue Prince of Persia: main+0x14f65f0), so at 120 every frame came two ticks late and the hold fell back to 60. The hold's trial at 120 must already have the event at 120 int nvn_vsync_hz(void) { return fps_high() && atomic_load(&g_fs_k) == 1 && atomic_load(&g_high_cap) == 120 ? 120 : 60; } +int nvn_fstep_fine(void) { return atomic_load(&g_fs_fine); } // nvn_mtl.m link_latency: the 60 Hz link's lead while it shows a 30 fps cap's late frames int nvn_game_fps(void) { int c = fps_present_cap(); if (c) return c == 30 && (atomic_load(&g_fs_k) == 1 || atomic_load(&g_fs_fine)) && fstep_on() ? 60 : c; uint64_t s = atomic_load(&g_itv2_since); return g_hz > 0 && !g_display_mode && s && now_ns() >= s + 1000000000ull ? 30 : 60; } static void present_cfg(const win_t *o) { if (g_hz >= 0) return; diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index f97213c4a5..96c6749f3e 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -17015,9 +17015,14 @@ static CAFrameRateRange link_range(int mode) { // and why a fixed-rate screen still has about 20 ms more presentation delay with the link at either latency, remain open (docs/reference/frame-rate.md) __attribute__((weak)) int front_fullscreen(void) { return 0; } // front_mac.m; graphics unit tests and iOS have none __attribute__((weak)) int front_screen_variable_refresh(void) { return 0; } // front_mac.m; graphics unit tests and iOS have none +// Latency 2 also while the 60 Hz link shows the late frames of a 30 fps cap at the next vsync (hle_nvn.c nvn_fstep_fine): at latency 1 it calls back 16.5 ms before the target +// time and takes a frame whose GPU work is still running, so on a ProMotion screen the present lands one 120 Hz refresh late and the link skips its next tick; the acquire, which +// counts ticks, then held each frame 50 ms (Ocarina of Time's title on an M1 iPad: 40 ms frames, 25 → 20 fps) or the screen showed them 33 / 38 / 42 ms apart. Latency 2 is the +// 33 ms lead the 30 Hz link has at latency 1, so the input lag is that of the cap's own link. +__attribute__((weak)) int nvn_fstep_fine(void) { return 0; } // hle_nvn.c; graphics unit tests have no frame step // CAMetalDisplayLink accepts only 1 or 2. Compare its readback with the supported request, or an oversized override rewrites and logs every frame. -static int link_latency(void) { return g_present_latency > 0 ? MIN(g_present_latency, 2) : front_fullscreen() && front_screen_variable_refresh() ? 2 : 1; } -static const char *link_latency_why(void) { return g_present_latency > 0 ? "DM_PRESENT_LATENCY" : !front_fullscreen() ? "window" : front_screen_variable_refresh() ? "fullscreen, variable refresh" : "fullscreen, fixed refresh"; } +static int link_latency(void) { return g_present_latency > 0 ? MIN(g_present_latency, 2) : (front_fullscreen() && front_screen_variable_refresh()) || nvn_fstep_fine() ? 2 : 1; } +static const char *link_latency_why(void) { return g_present_latency > 0 ? "DM_PRESENT_LATENCY" : nvn_fstep_fine() ? "late frames of a 30 fps cap at the next vsync" : !front_fullscreen() ? "window" : front_screen_variable_refresh() ? "fullscreen, variable refresh" : "fullscreen, fixed refresh"; } // The 30 Hz link at a 30 fps target, except in fullscreen on a fixed-rate screen. There the link skipped ticks: Ocarina of Time at a 30 fps cap on a MacBook Air M4 (60 Hz) got // 16.5-25 callbacks a second over stretches of 300 frames, also on frames with under 1 ms of GPU, and each skipped tick kept a frame on screen 66.7 ms (a 180 s walk: 388 and 996 // frames over 50 ms; 3-4 in a window, where the same link held 30.0). Why is open (docs/reference/frame-rate.md). The 60 Hz link paces the 30 through the present interval, as it From 3a625e51446d21aefc07e0a80197b3bd9de39990 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:52:51 +0100 Subject: [PATCH 28/40] OoT-iPhone B-1: why a 30 fps cap's late frames left a 66.7 ms gap about every 10, written down with the iPhone's and the M1 iPad's numbers docs/reference/frame-rate.md, "Late frames at a 30 fps cap on the 30 Hz display link": the 30 cap holds at most 2 vsyncs because its link runs at 30 Hz. A 36.5 ms frame comes 3.2 ms later on the tick grid each frame, so after about 10 a tick finds the queue empty and the previous frame stays 66.7 ms: 300 frames over ~333 ticks, 27 fps, 11% of the gaps at 66.7 ms, nothing discarded (iPhone empty ticks 31-33 per 300; "dropped" in the on-screen line is a gap over 1.5 holds). The frame step's paybacks (steps of 3 on 385 of 1,800 frames) do not fall on those gaps. The entry also holds the fix (B-2), the M1 iPad's three runs, and what is open: the iPad title's uneven gaps at 40 ms frames, the title movie's pictures (~210 of 300 frames get one), hold 3 under the cap (not needed), the power cost, and the iPhone, not yet measured. The game note's iPhone/iPad section and "Found on this game" link to it; the symptom index is regenerated. test_docs passes. --- dm/games/0400486024DEC000/README.md | 2 ++ docs/agents/symptoms.md | 1 + docs/reference/frame-rate.md | 17 +++++++++++++++++ 3 files changed, 20 insertions(+) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 225265eedd..0e335bbb00 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -87,6 +87,7 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, - **Memory.** iOS gives the app 7.97 GB of room against the game's 9.00 GB pool, so the guest goes into files by itself ([the guest in files](../../../docs/reference/memory.md#ios-ends-the-app-once-the-guests-touched-pages-reach-its-memory-limit)): footprint 2.2-2.3 GB, headroom to the memory limit 5.87 GB at its lowest, 4.4 GB of guest pages on disk. Metal's working set is 5,461 MB: [textures in GPU-only memory share bytes](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description) there by themselves, and the residency set holds 3.2 GB. With each render target in a recycled heap of its own, the set reached 5,184 MB at an outdoor save load and every command buffer failed out of memory. - **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. - **iPhone** (current, 2026-11-09): plays on an iPhone 17 (`iPhone18,3`, 8 GB, A19, iOS 26.6.1) from the owner's save in Kokiri Forest, with the launcher's 30 fps cap and Dynamic resolution on. Built from upstream `main` 04811a63 with this branch and, for testing, PRs #1842, #1901, #1942 and #1751, chain `39eb2f781d07`. iOS gives the app 5.96 GB of room (the iPad 7.97), Metal's working set is 5,461 MB. Before [the pool table left shader-code pools out](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) it stuttered too much to play: the guest's file pages were read back all the time, 14-17 fps. Now 27 fps standing, 27-30 walking, frame p50 36.5 / p95 40 ms, GPU p50 32 ms, the guest file's page-ins at 100-400 every 10 s. Frames just over the cap's 33.3 ms are late: about 30 of 300 are dropped, a light stutter walking. The title, with its movie on its own clock, holds 30 fps. An earlier build without these fixes ended in a kernel panic after the save loaded (`wait for termination failed … AppleCentauriManager`). +- **Pacing at the launcher's 30 fps cap** (2026-11-10, branch `iphone-b`). The hold cannot rise above 2 vsyncs there, and frames of 36 ms took one 66.7 ms gap about every 11 frames on the 30 Hz link: iPhone walk in Kokiri Forest 30-33 per 300; M1 iPad at the Deku Tree save, dynamic resolution on, 22.8 per 300 at 27.9 fps. With late frames shown at the next vsync of a 60 Hz link at frame latency 2, the M1 iPad showed 27.7 fps with gaps of 33 / 50 ms (81 / 17%) and 0.2 of 66.7 ms per 300 ([why and the three runs](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link)). The owner found the camera turning smooth on the 30 Hz link and did not judge the new build by eye. Open: the iPad's title (40 ms frames) still shows uneven 29-50 ms gaps, and its movie gets a new picture on only about 210 of 300 game frames (`[codec] AV1 decoder` lines: about 600 dropped to the movie's clock per 300 frames), where the iPhone's 33 ms title gets one every frame; the iPhone is not yet measured with the change. - **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. - Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needed [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` was linked otherwise); main has it since 9983d659. @@ -220,6 +221,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [The GPU idles with the next command buffer already committed](../../../docs/reference/frame-rate.md#the-gpu-idles-with-the-next-command-buffer-already-committed): the 25-40 ms frames. - [The uniform copy thread's hand-off convoys on its mutex](../../../docs/reference/frame-rate.md#the-uniform-copy-threads-hand-off-convoys-on-its-mutex): resolved. On one host build (M4 Pro) the replay thread spent half its busy samples in `ubc_wait`, 221-333k instructions a draw and 23-57 ms of submit CPU a frame against 38.9k and 7.1 ms with `DM_NVN_UB_COPY_THREAD=0`; about 35 shadows a frame that `um_overlay` writes into are now copied on the submit thread (PR #2078). - [The 30 Hz display link in fullscreen on a fixed-rate screen](../../../docs/reference/frame-rate.md#the-30-hz-display-link-in-fullscreen-on-a-fixed-rate-screen): 66.7 ms frames at a 30 fps cap in fullscreen on a MacBook Air M4, with the frame-delta change that applies the cap. +- [Late frames at a 30 fps cap on the 30 Hz display link](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link): one 66.7 ms gap about every 11 frames at the launcher's 30 fps cap on an iPhone and an M1 iPad, and on ProMotion the frame latency the 60 Hz link needs. - [Pool tables and readback buffers made anew each frame](../../../docs/reference/memory.md#pool-tables-and-readback-buffers-made-anew-each-frame): about 1,500 residency changes per 300 frames. - [Pool tables patched instead of built whole](../../../docs/reference/nvn-objects.md#pool-tables-patched-instead-of-built-whole): the game registers something between most of its draws, its streamed textures map and unmap virtual pool runs dozens of times a second, finalized descriptors stay registered and textures take over each other's storage; each of these walked the 33,025-entry texture table again, about 53 s of a seven-minute walk. - [Pool shaders taken as depth readers](../../../docs/reference/rendering.md#pool-shaders-taken-as-depth-readers): about 80 extra passes a frame, and half the frames over budget. diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index b94c109564..364a5bd677 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -557,6 +557,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - in fullscreen on a MacBook Pro's ProMotion screen, a 60 fps game held 40 fps with its GPU at 10 ms a frame: frame times alternated 16.7 and 33.3 ms, and each `on-screen` line counted half its frames as held too long (`dropped 150` of 300, `discarded 0`). The same route in a window held 60.0 fps, and the title screen showed the same 40 fps in fullscreen. → [The 60 Hz link at 40 fps in fullscreen on a ProMotion screen](../reference/frame-rate.md#the-60-hz-link-at-40-fps-in-fullscreen-on-a-promotion-screen) - on a MacBook Air M2's fixed 60 Hz built-in display, the reporter measured about 50 ms from the game's present call to on-screen presentation with the display link, at both requested frame latencies 1 and 2, versus 29 ms with `DM_PRESENT_LINK=0`. After loading, all three runs held 60 fps with 0.3% of on-screen intervals outside 16.7 ± 2 ms and no frame-step changes. → [Fixed-rate display-link delay remains at latency 1](../reference/frame-rate.md#fixed-rate-display-link-delay-remains-at-latency-1) - at a 30 fps target, in fullscreen on a fixed 60 Hz screen, frames stay on screen 66.7 ms, often many in a row; in a window a late frame stays 50 ms. The `display link up to tick` lines count 300 ticks per 300 presents, but over 12-18 s instead of 10. → [The 30 Hz display link in fullscreen on a fixed-rate screen](../reference/frame-rate.md#the-30-hz-display-link-in-fullscreen-on-a-fixed-rate-screen) +- at a 30 fps cap (the iOS launcher's), frames a little longer than 33.3 ms show 33.3 ms apart with one 66.7 ms gap about every 10 frames. The `on-screen up to frame` lines count 30-33 dropped frames per 300 (p99 66.7 ms) and none discarded. The frame step holds 2 vsyncs throughout (`held 1 / 2 / 3 vsyncs 0 / 1800 / 0`), with most frames late. Game time keeps to real time (1.000), so it is pacing, not slow motion. On a ProMotion screen, frames of 40 ms show at irregular 29 / 38 / 42 / 50 ms gaps instead. → [Late frames at a 30 fps cap on the 30 Hz display link](../reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link) - autorelease work takes a measurable share of replay time, with temporary arena-buffer ownership traffic a few times per draw. → [Arena buffer ownership on allocation](../reference/frame-rate.md#arena-buffer-ownership-on-allocation) - a game runs at about 10 fps with little GPU work: the submit thread waits for a command allocator (`mtl_sub_begin` → `mtl_wait_at`), the GPU sits idle between encoders for hundreds of ms (`DM_NVN_GPUPROF`), the Metal 4 `SubmissionQueue` thread spends nearly all its time in `iokit_user_client_trap` (almost all system time), and the `memory up to frame` line shows a residency set of 10,000 objects or more. → [Dedicated c1 constants as slices of a bank](../reference/frame-rate.md#dedicated-c1-constants-as-slices-of-a-bank) - the submit thread waits for the GPU at the start of a submission (`mtl_sub_begin` → `mtl_wait_at`), many times a frame, in a game that makes many small submissions a frame; the `translation detail` line's `GPU waits so far` climbs with all 8 allocator pools in use. → [One command allocator for each of a frame's submissions](../reference/frame-rate.md#one-command-allocator-for-each-of-a-frames-submissions) diff --git a/docs/reference/frame-rate.md b/docs/reference/frame-rate.md index d7b1124f48..0fbda4ac8e 100644 --- a/docs/reference/frame-rate.md +++ b/docs/reference/frame-rate.md @@ -449,6 +449,23 @@ Frame pacing: the host's frame step, engine frame limiters, present intervals an - **Check:** `make -C dm/gfx/tests GAME=<id> check_t_present_rescue` (`thirty`: the link's mode and range at 30 fps in a window, in fixed-rate and variable-refresh fullscreen, and back). - **Not checked:** variable-refresh fullscreen at 30 fps, external fixed-rate screens, other games at a 30 fps target, and the power cost of 60 callbacks a second instead of 30. +### Late frames at a 30 fps cap on the 30 Hz display link + +- **Symptom:** at a 30 fps cap (the iOS launcher's), frames a little longer than 33.3 ms show 33.3 ms apart with one 66.7 ms gap about every 10 frames. The `on-screen up to frame` lines count 30-33 dropped frames per 300 (p99 66.7 ms) and none discarded. The frame step holds 2 vsyncs throughout (`held 1 / 2 / 3 vsyncs 0 / 1800 / 0`), with most frames late. Game time keeps to real time (1.000), so it is pacing, not slow motion. On a ProMotion screen, frames of 40 ms show at irregular 29 / 38 / 42 / 50 ms gaps instead. +- **Cause:** a 30 fps cap holds at most 2 vsyncs (`fstep_kmax` in `hle_nvn.c`), because its display link runs at 30 Hz (`link_30hz`), where a hold of 3 would take two ticks, 66.7 ms. A game slower than the tick never waits at the acquire, and the link takes one frame per tick, in order. Each frame is ready a few ms later on the tick grid than the one before (36.5 − 33.3 = 3.2 ms on the iPhone), so after about 10 frames a tick finds the queue empty (`empty ticks` in the `display link up to tick` lines: 31-33 per 300 ticks), and the previous frame stays for a second tick. That gives 300 frames over about 333 ticks: 27 fps, 11% of the gaps at 66.7 ms. No frame is lost (`discarded 0`, `old frames dropped by merging` unchanged); the on-screen count calls a gap of more than 1.5 holds dropped. The frame step pays the time back with steps of 3 vsyncs (385 of 1,800 frames), which do not fall on the 66.7 ms gaps, so the motion is uneven twice over. +- **Fix:** late frames that cannot step the hold up set `nvn_fstep.fine` (`nvn_fstep.h`): the same late frames that would step up, 3 in a row or `FSTEP_RUN` with the GPU idle among them, at the longest hold allowed. While it is set at a 30 fps cap, `nvn_game_fps` reports 60, so `link_sync` runs the 60 Hz link. That link holds each frame its two vsyncs (`g_link_hold_until`) and shows a late one at the next vsync, 50 ms; it is never faster than 30. The mode ends after 8 s with no late frame, when the hold may rise again (the cap rose), or while the game fixes its own step, and both switches are logged. The 60 Hz link then runs at frame latency 2 (`link_latency` in `nvn_mtl.m`): at latency 1 it calls back 16.5 ms before the target time, and on a ProMotion screen it took a frame whose GPU work was still running. The present landed one 120 Hz refresh late and the link skipped its next tick, so the acquire, which counts ticks, held the title's 40 ms frames at 50 ms (25 → 20 fps), and gameplay frames of 36 ms showed 33 / 42 ms apart. Latency 2 is the lead the 30 Hz link has at latency 1, so the input lag is the cap's own. The `on-screen` lines count the gaps at the hold, one vsync longer, and two or more longer; the frame step's lines name the cap, the longest hold it allows and why. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md), on an iPhone 17 (`iPhone18,3`) walking Kokiri Forest (frames p50 36.5 / p95 40 ms, GPU p50 32 ms), and on an M1 iPad Pro at the Deku Tree save, with dynamic resolution on and 9,000 frames of camera turns per run: + + | Build | fps shown | gaps of 33 / 42 / 50 / 67 ms | 66.7 ms gaps per 300 | on-screen p99 | + |---|---|---|---|---| + | 30 Hz link (before) | 27.9 | 88.6 / 3.8 / 0 / 7.5 % | 22.8 | 66.7 ms | + | 60 Hz link, latency 1 | 27.2 | 40.8 / 53.7 / 5.3 / 0.1 % | 0.2 | 50.0 ms | + | 60 Hz link, latency 2 | 27.7 | 81.0 / 2.3 / 16.7 / 0.1 % | 0.2 | 50.0 ms | + + The frames took 36 ms (GPU 34 ms), and game time was 1.000 of real time in all three runs. +- **Open:** the iPad's title (40 ms frames, GPU 38 ms) still shows irregular 29-50 ms gaps with the 60 Hz link at latency 2, as it did on the 30 Hz link. A 30 fps cap could instead fall to a steady 20 (hold 3 on the 60 Hz link) when most frames are late; it was not needed here, since the 33 / 50 pacing removed the 66.7 ms gaps. The power cost of 60 callbacks a second while late frames are paced this way is not measured, and the iPhone has not yet been measured with the change. +- **Check:** `make -C dm/gfx/tests GAME=<id> t_fstep` (`30 fps cap, …`: the switch on the third late frame, no flapping over 400 vsyncs, back after 8 s, idle-GPU frames, the 60 fps cap's hold 3, the game's own step and a cap that rises). + ## CPU cost and pacing ### Arena buffer ownership on allocation From 47412a7933514b6a7e1a105bf1f116f6367e2ba0 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:53:16 +0100 Subject: [PATCH 29/40] =?UTF-8?q?OoT-iPhone=20B-3:=20the=20game-time=20lin?= =?UTF-8?q?e=20counts=20the=20switches=20to=20the=20next=20vsync=20as=20"N?= =?UTF-8?q?=C3=97=20so=20far",=20not=20"1=20times"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- dm/gfx/nvn/hle_nvn.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index d8eb53b17c..bc5872d621 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -2453,7 +2453,7 @@ static void fstep_tick(const win_t *o, int c) { if (o->presents % 1800 == 0) { LOG("[nvn] 120 fps: held 1 / 2 ticks %llu / %llu frames, %llu late, %llu up / %llu down (%llu probes, %llu failed trials)", (unsigned long long)held[1], (unsigned long long)held[2], (unsigned long long)g_fs.late, (unsigned long long)g_fs.ups, (unsigned long long)g_fs.downs, (unsigned long long)g_fs.probes, (unsigned long long)g_fs.failed); memset(held, 0, sizeof held); } } else if (o->presents % 1800 == 0) { // game time against real time (1.000 = normal speed; lower when a device cannot hold 20 fps or after long stalls) uint64_t vs = __atomic_load_n((uint64_t *)(dm_fps_state + 0x30), __ATOMIC_RELAXED) + fixed_sum, t = now_ns(); - char fn[96] = ""; if (kmax == 2 && kmin == 2) snprintf(fn, sizeof fn, ", now %s (%llu times so far)", g_fs.fine ? "at the next vsync" : "on the 30 Hz link", (unsigned long long)g_fs.fines); + char fn[96] = ""; if (kmax == 2 && kmin == 2) snprintf(fn, sizeof fn, ", now %s (switched to the next vsync %llu× so far)", g_fs.fine ? "at the next vsync" : "on the 30 Hz link", (unsigned long long)g_fs.fines); if (t0) LOG("[nvn] frame step: game time %.2f s in %.2f s (%.3f of real time); held 1 / 2 / 3 vsyncs %llu / %llu / %llu frames, %llu late, %llu vsyncs paid back, %.0f dropped, %llu up / %llu down (%llu probes, %llu failed trials); cap %d fps, hold at most %d vsyncs: %s%s", (vs - v0) / 60.0, (t - t0) / 1e9, (vs - v0) / 60.0 / ((t - t0) / 1e9), (unsigned long long)held[1], (unsigned long long)held[2], (unsigned long long)held[3], (unsigned long long)g_fs.late, (unsigned long long)g_fs.paid, g_fs.dropped, (unsigned long long)g_fs.ups, (unsigned long long)g_fs.downs, (unsigned long long)g_fs.probes, (unsigned long long)g_fs.failed, c, kmax, kwhy, fn); From 2fa00c2dc05cf4778e1b97e8febc2076ee5b4861 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 02:32:34 +0100 Subject: [PATCH 30/40] OoT-iPhone A (merge): the 20 graphics tests that build nvn_mtl.m and hle_nvn.c as one file compile again B-2 (effae9a5) gave nvn_mtl.m a weak default nvn_fstep_fine for the tests that have no frame step, and hle_nvn.c its strong definition. The tests that include both files in one translation unit (nvn_replay and 19 t_* tests) then stopped compiling: "redefinition of 'nvn_fstep_fine'". They rename nvn_mtl.m's weak nvn_game_fps around its include for the same reason; they now rename nvn_fstep_fine the same way. All of them build and pass again, including check_t_present_snap (8 runs) and check_t_capture_live (2 of 2 replays match). --- dm/gfx/tests/nvn_replay.m | 2 ++ dm/gfx/tests/t_capture_live.m | 2 ++ dm/gfx/tests/t_copy_alias.m | 2 ++ dm/gfx/tests/t_cpu_tex.m | 2 ++ dm/gfx/tests/t_large_mdi.m | 2 ++ dm/gfx/tests/t_linear_readback.m | 2 ++ dm/gfx/tests/t_link_ticks.m | 2 ++ dm/gfx/tests/t_mesh_first_use.m | 2 ++ dm/gfx/tests/t_mip_extent.m | 2 ++ dm/gfx/tests/t_pool_notes.m | 2 ++ dm/gfx/tests/t_present_crop.m | 2 ++ dm/gfx/tests/t_present_snap.m | 2 ++ dm/gfx/tests/t_queue_lifetime.m | 2 ++ dm/gfx/tests/t_queue_origin.m | 2 ++ dm/gfx/tests/t_readback.m | 2 ++ dm/gfx/tests/t_render_counter_overlap.m | 2 ++ dm/gfx/tests/t_render_enable_queue.m | 2 ++ dm/gfx/tests/t_rt_lifetime.m | 2 ++ dm/gfx/tests/t_sparse_tiles.m | 2 ++ dm/gfx/tests/t_wait_event.m | 2 ++ 20 files changed, 40 insertions(+) diff --git a/dm/gfx/tests/nvn_replay.m b/dm/gfx/tests/nvn_replay.m index 27e6752ca6..4bbfce5a06 100644 --- a/dm/gfx/tests/nvn_replay.m +++ b/dm/gfx/tests/nvn_replay.m @@ -11,8 +11,10 @@ #include <spawn.h> #include <sys/wait.h> #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_capture_live.m b/dm/gfx/tests/t_capture_live.m index bfbb6d0270..f87d033117 100644 --- a/dm/gfx/tests/t_capture_live.m +++ b/dm/gfx/tests/t_capture_live.m @@ -6,8 +6,10 @@ // ./t_capture_live <capture path> #import <Foundation/Foundation.h> #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_copy_alias.m b/dm/gfx/tests/t_copy_alias.m index 24254f2b06..9dafab322e 100644 --- a/dm/gfx/tests/t_copy_alias.m +++ b/dm/gfx/tests/t_copy_alias.m @@ -1,7 +1,9 @@ // Copies through distinct Metal buffers and sparse guest aliases must see earlier GPU writes. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_cpu_tex.m b/dm/gfx/tests/t_cpu_tex.m index 0888da9d8b..c879ab0e83 100644 --- a/dm/gfx/tests/t_cpu_tex.m +++ b/dm/gfx/tests/t_cpu_tex.m @@ -2,8 +2,10 @@ // not in it staying cheap with thousands of consecutive serials in it (a CFSet that hashed serials as numbers probed a whole bucket run per miss: half of Batman: Arkham Knight's // rendering thread with 6,442 CPU-written textures) #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_large_mdi.m b/dm/gfx/tests/t_large_mdi.m index 36183ebb13..9bacda0bda 100644 --- a/dm/gfx/tests/t_large_mdi.m +++ b/dm/gfx/tests/t_large_mdi.m @@ -1,8 +1,10 @@ // Issue #941: the last live indirect draw paints the target. The count is written on the GPU after recording. #import <Foundation/Foundation.h> #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_linear_readback.m b/dm/gfx/tests/t_linear_readback.m index 87ac1654d4..f33ee832a0 100644 --- a/dm/gfx/tests/t_linear_readback.m +++ b/dm/gfx/tests/t_linear_readback.m @@ -1,8 +1,10 @@ // NVN linear image storage: GPU image writes must be visible through the memory-pool CPU mapping. // Synthetic 160x90 RGBA8_SRGB IMAGE|LINEAR textures use a valid 640-byte row pitch. A linear texture whose rows no view can take (DM_TBUF_ALIGN=64) is uploaded at its stride. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_link_ticks.m b/dm/gfx/tests/t_link_ticks.m index 9cdee02a5a..a0d0c4e176 100644 --- a/dm/gfx/tests/t_link_ticks.m +++ b/dm/gfx/tests/t_link_ticks.m @@ -19,8 +19,10 @@ static int test_clock_gettime(clockid_t clock, struct timespec *t) { #define clock_gettime test_clock_gettime #define pthread_cond_timedwait test_pthread_cond_timedwait #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #undef pthread_cond_timedwait #undef clock_gettime #define g_ft g_ft_hle diff --git a/dm/gfx/tests/t_mesh_first_use.m b/dm/gfx/tests/t_mesh_first_use.m index 415838a1ef..def78eb1dc 100644 --- a/dm/gfx/tests/t_mesh_first_use.m +++ b/dm/gfx/tests/t_mesh_first_use.m @@ -3,8 +3,10 @@ // A mesh shader with no task draws more workgroups than Metal's mesh grid holds (DMSH_MESH_GRID) through its object // function's blocks (dmsh_meshstage.py main0_o), directly and indirectly: every workgroup draws its own pixel once. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_mip_extent.m b/dm/gfx/tests/t_mip_extent.m index bab3bcf737..7fb4d504e5 100644 --- a/dm/gfx/tests/t_mip_extent.m +++ b/dm/gfx/tests/t_mip_extent.m @@ -1,8 +1,10 @@ // Real NVN recording entry → production Metal replay; each mip of 8x6 is hand-calculated independently; with no viewport set, an explicit viewport is not used to dodge it. // nvn_mtl.m's nvn_game_fps is a weak default and hle_nvn.c gives the strong definition; in one translation unit the weak one is renamed (as in t_present_crop.m) #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle // both files have a static g_ft, so hle_nvn.c's is renamed too (as in t_present_crop.m) #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_pool_notes.m b/dm/gfx/tests/t_pool_notes.m index 8af3b9d877..5b85b06cef 100644 --- a/dm/gfx/tests/t_pool_notes.m +++ b/dm/gfx/tests/t_pool_notes.m @@ -5,8 +5,10 @@ // what a table built whole holds. // Ocarina of Time maps and unmaps virtual pool runs dozens of times a second and registers something between most of its draws; each of those walked its 33,025-entry texture table #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_present_crop.m b/dm/gfx/tests/t_present_crop.m index b13edd0170..9cb043b62e 100644 --- a/dm/gfx/tests/t_present_crop.m +++ b/dm/gfx/tests/t_present_crop.m @@ -1,8 +1,10 @@ // Present crop regression: colored 8×8 texture, closed-form RGBA expectations; real Metal rendering, no dependence on the game or on cross-run determinism. // nvn_mtl.m's nvn_game_fps is a weak default and hle_nvn.c gives the strong definition; in one translation unit that is a redefinition, so the weak one is renamed (this test does not look at the frame-rate target) #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle // both files have a static g_ft (nvn_mtl.m's fragment argument table, hle_nvn.c's textures a flush can mark CPU-written), so hle_nvn.c's is renamed too #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_present_snap.m b/dm/gfx/tests/t_present_snap.m index 081010dc92..821b6df0e5 100644 --- a/dm/gfx/tests/t_present_snap.m +++ b/dm/gfx/tests/t_present_snap.m @@ -1,7 +1,9 @@ // Snapshot lifecycle: real NVN acquire/present/suspend, real Metal copies, and controlled encoder/GPU delays. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_queue_lifetime.m b/dm/gfx/tests/t_queue_lifetime.m index 1e0b22ca9b..f9e63cf628 100644 --- a/dm/gfx/tests/t_queue_lifetime.m +++ b/dm/gfx/tests/t_queue_lifetime.m @@ -1,8 +1,10 @@ // Submitted commands own their CPU descriptors. Pool storage and shader code stay // alive while a wait parks the queue; only program/texture objects are reused. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_queue_origin.m b/dm/gfx/tests/t_queue_origin.m index d0e5d89076..6e48c1fa33 100644 --- a/dm/gfx/tests/t_queue_origin.m +++ b/dm/gfx/tests/t_queue_origin.m @@ -2,8 +2,10 @@ // used to construct later viewports can change independently. Check actual // culling and front_facing pixels while alternating two queues on one device. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_readback.m b/dm/gfx/tests/t_readback.m index 22a50245c5..6351d0c0dc 100644 --- a/dm/gfx/tests/t_readback.m +++ b/dm/gfx/tests/t_readback.m @@ -1,8 +1,10 @@ // NVN readback contract: a pending clear must reach both GPU buffer copies and CPU reads. // Expected 8x4 RGBA values are literal bytes, independent of the Metal replay's state. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_render_counter_overlap.m b/dm/gfx/tests/t_render_counter_overlap.m index 914a05a507..9166c617cc 100644 --- a/dm/gfx/tests/t_render_counter_overlap.m +++ b/dm/gfx/tests/t_render_counter_overlap.m @@ -5,8 +5,10 @@ // (it did in 3 of 60 runs); the predicates, which the host's encode-time values decide, are checked. #include "game_profile.h" #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_render_enable_queue.m b/dm/gfx/tests/t_render_enable_queue.m index b90ec2fffd..f64bed0fdd 100644 --- a/dm/gfx/tests/t_render_enable_queue.m +++ b/dm/gfx/tests/t_render_enable_queue.m @@ -2,8 +2,10 @@ // Every path uses public NVN recording; the last case gates only GPU completion. #include "game_profile.h" #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_rt_lifetime.m b/dm/gfx/tests/t_rt_lifetime.m index aba5af14ca..59ef1d3b25 100644 --- a/dm/gfx/tests/t_rt_lifetime.m +++ b/dm/gfx/tests/t_rt_lifetime.m @@ -1,8 +1,10 @@ // Queue bindings outlive completed command recordings. Exercise the public NVN // interface and verify pixels after the caller reuses the recording's memory. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_sparse_tiles.m b/dm/gfx/tests/t_sparse_tiles.m index 4526c2acb9..685236d35a 100644 --- a/dm/gfx/tests/t_sparse_tiles.m +++ b/dm/gfx/tests/t_sparse_tiles.m @@ -5,8 +5,10 @@ // its last place and each new place reads the memory (the CPU's fill behind a GPU-written place, the GPU's write to the memory as a buffer), a tile cut by the level's edge shares nothing, and a // new pool over the same memory never finds the old pool's tile. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #include "../nvn/nvn_mtl.m" #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft diff --git a/dm/gfx/tests/t_wait_event.m b/dm/gfx/tests/t_wait_event.m index c620a49970..2d321e5999 100644 --- a/dm/gfx/tests/t_wait_event.m +++ b/dm/gfx/tests/t_wait_event.m @@ -1,10 +1,12 @@ // Contract fixture: real public NVN lookup -> recording -> QueueSubmit -> Metal replay. // An event wait gates GPU work, but never blocks CPU submission or another NVN queue. #define nvn_game_fps nvn_game_fps_mtl_default +#define nvn_fstep_fine nvn_fstep_fine_mtl_default #define nvn_event_publish_gpu event_test_publish #include "../nvn/nvn_mtl.m" #undef nvn_event_publish_gpu #undef nvn_game_fps +#undef nvn_fstep_fine #define g_ft g_ft_hle #include "../nvn/hle_nvn.c" #undef g_ft From 1d91b35614d9e3e1359affbaec7f592a74bf5d49 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 02:35:37 +0100 Subject: [PATCH 31/40] OoT-iPhone B-1: the iPhone's numbers with late frames at the next vsync, and why a hold of 3 would not help its slow gameplay Session A's iPhone 17 run of the merged build at the 30 cap (iphoneA4_merged30): the first gameplay at thermal 1 showed gaps at the hold / one vsync longer / two or more of 1,098 / 997 / 1 (p99 45.8-50 ms), where the 30 Hz link gave 30-33 gaps of 66.7 ms per 300. Later, at thermal 2, gameplay ran CPU-bound at 13-22 fps and game time fell to 0.731: frames longer than the 3 vsyncs a step can carry (FSTEP_MAX), which the frame step already hands the game at hold 2 while it pays back, so option (a), hold 3 under the cap, would change neither game time nor what the 60 Hz link shows. The frame-rate entry and the game note say so. --- dm/games/0400486024DEC000/README.md | 2 +- docs/reference/frame-rate.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 0e335bbb00..ddf5699171 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -87,7 +87,7 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, - **Memory.** iOS gives the app 7.97 GB of room against the game's 9.00 GB pool, so the guest goes into files by itself ([the guest in files](../../../docs/reference/memory.md#ios-ends-the-app-once-the-guests-touched-pages-reach-its-memory-limit)): footprint 2.2-2.3 GB, headroom to the memory limit 5.87 GB at its lowest, 4.4 GB of guest pages on disk. Metal's working set is 5,461 MB: [textures in GPU-only memory share bytes](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description) there by themselves, and the residency set holds 3.2 GB. With each render target in a recycled heap of its own, the set reached 5,184 MB at an outdoor save load and every command buffer failed out of memory. - **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. - **iPhone** (current, 2026-11-09): plays on an iPhone 17 (`iPhone18,3`, 8 GB, A19, iOS 26.6.1) from the owner's save in Kokiri Forest, with the launcher's 30 fps cap and Dynamic resolution on. Built from upstream `main` 04811a63 with this branch and, for testing, PRs #1842, #1901, #1942 and #1751, chain `39eb2f781d07`. iOS gives the app 5.96 GB of room (the iPad 7.97), Metal's working set is 5,461 MB. Before [the pool table left shader-code pools out](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) it stuttered too much to play: the guest's file pages were read back all the time, 14-17 fps. Now 27 fps standing, 27-30 walking, frame p50 36.5 / p95 40 ms, GPU p50 32 ms, the guest file's page-ins at 100-400 every 10 s. Frames just over the cap's 33.3 ms are late: about 30 of 300 are dropped, a light stutter walking. The title, with its movie on its own clock, holds 30 fps. An earlier build without these fixes ended in a kernel panic after the save loaded (`wait for termination failed … AppleCentauriManager`). -- **Pacing at the launcher's 30 fps cap** (2026-11-10, branch `iphone-b`). The hold cannot rise above 2 vsyncs there, and frames of 36 ms took one 66.7 ms gap about every 11 frames on the 30 Hz link: iPhone walk in Kokiri Forest 30-33 per 300; M1 iPad at the Deku Tree save, dynamic resolution on, 22.8 per 300 at 27.9 fps. With late frames shown at the next vsync of a 60 Hz link at frame latency 2, the M1 iPad showed 27.7 fps with gaps of 33 / 50 ms (81 / 17%) and 0.2 of 66.7 ms per 300 ([why and the three runs](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link)). The owner found the camera turning smooth on the 30 Hz link and did not judge the new build by eye. Open: the iPad's title (40 ms frames) still shows uneven 29-50 ms gaps, and its movie gets a new picture on only about 210 of 300 game frames (`[codec] AV1 decoder` lines: about 600 dropped to the movie's clock per 300 frames), where the iPhone's 33 ms title gets one every frame; the iPhone is not yet measured with the change. +- **Pacing at the launcher's 30 fps cap** (2026-11-10). The hold cannot rise above 2 vsyncs there, and frames of 36 ms took one 66.7 ms gap about every 11 frames on the 30 Hz link: iPhone walk in Kokiri Forest 30-33 per 300; M1 iPad at the Deku Tree save, dynamic resolution on, 22.8 per 300 at 27.9 fps. With late frames shown at the next vsync of a 60 Hz link at frame latency 2, the M1 iPad showed 27.7 fps with gaps of 33 / 50 ms (81 / 17%) and 0.2 of 66.7 ms per 300 ([why and the three runs](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link)). The owner found the camera turning smooth on the 30 Hz link and did not judge the new build by eye. Open: the iPad's title (40 ms frames) still shows uneven 29-50 ms gaps, and its movie gets a new picture on only about 210 of 300 game frames (`[codec] AV1 decoder` lines: about 600 dropped to the movie's clock per 300 frames), where the iPhone's 33 ms title gets one every frame. On the iPhone 17 with the change (Kokiri Forest, camera turns), the first gameplay at thermal 1 showed gaps of 33 / 50 ms (52 / 48%) and 1 longer one in 2,096, p99 50 ms, game time 0.999; at thermal 2 it ran CPU-bound at 13-22 fps (frames over 50 ms, footprint up to 3.2 GiB) and game time fell to 0.731, more than the 3 vsyncs a step can carry, which a hold of 3 would not change. - **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. - Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needed [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` was linked otherwise); main has it since 9983d659. diff --git a/docs/reference/frame-rate.md b/docs/reference/frame-rate.md index 0fbda4ac8e..57f3f51fc1 100644 --- a/docs/reference/frame-rate.md +++ b/docs/reference/frame-rate.md @@ -463,7 +463,9 @@ Frame pacing: the host's frame step, engine frame limiters, present intervals an | 60 Hz link, latency 2 | 27.7 | 81.0 / 2.3 / 16.7 / 0.1 % | 0.2 | 50.0 ms | The frames took 36 ms (GPU 34 ms), and game time was 1.000 of real time in all three runs. -- **Open:** the iPad's title (40 ms frames, GPU 38 ms) still shows irregular 29-50 ms gaps with the 60 Hz link at latency 2, as it did on the 30 Hz link. A 30 fps cap could instead fall to a steady 20 (hold 3 on the 60 Hz link) when most frames are late; it was not needed here, since the 33 / 50 pacing removed the 66.7 ms gaps. The power cost of 60 callbacks a second while late frames are paced this way is not measured, and the iPhone has not yet been measured with the change. + + On the iPhone 17 at the 30 cap (10 minutes of camera turns in Kokiri Forest), the first gameplay at thermal 1 (frames 2,400-4,200, GPU about 40 ms) showed 1,098 / 997 / 1 gaps at the hold / one vsync longer / two or more (52 / 48 / 0.05%), at p99 45.8-50 ms and 24-27 fps, with game time at 0.999 of real time. Before the change it had 30-33 gaps of 66.7 ms per 300 (p99 66.7 ms). The switch to the 60 Hz link came on the third late frame, the switch back after 8 s on time on the title, and the switch to the 60 Hz link again with the first late gameplay frames, each with its `display link: frame latency` line. +- **Open:** the iPad's title (40 ms frames, GPU 38 ms) still shows irregular 29-50 ms gaps with the 60 Hz link at latency 2, as it did on the 30 Hz link. A 30 fps cap could instead fall to a steady 20 (hold 3 on the 60 Hz link) when most frames are late. It was not needed at 36-40 ms, since the 33 / 50 pacing removed the 66.7 ms gaps, and it would not help slower frames either: while it pays back, the frame step already hands the game a step of 3 vsyncs (`FSTEP_MAX`) at hold 2, and the 60 Hz link already shows each frame at its first vsync after 2. The iPhone's later gameplay at thermal 2 ran CPU-bound at 13-22 fps (frames 65-200 ms, footprint up to 3.2 GiB): more than 3 vsyncs a frame, so game time fell to 0.731 of real time (8,216 vsyncs dropped from the debt) whatever the hold. The power cost of 60 callbacks a second while late frames are paced this way is not measured. - **Check:** `make -C dm/gfx/tests GAME=<id> t_fstep` (`30 fps cap, …`: the switch on the third late frame, no flapping over 400 vsyncs, back after 8 s, idle-GPU frames, the 60 fps cap's hold 3, the game's own step and a cap that rises). ## CPU cost and pacing From d17df17ee0fa0ca987d2779bad6bd2dee6fa970e Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 03:48:02 +0100 Subject: [PATCH 32/40] OoT-iPhone A-2 docs: the iPhone 17 frame's profile, the fix's A/B, and the allocator pool tried and reverted Game note, "For an iPhone or iPad build": a new current bullet for the iPhone 17 at the owner's Kokiri spot. Its table gives the GPU span, the GPU work and the gaps before and inside submissions, before and after A-2. Gameplay GPU p50 is 21-24 ms, under the cap's 33.3 ms, so shader pull requests #1752 and #1833 were not tried. The frame is CPU-bound at 16-20 fps under thermal state 1-2. The bullet records the uncapped A/B of A-2, and a command allocator pool of 128 that removed its waits but no frame time, at a cost of about 300 MB (not kept). Open: translation cost, samples-passed reports, footprint growth. The CPU timeline the table was measured with is a local diagnostic, not in this pull request. rendering.md: the A-2 entry gets its measured result. A Found-on line in the note, and the outdated iPhone line under "Not run on an iPhone yet" now points to the section. --- dm/games/0400486024DEC000/README.md | 21 +++++++++++++++++++-- docs/reference/rendering.md | 2 +- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index ddf5699171..66ff8a7d00 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -88,8 +88,24 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, - **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. - **iPhone** (current, 2026-11-09): plays on an iPhone 17 (`iPhone18,3`, 8 GB, A19, iOS 26.6.1) from the owner's save in Kokiri Forest, with the launcher's 30 fps cap and Dynamic resolution on. Built from upstream `main` 04811a63 with this branch and, for testing, PRs #1842, #1901, #1942 and #1751, chain `39eb2f781d07`. iOS gives the app 5.96 GB of room (the iPad 7.97), Metal's working set is 5,461 MB. Before [the pool table left shader-code pools out](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) it stuttered too much to play: the guest's file pages were read back all the time, 14-17 fps. Now 27 fps standing, 27-30 walking, frame p50 36.5 / p95 40 ms, GPU p50 32 ms, the guest file's page-ins at 100-400 every 10 s. Frames just over the cap's 33.3 ms are late: about 30 of 300 are dropped, a light stutter walking. The title, with its movie on its own clock, holds 30 fps. An earlier build without these fixes ended in a kernel panic after the save loaded (`wait for termination failed … AppleCentauriManager`). - **Pacing at the launcher's 30 fps cap** (2026-11-10). The hold cannot rise above 2 vsyncs there, and frames of 36 ms took one 66.7 ms gap about every 11 frames on the 30 Hz link: iPhone walk in Kokiri Forest 30-33 per 300; M1 iPad at the Deku Tree save, dynamic resolution on, 22.8 per 300 at 27.9 fps. With late frames shown at the next vsync of a 60 Hz link at frame latency 2, the M1 iPad showed 27.7 fps with gaps of 33 / 50 ms (81 / 17%) and 0.2 of 66.7 ms per 300 ([why and the three runs](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link)). The owner found the camera turning smooth on the 30 Hz link and did not judge the new build by eye. Open: the iPad's title (40 ms frames) still shows uneven 29-50 ms gaps, and its movie gets a new picture on only about 210 of 300 game frames (`[codec] AV1 decoder` lines: about 600 dropped to the movie's clock per 300 frames), where the iPhone's 33 ms title gets one every frame. On the iPhone 17 with the change (Kokiri Forest, camera turns), the first gameplay at thermal 1 showed gaps of 33 / 50 ms (52 / 48%) and 1 longer one in 2,096, p99 50 ms, game time 0.999; at thermal 2 it ran CPU-bound at 13-22 fps (frames over 50 ms, footprint up to 3.2 GiB) and game time fell to 0.731, more than the 3 vsyncs a step can carry, which a hold of 3 would not change. +- **iPhone 17: GPU time and what limits the frame** (current, 2026-11-10). The owner's launcher settings (30 fps cap, dynamic resolution on, encoding on its own thread), Kokiri Forest from the owner's save, camera turns by `iphone_cam.sh` in the work folder. GPU frames from `DM_NVN_GPUPROF` with a CPU timeline of the host (a local diagnostic build): + + | Frame | Build | GPU span | GPU work | Gaps before a submission / inside one | Submissions, encoders | + |---|---|---|---|---|---| + | 3000 | before | 37.9 ms | 18.9 ms | 17.9 / 1.1 ms | 32, 152 | + | 3600 | before | 34.9 ms | 20.2 ms | 13.6 / 1.1 ms | 30, 153 | + | 8100 | after | 27.7 ms | 19.7 ms | 7.4 / 0.7 ms | 31, 153 | + | 9000 | after | 27.0 ms | 21.0 ms | 5.3 / 0.7 ms | 31, 164 | + | 12600 | after | 25.3 ms | 20.1 ms | 4.7 / 0.5 ms | 31, 160 | + | 13500 | after | 32.9 ms | 23.2 ms | 7.6 / 2.1 ms | 31, 156 | + + The gaps before each submission were the replay waiting for the encode thread at every submission's end: [the GPU idles before each submission while the replay waits for the encode thread](../../../docs/reference/rendering.md#the-gpu-idles-before-each-submission-while-the-replay-waits-for-the-encode-thread). What remains is the GPU waiting while the host translates a long submission (2-8 ms for some), whose first segment goes out only after 64 draws. GPU p50 in gameplay is 21-24 ms (median of the 300-frame timing windows), under the 33.3 ms the cap allows, so shader pull requests #1752 and #1833 were not tried. + - The frame is CPU-bound: 16-20 fps, frame p50 40-54 ms, p95 100-118 ms, 114-196 frames dropped per 300 (before the fix: 15.0 fps, p50 48.6 ms, 132 dropped; that run also built pipelines). The submit thread spends 18-25 ms of CPU a frame in 25-32 ms of wall clock (about 30 M cycles at 0.5 cycles per instruction), and the game's main thread yields 13,000-22,000 times a frame (5.5-9.4 ms). The phone reaches thermal state 1 after 44-64 s and 2 after 360-390 s; from then on about 30% of CPU time runs on performance cores, at a mean 1.5 GHz on the submit thread. The footprint grows from 0.7 GiB at the title to 3.2 GiB in play. + - A/B of the fix (`DM_NVN_SUB_DRAIN=ab`, `DM_AB=600`): uncapped, the frame step settles on the 3-vsync hold in both modes, mean frame p50 49.0 against 50.4 ms over 12 phase pairs; at the cap the title phases 33.3 against 36.6 ms. It removes about 37 waits a frame from the submit thread, but the frame rate is set by the CPU work. + - **Not this:** the command allocator pool. Its cap of 8 submissions in flight made the submit thread wait about 4 times a frame (3.4-8.2 ms). A cap of 128 (an A/B against 8) removed the waits but no frame time (gameplay frame p50 54.7 against 56.0 ms over 11 phase pairs), and each allocator keeps 8-13 MB, so the footprint grew from 3,230 to 3,535 MiB. Reverted. + - Open: the translation's CPU cost on the phone, the 1,260-1,710 samples-passed reports a frame written at the GPU's end, and the footprint's growth in play. - **Not this:** no thread of the game makes a [clock wait](../../../docs/reference/threads.md#clock-waits-sleep-to-their-predicted-end) (no `clock waits up to frame` line in either run), so that rule's predicted sleep plays no part in its speed. -- Open: the GPU time of the heavy views (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needed [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` was linked otherwise); main has it since 9983d659. +- Open: the GPU time of the heavy views on the M1 iPad (the frame leaves the GPU idle a quarter of the time, and its 110 render passes reload their attachments; on the iPhone 17 most of that idle time was the replay's wait, resolved above). A GPU hang after 18.5 minutes of play on an earlier build (`kIOGPUCommandBufferCallbackErrorHang`, then GPU access revoked): both hung command buffers held the Hi-Z task `a5288d0a6c4971fd` and its mesh `3439aaa071d868ae`, the draws of the M1 Mac's hangs in [PR #1842](https://github.com/seedds/nxnative/pull/1842). One kernel data abort on the iPad (2026-11-09 03:57) in kernel code that handles GPU-wired memory, cause unknown. Building for iOS on 7ffa63cc needs [PR #1906](https://github.com/seedds/nxnative/pull/1906) (`IOBluetooth` is linked otherwise). ## Screen shape @@ -145,7 +161,7 @@ From an empty save: at about 45 s `A` (title), `A` (New Game), then the movie. F - The "Navi" name label in the Great Deku Tree scene was missing on an earlier chain and has not been checked since. Not played yet: the Deku Tree dungeon, combat, later areas. Sound opens with no red line; movies and speech were not listened to. Physics issues reported in PR 1299 were not seen on this route (walking, the ladder, the camera). - Resolved: dotted horizontal rows across the picture in play, in fog and light (GitHub issue 1537): the agl_ndsr_vr networks' HMMA calls outside a chain overwrote an accumulator a chain held ([An HMMA outside a chain overwrote the chain's accumulator](../../../docs/reference/shaders.md#an-hmma-outside-a-chain-overwrote-the-chains-accumulator)), since chain 23a1ea1db092; found by bisecting chains on the Kokiri Forest autosave (0f510fd5's chain clean, bd9f422d's not). Under the game's upscaler they had also grown into white blotches: [LDSM reads rows other lanes stored](../../../docs/reference/shaders.md#ldsm-reads-rows-other-lanes-stored). - Open: GPU hangs in play ([#1326](https://github.com/seedds/nxnative/issues/1326), [#1605](https://github.com/seedds/nxnative/issues/1605)), repeatable in a Kokiri Forest conversation and the Impa escort on an M2 Pro and gone with the pipelines of task `a5288d0a6c4971fd` dropped ([#2011](https://github.com/seedds/nxnative/issues/2011)). They are not [threads returning before a barrier](../../../docs/reference/shaders.md#compute-threads-that-return-before-a-barrier): that task, the culling tasks `47f9b38490dcf9a2` and `99280b7b8aae78df` and mesh `3439aaa071d868ae` have no barrier in their bodies (none of the game's 520 task and mesh shaders has one). The lead in #2011 is pixel shader `bd09ab5fc4fd7680`'s light loop, which runs `min(count, 1000) + 1` times per pixel with per-tile counts the GPU reads. -- On an iPhone 17 and an M1 iPad: [For an iPhone or iPad build](#for-an-iphone-or-ipad-build). Open on the iPhone: frames a little over the 30 fps cap's 33.3 ms (GPU p50 32 ms in Kokiri Forest) make a light stutter. +- The iPhone 17 and the M1 iPad: [For an iPhone or iPad build](#for-an-iphone-or-ipad-build). ## Unused border corrections @@ -171,6 +187,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [The blend helper's output reference on iOS](../../../docs/reference/shaders.md#the-blend-helpers-output-reference-on-ios): resolved, 22,296 of 42,848 Mac shaders failed to compile on macOS 26 ([GitHub issue #2272](https://github.com/seedds/nxnative/issues/2272)); the Mac compile now names the host's minimum macOS. - [A NaN colour under a zero blend weight](../../../docs/reference/shaders.md#a-nan-colour-under-a-zero-blend-weight): resolved, white clouds over the ground mist between Kokiri Forest and Hyrule Field and over the ferns at the camera in Kokiri Forest ([issue #1839](https://github.com/seedds/nxnative/issues/1839)). The mist shader `5ad65c0eefd09c7f` writes NaN at alpha 0 where its layer has density 0; NVIDIA's blender, with the option NVN's driver sets, adds nothing there, Apple's stored the NaN. - [A first start builds every base of the prewarm list before the first frame](../../../docs/reference/rendering.md#a-first-start-builds-every-base-of-the-prewarm-list-before-the-first-frame): the 450-475 s first-start prepare on an M5 Max; the Mac app now bundles the list's pipelines (`--prebake`, on by default) and it took 4.3 s. +- [The GPU idles before each submission while the replay waits for the encode thread](../../../docs/reference/rendering.md#the-gpu-idles-before-each-submission-while-the-replay-waits-for-the-encode-thread): resolved on the iPhone 17, about 30 submissions a frame, 13.6-17.9 ms of GPU gaps before submissions in a 35-38 ms span ([For an iPhone or iPad build](#for-an-iphone-or-ipad-build)). - [The 60 Hz link at 40 fps in fullscreen on a ProMotion screen](../../../docs/reference/frame-rate.md#the-60-hz-link-at-40-fps-in-fullscreen-on-a-promotion-screen): 40 fps in fullscreen on an M5 Pro MacBook Pro with the GPU at 10 ms of the 16.7, every present one 120 Hz refresh late and every other display-link tick skipped; the screen's fixed 60 Hz mode held 60.0 fps there and fullscreen now switches to it, with the 120 Hz link (59 fps) for screens with no such mode. - [A screen-shape stub on the projection setter changes nothing: the camera caches its matrix](../../../docs/reference/rendering.md#a-screen-shape-stub-on-the-projection-setter-changes-nothing-the-camera-caches-its-matrix): the camera copy and the HUD's uniform block copy ([Screen shape](#screen-shape)). - [Textures in GPU-only memory took an allocation per description](../../../docs/reference/memory.md#textures-in-gpu-only-memory-took-an-allocation-per-description): the M1 iPad's GPU out-of-memory failure at the outdoor save load. diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index 9d61f44279..c6a4052ca3 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -1212,7 +1212,7 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Symptom:** with Metal encoding on its own thread (`DM_NVN_ENC_THREAD=1`, the default on iOS), a frame's GPU span is far above its GPU work, and the gaps sit before the first encoder of a submission: `DM_NVN_GPUPROF` with its CPU timeline shows the GPU done with everything committed while the next submission is still being encoded. The wait ledger (`DM_NVN_WAITS=1`) shows `encode thread drain` blocked about once per submission. - **Cause:** `mtl_sub_commit` waited for the encode thread to finish the submission (`rec_drain`) before returning, so the replay never translated the next submission while the encode thread encoded the last one, and the GPU got each submission only once both had finished it. A game that splits its frame at every event signal makes many submissions (each `nvnQueueSubmitCommands` is cut into one per wait, signal and fence by the queue scheduler, `nq_flatten`), and a submission under `cut_draws()` draws has no [segment](frame-rate.md#submissions-handed-to-the-gpu-in-segments) to start the GPU early. - **Fix:** the submission's records go to the encode thread with a note of its fence (`RO_NOTE` on `g_sub_cmt`), and the replay goes on. Everything else that goes to the queue after it is a record too (the next submission's commits, upload batches, snapshot copies, sparse mappings), a CPU wait for the GPU waits for the fence's signal, which comes after the commit, and graphics suspend waits for the last fence. The one direct commit that reads the game's work is the display command buffer of a present without a snapshot: it waits for the note of the submissions before it (`sub_commit_wait`, `pq_item.sub_v`). `DM_NVN_SUB_DRAIN=1` brings the wait back (`ab` alternates with `DM_AB`). -- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) on the iPhone 17 (Kokiri Forest, 2026-11-10): about 30 submissions a frame; frame 3600 spanned 34.9 ms for 20.2 ms of GPU work, and 13.6 of its 14.75 ms of gaps came before a submission's first encoder; the replay thread waited for the encode thread 36 times a frame, 10 ms in all. +- **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) on the iPhone 17 (Kokiri Forest, 2026-11-10): about 30 submissions a frame; frame 3600 spanned 34.9 ms for 20.2 ms of GPU work, and 13.6 of its 14.75 ms of gaps came before a submission's first encoder; the replay thread waited for the encode thread 36 times a frame, 10 ms in all. With the fix, gameplay frames span 25.3-32.9 ms for 19.7-23.2 ms of work, with 4.7-7.6 ms of gaps before submissions; what remains is the GPU waiting while the replay translates a long submission. The frame rate did not change there (A/B uncapped: frame p50 49.0 against 50.4 ms over 12 phase pairs), as the game's CPU work sets it; the title phases at the 30 fps cap went from 36.6 to 33.3 ms. - **Check:** `make -C dm/gfx/tests GAME=<id> t_sub_overlap && dm/gfx/tests/t_sub_overlap` holds the encode thread inside a submission (a queue block) and checks that the submission returns, that its note waits for the encode thread's commit, and that a present's wait holds until then (3 of 8 checks fail with the old wait). `check_t_present_crop` and `check_t_present_snap` run presents with and without the encode thread and snapshots. ### Mesh draws with no task shader above 1024 workgroups From f0e390e30a7085c6e55bfcc887ca2f9a1b451503 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Sat, 7 Nov 2026 20:09:39 -0500 Subject: [PATCH 33/40] DM_NVN_GPUPROF_GEOM=1 stamps each render pass where its vertex and mesh stages finish, so a profile splits a pass between geometry and pixels: Ocarina of Time's colour pass with its 11 mesh draws is 5.8 ms of geometry and 0.2 ms after it A second counter heap takes two relaxed stamps at each render pass's end (after the vertex stage and after the mesh stage); the pass line ends in "geom <us>", the time from the pass's start to the later of the two that falls inside the pass. gpuprof.py's timeline splits each pass into geometry and the time after it, and its table adds a geom ms column. Off by default: without the switch nothing changes. (cherry picked from commit 69552d8f2acb075ed7fbcaf51b92444a0cd18d7f) --- dm/gfx/nvn/nvn_mtl.m | 18 ++++++++++++++++-- dm/gfx/tests/t_gpuprof.m | 10 ++++++++-- dm/tools/gpuprof.py | 32 ++++++++++++++++++++++++++------ dm/tools/tests/test_gpuprof.py | 19 +++++++++++++++++++ docs/env_switches.tsv | 1 + 5 files changed, 70 insertions(+), 10 deletions(-) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 96c6749f3e..f215d0daff 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -1735,6 +1735,10 @@ static inline void m4_end(id e) { enum { GP_MAX = 4096 }; // the largest timestamp heap Metal makes typedef struct { char kind; uint16_t w, h; uint8_t nc, depth, loads; uint32_t draws, seg; uint64_t pass; uint32_t ld_kb, st_kb; uint16_t cfmt, dfmt; } gp_lab; // seg: the command buffer (segment) of the submission it is in static id<MTL4CounterHeap> g_gp_heap; static gp_lab g_gp_lab[GP_MAX / 2]; static int g_gp_n, g_gp_state, g_gp_cur = -1; static long g_gp_f0 = -2, g_gp_f1, g_gp_every; static uint64_t g_gp_frame; +// DM_NVN_GPUPROF_GEOM=1 (with DM_NVN_GPUPROF) also stamps each render pass where its vertex and its mesh stages finish (a second heap, at the pass's end), and the pass line gives +// "geom <us>": the time from the pass's start to the later of the two, inside the pass. On a tile-based GPU a pass's vertex, object and mesh work (binning) runs ahead of most of its +// fragment work, so geom against the rest splits a pass's time between geometry and pixels. Relaxed like the other stamps: a guide, not an exact boundary +static id<MTL4CounterHeap> g_gp_gheap; static int g_gp_geom = -1; static const void *g_gp_e; // the encoder g_gp_cur belongs to: m4_end also ends the upload / diagnostics / present encoders, which may end while a profiled pass is open (#531) static _Atomic int g_gp_busy; // a periodic profile's report has not been printed yet: the heap is still being read static uint64_t g_bw_load, g_bw_store, g_gp_bw_l0, g_gp_bw_s0; // the bandwidth ledger (defined with the pass statistics below) and its values at the last render pass end @@ -1768,7 +1772,10 @@ static void gp_begin(id e, char kind, MTL4RenderPassDescriptor *d) { // right if (!gp_active() || g_gp_state == 2) return; if (!g_gp_heap) { MTL4CounterHeapDescriptor *hd = [MTL4CounterHeapDescriptor new]; hd.type = MTL4CounterHeapTypeTimestamp; hd.count = GP_MAX; NSError *er = nil; g_gp_heap = [g_dev newCounterHeapWithDescriptor:hd error:&er]; if (!g_gp_heap) { MLOG("[RED] GPUPROF: no counter heap: %s", er.description.UTF8String); g_gp_f0 = -1; return; } } - if (!g_gp_state) { g_gp_state = 1; g_gp_n = 0; g_gp_frame = atomic_load(&g_cur_frame); [g_gp_heap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; } + if (g_gp_geom < 0) { const char *ge = getenv("DM_NVN_GPUPROF_GEOM"); g_gp_geom = ge && atoi(ge) > 0; if (g_gp_geom) MLOG("DM_NVN_GPUPROF_GEOM: render passes also stamped where their vertex and mesh stages finish"); } + if (g_gp_geom && !g_gp_gheap) { MTL4CounterHeapDescriptor *hd = [MTL4CounterHeapDescriptor new]; hd.type = MTL4CounterHeapTypeTimestamp; hd.count = GP_MAX; NSError *er = nil; + g_gp_gheap = [g_dev newCounterHeapWithDescriptor:hd error:&er]; if (!g_gp_gheap) { MLOG("[RED] GPUPROF_GEOM: no counter heap: %s", er.description.UTF8String); g_gp_geom = 0; } } + if (!g_gp_state) { g_gp_state = 1; g_gp_n = 0; g_gp_frame = atomic_load(&g_cur_frame); [g_gp_heap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; if (g_gp_geom) [g_gp_gheap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; } if (2 * g_gp_n + 2 > GP_MAX) return; gp_lab l = {kind, 0, 0, 0, 0, 0, kind == 'R' ? (uint32_t)g_stat_draw : g_gp_copies, g_seg, g_stat_pass, 0, 0, 0, 0}; if (kind != 'R') { l.w = (uint16_t)g_gp_disp; l.h = (uint16_t)g_gp_ubd; } if (d) { for (int i = 0; i < 8; i++) { MTLRenderPassColorAttachmentDescriptor *c = d.colorAttachments[i]; if (!c.texture) continue; @@ -1786,17 +1793,24 @@ static void gp_end(id e) { // right before that encoder ends int k = g_gp_cur; g_gp_cur = -1; g_gp_e = NULL; if (g_gp_lab[k].kind == 'R') { g_gp_lab[k].ld_kb = (uint32_t)((g_bw_load - g_gp_bw_l0) >> 10); g_gp_lab[k].st_kb = (uint32_t)((g_bw_store - g_gp_bw_s0) >> 10); } // since the previous pass end (rp_end); sd_finish has set the stores if (g_gp_lab[k].kind == 'R') g_gp_lab[k].draws = (uint32_t)g_stat_draw - g_gp_lab[k].draws; else { g_gp_lab[k].draws = g_gp_copies - g_gp_lab[k].draws; g_gp_lab[k].w = (uint16_t)(g_gp_disp - g_gp_lab[k].w); g_gp_lab[k].h = (uint16_t)(g_gp_ubd - g_gp_lab[k].h); } + if (g_gp_lab[k].kind == 'R' && g_gp_geom > 0) { // where this pass's geometry work finishes (DM_NVN_GPUPROF_GEOM) + [(id<MTL4RenderCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed afterStage:MTLRenderStageVertex intoHeap:g_gp_gheap atIndex:2 * k]; + [(id<MTL4RenderCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed afterStage:MTLRenderStageMesh intoHeap:g_gp_gheap atIndex:2 * k + 1]; } if (g_gp_lab[k].kind == 'R') [(id<MTL4RenderCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed afterStage:MTLRenderStageFragment intoHeap:g_gp_heap atIndex:2 * k + 1]; else [(id<MTL4ComputeCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed intoHeap:g_gp_heap atIndex:2 * k + 1]; } static void gp_report(int n, gp_lab *lab, uint64_t frame) { // the frame's last command buffer has completed NSData *r = [g_gp_heap resolveCounterRange:NSMakeRange(0, 2 * n)]; const uint64_t *t = r.bytes; if (!t) { free(lab); atomic_store(&g_gp_busy, 0); return; } + NSData *gr = g_gp_geom > 0 ? [g_gp_gheap resolveCounterRange:NSMakeRange(0, 2 * n)] : nil; const uint64_t *gt = gr.bytes; // DM_NVN_GPUPROF_GEOM's stamps, two per encoder double tick_ns = 1e9 / (double)[g_dev queryTimestampFrequency], tot[2] = {0}, span = 0; uint64_t t0 = 0, t1 = 0, ld = 0, st = 0; int np = 0; for (int i = 0; i < n; i++) { uint64_t a = t[2 * i], b = t[2 * i + 1]; if (!a || !b || b < a) continue; double us = (b - a) * tick_ns / 1e3; const gp_lab *l = &lab[i]; tot[l->kind != 'R'] += us; if (!t0 || a < t0) t0 = a; if (b > t1) t1 = b; if (l->kind == 'R') { np++; ld += l->ld_kb; st += l->st_kb; - MLOG("[gpuprof] F%llu %4d %8.1f us render pass %llu %ux%u colors %u depth %u loads %u draws %u start %+.1f seg %u load %u KB store %u KB fmt %u/%u", (unsigned long long)frame, i, us, (unsigned long long)l->pass, l->w, l->h, l->nc, l->depth, l->loads, l->draws, (a - t[0]) * tick_ns / 1e3, l->seg, l->ld_kb, l->st_kb, l->cfmt, l->dfmt); } + char geom[32] = ""; // the later of the vertex and mesh stamps that falls inside the pass; none when neither does + if (gt) { uint64_t g = 0; for (int j = 0; j < 2; j++) { uint64_t v = gt[2 * i + j]; if (v >= a && v <= b && v > g) g = v; } + if (g) snprintf(geom, sizeof geom, " geom %.1f", (g - a) * tick_ns / 1e3); } + MLOG("[gpuprof] F%llu %4d %8.1f us render pass %llu %ux%u colors %u depth %u loads %u draws %u start %+.1f seg %u load %u KB store %u KB fmt %u/%u%s", (unsigned long long)frame, i, us, (unsigned long long)l->pass, l->w, l->h, l->nc, l->depth, l->loads, l->draws, (a - t[0]) * tick_ns / 1e3, l->seg, l->ld_kb, l->st_kb, l->cfmt, l->dfmt, geom); } else MLOG("[gpuprof] F%llu %4d %8.1f us compute / copy after pass %llu start %+.1f seg %u copies %u (uniform updates %u) dispatches %u", (unsigned long long)frame, i, us, (unsigned long long)l->pass, (a - t[0]) * tick_ns / 1e3, l->seg, l->draws, l->h, l->w); } span = (t1 - t0) * tick_ns / 1e3; MLOG("[gpuprof] frame %llu: render %.2f ms, compute / copy %.2f ms over %d encoders (%d render passes, attachments load %.1f MB store %.1f MB), first start to last end %.2f ms", (unsigned long long)frame, tot[0] / 1e3, tot[1] / 1e3, n, np, ld / 1024.0, st / 1024.0, span / 1e3); diff --git a/dm/gfx/tests/t_gpuprof.m b/dm/gfx/tests/t_gpuprof.m index 8a0f96edaf..1929aee847 100644 --- a/dm/gfx/tests/t_gpuprof.m +++ b/dm/gfx/tests/t_gpuprof.m @@ -43,10 +43,10 @@ - (void)complete { @end // A draw pass answers only the render encoder's timestamp selector, an upload encoder only the compute one: a stamp sent to the wrong kind throws, as Metal 4's encoders do. @interface ProfileRender : NSObject -@property int stamps; @property NSUInteger last; +@property int stamps, geom; @property NSUInteger last; @property MTLRenderStages geom_stages; @end @implementation ProfileRender -- (void)writeTimestampWithGranularity:(MTL4TimestampGranularity)g afterStage:(MTLRenderStages)s intoHeap:(id)h atIndex:(NSUInteger)i { (void)g; (void)s; (void)h; _stamps++; _last = i; } +- (void)writeTimestampWithGranularity:(MTL4TimestampGranularity)g afterStage:(MTLRenderStages)s intoHeap:(id)h atIndex:(NSUInteger)i { (void)g; _stamps++; _last = i; if (h && h == (id)g_gp_gheap) { _geom++; _geom_stages |= s; } } @end @interface ProfileCompute : NSObject @property int stamps; @@ -85,5 +85,11 @@ int main(void) { @autoreleasepool { @try { gp_end(upload); } @catch (NSException *x) { CHECK(0, "another queue's encoder got the open pass's timestamp"); } CHECK(upload.stamps == 0 && g_gp_cur == k, "another queue's encoder does not close the open pass"); gp_end(pass); CHECK(pass.stamps == 2 && pass.last == 2 * (NSUInteger)k + 1 && g_gp_cur < 0, "the pass's own end writes its end stamp"); + CHECK(pass.geom == 0, "without DM_NVN_GPUPROF_GEOM a pass writes no geometry stamps"); + // DM_NVN_GPUPROF_GEOM: a pass's end also stamps where its vertex and mesh stages finish, in the second heap, before its fragment end stamp + g_gp_gheap = (id)[ProfileHeap new]; g_gp_geom = 1; ProfileRender *gpass = [ProfileRender new]; + gp_begin(gpass, 'R', nil); int k2 = g_gp_cur; gp_end(gpass); + CHECK(gpass.stamps == 4 && gpass.geom == 2 && gpass.geom_stages == (MTLRenderStageVertex | MTLRenderStageMesh), "a pass's end stamps its vertex and mesh stages"); + CHECK(gpass.last == 2 * (NSUInteger)k2 + 1, "the fragment end stamp still comes last, at the pass's own index"); printf("t_gpuprof: %d checks, %d failed\n", checks, failed); return failed != 0; } } diff --git a/dm/tools/gpuprof.py b/dm/tools/gpuprof.py index b373c4e21b..5398b71eac 100755 --- a/dm/tools/gpuprof.py +++ b/dm/tools/gpuprof.py @@ -11,13 +11,16 @@ gpuprof.py log.txt --timeline 600 every encoder of frame 600 in GPU order, with the idle gaps between them gpuprof.py log.txt --ab the [ab] phases: medians per configuration and the difference from the base +With DM_NVN_GPUPROF_GEOM=1 a render pass line ends in "geom <us>" (from the pass's start to where its vertex and mesh stages finish): the timeline splits each pass into +geometry and the time after it, and the table adds the mean geometry ms per frame of each group. + Times are GPU timestamps taken where each encoder begins and ends (relaxed: encoders may overlap), so the sum over encoders is not the frame's GPU time; the frame line's span (first start to last end) is closer to it. See docs/agents/05-debugging.md, "GPU time on the device". """ import argparse, collections, re, statistics, sys ENC = re.compile(r'\[gpuprof\] F(\d+)\s+(\d+)\s+([\d.]+) us (render pass|compute / copy)(.*)') -RP = re.compile(r'^ (\d+) (\d+)x(\d+) colors (\d+) depth (\d+) loads (\d+) draws (\d+) start ([+-][\d.]+) seg (\d+)(?: load (\d+) KB store (\d+) KB)?(?: fmt (\d+)/(\d+))?') +RP = re.compile(r'^ (\d+) (\d+)x(\d+) colors (\d+) depth (\d+) loads (\d+) draws (\d+) start ([+-][\d.]+) seg (\d+)(?: load (\d+) KB store (\d+) KB)?(?: fmt (\d+)/(\d+))?(?: geom ([\d.]+))?') CP = re.compile(r'^ after pass (\d+) start ([+-][\d.]+) seg (\d+) copies (\d+) \(uniform updates (\d+)\) dispatches (\d+)') FRAME = re.compile(r'\[gpuprof\] frame (\d+): render ([\d.]+) ms, compute / copy ([\d.]+) ms over (\d+) encoders.*?first start to last end ([\d.]+) ms') AB = re.compile(r'\[ab\] phase (\d+) (\S+): (\d+) frames, GPU p50 ([\d.]+) / p90 ([\d.]+) / mean ([\d.]+) ms, frame p50 ([\d.]+) / p90 ([\d.]+) / mean ([\d.]+) ms' @@ -39,7 +42,8 @@ def fmt_name(v): return MTL_FORMATS.get(v, str(v)) if v else '-' -Enc = collections.namedtuple('Enc', 'frame idx us kind pass_ w h colors depth loads draws start seg load_kb store_kb cfmt dfmt copies ubd dispatches') +# geom: us from a render pass's start to where its vertex and mesh stages finish (DM_NVN_GPUPROF_GEOM=1), None without +Enc = collections.namedtuple('Enc', 'frame idx us kind pass_ w h colors depth loads draws start seg load_kb store_kb cfmt dfmt copies ubd dispatches geom', defaults=(None,)) def parse(lines): @@ -55,7 +59,8 @@ def parse(lines): if not r: continue g = [int(x) if x is not None else 0 for x in (r.group(1), r.group(2), r.group(3), r.group(4), r.group(5), r.group(6), r.group(7))] e = Enc(f, idx, us, 'R', g[0], g[1], g[2], g[3], g[4], g[5], g[6], float(r.group(8)), int(r.group(9)), - int(r.group(10) or 0), int(r.group(11) or 0), int(r.group(12) or 0), int(r.group(13) or 0), 0, 0, 0) + int(r.group(10) or 0), int(r.group(11) or 0), int(r.group(12) or 0), int(r.group(13) or 0), 0, 0, 0, + float(r.group(14)) if r.group(14) else None) else: c = CP.match(rest) if not c: continue @@ -119,6 +124,18 @@ def aggregate(frames, by='shape', only=None): return rows, nf +def geom_means(frames, by='shape', only=None): + """Mean geometry us per frame of each group that has DM_NVN_GPUPROF_GEOM stamps ({group: us}; empty without them)""" + tot, nf = collections.defaultdict(float), 0 + for f, es in frames.items(): + if only and f not in only: continue + nf += 1 + ref = max((e.w * e.h for e in es if e.kind == 'R' and e.colors), default=0) + for e in es: + if e.geom is not None: tot[shape_key(e) if by == 'shape' else class_key(e, ref)] += e.geom + return {k: v / nf for k, v in tot.items()} if nf else {} + + def parse_ab(lines): out = [] for line in lines: @@ -188,7 +205,8 @@ def main(argv=None): gap = max(0.0, e.start - end); end = max(end, e.start + e.us) if e.kind == 'R': what = (f'pass {e.pass_} {e.w}x{e.h} {e.colors}c{"+d" if e.depth else ""} draws {e.draws} loads {e.loads} ' - f'load {e.load_kb} KB store {e.store_kb} KB {fmt_name(e.cfmt)}/{fmt_name(e.dfmt)}') + f'load {e.load_kb} KB store {e.store_kb} KB {fmt_name(e.cfmt)}/{fmt_name(e.dfmt)}' + + (f' | geometry {e.geom:.1f} us, after it {e.us - e.geom:.1f} us' if e.geom is not None else '')) else: what = f'compute / copy after pass {e.pass_}: copies {e.copies} (uniform updates {e.ubd}) dispatches {e.dispatches}' print(f' {e.idx:4d} +{e.start:9.1f} {e.us:8.1f}{" gap %.1f" % gap if gap >= 1 else "":>12s} {what}') @@ -203,11 +221,13 @@ def main(argv=None): rows, nf = aggregate(frames, a.by, only) tot = sum(r[2] for r in rows) or 1 print(f'\nmean per frame over {nf} frames, by {a.by}:') - print(f' {"group":52s} {"passes":>7s} {"ms":>7s} {"share":>6s} {"us/pass":>8s} {"draws":>7s} {"load MB":>8s} {"store MB":>8s}') + geo = geom_means(frames, a.by, only) + print(f' {"group":52s} {"passes":>7s} {"ms":>7s} {"share":>6s} {"us/pass":>8s} {"draws":>7s} {"load MB":>8s} {"store MB":>8s}' + (f' {"geom ms":>8s}' if geo else '')) shown, rest = rows[:a.top], rows[a.top:] if rest: shown = shown + [('other (%d groups)' % len(rest), sum(r[1] for r in rest), sum(r[2] for r in rest), sum(r[3] for r in rest), sum(r[4] for r in rest), sum(r[5] for r in rest))] for k, n, us, dr, ld, st in shown: - print(f' {k:52s} {n:7.1f} {us / 1e3:7.2f} {100 * us / tot:5.1f}% {us / n if n else 0:8.1f} {dr:7.0f} {ld / 1024:8.1f} {st / 1024:8.1f}') + g = (f' {geo[k] / 1e3:8.2f}' if k in geo else f' {"-":>8s}') if geo else '' + print(f' {k:52s} {n:7.1f} {us / 1e3:7.2f} {100 * us / tot:5.1f}% {us / n if n else 0:8.1f} {dr:7.0f} {ld / 1024:8.1f} {st / 1024:8.1f}' + g) print(f' {"total":52s} {sum(r[1] for r in rows):7.1f} {tot / 1e3:7.2f}') return 0 diff --git a/dm/tools/tests/test_gpuprof.py b/dm/tools/tests/test_gpuprof.py index 243e64d810..f06d42b95e 100644 --- a/dm/tools/tests/test_gpuprof.py +++ b/dm/tools/tests/test_gpuprof.py @@ -55,6 +55,25 @@ def test_parse(self): self.assertEqual(self.flines[600], {'render_ms': 0.71, 'compute_ms': 0.03, 'encoders': 5, 'span_ms': 0.76}) self.assertEqual(self.flines[1200]['span_ms'], 0.91) + def test_geom(self): + # DM_NVN_GPUPROF_GEOM: a pass line ending in its geometry time parses and is grouped; lines without it give None and no column + self.assertIsNone(self.frames[600][0].geom); self.assertEqual(gpuprof.geom_means(self.frames), {}) + frames, _ = gpuprof.parse([rp(600, 0, 300.0, 24, 1600, 900, 1, 1, 50, 0.0, 5625, 5625, 92, 252) + ' geom 120.5', + rp(600, 1, 100.0, 25, 1600, 900, 1, 0, 1, 300.0, 5625, 5625, 115, 0)]) + e = frames[600][0] + self.assertEqual((e.geom, e.dfmt, e.draws), (120.5, 252, 50)) + self.assertIsNone(frames[600][1].geom) + self.assertEqual(gpuprof.geom_means(frames), {'1600x900 1c+d RG11B10Float/Depth32Float': 120.5}) + out = io.StringIO() + with tempfile.NamedTemporaryFile('w', suffix='.log', delete=False) as f: f.write('\n'.join(LOG[:1] + [rp(600, 0, 300.0, 24, 1600, 900, 1, 1, 50, 0.0, 5625, 5625, 92, 252) + ' geom 120.5']) + '\n') + try: + with contextlib.redirect_stdout(out): self.assertEqual(gpuprof.main([f.name, '--timeline', '600']), 0) + self.assertIn('geometry 120.5 us, after it 179.5 us', out.getvalue()) + out = io.StringIO() + with contextlib.redirect_stdout(out): self.assertEqual(gpuprof.main([f.name]), 0) + self.assertIn('geom ms', out.getvalue()); self.assertIn(' 0.12', out.getvalue()) + finally: os.unlink(f.name) + def test_frame_stats(self): s = gpuprof.frame_stats(self.frames[600]) self.assertEqual((s['encoders'], s['passes'], s['draws']), (5, 4, 802)) diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 33067c55da..2eb213ed50 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -174,6 +174,7 @@ DM_NVN_GEO_LEAN switch `0` draws passthrough geometry shaders through their full DM_NVN_GEO_MESH switch `0` draws passthrough geometry pairs that have a mesh library (`geo_mesh.tsv`) through the compute path; by default they are mesh draws in the render pass (`geo_mesh_draw`) DM_NVN_GEO_MESH_RUN switch `0` dispatches every mesh geometry draw on its own; by default consecutive mesh draws of one pair whose result does not depend on their order are drawn as one dispatch (`geo_mesh_take`) DM_NVN_GPUPROF diagnostic `=<f>`, `<f1>-<f2>` or `every:<n>`: logs the GPU start and duration of every encoder of those frames (`[gpuprof]` lines from Metal 4 counter heap timestamps) +DM_NVN_GPUPROF_GEOM diagnostic `=1` (with `DM_NVN_GPUPROF`) also stamps each render pass where its vertex and mesh stages finish; the pass line ends in `geom <us>` and `gpuprof.py` splits each pass into geometry and the time after it DM_NVN_GPU_STORAGE switch `0` stops what the GPU writes into a texture in storage the CPU can read (a GPU texture, or one uploaded from its storage) from being written into that storage on the GPU (on by default; `gsb_target`, `gsb_store`), to measure it DM_NVN_GPU_TS switch `submission` or `segment_gaps`: what timestamp reports record instead of the default `segment` (the GPU start / end of the segment holding each report, without the GPU's idle time between the submission's command buffers): one time for the whole submission, or segment times with that idle time DM_NVN_HILITE diagnostic `=<fragment shader key>,…`: those draws use a solid colour instead (magenta, green, cyan, yellow, blue, white in order), to tell wrong geometry from wrong shading From 0ca0186203e4f789561ce8b18b9c357c0d340202 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 00:29:19 +0100 Subject: [PATCH 34/40] OoT-iPhone A-1: DM_NVN_GPUPROF logs the host's CPU timeline beside a profiled frame's encoders While a frame is profiled, each submission's encoding, each real commit to the queue (on the encode thread, from before its uniform-copy wait), every blocking CPU wait for the GPU by its wait-ledger site, and the NVN queue worker's parks (no work, or held by an event or sync wait) are logged as "[gpuprof] F<n> cpu" lines on the encoders' clock; encoder lines name their submission ("sub"). Metal 4's counter-heap stamps are Mach ticks (checked on an M4 Mac: a stamp falls between the commit's and the completion's mach_absolute_time), so no conversion is needed; the report says whether the clocks agree. First iPhone 17 run (Kokiri Forest, this build's pipelines still compiling): frame 3600 spans 34.9 ms for 20.2 ms of GPU work over 30 submissions; 13.6 of the 14.75 ms of gaps come before the first encoder of a submission, with the GPU done and the replay still encoding that submission (A-2). t_gpuprof checks the timeline through the real wait, commit and park paths. --- dm/gfx/nvn/nvn_mtl.m | 78 ++++++++++++++++++++++++++++++++-------- dm/gfx/nvn/nvn_queue.inc | 11 ++++-- dm/gfx/nvn/nvn_rec.h | 8 +++-- dm/gfx/tests/t_gpuprof.m | 27 ++++++++++++++ 4 files changed, 104 insertions(+), 20 deletions(-) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index f215d0daff..1a7b9f8554 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -278,7 +278,22 @@ static void ws_report(uint64_t frame, uint64_t frames) { if (n && o < sizeof line - 160) o += snprintf(line + o, sizeof line - o, " %s %.2f/frame (blocked %.2f, %.3f ms);", ws_name[i], (double)n / frames, (double)b / frames, ns / 1e6 / frames); } MLOG("[waits] up to frame %llu, per frame:%s", (unsigned long long)frame, o ? line : " none"); } +// DM_NVN_GPUPROF's CPU timeline: while a frame is profiled, the host's CPU side is logged beside its encoders' GPU stamps, so a gap between two encoders says what the GPU waited +// for: each submission's encoding (its begin to its end on the replay thread), each real commit to the queue (from before its uniform-copy wait to the commit's return; the encode +// thread makes it), every blocking CPU wait for the GPU by its wait-ledger site (the game's SyncWait, the allocator, ...), and the NVN queue worker parked with nothing it can +// replay (no work, or the first job of each queue held by an event or sync wait). Mach ticks: Metal 4's counter-heap timestamps are the same clock (measured on an M-series Mac: +// a stamp falls between the commit's and the completion's mach_absolute_time), so the report places them on the encoders' timeline with no conversion +enum { GT_SUB, GT_COMMIT, GT_WAIT, GT_QPARK, GT_MAX = 2048 }; +typedef struct { uint64_t t0, t1, a, b; uint32_t kind; } gt_ev; +static gt_ev g_gt[GT_MAX]; static _Atomic int g_gt_n, g_gt_on; +static uint64_t g_sub_serial; static uint32_t g_seg; // tentative: defined with the submission state below static _Atomic uint64_t g_sub_cmt; // the last submission whose commit the encode thread has made (its fence; with no encode thread, the last committed: mtl_sub_commit) +static inline void gt_put(uint32_t kind, uint64_t t0, uint64_t a, uint64_t b) { + if (__builtin_expect(!atomic_load_explicit(&g_gt_on, memory_order_relaxed), 1)) return; + int i = atomic_fetch_add_explicit(&g_gt_n, 1, memory_order_relaxed); if (i < GT_MAX) g_gt[i] = (gt_ev){t0, mach_absolute_time(), a, b, kind}; +} +// The NVN queue worker (nvn_queue.inc) parked from t0 until now: why (0 no work queued, 1 an event wait, 2 a sync wait) and the held queue's serial +void mtl_gp_note_park(uint64_t t0, unsigned why, uint64_t queue) { gt_put(GT_QPARK, t0, why, queue); } #include "nvn_rec.h" // deferred encoding (DM_NVN_ENC_THREAD) static NSHashTable *g_res_objects; static char g_tex_range_key; // res_top: a GPU-made texture's storage (guest address, NVN size, pool flags) @@ -1627,7 +1642,10 @@ void mtl_prepare(void) { } static void sub_cut_begin(void); static void ts_sub_begin(void), ts_boundary(void); static int ts_on(void); static void gmem_commit_patch(void); // the raw pool table words of the arguments encoded so far follow the pools (gmem_slot_note) +static uint64_t g_gt_sub0; // DM_NVN_GPUPROF's timeline: when this submission's encoding began +static uint64_t g_gt_queue; // the NVN queue being replayed (mtl_queue_activate) void mtl_sub_begin(void) { + g_gt_sub0 = mach_absolute_time(); mtl_init(); g_sub_serial++; g_seg = 0; if (g_pkg_kept) { g_pkg_kept = 0; nvn_dirty_touch(); } seg_clk_put(g_sclk); g_sclk = NULL; sub_cut_begin(); @@ -1733,7 +1751,7 @@ static inline void m4_end(id e) { // A render pass line also gives what it loaded and stored of its attachments (KB, the bandwidth ledger's view: DontCare stores are not counted) and the Metal pixel formats of // colour attachment 0 and of the depth / stencil attachment (0: none) enum { GP_MAX = 4096 }; // the largest timestamp heap Metal makes -typedef struct { char kind; uint16_t w, h; uint8_t nc, depth, loads; uint32_t draws, seg; uint64_t pass; uint32_t ld_kb, st_kb; uint16_t cfmt, dfmt; } gp_lab; // seg: the command buffer (segment) of the submission it is in +typedef struct { char kind; uint16_t w, h; uint8_t nc, depth, loads; uint32_t draws, seg; uint64_t pass; uint32_t ld_kb, st_kb; uint16_t cfmt, dfmt; uint32_t sub; } gp_lab; // seg: the command buffer (segment) of the submission it is in; sub: the submission (g_sub_serial) static id<MTL4CounterHeap> g_gp_heap; static gp_lab g_gp_lab[GP_MAX / 2]; static int g_gp_n, g_gp_state, g_gp_cur = -1; static long g_gp_f0 = -2, g_gp_f1, g_gp_every; static uint64_t g_gp_frame; // DM_NVN_GPUPROF_GEOM=1 (with DM_NVN_GPUPROF) also stamps each render pass where its vertex and its mesh stages finish (a second heap, at the pass's end), and the pass line gives // "geom <us>": the time from the pass's start to the later of the two, inside the pass. On a tile-based GPU a pass's vertex, object and mesh work (binning) runs ahead of most of its @@ -1753,14 +1771,21 @@ static int gp_active(void) { if (g_gp_every > 0) return f > 0 && f % g_gp_every == 0; return f >= g_gp_f0 && f <= g_gp_f1; } -static void gp_report(int n, gp_lab *lab, uint64_t frame); +typedef struct { int n, ngt; gp_lab *lab; gt_ev *gt; uint64_t frame; } gp_snap; +static void gp_report(gp_snap s); +// The profiled frame is over: its labels and CPU timeline are copied for the report, which runs once its last command buffer completes, and the heap is busy until then +static gp_snap gp_close(void) { + atomic_store(&g_gt_on, 0); int ngt = atomic_load(&g_gt_n); if (ngt > GT_MAX) ngt = GT_MAX; + gp_snap r = {g_gp_n, ngt, malloc(sizeof(gp_lab) * (g_gp_n ? g_gp_n : 1)), malloc(sizeof(gt_ev) * (ngt ? ngt : 1)), g_gp_frame}; + memcpy(r.lab, g_gp_lab, sizeof(gp_lab) * r.n); memcpy(r.gt, g_gt, sizeof(gt_ev) * ngt); + g_gp_state = 2; atomic_store(&g_gp_busy, 1); return r; +} static void gp_periodic_finish(void) { // on the submit thread, before the next frame's first encoder can touch the heap if (!g_gp_every || g_gp_state != 1 || g_gp_frame == atomic_load(&g_cur_frame)) return; - int n = g_gp_n; gp_lab *lab = malloc(sizeof *lab * (n ? n : 1)); memcpy(lab, g_gp_lab, sizeof *lab * n); uint64_t frame = g_gp_frame; - g_gp_state = 2; atomic_store(&g_gp_busy, 1); + gp_snap snap = gp_close(); static MTLSharedEventListener *listener; if (!listener) listener = [MTLSharedEventListener new]; // g_fence covers all submissions of the sampled frame. No GPU wait or extra command buffer is needed, including for every:1. - [g_ev notifyListener:listener atValue:g_fence block:^(id<MTLSharedEvent> event, uint64_t value) { (void)event; (void)value; gp_report(n, lab, frame); }]; + [g_ev notifyListener:listener atValue:g_fence block:^(id<MTLSharedEvent> event, uint64_t value) { (void)event; (void)value; gp_report(snap); }]; } static id<MTL4CounterHeap> g_ts_heap; static int g_ts_pend = -1; // ts_on; pend: the boundary the next encoder writes (ts_stamp) static void gp_begin(id e, char kind, MTL4RenderPassDescriptor *d) { // right after an encoder of the main command buffer is made @@ -1775,9 +1800,10 @@ static void gp_begin(id e, char kind, MTL4RenderPassDescriptor *d) { // right if (g_gp_geom < 0) { const char *ge = getenv("DM_NVN_GPUPROF_GEOM"); g_gp_geom = ge && atoi(ge) > 0; if (g_gp_geom) MLOG("DM_NVN_GPUPROF_GEOM: render passes also stamped where their vertex and mesh stages finish"); } if (g_gp_geom && !g_gp_gheap) { MTL4CounterHeapDescriptor *hd = [MTL4CounterHeapDescriptor new]; hd.type = MTL4CounterHeapTypeTimestamp; hd.count = GP_MAX; NSError *er = nil; g_gp_gheap = [g_dev newCounterHeapWithDescriptor:hd error:&er]; if (!g_gp_gheap) { MLOG("[RED] GPUPROF_GEOM: no counter heap: %s", er.description.UTF8String); g_gp_geom = 0; } } - if (!g_gp_state) { g_gp_state = 1; g_gp_n = 0; g_gp_frame = atomic_load(&g_cur_frame); [g_gp_heap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; if (g_gp_geom) [g_gp_gheap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; } + if (!g_gp_state) { g_gp_state = 1; g_gp_n = 0; g_gp_frame = atomic_load(&g_cur_frame); [g_gp_heap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; if (g_gp_geom) [g_gp_gheap invalidateCounterRange:NSMakeRange(0, GP_MAX)]; + atomic_store(&g_gt_n, 0); atomic_store(&g_gt_on, 1); if (g_gt_sub0) gt_put(GT_SUB, g_gt_sub0, g_sub_serial, g_gt_queue | 1ull << 63); } // the timeline opens with the submission being encoded (its end not reached: bit 63) if (2 * g_gp_n + 2 > GP_MAX) return; - gp_lab l = {kind, 0, 0, 0, 0, 0, kind == 'R' ? (uint32_t)g_stat_draw : g_gp_copies, g_seg, g_stat_pass, 0, 0, 0, 0}; if (kind != 'R') { l.w = (uint16_t)g_gp_disp; l.h = (uint16_t)g_gp_ubd; } + gp_lab l = {kind, 0, 0, 0, 0, 0, kind == 'R' ? (uint32_t)g_stat_draw : g_gp_copies, g_seg, g_stat_pass, 0, 0, 0, 0, (uint32_t)g_sub_serial}; if (kind != 'R') { l.w = (uint16_t)g_gp_disp; l.h = (uint16_t)g_gp_ubd; } if (d) { for (int i = 0; i < 8; i++) { MTLRenderPassColorAttachmentDescriptor *c = d.colorAttachments[i]; if (!c.texture) continue; if (!l.nc) l.cfmt = (uint16_t)c.texture.pixelFormat; l.nc++; l.w = (uint16_t)MAX(c.texture.width >> c.level, 1); l.h = (uint16_t)MAX(c.texture.height >> c.level, 1); l.loads += c.loadAction == MTLLoadActionLoad; } MTLRenderPassDepthAttachmentDescriptor *z = d.depthAttachment; @@ -1799,8 +1825,10 @@ static void gp_end(id e) { // right before that encoder ends if (g_gp_lab[k].kind == 'R') [(id<MTL4RenderCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed afterStage:MTLRenderStageFragment intoHeap:g_gp_heap atIndex:2 * k + 1]; else [(id<MTL4ComputeCommandEncoder>)e writeTimestampWithGranularity:MTL4TimestampGranularityRelaxed intoHeap:g_gp_heap atIndex:2 * k + 1]; } -static void gp_report(int n, gp_lab *lab, uint64_t frame) { // the frame's last command buffer has completed - NSData *r = [g_gp_heap resolveCounterRange:NSMakeRange(0, 2 * n)]; const uint64_t *t = r.bytes; if (!t) { free(lab); atomic_store(&g_gp_busy, 0); return; } +static int gt_cmp(const void *x, const void *y) { const gt_ev *a = x, *b = y; return a->t0 < b->t0 ? -1 : a->t0 > b->t0; } +static void gp_report(gp_snap snap) { // the frame's last command buffer has completed + int n = snap.n; const gp_lab *lab = snap.lab; uint64_t frame = snap.frame; + NSData *r = [g_gp_heap resolveCounterRange:NSMakeRange(0, 2 * n)]; const uint64_t *t = r.bytes; if (!t) { free(snap.lab); free(snap.gt); atomic_store(&g_gp_busy, 0); return; } NSData *gr = g_gp_geom > 0 ? [g_gp_gheap resolveCounterRange:NSMakeRange(0, 2 * n)] : nil; const uint64_t *gt = gr.bytes; // DM_NVN_GPUPROF_GEOM's stamps, two per encoder double tick_ns = 1e9 / (double)[g_dev queryTimestampFrequency], tot[2] = {0}, span = 0; uint64_t t0 = 0, t1 = 0, ld = 0, st = 0; int np = 0; for (int i = 0; i < n; i++) { uint64_t a = t[2 * i], b = t[2 * i + 1]; if (!a || !b || b < a) continue; @@ -1810,11 +1838,28 @@ static void gp_report(int n, gp_lab *lab, uint64_t frame) { // the frame's las char geom[32] = ""; // the later of the vertex and mesh stamps that falls inside the pass; none when neither does if (gt) { uint64_t g = 0; for (int j = 0; j < 2; j++) { uint64_t v = gt[2 * i + j]; if (v >= a && v <= b && v > g) g = v; } if (g) snprintf(geom, sizeof geom, " geom %.1f", (g - a) * tick_ns / 1e3); } - MLOG("[gpuprof] F%llu %4d %8.1f us render pass %llu %ux%u colors %u depth %u loads %u draws %u start %+.1f seg %u load %u KB store %u KB fmt %u/%u%s", (unsigned long long)frame, i, us, (unsigned long long)l->pass, l->w, l->h, l->nc, l->depth, l->loads, l->draws, (a - t[0]) * tick_ns / 1e3, l->seg, l->ld_kb, l->st_kb, l->cfmt, l->dfmt, geom); } - else MLOG("[gpuprof] F%llu %4d %8.1f us compute / copy after pass %llu start %+.1f seg %u copies %u (uniform updates %u) dispatches %u", (unsigned long long)frame, i, us, (unsigned long long)l->pass, (a - t[0]) * tick_ns / 1e3, l->seg, l->draws, l->h, l->w); } + MLOG("[gpuprof] F%llu %4d %8.1f us render pass %llu %ux%u colors %u depth %u loads %u draws %u start %+.1f seg %u load %u KB store %u KB fmt %u/%u%s sub %u", (unsigned long long)frame, i, us, (unsigned long long)l->pass, l->w, l->h, l->nc, l->depth, l->loads, l->draws, (a - t[0]) * tick_ns / 1e3, l->seg, l->ld_kb, l->st_kb, l->cfmt, l->dfmt, geom, l->sub); } + else MLOG("[gpuprof] F%llu %4d %8.1f us compute / copy after pass %llu start %+.1f seg %u copies %u (uniform updates %u) dispatches %u sub %u", (unsigned long long)frame, i, us, (unsigned long long)l->pass, (a - t[0]) * tick_ns / 1e3, l->seg, l->draws, l->h, l->w, l->sub); } span = (t1 - t0) * tick_ns / 1e3; MLOG("[gpuprof] frame %llu: render %.2f ms, compute / copy %.2f ms over %d encoders (%d render passes, attachments load %.1f MB store %.1f MB), first start to last end %.2f ms", (unsigned long long)frame, tot[0] / 1e3, tot[1] / 1e3, n, np, ld / 1024.0, st / 1024.0, span / 1e3); - free(lab); atomic_store(&g_gp_busy, 0); + // The CPU timeline (gt_put), on the encoders' clock: microseconds from the frame's first stamp + if (snap.ngt && n && t[0]) { + mach_timebase_info_data_t tb; mach_timebase_info(&tb); double cpu_ns = (double)tb.numer / tb.denom; + #define GT_US(v) (((double)(int64_t)((v) - t[0])) * cpu_ns / 1e3) + qsort(snap.gt, (size_t)snap.ngt, sizeof *snap.gt, gt_cmp); + static const char *const park[3] = {"no work queued", "an event wait holds queue", "a sync wait holds queue"}; + for (int i = 0; i < snap.ngt; i++) { const gt_ev *e = &snap.gt[i]; char what[160]; + switch (e->kind) { + case GT_SUB: snprintf(what, sizeof what, "submission %llu (NVN queue %llu) encoded%s", (unsigned long long)e->a, (unsigned long long)(e->b & ~(1ull << 63)), e->b >> 63 ? " (still encoding when the profile began)" : ""); break; + case GT_COMMIT: if (e->b == UINT32_MAX) snprintf(what, sizeof what, "commit of an upload batch of submission %llu", (unsigned long long)e->a); + else snprintf(what, sizeof what, "commit of submission %llu segment %llu (from before its uniform-copy wait)", (unsigned long long)e->a, (unsigned long long)e->b); break; + case GT_WAIT: snprintf(what, sizeof what, "CPU wait for fence %llu: %s", (unsigned long long)e->b, e->a < WS_N ? ws_name[e->a] : "?"); break; + default: snprintf(what, sizeof what, "NVN queue worker parked: %s", park[e->a < 3 ? e->a : 0]); if (e->a) snprintf(what + strlen(what), sizeof what - strlen(what), " %llu", (unsigned long long)e->b); break; } + MLOG("[gpuprof] F%llu cpu %+.1f..%+.1f us %s", (unsigned long long)frame, GT_US(e->t0), GT_US(e->t1), what); } + MLOG("[gpuprof] F%llu cpu timeline: %d events%s; this report at %+.1f us (after the last end %+.1f: the CPU and GPU clocks agree when it is later)", (unsigned long long)frame, snap.ngt, snap.ngt == GT_MAX ? " (the limit: later ones lost)" : "", GT_US(mach_absolute_time()), (t1 - t[0]) * tick_ns / 1e3); + #undef GT_US + } + free(snap.lab); free(snap.gt); atomic_store(&g_gp_busy, 0); } static inline id<MTL4RenderCommandEncoder> enc_r_begin_pass(MTL4RenderPassDescriptor *d) { g_cm_enc++; cm_enc_mark(2); // the draw passes (rp_begin_ex); see barrier_game id<MTL4RenderCommandEncoder> e = [g_cb renderCommandEncoderWithDescriptor:d]; cm_enc_mark(0); @@ -2680,8 +2725,7 @@ uint64_t mtl_sub_commit(void) { }]; void arena_commit(uint64_t fence); arena_commit(fence); res_commit(); - if (g_gp_state == 1 && !gp_active()) { int gn = g_gp_n; gp_lab *gl = malloc(sizeof(gp_lab) * (gn ? gn : 1)); memcpy(gl, g_gp_lab, sizeof(gp_lab) * gn); uint64_t gf = g_gp_frame; g_gp_state = 2; atomic_store(&g_gp_busy, 1); - [o addFeedbackHandler:^(id<MTL4CommitFeedback> f) { (void)f; gp_report(gn, gl, gf); }]; } + if (g_gp_state == 1 && !gp_active()) { gp_snap snap = gp_close(); [o addFeedbackHandler:^(id<MTL4CommitFeedback> f) { (void)f; gp_report(snap); }]; } [g_cb endCommandBuffer]; rec_commit(g_cb, o); g_cbgen++; // the stores it holds come before any later upload batch (gsb_held) if (!lost_test) rec_signal(g_ev, fence); @@ -2694,6 +2738,7 @@ uint64_t mtl_sub_commit(void) { if (rec_on() && g_rb_started) { rec_t *r = rec_put(RO_NOTE, 0); r->a[0] = (uint64_t)(uintptr_t)&g_sub_cmt; r->a[1] = fence; if (drain && sw_on(drain)) rec_drain(); else rec_publish(); } else atomic_store(&g_sub_cmt, fence); g_alloc_fence[g_alloc_cur] = fence; + gt_put(GT_SUB, g_gt_sub0, g_sub_serial, g_gt_queue); g_cb = nil; atomic_store(&g_sub_on, 0); // after ++g_fence and the last encoding: a buffer retired from now on waits for this fence at most defer_idle_commit(); // a feedback handler that drained since this submission's own commit saw it open @@ -2716,11 +2761,12 @@ static void post_early(uint64_t fence) { } static int mtl_wait_at(uint64_t fence, uint64_t timeout_ns, int site) { if (g_done >= fence) { if (ws_on()) ws_put(site, 0, 0); return 1; } - uint64_t w0 = ws_t0(); + uint64_t w0 = ws_t0(), gt0 = atomic_load_explicit(&g_gt_on, memory_order_relaxed) ? mach_absolute_time() : 0; // gt0: DM_NVN_GPUPROF's timeline if ((site == WS_SYNC || site == WS_SYNC_NEWEST || site == WS_ACQ) && fast_wake() && g_ev && fence <= g_fence) { // the game's own wait: wake on the GPU's signal (rec_signal(g_ev, fence) after each submission) uint64_t ms = timeout_ns / 1000000ull; if (ms > 3600000ull) ms = 3600000ull; if ([g_ev waitUntilSignaledValue:fence timeoutMS:ms]) post_early(fence); int ok = g_done >= fence; + if (gt0) gt_put(GT_WAIT, gt0, (uint64_t)site, fence); if (ws_on()) { ws_put(site, w0, 1); ws_wake(fence); } if (ok || timeout_ns < 1000000ull) return ok; // The event and feedback waits share a finite budget; a timeout must not buy another full wait. @@ -2751,6 +2797,7 @@ static int mtl_wait_at(uint64_t fence, uint64_t timeout_ns, int site) { } int ok = g_done >= fence; pthread_mutex_unlock(&g_mu); + if (gt0) gt_put(GT_WAIT, gt0, (uint64_t)site, fence); if (ws_on()) { ws_put(site, w0, 1); if (site == WS_SYNC_NEWEST || site == WS_SYNC || site == WS_ACQ) ws_wake(fence); } return ok; } @@ -7700,6 +7747,7 @@ static void queue_program_preserve(void) { S.cprog=index[5]<0?NULL:&q->programs[index[5]]; S.prog=bind_stages(NULL,0); args_dirty_all(); } void mtl_queue_activate(uint64_t serial) { // caller holds the submission lock + g_gt_queue = serial; if (g_active_queue && g_active_queue->serial == serial) return; if (g_active_queue) { g_active_queue->s = S; g_active_queue->r = R; g_active_queue->rc_skip=g_rc_skip; } queue_state *q = g_queue_states; diff --git a/dm/gfx/nvn/nvn_queue.inc b/dm/gfx/nvn/nvn_queue.inc index bdb2beb5f0..755b50223b 100644 --- a/dm/gfx/nvn/nvn_queue.inc +++ b/dm/gfx/nvn/nvn_queue.inc @@ -109,16 +109,18 @@ static void nq_cost_end(nq_cost c) { atomic_fetch_add(&g_sub_instr,ins-c.instr); atomic_fetch_add(&g_cum_sub_instr,ins-c.instr); atomic_fetch_add(&g_cum_sub_cyc,cyc-c.cycles); atomic_fetch_add(&g_cum_sub_sys_ns,thr_sys_ns()-c.sys); } +void mtl_gp_note_park(uint64_t t0,unsigned why,uint64_t queue); // nvn_mtl.m: DM_NVN_GPUPROF's CPU timeline static void *nq_worker(void *unused) { (void)unused; pthread_setname_np("NVN queue replay"); pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE,0); uint64_t frame=UINT64_MAX; + uint64_t park0=0,park_q=0; unsigned park_why=0; // parked since park0 (mach ticks) and why: 0 no work queued, 1 an event wait, 2 a sync wait holds queue park_q (the first held) for(;;) { pthread_mutex_lock(&nq_mu); nq_queue *q=NULL; nq_node *n=NULL; nq_job *j=NULL; for(;;) { scan_again:; if(nq_test_held) { pthread_cond_wait(&nq_cv,&nq_mu); continue; } - int mapped_event_wait=0; + int mapped_event_wait=0; unsigned held_why=0; uint64_t held_q=0; for(q=nq_queues;q;q=q->next) { j=q->first; if(!j) continue; n=j->first; if(!n) { nq_publish(j->receipt,q->physical); q->first=j->next; if(!q->first) q->last=NULL; nq_put(j->receipt); @@ -136,14 +138,14 @@ static void *nq_worker(void *unused) { else if(t-n->wait_since>5000000000ull && !n->wait_said) { static _Atomic int said; n->wait_said=1; if(atomic_fetch_add(&said,1)<8) LOG("[RED] queue %llu: an event wait has held it %.1f s: word %p is %#x, waiting for %s %#x; a GPU write of it is %s",(unsigned long long)q->serial,(t-n->wait_since)/1e9,(void *)n->event, atomic_load_explicit((_Atomic uint32_t *)n->event,memory_order_acquire),n->mode==0?"equal to":"at least",n->value,predecessor>mtl_completed_fence()?"queued, not completed":"not queued"); } - n->waiting=1; mapped_event_wait=1; continue; } + n->waiting=1; mapped_event_wait=1; if(!held_why) { held_why=1; held_q=q->serial; } continue; } } if(n->kind==NQ_SYNC) { if(n->receipt && !n->receipt->published) { uint64_t t=now_ns(); if(!n->wait_since) n->wait_since=t; else if(t-n->wait_since>5000000000ull && !n->wait_said) { static _Atomic int said; n->wait_said=1; if(atomic_fetch_add(&said,1)<8) LOG("[RED] queue %llu: a sync wait has held it %.1f s: the fence it waits for (receipt %p) is not published yet",(unsigned long long)q->serial,(t-n->wait_since)/1e9,(void *)n->receipt); } - continue; } + if(!held_why) { held_why=2; held_q=q->serial; } continue; } // Targets are immutable snapshots. One Metal queue orders an encoded producer before this continuation. A sync not yet fenced when the wait was submitted // (wait_kind 0) is a snapshot of nothing to wait for, and the wait passes. Sparking! ZERO, on one NVN queue, waits on a sync before that queue's own first // fence of it at its first load, which runs on the console; parked here, the fence behind the wait never published and its SyncWait stalled the game (#1808) @@ -151,11 +153,14 @@ static void *nq_worker(void *unused) { break; } if(q) break; + if(park0 && (held_why!=park_why || held_q!=park_q)) { mtl_gp_note_park(park0,park_why,park_q); park0=0; } + if(!park0) { park0=mach_absolute_time(); park_why=held_why; park_q=held_q; } // NVN events can also be changed by plain stores through mapped pool memory. // A bounded notification poll only while such a word is awaited complements API/callback wakes. if(mapped_event_wait) { struct timespec tick={0,1000000}; pthread_cond_timedwait_relative_np(&nq_cv,&nq_mu,&tick); } else pthread_cond_wait(&nq_cv,&nq_mu); } + if(park0) { mtl_gp_note_park(park0,park_why,park_q); park0=0; } j->first=n->next; if(!j->first) j->last=NULL; // Round-robin ready queues: a continually fed independent producer cannot starve another queue. if(q->next) { nq_queue **link=&nq_queues; while(*link!=q) link=&(*link)->next; *link=q->next; nq_queue *tail=nq_queues; while(tail->next) tail=tail->next; tail->next=q; q->next=NULL; } diff --git a/dm/gfx/nvn/nvn_rec.h b/dm/gfx/nvn/nvn_rec.h index b008d3da47..bec6cc26ee 100644 --- a/dm/gfx/nvn/nvn_rec.h +++ b/dm/gfx/nvn/nvn_rec.h @@ -138,12 +138,14 @@ static void rec_exec(const rec_t *r) { case RO_CB_END: [(id<MTL4CommandBuffer>)g_rx_cb endCommandBuffer]; break; case RO_CB_BLOCK: ((void (^)(id<MTL4CommandBuffer>))rec_id(a[0]))(g_rx_cb); break; case RO_COMMIT: case RO_COMMIT_REAL: { + uint64_t gt0 = atomic_load_explicit(&g_gt_on, memory_order_relaxed) ? mach_absolute_time() : 0; if (r->op == RO_COMMIT) ubc_wait(a[2]); // the uniform shadows this command buffer reads id<MTL4CommandBuffer> buffers[] = {r->op == RO_COMMIT ? g_rx_cb : rec_id(a[1])}; MTL4CommitOptions *o = rec_id(a[0]); pthread_mutex_lock(&g_res_commit_mu); // apart from the residency set's commit, as the present snapshot's queue commit (nvn_mtl.m res_commit) if (o) [g_q4 commit:buffers count:1 options:o]; else [g_q4 commit:buffers count:1]; pthread_mutex_unlock(&g_res_commit_mu); if (r->op == RO_COMMIT) g_rx_cb = nil; + if (gt0) gt_put(GT_COMMIT, gt0, a[3], r->op == RO_COMMIT ? a[4] : UINT32_MAX); // DM_NVN_GPUPROF's timeline: submission, segment (an upload batch: none) break; } case RO_SIGNAL: [g_q4 signalEvent:rec_id(a[0]) value:a[1]]; break; case RO_Q_BLOCK: ((void (^)(void))rec_id(a[0]))(); break; @@ -367,14 +369,16 @@ static void rec_cb_block(id cb, void (^block)(id<MTL4CommandBuffer>)) { static id rec_table(id t) { if (!rec_on() || !t) return t; DmRecTable *s = [DmRecTable new]; s->real = t; return s; } // The replay's main command buffer: committed once the uniform shadows queued so far are made (ubc_wait) static void rec_commit(id cb, MTL4CommitOptions *o) { - if ([cb isKindOfClass:[DmRecCB class]]) { rec_t *r = rec_put(RO_COMMIT, 1); r->a[0] = rec_obj(o); r->a[2] = ubc_head(); return; } + if ([cb isKindOfClass:[DmRecCB class]]) { rec_t *r = rec_put(RO_COMMIT, 1); r->a[0] = rec_obj(o); r->a[2] = ubc_head(); r->a[3] = g_sub_serial; r->a[4] = g_seg; return; } + uint64_t gt0 = atomic_load_explicit(&g_gt_on, memory_order_relaxed) ? mach_absolute_time() : 0; ubc_wait(ubc_head()); id<MTL4CommandBuffer> buffers[] = {cb}; if (o) [g_q4 commit:buffers count:1 options:o]; else [g_q4 commit:buffers count:1]; + if (gt0) gt_put(GT_COMMIT, gt0, g_sub_serial, g_seg); } // An upload batch: its place in the queue is after what is recorded so far (it reads no uniform shadow) static void rec_commit_upload(id cb, MTL4CommitOptions *o) { - if (rec_on() && g_rb_started) { rec_t *r = rec_put(RO_COMMIT_REAL, 2); r->a[0] = rec_obj(o); r->a[1] = rec_obj(cb); return; } + if (rec_on() && g_rb_started) { rec_t *r = rec_put(RO_COMMIT_REAL, 2); r->a[0] = rec_obj(o); r->a[1] = rec_obj(cb); r->a[3] = g_sub_serial; return; } id<MTL4CommandBuffer> buffers[] = {cb}; if (o) [g_q4 commit:buffers count:1 options:o]; else [g_q4 commit:buffers count:1]; } diff --git a/dm/gfx/tests/t_gpuprof.m b/dm/gfx/tests/t_gpuprof.m index 1929aee847..1bcf221782 100644 --- a/dm/gfx/tests/t_gpuprof.m +++ b/dm/gfx/tests/t_gpuprof.m @@ -48,6 +48,13 @@ @interface ProfileRender : NSObject @implementation ProfileRender - (void)writeTimestampWithGranularity:(MTL4TimestampGranularity)g afterStage:(MTLRenderStages)s intoHeap:(id)h atIndex:(NSUInteger)i { (void)g; _stamps++; _last = i; if (h && h == (id)g_gp_gheap) { _geom++; _geom_stages |= s; } } @end +@interface ProfileQueue : NSObject +@property int commits; +@end +@implementation ProfileQueue +- (void)commit:(id const *)b count:(NSUInteger)n { (void)b; (void)n; _commits++; } +- (void)commit:(id const *)b count:(NSUInteger)n options:(id)o { (void)b; (void)n; (void)o; _commits++; } +@end @interface ProfileCompute : NSObject @property int stamps; @end @@ -91,5 +98,25 @@ int main(void) { @autoreleasepool { gp_begin(gpass, 'R', nil); int k2 = g_gp_cur; gp_end(gpass); CHECK(gpass.stamps == 4 && gpass.geom == 2 && gpass.geom_stages == (MTLRenderStageVertex | MTLRenderStageMesh), "a pass's end stamps its vertex and mesh stages"); CHECK(gpass.last == 2 * (NSUInteger)k2 + 1, "the fragment end stamp still comes last, at the pass's own index"); + // The CPU timeline: recorded only while a frame is profiled, through the real wait and commit paths, and handed to the report with the labels + if (g_gp_state == 1) { gp_snap open = gp_close(); free(open.lab); free(open.gt); atomic_store(&g_gp_busy, 0); } // the pass frame above, left open + g_gp_every = 1; g_gp_state = 0; g_gp_geom = 0; + atomic_store(&g_gt_n, 0); + g_done = 0; mtl_wait_at(5, 1000000ull, WS_ALLOC); CHECK(atomic_load(&g_gt_n) == 0, "no timeline is kept outside a profiled frame"); + g_sub_serial = 41; g_gt_queue = 7; g_gt_sub0 = mach_absolute_time(); + atomic_store(&g_cur_frame, 20); gp_begin(nil, 'C', nil); gp_end(nil); + CHECK(atomic_load(&g_gt_on) && atomic_load(&g_gt_n) == 1 && g_gt[0].kind == GT_SUB && g_gt[0].a == 41 && g_gt[0].b == (7 | 1ull << 63), "the timeline opens with the submission being encoded"); + CHECK(g_gp_lab[g_gp_n - 1].sub == 41, "an encoder's label names its submission"); + uint64_t w0 = mach_absolute_time(); mtl_wait_at(5, 2000000ull, WS_ALLOC); // a fence the GPU never reaches: the wait times out after 2 ms + CHECK(atomic_load(&g_gt_n) == 2 && g_gt[1].kind == GT_WAIT && g_gt[1].a == WS_ALLOC && g_gt[1].b == 5 && g_gt[1].t0 >= w0 && g_gt[1].t1 > g_gt[1].t0, "a blocking wait is placed on the timeline with its site and fence"); + ProfileQueue *queue = [ProfileQueue new]; g_q4 = (id)queue; g_seg = 3; g_rec = 0; + rec_commit((id)[NSObject new], nil); + CHECK(queue.commits == 1 && atomic_load(&g_gt_n) == 3 && g_gt[2].kind == GT_COMMIT && g_gt[2].a == 41 && g_gt[2].b == 3, "a commit is placed on the timeline with its submission and segment"); + mtl_gp_note_park(mach_absolute_time(), 1, 9); + CHECK(atomic_load(&g_gt_n) == 4 && g_gt[3].kind == GT_QPARK && g_gt[3].a == 1 && g_gt[3].b == 9, "the queue worker's park is placed on the timeline"); + gp_snap snap = gp_close(); + CHECK(snap.ngt == 4 && snap.n == 1 && snap.frame == 20 && !atomic_load(&g_gt_on), "closing the frame hands its timeline to the report and stops recording"); + mtl_wait_at(5, 1000000ull, WS_ALLOC); CHECK(atomic_load(&g_gt_n) == 4, "nothing is recorded after the frame closed"); + gp_report(snap); CHECK(!atomic_load(&g_gp_busy), "the report frees the heap"); printf("t_gpuprof: %d checks, %d failed\n", checks, failed); return failed != 0; } } From e66db18bcf3178cf2cf1b75af0338329d64cae5a Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 08:31:41 +0100 Subject: [PATCH 35/40] OoT-iPhone A-1 docs: the game note's iPhone 17 table names DM_NVN_GPUPROF's CPU timeline, now in this pull request --- dm/games/0400486024DEC000/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 66ff8a7d00..a78709f3f3 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -88,7 +88,7 @@ Current (2026-11-09): plays on an M1 iPad Pro (11-inch, 3rd generation, `iPad13, - **Frame rate and GPU time.** At the Deku Tree's entrance: 20.0 fps in every 300-frame window of both runs (frame p50 50.0 ms, the frame step's 3-vsync hold; worst p95 51.0 and 66.6 ms), GPU 34-35 ms a frame (29 ms of work over 127 encoders, 205 MB of attachments loaded and 215 MB stored), game speed 0.97-1.00 after the load. Views deeper in that take 54-67 ms of GPU fall to 11-12 fps, and the game runs slow (game speed 0.53-0.61), as no hold is longer than 3 vsyncs. The title scene takes 61 ms of GPU, the game over screen 66-74 ms. Kokiri Forest by the owner's eye: 20 fps in handheld mode, 15 docked. - **iPhone** (current, 2026-11-09): plays on an iPhone 17 (`iPhone18,3`, 8 GB, A19, iOS 26.6.1) from the owner's save in Kokiri Forest, with the launcher's 30 fps cap and Dynamic resolution on. Built from upstream `main` 04811a63 with this branch and, for testing, PRs #1842, #1901, #1942 and #1751, chain `39eb2f781d07`. iOS gives the app 5.96 GB of room (the iPad 7.97), Metal's working set is 5,461 MB. Before [the pool table left shader-code pools out](../../../docs/reference/memory.md#shader-code-pools-wired-by-the-raw-pool-table) it stuttered too much to play: the guest's file pages were read back all the time, 14-17 fps. Now 27 fps standing, 27-30 walking, frame p50 36.5 / p95 40 ms, GPU p50 32 ms, the guest file's page-ins at 100-400 every 10 s. Frames just over the cap's 33.3 ms are late: about 30 of 300 are dropped, a light stutter walking. The title, with its movie on its own clock, holds 30 fps. An earlier build without these fixes ended in a kernel panic after the save loaded (`wait for termination failed … AppleCentauriManager`). - **Pacing at the launcher's 30 fps cap** (2026-11-10). The hold cannot rise above 2 vsyncs there, and frames of 36 ms took one 66.7 ms gap about every 11 frames on the 30 Hz link: iPhone walk in Kokiri Forest 30-33 per 300; M1 iPad at the Deku Tree save, dynamic resolution on, 22.8 per 300 at 27.9 fps. With late frames shown at the next vsync of a 60 Hz link at frame latency 2, the M1 iPad showed 27.7 fps with gaps of 33 / 50 ms (81 / 17%) and 0.2 of 66.7 ms per 300 ([why and the three runs](../../../docs/reference/frame-rate.md#late-frames-at-a-30-fps-cap-on-the-30-hz-display-link)). The owner found the camera turning smooth on the 30 Hz link and did not judge the new build by eye. Open: the iPad's title (40 ms frames) still shows uneven 29-50 ms gaps, and its movie gets a new picture on only about 210 of 300 game frames (`[codec] AV1 decoder` lines: about 600 dropped to the movie's clock per 300 frames), where the iPhone's 33 ms title gets one every frame. On the iPhone 17 with the change (Kokiri Forest, camera turns), the first gameplay at thermal 1 showed gaps of 33 / 50 ms (52 / 48%) and 1 longer one in 2,096, p99 50 ms, game time 0.999; at thermal 2 it ran CPU-bound at 13-22 fps (frames over 50 ms, footprint up to 3.2 GiB) and game time fell to 0.731, more than the 3 vsyncs a step can carry, which a hold of 3 would not change. -- **iPhone 17: GPU time and what limits the frame** (current, 2026-11-10). The owner's launcher settings (30 fps cap, dynamic resolution on, encoding on its own thread), Kokiri Forest from the owner's save, camera turns by `iphone_cam.sh` in the work folder. GPU frames from `DM_NVN_GPUPROF` with a CPU timeline of the host (a local diagnostic build): +- **iPhone 17: GPU time and what limits the frame** (current, 2026-11-10). The owner's launcher settings (30 fps cap, dynamic resolution on, encoding on its own thread), Kokiri Forest from the owner's save, camera turns by `iphone_cam.sh` in the work folder. GPU frames from `DM_NVN_GPUPROF` with its CPU timeline: | Frame | Build | GPU span | GPU work | Gaps before a submission / inside one | Submissions, encoders | |---|---|---|---|---|---| From 3a181b61662e88a1f61658aeda32def24ca68a50 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 10:12:15 +0100 Subject: [PATCH 36/40] test_env_switches passes again: DM_NVN_SUB_DRAIN has its row, and three switches that are not on main are no longer quoted as switches docs/env_switches.tsv: DM_NVN_SUB_DRAIN (A-2's switch to restore the old wait at each submission's end) gets its class and meaning. The docs quoted three names that no code reads. Each mention now drops its backticks and says the switch is not on main: - DM_NVN_VC_WAYS: another fork's switch, in memory.md. - DM_NVN_CUT_FIRST: a probe switch since removed, in Breath of the Wild's note. - DM_NVN_RT_RECLAIM: PR #1588's switch, in rendering.md, and so in the symptom index too (docs_symptoms.py). --- docs/env_switches.tsv | 1 + docs/reference/memory.md | 2 +- docs/reference/rendering.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index 2eb213ed50..7bfed21340 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -217,6 +217,7 @@ DM_NVN_SLICE_MERGE switch `0` keeps passes into other layers of one array or cub DM_NVN_SNAPRAW diagnostic `=1` makes `DM_NVN_TEXDUMP` also save each target's raw bytes, with a text header line, since converting an HDR target to an 8-bit image truncates DM_NVN_SPARSE_TILES switch `0` makes a sparse texture of 64 MB or more whole, as before; by default it is a Metal placement sparse texture with only the mapped tiles backed DM_NVN_STORAGE_FIRST_USE switch `0` keeps a sampled block-compressed texture in a CPU-writable pool to the one-time fill instead of making it CPU-written when its storage holds data at first use (on by default; `tex_cpu_content`), to measure it +DM_NVN_SUB_DRAIN switch `1` makes the replay wait for the encode thread at the end of every submission, as before (with `DM_NVN_ENC_THREAD`; by default a submission's end hands its records over and the replay goes on, and a present of the game's own texture waits for its commit); `ab` alternates with the `DM_AB` phases, to measure it DM_NVN_T2T_FROM diagnostic `=<frame>`: `DM_NVN_T2T_TRACE` records copies only from that frame on (default 0) DM_NVN_T2T_TRACE diagnostic `=<W>x<H>:<file>`: after each copy into an RGBA16F texture of that size, copies the result into a trace buffer on the GPU timeline; 240 entries are written to the file DM_NVN_TBDR switch `=1` turns on the replay's TBDR pass optimizations (store / load elision, pass merging, same-target continuation); off by default since bindless textures defeat their read inference diff --git a/docs/reference/memory.md b/docs/reference/memory.md index 6ca466c8a6..0698552b91 100644 --- a/docs/reference/memory.md +++ b/docs/reference/memory.md @@ -541,7 +541,7 @@ The guest address space, heaps, pools, virtual address memory and the allocators - **Symptom:** on an iPhone the app is ended by the system (a jetsam event: no crash report, no `[dm] CRASH` line) once its footprint reaches the limit: about 6 GB on a 12 GB iPhone with the increased memory limit (6,143 MiB measured in LiveContainer), 3.3 GB without it. The log stops mid-frame. - **Cause:** every guest page the game has touched is anonymous memory: `phys_footprint` charges it, and iOS cannot write it out as a Mac swaps it. What the game gives back is charged again as soon as it maps the range anew. - **Fix:** the guest block in sparse files, unlinked at once, mapped shared over the reservation at the same addresses (`hle_svc.c`; `DM_GUEST_FILE_DIR`, which the iOS app sets to `NSTemporaryDirectory()`). File pages take the same memory while they stay in it, but they are out of the footprint and the kernel writes them out when it needs the room. The cost is disk (Mario Kart World on a 16 GB Mac: 5.6 GB on disk and 8-10 GB of logical writes in 5.5 minutes), so a game that fits stays anonymous: on iOS the files are on when the game's memory pool is larger than `os_proc_available_memory()` at start, and the log's `[mem] guest memory` line gives both; on the Mac only with `DM_GUEST_FILE=1`, and `0` keeps them off anywhere. A game whose memory is mostly outside the block is not helped: Unreal's heap is a host malloc zone ([Thread stacks from the host's Unreal heap](#thread-stacks-from-the-hosts-unreal-heap)). The files are 1 GB each (`DM_GUEST_FILE_CHUNK`), since the stall at the system's periodic flush, about every 30 s, grows with the file (on a 16 GB M5 Mac, with 1 GB touched, up to 148 ms with one 20 GB file, 4 ms with 20 files of 1 GB). `guest_release` makes a range the game gives back `PROT_NONE` there, since a fresh anonymous mapping would bring it back into the footprint, and its disk blocks are punched out ([Guest files keep the blocks of memory the game gave back](#guest-files-keep-the-blocks-of-memory-the-game-gave-back)); a page that keeps its bytes is zeroed by `alias_initialize` and `SetHeapSize` when it is opened again, leaving the 4 KB pages that already read 0. `DM_GUEST_FILE_MIN_FREE` (default 2 GB of room after the dirty pages, the same entry) stops the game (exit code 87) before the disk fills; under it at start the guest stays anonymous, and on the Mac, which can swap, under that plus 8 GB. `make -C dm/host/tests memory-file` runs the memory SVC tests with the guest in files. -- **Found on:** [Mario Kart World](../../dm/games/0400C3F00006E000/README.md) 1.0.0 in handheld mode, on the switch2ios fork (a 12 GB iPhone, iOS 27.3, LiveContainer's 6,143 MiB limit; host from nxnative `8c0b8083` with that fork's changes): ended at 90 s with the guest anonymous; with one file, 7 min 33 s played, peak 5.56 GiB; with 1 GB files (and a larger texture view cache), a 14-minute session held, peak 5,663 MB, no 30-s stall left. On a 16 GB Mac the peak footprint fell from 11.6 GiB to 5.6. On this host (`975bb4c1` with this switch, 1 GB files, a larger texture view cache and the profile's map limit), a 16 GB M5 Mac with the iPhone's settings peaked at 4,365 and 4,792 MiB over an 18- and a 30-minute session, and the same build plays on the 12 GB iPhone. On iPhones with an 8 GiB limit ([#1184](https://github.com/seedds/nxnative/issues/1184)), where the game was ended at course load, or 30 s in at 8.1 GB, the fork's version of this switch merged by hand let it finish races: 4.41 GB at most over 13 minutes and two races on an iPhone 18 Pro with 512 MB files. Those phone runs also had a larger texture view cache (`DM_NVN_VC_WAYS=64`), which is not on main. +- **Found on:** [Mario Kart World](../../dm/games/0400C3F00006E000/README.md) 1.0.0 in handheld mode, on the switch2ios fork (a 12 GB iPhone, iOS 27.3, LiveContainer's 6,143 MiB limit; host from nxnative `8c0b8083` with that fork's changes): ended at 90 s with the guest anonymous; with one file, 7 min 33 s played, peak 5.56 GiB; with 1 GB files (and a larger texture view cache), a 14-minute session held, peak 5,663 MB, no 30-s stall left. On a 16 GB Mac the peak footprint fell from 11.6 GiB to 5.6. On this host (`975bb4c1` with this switch, 1 GB files, a larger texture view cache and the profile's map limit), a 16 GB M5 Mac with the iPhone's settings peaked at 4,365 and 4,792 MiB over an 18- and a 30-minute session, and the same build plays on the 12 GB iPhone. On iPhones with an 8 GiB limit ([#1184](https://github.com/seedds/nxnative/issues/1184)), where the game was ended at course load, or 30 s in at 8.1 GB, the fork's version of this switch merged by hand let it finish races: 4.41 GB at most over 13 minutes and two races on an iPhone 18 Pro with 512 MB files. Those phone runs also had a larger texture view cache (DM_NVN_VC_WAYS=64, that fork's switch: it is not on main). ### Guest files keep the blocks of memory the game gave back diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index c6a4052ca3..95ffd72175 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -397,7 +397,7 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Symptom:** a command buffer fails with a GPU page fault on the same frames in every run, in a compute or draw command buffer whose shaders sample through the texture pool. Ocarina of Time with `DM_NVN_RT_RECLAIM=1` (PR #1588's render-target reclaim; the switch was never on main): f3,677 and f4,851, compute shader 9e6b00b050e30801, each run. - **Cause:** `tex_by_id` (`nvn_mtl.m`) keeps an ID's cached view when the slow path finds the same texture behind it: `cur == v || v.parentTexture == cur`. When the ID has no texture now, `treg_tex` returns nil as `cur`. That happens for a texture that went and was not made again, a texture that could not be made, or a reclaimed render target. A cached view that is the base texture itself has no parent either, so nil == nil kept it and marked it valid again. The pool tables' fast path (`tex_fast_rid`) then wrote its resource ID after the host had released the texture and dropped its residency. With the reclaim it happened about 60 times in a 5,000-frame run. - **Fix:** general, in `tex_by_id`: the cached view is kept only when `cur` is non-nil. An ID with no texture drops its view, and its slot takes the typed zero texture an empty slot of that type gets. -- **Check:** Ocarina of Time on an M2 Pro 16 GB with `DM_NVN_RT_RECLAIM=1`, to f5,241 and f7,750. The f3,677 and f4,851 faults are gone, with no trimmed texture handed back (checked by marking every texture the reclaim's trim released). Pace is the same (7,750 frames in 202.5 s against 7,731 in 202.75 s), and reclaim counts are alike. A fault at f786 has another cause and is there without the reclaim too. The picture there was not compared: a dead ID now reads the typed zero texture instead of the released one. +- **Check:** Ocarina of Time on an M2 Pro 16 GB with DM_NVN_RT_RECLAIM=1, to f5,241 and f7,750. The f3,677 and f4,851 faults are gone, with no trimmed texture handed back (checked by marking every texture the reclaim's trim released). Pace is the same (7,750 frames in 202.5 s against 7,731 in 202.75 s), and reclaim counts are alike. A fault at f786 has another cause and is there without the reclaim too. The picture there was not compared: a dead ID now reads the typed zero texture instead of the released one. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md) (Switch 2, `0400486024DEC000`, v131072). ### Live resolution variants share physical image storage From 1eac389e10543473956dbc440fbfb92a40ad64b0 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 12:24:04 +0100 Subject: [PATCH 37/40] OoT-iPhone: test_oot_aspect's camera and shape helpers take a float cap, as the test passes 0.75 and 16/9 (pyright ratchet in CI) --- dm/tools/tests/test_oot_aspect.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dm/tools/tests/test_oot_aspect.py b/dm/tools/tests/test_oot_aspect.py index a67c5186ca..a4241c4196 100644 --- a/dm/tools/tests/test_oot_aspect.py +++ b/dm/tools/tests/test_oot_aspect.py @@ -24,7 +24,7 @@ patch = importlib.util.module_from_spec(spec); spec.loader.exec_module(patch) -def camera(r, cap=1): +def camera(r, cap: float = 1.0): """The camera's x / y multipliers the loader writes for a shape (loader.c screen_camera), upright held to the vertical angle of a picture of shape cap (screen_upright_cap, DM_SCREEN_UPRIGHT_FOV: 1 by default, 0 off)""" if not r: return (1, 1) @@ -139,7 +139,7 @@ def setUp(self): emit, put = OutputSize.emit, OutputSize.put - def shape(self, ratio, asked=None, cap=1): + def shape(self, ratio, asked=None, cap: float = 1.0): """The factors the output size was made with (state +0xb8 / +0xbc, 0 = 16:9) and, apart, the shape the host asks for now (+0xa0..: the camera ignores it)""" f = lambda r: (r, max(r * 9 / 16, 1), max(16 / (9 * r), 1)) if r else (0, 1, 1) self.u.mem_write(self.state + 0xa0, struct.pack('<3f', *f(ratio if asked is None else asked))) From 8a8fb0bbd7bb5fe055ff851e6874ec5e672ffd11 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 12:26:56 +0100 Subject: [PATCH 38/40] OoT-iPhone: the Python lint baseline drops test_gpuprof.py's unused os import, which this pull request removed --- dm/tools/lint_python_baseline.tsv | 1 - 1 file changed, 1 deletion(-) diff --git a/dm/tools/lint_python_baseline.tsv b/dm/tools/lint_python_baseline.tsv index ad81da4a0d..96a17204f0 100644 --- a/dm/tools/lint_python_baseline.tsv +++ b/dm/tools/lint_python_baseline.tsv @@ -207,7 +207,6 @@ ruff dm/tools/patch_nso.py B023 1 Function definition does not bind loop variabl ruff dm/tools/patch_nso.py B023 2 Function definition does not bind loop variable `n` ruff dm/tools/patch_nso.py PIE808 1 Unnecessary `start` argument in `range` ruff dm/tools/sdis.py F841 1 Local variable `lo` is assigned to but never used -ruff dm/tools/tests/test_gpuprof.py F401 1 `os` imported but unused ruff dm/tools/tests/test_gpuprof.py F401 1 `sys` imported but unused ruff dm/tools/tests/test_image_atomic_bounds.py B023 1 Function definition does not bind loop variable `dependencies` ruff dm/tools/tests/test_nsp_incremental.py B023 1 Function definition does not bind loop variable `build` From ed769e9c7fbd890b3b681158b13fec2622e4ac92 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 13:05:38 +0100 Subject: [PATCH 39/40] OoT-iPhone B-4: Ocarina of Time's smoke route, so regress.py check boots it An empty save to the title: the cold pipeline prepare (265 s for 14,748 pipelines on an M4 in quick mode, with no game frame meanwhile), the logos, then the title with its movie, where the frame step and the movie clock work. No input. Calibrated with one quick baseline run of upstream main c4ffb9df: first frame at 268 s, frame 897 at 300 s. --- dm/games/0400486024DEC000/smoke.tsv | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 dm/games/0400486024DEC000/smoke.tsv diff --git a/dm/games/0400486024DEC000/smoke.tsv b/dm/games/0400486024DEC000/smoke.tsv new file mode 100644 index 0000000000..9126c5d336 --- /dev/null +++ b/dm/games/0400486024DEC000/smoke.tsv @@ -0,0 +1,10 @@ +# Empty save to the title: the pipeline prepare of a cold start (265 s for 14,748 pipelines on an M4 in quick mode, no game frame meanwhile), the logos, then the title +# with its movie (compute-heavy: the frame step and the movie clock at work). No input. quick has no prebuild, so it pays the prepare too; the full mode prebuilds first. +save new +keys +secs 400 +quick 400 +reach 900@360 +window 340 50 +shots 350,385 +hang 30 From 4283410dd32df702166b3700313e2c49b63002c6 Mon Sep 17 00:00:00 2001 From: seedds <seedds@gmail.com> Date: Tue, 10 Nov 2026 14:19:06 +0100 Subject: [PATCH 40/40] Ocarina of Time's output-size stub keeps the size it hands the setter at state +0xe0, a free word, instead of +0xb0 Review on #1971 (lunks): state +0xb0 is the dynamic-resolution flag on their side, so a non-16:9 window turned dynamic resolution on whatever the setting; they moved the scratch to +0xe0. This tree has no reader of +0xb0 (loader.c, hle_nvn.c, the profile's other stubs; the setter main+0x1087f40 reads only the 8 bytes at x1), but +0xe0 is unused here too, so the move costs nothing. test_oot_aspect checks the setter now gets state +0xe0. --- dm/games/0400486024DEC000/patches.py | 4 ++-- dm/tools/tests/test_oot_aspect.py | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/dm/games/0400486024DEC000/patches.py b/dm/games/0400486024DEC000/patches.py index 8c531c0760..64ca35e8d8 100644 --- a/dm/games/0400486024DEC000/patches.py +++ b/dm/games/0400486024DEC000/patches.py @@ -79,7 +79,7 @@ def dynamic_resolution(ctx): # init (main+0x1087b70, the size from its config struct) and the mode change (main+0x6502e0: a level's size × its scale, capped by a table). The dynamic resolution scales from the # output height by one factor for both axes (main+0x31d3f64..), so its sizes follow the output's shape. The stub scales the requested size by the factors (the width for a wider # screen, the height for a taller one); when that is more than 1.5 times the requested size's pixels (beyond 2.67:1 or 1.19:1, a portrait window) it scales both sides down to -# 1.5 times, so the pixel count stays in bounds whatever the shape. Each side is rounded to a multiple of 8, into state +0xb0 (scratch, never the caller's struct: the init passes +# 1.5 times, so the pixel count stays in bounds whatever the shape. Each side is rounded to a multiple of 8, into state +0xe0 (scratch, never the caller's struct: the init passes # its config), and the setter runs on it. It also copies the camera's multipliers for that shape (state +0xc4 / +0xc8, from the loader) to +0xb8 / +0xbc (0 at 16:9): the camera # (camera_projection) follows the size the game renders at, not the shape the host asks for, as the game sets its size only at its init, at a mode change and when shape_change # asks it to. A leaf: x9 / x10 / x16 / s16..s19 are free (the setter loads w8..w11 before it reads them; s16..s19 are caller-saved), x1 an argument it only reads @@ -90,7 +90,7 @@ def dynamic_resolution(ctx): 0x1e330a52, 0x1e2f1013, 0x1e330a52, 0x1e310a13, 0x1e322260, 0x540000ad, # more than 1.5 × the requested pixels? 0x1e331a52, 0x1e21c252, 0x1e320a10, 0x1e320a31, # both sides × sqrt(limit / pixels) 0x1e390209, 0x11001129, 0x121d7129, 0x1e39022a, 0x1100114a, 0x121d714a, # each to a multiple of 8 - 0x29162a09, 0x9102c201) # into state +0xb0; x1 -> it + 0x291c2a09, 0x91038201) # into state +0xe0; x1 -> it def screen_shape(ctx): here, emit, fs_site, fs_put, FS_STATE_OFF = ctx.here, ctx.emit, ctx.site, ctx.put, ctx.state site = ctx.others.get('output size') diff --git a/dm/tools/tests/test_oot_aspect.py b/dm/tools/tests/test_oot_aspect.py index a4241c4196..94a66a24e8 100644 --- a/dm/tools/tests/test_oot_aspect.py +++ b/dm/tools/tests/test_oot_aspect.py @@ -95,14 +95,14 @@ def test_the_camera_multipliers_go_with_the_size(self): def test_wider_keeps_the_height(self): for ratio, docked, handheld in ((2.2, (2536, 1152), (1904, 864)), (2.4, (2768, 1152), (2072, 864)), (21 / 9, (2688, 1152), (2016, 864))): self.shape(ratio) - self.assertEqual(self.call(2048, 1152), (self.state + 0xb0, docked), ratio) - self.assertEqual(self.call(1536, 864), (self.state + 0xb0, handheld), ratio) + self.assertEqual(self.call(2048, 1152), (self.state + 0xe0, docked), ratio) + self.assertEqual(self.call(1536, 864), (self.state + 0xe0, handheld), ratio) def test_taller_keeps_the_width(self): for ratio, docked, handheld in ((1.43, (2048, 1432), (1920, 1344)), (1.3, (2048, 1576), (1920, 1480)), (4 / 3, (2048, 1536), (1920, 1440))): self.shape(ratio) - self.assertEqual(self.call(2048, 1152), (self.state + 0xb0, docked), ratio) - self.assertEqual(self.call(1920, 1080), (self.state + 0xb0, handheld), ratio) + self.assertEqual(self.call(2048, 1152), (self.state + 0xe0, docked), ratio) + self.assertEqual(self.call(1920, 1080), (self.state + 0xe0, handheld), ratio) def test_any_shape_keeps_the_pixels_in_bounds(self): for ratio in (1179 / 2556, 0.75, 1.0, 3.0, 5.0): # an iPhone held upright, a portrait window, a square, ultrawide beyond 21:9