diff --git a/dm/games/010015100B514000/README.md b/dm/games/010015100B514000/README.md index 143bda1557..2c0b12c8b2 100644 --- a/dm/games/010015100B514000/README.md +++ b/dm/games/010015100B514000/README.md @@ -100,10 +100,16 @@ Game-specific findings; the general host changes they led to are listed under [F ## Found on this game +- **Resolved: raw companion title/menu holds 30 fps with GPU time to spare**: [Acquire fences keep pacing out of frame-cost samples](../../../docs/reference/frame-rate.md#acquire-fences-keep-pacing-out-of-frame-cost-samples), measured 29.7 → 59.9 fps with the game's own interval controller. + - [Offline pipeline archive missing for the device GPU](../../../docs/reference/shaders.md#offline-pipeline-archive-missing-for-the-device-gpu): archive targets can follow profile metadata while iOS Metal 4 remains opt-in; [the probe above](#offline-pipeline-archive-probe) records the build evidence and open device checks. General host changes first needed here; each applies to every game: - **Resolved: `Ver. ?` on the title screen of a build packed from an update-only NUP**: [A game packed from a NUP or a dump shows "Ver. ?" for a build its profile registers](../../../docs/reference/packing.md#a-game-packed-from-a-nup-or-a-dump-shows-ver--for-a-build-its-profile-registers). One Mac boot before and after. + +- **SDK 21 movie startup after New Game**: [Virtual output properties and the event/no-flags overload](../../../docs/reference/audio.md#sdk-21-virtual-output-properties-and-the-eventno-flags-overload). + +- **Resolved: raw companion shader build failures**: [Metal parser nesting](../../../docs/reference/shaders.md#metals-parser-rejects-deeply-nested-control-flow) and [BAHS control data mistaken for descriptors](../../../docs/reference/shaders.md#a-control-sections-words-read-as-a-bahs-entry). - **Resolved: Mac launcher metadata matches native defaults**: [A Mac front end's default settings change Metal execution paths](../../../docs/reference/packing.md#a-mac-front-ends-default-settings-change-metal-execution-paths), found by metadata generation and source inspection; no gameplay measurement. - [A capture refused after a change to the reflection structs](../../../docs/reference/nvn-objects.md#a-capture-refused-after-a-change-to-the-reflection-structs). - **zstd files inside a RomFS image** (`shader_split.py` `unzs`): [The zstd command given a path inside the image](../../../docs/reference/packing.md#the-zstd-command-given-a-path-inside-the-image). @@ -123,3 +129,15 @@ General host changes first needed here; each applies to every game: - dm/games/README.md says the heap check turns "Wonder's 1.44 GB" request into a 0xc0000000 alias region; 03 says it asks for 1.47 GB, and the profile's alias region is 0xB8000000. - Several addresses in 02 and 03 (`main+0x1070910`, `main+0x107dbc0`, `main+0x4da5c`, `main+0xb8dec`, `main+0xcfd24`, `main+0x36ceff0`) do not name their build; the present-interval ones are from 1.2.1. - 2 agl post-effect compute shaders do not translate. + +## Raw Switch 2 companion + +**Bring-up, separate from the tested Switch 1 builds above.** The unpacked 1.2.1 update can contain Switch 2 program `040015100B514001` (main build `ec47e6f8846d2a7b6892e2682845c725`) in `updates/327680/program1/`, with the original program's files beside it and DLC `010015100B515001` under nested base/update snapshots. The dump builder selects this complete Switch 2 companion by default; `--program 0` explicitly selects the primary. The companion mounts its own RomFS as `content`, the primary's as `conorig` (`DM_ROMFS_PROGRAM_0`, [indexed game files](../../../docs/reference/services.md#raw-switch-2-companion-fonts-and-indexed-game-files)) and the raw DLC as `aocup` (index 1). Unknown companion builds receive their own provisional profile outside the repository; this build needs `render_cond on` in it (below). Mac only: iOS indexed RomFS bundling is not implemented. + +**New Game / Continue crash: resource heap slack.** Without it, a fresh save crashes about 3 s into New Game with `BufferInitialize: pool has no storage` and `CRASH sig=11 … h_nvnMemoryPoolGetFlags` ([#1580](https://github.com/seedds/nxnative/issues/1580)). The resource loader (`main+0x130460`) sizes each resource's ExpHeap as resource-size-table value + max(alignment, 0x20) + 8 (`main+0x1304e8..0x130514`, then `main+0x131e00`; constructor `main+0x2464d0`, vtable `main+0x394f220`). `Model/NpcCommonerA.bfres` gets `0x8d03c8` bytes; its combined MatBlock + DisplayList request `0x5424` (`main+0x1ff04c` / `+0x1ff068`) rounds to a `0x6000` GPU chunk (`main+0x287048`), leaving `0x3a8` for a `0x418` descriptor. [`patches.py`](patches.py) registers one cut for this build, `main+0x130514` `adds w0,w8,#8` → `adds w0,w8,#4,lsl #12` (+16 KB per resource heap), the allowance the Switch 1 builds carried before their command records were compacted. With the original DLC and no resource-size-table overlay, a fresh save plays the Flower Kingdom opening (its NPCs, Yoshi, the castle scene) and a copied save continues into a course. The native NVN2 byte accounting asked for in #1580 remains open; this is a heap allowance, not that correction. + +**Overworld blue tint: `render_cond on`.** With the key off, ignored conditional clears wipe the half-resolution depth and occlusion targets between their full-screen draws; the occlusion mask goes black, flowerbeds turn saturated blue, nearby pipes dark blue. Frame-3300 pass dumps with the key on show equal timestamp counters skipping those repeated clears, and the map shows lavender flowerbeds, white pipes and its bright lighting ([conditional rendering](../../../docs/reference/rendering.md#conditional-rendering-evaluated-at-replay)). + +**Course lighting pulsing with `DM_NVN_ENC_THREAD=1`.** With `render_cond` on, a timestamp report's 16-byte record was written in the submission's completion handler; with Metal encoding on its own thread that landed after the game had already set the counter word to 1 for a later frame's conditional clear, so the next check read `0 / 0` and skipped that frame's lighting blocks (24 of 90 course frames had four blocks skipped, the block count alternating 22 / 18). The counter word (always 0) is now stored at replay, in order with the game's writes, and the completion handler adds only the time (`nvn_mtl.m` `OP_REPORT_COUNTER`, `post_run`): 1 of 90 frames with one block, a steady 21 blocks a frame. Without the encode thread the pattern did not occur before or after. + +**Title and shaders.** A 25-second native boot with isolated saves mounted `content`, `conorig` and `aocup` and reached frame 600; a diagnostic shader build rendered the Switch 2 Edition / Bellabel Park 1.2.1 title with L + R glyphs and the Start/Assist Mode menu. The full Mac shader build produces all 30,630 valid shaders with no failures: two compute shaders formerly blocked by [Metal's parser nesting limit](../../../docs/reference/shaders.md#metals-parser-rejects-deeply-nested-control-flow) build, and the apparent unsupported Ampere instruction came from [control metadata mistaken for a BAHS descriptor](../../../docs/reference/shaders.md#a-control-sections-words-read-as-a-bahs-entry) (three false descriptors now excluded). Individual Bellabel Park modes and long play are not verified. diff --git a/dm/games/010015100B514000/patches.py b/dm/games/010015100B514000/patches.py new file mode 100644 index 0000000000..3ddb9b477d --- /dev/null +++ b/dm/games/010015100B514000/patches.py @@ -0,0 +1,23 @@ +# Super Mario Bros. Wonder (010015100B514000): the rewriter's patches of this game's own builds, keyed by NSO build ID +# (dm/tools/patch_nso.py, "Registered patches"). SDK cuts live in dm/tools/sdk_patches.py. +CUTS = { + # Nintendo Switch 2 Edition, Program 1 (040015100B514001) main, display version 1.2.1. + # Resource heap slack, as the Switch 1 builds once carried (README, Memory, "Removed workaround"; issue #1580). + # The resource loader (main+0x130460) sizes each resource's sead::ExpHeap as the resource-size-table value + # + max(alignment, 0x20) + 8 (main+0x1304e8..0x130514) and passes it to the heap's create (main+0x131e00). + # On the host, Model/NpcCommonerA.bfres (heap 0x8d03c8) runs short: the model's combined MatBlock + DisplayList + # request 0x5424 (main+0x1ff04c / +0x1ff068) rounds to a 0x6000 GPU chunk (main+0x287048), leaving 0x3a8 for + # a 0x418 descriptor, and the game then initializes a buffer on a null memory pool (crash in + # nvnMemoryPoolGetFlags). The native NVN2 command sizes that would keep the request at or below 0x5000 are + # not known (#1580). Every resource heap gets 16 KB more instead: adds w0,w8,#8 -> adds w0,w8,#4,lsl #12 + # (+0x4000 in place of +8). The b.eq that follows still sees the flags of the same adds. + 'ec47e6f8846d2a7b6892e2682845c725': [ + (0x130514, '00210031', '00114031', 'resource heap size: adds w0,w8,#8 -> adds w0,w8,#0x4000 (ExpHeap size +16 KB - 8)'), + ], +} + +BINDING = [ + {'what': 'resource heap size +16 KB (ExpHeap create size, main+0x130514)', 'now': 'build', 'goal': 'host', 'purpose': 'memory', + 'why': "the shortfall comes from the host's NVN2 command-memory accounting (#1580); a correct host representation would make the cut unnecessary for every build", + 'blocked': 'the native NVN2 sizes of a recorded indexed draw are not established (#1580)'}, +] diff --git a/dm/gfx/nvn/hle_nvn.c b/dm/gfx/nvn/hle_nvn.c index 7255e2cf5d..cfca856026 100644 --- a/dm/gfx/nvn/hle_nvn.c +++ b/dm/gfx/nvn/hle_nvn.c @@ -776,14 +776,19 @@ NVN(nvnProgramSetShaders, NVNboolean, NVNprogram *p, int n, const NVNshaderData } // ---------- Sync / events ---------- -NVN(nvnSyncInitialize, NVNboolean, NVNsync *s, NVNdevice *d) { (void)d; *(sync_t *)s = (sync_t){MAGIC('S', 'Y', 'N', 'C'), 0, 0}; return 1; } -NVN(nvnSyncFinalize, void, NVNsync *s) { ((sync_t *)s)->magic = 0; } +static void window_acquire_wait(win_t *o, int idx, int mark_frame); +static int sync_acquire_wait(sync_t *s, uint64_t timeout); +NVN(nvnSyncInitialize, NVNboolean, NVNsync *s, NVNdevice *d) { (void)d; *(sync_t *)s = (sync_t){.magic = MAGIC('S', 'Y', 'N', 'C')}; return 1; } +NVN(nvnSyncFinalize, void, NVNsync *s) { ((sync_t *)s)->magic = 0; atomic_store(&((sync_t *)s)->acquire_win, NULL); } NVN(nvnSyncSetDebugLabel, void, NVNsync *s, const char *l) { (void)s; (void)l; } // Fence = "all commands previously submitted to the queue have completed": record the fence value of the most recent submit -NVN(nvnQueueFenceSync, void, NVNqueue *q, NVNsync *s, NVNsyncCondition c, int f) { (void)q; (void)c; (void)f; ((sync_t *)s)->fenced = 1; ((sync_t *)s)->fence = mtl_last_fence(); } +NVN(nvnQueueFenceSync, void, NVNqueue *q, NVNsync *s, NVNsyncCondition c, int f) { (void)q; (void)c; (void)f; ((sync_t *)s)->fenced = 1; ((sync_t *)s)->fence = mtl_last_fence(); atomic_store(&((sync_t *)s)->acquire_win, NULL); } // A sync object that was never fenced waits until timeout, as on a real device; a fenced one is released only after the GPU has actually completed up to that fence value -// A GPU-side wait on a sync: the host has a single Metal queue that executes submissions in order, so commands before the fence are always done first and nothing needs waiting. A sync not fenced yet (fenced later by another thread) is not waited for here; logged once -NVN(nvnQueueWaitSync, NVNboolean, NVNqueue *q, const NVNsync *s) { (void)q; static int once; if (!((const sync_t *)s)->fenced && !once++) LOG("[nvn] QueueWaitSync: the sync object has no fence yet, the host does not wait"); return 1; } +// GPU command fences are ordered by the Metal queue. Acquire leases also depend on the host presenter, so the inline interpreter resolves those before using the texture. +// A sync not fenced yet (fenced later by another thread) is not waited for here; logged once. +NVN(nvnQueueWaitSync, NVNboolean, NVNqueue *q, const NVNsync *s) { (void)q; (void)sync_acquire_wait((sync_t *)s, UINT64_MAX); static int once; if (!((const sync_t *)s)->fenced && !once++) LOG("[nvn] QueueWaitSync: the sync object has no fence yet, the host does not wait"); return 1; } +// The CPU interpreter cannot submit a draw against a texture still owned by the presenter. Resolve an acquire dependency before recording / submitting that draw. +NVN(nvnCommandBufferWaitSync, void, NVNcommandBuffer *c, const NVNsync *s) { (void)c; (void)sync_acquire_wait((sync_t *)s, UINT64_MAX); } // ZCull data: the host does no ZCull, so there is nothing to save / restore NVN(nvnCommandBufferRestoreZCullData, void, NVNcommandBuffer *c, NVNbufferAddress a, size_t n) { (void)c; (void)a; (void)n; } NVN(nvnCommandBufferSaveZCullData, void, NVNcommandBuffer *c, NVNbufferAddress a, size_t n) { (void)c; (void)a; (void)n; } @@ -793,11 +798,18 @@ NVN_NOPROTO(nvnCommandBufferSetStencilCullCriteria, void, NVNcommandBuffer *c, i static _Atomic uint64_t g_sync_ns; NVN(nvnSyncWait, NVNsyncWaitResult, const NVNsync *s, uint64_t timeout) { const sync_t *o = (const sync_t *)s; - if (!o->fenced) return NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; + uint64_t begin = now_ns(); int waited = !sync_acquire_wait((sync_t *)s, 0); NVNsyncWaitResult result; + uint64_t left = timeout; + if (waited && timeout != UINT64_MAX) { uint64_t spent = now_ns() - begin; left = spent < timeout ? timeout - spent : 0; } + if (waited && (!left || !sync_acquire_wait((sync_t *)s, left))) { result = NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; goto done; } + if (!o->fenced) { result = NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; goto done; } int (*w)(uint64_t, uint64_t) = o->fenced == 2 ? mtl_wait_acq : mtl_wait; - if (w(o->fence, 0)) return NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED; - uint64_t t0 = now_ns(); int ok = w(o->fence, timeout); atomic_fetch_add(&g_sync_ns, now_ns() - t0); // the frame's blocked SyncWait time (@sync_ms) - return ok ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; + if (w(o->fence, 0)) { result = waited ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED; goto done; } + if (timeout != UINT64_MAX) { uint64_t spent = now_ns() - begin; timeout = spent < timeout ? timeout - spent : 0; } + result = w(o->fence, timeout) ? NVN_SYNC_WAIT_RESULT_CONDITION_SATISFIED : NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED; +done: + atomic_fetch_add(&g_sync_ns, now_ns() - begin); // includes the acquire fence, within the same caller timeout + return result; } NVN(nvnEventBuilderSetDefaults, void, NVNeventBuilder *b) { memset(b, 0, sizeof(ev_builder_t)); } NVN(nvnEventBuilderSetStorage, void, NVNeventBuilder *b, const NVNmemoryPool *p, int64_t off) { *(ev_builder_t *)b = (ev_builder_t){(const pool_t *)p, off}; } @@ -1394,7 +1406,6 @@ static void nvn_nop(void) {} static const char *const g_nops[] = { "nvnCommandBufferSetPatchSize", "nvnCommandBufferSetDepthBounds", - "nvnCommandBufferWaitSync", "nvnCommandBufferSetShaderScratchMemory", "nvnCommandBufferPushDebugGroupDynamic", "nvnCommandBufferPopDebugGroup", @@ -1491,11 +1502,9 @@ static uint64_t g_acq_wait_ns, g_pq_wait_ns; // Accumulated display-link wait static uint64_t g_fs_start, g_fs_wait; // frame step (fstep_tick): when the first acquire after a present returned (the frame's start, 0 = none yet), and its pacing wait uint64_t dm_nvn_frame_count(void) { return g_frame; } extern uint64_t g_t0_ns; // Host process start point (main.c; same base as the UIMSG timestamps) -NVN(nvnWindowAcquireTexture, NVNwindowAcquireTextureResult, NVNwindow *w, NVNsync *s, int *idx) { - win_t *o = (win_t *)w; - *idx = o->ntex ? o->next++ % o->ntex : 0; +static void window_acquire_wait(win_t *o, int idx, int mark_frame) { // Vsync blocks here (real-device NVN: when all textures in the swapchain are waiting for display, wait for the next one to be released), not in present -- the game often calls present while holding a lock - uint64_t at = *idx < 4 ? o->avail[*idx] : 0, now = dm_detclock_enabled() ? guest_ns() : now_ns(), wait = at > now ? at - now : 0, pace_ns = 0; + uint64_t at = idx < 4 ? o->avail[idx] : 0, now = dm_detclock_enabled() ? guest_ns() : now_ns(), wait = at > now ? at - now : 0, pace_ns = 0; if (!dm_detclock_enabled()) { // Register the acquire thread as a time-constraint thread (no longer registered only when it "arrived early": it must also declare itself when it is constantly behind in the hot state) static uint64_t rt_period; uint64_t period = present_period(o), rtp = rt_period_ns(o, period); if (rtp && rt_period != rtp) { rt_period = rtp; set_time_constraint(rtp); } @@ -1517,10 +1526,48 @@ NVN(nvnWindowAcquireTexture, NVNwindowAcquireTextureResult, NVNwindow *w, NVNsyn pace_ns += mtl_snap_acquire_wait(); // ring capacity is waited for at acquire, where the game does not hold its present lock g_acq_wait_ns += wait; trow(g_frame, "@late_ms", wait ? (now_ns() - at) / 1e6 : 0); // The previous present of this texture has not yet been processed by the present thread (display cannot keep up, a whole swapchain round behind): wait for it to be returned, then hand it to the game to draw - uint64_t tk = *idx < 4 ? o->ticket[*idx] : 0; + uint64_t tk = idx < 4 ? o->ticket[idx] : 0; if (tk) { uint64_t w0 = now_ns(); mtl_present_wait(tk); g_pq_wait_ns += now_ns() - w0; } - if (s) { ((sync_t *)s)->fenced = 2; ((sync_t *)s)->fence = mtl_last_fence(); } // The swapchain texture becomes available once the commits before this point have completed (fenced 2: an acquire sync, for the wait ledger) - if (!g_fs_start) { g_fs_start = now_ns(); g_fs_wait = pace_ns; } // the frame starts when the first acquire after a present returns: on a display-link tick when on time + if (mark_frame && !g_fs_start) { g_fs_start = now_ns(); g_fs_wait = pace_ns; } +} +static int window_acquire_ready(const win_t *o, int idx) { + uint64_t at = idx < 4 ? o->avail[idx] : 0, tk = idx < 4 ? o->ticket[idx] : 0; + return at <= now_ns() && (!tk || mtl_present_ready(tk)); +} +// Acquisition publishes a fence; its consumer waits for the buffer. Blocking in AcquireTexture itself makes sead count the vsync sleep as CPU work and hold interval 2. +// Infinite waits keep the present watchdog and display-link pacing; finite waits poll the same buffer ownership / deadline conditions within the caller's budget. +static int sync_acquire_wait(sync_t *s, uint64_t timeout) { + if (!s || !atomic_load(&s->acquire_win)) return 1; + uint64_t begin = now_ns(), end = timeout > UINT64_MAX - begin ? UINT64_MAX : begin + timeout; + for (;;) { + if (!atomic_load(&s->acquire_win)) return 1; + int idle = 0; + if (atomic_compare_exchange_strong(&s->acquire_busy, &idle, 1)) break; + uint64_t now = now_ns(); if (!timeout || now >= end) return 0; + sleep_until_ns(end - now < 1000000 ? end : now + 1000000); + } + win_t *o = atomic_load(&s->acquire_win); int ready = !o || window_acquire_ready(o, s->acquire_idx); + if (!ready && timeout == UINT64_MAX) { window_acquire_wait(o, s->acquire_idx, 0); ready = window_acquire_ready(o, s->acquire_idx); } + while (!ready && timeout) { + uint64_t now = now_ns(); if (now >= end) break; + uint64_t wake = end - now < 1000000 ? end : now + 1000000, at = o->avail[s->acquire_idx]; + if (at > now && at < wake) wake = at; + sleep_until_ns(wake); + ready = window_acquire_ready(o, s->acquire_idx); + } + if (ready) { + atomic_store(&s->acquire_win, NULL); + if (o && !g_fs_start) { g_fs_start = now_ns(); g_fs_wait = g_fs_start - begin; } + } + atomic_store(&s->acquire_busy, 0); + return ready; +} +NVN(nvnWindowAcquireTexture, NVNwindowAcquireTextureResult, NVNwindow *w, NVNsync *s, int *idx) { + win_t *o = (win_t *)w; + *idx = o->ntex ? o->next++ % o->ntex : 0; + int defer = s && !mtl_snap_on() && !dm_detclock_enabled(); // snapshot-ring and deterministic test modes retain their synchronous acquire path + if (!defer) window_acquire_wait(o, *idx, 1); + if (s) { sync_t *a = (sync_t *)s; a->fenced = 2; a->fence = mtl_last_fence(); a->acquire_idx = *idx; atomic_store(&a->acquire_win, defer ? o : NULL); } return NVN_WINDOW_ACQUIRE_TEXTURE_RESULT_SUCCESS; } // The older API (NVN 53, Super Mario 3D World + Bowser's Fury): the queue waits for the texture on the GPU and no sync object is handed out. The host replays in submission order, so later commands already come after earlier presents; the same as WindowAcquireTexture without a sync object @@ -3072,6 +3119,6 @@ int nvn_host_write_texture(const void *nvntex, const void *src, int w, int h, si int nvn_host_signal_sync(void *nvnsync) { sync_t *o = nvnsync; if (!o || o->magic != MAGIC('S', 'Y', 'N', 'C')) return -1; - o->fence = 0; o->fenced = 1; return 0; + o->fence = 0; o->fenced = 1; atomic_store(&o->acquire_win, NULL); return 0; } NVN(nvnTextureFlushTexels, void, const NVNtexture *t, const NVNtextureView *v, const NVNcopyRegion *r) { (void)t; (void)v; (void)r; } // unified memory + dirty page tracking: nothing to flush diff --git a/dm/gfx/nvn/nvn_capture.c b/dm/gfx/nvn/nvn_capture.c index dc6de22364..6e97ac52ca 100644 --- a/dm/gfx/nvn/nvn_capture.c +++ b/dm/gfx/nvn/nvn_capture.c @@ -10,7 +10,8 @@ __asm__(".linker_option \"-lcompression\""); // libcompression (LZFSE) links i #define CAP_SIZE(T, n) _Static_assert(sizeof(T) == (n), "nvn_capture.c: " #T " changed size: update cap_rec_fields and cap_layout") #define CAP_OFF(T, f, n) _Static_assert(offsetof(T, f) == (n), "nvn_capture.c: " #T "." #f " moved: update cap_rec_fields and cap_layout") CAP_SIZE(tex_t, 128); CAP_OFF(tex_t, mem, 88); CAP_OFF(tex_t, pool, 96); CAP_OFF(tex_t, serial, 120); -CAP_SIZE(pool_t, 40); CAP_SIZE(dpool_t, 24); CAP_SIZE(prog_t, 112); CAP_SIZE(sync_t, 16); CAP_SIZE(samp_par, 40); CAP_SIZE(view_t, 16); +// CAP_SYNC serializes only an object count; replay initializes syncs to zero, so their live acquire pointers are never written to a capture. +CAP_SIZE(pool_t, 40); CAP_SIZE(dpool_t, 24); CAP_SIZE(prog_t, 112); CAP_SIZE(sync_t, 32); CAP_SIZE(samp_par, 40); CAP_SIZE(view_t, 16); CAP_SIZE(p_set_rt, 240); CAP_OFF(p_set_rt, n, 0); CAP_OFF(p_set_rt, color, 8); CAP_OFF(p_set_rt, depth, 208); CAP_SIZE(p_bind_program, 16); CAP_OFF(p_bind_program, prog, 0); CAP_SIZE(NVNbufferRange, 16); CAP_SIZE(p_bind_vbuf, 264); CAP_OFF(p_bind_vbuf, n, 4); CAP_OFF(p_bind_vbuf, r, 8); @@ -42,7 +43,7 @@ _Static_assert(sizeof(p_copy_strides) == 16 && COPY_STRIDES_OFF == 192, "nvn_cap const char *cap_layout(void) { // sizes and pointer offsets as the walk knows them (the asserts above hold them to nvn_obj.h); bump this text with any change to the walk return "format 1\nops 54\n" - "tex_t 128 mem 88 pool 96 serial 120\npool_t 40\ndpool_t 24\nprog_t 112\nsync_t 16\nsamp_par 40\nview_t 16\n" + "tex_t 128 mem 88 pool 96 serial 120\npool_t 40\ndpool_t 24\nprog_t 112\nsync_t 32\nsamp_par 40\nview_t 16\n" "p_set_rt 240 color 8 depth 208\np_bind_program 16\np_bind_vbuf 264 r 8\np_bind_vbuf1 16 addr 8\np_bind_buf 24 addr 8\np_bind_ubo_small 16 addr 8\n" "p_bind_handles 280 src 16\np_bind_tex1 24 nobase 0x100 short 16\np_bind_state short\np_draw 48 ind 24 mdi_cnt 32\np_draw_idx 56 ibuf 24 ind 32 mdi_cnt 40\np_dispatch 32 ind 24\np_set_pool 8\n" "p_copy_b2t 64 src 0 tex 8\np_copy_t2t 104 src 0 dst 8\np_clear_tex 72 tex 0\np_clear_buf 24 addr 0\np_tf_bind 24 addr 8\np_tf_state 16 ctl 8\np_render_cond 16 addr 8\n" diff --git a/dm/gfx/nvn/nvn_mtl.h b/dm/gfx/nvn/nvn_mtl.h index b7dca8985a..e124818994 100644 --- a/dm/gfx/nvn/nvn_mtl.h +++ b/dm/gfx/nvn/nvn_mtl.h @@ -56,3 +56,4 @@ uint64_t mtl_present_ext(uint64_t serial, double period, void *ev, uint64_t ev_v void mtl_present_layer(uint32_t w, uint32_t h); // a sized vi layer (hle_vi.c): the picture's shape instead of the display's 16:9 void mtl_present_screen(uint32_t w, uint32_t h); // a screen taller than 16:9 (loader.c, DM_SCREEN_ASPECT): the picture's shape for a texture of that shape void mtl_present_wait(uint64_t ticket); // as in nvn_obj.h: wait until that present has been handed to the GPU +int mtl_present_ready(uint64_t ticket); diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 0482da208d..98c2898191 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -116,6 +116,7 @@ static inline uint64_t tex_slot_h(const uint64_t *tex, const uint64_t *img, unsi // it to its own ticket. Waiting on g_pq_done, a display-link present never waited here: with three presents in flight, or two after a dropped frame, the allocator was reset // under a present the GPU had not run, and that present's drawable read back black (t_present_alloc; docs/reference/frame-rate.md) static uint64_t g_present_alloc_done[3]; +static uint64_t g_present_gpu_ticket[3]; // under g_pq_mu: the source texture remains GPU-owned until this present's feedback static pthread_mutex_t g_pq_mu = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t g_pq_cv = PTHREAD_COND_INITIALIZER; static pthread_cond_t g_link_cv = PTHREAD_COND_INITIALIZER; @@ -1859,7 +1860,9 @@ static void post_run(post_rec *r, uint64_t c0, uint64_t c1) { free(l); uint64_t rec[2] = {sum, gpu_ticks()}; memcpy((void *)ops[i].addr, rec, sizeof rec); } else if (ops[i].kind == 4) mirror_rb_land(ops[i].vis); - else { uint64_t t = rep_ticks(&ops[i], sc, seg_last, c0, c1), rec[2] = {(ops[i].type == 0 || ops[i].type == 0x10) && !DM_GAME_RENDER_COND ? t : 0, t}; memcpy((void *)ops[i].addr, rec, sizeof rec); } // render_cond: a timestamp report's counter field is 0, as on the GPU (see rc_value) + else { uint64_t t = rep_ticks(&ops[i], sc, seg_last, c0, c1), rec[2] = {(ops[i].type == 0 || ops[i].type == 0x10) && !DM_GAME_RENDER_COND ? t : 0, t}; + if (DM_GAME_RENDER_COND && (ops[i].type == 0 || ops[i].type == 0x10)) memcpy((uint8_t *)ops[i].addr + 8, &rec[1], 8); // its counter word 0 was stored at replay (OP_REPORT_COUNTER) + else memcpy((void *)ops[i].addr, rec, sizeof rec); } // render_cond: a timestamp report's counter field is 0, as on the GPU (see rc_value) } free(ops); seg_clk_put(sc); r->ops = NULL; r->sc = NULL; atomic_store(&r->claimed, 1); @@ -13395,6 +13398,10 @@ void mtl_exec(const uint8_t *p) { case OP_REPORT_COUNTER: { uint64_t t, ad; memcpy(&t, a, 8); memcpy(&ad, (const uint8_t *)a + 8, 8); g_cnt_report[t & 31]++; PD("Q report type=%llu cmd=%p dst=%#llx", (unsigned long long)t, p, (unsigned long long)ad); if (DM_GAME_RENDER_COND) rc_note(ad, t == 0 || t == 0x10 ? 1 : 2); + // render_cond: a timestamp report's counter word (always 0) is stored now, in order with the game's own writes of it; the completion handler adds only the time + // (post_run). Stored there, it landed after the game had already set the word to 1 for a later frame's conditional clear: with Metal encoding on its own thread, + // Super Mario Bros. Wonder's Switch 2 Edition then saw 0 / 0, skipped that frame's lighting blocks and its courses' lighting pulsed (about one frame in four) + if (DM_GAME_RENDER_COND && (t == 0 || t == 0x10)) *(volatile uint64_t *)ad = 0; if (t == NVN_COUNTER_TYPE_SAMPLES_PASSED) { post((post_op){3, NULL, 0, 0, ad, (uint32_t)t, vis_snapshot()}); g_vis_cur = -1; } else mtl_sub_counter(ad, (uint32_t)t); break; } @@ -13404,7 +13411,7 @@ void mtl_exec(const uint8_t *p) { g_rc_skip = known && (rc->mode ? x == y : x != y); PD("E %llu conditional render %s addr=%#llx values %llu / %llu%s%s", g_stat_pass, rc->mode ? "draw if not equal" : "draw if equal", (unsigned long long)rc->addr, (unsigned long long)x, (unsigned long long)y, known ? "" : " (unknown, drawn)", g_rc_skip ? " → not drawn" : ""); } break; } - case OP_FENCE_SYNC: { sync_t *s; memcpy(&s, a, 8); s->fence = g_fence + 1; s->fenced = 1; // the fence value this submission (segment) commits with + case OP_FENCE_SYNC: { sync_t *s; memcpy(&s, a, 8); s->fence = g_fence + 1; s->fenced = 1; atomic_store(&s->acquire_win, NULL); // the fence value this submission (segment) commits with PD("Y fence_sync cmd=%p sync=%p fence=%llu", p, (void *)s, (unsigned long long)s->fence); break; } case OP_RESET_COUNTER: { uint64_t t; memcpy(&t, a, 8); g_cnt_reset[t & 31]++; PD("Q reset type=%llu cmd=%p", (unsigned long long)t, p); if (t == NVN_COUNTER_TYPE_SAMPLES_PASSED) { @@ -14343,11 +14350,11 @@ static int present_now(const tex_t *t, double period, uint64_t frame, uint64_t t MTL4CommitOptions *o = [MTL4CommitOptions new]; [o addFeedbackHandler:^(id f) { if (f.error) { MLOG("[RED] display command buffer error: %s", f.error.description.UTF8String); gpu_lost_check(f.error, "display"); } - pthread_mutex_lock(&g_pq_mu); if (ticket > g_pq_done) g_pq_done = ticket; if (ticket > g_present_alloc_done[alloc_i]) g_present_alloc_done[alloc_i] = ticket; g_pq_inflight--; pthread_cond_broadcast(&g_pq_cv); + pthread_mutex_lock(&g_pq_mu); if (ticket > g_pq_done) g_pq_done = ticket; if (ticket > g_present_alloc_done[alloc_i]) g_present_alloc_done[alloc_i] = ticket; if (g_present_gpu_ticket[alloc_i] == ticket) g_present_gpu_ticket[alloc_i] = 0; g_pq_inflight--; pthread_cond_broadcast(&g_pq_cv); present_gpu_note(f.GPUEndTime - f.GPUStartTime, fx_kind); pthread_mutex_unlock(&g_pq_mu); }]; id buffers[] = {cb}; - pthread_mutex_lock(&g_pq_mu); g_pq_inflight++; pthread_mutex_unlock(&g_pq_mu); + pthread_mutex_lock(&g_pq_mu); g_pq_inflight++; g_present_gpu_ticket[alloc_i] = ticket; pthread_mutex_unlock(&g_pq_mu); [g_q4 commit:buffers count:1 options:o]; if (target_ts > 0) { static double leads[300], e2e[300]; // e2e: game enqueue -> system-scheduled present (including the time waiting in the queue for the display-link callback), the full span of input latency that our side can read @@ -14762,6 +14769,12 @@ static uint64_t present_enqueue(const tex_t *t, double period, const int crop[4] void mtl_present_progress(uint64_t *queued, uint64_t *done) { pthread_mutex_lock(&g_pq_mu); *queued = g_pq_in; *done = g_pq_done; pthread_mutex_unlock(&g_pq_mu); } // Do not wait during discard: the frame enqueued only after the flag was set (the present thread had just passed the flag check) has nobody to take it, and waiting for it would stall again; it is left for the following mtl_drain to clear // Do not wait indefinitely in display-link mode: see the note at STALL_NS. Only frames "not yet taken by the display link" are dropped; the one already on its way to the screen is waited for until handed to the GPU (same dropping as when entering the background) +int mtl_present_ready(uint64_t tk) { + pthread_mutex_lock(&g_pq_mu); + int ready = !tk || ((g_pq_done >= tk || atomic_load(&g_present_discard)) && g_present_gpu_ticket[tk % 3] != tk); + pthread_mutex_unlock(&g_pq_mu); + return ready; +} void mtl_present_wait(uint64_t tk) { pthread_mutex_lock(&g_pq_mu); while (g_pq_done < tk && !atomic_load(&g_present_discard)) { diff --git a/dm/gfx/nvn/nvn_obj.h b/dm/gfx/nvn/nvn_obj.h index 8d69dce407..06e76c97b8 100644 --- a/dm/gfx/nvn/nvn_obj.h +++ b/dm/gfx/nvn/nvn_obj.h @@ -55,7 +55,7 @@ typedef struct { uint32_t magic; samp_par p; } samp_t; // The per-stage shader keys are not kept in prog_t: the game's real NVNprogram may be smaller than the header's 0xC0 (growing prog_t corrupted the adjacent object in Mario); see prog_keys in hle_nvn.c typedef struct { uint32_t magic, nstage; NVNshaderData sd[6]; uint64_t serial; } prog_t; FITS(prog_t, NVNprogram); // serial: table key for the replay layer's resolved result -typedef struct { uint32_t magic, fenced; uint64_t fence; } sync_t; /* fenced 2: by WindowAcquireTexture */ FITS(sync_t, NVNsync); +typedef struct { uint32_t magic, fenced; uint64_t fence; _Atomic(void *) acquire_win; int acquire_idx; _Atomic int acquire_busy; } sync_t; /* fenced 2: by WindowAcquireTexture */ FITS(sync_t, NVNsync); typedef struct { const pool_t *pool; int64_t off; } ev_builder_t; FITS(ev_builder_t, NVNeventBuilder); typedef struct { uint32_t magic; volatile uint32_t *val; const pool_t *pool; int64_t off; } ev_t; FITS(ev_t, NVNevent); // pool / off: EventGetMemoryPool / Offset @@ -216,6 +216,7 @@ void mtl_exec_rest(const uint64_t *handles, int n); // Tells which handles in // Present: draw the swapchain texture to the front-end CAMetalLayer (only counts when there is no window) uint64_t mtl_present(const tex_t *t, double period_s, const int crop[4]); // Enqueue a present and return the ticket immediately; period_s: minimum time this image is displayed (= present period, 0 = unlimited) void mtl_present_wait(uint64_t ticket); // Wait until the present with this ticket has been processed (blit committed to the GPU queue) +int mtl_present_ready(uint64_t ticket); // Nonblocking acquire-fence poll: ticket retired AND any GPU read of its source finished void mtl_present_flush(void); // Wait until all enqueued presents have been processed void mtl_present_test_stall(uint64_t ns); // Unit test: extra pause in ns before each present on the present thread int mtl_display_hz(void); // Maximum refresh rate of the screen (Hz) when there is a window, 0 without a window diff --git a/dm/gfx/tests/t_nvn.c b/dm/gfx/tests/t_nvn.c index 78c38d144e..308326000e 100644 --- a/dm/gfx/tests/t_nvn.c +++ b/dm/gfx/tests/t_nvn.c @@ -19,6 +19,11 @@ static int g_fail, g_n; printf("%-44s expected %-12lld → got %-12lld %s\n", name, w_, g_, w_ == g_ ? "✓" : "✗"); if (w_ != g_) g_fail++; } while (0) #define P(name) ((NVNT_##name)proc(#name)) +typedef struct { NVNsync *sync; int result; uint64_t returned; } acquire_wait_test; +static void *acquire_wait_thread(void *arg) { + acquire_wait_test *t = arg; + t->result = h_nvnSyncWait(t->sync, UINT64_MAX); t->returned = now_ns(); return NULL; +} static PFNNVNDEVICEGETPROCADDRESSPROC gpa; static NVNdevice dev; static void *proc(const char *n) { void *p = (void *)gpa(&dev, n); if (!p) { printf("address lookup failed %s\n", n); exit(1); } return p; } @@ -662,11 +667,44 @@ int main(void) { int idx[4]; for (int f = 0; f < 4; f++) { P(nvnWindowAcquireTexture)(&win, &s2, &idx[f]); + P(nvnSyncWait)(&s2, UINT64_MAX); // consume the acquired buffer's fence before reusing it for (int b = 0; b < 5; b++) P(nvnCommandBufferBarrier)(&cb, 0); P(nvnQueuePresentTexture)(&q, &win, idx[f]); } EXPECT("swapchain rotates 0,1,0,1", 0x0101, idx[0] << 12 | idx[1] << 8 | idx[2] << 4 | idx[3]); EXPECT("sync signaled after acquire (0)", 0, P(nvnSyncWait)(&s2, 0)); + // A buffer not yet available is acquired asynchronously. Polling / finite waits must not consume its fence or overrun the timeout; later consumers still see the dependency. + { + NVNwindowBuilder awb; NVNwindow aw; NVNsync as; + P(nvnWindowBuilderSetDevice)(&awb, &dev); P(nvnWindowBuilderSetDefaults)(&awb); P(nvnWindowBuilderSetTextures)(&awb, 2, scp); P(nvnWindowInitialize)(&aw, &awb); + P(nvnSyncInitialize)(&as, &dev); win_t *a = (win_t *)&aw; int ix; + a->avail[0] = UINT64_MAX; + P(nvnWindowAcquireTexture)(&aw, &as, &ix); + EXPECT("acquire returns the buffer before its fence signals", 0, ix); + EXPECT("acquire fence poll is not ready", NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED, P(nvnSyncWait)(&as, 0)); + EXPECT("finite acquire wait expires", NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED, P(nvnSyncWait)(&as, 1000000)); + EXPECT("timeout preserves the pending acquire", NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED, P(nvnSyncWait)(&as, 0)); + a->avail[0] = 0; + EXPECT("the same fence becomes signaled when the buffer is ready", NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED, P(nvnSyncWait)(&as, 0)); + a->avail[1] = now_ns() + 20000000; + P(nvnWindowAcquireTexture)(&aw, &as, &ix); + acquire_wait_test wa = {&as, -1, 0}, wb = {&as, -1, 0}; pthread_t ta, tb; + pthread_create(&ta, NULL, acquire_wait_thread, &wa); pthread_create(&tb, NULL, acquire_wait_thread, &wb); + pthread_join(ta, NULL); pthread_join(tb, NULL); + EXPECT("both acquire waiters observe success", 1, wa.result <= 1 && wb.result <= 1 && wa.result >= 0 && wb.result >= 0); + EXPECT("neither waiter returns before buffer availability", 1, wa.returned >= a->avail[1] && wb.returned >= a->avail[1]); + a->avail[0] = now_ns() + 2000000; + P(nvnWindowAcquireTexture)(&aw, &as, &ix); P(nvnQueueWaitSync)(&q, &as); + EXPECT("queue wait resolves acquire ownership before GPU use", NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED, P(nvnSyncWait)(&as, 0)); + a->avail[1] = now_ns() + 2000000; + P(nvnWindowAcquireTexture)(&aw, &as, &ix); P(nvnCommandBufferWaitSync)(&cb, &as); + EXPECT("command-buffer wait resolves acquire ownership", NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED, P(nvnSyncWait)(&as, 0)); + a->avail[0] = UINT64_MAX; + P(nvnWindowAcquireTexture)(&aw, &as, &ix); P(nvnQueueFenceSync)(&q, &as, NVN_SYNC_CONDITION_ALL_GPU_COMMANDS_COMPLETE, 0); + EXPECT("a new queue fence replaces the old acquire dependency", NVN_SYNC_WAIT_RESULT_ALREADY_SIGNALED, P(nvnSyncWait)(&as, 0)); + P(nvnSyncFinalize)(&as); P(nvnSyncInitialize)(&as, &dev); + EXPECT("reinitialized sync has no stale acquire", NVN_SYNC_WAIT_RESULT_TIMEOUT_EXPIRED, P(nvnSyncWait)(&as, 0)); + } unsigned bi = 0; while (strcmp(nvn_names[bi], "nvnCommandBufferBarrier")) bi++; EXPECT("Barrier count = 4 frames × 5", 20, nvn_slot[bi].cnt); diff --git a/dm/gfx/tests/t_present_rescue.m b/dm/gfx/tests/t_present_rescue.m index 0df9fd3302..b8e9305d1f 100644 --- a/dm/gfx/tests/t_present_rescue.m +++ b/dm/gfx/tests/t_present_rescue.m @@ -144,10 +144,24 @@ static void metal_lifecycle(void) { CHECK("present thread can take a real drawable after rescue", drawable != nil); [drawable present]; g_link_started = 0; } +static void acquire_lease(void) { + reset(); g_pq_done = 5; g_present_gpu_ticket[5 % 3] = 5; + CHECK("retired present with GPU-owned source is not acquirable", !mtl_present_ready(5)); + atomic_store(&g_present_discard, 1); + CHECK("discard cannot release a source still read by the GPU", !mtl_present_ready(5)); + atomic_store(&g_present_discard, 0); + CHECK("no present ticket needs no GPU wait", mtl_present_ready(0)); + g_present_gpu_ticket[5 % 3] = 0; + CHECK("GPU completion releases the retired source", mtl_present_ready(5)); + CHECK("a future present remains unavailable", !mtl_present_ready(6)); + g_present_gpu_ticket[5 % 3] = 8; + CHECK("a reused allocator slot does not retain an older source", mtl_present_ready(5)); + g_present_gpu_ticket[5 % 3] = 0; g_pq_done = 0; +} int main(int argc, char **argv) { @autoreleasepool { alarm(10); setbuf(stdout, NULL); if (argc > 2 || (argc == 2 && strcmp(argv[1], "--metal"))) return 2; - thresholds(); backoff(); settings(); lifecycle(); latency(); + thresholds(); backoff(); settings(); lifecycle(); latency(); acquire_lease(); if (argc == 2) metal_lifecycle(); printf("present rescue: %d checks, %d failed\n", checks, failed); return failed != 0; } } diff --git a/dm/gfx/tools/dmsh_pch.py b/dm/gfx/tools/dmsh_pch.py index c18c573868..4d22cec7d3 100644 --- a/dm/gfx/tools/dmsh_pch.py +++ b/dm/gfx/tools/dmsh_pch.py @@ -24,6 +24,9 @@ } ''' _lock, _headers = threading.Lock(), {} +# A translated control-flow tree can exceed clang's default 256 bracket levels (Wonder's agl pfx compute shaders). This only raises a parser acceptance limit; +# it changes neither the source nor code generation, as the byte-identity tests verify, so existing successful compile-cache entries remain valid. +PARSER = ['-fbracket-depth=4096'] def make(metal): @@ -58,7 +61,7 @@ def header(metal): # the path of the convention's header, None when it is not def _run(metal, msl, air, **kw): # The source on standard input, so no file name reaches the compiler: clang names a source's static initializer after its file (_GLOBAL__sub_I_, which any function constant brings), # so the same text under two names gave two libraries, and shaders sharing a source (3,936 of Breath of the Wild's 10,560 pixel shaders, in their _bl) took the bytes of whichever compiled first - with open(msl, 'rb') as f: return subprocess.run([*metal, '-x', 'metal', '-', '-o', air], stdin=f, **kw) + with open(msl, 'rb') as f: return subprocess.run([*metal, *PARSER, '-x', 'metal', '-', '-o', air], stdin=f, **kw) def compile(metal, msl, air): diff --git a/dm/host/hle_audio.c b/dm/host/hle_audio.c index 40a3b7678c..2a561fb187 100644 --- a/dm/host/hle_audio.c +++ b/dm/host/hle_audio.c @@ -602,6 +602,16 @@ static void os_reclaim(ostream *o) { // under o->m: the blocks the output play HLE_FORCE("_ZN2nn5audio29GetDefaultVirtualOutputDeviceEPNS0_15AudioDeviceNameE", "connect to auddmg:u; the host has one output: its name", void, h_os_DefaultDev, char *name) { memset(name, 0, 0x100); strlcpy(name, "AudioVirtualOutput", 0x100); } +// SDK 21 exposes properties for the already-selected virtual output device. +// Wonder's opening movie probes this before opening a stream. Without a hook +// it connects to auddmg:u, which the host does not serve, and aborts (0xa99). +// The SDK's output structure is a 0x100-byte name followed by 12 property bytes. +HLE_FORCE("_ZN2nn5audio32GetVirtualOutputDevicePropertiesEPNS0_23VirtualDevicePropertiesERKNS0_15AudioDeviceNameE", "the host's virtual output properties", Result, h_os_Props, void *props, const char *dev) { + if (!props || !dev || strncmp(dev, "AudioVirtualOutput", 0x100)) return RES_FAIL; + memset(props, 0, 0x10c); + memcpy(props, dev, 0x100); + return RES_OK; +} static Result os_open(void *g, void *ev, const int32_t *fmt, int frames, void *lr) { int f = fmt[0], ch = fmt[1], rate = fmt[2]; if (f != OS_FLOAT && f != OS_PCM16) { LOG("[RED] OutputStream sample format %d: not supported, the open fails", f); return RES_FAIL; } @@ -620,6 +630,13 @@ HLE_FORCE("_ZN2nn5audio16OpenOutputStreamEPNS0_16OutputStreamTypeEPNS_2os11Syste (void)buf; (void)size; (void)dev; (void)threshold; (void)flags; return os_open(g, ev, fmt, frames, __builtin_return_address(0)); } +// SDK 21 also exports the event-taking overload without an OutputStreamFlag. +// Wonder calls this after querying the properties (opening movie startup). +HLE_FORCE("_ZN2nn5audio16OpenOutputStreamEPNS0_16OutputStreamTypeEPNS_2os11SystemEventEPvmRKNS0_15AudioDeviceNameERKNS0_12StreamFormatEii", + "the host's AudioOut (event, no flags)", Result, h_os_OpenEvent, void *g, void *ev, void *buf, size_t size, const char *dev, const int32_t *fmt, int frames, int threshold) { + (void)buf; (void)size; (void)dev; (void)threshold; + return os_open(g, ev, fmt, frames, __builtin_return_address(0)); +} HLE_FORCE("_ZN2nn5audio16OpenOutputStreamEPNS0_16OutputStreamTypeEPvmRKNS0_15AudioDeviceNameERKNS0_12StreamFormatEiNS_4util10BitFlagSetILi32ENS0_16OutputStreamFlagEEE", "connect to auddmg:u (no event: the game polls); an AudioOut of the host's", Result, h_os_OpenNoEv, void *g, void *buf, size_t size, const char *dev, const int32_t *fmt, int frames, uint32_t flags) { (void)buf; (void)size; (void)dev; (void)flags; diff --git a/dm/host/hle_fs.c b/dm/host/hle_fs.c index 7f40189c31..79000e8f2a 100644 --- a/dm/host/hle_fs.c +++ b/dm/host/hle_fs.c @@ -1136,10 +1136,14 @@ HLE_FORCE("_ZN2nn2fs14CloseDirectoryENS0_15DirectoryHandleE", "fsp-srv → close // ---------- Overloads from newer SDKs (SDK 16: MountRom without a cache, ReadFile without the bytes-read output / with read options, directory entry count) ---------- HLE_FORCE("_ZN2nn2fs8MountRomEPKc", "fsp-srv; rom → host RomFS directory (overload without the cache arguments)", Result, h_fs_MountRom1, const char *name) { return h_fs_MountRom(name, NULL, 0); } -// SDK 19 (Metroid Prime 4: Beyond): MountRom and QueryMountRomCacheSize with an int after the first argument (the program index); the host has one RomFS, so it is ignored +// Indexed RomFS: a companion can read another program through DM_ROMFS_PROGRAM_. Without a mapping, index 0 keeps the historical single-program mount. HLE_FORCE("_ZN2nn2fs8MountRomEPKciPvm", "fsp-srv; rom → host RomFS directory (SDK 19 overload with an index)", Result, h_fs_MountRomIdx, const char *name, int idx, void *cache, size_t sz) { - if (idx) LOG("[RED] fs::MountRom(%s, program index %d): the host has one RomFS, this program's", name, idx); - return h_fs_MountRom(name, cache, sz); + char key[64]; snprintf(key, sizeof key, "DM_ROMFS_PROGRAM_%d", idx); + const char *dir = getenv(key); + if (dir && *dir) { dm_thread_service_request(); return mount(name, dir, 1); } + if (!idx) return h_fs_MountRom(name, cache, sz); + LOG("[RED] fs::MountRom(%s, program index %d): %s is unset", name, idx, key); + return RES_TARGET_NOT_FOUND; } HLE_FORCE("_ZN2nn2fs22QueryMountRomCacheSizeEPmi", "fsp-srv; the cache gets only the minimum size (SDK 19 overload with an index)", Result, h_fs_QueryRomCacheIdx, size_t *out, int idx) { (void)idx; return h_fs_QueryRomCache(out); } // ReadFile without the bytes-read output: the read must be complete; a short read (past the end of file) is returned as a failure diff --git a/dm/host/hle_sdk16.c b/dm/host/hle_sdk16.c index 6ad1c76477..d9c526e1b8 100644 --- a/dm/host/hle_sdk16.c +++ b/dm/host/hle_sdk16.c @@ -184,17 +184,22 @@ HLE_FORCE("_ZN2nn2pl13GetSharedFontEPiPNS0_14SharedFontDataEiNS_8settings12Langu } // One built-in font (SDK 21's pl, Switch 2 and Switch 1 overloads; Splatoon Raiders, sdk+0x59a2b0 and sdk+0x59a670): fills {u8, u8, pad, const void *address, u64 size} from the service's reply and returns whether // it is loaded. Unhooked, the SDK connects to pl:u and aborts. The two bytes are the low bits of the reply's first word, which the game does not read (main+0x36c8518). Types 0-5 are the shared font types, by the -// file names main maps to them (main+0x36c8020: nintendo_udsg-r_std, _org_zh-cn, _ext_zh-cn, udjxh-db_zh-tw, udsg-r_ko, nintendo_ext). The address is the same form as GetSharedFontAddress +// file names main maps to them (main+0x36c8020: nintendo_udsg-r_std, _org_zh-cn, _ext_zh-cn, udjxh-db_zh-tw, udsg-r_ko, nintendo_ext). Built-in addresses point to the plain TTF, independent of GetSharedFontAddress's Decode convention typedef struct { uint8_t type, flag, pad[6]; const void *address; uint64_t size; } pl_builtin_font_t; static unsigned pl_builtin_font(pl_builtin_font_t *out, int type) { font_t f = font_get(type); if (!f.p) return 0; - *out = (pl_builtin_font_t){ .type = (uint8_t)type, .address = f.p + font_skip(), .size = f.n - font_skip() }; + // BuiltInFontData always describes a plain TTF, independently of the older shared-font Decode API. + *out = (pl_builtin_font_t){ .address = f.p + 8, .size = f.n - 8 }; return 1; } HLE_FORCE("_ZN2nn2pl18GetBuiltInFontDataEPNS0_17BuiltInFontDataNxENS0_17BuiltInFontTypeNxEb", "one built-in font (Switch 1 type): the stand-in TTFs", unsigned, h_pl_BuiltInNx, pl_builtin_font_t *out, int type, unsigned b) { (void)b; return pl_builtin_font(out, type); } HLE_FORCE("_ZN2nn2pl18GetBuiltInFontDataEPNS0_20BuiltInFontDataOunceENS0_20BuiltInFontTypeOunceEb", "one built-in font (Switch 2 type): the stand-in TTFs", unsigned, h_pl_BuiltInOunce, pl_builtin_font_t *out, int type, unsigned b) { (void)b; return pl_builtin_font(out, type); } -// Whether running on a Switch 2 (codename Ounce) (returns bool): the host reports false, as a Switch 1. Mario 1.2.1 uses it to decide whether to count add-on content (the upgrade pack) and suggest the Switch 2 Edition -HLE_FORCE("_ZN2nn2pl16IsRunningOnOunceEv", "running on Switch 2: no", unsigned, h_pl_IsOunce, void) { return 0; } +// Whether running on a Switch 2 (codename Ounce): follow the loaded SDK, rather than forcing Switch 1 for a Switch 2 companion. +HLE_FORCE("_ZN2nn2pl16IsRunningOnOunceEv", "console generation follows the loaded SDK build spec", unsigned, h_pl_IsOunce, void) { return dm_sdk_ounce(); } +HLE_FORCE("_ZN2nn2pl17RequestLoadNxFontEPvm", "built-in font getters return host-owned font memory directly; prepare it before the first query", void, h_pl_LoadNxFont, void *work, size_t size) { + (void)work; (void)size; + for (int type = 0; type < 6; type++) (void)font_get(type); +} // The Legend of Zelda: Breath of the Wild (SDK 20): asks for the shared fonts to be loaded before querying their state; the host's stand-in fonts are there on demand. VR mode (the Labo VR goggles): the host never enters VR; the query is always false, // and BeginVrMode returns what the SDK's own returns when pctl::CheckStereoVisionPermission refuses (hle_misc.c), 0xd08e (BotW's sdk+0x412970; Super Smash Bros. Ultimate tests it with cbz); EndVrMode is a no-op HLE_FORCE("_ZN2nn2pl21RequestSharedFontLoadENS0_14SharedFontTypeE", "request shared font load; the stand-in font is always available, no-op", void, h_pl_ReqLoad, int type) { (void)type; } diff --git a/dm/host/tests/t_fs.c b/dm/host/tests/t_fs.c index cd17270077..73d9eb8e6b 100644 --- a/dm/host/tests/t_fs.c +++ b/dm/host/tests/t_fs.c @@ -75,6 +75,13 @@ int main(void) { setenv("DM_ROMFS", rom, 1); setenv("DM_ROMFS_OVERLAY", overlay, 1); CHECK(h_fs_MountRom("rom", NULL, 0) == RES_OK, "mount RomFS with two layers (base + overlay)"); size_t rom_cache = 0; CHECK(h_fs_QueryRomCacheIdx(&rom_cache, 0) == RES_OK && rom_cache == 0x1000, "SDK 19: the cache size overload with a program index"); CHECK(h_fs_MountRomIdx("rom2", 0, NULL, 0) == RES_OK && read_fs("rom2:/data/base", got, sizeof got) && !strcmp(got, "only-base"), "SDK 19: MountRom with a program index mounts the RomFS"); + uint64_t ih; + setenv("DM_ROMFS_PROGRAM_0", overlay, 1); + CHECK(h_fs_MountRomIdx("other", 0, NULL, 0) == RES_OK && read_fs("other:/data/over", got, sizeof got) && !strcmp(got, "only-overlay"), "indexed mount selects the mapped program, not the current RomFS"); + CHECK(h_fs_OpenFile(&ih, "other:/data/over", MODE_WRITE) == RES_FAIL, "indexed RomFS is read-only"); + CHECK(h_fs_OpenFile(&ih, "other:/data/base", MODE_READ) != RES_OK, "indexed program does not fall back to the current program"); + CHECK(h_fs_MountRomIdx("missing", 7, NULL, 0) == RES_TARGET_NOT_FOUND, "missing program index fails instead of mounting the wrong program"); + unsetenv("DM_ROMFS_PROGRAM_0"); CHECK(read_fs("rom:/data/same", got, sizeof got) && !strcmp(got, "overlay"), "the overlay file wins"); CHECK(read_fs("rom:/data/base", got, sizeof got) && !strcmp(got, "only-base"), "falls back to the base when the overlay lacks the file"); uint64_t rh; CHECK(h_fs_OpenFile(&rh, "rom:/data/same", MODE_WRITE) == RES_FAIL, "the two-layer mount refuses writes"); CHECK(read_fs("rom:\\data\\over", got, sizeof got) && !strcmp(got, "only-overlay") && read_fs("rom:\\data\\base", got, sizeof got) && !strcmp(got, "only-base"), "backslashes separate as slashes do, in both layers"); diff --git a/dm/host/tests/t_output_stream.c b/dm/host/tests/t_output_stream.c index da6e2ccee2..689b1a93d5 100644 --- a/dm/host/tests/t_output_stream.c +++ b/dm/host/tests/t_output_stream.c @@ -24,8 +24,18 @@ int main(void) { if (!name[0]) FAIL("no default virtual device name"); if (h_os_State(stream) != OS_NONE) FAIL("state of a stream never opened"); int32_t fmt[3] = {OS_FLOAT, 2, 48000}; + // The properties call writes only 0x10c bytes of its output struct. + uint8_t props[0x110]; + memset(props, 0xa5, sizeof props); + if (h_os_Props(props, name) != RES_OK || memcmp(props, name, 0x100) || props[0x10c] != 0xa5) + FAIL("default virtual output device properties/name/size"); + if (h_os_Props(props, "UnrecognizedDevice") == RES_OK) FAIL("unknown virtual output device succeeded"); if (h_os_Open(stream, ev, NULL, 0, name, fmt, CAP, CAP - FRAMES, 0)) FAIL("open"); if (h_os_State(stream) != OS_STOPPED || h_os_Free(stream) != CAP) FAIL("after open: state %d, free %d", h_os_State(stream), h_os_Free(stream)); + char noflags[0x350] = {0}; + if (h_os_OpenEvent(noflags, ev, NULL, 0, name, fmt, CAP, CAP - FRAMES)) FAIL("open event/no-flags"); + if (h_os_State(noflags) != OS_STOPPED || h_os_Free(noflags) != CAP) FAIL("event/no-flags stream state or capacity"); + h_os_Close(noflags); int32_t bad[3] = {OS_FLOAT, 2, 44100}; char other[0x350] = {0}; if (!h_os_Open(other, NULL, NULL, 0, name, bad, CAP, 0, 0) || h_os_State(other) != OS_NONE) FAIL("a 44.1 kHz stream opened"); diff --git a/dm/host/tests/t_shared_font.c b/dm/host/tests/t_shared_font.c index 186ed769fc..85febf1c96 100644 --- a/dm/host/tests/t_shared_font.c +++ b/dm/host/tests/t_shared_font.c @@ -28,6 +28,7 @@ int g_nmod; uint64_t g_t0_ns; uint64_t now_ns(void) { return 0; } int dm_sdk_major(void) { return 20; } +int dm_sdk_ounce(void) { return 1; } int dm_game_imports(const char *prefix) { (void)prefix; return 1; } module_t *module_of(const void *p) { (void)p; return NULL; } void dm_backtrace(uint64_t fp) { (void)fp; } @@ -65,6 +66,19 @@ int main(void) { CHECK(load(&self, name, before[i], 17, sizeof name) == status, "uninitialized fonts preserve result"); CHECK(calls == n + 1 && seen_data == before[i], "uninitialized fonts leave all data unchanged"); } + char path[] = "/tmp/nxnative-font-XXXXXX"; + int fd = mkstemp(path); + CHECK(fd >= 0, "temporary font source created"); + CHECK(write(fd, standard, sizeof standard) == sizeof standard, "font source written"); + close(fd); setenv("DM_SHARED_FONT", path, 1); setenv("DM_SHARED_FONT_EXT", path, 1); + pl_builtin_font_t builtin = {0}; + CHECK(h_pl_BuiltInNx(&builtin, 5, 0) == 1, "Switch 1 built-in font available"); + CHECK(builtin.address == ext_f.p + 8 && builtin.size == sizeof extension, "built-in Nx descriptor excludes Decode header even when main imports Decode"); + CHECK(h_pl_BuiltInOunce(&builtin, 0, 0) == 1, "Switch 2 built-in font available"); + CHECK(builtin.address == std_f.p + 8 && builtin.size == sizeof standard, "built-in Ounce descriptor starts at TTF"); + h_pl_LoadNxFont(NULL, 0); + CHECK(h_pl_IsOunce() == 1, "console query follows SDK generation"); + unlink(path); munmap(std_f.p, std_f.n); munmap(ext_f.p, ext_f.n); std_f = (font_t){standard, sizeof standard}; ext_f = (font_t){extension, sizeof extension}; const void *input[] = {standard, extension, standard + 8, extension + 8, standard + 1, extension + 9, unrelated, NULL}; const void *expect[] = {standard + 8, extension + 8, standard + 8, extension + 8, standard + 1, extension + 9, unrelated, NULL}; diff --git a/dm/mac/launcher.c b/dm/mac/launcher.c index 1268b1837c..cfdb528f60 100644 --- a/dm/mac/launcher.c +++ b/dm/mac/launcher.c @@ -176,7 +176,15 @@ int main(int argc, char **argv) { if (!stat(p, &st)) { romfs_img *image = romfs_image_open(p); valid_romfs = image != NULL; romfs_image_close(image); snprintf(romfs, sizeof romfs, "%s", p); - } else { snprintf(p, sizeof p, "%s/%s", romfs, marker); valid_romfs = !stat(p, &st); } + } else { + snprintf(p, sizeof p, "%s/%s", romfs, marker); valid_romfs = !stat(p, &st); + if (!valid_romfs && !stat(romfs, &st) && S_ISDIR(st.st_mode)) { + for (unsigned i = 0; i < 256 && !valid_romfs; i++) { + snprintf(p, sizeof p, "%s/romfs_programs/%u/%s", res, i, marker); + valid_romfs = !stat(p, &st); + } + } + } if (access(host, X_OK) || !valid_romfs || !modules_present(dm)) { alert(name, "The game data inside this app is missing or incomplete (Contents/Resources/romfs.img, Contents/Resources/dm or Contents/MacOS/dmhost): pack it again with build_ipa_from_nsp.py --platform mac-app."); return 1; @@ -206,6 +214,11 @@ int main(int argc, char **argv) { if (shown == 2) saved_choices(); cloud_saves_folder(); setenv("DM_ROOT", sup, 0); setenv("DM_ROMFS", romfs, 0); setenv("DM_SAVE", save, 0); setenv("DM_TEMP", temp, 0); + for (unsigned i = 0; i < 256; i++) { + char key[64]; snprintf(key, sizeof key, "DM_ROMFS_PROGRAM_%u", i); + snprintf(p, sizeof p, "%s/romfs_programs/%u", res, i); + if (!stat(p, &st) && S_ISDIR(st.st_mode)) setenv(key, p, 0); + } snprintf(p, sizeof p, "%s/msl", res); setenv("DM_MSL_DIR", p, 0); snprintf(p, sizeof p, "%s/font", dm); setenv("DM_FONT_DIR", p, 0); snprintf(p, sizeof p, "%s/shader_map.tsv", dm); if (!stat(p, &st)) setenv("DM_NVN_SHMAP", p, 0); diff --git a/dm/mac/pack_app.py b/dm/mac/pack_app.py index 56f82c5ef5..eca14f2a36 100755 --- a/dm/mac/pack_app.py +++ b/dm/mac/pack_app.py @@ -205,7 +205,8 @@ def selected_inputs(a): for name in ('patched', 'romfs', 'msl', 'tables', 'font', 'aoc', 'pso'): p = getattr(a, name) if p and not (rfs.Path(p).is_dir() if name == 'romfs' else shader_lib.exists(p) if name == 'msl' else Path(p).is_dir()): raise SystemExit(f'no {name} directory at {p}') - if a.marker and not (rfs.Path(a.romfs) / a.marker).exists(): raise SystemExit(f'no RomFS at {a.romfs} (missing {a.marker})') + roots = [a.romfs] + [row.partition('\t')[2] for row in getattr(a, 'romfs_program', None) or []] + if a.marker and not any((rfs.Path(root) / a.marker).exists() for root in roots): raise SystemExit(f'no RomFS at {a.romfs} (missing {a.marker})') mods = sorted({p for pat in MODULE_FILES for p in Path(a.patched).glob(pat) if not detritus(p.name)}) if not any(p.name.endswith('.patched.bin') for p in mods): raise SystemExit(f'no rewritten module in {a.patched}') for p in mods: @@ -272,6 +273,13 @@ def pack(a): if getattr(a, name): setattr(a, name, str(Path(getattr(a, name)).resolve())) mods, msls, pso, tables, pairs, fonts, files = selected_inputs(a) # --program: the other programs of a game of several programs (index, rewritten modules, RomFS, shader libraries, tables, offline pages; tab-separated), which go in beside the first's as one app + indexed_romfs = {} + for row in getattr(a, 'romfs_program', None) or []: + index, path = row.split('\t', 1) + if not index.isdecimal() or not 0 <= int(index) < 256: raise SystemExit(f'invalid RomFS program index: {index}') + index = str(int(index)); path = Path(path).resolve() + if index in indexed_romfs or not path.is_dir(): raise SystemExit(f'duplicate program index or missing RomFS: {row}') + indexed_romfs[index] = path programs = [] for row in a.program or []: n, patched, prom, pmsl, ptab, phtml = row.split('\t') @@ -288,7 +296,8 @@ def pack(a): minos = min_os(a.host) sources = [Path(__file__).resolve(), HERE / 'launcher.c', HERE.parent / 'host/romfs_image.h', HERE.parent / 'host/save_sync.c', HERE.parent / 'host/save_sync.h', HERE.parent / 'tools/romfs.py'] + swift_sources() inputs = {'options': vars(a), 'files': [(str(p), p.stat().st_mode, digest(p), os.readlink(p) if p.is_symlink() else None) for p in files + sources], - 'trees': [(p, inventory(Path(p))) for p in (a.romfs, a.aoc, a.html, *(str(x[1]) for x in programs), *(str(x[3]) for x in programs)) if p and Path(p).exists()], + 'aoc_links': [(p.name, str(p.resolve()), inventory(p.resolve())) for p in Path(a.aoc).iterdir() if p.is_symlink()] if a.aoc else [], + 'trees': [(p, inventory(Path(p))) for p in (a.romfs, a.aoc, a.html, *(str(x[1]) for x in programs), *(str(x[3]) for x in programs), *(str(p) for p in indexed_romfs.values())) if p and Path(p).exists()], 'tools': [[str(c), c.stat().st_mode, c.stat().st_size, c.stat().st_mtime_ns, c.stat().st_ctime_ns, run([c, '--version']).strip()] for c in (clang, swiftc)] + [ str(sdk), run(['xcrun', '--sdk', 'macosx', '--show-sdk-version']).strip(), str(codesign), codesign.stat().st_mode, digest(codesign), minos]} @@ -306,7 +315,8 @@ def pack(a): for d in (mac, dm / 'font'): d.mkdir(parents=True) romfs, msl, aoc = Path(a.romfs), Path(a.msl), Path(a.aoc) if a.aoc else None # each source is probed on its own, as they may lie on different volumes; cp -c copies what it cannot clone, so a probe decides only the free-space check and the step line - trees, lists = {'romfs': romfs, 'aoc': aoc}, {'mods': mods, 'msl': msls, 'pso': pso} + aoc_sources = {p.name: p.resolve() for p in aoc.iterdir()} if aoc else {} + trees, lists = {'romfs': romfs, **{f'aoc/{n}': p for n, p in aoc_sources.items()}, **{f'program{n}': p for n, p in indexed_romfs.items()}}, {'mods': mods, 'msl': msls, 'pso': pso} clone = {k: bool(d) and can_clone(d, res) for k, d in trees.items()} | {k: bool(ps) and can_clone(ps[0], res) for k, ps in lists.items()} if is_chain_library(msl): clone['msl'] = False # copy_msl writes it, which the free-space check below counts need = sum(tree_bytes(d) for k, d in trees.items() if d and not clone[k]) + sum(p.stat().st_size for k, ps in lists.items() if not clone[k] for p in ps) @@ -323,9 +333,14 @@ def pack(a): if a.env_file: shutil.copyfile(a.env_file, dm / 'env.txt') if a.settings: shutil.copyfile(a.settings, dm / 'launcher.json') step('shader libraries'); msl_dst = res / 'msl' / (a.msl_name or msl.name); msl_dst.parent.mkdir(exist_ok=True); copy_msl(msl, msls, msl_dst, clone['msl']) - if aoc: step('add-on content'); copy([aoc], res / 'aoc', clone['aoc']) + if aoc: + step('add-on content'); (res / 'aoc').mkdir() + for n, source in aoc_sources.items(): copy([source], res / 'aoc' / n, clone[f'aoc/{n}']) if pso: step('pipeline archive'); (res / 'pso').mkdir(); copy(pso, res / 'pso', clone['pso']) step('game files'); copy([romfs], res / ('romfs.img' if romfs.is_file() else 'romfs'), clone['romfs']) + for n, source in indexed_romfs.items(): + (res / 'romfs_programs').mkdir(exist_ok=True) + copy([source], res / 'romfs_programs' / n, clone[f'program{n}']) if a.html and Path(a.html).is_dir(): copy([Path(a.html)], res / 'html', False) # the offline pages, beside the RomFS where the host looks for them (hle_web.c) # The other programs: p/ (RomFS, offline pages), msl/p/, dm/p/ (modules, tables), and the table of them all, which the launcher makes DM_PROGRAMS (launcher.c, hle_oe.c): @R@ Resources, @C@ Contents, @S@ the app's Application Support folder if programs: @@ -379,6 +394,7 @@ def parse(argv): ap.add_argument('--pso', help='the pipeline archive made offline for this Mac (pso_offline.py --platform macos output: .mtl4a, prebuilt.tsv)') ap.add_argument('--pso-pairs', help='pso_pairs.tsv for a profile that has none (the shader split\'s copy)') ap.add_argument('--html', help='the offline HTML pages (nx_extract --html), which go beside the RomFS') + ap.add_argument('--romfs-program', action='append', help='read-only RomFS of another program: index and directory, tab-separated') ap.add_argument('--program', action='append', help='another program of a game of several programs: index, rewritten modules, RomFS, Mac shader libraries, tables, offline pages (tab-separated)') ap.add_argument('--env-file', help='environment switches baked in (Resources/dm/env.txt)') ap.add_argument('--settings', help='the game\'s launcher settings (dm/games/settings.py output; Resources/dm/launcher.json, which the Mac front end reads)') diff --git a/dm/tools/tests/test_nsp_dump.py b/dm/tools/tests/test_nsp_dump.py index 25a2068aa0..4ddf7432b3 100644 --- a/dm/tools/tests/test_nsp_dump.py +++ b/dm/tools/tests/test_nsp_dump.py @@ -139,6 +139,56 @@ def test_dlc_images_follow_the_folder(self): self.assertIn('dlcfile', romfs_image.read_index(aoc / '1.img')) +class Companion(Base): + def test_nested_dlc_uses_numeric_newest_complete_snapshot(self): + d = self.dump / 'dlc/040074a01bf13001/0' + # A retained flat/base tree must not hide a newer extracted update. + for v in (9, 100): + p = d / f'updates/{v}/program/0/romfs'; p.mkdir(parents=True) + (p / 'data').write_text(str(v)) + (d / 'updates/200/program/0').mkdir(parents=True) + self.assertEqual(pack_dump.scan(self.dump)[2][TID + 0x1001], d / 'updates/100/program/0/romfs') + + def test_companion_identity_and_cross_platform_dlc_index(self): + root = self.dump / f'updates/{VER}' + primary = TID & 0x00FFFFFFFFFFFFFF | 0x0100000000000000 + for p in (self.dump / 'base', root): (p / 'exefs/main.npdm').write_bytes(npdm(primary)) + version(root / 'program1', tid=TID + 1) + self.assertEqual(pack_dump.default_program(self.dump), 1) + app, vers, dlc = pack_dump.scan(self.dump, 1) + self.assertEqual(app, TID + 1) + self.assertEqual(vers, {VER: root / 'program1'}) + self.assertEqual(pack_dump.dlc_index(app, next(iter(dlc))), 1) + self.assertEqual(pack_dump.scan(self.dump, 0)[0], primary) + app, fam = nsp.scan_dump(self.dump, want=primary, program=1) + self.assertIsNone(fam['base']) + self.assertEqual(fam['shared_romfs'], root / 'romfs') + with self.assertRaisesRegex(pack_dump.Error, 'no complete program 2'): pack_dump.scan(self.dump, 2) + (root / 'program1/exefs/main.npdm').write_bytes(npdm(TID + 3)) + with self.assertRaisesRegex(pack_dump.Error, 'is not program 1'): pack_dump.scan(self.dump, 1) + + def test_primary_update_without_companion_does_not_launch_an_old_companion(self): + root = self.dump / f'updates/{VER}' + version(root / 'program1', tid=TID + 1) + version(self.dump / f'updates/{VER + 1}', tid=TID) + self.assertEqual(pack_dump.default_program(self.dump), 0) + + def test_raw_links_keep_game_files_unpacked(self): + out = self.dump / 'raw' + pack_dump.pack_version(self.dump / f'updates/{VER}', TID, out) + self.assertTrue((out / 'RomFS.img').is_file()) + pack_dump.pack_version(self.dump / f'updates/{VER}', TID, out, raw=True) + self.assertTrue((out / 'RomFS').is_symlink()) + self.assertEqual((out / 'RomFS/dir/new').read_text(), 'data') + self.assertFalse((out / 'RomFS.img').exists()) + _, _, dlc = pack_dump.scan(self.dump) + pack_dump.pack_dlc(TID, dlc, out / 'aoc', raw=True) + self.assertEqual((out / 'aoc/1/dlcfile').read_text(), 'dlc') + self.assertFalse((out / 'aoc/1.img').exists()) + pack_dump.pack_dlc(TID, {}, out / 'aoc', raw=True) + self.assertTrue(next(iter(dlc.values())).is_dir()) + + class Stop(Exception): pass @@ -194,6 +244,13 @@ def test_files_as_the_manifest_pass_leaves_them(self): self.assertEqual(sorted(rows[1:]), sorted([f'/dir/a.bin\t4\t{b"data".hex()}', f'/nro/m.NRO\t{len(nro)}\t{nro[:32].hex()}'])) self.assertEqual((out / 'romfs_nro/nro/m.NRO').read_bytes(), nro) + def test_survey_preserves_supplied_control_metadata(self): + src = self.dump / f'updates/{VER}' + (src / 'control').mkdir(); (src / 'control/control.nacp').write_bytes(b'original control') + out = self.dump.parent / 'control-survey' + nsp.survey_dump(src, out) + self.assertEqual((out / 'control.nacp').read_bytes(), b'original control') + def test_a_version_without_code_or_romfs_stops(self): src = self.dump.parent / 'v'; (src / 'romfs').mkdir(parents=True) with self.assertRaisesRegex(nsp.BuildError, 'no exefs/main.npdm and romfs'): nsp.survey_dump(src, self.dump.parent / 'out') diff --git a/dm/tools/tests/test_nsp_mac_app.py b/dm/tools/tests/test_nsp_mac_app.py index 47ada517ff..5c9a8709cc 100644 --- a/dm/tools/tests/test_nsp_mac_app.py +++ b/dm/tools/tests/test_nsp_mac_app.py @@ -176,6 +176,20 @@ def assert_repacked(self): self.assertTrue(any('--sign' in c for c in self.commands)) self.assertTrue(self.key.is_file()) + def test_indexed_romfs_and_raw_dlc_links_are_portable_and_invalidate(self): + shared = self.tmp / 'primary'; shared.mkdir(); (shared / 'marker').write_text('primary') + self.args.romfs_program = [f'0\t{shared}'] + (self.tmp / 'romfs/marker').unlink() # the game's marker belongs to the indexed primary, not the companion + target = self.tmp / 'dlc'; (self.tmp / 'aoc/1').rename(target) + (self.tmp / 'aoc/1').symlink_to(target, target_is_directory=True) + self.pack(); self.assert_reused() + resources = Path(self.args.out) / 'Contents/Resources' + self.assertEqual((resources / 'romfs_programs/0/marker').read_text(), 'primary') + self.assertFalse((resources / 'aoc/1').is_symlink()) + self.assertEqual((resources / 'aoc/1/data').read_bytes(), b'dlc') + (shared / 'marker').write_text('changed'); self.assert_repacked() + (target / 'data').write_text('updated DLC'); self.assert_repacked() + def test_retargeted_input_symlink_repacks(self): msl = self.tmp / 'msl' for name in ('left.metallib', 'right.metallib'): (msl / name).write_bytes(b'same library') @@ -446,7 +460,7 @@ def setUpClass(cls): cls.args = pack_app.parse([str(cls.app), '--host', str(t / 'dmhost'), '--patched', str(t / 'patched'), '--romfs', str(t / 'romfs'), '--msl', str(t / 'cache/b-0123456789ab'), '--tables', str(t / 'profile'), '--font', str(t / 'font'), '--aoc', str(t / 'aoc'), '--pso', str(t / 'pso'), '--env-file', str(t / 'env.txt'), '--settings', str(t / 'launcher.json'), '--title-id', TID.lower(), '--name', 'Game: Test', '--version', '1.0.1', '--lang', 'fr', '--marker', 'region_common/audio/edward.bfsar', '--icon', str(t / 'icon.jpg'), - '--category', 'public.app-category.games']) + '--category', 'public.app-category.games', '--romfs-program', f'0\t{t / "romfs"}']) with contextlib.redirect_stdout(io.StringIO()): pack_app.pack(cls.args) @classmethod @@ -481,6 +495,15 @@ def test_layout(self): self.assertEqual(sorted(p.name for p in (c / 'Resources/pso').iterdir()), ['applegpu_g14s.mtl4a', 'prebuilt.tsv']) # the pipeline archive and its stamp only self.assertEqual((c / 'Resources/AppIcon.icns').read_bytes()[:4], b'icns') + def test_launcher_accepts_marker_in_indexed_primary(self): + with tempfile.TemporaryDirectory(dir=TEST_ROOT) as folder: + app = Path(folder) / 'Companion.app'; shutil.copytree(self.app, app) + (app / 'Contents/Resources/romfs/region_common/audio/edward.bfsar').unlink() + pack_app.sign(app, '-', shutil.which('codesign')) + p = self.launch(app=app); p.communicate(timeout=30) + self.assertEqual(p.returncode, 7) + self.assertEqual(self.host_run()['DM_ROMFS_PROGRAM_0'], str(app.resolve() / 'Contents/Resources/romfs_programs/0')) + def test_pipeline_pair_tables(self): # the pipeline prebuild reads them beside DM_NVN_SHMAP: without them it builds nothing, though every shader library is there dm = self.app / 'Contents/Resources/dm' self.assertEqual((dm / 'shader_pairs.tsv').read_text(), 'vertex-key\tpixel-key\n') @@ -586,6 +609,7 @@ def test_launch(self): self.assertEqual(p.returncode, 7) # the host's exit code r, res = self.host_run(), self.app / 'Contents/Resources' self.assertEqual(r['argv0'], str(self.app / 'Contents/MacOS/dmhost')); self.assertEqual(r['argv1'], str(res / 'dm')) + self.assertEqual(r['DM_ROMFS_PROGRAM_0'], str(res / 'romfs_programs/0')) self.assertEqual((r['DM_ROMFS'], r['DM_MSL_DIR'], r['DM_NVN_SHMAP'], r['DM_FONT_DIR'], r['DM_AOC_DIR']), (str(res / 'romfs'), str(res / 'msl'), str(res / 'dm/shader_map.tsv'), str(res / 'dm/font'), str(res / 'aoc'))) self.assertEqual(r['DM_PSO_BUNDLE'], str(res / 'pso')) # the pipeline archive made offline, as the iOS app's main.m sets it @@ -698,7 +722,7 @@ def tree_bytes(d): counted.append(d); return 0 with mock.patch.object(pack_app, 'can_clone', can_clone), mock.patch.object(pack_app, 'tree_bytes', tree_bytes), \ mock.patch.object(shutil, 'disk_usage', lambda p: types.SimpleNamespace(free=(1 << 30) + lists - 1)), contextlib.redirect_stdout(io.StringIO()): with self.assertRaisesRegex(SystemExit, 'the app needs'): pack_app.pack(a) - self.assertEqual(len(probed), 5); self.assertEqual(counted, [(self.tmp / 'aoc').resolve()]) + self.assertEqual(len(probed), 6); self.assertEqual(counted, [(self.tmp / 'aoc/1').resolve()]) def test_copies(self): # another volume (no APFS clone): plain copies, the same layout; a new app replaces the old one whole with mock.patch.object(pack_app, 'can_clone', lambda *a: False), mock.patch.object(pack_app, 'sign', lambda *a: None), mock.patch('sys.stdout'): diff --git a/dm/tools/tests/test_shader_pch.py b/dm/tools/tests/test_shader_pch.py index 47e0a21cad..ea188dc646 100644 --- a/dm/tools/tests/test_shader_pch.py +++ b/dm/tools/tests/test_shader_pch.py @@ -53,12 +53,12 @@ def test_a_good_header_is_used_and_the_compile_runs_once(self): with mock.patch.object(dmsh_pch, 'make', return_value=('/p.pch', None)) as make, mock.patch.object(subprocess, 'run', return_value=self.result()) as run: dmsh_pch.compile(['metal', '-c'], 'a.metal', 'a.air'); dmsh_pch.compile(['metal', '-c'], 'b.metal', 'b.air') make.assert_called_once() # one header per convention - self.assertEqual([c.args[0] for c in run.call_args_list], [['metal', '-c', '-include-pch', '/p.pch', '-x', 'metal', '-', '-o', n + '.air'] for n in 'ab']) + self.assertEqual([c.args[0] for c in run.call_args_list], [['metal', '-c', '-include-pch', '/p.pch', *dmsh_pch.PARSER, '-x', 'metal', '-', '-o', n + '.air'] for n in 'ab']) def test_a_failure_with_the_header_runs_the_plain_compile(self): with mock.patch.object(dmsh_pch, 'make', return_value=('/p.pch', None)), mock.patch.object(subprocess, 'run', side_effect=[self.result(1), self.result()]) as run: dmsh_pch.compile(['metal', '-c'], 'a.metal', 'a.air') - self.assertEqual(run.call_args_list[1].args[0], ['metal', '-c', '-x', 'metal', '-', '-o', 'a.air']) + self.assertEqual(run.call_args_list[1].args[0], ['metal', '-c', *dmsh_pch.PARSER, '-x', 'metal', '-', '-o', 'a.air']) self.assertTrue(run.call_args_list[1].kwargs.get('check')) def test_a_header_that_cannot_be_used_is_said_once_and_compiles_stay_plain(self): @@ -72,7 +72,7 @@ def test_dmsh_pch_0_turns_it_off(self): with mock.patch.dict(os.environ, {'DMSH_PCH': '0'}), mock.patch.object(dmsh_pch, 'make') as make, mock.patch.object(subprocess, 'run', return_value=self.result()) as run: dmsh_pch.compile(['metal', '-c'], 'a.metal', 'a.air') make.assert_not_called() - self.assertEqual(run.call_args.args[0], ['metal', '-c', '-x', 'metal', '-', '-o', 'a.air']) + self.assertEqual(run.call_args.args[0], ['metal', '-c', *dmsh_pch.PARSER, '-x', 'metal', '-', '-o', 'a.air']) def test_each_run_reads_the_source_from_its_start(self): read = [] @@ -116,6 +116,24 @@ def test_mac_output_is_byte_identical(self): def test_ios_output_is_byte_identical(self): self.check_identical(ios_metal.METAL) + def check_deep_nesting(self, metal): + src = self.tmp / 'deep.metal' + src.write_text('#include \nusing namespace metal;\nfragment float4 main0(float4 p [[position]]) {' + + ''.join(f'if (p.x > {i}.0f) {{' for i in range(300)) + 'return p;' + '}' * 300 + 'return float4(0);}') + plain = subprocess.run([*metal, '-x', 'metal', str(src), '-o', str(self.tmp / 'plain.air')], capture_output=True) + self.assertNotEqual(plain.returncode, 0) + self.assertIn(b'bracket nesting level exceeded', plain.stderr) + dmsh_pch.compile(metal, str(src), str(self.tmp / 'deep.air')) + self.assertTrue((self.tmp / 'deep.air').is_file()) + + @unittest.skipUnless(has_sdk('macosx'), 'no macOS Metal compiler') + def test_mac_deep_control_flow_compiles(self): + self.check_deep_nesting(MAC) + + @unittest.skipUnless(has_sdk('iphoneos'), 'no iOS Metal compiler') + def test_ios_deep_control_flow_compiles(self): + self.check_deep_nesting(ios_metal.METAL) + def check_no_name(self, metal, header): pch = dmsh_pch.make(metal)[0] if header else None if header: self.assertTrue(pch) diff --git a/dm/tools/tests/test_split_bahs_control.py b/dm/tools/tests/test_split_bahs_control.py new file mode 100644 index 0000000000..eb399150dd --- /dev/null +++ b/dm/tools/tests/test_split_bahs_control.py @@ -0,0 +1,30 @@ +"""BAHS control words that resemble binary descriptors must not become extra shaders. Synthetic NVN2 bytes only.""" +import struct +import unittest + +from dm.tools.tests.test_split_gzip import ss + + +class ControlDescriptors(unittest.TestCase): + def test_control_words_are_not_descriptors_and_shared_controls_still_work(self): + region = 0x400 + data = bytearray(region + 0x200) + struct.pack_into(' adds w0,w8,#0x4000 (ExpHeap size +16 KB - 8) + ### Super Smash Bros. Ultimate [Super Smash Bros. Ultimate](../../dm/games/01006A800016E000/README.md), `01006A800016E000`, [patches.py](../../dm/games/01006A800016E000/patches.py). diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 8563b09f7e..18a891c5f3 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -338,6 +338,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - a Switch 2 game presents at 720p, with its upscaler at the matching low size. → [The display's largest buffer](../reference/services.md#the-displays-largest-buffer) - `registered NROs not found` from `nro_collect.py` in a game whose RomFS has an NRR, or `[RED] ro::LoadModule: no rewritten NRO with SHA-256 …`. `main` is small (about 600 KB of code) beside a data segment of random-looking bytes. → [A game that decodes its code at run time](../reference/services.md#a-game-that-decodes-its-code-at-run-time) - a crash in `ro::RegisterModuleInfo`, or an NRR that seems to register millions of hashes: the words at +0x340 and +0x344 are random. → [NRR version 1](../reference/services.md#nrr-version-1) +- a raw upgrade dump is rejected as missing DLC, or its companion reads its own small RomFS when requesting program 0. Built-in font lookups can receive a BFTTF header instead of a TTF. → [Raw Switch 2 companion fonts and indexed game files](../reference/services.md#raw-switch-2-companion-fonts-and-indexed-game-files) ## Movies @@ -420,6 +421,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - `[RED] sf: the game connects to service auddmg:u itself, which the host does not serve`, then `game called nn::diag abort: Result 0xa99` at `sdk+0x1f61ac` under `audio::ListVirtualInputDevices`. → [No virtual audio input devices (SDK 22)](../reference/audio.md#no-virtual-audio-input-devices-sdk-22) - `game called nn::diag abort: Result 0xe15, call site sdk+0x1f8704`, i.e. inside `nn::audio::CreateAudioOutManagerByHipc+0xb4`, before any audio device opened. → [`OpenAudioOut` without a release event (SDK 22)](../reference/audio.md#openaudioout-without-a-release-event-sdk-22) - `[RED] sf: the game connects to service auddmg:u itself, which the host does not serve`, then `game called nn::diag abort: Result 0xa99` under `nn::audio::GetDefaultVirtualOutputDevice` (`sdk+0x221520`, `audio::detail::GetVirtualOutputDeviceProperties`). → [SDK 21's OutputStream on a virtual output device](../reference/audio.md#sdk-21s-outputstream-on-a-virtual-output-device) +- Super Mario Bros. Wonder Nintendo Switch 2 Edition's Program 1 (v327680) called `audio::GetVirtualOutputDeviceProperties` while opening its online-selection movie and connected to the unsupported `auddmg:u` service, aborting with `0xa99` at `main+0x2298684` (the later stream-open call was at `main+0x2298728`). → [SDK 21 virtual output properties and the event/no-flags overload](../reference/audio.md#sdk-21-virtual-output-properties-and-the-eventno-flags-overload) - an abort in `sm::Initialize` under the hardware Opus decoder's C interface; for the multi-stream one, under `nn::codec::GetHardwareOpusMultiStreamDecoderWorkBufferSize(int, int, int, int, int)` (at `nnMain`, CRI's Opus streams). An abort in `sf::hipc::InitializeHipcServiceResolution` under `codec::GetHardwareOpusDecoderWorkBufferSize(int, int)` (`main+0x31f8e8`, early in `nnMain`). A game that builds `HardwareOpusMultiStreamDecoder` with the imported constructor and then calls it through its vtable. → [Hardware Opus decoders built on the SDK's software decoders](../reference/audio.md#hardware-opus-decoders-built-on-the-sdks-software-decoders) - `game called nn::diag abort: Result 0xe15` under `nn::codec::GetHardwareOpusDecoderManagerWorkBufferSize` (`sdk+0x323180`), the game's sound setup. → [SDK 21's hardware Opus decoder manager](../reference/audio.md#sdk-21s-hardware-opus-decoder-manager) - `CRASH sig=11 addr=0` in `h_codec_HwOpusFini`, the first time the game finalizes a decoder. → [One key for the two forms of the Opus decoder](../reference/audio.md#one-key-for-the-two-forms-of-the-opus-decoder) @@ -466,6 +468,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - a Mac game started with `DM_FULLSCREEN=1` from an app that is itself in fullscreen (the launcher's console mode) came up as a window. → [A game opened from a fullscreen app stays a window](../reference/frame-rate.md#a-game-opened-from-a-fullscreen-app-stays-a-window) - on a ProMotion built-in screen, only in fullscreen, the frame step went 1 → 2 → 3 vsyncs on frames costing 0.1–0.6 vsyncs (`3 late frames; cost 0.32 vsyncs`) and the game presented 15 frames a second; a window, or the screen set to a fixed 60 Hz, did not do it. The `DM_NVN_TSV` rows show presents shown one 120 Hz refresh late (`@pres_at_ms` − `@link_target_ms` = 8.3 ms) with a skipped tick after each (`@link_gap_ms` 33 ms), and `@link_lead_ms` about 16 ms against about 50 ms in a window. → [Frame latency 2 for the display link in fullscreen](../reference/frame-rate.md#frame-latency-2-for-the-display-link-in-fullscreen) - on a MacBook Air M2's fixed 60 Hz built-in display, the reporter measured about 50 ms from the game's present call to on-screen presentation with the display link, at both requested frame latencies 1 and 2, versus 29 ms with `DM_PRESENT_LINK=0`. After loading, all three runs held 60 fps with 0.3% of on-screen intervals outside 16.7 ± 2 ms and no frame-step changes. → [Fixed-rate display-link delay remains at latency 1](../reference/frame-rate.md#fixed-rate-display-link-delay-remains-at-latency-1) +- Wonder's Switch 2 title/menu stays near 30 fps with about 5 ms of GPU work. Its dynamic interval controller allows intervals 1–2 but holds 2. → [Acquire fences keep pacing out of frame-cost samples](../reference/frame-rate.md#acquire-fences-keep-pacing-out-of-frame-cost-samples) - autorelease work takes a measurable share of replay time, with temporary arena-buffer ownership traffic a few times per draw. → [Arena buffer ownership on allocation](../reference/frame-rate.md#arena-buffer-ownership-on-allocation) - a game runs at about 10 fps with little GPU work: the submit thread waits for a command allocator (`mtl_sub_begin` → `mtl_wait_at`), the GPU sits idle between encoders for hundreds of ms (`DM_NVN_GPUPROF`), the Metal 4 `SubmissionQueue` thread spends nearly all its time in `iokit_user_client_trap` (almost all system time), and the `memory up to frame` line shows a residency set of 10,000 objects or more. → [Dedicated c1 constants as slices of a bank](../reference/frame-rate.md#dedicated-c1-constants-as-slices-of-a-bank) - the submit thread waits on the texture / sampler registry lock while a lower-priority registration thread is preempted holding it. → [Registry lock priority inheritance](../reference/frame-rate.md#registry-lock-priority-inheritance) @@ -817,6 +820,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - 12 compute shaders (particles, post effects) translated into an endless loop that hung the GPU at the first gameplay scene; SPIRV-Cross failed on 7 shaders instead of 2 (SPIRV-Cross parse failures). → [Exact program start for the decoder](../reference/shaders.md#exact-program-start-for-the-decoder) - GLSLC outputs fail the reader's check and are dropped (44 of DARK SOULS: REMASTERED's `main`'s 72 outputs and most of its archives' pixel shaders). → [GLSLC outputs read through their section headers](../reference/shaders.md#glslc-outputs-read-through-their-section-headers) - `shader_split.py` finds no stage for vertex shaders and stops at the first captured output (`no GLSLC output`). Translated, every post pass and the window draw one texel's colour (a black screen with sound); every varying at (0, 0, 0, 1). → [VertexA and VertexB parts merged into one vertex shader](../reference/shaders.md#vertexa-and-vertexb-parts-merged-into-one-vertex-shader) +- `metal` reports `bracket nesting level exceeded maximum of 256` on otherwise translated compute shaders. → [Metal's parser rejects deeply nested control flow](../reference/shaders.md#metals-parser-rejects-deeply-nested-control-flow) - a shader build accepts tools that differ from `dmsh_chain.lock`; after an override build, returning to the tested Metal compiler can still restore the override's libraries. → [Tested tools and compiler-specific shader caches](../reference/shaders.md#tested-tools-and-compiler-specific-shader-caches) - after an Xcode update the build stops with `[RED] this Mac's Metal compiler is not the one the shaders were tested with (dm/gfx/tools/dmsh_chain.lock): metal_macosx Apple metal version 32023.922 …`; `xcodebuild -downloadComponent MetalToolchain`, as the message said, downloads the same untested compiler again. → [Metal compiler of a newer Xcode](../reference/shaders.md#metal-compiler-of-a-newer-xcode) - `regress.py` waits for other jobs to end while the only reported jobs are sleeping `MSBuild.dll /nodemode:1 /nodeReuse:true` workers using 0% CPU. → [Idle MSBuild workers block regression measurements](../reference/shaders.md#idle-msbuild-workers-block-regression-measurements) diff --git a/docs/development.md b/docs/development.md index 975a86340d..6a476aa15c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -873,3 +873,10 @@ python3 dm/tools/il2cpp_sym.py "$W/extract/exefs/main" "$W/il2cpp/script.json" - The tool loads NSO/NRO segments with `nso.py`, prints the `ScriptMethod` name plus the offset into it, and annotates direct `b`/`bl` targets with Capstone. Aliases at one address are sorted and shown together. The table has no method sizes and leaves out native code, so the tool bounds each name by the module's `.eh_frame` functions (`nso_code.fde_ranges`; IL2CPP keeps unwind tables for its C++ exceptions, and they cover 98% of BALL x PIT's text): the nearest preceding method is named only when it starts in the function that holds the address. An address in a function no method starts prints `unknown (function 0x…+0x…)`, and one outside every function `unknown`; a module without `.eh_frame` ranges is an error. Use the exact original module/build that produced the table: inputs and `ScriptMethod.Address` must be module-relative offsets, not compressed-file offsets or absolute guest/Ghidra addresses. It rejects out-of-text addresses. For NSOs with separate file and memory offsets, Il2CppDumper's [NSO reader](https://github.com/Perfare/Il2CppDumper/blob/master/Il2CppDumper/ExecutableFormats/NSO.cs) may fail with `Sequence contains no matching element` while mapping MOD0. If its source still has `Position = header.TextSegment.FileOffset + modOffset + 4;`, change that line to `Position = MapVATR(modOffset) + 4;` and rebuild the dumper with the SDK its project targets. The correction is conditional and applies to the external dumper; it is not vendored here. Keep metadata, dumper output and all extracted files in the game's NSP work folder. + + +### Raw companion dumps on the Mac + +`build_ipa_from_nsp.py --platform mac` keeps raw RomFS and DLC directories linked into the build, without creating an NSP or RomFS image. A complete Switch 2 `program1` companion is selected automatically; `--program 0` selects the primary, and `--program N` selects `programN` or `program/N`. Nested DLC snapshots under `dlc//<instance>/{base,updates/<version>}/program/0/romfs` use the newest complete numeric update. These snapshots are complete extracted trees, as with the existing dump format. + +The companion reads its own RomFS normally and the primary through `DM_ROMFS_PROGRAM_0`. Mac apps copy indexed trees into `Contents/Resources/romfs_programs/<index>`; the launcher supplies their paths. Raw companions currently require `--platform mac` or `mac-app`; iOS indexed RomFS bundling is not implemented. Original dump files are retained. diff --git a/docs/reference/audio.md b/docs/reference/audio.md index 7027e3bafc..4bd3a050ba 100644 --- a/docs/reference/audio.md +++ b/docs/reference/audio.md @@ -343,6 +343,13 @@ For a replaced DSP renderer the size query keeps the service's work-buffer contr - **Focus:** Star Fox closes its stream when its window loses focus and opens a new one when it regains it; 14 rounds went by with the sound intact. - **Found on:** Xenoblade Chronicles 2, Nintendo Switch 2 Edition (`0400E95004038001`) (SDK 21.4.4, one or two 2-channel streams at 48 kHz, 10 ms periods, 40 ms capacity; the second when the active device is `AudioExternalOutput`); [Star Fox](../../dm/games/0400E88025B1A000/README.md) (Wwise 2024.1, SDK 21.4.4). +### SDK 21 virtual output properties and the event/no-flags overload + +- **Symptom:** Super Mario Bros. Wonder Nintendo Switch 2 Edition's Program 1 (v327680) called `audio::GetVirtualOutputDeviceProperties` while opening its online-selection movie and connected to the unsupported `auddmg:u` service, aborting with `0xa99` at `main+0x2298684` (the later stream-open call was at `main+0x2298728`). +- **Cause:** The host's SDK 21 `GetDefaultVirtualOutputDevice` and `OpenOutputStream` hooks did not include the separate `GetVirtualOutputDeviceProperties` export or the event-taking `OpenOutputStream` overload without `OutputStreamFlag`; the SDK's default implementation thus called `auddmg:u` itself. +- **Fix:** Answer the properties call for the host's one named virtual output (`AudioVirtualOutput`) in the SDK's 0x10c-byte result structure, with a zeroed 12-byte property tail, and route the alternate event/no-flags stream-open overload through the same AudioOut path as the flags overload. Unknown names fail instead of inventing output devices. `t_output_stream` checks both exports, capacity/state and the exact property write length. +- **Found on:** Super Mario Bros. Wonder Nintendo Switch 2 Edition (the separately proposed Program 1 game profile) (Program 1, SDK 21), opening movie after New Game. + ## Codecs ### Hardware Opus decoders built on the SDK's software decoders diff --git a/docs/reference/frame-rate.md b/docs/reference/frame-rate.md index d644ee7a0a..d56194733e 100644 --- a/docs/reference/frame-rate.md +++ b/docs/reference/frame-rate.md @@ -329,6 +329,14 @@ Frame pacing: the host's frame step, engine frame limiters, present intervals an ## CPU cost and pacing +### Acquire fences keep pacing out of frame-cost samples + +- **Symptom:** Wonder's Switch 2 title/menu stays near 30 fps with about 5 ms of GPU work. Its dynamic interval controller allows intervals 1–2 but holds 2. +- **Cause:** the host slept for presentation inside `nvnWindowAcquireTexture`, before the engine finished its CPU timing sample. The engine then waited on the returned acquire sync after taking that sample. The measured CPU cost included the host's vsync sleep, so the controller kept requesting 30 fps even with spare CPU/GPU capacity. The system-tick frequency was correct; GPU-segment timestamps and a 60 Hz display-link override did not resolve it. +- **Fix:** normal sync-returning acquires publish a pending buffer lease. `nvnSyncWait` consumes it with the caller's timeout, retaining it after a timeout; queue and command-buffer waits resolve it before the inline interpreter uses the texture. A poll requires the slot deadline, retired present ticket and completion of any GPU read of that ticket's source. Atomic ownership allows concurrent waiters. Acquires without a sync, snapshot-ring mode and deterministic test mode retain their synchronous path. Capture layout records the larger sync object; live acquire pointers are not serialized. +- **Check:** `t_nvn.c` covers pending polls, finite expiration, later readiness, two waiters, both GPU-wait APIs, replacement and reinitialization. `t_present_rescue.m` covers a retired ticket whose source is still GPU-owned. A visible-window 20-second title benchmark, warm pipelines and no screenshots within the measurement window, changed from 29.7 fps / 33.34 ms median frame time to 59.9 fps / 16.70 ms, without a frame-rate override. Full DLC gameplay and other games remain unmeasured. +- **Found on:** [Wonder's raw Switch 2 companion](../../dm/games/010015100B514000/README.md#raw-switch-2-companion), main build `ec47e6f8846d2a7b6892e2682845c725`: the CPU timing sample at `main+0x1bbb58`, acquire-fence wait from `main+0x184708`, and frame-cost publication at `main+0x90a20` were checked by disassembly, sampling and a write watch. Logs and benchmarks are in the raw dump's `040015100B514001_build/fps-check/`. + ### Arena buffer ownership on allocation - **Symptom:** autorelease work takes a measurable share of replay time, with temporary arena-buffer ownership traffic a few times per draw. diff --git a/docs/reference/rendering.md b/docs/reference/rendering.md index bb199387f5..ba1a572d1d 100644 --- a/docs/reference/rendering.md +++ b/docs/reference/rendering.md @@ -1226,6 +1226,7 @@ The replay on Metal (`dm/gfx/nvn/nvn_mtl.m`) and the Vulkan and OpenGL layers. E - **Found on:** - [The Legend of Zelda: Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md): TOTK wraps nearly every `ClearColor` / `ClearDepthStencil` this way; each clear is followed by `ReportCounter(TIMESTAMP_TOP, rec)`, the CPU sets `rec` to 1 when a target needs a clear, and `rec + 16` is never written, so after one clear the two are equal and later conditional clears of that target are skipped. The first tile-lighting round, the half-resolution scene depth the light volumes test against (back faces `GREATER` against a buffer just cleared to 1.0 can never pass) and the light accumulation were all cleared before the composite read them. - [Mario vs. Donkey Kong](../../dm/games/0100B99019412000/README.md) (it skips clears under render conditions, as Tears of the Kingdom and Animal Crossing: New Horizons do). + - [Super Mario Bros. Wonder, Switch 2 companion](../../dm/games/010015100B514000/README.md#raw-switch-2-companion): repeated conditional clears of the depth and occlusion targets produce blue flowerbeds and pipes; enabling the companion profile's `render_cond` restores the map lighting. - Drawn unconditionally with no `render_cond`, nothing looked wrong in [Big Brain Academy: Brain vs. Brain](../../dm/games/0100620012D6E000/README.md) (no screen played looked wrong) or [DARK SOULS: REMASTERED](../../dm/games/01004AB00A260000/README.md) (nothing seen). - [Portal 2](../../dm/games/0100ABD01785C000/README.md) needs no `render_cond` and must not get it: NVIDIA's occlusion culling in its D3D9 layer (`IDirect3DDevice9::SetOcclusionShape`, `materialsystem.nro+0xc1130`) puts a mesh under `SetRenderEnableConditional(RENDER_IF_NOT_EQUAL, pair)` when its shape has a query, else `SetRenderEnable(true)`, so drawing everything only costs GPU time. With the key on, those conditions read 0 / 0 at replay (no report in the submission writes the pairs; memory later held `0xffff` / 0) and visible furniture vanished. **Open:** who writes those pairs and when. diff --git a/docs/reference/services.md b/docs/reference/services.md index 648dfce35e..ff90a61799 100644 --- a/docs/reference/services.md +++ b/docs/reference/services.md @@ -1100,3 +1100,11 @@ Most service calls can be found before they crash, before the first boot or the - **Cause:** an NRR with 1 in the byte at +5 (the Switch 2's) has a header 0x180 bytes longer. The same fields follow it: program ID +0x4b0, size +0x4b8, hash table +0x4c0, its count +0x4c4. - **Fix:** `hle_ro.c` and `nro_collect.py` read the table by that byte. - **Found on:** [Mario Tennis Fever](../../dm/games/0400D4201C6EA000/README.md). + + +### Raw Switch 2 companion fonts and indexed game files + +- **Symptom:** a raw upgrade dump is rejected as missing DLC, or its companion reads its own small RomFS when requesting program 0. Built-in font lookups can receive a BFTTF header instead of a TTF. +- **Cause:** the dump scanner understood only flat DLC trees and one program. The indexed MountRom overload ignored its index. BuiltInFontData reused the older shared-font Decode address convention. +- **Fix:** `pack_dump.py` discovers `programN` / `program/N`, and the newest complete numeric DLC snapshot under `<instance>/updates/<version>/program/0/romfs` (falling back to base). Mac development builds link raw directories; Mac apps copy them into the bundle. `DM_ROMFS_PROGRAM_<index>` selects an indexed read-only filesystem, bundled in `Resources/romfs_programs/<index>`. Built-in font getters always return the plain TTF after the 8-byte header; RequestLoadNxFont prepares the host-owned fonts. The console-generation query follows the loaded SDK build spec. +- **Found on:** [Wonder's Switch 2 companion](../../dm/games/010015100B514000/README.md#raw-switch-2-companion). This adds Mac bring-up support; it does not establish iOS or all DLC gameplay compatibility. diff --git a/docs/reference/shaders.md b/docs/reference/shaders.md index f55a94ed00..d85fb3aa57 100644 --- a/docs/reference/shaders.md +++ b/docs/reference/shaders.md @@ -80,7 +80,8 @@ Finding, keying and translating shaders (`dm/gfx/tools/shader_split.py`, `dmsh_b - **Cause:** `bahs_shaders` finds the entries by scanning the table area 4 bytes at a time. An NVN2 control section's +0x38..+0x47 (instruction length, c1 start and end, code length: `L, 0, L, L`) reads as an entry `{code offset L, 0, control size L, control L bytes on}`. When L is a multiple of 0x100 and a later program's control section lies L bytes on, every check passes: the shader pairs that control with whatever code lies at L in the code region. `bahs_variations` finds the binary array at the lowest entry taken minus 0x18, so such an entry below the real ones also hid that file's variation pairs. - **Fix:** an entry that lies inside a control section another entry names, measured by that section's own length (`ctl_len`, 0x48 key bytes for NVN2) rather than the length an entry claims, is dropped. Whether a false entry's section is genuine does not matter, as its first 0x48 bytes are a real control section. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md): six such shaders in `agl_common`, `agl_technique`, `agl_technique_pfx` and `tera_common`, none of them ever loaded; one failed the build. `tera_common` gave 175 variation pairs once its false entry was gone. [Splatoon Raiders](../../dm/games/0400380020056000/README.md)' map holds the same three agl keys (`68bbe98b3bfd9876`, `18be578dfdaaffa0`, `7a4936fccf82094e`); it and Donkey Kong Bananza need their tables split again. [Mario Kart World](../../dm/games/0400C3F00006E000/README.md): one compute shader in `agl_common` (`797e77c2ec4341c5`), dropped from its tables ([issue #1592](https://github.com/seedds/nxnative/issues/1592)). -- **Check:** `dm/tools/tests/test_split_bahs_entries.py` (two programs and the false entry between them, taken on `main`). +- **Also found on:** [Super Mario Bros. Wonder's raw Switch 2 companion](../../dm/games/010015100B514000/README.md#raw-switch-2-companion): `agl_common`, false entry `0x35428` (key `68bbe98b3bfd9876`), one compute shader failing on Ampere instruction `481` at address 0. +- **Check:** `dm/tools/tests/test_split_bahs_entries.py` (two programs and the false entry between them, taken on `main`); `dm/tools/tests/test_split_bahs_control.py` (a false descriptor pointing into another real control, then entries sharing a control section). ### Zstd files recognised by their magic @@ -621,6 +622,13 @@ Finding, keying and translating shaders (`dm/gfx/tools/shader_split.py`, `dmsh_b ## Translation +### Metal's parser rejects deeply nested control flow + +- **Symptom:** `metal` reports `bracket nesting level exceeded maximum of 256` on otherwise translated compute shaders. +- **Cause:** the generated structured control flow exceeds clang's default parser nesting budget. This is a parser limit, not an unsupported GPU instruction. +- **Fix:** the shared Mac/iOS compile runner in `dmsh_pch.py` passes `-fbracket-depth=4096`. It changes only parser acceptance, leaving source and code generation unchanged; successful existing cache entries remain usable. `test_shader_pch.py` checks byte-identical AIR for ordinary shaders and a 300-level shader that fails at the default limit but compiles with the increased budget, on both SDKs. +- **Found on:** [Wonder's raw Switch 2 companion](../../dm/games/010015100B514000/README.md#raw-switch-2-companion), `agl_technique_pfx` compute keys `441c6cb49521c923` and `5bfc92350157b3d0`. + ### Tested tools and compiler-specific shader caches - **Symptom:** a shader build accepts tools that differ from `dmsh_chain.lock`; after an override build, returning to the tested Metal compiler can still restore the override's libraries. diff --git a/scripts/build_ipa_from_nsp.py b/scripts/build_ipa_from_nsp.py index 4ee1706e66..034d082fa2 100755 --- a/scripts/build_ipa_from_nsp.py +++ b/scripts/build_ipa_from_nsp.py @@ -17,7 +17,7 @@ # or in a terminal: python3 scripts/build_ipa_from_nsp.py <NSP folder> -k <prod.keys> [options] (values given on the command line override "Settings", -h lists every option; # without keyboard input a missing value is not asked for but reported as an error; exit code 0 on success, 1 on error, 130 when interrupted). Command-line options entered in Spyder's run configuration work the same way. # Uses only three things: this repository, a folder holding one game's NSPs (the base game, with or without an update; DLC NSPs, when present, are packed too and unlock the add-on content already in the game), and prod.keys. -# Instead of NSPs, the folder may hold the game's files as a dump (base/, updates/<version>/, dlc/; scripts/pack_dump.py lists the layout): that needs no prod.keys. A game with no profile is surveyed from those files. A dump has no control data: the name comes from the profile (a provisional one takes the folder's name), and the app has no icon. +# Instead of NSPs, the folder may hold the game's files as a dump (base/, updates/<version>/, dlc/; scripts/pack_dump.py lists the layout): that needs no prod.keys. A game with no profile is surveyed from those files. Optional control/control.nacp supplies the title and version; without it the name comes from the profile (a provisional one takes the folder's name). # Or the folder may hold the game as one NUP container (a .nup file: base game, updates and DLC; scripts/pack_nup.py), or the .nup file may be given in its place: it needs no prod.keys, like a dump; a game with no profile is surveyed from the container's files (its newest version's code modules and RomFS file headers), # and its RomFS images are written straight from it; the name, display version and icon come from the NACP and icon it carries, else as for a dump (its name record is a packer's label, not the game's). # The newest version in the folder is packed. A version the profile registers (with every version in its shader_versions in the folder) is built from the profile as it is; any other @@ -764,17 +764,20 @@ def package_size(fam, k): # (name, bytes) of a package for the survey report: MANIFEST_HEAD = '# path\tbytes\tfirst 32 bytes (hex)\n' # nx_extract --manifest's header -def survey_dump(src, out): - """A dump's version folder as survey_nup leaves a NUP package's, with no control data (a dump has none). Returns the files listed""" +def survey_dump(src, out, shared=None): + """A dump's version folder as survey_nup leaves a NUP package's, including optional control metadata. Returns the files listed""" if not pack_dump.is_version(src): raise BuildError(f'{src}: no exefs/main.npdm and romfs') out.mkdir(parents=True) pack_dump.exefs(src / 'exefs', out / 'ExeFS') - rows = [] - for p, size, head in pg.romfs_files(src / 'romfs'): + if (src / 'control/control.nacp').is_file(): pack_dump.clone(src / 'control/control.nacp', out / 'control.nacp') + rows, files = [], {} + for root in ([shared] if shared else []) + [src / 'romfs']: + for p, size, head in pg.romfs_files(root): files[p] = (root, size, head) + for p, (root, size, head) in sorted(files.items()): rows.append(f'{p}\t{size}\t{head.hex()}\n') if p.lower().endswith('.nro'): o = out / 'romfs_nro' / p.lstrip('/'); o.parent.mkdir(parents=True, exist_ok=True) - pack_dump.clone(src / 'romfs' / p.lstrip('/'), o) + pack_dump.clone(root / p.lstrip('/'), o) (out / 'romfs_manifest.tsv').write_text(MANIFEST_HEAD + ''.join(rows)) return len(rows) @@ -834,7 +837,7 @@ def listed(): # files in the manifest so far except OSError: return None PROG.stage('survey: RomFS manifest', poll=None if files else listed) say(f'extracting the code modules and a RomFS manifest (no game files written) → {ex}') - if files: say(f' {survey_nup(nup, upd or fam["base"], fam["base"], tmp) if nup else survey_dump(upd or fam["base"], tmp):,} RomFS files listed') + if files: say(f' {survey_nup(nup, upd or fam["base"], fam["base"], tmp) if nup else survey_dump(upd or fam["base"], tmp, fam.get("shared_romfs")):,} RomFS files listed') else: run([nx, keys, fam['base'], upd or '-', tmp, '--manifest'] + extract_flags(app), logs / 'extract.log') swap_in(tmp, ex) PROG.stage('survey: code modules', step='reading') @@ -1330,26 +1333,32 @@ def check_formats(dirs): say(f'[RED] {msg}\npacked as they are (--allow-damaged)') -def scan_dump(dump, want=None): # a dump folder as scan_nsps gives a folder of NSPs, with folders for the packages: (title ID, {'base', 'updates', 'dlc', 'dump': True}) - try: app, vers, dlc = pack_dump.scan(dump) +def scan_dump(dump, want=None, program=0): # a dump folder as scan_nsps gives a folder of NSPs, with folders for the packages: (title ID, {'base', 'updates', 'dlc', 'dump': True}) + try: app, vers, dlc = pack_dump.scan(dump, program) except pack_dump.Error as e: raise BuildError(str(e)) from e - if want is not None and want != app: raise BuildError(f'no game {want:016X} (--title-id / TITLE_ID) in {dump}, which holds {app:016X}') - return app, {'base': pack_dump.base_of(dump), 'updates': sorted(vers.items()), 'dlc': [(tid, None, roms) for tid, roms in dlc.items()], 'dump': True} + if want is not None and want != app and (not program or want != pack_dump.scan(dump)[0]): raise BuildError(f'no game {want:016X} (--title-id / TITLE_ID) in {dump}, which holds {app:016X}') + base = pack_dump.base_of(dump) + selected_base = pack_dump.program_dir(base, program) if base else None + if selected_base and not pack_dump.has_code(selected_base): selected_base = None + selected = max(vers.items(), default=(0, selected_base))[1] if program else None + shared = (selected.parent.parent if selected.parent.name == 'program' else selected.parent) if selected else None + return app, {'base': selected_base, 'updates': sorted(vers.items()), 'dlc': [(tid, None, roms) for tid, roms in dlc.items()], 'dump': True, + 'program': program, 'shared_romfs': shared / 'romfs' if shared else None} -def extract_dump(ex, name, src, app): # one version of a dump folder (src: base/ or updates/<version>/) into the layout extract makes; the code modules and the image are kept while unchanged +def extract_dump(ex, name, src, app, raw=False): # one version of a dump folder (src: base/ or updates/<version>/) into the layout extract makes; the code modules and the image are kept while unchanged dst = ex / name; part = dst / 'RomFS.img.part' def poll(): try: return {'done': part.stat().st_size} except OSError: return None PROG.stage(f'pack {name}', poll=poll, unit='B', total=tree_size(src / 'romfs')[1], est=True) - try: pack_dump.pack_version(src, app, dst, say=lambda s: None) + try: pack_dump.pack_version(src, app, dst, say=lambda s: None, raw=raw) except pack_dump.Error as e: raise BuildError(str(e)) from e - PROG.end(amount((dst / 'RomFS.img').stat().st_size, 'B')) + PROG.end('raw directory' if raw else amount((dst / 'RomFS.img').stat().st_size, 'B')) return dst -def extract_aoc(nx, keys, repo, work, app, fam, logs): +def extract_aoc(nx, keys, repo, work, app, fam, logs, raw=False): """DLC NSPs in the folder → <id>_build/aoc/<index>.img (the RomFS of each one's data NCA; index = title ID − base − 0x1000). The host reports these indices as owned and mounts them (DM_AOC_DIR). Returns None without DLC. Reused when the DLC NSPs (names, sizes and times) and nx_extract's sources are as at the last extraction (Monster Hunter Rise: 250 DLC, 30 s), and aoc/ holds the files that extraction wrote (names, sizes and times): a NUP or a dump folder packs its DLC into the same aoc/""" @@ -1362,7 +1371,7 @@ def extract_aoc(nx, keys, repo, work, app, fam, logs): return work / 'aoc' if fam.get('dump'): PROG.stage('pack DLC', step='packing') - try: pack_dump.pack_dlc(app, {tid: roms for tid, _, roms in fam['dlc']}, work / 'aoc', say=lambda s: None) + try: pack_dump.pack_dlc(app, {tid: roms for tid, _, roms in fam['dlc']}, work / 'aoc', say=lambda s: None, raw=raw) except pack_dump.Error as e: raise BuildError(str(e)) from e PROG.end(f'{len(fam["dlc"])} DLC') return work / 'aoc' @@ -2788,6 +2797,7 @@ def mac_app(py, repo, out, app, work, game, patched, host, font, msl, aoc, gfx, run([py, repo / 'dm/games/settings.py', g.id, patched, control or game / 'control.nacp', settings], logs / 'mac_settings.log', env=dict(env, GFXAPI=gfx, SETTINGS_PLATFORM='mac', DM_LANG=lang)) cmd = [py, repo / 'dm/mac/pack_app.py', out, '--host', host, '--patched', patched, '--romfs', rfs.source(game), '--msl', lib, '--tables', g.table('shader_map.tsv').parent, '--font', font, '--title-id', f'{app:016X}', '--name', name, '--bundle-id', bundle_id, '--lang', lang, '--sign', sign or '-'] + if env.get('DM_ROMFS_PROGRAM_0'): cmd += ['--romfs-program', f'0\t{env["DM_ROMFS_PROGRAM_0"]}'] split_pairs = work / 'shader/pso_pairs.tsv' # the split's copy of pso_pairs.tsv, for a profile without the table (as mac_run's DM_PSO_PAIRS) for opt, v in (('--version', version), ('--category', g.get('app_category')), ('--marker', g.get('romfs_marker')), ('--aoc', aoc), ('--icon', icon), ('--pso', pso), ('--pso-pairs', split_pairs if gfx == 'nvn' and not g.table('pso_pairs.tsv').is_file() and split_pairs.is_file() else None), @@ -2846,7 +2856,7 @@ def parse_args(argv): ap.add_argument('--show-fps', action=argparse.BooleanOptionalAction, help='show the game\'s frame rate on the iPhone, frames on screen per second at the top left (default: the SHOW_FPS setting, on)') ap.add_argument('--motion', action=argparse.BooleanOptionalAction, help='pass a gyro controller\'s motion (on iOS also the phone\'s own) to games that read it; --no-motion builds with DM_MOTION=0, for when motion misbehaves (default: the MOTION setting, on)') ap.add_argument('--survey-only', action='store_true', help='a game with no profile: only the survey, no provisional profile and no IPA') - ap.add_argument('--program', type=int, default=None, help='a multi-program application (Super Mario 3D All-Stars): pack program N of the NSPs as a game of its own, title ID = the application\'s + N (default: the PROGRAM_INDEX setting, 0; a game whose profile lists programs packs them all as one from program 0: this is for bring-up)') + ap.add_argument('--program', type=int, default=None, help='select program N from a multi-program application or raw dump; raw dumps default to a complete Switch 2 companion when present, otherwise program 0') ap.add_argument('--allow-damaged', action='store_true', help='a damaged NSP (files failing the NCA hash check), or files whose bytes do not start as their names promise (a .zs file that is no zstd stream): write them as read and go on instead of stopping') ap.add_argument('--allow-shader-regressions', action='store_true', help='shaders that translated in the game\'s previous build and fail with the current translation chain: ' 'pack the app without them instead of stopping') @@ -2977,14 +2987,19 @@ def main(args=None): py = sys.executable want = args.title_id or TITLE_ID if want and not re.fullmatch(r'[0-9A-Fa-f]{16}', want): raise BuildError(f'--title-id / TITLE_ID takes a title ID, 16 hex digits: {want}') - if PROGRAM_INDEX and dump: raise BuildError('--program / PROGRAM_INDEX picks a program from a multi-program application\'s NSPs; ' + if PROGRAM_INDEX and nup: raise BuildError('--program / PROGRAM_INDEX picks a program from a multi-program application\'s NSPs; ' + ('a NUP' if nup else 'a dump folder') + ' is read as one program\'s files') if nup: app, fam = scan_nup(nup, int(want, 16) if want else None) say(f'a NUP: {nup.name}, {app:016X}, {len(fam["updates"])} update(s), {len(fam["dlc"])} DLC') if fam['base'] is None: say(pack_nup.NO_BASE) elif dump: - app, fam = scan_dump(nsp_dir, int(want, 16) if want else None) + if args.program is None and not PROGRAM_INDEX: PROGRAM_INDEX = pack_dump.default_program(nsp_dir) + app, fam = scan_dump(nsp_dir, int(want, 16) if want else None, PROGRAM_INDEX) + if PROGRAM_INDEX: say(f'raw dump program {PROGRAM_INDEX}: {app:016X}') + if fam.get('shared_romfs'): + if not mac: raise BuildError('raw companions require --platform mac or mac-app; iOS indexed RomFS bundling is not implemented') + env['DM_ROMFS_PROGRAM_0'] = str(fam['shared_romfs']) say(f'a dump folder: {app:016X}, {len(fam["updates"])} update(s), {len(fam["dlc"])} DLC') else: with package_title_reader(repo, keys, env) as from_package: @@ -3060,6 +3075,11 @@ def main(args=None): logs = work / 'logs' check_prereqs(env, xlate, mac) picks = pick_nsps(fam, have, versions) + if fam.get('shared_romfs'): + selected = picks[pkg_ver][1] or picks[pkg_ver][0] + snapshot = selected.parent.parent if selected.parent.name == 'program' else selected.parent + fam['shared_romfs'] = snapshot / 'romfs' + env['DM_ROMFS_PROGRAM_0'] = str(fam['shared_romfs']) ex = work / 'extract'; ex.mkdir(parents=True, exist_ok=True); (work / 'tmp').mkdir(exist_ok=True) gb = (sum(fam['nup'].tree_bytes(pack_nup.romfs_of(u or b, b)) for b, u in picks.values()) if nup else sum(tree_size((u or b) / 'romfs')[1] for b, u in picks.values()) if dump else extract_bytes(picks, False)) / 2**30 + 6 # every merged image is complete, including on APFS; shader cache + slack @@ -3074,7 +3094,7 @@ def main(args=None): dirs = {} for ver in versions: # each merged version streams directly into its own image b, u = picks[ver] - dirs[ver] = extract_nup(ex, ver, u or b, b, app, fam['nup']) if nup else extract_dump(ex, ver, u or b, app) if dump else extract(nx, keys, ex, ver, b, u, None, logs, g, app) + dirs[ver] = extract_nup(ex, ver, u or b, b, app, fam['nup']) if nup else extract_dump(ex, ver, u or b, app, raw=mac) if dump else extract(nx, keys, ex, ver, b, u, None, logs, g, app) if not nup and not dump: extract_html(nx, keys, dirs[ver], b, u, logs, app) check_formats(dirs) check_versions_aarch64(dirs, logs, g, nacp.name(dirs[versions[0]] / 'control.nacp') or f'{app:016X}') # before build_mismatch, derive_profile and build_sources read the modules @@ -3108,8 +3128,9 @@ def main(args=None): for on, k in ((glslc, 'glslc_capture'), (vkcap, 'vk_capture'), (glcap, 'gl_capture'), (uedec, 'ue_decode'), (gfx == 'nvn' and src['oodle_packs'][0], 'oodle_packs'), (gfx == 'nvn' and src['ue_legacy'][0], 'ue_legacy')): if on: say(src[k][1]) PROG.add(glslc + vkcap + glcap + uedec + (gfx == 'nvn')) - aoc = extract_aoc(nx, keys, repo, work, app, fam, logs) + aoc = extract_aoc(nx, keys, repo, work, app, fam, logs, raw=mac) dlc_roms = sorted(p for p in aoc.iterdir() if p.is_dir() or p.suffix == '.img') if aoc else [] + if fam.get('shared_romfs'): dlc_roms.append(fam['shared_romfs']) base = next((dirs[v] for v in versions if picks[v][1] is None), dirs[versions[0]]) # the folder the base game was extracted to alone # the app name and bundle ID both come from the base NSP: the game name in its control data, and a bundle ID from its title ID; a setting / the command line overrides them app_name = args.app_name or APP_NAME or nacp.name(base / 'control.nacp') or (g.get('name') if dump else '') # a dump has no control data, a NUP may have no NACP @@ -3177,6 +3198,11 @@ def derive_shader_tables(reason): # called only after the strict DLC-only or a jit = uses_jit(game / 'ExeFS') if jit and not mac and not mac_pack: capture_jit(py, repo, game, patched, work, env, logs, g) # the iPhone runs no code made at run time: it serves the plugin's output from an image env_file = build_env_file(work, args.env_file or DM_ENV_FILE, motion) + if fam.get('shared_romfs') and not mac_pack: + shared_env = work / 'companion.env' + text = Path(env_file).read_text() if env_file else '' + shared_env.write_text(text + f'\nDM_ROMFS_PROGRAM_0={fam["shared_romfs"]}\n') + env_file = str(shared_env) if not motion: say('motion (gyro): off, DM_MOTION=0 is built in (on the iPhone, DM_MOTION=1 in the app\'s Documents/env.txt turns it back on)') if mac_pack: # a Mac app: the Mac host, then the app packed from it and the build's files if getattr(args, 'in_programs', None) is not None: # a program of the game: the app its first program's build packs takes these diff --git a/scripts/pack_dump.py b/scripts/pack_dump.py index 50225fa737..ea98106e69 100644 --- a/scripts/pack_dump.py +++ b/scripts/pack_dump.py @@ -13,7 +13,7 @@ # extract/v<version>/ExeFS/ the newest version's code modules (APFS clones, or copies on another volume) # extract/v<version>/RomFS.img its RomFS as one image (dm/tools/romfs_image.py: the host's DM_ROMFS, the IPA's romfs.img) # aoc/<index>.img each DLC's RomFS (index = DLC title ID − base title ID − 0x1000), the host's DM_AOC_DIR -# A dump has no control data, so there is no control.nacp or icon.jpg beside them, as there is after nx_extract. +# Optional control/control.nacp is preserved beside the code modules; no icon is synthesized. # The title ID is the program ID in main.npdm. A repeat run keeps code modules that are the same and an image whose source tree, packer and image are unchanged (romfs_image.py --reuse). # Usage: pack_dump.py <dump folder> [--version N] [--work-dir DIR] (exit 1 on error) import argparse, filecmp, shutil, struct, subprocess, sys @@ -78,8 +78,39 @@ def is_dump(folder): # whether a folder has the layout above: a base game's co return any(is_version(p) for p in ups) -def scan(dump): - """What a dump folder holds: (title ID, {update version: its folder}, {DLC title ID: its RomFS folder}). The title ID is the base game's program ID, else the newest whole update's""" +def program_dir(snapshot, index): + if not index: return snapshot + return next((p for p in (snapshot / f'program{index}', snapshot / 'program' / str(index)) if has_code(p)), snapshot / f'program{index}') + + +def default_program(dump): + snapshots = [p for _, p in sorted(updates(dump).items(), reverse=True)] + if (base := base_of(dump)): snapshots.append(base) + for snapshot in snapshots: + if not is_version(snapshot): continue + p = program_dir(snapshot, 1) + return 1 if is_version(p) and program_id(p / 'exefs/main.npdm') >> 56 == 4 else 0 + return 0 + + +def dlc_romfs(content): + candidates = [content / 'romfs'] if (content / 'romfs').is_dir() else [] + for instance in sorted(p for p in content.iterdir() if p.is_dir() and p.name != 'romfs'): + snapshots = [p for _, p in sorted(updates(instance).items(), reverse=True)] + [instance / 'base', instance] + for snapshot in snapshots: + rom = next((p for p in (snapshot / 'program/0/romfs', snapshot / 'program0/romfs', snapshot / 'romfs') if p.is_dir()), None) + if rom: candidates.append(rom); break + if len(candidates) != 1: raise Error(f'{content}: expected one <n>/romfs folder or extracted DLC snapshot, found {len(candidates)}') + return candidates[0] + + +def dlc_index(app, tid): + # The platform byte and program index may differ between a Switch 2 companion and its Switch 1 DLC. + return (tid & 0x00FFFFFFFFFFFFFF) - (app & 0x00FFFFFFFFFFF000) - 0x1000 + + +def scan(dump, program=0): + """What a dump folder holds: (title ID, {update version: its folder}, {DLC title ID: its RomFS folder}). The title ID is the selected program's ID (program 0 by default)""" if not is_dump(dump): raise Error(f'{dump}: no base/exefs/main.npdm with a RomFS in base/ or updates/<version>/, nor exefs/main.npdm and romfs in the folder or in updates/<version>/') vers = updates(dump) app = program_id((base_of(dump) or max((p for p in vers.values() if is_version(p)), key=lambda p: int(p.name))) / 'exefs/main.npdm') @@ -88,26 +119,50 @@ def scan(dump): for d in sorted(p for p in (dump / 'dlc').iterdir() if p.is_dir()) if (dump / 'dlc').is_dir() else []: try: tid = int(d.name, 16) except ValueError: raise Error(f'{d}: not a DLC title ID') from None - if not 0 < tid - app - 0x1000 <= 2000: raise Error(f'{d}: {tid:016X} is not a DLC of {app:016X}') - roms = sorted(d.glob('*/romfs')) or ([d / 'romfs'] if (d / 'romfs').is_dir() else []) - if len(roms) != 1: raise Error(f'{d}: expected one <n>/romfs folder, found {len(roms)}') - dlc[tid] = roms[0] + if not 0 < dlc_index(app, tid) <= 2000: raise Error(f'{d}: {tid:016X} is not a DLC of {app:016X}') + dlc[tid] = dlc_romfs(d) + if program: + base = program_dir(base_of(dump), program) if base_of(dump) else None + vers = {v: program_dir(p, program) for v, p in vers.items() if is_version(program_dir(p, program))} + candidates = [p for _, p in sorted(vers.items(), reverse=True)] + ([base] if base and has_code(base) else []) + if not candidates: raise Error(f'{dump}: no complete program {program}') + pid = program_id(candidates[0] / 'exefs/main.npdm') + if (pid & 0x00FFFFFFFFFFFFFF) != (app & 0x00FFFFFFFFFFFFFF) + program: + raise Error(f'{candidates[0]}: program ID {pid:016X} is not program {program} of {app:016X}') + app = pid return app, vers, dlc -def pack_version(src, app, out, say=print): # one version's folder (base/ or updates/<version>/) → out/ExeFS and out/RomFS.img +def link_directory(src, dst): + if dst.is_symlink() and dst.resolve() == src.resolve(): return + if dst.is_symlink() or dst.is_file(): dst.unlink() + elif dst.exists(): shutil.rmtree(dst) + dst.parent.mkdir(parents=True, exist_ok=True) + dst.symlink_to(src.resolve(), target_is_directory=True) + + +def pack_version(src, app, out, say=print, raw=False): # one version's folder (base/ or updates/<version>/) → out/ExeFS and out/RomFS.img if not (src / 'exefs/main').is_file() or not (src / 'exefs/main.npdm').is_file() or not (src / 'romfs').is_dir(): raise Error(f'{src}: no exefs/main, exefs/main.npdm and romfs') if (pid := program_id(src / 'exefs/main.npdm')) != app: raise Error(f'{src}: program ID {pid:016X}, the base is {app:016X}') exefs(src / 'exefs', out / 'ExeFS') - pack(src / 'romfs', out / 'RomFS.img', say) + if raw: + for name in ('RomFS.img', 'RomFS.img.inputs.json'): (out / name).unlink(missing_ok=True) + link_directory(src / 'romfs', out / 'RomFS') + else: pack(src / 'romfs', out / 'RomFS.img', say) + control = src / 'control/control.nacp' + if control.is_file(): clone(control, out / 'control.nacp') -def pack_dlc(app, dlc, aoc, say=print): # {DLC title ID: its RomFS folder} → aoc/<index>.img; images of DLC no longer in the folder go - want = {f'{tid - app - 0x1000}.img': roms for tid, roms in dlc.items()} +def pack_dlc(app, dlc, aoc, say=print, raw=False): # {DLC title ID: its RomFS folder} → aoc/<index>.img; images of DLC no longer in the folder go + want = {f'{dlc_index(app, tid)}' + ('' if raw else '.img'): roms for tid, roms in dlc.items()} if aoc.is_dir(): for f in aoc.iterdir(): - if f.name.removesuffix('.inputs.json') not in want: f.unlink() - for name, roms in want.items(): pack(roms, aoc / name, say) + if f.name.removesuffix('.inputs.json') not in want: + if f.is_dir() and not f.is_symlink(): shutil.rmtree(f) + else: f.unlink() + for name, roms in want.items(): + if raw: link_directory(roms, aoc / name) + else: pack(roms, aoc / name, say) def main():