From d9247dd3896232765e5bd25cb78c41f4377c7b0f Mon Sep 17 00:00:00 2001 From: seedds Date: Wed, 11 Nov 2026 00:06:13 +0100 Subject: [PATCH] A re-upload of a texture taken from guest memory reads only the images written, and a submission looks at a texture's pages once: the images (a layer's level) with a page written since the last upload are read from guest memory and the others copied on the GPU from the texture it replaces (pkg_written, up_copy_add; DM_NVN_PARTIAL_UPLOAD=0 reads whole), where one written page sent every layer and level through the upload again, at every use. A write another thread makes while the queue's thread replays a submission is found at the next submission's first look (nvn_dirty_touch then, so that generation caches do not keep the texture); writes by the replaying thread (nvn_dirty_own_writes) and looks outside a submission stay exact. Ocarina of Time's loader fills a 1800x1380x8 array of 38.5 MB piece by piece while the game draws with it: on the walk into Hyrule Castle Town 31-52 re-uploads a run, the array 16-17 times whole, frames of 30-64 ms; now 16-17 re-uploads, the array 7-9 times at 4.7-14.2 MB, frames of 25 ms at most (M4 Pro, on base 8db5dd45). A texture holding copied images is read whole once an upload batch was dropped (g_up_lost_gen), and the recovery of a dropped batch is no longer skipped for a texture with edge copies (pkg_edges_clean). The DM_NVN_REUPLOADS line names what was read, the time and the submission; t_partial_upload compares every image with a CPU reference after each write --- dm/games/0400486024DEC000/README.md | 2 + dm/gfx/nvn/nvn_dirty.c | 14 +- dm/gfx/nvn/nvn_dirty.h | 3 + dm/gfx/nvn/nvn_mtl.m | 131 +++++++++++++---- dm/gfx/tests/Makefile | 3 + dm/gfx/tests/t_direct_upload.m | 1 + dm/gfx/tests/t_partial_upload.m | 215 ++++++++++++++++++++++++++++ docs/agents/symptoms.md | 1 + docs/env_switches.tsv | 3 +- docs/reference/frame-rate.md | 13 ++ 10 files changed, 356 insertions(+), 30 deletions(-) create mode 100644 dm/gfx/tests/t_partial_upload.m diff --git a/dm/games/0400486024DEC000/README.md b/dm/games/0400486024DEC000/README.md index 0d219715f3..8d6fb31af1 100644 --- a/dm/games/0400486024DEC000/README.md +++ b/dm/games/0400486024DEC000/README.md @@ -46,6 +46,7 @@ Measured on an M6 Mac with 16 GB, headless, on one walk with camera turns (`walk - In a run that takes shots, each shot costs its frame 60–100 ms, and `DM_NVN_GPUPROF` and pass dumps slow the frames they cover. - Resolved: the castle town market at 9–12 fps on a first visit, with the GPU at 24–26 ms a frame and the main thread busy searching the pipeline archives: [archive searches repeated while the build table is full](../../../docs/reference/frame-rate.md#archive-searches-repeated-while-the-build-table-is-full). Game v131072. On an M5 Mac (macOS 27.0.1, a host near `bd979ced`), CPU translation 113.8 → 6.4 ms a frame and failed searches 9,208 (30.5 s) → 5 per 300 frames, frame p50 107 → 28.5 ms. On an M4 Pro MacBook Pro (24 GB, a host near `1794a35`, real play), failed searches about 49,000 → 5–57 per 300 frames, frame p50 84 → 16–20 ms, main thread 854 → 440–600 ms/s, game time 0.605 → 1.000 of real time. In both, the build table stayed full (384 in flight). Neither is a controlled A/B: the second visits started with the first visit's pipelines prepared (489 more on the M4 Pro), which explains part of the lower GPU time. The drop in failed searches is the change's direct effect. - Resolved: frames of 50–170 ms on entering the castle town, in a second run as in the first, with `new pipelines 1` or `0` and the time under `predictions`: [pulling stand-ins made for predicted keys that are never drawn](../../../docs/reference/frame-rate.md#pulling-stand-ins-made-for-predicted-keys-that-are-never-drawn). Game v131072, M4 Pro, the town entry route (headless, `DM_FPS_HOLD=off`): the three predictions of the route 30.8, 104.4 and 54.5 ms → 0.5, 2.0 and 1.4 ms, pulling pipelines made 244 → 1; frames 910, 1458 and 1787 (48, 132 and 71 ms) are no longer long. The long frames left on the route are texture uploads. +- Resolved: frames of 30–64 ms while walking into the castle town, with `new pipelines 0` and `6 re-uploads, 227.4 MB / 32.0 ms`: the loader fills a 1800x1380x8 R16 array (38.5 MB) and three block-compressed textures over several frames while they are drawn, and each use read the whole texture again, up to 8 times in one submission: [a texture written while it is drawn read whole at every use](../../../docs/reference/frame-rate.md#a-texture-written-while-it-is-drawn-read-whole-at-every-use). Now one look a submission and only the written layers: 25 ms at most there. The single-image textures (3072x2096 BC5, 3056x1152) are still read whole, once a submission (open). - Open: frames of 100–650 ms when an area first loads (640 first uploads, 321 MB, in one frame of 294 ms; another 640 ms of submit with 68 MB). - `DM_NVN_TBDR=1` was tried and does not help: p95 19 ms on the same walk, against 17.8. - The first long run on a new chain is slower (p50 60–114 ms), while `MTLCompilerService` compiles pipelines; the next run is not. @@ -132,6 +133,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25 - [Early fragment tests from the control section](../../../docs/reference/shaders.md#early-fragment-tests-from-the-control-section): 746 of its 22,352 pixel shaders set the NVN2 flag (control +0x5b), none of them writes depth; before, no NVN2 shader got the attribute. - [Archive searches repeated while the build table is full](../../../docs/reference/frame-rate.md#archive-searches-repeated-while-the-build-table-is-full): the castle town market at about 12 fps while the GPU had time to spare. - [Pulling stand-ins made for predicted keys that are never drawn](../../../docs/reference/frame-rate.md#pulling-stand-ins-made-for-predicted-keys-that-are-never-drawn): resolved, frames of 50–170 ms on entering the castle town in every run, with one new pipeline or none. +- [A texture written while it is drawn read whole at every use](../../../docs/reference/frame-rate.md#a-texture-written-while-it-is-drawn-read-whole-at-every-use): resolved, frames of 31–59 ms on entering Hyrule Castle Town, a 1800x1380x8 R16 array (38.5 MB) and three block-compressed textures re-uploaded 31–52 times while the loader wrote them, the array 16–17 times whole; 16–17 re-uploads after, the array's 7–9 at 4.7–14.2 MB, one frame of 25 ms at most. - [One Pro Controller read as two players](../../../docs/reference/services.md#one-pro-controller-read-as-two-players): resolved, no music in Kokiri Forest gameplay and its cutscenes; the controller applet opened over and over and the game kept its music suspended while it had focus. - [A keyboard receiver that declares a gamepad takes player 1](../../../docs/reference/services.md#a-keyboard-receiver-that-declares-a-gamepad-takes-player-1): current until the reporter retests ([issue #1657](https://github.com/seedds/nxnative/issues/1657)). A Keychron Link receiver was player 1 and the Pro Controller player 2, so the game took no input; a raw HID pad is now a player from its first button press. - [Compute bindings prepare texture copies before the dispatch encoder](../../../docs/reference/rendering.md#compute-bindings-prepare-texture-copies-before-the-dispatch-encoder): resolved, `DM_NVN_DISPATCH_OWN=1` crashed in the Metal driver (`AGX::ComputeContext::prepareForEnqueue`) during the Kokiri Forest load ([issue #1743](https://github.com/seedds/nxnative/issues/1743)); on an M2 Pro 5 of 5 runs crashed before the fix and 3 of 3 loaded the forest after it. diff --git a/dm/gfx/nvn/nvn_dirty.c b/dm/gfx/nvn/nvn_dirty.c index 21b5a872e5..de194b5557 100644 --- a/dm/gfx/nvn/nvn_dirty.c +++ b/dm/gfx/nvn/nvn_dirty.c @@ -76,11 +76,17 @@ static void stamp_page(range_t *r, uintptr_t p) { // must invalidate that cached clock, as the page stays writable until the next upload protects it again. atomic_fetch_add(&g_now, 1); } +// Writes by the thread that replays submissions (nvn_dirty_own_thread), by fault or as host writes into a tracked range: they are in order with the commands it replays, where a write by any other thread +// races it (nvn_mtl.m looks at a texture's pages once a submission unless this moved). Not the pages nvn_dirty_close finds changed: another thread or the GPU wrote those +static _Atomic(pthread_t) g_own; static _Atomic uint64_t g_own_writes; +static void own_note(void) { pthread_t o = atomic_load_explicit(&g_own, memory_order_relaxed); if (o && pthread_equal(o, pthread_self())) atomic_fetch_add(&g_own_writes, 1); } +void nvn_dirty_own_thread(void) { atomic_store(&g_own, pthread_self()); } +uint64_t nvn_dirty_own_writes(void) { return atomic_load(&g_own_writes); } static _Atomic uint64_t g_sparse_writes; // writes to pages protected for a placement sparse tile (nvn_dirty_sparse_writes) static void sparse_note(range_t *r, uintptr_t p) { _Atomic uint8_t *sp = atomic_load(&r->sparse); if (sp && atomic_exchange(&sp[p], 0)) atomic_fetch_add(&g_sparse_writes, 1); } static void unprotect_page(range_t *r, uintptr_t p) { // The caller has already changed the state from PROT to BUSY mp(r->base + (p << g_shift), g_pg, PROT_READ | PROT_WRITE); - stamp_page(r, p); sparse_note(r, p); + stamp_page(r, p); sparse_note(r, p); own_note(); atomic_store(&r->state[p], PG_NONE); } // Take a protected or open page for unprotecting (-> BUSY): 1 = taken; 0 = not, with the state found in *st @@ -124,14 +130,15 @@ static void release_pages(range_t *r, uintptr_t p, uintptr_t q, _Atomic uint64_t } } static void prep_write(void *addr, size_t len) { - uintptr_t a = (uintptr_t)addr, e = a + len; int n = atomic_load(&g_nr); + uintptr_t a = (uintptr_t)addr, e = a + len; int n = atomic_load(&g_nr), in = 0; if (!len) return; for (int i = 0; i < n; i++) { range_t *r = rg(i); if (!live_at(i) || e <= r->base || a >= r->end) continue; - uintptr_t lo = a > r->base ? a : r->base, hi = e < r->end ? e : r->end; + uintptr_t lo = a > r->base ? a : r->base, hi = e < r->end ? e : r->end; in = 1; release_pages(r, (lo - r->base) >> g_shift, (hi - 1 - r->base) >> g_shift, &g_preps); } + if (in) own_note(); // whatever its pages' state: one another thread wrote first is writable already } void nvn_dirty_add_range(const void *mem, size_t size) { @@ -296,6 +303,7 @@ void nvn_dirty_protect_sparse(const void *addr, size_t len) { } uint64_t nvn_dirty_sparse_writes(void) { return atomic_load(&g_sparse_writes); } uint64_t nvn_dirty_now(void) { return atomic_load(&g_now); } +void nvn_dirty_touch(void) { atomic_fetch_add(&g_now, 1); } // As nvn_dirty_since, with the caller's last range kept in *span (rechecked, scanned again only if it no longer holds the span; a slot reused by another pool over the same memory is as // good): a packaged texture keeps its handle (the scan over every range ran twice per texture lookup), and a sparse texture's page-table walk (hle_nvn.c vpt_walk) asks once per run of // physical pages, which mostly lie in one pool, where find_span scans every registered range (about 4,300 in a Mario Kart World race) diff --git a/dm/gfx/nvn/nvn_dirty.h b/dm/gfx/nvn/nvn_dirty.h index f14aa29280..2bccb8119f 100644 --- a/dm/gfx/nvn/nvn_dirty.h +++ b/dm/gfx/nvn/nvn_dirty.h @@ -13,6 +13,9 @@ int nvn_dirty_tracked(const void *addr, size_t len); void nvn_dirty_protect(const void *addr, size_t len); // Set the covered pages read-only void nvn_dirty_protect_sparse(const void *addr, size_t len); // The same, for the pages behind a placement sparse tile: a later write to one moves nvn_dirty_sparse_writes uint64_t nvn_dirty_sparse_writes(void); // Writes so far to pages protected that way +void nvn_dirty_own_thread(void); // The calling thread is the one that replays submissions +uint64_t nvn_dirty_own_writes(void); // Writes to protected pages by that thread so far (faults and host writes): the ones in order with what it replays +void nvn_dirty_touch(void); // Moves the generation as a write does, with no page written: whoever remembers "nothing written since" by it asks its pages again uint64_t nvn_dirty_now(void); // Current generation: take the content's stamp before nvn_dirty_protect and before reading it int nvn_dirty_since(const void *addr, size_t len, uint64_t stamp); // Covered pages were written after the stamp -> 1 const void *nvn_dirty_span(const void *addr, size_t len); // The tracked range holding the span (a handle to keep; NULL = untracked) diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 876fad3739..2345365738 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -1396,6 +1396,7 @@ static void up_owner_check(const char *where) { static void treg_finish_submission(void); static uint64_t g_sub_serial; // Submission index (used by the parallel precompile's 'predict only once per submission') +static int g_pkg_kept; // a texture written during this submission kept its texture (tex_mtl_): the next submission starts a write generation, as a write would // The first submission: the prepare step (warm_start), before anything of it is encoded. No GPU work: the submit entry (hle_nvn.c) calls it outside the graphics gate void mtl_prepare(void) { if (!g_warm_started && prepare_on()) { mtl_init(); shmap_load(); } @@ -1403,7 +1404,8 @@ void mtl_prepare(void) { static void sub_cut_begin(void); static void ts_sub_begin(void), ts_boundary(void); static int ts_on(void); static void gmem_commit_patch(void); // the raw pool table words of the arguments encoded so far follow the pools (gmem_slot_note) void mtl_sub_begin(void) { - mtl_init(); g_sub_serial++; g_seg = 0; seg_clk_put(g_sclk); g_sclk = NULL; + mtl_init(); g_sub_serial++; g_seg = 0; + if (g_pkg_kept) { g_pkg_kept = 0; nvn_dirty_touch(); } seg_clk_put(g_sclk); g_sclk = NULL; sub_cut_begin(); defer_drain(); rh_trim(); @@ -1411,6 +1413,7 @@ void mtl_sub_begin(void) { mtl_prepare(); // done by then in the submit entry; the replay tool (nvn_replay) reaches it here warm_claim(); if (g_cb) MDIE("submit re-entered"); + if (!atomic_load(&g_sub_thread_set) || !pthread_equal(g_sub_thread, pthread_self())) nvn_dirty_own_thread(); g_sub_thread = pthread_self(); atomic_store(&g_sub_thread_set, 1); atomic_store(&g_last_commit_error, 0); g_sub_pool = objc_autoreleasePoolPush(); @@ -2764,13 +2767,15 @@ static void tex_put(uint64_t k, id o) { // serials from 2^62 up are external t // Host record of a packaged texture: the current MTLTexture, the upload stamp, and the global write generation at the last check // spare: the texture the last re-upload replaced, kept resident and written by the next re-upload once the GPU has passed spare_fence (a texture the CPU rewrites every frame, a movie frame, // alternates between two textures instead of allocating one per frame: Link's Awakening's title movie made and freed an 8 MB texture a frame) -@interface NvPkg : NSObject { @public id tex, spare, view; uint64_t stamp, seen, hash, spare_fence, vgen, vseen, sub; const void *span; int sparse, vbase; uint8_t *edge[2]; uint32_t edge_n[2], stored; id ps; id eb; } @end // hash: full hash of the uploaded content under DM_NVN_DIRTYCHECK; span: its tracked range (nvn_dirty_span, NULL = untracked); +@interface NvPkg : NSObject { @public id tex, spare, view; uint64_t stamp, seen, hash, spare_fence, vgen, vseen, sub, own, lost_gen; const void *span; int sparse, vbase, part; uint8_t *edge[2]; uint32_t edge_n[2], stored; id ps; id eb; } @end // hash: full hash of the uploaded content under DM_NVN_DIRTYCHECK; span: its tracked range (nvn_dirty_span, NULL = untracked); // sparse: in a sparse virtual pool, read through its page table (hle_nvn.c nvn_sparse_*), vgen the page table's generation at the upload and vseen at the last look; view: what it is bound as (sparse_view) // ps: its placement sparse texture (NvSparse, sparse_make), which tex is: mapped and uploaded tile by tile (sparse_sync) // edge: the uploaded bytes of the storage's first and last host pages where other data shares them (pkg_edges_take); in eb from the first GPU store into its storage on (pkg_edges_gpu), // where the store writes them too // Its release retires its textures (defer_obj_release, not res_remove at once): serials that took its texture over (cpu_unc_drop) keep it resident (defer_drain) // stored: GPU writes stored into its storage since its upload (gsb_swz; the DM_NVN_REUPLOADS line), which DM_NVN_DIRTYCHECK does not count as missed CPU writes +// sub, own: the submission of its last look at its pages and nvn_dirty_own_writes then (a sparse one: sub only, sparse_checked) +// part: tex holds images copied from the texture it replaced (pkg_written), so through it from every upload back to the last whole one, when g_up_lost_gen was lost_gen @implementation NvPkg - (void)dealloc { defer_obj_release(tex); defer_obj_release(spare); if (eb) defer_obj_release(eb); else { free(edge[0]); free(edge[1]); } } @end @@ -2847,6 +2852,10 @@ static void pkg_edges_take(NvPkg *pk, const tex_t *t) { // right after an uplo *e0n = n0; if (n1) { *e1s = r[1][0] - (uintptr_t)t->mem; *e1n = n1; } return pk->eb; } +static int pkg_inner_written(const NvPkg *pk, const tex_t *t) { // a page wholly inside the storage was written since the upload + uintptr_t r[2][2], ia, ie; pkg_edge_ranges(t, r, &ia, &ie); + return ie > ia && nvn_dirty_since_span(pk->span, (const void *)ia, ie - ia, pk->stamp); +} static int pkg_edges_clean(NvPkg *pk, const tex_t *t) { // the covered pages were written since the upload: 1 = only edge pages, and the texture's bytes there are as uploaded if (!pk->edge_n[0] && !pk->edge_n[1]) return 0; uintptr_t r[2][2], ia, ie; pkg_edge_ranges(t, r, &ia, &ie); @@ -3045,6 +3054,7 @@ static void up_wait_zero(_Atomic int *v, int site) { __unsafe_unretained id dst; // Strongly referenced by g_up_texs until the flush is encoded NSUInteger slice, level; MTLOrigin at; // where in the level (a sparse texture's tile, sparse_sync); 0 otherwise + __unsafe_unretained id tsrc; // the image is copied from this texture's same slice and level instead of from src (up_copy_add); kept by g_up_views } up_blit; static up_blit *g_up_blits; static int g_nup_blits, g_up_blitcap; static NSMutableArray *g_up_texs, *g_up_views; // Targets and temporary integer-write views kept through their upload @@ -3061,6 +3071,7 @@ static void up_wait_zero(_Atomic int *v, int site) { // The upload command buffer was discarded by the system (dropped along during GPU error recovery; the first frame on entering Travincal drops one every run): the objects of this batch of textures are built and the timestamps updated, but the content was never written in, // and if the pool memory on the game side no longer changes it will never be re-uploaded. Record this batch of target textures, and the next time any of them is fetched treat it as dirty and re-upload once. // DM_TEST_UP_LOST= (for testing): treat that batch of uploads as dropped, to verify the re-upload path. +static _Atomic uint64_t g_up_lost_gen; // batches dropped so far: a texture holding images copied from its predecessor (NvPkg part) cannot tell whether the drop reached them, and is uploaded whole static NSHashTable *g_up_lost; static os_unfair_lock g_up_lost_lk = OS_UNFAIR_LOCK_INIT; static _Atomic int g_up_lost_n; static _Atomic uint64_t g_up_lost_redo; // Weak-reference table: if a texture in the record is never fetched again and is released normally, this table does not keep it alive static int up_lost_take(id tex) { // This texture's upload was dropped: take it out of the record and return 1 if (!tex) return 0; os_unfair_lock_lock(&g_up_lost_lk); int hit = [g_up_lost containsObject:tex]; if (hit) { [g_up_lost removeObject:tex]; atomic_store(&g_up_lost_n, (int)g_up_lost.allObjects.count); atomic_fetch_add(&g_up_lost_redo, 1); } os_unfair_lock_unlock(&g_up_lost_lk); return hit; @@ -3074,9 +3085,13 @@ static void up_flush(void) { } id b = g_nup_blits ? g_up_ce = [g_up_cb computeCommandEncoder] : nil; if (b) [b barrierAfterQueueStages:MTLStageAll beforeStages:MTLStageAll visibilityOptions:resource_vis()]; + id from = nil; for (int i = 0; b && i < g_nup_blits; i++) { up_blit *x = &g_up_blits[i]; - enc_b_copy_buffer_texture(b, x->src, x->off, x->row, x->image, x->size, x->dst, x->slice, x->level, x->at); + if (!x->tsrc) { enc_b_copy_buffer_texture(b, x->src, x->off, x->row, x->image, x->size, x->dst, x->slice, x->level, x->at); continue; } + // The source may be the target of this encoder's earlier copies (a texture uploaded and replaced within one batch) + if (x->tsrc != from) { from = x->tsrc; [b barrierAfterEncoderStages:MTLStageBlit beforeEncoderStages:MTLStageBlit visibilityOptions:MTL4VisibilityOptionDevice]; } + enc_b_copy_texture(b, x->tsrc, x->slice, x->level, MTLOriginMake(0, 0, 0), x->size, x->dst, x->slice, x->level, MTLOriginMake(0, 0, 0)); } if (b) { m4_end(b); g_up_ce = nil; } id cb = g_up_cb; @@ -3096,6 +3111,7 @@ static void up_flush(void) { [o addFeedbackHandler:^(id f) { if (f.error) { MLOG("[RED] upload command buffer execution error: %s", f.error.description.UTF8String); res_top_once(f.error); gpu_lost_check(f.error, "upload"); } if (f.error || fake_lost) { + atomic_fetch_add(&g_up_lost_gen, 1); os_unfair_lock_lock(&g_up_lost_lk); if (!g_up_lost) g_up_lost = [NSHashTable weakObjectsHashTable]; for (id d in dsts) [g_up_lost addObject:d]; int n = (int)g_up_lost.allObjects.count; atomic_store(&g_up_lost_n, n); os_unfair_lock_unlock(&g_up_lost_lk); pthread_mutex_lock(&g_smu); g_ser_gen++; pthread_mutex_unlock(&g_smu); // Make the view cache of every texture ID take the slow path once (on a cache hit tex_mtl is not passed through and this record is not seen) MLOG("%supload command buffer dropped: the %lu textures of this batch are marked unwritten and re-uploaded on next use (%d awaiting re-upload)", fake_lost ? "DM_TEST_UP_LOST: " : "", (unsigned long)dsts.count, n); @@ -3224,12 +3240,18 @@ static void up_blit_add_at(id src, size_t off, size_t row, size_t ima g_up_blitcap = g_up_blitcap ? g_up_blitcap * 2 : 256; g_up_blits = realloc_or_die(g_up_blits, g_up_blitcap * sizeof *g_up_blits, "the upload blits"); } - g_up_blits[g_nup_blits++] = (up_blit){src, off, row, image, size, dst, slice, level, at}; + g_up_blits[g_nup_blits++] = (up_blit){src, off, row, image, size, dst, slice, level, at, nil}; +} +static void up_copy_add(id src, MTLSize size, id dst, NSUInteger slice, NSUInteger level) { + up_blit_add_at(nil, 0, 0, 0, size, dst, slice, level, MTLOriginMake(0, 0, 0)); + g_up_blits[g_nup_blits - 1].tsrc = src; + if (!g_up_views) g_up_views = [NSMutableArray array]; + if (g_up_views.lastObject != src) [g_up_views addObject:src]; } static void up_blit_add(id src, size_t off, size_t row, size_t image, MTLSize size, id dst, NSUInteger slice, NSUInteger level) { up_blit_add_at(src, off, row, image, size, dst, slice, level, MTLOriginMake(0, 0, 0)); } -static id pkg_upload_(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep); +static id pkg_upload_(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep, id prev, const void *span, uint64_t since, unsigned *taken); static int g_direct_upload_on = -1; static uint64_t g_direct_uploads, g_direct_dispatches; static int direct_upload_on(void) { if (g_direct_upload_on < 0) { const char *e=getenv("DM_NVN_DIRECT_UPLOAD"); g_direct_upload_on=!e||atoi(e); } return g_direct_upload_on; } int nvn_host_readable(uint64_t a, uint64_t n); // hle_nvn.c: guest memory and sparse pools @@ -3238,9 +3260,12 @@ static void up_blit_add(id src, size_t off, size_t row, size_t image, // reuse: a texture of the same description, idle on the GPU, to upload into instead of a new one; keep: usage the texture it replaces was upgraded to (tex_upgrade), which the new one // is made with: a texture drawn into and re-uploaded (a CPU write into one the game renders into) was made without render-target use again and upgraded at its next attachment, a // second texture and a copy of every level for each re-upload, and its spare (of other usage) was never reused -static id pkg_upload(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep) { +// prev: the texture this upload replaces, made of the same storage as it was at stamp `since` (span: its tracked range). Only the images written since are read from guest memory +// (pkg_written); the others are copied from prev on the GPU, *taken of them (0: read whole) +static id pkg_upload(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep, id prev, const void *span, uint64_t since, unsigned *taken) { struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a); - id r = pkg_upload_(t, stamp, reuse, keep); + if (taken) *taken = 0; + id r = pkg_upload_(t, stamp, reuse, keep, prev, span, since, taken); clock_gettime(CLOCK_MONOTONIC, &b); uint64_t d_ = (uint64_t)((b.tv_sec - a.tv_sec) * 1000000000ll + b.tv_nsec - a.tv_nsec); atomic_fetch_add(&g_upload_ns, d_); g_up_total_ns += d_; return r; @@ -3311,7 +3336,27 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r res_add(buf); [g_up_keep addObject:buf]; *off = (uint64_t)addr - a; return buf; } -static id pkg_upload_(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep) { +// A re-upload read every layer and level of a texture from guest memory when one page of it was written. Ocarina of Time fills a 1800x1380x8 RGBA8 array (38.5 MB) layer by layer +// from its CPU while it draws with it: 13 whole uploads in 6 frames, six of them in one frame, for 3 to 936 of its 2,466 pages each (docs/reference/frame-rate.md). A layer's level +// (a 3D texture's level) lies in a byte range of its own (nvn_tex.c), so the images to read are those with a page written since the stamp; pages are 16 KB, so an image that shares +// one with a written neighbour is read too. A byte each (level * layers + layer), *nup of them set; NULL when the layout does not say (read whole) +static int g_partial_upload_on = -1; +static uint8_t *pkg_written(const tex_t *t, const void *span, uint64_t since, unsigned levels, unsigned layers, unsigned *nup) { + if (g_partial_upload_on < 0) { const char *e = getenv("DM_NVN_PARTIAL_UPLOAD"); g_partial_upload_on = !e || atoi(e); } + if (!g_partial_upload_on) return NULL; + nvn_texlayout L; nvn_tex_layout(&t->d, &L); + if (!span || !levels || levels > 16 || levels > t->d.levels || !L.layer_size || L.size != t->L.size || L.layer_size != t->L.layer_size || (uint64_t)layers * L.layer_size != L.size) return NULL; + uint8_t *need = calloc((size_t)levels * layers, 1); if (!need) return NULL; + uint64_t lo[17]; for (unsigned l = 0; l <= levels; l++) lo[l] = l < t->d.levels ? nvn_tex_level_offset(&t->d, l) : L.layer_size; + *nup = 0; + for (unsigned y = 0; y < layers; y++) { + const uint8_t *m = (const uint8_t *)t->mem + (uint64_t)y * L.layer_size; + if (!nvn_dirty_since_span(span, m, L.layer_size, since)) continue; + for (unsigned l = 0; l < levels; l++) if (lo[l + 1] <= lo[l] || nvn_dirty_since_span(span, m + lo[l], lo[l + 1] - lo[l], since)) { need[l * layers + y] = 1; ++*nup; } + } + return need; +} +static id pkg_upload_(const tex_t *t, uint64_t *stamp, id reuse, MTLTextureUsage keep, id prev, const void *span, uint64_t since, unsigned *taken) { MTLTextureDescriptor *d = tex_desc(t, 1); if (!d) return nil; d.usage |= keep & (MTLTextureUsageRenderTarget | MTLTextureUsagePixelFormatView); @@ -3334,10 +3379,18 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r if (sparse) nvn_sparse_protect(t->pool, (uint64_t)(uintptr_t)t->mem, t->L.size); else nvn_dirty_protect(t->mem, t->L.size); } int bw, bh, bpb = nvn_fmt_bpb(t->d.format, &bw, &bh); unsigned layers = (t->d.target == 4 || t->d.target == 9) ? t->d.depth : t->d.target == 8 ? 6 : t->d.target == 3 ? t->d.height : 1; // a cube array's depth counts layer-faces (tex_desc) + // Taken after the stamp and the protect: a write that lands from here on faults and is found at the next look + uint8_t *need = NULL; unsigned nimg = (unsigned)d.mipmapLevelCount * layers, nup = 0; + if (prev && prev != mt && stamp && !sparse && !linear && bpb && d.sampleCount <= 1 && prev.pixelFormat == mt.pixelFormat && prev.textureType == mt.textureType && prev.sampleCount == mt.sampleCount + && prev.width == mt.width && prev.height == mt.height && prev.depth == mt.depth && prev.arrayLength == mt.arrayLength && prev.mipmapLevelCount == mt.mipmapLevelCount) + need = pkg_written(t, span, since, (unsigned)d.mipmapLevelCount, layers, &nup); + if (need && nup == nimg) { free(need); need = NULL; } + if (need && taken) *taken = nimg - nup; size_t total = 0; for (unsigned l = 0; l < d.mipmapLevelCount; l++) { uint32_t w = MAX(t->d.width >> l, 1u), h = MAX(t->d.height >> l, 1u), z = t->d.target == 2 ? MAX(t->d.depth >> l, 1u) : 1; - total += (size_t)((w + bw - 1) / bw) * bpb * ((h + bh - 1) / bh) * z * layers; + unsigned n = layers; if (need) { n = 0; for (unsigned y = 0; y < layers; y++) n += need[l * layers + y]; } + total += (size_t)((w + bw - 1) / bw) * bpb * ((h + bh - 1) / bh) * z * n; } g_bp_up_bytes += total; void *mb; uint64_t mo; @@ -3351,7 +3404,7 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r if (use_gpu) { // The GPU reads the memory pool at execution time, which is closer to a real Switch than a snapshot at CPU upload time; the dirty-page stamp is still taken before encoding, and later writes still trigger a re-upload. up_begin(); - if (direct && !(mo & 7)) { + if (direct && !need && !(mo & 7)) { id view = [mt newTextureViewWithPixelFormat:MTLPixelFormatRGBA16Uint textureType:MTLTextureType2DArray levels:NSMakeRange(0,mt.mipmapLevelCount) slices:NSMakeRange(0,mt.arrayLength)]; if (view) { if (!g_up_views) g_up_views=[NSMutableArray array];[g_up_views addObject:view]; @@ -3375,6 +3428,7 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r uint32_t w = MAX(t->d.width >> l, 1u), h = MAX(t->d.height >> l, 1u), z = t->d.target == 2 ? MAX(t->d.depth >> l, 1u) : 1; size_t pitch = (size_t)((w + bw - 1) / bw) * bpb, rows = (h + bh - 1) / bh; for (unsigned y = 0; y < layers; y++) { + if (need && !need[l * layers + y]) { up_copy_add(prev, MTLSizeMake(w, h, z), mt, y, l); continue; } size_t poff; id params = up_param_alloc(&poff); // Metal 4 has no setBytes: the parameter block of each deswizzle is cut from a shared block (no new GPU buffer each time) if (nvn_deswz_gpu_encode_m4(g_up_ce, g_up_ct, params, poff, (__bridge id)mb, mo, &t->d, l, y, stg, stg_off + off, pitch)) { enc_ok = 0; break; } up_blit_add(stg, stg_off + off, pitch, pitch * rows, MTLSizeMake(w, h, z), mt, y, l); @@ -3384,6 +3438,7 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r if (enc_ok) { [g_up_texs addObject:mt]; if (!g_cb) up_flush(); + free(need); return mt; } g_nup_blits = nb0; // Roll back the copies already recorded for this texture: the already-encoded deswizzle only writes the staging area and is harmless; this texture takes the CPU path below instead @@ -3406,6 +3461,7 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r uint32_t w = MAX(t->d.width >> l, 1u), h = MAX(t->d.height >> l, 1u), z = t->d.target == 2 ? MAX(t->d.depth >> l, 1u) : 1; size_t pitch = (size_t)((w + bw - 1) / bw) * bpb, rows = (h + bh - 1) / bh; for (unsigned y = 0; y < layers; y++) { + if (need && !need[l * layers + y]) { up_copy_add(prev, MTLSizeMake(w, h, z), mt, y, l); continue; } if (linear) { // one 2D level (linear_fit): rows of the storage's stride, the same pitch GetStorageSize counts size_t srow = t->L.layer_size / (t->d.height ? t->d.height : 1); if (l || y || z != 1 || srow < pitch || srow * rows > t->L.size) { RED_ONCE(153, "linear texture upload: %ux%u format %#x target %d levels %u is not one 2D level of rows (stride %zu)", t->d.width, t->d.height, t->d.format, t->d.target, t->d.levels, srow); break; } @@ -3419,7 +3475,7 @@ static void rb_swap_rows(uint8_t *p, size_t wb, size_t rows, size_t nz, size_t r off += pitch * rows * z; } } - free(part); + free(part); free(need); up_flush(); return mt; } @@ -3850,7 +3906,7 @@ static int tex_init_fill(const tex_t *t, id mt) { static int flush_fill = -1; if (flush_fill < 0) { const char *e = getenv("DM_NVN_FLUSH_FILL"); flush_fill = !e || atoi(e) != 0; } if (!no_cpu && !uncached && (!flush_fill || !nvn_flushed(t->mem, t->L.size))) return 0; if (!atomic_load(&g_sub_thread_set) || !on_sub_thread() || !mem_any(t->mem, t->L.size)) return 0; // the upload batch belongs to the submit thread - if (pkg_upload(t, NULL, mt, 0) != mt) return 0; + if (pkg_upload(t, NULL, mt, 0, nil, NULL, 0, NULL) != mt) return 0; int why = no_cpu ? 0 : uncached ? 1 : 2; static uint64_t n[3]; if (n[why]++ < 4) MLOG("initial content: ser=%llu %ux%ux%u format %#x filled from its storage, %s (%llu so far)", (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, (const char *[]){"a CPU_NO_ACCESS pool's", "a CPU_UNCACHED pool's", "flushed by the CPU before its first use"}[why], (unsigned long long)n[why]); @@ -4256,9 +4312,10 @@ static void cpu_unc_drop(void) { } static id tex_mtl_(const tex_t *ct); static id tex_mtl(const tex_t *ct) { AP_B; id r = tex_mtl_(ct); AP_E(4); return r; } -// Diagnostic DM_NVN_REUPLOADS=1: a line for each re-upload of a texture taken from its storage (the first 400): its shape and size, the pages written since its upload and where the -// first of them lies, and the GPU writes stored into its storage since then (gsb_swz) -static void reup_note(const NvPkg *pk, const tex_t *t) { +// Diagnostic DM_NVN_REUPLOADS=1: a line for each re-upload of a texture taken from its storage (the first 400): its shape and size, how much of it was read from guest memory +// (pkg_written), the time it took on the submit thread, the frame and the submission (a frame's replay is many, and each looks at the pages once), the pages written since its upload and where the first of them lies, +// and the GPU writes stored into its storage since then (gsb_swz) +static void reup_note(const NvPkg *pk, const tex_t *t, unsigned taken, uint64_t bytes, uint64_t ns) { static int on = -1, nl; if (on < 0) on = getenv("DM_NVN_REUPLOADS") != NULL; if (!on || nl >= 400) return; nl++; @@ -4268,8 +4325,10 @@ static void reup_note(const NvPkg *pk, const tex_t *t) { while (lv + 1 < t->d.levels && nvn_tex_level_offset(&t->d, lv + 1) <= o) lv++; } char w[96]; if (pk->sparse) snprintf(w, sizeof w, "a sparse pool's"); else if (n == SIZE_MAX) snprintf(w, sizeof w, "untracked"); else snprintf(w, sizeof w, "%zu of %zu, the first in layer %u level %u", n, (size_t)((t->L.size + pg - 1) / pg), layer, lv); - MLOG("re-upload ser=%llu %ux%ux%u format %#x target %d levels %u: %.1f MB, frame %llu; pages written since its upload: %s; GPU writes stored into its storage since then: %u", - (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, t->L.size / 1048576.0, (unsigned long long)atomic_load(&g_cur_frame), w, pk->stored); + char u[64]; if (taken) snprintf(u, sizeof u, "%.1f MB read, %u images copied from the texture it replaces", bytes / 1048576.0, taken); else snprintf(u, sizeof u, "read whole"); + MLOG("re-upload ser=%llu %ux%ux%u format %#x target %d levels %u: %.1f MB, %s, %.2f ms, frame %llu submission %llu; pages written since its upload: %s; GPU writes stored into its storage since then: %u", + (unsigned long long)t->serial, t->d.width, t->d.height, t->d.depth, t->d.format, t->d.target, t->d.levels, t->L.size / 1048576.0, u, ns / 1e6, (unsigned long long)atomic_load(&g_cur_frame), + (unsigned long long)g_sub_serial, w, pk->stored); } // A sparse texture is bound from its first level whose pages, and those of every smaller level, are all mapped: a view of levels [r, n) samples as with a minimum LOD of r. // The translator drops TEX's LOD clamp and residency predicate, so Monster Hunter Rise's streamed textures were sampled at levels it had not mapped yet; those read 0, which ASTC @@ -4411,28 +4470,46 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { // queued it, which present_enqueue resolved there. Its pages are not looked at here, so the submit thread's next look still finds a write since (pk->seen stays) if (atomic_load(&g_sub_thread_set) && !on_sub_thread()) return pk->view ? pk->view : pk->tex; if (gsb_held(t->serial)) return pk->view ? pk->view : pk->tex; // its storage lacks a GPU write this command buffer stores (gsb_store): uploaded at its next use if the CPU wrote it - uint64_t now = nvn_dirty_now(); + uint64_t now = nvn_dirty_now(), own = nvn_dirty_own_writes(); int tracked = pk->span != NULL; // kept from the upload: a texture's pool stays registered while the texture lives - int clean = tracked && (now == pk->seen || !nvn_dirty_since_span(pk->span, t->mem, t->L.size, pk->stamp)); + // Looked at already in this submission: it keeps that look's texture. The queue's thread replays a submission after the game queued it, so the writes the game made + // before are all in the first look, and a write another thread makes during the replay races the GPU's run of that submission on the console: every draw of it reading + // the memory as of one moment after the submit is one of its outcomes. The next submission's first look finds the write (seen stays, and g_pkg_kept sends every cache + // of the write generation through its check again, mtl_sub_begin), so nothing stays stale once the writer is done. Looking at each use fed on itself: a look that + // found a few pages written took a texture and uploaded, the writer wrote a few more meanwhile, and the next draw's look found those (Ocarina of Time, 22 uploads of + // one texture in one submission, docs/reference/frame-rate.md). A write by this thread is in order with the commands it replays and is looked for (own); outside a + // submission (a present) every look reads the pages + int written = !tracked || (now != pk->seen && nvn_dirty_since_span(pk->span, t->mem, t->L.size, pk->stamp)); + int kept = written && tracked && !pk->sparse && atomic_load_explicit(&g_sub_on, memory_order_relaxed) && pk->sub == g_sub_serial && pk->own == own; + int clean = !written || kept; + // Only the pages it shares with other data, its bytes there as uploaded: clean, and no write generation started for a neighbour's writes (pkg_edges_clean protects + // the storage again, so not while a page inside it is being written) + if (kept && (pk->edge_n[0] || pk->edge_n[1]) && !pkg_inner_written(pk, t) && pkg_edges_clean(pk, t)) return pk->view ? pk->view : pk->tex; + if (kept) { now = pk->seen; g_pkg_kept = 1; } else if (!pk->sparse) { pk->sub = g_sub_serial; pk->own = own; } // Checked already in this submission: seen and vseen stay those of that check, so a write or a mapping another thread makes during the replay is found at the next submission's if (pk->sparse && sparse_checked(pk)) { clean = 1; now = pk->seen; } else if (pk->sparse) { uint64_t g = nvn_sparse_gen(); clean = (now == pk->seen && g == pk->vseen) || !nvn_sparse_since(t->pool, (uint64_t)(uintptr_t)t->mem, t->L.size, pk->vgen, pk->stamp); pk->vseen = g; pk->sub = g_sub_serial; } pk->seen = now; - if (clean && atomic_load_explicit(&g_up_lost_n, memory_order_relaxed) && up_lost_take(pk->tex)) { clean = 0; uint64_t k = atomic_load(&g_up_lost_redo); if (k <= 3 || !atomic_load(&g_up_lost_n)) MLOG("re-uploading a texture whose upload was dropped ser=%llu (%llu in total, %d left in the record)", (unsigned long long)t->serial, (unsigned long long)k, atomic_load(&g_up_lost_n)); } + // Images copied from a predecessor are only as good as its upload, and the report of a dropped batch comes after later uploads were encoded + int whole = pk->part && pk->lost_gen != atomic_load(&g_up_lost_gen); + if (whole && clean) { clean = 0; static int nl; if (nl++ < 8) MLOG("re-uploading a texture holding images copied from one uploaded before a batch was dropped ser=%llu", (unsigned long long)t->serial); } + if (!clean && atomic_load_explicit(&g_up_lost_n, memory_order_relaxed) && up_lost_take(pk->tex)) whole = 1; + if (clean && atomic_load_explicit(&g_up_lost_n, memory_order_relaxed) && up_lost_take(pk->tex)) { clean = 0; whole = 1; uint64_t k = atomic_load(&g_up_lost_redo); if (k <= 3 || !atomic_load(&g_up_lost_n)) MLOG("re-uploading a texture whose upload was dropped ser=%llu (%llu in total, %d left in the record)", (unsigned long long)t->serial, (unsigned long long)k, atomic_load(&g_up_lost_n)); } if (clean) { - if (dirtycheck() && !pk->sparse && !pk->stored && full_hash(t->mem, t->L.size) != pk->hash && !g_pkg_miss++) + if (dirtycheck() && !kept && !pk->sparse && !pk->stored && full_hash(t->mem, t->L.size) != pk->hash && !g_pkg_miss++) MLOG("[RED] dirty-page miss: texture ser=%llu pool content changed but was not detected", t->serial); return pk->view ? pk->view : pk->tex; } - if (!pk->sparse && pkg_edges_clean(pk, t)) return pk->view ? pk->view : pk->tex; // only the pages it shares with other data were written + if (!whole && !pk->sparse && pkg_edges_clean(pk, t)) return pk->view ? pk->view : pk->tex; // only the pages it shares with other data were written // Pool content changed (streamed assets written into the same storage, a movie frame): switch to another texture, as in-flight draws still read the old one. That is the spare // when the GPU is done with it (see NvPkg), else a new one; the old one becomes the spare (pkg_upload_ made nt resident) id reuse = nil; if (pk->spare) { pthread_mutex_lock(&g_mu); uint64_t done = g_done; pthread_mutex_unlock(&g_mu); if (pk->spare_fence <= done && pk->spare.usage == pk->tex.usage) reuse = pk->spare; } if (reuse) { pthread_mutex_lock(&g_smu); if (entry_holds((__bridge void *)reuse)) reuse = nil; pthread_mutex_unlock(&g_smu); } // serials that took it over (cpu_unc_drop) still draw it - reup_note(pk, t); - uint64_t st, vg = nvn_sparse_gen(); id nt = pkg_upload(t, &st, reuse, pk->tex.usage); - if (nt) { id old = pk->tex; pk->tex = nt; pk->stored = 0; + uint64_t st, vg = nvn_sparse_gen(), b0 = g_bp_up_bytes, lg = atomic_load(&g_up_lost_gen), t0 = bp_now(); unsigned taken = 0; + id nt = pkg_upload(t, &st, reuse, pk->tex.usage, whole ? nil : pk->tex, pk->span, pk->stamp, &taken); + reup_note(pk, t, taken, g_bp_up_bytes - b0, bp_now() - t0); + if (nt) { id old = pk->tex; pk->tex = nt; pk->stored = 0; pk->part = taken != 0; if (!taken) pk->lost_gen = lg; if (pk->spare && pk->spare != nt) defer_obj_release(pk->spare); // still in use, or of another shape: retired as before if (pk->sparse) sparse_view(pk, t); zb_report(t, pk->sparse ? (unsigned)pk->vbase : 0, "re-upload"); @@ -4441,7 +4518,9 @@ static void sparse_view(NvPkg *pk, const tex_t *t) { } pk = [NvPkg new]; pk->sparse = t->pool && nvn_sparse_pool(t->pool); pk->vgen = pk->vseen = nvn_sparse_gen(); - pk->tex = pkg_upload(t, &pk->stamp, nil, 0); + pk->lost_gen = atomic_load(&g_up_lost_gen); + if (!pk->sparse) { pk->sub = g_sub_serial; pk->own = nvn_dirty_own_writes(); } + pk->tex = pkg_upload(t, &pk->stamp, nil, 0, nil, NULL, 0, NULL); if (!pk->tex) return nil; pk->span = nvn_dirty_span(t->mem, t->L.size); pk->seen = pk->stamp; g_pkg_uploads++; if (dirtycheck() && !pk->sparse) pk->hash = full_hash(t->mem, t->L.size); // "nothing written since" starts at the stamp: a write that faulted during the upload moved "now" past it diff --git a/dm/gfx/tests/Makefile b/dm/gfx/tests/Makefile index 30a80c6f99..10f0b53027 100644 --- a/dm/gfx/tests/Makefile +++ b/dm/gfx/tests/Makefile @@ -471,6 +471,9 @@ check_t_upgrade_alias: RUN_ENV = MTL_DEBUG_LAYER=1 t_alias_takeover: t_alias_takeover.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_alias_takeover.m $(MTL_T_SRC) $(FW) check_t_alias_takeover: RUN_ENV = MTL_DEBUG_LAYER=1 +t_partial_upload: t_partial_upload.m $(MTL_T_DEP) + clang $(CFLAGS) -fobjc-arc -o $@ t_partial_upload.m $(MTL_T_SRC) $(FW) +check_t_partial_upload: RUN_ENV = MTL_DEBUG_LAYER=1 t_srgb_alias: t_srgb_alias.m $(MTL_T_DEP) clang $(CFLAGS) -fobjc-arc -o $@ t_srgb_alias.m $(MTL_T_SRC) $(FW) check_t_srgb_alias: RUN_ENV = MTL_DEBUG_LAYER=1 diff --git a/dm/gfx/tests/t_direct_upload.m b/dm/gfx/tests/t_direct_upload.m index 762858ab21..02b492cb49 100644 --- a/dm/gfx/tests/t_direct_upload.m +++ b/dm/gfx/tests/t_direct_upload.m @@ -30,6 +30,7 @@ static void run(int target,unsigned width,unsigned height,unsigned layers,unsign int fast=direct&&!(source_offset&7)&&nvn_deswz_gpu_direct_supported(&t.d);CHECK("direct upload selected only when eligible",g_direct_uploads-uploads==(uint64_t)fast);CHECK("one dispatch per mip across all faces",g_direct_dispatches-dispatches==(fast?levels:0)); if(!round)first=image;else CHECK("pending/read texture is not overwritten",image!=previous); if(round==2)CHECK("idle spare texture is reused",image==first);previous=image; + up_wait_zero(&g_up_pending,WS_OLDQ); // a dropped batch is reported by its completion handler, after the submission's fence if(atomic_load(&g_up_lost_n)) { uint64_t recovered=atomic_load(&g_up_lost_redo);mtl_sub_begin();image=tex_mtl(&t);finish(); CHECK("direct target participates in dropped-upload recovery",atomic_load(&g_up_lost_redo)==recovered+1);CHECK("dropped upload marker consumed",!atomic_load(&g_up_lost_n));first=image;previous=image; diff --git a/dm/gfx/tests/t_partial_upload.m b/dm/gfx/tests/t_partial_upload.m new file mode 100644 index 0000000000..70acea7031 --- /dev/null +++ b/dm/gfx/tests/t_partial_upload.m @@ -0,0 +1,215 @@ +// Re-uploads of a texture taken from guest memory read only the images (a layer's level, a 3D level) whose pages were written since its upload and copy the others from the +// texture they replace (pkg_written in nvn_mtl.m), through the production lookup (tex_mtl): after each write every image of the texture the host returns is compared with a CPU +// reference, and the bytes read from guest memory with those of the images whose pages the write touched. Ocarina of Time's 1800x1380x8 R16 array, written layer by layer while +// drawn, was read whole 13 times in 6 frames (docs/reference/frame-rate.md) +#include "../nvn/nvn_mtl.m" +void *front_metal_layer(void) { return NULL; } +module_t *module_of(const void *p) { (void)p; return NULL; } +module_t *g_mod; int g_nmod; +void svc_thread_dump(void) {} +uint64_t now_ns(void) { return bp_now(); } +static int checks, failed; +static const char *g_case = ""; +#define CHECK(c, label) do { checks++; if (!(c)) { failed++; fprintf(stderr, "FAIL: %s: %s (line %d)\n", g_case, label, __LINE__); } } while (0) +static void finish(void) { CHECK(mtl_wait(mtl_sub_commit(), 5000000000ull), "submission completes"); CHECK(!mtl_last_commit_error(), "GPU has no submission error"); } + +typedef struct { + tex_t t; pool_t pool; void *mem; size_t poolsize; + unsigned levels, layers, bpb, bw, bh; int d3; + size_t bytes, *at, *len; uint64_t *lo, *hi; // per image (level * layers + layer): its place in the linear reference, and its byte range in the storage + uint8_t *expected, *written; unsigned fill; +} fix_t; +static unsigned img(const fix_t *f, unsigned l, unsigned a) { return l * f->layers + a; } +static void geom(const fix_t *f, unsigned l, unsigned *w, unsigned *h, unsigned *z, size_t *pitch, size_t *rows) { + *w = MAX(f->t.d.width >> l, 1u); *h = f->t.d.target == NVN_TEXTURE_TARGET_1D_ARRAY ? 1 : MAX(f->t.d.height >> l, 1u); *z = f->d3 ? MAX(f->t.d.depth >> l, 1u) : 1; + *pitch = (size_t)((*w + f->bw - 1) / f->bw) * f->bpb; *rows = (*h + f->bh - 1) / f->bh; +} +static fix_t *make(int target, NVNformat format, unsigned w, unsigned h, unsigned depth, unsigned levels, size_t source_offset) { + static uint64_t serial = 97000; + fix_t *f = calloc(1, sizeof *f); int bw, bh; + f->t = (tex_t){.magic = MAGIC('T','E','X',0), .has_layout = 1, .serial = serial++}; + f->t.d = (nvn_texdesc){.target = target, .format = format, .width = w, .height = h, .depth = depth, .levels = levels, .samples = 1}; + nvn_tex_layout(&f->t.d, &f->t.L); + f->bpb = (unsigned)nvn_fmt_bpb(format, &bw, &bh); f->bw = (unsigned)bw; f->bh = (unsigned)bh; + f->d3 = target == NVN_TEXTURE_TARGET_3D; f->levels = levels; + f->layers = target == NVN_TEXTURE_TARGET_2D_ARRAY || target == NVN_TEXTURE_TARGET_CUBEMAP_ARRAY ? depth : target == NVN_TEXTURE_TARGET_CUBEMAP ? 6 : 1; + unsigned n = levels * f->layers; f->at = calloc(n, sizeof *f->at); f->len = calloc(n, sizeof *f->len); f->lo = calloc(n, 8); f->hi = calloc(n, 8); f->written = calloc(n, 1); + for (unsigned l = 0; l < levels; l++) for (unsigned a = 0; a < f->layers; a++) { + unsigned lw, lh, lz; size_t pitch, rows; geom(f, l, &lw, &lh, &lz, &pitch, &rows); + unsigned i = img(f, l, a); f->at[i] = f->bytes; f->len[i] = pitch * rows * lz; f->bytes += f->len[i]; + f->lo[i] = a * f->t.L.layer_size + nvn_tex_level_offset(&f->t.d, l); + f->hi[i] = a * f->t.L.layer_size + (l + 1 < levels ? nvn_tex_level_offset(&f->t.d, l + 1) : f->t.L.layer_size); + } + size_t pg = (size_t)getpagesize(); f->poolsize = (f->t.L.size + source_offset + pg + pg - 1) & ~(pg - 1); + if (posix_memalign(&f->mem, pg, f->poolsize)) abort(); + memset(f->mem, 0, f->poolsize); + f->pool = (pool_t){.magic = MAGIC('P','O','O','L'), .mem = f->mem, .size = f->poolsize, .flags = NVN_MEMORY_POOL_FLAGS_CPU_UNCACHED | NVN_MEMORY_POOL_FLAGS_GPU_CACHED}; + f->t.pool = &f->pool; f->t.mem = (uint8_t *)f->mem + source_offset; + mtl_pool_add(f->mem, f->poolsize); nvn_dirty_add_range(f->mem, f->poolsize); + f->expected = malloc(f->bytes); + return f; +} +static void drop(fix_t *f) { + mtl_tex_release(&f->t); mtl_drain(); mtl_alias_drop_pool(&f->pool); nvn_dirty_remove_range(f->mem); mtl_pool_del(f->mem); mtl_drain(); + free(f->mem); free(f->expected); free(f->at); free(f->len); free(f->lo); free(f->hi); free(f->written); free(f); +} +// The game writes one image: new bytes in the reference, swizzled into the storage, its pages counted as written +static void write_image(fix_t *f, unsigned l, unsigned a) { + unsigned i = img(f, l, a), w, h, z; size_t pitch, rows; geom(f, l, &w, &h, &z, &pitch, &rows); + uint8_t *p = f->expected + f->at[i]; unsigned k = ++f->fill; + for (size_t b = 0; b < f->len[i]; b++) p[b] = (uint8_t)(b * 131u + (b >> 8) * 17u + a * 29u + l * 71u + k * 83u); + dm_guest_write_prepare((uint8_t *)f->t.mem + f->lo[i], f->hi[i] - f->lo[i]); + CHECK(!nvn_swizzle(&f->t.d, l, a, p, (uint32_t)pitch, f->t.mem), "reference swizzle succeeds"); + f->written[i] = 1; +} +typedef struct { fix_t *f; unsigned l, a; } wr_t; +static void *wr_main(void *p) { wr_t *w = p; write_image(w->f, w->l, w->a); return NULL; } +static void elsewhere(fix_t *f, unsigned l, unsigned a) { wr_t w = {f, l, a}; pthread_t th; pthread_create(&th, NULL, wr_main, &w); pthread_join(th, NULL); } +static void *beside_main(void *p) { fix_t *f = p; dm_guest_write_prepare(f->mem, 16); memset(f->mem, 0x5a, 16); return NULL; } +static id by_id(int tid) { os_unfair_lock_lock(&g_rmu); id v = tex_by_id(tid); os_unfair_lock_unlock(&g_rmu); return v; } +// The images a look reads from guest memory after those writes: every one with a byte in a host page a written image has a byte in +static size_t expect_read(fix_t *f, unsigned *count) { + uintptr_t pg = (uintptr_t)getpagesize(), base = (uintptr_t)f->t.mem; size_t n = 0; unsigned c = 0, ni = f->levels * f->layers; + for (unsigned i = 0; i < ni; i++) { int hit = 0; + for (unsigned j = 0; j < ni && !hit; j++) hit = f->written[j] && ((base + f->lo[i]) & ~(pg - 1)) <= ((base + f->hi[j] - 1) & ~(pg - 1)) && ((base + f->lo[j]) & ~(pg - 1)) <= ((base + f->hi[i] - 1) & ~(pg - 1)); + if (hit) { n += f->len[i]; c++; } } + memset(f->written, 0, ni); *count = c; + return n; +} +static int content_ok(fix_t *f, id image) { + id rb = [g_dev newBufferWithLength:f->bytes + 512 options:MTLResourceStorageModeShared]; memset(rb.contents, 0xcd, rb.length); + id cb = q_cb(); id ce = [cb blitCommandEncoder]; + for (unsigned l = 0; l < f->levels; l++) for (unsigned a = 0; a < f->layers; a++) { + unsigned w, h, z; size_t pitch, rows; geom(f, l, &w, &h, &z, &pitch, &rows); + [ce copyFromTexture:image sourceSlice:a sourceLevel:l sourceOrigin:MTLOriginMake(0, 0, 0) sourceSize:MTLSizeMake(w, h, z) toBuffer:rb destinationOffset:256 + f->at[img(f, l, a)] destinationBytesPerRow:pitch destinationBytesPerImage:pitch * rows]; + } + [ce endEncoding]; [cb commit]; [cb waitUntilCompleted]; + if (cb.status != MTLCommandBufferStatusCompleted) return 0; + int sh = rb_packed_shift(f->t.d.format); // Metal's order of a packed 16-bit format, back to the guest's + if (sh) for (unsigned l = 0; l < f->levels; l++) for (unsigned a = 0; a < f->layers; a++) { unsigned w, h, z; size_t pitch, rows; geom(f, l, &w, &h, &z, &pitch, &rows); + rb_swap_rows((uint8_t *)rb.contents + 256 + f->at[img(f, l, a)], pitch, rows, z, pitch, pitch * rows, sh); } + int ok = !memcmp((uint8_t *)rb.contents + 256, f->expected, f->bytes); + if (!ok) for (unsigned i = 0; i < f->levels * f->layers; i++) if (memcmp((uint8_t *)rb.contents + 256 + f->at[i], f->expected + f->at[i], f->len[i])) fprintf(stderr, " %s: level %u layer %u differs\n", g_case, i / f->layers, i % f->layers); + return ok; +} +// One look in a submission of its own. whole: every image is expected read from guest memory +static id look(fix_t *f, int whole, const char *what) { + unsigned n; size_t want = expect_read(f, &n); uint64_t b0 = mtl_upload_bytes(), r0 = g_pkg_reuploads; + mtl_sub_begin(); id image = tex_mtl(&f->t); finish(); + char label[160]; + snprintf(label, sizeof label, "%s: resolves", what); CHECK(image != nil, label); + snprintf(label, sizeof label, "%s: read %llu bytes from guest memory, expected %zu of %zu", what, (unsigned long long)(mtl_upload_bytes() - b0), whole ? f->bytes : want, f->bytes); + CHECK(mtl_upload_bytes() - b0 == (whole ? f->bytes : want), label); + if (!whole) { snprintf(label, sizeof label, "%s: fewer than all images written", what); CHECK(n < f->levels * f->layers, label); } + snprintf(label, sizeof label, "%s: counted as one re-upload", what); CHECK(g_pkg_reuploads == r0 + 1 || (whole && g_pkg_reuploads == r0), label); + snprintf(label, sizeof label, "%s: every image holds the reference", what); CHECK(image && content_ok(f, image), label); + return image; +} +static void run(const char *name, int target, NVNformat format, unsigned w, unsigned h, unsigned depth, unsigned levels, size_t source_offset, int direct) { + g_case = name; g_direct_upload_on = direct; + fix_t *f = make(target, format, w, h, depth, levels, source_offset); + unsigned last = f->layers - 1, mid = f->layers / 2, top = levels - 1; + for (unsigned l = 0; l < levels; l++) for (unsigned a = 0; a < f->layers; a++) write_image(f, l, a); + id first = look(f, 1, "first upload"); + write_image(f, 0, mid); + id second = look(f, 0, "one image written"); + CHECK(second != first, "the texture in-flight draws read is not written"); + CHECK(tex_mtl(&f->t) == second, "unchanged storage keeps its texture"); + // The spare is the first texture: it lacks the image written before this one + write_image(f, top, last); + id third = look(f, 0, "another image written"); + CHECK(third == first, "the idle spare is written"); + if (f->layers > 1) { write_image(f, 0, 0); write_image(f, top, mid); look(f, 0, "two images written"); } + // Two looks in one submission: the second texture's copies read the first's, which the same batch uploads + mtl_sub_begin(); + write_image(f, 0, last); unsigned n; expect_read(f, &n); id a = tex_mtl(&f->t); + write_image(f, top, 0); expect_read(f, &n); id b = tex_mtl(&f->t); + finish(); + CHECK(a && b && a != b, "a look after another write in the same submission takes another texture"); + CHECK(content_ok(f, b), "two re-uploads in one submission: every image holds the reference"); + // A write by another thread while the submit thread replays a submission races it: the submission keeps the texture of its first look, and the next one reads the pages + for (unsigned l = 0; l < levels; l++) for (unsigned y = 0; y < f->layers; y++) write_image(f, l, y); + look(f, 1, "before the racing write"); + mtl_sub_begin(); id kept = tex_mtl(&f->t); uint64_t r0 = g_pkg_reuploads; + elsewhere(f, 0, mid); + CHECK(tex_mtl(&f->t) == kept && g_pkg_reuploads == r0, "a write by another thread during a submission leaves the submission its texture"); + write_image(f, top, last); + expect_read(f, &n); id own = tex_mtl(&f->t); + CHECK(own != kept && g_pkg_reuploads == r0 + 1, "a write by the submit thread is looked for at the next use"); + finish(); + CHECK(content_ok(f, own), "and that look reads the other thread's write too"); + mtl_sub_begin(); kept = tex_mtl(&f->t); elsewhere(f, 0, 0); CHECK(tex_mtl(&f->t) == kept, "the submission keeps its texture again"); finish(); + look(f, 0, "the next submission after a racing write"); + elsewhere(f, top, mid); r0 = g_pkg_reuploads; expect_read(f, &n); + id out = tex_mtl(&f->t); + CHECK(g_pkg_reuploads == r0 + 1 && content_ok(f, out), "outside a submission every look reads the pages"); + // A racing write to the data beside it, in a host page they share, is no write to it: the texture stays, and the next submission starts no write generation for it + if ((uint8_t *)f->t.mem != (uint8_t *)f->mem) { + mtl_sub_begin(); kept = tex_mtl(&f->t); r0 = g_pkg_reuploads; uint64_t skips = g_pkg_edge_skips; + pthread_t th; pthread_create(&th, NULL, beside_main, f); pthread_join(th, NULL); + CHECK(tex_mtl(&f->t) == kept && g_pkg_reuploads == r0 && g_pkg_edge_skips == skips + 1 && !g_pkg_kept, "a racing write beside the texture leaves it clean"); + finish(); + } + // The same through a texture ID, whose view is remembered by write generation: no page anywhere is written after the racing write, and the next submission reads it all the same + view_t rrr1 = {0}; rrr1.set_swizzle = 1; rrr1.swz[0] = rrr1.swz[1] = rrr1.swz[2] = MTLTextureSwizzleRed; rrr1.swz[3] = MTLTextureSwizzleOne; + mtl_tex_init(&f->t); mtl_reg_tex(9, &f->t, &rrr1); // a live object: its ID keeps its view from one submission to the next + mtl_sub_begin(); id v1 = by_id(9); elsewhere(f, 0, mid); id v2 = by_id(9); finish(); + CHECK(v1.parentTexture && g_treg[9].view && v2 == v1, "a texture ID keeps its view through the submission of a racing write"); + expect_read(f, &n); r0 = g_pkg_reuploads; + mtl_sub_begin(); id v3 = by_id(9); finish(); + CHECK(v3.parentTexture && v3.parentTexture != v1.parentTexture && g_pkg_reuploads == r0 + 1, "and takes a view of another texture in the next submission, with nothing else written since"); + CHECK(content_ok(f, v3.parentTexture), "which holds the written content"); + mtl_reg_tex(9, NULL, NULL); + // A dropped batch is reported after later uploads were encoded: images copied since the last whole upload are not trusted + NvPkg *pk = tex_get(f->t.serial); + CHECK([pk isKindOfClass:[NvPkg class]] && pk->part, "the texture is known to hold copied images"); + atomic_fetch_add(&g_up_lost_gen, 1); + look(f, 1, "after a dropped batch"); + CHECK(!pk->part, "a whole upload holds no copied image"); + write_image(f, 0, mid); look(f, 0, "one image written after the whole upload"); + for (unsigned l = 0; l < levels; l++) for (unsigned y = 0; y < f->layers; y++) write_image(f, l, y); + look(f, 1, "every image written"); + drop(f); +} +// The cost on the texture this was found on, read whole against one layer's: a look a submission (the idle spare is written), and three, each after a write, as it was found +// (a texture made for each) +static void cost(void) { + g_case = "cost"; g_direct_upload_on = 1; + fix_t *f = make(NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_R16, 1800, 1380, 8, 1, 0); + for (unsigned a = 0; a < 8; a++) write_image(f, 0, a); + look(f, 1, "first upload"); + double ms[2][2][2] = {{{0}}}; unsigned n; + for (int looks = 1; looks <= 3; looks += 2) for (int part = 0; part < 2; part++) for (int r = 0; r < 6; r++) { + mtl_sub_begin(); uint64_t t0 = bp_now(), cpu = 0; + for (int k = 0; k < looks; k++) { + if (part) write_image(f, 0, (unsigned)(r + k) % 8); else for (unsigned a = 0; a < 8; a++) write_image(f, 0, a); + expect_read(f, &n); + uint64_t c0 = bp_now(); CHECK(tex_mtl(&f->t) != nil, "resolves"); cpu += bp_now() - c0; + } + finish(); uint64_t t2 = bp_now(); + if (r >= 2) { ms[looks / 2][part][0] += cpu / (4e6 * looks); ms[looks / 2][part][1] += (t2 - t0) / 4e6; } + } + CHECK(content_ok(f, tex_mtl(&f->t)), "every image holds the reference"); + for (int i = 0; i < 2; i++) + printf("t_partial_upload: 1800x1380x8 R16 (%.1f MB), %d re-upload%s a submission: read whole %.2f ms each on the submit thread, %.1f ms a submission to completion; one layer written %.2f ms, %.1f ms\n", + f->t.L.size / 1048576.0, 2 * i + 1, i ? "s" : "", ms[i][0][0], ms[i][0][1], ms[i][1][0], ms[i][1][1]); + drop(f); +} +int main(void) { @autoreleasepool { + mtl_init(); + for (int direct = 0; direct < 2; direct++) { + run("R16 array", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_R16, 256, 192, 8, 1, 0, direct); + run("R16 array beside other data", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_R16, 256, 192, 8, 1, 4096, direct); + run("RGBA8 array with levels", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_RGBA8, 200, 120, 5, 6, 0, direct); + run("RGBA16F array with levels", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_RGBA16F, 64, 64, 12, 4, 4096, direct); + run("BC5 array", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_RGTC2_UNORM, 512, 256, 3, 1, 0, direct); + run("BC7 array with levels", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_BPTC_UNORM, 256, 256, 3, 5, 4096, direct); + run("BC7 2D with levels", NVN_TEXTURE_TARGET_2D, NVN_FORMAT_BPTC_UNORM, 1024, 512, 1, 8, 0, direct); + run("RGBA8 cube with levels", NVN_TEXTURE_TARGET_CUBEMAP, NVN_FORMAT_RGBA8, 128, 128, 1, 5, 0, direct); + run("RGBA8 cube array", NVN_TEXTURE_TARGET_CUBEMAP_ARRAY, NVN_FORMAT_RGBA8, 64, 64, 12, 3, 0, direct); + run("RGBA8 3D with levels", NVN_TEXTURE_TARGET_3D, NVN_FORMAT_RGBA8, 64, 64, 32, 4, 0, direct); + run("RGB565 array", NVN_TEXTURE_TARGET_2D_ARRAY, NVN_FORMAT_RGB565, 256, 128, 4, 1, 0, direct); + } + cost(); + printf("t_partial_upload: %d checks, %d failed\n", checks, failed); return failed != 0; +} } diff --git a/docs/agents/symptoms.md b/docs/agents/symptoms.md index 80462e5fef..499ac9e877 100644 --- a/docs/agents/symptoms.md +++ b/docs/agents/symptoms.md @@ -540,6 +540,7 @@ One line per symptom in the [subsystem references](../reference/): what you see - passes that read textures through a pool (Shaders report which textures they read through a pool) cost far more GPU time than their texture reads explain, most on an iPad: Ocarina of Time's colour pass about 13 ms on an iPad Pro M5, 6 times the Mac's, while its prepass cost 1.3 times. → [Pool marks queue on one word when they test it with an atomic load](../reference/frame-rate.md#pool-marks-queue-on-one-word-when-they-test-it-with-an-atomic-load) - a busy scene runs far below what its GPU time allows, with the main thread busy for most of each second. The `pipelines up to frame` line shows archive lookups that found none in the thousands per 300 frames, at about 3 ms each, many draws with a hybrid pipeline, and `in flight 384`. Ocarina of Time's castle town market: 9,208 searches, 30.5 s in 300 frames, CPU translation 114 ms a frame. → [Archive searches repeated while the build table is full](../reference/frame-rate.md#archive-searches-repeated-while-the-build-table-is-full) - frames of 50–170 ms on entering an area, in every run again, although the prewarm lists and the pipeline archive hold everything the area draws. The long-frame line shows `new pipelines 0` or `1` with tens of milliseconds, and the `pipelines up to frame` line shows that time under `predictions`, with a hundred or more `archive lookups found` at about 1 ms each and as many `pulling pipelines made`, against one or two `draws with a hybrid pipeline`. Ocarina of Time's castle town: one prediction made 160 stand-ins in 104 ms, three predictions 244 in 191 ms. → [Pulling stand-ins made for predicted keys that are never drawn](../reference/frame-rate.md#pulling-stand-ins-made-for-predicted-keys-that-are-never-drawn) +- frames of 30–150 ms as an area loads, with `new pipelines 0` and a long-frame line such as `texture uploads 31 first + 6 re-uploads, 235.0 MB / 23.7 ms`. With `DM_NVN_REUPLOADS=1` the same large texture is re-uploaded many times within a few frames, several times with the same `submission` number, each for a few written pages (`pages written since its upload: 7 of 2466`). → [A texture written while it is drawn read whole at every use](../reference/frame-rate.md#a-texture-written-while-it-is-drawn-read-whole-at-every-use) ## NVN objects diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index c6c89c8007..a9e582767b 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -174,6 +174,7 @@ DM_NVN_LAZY_CLEAR switch `0` makes clears with no draw after them open a pass of DM_NVN_LOADPROBE diagnostic `=1`, `2` or `3`: attachments whose load is skipped are cleared to a sentinel colour (`3` NaN; `1` NaN for float formats) and the sentinel pixels left at pass end are counted (should be 0) DM_NVN_LOG diagnostic `=1` logs the creation of NVN objects DM_NVN_MAGSCAN diagnostic `=[:every[:phase]]`: device self-check that reads each pass's colour attachments every `every` frames (default 30) and logs `[mag]` lines for solid magenta 16×16 blocks or non-finite pixels +DM_NVN_PARTIAL_UPLOAD switch `0` reads a texture whole from its storage at every re-upload; by default a re-upload reads only the images (a layer's level) with a page written since the last upload and copies the others from the texture it replaces DM_NVN_PASSDUMP diagnostic `=`, `-` or `every:`: logs every render pass of those frames (attachments, clears, shader keys, bindings, draw state, presents, and the host function and line that ended each pass) for the `tbdr_*.py` tools DM_NVN_PASSDUMP_ALL diagnostic `=1` makes `DM_NVN_PASSDUMP` record every draw and binding change, not only the first per pass and stage, including copies and inline uniform updates DM_NVN_PASSGPU diagnostic `=` writes the GPU time of each render and compute pass from stage-boundary timestamp counters (off under Metal 4); also gives each dispatch its own encoder @@ -187,7 +188,7 @@ DM_NVN_PROF diagnostic `=1` wraps the implemented NVN functions in a timing wrap DM_NVN_PROGTRACE diagnostic `=` logs matching ProgramSetShaders calls and their guest backtraces DM_NVN_REGLUM diagnostic `=x0,y0,x1,y1:` (normalized coordinates): on every present records the present number, time, mean luminance and a grayscale thumbnail of that region, to check animation speed against frame rate DM_NVN_REPORT_EXACT switch `0` refuses a partly covered uniform shadow over the whole 4 KB page of a counter report of a running submission, as before; by default only over the report's own 16 bytes, which the host writes when that submission completes -DM_NVN_REUPLOADS diagnostic `=1` logs the first 400 re-uploads of textures taken from their storage: shape, size, the pages written since the upload and where the first lies, and the GPU copies into it since then +DM_NVN_REUPLOADS diagnostic `=1` logs the first 400 re-uploads of textures taken from their storage: shape, size, how much was read from guest memory and how many images were copied from the texture it replaces, the frame and the submission, the pages written since the upload and where the first lies, and the GPU copies into it since then DM_NVN_RP_AREA switch `0` makes passes on targets of 16 M pixels or more cover the whole target, as before; by default a draw pass that loads what it keeps stops at the far corner of its scissor when that is at most half the target DM_NVN_RT_CONTINUE switch `0` ends the pass whenever the game sets render targets, as before; by default SetRenderTargets naming the open pass's targets continues the pass DM_NVN_RT_RANGE switch `=1` (experimental, off by default; with `DM_NVN_TBDR=1`) keeps the pass open when the game sets the same colour and depth target again with a view that differs only in its level count diff --git a/docs/reference/frame-rate.md b/docs/reference/frame-rate.md index 6abd2c7f42..e983ac669f 100644 --- a/docs/reference/frame-rate.md +++ b/docs/reference/frame-rate.md @@ -626,4 +626,17 @@ Frame pacing: the host's frame step, engine frame limiters, present intervals an - **Cause:** after a submission's first pipeline miss the prediction (`pso_predict_rest_`) walks the rest of its draws. Besides each draw's key it proposes that key without the depth and stencil attachment, for a pass that attaches them on demand. Such a key is rarely drawn, so it never enters the pipeline cache or a prewarm list, and every prediction proposes it again. With hybrid pipelines the prediction made the pulling stand-in of every proposed key, to learn whether its draw would wait: an archive lookup of the pulling base on the submit thread each, outside the budget of two builds. A draw asks for its own pipeline without compiling first, and a Metal 4 base does not depend on the depth and stencil formats, so each of those keys was one specialization of a base already made; the stand-in was never needed. - **Fix:** general, in the host (`nvn_mtl.m`, `pso_base_made`, `g_pp_easy`): the prediction first asks whether a proposed key's base is made. If it is, the key needs no compile and no stand-in: nothing is made or handed to the background for it, and it is left out of later predictions' candidates, where it used to fill the 160 places for alternatives. Open: a proposed key with no base made still gets its stand-in made on the submit thread, an archive lookup each; on a first visit with no prewarm row that is the lookup its draw does next, but for a key that is not drawn it is wasted. - **Check:** `make -C dm/gfx/tests GAME= check_t_pso_depth_stencil`: with a draw's pipeline cached, the prediction hands nothing over for its attachment-free variant, asks no archive, makes no pipeline, and proposes it once only; without Metal 4 bases it builds it as before. On the castle town route the same three predictions took 0.5, 2.0 and 1.4 ms, with 1 pulling pipeline made. + +### A texture written while it is drawn read whole at every use + +- **Symptom:** frames of 30–150 ms as an area loads, with `new pipelines 0` and a long-frame line such as `texture uploads 31 first + 6 re-uploads, 235.0 MB / 23.7 ms`. With `DM_NVN_REUPLOADS=1` the same large texture is re-uploaded many times within a few frames, several times with the same `submission` number, each for a few written pages (`pages written since its upload: 7 of 2466`). +- **Cause:** two things in the host's look at a texture taken from guest memory (`tex_mtl_` in `nvn_mtl.m`), found where a game's loader thread fills a texture over several frames while the game already draws with it. (1) One written page sent every layer and level through the upload again, into another Metal texture. (2) The pages were asked at every use. The queue's thread replays a submission while the game's threads run on, so each look found the few pages written since the last, took a texture and uploaded, and the next draw's look found the pages written meanwhile: Ocarina of Time re-uploaded a 1800x1380x8 R16 array (38.5 MB) 5 times and a 3072x2096 BC5 texture 8 times in one submission. Fixing (1) alone made (2) worse, as a shorter upload lets more looks see a fresh write: 17 uploads of the array in one submission, 22 of the BC5 texture, a frame of 148.5 ms. +- **Fix:** general, in the host. + - A re-upload reads only the images written (`pkg_written`): a layer's level, or a 3D texture's level, lies in a byte range of its own, so the images with a page written since the last upload's stamp are read from guest memory and the others are copied on the GPU from the texture being replaced (`up_copy_add`, in the upload batch's copy encoder, after a barrier where the source is itself that batch's target). The texture in-flight draws read is still never written. Pages are 16 KB, so an image sharing one with a written neighbour is read too. Sparse, linear and multisampled textures, and a texture whose layout does not say, are read whole. A texture holding copied images is read whole again once any upload batch has been dropped since its last whole upload (`g_up_lost_gen`), as the report of a dropped batch comes after later uploads were encoded. `DM_NVN_PARTIAL_UPLOAD=0` reads whole. + - A submission looks at a texture's pages once. A write another thread makes while the queue's thread replays a submission races the GPU's run of that submission on the console, where every draw of it reading the memory as of one moment after the submit is one of the outcomes; the writes the game made before it queued the submission are all in the first look. The next submission's first look reads what was written since, so the texture is current one submission after the writer stops. A write by the replaying thread itself (a fault or a host write into tracked memory, `nvn_dirty_own_writes`) is in order with the commands it replays and is looked for at the next use; outside a submission (a present) every look reads the pages. When a submission kept a texture that was written, the next one starts a write generation (`nvn_dirty_touch`): the caches that remember "nothing written since" by generation (a texture ID's view, the argument blocks) would otherwise keep the old texture until some other page was written. + - Found on the way: the recovery of a dropped upload batch was skipped for a texture whose storage shares its first or last host page with other data (`pkg_edges_clean` found those bytes unchanged and kept the unwritten texture). `t_direct_upload` passed because its recovery block ran only when the report arrived before the test looked. +- **Measured:** Ocarina of Time on an M4 Pro, headless, walking into Hyrule Castle Town (70 s, `DM_FPS_HOLD=off`, the runs in which the array is rewritten). Before, two runs: 52 and 31 re-uploads, the array 16 and 17 times whole (616 and 654 MB), frames of 31, 59 and 30 ms and of 64 and 37 ms while it is written. After, three runs: 16–17 re-uploads, the array 7–9 times, 66–76 MB read in all (4.7–14.2 MB each, 0.04–0.15 ms each on the queue's thread), and one frame of 25 ms in two of the runs, none over 22 ms in the third. In `t_partial_upload` three re-uploads a submission of a 1800x1380x8 R16 array complete in 13–17 ms with one layer written each, against 70–75 ms read whole. +- **Check:** `make -C dm/gfx/tests GAME= check_t_partial_upload`: 2D arrays, cube maps and arrays, 2D and 3D textures with levels, in R16, RGBA8, RGBA16F, RGB565, BC5 and BC7; after each write every image of the texture the host returns is compared with a CPU reference and the bytes read from guest memory with those of the images whose pages the write touched; the idle spare (two uploads old) and two re-uploads in one submission; a racing write (kept for the submission, read by the next, also through a texture ID's cached view with nothing else written), a write by the submit thread, a look outside a submission, a dropped batch. `check_t_direct_upload` covers whole uploads and the dropped-batch recovery. +- **Open:** a single-image texture (the 3072x2096 BC5 one) is still read whole: nothing finer than an image is tracked. Each re-upload while the spare is busy makes a Metal texture and copies the unwritten images; writing the written images into the current texture would save both, and needs every reader of it (in-flight draws of the open command buffer, a present on another thread) ordered after the upload. `DM_NVN_DIRTYCHECK=1` reports one `dirty-page miss` on this route with and without the change (`DM_NVN_PARTIAL_UPLOAD=0`, before the one look per submission): not found yet. +- **Not checked:** other games (no `scripts/regress.py check`: Ocarina of Time is the only game built on the Mac this was measured on, though every NVN game with textures taken from guest memory passes through this path); iOS. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md#frame-rate).