Pipelines a hybrid draw stands in for are listed in pso_user.tsv when first asked for (from HookLink)

dm/games/0400486024DEC000/README.md

@@ -145,6 +145,7 @@ Its 37 kernels that correct samples, NDSR's network among them, gave about 0.25
 - [Archive searches repeated while the build table is full](../../../docs/reference/frame-rate.md#archive-searches-repeated-while-the-build-table-is-full): the castle town market at about 12 fps while the GPU had time to spare.
 - [Pulling stand-ins made for predicted keys that are never drawn](../../../docs/reference/frame-rate.md#pulling-stand-ins-made-for-predicted-keys-that-are-never-drawn): resolved, frames of 50–170 ms on entering the castle town in every run, with one new pipeline or none.
 - [A texture written while it is drawn read whole at every use](../../../docs/reference/frame-rate.md#a-texture-written-while-it-is-drawn-read-whole-at-every-use): resolved, frames of 31–59 ms on entering Hyrule Castle Town, a 1800x1380x8 R16 array (38.5 MB) and three block-compressed textures re-uploaded 31–52 times while the loader wrote them, the array 16–17 times whole; 16–17 re-uploads after, the array's 7–9 at 4.7–14.2 MB, one frame of 25 ms at most.
+- [A pipeline a hybrid draw stood in for listed only once built](../../../docs/reference/shaders.md#a-pipeline-a-hybrid-draw-stood-in-for-listed-only-once-built): on an iPhone 18 Pro Max, found by HookLink, the pipelines Castle Town asked for last were not in `pso_user.tsv` for the next start; they are listed when first asked for (`DM_PSO_DEMAND_RECORD=0` restores the old order). Synthetic hybrid-lookup tests now cover pending compilation, canonical persistence/reload, completion deduplication and recording limits; the iPhone timing claim is not locally reproduced.
 - [One Pro Controller read as two players](../../../docs/reference/services.md#one-pro-controller-read-as-two-players): resolved, no music in Kokiri Forest gameplay and its cutscenes; the controller applet opened over and over and the game kept its music suspended while it had focus.
 - [A keyboard receiver that declares a gamepad takes player 1](../../../docs/reference/services.md#a-keyboard-receiver-that-declares-a-gamepad-takes-player-1): current until the reporter retests ([issue #1657](https://github.com/seedds/nxnative/issues/1657)). A Keychron Link receiver was player 1 and the Pro Controller player 2, so the game took no input; a raw HID pad is now a player from its first button press.
 - [The controller applet leaves out the players past the maximum](../../../docs/reference/services.md#the-controller-applet-leaves-out-the-players-past-the-maximum): resolved in the host's tests, not run on the game. One Bluetooth Pro Controller came back as players 1 and 2 after a long disconnect, and the game opened the controller applet 1,671 times with no input ([issue #2237](https://github.com/seedds/nxnative/issues/2237)); the second player's controller now feeds player 1.

dm/gfx/nvn/nvn_mtl.m

@@ -9220,6 +9220,18 @@ static int pso_pend_kick(const pso_key *k, NvShader *vs, NvShader *fs) {   // No
     pso_pend_start(slot, gen, pso_desc(k, vs, fs, nil, 0), !k->border, dispatch_get_global_queue(QOS_CLASS_USER_INITIATED, 0));
     return 1;
 }
+// A pipeline a hybrid draw had to stand in for goes into pso_user.tsv when first asked for, not only once built. During play its build took 5-50 s on an iPhone 18 Pro Max
+// (2 performance cores shared with the game), about 1 s in the prepare step before the first frame; Castle Town asked for ~400 and finished ~60 a minute, and what had not
+// finished by the end of the run was never listed. Listed at once, the next start's prepare step builds every one of them. The later line from the built pipeline is a
+// duplicate, dropped by pso_user_record. DM_PSO_DEMAND_RECORD=0 records only built pipelines, as before (submit thread)
+static nvn_hmap g_demand_rec = {.klen = sizeof(pso_key)};
+static void demand_record(const pso_key *k, NvShader *vs, NvShader *fs) {
+    static int on = -1; if (on < 0) { const char *ev = getenv("DM_PSO_DEMAND_RECORD"); on = !(ev && !strcmp(ev, "0")); }
+    if (!on || nvn_hm_get(&g_demand_rec, k) || g_demand_rec.n >= (1u << 16)) return;
+    nvn_hm_put(&g_demand_rec, k, (void *)1);
+    pso_skey sk; char shex[sizeof sk * 2 + 1]; pso_skey_from(k, vs, fs, 0, NULL, &sk); pso_skey_hex(&sk, shex);
+    pso_user_record("R\t%s\n", shex);
+}
 // The build a hybrid draw asked for while it draws with its stand-in: outside the speculative limit, on pp_bg_queue
 static int pso_pend_demand(const pso_key *k, NvShader *vs, NvShader *fs) {
     if (hm_get_id(&g_pso, k)) return 0;
@@ -9609,6 +9621,7 @@ static void pso_prebuild_bg(void) {   // after the prewarm, in the app without a
                 __unsafe_unretained id<MTLRenderPipelineState> h = hybrid_get(&k, vs, fs);
                 if (h) {
                     if (st == 0) pso_pend_demand(&k, vs, fs);
+                    demand_record(&k, vs, fs);
                     g_pso_pull = 1; g_hy_ns += bp_now() - bt;
                     if (g_pp_submit_mark != g_sub_serial) { g_pp_submit_mark = g_sub_serial; uint64_t t0 = bp_now(); pso_predict_rest(); g_bp_pso_ns += bp_now() - t0; }
                     return h;

dm/gfx/tests/Makefile

@@ -354,6 +354,13 @@ t_rt_depth_ro: t_rt_depth_ro.m $(MTL_T_DEP)
 	clang $(CFLAGS) -fobjc-arc -o $@ t_rt_depth_ro.m $(MTL_T_SRC) $(FW)
 t_pso_pending: t_pso_pending.m $(MTL_T_DEP)
 	clang $(CFLAGS) -fobjc-arc -o $@ t_pso_pending.m $(MTL_T_SRC) $(FW)
+t_pso_demand_record: t_pso_demand_record.m $(MTL_T_DEP)
+	clang $(CFLAGS) -fobjc-arc -o $@ t_pso_demand_record.m $(MTL_T_SRC) $(FW)
+.PHONY: check_t_pso_demand_record
+check_t_pso_demand_record: t_pso_demand_record
+	MTL_DEBUG_LAYER=1 ./t_pso_demand_record
+	MTL_DEBUG_LAYER=1 ./t_pso_demand_record --off
+	MTL_DEBUG_LAYER=1 ./t_pso_demand_record --cap
 t_shmap_warm: t_shmap_warm.m $(MTL_T_DEP)
 	clang $(CFLAGS) -fobjc-arc -o $@ t_shmap_warm.m $(MTL_T_SRC) $(FW)
 t_pso_depth_stencil: t_pso_depth_stencil.m $(MTL_T_DEP)

dm/gfx/tests/t_pso_demand_record.m

@@ -0,0 +1,117 @@
+// Real pso_get hybrid misses must record demand before their queued compiles run.
+// Synthetic functions and cached stand-ins need no translated/game libraries.
+#include "../nvn/nvn_mtl.m"
+#include <spawn.h>
+#include <sys/wait.h>
+#include <unistd.h>
+extern char **environ;
+void *front_metal_layer(void) { return NULL; }
+module_t *module_of(const void *p) { (void)p; return NULL; }
+module_t *g_mod; int g_nmod;
+void svc_thread_dump(void) {}
+uint64_t now_ns(void) { return bp_now(); }
+static int checks, failed;
+#define CHECK(name, cond) do { checks++; if (!(cond)) { failed++; fprintf(stderr,"FAIL: %s (%d)\n",name,__LINE__); } } while (0)
+static const char *vkey="1111111111111111_b", *fkey="2222222222222222_b";
+static NSArray<NSString *> *rows(const char *path) {
+    NSString *text=[NSString stringWithContentsOfFile:@(path) encoding:NSUTF8StringEncoding error:nil];
+    CHECK("manifest is independently readable",text != nil);
+    NSMutableArray<NSString *> *r=[NSMutableArray array];
+    for(NSString *line in [text componentsSeparatedByString:@"\n"]) if([line hasPrefix:@"R\t"]) [r addObject:line];
+    return r;
+}
+static void check_rows(NSArray<NSString *> *r, unsigned n) {
+    CHECK("exact number of canonical render rows",r.count == n);
+    for(NSUInteger i=0;i<r.count;i++) {
+        pso_skey k; int ok=!pso_skey_parse([r[i] substringFromIndex:2].UTF8String,&k);
+        CHECK("canonical key parses",ok);if(!ok)continue;
+        CHECK("synthetic shader names and real stages survive serialization",!strcmp(k.vs,vkey)&&!strcmp(k.fs,fkey)&&k.vs_stage==0&&k.fs_stage==1);
+        CHECK("ordinary key contains no process pointers or merged state",!k.k.vs&&!k.k.fs&&!k.k.fsfn&&!k.merged&&!k.fz);
+        CHECK("pipeline output state survives serialization",k.k.cfmt[0]==MTLPixelFormatRGBA8Unorm&&k.k.samples==1&&k.k.chmask==(i?7u:15u));
+    }
+}
+static int reload(const char *path, unsigned n) {
+    NSArray<NSString *> *r=rows(path);check_rows(r,n);
+    g_shmap=[@{@"1111111111111111":@"vertex",@"2222222222222222":@"pixel"} mutableCopy];
+    long lines=-1;NSArray<WarmTask *> *tasks=warm_parse(@[@(path)],&lines);
+    CHECK("fresh process accepts recorded rows as prewarm tasks",lines==n&&tasks.count==(n?1u:0u));
+    if(tasks.count) {
+        WarmTask *t=tasks[0];CHECK("rows group into the correct render pair",t->kind==1&&[t->vkey isEqualToString:@(vkey)]&&[t->fkey isEqualToString:@(fkey)]&&t->rs.count==n);
+        for(NSUInteger i=0;i<t->rs.count&&i<r.count;i++) {
+            pso_skey key;int ok=!pso_skey_parse([r[i] substringFromIndex:2].UTF8String,&key);
+            CHECK("prewarm retains the complete canonical key",ok&&[t->rs[i] isEqualToData:[NSData dataWithBytes:&key length:sizeof key]]);
+        }
+    }
+    printf("reload: %d checks, %d failed\n",checks,failed);return failed!=0;
+}
+static void child_reload(const char *exe,const char *path,unsigned n) {
+    char count[16];snprintf(count,sizeof count,"%u",n);
+    char *args[]={(char *)exe,"--reload",(char *)path,count,NULL};pid_t pid;int status=0;
+    int error=posix_spawn(&pid,exe,NULL,NULL,args,environ);CHECK("fresh process starts",!error);
+    if(!error){CHECK("fresh process exits normally",waitpid(pid,&status,0)==pid&&WIFEXITED(status)&&!WEXITSTATUS(status));}
+}
+static int pending(const pso_key *key) {
+    pthread_mutex_lock(&g_pp_mu);int i=pso_pend_find(key);int yes=i>=0&&!g_pp[i].done&&!g_pp[i].run&&g_pp[i].d!=nil;pthread_mutex_unlock(&g_pp_mu);return yes;
+}
+int main(int argc,char **argv) { @autoreleasepool {
+    alarm(90);
+    if(argc==4&&!strcmp(argv[1],"--reload"))return reload(argv[2],(unsigned)strtoul(argv[3],NULL,10));
+    int off=argc==2&&!strcmp(argv[1],"--off"),cap=argc==2&&!strcmp(argv[1],"--cap");
+    if(argc>2||(argc==2&&!off&&!cap))return 2;
+    char path[]="/tmp/t_pso_demand_record.XXXXXX";int fd=mkstemp(path);if(fd<0)return 2;close(fd);
+    setenv("DM_PSO_USER",path,1);setenv("DM_PSO_ARCHIVE","off",1);setenv("DM_PSO_MANIFEST","off",1);setenv("DM_PSO_PREBUILD","0",1);
+    setenv("DM_PSO_DEMAND_RECORD",off?"0":"1",1);setenv("DM_PSO_HYBRID","1",1);setenv("DM_NVN_PREDICT","0",1);unsetenv("DM_SAVE");
+    mtl_init();CHECK("hybrid path is available",hybrid_on());if(!hybrid_on())return 2;
+    NSError *error=nil;
+    id<MTLLibrary> lib=[g_dev newLibraryWithSource:@"#include <metal_stdlib>\nusing namespace metal;\nvertex float4 v(uint i[[vertex_id]]){return float4(i==1?3.0:-1.0,i==2?3.0:-1.0,0,1);}\nfragment float4 f(){return float4(0,1,0,1);}\n" options:nil error:&error];
+    CHECK("synthetic library compiles",lib!=nil);if(!lib)return 2;
+    NvShader *vs=[NvShader new],*fs=[NvShader new];vs->stage=0;fs->stage=1;vs->has_pull=1;fs->rf.color_known=1;fs->rf.color_mask=1;
+    strlcpy(vs->key,vkey,sizeof vs->key);strlcpy(fs->key,fkey,sizeof fs->key);
+    vs->fn=fn4_note([lib newFunctionWithName:@"v"],lib,@"v",nil);fs->fn=fn4_note([lib newFunctionWithName:@"f"],lib,@"f",nil);
+    CHECK("synthetic function entries exist",vs->fn&&fs->fn);if(!vs->fn||!fs->fn)return 2;
+    NvProg *pr=[NvProg new];pr->sh[0]=vs;pr->sh[1]=fs;
+    state_reset();memset(&g_pass,0,sizeof g_pass);g_pass.cfmt[0]=MTLPixelFormatRGBA8Unorm;g_pass.samples=1;
+    pso_key keys[2];id<MTLRenderPipelineState> stand[2];
+    for(int i=0;i<2;i++) {
+        S.chmask.mask=i?7:15;pso_key_fill(&keys[i],vs,fs,g_pass.cfmt,0,0,1,0);
+        // Build stand-ins outside the production base cache: the real pso_get
+        // archive-only lookup must miss and admit its own background demand.
+        stand[i]=[g_dev newRenderPipelineStateWithDescriptor:pso_desc(&keys[i],vs,fs,nil,0) error:&error];
+        CHECK("real cached stand-in compiles",stand[i]!=nil);if(!stand[i])return 2;
+        hm_put_id(&g_pso_hy,&keys[i],stand[i]);
+    }
+    if(cap) {
+        for(uint32_t i=0;i<(1u<<16)-1;i++){pso_key dummy={0};dummy.samples=i+2;nvn_hm_put(&g_demand_rec,&dummy,(void *)1);}
+        CHECK("admission map starts one entry below its bound",g_demand_rec.n==(1u<<16)-1);
+    }
+    dispatch_queue_t queues[8];int nq=0;
+    for(int i=0;i<9;i++){dispatch_queue_t q=pp_bg_queue();int seen=0;for(int j=0;j<nq;j++)seen|=q==queues[j];if(seen)break;CHECK("bounded background queue count",nq<8);if(nq==8)return 2;queues[nq++]=q;dispatch_suspend(q);}
+    S.chmask.mask=15;id<MTLRenderPipelineState> a=pso_get(pr,nil);
+    CHECK("first real lookup returns the cached stand-in",a==stand[0]&&g_pso_pull);
+    CHECK("new demand remains queued, not compiled",pending(&keys[0]));
+    unsigned early=off?0:1;check_rows(rows(path),early);
+    CHECK("already-pending lookup still returns stand-in",pso_get(pr,nil)==stand[0]&&g_pso_pull);
+    CHECK("duplicate demand does not append a row",rows(path).count==early);
+    // The second pipeline was queued independently: its *first* pso_get sees
+    // st==1, covering placement of demand_record outside the st==0 branch.
+    CHECK("second demand can be independently queued",pso_pend_demand(&keys[1],vs,fs));
+    S.chmask.mask=7;CHECK("prequeued lookup returns its stand-in",pso_get(pr,nil)==stand[1]&&g_pso_pull);
+    CHECK("both compiles remain gated",pending(&keys[0])&&pending(&keys[1]));
+    early=off?0:cap?1:2;NSArray<NSString *> *before=rows(path);check_rows(before,early);
+    if(cap)CHECK("map cap declines the next early record",g_demand_rec.n==(1u<<16));
+    child_reload(argv[0],path,early); // fflush visibility across exec; not power-loss/fsync durability.
+    CHECK("reload did not need compilation to complete",pending(&keys[0])&&pending(&keys[1]));
+    for(int i=0;i<nq;i++)dispatch_resume(queues[i]);
+    for(int i=0;i<nq;i++)dispatch_sync(queues[i],^{});
+    for(int i=0;i<2;i++) {
+        S.chmask.mask=i?7:15;id<MTLRenderPipelineState> ps=pso_get(pr,nil);
+        CHECK("eventual completed pipeline replaces stand-in",ps&&ps!=stand[i]&&!g_pso_pull);
+        CHECK("completed pipeline is retained in the normal cache",hm_get_id(&g_pso,&keys[i])==ps);
+    }
+    NSArray<NSString *> *after=rows(path);check_rows(after,2);
+    for(NSUInteger i=0;i<before.count&&i<after.count;i++)CHECK("early row exactly equals eventual completion row",[before[i] isEqualToString:after[i]]);
+    CHECK("completion deduplicates or records off/capped demand",after.count==2);
+    child_reload(argv[0],path,2);
+    if(g_pso_user){fclose(g_pso_user);g_pso_user=NULL;}unlink(path);
+    printf("t_pso_demand_record %s: %d checks, %d failed\n",off?"off":cap?"cap":"on",checks,failed);return failed!=0;
+} }

docs/agents/symptoms.md

@@ -1168,6 +1168,7 @@ One line per symptom in the [subsystem references](../reference/): what you see
 - the first sight of a scene drawn with task and mesh shaders freezes for seconds on a cold Metal cache, on every start once the system's shader cache has dropped the pipelines, although the prewarm finished: `task / mesh pipeline: task … mesh … fragment …, N ms required at frame F`. Before ea0c98df the draw was left out instead, and the foliage stayed missing or white until the pipeline was ready. → [Native mesh pipelines compiled at their first draw](../reference/shaders.md#native-mesh-pipelines-compiled-at-their-first-draw)
 - a frame of 200 ms or more the first time an effect appears, although the bundled pipeline archive was trusted and the prewarm finished. With `DM_PSO_WAITLOG=1` the frame shows waits on render pipelines with no pulling entry, 34-74 ms each, for vertex shaders that have no stage inputs. → [Vertex shaders with layer entry points left out of the prebuild](../reference/shaders.md#vertex-shaders-with-layer-entry-points-left-out-of-the-prebuild)
 - `[RED] pipeline prewarm: seed list does not exist: …/pso_manifest.tsv` at every start, no `pipeline prewarm list …/pso_user.tsv` line, and the same compile stalls every session in places already visited, while `pso_user.tsv` keeps growing; the pipeline archive gains files every start and is never compacted (Donkey Kong Bananza: 88 files, 7 s to open them). → [The device's prewarm list unread without a seed list](../reference/shaders.md#the-devices-prewarm-list-unread-without-a-seed-list)
+- on an iPhone the next start still waits on, or draws with stand-ins for, the pipelines the last session asked for last. Ocarina of Time on an iPhone 18 Pro Max asked for about 400 pipelines in Castle Town and finished about 60 a minute; whatever was still unbuilt when the run ended was not in `pso_user.tsv`, so the next start's prepare step did not build it. HookLink's measurement; we have not reproduced it. → [A pipeline a hybrid draw stood in for listed only once built](../reference/shaders.md#a-pipeline-a-hybrid-draw-stood-in-for-listed-only-once-built)
 - the first frame that draws with new programs loads their shader libraries on the submit thread (`[longframe] ... shader libraries N, X ms`): tens of milliseconds with the files cached, seconds when the drive is slow. → [Shader libraries loaded when the program is made](../reference/shaders.md#shader-libraries-loaded-when-the-program-is-made)
 - at boot, `CRASH sig=5 … pc=_dispatch_once_wait.cold.1` right after the `pipeline bases so far` line, with `bt: pso_prebuild __shmap_load_block_invoke … mtl_prog_prefetch h_nvnProgramSetShaders`. → [The prepare step inside the shader map's once](../reference/shaders.md#the-prepare-step-inside-the-shader-maps-once)
 - after an IPA packed without `_bp.metallib` ran once with `DM_PSO_MTL4=1`, an app with those libraries still stalls on new pipelines: `pipelines up to frame` lines show `draws with a hybrid pipeline 0` and `draws that waited: pulling base not made yet`, and the log has no `pipeline prebuild:` line. The run before it logged `[RED] pulling library of … not usable` and `pipeline prebuild done: … 19838 left to the draws`. → [A prebuild stamp written without the pulling libraries](../reference/shaders.md#a-prebuild-stamp-written-without-the-pulling-libraries)

docs/env_switches.tsv

@@ -294,6 +294,7 @@ DM_PROGRAMS	path	Table of a game's programs on the Mac (`programs.tsv`: index, h
 DM_PROGRAM_ID	switch	`=<hex>`: the program ID the host reports from `svcGetInfo(ProgramId)` (default: the profile's `program_id`)
 DM_PSO_ARCHIVE	path	The folder of the per-game Metal 4 pipeline archive of pipeline bases (default `pso_archive/` beside `pso_user.tsv`); `off` keeps no archive
 DM_PSO_BUNDLE	path	The pipeline archive made offline (`<GPU architecture>.mtl4a`, `prebuilt.tsv`), read only; the iOS app and Mac launcher set it to their `pso/` folder; `off` ignores it
+DM_PSO_DEMAND_RECORD	switch	`0`: a pipeline a hybrid draw stands in for goes into `pso_user.tsv` only once built, not when first asked for (default: at once, so the next start's prepare step builds it)
 DM_PSO_GPUS	switch	`build_ipa.sh`: the GPU architectures, comma-separated, to build the offline pipeline archive for (`none` builds none); unset, a profile with `pso_mtl4 on` gets its `pso_gpu` list or `g18p` when absent; profiles with `pso_mtl4 off` get none
 DM_PSO_HYBRID	switch	`0`: a draw whose pipeline is not built yet waits for it instead of drawing with its shader pair's pulling pipeline meanwhile (on by default with the Metal 4 path)
 DM_PSO_IOS_MIN	switch	`build_ipa.sh`: the minimum iOS version of the offline pipeline archives (default the SDK's major version); a device of another major version cannot open them and builds the bases itself