Repository navigation
Ocarina of Time: a prewarm seed list (9,784 rows), so a first start prepares its pipelines instead of freezing for them in play - #1488
aksmith-git wants to merge 2 commits into
Conversation
…so a first start prepares its pipelines before the first frame instead of freezing for them in play 8,355 shaders, 1,227 render, 144 compute and 58 task / mesh pipelines, merged from an iPad Pro's pso_user.tsv over several hours of play and the Mac's DM_PSO_RECORD runs; every shader named is in shader_map.tsv and every R row parses. The bundled list was empty. On the Mac (M4 Pro, DM_PSO_MTL4=0, a 5-minute route from a save) a fresh start compiled 1,056 pipelines on the submit thread (52 s, one frame of 17.8 s entering Kokiri Forest); with the list it prepares it in 5.1 s and builds 6 in play (1.1 s), frames over 100 ms 45 -> 5. The M rows wait for a prewarm that builds task / mesh pipelines (PR seedds#1368); main's skips them.
|
Both lists compared, since they conflict on the same file: #1488 (9,784 rows: S 8,355, R 1,227, C 144, M 58) and #1493 (9,620 rows: S 8,376, R 1,112, C 132) share 9,558 rows; #1493 adds 21 shaders (its S set is a superset of #1488's) and 40 render pipelines #1488 lacks, #1488 adds 155 render pipelines, 12 compute pipelines and the 58 M rows. A third list, this Mac's own Cost of the prepare step with the union list, Ocarina of Time v131072 Mac app on main bd9f422 + #1368 + #1307, M3 Max (36 GB, macOS 27.0.1), the device archive empty each time:
|
|
Compared both lists on a true first start, M4 MacBook Air (24 GB, macOS 27.2, 10 cores), Mac app from main b2d46b8 (+ #1544, #1543, #1555),
Both lists miss the same five compute pipelines: The 7 render pipelines #1488 has and #1493 lacks on this route never made a frame over 100 ms (hybrid pipelines cover them), and #1488's longer list costs about 16 s more before the first frame here. On this route, then, #1493's list (or the union, plus the five |
…g builds on a first start (Deku Scrub, Business Scrub, Deku Baba and two more), as C rows
|
Thanks for the first-start comparison. Added the five compute pipelines you found as |
|
Compute pipelines are now in the offline archive (30fb213). The title movie's network kernels and the area effects' compute pipelines come from the bundle on a first start instead of being compiled, which was most of the cold prepare time measured above (85-150 s). So the seed list's value needs measuring again on main. When it lands, the plan is one list recorded as docs/development.md says: the scenes a new player reaches first and a save load (#1493's route), plus the five |
…ipelines before the first frame instead of building them at the title and in play (PRs #1493, #1488) pso_manifest.tsv was header-only, so a first start with a cold Metal shader cache built the title's pipelines in one submission: 252.6 s of black picture on an M3 Pro (issue #1477). The list is #1493's, recorded as docs/development.md asks (boot, the title and its movie, a new game and the opening, Link's house, Kokiri Forest, a save load; 8,376 shaders, 1,112 render and 132 compute pipelines), and the five compute pipelines Saria's greeting builds, which PucaVaz found on #1488. 9,625 rows; every shader is in the map. #1488's longer list (hours of iPad play, 58 M rows main's prewarm skips) prepared 16 s longer on an M4 MacBook Air and avoided only 7 render builds that made no frame over 100 ms. Open, in the game note: the first start with compute pipelines in the offline archive (30fb213) and on iOS, where this game has no bundled archive, and the render bases the R rows need. Co-authored-by: Yunus ACAR <125305588+itsyunusacar@users.noreply.github.com>
|
Thank you. The list that landed (540f797) is #1493's, recorded over the first scenes and a save load as docs/development.md asks, plus the five |
…loading Merge adjacent GPU memory pools with contiguous guest virtual addresses and matching GPU address deltas into single gmem_map run entries (up to GMEM_RUN_MAX = 1,024 runs) backed by a reusable MTLBuffer ring (gmem_table_take / gmem_table_give), and reuse g_gmem_buf across encode_args_ calls within a submission since gmem_commit_patch updates the table at commit time. Also zero-fill unmapped virtual pool spans with PROT_READ | PROT_WRITE in vpool_map so late CPU reads do not fault. For MTL4Archive pipeline caches (related to PRs seedds#1307, seedds#1488, seedds#1493, seedds#1546), separate vertex-descriptor (vd) and pulling (non-vd) archive epochs so specialized vertex-descriptor pipelines are not written into the pulling archive set; guard against macOS 27's crash in MTL4ArchiveImpl::newArchiveReplyForKey when deserializing an archive entry with a zero-length function reflection string; preserve the active OP_CALL return stack when starting shader_prefetch_rest and pso_predict_rest_; and support falling back to the bundled shader chain directory when DMSH_CHAIN advances. Verified in The Legend of Zelda: Ocarina of Time (0400486024DEC000) on Apple M-series (macOS).
…ipelines before the first frame instead of building them at the title and in play (PRs seedds#1493, seedds#1488)
…pipeline archive: Ocarina of Time's foliage, shadows and enemies no longer stall at their first draw (59.5 s of builds at the Kokiri Forest entry on an M5 Max with a cold Metal cache, one frame of 42.9 s), and its profile seeds 123 of them (PR #1368) A native mesh pipeline was built only at its first draw, through the MTLDevice, and nothing kept it: no prewarm row named one and no pipeline serializer saw it. Since ea0c98d the draw waits for its pipeline, so each one met in play was a stall on the submit thread. - mtk_pso records each pipeline it builds as an M row (the task shader's key or -, then the pso_skey hex with the mesh shader in the vertex place); pso_mesh_skey_parse refuses rows with a vertex layout, a specialized fragment or the wrong stages, and the prewarm builds the rows on its workers (kind 3), so the prepare step has them before the first frame. - warm_claim takes a prewarmed pipeline under the live shader objects when they come from the same library and entry (warm_mesh_shader_equal): the program prefetch loads objects of its own for the same keys, which claiming by identity dropped; a mesh shader loaded with its other depth entry is not taken. - mtk_build keeps the pipelines in the pipeline archive as cps_new keeps compute pipelines: looked up first (a p file when a flush holds only them), built through the MTLDevice on a miss (the system's shader cache serves it when it can) and captured into the flush period in the background (cps_harvest generalized to arch_harvest). Compaction captures one taken from an old plain file again; the flush line counts them on their own. - Ocarina of Time's pso_manifest.tsv carries 123 M rows (Kokiri Forest, Hyrule Field, Castle Town, Hyrule Castle). - prog_get returns nil for no program (draw_'s passdump line tests S.prog, the lines after it did not; clang-tidy found it once mtk_pso grew). Measured on an M6 (16 GB), Ocarina of Time's village walk with the prepare step, hosts pinned to the libraries on disk (f9f01e4f9994): main built 56 mesh pipelines at their draws; this change's first start prewarmed 121 (two seed rows need a newer chain's paired mesh outputs, 5e344a8) and built none at a draw, prepare 19.0 s against 2.7 s as 65 were not in the system's cache; the second start took all 121 from the archive, prepare 2.8 s. Contributors' runs on the rows and prewarm: M1 cold 103.5 s of builds in a 300 s walk; iPad Pro (M5) 1.9-5 s per pipeline cold. Checks: t_pso_skey, t_pso_warm (both compiler paths), t_pso_archive (mesh_pipelines), t_mesh_first_use, t_pso_bundle, t_pso_stall. From PR #1368 (nanotchec): the M row format, its parser, the prewarm task and the claim fix; its source-expansion tool is left out. Seed rows also from padsiwono (50 new) and PR #1488; Amost- diagnosed the dropped claims. Co-authored-by: nanotchec <145017345+nanotchec@users.noreply.github.com>
…ipelines before the first frame instead of building them at the title and in play (PRs seedds#1493, seedds#1488) pso_manifest.tsv was header-only, so a first start with a cold Metal shader cache built the title's pipelines in one submission: 252.6 s of black picture on an M3 Pro (issue seedds#1477). The list is seedds#1493's, recorded as docs/development.md asks (boot, the title and its movie, a new game and the opening, Link's house, Kokiri Forest, a save load; 8,376 shaders, 1,112 render and 132 compute pipelines), and the five compute pipelines Saria's greeting builds, which PucaVaz found on seedds#1488. 9,625 rows; every shader is in the map. seedds#1488's longer list (hours of iPad play, 58 M rows main's prewarm skips) prepared 16 s longer on an M4 MacBook Air and avoided only 7 render builds that made no frame over 100 ms. Open, in the game note: the first start with compute pipelines in the offline archive (bb0ce6c) and on iOS, where this game has no bundled archive, and the render bases the R rows need. Co-authored-by: Yunus ACAR <125305588+itsyunusacar@users.noreply.github.com>
…pipeline archive: Ocarina of Time's foliage, shadows and enemies no longer stall at their first draw (59.5 s of builds at the Kokiri Forest entry on an M5 Max with a cold Metal cache, one frame of 42.9 s), and its profile seeds 123 of them (PR seedds#1368) A native mesh pipeline was built only at its first draw, through the MTLDevice, and nothing kept it: no prewarm row named one and no pipeline serializer saw it. Since 5e5e7e9 the draw waits for its pipeline, so each one met in play was a stall on the submit thread. - mtk_pso records each pipeline it builds as an M row (the task shader's key or -, then the pso_skey hex with the mesh shader in the vertex place); pso_mesh_skey_parse refuses rows with a vertex layout, a specialized fragment or the wrong stages, and the prewarm builds the rows on its workers (kind 3), so the prepare step has them before the first frame. - warm_claim takes a prewarmed pipeline under the live shader objects when they come from the same library and entry (warm_mesh_shader_equal): the program prefetch loads objects of its own for the same keys, which claiming by identity dropped; a mesh shader loaded with its other depth entry is not taken. - mtk_build keeps the pipelines in the pipeline archive as cps_new keeps compute pipelines: looked up first (a p file when a flush holds only them), built through the MTLDevice on a miss (the system's shader cache serves it when it can) and captured into the flush period in the background (cps_harvest generalized to arch_harvest). Compaction captures one taken from an old plain file again; the flush line counts them on their own. - Ocarina of Time's pso_manifest.tsv carries 123 M rows (Kokiri Forest, Hyrule Field, Castle Town, Hyrule Castle). - prog_get returns nil for no program (draw_'s passdump line tests S.prog, the lines after it did not; clang-tidy found it once mtk_pso grew). Measured on an M6 (16 GB), Ocarina of Time's village walk with the prepare step, hosts pinned to the libraries on disk (f9f01e4f9994): main built 56 mesh pipelines at their draws; this change's first start prewarmed 121 (two seed rows need a newer chain's paired mesh outputs, 7db6904) and built none at a draw, prepare 19.0 s against 2.7 s as 65 were not in the system's cache; the second start took all 121 from the archive, prepare 2.8 s. Contributors' runs on the rows and prewarm: M1 cold 103.5 s of builds in a 300 s walk; iPad Pro (M5) 1.9-5 s per pipeline cold. Checks: t_pso_skey, t_pso_warm (both compiler paths), t_pso_archive (mesh_pipelines), t_mesh_first_use, t_pso_bundle, t_pso_stall. From PR seedds#1368 (nanotchec): the M row format, its parser, the prewarm task and the claim fix; its source-expansion tool is left out. Seed rows also from padsiwono (50 new) and PR seedds#1488; Amost- diagnosed the dropped claims. Co-authored-by: nanotchec <145017345+nanotchec@users.noreply.github.com>
What and why
Ocarina of Time's
pso_manifest.tsvis empty (the provisional header only). So on a first start, or once the system has dropped the Metal cache, every pipeline is compiled on the submit thread when first drawn, at 0.1-2.8 s each, and the prepare step has nothing to prepare.This adds a seed list of 9,784 rows: 8,355 shaders, 1,227 render pipelines, 144 compute pipelines and 58 task / mesh pipelines (
Mrows). It merges an iPad Pro'spso_user.tsv(several hours of play from the title through Kokiri Forest and on) with the Mac'sDM_PSO_RECORDruns over a 5-minute route from a save. Every shader it names is in the profile'sshader_map.tsv, and everyRrow parses with main'spso_skey_parse. The rows name shaders by content key and pipelines by NVN state, so no chain change invalidates them.Main's prewarm skips the 58
Mrows (warm_parsereads onlyS,RandC). They have the format #1368 reads (M, task key or-, then thepso_skeyhex with the mesh shader in the vertex place), so they take effect if that lands.Checked
test_docs.py,test_game_profile.py,test_nsp_incremental.py,test_nsp_mac_app.py.0400486024DEC000v131072), 5 minutes from a Kokiri Forest save in handheld mode on an M4 Pro Mac, withDM_PSO_MTL4=0(the iPad's path) andDM_PSO_PREPARE=1. These runs used our stack (main 6121fd7 with other changes, not main 0f510fd). In the runs with this list, the host also read theMrows with a change like Ocarina of Time: prewarm native mesh pipelines before attacks and enemies appear #1368's: 56 and 58 task / mesh pipelines prepared, none built in play. Without it, the second start built its 56 in play. Harness frames are left out of the counts (a periodic screenshot and stats frame of about 440 ms).pso_user.tsvregress.pynot run: profile data only.Docs