Skip to content

Pipelines a hybrid draw stands in for are listed in pso_user.tsv when first asked for (from HookLink) - #2018

Open
oliverames wants to merge 3 commits into
seedds:mainfrom
oliverames:pso-demand-record-20261109
Open

oliverames wants to merge 3 commits into
seedds:mainfrom
oliverames:pso-demand-record-20261109

Conversation

@oliverames

@oliverames oliverames commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What and why

pso_user.tsv lists the pipelines a device needs so the next start's prepare step builds them before the first frame. It took a pipeline's row only once the build had finished. When a hybrid draw stands in for a pipeline (it draws with its shader pair's pulling pipeline), the pipeline it wants is built in the background (pso_pend_demand). On HookLink's iPhone 18 Pro Max those builds took 5-50 s during play, with the game on two performance cores, against about 1 s in the prepare step. Castle Town asked for about 400 pipelines and finished about 60 a minute, so whatever was unbuilt when the run ended was never listed, and the next start did not build it.

demand_record (dm/gfx/nvn/nvn_mtl.m) writes the row when the hybrid draw first asks for the pipeline, once per key (at most 65,536 keys remembered). The later row from the built pipeline is the same line, and pso_user_record already drops duplicates. DM_PSO_DEMAND_RECORD=0 records a pipeline only once built, as before.

Ported from HookLink's commit 0ad4e1a (engine branch: https://github.com/dsi-hub/nxnative/tree/hooklink, project: https://github.com/dsi-hub/HookLink). Their commit sits on two others that are not ported here (archive-miss memory, keyed demand table), so I adapted it: the call goes beside main's pso_pend_demand in the hybrid branch and runs for every stand-in draw (theirs also ran when a build was already pending), without the archive-miss lines. Main has since gained an archive-miss memory of its own (g_pbase_miss, PR #1825). That doesn't change this PR: the row is written when the draw takes its stand-in, whether or not the archive search ran, and g_pbase_miss lives only in the process, so it lists nothing in pso_user.tsv. Added: a test, the reference entry (docs/reference/shaders.md), the switch row and a line in the game's note.

Checked

  • Tests: make -C dm/gfx/tests GAME=0400486024DEC000 check_t_pso_demand_record passes (the new t_pso_demand_record, run twice: default and DM_PSO_DEMAND_RECORD=0). It calls demand_record twice for one key and checks one row, a second key gets its own row, and with the switch at 0 no row is written. With the pso_user_record call in demand_record removed, both default-run checks fail; restored afterwards. The test does not draw, so the call site in the hybrid path is not run by any test: that needs translated shader libraries with pulling variants. dm/tools/tests/test_env_switches.py and test_docs.py pass.
  • Games run: none. We did no phone run and did not play the game. The 5-50 s build times and the Castle Town counts are HookLink's, from their iPhone 18 Pro Max; we have not reproduced them.
  • Devices and OS: none for this change. The test ran on a Mac (Apple silicon).
  • Not measured: how many extra rows a session adds to pso_user.tsv, or how much longer the next start's prepare step takes with them. The full GPU test run (make -C dm/gfx/tests run) and a full host build were not run.

Front ends

No front-end interface change.

Docs

  • The finding is in docs/reference/shaders.md ("A pipeline a hybrid draw stood in for listed only once built") and the game's note, and python3 dm/tools/docs_symptoms.py was run
  • No chain tool changed
  • No game files, keys or data extracted from a game in the change or the attachments
  • docs/env_switches.tsv has a row for DM_PSO_DEMAND_RECORD.

@oliverames
oliverames force-pushed the pso-demand-record-20261109 branch from 1e1e0d5 to c7da101 Compare November 10, 2026 00:49
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation game: 0400486024DEC000 A game labels Nov 10, 2026
@oliverames
oliverames force-pushed the pso-demand-record-20261109 branch from c7da101 to 6cec172 Compare November 10, 2026 02:36
… first asked for, so the next start's prepare step builds it (from HookLink)

pso_user.tsv took a pipeline's row only once its build finished. While a hybrid draw drew with its shader pair's pulling pipeline, the pipeline it wanted was built in the background; on an iPhone 18 Pro Max (HookLink's measurement) that took 5-50 s with the game running, against about 1 s in the prepare step. Castle Town asked for about 400 pipelines and finished about 60 a minute, so what was unbuilt when the run ended was never listed and the next start did not build it.

demand_record writes the row when the hybrid draw first asks for the pipeline, once per key. The later row from the built pipeline is the same line and pso_user_record drops it. DM_PSO_DEMAND_RECORD=0 keeps the old order. Added t_pso_demand_record, the reference entry, the switch row and the game note line.

Ported from HookLink's commit 0ad4e1a (https://github.com/dsi-hub/nxnative/tree/hooklink, project https://github.com/dsi-hub/HookLink), without its archive-miss memory (b3471d5) and keyed demand table (b15d36f): main has an archive-miss memory of its own (g_pbase_miss, PR seedds#1825) and no keyed demand table, so the call goes next to main's pso_pend_demand and runs for every stand-in draw, not only when no build is pending.

Co-authored-by: dsi-hub <260802842+dsi-hub@users.noreply.github.com>
@oliverames
oliverames force-pushed the pso-demand-record-20261109 branch from 6cec172 to 2e74684 Compare November 10, 2026 16:30
@akratch

akratch commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

I prepared a test-only follow-up on this exact head. Production and chain are unchanged.

The previous direct-helper test passed even if hybrid lookup omitted or misplaced the record call. The new fixture enters actual pso_get with synthetic Metal functions and real cached stand-ins, gates the production compile queues, and checks both newly queued and independently pending demands. An exec child validates canonical rows through warm_parse before compilation finishes; released compilation replaces the stand-ins and preserves row deduplication. Separate cases check recording off and the 65,536-entry boundary.

Checked head 32484bec8ec33fc169bb2eec640670170aa0788b passes on/off/cap (56/42/53 parent checks plus fresh-process checks), under both encoder flags with Metal validation. Source-copy controls fail both flags: removing call-site recording gives four parent/two early-child failures; recording only newly queued demand gives two parent/three early-child failures. No bug-restoring production switch was added. Focused LLVM23 has zero fixture findings/zero new above unchanged baseline223; six final-head documentation tests pass.

This proves actual lookup placement and process-reload visibility, not a game draw, fresh-process prewarm compilation, power-loss durability or the reported iPhone performance. Flag variation does not independently test replay-thread concurrency. Apple M4 Max / macOS27.2 / Xcode27.2. All inputs synthetic. Feel free to fold these cases or the four-file commit into this PR.

Oliver Ames and others added 2 commits November 12, 2026 11:52
Only the game note's "Found on this game" list conflicted; both sides kept.
The earlier fixture called demand_record directly and could not detect a missing or misplaced pso_get call site. Compile synthetic Metal functions, cache real stand-ins, and suspend the production background queues while real lookups encounter both new and independently prequeued demands. An exec child checks canonical rows through warm_parse before either compile can run; completion replaces the stand-ins and preserves identical deduplicated rows.

Cover recording disabled and the 65536-entry admission boundary, including eventual recording beyond that early-demand limit. With API validation, both encoder configurations pass on/off/cap cases. Removing call-site recording fails the matched control, as does limiting it to newly queued demands. Six docs tests pass and focused LLVM23 analysis adds no findings. This is test/docs only: no game assets, gameplay timing, power-loss durability or fresh-process prewarm compilation claim.

Co-authored-by: Adam Kratch <70729875+akratch@users.noreply.github.com>

Co-authored-by: akratch <70729875+akratch@users.noreply.github.com>
@oliverames

Copy link
Copy Markdown
Contributor Author

Thanks @akratch, these are much better tests than mine: mine passed even with the record call missing or in the wrong place. I've folded your commit into this PR (f6eac81, with you as co-author) and merged main 1992edb in. Only the game note's list conflicted. On an M2 Pro, check_t_pso_demand_record passes in all three modes, with Metal validation on: 54 checks with recording on, 40 off, 51 at the cap. test_docs, test_env_switches and test_gfx_tests_makefile pass too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400486024DEC000 A game nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants