[computerguy526]

Replaces our earlier #2221.

Symptom
Every Ocarina of Time run compiled the same few dozen to few hundred bases again, although the run before had compiled them. Area loads stalled on them: a draw waits in pso_get → arch_capture → MTLCompiler. Each run ended with one of these:

pipeline archive at exit: 425 pipelines not written yet, not all written in time: those left are built again at the next start (968 ms)

[RED] graceful exit not done in 1.5 seconds, exiting at once

The second has no archive line at all.

Cause

• Since Timer flushes wait during any prewarm, so the start after a cold start does not compact (Ocarina of Time 762 s → 3.3 s) #1879, the timer's flushes wait for the end of the prewarm. A period is written only when it is full (ARCH_PERIOD_MAX, 2,000) or at exit.
• Ocarina of Time's prewarm (31,663 lines after the first frame, on 5 workers) outlives a play session, so everything a session compiled waited for the exit.
• The graceful exit (SIGTERM) gave the archive what was left of 1.5 s. A write takes 1.8–4 s per 900 bases.
• When a draw was still compiling its pipeline, dm_gfx_suspend waited on that draw, and the 1.5 s fallback ended the process before any write. A harness that stops a run for "no new frame" sends SIGTERM at exactly that moment.

What changes

• nvn_mtl.m: during a prewarm that is not compacting, the timer's flush hands the period over once it is DM_PSO_ARCH_WARM_FLUSH_MS old.
  • The default is 60 s, and 0 turns the flush off.
  • Each later flush waits twice as long, up to 16×: 120 s, 240 s, and so on up to 16 min. A long cold start therefore adds about one file per doubling, not one per 10 s, so Ocarina of Time (0400486024DEC000 v131072, Mac M4 Pro): every start spends about 5 minutes in the prepare step, 309 s on a warm start with nothing compiled #1747's 80 files do not come back.
  • While captures wait, the timer looks again every 10 s.
  • The flush is off while a compaction runs.
• hle_lifecycle.c, the graceful exit:
  • The archive starts writing first (dm_gfx_quit_begin), before the drain.
  • It gets DM_EXIT_ARCH_MS beyond the 1.5 s: 8 s on the Mac, 3 s on iOS, where the app is ended about 5 s after its termination callback.
  • If graphics are not suspended within 1.5 s, the fallback still waits for the archive's write within the same deadline. One claim flag makes sure only one of the two paths writes, runs the exit's continuation (then) and ends the process.
  • A draw that is still compiling its pipeline holds its period open. That period is written only if the compile returns within the budget; the periods before it are written either way.
  • A hard deadline at 1.5 s + the budget + 0.5 s ends the process whatever happens.
  • Files are still written as .tmp and renamed after their index, so a write the deadline cuts short leaves the files already there valid.
• nvn_mtl.m pso4_init: a start removes a <file>.mtl4a.tmp that a stopped write left. Main kept these leftovers, and the hard deadline can now end the process inside a write.
• The normal quit path (atexit: ⌘Q, closing the window) already waited 10 s and is unchanged.
• Reference entry: A run that ends in its prewarm loses what it compiled, plus its line in the symptom index, the archive paragraph in docs/development.md, and rows in docs/env_switches.tsv for DM_PSO_ARCH_WARM_FLUSH_MS, DM_EXIT_ARCH_MS and the test-only DM_TEST_STUCK_DRAW.

Tests
make -C dm/gfx/tests GAME=0400486024DEC000 check_t_pso_archive runs t_pso_archive and test_pso_archive.py. The new cases each use fresh processes.

case	with the change	without (main's nvn_mtl.m, hle_nvn.c, dm.h, hle_lifecycle.c)
five new bases in a prewarm, the timer's flush with DM_PSO_ARCH_WARM_FLUSH_MS=300, then _exit	one file; the next start takes all 5 and compiles none	FAIL: no file, so all 5 are compiled again
SIGTERM in a prewarm while a draw holds the graphics layer (dm_gfx_enter with no leave)	one file; all 5 taken at the next start	FAIL: no file
SIGTERM in a prewarm, no draw held	one file	passes (5 bases fit in the old 1.3 s)
a torn write's .tmp and an index without its file, beside a file that is not an archive	leftovers removed, the bad file skipped, all 5 bases from the good file	FAIL: the .tmp stays

Totals: t_pso_archive in process, 355 checks, 0 failed; test_pso_archive.py, 11 tests OK with the change and 3 of 11 failing without it. test_docs and test_env_switches pass.

Local CI

• The targeted tests pass (above): t_pso_archive, test_pso_archive.py, test_docs and test_env_switches.
• ci_local.sh --full, this commit merged into main f31d607: every stage passes, including the host tests, ASan, TSan and the iOS build, except one tool test. lint-c's findings and the CPU-only gfx test failures are only in files this change does not touch, and are the same on main.
• The tool test is test_bench's test_cli_relative_paths, which this change does not touch (bench.py and the test are the same as on main). It is flaky here: it failed in 1 of 10 runs of the branch alone.

In game
Ocarina of Time v131072, M4 Pro, macOS 26.6, 2026-11-10, on our fork's host with this change (oot_r13af_host), not on main. These are warm-up runs on the walkcam route, each ended by SIGTERM, each starting from the archive the one before left. Logs are in Zelda_OoT/0400486024dec000-work/runs/<run>/.

	before: r13w2101343_*, r13w3101353..101417_* (same chain, without the change)	with the change: r13w4101505_{fog,hyrule-field,kakariko,kokiri-house,zora-domain}
writes during play	none: the prewarm runs the whole session	one per run at about 60 s, e.g. 00152-713.mtl4a, 00153-1677, 00155-447, 00157-358, 00159-52
at exit	r13w2101343_hyrule-field: 425 pipelines not written yet … (968 ms); r13w2101343_kakariko: graceful exit not done in 1.5 seconds and no write; about 500 pipelines unwritten per run	written in 0.5–6.0 s: 522 (Hyrule Field, 6.0 s), 209 (Kakariko, 3.9 s), 71, 28
prepare before the first frame	3.8–4.0 s	1.4–4.4 s

What the next run takes from the archive was checked separately on 2026-11-10 (runs kmA1 / kmA2, which have since been deleted; the numbers are from our notes):

• kmA2 found all 54 of the bases kmA1 had written and later asked for.
• Its 32 misses were all bases that were never written.
• No key mismatched.

So what a run writes, the next run uses. The before runs compiled 57–167 bases at each area load, and area loads at Hyrule Field, Zora's Domain and Kakariko stalled for more than 20 s.

Not measured with the change: the area-load compile counts and stall times. Later verify runs still compiled 33–167 bases per area load. The keymatch runs show those are mostly content the earlier passes never reached, not lost writes. I don't claim a stall number for the change.

Not covered yet:

• Most of a run's compiles are now kept, but not all. The fog run (r13w4101505_fog, the first of the series) still ended with 1,378 pipelines unwritten after 9.3 s: its second period had not reached the 120 s flush when the run ended, and compute-heavy periods take tens of seconds to write.
• A few pipelines (0–8) captured after the exit's hand-over stay unwritten.

Regression

• regress.py impact HEAD~1..HEAD picks The Legend of Zelda: Breath of the Wild, Pokémon Scarlet and The Legend of Zelda: Tears of the Kingdom. All three use the pipeline archive, and BotW and TotK also use the prepare step. Hades is not affected, and 163 profiles have no usage manifest. None of the three games was run for this PR (I don't have them).
• dm/host/dm.h and dm/host/hle_lifecycle.c have no row in impact_map.tsv. They change the graceful exit for every game: a SIGTERM exit with unwritten captures can now take up to 10 s instead of 1.5 s (2.0 s with DM_EXIT_ARCH_MS=0). An oe::ExecuteProgram hand-off goes through the same exit, so it waits the same way.
• A prewarm longer than 60 s writes up to about log2(prewarm / 60 s) more run files, which brings the next tier compaction (PSO_ARCH_COMPACT) that many files closer.
• DM_PSO_ARCH_WARM_FLUSH_MS=0 turns the prewarm flush off. DM_EXIT_ARCH_MS=0 gives the main exit path the old 1.3 s, but the timings are not quite the old ones: a fallback at 1.5 s hands the archive over without waiting (dm_gfx_quit(0)), and the hard deadline is at 2.0 s.

Co-authored with UncraftableXD.