[FifteenPen]

What and why

Since e45694b (#2066), a game uniform read at a fixed offset is a plain load. On Nintendo Switch Sports Resort (04008E001F098000, v65536) that makes Metal's backend take minutes and gigabytes for some material pipelines. The prebuild compiles them side by side and MTLCompilerService runs out of memory:

• In the log: [RED] pipeline prebuild: the base of 0eb245e18bea1bac + fbe6bb186641245c failed: Compilation failed due to an interrupted connection: XPC_ERROR_CONNECTION_INTERRUPTED, 140 failed pairs in one start.
• Crash reports: 32 MTLCompilerService crashes in one evening, each an operator new failure under llvm::CloneBasicBlock.
• The archive loses whole files: the period those compiles were captured in fails to write: [RED] pipeline archive: writing …/p00130.mtl4a failed: Failed to generate machO:warning: units in '…/com.apple.gpuarchiver/…/air64_v29' are already packed. Two files (about 3,400 bases) were lost in one start, so the next start built them again. The game compiled pipelines for minutes at every launch.

Cause. These pixel shaders read integer flags from a game uniform (as_type<int>((*spvDescriptorSet0.fp_c5)._m0[40].x), 455 such reads in fbe6bb186641245c) and test them at each level of branches nested 36 deep. With the comparison gone, the backend takes every read of one word as one value, and its instruction selection blows up. I narrowed it on the shader's text:

• The shader as the tree before e45694b wrote it: 0.6 s.
• The same source with only its c5 comparisons taken out: slow.
• The same source with only the comparisons of c6 taken out (read as floats): 0.6 s.
• The front end's IR has the same shape either way (largest block 239 lines), so the cost is in the backend.

Fix. dmsh_rewrite.flag_reads (called after select_guards in dmsh_build.py) reads a game uniform (c3 and up) that the shader takes as an integer at a fixed offset through a volatile pointer:

as_type<int>(dm_vl((*spvDescriptorSet0.fp_c5)._m0[40]).x)

Each read stays a load of its own, as the compared read was, with no size table and no comparison, so #2066's change stands. Float reads and the driver's banks (c0 to c2) are left alone: they did not slow the compile, and dmsh_fetch reads the float reads' text. Chain change: 3695a7c258bb.

Measured

One render pipeline per pair, built from the shader build's own .metallib files with newRenderPipelineStateWithDescriptor, the Metal function cache emptied before each. M4 Pro, macOS 27.2. Another build was running on the machine during these, so the main column is somewhat inflated; the first pair took 78.9 s on the idle machine.

Vertex + pixel shader	main f20f650	this PR
0eb245e18bea1bac + fbe6bb186641245c	85.6 s	1.1 s
00a37c8963c861be + 8bdcdee8e27bd271	141.6 s	1.4 s
03e62a4a5e7ece89 + 0d57ffc025e0df8f	not finished after 150 s, compiler at 3.9 GB	3.3 s
03e62a4a5e7ece89 + 6bce362cc76ee226	not finished after 150 s, compiler at 3.7 GB	3.0 s
07275c719cb102f9 + a0590cfb10248864	81.7 s	0.9 s
0fe3dadc985ff916 + dea5bd6c61bfddd3	84.2 s	1.1 s

These are the six pairs the game's prebuild logged as failed.

Checked

• test_dmsh_flag_reads.py (new): the integer reads of game slots are wrapped; float reads, a driver bank and a computed offset are not; a shader with none is unchanged; the result compiles with the build's Metal flags.
• The other dm/gfx/tests Python tests pass (53 of 54 run this way; test_pso_archive.py needs its executable argument and was not run).
• test_docs.py, test_env_switches.py, test_architecture.py and the symptom index pass.
• The game's whole shader map builds with the rewrite: 11,059 of 11,059. That build was on a tree with this change plus the game's other open PRs, not on this branch alone.

Not checked

• GPU cost. A volatile load is not merged with another read of the same word. I have not measured what that costs in a frame, on this game or on Donkey Kong Bananza, where The uniform bounds guard (26062f9e) halves Donkey Kong Bananza's frame rate on an M4: 55 → 30 fps #2066 measured the comparisons. If it shows there, the rewrite could be limited to reads that feed a comparison.
• A full prebuild of the game after the fix. The six pairs above are measured; a whole start with the archive filling is not yet.
• Other games. Every game's shaders change with the chain; none was run.
• The host side. One crashed compile still loses its whole archive period, and a quit during the prebuild drops what is not written within 10 s. This PR removes the crashes' cause here, not that.

The game's profile is in #2107.

[seedds]

Thanks, this found the same cause as #2372, which landed on main as dfc5a15. That change makes a uniform word volatile only where it is tested for equality against constants more than once, rather than every integer fixed-offset read: on older-chain MSL your form would change 32 to 56% of shaders, and #2372's under 1%, which keeps more of #2066's GPU saving.

Before closing this: could you rerun your six pairs (0eb245e18bea1bac + fbe6bb186641245c, 00a37c8963c861be + 8bdcdee8e27bd271, 03e62a4a5e7ece89 + 0d57ffc025e0df8f / 6bce362cc76ee226, 07275c719cb102f9 + a0590cfb10248864, 0fe3dadc985ff916 + dea5bd6c61bfddd3) on main's chain? #2372 rewrites ==/!= tests against constants only, so flag checks written as bit tests ((w & m) != 0) would not be covered. If any pair is still slow, its form would show what to add.

[FifteenPen]

Reran the six pairs on main 6815810 (with #2372, dfc5a15): the 11 shaders translated from Switch Sports Resort's packages on main's chain, one render pipeline per pair from the build's .metallib files with newRenderPipelineStateWithDescriptor, Metal's function cache emptied before each, M4 Pro, macOS 27.2. Main's pixel shaders now declare dm_border_off (function constant 216), so the functions were taken through newFunctionWithName:constantValues: with no constant set (front includes that specialization).

Vertex + pixel shader	main f20f650 (this PR's table)	main 6815810
0eb245e18bea1bac + fbe6bb186641245c	85.6 s	0.4 + 0.5 s
00a37c8963c861be + 8bdcdee8e27bd271	141.6 s	0.5 + 0.5 s
03e62a4a5e7ece89 + 0d57ffc025e0df8f	not finished after 150 s	0.9 + 2.0 s
03e62a4a5e7ece89 + 6bce362cc76ee226	not finished after 150 s	0.9 + 1.9 s
07275c719cb102f9 + a0590cfb10248864	81.7 s	0.4 + 0.5 s
0fe3dadc985ff916 + dea5bd6c61bfddd3	84.2 s	0.3 + 0.4 s

(front + back.) None is slow on main. The pixel shaders carry #2372's dm_uword reads (259 to 429 sites each), and none of them tests a flag as a bit test ((w & m) != 0), so there is nothing to add for these. Fine to close this in favour of #2372.

Not run: the game's whole prebuild on main.

[seedds]

Closing in favour of dfc5a15 (#2372). Thanks for the rerun: all six pairs now compile in 0.7–2.9 s on main (front + back), none of their pixel shaders tests a flag as a bit test, so there is nothing to add. Thanks also for tracing the cause.