Skip to content

DM_GPU_CANARY=1 (opt-in): a fixed compute dispatch once a second, to tell a GPU clock fall from a heavier scene - #1709

Open
oliverames wants to merge 2 commits into
seedds:mainfrom
oliverames:gpu-canary
Open

oliverames wants to merge 2 commits into
seedds:mainfrom
oliverames:gpu-canary

Conversation

@oliverames

@oliverames oliverames commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Opt-in, off by default. Read on one device so far (below); the Mac test and one iPhone run are the only checks.

What and why

Mario Kart World 1.8.0 (#1383's build) on a physical iPhone goes from about 9 to 30 ms of GPU time a frame, a tight plateau, 30 to 50 s into a race: seen in six runs with HDR on and off, with the game's present interval and with DM_PRESENT_INTERVAL=display, with and without a scheduler declaration change, while the CPU stayed at a full clock and 55 to 63% on performance cores (and the ProcessInfo thermal state did not track it). cheemu21/plumber-racer-2 reports the same plateau on a cold phone. The game's GPU time moves with its scene and with the GPU clock, so it cannot say which of the two stepped.

DM_GPU_CANARY=1 submits a fixed compute dispatch (about 1 G dependent FMAs, no memory traffic) on the app's own queue once a second, after five seconds, and logs its GPU time: gpu canary #n at s s: GPU x ms, submit to done y ms. It does not move with the scene, so a step in it at the moment the game's GPU time steps says the GPU clock fell, and a flat canary says the scene got heavier. It shares the GPU with the game, so read the smallest of a few samples. It costs a few milliseconds of GPU a second. Without the variable nothing is created or submitted. Switch row in docs/env_switches.tsv; reading guide in docs/development.md.

Checked

  • Tests: new t_gpu_canary runs the real Metal path (mtl_init with the variable set): the start line is logged, three samples arrive in 8 s, GPU time positive and bounded (7.6 ms on an M-series Mac), built with make -C dm/gfx/tests GAME=0400C3F00006E000 t_gpu_canary. It caught a real bug while writing it (the timer source was a local and was released at once, so nothing ever fired); fixed. nvn_mtl.m also syntax-checks for the iPhoneOS SDK. dm/tools/tests/test_docs.py passes; the new switch is in docs/env_switches.tsv with its meaning (at the time env_switches.py --check reported DM_NVN_BC_ALIAS, which was missing from the table on main too; main's d8b2b64a1 has since added that row).
  • Games run, with title ID and version: Mario Kart World (0400C3F00006E000) 1.8.0, built from WIP: Mario Kart World 1.8.0 on iOS - rendering, memory and presentation #1383's tree with the same function, one run of 62 s with the canary on, handheld, HDR off.
  • Devices and OS: Apple-silicon Mac, macOS 27.3, for the test; iPhone 19,4 (iOS 27.3) for the run.
  • Measurements: the canary took 4.3 to 4.8 ms from 11 to 28 s, ramped over 4 s, then 13.8 to 17.9 ms (3.1 to 3.3 times) from 34 s to the end; the game's own GPU p50 went 7.5 to 8.9 ms, then 29 to 31 ms (3.3 to 3.5 times) at the same moment; the ProcessInfo thermal state stayed 1 and the CPU stayed at a full clock. So the GPU clock fell to about a third and the scene was not the cause.
  • scripts/regress.py check --games: not run: off by default, so every game runs as before without the variable.
  • Not checked: whether the canary's cost changes the game's frame time (about 4 ms of GPU a second on the phone, small next to a 16.7 ms frame; unmeasured); a second run; a run on another game or device.

Front ends

None: no front end has to act on this.

Docs

  • The reading guide is in docs/development.md and the switch in docs/env_switches.tsv; no game finding, so no symptom entry
  • A chain tool changed: dmsh_chain.py --header was run and dm/gfx/nvn/dmsh_chain.h is part of the change. Not applicable: no chain tool changed.
  • No game files, keys or data extracted from a game in the change or the attachments

@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation labels Nov 7, 2026
@oliverames oliverames changed the title DM_GPU_CANARY=1 (opt-in, not yet read on a device): a fixed compute dispatch once a second, to tell a GPU clock fall from a heavier scene DM_GPU_CANARY=1 (opt-in): a fixed compute dispatch once a second, to tell a GPU clock fall from a heavier scene Nov 7, 2026
@github-actions github-actions Bot added the game: 0400C3F00006E000 A game profile label Nov 7, 2026
@oliverames

Copy link
Copy Markdown
Contributor Author

Device read of the canary (iPhone 18 Pro class, 2P+4E cores, MKW 1.8.0, HDR off, thermal state 1 throughout), not fully tested:

  • Across two device runs the canary shows the GPU running 3.1 to 3.3x slower about 30 s into sustained MKW load, while ProcessInfo.thermalState stays at 1 and the CPU keeps its clock. So the slowdown is not visible to the thermal API.
  • In a third run (DM_PRESENT_HZ=30) the canary read 4.4 to 7 ms from 13 to 25 s, then 14 to 16 ms from 29 s on, the same step.
  • HDR on or off, game or display present interval, and DM_RT60 made no difference.

Conclusion: the canary separates a GPU clock drop from the game's own pacing, which is what it was added for. It does not explain the drop, so this PR is a diagnostic only.

… GPU time logged, to tell a GPU clock fall from a heavier scene

Mario Kart World on an iPhone goes from about 9 to 30 ms of GPU time a frame, a tight plateau, 30 to 50 s into a race, with the CPU
at a full clock, with HDR on or off, and with the game's interval or the display's. The game's own GPU time moves with its scene and
with the GPU clock, so it cannot say which. This dispatch does not move with the scene: a step in it at the same moment says the
clock fell. t_gpu_canary runs the real Metal path on the Mac (three samples in 8 s, 7.6 ms of GPU each).
oliverames pushed a commit to oliverames/nxnative that referenced this pull request Nov 10, 2026
…me on an iPhone, and the phone restarts some minutes later (Mario Kart World 1.8.0, iOS 27.3 beta); open

On an iPhone 19,4 the picture stopped 2 to 3 s into a race, 184 s after launch, with the overlay at 0 fps, the process alive, no [RED] line and no GPU error. Every thread waited and none spun. The state dump has the replay thread in iokit_user_client_trap under res_commit (pool_res, with g_mu held by its caller and g_res_mu across the call) and the GPU's completion queue waiting for a mutex in post_complete, with the main fence at submitted 206262 / completed 206256.

Ending the app did not end the process. The system's stack report 15 minutes later still has it with one thread, NVN queue replay, in an uninterruptible kernel wait with a read-write lock held and a kernel thread blocked on it, and the phone at 7.2 GB wired and 34 MB free. The phone then restarted itself.

It happened twice in four runs of one build (main 04811a6 with PRs seedds#1709, seedds#1695, seedds#1262, seedds#1599 and seedds#1713 merged): in that race, and an hour later at the title's demo with no button pressed, 353 s after launch, with the same thread state. Both came in a scene load (1,840 allocations added to the residency set in 678 commits in the 300 frames before the second). With main's 6f50de0 and 12a44e0 added, four races did not freeze, which is too few to say it is gone. The game makes about five residency commits a frame in a race. No fix: the entry says what was ruled out, what the host could change, and what to collect from the phone (SIGALRM for the host's dump, then systemCrashLogs). The game note links it under Open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 0400C3F00006E000 A game profile nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant