Skip to content

iOS: run depth/stencil regression checks on physical iPhones - #1153

Closed
oliverames wants to merge 1 commit into
seedds:mainfrom
oliverames:iphone-gfx-regression-runner
Closed

oliverames wants to merge 1 commit into
seedds:mainfrom
oliverames:iphone-gfx-regression-runner

Conversation

@oliverames

@oliverames oliverames commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What and why

The GPU tests under dm/gfx/tests only run on a Mac, so a depth/stencil fault that shows on an iPhone's GPU alone has no test to run there. This adds dm/ios/tools/test_gfx_on_iphone.py, which builds a small signed app around t_clear_ds.m, compiled unchanged, and with --run installs it on a connected iPhone, starts it and reads the result back. --generate-only stops after the Xcode project and --build-only after the signed app. It reads no game files or keys.

It came from testing Donkey Kong Bananza on an iPhone 17 Pro Max. The fixture covers the shared depth/stencil clear behaviour, so it can help check renderer changes other titles use. It doesn't change how any game renders.

I've cut this PR down to that one change. Earlier versions also carried a window_copy fixture, a push_capture fixture and a --measurement metal_conversion mode. Those are gone from this branch: window_copy came from #1140, which was closed, push_capture only exists in #1160, and the measurement probe belongs to separate HDR work. The old head (92b6d73) is kept on my fork as archive/iphone-gfx-runner-extras-20261110 in case any of it is wanted later.

The branch is now one commit on main fd8188c9 (head 2dba6cd1). clear_ds is the only fixture offered.

While trimming I moved what the tool decides into four small functions so it can be tested without a phone:

  • results.json is accepted only when it's a JSON object with this build's run ID and complete set. The same bundle ID keeps its Documents folder across installs, so an earlier run's report is still there until the new app writes its own. A copy that doesn't parse is polled again instead of ending the run.
  • A pass needs a zero exit code, at least one check, no failed check and the app's own passed field. Before, the tool trusted passed alone.

Checked

  • Tests: python3 dm/ios/tests/test_gfx_phone_runner.py (11 tests, new: which report is accepted, what counts as a pass, the names of what it builds, and that the support source list still matches the tests' Makefile). CI's tool tests job picks it up with the other dm/ios/tests files. python3 dm/tools/tests/test_docs.py (6 tests). The Python lint ratchet against main: 323 findings, all in the baseline, none new.
  • Build: --generate-only on main fd8188c9, then an unsigned xcodebuild of the generated project for generic/platform=iOS (Xcode 27.3, profile 040074A01BF12000). It compiled and linked.
  • Games run: none. The tool uses a profile for compile-time constants only.
  • Devices and OS: I haven't run this head on a phone. Nothing was signed, installed or launched, so --build-only's signing steps and all of --run are untested at this commit. The only phone run was on 2026-11-04 with the first version of the tool, on an iPhone 17 Pro Max (A19 Pro GPU, iOS 27.2): all 11 cases and 59 checks passed with Metal API Validation Enabled in the collected log, and the full --run path built, signed, installed, launched and pulled the matching report (Debug app, upstream renderer at 1d4b8e82). The report handling has changed since then and only has the unit test behind it.
  • No regress.py report: the change adds a tool and a test and doesn't touch the host, the shader chain or any game.

Front ends

None: nothing here is for a front end to act on.

Docs

  • docs/development.md (Continuous integration) says how to run the tool, what a pass means and what the test covers. There's no game finding or reference entry in this change, so docs_symptoms.py wasn't needed.
  • A chain tool changed: not applicable.
  • No game files, keys or data extracted from a game in the change or the attachments

@github-actions github-actions Bot added ios The iPhone and iPad app (dm/ios) docs Documentation labels Nov 4, 2026
@github-actions github-actions Bot added the nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) label Nov 4, 2026
@oliverames

Copy link
Copy Markdown
Contributor Author

Commit 54e15b37 adds --fixture window_copy and --fixture push_capture, with documentation linking the fixture sources in #1140 and #1160. clear_ds remains the default. Use --tree to select the checkout containing the requested fixture; a missing source fails before output creation or any device operation.

Both new choices passed project/header generation using the iPhoneOS SDK. Ruff and pyright report zero findings for the runner, all 11 documentation tests pass, and the missing-source check passes. The initial sandboxed generation failed on access to Xcode's compiler cache; generation succeeded with cache access.

The local runner previously exercised these choices on iPhone 17 Pro Max: 31 window-copy checks and 28 push-capture checks passed. Those results apply to their recorded fixture hashes, not the newer 32-check window-copy completion case. No new phone run occurred for this commit. The PR remains draft, and neither fixture establishes sustained gameplay, visible-screen output or a fur-shimmer improvement.

@oliverames

Copy link
Copy Markdown
Contributor Author

@seedds, new head edd9817d adds a separate raw-measurement mode for the Switch 2 games running on iPhone investigation. It uses #1168's standalone conversion probe, embeds the same 95 exact inputs, and records raw output without treating rounding as a regression pass. The iOS 26 app builds and signs, source/input hashes are checked, report-contract tests and Python/docs checks pass, and independent review found no actionable issues. It has not been installed or measured on iPhone 17 Pro Max. Existing fixture behavior remains separate; this stays draft pending physical measurement and hosted CI.

@oliverames

Copy link
Copy Markdown
Contributor Author

@seedds Flagging from my open-PR index of my 32 open PRs here: this one stays draft pending hosted CI and the remaining device verification. The description above is still the full picture, nothing new from me. Happy to rebase, split or retitle whenever review suits.

@oliverames
oliverames marked this pull request as ready for review November 5, 2026 22:15
@oliverames

Copy link
Copy Markdown
Contributor Author

@seedds Marking this ready for review now, so it stops waiting in draft on me. The remaining item from the description is still open: hosted CI and the remaining device verification. It can carry on alongside a review instead of ahead of it. The description stays the source of truth; happy to rebase, split or retitle whenever review suits.

@oliverames
oliverames force-pushed the iphone-gfx-regression-runner branch 3 times, most recently from 9f02a07 to 92b6d73 Compare November 10, 2026 03:12
…(dm/gfx/tests/t_clear_ds.m) on a connected iPhone and reads its result back

The GPU tests under dm/gfx/tests run on the Mac only, so a depth/stencil fault that shows on an iPhone's GPU alone had no test to run there. This tool builds a small signed app around t_clear_ds.m, compiled unchanged (a wrapper file renames its main and returns its two counters), with the support sources of the tests' Makefile (MTL_T_SRC), the profile header and the host's Metal shaders made for the iPhone SDK. With --run it installs the app, starts it and copies results.json and the test's log back from the app's Documents folder. --generate-only stops after the Xcode project and --build-only after the signed app. No game files or keys are read. clear_ds is the only fixture offered.

What the tool decides is in four small functions. accept_report takes results.json only when it is a JSON object with this build's run ID and complete set: the same bundle ID keeps its Documents folder across installs, so an earlier run's report is still there until the new app writes its own, and a copy that does not parse is polled again rather than ending the run. report_passed wants a zero exit code, at least one check, no failed check and the app's own passed field, with the counts as integers. fixture_path and app_path name the test source and the built app.

Checked on a Mac at this commit, on main fd8188c: dm/ios/tests/test_gfx_phone_runner.py (11 tests: the report cases above, the two names, and that support_sources still matches the tests' Makefile), which CI's tool tests job picks up with the other dm/ios/tests files; dm/tools/tests/test_docs.py (6 tests); the Python lint ratchet against origin/main (323 findings, all in the baseline, none new); --help; and --generate-only followed by an unsigned xcodebuild of the generated project for generic/platform=iOS (CODE_SIGNING_ALLOWED=NO, Xcode 27.3, profile 040074A01BF12000), which compiled and linked.

Not checked at this commit: nothing was signed, installed or run on a phone, so --build-only's signing steps and all of --run are as they were on 2026-11-04, when the first form of this tool ran clear_ds on an iPhone (the pull request's description has what that run showed). The report handling has changed since then and has only the test above behind it.

docs/development.md (Continuous integration) says how to run it and what a pass means.
@oliverames
oliverames force-pushed the iphone-gfx-regression-runner branch from 92b6d73 to 2dba6cd Compare November 10, 2026 13:21
@oliverames

Copy link
Copy Markdown
Contributor Author

Closing this. It only runs on a physical iPhone, and the checks it would add are better made on the Mac, where you work. Thanks!

@oliverames oliverames closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation ios The iPhone and iPad app (dm/ios) nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant