Repository navigation
Description
Building on @jthrilly's analysis in #272: one translator and one host serve every profile, but each of us can only test the games we own, and CI has no game files and no Metal 4 (#112). So a change that is well tested on its own games can still break others. Examples: c3a0880 broke SMT V and Yoshi (#173, #218, #272); my prepare step hangs on Odyssey (#294, #276); my offline pipeline archive fails on TotK (#283).
I'm pausing my own feature work (Arkham Knight, 4K) to help put a foundation in place first. Nothing here needs a feature freeze. Every step below is a small PR that stands on its own, keeps the current behaviour, and is proven before it lands.
What the numbers say (upstream/main, last ~460 commits)
- Merge conflicts sit in a few files.
docs/development.md(49 PRs), the generateddocs/agents/symptoms.md(47), the games table (41),dm/gfx/tests/Makefile(36) and its.gitignore(28), plus 80 commits that edited the game count line inAGENTS.md. 28 of 58 open PRs are conflicting. - The host only accepts libraries built by its exact chain version.
DMSH_CHAINwas bumped 17 times in 15 days. Every bump rebuilds every game and moves every game onto an unvalidated translator. A cold rebuild of TotK takes 45 min for the Mac, 42 min for iOS and 7 min for the pipeline archive. The driver segment layout is written down twice:nvn_mtl.manddmsh_build.py. nvn_mtl.mhas 10,617 lines and 1 MB (~290k tokens) (10,961 on Oct 27, 11,178 on Oct 28, 11,944 on Oct 31). It holds about 780g_*globals and 125getenvcalls; 40 tests include it, and it was touched in 144 commits and 75 PRs. An agent has to load all of it for a small change.- Only 14 of 63 gfx tests run in CI. No test works at the frame level.
- There are 229
DM_*variables, 66 of them undocumented. The 39 profile keys, by contrast, are all documented.
What stays
These are the right decisions and remain:
- one translator for every game;
- profiles with every fixed address gated by build ID;
- the per-service host;
- the record format with its static asserts;
- fail-soft sidecars and the
[RED]discipline; - the content-addressed Metal compile cache;
- the docs culture, including
test_docs.py.
The problem is coupling and centralisation, not these decisions.
Steps (each one a PR; the order puts safety before refactoring)
Conflicts out of the way
- Issue forms, PR template, CONTRIBUTING.md (Add issue forms, a pull request template and CONTRIBUTING.md #306, landed as 81d53bf)
- gfx tests: the run list and
.gitignorecome from the sources (Take the gfx tests' run list and .gitignore from the test sources #307, landed as 6c4a81d) - Docs de-conflict: drop the game count from
AGENTS.md(test_docsderives it), regeneratesymptoms.mdin CI, one file per reference entry for new findings (the game count: AGENTS.md states no game count; test_docs keeps it out #398, landed as 3cf4c2d; reference entries and the index merged with git's union driver, guarded bytest_docs: Merge docs/reference and the symptom index with git's union driver #400; README's gallery written from a table: README's games gallery is written from docs/images/gallery.tsv #401, landed as 9c52eaa; the symptom index regenerated onmainby CI: The symptom index is regenerated on main by CI, so a reference entry no longer needs it in the PR #472; one file per entry left out, see The symptom index is regenerated on main by CI, so a reference entry no longer needs it in the PR #472)
A safety net that knows every game
-
scripts/regress.py(scripts/regress.py: check a change on every local game, building on #272's shader regression check; the developer loop #328, building on Catch translator and host changes that break games nobody on the PR owns #272's shader regression check; first results: it confirmed the pso_offline: metal-tt fails on ToTK's pso_pairs.tsv pairs whose pixel shader reads an output the vertex shader does not write #283 fix on TotK and found Hades spins on the offline nifm request: ~952 million IsNetworkRequestOnHold calls in 60 s #334 on Hades), run locally on the games you own. It reports, per game, the translator diff (shaders newly failing or with changed MSL, by hash), boot smoke (frames reached, crash or hang, new[RED]lines, prepare time, frame times) and screenshot hashes. The report holds only numbers, hashes and shader keys; no game data leaves the machine. - One documented developer loop for people and agents: a single page (docs/agents) that lists every tool and when to use it (build and run, the GPU diagnostics from GPU measurement tools: a profiled frame in every n, A/B phases in one run, GPU captures, and gpuprof.py #225 and the dumps,
bench.py,regress.py, what CI covers and what runs locally), with a "before you open a PR" checklist tiered by what changed.AGENTS.md, CONTRIBUTING.md and the PR template point to it. It comes with theregress.pyPR (scripts/regress.py: check a change on every local game, building on #272's shader regression check; the developer loop #328). - Usage manifests (regress.py usage / impact: which games a change can reach, from per-game usage manifests #329, landed as b3f97ef): each profile lists the NVN functions, services and shader features its game uses (names and counts only).
regress.py impactthen tells a PR author which games their change can reach, including games they don't own. -
DM_NVN_CAPTUREplus a replayer (design: Design: capture a few frames per game and replay them against new host code (DM_NVN_CAPTURE + replayer) #402): capture a few fixed frames per game once, then replay them against new code in seconds. Captures stay on the owner's disk; the repo keeps only image digests. (step 1, one frame: Capture one NVN frame and replay it against new code (DM_NVN_CAPTURE, nvn_replay), step 1 of #402 #473, landed as d9c2d5e, with atex_fitview fix the replay found. Open, in Breath of the Wild's note: fresh replays of one outdoors frame differ from the live frame and from each other in 3 to 124 pixels of two G-buffer targets, with segmented submission on or off. The variation follows the outcome of the G-buffer pass's depth-equal tests and points to the GPU (5d2bf73). It stays open in the note. Step 3,regress.py captureandregress.py replay, landed as 8325eef: one frame per game, and a candidate beyond twice the capture's noise is aREGRESSION. Next: several frames with pool deltas, step 2 of Design: capture a few frames per game and replay them against new host code (DM_NVN_CAPTURE + replayer) #402) - Reproducible bug reports without sharing game data:
DM_RECrecords a player's inputs with timestamps inDM_KEYSsyntax. With the reporter's save and the same build, that replays the path to the bug. (Record player input as DM_KEYS steps (DM_REC), for bug reports and smoke routes #448, landed as 96820ed, withDM_KEYS=@<file>for long recordings. Replay is frame-exact since 710559c.)- A diagnostic bundle holds the log, the build and environment, and the pass list of the broken frame (shader keys, formats, draw state), with automatic per-pass findings: white or black output, NaN, flicker. It never contains textures, shaders or other game assets, and the bug report form asks for it (the form itself is in Add issue forms, a pull request template and CONTRIBUTING.md #306). First part, the log, build, device and settings without the pass list: scripts/bug_report.py: the logs, versions and settings a bug report needs, in one zip without game files #465, landed as f96d01e.
- Nightly Metal tests on Apple silicon (all 63 gfx tests and the translator GPU tests), on a self-hosted runner or a scheduled job on the maintainer's machine.
A reproducible setup
- One command each for the tools: a Brewfile,
requirements.txtandglobal.json, plus Dependabot for the Actions and a check that the committed translator matches its lock (Add Brewfile, requirements.txt, global.json, dependabot, translator l… #322, by @PrashantJi02)
Code quality checks in CI (free on GitHub's runners for public repos; each one is a ratchet, so nothing goes red today, only regressions do)
- ASan, UBSan and TSan over every host and CPU gfx test, plus clang's static analyzer (host and texture tests: CI: ASan, UBSan and TSan over the host and CPU texture tests, as a ratchet #412, landed as ce99604 with root fixes for hle_audren.c: misaligned loads while capture() probes the command header layout (UBSan) #409, hle_audio.c: a->au written outside g_aos_lock while the watchdog reads it (TSan) #410 and hle_thread.c: host_work, mport and qos written without G after the thread is in g_all (TSan) #411; the CPU gfx tests follow, now that Take the gfx tests' run list and .gitignore from the test sources #307 has landed; the static analyzer landed with Lint C and Objective-C in CI: a clang-tidy ratchet and a clang-format report #425 as a345f5b)
- Proposed by @relay4 below:
implicit-conversionandlocal-boundsin the CI's ASan and UBSan builds, and a separate integer target for buffer and count arithmetic. LeakSanitizer, RealtimeSanitizer and TypeSanitizer as local audit tools with Homebrew Clang, not in CI. The RealtimeSanitizer finding, logging in the audio callback, is in audio: move callback diagnostics to the watchdog #731 by @thehobbit85.
- Proposed by @relay4 below:
- A pre-commit hook with the fast checks (game files and keys refused, the lint ratchet with CI's pins,
test_docs, theDM_*registry, the chain header), somaindoesn't go red after a push (A pre-commit hook runs CI's fast checks before each commit #407; the lint ratchet is on the dev loop's pre-PR list since 9333825) - ruff and pyright for the Python toolchain (
dmsh_build.py, the NSP script, the split) (Lint and type-check the Python tools in CI with a ratchet (ruff, pyright) #323, landed as 7f5ef41) - Coverage for the host, the CPU gfx tests and the Python tools, as a job summary with diff coverage per PR (Codecov becomes optional once the owner installs its app) (CI: line coverage of the host, CPU texture and tool tests, with diff coverage per PR #439, landed as 68db795: host 61.7%, CPU textures 85.4%, Python tools 51.9% at landing. The translator tests stay out of it for now.)
- CodeQL for Python and C: no workflow needed. The owner turns on CodeQL's default setup (Settings → Code security), which analyses Python, and C/C++ without a build step. CodeQL doesn't read Objective-C, so the
.mhalf of the NVN layer stays with clang-tidy (Lint C and Objective-C in CI: a clang-tidy ratchet and a clang-format report #425). Not enabled on 2026-10-31: the Actions workflow list has no CodeQL entry, and no CodeQL check runs onmain. - Fuzzing of the file parsers (NSP/NCA/NSO and the shader split's container readers), with synthetic seeds only (Fuzz the file parsers nightly, with a regression corpus in the tool tests (found 25 bugs: #451–#454) #455, landed as 7a3a875: five harnesses, nightly; found 25 bugs, filed as NSO loaders: an 8 GiB allocation from a 256-byte file, and a silently short image (fuzzing) #451–nacp and romfs: struct.error on files cut short (fuzzing) #454; NSO loaders: an 8 GiB allocation from a 256-byte file, and a silently short image (fuzzing) #451 fixed in 3950976, one NSO loader for every tool that checks a module as Horizon does; shader_split: runaway recursion on a self-containing SARC, and readers that raise instead of red() (fuzzing) #452 in 625a260, uepak: multi-GB allocations from pak sizes, and raw exceptions on malformed paks (fuzzing) #453 in 966f201, nacp and romfs: struct.error on files cut short (fuzzing) #454 in 713ecaa from nacp, romfs: reject a control.nacp or RomFS image cut short within their contracts (#454) #605 by @mrkayhyun)
- clang-tidy~~, plus clang-format on changed lines only (no reformatting of existing code)~~ (Lint C and Objective-C in CI: a clang-tidy ratchet and a clang-format report #425, landed as a345f5b: a clang-tidy ratchet with the static analyzer that only shrinks, 273 findings as the baseline at landing. It found Four small host bugs from clang's static analyzer (TOCTOU on a controller and the frame cap, two unchecked null pointers) #424. The clang-format part is dropped.
.clang-formatcannot state the house style, and a root.clang-formatwould let editors reformat dense lines on save.) - Work the baselines down once the checks are in: the lint ratchet's findings, clang-tidy's baseline (Lint C and Objective-C in CI: a clang-tidy ratchet and a clang-format report #425), the sanitizer ignorelist and
tsan.suppentries that are not test fixtures (CI: ASan, UBSan and TSan over the host and CPU texture tests, as a ratchet #412's real-bug suppressions are already gone: ce99604), the fuzzer's expected failures (Fuzz the file parsers nightly, with a regression corpus in the tool tests (found 25 bugs: #451–#454) #455; all rows are gone since 713ecaa, andknown.tsvis empty). One small PR per finding or file, each fixed at its cause; the ratchets keep the numbers from growing back. - Rule for every PR: a bug found along the way gets its own issue or PR instead of riding along (followed for the sanitizer findings: hle_audren.c: misaligned loads while capture() probes the command header layout (UBSan) #409–hle_thread.c: host_work, mport and qos written without G after the thread is in g_all (TSan) #411 and t_hid: pass a three-axis rotation to dm_motion_set_switch (read past the array) #443, the analyzer's: Four small host bugs from clang's static analyzer (TOCTOU on a controller and the frame cap, two unchecked null pointers) #424, fixed in 18d65b3, the fuzzer's: NSO loaders: an 8 GiB allocation from a 256-byte file, and a silently short image (fuzzing) #451–nacp and romfs: struct.error on files cut short (fuzzing) #454, all fixed (3950976, 625a260, 966f201, 713ecaa), a stale generated header in the host tests: Host tests regenerate the game profile header instead of keeping a stale one #466, landed as 201dfd1, and an ASan finding in
t_fpslock: t_fpslock: copy the Frostbite names literal by its own length (ASan on main) #648, landed as 9474a63) - Flaky tests fixed at their cause, starting with the audio device-stall check (Wait for the restarted AudioUnit in the audio device-stall test #317, landed as b7654d4) and the Mac launcher's layout check (test_launcher_storage: the Mac window's layout check is flaky on CI (fixed 50 ms waits) #445, fixed in e8f3a8f)
Decouple the chain
-
dmsh_abi.has the single source for the driver layout and binding kinds, read by bothdmsh_build.pyand the host, with a cross-check test (dmsh_abi.h: one definition of the shader library ABI for the build and the host #391, landed as 03628c8: identical host objects, all 19,592 BotW shaders unchanged; reviewing it found a binding-kind mismatch in theDM_NVN_RT_RANGEcheck, fixed in 2f1a044) The host loads libraries by ABI compatibility.Dropped: the host is built into the IPA or Mac app, and the NSP script builds it and the shader libraries in the same run, so a new host never meets old libraries in practice. The real cost of a chain change is rebuilding the shaders, which the fingerprints below address.- Per-pass fingerprints in
dmsh_build.py, so a tool change rebuilds only the shaders whose output changes (dmsh_build: a comment edit no longer translates every shader again (CODE_FP from the code form) #467, landed as d0b7c84: a comment, docstring or layout edit no longer translates every shader again; a split per pass was analysed and left out, since most edits land in code every shader runs through. For the same reason @seedds moved the tool lock checks out of the chain in faabe05.) - Game files stay on the device for repeated iPhone builds: the RomFS, which changes only with the game's version, goes to the app's
Documents/romfs.imgonce per game and version, and every later build is a slim IPA (host, modules, shaders). A cold Breath of the Wild build on an M2 Max took 53 min: shaders 27.2 (19,592, Mac and iOS), IPA pack 10.9 (14.7 GB, nearly all RomFS), shader split 7.3, app build 5.0 (pipeline archive about 4.5), extraction 2.7. The pack is the step every rebuild pays. The parts exist (push_romfs.sh,main.mreading the image,DM_COPY_ROMFS=NOin build_ipa.sh: DM_COPY_ROMFS=NO for the unsigned IPA; entitlements in the executable's signature #478); missing are an NSP script option (--romfs separate), a RomFS identity (title, version, digest) so a mismatched image is refused with a clear message instead of a crash, and docs that make it the route for repeated builds. build_ipa.sh: DM_COPY_ROMFS=NO for the unsigned IPA; entitlements in the executable's signature #478 and NSP script --romfs separate: game files stay on the phone between builds, and the app refuses a wrong image #493, landed as 32ae3b1 and 9ac535b:--romfs separate, andromfs.idchecked by the app at start. The add-on content follows the same way (Documents/aoc, iOS: find a game's DLC (aoc/) in Documents before the app bundle #486 and its follow-up), and NX2Apple's installer pushes the image once per game and version, then installs slim IPAs; the identity file is the interface between the two. - An incremental pipeline archive, keyed per pair (about 4.5 of the 53 min, so behind the item above and dmsh_build: build only the metallibs a target needs (an iPhone build skips the macOS half) #275)
- Incremental IPA and Mac app builds: unchanged IPA entries and derived-profile shader outputs are reused (Incremental builds: rebuild only what changed (IPA repack, Mac app) #279, by @surajmandalcell, landed as 09f3778)
- Shader builds compile only the metallibs a target needs, so an iPhone build skips the macOS half (dmsh_build: build only the metallibs a target needs (an iPhone build skips the macOS half) #275, by @JMRBDev, landed as 712eb81)
- A
chain_validatedrow per profile, so each game shows which chain it was last green on (regress.py --record: each game keeps the chain it was last green on (validated.tsv) #464:regress.py --record,dm/games/<id>/validated.tsv; landed as 76c5453)
Rules that enforce themselves
- Docs that still name removed switches corrected (Correct docs that still name removed environment switches #312, landed as de13572)
- An environment registry, with a test that fails on an unregistered
getenv("DM_…"). List every DM_* environment switch the code reads, with its meaning #318, everyDM_*switch listed with its meaning taken from the commit history (landed as cc626e7, withtest_env_switches.py). - Architecture checks as ordinary tests in
dm/tools/tests, written as ratchets: today's state is the baseline, and a check fails only when things get worse. Examples: file size limits, no game-specific branches in the host core. First: no title IDs in the host (Architecture checks: no title IDs in the host, build IDs only in its per-build tables #422, landed as c7da003; build IDs outside the classified host rows are stopped bydocs_patches.pythroughtest_docs). A generated index of every game-specific patch, with a note perpatches.pyon whether a general rule could replace it, landed as 68fbe84 (docs/agents/game-patches.md, checked bytest_docs). @seedds's Patch design: bind every patch as loosely as what it depends on (host, sdk, engine, game, build) #635 goes further: every patch states what it depends on and how loosely it could be bound (ed0360b and later); file size limits wait for a decision (nvn_mtl.m grew 2,221 lines in three days). - Size limits for C, Objective-C and Python, as a ratchet so nobody, person or agent, piles another 10,000 lines into one file: a file over the limit (aiming under 1,000 lines per file, new files at most about 800, lowered step by step as modules come out; third-party code excluded) may not grow, and a new file may not start over it; functions over a length limit are flagged by clang-tidy's
readability-function-sizein Lint C and Objective-C in CI: a clang-tidy ratchet and a clang-format report #425's ratchet (lizard does the same for Python). The numbers are @seedds's call. Comes after thenvn_mtl.msplit, which brings the largest file from 11,923 lines to topic parts of at most 1,467.
Extract, one module at a time, each proven by identical object code or replay digests
- A mechanical split of
nvn_mtl.minto included fragments, still one translation unit (an updated Split nvn_mtl.m into topic parts (mechanical, identical machine code) #133, re-cut by a script,dm/tools/split_nvn_mtl.py, so it can be redone on anymaininstead of waiting for a quiet hour). On its own this only helps reading: every part still sees every static. It is the first step, not the goal. (Split nvn_mtl.m into topic parts with a script that re-cuts it on any main (mechanical, same machine code) #479, re-cut on 2006d5e: 632 lines stay innvn_mtl.m, 14 parts from 342 to 1,467 lines, same machine code apart fromres_add_atline numbers. It conflicts with today'smainagain.) - A coupling measure as a second fitness function, since line counts are only a proxy.
dm/tools/coupling.pyreads clang's JSON AST (clang -fsyntax-only -Xclang -ast-dump=json, which works while the parts are still one translation unit) and, per part or module, counts:- the mutable file-scope variables it touches that another part defines (an extracted module should reach zero: everything through its interface);
- which parts it calls and is called from, and the cycles between them;
- once modules are files of their own, the symbols each imports and exports (
nm) and its includes (clang -MM).
A baseline table and a test indm/tools/testsfail when a number grows, like the lint ratchet. It comes right after the split (before it, everything is one file) and orders the decoupling: the least entangled part goes first, and each decoupling PR shows how far its numbers fell.
- Right after the split, the decoupling:
shlib(the shader library loader, ~500 lines; it is the ABI boundary), thenpso,presentanddiag, each a module with its own header and interface instead of shared statics, one PR each. Each is checked with replayed frames (DM_NVN_CAPTURE, Capture one NVN frame and replay it against new code (DM_NVN_CAPTURE, nvn_replay), step 1 of #402 #473) before and after, plusregress.py. The size limit below goes down as modules come out. - Per-build host tables (
hle_fpslock.c,hle_uemalloc.c) become profile rows (Unreal Engine 3 support: shader caches in the split, shader_key_skip, the UE3 allocator, its frame-rate cap #339 adds Arkham Knight's rows to both, in today's form; they move with the rest). Unreal Engine 3 support: shader caches in the split, shader_key_skip, the UE3 allocator, its frame-rate cap #339 is closed. Arkham Knight landed separately in d8bc921, with one UE3 row inhle_uemalloc.c. Patch design: bind every patch as loosely as what it depends on (host, sdk, engine, game, build) #635 removes most Unreal rows another way: the rewriter finds FMallocBinned3 and the console variables by name, with no build ID (51904e7, 602ed11). The rows left are for builds that have nothing to find by name. On 2026-10-31hle_fpslock.cholds 11 rows for 7 builds andhle_uemalloc.cone row (Arkham Knight). Moving them to profile rows waits on Proposal: a game profile the host reads at run time (design note and dm_profile() accessor) #492, the run-time profile proposal, which has no reply yet.
A clear platform scope
- A short "Supported platforms" section in the README, as tiers. Tier 1 is supported and tested: Apple silicon Macs on macOS 26+, iPhones with an A17 Pro or later on iOS 26+, iPads with an M1 or later. Everything else (Intel Macs, GPUs without Metal 4, iOS below 26, tvOS) is out of scope upstream; forks are welcome to carry it. Changing a tier goes through an issue. PRs like Add iOS 16.1/17 compatibility and tvOS support for Pokémon Legends: Z-A #310 and Intel Mac: build the shader translator for x86_64 and document the pipeline #189 show the question keeps coming up, and the answer is the maintainer's to give. (README states what nxnative runs on, in one place, and the bug form asks for it #446)
- Make the requirements hard to miss: today they are one sentence under the games list, and the same question keeps coming in as issues (Question: what's the minimum OS version for this project? #165, 如何兼容iOS18的设备 #8, 不支持 BC 的设备(A16)上 iOS 版闪退:贴图视图 R8Unorm → RGBA16Uint 断言失败 #4, 构建时报错:OSError: [Errno 86] Bad CPU type in executable: '~/nxnative/dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate' #54). A "Requirements" line near the top of the README and again in the install steps, the same tiers in CONTRIBUTING.md and the issue forms (Add issue forms, a pull request template and CONTRIBUTING.md #306), with a required "my device meets the requirements" checkbox in the bug form. In part: Add issue forms, a pull request template and CONTRIBUTING.md #306 (CONTRIBUTING.md and the forms), Require the Metal compiler the shaders were tested with #333 (the build requirements in the README). (README states what nxnative runs on, in one place, and the bug form asks for it #446)
- The build Mac's toolchain is part of the scope: the README states the tested one (macOS 26.6+, Xcode 27.1 / Metal 32023.921, the chain's lock), the shader build stops on a different Metal compiler unless overridden, and
spirv-crossis pinned with the chain like the translator, so every Mac produces the same MSL (Legend of Zelda: Breath of The Wild #242: corruption on an M5 and an iPhone 17 built with Xcode 26). PRs: Require the Metal compiler the shaders were tested with #333 (the tested Metal compiler), Pin spirv-cross with the translation chain #337 (spirv-cross pinned), landed together as 4eeac03. - CI builds and tests with that toolchain: the macOS jobs run on GitHub's
xcode-27image with Xcode 27.1 named inDEVELOPER_DIR, the build check compares the runner's Metal compiler withdmsh_chain.lock, and a test keeps the workflow, README and the NSP script naming the same Xcode (CI builds and tests with the toolchain the project supports (Xcode 27.1) #421, landed as ce99604). Until now CI used Xcode 26.6, which inlined differently and made known TSan races appear under other names in CI: ASan, UBSan and TSan over the host and CPU texture tests, as a ratchet #412.
Triage, so issues and PRs find each other
When this was written, 61 open issues had no label, 24 had no reply, none of the 74 open PRs linked the issue it fixes, some were already covered by main (#315, #316, both closed since), and 33 of the 74 conflicted with it. On Oct 27: 78 open issues, none labelled, 33 without a reply; 121 open PRs, 60 of them conflicting. On Oct 31: 112 open issues, one of them labelled, 64 without a reply, and 183 open PRs.
- Area labels from the changed paths:
host,nvn,shader chain,packing,tools,docs, …,game: <title ID>andnew game, applied by a workflow that never runs PR code (Area labels on pull requests from the paths they change #408). The issue forms set their own label. - A keyword check on new issues for devices outside Tier 1 (Intel, iOS below 26, A16 and older) that answers with the requirements from the README. No model, no secret. (Bug reports from a device README's Requirements excludes get a label and one comment pointing there #462, on README states what nxnative runs on, in one place, and the bug form asks for it #446)
- A regular triage report on open PRs and issues: which PRs are already in
main, which conflict and since when, which issue each PR fixes, and likely duplicates (several profiles for one title ID, for example). Generated by a script from the API by fixed rules, not by anyone's judgement, so it can run on a schedule and anyone can check it; the maintainer decides what to close. Where it is posted (its own issue, a Discussion or a page) is still open. Later, after the items above.
What only the repository owner can switch on
Each of these is optional and takes one click:
- Discussions, for design questions like this one;
- branch protection with a merge queue;
- Copilot code review on all PRs, via a ruleset (Settings → Rulesets → "Automatically request Copilot code review"). It needs a Copilot plan on your account, and GitHub offers Copilot Pro at no cost to maintainers of popular open source repositories: if you qualify, Copilot settings (avatar → Copilot settings) shows "Get access to GitHub Copilot" (docs). A second, independent review on every PR would help with the volume of PRs coming in;
- CodeQL's default setup (see the item above);
- a weekly release tag.
I'm happy to set these up and to do the work above.
@JMRBDev, your Wonder data in #272 is exactly what the translator diff would report automatically. Anyone who owns games we don't have: running regress.py on them (it landed with #328) is the most useful thing you can do.
Status (Oct 31): the boxes are ticked once the PRs land. Landed since Oct 28: #448 (96820ed, frame-exact replay in 710559c), #425 (a345f5b, without the clang-format report), #439 (68db795), the fix for #454 (713ecaa, from #605) and regress.py capture and replay (8325eef). #339 is closed, since Arkham Knight landed separately in d8bc921. Open now: #322, #400, #407, #408, #422, #446, #457, #462, #472, #479, #486. Next, in this order: the nvn_mtl.m split (#479, re-cut on 2006d5e, conflicting again with today's main), the coupling measure, then shlib as the first module, checked with regress.py replay. Also open: several frames per capture (step 2 of #402), the Breath of the Wild replay noise from #473, and two more timed rebuilds, one with nothing changed and one with a real chain change, to complete the picture of where a rebuild spends its time. Graphical front ends have their own issue: #388.
Activity
v0lumehi commentedon Oct 26, 2026
@seedds, a few quick yes/no questions so I don't step on your toes:
main(never PR code) or as a scheduled job? Or how would you like to solve that?No pressure on any of these; the PRs work without them.
seedds commentedon Oct 26, 2026
Great idea. Please continue this work
v0lumehi commentedon Oct 26, 2026
On it, will keep you updated
v0lumehi commentedon Oct 27, 2026
Progress update. Thanks @seedds for taking in so much in one day, and for the fixes on top. The review notes on #325, #329 and #333 caught real problems.
Landed (boxes ticked above):
regress.pyand the developer loop (scripts/regress.py: check a change on every local game, building on #272's shader regression check; the developer loop #328, 6f58a7d)regress.py impact(regress.py usage / impact: which games a change can reach, from per-game usage manifests #329, b3f97ef)DM_*registry with its test (List every DM_* environment switch the code reads, with its meaning #318, cc626e7) and the docs for removed switches (Correct docs that still name removed environment switches #312, de13572)Rebased on
main75065df, conflict-free again: #306, #307, #303, #327, #339, #340, #352.mainhad added seven tests to the hand-kept run list since. After the rebase,make -n -B runexecutes the same 87 test commands asmain's Makefile, and every Metal test passes.New: #391, the first step of "Decouple the chain".
dmsh_abi.his the one definition of the shader library ABI: the driver segment layout and the 19 binding kinds, which the host anddmsh_build.pyeach kept their own copy of. A test guards it with a ratchet. The host objects are identical, and all 19,592 Breath of the Wild shaders are unchanged (regress.py chain: PASS). It bumps the chain once. The next step (loading libraries by ABI compatibility) is what stops later tool changes from forcing a rebuild of every game. It ends with two questions for you about things that look off between the two sides.Split off: graphical front ends have their own issue now, #388. Steps already open there: #306 (NX2Apple first in the issue chooser), #387 (README,
docs/frontends.md), and the scope question on #374.Next here: the docs de-conflict (the game count out of
AGENTS.md,symptoms.mdregenerated in CI), then ABI-compatible library loading on top of #391.19 remaining items
v0lumehi commentedon Oct 28, 2026
@seedds the roadmap PRs are all rebased on today's
main, green and mergeable. I'm pausing here until you've had a chance to go through them; no new roadmap PRs from me in the meantime (the coupling measure and the first module,shlib, wait for #479).Ready for review, in a suggested order (smallest and most independent first):
docs/referenceand the symptom index) → The symptom index is regenerated on main by CI, so a reference entry no longer needs it in the PR #472 (the index regenerated onmainby CI).main).DM_RECrecords input asDM_KEYSsteps).nvn_mtl.msplit, best at a quiet moment: it conflicts with any open PR that editsnvn_mtl.m. If you'd rather cut it yourself, take only its first commit and runpython3 dm/tools/split_nvn_mtl.pyon yourmain.Also rebased and green, outside the roadmap: #339 (Unreal Engine 3) with #340 on top, #303 (
DM_NVN_POOLPEAK) and #76 (Breath of the Wild above 60 fps and 1080p).The checklist above is current. Thanks!
Record player input as DM_KEYS steps (DM_REC)
Record player input as DM_KEYS steps (DM_REC), DM_KEYS=@<file> (PR #448)
relay4 commentedon Oct 30, 2026
Following up on the sanitizer work in #412, I’d suggest extending the checks in two ways. I tested these additions locally.
For CI, using Apple Clang from the existing Xcode toolchain:
-fsanitize=implicit-conversion,local-boundsto the existing ASan/UBSan builds. These catch lossy integer conversions, sign changes and local-array bounds violations that-fsanitize=undefineddoes not include.-fsanitize=undefined,integer,implicit-conversion,local-boundstarget for buffer/count arithmetic: heap sizes, audio renderer command/work buffers, HID reports, keyboard/font buffers and CPU texture layout/deswizzle.The broader integer checks need selective coverage: unsigned wrapping is intentional in hashes, ring counters and codec bitstreams. I’d document narrow exceptions for deliberate operations.
These targets passed locally with Apple Clang, including the host ASan/UBSan and TSan suites.
For occasional local audits, Homebrew Clang provides additional options:
-fsanitize=leak) checks allocation ownership and cleanup.-fsanitize=realtime, with audio callbacks marked[[clang::nonblocking]]) detects allocation, locking and other unsafe operations on the real-time callback path.-fsanitize=type) checks strict-aliasing violations.All three ran on my arm64 Mac with Homebrew Clang 23.1.4. The audit found a concrete RTSan issue (however I'm unsure how severe that is): the audio callback’s periodic logging takes stdio/internal locks and can allocate. It also found RomFS image leaks in the filesystem test’s simulated process restarts.
The Homebrew checks need some caution: TySan produced false positives for valid accesses to array members of static structs, and standalone LSan hit an internal allocator assertion during CoreAudio startup, reproducible in a minimal program without host code. I’d therefore keep these as local audit tools initially, with findings reduced and verified before changing production code.
GCC's static
-fanalyzer(Homebrew GCC 16.2.0)dm/host/hle_os.c:126–129,game_env_init.GCC reports
-Wanalyzer-malloc-leak. The initializer always callsstrdup(e).If the input is empty or contains no accepted assignments, no pointer into that
buffer is stored in
g_env_kv, and the duplicate is never freed. Nonemptyaccepted assignments intentionally retain their backing storage for the
process lifetime. Initialization is protected by
pthread_once, so this isnot a leak that grows per frame or call.
Fix: avoid duplicating input with no assignments, or free the duplicate when
no entries were accepted. This is a low-priority cleanup, not a demonstrated
gameplay failure.
dm/host/hle_fs.c:985,h_fs_OpenDirectory.After
opendirsucceeds, the result ofcallocis immediately dereferencedwithout checking for failure. GCC reports
-Wanalyzer-possible-null-dereference. A real allocation failure thereforecrashes the host instead of being handled as an allocation/resource error.
Fix: check the allocation, close the already opened
DIRon failure and returnan appropriate failure result. Similar unchecked allocations exist in thread
creation/startup and shader diagnostic code; these form a broader allocation-
failure handling category.
GCC's static analyzer could be added in addition to other static analysis methods, since it's complementary.
relay4 commentedon Oct 31, 2026
Consider cppcheck and flawfinder alongside clang-tidy
As an addition to the roadmap's existing static-analysis work, I'd suggest evaluating cppcheck and flawfinder alongside clang-tidy and Clang's static analyzer.
I tried cppcheck 2.22.0 on the first-party C code and flawfinder 2.0.20 on the C/Objective-C sources and headers. The audit led to three confirmed issues:
Audio command logging can read beyond a stack buffer
Cppcheck flagged an uninitialized histogram string in
hle_audren.c:capture(). A malformed command list can trigger an out-of-bounds read during logging, reproduced under ASan. Histogram formatting can also exceed its buffer after truncation.Fix: validate the complete list before consuming histogram/sink discovery, initialize the string and bound formatting.
Save-sync metadata writes can leak a file descriptor
Reviewing a cppcheck report exposed an error path in
save_sync.c:write_base(): iffprintf()fails, short-circuit evaluation skipsfclose(). Cleanup can also overwrite the original error.Fix: always close the stream, preserve the first error and replace the previous metadata only after writing and closing succeed.
Watchdog sampling fails for paths containing apostrophes
Flawfinder flagged
system()incrash.c.DM_WATCH_SAMPLEis interpolated into a single-quoted shell command, so a path such as/tmp/John's Mac/hang.txtbreaks parsing.Fix: launch
/usr/bin/samplewithposix_spawn()and a separate argument vector, retaining output redirection and error handling.Both tools produced substantial noise, so I'd start with non-blocking local or scheduled audits. After configuring Apple API models and triaging false positives, selected cppcheck checks could become a ratchet like clang-tidy's. Flawfinder is better treated as a source of manual-review hints. The findings themselves can be fixed separately, each with a regression test.
fs: close directories when handle allocation fails
audren: validate command lists before capture logging
os: free unused game environment backing storage
A daily triage report on open pull requests and issues (#770)
Fuzz the file parsers nightly, with a regression corpus in the tool t…
dmsh_abi.h: one definition of the shader library ABI for the build an…
Record player input as DM_KEYS steps (DM_REC), DM_KEYS=@<file> (PR se…
A daily triage report on open pull requests and issues (seedds#770)