Repository navigation
Yakuza Kiwami plays on the Mac and the iPhone; ADR jump tables with a far bounds check; PARC/SLLZ shader split - #912
Open
joshhhhyyyy wants to merge 4 commits into
Open
joshhhhyyyy wants to merge 4 commits into
joshhhhyyyy wants to merge 4 commits into
Conversation
joshhhhyyyy
force-pushed
the
yakuza-kiwami
branch
4 times, most recently
from
November 2, 2026 18:07
fd279cb to
c4665a3
Compare
joshhhhyyyy
force-pushed
the
yakuza-kiwami
branch
2 times, most recently
from
November 5, 2026 17:02
ce852f0 to
0df5df6
Compare
…scheduled far before their branch Yakuza Kiwami 1.01's main stopped the rewrite with `AssertionError: jump table 91b24c`. The dispatch at main+0x91b24c (b.hi; mov w16, w21; adrp / add of the table; adr x18; ldrsw; add; br x18) checks its index with `cmp w21, seedds#3` 31 instructions before the br: the compiler scheduled 23 floating-point and bitfield instructions between the compare and its b.hi. jump_adr_offsets looks 16 instructions back for the compare (guarded_count then checks that a branch acts on it) and the other forms expect it right before the b.hi. Over 31 instructions more must hold than the flags: nothing may rewrite the index, the entry or x18 on the way, and nothing may branch into it. jump_adr_far_check, tried after every other form so the tables they recognize keep their targets, finds the check by the flags: back from the load through straight-line code the first control flow must be the b.hi, and before it the first instruction that writes the flags must be a cmp of the index with an immediate. Between that compare and the br nothing may call, branch, trap or write the index, the entry or x18, no branch or known table of the function may enter the run, and every target must be an instruction of the function outside it. The entry is in docs/reference/rewriter.md, "ADR-relative jump table whose bounds check is scheduled far before its branch". Checked: test_jump_copies.py's AdrFarCheck (the far and the near check, every kind of flag writer between the check and its branch, writes of each register involved, control flow into and out of the run, undecodable words, the real rewriter on a synthetic module); with jump_adr_far_check taken out, its far case stops with the assertion on this main too. Yakuza Kiwami's main rewrites, the table resolving to four entries of which 1 and 3 share a case block; a census of its modules found no other unrecognized table. On main of 2026-11-02 the rewrite of that main stopped with the assertion, and with this change its seven output files were byte for byte those of the build the game was played on. The new form runs only where every older one failed, which stopped the rewrite before, so a module that rewrote before rewrites the same: Tomodachi Life: Living the Dream 1.0.4's main and sdk, the other game on hand, did byte for byte. ruff and pyright find nothing new. Co-Authored-By: Claude <claude@users.noreply.github.com>
Yakuza Kiwami's NSP build stopped at `the shader split found no shaders in main or the RomFS`, after `[RED] main control section @... no code area` lines. Ryu Ga Gotoku Studio's engine keeps nearly all its files in PARC archives, most of them SLLZ-compressed, and its shaders are shader objects (GSVS4 .vso, GSPS4 .pso, GSFX4 .fxo) that hold whole GLSLC outputs, all SLLZ version 1, so a scan of the stored bytes finds no magic. The engine's own objects are SLLZ streams in main's data, whose literals copy the GLSLC and control magics verbatim: those were the control sections with no code area. shader_split.py lists a file that starts with PARC by its tables (par_entries) and decompresses only the files whose first 8 decoded bytes are a shader object, a GLSLC output or another PARC archive (par_shaders; sllz peeks), so the archives of models, textures and motions are only listed. Each GLSLC output is taken whole, one stage each, and the pixel-input cover pairs them. sllz_main_objects reads the objects in main, raw_shaders leaves out control magics inside SLLZ streams, and a file that is SLLZ as a whole is decompressed first. One damaged file is left out with a red line, not its archive. The layouts follow ParManager's ParLibrary (Kaplas80), whose SLLZ decoder credits Rick Gibbed. The entry is in docs/reference/shaders.md, "GLSLC outputs in Ryu Ga Gotoku's PARC archives and SLLZ streams". Yakuza Kiwami 1.01: 0 -> 2,462 shaders (1,259 vertex, 1,193 pixel, 9 geometry, 1 compute). 2,399 are in the four archives under data/shader (3,172 objects with 3,752 outputs) and 63 more in 59 of main's 60 SLLZ streams, among them the only geometry and compute shaders. Its other 2,969 archives (17 GB) hold no shader object; data/chara.par (1.5 GB) is listed in 1.6 s. The new paths run only on files that start with PARC or SLLZ and on SLLZ streams. Tomodachi Life: Living the Dream, the other game on hand, has none: no file of its RomFS (66,343) starts with either, and its main (54 MB) holds no SLLZ byte, so its split is unchanged. Checked: test_split_par.py, 11 tests (SLLZ versions 1 and 2 in both byte orders, overlapping copies, peeks and damaged streams; PARC listings in both byte orders with subfolders; stored and compressed nested archives; only shader objects decompressed; one damaged file left out alone; .par and SLLZ-wrapped files through scan_file_in; main's streams and raw_shaders' skip). Co-Authored-By: Claude <claude@users.noreply.github.com>
Update 1.01 (v65536), Ryu Ga Gotoku Studio's engine with CRIWARE, SDK 18.3.2, on NVN: a profile with its note and games table row. Played on the Mac (M3 Pro) and an iPhone19,4 (A20 Pro, iOS 27.2): boot, the title, the story with its cutscenes and fights. No game-specific rewriter cut or host change: it needed the two general fixes before this commit. The game maps all its memory itself. main reserves a 32 GB address region (main+0x379938) and its own allocators carve 13 GB of reservations from it (a 4 GB growable heap and pools of 1, 4, 2 and 2 GB), about 1.5 GB of them used at the title. The profile holds them unchanged in a 36 GB alias region, one 36.62 GB block (mem_reserve 0x928000000, mem_alias 0x900000000): the regions reached +33.72 GB in the NSP script's address region check, the highest mapped page +33.75 GB in play, with at most 2.25 GB mapped. A standalone mmap probe on the iPhone19,4 found one 64 GB range with the Increased Memory Limit entitlement the builds carry by default (6 GB with neither entitlement), so the block fits there; docs/reference/memory.md records it. A phone limited to about 6 GB would need the game's sizes cut (the note's Open). The other keys are the provisional profile's static guesses, kept because they boot and play; profile_guess.py --check differed only in mem_reserve, the measured layout. One key is added: depth_mode zero, as the game selects near = 0 (one nvnDeviceSetDepthMode call at startup in every run), so the Mac prebuild makes its pipeline bases for the main0_z entry the shaders run; it is no shader input. The shader tables are the provisional profile's, byte for byte (the split of the commit before: 2,462 shaders), and shader_groups / shader_note stay as it wrote them, as every shader_* key is in each shader's cache key. The version is v65536, the update's, which the NSP script reads from the [v65536] tag in an update's file name (a single update without one is taken for it). The Mac run (24,600 frames) and the iPhone run (4,500) printed no [RED] line, no unimplemented call and no shader key missing from the map, and neither crashed. Most frames take 33.3 ms (30 fps), with stretches at 60: the game paces itself under present interval 1. Peak memory: Mac 2.54 GB resident and 1.39 GB of GPU allocations, iPhone 2.04 GB and 1.36 GB, with 6.15 GB still below its memory limit at the lowest. Those runs had an older form of the sdk's cuts than this tree's SDK-wide rule (regions of 1 GB and more aligned to 32 MB, instead of 2 MB from 4 MB up); the note says so. A long session on the same iPhone (2026-11-04, the build of the first iPhone run): 40,199 frames over about 22 minutes of play, the median 300-frame window's frame time p50 33.31 ms and p95 35.05 ms, the GPU 7.82 ms of the 33.3 ms budget, thermal state 0 throughout, peak 2.24 GB resident and 1.34 GB of GPU allocations. With the phone's own prewarm list (Documents/pso_user.tsv, 1,300 lines) the longest frame of play took 185 ms, against 1.06 s on the first run; two 25-minute stays in the background resumed where they were; no [RED] line, unimplemented call, missing shader key or crash. The note's Frame rate has it. Co-Authored-By: Claude <claude@users.noreply.github.com>
The runs in the note before this one had an older form of the sdk's cuts (regions of 1 GB and more aligned to 32 MB). A Mac run of the branch's own build (2026-11-02) took the SDK-wide ones, 2 MB alignment from 4 MB up: 8,400 frames from a save into street fights, the regions' highest mapped page at +33.65 GB of the 36 GB alias region (+33.75 GB before), peak 2.40 GB resident and 1.38 GB of GPU allocations, no crash, no unimplemented call, no shader key missing from the map. The note's Open item that asked for this run gives way to its one red line, a 3D texture bound in a shader's cube slot once in a fight. Co-Authored-By: Claude <claude@users.noreply.github.com>
joshhhhyyyy
force-pushed
the
yakuza-kiwami
branch
from
November 9, 2026 15:57
0df5df6 to
ec428f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Yakuza Kiwami (
0100C9801FEE6000, update 1.01 = v65536, SDK 18.3.2, Ryu Ga Gotoku Studio's engine with CRIWARE, NVN) plays on the Mac and on an iPhone: boot, the title, the story with its cutscenes and fights. Four commits, the first two standing on their own:Rewriter (
dm/tools/patch_nso.py,jump_adr_far_check): an ADR-relative jump table whosecmpis scheduled 31 instructions before itsbr(main+0x91b24c: 23 float and bitfield instructions between thecmpand itsb.hi) stops the rewrite onmainwithAssertionError: jump table 91b24c. The new form finds the check by the flags theb.hireads, not by a distance, and is tried only after every other form, that is only where the rewrite used to stop. Entry:docs/reference/rewriter.md, "ADR-relative jump table whose bounds check is scheduled far before its branch".Shader split (
dm/gfx/tools/shader_split.py): the game keeps its shaders as whole GLSLC outputs in SLLZ-compressed shader objects (.vso/.pso/.fxo) inside PARC archives, and the engine's own in SLLZ streams inmain, so the split found none and the NSP script stopped. It now lists PARC archives by their tables, decompresses only the shader objects (SLLZ versions 1 and 2, after ParManager's ParLibrary), and readsmain's streams: 0 → 2,462 shaders. Entry:docs/reference/shaders.md, "GLSLC outputs in Ryu Ga Gotoku's PARC archives and SLLZ streams".Profile and note (
dm/games/0100C9801FEE6000/, games table row): the provisional profile's keys, except the memory layout. The game reserves a 32 GB address region and carves 13 GB of reservations from it (a 4 GB heap and pools of 1, 4, 2 and 2 GB), so the profile gives it one 36.62 GB block with a 36 GB alias region (mem_reserve 0x928000000,mem_alias 0x900000000). An iPhone19,4 on iOS 27.2 reserves that with the Increased Memory Limit entitlement the builds carry by default: a standalonemmapprobe found a 64 GB largest block with it and 6 GB without (docs/reference/memory.md). The shader tables and theshader_*keys are the provisional profile's, unchanged, as everyshader_*key is in each shader's cache key (as in The Legend of Zelda: Echoes of Wisdom (01008CF01BAAC000, base game 1.0.0): game profile #152). It also declares depth_mode zero: the game selects near = 0 at startup in every run (as main’s a861f1a does for four games)Gallery (
docs/images/yakuza-kiwami.jpg,gallery.tsv, README regenerated): a frame of a street fight from a Mac run of this branch.No game-specific rewriter cut, no host change, no shader-chain change.
Checked
DM_GAME=010015100B514000as in CI): every file ofdm/tools/tests,dm/ios/testsanddm/gfx/tests.test_docs.py, the newtest_split_par.py(11 tests),test_jump_copies.pywith the newAdrFarCheckcases andtest_game_profile.pypass; 23 other files fail there with the same counts onmain(they need macOS tools or a newer Python). ruff 0.17.2 and pyright 1.1.414, the pinned versions: no new finding in the changed files againstmain. Host and GPU tests not run: no host or graphics code changed.main(all seven output files),subsdk0and the game's NRO rewrite byte for byte the same; onmain, the rewrite of thismainstops with the assertion. Reach: the new form runs only where every older one failed, so a module that rewrote before rewrites the same: Tomodachi Life: Living the Dream 1.0.4'smainandsdk, the other game on hand, do byte for byte.shader_split.pyhere is byte for byte the version that split the game (2,462 shaders: 1,259 vertex, 1,193 pixel, 9 geometry, 1 compute), and the profile's tables are that split's. Reach: the new paths run only on files that start withPARCorSLLZand on SLLZ streams; Tomodachi Life's 66,343 RomFS files and itsmainhave none.profile_guess.py --check: the guesses match the profile exceptmem_reserve, the measured layout (mem_aliasshows under "heap check / measured").mainwith other PRs merged (below):[RED]line, no unimplemented call, no shader key missing from the map. Most frames take 33.3 ms (30 fps), with stretches at 60, as the game paces itself under present interval 1. Peak 2.54 GB resident and 1.39 GB of GPU allocations; the regions' highest mapped page +33.75 GB of the 36 GB alias region, 2.25 GB mapped at most.0x7000000000. No[RED]line, no crash; peak 2.04 GB resident and 1.36 GB of GPU allocations, 6.15 GB below the memory limit at the lowest; thermal state 0.build_ipa_from_nsp.py --platform macfrom this tree): extraction reused;main,subsdk0and the NRO rewritten byte for byte as above, thesdkwithmain's 5 SDK-wide cuts; the split 2,462 shaders, its tables passing the check against this profile's; 2,462 Mac shaders and the host built. Then 8,400 frames with a window, from a save into street fights: no crash, no unimplemented call, no shader key missing from the map; the regions' highest mapped page +33.65 GB (+33.75 GB with the older cuts below); peak 2.40 GB resident, 1.38 GB of GPU allocations. One red line, once, at frame 2614 as a new pipeline was made:texture type mismatch: shader d303b2b1a0f401f7_b (stage 1) declares slot 134 as Metal type 5, texture ID 6493 is type 7 (NVN target 2, format 0x9a, Metal format 80); bound as is(a 3D texture in a cube slot; nothing wrong on screen; the note's Open). The gallery frame is from this run.regress.pyneeds macOS, and the reach of both tool changes is nil by construction (the two "Reach" checks above), so no game's run can change.Tested with
The earlier Mac and iPhone runs were made on a local
main(2026-11-02, before #855 and #689) with these PRs merged, none of them part of this PR or needed by it: #627 (Port Tomodachi Life: Living the Dream 1.0.4), #683 (iOS: adapt gameplay to Duo portrait and tabletop poses) and #812 (translate shaders the split missed at run time with maxmetal), plus a small local iOS build fix. #812 is on by default, and its host prints a[RED] shader … is not in the shader map: maxmetal translated it at run timeline for each shader it translates; neither run printed one, so every shader came from this profile's map.One difference follows from #627: its branch predates 17f670a, and merging it brought back the old per-build cut rows for this game's
sdk(f118ae1c…, Absolum's SDK 18.3.2 build). So the earlier runs had regions of 1 GB and more aligned to 32 MB, where this branch hasmain's SDK-wide rule (2 MB alignment from 4 MB up, as SDK 20), which only takes away alignment slack in front of the game's regions: the run of this branch above reached +33.65 GB where the older cuts reached +33.75. The game note records it.Not checked
sdkcuts; the new rule only lowers the regions' reach).Docs
docs/reference/rewriter.md,docs/reference/shaders.md,docs/reference/memory.mdand the game note, andpython3 dm/tools/docs_symptoms.pywas run (the symptom index)dm/gfx/nvn/dmsh_chain.huntouched)🤖 Generated with Claude Code