Skip to content

shader_split: score only complete NDSR records - #1298

Closed
thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/ndsr-complete-records
Closed

thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/ndsr-complete-records

Conversation

@thehobbit85

@thehobbit85 thehobbit85 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1296. Short NDSR shader packs can end in a partial 16-byte record. The scanner now scores only complete records when choosing between header offsets 0x10 and 0x14. It also retains main’s 8-byte walk past non-shader entries.

Current validation

Tested cd03af43 against main b2d46b86 on Apple silicon. The saved 127-byte input triggered Python’s record-length error on that main revision. The rebased scanner passed 128 tests covering NDSR layout and pairing, NDVE, related splitters, fuzz seeds, docs and the shader-chain header. Pinned Ruff and Pyright reported 371 baseline findings, zero new and zero stale. The symptom index was regenerated from the combined reference entries.

Donkey Kong Bananza and Splatoon Raiders use the two header layouts. Star Fox embeds these packs inside VIPR containers. The regression harness found no Mac builds of those three games. No game resplit, gameplay or device result is available for this revision.

Earlier implementation and validation

Short NDSR packs can leave a partial 16-byte record in either candidate header slice. struct.iter_unpack then raises before the reader can choose the layout, including for the saved 127-byte synthetic fuzz input. Score at most 16 complete records at each offset, preserving layout selection, shader contents and vertex/pixel pairing.

Fixes #1296. Related to #313's parser robustness work.

Adds the saved input to the automatically replayed fuzz corpus, plus tests for short lengths, both record starts, trailing-byte remainders, valid shader extraction and pairing. Updates the existing NDSR reference and symptom index; no shader translation chain or runtime changes.

Validation on Apple silicon macOS, using existing tools and synthetic fixtures only:

  • The exact saved input failed unchanged main f8a5618d with struct.error@shader_split.py:ndsr_shaders>records_from; it passes with this change. The additional boundary tests failed 268 subcases before the fix and pass afterward.
  • All 35 corpus tests and all 3 NDSR boundary tests pass.
  • Ruff 0.17.2 / Pyright 1.1.414 ratchet: 389 existing findings, no new findings and no stale baseline entries.
  • Shader-parser random fuzz engine: 266,803 cases over 60 seconds, no findings.
  • Python CI file loop: all 151 files pass (1,564 tests, 13 optional skips). An initial Rosetta-launched run failed 14 architecture-sensitive files; all 14 pass when explicitly launched with arch -arm64 using the existing ARM64 .NET 10 runtime, without code changes.
  • make -C dm/gfx/tests GAME=010015100B514000 -j3 run: passed on the Mac (existing compiler warnings and optional external reflection samples reported by the suite).

Impacted-game rationale: Donkey Kong Bananza uses the 0x10 record layout, Splatoon Raiders uses 0x14, and Star Fox embeds NDSR packs in VIPR containers. These are the narrow affected paths. No local game inputs are available for this task, so no shader resplit, gameplay, or device validation is claimed. The original archived fix and seed remain preserved.

regress.py impact f8a5618d..HEAD reports 3 broad matches, 1 unaffected profile and 141 unknown profiles; its file-level shader-split rule is broader than this NDSR-only change. regress.py check --baseline f8a5618d --candidate f72ee4e7 --games 040074A01BF12000,0400380020056000,0400E88025B1A000 was attempted against an explicitly empty input directory: exit 2, no Mac builds found, so no gameplay report or validation stamps were produced.

Coverage check before implementation: all 426 issues and 863 PR descriptions, relevant issue timelines, and 104 relevant open/draft/closed/merged PR diffs. #1255/#1258 retain the old scorer; #1259's embedded-pack support and #1278's NARC bounds fix do not resolve this failure. Coverage was rechecked after validation: main remains f8a5618d; the 865-PR inventory and the issue timeline contain no equivalent solution. PRs that changed during this pass were inspected too, including #1297's separate NDVE reader, which leaves the NDSR scorer unchanged.

@github-actions github-actions Bot added packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 5, 2026
Fixes seedds#1296. Preserve both record starts and shader pairing; replay the saved synthetic fuzz input.
@thehobbit85
thehobbit85 force-pushed the codex/ndsr-complete-records branch from f72ee4e to cd03af4 Compare November 6, 2026 19:05
seedds added a commit that referenced this pull request Nov 7, 2026
…score counts only complete 16-byte records (PR #1298)

ndsr_shaders chooses between record starts +0x10 and +0x14 by how many of the first 16 records point at an NVN2 control section. It read
d[o:o + 256] with struct.iter_unpack, which raises struct.error when fewer than 16 records remain and the slice ends in a partial one; the two
starts leave different remainders, so no single length is safe for both. The score now reads min(16, (len - o) // 16) records. Wherever the old
score did not raise it gives the same count, so no game's split changes. shader_split.py is not a chain tool.

A 127-byte fuzz input that raised is in the replayed corpus; test_split_ndsr covers short packs, both starts and vertex/pixel pairing.

Checked: test_split_ndsr (its 268 subcases all raise on the old scorer), test_split_ndve, test_fuzz_corpus, test_docs. No regression games run:
the split output is unchanged for every input the old scorer accepted.

Fixes #1296

Co-authored-by: Eliran Zach <4012041+thehobbit85@users.noreply.github.com>
@seedds

seedds commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Landed on main as ff33639, unchanged, with you as co-author. Thanks! I confirmed the new test's 268 subcases and the saved corpus input fail on the old scorer and pass with yours. Wherever the old score did not raise, the new one gives the same count, so no game's shader split changes. #1296 is closed by the commit.

@seedds seedds closed this Nov 7, 2026
gnosticgeek pushed a commit to gnosticgeek/nxforge that referenced this pull request Nov 7, 2026
…cing (seedds#1555), device-shared memory pool storage, short NDSR records in the shader split (seedds#1298, took upstream's form of our own fix)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qteK3qWJVi2jG3Gz4i1eY
Know-It-All-Brother pushed a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…score counts only complete 16-byte records (PR seedds#1298)

ndsr_shaders chooses between record starts +0x10 and +0x14 by how many of the first 16 records point at an NVN2 control section. It read
d[o:o + 256] with struct.iter_unpack, which raises struct.error when fewer than 16 records remain and the slice ends in a partial one; the two
starts leave different remainders, so no single length is safe for both. The score now reads min(16, (len - o) // 16) records. Wherever the old
score did not raise it gives the same count, so no game's split changes. shader_split.py is not a chain tool.

A 127-byte fuzz input that raised is in the replayed corpus; test_split_ndsr covers short packs, both starts and vertex/pixel pairing.

Checked: test_split_ndsr (its 268 subcases all raise on the old scorer), test_split_ndve, test_fuzz_corpus, test_docs. No regression games run:
the split output is unchanged for every input the old scorer accepted.

Fixes seedds#1296

Co-authored-by: Eliran Zach <4012041+thehobbit85@users.noreply.github.com>
@thehobbit85
thehobbit85 deleted the codex/ndsr-complete-records branch November 10, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shader_split: short NDSR packs raise struct.error while scoring record starts

2 participants