[thehobbit85]

What and why

Addresses the NARC archive-reader portion (part 2) of #1131. The issue stays open for vertex-stage ViewportMaskNV lowering and game validation; the movie decoder constructors are already on main.

Camelot archives store shader banks behind per-member compression, so scanning the stored bytes misses those shaders. Read BTAF/BTNF/GMIF sections and decode zlib, raw deflate, Camelot LZSS, stored and LZ4 members before using the existing shader readers. Preserve shader keys and vertex/pixel pairs, identify archives by magic rather than extension, and skip recognized texture/vertex-buffer payloads. Reject invalid ranges, truncated streams and inconsistent lengths; bound decoded bytes and nested archive work.

Adapted from PikaMan80's format description and decoder in the issue's supplied patch, with co-author credit. Synthetic regressions cover all eight codec tags, overlapping/extended LZSS references, corrupt archives, shared byte budgets, nested discovery and identical CLI outputs for compressed versus plain members. No game profile or generated game tables are included.

Checked

The latest validated head is 7d50c24fa46045d53c7f80bda4d1576a91f69952, rebased onto main 019a822d. This update also carries the exact synthetic scheduler-test prerequisite from #1170 (3b4fcabb), which corrects the service-request fixture's host-dispatch timing assumption without changing production scheduling. The nullable-regex and Xcode fixture fixes are already on main (e556f7a1); obsolete #1157/#1164 prerequisite patches are not carried. The NARC feature patch and its co-author credit are preserved.

Fresh validation on this integration:

• 74 focused NARC, docs, chain, Xcode workflow, and upstream issue-labeling tests pass.
• Full Python lint: 391 recorded findings, zero new or stale.
• Full LLVM 23.1.4 C lint: 230 sources, 241 recorded findings, zero counts above or below baseline.
• make -C dm/host/tests GAME=010015100B514000 B=<scratch> preempt passes both scheduling modes (98 PASS results); the same target with SAN="-g -fsanitize=thread -fno-inline" also passes both modes, with no TSan diagnostics. The native code and dependencies are byte-identical before and after the final issue-labeling-only main update.
• git diff --check passes. Stable patch IDs confirm the original NARC feature and Make service-request scheduler tests deterministic #1170 prerequisite are unchanged, and the difference from the CI agent's prepared tree is exactly the intervening upstream commits.

These are local results; the replacement head still requires remote CI. No gameplay or full-game regression claim is added.

Initial NARC validation and remaining game/toolchain limitations

• Apple Silicon, macOS 26.6 (25G5028f), existing Homebrew Python 3.14.6 and LZ4.
• Failing-before/passing-after: compressed NARC fixtures yield no shaders or pairs on unchanged main, while the raw fixture yields both stages and one pair. The completed reader passes those cases. Nested-budget tests also failed before their checks were added.
• After rebasing onto 75e5ddd8: python3 -m unittest dm.tools.tests.test_split_narc dm.tools.tests.test_split_gzip dm.tools.tests.test_split_nesting dm.tools.tests.test_docs dm.tools.tests.test_nsp_chain dm.tools.tests.test_nsp_mac_app dm.gfx.tests.test_gfpak — 97 tests pass.
• Broader tool/iOS run, one fresh process per test file as in CI: 94 of 95 files pass, 1,055 tests exercised, 24 skipped. Three derived-profile cases in test_xcode_workflow.py fail because their fixture SimpleNamespace lacks modules_in; the same three failures reproduce in an exported copy of pinned main 1d4b8e82.
• lint_python.py with pinned ruff 0.17.2 and pyright 1.1.414: 392 findings against 391 recorded, with the same single finding on pinned main (dmsh_meshstage.py:101, optional regex result subscript). No additional finding from this diff; the baseline was not changed.
• 8,000 deterministic one-byte mutations of synthetic codec payloads, retaining their small declared sizes: valid results or named [RED] rejections, no unexpected exceptions.
• make -C dm/gfx/tests GAME=010015100B514000 -j3 run built the native test targets but stopped at check_t_alpha: the installed Xcode cannot execute its Metal compiler component. The broad gfx Python suite is likewise blocked (283 tests, 41 failing compiler checks, one compiler error, 56 skips). xcrun metal --version independently reports the missing component. No toolchain was installed for this change.
• git diff --check origin/main..HEAD passes. All PR states, issue links/comments, and patches of 27 potentially overlapping open PRs were checked before publication; no existing NARC solution was found.

Games and limitations: no game/system files are available; no gameplay, full shader build or iOS-device validation is claimed. Mario Tennis Aces (0100BDE00862A000) is directly affected and has no registered profile on this base. The new branch is selected only by NARC magic, so the game check was scoped to that title. scripts/regress.py check --games 0100BDE00862A000 cannot run without REGRESS_ROOT and REGRESS_GAMES. Other games remain unchecked. Existing BNSH, GFPAK, SARC/Yaz0 and gzip paths have synthetic coverage above.

regress.py impact origin/main..HEAD reports:

0 directly affected, 3 broad, 1 not affected, 131 unknown (no usage manifest)
5 changed files; 0 NVN/HLE entry points; 1 impact rule
broad: Breath of the Wild, Pokémon Scarlet, Tears of the Kingdom
not affected: Hades

The broad rows come from the shared splitter rule; they are not gameplay results.

Docs

• Added the durable format finding in docs/reference/shaders.md (Compressed shader banks in Camelot NARC archives), linked issue [PATCH] Mario Tennis Aces (0100BDE00862A000): fix movie::Decoder C2 constructor hook, Camelot NARC shader extraction, and VS ViewportMaskNV lowering #1131, and regenerated the symptom index.
• Updated docs/development.md with archive detection and extraction behavior.
• No chain tool changed; test_nsp_chain confirms the existing chain header is current.
• No game files, keys, extracted game tables, agent plans or execution journals in this diff.

[seedds]

Thanks for this. I'm closing it because main gained its own NARC reader in b6dbb85, right after this branch's base, and the remaining gap turned out to be one condition in the split's prefilter, now on main as 4447be1. Your archive fixtures are adapted in test_split_narc.py, and you're credited as co-author.

Landed on top of that reader, this branch's check would run first and take every NARC. On synthetic archives, it found no shaders in members using MMMM chunks or 0x5a zstd, both of which Mario Tennis Fever's reader handles.

The t_preempt.c changes belong to #1170, which stays open.