Skip to content

Read compiled shaders from Larian v13 archives - #2117

Open
thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/lspk-shader-archives
Open

thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/lspk-shader-archives

Conversation

@thehobbit85

@thehobbit85 thehobbit85 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Read bounded Larian v13 archive members and decode their compressed .bshd shader programs. Numbered parts and GLSLC vertex/pixel pairing are preserved. Solid archives and other archive versions remain unsupported.

Rebased b024e20b6056e3d7a48d09ee4248545d1c475a3f onto main a9005a51e3e331e0fa3e9a878905c4b09533a40d, producing b775c758529ff295fc47d35a4b2fc1fce48204d5. The archive-dispatch conflict retains main’s RPF6, D3 and CASC jobs and file filters alongside LSPK. The original author and co-author commit message is preserved. Only a trailing test-file blank line was trimmed.

38 archive and documentation tests passed, including the real LSPK CLI fixture and RPF6/D3/CASC dispatcher coverage. Native arm64 Python 3.14.6 with the repository-pinned tools was used. Generated docs, remaining diff, range-diff, main ancestry and git diff --check were reviewed. lint_python: 321 findings, 321 in the baseline, 0 new, 0 fixed but still listed

Divinity is the reported format. Wonder and Breath of the Wild cover Nintendo containers, Rise Capcom containers and Engage Unity containers in the shared dispatcher. No game files or game shader comparison was run. The title-screen loading loop in #1679 remains unresolved. Earlier fuzz and native-suite evidence below belongs to its named base.

Prior description, validation limits and author context

The following results retain their original revision and environment.

What and why

Addresses the missing archived shaders in issue #1679. The splitter scanned Larian .pak files as raw bytes, so it could not find the GLSLC programs inside their compressed .bshd members.

Read the footer-based LSPK v13 file list, select compiled shader members, and decode stored/zlib/LZ4 entries from the primary or numbered archive parts. Decode each member's size-prefixed zlib wrapper and pass its bytes to the existing GLSLC reader, preserving vertex/pixel pairs. Validate ranges, compression and exact decoded lengths, with bounded allocations; damaged and unsupported archives are reported as unreadable. Shaderless archives and external parts are excluded from loose scanning.

Thanks to Zatteo for the diagnosis, .bshd layout and initial reader attached to the issue. The archive structures were independently checked against LSLib's LSPKHeader13 and FileEntry10.

This fixes shader discovery for that format. The separate title-screen loading loop remains unresolved, and no game files or gameplay testing were available. This PR therefore references, rather than closes, #1679. Solid archives and versions other than 13 remain unsupported.

Checked

Apple M2 Max, macOS 26.6, Xcode 27.2, Python 3.14.6.

  • The new end-to-end CLI regression fails on pristine base 6c676444: zero shaders instead of the synthetic vertex/pixel pair. All 10 new tests pass with the fix, including folder and RomFS-image input, all supported compressions, numbered parts, pairing, compute and reflection-only members, empty lists, invalid ranges/sizes, truncated streams, unsupported flags and allocation limits. No GLSLC parser or pairing mocks.
  • Five minutes of deterministic fuzzing (seed 1679): 3,369,753 inputs, no findings. The harness exercises archive metadata, member decode and the real shader parser with allocation guards and per-input timeouts; it uses synthetic seeds only.
  • Full make -C dm/gfx/tests -j2 GAME=010015100B514000 run: PASS.
  • Native /usr/bin/arch -arm64 make -C dm/host -j2 GAME=010015100B514000 dmhost: PASS. An initial Rosetta invocation produced two incompatible generated libvpx objects; those were rebuilt natively before the successful run.
  • Documentation tests: 6 PASS. iOS Python tests: 52 PASS.
  • Full tools suite: 1433 tests, 23 skips, one inherited ABI-literal ratchet failure in unchanged nvn_mtl.m; existing PR test_dmsh_abi's ratchet no longer counts "?": the unknown binding kind is also what log lines write for a name they do not have, and the third such placeholder in nvn_mtl.m fails the Tool tests on main (issue #1656) #1981 addresses that failure.
  • Graphics Python suite: 467 tests, 140 skips, six inherited IntegerPassThrough fixture errors. Python lint reports three inherited dmsh_build.py:166–167 type findings. These failures reproduce on pristine base 6c676444; Fix shader-tool lint and stale shader test fixtures #2111 addresses the graphics fixture and Python lint failures. No new lint findings in this change.
  • regress.py impact HEAD: three broadly affected shader profiles, one unaffected profile and 149 without usage manifests. Attempted regress.py check --games 010027400CDC6000,010015100B514000,01007EF00011E000,0100B04011742000,0100A6301214E000: exit 2 because no REGRESS_ROOT, REGRESS_GAMES or game files are available. Divinity is the reported archive format (no repository profile yet); Wonder and Breath of the Wild exercise Nintendo archives, Rise exercises Capcom archives, and Engage exercises Unity archives in the shared dispatcher. None was run. The synthetic CLI test covers the new format through both filesystem backends.

Immediately before publication, fresh main, all 1469 open/draft/closed/merged PR descriptions and the issue timeline had no competing LSPK solution.

Docs

  • General mechanism, source and limitations in docs/reference/shaders.md; packing behavior in docs/development.md; symptom index regenerated.
  • No shader-chain tool changed; no chain version bump is required.
  • No game files, extracted data, agent plans or execution journals in the diff.

@github-actions github-actions Bot added packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 10, 2026
The shader split scanned compressed LSPK files as raw bytes and missed their .bshd programs. Read the bounded v13 file list and multipart members, decode both compression layers, and retain GLSLC stage pairs. Synthetic archives exercise the real parser and CLI without game files.

Refs seedds#1679. The title-screen loading loop remains unresolved.

Co-authored-by: Zatteo <64840342+zatteogit@users.noreply.github.com>
@thehobbit85
thehobbit85 force-pushed the codex/lspk-shader-archives branch from b024e20 to b775c75 Compare November 13, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants