Skip to content

Allow sparse text to use disjoint patch-area reaches - #2119

Open
thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/sparse-patch-areas
Open

thehobbit85 wants to merge 1 commit into
seedds:mainfrom
thehobbit85:codex/sparse-patch-areas

Conversation

@thehobbit85

Copy link
Copy Markdown
Contributor

What and why

Addresses the rewriter assertion in issue #1504. Spelunky 2 has hundreds of megabytes of embedded data between small code islands. The rewriter rejected its layout before examining the actual patch sites because the front and back patch areas could not reach a continuous interval.

Retain the normal front reservation when those reaches do not overlap. Early sites use the front area and late sites use the back area. Every emitted entry and return branch still passes the existing range check; a required site in the unreachable middle still fails. Existing overlapping layouts and explicit reservations retain their behavior. No addresses are relocated and no instructions are silently skipped.

This is separate from #1288's middle-area solution: that PR requires a function start in the middle interval where it can insert a new area. The reported sparse data gap has none. Its current implementation was inspected, together with #856 and fresh main, before starting and again before publication.

This PR does not close #1504: compressed embedded shaders and large allocator regions remain separate unresolved problems, and gameplay has not been tested.

Checked

Apple M2 Max, macOS 26.6, Python 3.14.6, Capstone 5.

  • New layout/emitter regression: four of five tests fail on the original continuous-reach assertion; all five pass after the fix. Tests execute the production layout and real x18 scratch/TLS emitter and disassemble both entry and return branches. They cover sparse islands, an unreachable middle instruction, explicit reservations, no patch sites, and the prior FC27/Hello Kitty layouts and stub bytes. A size-only image keeps the automated test small; no game files are used.
  • Full CLI on an independently generated NSO with 285,220,864 bytes of text: PASS in 141.4 seconds. The actual loader, data-block classifier and rewriter identified the large data gap and rewrote both x18 sites at 0x100 and 0x11001800, emitting 40-byte front and back stubs. Temporary synthetic binary and outputs were removed afterward; only validation metadata remains outside the repository.
  • Full tools suite: 1428 tests, 23 skips, one inherited ABI-literal ratchet failure in unchanged nvn_mtl.m, independently reproduced on pristine base 6c676444; existing test_dmsh_abi's ratchet no longer counts "?": the unknown binding kind is also what log lines write for a name they do not have, and the third such placeholder in nvn_mtl.m fails the Tool tests on main (issue #1656) #1981 addresses it. Documentation tests in that suite pass.
  • iOS Python suite: all 52 pass.
  • Python lint: only the same three inherited dmsh_build.py:166–167 type findings, independently reproduced on base and addressed by existing Fix shader-tool lint and stale shader test fixtures #2111. No new findings.
  • git diff --check: PASS. No host or shader code changes, and no shader-chain version change.
  • regress.py impact HEAD: four broad rewriter profiles and 149 without usage manifests. Attempted regress.py check --games 0100C49025D3E000,0100B04011742000,010093801237C000,010015100B514000,01007EC013ABC000: exit 2 because no REGRESS_ROOT, REGRESS_GAMES or game files are available. FC27 exercises two patch areas, Rise a compact front reservation, Metroid Dread large BSS with a front area, Wonder the ordinary layout, and Spelunky 2 the reported sparse layout (no profile yet). None was run; no gameplay validation is claimed.

Fresh main and all 1471 open/draft/closed/merged PR descriptions plus the issue timeline were checked before publication. No existing sparse-layout solution was found.

Docs

  • General mechanism and limits in docs/reference/rewriter.md; packing behavior in docs/development.md; symptom index regenerated.
  • No game files, extracted data, agent plans or journals in the diff.

Spelunky 2 embeds a large data region between small code islands. Requiring a continuous front/back reach rejected it before checking any actual rewrite site. Keep the normal front reservation when the reaches do not overlap, and retain all entry and return branch range checks. Synthetic regressions exercise real layout and x18 stub emission, including rejection of an unreachable middle site.

Refs seedds#1504. Embedded shader extraction and allocator requirements remain separate.
@github-actions github-actions Bot added rewriter The NSO rewriter and per-build patches tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation rewriter The NSO rewriter and per-build patches tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spelunky 2 (01007EC013ABC000): rewriter split range assertion and shader split finds no shaders

2 participants