Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# The games gallery's table (dm/tools/docs_gallery.py writes README.md's gallery from it): a new game is a row at its end. README.md itself is not merged this way; after a
# conflict in its gallery, run docs_gallery.py. test_docs fails on a game in the table twice
docs/images/gallery.tsv merge=union
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
| Another Switch title | the NSP script (below), then [docs/agents](docs/agents/README.md) |
| Packing, building, prewarm lists, power | [docs/development.md](docs/development.md) |
| Checking a change before a pull request | [06-dev-loop.md](docs/agents/06-dev-loop.md) |
- Where a new finding goes, once: a fact about one game (its passes, shader keys, measurements, why its profile needs a key or a cut, its status) in that game's note, in the section it belongs to and marked current or resolved; a general mechanism (a host or tool change that holds for other games) as an entry in its `docs/reference/<subsystem>.md` section (Symptom, Cause, Fix, Found on with the game's note linked), then `python3 dm/tools/docs_symptoms.py` to update the symptom index, and a line under the game note's "Found on this game". Link to a finding instead of repeating it. Adding a game takes its note and its row in the games table. Once Mac gameplay testing passes, add it to README's regular games gallery with a screenshot and IPA size; Mac testing is sufficient for inclusion. Add caption notes only for noteworthy features or limitations (such as a 60 fps upgrade or controller requirement); otherwise show only the size. Keep testing milestones, platform and remaining validation in the game note. This file changes only when a rule does; it states no game count (the games table is the count). `dm/tools/tests/test_docs.py` checks the links and anchors, the notes and their table rows, the symptom index, and the length of this file (1,500 words) and of each guide in `docs/agents` (6,000).
- Where a new finding goes, once: a fact about one game (its passes, shader keys, measurements, why its profile needs a key or a cut, its status) in that game's note, in the section it belongs to and marked current or resolved; a general mechanism (a host or tool change that holds for other games) as an entry in its `docs/reference/<subsystem>.md` section (Symptom, Cause, Fix, Found on with the game's note linked), then `python3 dm/tools/docs_symptoms.py` to update the symptom index, and a line under the game note's "Found on this game". Link to a finding instead of repeating it. Adding a game takes its note and its row in the games table. Once Mac gameplay testing passes, add it to README's regular games gallery with a screenshot and IPA size (`docs/images/gallery.tsv`, then `dm/tools/docs_gallery.py`); Mac testing is sufficient for inclusion. Add caption notes only for noteworthy features or limitations (such as a 60 fps upgrade or controller requirement); otherwise show only the size. Keep testing milestones, platform and remaining validation in the game note. This file changes only when a rule does; it states no game count (the games table is the count). `dm/tools/tests/test_docs.py` checks the links and anchors, the notes and their table rows, the symptom index, and the length of this file (1,500 words) and of each guide in `docs/agents` (6,000).
- Each game's constants and shader tables form a game profile in `dm/games/<id>/`, where `<id>` is the base game's title ID: `game.tsv` plus the tables (and `patches.py` when the rewriter has patches for its builds). Mario's profile is `010015100B514000`. Tools pick the profile with `DM_GAME`; `make -C dm/host` and `build_ipa.sh` pick it with `GAME`. There is no default game: set it every time (the tools stop and list the profiles otherwise). Put per-game values in the profile, not in code ([dm/games/README.md](dm/games/README.md)).
- Prefer general optimizations and fixes to game-specific rules. Performance work (frame pacing, power, caches, load times) counts here as much as correctness. First look for a change that holds for every game: Horizon / NVN / Vulkan semantics, or what the binary, its SDK version, its imports or its files show. Use a game-specific rule (a profile key, a rewriter cut, a build-ID gate, a per-game branch in the host) only when it is absolutely necessary, that is, when no general rule can do it.
- Find a bug's root cause and fix it there; do not just work around it. Before changing code, find why it happens: the faulting instruction and its decompilation, the value that went wrong and who wrote it, the Horizon / NVN / Vulkan / Metal semantics the host or the translator gets wrong. Then fix that place. A change that only hides the symptom (a stub, a skipped call, a clamp, a retry, a sleep or a longer timeout, a cache flush, a profile key that turns a feature off) is a workaround: use one only when the cause is known and lies where it cannot be fixed (the game's own code, which a rewriter cut changes deliberately, or a platform limit), or when the user asks for a stopgap. Then say so in the code comment and in the game note or reference entry: the cause, or what is still unknown, marked open.
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Never commit or attach game files, keys, title keys or anything extracted from a
- **One change per PR, off `main`.** Rebase on `main` rather than merging it in.
- **Commits.** Each commit message says what changed and why, in plain English sentences, as the history does: the symptom, the cause, the fix, and what was measured or checked.
- **Where findings go.** A fact about one game goes in its note (`dm/games/<id>/README.md`). A general mechanism gets an entry in `docs/reference/<subsystem>.md` (Symptom, Cause, Fix, Found on), followed by `python3 dm/tools/docs_symptoms.py`. A new game takes its profile, its note and its row in the games table ([dm/games/README.md](dm/games/README.md)).
- **The README gallery.** After Mac gameplay testing, add its screenshot to `docs/images/` and its row to [gallery.tsv](docs/images/gallery.tsv), then run `python3 dm/tools/docs_gallery.py`. The table preserves your chosen order; notes hold the IPA size and noteworthy features or limitations. Leave the note empty until the IPA size is measured. Concurrent additions merge in the TSV; regenerate README to resolve a conflict inside its gallery. Conflicting rows for one game need a manual resolution before regeneration.
- **The shader chain.** After changing one of its tools, run `python3 dm/gfx/tools/dmsh_chain.py --header` and commit `dm/gfx/nvn/dmsh_chain.h` with the change. Every game's shaders are then rebuilt, so say which games you checked.
- **Tests.** CI runs the host tests, the CPU-only graphics tests, the tool tests and the builds ([Continuous integration](docs/development.md#continuous-integration)). GitHub's runners have no Metal 4, so run the GPU tests on an Apple-silicon Mac: `make -C dm/gfx/tests GAME=<id> run`.
- **The PR text.** Fill in the template: what and why, what was checked (tests, games, devices), and which docs changed.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Needs iOS / iPadOS 26 or later on an iPhone 15 Pro or newer (A17 Pro chip or lat

Under each game: the IPA's size, and "60 fps with 1.2.0" where the game runs at 60 fps (30 on the Switch) with that update.

<!-- gallery: written by dm/tools/docs_gallery.py from docs/images/gallery.tsv; change the table, then run it -->
<table>
<tr>
<td align="center" valign="top" width="33%"><img src="docs/images/mario-kart-8-deluxe.jpg" width="256" alt="Mario Kart 8 Deluxe"><br>Mario Kart 8 Deluxe<br><sub>~11.9 GB · no Mii heads</sub></td>
Expand Down Expand Up @@ -166,6 +167,7 @@ Under each game: the IPA's size, and "60 fps with 1.2.0" where the game runs at
<td align="center" valign="top" width="33%"><img src="docs/images/final-fantasy-crystal-chronicles.jpg" width="256" alt="FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition"><br>FINAL FANTASY CRYSTAL CHRONICLES Remastered Edition</td>
</tr>
</table>
<!-- end of gallery -->

## Pack an IPA

Expand Down
68 changes: 68 additions & 0 deletions dm/tools/docs_gallery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
# Writes the games gallery in README.md (between its gallery markers) from docs/images/gallery.tsv: one cell per row of the table, three to a row, in the table's order.
# The table is the source: a new game is a row in it (usually at the end), so two pull requests that each add a game change one line each instead of moving every cell after
# the first one; .gitattributes merges the table with the union driver. Columns: image (a file in docs/images), name (the caption and the image's alt text), note (the small
# line under it: the IPA's size, then " · " and a remark if any; omit when the size is unknown) and, optionally, the base game's title ID, which links to its game note.
# Run it after changing the table; --check only reports whether README.md is current (dm/tools/tests/test_docs.py runs it).
# Usage: docs_gallery.py [--check]
import html, pathlib, re, sys

REPO = pathlib.Path(__file__).resolve().parents[2]
TABLE = REPO / 'docs' / 'images' / 'gallery.tsv'
OUT = REPO / 'README.md'
BEGIN = '<!-- gallery: written by dm/tools/docs_gallery.py from docs/images/gallery.tsv; change the table, then run it -->'
END = '<!-- end of gallery -->'
PER_ROW = 3


def rows():
out = []
seen = {'image': {}, 'name': {}, 'title ID': {}}
for n, line in enumerate(TABLE.read_text(encoding='utf-8').splitlines(), 1):
if not line.strip() or line.startswith('#'): continue
c = line.split('\t')
if len(c) not in (2, 3, 4): sys.exit(f'{TABLE.name}:{n}: 2 to 4 columns expected, {len(c)} found')
image, name, note, tid = c + [''] * (4 - len(c))
if not (REPO / 'docs' / 'images' / image).is_file(): sys.exit(f'{TABLE.name}:{n}: no image docs/images/{image}')
if tid and not re.fullmatch(r'[0-9A-F]{16}', tid): sys.exit(f'{TABLE.name}:{n}: a title ID is 16 hex digits: {tid}')
# A union merge can retain two different captions for the same game, so check its image and title ID too.
for field, value in (('image', image), ('name', name), ('title ID', tid)):
if not value: continue
if value in seen[field]:
sys.exit(f'{TABLE.name}:{n}: duplicate {field} {value!r} (first at line {seen[field][value]}): merge the rows by hand')
seen[field][value] = n
out.append((image, name, note, tid))
return out


def cell(image, name, note, tid):
# Attributes use double quotes; apostrophes in existing captions need no escaping.
image, name, note = (html.escape(s, quote=False).replace('"', '&quot;') for s in (image, name, note))
caption = f'<a href="dm/games/{tid}/README.md">{name}</a>' if tid else name
detail = f'<br><sub>{note}</sub>' if note else '' # a game whose IPA size is still open has only its caption
return f'<td align="center" valign="top" width="33%"><img src="docs/images/{image}" width="256" alt="{name}"><br>{caption}{detail}</td>'


def gallery():
r = rows()
lines = ['<table>']
for i in range(0, len(r), PER_ROW):
row = [cell(*x) for x in r[i:i + PER_ROW]]
lines += ['<tr>'] + row + ['<td></td>'] * (PER_ROW - len(row)) + ['</tr>'] # a short last row is filled with empty cells, as README had it
return '\n'.join(lines + ['</table>'])


def readme():
text = OUT.read_text(encoding='utf-8')
if text.count(BEGIN) != 1 or text.count(END) != 1: sys.exit(f'README.md: the gallery markers ({BEGIN} … {END}) must appear once each')
head, rest = text.split(BEGIN)
if END not in rest: sys.exit('README.md: the gallery end marker must follow the begin marker')
return head + BEGIN + '\n' + gallery() + '\n' + END + rest.split(END, 1)[1]


if __name__ == '__main__':
text = readme()
if '--check' in sys.argv[1:]:
if OUT.read_text(encoding='utf-8') != text: sys.exit('README.md: the gallery is out of date: run dm/tools/docs_gallery.py')
else:
OUT.write_text(text, encoding='utf-8')
87 changes: 86 additions & 1 deletion dm/tools/tests/test_docs.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
"""The repository's own Markdown stays navigable: every relative link and #anchor resolves (GitHub's heading anchors), every game profile has its note
(dm/games/<id>/README.md) and a row in the games table of dm/games/README.md, the symptom index (docs/agents/symptoms.md) is the one dm/tools/docs_symptoms.py writes from
docs/reference, and the entry points stay short: AGENTS.md at most 1,500 words, each numbered guide in docs/agents at most 6,000 (a new finding goes into a game note or a
reference, not into these). Third-party trees are left out. No game files needed."""
reference, not into these), and README's games gallery is the one dm/tools/docs_gallery.py writes from docs/images/gallery.tsv. Third-party trees are left out. No game files needed."""
from pathlib import Path
import re
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
from urllib.parse import unquote

REPO = Path(__file__).resolve().parents[3]
sys.path.insert(0, str(REPO / 'dm/tools'))
import docs_symptoms # noqa: E402
import docs_gallery # noqa: E402

SKIP = ('third_party', 'node_modules')
AGENTS_WORDS, GUIDE_WORDS = 1500, 6000
Expand Down Expand Up @@ -84,6 +87,88 @@ def test_index_is_current(self):
self.assertEqual(docs_symptoms.OUT.read_text(), docs_symptoms.index(), 'docs/agents/symptoms.md is out of date: run dm/tools/docs_symptoms.py')


class Gallery(unittest.TestCase):
def test_gallery_is_current(self):
self.assertEqual(docs_gallery.OUT.read_text(encoding='utf-8'), docs_gallery.readme(), 'README.md: the gallery is out of date: run dm/tools/docs_gallery.py')

def test_plain_text_captions_and_unknown_size(self):
image = docs_gallery.rows()[0][0]
with tempfile.TemporaryDirectory() as folder:
table = Path(folder) / 'gallery.tsv'
table.write_text(f'{image}\tA & B: "Adventure" <2>\n', encoding='utf-8')
with patch.object(docs_gallery, 'TABLE', table):
text = docs_gallery.gallery()
self.assertIn('alt="A &amp; B: &quot;Adventure&quot; &lt;2&gt;"', text)
self.assertIn('<br>A &amp; B: &quot;Adventure&quot; &lt;2&gt;</td>', text)
self.assertNotIn('<sub>', text)
self.assertIn('<sub>~2 GB &amp; more</sub>', docs_gallery.cell(image, 'Game', '~2 GB & more', ''))

def test_reversed_markers_fail_before_writing(self):
with tempfile.TemporaryDirectory() as folder:
out = Path(folder) / 'README.md'
text = f'{docs_gallery.END}\nKeep this text\n{docs_gallery.BEGIN}\n'
out.write_text(text, encoding='utf-8')
with patch.object(docs_gallery, 'OUT', out), self.assertRaisesRegex(SystemExit, 'end marker must follow'):
docs_gallery.readme()
self.assertEqual(out.read_text(encoding='utf-8'), text)

def test_concurrent_additions_merge_and_regenerate(self):
games = docs_gallery.rows()[:5]
with tempfile.TemporaryDirectory() as folder:
root = Path(folder)
table, out = root / 'docs/images/gallery.tsv', root / 'README.md'
table.parent.mkdir(parents=True)
(root / '.gitattributes').write_bytes((REPO / '.gitattributes').read_bytes())

def git(*args, check=True):
return subprocess.run(['git', '-c', 'user.name=seedds', '-c', 'user.email=seedds@gmail.com',
'-c', 'commit.gpgSign=false', '-c', 'core.hooksPath=/dev/null', *args],
cwd=root, check=check, capture_output=True, text=True)

def write(rows):
table.write_text(''.join('\t'.join(row).rstrip('\t') + '\n' for row in rows), encoding='utf-8')
out.write_text(f'Before\n{docs_gallery.BEGIN}\n{docs_gallery.END}\nAfter\n', encoding='utf-8')
with patch.object(docs_gallery, 'TABLE', table), patch.object(docs_gallery, 'OUT', out):
out.write_text(docs_gallery.readme(), encoding='utf-8')
git('add', '.')
git('commit', '-m', 'Gallery fixture')

git('init', '-b', 'base')
write(games[:3])
git('switch', '-c', 'ours')
write(games[:4])
git('switch', '-c', 'theirs', 'base')
write(games[:3] + games[4:])
self.assertEqual(git('merge', '--no-commit', '--no-ff', 'ours', check=False).returncode, 1)
self.assertIn('<<<<<<<', out.read_text(encoding='utf-8'))
self.assertEqual(git('diff', '--name-only', '--diff-filter=U').stdout.strip(), 'README.md')
with patch.object(docs_gallery, 'TABLE', table), patch.object(docs_gallery, 'OUT', out):
self.assertCountEqual(docs_gallery.rows(), games)
text = docs_gallery.readme()
self.assertNotIn('<<<<<<<', text)
self.assertTrue(text.startswith(f'Before\n{docs_gallery.BEGIN}\n'))
self.assertTrue(text.endswith(f'\n{docs_gallery.END}\nAfter\n'))
self.assertEqual(text.count('<img '), len(games))

def test_union_merge_rejects_conflicting_game_rows(self):
first, second = docs_gallery.rows()[:2]
# Concurrent caption/screenshot and size edits must require a manual resolution even when the names differ.
for field, changes in (('image', {1: 'Corrected caption'}), ('name', {0: second[0]}),
('title ID', {0: second[0], 1: 'Corrected caption'})):
with self.subTest(field=field), tempfile.TemporaryDirectory() as folder:
base = [first[0], first[1], first[2], '0100000000000000' if field == 'title ID' else '']
ours, theirs = base.copy(), base.copy()
for i, value in changes.items(): ours[i] = value
theirs[2] = '~2 GB'
paths = [Path(folder) / f'{side}.tsv' for side in ('ours', 'base', 'theirs')]
for path, row in zip(paths, (ours, base, theirs)): path.write_text('\t'.join(row) + '\n', encoding='utf-8')
result = subprocess.run(['git', 'merge-file', '--union', '-p', *map(str, paths)], check=True, capture_output=True, text=True)
merged = Path(folder) / 'merged.tsv'
merged.write_text(result.stdout, encoding='utf-8')
with patch.object(docs_gallery, 'TABLE', merged), self.assertRaisesRegex(SystemExit, f'duplicate {field}'):
docs_gallery.readme()


class Length(unittest.TestCase):
def test_entry_points_stay_short(self):
words = lambda p: len(p.read_text().split())
Expand Down
Loading
Loading