Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/psprecomp-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ jobs:
path: psprecomp/dist/*

android-app:
# The Android setup app (DissidiaRecomp-Setup-<version>-android.apk): a wizard that builds the game ON the phone from the player's own disc.
# The Android setup app (DissidiaRecomp-Setup-<version>-android.apk): an Android app (native views + the C++ install engine over JNI) that builds the game ON the phone from the player's own disc.
# It carries its own compiler (zig), recompiler, Python and the prebuilt Android core, so nothing but the APK is downloaded.
# Design and the verified steps: docs/android.md "Building on the phone".
runs-on: ubuntu-22.04
Expand Down
1 change: 1 addition & 0 deletions psprecomp/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ All under `runtime/` (headers in `runtime/include/`, sources in `runtime/src/`):
| Core library + SDK | `runtime/CMakeLists.txt`, `runtime/sdk/CMakeLists.txt` | Everything under `runtime/src` (plus the selected game module) builds as the static library `psprecomp_core`; `psprecomp_runtime` is `main.cpp` (the only file that includes the generated `funcs.h`) linked with it, the generated libraries and the overlay libraries in one link group. `cmake --install --component sdk` writes an SDK (`lib/libpsprecomp_core.a`, headers, `src/main.cpp`, `sdk/CMakeLists.txt`) that builds a game from a recompiled output directory without the runtime sources: a release ships the prebuilt core and the player's machine compiles only the generated C++ (design: [docs/installer.md](docs/installer.md)) |
| Toolchain bundle | `toolchain/` (`build_bundle.py`, `versions.json`, `zigwrap.c`) | The portable compiler a release ships and CI builds the core with: pinned zig (clang + lld + libc++) with cmake, ninja and static zlib/FreeType/SDL2/libavcodec built by it, driven through a generated `toolchain.cmake` + `deps.cmake` (`PSPRECOMP_DEPS_CMAKE` makes the runtime CMake use them instead of pkg-config). Reason: the prebuilt core and the code the player compiles are linked into one program, so both must come from the same compiler and C++ library; `toolchain.json` is installed in the SDK and `psp_build.py` refuses a mismatched bundle (details: [docs/installer.md](docs/installer.md)) |
| Setup wizard | `setup/` (`setup_main.cpp`, `install.cpp`, `proc.cpp`) | `psprecomp_setup`: the installer / launcher, an SDL2 program for Linux and Windows that reuses the runtime's menu renderer (`runtime/src/hle/psp_savedata_dialog.cpp`), picks files with the system dialog (portable-file-dialogs) and imports saves (`psp_savemgr`) as an optional last step, drives `games/dissidia/tools/setup_from_iso.py` + `psp_build.py` (+ optional `fetch_textures.py`, `jp_audio.py`) through a small cross-platform process helper; shows progress/ETA, runs the update check, starts the game. Independent of the game's generated code (design: [docs/installer.md](docs/installer.md)) |
| Android setup app | `android/setup/` (Java pages), `setup/android_engine.cpp` (JNI), `android/build_setup_app.py`, `android/pack_setup_app.py`, `release/build_android_app.py` | Builds the game on the phone. Native Android views (no SDL) for the pages, controller and touch input, the live build log, log sharing and crash reports; the install logic is not duplicated: `setup/install.cpp` (plans, progress, `install.ini`), `proc.cpp` and `payload.cpp` are compiled into `libsetupcore.so` and reached through JNI, so the desktop wizard and the app run the same steps. The extras are one list (`setup/extras.cpp`) that both installers build their pages, install.ini and steps from, so an extra is added once for both. The SDL wizard's Android build was replaced by it (rotation, controller focus and Android-native pickers / share sheets are simpler with views than with the canvas renderer) ([docs/android.md](docs/android.md)) |
| Release pipeline | `release/` (`build_release.py`, `assemble.py`, `make_manifest.py`, `sign_manifest.py`, `third_party_notices.py`), `.github/workflows/release.yml` | Builds a platform's archive from source: bundle -> header-only stub (`psprecomp recompile --headers-only`, so CI needs no game) -> runtime core + SDK -> wizard -> archive with `release.json` (file hashes + rebuild/relink fingerprints) and a `manifest.json` (+ Ed25519 `.sig` when a signing key is configured) per release, plus `licenses/THIRD-PARTY-NOTICES.txt` (incl. the FFmpeg LGPL source offer). The player's machine only compiles the generated code (`psp_build.py --sdk --toolchain`) |
| Vulkan backend | `runtime/src/vk/` (`psp_vk_context`, `psp_vk_glsl`, `vkgl_core`, `vkgl_api`), `runtime/include/psp_gl_table.h`, `runtime/include/psp_gl_window.h`, `runtime/third_party/vk_mem_alloc.h` | `PSPRECOMP_RENDERER=vulkan` replaces glad's GL function pointers (on Android a pointer table behind the `gl*` names) with an implementation on Vulkan, so the GE renderer is one code base for both APIs. Images are stored in GL orientation (row 0 = bottom) and only the window present flips; GL-style GLSL (loose uniforms, location-less varyings) is converted to Vulkan GLSL (one std140 block, locations by name, clip-depth fix) and compiled to SPIR-V with glslang, cached on disk by hash; Android carries glslang too: a shader pack (`android/shaders/` -> `assets/shaders.pack`, kept current from the shader sources phones record) is compiled at start behind a "Preparing graphics" screen (`vk/psp_vk_warmup.cpp`), and anything else is compiled when first needed. State becomes pipelines (cached by key), render passes are begun lazily, uniforms/vertices go through per-frame rings, and GL's sequential buffer semantics are kept by copy-on-conflict. [docs/VULKAN_PORT.md](docs/VULKAN_PORT.md) |
| Custom GPU drivers (Android) | `runtime/src/psp_gpu_driver.{h,cpp}`, `runtime/src/vk/psp_vk_loader.cpp`, `android/build_deps.py` (libadrenotools) | On an Adreno phone the Vulkan loader is opened through libadrenotools (`adrenotools_open_libvulkan`) with a community driver (Mesa Turnip) chosen in Settings instead of the phone's own; the surface is then created with `vkCreateAndroidSurfaceKHR` because SDL's helper talks to the system loader. Drivers are adrenotools packages (zip: `meta.json` + `vulkan.*.so`) unpacked under the app's internal storage (dlopen refuses the sd card); the Settings > Drivers tab (only listed where `psp_gpu::supported()`) lists them from the GitHub releases of `K11MCH1/AdrenoToolsDrivers` (`PSPRECOMP_GPU_DRIVER_FEEDS`), downloads over `psp_http` and unpacks without trusting zip paths. A crash marker skips a driver that killed the process while Vulkan was brought up. The four libadrenotools hook libraries ship in the APK next to `libmain.so`. List/zip/meta code is host-testable (`tests/test_gpu_driver.cpp`). |
Expand Down
27 changes: 11 additions & 16 deletions psprecomp/android/build_setup_app.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#!/usr/bin/env python3
"""Build libmain.so of the Android setup app (the wizard in setup/, as a shared library for SDL's Java activity) with zig and the NDK sysroot.
"""Build libsetupcore.so of the Android setup app: the install engine of setup/ (plans, processes, progress, payload; setup/android_engine.cpp
is its JNI face) for the app's Java UI (android/setup/java). Built with zig and the NDK sysroot.

build_setup_app.py --zig ZIG --ndk NDK_ROOT --deps DEPS_DIR --out libmain.so [--jobs N]
build_setup_app.py --zig ZIG --ndk NDK_ROOT --out libsetupcore.so [--jobs N]

DEPS_DIR is the Android part's deps/deps (SDL2 headers + libSDL2.so, FreeType). The same compiler and flags the phone uses to build the game
(games/dissidia/tools/android_build.py device_build): zig as clang + lld against the NDK sysroot, linked with c++_static. Standard library only.
The same compiler and flags the phone uses to build the game (games/dissidia/tools/android_build.py device_build): zig as clang + lld against
the NDK sysroot, linked with c++_static. No SDL: the pages are Android views. Standard library only.
"""
import argparse, concurrent.futures, glob, hashlib, os, subprocess, sys, tempfile

Expand All @@ -17,26 +18,21 @@

def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--zig", required=True); ap.add_argument("--ndk", required=True); ap.add_argument("--deps", required=True)
ap.add_argument("--zig", required=True); ap.add_argument("--ndk", required=True)
ap.add_argument("--deps", help="(ignored; the engine needs no SDL)")
ap.add_argument("--out", required=True); ap.add_argument("--jobs", type=int, default=os.cpu_count() or 2)
a = ap.parse_args()
sysroot = os.path.join(a.ndk, SYSROOT)
lib = sysroot + "/usr/lib/aarch64-linux-android"
rt = os.path.join(a.ndk, CLANG_RT)
work = tempfile.mkdtemp(prefix="setupapp-")
env = dict(os.environ, ZIG_GLOBAL_CACHE_DIR=os.path.join(work, "zig-cache"), HOME=work)
# the menu fonts, embedded as C++ (the same cmake script setup/CMakeLists.txt uses)
fonts = os.path.join(work, "psp_ui_fonts.cpp")
subprocess.check_call(["cmake", "-DFONT_DIR=" + REPO + "/runtime/assets/fonts", "-DOUT=" + fonts, "-P", REPO + "/runtime/assets/embed_fonts.cmake"])
rtsrc = REPO + "/runtime/src/hle/"
cxx = [REPO + "/setup/" + f for f in ("setup_main.cpp", "install.cpp", "proc.cpp", "payload.cpp", "android.cpp")] + \
[rtsrc + f for f in ("psp_savedata_dialog.cpp", "psp_savemgr.cpp", "psp_savedata_crypto.cpp")] + [fonts]
cxx = [REPO + "/setup/" + f for f in ("install.cpp", "extras.cpp", "proc.cpp", "payload.cpp", "android_engine.cpp")]
c = [REPO + "/setup/third_party/miniz.c"]
common = ["-target", "aarch64-linux-android", "-DANDROID", "-U__ANDROID_API__", "-D__ANDROID_API__=%d" % API, "-fPIC", "-O2",
"-fdata-sections", "-ffunction-sections", "-funwind-tables", "-fstack-protector-strong", "-Wno-everything",
"-isystem", sysroot + "/usr/include/aarch64-linux-android", "-isystem", sysroot + "/usr/include",
"-I" + REPO + "/runtime/include", "-I" + rtsrc, "-isystem", a.deps + "/include/SDL2", "-isystem", a.deps + "/include/freetype2",
"-isystem", a.deps + "/include"]
"-I" + REPO + "/setup"]
cxxflags = ["-std=gnu++17", "-nostdinc++", "-I" + sysroot + "/usr/include/c++/v1"]
objs = []

Expand All @@ -52,11 +48,10 @@ def build(src):

with concurrent.futures.ThreadPoolExecutor(a.jobs) as ex:
objs = list(ex.map(build, cxx + c))
d = a.deps + "/lib"
cl = lib + "/%d" % API
cmd = [a.zig, "ld.lld", "-m", "aarch64linux", "-shared", "-z", "noexecstack", "-z", "relro", "-z", "now", "-z", "max-page-size=16384", "--hash-style=gnu",
"--eh-frame-hdr", "--build-id=sha1", "--gc-sections", "--threads=1", "-s", "--no-undefined", "-soname", "libmain.so", "-o", a.out,
cl + "/crtbegin_so.o"] + objs + [d + "/libSDL2.so", d + "/libfreetype.a", cl + "/libz.so", "-L" + cl, "-L" + lib, "-llog", "-landroid", "-lGLESv3",
"--eh-frame-hdr", "--build-id=sha1", "--gc-sections", "--threads=1", "-s", "--no-undefined", "-soname", os.path.basename(a.out), "-o", a.out,
cl + "/crtbegin_so.o"] + objs + ["-L" + cl, "-L" + lib, "-llog",
"-lm", lib + "/libc++_static.a", lib + "/libc++abi.a", rt + "/aarch64/libunwind.a", rt + "/aarch64/libatomic.a", "-ldl", "-lc",
rt + "/libclang_rt.builtins-aarch64-android.a", cl + "/crtend_so.o"]
r = subprocess.run(cmd, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
Expand Down
18 changes: 14 additions & 4 deletions psprecomp/android/make_skeleton.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"""Build the unsigned APK skeleton (manifest + compiled resources + classes.dex of SDL's Java activity), zip-aligned, for apk_pack.py.
Release-time tool (needs a JDK, the Android SDK build-tools + platform, SDL2 source): the player's machine only runs apk_pack.py.

make_skeleton.py --sdk SDK --sdl-src SDL2_SRC [--package org.psprecomp.game] [--label "Dissidia 012"] [--api 29] -o skeleton.apk
make_skeleton.py --sdk SDK (--sdl-src SDL2_SRC | --no-sdl) [--package org.psprecomp.game] [--label "Dissidia 012"] [--api 29] -o skeleton.apk
"""
import argparse, glob, os, shutil, subprocess, sys, tempfile

Expand All @@ -14,7 +14,10 @@ def run(cmd):

def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--sdk", required=True); ap.add_argument("--sdl-src", required=True)
ap.add_argument("--sdk", required=True); ap.add_argument("--sdl-src")
ap.add_argument("--no-sdl", action="store_true", help="no SDL Java classes (the setup app: plain Android views)")
ap.add_argument("--extra-res", help="a second resource folder (the setup app's theme)")
ap.add_argument("--assets", help="a folder packed as the APK's assets/ (the setup app's fonts)")
ap.add_argument("--package", default="org.psprecomp.game"); ap.add_argument("--label", default="Dissidia 012")
ap.add_argument("--api", default="29"); ap.add_argument("-o", "--output", required=True)
ap.add_argument("--manifest", help="manifest template (default: hello/AndroidManifest.xml); its package is replaced by --package")
Expand All @@ -28,7 +31,9 @@ def main():
jar = sorted(glob.glob(sdk + "/platforms/android-*/android.jar"))[-1]
work = tempfile.mkdtemp(prefix="skeleton-")
classes = work + "/classes"; os.makedirs(classes)
srcs = glob.glob(os.path.abspath(a.sdl_src) + "/android-project/app/src/main/java/org/libsdl/app/*.java")
if not a.no_sdl and not a.sdl_src:
ap.error("--sdl-src is needed (or --no-sdl)")
srcs = [] if a.no_sdl else glob.glob(os.path.abspath(a.sdl_src) + "/android-project/app/src/main/java/org/libsdl/app/*.java")
if a.extra_java:
srcs += glob.glob(os.path.abspath(a.extra_java) + "/**/*.java", recursive=True)
run(["javac", "-source", "8", "-target", "8", "-Xlint:-options", "-cp", jar, "-d", classes] + srcs)
Expand All @@ -46,7 +51,12 @@ def main():
base = work + "/base.apk"
compiled = work + "/res.zip"
run([bt + "/aapt2", "compile", "--dir", os.path.abspath(a.icon_res) if a.icon_res else here + "/icon/res", "-o", compiled]) # launcher icon (icon/make_icon.py)
run([bt + "/aapt2", "link", "-o", base, "-I", jar, "--manifest", man, "-R", compiled, "--min-sdk-version", a.api, "--target-sdk-version", "34"])
extra = []
if a.extra_res:
extra = [work + "/res_extra.zip"] # a base input (not an overlay): it adds resources
run([bt + "/aapt2", "compile", "--dir", os.path.abspath(a.extra_res), "-o", extra[0]])
run([bt + "/aapt2", "link", "-o", base, "-I", jar, "--manifest", man, "-R", compiled] + extra + [ "--min-sdk-version", a.api, "--target-sdk-version", "34"] +
(["-A", os.path.abspath(a.assets)] if a.assets else []))
import zipfile
unaligned = work + "/unaligned.apk"
shutil.copy(base, unaligned)
Expand Down
11 changes: 5 additions & 6 deletions psprecomp/android/pack_setup_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
"""Assemble the Android setup app (APK): the wizard, the tools it runs on the phone, and the payload they work from.

pack_setup_app.py --android-part DIR --ndk NDK --zig-tree DIR --python-tree DIR --musl FILE --psprecomp FILE --pspdecrypt FILE
--skeleton APK --libmain SO [--game DIR] [--patchelf PATCHELF] -o setup.apk
--skeleton APK --engine SO [--game DIR] [--patchelf PATCHELF] -o setup.apk

Executables go into lib/arm64-v8a/ as lib*.so (the only place an app may run files from since Android 10): libzig.so (zig 0.15, static,
clang + lld in one), libpsprecomp.so (the recompiler, static musl), libpspdecrypt.so, libpython.so (python-build-standalone, musl; its
interpreter is patched to /data/data/org.psprecomp.setup/files/ld-musl, a link the app makes to libmusl.so = Alpine's musl loader), libmain.so
(the wizard) and libSDL2.so. Everything else (Python's library, zig's lib, the NDK sysroot, the Android core + libraries, the game's tools
interpreter is patched to /data/data/org.psprecomp.setup/files/ld-musl, a link the app makes to libmusl.so = Alpine's musl loader) and
libsetupcore.so (the install engine behind the app's Java pages, android/build_setup_app.py). Everything else (Python's library, zig's lib, the NDK sysroot, the Android core + libraries, the game's tools
and analysis) is one zip, lib/arm64-v8a/libpayload.so, that the wizard unpacks on its first start. Standard library only (+ patchelf).
"""
import argparse, hashlib, os, shutil, subprocess, sys, tempfile, zipfile
Expand All @@ -32,7 +32,7 @@ def add_tree(z, src, arc, skip=lambda rel: False):

def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
for n in ("android-part", "ndk", "zig-tree", "python-tree", "musl", "psprecomp", "pspdecrypt", "skeleton", "libmain"):
for n in ("android-part", "ndk", "zig-tree", "python-tree", "musl", "psprecomp", "pspdecrypt", "skeleton", "engine"):
ap.add_argument("--" + n, required=True)
ap.add_argument("--game", default=os.path.join(REPO, "games", "dissidia"))
ap.add_argument("--patchelf", default="patchelf")
Expand Down Expand Up @@ -117,8 +117,7 @@ def main():
shutil.copy2(os.path.join(a.python_tree, "bin", "python3.13"), py)
subprocess.check_call([a.patchelf, "--set-interpreter", INTERP, py])

libs = [("libmain.so", a.libmain), ("libSDL2.so", os.path.join(a.android_part, "deps", "deps", "lib", "libSDL2.so")),
("libzig.so", os.path.join(a.zig_tree, "zig")), ("libpsprecomp.so", a.psprecomp), ("libpspdecrypt.so", a.pspdecrypt),
libs = [("libsetupcore.so", a.engine), ("libzig.so", os.path.join(a.zig_tree, "zig")), ("libpsprecomp.so", a.psprecomp), ("libpspdecrypt.so", a.pspdecrypt),
("libpython.so", py), ("libmusl.so", a.musl), ("libpayload.so", payload)]
cmd = [sys.executable, "-I", os.path.join(HERE, "apk_pack.py"), "--skeleton", a.skeleton, "--key", os.path.join(a.android_part, "apk", "debug_key.json"), "-o", a.output]
for name, path in libs:
Expand Down
Loading
Loading