Skip to content

Shared app: send a large game file in parts and join them on the device - #1027

Closed
Haseenesss wants to merge 1 commit into
LNPANDY:mainfrom
Haseenesss:fix/899-chunked-shell-data
Closed

Haseenesss wants to merge 1 commit into
LNPANDY:mainfrom
Haseenesss:fix/899-chunked-shell-data

Conversation

@Haseenesss

Copy link
Copy Markdown
Contributor

Fix or feature

  • Fix
  • Feature

What and why

Fixes #899.

Symptom. Sending a large game's data to the shared app on an iPhone or iPad stalls or fails: the romfs.img on the device stays at 0 KB, the Mac warns that it is out of application memory, and on a 29 GB image the copy ends with CoreDeviceError 3, "The connection was interrupted" (#899, MacBook Pro M1 Pro 8 GB).

Cause. xcrun devicectl device copy to does not stream the file. CoreDeviceService reads the whole source into memory before it sends anything. sample of CoreDeviceService during the copy shows TransferFilesActionImplementation → Data.init(contentsOf:) → read. I reproduced it on an 8 GB Mac (iPadOS 27.2, iPad mini A17 Pro) with a 4.39 GB image:

  • CoreDeviceService grew to 6.9 GB and top showed it as stuck.
  • Swap filled up (5.4 of 6 GB) and the internal disk paged at about 290 MB/s.
  • The tunnel to the iPad carried about 40 KB a minute, and devicectl printed nothing for minutes.

Small files never hit this, which is why only large games did.

Change.

  • Launcher (NXCore/ShellDataParts.swift, ShellInstall.copyGameData):
    • A file larger than 512 MiB goes as parts named <file>.nxpart.<index>.<count>, one devicectl copy each. Each part is cut into a temporary file 8 MiB at a time, so neither the launcher nor CoreDeviceService holds more than one part.
    • A part that is already on the device with its size is not sent again, so a send that stops halfway resumes. Before, a 29 GB send that failed started over.
    • For romfs.img, parts count only when they belong to the same image. A parts receipt sits next to the existing copy receipt, so the parts of an older image of the same size are sent again.
  • Shared app (ShellHost/DataParts.swift, called from ShellHostModel.init off the main actor before BackupExclusion):
    • At its start the app joins every complete set. It appends the parts to <file>.nxjoin, removes each part once its bytes are written, and renames the join over the file. The device needs at most one part of extra space.
    • A join that stops (the app quits, the device restarts) goes on at the next start. A set that is not complete, or whose part sizes do not fit, is left alone.
    • After a join the library is read again, so the game turns from "Needs game data" to playable.
  • Apps the launcher did not build. The app writes Documents/shell-features.txt (data-parts 1). sendData sends into a shared app the launcher did not just build (one signed by another tool, or built by an older launcher). In that case it splits only a file devicectl cannot copy whole (more than a quarter of the Mac's memory), unless the app says it joins parts. Install and Retry always split, because the app is the one just built.
  • Log. The data step logs one new line, shell.dataParts, in all seven languages.

Checked

  • Tests: the pre-push hook passed: swift build && swift test in frontend and in frontend-ios (176 tests), and the Qt front end's unittest for the l10n change.
  • Test for a fix:
    • ShellInstallTests run the copy against the stand-in devicectl with 4-byte parts. They check that the image goes as parts and never whole, that a send stopped on the second part resends only the short part and the rest, that parts on the device count as the data before and after the join, that the parts of another image of the same size go again, and that an app that never said it joins parts gets parts only for a file above the single-copy limit.
    • DataPartsTests (frontend-ios) check the join itself: a complete set replaces an older file; an incomplete set waits; a part of the wrong size is not joined; a join cut off mid-append resumes and truncates the extra bytes; a join that cannot resume is removed and waits for the parts to be sent again; and the model joins at its start and the game becomes playable.
  • Visual QA: no view changed.
  • Run on a device or Mac:
    • Setup: MacBook (8 GB, macOS 27.2) and iPad mini A17 Pro (iPadOS 27.2), 13 Sentinels: Aegis Rim (01008D7016438000) v1.00. The image is 4,389,308,865 bytes.
    • The shared app was built from this branch, with three games, through --install-shell. It went as 9 parts over the CoreDevice connection in about 4 minutes.
    • CoreDeviceService peaked at 715 MB per part, then dropped back, and swap stayed at about 0.85 GB. Before the fix, the same image stalled at 6.9 GB.
    • After the app opened, the parts were gone and romfs.img was whole. The SHA-256 of the whole file read back from the device matches the Mac's, and the game's saves in the folder were untouched.
    • Not checked: the 29 GB game from [iPad] Large game data transfer fails with CoreDeviceError 3 — Fire Emblem Fortune's Weave #899, and a Wi-Fi send that drops halfway on a real device (the stand-in covers the resume).

Screenshots

None, no visible change. The data step writes one more line to the install log.

Scope

  • No engine file (dm/, scripts/, docs/, ref/) is added to the repository: a change there is a patch in patches/ (a commit in engine/, then tools/engine.py export, and tools/engine.py check passes), with its Upstream-Status
  • A fix to the engine or a game profile also goes to seedds/nxnative as a PR in parallel: not applicable, the change is in frontend/ and frontend-ios/ only
  • A change to a launcher view has before and after screenshots above: no view changed
  • No keys, game files or data extracted from a game in the change or the attachments

🤖 Generated with Claude Code

`devicectl device copy to` does not stream: CoreDeviceService reads the
whole source into memory first (Data(contentsOf:) in
TransferFilesActionImplementation). With a romfs.img near the Mac's
memory it swaps until it stalls and the file on the device stays at
0 KB, or the connection drops with CoreDeviceError 3 (LNPANDY#899). On an 8 GB
Mac a 4.39 GB image made CoreDeviceService grow to 6.9 GB and stall.

The launcher now sends a file larger than 512 MiB as parts
(`<file>.nxpart.<index>.<count>`), one devicectl copy each, cut into a
temporary file 8 MiB at a time. Parts already on the device with their
size are not sent again, so a send that stops resumes. For romfs.img a
part counts only when it belongs to the same image (a parts receipt next
to the existing copy receipt).

The shared app joins a complete set at its start, off the main actor:
it appends the parts to `<file>.nxjoin`, removes each part once written,
and renames the join over the file, so it needs at most one part of
extra space. A join that stops goes on at the next start. The app writes
Documents/shell-features.txt, and a data send to an app the launcher did
not just build splits only what devicectl cannot copy whole (a quarter
of the Mac's memory) unless the app says it joins parts.

Fixes LNPANDY#899

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@v0lumehi

v0lumehi commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Thank you @Haseenesss. Your change is part of #1097, the launcher UI overhaul, with you as co-author. Closing this one in favor of it. Please have a look there if something of yours is missing or changed.

@v0lumehi v0lumehi closed this Nov 10, 2026
@Haseenesss
Haseenesss deleted the fix/899-chunked-shell-data branch November 13, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ios iPhone or iPad launcher The NX2Apple launcher app windows Windows (the Qt front end)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[iPad] Large game data transfer fails with CoreDeviceError 3 — Fire Emblem Fortune's Weave

2 participants