Repository navigation
Shared app: send a large game file in parts and join them on the device - #1027
Closed
Haseenesss wants to merge 1 commit into
Closed
Haseenesss wants to merge 1 commit into
Haseenesss wants to merge 1 commit into
Conversation
`devicectl device copy to` does not stream: CoreDeviceService reads the whole source into memory first (Data(contentsOf:) in TransferFilesActionImplementation). With a romfs.img near the Mac's memory it swaps until it stalls and the file on the device stays at 0 KB, or the connection drops with CoreDeviceError 3 (LNPANDY#899). On an 8 GB Mac a 4.39 GB image made CoreDeviceService grow to 6.9 GB and stall. The launcher now sends a file larger than 512 MiB as parts (`<file>.nxpart.<index>.<count>`), one devicectl copy each, cut into a temporary file 8 MiB at a time. Parts already on the device with their size are not sent again, so a send that stops resumes. For romfs.img a part counts only when it belongs to the same image (a parts receipt next to the existing copy receipt). The shared app joins a complete set at its start, off the main actor: it appends the parts to `<file>.nxjoin`, removes each part once written, and renames the join over the file, so it needs at most one part of extra space. A join that stops goes on at the next start. The app writes Documents/shell-features.txt, and a data send to an app the launcher did not just build splits only what devicectl cannot copy whole (a quarter of the Mac's memory) unless the app says it joins parts. Fixes LNPANDY#899 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 of 2 tasks
Collaborator
|
Thank you @Haseenesss. Your change is part of #1097, the launcher UI overhaul, with you as co-author. Closing this one in favor of it. Please have a look there if something of yours is missing or changed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix or feature
What and why
Fixes #899.
Symptom. Sending a large game's data to the shared app on an iPhone or iPad stalls or fails: the
romfs.imgon the device stays at 0 KB, the Mac warns that it is out of application memory, and on a 29 GB image the copy ends withCoreDeviceError 3, "The connection was interrupted" (#899, MacBook Pro M1 Pro 8 GB).Cause.
xcrun devicectl device copy todoes not stream the file. CoreDeviceService reads the whole source into memory before it sends anything.sampleof CoreDeviceService during the copy showsTransferFilesActionImplementation → Data.init(contentsOf:) → read. I reproduced it on an 8 GB Mac (iPadOS 27.2, iPad mini A17 Pro) with a 4.39 GB image:topshowed it asstuck.Small files never hit this, which is why only large games did.
Change.
NXCore/ShellDataParts.swift,ShellInstall.copyGameData):<file>.nxpart.<index>.<count>, one devicectl copy each. Each part is cut into a temporary file 8 MiB at a time, so neither the launcher nor CoreDeviceService holds more than one part.romfs.img, parts count only when they belong to the same image. A parts receipt sits next to the existing copy receipt, so the parts of an older image of the same size are sent again.ShellHost/DataParts.swift, called fromShellHostModel.initoff the main actor beforeBackupExclusion):<file>.nxjoin, removes each part once its bytes are written, and renames the join over the file. The device needs at most one part of extra space.Documents/shell-features.txt(data-parts 1).sendDatasends into a shared app the launcher did not just build (one signed by another tool, or built by an older launcher). In that case it splits only a file devicectl cannot copy whole (more than a quarter of the Mac's memory), unless the app says it joins parts. Install and Retry always split, because the app is the one just built.shell.dataParts, in all seven languages.Checked
swift build && swift testinfrontendand infrontend-ios(176 tests), and the Qt front end's unittest for thel10nchange.ShellInstallTestsrun the copy against the stand-in devicectl with 4-byte parts. They check that the image goes as parts and never whole, that a send stopped on the second part resends only the short part and the rest, that parts on the device count as the data before and after the join, that the parts of another image of the same size go again, and that an app that never said it joins parts gets parts only for a file above the single-copy limit.DataPartsTests(frontend-ios) check the join itself: a complete set replaces an older file; an incomplete set waits; a part of the wrong size is not joined; a join cut off mid-append resumes and truncates the extra bytes; a join that cannot resume is removed and waits for the parts to be sent again; and the model joins at its start and the game becomes playable.--install-shell. It went as 9 parts over the CoreDevice connection in about 4 minutes.romfs.imgwas whole. The SHA-256 of the whole file read back from the device matches the Mac's, and the game's saves in the folder were untouched.Screenshots
None, no visible change. The data step writes one more line to the install log.
Scope
dm/,scripts/,docs/,ref/) is added to the repository: a change there is a patch inpatches/(a commit inengine/, thentools/engine.py export, andtools/engine.py checkpasses), with itsUpstream-Statusfrontend/andfrontend-ios/only🤖 Generated with Claude Code