Repository navigation
repo-publish: keep the previous package set across a no-op publish - #492
Conversation
stage() kept the files of the new index and of the index it replaced. Publishing the same package set again (v1, then v2, then v2: the daily --from-main run when nothing changed) made v2 the "previous" index, so v1's files were pruned locally and then deleted from the mirror by the upload's --delete. The channel lost the one version a phone can go back to, on any day with no new builds. The same happened to stable on a promotion with nothing new. stage() now records the set it last staged and the one before it in out/publish/<channel>/aarch64.sets, beside the channel, which the upload doesn't send. Each run decides from that record, never from the index on disk: it rotates the record only when the new index lists other files than the recorded current set, and replaces it in one rename once the new index is in place, before pruning everything but the new set and the recorded previous one. So a no-op keeps v1, v3 drops it and keeps v2, a retry after a stage that failed part way (even after the database was copied) decides as the first try would, and a file no index lists, such as one a failed stage copied, goes at the next stage even when that one changes nothing. A channel staged before the record existed starts it, before anything is copied, from its index and the package files it has, so the first run after this change loses nothing. Each read and write here is checked on its own (the channel is listed with find, and the prune looks names up in the shell, not with grep), so a command that fails stops the stage rather than recording or keeping less. docs/RELEASES.md says the same. tests/repo-publish-stage runs stage() on its own against fictional packages and indexes (no keys, no signing, no server): v1 -> v2 -> v2 keeps v1, v3 drops v1 and keeps v2, a removed package stays for one version, an older set is a change, a stray file goes on a no-op, a failed stage (a missing package or signature, a fresh channel) changes neither the index nor what is kept, a retry after a failed index copy or a record that can't be replaced or is half written keeps the set before, a stage that can't list the channel deletes nothing, a failing grep deletes nothing, a channel whose files, index or directory can't be read gets no record, and a channel without the record keeps v1 on a no-op but not what a failed stage copied. Against the old stage() it fails at "publishing v2 again still keeps v1's files". tests/all runs it beside tests/repo-publish (the --from-main copy); it skips without bsdtar. Tested on the laptop: tests/repo-publish-stage, tests/all and tools/pr-check. Nothing runs on a phone (CONTRIBUTING.md 6.5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for finding this. You're right that a publish with nothing new made the current set its own "previous", so the one way back disappeared, first locally and then from the server through I read the test and ran it (25 ok). I also put main's behaviour back and broke a few other steps on purpose. Main's rotation fails exactly where you said, and dropping either On your questions:
One practical thing: #491 also edits |
|
Note 🤖 Opus 5.5 replying on behalf of Victor. Yes, please resolve it at merge. As we said on #491, we’re leaving the |
…erekross#492) stage() kept the files of the new index and of the index it replaced. Publishing the same package set again (v1, then v2, then v2: the daily --from-main run when nothing changed) made v2 the "previous" index, so v1's files were pruned locally and then deleted from the mirror by the upload's --delete. The channel lost the one version a phone can go back to, on any day with no new builds. The same happened to stable on a promotion with nothing new. stage() now records the set it last staged and the one before it in out/publish/<channel>/aarch64.sets, beside the channel, which the upload doesn't send. Each run decides from that record, never from the index on disk: it rotates the record only when the new index lists other files than the recorded current set, and replaces it in one rename once the new index is in place, before pruning everything but the new set and the recorded previous one. So a no-op keeps v1, v3 drops it and keeps v2, a retry after a stage that failed part way (even after the database was copied) decides as the first try would, and a file no index lists, such as one a failed stage copied, goes at the next stage even when that one changes nothing. A channel staged before the record existed starts it, before anything is copied, from its index and the package files it has, so the first run after this change loses nothing. Each read and write here is checked on its own (the channel is listed with find, and the prune looks names up in the shell, not with grep), so a command that fails stops the stage rather than recording or keeping less. docs/RELEASES.md says the same. tests/repo-publish-stage runs stage() on its own against fictional packages and indexes (no keys, no signing, no server): v1 -> v2 -> v2 keeps v1, v3 drops v1 and keeps v2, a removed package stays for one version, an older set is a change, a stray file goes on a no-op, a failed stage (a missing package or signature, a fresh channel) changes neither the index nor what is kept, a retry after a failed index copy or a record that can't be replaced or is half written keeps the set before, a stage that can't list the channel deletes nothing, a failing grep deletes nothing, a channel whose files, index or directory can't be read gets no record, and a channel without the record keeps v1 on a no-op but not what a failed stage copied. Against the old stage() it fails at "publishing v2 again still keeps v1's files". tests/all runs it beside tests/repo-publish (the --from-main copy); it skips without bsdtar. Tested on the laptop: tests/repo-publish-stage, tests/all and tools/pr-check. Nothing runs on a phone (CONTRIBUTING.md 6.5). Co-authored-by: only21mil <only21mil@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What and why
tools/repo-publish'sstage()kept the files of the new index and of the index it replaced. Publishing the same package set twice in a row (v1, then v2, then v2 again) made v2 the "previous" index, so v1's files were pruned locally and then deleted from the mirror by the upload'srsync --delete. That happens on every daily--from-mainrun with no new builds, and on a stable promotion with nothing new. Each time, the channel loses the one version a phone can go back to (docs/RELEASES.md, "Publishing"). Currentmain(7172296) still has thisstage(): the new test fails on it atnot ok - publishing v2 again still keeps v1's files.The fix:
stage()records the set it last staged and the one before it inout/publish/<channel>/aarch64.sets, beside the channel, outside the directory the upload sends.findand looks names up in the shell, not withgrep.So a no-op publish keeps v1, v3 drops v1 and keeps v2, and a file no index lists (such as one a failed stage copied) goes at the next stage.
docs/RELEASES.mdand the script's header say the same.For the maintainer
out/publish:tools/repo-publish --no-build --dry-run development. The first run createsout/publish/development/aarch64.setsfrom what the channel holds;catit to see the two sets. I haven't run the script against the real channel or server.--dry-runstill stages locally, as before: only the upload is skipped, so a dry-run set you then don't publish counts as "current" for the next run. Whether a dry run should stage at all is your decision.tests/repo-publish(from your tools: repo-publish --from-main publishes a tree with no prebuilt files (from #289) #476) still covers--from-main.tests/repo-publish-stagesits beside it because it testsstage()alone with bsdtar, not through the git and rsync stand-ins.Device and build
tools/repo-publishruns on the maintainer's laptop (CONTRIBUTING.md §6.5)uname -r): n/aChecks (no phone needed)
tools/pr-check origin/mainpassestests/allpasses, with the newtests/repo-publish-stagein it (it skips withoutbsdtar; CI's Arch container has it)tests/packagespasses (or the packages I touched: none, so not run)versionbumped: none changedCo-Authored-By:orAssisted-by:trailer, never aSigned-off-by:tests/repo-publish-stagerunsstage()alone on fictional packages and indexes, with no keys, signing or server. Its 25 cases (named in the after screenshot below) cover repeated and changed sets, a stray file, failures at each step of a stage and its retry, and a channel without a record.tests/release-signingalso passes: it runs the realtools/repo-publish --no-build --dry-run developmentand--dry-run stable --expect-indexwith throwaway keys and a stand-in rsync. In a run with its fixture kept, I checked by hand (the test doesn't assert it) that the record sits beside each channel and every rsync source is the channel directory alone.Before and after. Nothing here has a screen, so these are terminal runs of the new test, on made-up packages.
Before: the test on main's
stage()(7172296), with the two helpers it loads added. Publishing v2 again deletes v1's package and signature.After: the same test on this change: 25 ok lines, exit 0.
On the phone
out/publish. The upload and the server's copy rest on readingupload(), which syncsout/publish/<channel>/aarch64/alone, and on the stand-in rsync intests/release-signing.userdata, weakens the storage guard, or reads registers from userspaceGenerated with Claude Opus 5.5 and GPT-6.1 Sol in T3 Code Nightly