Skip to content

repo-publish: keep the previous package set across a no-op publish - #492

Merged
derekross merged 1 commit into
derekross:mainfrom
only21mil:build/c112-distinct-publish-rollback-r3
Nov 8, 2026
Merged

derekross merged 1 commit into
derekross:mainfrom
only21mil:build/c112-distinct-publish-rollback-r3

Conversation

@only21mil

Copy link
Copy Markdown
Contributor

What and why

tools/repo-publish's stage() kept the files of the new index and of the index it replaced. Publishing the same package set twice in a row (v1, then v2, then v2 again) made v2 the "previous" index, so v1's files were pruned locally and then deleted from the mirror by the upload's rsync --delete. That happens on every daily --from-main run with no new builds, and on a stable promotion with nothing new. Each time, the channel loses the one version a phone can go back to (docs/RELEASES.md, "Publishing"). Current main (7172296) still has this stage(): the new test fails on it at not ok - publishing v2 again still keeps v1's files.

The fix: stage() records the set it last staged and the one before it in out/publish/<channel>/aarch64.sets, beside the channel, outside the directory the upload sends.

  • Every run decides from the record, never from the index on disk. The current set moves to "previous" only when the new index names different files.
  • The record is replaced in one rename, after the new index is in place and before the prune. A stage that fails anywhere leaves it alone, so a retry decides as the first try would.
  • The prune keeps only the new set and the recorded previous one. It lists the channel with find and looks names up in the shell, not with grep.
  • A channel staged before this change gets its record from its index and package files before anything is copied, so the first run loses nothing.
  • Each read and write is checked on its own, so a failing command stops the stage instead of recording or keeping less.

So a no-op publish keeps v1, v3 drops v1 and keeps v2, and a file no index lists (such as one a failed stage copied) goes at the next stage. docs/RELEASES.md and the script's header say the same.

For the maintainer

  • Your call on the publisher's behaviour. This keeps "the last different package set" as the rollback, as the docs describe one version back. If you'd rather keep something else (the last n sets, or decide from the server), say so and I'll change it.
  • A real dry run is yours to run. It needs your keys and your out/publish: tools/repo-publish --no-build --dry-run development. The first run creates out/publish/development/aarch64.sets from what the channel holds; cat it to see the two sets. I haven't run the script against the real channel or server.
  • --dry-run still stages locally, as before: only the upload is skipped, so a dry-run set you then don't publish counts as "current" for the next run. Whether a dry run should stage at all is your decision.
  • tests/repo-publish (from your tools: repo-publish --from-main publishes a tree with no prebuilt files (from #289) #476) still covers --from-main. tests/repo-publish-stage sits beside it because it tests stage() alone with bsdtar, not through the git and rsync stand-ins.

Device and build

  • Phone: none: tools only. tools/repo-publish runs on the maintainer's laptop (CONTRIBUTING.md §6.5)
  • Kernel release (uname -r): n/a
  • Packages changed, with versions: none

Checks (no phone needed)

  • tools/pr-check origin/main passes
  • tests/all passes, with the new tests/repo-publish-stage in it (it skips without bsdtar; CI's Arch container has it)
  • tests/packages passes (or the packages I touched: none, so not run)
  • Each changed package has its version bumped: none changed
  • Commits have area prefixes and bodies that say why; any AI help is a Co-Authored-By: or Assisted-by: trailer, never a Signed-off-by:

tests/repo-publish-stage runs stage() alone on fictional packages and indexes, with no keys, signing or server. Its 25 cases (named in the after screenshot below) cover repeated and changed sets, a stray file, failures at each step of a stage and its retry, and a channel without a record.

tests/release-signing also passes: it runs the real tools/repo-publish --no-build --dry-run development and --dry-run stable --expect-index with throwaway keys and a stand-in rsync. In a run with its fixture kept, I checked by hand (the test doesn't assert it) that the record sits beside each channel and every rsync source is the channel directory alone.

Before and after. Nothing here has a screen, so these are terminal runs of the new test, on made-up packages.

Before: the test on main's stage() (7172296), with the two helpers it loads added. Publishing v2 again deletes v1's package and signature.

tests/repo-publish-stage on main's stage(): not ok - publishing v2 again still keeps v1's files, exit 1

After: the same test on this change: 25 ok lines, exit 0.

tests/repo-publish-stage on this change: 25 ok lines, exit 0

On the phone

  • I ran this on my own phone, booted into this build (n/a: tools only)
  • Runbook steps passed (n/a: no runbook)
  • What I did not test: a real publish, or a dry run against your out/publish. The upload and the server's copy rest on reading upload(), which syncs out/publish/<channel>/aarch64/ alone, and on the stand-in rsync in tests/release-signing.
  • Nothing in this change writes outside slot B and userdata, weakens the storage guard, or reads registers from userspace

Generated with Claude Opus 5.5 and GPT-6.1 Sol in T3 Code Nightly

stage() kept the files of the new index and of the index it replaced.
Publishing the same package set again (v1, then v2, then v2: the daily
--from-main run when nothing changed) made v2 the "previous" index, so
v1's files were pruned locally and then deleted from the mirror by the
upload's --delete. The channel lost the one version a phone can go back
to, on any day with no new builds. The same happened to stable on a
promotion with nothing new.

stage() now records the set it last staged and the one before it in
out/publish/<channel>/aarch64.sets, beside the channel, which the upload
doesn't send. Each run decides from that record, never from the index
on disk: it rotates the record only when the new index lists other
files than the recorded current set, and replaces it in one rename once
the new index is in place, before pruning everything but the new set
and the recorded previous one. So a no-op keeps v1, v3 drops it and
keeps v2, a retry after a stage that failed part way (even after the
database was copied) decides as the first try would, and a file no
index lists, such as one a failed stage copied, goes at the next stage
even when that one changes nothing. A channel staged before the record
existed starts it, before anything is copied, from its index and the
package files it has, so the first run after this change loses nothing.
Each read and write here is checked on its own (the channel is listed
with find, and the prune looks names up in the shell, not with grep),
so a command that fails stops the stage rather than recording or
keeping less.
docs/RELEASES.md says the same.

tests/repo-publish-stage runs stage() on its own against fictional
packages and indexes (no keys, no signing, no server): v1 -> v2 -> v2
keeps v1, v3 drops v1 and keeps v2, a removed package stays for one
version, an older set is a change, a stray file goes on a no-op, a
failed stage (a missing package or signature, a fresh channel) changes
neither the index nor what is kept, a retry after a failed index copy or
a record that can't be replaced or is half written keeps the set before,
a stage that can't list the channel deletes nothing, a failing grep
deletes nothing, a channel whose files, index or directory can't be read
gets no record, and a channel without the record keeps v1 on a no-op but
not what a failed stage copied. Against the old stage() it fails at
"publishing v2 again still keeps v1's files". tests/all runs it beside
tests/repo-publish (the --from-main copy); it skips without bsdtar.

Tested on the laptop: tests/repo-publish-stage, tests/all and
tools/pr-check.
Nothing runs on a phone (CONTRIBUTING.md 6.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekross

Copy link
Copy Markdown
Owner

Thank you for finding this. You're right that a publish with nothing new made the current set its own "previous", so the one way back disappeared, first locally and then from the server through --delete. Recording the two sets beside the channel, and deciding from that record instead of the index on disk, is a clean fix. It also survives the daily timer's fresh checkout, because out is the real tree's.

I read the test and ran it (25 ok). I also put main's behaviour back and broke a few other steps on purpose. Main's rotation fails exactly where you said, and dropping either save_sets call or seeding the record empty is caught too. Both call sites run stage() at the top level, so a failing read really does stop the stage. The upload only sends the channel directory, so the record never reaches the server.

On your questions:

  • The previous set. Keeping "the last different set" is what RELEASES says, so that's what I want.
  • Dry runs. Please keep --dry-run staging locally. The release runbook runs repo-publish --dry-run stable --expect-index … and then the real promotion. On main, that dry run put the new index in place first, so the real promotion pruned stable's previous set every time. My local stable channel holds exactly its 45 indexed packages and nothing older. With your record, the dry run moves the record, and the real run keeps the set the server had. So this fixes promotions as well as quiet days.
  • The real dry run is mine to do, and I'll do it once this is in.

One practical thing: #491 also edits tests/all (the comment and the suite loop), so the second of the two to land will conflict. The loop's end should read tests/repo-publish tests/repo-publish-stage tests/extract-vendor-firmware. I can resolve that at merge.

@derekross
derekross merged commit 025e6a2 into derekross:main Nov 8, 2026
2 checks passed
@only21mil

only21mil commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

Yes, please resolve it at merge. As we said on #491, we’re leaving the tests/all conflict to you; the loop should end tests/repo-publish tests/repo-publish-stage tests/extract-vendor-firmware.

robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
…erekross#492)

stage() kept the files of the new index and of the index it replaced.
Publishing the same package set again (v1, then v2, then v2: the daily
--from-main run when nothing changed) made v2 the "previous" index, so
v1's files were pruned locally and then deleted from the mirror by the
upload's --delete. The channel lost the one version a phone can go back
to, on any day with no new builds. The same happened to stable on a
promotion with nothing new.

stage() now records the set it last staged and the one before it in
out/publish/<channel>/aarch64.sets, beside the channel, which the upload
doesn't send. Each run decides from that record, never from the index
on disk: it rotates the record only when the new index lists other
files than the recorded current set, and replaces it in one rename once
the new index is in place, before pruning everything but the new set
and the recorded previous one. So a no-op keeps v1, v3 drops it and
keeps v2, a retry after a stage that failed part way (even after the
database was copied) decides as the first try would, and a file no
index lists, such as one a failed stage copied, goes at the next stage
even when that one changes nothing. A channel staged before the record
existed starts it, before anything is copied, from its index and the
package files it has, so the first run after this change loses nothing.
Each read and write here is checked on its own (the channel is listed
with find, and the prune looks names up in the shell, not with grep),
so a command that fails stops the stage rather than recording or
keeping less.
docs/RELEASES.md says the same.

tests/repo-publish-stage runs stage() on its own against fictional
packages and indexes (no keys, no signing, no server): v1 -> v2 -> v2
keeps v1, v3 drops v1 and keeps v2, a removed package stays for one
version, an older set is a change, a stray file goes on a no-op, a
failed stage (a missing package or signature, a fresh channel) changes
neither the index nor what is kept, a retry after a failed index copy or
a record that can't be replaced or is half written keeps the set before,
a stage that can't list the channel deletes nothing, a failing grep
deletes nothing, a channel whose files, index or directory can't be read
gets no record, and a channel without the record keeps v1 on a no-op but
not what a failed stage copied. Against the old stage() it fails at
"publishing v2 again still keeps v1's files". tests/all runs it beside
tests/repo-publish (the --from-main copy); it skips without bsdtar.

Tested on the laptop: tests/repo-publish-stage, tests/all and
tools/pr-check.
Nothing runs on a phone (CONTRIBUTING.md 6.5).

Co-authored-by: only21mil <only21mil@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants