From 4831fd7d5749fbc710df60d1fb89cf37198074c5 Mon Sep 17 00:00:00 2001 From: only21mil Date: Sat, 7 Nov 2026 15:11:45 +0000 Subject: [PATCH] repo-publish: keep the previous package set across a no-op publish stage() kept the files of the new index and of the index it replaced. Publishing the same package set again (v1, then v2, then v2: the daily --from-main run when nothing changed) made v2 the "previous" index, so v1's files were pruned locally and then deleted from the mirror by the upload's --delete. The channel lost the one version a phone can go back to, on any day with no new builds. The same happened to stable on a promotion with nothing new. stage() now records the set it last staged and the one before it in out/publish//aarch64.sets, beside the channel, which the upload doesn't send. Each run decides from that record, never from the index on disk: it rotates the record only when the new index lists other files than the recorded current set, and replaces it in one rename once the new index is in place, before pruning everything but the new set and the recorded previous one. So a no-op keeps v1, v3 drops it and keeps v2, a retry after a stage that failed part way (even after the database was copied) decides as the first try would, and a file no index lists, such as one a failed stage copied, goes at the next stage even when that one changes nothing. A channel staged before the record existed starts it, before anything is copied, from its index and the package files it has, so the first run after this change loses nothing. Each read and write here is checked on its own (the channel is listed with find, and the prune looks names up in the shell, not with grep), so a command that fails stops the stage rather than recording or keeping less. docs/RELEASES.md says the same. tests/repo-publish-stage runs stage() on its own against fictional packages and indexes (no keys, no signing, no server): v1 -> v2 -> v2 keeps v1, v3 drops v1 and keeps v2, a removed package stays for one version, an older set is a change, a stray file goes on a no-op, a failed stage (a missing package or signature, a fresh channel) changes neither the index nor what is kept, a retry after a failed index copy or a record that can't be replaced or is half written keeps the set before, a stage that can't list the channel deletes nothing, a failing grep deletes nothing, a channel whose files, index or directory can't be read gets no record, and a channel without the record keeps v1 on a no-op but not what a failed stage copied. Against the old stage() it fails at "publishing v2 again still keeps v1's files". tests/all runs it beside tests/repo-publish (the --from-main copy); it skips without bsdtar. Tested on the laptop: tests/repo-publish-stage, tests/all and tools/pr-check. Nothing runs on a phone (CONTRIBUTING.md 6.5). Co-Authored-By: Claude Opus 5.5 --- docs/RELEASES.md | 2 +- tests/all | 7 +- tests/repo-publish-stage | 222 +++++++++++++++++++++++++++++++++++++++ tools/repo-publish | 49 ++++++--- 4 files changed, 263 insertions(+), 17 deletions(-) create mode 100755 tests/repo-publish-stage diff --git a/docs/RELEASES.md b/docs/RELEASES.md index 2937eaa4d..27941295c 100644 --- a/docs/RELEASES.md +++ b/docs/RELEASES.md @@ -389,7 +389,7 @@ CI never holds the signing key and never sees a factory image. The maintainer do - **Users follow `stable`, builders and testers `development`** (decided, ยง12.9). Release images point `[omamobile]` at `stable` on `pkgs.omamobile.org`: the device package's `omamobile-mirrorlist` becomes that URL instead of a local `file://` path. - **The move from the original key.** The original package key (2026-10-12, `6C84FE3D240709E70E27CC94B301F5EA62B249FE`, `keys/gnupg`) still signs packages, the database and bundles. `omamobile-keyring` 20261026 trusts both keys and is itself signed by the original, so phones accept the release key with an ordinary update. Then: switch the build tools to the release key (`keys/gnupg.release`), and once published phones have that keyring, move the original key to `omamobile-revoked`. `tools/install-phone` accepts either key's signature on a bundle until then. - Rotation and revocation go through `omamobile-keyring`'s `omamobile-trusted` and `omamobile-revoked`, as Arch's keyring does. -- **Publishing** (`tools/repo-publish`): `pkgs.omamobile.org` has two channels, `development/aarch64` and `stable/aarch64`. `repo-publish development` builds what changed (`build-packages --all`), checks every package's checksum and signature against `omamobile-keyring`'s trusted keys, and uploads packages first, then the index, then the pruning; each channel keeps the current and the previous index's files. `repo-publish stable` promotes development's signed files unchanged, after asking. The maintainer's laptop runs it daily from a clean `origin/main` checkout (`--from-main`, `tools/systemd/omamobile-publish.timer`), and `--remind` opens a "Ready for stable" issue when development has differed from stable for three days. Experimental phones' device packages are published to development only. The server account needs the `deployers` group. +- **Publishing** (`tools/repo-publish`): `pkgs.omamobile.org` has two channels, `development/aarch64` and `stable/aarch64`. `repo-publish development` builds what changed (`build-packages --all`), checks every package's checksum and signature against `omamobile-keyring`'s trusted keys, and uploads packages first, then the index, then the pruning; each channel keeps the files of the current index and of the last one before it with a different package set, also after a publish that changed nothing. `repo-publish stable` promotes development's signed files unchanged, after asking. The maintainer's laptop runs it daily from a clean `origin/main` checkout (`--from-main`, `tools/systemd/omamobile-publish.timer`), and `--remind` opens a "Ready for stable" issue when development has differed from stable for three days. Experimental phones' device packages are published to development only. The server account needs the `deployers` group. - Never commit, paste or upload a key, a passphrase or `keys/`. CI gets no key. - **Contributors' builds** (review lesson L-7). A port's own packages are signed with a local key that only that contributor's machine trusts, and everything lands under `out//` and `keys//`, never in the maintainer's `keys/` or `out/repo`: diff --git a/tests/all b/tests/all index 06bb61eef..7417b0fe9 100755 --- a/tests/all +++ b/tests/all @@ -20,8 +20,9 @@ # a scratch server, tools/extract-camera-data's adrp decoding, # tools/phone-smoke's dmesg error classification, the OWL conformance # report's differences on the host, bin/lineage-ref sync's branch handling -# against stand-ins for docker and repo, and tools/repo-publish --from-main's -# copy of the prebuilt binaries into a fixture's publish tree. +# against stand-ins for docker and repo, tools/repo-publish --from-main's +# copy of the prebuilt binaries into a fixture's publish tree, and +# tools/repo-publish's staging. set -euo pipefail cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." for f in tools/install-phone tools/phone-provision tools/nv-backup tools/build-rootfs tools/build-packages tools/make-bundle \ @@ -38,7 +39,7 @@ sh -n device/flame/initramfs/build bash -n tools/build-flame-probe echo "ok - scripts parse" python3 tests/bt-uart-gpio.py -for t in tests/pr-check tests/kernel-pr-check tests/phone-smoke tests/alarm-wake-test tests/install-phone tests/build-rootfs tests/rootfs-resolve tests/phone-provision tests/nv-backup tests/initramfs-grow tests/images-layout tests/boot-update-parity tests/stage-boot-files tests/initramfs-reproducible tests/device-overlays tests/simplefb-trigger tests/initramfs-steps tests/storage-guard tests/usb-product tests/boot-method tests/modules-push tests/make-bundle-keys tests/make-bundle tests/release-signing tests/pinned-packages tests/flame-probe packages/omamobile-firstboot/test/all packages/omamobile-device-felix/test/all packages/omamobile-device-lynx/test/all packages/omamobile-device-panther/test/all tests/local-builds tests/phone-install-package tests/publish-kernel tests/fetch-toolchain tests/release-upload tests/extract-camera-data tests/release tests/lineage-ref tests/repo-publish; do +for t in tests/pr-check tests/kernel-pr-check tests/phone-smoke tests/alarm-wake-test tests/install-phone tests/build-rootfs tests/rootfs-resolve tests/phone-provision tests/nv-backup tests/initramfs-grow tests/images-layout tests/boot-update-parity tests/stage-boot-files tests/initramfs-reproducible tests/device-overlays tests/simplefb-trigger tests/initramfs-steps tests/storage-guard tests/usb-product tests/boot-method tests/modules-push tests/make-bundle-keys tests/make-bundle tests/release-signing tests/pinned-packages tests/flame-probe packages/omamobile-firstboot/test/all packages/omamobile-device-felix/test/all packages/omamobile-device-lynx/test/all packages/omamobile-device-panther/test/all tests/local-builds tests/phone-install-package tests/publish-kernel tests/fetch-toolchain tests/release-upload tests/extract-camera-data tests/release tests/lineage-ref tests/repo-publish tests/repo-publish-stage; do echo "# $t" "$t" done diff --git a/tests/repo-publish-stage b/tests/repo-publish-stage new file mode 100755 index 000000000..8bef0715b --- /dev/null +++ b/tests/repo-publish-stage @@ -0,0 +1,222 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT +# tools/repo-publish's stage() (no keys, no signing, no server): it brings a +# channel directory to a new index and keeps the files of the last index +# that listed a different package set, so a phone can go back one version +# even after a publish that changed nothing, or after a stage that failed +# part way. Fictional packages and indexes only; stage() is taken from the +# script and run on its own. +set -euo pipefail +cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." +fail() { printf 'not ok - %s\n' "$1" >&2; exit 1; } +pass() { printf 'ok - %s\n' "$1"; } +command -v bsdtar >/dev/null || { pass "# skip: no bsdtar"; exit 0; } +t=$(mktemp -d) +trap 'rm -rf "$t"' EXIT + +{ + grep -m1 '^DB=' tools/repo-publish + grep -m1 '^db_files() ' tools/repo-publish + grep -m1 '^pkg_files() ' tools/repo-publish + grep -m1 '^save_sets() ' tools/repo-publish + sed -n '/^stage() {$/,/^}$/p' tools/repo-publish +} > "$t/fn.sh" +if ! grep -q '^stage() {$' "$t/fn.sh" || ! grep -q '^db_files() ' "$t/fn.sh" || ! grep -q '^pkg_files() ' "$t/fn.sh" || + ! grep -q '^save_sets() ' "$t/fn.sh" || ! bash -n "$t/fn.sh"; then + fail "stage() and the helpers it uses are in tools/repo-publish" +fi +# shellcheck source=/dev/null +. "$t/fn.sh" + +# make_repo NAME PACKAGE...: a built repository in $t/NAME, as +# tools/build-packages leaves out/repo: each package with its .sig, and a +# (fictionally signed) index listing them +make_repo() { + local p f sum d=$t/$1 + shift + rm -rf "$d" "$t/db" && mkdir -p "$d" "$t/db" + for p in "$@"; do + echo "fictional $p" > "$d/$p-aarch64.pkg.tar.zst" + echo "fictional signature" > "$d/$p-aarch64.pkg.tar.zst.sig" + sum=$(sha256sum < "$d/$p-aarch64.pkg.tar.zst" | cut -d' ' -f1) + mkdir -p "$t/db/$p" + printf '%%FILENAME%%\n%s\n\n%%SHA256SUM%%\n%s\n' "$p-aarch64.pkg.tar.zst" "$sum" > "$t/db/$p/desc" + done + for f in omamobile.db omamobile.files; do + (cd "$t/db" && bsdtar -czf "$d/$f.tar.gz" -- *) + echo "fictional signature" > "$d/$f.tar.gz.sig" + ln -sfn "$f.tar.gz" "$d/$f" + ln -sfn "$f.tar.gz.sig" "$d/$f.sig" + done +} +ch=$t/channel/aarch64 +# run_stage NAME: stage $t/NAME into the channel, in a shell of its own so +# set -e stops stage() at a failure as in the script (in a caller's `if` +# or `||` it would be ignored). FAULT, if set, is run first: a function +# that stands in for a command and fails. +run_stage() { FAULT=${FAULT:-} bash -c 'set -euo pipefail; . "$1"; eval "$FAULT"; stage "$2" "$3"' _ "$t/fn.sh" "$t/$1" "$ch" > "$t/log" 2>&1; } +# stages NAME WHAT: run_stage NAME, which must succeed +stages() { run_stage "$1" || { cat "$t/log" >&2; fail "$2"; }; } +# The package files (and signatures) wanted, and those the channel has +want() { local p; for p in "$@"; do printf '%s\n' "$p-aarch64.pkg.tar.zst" "$p-aarch64.pkg.tar.zst.sig"; done | sort; } +have() { (cd "$ch" && printf '%s\n' *.pkg.tar.*) | sort; } +# expect WHAT PACKAGE...: the channel holds exactly these packages +expect() { + local what=$1 + shift + diff <(want "$@") <(have) >&2 || fail "$what" + pass "$what" +} +# kept WHAT PACKAGE...: these packages are all still there +kept() { + local what=$1 + shift + [[ -z $(comm -23 <(want "$@") <(have)) ]] || fail "$what" + pass "$what" +} +# expect_index NAME: the channel's index files are NAME's +expect_index() { + diff <(db_files "$t/$1") <(db_files "$ch") >&2 || fail "the channel's index is $1's" + local f s + for f in omamobile.db omamobile.files; do + for s in "" .sig; do + if ! cmp -s "$ch/$f.tar.gz$s" "$t/$1/$f.tar.gz$s" || ! cmp -s "$ch/$f$s" "$t/$1/$f.tar.gz$s"; then + fail "$f$s and $f.tar.gz$s are $1's" + fi + done + done +} + +# v1 -> v2 -> v2 -> v3: base stays, app moves 1 -> 2 -> 3 +make_repo v1 base-1-1 app-1-1 +make_repo v2 base-1-1 app-2-1 +make_repo v3 base-1-1 app-3-1 +stages v1 "stages v1" +expect_index v1 +expect "the first publish holds v1" base-1-1 app-1-1 +stages v2 "stages v2" +expect_index v2 +expect "v2 keeps v1's files" base-1-1 app-1-1 app-2-1 +stages v2 "stages v2 again" +expect_index v2 +expect "publishing v2 again still keeps v1's files" base-1-1 app-1-1 app-2-1 +stages v2 "stages v2 a third time" +expect "and so does a third publish of v2" base-1-1 app-1-1 app-2-1 +stages v3 "stages v3" +expect_index v3 +expect "v3 keeps v2's files and drops v1's" base-1-1 app-2-1 app-3-1 +stages v3 "stages v3 again" +expect "publishing v3 again keeps v2's files" base-1-1 app-2-1 app-3-1 + +# A package removed from the set stays for one version, as the rollback +make_repo v4 base-1-1 +stages v4 "stages v4" +expect_index v4 +expect "a set without app keeps v3's app as the way back" base-1-1 app-3-1 +stages v4 "stages v4 again" +expect "and still after publishing that set again" base-1-1 app-3-1 +make_repo v5 base-2-1 +stages v5 "stages v5" +expect "the next change drops the removed package" base-1-1 base-2-1 + +# Going back to an older set is a change too: the set it replaces stays +stages v4 "stages v4 after v5" +expect_index v4 +expect "staging an older set keeps the newer one it replaces" base-1-1 base-2-1 +stages v5 "stages v5 after v4" +expect_index v5 + +# A file no index lists goes, with its signature, also on a no-op +echo stray > "$ch/stray-1-1-aarch64.pkg.tar.zst" +echo stray > "$ch/stray-1-1-aarch64.pkg.tar.zst.sig" +stages v5 "stages v5 with a stray file" +expect "a no-op drops a file neither set lists" base-1-1 base-2-1 + +# A failed stage (a package or signature the new index lists is missing) +# changes neither the index nor what is kept; the files cp copied before it +# stopped go at the next stage, also when that one changes nothing +make_repo v6 base-2-1 app-6-1 +mv "$t/v6/app-6-1-aarch64.pkg.tar.zst" "$t/app-6-1.held" +if run_stage v6; then fail "refuses an index whose package is missing"; fi +expect_index v5 +kept "a failed stage keeps the index and every file" base-1-1 base-2-1 +stages v5 "stages v5 after the failed v6" +expect_index v5 +expect "the no-op after it drops what the failed stage copied" base-1-1 base-2-1 +mv "$t/v5/base-2-1-aarch64.pkg.tar.zst.sig" "$t/base-2-1.sig.held" +if run_stage v5; then fail "refuses a package without its signature"; fi +expect_index v5 +kept "a failed no-op stage keeps the index and every file" base-1-1 base-2-1 +mv "$t/base-2-1.sig.held" "$t/v5/base-2-1-aarch64.pkg.tar.zst.sig" +mv "$t/app-6-1.held" "$t/v6/app-6-1-aarch64.pkg.tar.zst" +stages v6 "stages v6 once its package is there" +expect_index v6 +expect "the next good stage holds v6 and v5 and nothing else" base-2-1 app-6-1 +extra=$(cd "$ch" && shopt -s nullglob dotglob && printf '%s\n' * | grep -vE '\.pkg\.tar\.|^omamobile\.(db|files)(\.tar\.gz)?(\.sig)?$' || true) +[[ -f $ch.sets && -z $extra ]] || fail "the record is beside the channel, which holds only packages and the index: $extra" +pass "the record of the sets is beside the channel, so it isn't uploaded" +rm -rf "$t/channel" +rm "$t/v6/app-6-1-aarch64.pkg.tar.zst" +if run_stage v6; then fail "refuses v6 in a fresh channel"; fi +[[ ! -e $ch/omamobile.db && ! -s $ch.sets ]] || fail "a failed first stage leaves no index" +pass "a failed first stage leaves no index and an empty record" + +# A stage that fails while copying the index leaves the database on disk +# new and the record as it was: the retry decides from the record, so it +# keeps the set before it. The same when the record can't be replaced. +rm -rf "$t/channel" +stages v1 "stages v1 in a new channel" +stages v2 "stages v2 in a new channel" +mv "$t/v3/omamobile.files.tar.gz" "$t/files.held" +if run_stage v3; then fail "refuses v3 without its .files archive"; fi +diff <(db_files "$t/v3") <(db_files "$ch") >&2 || fail "the failed v3 got as far as the database" +mv "$t/files.held" "$t/v3/omamobile.files.tar.gz" +stages v3 "retries v3" +expect_index v3 +expect "a retry after a failed index copy keeps v2's files" base-1-1 app-2-1 app-3-1 +cp "$ch.sets" "$t/sets.before" +if FAULT='mv() { return 75; }' run_stage v4; then fail "stops when the record can't be replaced"; fi +cmp -s "$ch.sets" "$t/sets.before" || fail "a record that can't be replaced stays as it was" +stages v4 "retries v4" +expect "a retry after a failed record keeps v3's files" base-1-1 app-3-1 +cp "$ch.sets" "$t/sets.before" +# shellcheck disable=SC2016 # FAULT is expanded by the stage's own shell +if FAULT='sed() { [[ $2 != "s/^./current &/p" ]] || return 75; command sed "$@"; }' run_stage v5; then + fail "stops when the record's current set can't be written" +fi +cmp -s "$ch.sets" "$t/sets.before" || fail "a record whose current set can't be written stays as it was" +stages v5 "retries v5" +stages v5 "stages v5 again" +expect "a retry after a half-written record keeps v4's files" base-1-1 base-2-1 +if FAULT='find() { return 1; }' run_stage v4; then fail "stops when the channel can't be listed for pruning"; fi +kept "a stage that can't list the channel deletes nothing" base-1-1 base-2-1 +FAULT='grep() { return 2; }' stages v5 "stages v5 with a failing grep" +expect "a failing grep can't make the prune delete what it keeps" base-1-1 base-2-1 + +# A channel staged before the record existed (v1 and v2's files, v2's +# index) gets one from what it has before anything is copied: a failed +# stage adds nothing to it, a no-op keeps v1, the next change rotates +rm -rf "$t/channel" +stages v1 "stages v1 for a channel without a record" +stages v2 "stages v2 for a channel without a record" +rm "$ch.sets" +if FAULT='pkg_files() { return 75; }' run_stage v2; then fail "stops when the channel's files can't be listed"; fi +if FAULT="bsdtar() { [[ \$2 != '$ch/'* ]] || return 75; command bsdtar \"\$@\"; }" run_stage v2; then + fail "stops when the channel's index can't be read" +fi +if ((EUID)); then + chmod 300 "$ch" + if run_stage v2; then chmod 755 "$ch"; fail "stops when the channel's directory can't be read"; fi + chmod 755 "$ch" +fi +[[ ! -e $ch.sets ]] || fail "a failed read records nothing" +pass "a channel whose files, index or directory can't be read gets no record" +mv "$t/v3/app-3-1-aarch64.pkg.tar.zst.sig" "$t/sig.held" +if run_stage v3; then fail "refuses v3 without app's signature"; fi +[[ -f $ch/app-3-1-aarch64.pkg.tar.zst ]] || fail "the failed v3 copied app-3-1 before it stopped" +mv "$t/sig.held" "$t/v3/app-3-1-aarch64.pkg.tar.zst.sig" +stages v2 "stages v2 after the failed v3" +expect_index v2 +expect "without a record, a no-op keeps v1 and drops what the failed v3 copied" base-1-1 app-1-1 app-2-1 +stages v3 "stages v3 after that" +expect "and the next change keeps only the set it replaces" base-1-1 app-2-1 app-3-1 diff --git a/tools/repo-publish b/tools/repo-publish index 283f4e38d..e87ee3045 100755 --- a/tools/repo-publish +++ b/tools/repo-publish @@ -27,7 +27,7 @@ # on GitHub, unless one is open already # # Each channel is out/publish//aarch64 here, mirrored to the server. -# It keeps the files of the current index and of the one before it, so a +# It keeps the files of the current index and the last different one, so a # phone can go back one version. Upload order: packages and signatures, then # the index, then the pruning, so a phone never sees an index that names a # file not there yet. Nothing is uploaded unless every package the index lists @@ -111,6 +111,10 @@ stb=$pub/stable/$ARCH # The package file names an index lists, one per line db_files() { [[ -f $1/$DB ]] && bsdtar -xOf "$1/$DB" --include='*/desc' | awk '/^%FILENAME%$/ { getline; print }' | sort; } +# The package file names a channel directory holds, one per line +pkg_files() { find "$1" -mindepth 1 -maxdepth 1 -name '*.pkg.tar.*' -printf '%f\n' | sed 's/\.sig$//' | sort -u; } +# save_sets FILE CURRENT PREVIOUS: record two package sets in FILE at once +save_sets() { { sed -n 's/^./current &/p' <<<"$2" && sed -n 's/^./previous &/p' <<<"$3"; } > "$1.new" && mv -f -- "$1.new" "$1"; } # "file sha256" for each package an index lists db_sums() { bsdtar -xOf "$1/$DB" --include='*/desc' | @@ -175,28 +179,47 @@ verify() { echo " $n packages and the index: checksums good, signed by the release key" } -# Bring a channel directory to a new index: the new files, plus the ones the -# previous index listed, nothing else. +# Bring a channel directory to a new index: the new files, plus those of the +# last different package set before it, nothing else. $to.sets, beside the +# channel (never uploaded), records the set last staged and the one before +# it. Every run decides from that record, never from the index on disk, and +# replaces it in one rename once the new index is in place, so a stage that +# fails anywhere changes nothing a retry decides by. Staging the same set +# again keeps the one before it, so a phone can still go back. A channel +# staged before the record existed starts it, before copying anything, from +# its index and every package file it has. Each read and write here is +# checked on its own, so one that fails stops the stage instead of +# recording, or keeping, less. stage() { local from=$1 to=$2 f mkdir -p "$to" - local previous - previous=$(db_files "$to" || true) + local new current="" previous + new=$(db_files "$from") + if [[ -f $to.sets ]]; then + current=$(sed -n 's/^current //p' "$to.sets") + previous=$(sed -n 's/^previous //p' "$to.sets") + else + if [[ -f $to/$DB ]]; then current=$(db_files "$to"); fi + previous=$(pkg_files "$to") + save_sets "$to.sets" "$current" "$previous" + fi + [[ $new == "$current" ]] || previous=$current while read -r f; do cp -p "$from/$f" "$from/$f.sig" "$to/" - done < <(db_files "$from") + done <<<"$new" for f in omamobile.db omamobile.files; do cp -pL "$from/$f.tar.gz" "$from/$f.tar.gz.sig" "$to/" cp -pL "$from/$f.tar.gz" "$to/$f" cp -pL "$from/$f.tar.gz.sig" "$to/$f.sig" done - local keep - keep=$( { db_files "$to"; if [[ -n $previous ]]; then echo "$previous"; fi; } | sort -u) - for f in "$to"/*.pkg.tar.*; do - [[ -e $f ]] || continue - f=${f##*/}; f=${f%.sig} - grep -qxF "$f" <<<"$keep" || rm -f "$to/$f" "$to/$f.sig" - done + save_sets "$to.sets" "$new" "$previous" + local -A keep=() + local have + while read -r f; do [[ -z $f ]] || keep[$f]=1; done <<<"$new"$'\n'"$previous" + have=$(pkg_files "$to") + while read -r f; do + [[ -z $f || -n ${keep[$f]:-} ]] || rm -f "$to/$f" "$to/$f.sig" + done <<<"$have" } write_index() {