Skip to content

install-phone: the next run removes the verify directory a killed run left - #682

Merged
derekross merged 1 commit into
derekross:mainfrom
only21mil:build/install-phone-sweep-verify-dirs
Nov 12, 2026
Merged

derekross merged 1 commit into
derekross:mainfrom
only21mil:build/install-phone-sweep-verify-dirs

Conversation

@only21mil

Copy link
Copy Markdown
Contributor

What and why

This follows the maintainer's note on #648: a run killed outright leaves its verify.XXXXXX behind, and nothing sweeps old ones.

Since #648 the shipped ramdisk is read in a verify.XXXXXX directory of the run's own in --work, removed when the run exits. A run killed with SIGKILL, a crash or a power cut never gets there, so its directory (the decompressed ramdisk) stays for good, one more per such run.

Age can't tell a dead run's directory from a live one's: an install can wait at the codename prompt for hours, and it needs its directory until it exits. So each run now takes a shared flock on --work itself (a descriptor on the directory, so nothing new is left in --work) and keeps it until it exits. It marks its own directory only once it holds that lock. A run that gets the lock exclusively, so no other run of this installer is live, first removes the marked directories: their runs have all ended. These are never removed: a directory not yet marked, one left by an installer from before this change (its run may still be going), and anything while another run is live. The lock comes from python3's fcntl, which the installer already needs. flock(1) is util-linux and not in its tool list, and the bundle ships install alone (L-497). The run sweeps only after acquiring the exclusive lock and marks its directory only after acquiring the shared lock. All of this assumes a flock that really excludes, on every host that shares the work directory.

Children inherit the descriptor, so a directory stays while any process of its run lives. gpg, whose agent can outlive it, runs before the lock is taken. Leaving the unmarked directories of today's installer alone is my choice; sweeping them would mean a still-running install from before this change could lose its directory.

I first reproduced this on main e6bc8e3 before writing the fix. The first pair below repeats that original recipe: SIGKILL goes only to the installer process. At the codename prompt, its answer-reading child survives and retains the inherited lock descriptor, so the candidate leaves that directory while the child remains alive. Step 2 removes step 1's directory, but its surviving child prevents every later sweep, leaving both step 2's and step 4's directories at the end.

I then changed the fault to kill the installer and all its descendants. The supplemental control repeats that revised recipe on main and b4cb0f9 after the fix was written; it is a post-fix control.

Both recipes run the bundle’s real ./install in bwrap with the checkout mounted at /mnt. Fastboot is a stand-in. The script removes colour escapes and selects install output with grep -E '==>|error'; parenthesized lines are the script’s notes.

Before, main e6bc8e3, the original recipe (SIGKILL to the installer process only), taken before I wrote the fix:

$ git rev-parse --short origin/main   # the bundle's install is this revision's tools/install-phone
e6bc8e3d
(bundle install == origin/main:tools/install-phone)
$ cd bundle/omamobile-twelve-2026.11.09   # each install below runs from here with --work work (a directory beside the bundle)

## 1. ./install --unsigned --verify-only --work work, killed while it reads the ramdisk
(killed with SIGKILL while its verify.* directory existed)
$ ls work
verify.YMTJu2

## 2. ./install --unsigned --work work, killed at "Type the phone's codename"
(killed with SIGKILL at the codename prompt)
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.8vHzKt
verify.YMTJu2

## 3. run again
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.8vHzKt
verify.YMTJu2

## 4. ./install --unsigned --verify-only --work work, killed while it reads the ramdisk
(killed with SIGKILL while its verify.* directory existed)
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.8vHzKt
verify.OlHLc7
verify.YMTJu2

## 5. run again
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.8vHzKt
verify.OlHLc7
verify.YMTJu2

## 6. a full install waits at the codename; --verify-only runs from a second terminal; the install goes ahead
(terminal 1: ./install --unsigned --work work, waiting at "Type the phone's codename")
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.7RnsO1
verify.8vHzKt
verify.OlHLc7
verify.YMTJu2

(terminal 2)
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.7RnsO1
verify.8vHzKt
verify.OlHLc7
verify.YMTJu2

(terminal 1: the codename typed)
==> Checking the files (bundle mode)
==> Checking the phone
==> About the phone's owner
Host name [alice-phone]: Time zone [Etc/UTC]: Full name for git (optional): Email for git (optional): Wi-Fi network name (optional; Enter to skip): SSH public key for root and alice [/mnt/logs/repro/fvo2b-origin-main/home/.ssh/id_ed25519.pub]: ==> Preparing a local copy of the root image in /mnt/logs/repro/fvo2b-origin-main/work
==> The plan
Type the phone's codename (twelve) to go ahead: ==> Flashing
==> Done. The phone boots slot B; the first boot sets up alice (a few minutes).
exit=0
fastboot calls: 10; partitions flashed: 6
$ ls work
verify.8vHzKt
verify.OlHLc7
verify.YMTJu2

After, this branch b4cb0f9, the original recipe (SIGKILL to the installer process only):

$ git rev-parse --short install-phone-sweep-verify-dirs   # the bundle's install is this revision's tools/install-phone
b4cb0f94
(bundle install == install-phone-sweep-verify-dirs:tools/install-phone)
$ cd bundle/omamobile-twelve-2026.11.09   # each install below runs from here with --work work (a directory beside the bundle)

## 1. ./install --unsigned --verify-only --work work, killed while it reads the ramdisk
(killed with SIGKILL while its verify.* directory existed)
$ ls work
verify.esb8e8

## 2. ./install --unsigned --work work, killed at "Type the phone's codename"
(killed with SIGKILL at the codename prompt)
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv

## 3. run again
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv

## 4. ./install --unsigned --verify-only --work work, killed while it reads the ramdisk
(killed with SIGKILL while its verify.* directory existed)
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv
verify.jeRBMS

## 5. run again
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv
verify.jeRBMS

## 6. a full install waits at the codename; --verify-only runs from a second terminal; the install goes ahead
(terminal 1: ./install --unsigned --work work, waiting at "Type the phone's codename")
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv
verify.jeRBMS
verify.mgVZAq

(terminal 2)
$ ./install --unsigned --verify-only --work work
==> Checking the files (bundle mode)
==> The files check out (the phone was not contacted)
exit=0
$ ls work
authorized_keys
boot-update.conf
firstboot.conf
ramdisk
root.simg
verify.Bjpygv
verify.jeRBMS
verify.mgVZAq

(terminal 1: the codename typed)
==> Checking the files (bundle mode)
==> Checking the phone
==> About the phone's owner
Host name [alice-phone]: Time zone [Etc/UTC]: Full name for git (optional): Email for git (optional): Wi-Fi network name (optional; Enter to skip): SSH public key for root and alice [/mnt/logs/repro/fvo2b-install-phone-sweep-verify-dirs/home/.ssh/id_ed25519.pub]: ==> Preparing a local copy of the root image in /mnt/logs/repro/fvo2b-install-phone-sweep-verify-dirs/work
==> The plan
Type the phone's codename (twelve) to go ahead: ==> Flashing
==> Done. The phone boots slot B; the first boot sets up alice (a few minutes).
exit=0
fastboot calls: 10; partitions flashed: 6
$ ls work
verify.Bjpygv
verify.jeRBMS

As a post-fix control, I used a recipe written after the fix, sending SIGKILL to the installer and all its descendants: the listings contain 1 2 2 3 3 | 4 4 3 verify.* directories on main e6bc8e3 and 1 1 0 1 0 | 1 1 0 on b4cb0f9; in both runs, the waiting install still flashes its six partitions.

Regression test (a stand-in, not the reproduction): in tests/install-phone-initramfs, an install waits at the codename and a second one is killed there with SIGKILL. --verify-only beside the waiting one removes nothing. The waiting one then flashes the hook's vendor_boot.img byte for byte, which its key append reads from its directory. A later --verify-only on its own removes the killed one's directory and leaves an unmarked one. With main's tool the test fails at "a run with no other live removes a killed install's directory, and nothing unmarked". Each of these mutants also fails it: no exclusive check, the shared lock let go after marking, no mark check, no mark written, or a blocking exclusive lock. No test covers marking before locking (a race).

The body is longer than usual because of the two primary transcripts.

Device and build

  • Phone: none: the laptop-side installer only
  • Kernel release (uname -r): not applicable
  • Packages changed, with versions: none

Checks (no phone needed)

  • tools/pr-check origin/main passes
  • tests/all passes (tests/install-phone-initramfs, with its byte comparisons, also under umask 077, 022 and 002)
  • tests/packages passes (or the packages I touched: ...): not run, no package changed
  • Each changed package has its version bumped, and the new version is in the commit subject: no package changed
  • Commits have area prefixes and bodies that say why; any AI help is a Co-Authored-By: or Assisted-by: trailer, never a Signed-off-by:

On the phone

  • I ran this on my own phone, booted into this build: no, the change is in the laptop-side installer
  • Runbook steps passed (plan, step, result): none, no plan step
  • What I did not test: a phone; a network-mounted home; a run killed between creating its directory and marking it (that directory stays, as on main); a file system whose flock silently does nothing (the sweep relies on working exclusion between every host that shares the work directory)
  • Nothing in this change writes outside slot B and userdata, weakens the storage guard, or reads registers from userspace (AGENTS.md §1)
tools/phone-smoke report

Not applicable: a laptop-side installer change; nothing on the phone changes.

Generated with Claude Opus 5.5 in T3 Code Nightly

… left

Since derekross#648 the shipped ramdisk is read in a verify.XXXXXX directory of
the run's own in --work, removed when the run exits. A run killed
outright (SIGKILL, a closed terminal's kill, a crash) never exits that
way, so its directory, with the decompressed ramdisk, stayed in --work
for good, one more for every such run.

A run can't tell a dead run's directory from a live one's by age (an
install can wait at the codename prompt for hours), and a full install
needs its directory until it exits. So each run holds a shared flock on
--work itself (a descriptor on the directory, no file of its own there)
until it exits, and marks its directory only once it holds the lock. A
run that gets the lock exclusively, so no other run of this installer is
live, first removes the marked directories: their runs have all ended.
A directory not yet marked, or left by an installer from before this
change (whose run may still be going), is never removed. flock comes
from python3's fcntl, which the installer already needs; flock(1) is
util-linux, not in its tool list, and the bundle ships install alone, so
the helper is inline (L-497). Where --work can't be locked (some network
file systems) nothing is removed and nothing is marked.

Tested: tests/install-phone-initramfs gains the case (an install waiting
at the codename, a second one killed with SIGKILL there, --verify-only
beside the waiting one, the waiting one then flashing the key's append,
and a later --verify-only alone), which fails with main's tool at "a run
with no other live removes a killed install's directory"; mutants
without the exclusive check, with the shared lock let go after marking,
without the mark check, or with no mark written each fail it, and a
blocking exclusive lock hangs it. Both install-phone suites pass, the
initramfs one under umask 077, 022 and 002; tools/pr-check origin/main;
tests/all. A bundle's ./install --unsigned, killed during
--verify-only's read and at the codename prompt, then run again: the
leftovers pile up on main and are gone on this branch, and an install
waiting at the codename keeps its directory and flashes. Not tested: on
a phone (laptop-side only); a network-mounted home; a run killed between
creating its directory and marking it (its directory stays, as on main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekross

Copy link
Copy Markdown
Owner

Thanks, @only21mil. That is the first of the two things I noted on #648, and you took the hard part seriously: age can't tell a dead run's directory from one waiting at the codename prompt, so a lock that dies with the run is the right witness. Marking the directory only once the shared lock is held, and sweeping only marked ones, is what makes the non-atomic lock change harmless. Thank you also for saying plainly what you didn't test.

I checked it merged onto today's main: tools/pr-check and tests/install-phone-initramfs pass, and the suite also passes with #681 applied next to it. With main's tool your case fails at "a run with no other live removes a killed install's directory, and nothing unmarked". Two changes of my own to the helper both fail the suite: sweeping without the exclusive lock, and sweeping directories that carry no mark.

On the case you name as untested, a shared home whose flock doesn't exclude between machines: I read what a live install does if its directory goes. Its only later read is the cat … shipped.cpio | lz4 when the key is appended, which stops the run before anything is flashed. So the worst case there is a refused install, and I'm fine with that.

Two small things for some later change, not this one:

  • A passing run of the suite now prints bash's job notice for the killed install ("line 136: … Killed … exec "$BUNDLE/install" …"), which in a CI log looks like a failure. disown on that job, or its stderr closed, would quiet it.
  • The header's line about --work could say that the directories of killed runs are removed there by the next run.

Merging.

@derekross
derekross merged commit 9804401 into derekross:main Nov 12, 2026
2 checks passed
@only21mil

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

#699 quiets the job notice. On its head the suite exited 0 with no "Killed" line, and the same run on main printed it. I closed the held job's stderr to /dev/null rather than use disown: with disown the notice still printed, and wait no longer waited for the killed run. The install's own output still goes to its OUT file, which the suite prints on a failure.

#700 adds one sentence to the --work entry: "The next run that finds no other run live removes killed runs' verify.* directories; ones left by installers without this cleanup are kept". It changes the help text only.

derekross pushed a commit that referenced this pull request Nov 13, 2026
#699)

Since #682 a passing run printed bash's job notice for the install the suite kills, which reads as a failure in a CI log. The notice comes from the held job's own shell, so that job's stderr now goes to `/dev/null` (its output is in its OUT file as before, which the suite prints on a failure) and the `wait` is plain. Keep the `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>` trailer.
derekross pushed a commit that referenced this pull request Nov 13, 2026
…ory (#700)

Since #682 a run that finds no other run of the installer live removes the `verify.*` directories killed runs left in `--work`, but `--help` described `--work` only as where the image copy goes. The entry now says so, and that directories left by installers without this cleanup are kept. Header comment only; the code is unchanged. Keep the `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>` trailer.
robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
… left (derekross#682)

A run killed outright never reaches its exit trap, so its verify.XXXXXX (the decompressed ramdisk) stayed in --work for good. Each run now holds a shared flock on --work until it exits and marks its directory once it holds it; a run that gets the lock exclusively, so no other run is live, first removes the marked directories. An unmarked directory, and anything while another run is live, is never removed; where --work can't be locked nothing is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants