Skip to content

extract-vendor-firmware: build each run in a new generation - #491

Merged
derekross merged 4 commits into
derekross:mainfrom
only21mil:build/c199-vendor-firmware-generation-r2
Nov 8, 2026
Merged

derekross merged 4 commits into
derekross:mainfrom
only21mil:build/c199-vendor-firmware-generation-r2

Conversation

@only21mil

@only21mil only21mil commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What and why

tools/extract-vendor-firmware wrote straight into its output directory. The plans run it into the default out/vendor-firmware and then tar that tree onto the phone (WIFI-BATTERY-PLAN §2.2, BLUETOOTH-PLAN, AOC-PLAN C8.0). So:

  • A rerun for another image or device kept the earlier run's firmware and sensor registry files, and the firmware went into vendor-firmware.sha256 as if this run had produced it.
  • On ext4, debugfs exits 0 and writes nothing when a file is missing, so an older copy with the same name passed the size check.
  • debugfs also exits 0 when a copy is cut short, for example by a full disk or a file size limit. It only says so on stderr, so a truncated file passed the size check too.

A run that failed half way also left old and new files mixed.

Each run now builds a whole new generation:

  • It extracts into an empty staging directory inside the output directory (the -c camera APEX into a new file beside it), and runs the sanity checks there. It also checks that the checksum list names exactly the mapped files; that check is a backstop, since no reachable state trips it, so no test covers it directly (the tests check the list's contents after every successful run).
  • Only then does it rename the previous outputs aside and the new ones into place, then the -c file. A failed run leaves the previous outputs and -c untouched; if a rename fails while publishing, what was replaced is put back.
  • A debugfs copy fails on anything debugfs prints to stderr, apart from its banner and rdump's notes that, as a user, it can't give files the image's owner (root).
  • A -c that names a directory or sits inside the replaced outputs is refused.
  • An output directory that resolves to / (-o /, //, /usr/.., a symlink to /) is refused before anything is written, so a mistyped -o can't make the run replace this machine's own /usr/lib/firmware/updates and /usr/share/omamobile/sensors.

The script replaces only what it writes: usr/lib/firmware/updates, usr/share/omamobile/sensors, vendor-firmware.sha256 and the -c file. The rest of the output directory stays, and tar -C OUTDIR -cf - usr works unchanged: the staging directory is hidden at the top and removed on exit. tools/phone-provision already uses a new directory per run, so it avoids stale-output effects and gains the stricter debugfs, checksum-list, and -c destination checks.

tests/extract-vendor-firmware (new, added to tests/all) runs the tool on made-up files, with stand-ins for debugfs and fsck.erofs that report errors the way debugfs does: on stderr, with exit 0. Where the host has mkfs.ext4 and debugfs, it also runs on a real ext4 image of the same files, with a root-owned registry, and with debugfs under a file size limit. It checks that:

  • a reused directory keeps no earlier firmware or registry file;
  • a husky run drops an earlier cheetah registry;
  • each of these leaves the previous outputs byte for byte: a missing file, a missing registry, a short write (also under a file size limit), a failed sanity check, a missing camera APEX, a bad -c and a failed rename;
  • -o /, //, /usr/.. and a symlink to / are refused before anything is written: no scratch directory is made and nothing is printed as done. The image in these checks is all zeros, so a tool without the refusal stops at its image check instead, and the test never writes under /.

EVF=path runs the same test against another copy of the tool.

Device and build

  • Phone: none: tools only. This runs on the laptop and never on a phone (CONTRIBUTING §6.5).
  • Kernel release (uname -r): n/a
  • Packages changed, with versions: none
  • Screenshots: this command-line tool runs on the laptop and changes nothing on a screen, so the before and after are terminal runs of the new test, on made-up files.

Before: the new test against main's tool (7172296): 2 ok and 26 not ok lines ("26 check(s) failed"). The captures below are from the first version of this PR, before the -o / checks were added (then 2 ok, 22 failed).

tests/extract-vendor-firmware run against main's extract-vendor-firmware: 22 check(s) failed

After: the same test against the first commit (8a456cb): 20 ok lines and exit 0. With the second commit the test has 24 ok lines and exits 0; the -o / captures follow.

tests/extract-vendor-firmware run against this change: 20 ok lines and exit 0

The -o / refusal, before: the test against the first commit's tool (8a456cb), which has no refusal: the four -o / checks fail ("4 check(s) failed"), each stopping at the image check, and nothing is written.

tests/extract-vendor-firmware against 8a456cb: the four -o / checks fail, 4 check(s) failed

After: the same test against this change: 24 ok lines and exit 0.

tests/extract-vendor-firmware against this change: 24 ok lines and exit 0

Checks (no phone needed)

  • tools/pr-check origin/main passes
  • tests/all passes. Skipped on this host for reasons unrelated to this change: phone-provision's container leg (no Docker), the AOSP mkbootimg layouts, dtc overlays, lz4, repo-add, bsdtar, and the root image smoke test.
  • tests/packages: not run; no package touched
  • Each changed package has its version bumped: none changed
  • Commits have area prefixes and bodies that say why; AI help is a Co-Authored-By: trailer

On the phone

  • I ran this on my own phone: not applicable, since the tool runs on the laptop
  • Runbook steps passed: none
  • What I did not test:
    • a real Google factory image (only made-up files);
    • real EROFS images (stand-in only; no erofs-utils on the test host);
    • the sparse and super.img paths, which are unchanged;
    • panther, raven and tegu maps, which the publish code doesn't depend on;
    • a failed write of the final success message after publishing (fixed by structure, no test).
  • Nothing in this change writes outside slot B and userdata, weakens the storage guard, or reads registers from userspace. It doesn't touch the phone.
tools/phone-smoke reportNot applicable: this tools-only change never runs on a phone.

Found in an audit of the tools by Kimi K3.1 High and Sol 6.1 xhigh.

Prior PRs: none make this change. The open modem-sit drafts (#374, #376, #378, #380) add tegu entries to the same file's map; those hunks merge with this change without a conflict, and their files land in the same generation.

Generated with Claude Opus 5.5 in T3 Code Nightly

The extractor wrote straight into OUTDIR. The plans run it into the
default out/vendor-firmware and tar that tree onto the phone, so a rerun
for another image or device kept the earlier run's firmware and sensor
registry files, and the firmware went into vendor-firmware.sha256 as if
it were this run's. On ext4, debugfs exits 0 when a copy fails: for a
missing file it writes nothing, so an older copy of the same name passed
the size check, and for a short write (a full disk, a file size limit)
it leaves a truncated file that passed too. A run that failed half way
left a mix of old and new files.

Each run now extracts into an empty staging directory inside OUTDIR
(the -c camera APEX into a new file beside it), runs the checks there,
checks that the checksum list names exactly the mapped files, and only
then moves the previous outputs aside and the new ones into place:
renames within OUTDIR, then the -c file. A failed run leaves the
previous outputs and the -c file as they were; if a rename fails while
publishing, what was already replaced is put back. The script replaces
only what it writes: usr/lib/firmware/updates,
usr/share/omamobile/sensors, vendor-firmware.sha256 and the -c file;
the rest of OUTDIR stays. An ext4 copy fails on any debugfs message but
its banner and the exact notes that it can't give a file the image's
owner (as a user); the messages are read in the shell, so a nearly full
disk can't hide them. A -c that names a directory, or a path inside the
outputs the run replaces, is refused before anything is extracted.
tools/phone-provision already extracts into a new directory per run;
it calls the tool the same way as before.

tests/extract-vendor-firmware (run by tests/all) runs the tool on
made-up files with stand-in debugfs and fsck.erofs that print real
debugfs's messages, and, where the host has mkfs.ext4 and debugfs, on a
real ext4 image of the same files with a root-owned registry: a reused
directory keeps no earlier firmware or registry file, a husky run drops
a cheetah registry, and a missing file or registry, a short write (also
under a file size limit), a failed sanity check, a missing camera APEX
or a failed rename leaves the previous outputs byte for byte. Against
the tool before this change, only 2 of its checks pass. No factory
image was read and nothing ran on a phone.

Found in an audit of the tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derekross

Copy link
Copy Markdown
Owner

Thank you, this is a careful piece of work. Reusing out/vendor-firmware really did carry another image's firmware into the checksum list. debugfs's exit 0 on a missing or truncated copy is exactly the kind of thing that hides until it hurts. Building each run in a fresh directory and renaming it into place fixes both.

I read the test, ran it (20 ok; against main's tool, 22 fail, as you said) and broke the tool on purpose in a few places. Making restore() do nothing, making the debugfs filter accept everything, and dropping the -c refusal are each caught. A first run on your fixture gives byte-for-byte the same tree as main's tool, so nothing changes for a phone that provisions from a fresh directory. I also ran plain debugfs 1.47.4, as a user, on the real Pixel 7 Pro vendor image I have. Every stderr line was the banner or one of the ownership notes your filter allows, so the stricter check passes on a real image too.

It's ready to merge. One practical thing: #492 also edits tests/all (the comment and the suite loop), so whichever of the two lands second will conflict. If this one goes second, the loop's end is tests/repo-publish tests/repo-publish-stage tests/extract-vendor-firmware. I can resolve it at merge time.

Two small, optional notes:

  • The tool now moves aside and deletes all of usr/share/omamobile/sensors and usr/lib/firmware/updates under OUTDIR. Nothing runs it with -o /, but a refusal for / would make that mistake impossible.
  • Removing the "checksum list names exactly the mapped files" check doesn't fail any test, because no reachable state can trip it. It's fine as a backstop. I'd just not describe it as tested.

derekross and others added 2 commits November 7, 2026 22:26
Publishing moves the previous usr/lib/firmware/updates and
usr/share/omamobile/sensors aside under OUTDIR and removes them once the
new generation is in place. Nothing calls the tool with -o /, but an
operator who did would have the host's own directories replaced. The
tool now refuses an OUTDIR whose resolved path is /, right after it is
resolved and before the scratch directory or anything else is written.

tests/extract-vendor-firmware runs the tool with -o /, //, /usr/.. and a
symlink to /, and checks the refusal message and that no scratch
directory was made. Its image is all zeros, so a tool without the
refusal stops at the image check before it could write under /: against
the previous commit's tool these four checks fail and nothing is
written there. The test header now says the checksum list is checked
after a successful run; the tool's own check that the list names exactly
the mapped files is a backstop no reachable state trips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@only21mil

only21mil commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Note

🤖 Opus 5.5 replying on behalf of Victor.

Implemented in 8d08b8c:

  • tools/extract-vendor-firmware now rejects an output directory that resolves to / before creating work, staging files or publishing.
  • tests/extract-vendor-firmware covers /, //, /usr/.. and a symlink to /. The repaired tool passes all 24 checks; the prior head fails exactly those four guard checks.
  • The PR description now identifies the checksum-list exact-mapped-files check as a backstop and does not claim that branch directly exercises it.

The existing checks and the original change remain unchanged; the shared tests/all merge conflict is still left for maintainer resolution.

@derekross
derekross merged commit 8237df5 into derekross:main Nov 8, 2026
2 checks passed
robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
…s#491)

The extractor wrote straight into OUTDIR. The plans run it into the
default out/vendor-firmware and tar that tree onto the phone, so a rerun
for another image or device kept the earlier run's firmware and sensor
registry files, and the firmware went into vendor-firmware.sha256 as if
it were this run's. On ext4, debugfs exits 0 when a copy fails: for a
missing file it writes nothing, so an older copy of the same name passed
the size check, and for a short write (a full disk, a file size limit)
it leaves a truncated file that passed too. A run that failed half way
left a mix of old and new files.

Each run now extracts into an empty staging directory inside OUTDIR
(the -c camera APEX into a new file beside it), runs the checks there,
checks that the checksum list names exactly the mapped files, and only
then moves the previous outputs aside and the new ones into place:
renames within OUTDIR, then the -c file. A failed run leaves the
previous outputs and the -c file as they were; if a rename fails while
publishing, what was already replaced is put back. The script replaces
only what it writes: usr/lib/firmware/updates,
usr/share/omamobile/sensors, vendor-firmware.sha256 and the -c file;
the rest of OUTDIR stays. An ext4 copy fails on any debugfs message but
its banner and the exact notes that it can't give a file the image's
owner (as a user); the messages are read in the shell, so a nearly full
disk can't hide them. A -c that names a directory, or a path inside the
outputs the run replaces, is refused before anything is extracted, and
so is an -o that resolves to / (/, //, /usr/.., a link to /), before
the scratch directory is made.
tools/phone-provision already extracts into a new directory per run;
it calls the tool the same way as before.

tests/extract-vendor-firmware (run by tests/all) runs the tool on
made-up files with stand-in debugfs and fsck.erofs that print real
debugfs's messages, and, where the host has mkfs.ext4 and debugfs, on a
real ext4 image of the same files with a root-owned registry: a reused
directory keeps no earlier firmware or registry file, a husky run drops
a cheetah registry, and a missing file or registry, a short write (also
under a file size limit), a failed sanity check, a missing camera APEX
or a failed rename leaves the previous outputs byte for byte. Against
the tool before this change, only 2 of its checks pass. No factory
image was read and nothing ran on a phone.

Found in an audit of the tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants