From 8a456cbef2c1a455cb7c2db2b4d3b153be73942b Mon Sep 17 00:00:00 2001 From: only21mil Date: Sat, 7 Nov 2026 15:11:38 +0000 Subject: [PATCH 1/2] extract-vendor-firmware: build each run in a new generation The extractor wrote straight into OUTDIR. The plans run it into the default out/vendor-firmware and tar that tree onto the phone, so a rerun for another image or device kept the earlier run's firmware and sensor registry files, and the firmware went into vendor-firmware.sha256 as if it were this run's. On ext4, debugfs exits 0 when a copy fails: for a missing file it writes nothing, so an older copy of the same name passed the size check, and for a short write (a full disk, a file size limit) it leaves a truncated file that passed too. A run that failed half way left a mix of old and new files. Each run now extracts into an empty staging directory inside OUTDIR (the -c camera APEX into a new file beside it), runs the checks there, checks that the checksum list names exactly the mapped files, and only then moves the previous outputs aside and the new ones into place: renames within OUTDIR, then the -c file. A failed run leaves the previous outputs and the -c file as they were; if a rename fails while publishing, what was already replaced is put back. The script replaces only what it writes: usr/lib/firmware/updates, usr/share/omamobile/sensors, vendor-firmware.sha256 and the -c file; the rest of OUTDIR stays. An ext4 copy fails on any debugfs message but its banner and the exact notes that it can't give a file the image's owner (as a user); the messages are read in the shell, so a nearly full disk can't hide them. A -c that names a directory, or a path inside the outputs the run replaces, is refused before anything is extracted. tools/phone-provision already extracts into a new directory per run; it calls the tool the same way as before. tests/extract-vendor-firmware (run by tests/all) runs the tool on made-up files with stand-in debugfs and fsck.erofs that print real debugfs's messages, and, where the host has mkfs.ext4 and debugfs, on a real ext4 image of the same files with a root-owned registry: a reused directory keeps no earlier firmware or registry file, a husky run drops a cheetah registry, and a missing file or registry, a short write (also under a file size limit), a failed sanity check, a missing camera APEX or a failed rename leaves the previous outputs byte for byte. Against the tool before this change, only 2 of its checks pass. No factory image was read and nothing ran on a phone. Found in an audit of the tools. Co-Authored-By: Claude Opus 5.5 --- tests/all | 8 +- tests/extract-vendor-firmware | 343 ++++++++++++++++++++++++++++++++++ tools/extract-vendor-firmware | 111 +++++++++-- 3 files changed, 445 insertions(+), 17 deletions(-) create mode 100755 tests/extract-vendor-firmware diff --git a/tests/all b/tests/all index 06bb61eef..3b460787b 100755 --- a/tests/all +++ b/tests/all @@ -20,8 +20,10 @@ # a scratch server, tools/extract-camera-data's adrp decoding, # tools/phone-smoke's dmesg error classification, the OWL conformance # report's differences on the host, bin/lineage-ref sync's branch handling -# against stand-ins for docker and repo, and tools/repo-publish --from-main's -# copy of the prebuilt binaries into a fixture's publish tree. +# against stand-ins for docker and repo, tools/repo-publish --from-main's +# copy of the prebuilt binaries into a fixture's publish tree, and +# tools/extract-vendor-firmware's output generations (made-up files, no +# factory image). set -euo pipefail cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." for f in tools/install-phone tools/phone-provision tools/nv-backup tools/build-rootfs tools/build-packages tools/make-bundle \ @@ -38,7 +40,7 @@ sh -n device/flame/initramfs/build bash -n tools/build-flame-probe echo "ok - scripts parse" python3 tests/bt-uart-gpio.py -for t in tests/pr-check tests/kernel-pr-check tests/phone-smoke tests/alarm-wake-test tests/install-phone tests/build-rootfs tests/rootfs-resolve tests/phone-provision tests/nv-backup tests/initramfs-grow tests/images-layout tests/boot-update-parity tests/stage-boot-files tests/initramfs-reproducible tests/device-overlays tests/simplefb-trigger tests/initramfs-steps tests/storage-guard tests/usb-product tests/boot-method tests/modules-push tests/make-bundle-keys tests/make-bundle tests/release-signing tests/pinned-packages tests/flame-probe packages/omamobile-firstboot/test/all packages/omamobile-device-felix/test/all packages/omamobile-device-lynx/test/all packages/omamobile-device-panther/test/all tests/local-builds tests/phone-install-package tests/publish-kernel tests/fetch-toolchain tests/release-upload tests/extract-camera-data tests/release tests/lineage-ref tests/repo-publish; do +for t in tests/pr-check tests/kernel-pr-check tests/phone-smoke tests/alarm-wake-test tests/install-phone tests/build-rootfs tests/rootfs-resolve tests/phone-provision tests/nv-backup tests/initramfs-grow tests/images-layout tests/boot-update-parity tests/stage-boot-files tests/initramfs-reproducible tests/device-overlays tests/simplefb-trigger tests/initramfs-steps tests/storage-guard tests/usb-product tests/boot-method tests/modules-push tests/make-bundle-keys tests/make-bundle tests/release-signing tests/pinned-packages tests/flame-probe packages/omamobile-firstboot/test/all packages/omamobile-device-felix/test/all packages/omamobile-device-lynx/test/all packages/omamobile-device-panther/test/all tests/local-builds tests/phone-install-package tests/publish-kernel tests/fetch-toolchain tests/release-upload tests/extract-camera-data tests/release tests/lineage-ref tests/repo-publish tests/extract-vendor-firmware; do echo "# $t" "$t" done diff --git a/tests/extract-vendor-firmware b/tests/extract-vendor-firmware new file mode 100755 index 000000000..423f0f2e1 --- /dev/null +++ b/tests/extract-vendor-firmware @@ -0,0 +1,343 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT +# Tests for tools/extract-vendor-firmware with made-up vendor files (no +# factory image, no Google firmware): a small file with the ext4 or EROFS +# magic stands in for vendor.img, and stand-in debugfs and fsck.erofs read +# a fixture tree instead (with real debugfs's stderr: a missing file or a +# failed write is an error message and exit 0). Checked: a run into a +# reused output directory keeps no firmware or sensor registry file from +# an earlier run (another device's, or one the new image doesn't have), +# and leaves the rest of the directory alone; a husky run drops an earlier +# registry; the checksum list names exactly the mapped files; a failed run +# (a missing file or registry, debugfs reporting a short write, also +# under a file size limit, a failed sanity check, a missing camera APEX) +# leaves the previous outputs and the -c file byte for byte, so an older +# file can't pass for a missing one; a -c that is a directory or inside +# the outputs is refused; and a move that fails while publishing puts the +# previous outputs back. With mkfs.ext4 +# and debugfs on the host, the same on a real ext4 image whose registry +# belongs to root, and a copy that a file size limit cuts short. EVF=path +# runs another copy of the tool (a negative control), so every check runs +# and the failures are counted rather than stopping at the first. +set -euo pipefail +cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." +EVF=$(realpath -- "${EVF:-tools/extract-vendor-firmware}") +failed=0 +fail() { printf 'not ok - %s\n' "$1" >&2; failed=$((failed + 1)); } +pass() { printf 'ok - %s\n' "$1"; } +# pass_since N MESSAGE: pass if nothing failed since the count was N +pass_since() { if ((failed == $1)); then pass "$2"; fi; } +t=$(mktemp -d) +trap 'rm -rf "$t"' EXIT +export PATH=$PATH:/usr/sbin:/sbin + +# --- Stand-ins ----------------------------------------------------------------- +mkdir -p "$t/bin" +cat > "$t/bin/debugfs" <<'EOF' +#!/bin/bash +# debugfs -R "dump /SRC OUT" IMG, or "rdump /DIR OUTDIR", from $FAKE_VENDOR, +# with real debugfs's stderr: its banner, rdump's ownership note (run as a +# user), and errors with exit 0; a dump of the path in .fail-dump, or the +# rdump when .fail-rdump exists, writes part and reports a write error +set -u +echo "debugfs 1.47.2 (stand-in)" >&2 +read -r cmd src out <<<"$2" +if [[ -f $FAKE_VENDOR/.fail-dump && $src == "/$(cat "$FAKE_VENDOR/.fail-dump")" ]] || + [[ $cmd == rdump && -f $FAKE_VENDOR/.fail-rdump ]]; then + if [[ $cmd == dump ]]; then + echo partial > "$out" + echo "dump_file: File too large while writing file \"$src\"" >&2 + else # after many real notes, an error whose file name holds their words + mkdir -p "$out/${src##*/}" && echo partial > "$out/${src##*/}/part.json" + for n in $(seq 800); do + echo "dump_file: Operation not permitted while changing ownership of $out/${src##*/}/e$n.json" + done >&2 + echo "rdump: Too many open files while opening $out/${src##*/}/x while changing ownership of y.json" >&2 + fi + exit 0 +fi +case $cmd in + dump) [[ -f $FAKE_VENDOR$src ]] && cp "$FAKE_VENDOR$src" "$out" ;; + rdump) [[ -d $FAKE_VENDOR$src ]] && cp -r "$FAKE_VENDOR$src" "$out/" && + echo "dump_file: Operation not permitted while changing ownership of $out/${src##*/}/x" >&2 && + echo "rdump: Operation not permitted while changing ownership of $out/${src##*/}" >&2 ;; +esac || echo "$cmd: File not found by ext2_lookup while looking up $src" >&2 +exit 0 +EOF +cat > "$t/bin/fsck.erofs" <<'EOF' +#!/bin/bash +# fsck.erofs --extract=DIR --no-preserve IMG, from $FAKE_VENDOR +set -eu +dir=${1#--extract=} +mkdir -p "$dir" && cp -r "$FAKE_VENDOR/." "$dir/" +EOF +chmod +x "$t/bin/debugfs" "$t/bin/fsck.erofs" + +# vendor.img stand-ins: only the magic the tool checks +head -c 4096 /dev/zero > "$t/ext4.img" +printf '\x53\xef' | dd of="$t/ext4.img" bs=1 seek=1080 conv=notrunc status=none +head -c 4096 /dev/zero > "$t/erofs.img" +printf '\xe2\xe1\xf5\xe0' | dd of="$t/erofs.img" bs=1 seek=1024 conv=notrunc status=none + +# --- Made-up vendor trees -------------------------------------------------------- +halo() { printf 'WMFW\0\0\0\0\0\0\004 %s' "$*"; } # WMFW, byte 10 = 4 (Halo core) +# make_vendor DIR DEVICE GENERATION: the files the tool maps, its sanity +# checks pass; GENERATION goes into every file +make_vendor() { + local v=$1 dev=$2 g=$3 f + mkdir -p "$v/firmware/brcm" "$v/etc/sensors/registry" "$v/apex" + printf 'ro.product.vendor.device=%s\nro.vendor.build.fingerprint=fictional/%s\n' "$dev" "$g" > "$v/build.prop" + if [[ $dev == husky ]]; then + printf 'fictional 4398d0-roml Version=1.0.3 (%s)\n' "$g" > "$v/firmware/fw_bcmdhd.bin_4398_d0" + printf 'NVRAMRev=%s\n' "$g" > "$v/firmware/bcmdhd.cal_USI_G5SN_9957_V17" + echo "clm $g" > "$v/firmware/bcmdhd_clm.blob_4398_d0" + echo "txcap $g" > "$v/firmware/bcmdhd_txcap.blob_4398_d0" + return + fi + printf 'fictional 4389c1-roml Version=1.0.1 (%s)\n' "$g" > "$v/firmware/fw_bcmdhd.bin" + printf 'NVRAMRev=%s\n' "$g" > "$v/firmware/bcmdhd.cal" + echo "clm $g" > "$v/firmware/bcmdhd_clm.blob" + echo "BCM4389C1 fictional $g" > "$v/firmware/brcm/BCM.hcd" + echo "aoc $g" > "$v/firmware/aoc.bin" + halo "FIRMWARE_CS40L26 BUZZ_EFFECTS2_BUZZ_LEVEL $g" > "$v/firmware/cs40l26.wmfw" + halo "calib $g" > "$v/firmware/cs40l26-calib.wmfw" + for f in cs40l26.bin cs40l26-svc.bin cs40l26-dvl.bin; do echo "WMDR $f $g" > "$v/firmware/$f"; done + halo "FIRMWARE_HALO_CSPL CAL_CHECKSUM $g" > "$v/firmware/cs35l41-dsp1-spk-prot.wmfw" + echo "WMDR top $g" > "$v/firmware/cs35l41-dsp1-spk-prot.bin" + echo "WMDR bottom $g" > "$v/firmware/R-cs35l41-dsp1-spk-prot.bin" + echo '3,1,2' > "$v/firmware/fast_switch3.txt" + echo '3,1,3' > "$v/firmware/fast_switch4.txt" + echo "{\"accel\": \"$g\"}" > "$v/etc/sensors/registry/fictional_accel.json" + echo "apex $g" > "$v/apex/com.google.pixel.camera.hal.apex" +} +make_vendor "$t/v1" cheetah gen1 +make_vendor "$t/v2" cheetah gen2 +make_vendor "$t/husky" husky gen3 + +FW=usr/lib/firmware/updates +REG=usr/share/omamobile/sensors/registry +CHEETAH_FILES="aoc.bin +brcm/BCM.google,gs201-cheetah.hcd +brcm/brcmfmac4389c1-pcie.google,gs201-cheetah.bin +brcm/brcmfmac4389c1-pcie.google,gs201-cheetah.clm_blob +brcm/brcmfmac4389c1-pcie.google,gs201-cheetah.txt +cirrus/cs35l41-cheetah-bottom-media-delta.txt +cirrus/cs35l41-cheetah-top-media-delta.txt +cirrus/cs35l41-dsp1-spk-prot-cheetah-bottom.bin +cirrus/cs35l41-dsp1-spk-prot-cheetah-top.bin +cirrus/cs35l41-dsp1-spk-prot-cheetah.wmfw +cs40l26-calib.wmfw +cs40l26-dvl.bin +cs40l26-svc.bin +cs40l26.bin +cs40l26.wmfw" + +# run VENDOR IMG ARGS...: the tool with the stand-ins first on PATH +run() { + local v=$1 img=$2; shift 2 + FAKE_VENDOR=$v PATH=$t/bin:$PATH "$EVF" -w "$t/work" "$@" "$img" > "$t/log" 2>&1 +} +# Everything in a directory, contents included, hidden staging dirs too +snap() { (cd "$1" && find . -printf '%y %m %p\n' | LC_ALL=C sort && find . -type f -print0 | LC_ALL=C sort -z | xargs -0r sha256sum); } +listed() { cut -c67- "$1/vendor-firmware.sha256" | LC_ALL=C sort; } +# A first generation in a fresh directory, plus files a previous run (or +# another device's) left behind, and a file that isn't the tool's +seed() { + local o=$1 + rm -rf "$o" + run "$t/v1" "$t/ext4.img" -o "$o" || { cat "$t/log"; fail "seed: a first extraction into $o succeeds"; return; } + echo stale > "$o/$FW/brcm/brcmfmac4389c1-pcie.google,gs201-panther.bin" + echo stale > "$o/$FW/old-only.bin" + echo stale > "$o/$REG/stale_sensor.json" + echo mine > "$o/notes.txt" +} +# A failed run over a seeded directory leaves it (and the -c file) exactly +# as it was +refuses() { # NAME EXPECTED-ERROR VENDOR IMG ARGS... + local name=$1 err=$2 o=$t/refused before; shift 2 + seed "$o" + [[ -s $o/vendor-firmware.sha256 ]] || { fail "$name: there is a previous generation to keep"; return; } + before=$(snap "$o") + if run "$@" -o "$o"; then fail "$name: refused"; return; fi + grep -qF -- "$err" "$t/log" || { cat "$t/log"; fail "$name: says '$err'"; return; } + [[ $(snap "$o") == "$before" ]] || { diff <(echo "$before") <(snap "$o") || true; fail "$name: the previous outputs stay as they were"; return; } + pass "$name: refused, the previous outputs untouched" +} + +# --- A fresh run --------------------------------------------------------------- +o=$t/fresh n=$failed +if run "$t/v1" "$t/ext4.img" -o "$o"; then + [[ $(listed "$o") == "$CHEETAH_FILES" ]] || fail "fresh: the checksum list names exactly the mapped files" + (cd "$o/$FW" && sha256sum --quiet -c "$o/vendor-firmware.sha256") || fail "fresh: the checksums match" + top=$(find "$o" -mindepth 1 -maxdepth 1 -printf '%f\n' | LC_ALL=C sort | tr '\n' ' ') + [[ $top == "usr vendor-firmware.sha256 " ]] || fail "fresh: nothing but usr and the checksum list in OUT (got: $top)" + [[ $(cat "$o/$REG/fictional_accel.json") == '{"accel": "gen1"}' ]] || fail "fresh: the registry is copied" + pass_since "$n" "fresh: a first run stages the mapped files, the registry and their checksums" +else + cat "$t/log"; fail "fresh: a first run succeeds" +fi + +# --- A reused output directory ------------------------------------------------- +o=$t/reuse n=$failed +seed "$o" +if run "$t/v2" "$t/ext4.img" -o "$o"; then + [[ ! -e "$o/$FW/brcm/brcmfmac4389c1-pcie.google,gs201-panther.bin" && ! -e "$o/$FW/old-only.bin" ]] || + fail "reuse: no firmware from an earlier run stays" + [[ ! -e "$o/$REG/stale_sensor.json" ]] || fail "reuse: no sensor registry file from an earlier run stays" + [[ $(listed "$o") == "$CHEETAH_FILES" ]] || fail "reuse: the checksum list names only this run's files" + [[ $(cat "$o/$FW/aoc.bin") == "aoc gen2" && $(cat "$o/$REG/fictional_accel.json") == '{"accel": "gen2"}' ]] || + fail "reuse: the files are this run's" + [[ $(cat "$o/notes.txt") == mine ]] || fail "reuse: a file that isn't the tool's stays" + [[ -z $(find "$o" -maxdepth 1 -name '.extract-vendor-firmware.*') ]] || fail "reuse: no staging directory left in OUT" + pass_since "$n" "reuse: a second run replaces the earlier generation and keeps nothing from it" +else + cat "$t/log"; fail "reuse: a second run succeeds" +fi + +o=$t/husky-reuse n=$failed +seed "$o" +if run "$t/husky" "$t/ext4.img" -d husky -o "$o"; then + [[ ! -e $o/usr/share/omamobile/sensors ]] || fail "husky: an earlier device's sensor registry is dropped" + [[ $(listed "$o" | tr '\n' ' ') == "brcm/brcmfmac4398d0-pcie.google,zuma-husky.bin brcm/brcmfmac4398d0-pcie.google,zuma-husky.clm_blob brcm/brcmfmac4398d0-pcie.google,zuma-husky.txcap_blob brcm/brcmfmac4398d0-pcie.google,zuma-husky.txt " ]] || + fail "husky: the checksum list names husky's four files only" + [[ $(find "$o/$FW" -type f | wc -l) == 4 ]] || fail "husky: no cheetah firmware stays" + pass_since "$n" "husky: a run over a cheetah generation leaves only husky's files" +else + cat "$t/log"; fail "husky: a run over a cheetah generation succeeds" +fi + +# --- Failed runs leave the previous generation ---------------------------------- +rm -rf "$t/v2-missing"; cp -r "$t/v2" "$t/v2-missing"; rm "$t/v2-missing/firmware/aoc.bin" +refuses "a missing ext4 file with an older copy in OUT" "/vendor/firmware/aoc.bin is missing" "$t/v2-missing" "$t/ext4.img" +refuses "a missing EROFS file with an older copy in OUT" "/vendor/firmware/aoc.bin is missing" "$t/v2-missing" "$t/erofs.img" + +rm -rf "$t/v2-noreg"; cp -r "$t/v2" "$t/v2-noreg"; rm -r "$t/v2-noreg/etc/sensors/registry" +refuses "a missing sensor registry with an older one in OUT" "/vendor/etc/sensors/registry is missing" "$t/v2-noreg" "$t/ext4.img" + +rm -rf "$t/v2-dumpfail"; cp -r "$t/v2" "$t/v2-dumpfail"; echo firmware/cs40l26.bin > "$t/v2-dumpfail/.fail-dump" +refuses "debugfs reporting a short write (exit 0)" "/vendor/firmware/cs40l26.bin is missing" "$t/v2-dumpfail" "$t/ext4.img" +rm -rf "$t/v2-rdumpfail"; cp -r "$t/v2" "$t/v2-rdumpfail"; touch "$t/v2-rdumpfail/.fail-rdump" +refuses "debugfs reporting a short write in the registry (exit 0)" "/vendor/etc/sensors/registry is missing" "$t/v2-rdumpfail" "$t/ext4.img" +# The same under a file size limit on the run (as on a nearly full disk): +# reading debugfs's messages must not need a file of its own +o=$t/refused-limited n=$failed +seed "$o" +before=$(snap "$o") +if (trap '' XFSZ; ulimit -f 8; run "$t/v2-rdumpfail" "$t/ext4.img" -o "$o"); then + fail "a registry short write under a file size limit: refused" +elif [[ $(snap "$o") != "$before" ]]; then + cat "$t/log"; fail "a registry short write under a file size limit: the previous outputs stay as they were" +fi +pass_since "$n" "a registry short write under a file size limit: refused, the previous outputs untouched" + +rm -rf "$t/v2-same"; cp -r "$t/v2" "$t/v2-same" +cp "$t/v2-same/firmware/cs35l41-dsp1-spk-prot.bin" "$t/v2-same/firmware/R-cs35l41-dsp1-spk-prot.bin" +refuses "a sanity check failing after every copy" "the top and bottom speaker tunings are the same file" "$t/v2-same" "$t/ext4.img" + +# The camera APEX (-c) is replaced only by a successful run +echo "old apex" > "$t/camera.apex" +rm -rf "$t/v2-noapex"; cp -r "$t/v2" "$t/v2-noapex"; rm "$t/v2-noapex/apex/com.google.pixel.camera.hal.apex" +refuses "a missing camera APEX with an older one at -c" "camera.hal.apex is missing" "$t/v2-noapex" "$t/ext4.img" -c "$t/camera.apex" +[[ $(cat "$t/camera.apex") == "old apex" ]] || fail "camera: the -c file stays after a failed run" +[[ -z $(find "$t" -maxdepth 1 -name 'camera.apex.*') ]] || fail "camera: no temporary APEX left beside -c" +o=$t/camera +seed "$o" +if run "$t/v2" "$t/ext4.img" -c "$t/camera.apex" -o "$o" && [[ $(cat "$t/camera.apex") == "apex gen2" ]]; then + pass "camera: a successful run replaces the -c file" +else + cat "$t/log"; fail "camera: a successful run replaces the -c file" +fi + +# -c naming a directory is refused before anything is extracted +o=$t/camera +before=$(snap "$o") +if run "$t/v2" "$t/ext4.img" -c "$t" -o "$o"; then + fail "camera: a -c that is a directory is refused" +elif ! grep -qF -- "-c needs a file name" "$t/log" || [[ $(snap "$o") != "$before" ]]; then + cat "$t/log"; fail "camera: a -c that is a directory is refused, nothing changed" +else + pass "camera: a -c that is a directory is refused, nothing changed" +fi +# -c inside the outputs the run replaces is refused too +if run "$t/v2" "$t/ext4.img" -c "$o/$FW/camera.apex" -o "$o"; then + fail "camera: a -c inside the replaced outputs is refused" +elif ! grep -qF -- "-c must not be inside the outputs" "$t/log" || [[ $(snap "$o") != "$before" ]]; then + cat "$t/log"; fail "camera: a -c inside the replaced outputs is refused, nothing changed" +else + pass "camera: a -c inside the replaced outputs is refused, nothing changed" +fi + +# A move that fails while publishing (the registry's parent not writable) +# puts back what was already replaced: OUT is left as it was +if ((EUID != 0)); then + o=$t/publish-fail n=$failed + seed "$o" + chmod a-w "$o/usr/share/omamobile" + before=$(snap "$o") + if run "$t/v2" "$t/ext4.img" -o "$o"; then + fail "publish failure: the run fails" + else + grep -qF "publishing failed; the previous outputs are back in place" "$t/log" || + { cat "$t/log"; fail "publish failure: says the previous outputs are back"; } + [[ $(snap "$o") == "$before" ]] || { diff <(echo "$before") <(snap "$o") || true; fail "publish failure: OUT is as it was"; } + fi + chmod u+w "$o/usr/share/omamobile" + pass_since "$n" "publish failure: the previous outputs put back, OUT as it was" +else + echo "ok - # skip: running as root, so a read-only directory can't make a move fail" +fi + +# The same on EROFS: a reused directory, and a failure after copying +o=$t/erofs +seed "$o" +if run "$t/v2" "$t/erofs.img" -o "$o" && [[ $(listed "$o") == "$CHEETAH_FILES" && ! -e $o/$REG/stale_sensor.json ]]; then + pass "erofs: a second run keeps nothing from the earlier generation" +else + cat "$t/log"; fail "erofs: a second run keeps nothing from the earlier generation" +fi +refuses "erofs: a sanity check failing after every copy" "the top and bottom speaker tunings are the same file" "$t/v2-same" "$t/erofs.img" + +# --- A real ext4 image (synthetic files), when the host can make one ----------- +# Its registry files belong to root, as on a real image, so a run as a user +# sees rdump's ownership notes; a 9 KiB aoc.bin is cut short by a file size +# limit on debugfs, which reports it and exits 0. +rm -rf "$t/v2-big"; cp -r "$t/v2" "$t/v2-big"; head -c 9216 /dev/zero | tr '\0' 'a' > "$t/v2-big/firmware/aoc.bin" +if realdfs=$(command -v debugfs) && mkfs.ext4 -q -F -d "$t/v2" "$t/real.img" 8M >/dev/null 2>&1 && + mkfs.ext4 -q -F -d "$t/v2-missing" "$t/real-missing.img" 8M >/dev/null 2>&1 && + mkfs.ext4 -q -F -d "$t/v2-big" "$t/real-big.img" 8M >/dev/null 2>&1 && + debugfs -w -R "sif /etc/sensors/registry uid 0" "$t/real.img" >/dev/null 2>&1 && + debugfs -w -R "sif /etc/sensors/registry/fictional_accel.json uid 0" "$t/real.img" >/dev/null 2>&1; then + mkdir -p "$t/limited" + printf '#!/bin/bash\ntrap "" XFSZ\nulimit -f 4\nexec %q "$@"\n' "$realdfs" > "$t/limited/debugfs" + chmod +x "$t/limited/debugfs" + o=$t/real + seed "$o" + if "$EVF" -w "$t/work" -o "$o" "$t/real.img" > "$t/log" 2>&1 && + [[ $(listed "$o") == "$CHEETAH_FILES" && ! -e $o/$REG/stale_sensor.json && $(cat "$o/$FW/aoc.bin") == "aoc gen2" ]]; then + pass "real ext4: a second run keeps nothing from the earlier generation" + else + cat "$t/log"; fail "real ext4: a second run keeps nothing from the earlier generation" + fi + before=$(snap "$o") + if "$EVF" -w "$t/work" -o "$o" "$t/real-missing.img" > "$t/log" 2>&1; then + fail "real ext4: a missing file with an older copy in OUT is refused" + elif ! grep -qF "/vendor/firmware/aoc.bin is missing" "$t/log"; then + cat "$t/log"; fail "real ext4: says the file is missing" + elif [[ $(snap "$o") != "$before" ]]; then + fail "real ext4: a refused run leaves the previous outputs" + else + pass "real ext4: a missing file with an older copy in OUT is refused, the previous outputs untouched" + fi + before=$(snap "$o") + if PATH=$t/limited:$PATH "$EVF" -w "$t/work" -o "$o" "$t/real-big.img" > "$t/log" 2>&1; then + fail "real ext4: a copy cut short by debugfs is refused" + elif ! grep -qF "/vendor/firmware/aoc.bin is missing" "$t/log" || [[ $(snap "$o") != "$before" ]]; then + cat "$t/log"; fail "real ext4: a copy cut short by debugfs is refused, the previous outputs untouched" + else + pass "real ext4: a copy cut short by debugfs is refused, the previous outputs untouched" + fi +else + echo "ok - # skip: no mkfs.ext4 -d and debugfs for the real ext4 checks" +fi + +((failed == 0)) || { echo "$failed check(s) failed" >&2; exit 1; } diff --git a/tools/extract-vendor-firmware b/tools/extract-vendor-firmware index 534c5d5d0..a964fbfc0 100755 --- a/tools/extract-vendor-firmware +++ b/tools/extract-vendor-firmware @@ -87,6 +87,14 @@ # # Output is a tree rooted at /, under usr/lib/firmware/updates (searched before # linux-firmware's files, and not owned by any package). Nothing touches the phone. +# Each run builds a whole new generation in a staging directory inside OUTDIR, +# checks every file and the checksum list, and only then replaces the paths +# this script owns there (usr/lib/firmware/updates, usr/share/omamobile/sensors, +# vendor-firmware.sha256 and the -c file). A failed run, a failed replacement +# included, leaves the previous outputs as they were (if putting them back +# fails too, the error names the directory holding them), and a reused OUTDIR +# keeps no file from an earlier run (another device's firmware, an old sensor +# registry); the rest of OUTDIR is left alone. # # Needs: unzip, debugfs (e2fsprogs); simg2img if vendor.img is sparse; # lpunpack if the image only has super.img; fsck.erofs (erofs-utils) if vendor is EROFS. @@ -128,7 +136,8 @@ Usage: extract-vendor-firmware [-o OUTDIR] [-w WORKDIR] [-f] SOURCE SOURCE Google factory zip (-*-factory-*.zip), the image--*.zip inside it, or a raw/sparse vendor.img - -o DIR staging root to write into (default: ./out/vendor-firmware) + -o DIR staging root to write into (default: ./out/vendor-firmware); a + successful run replaces this script's earlier outputs there -w DIR scratch directory for the ~1 GB vendor.img (default: a temp dir in \$TMPDIR) -d DEV cheetah (default), panther, raven, tegu or husky -s SKU husky: the hardware SKU (androidboot.hardware.sku), which picks the NVRAM @@ -247,6 +256,12 @@ SRC=$(readlink -f "$1") [[ -f "$SRC" ]] || die "no such file: $1" # Absolute, because the checksum step runs from inside the staging dir OUT=$(realpath -m -- "$OUT") +[[ -z $CAMERA_APEX || ( -d $(dirname -- "$CAMERA_APEX") && ! -d $CAMERA_APEX ) ]] || + die "-c needs a file name in an existing directory: $CAMERA_APEX" +case $CAMERA_APEX in + "$OUT/$DEST_REL"/*|"$OUT/usr/share/omamobile/sensors"/*|"$OUT/vendor-firmware.sha256") + die "-c must not be inside the outputs this script replaces in $OUT" ;; +esac if [[ -n "$WORK" ]]; then mkdir -p "$WORK" @@ -254,7 +269,25 @@ if [[ -n "$WORK" ]]; then else WORK=$(mktemp -d "${TMPDIR:-/tmp}/xvf.XXXXXX") fi -trap 'rm -rf "$WORK"' EXIT +stage="" cam_new="" publishing="" aside=() published=() +# A failed publish puts the previous outputs back (restore); only if that +# fails too does the staging directory stay, holding them. +restore() { + local p + for p in "${published[@]}"; do rm -rf -- "${OUT:?}/$p" || return 1; done + for p in "${aside[@]}"; do mv -T -- "$stage/old/$p" "$OUT/$p" || return 1; done +} +cleanup() { + [[ -z $cam_new ]] || rm -f -- "$cam_new" + if [[ -n $publishing ]] && ! restore; then + printf 'error: publishing failed; the previous outputs are in %s\n' "$stage/old" >&2 + else + [[ -z $publishing ]] || printf 'error: publishing failed; the previous outputs are back in place\n' >&2 + [[ -z $stage ]] || rm -rf "$stage" + fi + rm -rf "$WORK" +} +trap cleanup EXIT # Read 4 bytes at an offset as a little-endian hex word. magic_at() { od -An -tx4 -j "$2" -N4 "$1" | tr -d ' \n'; } @@ -316,11 +349,27 @@ fi if [[ "$(od -An -tx2 -j 1080 -N2 "$img" | tr -d ' ')" == ef53 ]]; then need debugfs "install e2fsprogs" say "vendor.img is ext4; reading it with debugfs (no mount)" - get_file() { - debugfs -R "dump /$1 $2" "$img" 2>/dev/null - [[ -s "$2" ]] + # debugfs exits 0 when a copy fails (a missing file, a short write on a + # full disk) and says so on stderr: anything there but its banner, or + # rdump's notes that it can't give the files the image's owners (as a + # user), fails the copy. The copy goes to a new path, so no older file + # can stand in for it. + dfs() { + local err line + err=$(debugfs -R "$1" "$img" 2>&1 >/dev/null) || return 1 + while [[ -n $err ]]; do # line by line in the shell: nothing here can fail + line=${err%%$'\n'*} + if [[ $err == *$'\n'* ]]; then err=${err#*$'\n'}; else err=""; fi + case $line in + "" | "debugfs "[0-9]*) ;; + "dump_file: Operation not permitted while changing ownership of "*) ;; + "rdump: Operation not permitted while changing ownership of "*) ;; + *) return 1 ;; + esac + done } - get_dir() { mkdir -p "$2" && debugfs -R "rdump /$1 $2" "$img" 2>/dev/null && [[ -d "$2/$(basename "$1")" ]]; } + get_file() { dfs "dump /$1 $2" && [[ -s "$2" ]]; } + get_dir() { mkdir -p "$2" && dfs "rdump /$1 $2" && [[ -d "$2/$(basename "$1")" ]]; } elif [[ "$(magic_at "$img" 1024)" == e0f5e1e2 ]]; then need fsck.erofs "install erofs-utils" say "vendor.img is EROFS; extracting it with fsck.erofs" @@ -340,11 +389,16 @@ if get_file build.prop "$WORK/build.prop"; then say "Vendor partition: $(sed -n 's/^ro.vendor.build.fingerprint=//p' "$WORK/build.prop" | head -1)" fi -dest="$OUT/$DEST_REL" +# The new generation: everything below is written into $new, which starts +# empty, so no file from an earlier run can stand in for a missing one. +mkdir -p "$OUT" +stage=$(mktemp -d "$OUT/.extract-vendor-firmware.XXXXXX") +new="$stage/new" +dest="$new/$DEST_REL" for entry in "${MAP[@]}"; do from=${entry%%|*} to=${entry#*|} mkdir -p "$dest/$(dirname "$to")" - get_file "$from" "$dest/$to" || die "/vendor/$from is missing from this image" + get_file "$from" "$dest/$to" || die "/vendor/$from is missing from this image or could not be copied" chmod 0644 "$dest/$to" printf ' /vendor/%-28s -> /%s/%s\n' "$from" "$DEST_REL" "$to" done @@ -353,16 +407,19 @@ done # settings), read by omamobile-sensor-proxy; without it only virtual sensors. # husky has no AoC driver yet, so it skips the registry. if [[ $DEVICE != husky ]]; then - reg="$OUT/usr/share/omamobile/sensors" - get_dir etc/sensors/registry "$reg" || die "/vendor/etc/sensors/registry is missing from this image" + reg="$new/usr/share/omamobile/sensors" + get_dir etc/sensors/registry "$reg" || + die "/vendor/etc/sensors/registry is missing from this image or could not be copied" chmod -R u=rwX,go=rX "$reg" printf ' /vendor/%-28s -> /%s\n' "etc/sensors/registry/" "usr/share/omamobile/sensors/registry/" fi +# The camera APEX goes to a new file beside -c, renamed into place below if [[ -n $CAMERA_APEX ]]; then - get_file apex/com.google.pixel.camera.hal.apex "$CAMERA_APEX" || - die "/vendor/apex/com.google.pixel.camera.hal.apex is missing from this image" - say "Camera HAL APEX copied to $CAMERA_APEX" + cam_new=$(mktemp "$CAMERA_APEX.XXXXXX") + get_file apex/com.google.pixel.camera.hal.apex "$cam_new" || + die "/vendor/apex/com.google.pixel.camera.hal.apex is missing from this image or could not be copied" + chmod 0644 "$cam_new" fi # Sanity checks: right chip, text NVRAM, Cirrus file formats. @@ -417,7 +474,12 @@ fi ver=$(grep -ao 'Version=[0-9.]* ([^)]*)' "$brcm/$FW_BASE.$BOARD.bin" | head -1) bt="" [[ -z $BT_CHIP ]] || bt=$(grep -ao "$BT_CHIP [^[]*" "$brcm/BCM.$BOARD.hcd" | head -1) -(cd "$dest" && find . -type f -printf '%P\n' | sort | xargs sha256sum -- > "$OUT/vendor-firmware.sha256") +(cd "$dest" && find . -type f -printf '%P\n' | sort | xargs sha256sum -- > "$new/vendor-firmware.sha256") +# The checksum list names exactly the mapped files +for entry in "${MAP[@]}"; do printf '%s\n' "${entry#*|}"; done | sort > "$stage/expected" +cut -c67- "$new/vendor-firmware.sha256" | cmp -s - "$stage/expected" || + die "the staged firmware is not exactly the mapped files" + say "Wi-Fi firmware: ${ver:-unknown version}" if [[ $DEVICE == husky ]]; then say "NVRAM: /vendor/firmware/$nvram${SKU:+ (SKU $SKU)}" @@ -430,5 +492,26 @@ fi say "AoC firmware: aoc-tegu.bin ($(stat -c %s "$dest/aoc-tegu.bin") bytes)" [[ $DEVICE == cheetah || $DEVICE == panther || $DEVICE == raven ]] && say "Speaker protection: cs35l41-dsp1-spk-prot-$DEVICE.wmfw and the top and bottom tunings" +# Publish: move the previous outputs aside, then the new ones into place, all +# renames within OUTDIR, then the -c file. A failed move puts it all back. +publishing=1 +for p in "$DEST_REL" usr/share/omamobile/sensors vendor-firmware.sha256; do + if [[ -e $OUT/$p || -L $OUT/$p ]]; then # -L: a dangling symlink too + mkdir -p "$stage/old/$(dirname "$p")" + mv -T -- "$OUT/$p" "$stage/old/$p" + aside+=("$p") + fi + if [[ -e $new/$p ]]; then + mkdir -p "$OUT/$(dirname "$p")" + mv -T -- "$new/$p" "$OUT/$p" + published+=("$p") + fi +done +if [[ -n $CAMERA_APEX ]]; then + mv -fT -- "$cam_new" "$CAMERA_APEX" + cam_new="" +fi +publishing="" # published: nothing below may undo it +[[ -z $CAMERA_APEX ]] || say "Camera HAL APEX copied to $CAMERA_APEX" say "Staged in $OUT (checksums in $OUT/vendor-firmware.sha256)" echo "This is Google's proprietary firmware from your own factory image. Do not redistribute it." From 8d08b8c5de7ac2b4e9b93f3ec33a6fd20af42b27 Mon Sep 17 00:00:00 2001 From: only21mil Date: Sun, 8 Nov 2026 09:59:46 +0000 Subject: [PATCH 2/2] extract-vendor-firmware: refuse an -o that resolves to / Publishing moves the previous usr/lib/firmware/updates and usr/share/omamobile/sensors aside under OUTDIR and removes them once the new generation is in place. Nothing calls the tool with -o /, but an operator who did would have the host's own directories replaced. The tool now refuses an OUTDIR whose resolved path is /, right after it is resolved and before the scratch directory or anything else is written. tests/extract-vendor-firmware runs the tool with -o /, //, /usr/.. and a symlink to /, and checks the refusal message and that no scratch directory was made. Its image is all zeros, so a tool without the refusal stops at the image check before it could write under /: against the previous commit's tool these four checks fail and nothing is written there. The test header now says the checksum list is checked after a successful run; the tool's own check that the list names exactly the mapped files is a backstop no reachable state trips. Co-Authored-By: Claude Opus 5.5 --- tests/extract-vendor-firmware | 33 ++++++++++++++++++++++++++------- tools/extract-vendor-firmware | 6 +++++- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/tests/extract-vendor-firmware b/tests/extract-vendor-firmware index 423f0f2e1..a4b13271f 100755 --- a/tests/extract-vendor-firmware +++ b/tests/extract-vendor-firmware @@ -8,13 +8,14 @@ # reused output directory keeps no firmware or sensor registry file from # an earlier run (another device's, or one the new image doesn't have), # and leaves the rest of the directory alone; a husky run drops an earlier -# registry; the checksum list names exactly the mapped files; a failed run -# (a missing file or registry, debugfs reporting a short write, also -# under a file size limit, a failed sanity check, a missing camera APEX) -# leaves the previous outputs and the -c file byte for byte, so an older -# file can't pass for a missing one; a -c that is a directory or inside -# the outputs is refused; and a move that fails while publishing puts the -# previous outputs back. With mkfs.ext4 +# registry; a successful run's checksum list names exactly the mapped +# files; a failed run (a missing file or registry, debugfs reporting a +# short write, also under a file size limit, a failed sanity check, a +# missing camera APEX) leaves the previous outputs and the -c file byte +# for byte, so an older file can't pass for a missing one; a -c that is a +# directory or inside the outputs is refused; an -o that resolves to / is +# refused before anything is written; and a move that fails while +# publishing puts the previous outputs back. With mkfs.ext4 # and debugfs on the host, the same on a real ext4 image whose registry # belongs to root, and a copy that a file size limit cuts short. EVF=path # runs another copy of the tool (a negative control), so every check runs @@ -267,6 +268,24 @@ else pass "camera: a -c inside the replaced outputs is refused, nothing changed" fi +# -o naming / (or a path that resolves to it) is refused before anything is +# written, the scratch directory included. The image is all zeros, neither +# ext4 nor EROFS, so a tool without the refusal stops at the image check +# instead, before it could write under /: this check never writes there. +head -c 4096 /dev/zero > "$t/junk.img" +ln -s / "$t/link-to-root" +for root in / // /usr/.. "$t/link-to-root"; do + name="-o ${root#"$t/"}" + rm -rf "$t/work-root" + if "$EVF" -w "$t/work-root" -o "$root" "$t/junk.img" > "$t/log" 2>&1; then + fail "$name: refused" + elif ! grep -qF -- "-o resolves to /" "$t/log" || [[ -e $t/work-root ]]; then + cat "$t/log"; fail "$name: refused before anything is written" + else + pass "$name: refused before anything is written" + fi +done + # A move that fails while publishing (the registry's parent not writable) # puts back what was already replaced: OUT is left as it was if ((EUID != 0)); then diff --git a/tools/extract-vendor-firmware b/tools/extract-vendor-firmware index a964fbfc0..702c77b0f 100755 --- a/tools/extract-vendor-firmware +++ b/tools/extract-vendor-firmware @@ -94,7 +94,8 @@ # included, leaves the previous outputs as they were (if putting them back # fails too, the error names the directory holding them), and a reused OUTDIR # keeps no file from an earlier run (another device's firmware, an old sensor -# registry); the rest of OUTDIR is left alone. +# registry); the rest of OUTDIR is left alone. An OUTDIR that is / (or +# resolves to it) is refused before anything is written. # # Needs: unzip, debugfs (e2fsprogs); simg2img if vendor.img is sparse; # lpunpack if the image only has super.img; fsck.erofs (erofs-utils) if vendor is EROFS. @@ -256,6 +257,9 @@ SRC=$(readlink -f "$1") [[ -f "$SRC" ]] || die "no such file: $1" # Absolute, because the checksum step runs from inside the staging dir OUT=$(realpath -m -- "$OUT") +# Publishing replaces whole directories under OUT, so OUT=/ would replace this +# machine's own; refuse before anything is written (WORK included) +[[ $OUT != / ]] || die "-o resolves to /: publishing would replace this machine's own directories" [[ -z $CAMERA_APEX || ( -d $(dirname -- "$CAMERA_APEX") && ! -d $CAMERA_APEX ) ]] || die "-c needs a file name in an existing directory: $CAMERA_APEX" case $CAMERA_APEX in