Skip to content

tools/fetch-ndk: the pinned Android NDK for the bionic programs, checked before it is unpacked - #900

Open
W4HO-ham wants to merge 3 commits into
derekross:mainfrom
W4HO-ham:tools-fetch-ndk
Open

W4HO-ham wants to merge 3 commits into
derekross:mainfrom
W4HO-ham:tools-fetch-ndk

Conversation

@W4HO-ham

Copy link
Copy Markdown
Contributor

As you asked on #864: the NDK download with a pinned checksum, as its own pull request.

The 9a's GNSS HAL client is built on the computer with the Android NDK (make -C packages/omamobile-gnss/bionic NDK=...), and nothing in the tree said which NDK or where it comes from. That is how a build machine whose bionic/prebuilt/ predated the client published omamobile-gnss without it (#748).

  • tools/ndk.lock pins r30 (30.0.16248370, the release the client was built with so far): the archive's SHA-256 and size as dl.google.com served it today, and its URL.
  • tools/fetch-ndk downloads it, checks the size and the SHA-256 before anything is unpacked, unpacks it into ~/.cache/omamobile/ndk (outside the repository) with its compilers executable, removes the archive, and prints the path. An NDK already there with the pinned revision in its source.properties is used as it is; another NDK in the way is refused, not overwritten. --print prints only the path, for make -C packages/omamobile-gnss/bionic NDK="$(tools/fetch-ndk --print)".
  • The bionic Makefile's header and build-packages' warning for a package without the client name it.

It is about 740 MB to download and 2.3 GB unpacked, once. The NDK's compilers are x86_64 Linux (and macOS) binaries, so the make runs on an x86_64 build machine; the tool says so on another host but still fetches.

Tested:

  • tests/fetch-ndk against a file:// fixture: a matching archive unpacked with its compiler executable and the archive removed; an NDK already there used without a download; a wrong size, a wrong SHA-256, another revision, another NDK in the way, a host other than Google's and a broken lock refused, with nothing left behind. tests/tools-help, tests/all and tools/pr-check origin/main pass.
  • The real archive, downloaded on an arm64 build machine, went through the tool by a file:// lock with the real size and checksum: verified, unpacked and found again on a second run without a download.
  • The client built with the same release on my x86_64 laptop: omamobile-gnss-client, 15656 bytes, aarch64 PIE, as the one my palmtop runs.

Not tested: a download from dl.google.com through the tool itself (I fetched the archive with curl on the build machine, then ran the tool on it), and the tool on your build machine.

@derekross derekross left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for sending this on its own as I asked, and for building it the way fetch-toolchain is built: size and checksum before anything is unpacked, nothing overwritten, nothing left behind.

I checked the pin from scratch. I downloaded android-ndk-r30-linux.zip from dl.google.com myself: 738,633,529 bytes, SHA-256 753611f4…64df2, both as in your lock, and Pkg.Revision = 30.0.16248370 inside. Its SHA-1 (5107f898…2dee9) is the one Google lists on the NDK downloads page and in the r30 release notes, so the pin is Google's published archive and not only what the host served today. I also confirmed nothing automated calls the tool: only the Makefile's comment and build-packages' warning text name it. And I took out each of your checks in turn to see your test notice: it catches the size, the checksum, the revision and the NDK in the way.

Four things before it goes in:

  1. Google's terms. The NDK downloads page asks for agreement to its terms and conditions before a download. The tool says nothing about them. For factory images we have the owner accept Google's terms first; I need to decide what the equivalent is for a builder's tool (probably a line naming the terms page and an explicit option on the first fetch). I'll come back on that one; no need to guess at it.
  2. The "another host" case can't fail. With the dl.google.com check removed from the tool, tests/fetch-ndk still passes, because curl then really asks example.org and finds nothing. Please assert the refusal's message, and give that case a fake curl that records a call and fails, so the test shows no request was made.
  3. Hold curl to HTTPS, as fetch-toolchain does: --proto '=https,file' --proto-redir '=https'. The checksum would catch a wrong file, but a redirect shouldn't be able to leave HTTPS at all.
  4. --print shouldn't download. make … NDK="$(tools/fetch-ndk --print)" on a machine without the NDK starts a silent 740 MB fetch inside the command substitution, and the new warning in build-packages prints that very command on the build machine. I'd like --print to stop with "run tools/fetch-ndk first" when the NDK isn't there, and the Makefile comment and the warning to name the two steps.

Smaller, as you like: tests/fetch-ndk on its own line in the "one suite per line" block at the end of tests/all (the long loop line is edited by a dozen open pull requests and conflicts every time); a lock or a per-run name for the .part file; and stopping on a host that isn't x86_64 unless asked, since the download is no use there.

One thing I'm weighing on my side and you needn't act on: an NDK already in the cache is accepted on its source.properties line. fetch-toolchain keeps a digest of what it unpacked and can verify it again; since this compiler's output ends up in a signed package I may want the same here.

W4HO-ham added a commit to W4HO-ham/omarchy-mobile that referenced this pull request Nov 18, 2026
…ter that isn't x86_64 refused

From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
@W4HO-ham

Copy link
Copy Markdown
Contributor Author

Thank you for checking the pin against Google's published SHA-1, and for taking each check out in turn. The four points are in eb0757e, on top of the original commit, rebased onto main:

  1. Google's terms: left for your decision, nothing changed.
  2. Another host: tests/fetch-ndk has a stand-in curl that records any call and fails. That case now checks the refusal's message ("the NDK comes from https://dl.google.com/android/repository/") and that no request was made. With the host check taken out of the tool, the test fails.
  3. HTTPS: curl runs with --proto '=https,file' --proto-redir '=https', as in fetch-toolchain (file only for the test's fixture). The test reads the options from the stand-in's record, so dropping them fails it too.
  4. --print: it only prints the path of an NDK already fetched. Without one it stops with "the NDK r30 isn't in … yet: run tools/fetch-ndk first", creates nothing and calls nothing. The Makefile's comment and the warning in build-packages name the two steps: tools/fetch-ndk, then the make with --print.

The smaller ones:

  • tests/fetch-ndk is now on its own line at the end of tests/all, in the one-suite-per-line block, and it runs bash -n on the tool itself, so the branch touches no shared line there (the loop, the syntax list and the header comment are as on main).
  • Each run downloads to .android-ndk-r30.zip.<pid>.part and unpacks into its own folder, then moves the NDK into place with mv -T. A second run that finishes later finds the NDK there and leaves it.
  • A computer that isn't x86_64 stops before any download ("the NDK's compilers run on x86_64 Linux only (--any-host to fetch it all the same)").

Taking out any of the four checks (host, HTTPS options, --print stop, x86_64 stop) makes tests/fetch-ndk fail; I tried each. tests/all (817 ok) and tools/pr-check origin/main pass.

On the cached NDK and a digest of what was unpacked: understood, I'll wait for your call on that and on the terms.

@derekross

Copy link
Copy Markdown
Owner

On Google's terms, the one I owed you: please do it the way the factory image flow does. The tool names the NDK terms page (https://developer.android.com/ndk/downloads) and downloads only when the builder has passed an explicit option that says they accept them (--accept-google-terms or the like); without it, the first fetch stops and prints the page and the option. A cached NDK needs no option. With that and the other three points it can go in.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

Done in d355ec7, as the factory image flow does it: a fetch that needs a download stops unless --accept-google-terms is given, and says why:

error: the Android NDK is Google's, and Google asks everyone who downloads it to agree to its terms and conditions first. Read them on its download page, https://developer.android.com/ndk/downloads, and if you accept them, run: tools/fetch-ndk --accept-google-terms

An NDK already in the cache is used with no option. --print still never downloads; without the NDK it now gives the whole command ("fetch it first with tools/fetch-ndk --accept-google-terms", with the page). The Makefile's comment and the warning in build-packages name the option too. tests/fetch-ndk checks that a fetch without the option makes no request (the stand-in curl records none), leaves nothing in the cache and names the page and the option, and that the cached NDK needs no option; taking the check out of the tool makes the test fail. tests/all and tools/pr-check origin/main pass.

W4HO-ham added a commit to W4HO-ham/omarchy-mobile that referenced this pull request Nov 18, 2026
…ter that isn't x86_64 refused

From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
…ked before it is unpacked

The Pixel 9a's GNSS HAL client is built on the computer with the Android NDK (make -C packages/omamobile-gnss/bionic NDK=...), and nothing in the tree said which NDK or where to get it, so a build machine whose bionic/prebuilt/ predated the client published omamobile-gnss without it (derekross#748).

tools/fetch-ndk fetches the NDK tools/ndk.lock pins (r30, 30.0.16248370, the release that built the client so far) from dl.google.com, checks the archive's size and SHA-256 before anything is unpacked, unpacks it into ~/.cache/omamobile/ndk with its compilers executable, removes the archive, and prints the path; an NDK already there with the pinned revision is used as it is. The bionic Makefile's header and build-packages' warning for a package without the client name it: make -C packages/omamobile-gnss/bionic NDK="$(tools/fetch-ndk --print)".

Tested: tests/fetch-ndk against a file:// fixture (a matching archive unpacked with its compiler executable and the archive removed; an NDK already there used without a download; a wrong size, a wrong SHA-256, another revision, another NDK in the way, another host and a broken lock refused with nothing left behind), tests/tools-help, tests/all. The real archive, already downloaded on an arm64 build machine, went through the tool through a file:// lock with the real size and checksum: verified, unpacked (2.3 GB) and found again on a second run. The client built with the same NDK release on an x86_64 laptop.
…ter that isn't x86_64 refused

From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
As for the factory images, nothing of Google's is downloaded until the builder says they accept Google's terms. A fetch that needs a download stops unless --accept-google-terms is given, naming the NDK's download page, where Google shows its terms and conditions (https://developer.android.com/ndk/downloads), and the option. An NDK already fetched needs no option, and --print, which never downloads, now gives the whole command. The Makefile's comment and build-packages' warning name the option. tests/fetch-ndk checks that a fetch without the option makes no request, leaves nothing and names the page and the option, and that the cached NDK is used without it; taking the check out makes the test fail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants