Repository navigation
Conversation
derekross
left a comment
There was a problem hiding this comment.
Thank you for sending this on its own as I asked, and for building it the way fetch-toolchain is built: size and checksum before anything is unpacked, nothing overwritten, nothing left behind.
I checked the pin from scratch. I downloaded android-ndk-r30-linux.zip from dl.google.com myself: 738,633,529 bytes, SHA-256 753611f4…64df2, both as in your lock, and Pkg.Revision = 30.0.16248370 inside. Its SHA-1 (5107f898…2dee9) is the one Google lists on the NDK downloads page and in the r30 release notes, so the pin is Google's published archive and not only what the host served today. I also confirmed nothing automated calls the tool: only the Makefile's comment and build-packages' warning text name it. And I took out each of your checks in turn to see your test notice: it catches the size, the checksum, the revision and the NDK in the way.
Four things before it goes in:
- Google's terms. The NDK downloads page asks for agreement to its terms and conditions before a download. The tool says nothing about them. For factory images we have the owner accept Google's terms first; I need to decide what the equivalent is for a builder's tool (probably a line naming the terms page and an explicit option on the first fetch). I'll come back on that one; no need to guess at it.
- The "another host" case can't fail. With the
dl.google.comcheck removed from the tool,tests/fetch-ndkstill passes, becausecurlthen really asks example.org and finds nothing. Please assert the refusal's message, and give that case a fakecurlthat records a call and fails, so the test shows no request was made. - Hold
curlto HTTPS, asfetch-toolchaindoes:--proto '=https,file' --proto-redir '=https'. The checksum would catch a wrong file, but a redirect shouldn't be able to leave HTTPS at all. --printshouldn't download.make … NDK="$(tools/fetch-ndk --print)"on a machine without the NDK starts a silent 740 MB fetch inside the command substitution, and the new warning inbuild-packagesprints that very command on the build machine. I'd like--printto stop with "run tools/fetch-ndk first" when the NDK isn't there, and the Makefile comment and the warning to name the two steps.
Smaller, as you like: tests/fetch-ndk on its own line in the "one suite per line" block at the end of tests/all (the long loop line is edited by a dozen open pull requests and conflicts every time); a lock or a per-run name for the .part file; and stopping on a host that isn't x86_64 unless asked, since the download is no use there.
One thing I'm weighing on my side and you needn't act on: an NDK already in the cache is accepted on its source.properties line. fetch-toolchain keeps a digest of what it unpacked and can verify it again; since this compiler's output ends up in a signed package I may want the same here.
…ter that isn't x86_64 refused From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
df9abfe to
eb0757e
Compare
|
Thank you for checking the pin against Google's published SHA-1, and for taking each check out in turn. The four points are in eb0757e, on top of the original commit, rebased onto main:
The smaller ones:
Taking out any of the four checks (host, HTTPS options, On the cached NDK and a digest of what was unpacked: understood, I'll wait for your call on that and on the terms. |
|
On Google's terms, the one I owed you: please do it the way the factory image flow does. The tool names the NDK terms page (https://developer.android.com/ndk/downloads) and downloads only when the builder has passed an explicit option that says they accept them ( |
|
Done in d355ec7, as the factory image flow does it: a fetch that needs a download stops unless An NDK already in the cache is used with no option. |
…ter that isn't x86_64 refused From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
d355ec7 to
5fa2734
Compare
…ked before it is unpacked The Pixel 9a's GNSS HAL client is built on the computer with the Android NDK (make -C packages/omamobile-gnss/bionic NDK=...), and nothing in the tree said which NDK or where to get it, so a build machine whose bionic/prebuilt/ predated the client published omamobile-gnss without it (derekross#748). tools/fetch-ndk fetches the NDK tools/ndk.lock pins (r30, 30.0.16248370, the release that built the client so far) from dl.google.com, checks the archive's size and SHA-256 before anything is unpacked, unpacks it into ~/.cache/omamobile/ndk with its compilers executable, removes the archive, and prints the path; an NDK already there with the pinned revision is used as it is. The bionic Makefile's header and build-packages' warning for a package without the client name it: make -C packages/omamobile-gnss/bionic NDK="$(tools/fetch-ndk --print)". Tested: tests/fetch-ndk against a file:// fixture (a matching archive unpacked with its compiler executable and the archive removed; an NDK already there used without a download; a wrong size, a wrong SHA-256, another revision, another NDK in the way, another host and a broken lock refused with nothing left behind), tests/tools-help, tests/all. The real archive, already downloaded on an arm64 build machine, went through the tool through a file:// lock with the real size and checksum: verified, unpacked (2.3 GB) and found again on a second run. The client built with the same NDK release on an x86_64 laptop.
…ter that isn't x86_64 refused From the review of derekross#900. --print now only prints the path of an NDK already fetched and otherwise stops with "run tools/fetch-ndk first", so a make that asks for the path can't start a 740 MB download inside a command substitution; the Makefile's comment and build-packages' warning name the two steps. curl is held to HTTPS, redirects included (--proto '=https,file' --proto-redir '=https', as in fetch-toolchain). A fetch on a computer that isn't x86_64 stops unless --any-host is given. Each run downloads under a name of its own and moves the unpacked NDK into place with mv -T, so two runs at once can't mix their files. tests/fetch-ndk gets a stand-in curl that records any call and fails: the case of another host now checks the refusal's message and that no request was made, and new cases cover --print with no NDK, a computer that isn't x86_64 and the HTTPS options. Taking out any of the four checks makes the test fail. The suite moves to its own line at the end of tests/all, and checks its tool's syntax itself, so it touches no shared line there.
As for the factory images, nothing of Google's is downloaded until the builder says they accept Google's terms. A fetch that needs a download stops unless --accept-google-terms is given, naming the NDK's download page, where Google shows its terms and conditions (https://developer.android.com/ndk/downloads), and the option. An NDK already fetched needs no option, and --print, which never downloads, now gives the whole command. The Makefile's comment and build-packages' warning name the option. tests/fetch-ndk checks that a fetch without the option makes no request, leaves nothing and names the page and the option, and that the cached NDK is used without it; taking the check out makes the test fail.
5fa2734 to
082769a
Compare
As you asked on #864: the NDK download with a pinned checksum, as its own pull request.
The 9a's GNSS HAL client is built on the computer with the Android NDK (
make -C packages/omamobile-gnss/bionic NDK=...), and nothing in the tree said which NDK or where it comes from. That is how a build machine whosebionic/prebuilt/predated the client publishedomamobile-gnsswithout it (#748).tools/ndk.lockpins r30 (30.0.16248370, the release the client was built with so far): the archive's SHA-256 and size asdl.google.comserved it today, and its URL.tools/fetch-ndkdownloads it, checks the size and the SHA-256 before anything is unpacked, unpacks it into~/.cache/omamobile/ndk(outside the repository) with its compilers executable, removes the archive, and prints the path. An NDK already there with the pinned revision in itssource.propertiesis used as it is; another NDK in the way is refused, not overwritten.--printprints only the path, formake -C packages/omamobile-gnss/bionic NDK="$(tools/fetch-ndk --print)".Makefile's header andbuild-packages' warning for a package without the client name it.It is about 740 MB to download and 2.3 GB unpacked, once. The NDK's compilers are x86_64 Linux (and macOS) binaries, so the
makeruns on an x86_64 build machine; the tool says so on another host but still fetches.Tested:
tests/fetch-ndkagainst afile://fixture: a matching archive unpacked with its compiler executable and the archive removed; an NDK already there used without a download; a wrong size, a wrong SHA-256, another revision, another NDK in the way, a host other than Google's and a broken lock refused, with nothing left behind.tests/tools-help,tests/allandtools/pr-check origin/mainpass.file://lock with the real size and checksum: verified, unpacked and found again on a second run without a download.omamobile-gnss-client, 15656 bytes, aarch64 PIE, as the one my palmtop runs.Not tested: a download from
dl.google.comthrough the tool itself (I fetched the archive withcurlon the build machine, then ran the tool on it), and the tool on your build machine.