diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index d5dc6c2..57ef580 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -68,6 +68,16 @@ jobs: ./verify_binary.sh bazel-bin/hello_c aarch64 ./verify_binary.sh bazel-bin/hello_cc aarch64 + # Guards remote-execution compatibility: the hermetic sandbox bind-mounts only + # declared inputs, so any file the toolchain reaches without declaring it fails + # here exactly as it would on a remote executor. See tests/.bazelrc. + - name: Build under hermetic sandbox (remote-execution guard) + working-directory: tests + run: | + bazel build //:all --config=aarch64 --config=hermetic_x86_64 + bazel build //:all --config=jetpack_512 --config=hermetic_x86_64 + bazel build //:all --config=jetpack_62 --config=hermetic_x86_64 + native-x86_64: name: Native x86_64 runs-on: ubuntu-22.04 @@ -99,6 +109,10 @@ jobs: bazel-bin/hello_c bazel-bin/hello_cc + - name: Build under hermetic sandbox (remote-execution guard) + working-directory: tests + run: bazel build //:all --config=hermetic_x86_64 + native-arm64: name: Native ARM64 runs-on: ubuntu-22.04-arm @@ -130,6 +144,10 @@ jobs: bazel-bin/hello_c bazel-bin/hello_cc + - name: Build under hermetic sandbox (remote-execution guard) + working-directory: tests + run: bazel build //:all --config=hermetic + ci-ok: name: CI OK runs-on: ubuntu-slim diff --git a/README.hacks b/README.hacks index f26ffc3..1e64213 100644 --- a/README.hacks +++ b/README.hacks @@ -18,23 +18,51 @@ There are a few major issues to work around, both handled by: the sysroot in a way bazel can work with, in part or in whole due to some issue with symlinks. There's a sort-of-functional way to specify a sysroot in the bazel cc toolchain config, but there's no way to determine that path - dymanically in the correct scope without symlinks, the only way it works is - with a hard-coded path that is not at all portable. The workaround is to - use `find` to discover the path from the execroot (this is the working - directory when the `aarch64-linux-gnu-gcc` is executed,) which is not, itself, - a symlink, but its parent directory is, so run `realpath` on the output of - `dirname` and then append `/../../../` to remove the - `usr/aarch64-linux-gnu/lib` prefix and get to the root, then add that path - variable with `--sysroot` at the end of the `gcc` command, to override anything - bazel might set. This path could be hard-coded in the wrapper script to eliminate - several process spawns per gcc call, but that's less flexible. + dynamically in the correct scope, the only way it works is with a hard-coded + path that is not at all portable. The workaround is to compute it in the + wrapper: `realpath` the gcc binary and strip the trailing `/usr/bin/`, + then pass the result as `--sysroot` at the end of the `gcc` command, to + override anything bazel might set. + Deriving it from the gcc binary specifically is what makes this work in every + execution environment. gcc resolves its own binary before computing the library + search paths it hands to ld, so under bazel's default (symlinking) sandbox gcc + ends up working from the real output_base path while the execroot path is a + symlink. If `--sysroot` names the execroot path instead, ld no longer sees the + ld scripts under `usr/aarch64-linux-gnu/lib` (`libc.so` is a `GROUP(...)` + script naming absolute paths) as living inside the sysroot, stops rewriting + those paths, and the link fails on `__libc_start_main`. Anchoring on gcc's own + realpath always agrees with gcc, whether the inputs were symlinked (default + sandbox), bind-mounted (hermetic sandbox) or materialized outright (remote + execution). + An earlier version discovered the sysroot by running `find` for `libc.so.6` and + walking up three directories. Don't go back to that: the archives keep Ubuntu's + merged-/usr layout (top-level `lib` is a symlink to `usr/lib`), so once those + paths are declared as inputs libc.so.6 is reachable at more than one depth and + the result is ambiguous. The current form is also three fewer process spawns per + gcc call and drops the dependency on `find` and `head` being on PATH. 3. Ubuntu's compiler binaries aren't statically linked and require a few shared libraries from the bundle to execute on the host, which they don't seem to - be able to just pick up from the bazel sandbox enviroment, so use `find` to - set and export `$LD_LIBRARY_PATH`. This is also added to - `aarch64-linux-gnu-ar`. + be able to just pick up from the bazel sandbox enviroment, so set and export + `$LD_LIBRARY_PATH` pointing at the bundled host libraries (`host-libs/` in the + native archives, `usr/lib/x86_64-linux-gnu/` in the cross archive). This is + also added to `aarch64-linux-gnu-ar` and `aarch64-linux-gnu-as`. Minor issues: 1. Ubuntu gcc defaults to defining `-D_FORTIFY_SOURCE` so when we add our setting to the `feature_flags` in `toolchain/config.bzl` that's an error, to fix this add `-U_FORTIFY_SOURCE` immediately before `-D_FORTIFY_SOURCE=1` + +Remote execution: +1. None of the above is what makes the toolchain work remotely -- correct input + declaration is. The wrapper scripts only ever reach files that + `toolchain/ubuntu-22.04-arm64-cross.BUILD` and `toolchain/ubuntu-22.04-native.BUILD` + declare, which is why those `libraries` filegroups glob whole directories rather + than `*.so*`/`*.a`: the extension-matching version silently omitted gcc's + extensionless internals (`cc1`, `cc1plus`, `collect2`, `lto1`, `lto-wrapper`), + the CRT objects, the `*.spec` files and ld's linker scripts. Under the default + sandbox that went unnoticed, because `realpath` resolves a declared input back + out to the full unpacked archive in the output base and the undeclared files are + sitting right there next to the declared ones. A remote executor materializes + only the declared inputs, so anything missing from those filegroups is simply + absent. Build with `--config=hermetic` (see `tests/.bazelrc`) to catch that + locally; CI does this on every configuration. diff --git a/README.md b/README.md index 2b1c9d3..113f61d 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,57 @@ Finally, build a cc_binary target like so: bazel build --config jetpack62 //path/to/target ``` +## Remote execution + +The toolchains declare every file gcc, ld and the wrapper scripts touch as an action +input, so they work under remote execution as well as locally. Two things are required +of the RE setup: + +1. **The exec platform must be linux/x86_64** (or linux/aarch64 for the native aarch64 + toolchain) **and must name a container image.** The compilers in these archives are + Ubuntu 22.04 host binaries, so the executor has to be able to run them: + + ```python + platform( + name = "re_platform", + constraint_values = [ + "@platforms//os:linux", + "@platforms//cpu:x86_64", + ], + exec_properties = { + "container-image": "docker://", + }, + ) + ``` + + ```bash + bazel build //path/to/target --config=aarch64 \ + --remote_executor=grpc://: \ + --spawn_strategy=remote \ + --extra_execution_platforms=//:re_platform + ``` + +2. **The image must provide bash and coreutils.** The toolchain tools are shell wrappers + (see `README.hacks`) that run `bash`, `realpath`, `dirname` and `pwd`. Those are host + tools, not Bazel inputs, so the executor image has to supply them. Any standard + Ubuntu 22.04 base image does. + +### Checking hermeticity without an RE backend + +`tests/.bazelrc` defines a `hermetic` config that builds under Bazel's hermetic linux +sandbox. Unlike the default sandbox, which symlinks inputs back into the output base and +therefore lets an action reach files that were never declared, the hermetic sandbox +bind-mounts only the declared inputs — the same contract a remote executor gives an +action. An under-declared toolchain input fails there exactly as it would remotely: + +```bash +cd tests +bazel build //:all --config=aarch64 --config=hermetic_x86_64 # x86_64 host +bazel build //:all --config=hermetic # aarch64 host +``` + +CI runs this for every configuration, so under-declaration cannot regress silently. + ## How does this all work? Beginning from the project repository, the toolchain resolution process goes like: 1. `aarch64_linux_gnu_deps` is loaded from `deps.bzl` and invoked. diff --git a/tests/.bazelrc b/tests/.bazelrc index cdd04bb..8651372 100644 --- a/tests/.bazelrc +++ b/tests/.bazelrc @@ -8,3 +8,26 @@ build:jetpack_62 --platforms=@aarch64_linux_gnu//platforms:tegra_jetpack_62 # Show verbose failures for debugging build --verbose_failures + +# Remote-execution hermeticity guard. +# +# The default linux sandbox symlinks declared inputs back into the output base, so an +# action that reaches a file the toolchain never declared still finds it. The hermetic +# sandbox bind-mounts the declared inputs instead, with nothing else reachable, which is +# the same contract a remote executor gives an action. Building under it locally catches +# under-declared toolchain inputs without needing an RE backend. +# +# The mount pairs stand in for the executor's container image: the toolchain wrapper +# scripts shell out to bash, realpath and dirname, and those are host tools, not Bazel +# inputs. An RE exec platform has to supply them the same way (see README.md). +build:hermetic --spawn_strategy=linux-sandbox +build:hermetic --experimental_use_hermetic_linux_sandbox +build:hermetic --sandbox_add_mount_pair=/usr/bin:/bin +build:hermetic --sandbox_add_mount_pair=/usr/bin +build:hermetic --sandbox_add_mount_pair=/usr/lib:/lib +build:hermetic --sandbox_add_mount_pair=/usr/lib + +# x86_64 hosts also need /lib64, which holds the ELF interpreter every host binary names. +# arm64 Ubuntu has no /usr/lib64 to mount, so it keeps the plain --config=hermetic. +build:hermetic_x86_64 --config=hermetic +build:hermetic_x86_64 --sandbox_add_mount_pair=/usr/lib64:/lib64 diff --git a/toolchain/ubuntu-22.04-arm64-cross.BUILD b/toolchain/ubuntu-22.04-arm64-cross.BUILD index ca0fa91..03ada22 100644 --- a/toolchain/ubuntu-22.04-arm64-cross.BUILD +++ b/toolchain/ubuntu-22.04-arm64-cross.BUILD @@ -79,13 +79,17 @@ filegroup( ]), ) +# Every file gcc and ld reach at action time. Under remote execution an action +# only sees the inputs it declares, so this deliberately globs whole directories +# instead of matching *.so*/*.a: those two patterns miss gcc's extensionless +# internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects +# (crt1.o, crti.o, crtn.o, Scrt1.o, crtbegin.o, crtend.o), gcc's *.spec files, +# and the GNU ld text scripts named libc.so / libpthread.so. filegroup( name = "libraries", srcs = glob([ - "usr/aarch64-linux-gnu/lib/**/*.so*", - "usr/aarch64-linux-gnu/lib/*.a", - "usr/lib/gcc-cross/aarch64-linux-gnu/11/**/*.so*", - "usr/lib/gcc-cross/aarch64-linux-gnu/11/*.a", + "usr/aarch64-linux-gnu/lib/**", + "usr/lib/gcc-cross/aarch64-linux-gnu/11/**", ]), ) diff --git a/toolchain/ubuntu-22.04-arm64-cross/linux_x86_64/aarch64-linux-gnu-gcc b/toolchain/ubuntu-22.04-arm64-cross/linux_x86_64/aarch64-linux-gnu-gcc index 1aad305..b080f9b 100755 --- a/toolchain/ubuntu-22.04-arm64-cross/linux_x86_64/aarch64-linux-gnu-gcc +++ b/toolchain/ubuntu-22.04-arm64-cross/linux_x86_64/aarch64-linux-gnu-gcc @@ -9,10 +9,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" external_dir="$(cd "${script_dir}/../../../../" && pwd)" # ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's -# sandbox symlinks, so resolve the actual path where libraries reside: -libc_realpath="$(realpath "$(find "${external_dir}/ubuntu-22.04-arm64-cross/" -name libc.so.6 | head -n1)")" # This is the actual path to libc -# Remove the filename and usr/aarch64-linux-gnu/lib to get the sysroot: -sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")" +# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it +# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops +# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as +# sysroot-relative and looks for them at the real root. So derive the sysroot from the +# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under +# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking +# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6 +# sits at more than one depth and any depth-based guess is ambiguous. +gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-arm64-cross/usr/bin/aarch64-linux-gnu-gcc-11")" +sysroot="${gcc_realpath%/usr/bin/*}" # Required so the host binaries can find shared libraries they need. # Include usr/lib/x86_64-linux-gnu/ alongside host-libs/: the cross packages deposit @@ -25,7 +31,7 @@ export LD_LIBRARY_PATH="${sysroot}/usr/lib/x86_64-linux-gnu" # the command line as DT_NEEDED, which is required to make ld.so use the RPATH entries # to find those libraries: -"$sysroot/usr/bin/aarch64-linux-gnu-gcc-11" \ +"$gcc_realpath" \ -Wl,--no-as-needed \ "-B${sysroot}/usr/bin/" \ "-B${script_dir}/" \ diff --git a/toolchain/ubuntu-22.04-native.BUILD b/toolchain/ubuntu-22.04-native.BUILD index 69b6188..9e68f54 100644 --- a/toolchain/ubuntu-22.04-native.BUILD +++ b/toolchain/ubuntu-22.04-native.BUILD @@ -86,13 +86,39 @@ filegroup( ]), ) +# Every file gcc and ld reach at action time. Under remote execution an action +# only sees the inputs it declares, so this deliberately globs whole directories +# instead of matching *.so*/*.a: those two patterns miss gcc's extensionless +# internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects +# (crt1.o, crti.o, crtn.o, Scrt1.o, crtbegin.o, crtend.o), gcc's *.spec files, +# and ld's linker scripts (ldscripts/, and the text scripts named libc.so / +# libpthread.so). filegroup( name = "libraries", - srcs = glob([ - "usr/lib/*-linux-gnu/**/*.so*", - "usr/lib/*-linux-gnu/*.a", - "usr/lib/gcc/*-linux-gnu/11/*.a", - ]), + srcs = glob( + [ + "usr/lib/*-linux-gnu/**", + "usr/lib/gcc/*-linux-gnu/11/**", + # The archives keep Ubuntu's merged-/usr layout, where the top-level + # lib/ and lib64/ are symlinks into usr/. glibc's ld scripts name the + # /lib spelling -- libc.so is + # GROUP(/lib/x86_64-linux-gnu/libc.so.6 ... AS_NEEDED(/lib64/ld-linux-x86-64.so.2)) + # -- and ld resolves those against --sysroot, so the shared libraries + # they reach for have to be declared under that spelling too. + "lib/*-linux-gnu/*.so*", + "lib64/*.so*", + "usr/lib64/*.so*", + # aarch64 keeps the ELF interpreter directly in lib/ -- its libc.so + # says AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) where x86_64 says + # /lib64/... -- so that spelling has to be declared too. Matches + # nothing on x86_64, whose lib/ holds only directories. + "lib/*.so*", + ], + # The merged-/usr spellings are architecture-specific -- lib64/ is the + # x86_64 layout and the arm64 archive has no lib64 at all -- so some of + # these patterns are expected to match nothing for a given archive. + allow_empty = True, + ), ) # Shared libraries required to execute compiler binaries on the host system: diff --git a/toolchain/ubuntu-22.04-native/linux_aarch64/aarch64-linux-gnu-gcc b/toolchain/ubuntu-22.04-native/linux_aarch64/aarch64-linux-gnu-gcc index ae1a2a4..58f882d 100755 --- a/toolchain/ubuntu-22.04-native/linux_aarch64/aarch64-linux-gnu-gcc +++ b/toolchain/ubuntu-22.04-native/linux_aarch64/aarch64-linux-gnu-gcc @@ -9,10 +9,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" external_dir="$(cd "${script_dir}/../../../../" && pwd)" # ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's -# sandbox symlinks, so resolve the actual path where libraries reside: -libc_realpath="$(realpath "$(find "${external_dir}/ubuntu-22.04-aarch64-native/" -name libc.so.6 | head -n1)")" # This is the actual path to libc -# Remove the filename and usr/aarch64-linux-gnu/lib to get the sysroot: -sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")" +# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it +# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops +# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as +# sysroot-relative and looks for them at the real root. So derive the sysroot from the +# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under +# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking +# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6 +# sits at more than one depth and any depth-based guess is ambiguous. +gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-aarch64-native/usr/bin/aarch64-linux-gnu-gcc-11")" +sysroot="${gcc_realpath%/usr/bin/*}" # Due to https://github.com/bazelbuild/bazel/issues/16222 this is the only spot we can add # the --no-as-needed ld flag to make the final binary list every shared library provided on @@ -25,9 +31,9 @@ sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")" sanitizer_rpath=() [[ "$*" == *-fsanitize=* ]] && sanitizer_rpath=("-Wl,-rpath,${sysroot}/usr/lib/aarch64-linux-gnu/") -"$sysroot/usr/bin/aarch64-linux-gnu-gcc-11" \ +"$gcc_realpath" \ -Wl,--no-as-needed \ - "-B${external_dir}/ubuntu-22.04-aarch64-native/usr/bin/" \ + "-B${sysroot}/usr/bin/" \ "${sanitizer_rpath[@]}" \ "-B${script_dir}/" \ "$@" --sysroot="$sysroot" diff --git a/toolchain/ubuntu-22.04-native/linux_x86_64/x86_64-linux-gnu-gcc b/toolchain/ubuntu-22.04-native/linux_x86_64/x86_64-linux-gnu-gcc index 76df9e2..3541aeb 100755 --- a/toolchain/ubuntu-22.04-native/linux_x86_64/x86_64-linux-gnu-gcc +++ b/toolchain/ubuntu-22.04-native/linux_x86_64/x86_64-linux-gnu-gcc @@ -10,10 +10,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" external_dir="$(cd "${script_dir}/../../../../" && pwd)" # ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's -# sandbox symlinks, so resolve the actual path where libraries reside: -libc_realpath="$(realpath "$(find -L "${external_dir}/ubuntu-22.04-x86_64-native/" -name libc.so.6 | head -n1)")" # This is the actual path to libc -# Remove the filename and usr/x86_64-linux-gnu/lib to get the sysroot: -sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")" +# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it +# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops +# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as +# sysroot-relative and looks for them at the real root. So derive the sysroot from the +# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under +# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking +# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6 +# sits at more than one depth and any depth-based guess is ambiguous. +gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-x86_64-native/usr/bin/x86_64-linux-gnu-gcc-11")" +sysroot="${gcc_realpath%/usr/bin/*}" # Due to https://github.com/bazelbuild/bazel/issues/16222 this is the only spot we can add # the --no-as-needed ld flag to make the final binary list every shared library provided on @@ -26,9 +32,9 @@ sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")" sanitizer_rpath=() [[ "$*" == *-fsanitize=* ]] && sanitizer_rpath=("-Wl,-rpath,${sysroot}/usr/lib/x86_64-linux-gnu/") -"$sysroot/usr/bin/x86_64-linux-gnu-gcc-11" \ +"$gcc_realpath" \ -Wl,--no-as-needed \ - "-B${external_dir}/ubuntu-22.04-x86_64-native/usr/bin/" \ + "-B${sysroot}/usr/bin/" \ "${sanitizer_rpath[@]}" \ "-B${script_dir}/" \ "$@" --sysroot="$sysroot"