Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,16 @@ jobs:
./verify_binary.sh bazel-bin/hello_c aarch64
./verify_binary.sh bazel-bin/hello_cc aarch64

# Guards remote-execution compatibility: the hermetic sandbox bind-mounts only
# declared inputs, so any file the toolchain reaches without declaring it fails
# here exactly as it would on a remote executor. See tests/.bazelrc.
- name: Build under hermetic sandbox (remote-execution guard)
working-directory: tests
run: |
bazel build //:all --config=aarch64 --config=hermetic_x86_64
bazel build //:all --config=jetpack_512 --config=hermetic_x86_64
bazel build //:all --config=jetpack_62 --config=hermetic_x86_64

native-x86_64:
name: Native x86_64
runs-on: ubuntu-22.04
Expand Down Expand Up @@ -99,6 +109,10 @@ jobs:
bazel-bin/hello_c
bazel-bin/hello_cc

- name: Build under hermetic sandbox (remote-execution guard)
working-directory: tests
run: bazel build //:all --config=hermetic_x86_64

native-arm64:
name: Native ARM64
runs-on: ubuntu-22.04-arm
Expand Down Expand Up @@ -130,6 +144,10 @@ jobs:
bazel-bin/hello_c
bazel-bin/hello_cc

- name: Build under hermetic sandbox (remote-execution guard)
working-directory: tests
run: bazel build //:all --config=hermetic

ci-ok:
name: CI OK
runs-on: ubuntu-slim
Expand Down
54 changes: 41 additions & 13 deletions README.hacks
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,51 @@ There are a few major issues to work around, both handled by:
the sysroot in a way bazel can work with, in part or in whole due to some
issue with symlinks. There's a sort-of-functional way to specify a sysroot
in the bazel cc toolchain config, but there's no way to determine that path
dymanically in the correct scope without symlinks, the only way it works is
with a hard-coded path that is not at all portable. The workaround is to
use `find` to discover the path from the execroot (this is the working
directory when the `aarch64-linux-gnu-gcc` is executed,) which is not, itself,
a symlink, but its parent directory is, so run `realpath` on the output of
`dirname` and then append `/../../../` to remove the
`usr/aarch64-linux-gnu/lib` prefix and get to the root, then add that path
variable with `--sysroot` at the end of the `gcc` command, to override anything
bazel might set. This path could be hard-coded in the wrapper script to eliminate
several process spawns per gcc call, but that's less flexible.
dynamically in the correct scope, the only way it works is with a hard-coded
path that is not at all portable. The workaround is to compute it in the
wrapper: `realpath` the gcc binary and strip the trailing `/usr/bin/<gcc>`,
then pass the result as `--sysroot` at the end of the `gcc` command, to
override anything bazel might set.
Deriving it from the gcc binary specifically is what makes this work in every
execution environment. gcc resolves its own binary before computing the library
search paths it hands to ld, so under bazel's default (symlinking) sandbox gcc
ends up working from the real output_base path while the execroot path is a
symlink. If `--sysroot` names the execroot path instead, ld no longer sees the
ld scripts under `usr/aarch64-linux-gnu/lib` (`libc.so` is a `GROUP(...)`
script naming absolute paths) as living inside the sysroot, stops rewriting
those paths, and the link fails on `__libc_start_main`. Anchoring on gcc's own
realpath always agrees with gcc, whether the inputs were symlinked (default
sandbox), bind-mounted (hermetic sandbox) or materialized outright (remote
execution).
An earlier version discovered the sysroot by running `find` for `libc.so.6` and
walking up three directories. Don't go back to that: the archives keep Ubuntu's
merged-/usr layout (top-level `lib` is a symlink to `usr/lib`), so once those
paths are declared as inputs libc.so.6 is reachable at more than one depth and
the result is ambiguous. The current form is also three fewer process spawns per
gcc call and drops the dependency on `find` and `head` being on PATH.
3. Ubuntu's compiler binaries aren't statically linked and require a few shared
libraries from the bundle to execute on the host, which they don't seem to
be able to just pick up from the bazel sandbox enviroment, so use `find` to
set and export `$LD_LIBRARY_PATH`. This is also added to
`aarch64-linux-gnu-ar`.
be able to just pick up from the bazel sandbox enviroment, so set and export
`$LD_LIBRARY_PATH` pointing at the bundled host libraries (`host-libs/` in the
native archives, `usr/lib/x86_64-linux-gnu/` in the cross archive). This is
also added to `aarch64-linux-gnu-ar` and `aarch64-linux-gnu-as`.

Minor issues:
1. Ubuntu gcc defaults to defining `-D_FORTIFY_SOURCE` so when we add our setting
to the `feature_flags` in `toolchain/config.bzl` that's an error, to fix this
add `-U_FORTIFY_SOURCE` immediately before `-D_FORTIFY_SOURCE=1`

Remote execution:
1. None of the above is what makes the toolchain work remotely -- correct input
declaration is. The wrapper scripts only ever reach files that
`toolchain/ubuntu-22.04-arm64-cross.BUILD` and `toolchain/ubuntu-22.04-native.BUILD`
declare, which is why those `libraries` filegroups glob whole directories rather
than `*.so*`/`*.a`: the extension-matching version silently omitted gcc's
extensionless internals (`cc1`, `cc1plus`, `collect2`, `lto1`, `lto-wrapper`),
the CRT objects, the `*.spec` files and ld's linker scripts. Under the default
sandbox that went unnoticed, because `realpath` resolves a declared input back
out to the full unpacked archive in the output base and the undeclared files are
sitting right there next to the declared ones. A remote executor materializes
only the declared inputs, so anything missing from those filegroups is simply
absent. Build with `--config=hermetic` (see `tests/.bazelrc`) to catch that
locally; CI does this on every configuration.
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,57 @@ Finally, build a cc_binary target like so:
bazel build --config jetpack62 //path/to/target
```

## Remote execution

The toolchains declare every file gcc, ld and the wrapper scripts touch as an action
input, so they work under remote execution as well as locally. Two things are required
of the RE setup:

1. **The exec platform must be linux/x86_64** (or linux/aarch64 for the native aarch64
toolchain) **and must name a container image.** The compilers in these archives are
Ubuntu 22.04 host binaries, so the executor has to be able to run them:

```python
platform(
name = "re_platform",
constraint_values = [
"@platforms//os:linux",
"@platforms//cpu:x86_64",
],
exec_properties = {
"container-image": "docker://<your-ubuntu-22.04-based-image>",
},
)
```

```bash
bazel build //path/to/target --config=aarch64 \
--remote_executor=grpc://<host>:<port> \
--spawn_strategy=remote \
--extra_execution_platforms=//:re_platform
```

2. **The image must provide bash and coreutils.** The toolchain tools are shell wrappers
(see `README.hacks`) that run `bash`, `realpath`, `dirname` and `pwd`. Those are host
tools, not Bazel inputs, so the executor image has to supply them. Any standard
Ubuntu 22.04 base image does.

### Checking hermeticity without an RE backend

`tests/.bazelrc` defines a `hermetic` config that builds under Bazel's hermetic linux
sandbox. Unlike the default sandbox, which symlinks inputs back into the output base and
therefore lets an action reach files that were never declared, the hermetic sandbox
bind-mounts only the declared inputs — the same contract a remote executor gives an
action. An under-declared toolchain input fails there exactly as it would remotely:

```bash
cd tests
bazel build //:all --config=aarch64 --config=hermetic_x86_64 # x86_64 host
bazel build //:all --config=hermetic # aarch64 host
```

CI runs this for every configuration, so under-declaration cannot regress silently.

## How does this all work?
Beginning from the project repository, the toolchain resolution process goes like:
1. `aarch64_linux_gnu_deps` is loaded from `deps.bzl` and invoked.
Expand Down
23 changes: 23 additions & 0 deletions tests/.bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,26 @@ build:jetpack_62 --platforms=@aarch64_linux_gnu//platforms:tegra_jetpack_62

# Show verbose failures for debugging
build --verbose_failures

# Remote-execution hermeticity guard.
#
# The default linux sandbox symlinks declared inputs back into the output base, so an
# action that reaches a file the toolchain never declared still finds it. The hermetic
# sandbox bind-mounts the declared inputs instead, with nothing else reachable, which is
# the same contract a remote executor gives an action. Building under it locally catches
# under-declared toolchain inputs without needing an RE backend.
#
# The mount pairs stand in for the executor's container image: the toolchain wrapper
# scripts shell out to bash, realpath and dirname, and those are host tools, not Bazel
# inputs. An RE exec platform has to supply them the same way (see README.md).
build:hermetic --spawn_strategy=linux-sandbox
build:hermetic --experimental_use_hermetic_linux_sandbox
build:hermetic --sandbox_add_mount_pair=/usr/bin:/bin
build:hermetic --sandbox_add_mount_pair=/usr/bin
build:hermetic --sandbox_add_mount_pair=/usr/lib:/lib
build:hermetic --sandbox_add_mount_pair=/usr/lib

# x86_64 hosts also need /lib64, which holds the ELF interpreter every host binary names.
# arm64 Ubuntu has no /usr/lib64 to mount, so it keeps the plain --config=hermetic.
build:hermetic_x86_64 --config=hermetic
build:hermetic_x86_64 --sandbox_add_mount_pair=/usr/lib64:/lib64
12 changes: 8 additions & 4 deletions toolchain/ubuntu-22.04-arm64-cross.BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,17 @@ filegroup(
]),
)

# Every file gcc and ld reach at action time. Under remote execution an action
# only sees the inputs it declares, so this deliberately globs whole directories
# instead of matching *.so*/*.a: those two patterns miss gcc's extensionless
# internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects
# (crt1.o, crti.o, crtn.o, Scrt1.o, crtbegin.o, crtend.o), gcc's *.spec files,
# and the GNU ld text scripts named libc.so / libpthread.so.
filegroup(
name = "libraries",
srcs = glob([
"usr/aarch64-linux-gnu/lib/**/*.so*",
"usr/aarch64-linux-gnu/lib/*.a",
"usr/lib/gcc-cross/aarch64-linux-gnu/11/**/*.so*",
"usr/lib/gcc-cross/aarch64-linux-gnu/11/*.a",
"usr/aarch64-linux-gnu/lib/**",
"usr/lib/gcc-cross/aarch64-linux-gnu/11/**",
]),
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
external_dir="$(cd "${script_dir}/../../../../" && pwd)"

# ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's
# sandbox symlinks, so resolve the actual path where libraries reside:
libc_realpath="$(realpath "$(find "${external_dir}/ubuntu-22.04-arm64-cross/" -name libc.so.6 | head -n1)")" # This is the actual path to libc
# Remove the filename and usr/aarch64-linux-gnu/lib to get the sysroot:
sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")"
# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it
# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops
# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as
# sysroot-relative and looks for them at the real root. So derive the sysroot from the
# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under
# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking
# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6
# sits at more than one depth and any depth-based guess is ambiguous.
gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-arm64-cross/usr/bin/aarch64-linux-gnu-gcc-11")"
sysroot="${gcc_realpath%/usr/bin/*}"

# Required so the host binaries can find shared libraries they need.
# Include usr/lib/x86_64-linux-gnu/ alongside host-libs/: the cross packages deposit
Expand All @@ -25,7 +31,7 @@ export LD_LIBRARY_PATH="${sysroot}/usr/lib/x86_64-linux-gnu"
# the command line as DT_NEEDED, which is required to make ld.so use the RPATH entries
# to find those libraries:

"$sysroot/usr/bin/aarch64-linux-gnu-gcc-11" \
"$gcc_realpath" \
-Wl,--no-as-needed \
"-B${sysroot}/usr/bin/" \
"-B${script_dir}/" \
Expand Down
36 changes: 31 additions & 5 deletions toolchain/ubuntu-22.04-native.BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -86,13 +86,39 @@ filegroup(
]),
)

# Every file gcc and ld reach at action time. Under remote execution an action
# only sees the inputs it declares, so this deliberately globs whole directories
# instead of matching *.so*/*.a: those two patterns miss gcc's extensionless
# internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects
# (crt1.o, crti.o, crtn.o, Scrt1.o, crtbegin.o, crtend.o), gcc's *.spec files,
# and ld's linker scripts (ldscripts/, and the text scripts named libc.so /
# libpthread.so).
filegroup(
name = "libraries",
srcs = glob([
"usr/lib/*-linux-gnu/**/*.so*",
"usr/lib/*-linux-gnu/*.a",
"usr/lib/gcc/*-linux-gnu/11/*.a",
]),
srcs = glob(
[
"usr/lib/*-linux-gnu/**",
"usr/lib/gcc/*-linux-gnu/11/**",
# The archives keep Ubuntu's merged-/usr layout, where the top-level
# lib/ and lib64/ are symlinks into usr/. glibc's ld scripts name the
# /lib spelling -- libc.so is
# GROUP(/lib/x86_64-linux-gnu/libc.so.6 ... AS_NEEDED(/lib64/ld-linux-x86-64.so.2))
# -- and ld resolves those against --sysroot, so the shared libraries
# they reach for have to be declared under that spelling too.
"lib/*-linux-gnu/*.so*",
"lib64/*.so*",
"usr/lib64/*.so*",
# aarch64 keeps the ELF interpreter directly in lib/ -- its libc.so
# says AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) where x86_64 says
# /lib64/... -- so that spelling has to be declared too. Matches
# nothing on x86_64, whose lib/ holds only directories.
"lib/*.so*",
],
# The merged-/usr spellings are architecture-specific -- lib64/ is the
# x86_64 layout and the arm64 archive has no lib64 at all -- so some of
# these patterns are expected to match nothing for a given archive.
allow_empty = True,
),
)

# Shared libraries required to execute compiler binaries on the host system:
Expand Down
18 changes: 12 additions & 6 deletions toolchain/ubuntu-22.04-native/linux_aarch64/aarch64-linux-gnu-gcc
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
external_dir="$(cd "${script_dir}/../../../../" && pwd)"

# ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's
# sandbox symlinks, so resolve the actual path where libraries reside:
libc_realpath="$(realpath "$(find "${external_dir}/ubuntu-22.04-aarch64-native/" -name libc.so.6 | head -n1)")" # This is the actual path to libc
# Remove the filename and usr/aarch64-linux-gnu/lib to get the sysroot:
sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")"
# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it
# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops
# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as
# sysroot-relative and looks for them at the real root. So derive the sysroot from the
# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under
# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking
# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6
# sits at more than one depth and any depth-based guess is ambiguous.
gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-aarch64-native/usr/bin/aarch64-linux-gnu-gcc-11")"
sysroot="${gcc_realpath%/usr/bin/*}"

# Due to https://github.com/bazelbuild/bazel/issues/16222 this is the only spot we can add
# the --no-as-needed ld flag to make the final binary list every shared library provided on
Expand All @@ -25,9 +31,9 @@ sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")"
sanitizer_rpath=()
[[ "$*" == *-fsanitize=* ]] && sanitizer_rpath=("-Wl,-rpath,${sysroot}/usr/lib/aarch64-linux-gnu/")

"$sysroot/usr/bin/aarch64-linux-gnu-gcc-11" \
"$gcc_realpath" \
-Wl,--no-as-needed \
"-B${external_dir}/ubuntu-22.04-aarch64-native/usr/bin/" \
"-B${sysroot}/usr/bin/" \
"${sanitizer_rpath[@]}" \
"-B${script_dir}/" \
"$@" --sysroot="$sysroot"
18 changes: 12 additions & 6 deletions toolchain/ubuntu-22.04-native/linux_x86_64/x86_64-linux-gnu-gcc
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,16 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
external_dir="$(cd "${script_dir}/../../../../" && pwd)"

# ld (at least built and configured the way Ubuntu does,) doesn't work well with bazel's
# sandbox symlinks, so resolve the actual path where libraries reside:
libc_realpath="$(realpath "$(find -L "${external_dir}/ubuntu-22.04-x86_64-native/" -name libc.so.6 | head -n1)")" # This is the actual path to libc
# Remove the filename and usr/x86_64-linux-gnu/lib to get the sysroot:
sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")"
# sandbox symlinks: gcc resolves its own binary before deriving the library search paths it
# hands to ld, so unless --sysroot names the same directory gcc resolved to, ld stops
# treating the absolute paths inside glibc's ld scripts (libc.so is a GROUP(...) script) as
# sysroot-relative and looks for them at the real root. So derive the sysroot from the
# realpath of the gcc binary itself: it is the one path guaranteed to agree with gcc under
# the symlinking sandbox, the hermetic sandbox and remote execution alike. Don't go looking
# for it with `find` either -- the archives keep Ubuntu's merged-/usr layout, so libc.so.6
# sits at more than one depth and any depth-based guess is ambiguous.
gcc_realpath="$(realpath "${external_dir}/ubuntu-22.04-x86_64-native/usr/bin/x86_64-linux-gnu-gcc-11")"
sysroot="${gcc_realpath%/usr/bin/*}"

# Due to https://github.com/bazelbuild/bazel/issues/16222 this is the only spot we can add
# the --no-as-needed ld flag to make the final binary list every shared library provided on
Expand All @@ -26,9 +32,9 @@ sysroot="$(realpath "$(dirname "$libc_realpath")/../../../")"
sanitizer_rpath=()
[[ "$*" == *-fsanitize=* ]] && sanitizer_rpath=("-Wl,-rpath,${sysroot}/usr/lib/x86_64-linux-gnu/")

"$sysroot/usr/bin/x86_64-linux-gnu-gcc-11" \
"$gcc_realpath" \
-Wl,--no-as-needed \
"-B${external_dir}/ubuntu-22.04-x86_64-native/usr/bin/" \
"-B${sysroot}/usr/bin/" \
"${sanitizer_rpath[@]}" \
"-B${script_dir}/" \
"$@" --sysroot="$sysroot"
Loading