Repository navigation
Declare all toolchain inputs so builds work under remote execution - #21
Merged
Merged
Conversation
gartnera
marked this pull request as ready for review
August 28, 2026 18:24
The toolchains built only because the default linux sandbox symlinks declared inputs back into the output base: `realpath` in the gcc wrappers followed those symlinks out to the fully unpacked archive, and gcc was then driven with -B/--sysroot pointing there, so it freely read files that were never declared as action inputs. A remote executor materializes only the declared inputs, so those files are simply absent and the build fails. The `libraries` filegroups matched only *.so*/*.a, which missed everything gcc and ld actually reach at action time: the extensionless gcc internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects, gcc's *.spec files and ld's linker scripts. Glob the whole directories instead. The native archives additionally need the merged-/usr spelling of their shared libraries, because glibc's ld scripts name /lib/<triple>/libc.so.6 and /lib64/ld-linux-*.so, which ld resolves against --sysroot. Those merged-/usr spellings are architecture-specific -- lib64/ is the x86_64 layout and the arm64 archive has no lib64 at all -- so the glob needs allow_empty = True, as Bazel 8 rejects a pattern that matches nothing by default (see #23). Once those inputs are declared, the gcc wrappers can no longer find the sysroot by running `find` for libc.so.6 and walking up three directories: libc.so.6 is now reachable at two depths and the guess is ambiguous. Derive it from the realpath of the gcc binary instead. That also has to be a file rather than the archive directory -- gcc resolves its own binary before deriving the search paths it hands to ld, and if --sysroot names a different directory than gcc resolved to, ld stops treating the absolute paths in glibc's GROUP(...) scripts as sysroot-relative and the link fails on __libc_start_main. Add a `hermetic` config to tests/.bazelrc that builds under Bazel's hermetic linux sandbox, which bind-mounts only declared inputs and so reproduces the RE contract locally, and run it from CI for every configuration so this cannot regress. Document the RE exec-platform and container requirements in README.md. Verified on linux/x86_64 with Bazel 6.5.0: all three cross configs and the native x86_64 build pass under both the default and the hermetic sandbox, and the cross binaries still run under QEMU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `libraries` filegroup declared the merged-/usr spelling of the interpreter
as lib64/*.so* and usr/lib64/*.so*, which is the x86_64 layout. On arm64 there
is no lib64 at all: ubuntu-22.04-aarch64-native ships `lib -> usr/lib` plus
usr/lib/ld-linux-aarch64.so.1, and its libc.so reads
GROUP ( /lib/aarch64-linux-gnu/libc.so.6 ... AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) )
so ld resolves the interpreter directly under the sysroot's lib/, which no glob
matched. Under the hermetic sandbox or remote execution that file is absent and
the link fails on the AS_NEEDED, exactly as the x86_64 link did before
lib64/*.so* was declared.
Add lib/*.so* for that spelling. It matches nothing on x86_64, whose lib/ holds
only directories.
Verified on linux/x86_64: native x86_64 and all three cross configs still build
clean under both the default and the hermetic sandbox. The arm64 native case
needs an arm64 host, so CI's native-arm64 hermetic step is the check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gartnera
force-pushed
the
alexg/remote-execution-hermeticity
branch
from
September 9, 2026 16:19
7b4ca4d to
61aff32
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The toolchains built only because the default linux sandbox symlinks declared inputs back into the output base:
realpathin the gcc wrappers followed those symlinks out to the fully unpacked archive, and gcc was then driven with -B/--sysroot pointing there, so it freely read files that were never declared as action inputs. A remote executor materializes only the declared inputs, so those files are simply absent and the build fails.The
librariesfilegroups matched only .so/*.a, which missed everything gcc and ld actually reach at action time: the extensionless gcc internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects, gcc's .spec files and ld's linker scripts. Glob the whole directories instead. The native archives additionally need the merged-/usr spelling of their shared libraries, because glibc's ld scripts name /lib//libc.so.6 and /lib64/ld-linux-.so, which ld resolves against --sysroot.Once those inputs are declared, the gcc wrappers can no longer find the sysroot by running
findfor libc.so.6 and walking up three directories: libc.so.6 is now reachable at two depths and the guess is ambiguous. Derive it from the realpath of the gcc binary instead. That also has to be a file rather than the archive directory -- gcc resolves its own binary before deriving the search paths it hands to ld, and if --sysroot names a different directory than gcc resolved to, ld stops treating the absolute paths in glibc's GROUP(...) scripts as sysroot-relative and the link fails on __libc_start_main.Add a
hermeticconfig to tests/.bazelrc that builds under Bazel's hermetic linux sandbox, which bind-mounts only declared inputs and so reproduces the RE contract locally, and run it from CI for every configuration so this cannot regress. Document the RE exec-platform and container requirements in README.md.Verified on linux/x86_64 with Bazel 6.5.0: all three cross configs and the native x86_64 build pass under both the default and the hermetic sandbox, and the cross binaries still run under QEMU.