Skip to content

Declare all toolchain inputs so builds work under remote execution - #21

Merged
gartnera merged 2 commits into
masterfrom
alexg/remote-execution-hermeticity
Sep 9, 2026
Merged

gartnera merged 2 commits into
masterfrom
alexg/remote-execution-hermeticity

Conversation

@gartnera

Copy link
Copy Markdown
Member

The toolchains built only because the default linux sandbox symlinks declared inputs back into the output base: realpath in the gcc wrappers followed those symlinks out to the fully unpacked archive, and gcc was then driven with -B/--sysroot pointing there, so it freely read files that were never declared as action inputs. A remote executor materializes only the declared inputs, so those files are simply absent and the build fails.

The libraries filegroups matched only .so/*.a, which missed everything gcc and ld actually reach at action time: the extensionless gcc internals (cc1, cc1plus, collect2, lto1, lto-wrapper), the CRT objects, gcc's .spec files and ld's linker scripts. Glob the whole directories instead. The native archives additionally need the merged-/usr spelling of their shared libraries, because glibc's ld scripts name /lib//libc.so.6 and /lib64/ld-linux-.so, which ld resolves against --sysroot.

Once those inputs are declared, the gcc wrappers can no longer find the sysroot by running find for libc.so.6 and walking up three directories: libc.so.6 is now reachable at two depths and the guess is ambiguous. Derive it from the realpath of the gcc binary instead. That also has to be a file rather than the archive directory -- gcc resolves its own binary before deriving the search paths it hands to ld, and if --sysroot names a different directory than gcc resolved to, ld stops treating the absolute paths in glibc's GROUP(...) scripts as sysroot-relative and the link fails on __libc_start_main.

Add a hermetic config to tests/.bazelrc that builds under Bazel's hermetic linux sandbox, which bind-mounts only declared inputs and so reproduces the RE contract locally, and run it from CI for every configuration so this cannot regress. Document the RE exec-platform and container requirements in README.md.

Verified on linux/x86_64 with Bazel 6.5.0: all three cross configs and the native x86_64 build pass under both the default and the hermetic sandbox, and the cross binaries still run under QEMU.

@gartnera
gartnera marked this pull request as ready for review August 28, 2026 18:24
gartnera and others added 2 commits September 9, 2026 16:16
The toolchains built only because the default linux sandbox symlinks declared
inputs back into the output base: `realpath` in the gcc wrappers followed those
symlinks out to the fully unpacked archive, and gcc was then driven with
-B/--sysroot pointing there, so it freely read files that were never declared as
action inputs. A remote executor materializes only the declared inputs, so those
files are simply absent and the build fails.

The `libraries` filegroups matched only *.so*/*.a, which missed everything gcc
and ld actually reach at action time: the extensionless gcc internals (cc1,
cc1plus, collect2, lto1, lto-wrapper), the CRT objects, gcc's *.spec files and
ld's linker scripts. Glob the whole directories instead. The native archives
additionally need the merged-/usr spelling of their shared libraries, because
glibc's ld scripts name /lib/<triple>/libc.so.6 and /lib64/ld-linux-*.so, which
ld resolves against --sysroot.

Those merged-/usr spellings are architecture-specific -- lib64/ is the x86_64
layout and the arm64 archive has no lib64 at all -- so the glob needs
allow_empty = True, as Bazel 8 rejects a pattern that matches nothing by
default (see #23).

Once those inputs are declared, the gcc wrappers can no longer find the sysroot
by running `find` for libc.so.6 and walking up three directories: libc.so.6 is
now reachable at two depths and the guess is ambiguous. Derive it from the
realpath of the gcc binary instead. That also has to be a file rather than the
archive directory -- gcc resolves its own binary before deriving the search
paths it hands to ld, and if --sysroot names a different directory than gcc
resolved to, ld stops treating the absolute paths in glibc's GROUP(...) scripts
as sysroot-relative and the link fails on __libc_start_main.

Add a `hermetic` config to tests/.bazelrc that builds under Bazel's hermetic
linux sandbox, which bind-mounts only declared inputs and so reproduces the RE
contract locally, and run it from CI for every configuration so this cannot
regress. Document the RE exec-platform and container requirements in README.md.

Verified on linux/x86_64 with Bazel 6.5.0: all three cross configs and the
native x86_64 build pass under both the default and the hermetic sandbox, and
the cross binaries still run under QEMU.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `libraries` filegroup declared the merged-/usr spelling of the interpreter
as lib64/*.so* and usr/lib64/*.so*, which is the x86_64 layout. On arm64 there
is no lib64 at all: ubuntu-22.04-aarch64-native ships `lib -> usr/lib` plus
usr/lib/ld-linux-aarch64.so.1, and its libc.so reads

    GROUP ( /lib/aarch64-linux-gnu/libc.so.6 ... AS_NEEDED ( /lib/ld-linux-aarch64.so.1 ) )

so ld resolves the interpreter directly under the sysroot's lib/, which no glob
matched. Under the hermetic sandbox or remote execution that file is absent and
the link fails on the AS_NEEDED, exactly as the x86_64 link did before
lib64/*.so* was declared.

Add lib/*.so* for that spelling. It matches nothing on x86_64, whose lib/ holds
only directories.

Verified on linux/x86_64: native x86_64 and all three cross configs still build
clean under both the default and the hermetic sandbox. The arm64 native case
needs an arm64 host, so CI's native-arm64 hermetic step is the check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gartnera
gartnera force-pushed the alexg/remote-execution-hermeticity branch from 7b4ca4d to 61aff32 Compare September 9, 2026 16:19
@gartnera
gartnera merged commit d350391 into master Sep 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant