Skip to content

Trim toolchain file groups to the tools their actions run - #934

Merged
helly25 merged 3 commits into
bazel-contrib:masterfrom
patrickmscott:reduce-remote-inputs
Nov 6, 2026
Merged

helly25 merged 3 commits into
bazel-contrib:masterfrom
patrickmscott:reduce-remote-inputs

Conversation

@patrickmscott

Copy link
Copy Markdown
Contributor

all_files carried the LLVM distribution's whole bin/, and every file group carried the binary behind every wrapped tool (clang-tidy, clangd, ...), so remote executors staged gigabytes of unused inputs per action.

  • all_files takes the wrapped tools' binaries instead of all of bin/.
  • Other file groups keep only the wrappers and runtime_shared_libraries; each already lists the binaries its actions run.
  • Darwin-to-Darwin archive actions take llvm-libtool-darwin, which the libtool wrapper runs.

@helly25

helly25 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

all_files carried the LLVM distribution's whole bin/, and every file
group carried the binary behind every wrapped tool (clang-tidy, clangd,
...), so remote executors staged gigabytes of unused inputs per action.

- all_files takes the wrapped tools' binaries instead of all of bin/.
- Other file groups keep only the wrappers and runtime_shared_libraries;
  each already lists the binaries its actions run.
- Darwin-to-Darwin archive actions take llvm-libtool-darwin, which the
  libtool wrapper runs.
@patrickmscott

Copy link
Copy Markdown
Contributor Author

@helly25 updated

@helly25

helly25 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

@patrickmscott: The PR indeed reduces unnecessary compiler/linker inputs introduced by #914 while retaining the shared libraries required on remote workers.

Correctness issue remain: all_files drops llvm-as, although as_files still includes it through the distribution’s :as filegroup. Bazel requires all_files to be a superset of the action-specific sets. Ordinary compilation uses Clang’s integrated assembler, so existing tests can pass despite this omission. Changed input sets (https://github.com/bazel-contrib/toolchains_llvm/blob/01796b567d37d6e649f594d6d7129e9d12920c4a/toolchain/internal/configure.bzl#L900-L916), Bazel’s contract (https://bazel.build/reference/be/c-cpp#cc_toolchain.all_files).

The rest of the change looks good:

  • Compiler inputs retain Clang and the module dependency scanner.
  • Shared runtime libraries remain available.
  • The Darwin archiver change correctly supplies llvm-libtool-darwin, matching the existing tool selection.
  • All 99 CI checks passed.
  • The linker override remains available through both input paths:
    • Downloaded linkers such as mold: included through bin/linkers/ with directory inputs, or explicitly through linker_file with legacy inputs.
    • auto and absolute paths: resolution is unchanged; these still require the linker installed on the execution machine.
  • The PR’s mold, native-linker, feature-override, and Darwin ThinLTO CI tests all passed. The llvm-as omission I identified is separate from linker overrides.

Remaining work:

  1. Restore the all_files superset invariant, preferably by composing it from the action-specific groups.
  2. Add regression coverage checking both required inputs and exclusion of unrelated binaries.
  3. Provide one remote execution—or equivalent isolated staging—result using shared-library LLVM tools. The PR adds no tests demonstrating that scenario.

@patrickmscott

Copy link
Copy Markdown
Contributor Author

@helly25 I addressed all_files and added a test

@helly25
helly25 merged commit b9bec79 into bazel-contrib:master Nov 6, 2026
99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants