Skip to content

Support constrained LLVM catalogues, hermetic-llvm and MSan packages - #942

Open
helly25 wants to merge 2 commits into
bazel-contrib:masterfrom
helly25:feat/hermetic-llvm-packages
Open

helly25 wants to merge 2 commits into
bazel-contrib:masterfrom
helly25:feat/hermetic-llvm-packages

Conversation

@helly25

@helly25 helly25 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Support constrained LLVM distributions using provider-neutral catalogue-header metadata, with opt-in profiles for hermetic-llvm and the MSan packages used by mbo/xff.

  • Linux uses host/sysroot libstdc++ and libgcc; macOS uses Apple SDK libc++ with the native linker.
  • Catalogue headers can declare archive/platform mappings, extraction layout, required/excluded features, and allowed standard libraries.
  • The MSan profile pairs LLVM 22.1.8 with the exact matching instrumented libc++ automatically, with checksums for both native Linux x86_64 and AArch64 packages.
  • Incompatible configurations fail explicitly during toolchain selection, before obscure missing-file or runtime-library errors.
  • Default official LLVM selection is unchanged. In particular, this does not enable official LLVM's unsupported large-window zstd archives.

Implementation

Extend _meta without changing the existing archive: sha256 entries. Profiles are attached to the actual extracted compiler and follow feature-override archive reuse. Validation uses the consuming configuration, works with Bazel 7's exec-configured tool inputs, and ignores inactive variants/nonmatching target platforms.

Bring forward the optional-tool and libc++-header work from #935 by @ravi-pplx, plus the WORKSPACE Wasm dependency-order fix discovered while testing that PR. Also handle missing top-level headers, llvm-as, compiler-runtime directories, and optional dwp/profdata inputs. Required compiler/linker tools and Clang resource headers remain required.

Add a curated, checksum-pinned, opt-in hermetic_llvm.jsonc for release llvm-23.1.2-2. Its header requires libstdc++ on Linux and excludes sanitizer features because the packages contain no sanitizer runtimes. The regular LLVM catalogue remains the default; this is not a switch to hermetic-llvm's Bazel toolchain/module or a complete hermetic target environment.

Add msan.jsonc for RealtimeRoboticsGroup/toolchains LLVM 22.1.8 and its matched MSan libc++ packages, selected through the existing feature-override mechanism. The compiler/runtime pair requires msan and bundled libc++; ordinary target and execution configurations retain their default compiler. Migrate the existing MSan CI configuration from manual URLs/checksums to this catalogue. This PR does not modify mbo or xff.

Generic _meta.overlays entries select companion archives by exact LLVM version and execution platform, verify their SHA-256 and extract them into a named subdirectory. Missing matches fail rather than silently mixing runtime versions. A catalogued libcxx-msan overlay rejects a conflicting manual libcxx_url; the legacy optional overlay remains supported for ordinary compilers. Document both profiles and the generic schema, with a ready-to-use README MSan example.

Verification

Local environment Bazel/mode Result
macOS ARM64, Apple SDK libc++ 9.3.0 Bzlmod Sandbox C/C++ compilation, archives, C++ exceptions, linking and execution pass; ASan rejected during analysis
macOS ARM64, Apple SDK libc++ 7.7.2 WORKSPACE Same positive and negative checks pass
Linux ARM64, existing GCC 14 container, libstdc++/libgcc 9.3.0 Bzlmod Same build/run checks pass; ASan and incompatible libc++ override rejected during analysis
Linux ARM64, existing GCC 14 container, libstdc++/libgcc 7.7.2 WORKSPACE Same positive and negative checks pass
  • All 79 //toolchain/... tests pass with Bazel 9.3.0 Bzlmod and Bazel 7.7.2 WORKSPACE, including unchanged official-distribution golden tests.
  • New tests cover metadata/URL/checksum merging, custom host-aware version selection, canonical platform keys, required MSan features, explicit feature disabling, standard-library restrictions, optional-tool preservation, exact overlay matching and conflicting runtime sources.
  • The real LLVM 22.1.8/MSan libc++ pair passes sandboxed C instrumentation and C++ standard-library tests on Linux ARM64 with Bazel 9.3.0 Bzlmod and Bazel 7.7.2 WORKSPACE, including layering_check. Absent/explicitly disabled MSan restores the ordinary compiler. MSan+UBSan also passes with Bazel 9.3.0. The local Docker test uses a per-container seccomp exception for MSan's process-local personality(ADDR_NO_RANDOMIZE) call; no host/VM ASLR setting was changed.
  • Repository Trunk checks and git diff --check pass.
  • Add required hermetic-llvm CI coverage for macOS/Linux, Bazel 7 Bzlmod/WORKSPACE and Bazel 9 Bzlmod. Add focused MSan catalogue CI with Bazel 7 WORKSPACE and Bazel 9 Bzlmod, exercising the LLVM 23/libstdc++ default → LLVM 22/MSan libc++ override and disabled-feature fallback. Linux x86_64 is covered by CI, not claimed as locally tested.

Both opt-in catalogues are currently curated separately from the official updater. The hermetic-llvm profile does not supply sanitizer runtimes or an Apple LTO plugin; the MSan profile supplies its matched instrumented libc++ overlay.

Extend catalogue headers with platform mappings, archive layout and configuration requirements. Preserve optional-tool fixes from bazel-contrib#935 and verify real compiler-only packages with system C++ runtimes.

Co-authored-by: Ravi Agarwal <223930721+ravi-pplx@users.noreply.github.com>
Signed-off-by: helly25 <6420169+helly25@users.noreply.github.com>
Signed-off-by: helly25 <6420169+helly25@users.noreply.github.com>
@helly25 helly25 changed the title Support constrained distribution catalogues and hermetic-llvm packages Support constrained LLVM catalogues, hermetic-llvm and MSan packages Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant