Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 20 additions & 13 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,19 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
os: [ubuntu-24.04, ubuntu-latest, macos-latest]
config:
- name: default
flags: "--config=clang_release"
- name: asan
flags: "--config=clang --config=asan"
exclude:
# asan on Linux is enough; the macOS asan cell is redundant sanitizer coverage, so drop it.
# This leaves: ubuntu default, ubuntu asan, macos default.
# Exercise both Linux FUSE SONAMEs in default builds; one Linux asan cell is enough.
# This leaves: Ubuntu 24.04 default, latest default + asan, and macOS default.
- os: ubuntu-24.04
config:
name: asan
flags: "--config=clang --config=asan"
- os: macos-latest
config:
name: asan
Expand All @@ -90,7 +94,7 @@ jobs:
id: bazel-cache
uses: ./.github/actions/bazel-cache-restore
with:
namespace: ${{ matrix.config.name }}
namespace: ${{ matrix.os == 'ubuntu-24.04' && 'default-ubuntu24' || matrix.config.name }}
legacy-prefix: bazel-disk-${{ matrix.os }}-${{ matrix.config.name }}-
# Every matrix cell uses hermetic LLVM. Do not persist the repository cache: one LLVM archive
# is ~4.4 GB and would consume almost half of GitHub's per-repository cache budget. Compiled
Expand All @@ -105,13 +109,14 @@ jobs:
command -v mandoc
# The FUSE server's integration test mounts a real filesystem through the kernel, which only
# Linux runners can do (macOS has no fuse3; the test skips there by design). Install the
# runtime pieces the loader dlopens (libfuse3) and the unmount helper (fusermount3). The test
# fuse3 package, which pulls in the matching libfuse3 runtime (SONAME 3 on Ubuntu 24.04,
# SONAME 4 on Ubuntu 26.04) and the unmount helper (fusermount3). The test
# command below opts Linux into the Bazel capability contract, so mounting failures cannot
# silently become skips.
- name: Enable FUSE integration coverage (Linux)
run: |
if [ "${RUNNER_OS}" = "Linux" ]; then
sudo apt-get install -y fuse3 libfuse3-3
sudo apt-get install -y fuse3
fi
# Every job builds with EVERY extra on (--config=xff_docs, which is --config=xff_full plus the
# archive extra), so the composable extras get the SAME asan / tsan coverage as the core - our
Expand Down Expand Up @@ -355,7 +360,7 @@ jobs:
run: |
# Ubuntu's lcov package supplies the Perl dependencies, but its old genhtml ignores
# metric-specific color thresholds. Install the pinned current release over it below.
sudo apt-get install -y lcov mandoc fuse3 libfuse3-3
sudo apt-get install -y lcov mandoc fuse3
tools/install_lcov.sh "${RUNNER_TEMP}/lcov"
echo "${RUNNER_TEMP}/lcov/bin" >>"${GITHUB_PATH}"
- name: Collect and enforce coverage
Expand Down Expand Up @@ -477,7 +482,7 @@ jobs:
# This runner promises a working mount path through the named Bazel config below.
- name: Enable FUSE integration coverage
run: |
sudo apt-get install -y fuse3 libfuse3-3
sudo apt-get install -y fuse3
- name: bazel test
run: |
# shellcheck disable=SC2046 # deliberate word splitting: the tool prints bazel target patterns
Expand Down Expand Up @@ -507,25 +512,27 @@ jobs:
contents: read
actions: write # Retire replaced main cache generations after successful uploads.
needs: [pre-commit, trunk]
runs-on: ubuntu-latest
# LLVM 22.1.8 LLD requires libxml2.so.2; Ubuntu 26.04 only packages SONAME 16.
# Keep the matching MSan compiler/libc++ pair on a host with compatible libraries.
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- uses: bazel-contrib/setup-bazel@0.19.0
- name: Restore Bazel compiled outputs
id: bazel-cache
uses: ./.github/actions/bazel-cache-restore
with:
namespace: msan
namespace: msan-ubuntu24
legacy-prefix: bazel-disk-msan-
- name: Install roff formatter (mandoc)
run: sudo apt-get install -y mandoc
- name: Install LLVM 22 runtime dependency and roff formatter
run: sudo apt-get install -y libxml2 mandoc
# Installed for the same reason as the tsan cell, though every mounting test skips here: a
# mount runs through the dlopened SYSTEM libfuse3, and MSan reports everything it did not
# instrument as uninitialized. The skips key off --config=msan itself (the MEMORY_SANITIZER
# macro in C++, the XFF_MSAN env in the shell test), so they hold whether or not this is set.
- name: Enable FUSE integration coverage
run: |
sudo apt-get install -y fuse3 libfuse3-3
sudo apt-get install -y fuse3
- name: bazel test
run: |
# shellcheck disable=SC2046 # deliberate word splitting: the tool prints bazel target patterns
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
run: |
# Ubuntu's lcov package supplies the Perl dependencies, but its old genhtml ignores
# metric-specific color thresholds. Install the pinned current release over it below.
sudo apt-get install -y lcov mandoc fuse3 libfuse3-3
sudo apt-get install -y lcov mandoc fuse3
tools/install_lcov.sh "${RUNNER_TEMP}/lcov"
echo "${RUNNER_TEMP}/lcov/bin" >>"${GITHUB_PATH}"
- name: Check live repository tooling
Expand Down
25 changes: 23 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,16 +68,37 @@ No tests contact GitHub or recursively invoke Bazel; the separate site integrati

## Compiler and sanitizer versions

`--config=clang` uses LLVM 23.1.2. The `msan` feature selects LLVM 22.1.8 and its matching
`--config=clang` selects the newest stable LLVM 23 release available for the host from the pinned
`toolchains_llvm` catalogue (`latest:major==23`). Updating that module can adopt LLVM 23 fixes
without a separate compiler-version edit; LLVM 24 requires an explicit selector change. The
`msan` feature selects LLVM 22.1.8 and its matching
instrumented libc++ on Linux x86-64, independently of config flag order. ASan, TSan, coverage,
fuzzing, releases, build tools, and clang-tidy use LLVM 23. The existing `--config=lto` selection
continues to use native Apple ld on macOS and bundled LLD on Linux.

Pre-commit and `tools/clang_format.sh` run clang-format from the same pinned LLVM 23 distribution.
The official LLVM 22.1.8 LLD executable requires `libxml2.so.2`. The MSan CI job uses
Ubuntu 24.04 and explicitly installs `libxml2`; Ubuntu 26.04 only packages `libxml2.so.16`,
which cannot satisfy that dependency. Ordinary LLVM 23 builds and the other sanitizer jobs
continue on `ubuntu-latest`. Moving MSan to that host requires a compatible compiler/runtime
distribution; keep its compiler and instrumented libc++ versions matched.

Pre-commit and `tools/clang_format.sh` run clang-format from the same selected LLVM 23 distribution.
The first run fetches that distribution if needed; it does not compile XFF. CI runs clang-tidy
across the complete indexed source set when the compiler or lint policy changes; ordinary source
changes keep the affected-file scope.

## Linux FUSE runtimes

The FUSE extra loads either `libfuse3.so.4` (Ubuntu 26.04) or `libfuse3.so.3` (Ubuntu 24.04)
at runtime. Install `fuse3`; its package dependency selects the matching runtime library without
pinning a distribution-specific library package name. Both SONAMEs expose the FUSE3 API subset
used by xff, including the four-argument `fuse_session_new` compatibility entry point. No libfuse
headers or libraries are included in the build.

Linux CI runs the real mount tests on Ubuntu 24.04 and `ubuntu-latest`, with
`--config=xff_fuse_tests_required` so missing libraries or failed mounts fail the job. Hosts without
a usable FUSE runtime retain the existing extraction fallback.

## Local clang-tidy scope

The normal commit and push hooks run clang-tidy on their changed C++ files. Changed headers
Expand Down
20 changes: 14 additions & 6 deletions bazelmod/llvm.MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,14 @@

"""LLVM 23 by default; LLVM 22 with its matching instrumented libc++ for MSan."""

# The release provides feature-based linker selection and native macOS ThinLTO.
bazel_dep(name = "toolchains_llvm", version = "1.11.3")
# Pin the selector support merged after 1.12.0; use a release containing it when available.
# https://github.com/bazel-contrib/toolchains_llvm/pull/911
bazel_dep(name = "toolchains_llvm", version = "1.12.0")
git_override(
module_name = "toolchains_llvm",
commit = "b63941bcc533a52e1b81c81af5fe77d5a2b382f8",
remote = "https://github.com/bazel-contrib/toolchains_llvm.git",
)

# The MemorySanitizer-instrumented libc++ (`--config=msan`, Linux only). MSan reports a
# false positive for every read of memory written by code it did not instrument, and the
Expand All @@ -41,8 +47,10 @@ bazel_dep(name = "toolchains_llvm", version = "1.11.3")
# Keep the MSan compiler and instrumented standard library on the same release.
# The ordinary compiler can advance independently through the feature override below.
MSAN_LIBCXX_URL = "https://github.com/RealtimeRoboticsGroup/toolchains/releases/download/{v}/libcxx-msan-{v}-x86_64-linux-gnu-ubuntu-22.04.tar.zst"
LLVM_VERSION = "23.1.2"
MSAN_LLVM_VERSION = "22.1.8"
# Follow stable LLVM 23 releases from the pinned toolchains_llvm catalogue.
# Major compiler and clang-tidy upgrades require an explicit policy change.
LLVM_VERSION = "latest:major==23"
LLVM_VERSION_MSAN = "22.1.8"

llvm = use_extension("@toolchains_llvm//toolchain/extensions:llvm.bzl", "llvm", dev_dependency = True)
llvm.toolchain(
Expand Down Expand Up @@ -77,9 +85,9 @@ llvm.feature_override(
features = ["msan"],
targets = ["linux-x86_64"],
libcxx_sha256 = "fd1ac49eeda4edc6deeda2037998e558718a63a5054fff4fdace7751b270217a",
libcxx_url = MSAN_LIBCXX_URL.format(v = MSAN_LLVM_VERSION),
libcxx_url = MSAN_LIBCXX_URL.format(v = LLVM_VERSION_MSAN),
linker = {"linux-x86_64": ""},
llvm_version = MSAN_LLVM_VERSION,
llvm_version = LLVM_VERSION_MSAN,
)
use_repo(llvm, "llvm_toolchain_llvm")

Expand Down
10 changes: 7 additions & 3 deletions extra_modules/fuse/fuse_abi.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,16 @@
// `fuse_conn_info` are only ever passed through, so their layouts are irrelevant AND absent.
// What is not written cannot drift.
// 2. Anything whose fields we touch carries the exact layout, field for field, in libfuse's own
// order, transcribed against fuse-3.18.2. Order IS the ABI.
// order, transcribed against fuse-3.18.2. Order IS the ABI. The prefix we use also matches
// fuse-3.14.0 (Linux SONAME 3); fuse-3.18.2 uses SONAME 4. We only access the common
// fuse_file_info fields and lowlevel ops prefix, and use the exported four-argument
// fuse_session_new compatibility entry point on both runtimes.
//
// HOW A MISTAKE SURFACES: the loader resolves every entry point by NAME (see fuse_loader.h), so a
// wrong signature or field offset cannot be caught at link time - it corrupts at runtime. The guard
// is Linux CI, where fuse_server_test and the --archive-mount CLI cases drive a real fuse3: a
// mismatched `fuse_lowlevel_ops` prefix or a shifted `fuse_file_info::fh` fails there loudly.
// is Linux CI on Ubuntu 24.04 (SONAME 3) and ubuntu-latest (SONAME 4), where fuse_server_test
// and the --archive-mount CLI cases drive a real fuse3: a mismatched `fuse_lowlevel_ops` prefix or a shifted
// `fuse_file_info::fh` fails there loudly.
//
// EXTENDING IT: fields are only ever APPENDED to `fuse_lowlevel_ops`, and fuse_server passes
// `op_size` as the offset just past the LAST op it implements (never `sizeof`), so this prefix means
Expand Down
1 change: 1 addition & 0 deletions extra_modules/fuse/fuse_loader.cc
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ constexpr std::array kLibraryCandidates = std::to_array<std::string_view>({
"/usr/local/lib/libfuse3.dylib",
"libfuse3.dylib",
#else
"libfuse3.so.4",
"libfuse3.so.3",
"libfuse3.so",
#endif
Expand Down
7 changes: 4 additions & 3 deletions extra_modules/fuse/fuse_loader.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@
// mount server needs is resolved then. One binary runs everywhere; where the library is absent,
// `--archive-mount` degrades to extraction and this loader says why.
//
// Scope: the FUSE3 API only. macFUSE installations that ship only the legacy fuse2 library report
// unavailable (with the library name in `error()`); supporting the fuse2 lowlevel API is a separate
// decision to take when a real macFUSE user appears, not a default.
// Scope: the FUSE3 API, including Linux SONAMEs 3 and 4. The ABI subset in fuse_abi.h
// works with both; SONAME 4 is still the FUSE3 API. macFUSE installations that ship only the legacy fuse2 library
// report unavailable (with the library name in `error()`); supporting the fuse2 lowlevel API is a separate decision to
// take when a real macFUSE user appears, not a default.

#include <string>
#include <string_view>
Expand Down
4 changes: 3 additions & 1 deletion tools/fuse_linux_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ export DEBIAN_FRONTEND=noninteractive
apt-get update -qq >/dev/null
# git is not optional: MODULE.bazel pulls toolchains_llvm and hedron_compile_commands through
# git_repository, so a cold cache cannot even compute the repo mapping without it.
apt-get install -y -qq curl git gcc g++ python3 unzip zip fuse3 libfuse3-3 >/dev/null
apt-get install -y -qq curl git gcc g++ python3 unzip zip fuse3 >/dev/null
arch="$(uname -m)"; [ "${arch}" = "aarch64" ] && arch=arm64 || arch=amd64
curl -sSL -o /usr/local/bin/bazel "https://github.com/bazelbuild/bazelisk/releases/download/v1.20.0/bazelisk-linux-${arch}"
chmod +x /usr/local/bin/bazel
Expand All @@ -64,6 +64,8 @@ SETUP_EOF

if [[ "${MODE}" == "msan" ]]; then
script="${SETUP}
# The LLVM 22.1.8 linker needs SONAME 2, available in this Ubuntu 24.04 image.
apt-get install -y -qq libxml2 >/dev/null
bazel --output_user_root=/bzcache test --config=xff_full @xff_fuse//... //xff/cli:full_extras_test \\
--config=clang --config=msan --config=xff_fuse_tests_required \\
--nocache_test_results --test_output=errors"
Expand Down
4 changes: 3 additions & 1 deletion xff/cli/help_topic_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -398,8 +398,10 @@ test::bad_flags_are_hard_errors_even_with_help() {
test::meta_spelling_inside_exec_stays_a_child_argument() {
# Meta recognition follows parser boundaries. A child command is an opaque
# argument run, so its `--help` must not turn xff itself into the usage page.
# Use the shell builtin: external printf implementations can interpret --help themselves.
local out
out="$("$(_xff_bin)" --allow-exec . -maxdepth 0 -exec printf 'child:%s\n' --help ';' 2>&1)"
# shellcheck disable=SC2016 # The child shell expands its own positional argument.
out="$("$(_xff_bin)" --allow-exec . -maxdepth 0 -exec sh -c 'printf "child:%s\n" "$1"' child --help ';' 2>&1)"
expect_output_contains 'child:--help' "${out}"
expect_output_not_contains 'eXtended File Find' "${out}"
}
Expand Down
Loading