Isolate executor diagnostics and query download support

.bazelrc

@@ -140,6 +140,11 @@ common:clang_release --config=clang --config=release
 common:linux_perf --define=pfm=1
 common:linux_perf --downloader_config=bazelmod/linux_perf_downloader.cfg
 
+# Unconfigured source queries visit every select arm, including Google Benchmark's
+# optional libpfm dependency. Repair that one retired mirror for queries too,
+# without enabling hardware counters or changing default/release build settings.
+query --downloader_config=bazelmod/linux_perf_downloader.cfg
+
 # --config=clang-coverage : collect C++ coverage with the same hermetic Clang/libc++ toolchain
 # that owns the default build. Bazel otherwise defaults C++ coverage to the local gcov toolchain.
 common:clang-coverage --config=clang

bazelmod/clang_tidy_targets.bzl

@@ -12,6 +12,7 @@ CLANG_TIDY_MANUAL_TARGETS = [
     "//xff/cli:globals_benchmark",
     "//xff/engine:read_accounting_test",
     "//xff/engine:compare_benchmark",
+    "//xff/engine:executor_benchmark",
     "//xff/engine:rg_benchmark",
     "//xff/engine:expression_benchmark",
     "//xff/engine:expression_run_benchmark",

docs/benchmarks/executor-dispatch-probe.md

@@ -1,30 +1,29 @@
 # Executor lifecycle and task-grain probe
 
 This page retains the measurement evidence from [PR #1000](https://github.com/mboworks/xff/pull/1000)
-without merging its executor or diagnostic C++ targets. The probe target is not
-available on main: reproducing the experiment requires its exact measured source,
-`c8929d16c77e318b15d87c5bfcbf07af0fa7058e`. The lossless bundle can instead be
-verified offline from main without building or executing any measured binary.
-
-At that source, `//xff/engine:executor_benchmark` is an explicit diagnostic, not a
-shipping dependency or a new runtime admission rule. It separates thread-start calls,
-first dispatch, warmed work and executor teardown. Use a fresh executable process
-for every cold sample: a previous executor or other Abseil contention may already
-have calibrated the process's clock. Ordinary in-process repetition cannot
-establish fresh-process startup cost.
-
-In a separate checkout of the measured source, build before measuring, then
-invoke the resolved executable directly. Do not build this target from main or
-relabel a new build as verification of the historical executable bytes:
+and makes its executor probe available as a test-only diagnostic. It separates
+thread-start calls, first dispatch, warmed work and executor teardown. Use a
+fresh executable process for every cold sample.
+
+Build before measuring, then invoke the resolved executable directly:
 
 ```sh
-git fetch origin audit/executor-dispatch-costs
-git worktree add --detach /tmp/xff-executor-probe-repro c8929d16c77e318b15d87c5bfcbf07af0fa7058e
-cd /tmp/xff-executor-probe-repro
 bazel build --config=clang_release //xff/engine:executor_benchmark
 bazel-bin/xff/engine/executor_benchmark leaf 3 8192 16 128 32
 ```
 
+The candidate executor support and probe libraries are private `testonly`
+targets. Bazel prevents shipping binaries from depending on them. Production
+traversal and matching keep their existing scheduling implementations; no
+admission, shared-pool, frontier, caller-claim or remaining-work policy is
+installed by this infrastructure extraction.
+
+Historical results below describe the exact measured source
+`c8929d16c77e318b15d87c5bfcbf07af0fa7058e` and original flags. The current main
+includes the Abseil steady-clock correction, so new builds must retain their
+own provenance and must not be relabeled as verification of historical binary
+bytes or used to justify historical cold-start thresholds.
+
 The six arguments are dispatch style (`leaf`, `claimable`, `drain`), caller-inclusive
 participants (1..64), items per warmed sweep, sweep count, chunk grain and unsigned
 mixing iterations per item. All dimensions except mixing iterations must be positive;

docs/benchmarks/native-executor-dispatch-controls.json

@@ -147,6 +147,6 @@
     "Phase times include diagnostic bookkeeping; fastest seven selects warmed phase, not process wall time.",
     "CPU model, kernel version and per-sample thermal/load observations were not recorded by this harness.",
     "Guest-visible distinct package/core pairs do not independently verify physical-host cores or QEMU thread affinity.",
-    "Diagnostic C++ targets and their executor dependency are not part of this evidence-only extraction onto main."
+    "Historical evidence remains unchanged; the current diagnostic targets are isolated behind testonly support."
   ]
 }

docs/native-profiling.md

@@ -28,14 +28,21 @@ filter, or timing columns alone do not establish working counters. Use
 The short duration above is a capability probe, not performance evidence.
 
 Google Benchmark 1.9.5 pins libpfm 4.11.0.bcr.1. That BCR source names the retired
-`netcologne.dl.sourceforge.net` mirror. The opt-in downloader configuration
+`netcologne.dl.sourceforge.net` mirror. The downloader configuration
 redirects exactly that archive to `downloads.sourceforge.net`, preserving the
 pinned version, BCR patches and integrity check. No LLVM/toolchain dependency
 changes are involved. It does not rewrite other archives or disable checksum
 validation. If network access is unavailable, an independently verified copy of
 `libpfm-4.11.0.tar.gz` can be supplied through Bazel's `--distdir=PATH`; do not
 substitute an unverified system library.
 
+Unconfigured `bazel query` traverses all `select` arms, including the optional
+libpfm dependency, even when hardware counters are disabled. Queries therefore
+use the same narrowly scoped download repair so compile-source indexing can
+complete. This does not enable counters or change default/release builds,
+compiler/linker selection or macOS runtime behavior. Source indexing still
+fails closed on any unsuccessful query.
+
 ## Kernel and external perf capability
 
 The `perf` executable is a separate host profiling tool, not a libpfm dependency.

tools/repository_tooling_test.py

@@ -69,6 +69,21 @@ def test_libpfm_downloader_rewrites_only_the_retired_archive(self):
                           retired + "?download=1", canonical.removeprefix("https://")):
             self.assertIsNone(re.fullmatch(pattern, unrelated))
 
+    def test_unconfigured_queries_repair_libpfm_without_enabling_counters(self):
+        commands = [shlex.split(line, comments=True)
+                    for line in (self.workspace / ".bazelrc").read_text().splitlines()]
+        commands = [command for command in commands if command]
+        download_settings = [command for command in commands
+                             if any(option.startswith("--downloader_config=") for option in command[1:])]
+        self.assertEqual(download_settings, [
+            ["common:linux_perf", "--downloader_config=bazelmod/linux_perf_downloader.cfg"],
+            ["query", "--downloader_config=bazelmod/linux_perf_downloader.cfg"],
+        ])
+        query_settings = [option for command in commands if command[0] == "query"
+                          for option in command[1:]]
+        self.assertNotIn("--define=pfm=1", query_settings)
+        self.assertNotIn("--config=linux_perf", query_settings)
+
 
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(description=__doc__)

xff/engine/BUILD.bazel

@@ -19,6 +19,73 @@ load("//xff:cc.bzl", "cc_binary", "cc_library", "cc_test")
 
 package(default_visibility = ["//visibility:private"])
 
+# Experimental scheduling support is restricted to tests and diagnostics.
+cc_library(
+    name = "run_executor_cc",
+    testonly = True,
+    srcs = ["run_executor.cc"],
+    hdrs = ["run_executor.h"],
+    deps = [
+        "@abseil-cpp//absl/base:core_headers",
+        "@abseil-cpp//absl/functional:any_invocable",
+        "@abseil-cpp//absl/synchronization",
+    ],
+)
+
+cc_test(
+    name = "run_executor_test",
+    size = "small",
+    srcs = ["run_executor_test.cc"],
+    deps = [
+        ":run_executor_cc",
+        "@abseil-cpp//absl/synchronization",
+        "@abseil-cpp//absl/time",
+        "@googletest//:gtest",
+        "@googletest//:gtest_main",
+    ],
+)
+
+cc_library(
+    name = "executor_measurement_cc",
+    testonly = True,
+    srcs = ["executor_measurement.cc"],
+    hdrs = ["executor_measurement.h"],
+    deps = [
+        ":run_executor_cc",
+        "@abseil-cpp//absl/status",
+        "@abseil-cpp//absl/status:statusor",
+    ],
+)
+
+cc_test(
+    name = "executor_measurement_test",
+    size = "small",
+    srcs = ["executor_measurement_test.cc"],
+    deps = [
+        ":executor_measurement_cc",
+        "@abseil-cpp//absl/status",
+        "@googletest//:gtest",
+        "@googletest//:gtest_main",
+        "@mboworks_mbo//mbo/testing:status_cc",
+    ],
+)
+
+# Fresh-process lifecycle / warmed-dispatch diagnostic, not runtime admission.
+cc_binary(
+    name = "executor_benchmark",
+    testonly = True,
+    srcs = ["executor_benchmark.cc"],
+    tags = [
+        "clang-tidy",
+        "manual",
+    ],
+    deps = [
+        ":executor_measurement_cc",
+        "@abseil-cpp//absl/strings",
+        "@nlohmann_json//:json",
+    ],
+)
+
 # Directory traversal over a vfs::FileSystem: a worker pool reads (readdir+lstat)
 # in parallel while a single coordinator thread visits in --sort order.
 cc_library(

xff/engine/executor_benchmark.cc

@@ -0,0 +1,79 @@
+// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors
+// SPDX-License-Identifier: Apache-2.0
+
+#include <cstddef>
+#include <iostream>
+#include <span>
+#include <string>
+#include <string_view>
+#include <vector>
+
+#include "absl/strings/numbers.h"
+#include "nlohmann/json.hpp"
+#include "xff/engine/executor_measurement.h"
+
+namespace {
+
+using ::xff::engine::ExecutorProbePhase;
+
+nlohmann::json Phase(const ExecutorProbePhase& phase) {
+  return {
+      {"seconds", phase.seconds},
+      {"items", phase.items},
+      {"queued_jobs", phase.queued_jobs},
+      {"observed_participants", phase.observed_participants},
+      {"caller_participated", phase.caller_participated},
+      {"checksum", phase.checksum},
+  };
+}
+
+// XFF_HOST_IO: diagnostic executable's result/error stream adapter; no path I/O.
+int Error(std::string_view message) {
+  std::cerr << message << '\n';
+  return 2;
+}
+
+// XFF_HOST_IO: diagnostic executable's result stream adapter; no path I/O.
+void Emit(const nlohmann::json& report) {
+  std::cout << report.dump() << '\n';
+}
+
+}  // namespace
+
+// XFF_ABI_POINTER: standard process entry point; argv is converted inside this adapter.
+int main(int argc, char** argv) {
+  if (argc != 7) {
+    return Error("usage: executor_benchmark leaf|claimable|drain PARTICIPANTS ITEMS SWEEPS GRAIN WORK_UNITS");
+  }
+  const std::span argument_view(argv, static_cast<std::size_t>(argc));
+  const std::vector<std::string> arguments(argument_view.begin(), argument_view.end());
+  const auto dispatch = xff::engine::ParseExecutorProbeDispatch(arguments.at(1));
+  if (!dispatch.ok()) {
+    return Error(dispatch.status().message());
+  }
+  xff::engine::ExecutorProbeConfig config{.dispatch = *dispatch};
+  if (!absl::SimpleAtoi(arguments.at(2), &config.participants) || !absl::SimpleAtoi(arguments.at(3), &config.items)
+      || !absl::SimpleAtoi(arguments.at(4), &config.sweeps) || !absl::SimpleAtoi(arguments.at(5), &config.grain)
+      || !absl::SimpleAtoi(arguments.at(6), &config.work_units)) {
+    return Error("probe dimensions must be unsigned integers");
+  }
+  const auto result = xff::engine::MeasureExecutor(config);
+  if (!result.ok()) {
+    return Error(result.status().message());
+  }
+  Emit({
+      {"dispatch", arguments.at(1)},
+      {"participants", config.participants},
+      {"items", config.items},
+      {"sweeps", config.sweeps},
+      {"grain", config.grain},
+      {"work_units", config.work_units},
+      {"background_workers", result->background_workers},
+      {"start_seconds", result->start_seconds},
+      {"first_dispatch", Phase(result->first_dispatch)},
+      {"warmed", Phase(result->warmed)},
+      {"teardown_seconds", result->teardown_seconds},
+      {"measurement", "synthetic executor phases including bookkeeping; fresh process required for cold costs"},
+  });
+  return 0;
+}