Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,11 @@ common:clang_release --config=clang --config=release
common:linux_perf --define=pfm=1
common:linux_perf --downloader_config=bazelmod/linux_perf_downloader.cfg

# Unconfigured source queries visit every select arm, including Google Benchmark's
# optional libpfm dependency. Repair that one retired mirror for queries too,
# without enabling hardware counters or changing default/release build settings.
query --downloader_config=bazelmod/linux_perf_downloader.cfg

# --config=clang-coverage : collect C++ coverage with the same hermetic Clang/libc++ toolchain
# that owns the default build. Bazel otherwise defaults C++ coverage to the local gcov toolchain.
common:clang-coverage --config=clang
Expand Down
19 changes: 13 additions & 6 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,12 +144,19 @@ scope or declaring new flags implemented. See
useful task size, throughput, first output, memory and actual thread/child counts. Include serial
command/concurrent-discovery, strict dependent expressions and parallel command controls.
Do not accept a default or performance claim from hosted correctness checks alone.
- [ ] JC07: Separate directory-admission tuning from worker-budget tuning. PR #991's experimental
64-sibling gate is not an optimum across storage types. Design explicit serial operation and
minimum independent-work/cost controls in the jobs family, preserving semantic/safety barriers
and command resource bounds. Distinguish cold start from warm dispatch, and qualify settings
on small slow frontiers as well as broad memory-backed workloads before choosing defaults.
Keep spellings unsettled until designed; follow JC01's unstable gate and self-documentation.
- [ ] JC07: Separate directory-admission tuning from worker-budget tuning. After #1034 removed
Abseil clock calibration, #991's 4 ms saved-work gate and synchronous first-child sample
are not recommended: Linux cheap scans save about 0.21 ms, while slow-child controls regress
by 28-59%; macOS broad/nested 4,096-file scans regress by 7-9%. Current paired measurements,
raw evidence and the retention recommendation are in docs/performance-analysis.md. Keep
#991 draft; preserve its independent libpfm query-downloader repair if admission is discarded.
Investigate actual pool startup/queue cost and bounded run-owned worker reuse without
serializing small expensive frontiers. Qualify heterogeneous reads and real slow storage.
Design explicit serial operation and minimum independent-work/cost controls in the jobs
family, preserving semantic/safety barriers and command resource bounds. Distinguish cold
start from warm dispatch; qualify settings on small slow frontiers and broad memory-backed
workloads before choosing defaults. Keep spellings unsettled until designed; follow JC01's
unstable gate and self-documentation.
- [ ] JC08: Add opt-in, super-low-overhead scheduling analysis for admission tuning. Select diagnostics
once at command setup; keep disabled inner loops free of diagnostic checks, timers, counters,
atomics and allocations. Use bounded per-role/worker aggregates from existing observations:
Expand Down
2 changes: 1 addition & 1 deletion XFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,7 @@ See also: [--help=config](#topic-config), [--help=archive](#topic-archive), [--h
<a id="flag-jobs"></a>

- `-j N, -j=N, --jobs=N|all` - directory-read, eligible content-match, comparison and -exec workers _(global, find, xff, rg)_
`N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories on up to `N` worker threads; large directory listings can split required eager metadata reads across the same pool. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches also match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
`N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories on up to `N` worker threads; large directory listings can split required eager metadata reads across the same pool. Directory startup uses observed read cost and remaining independent work; small cheap batches can stay inline, while an existing pool is reused. `N` is a limit, not a promise to parallelize every listing. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches also match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
See also: [--help=output](#topic-output), [--help=stats](#topic-stats)

<a id="flag-sort"></a>
Expand Down
Binary file not shown.
Binary file not shown.
10 changes: 8 additions & 2 deletions docs/design-parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,14 @@ A bounded directory-read worker pool with a single coordinator:

- Workers perform only `ReadDir` plus metadata lookup and return complete listings.
The coordinator submits sibling-directory reads ahead, then consumes their
futures in the order required by `--sort`. Directory workers start only when at least
two sibling directories provide independent work. For a coordinator-read directory with at least
futures in the order required by `--sort`. At least two sibling directories must provide
independent work. Cold startup additionally uses the last coordinator read's cost and the
estimated number of parallel rounds saved. If the parent does not justify startup, one child
is sampled inline and its ready result is retained, so it is never read twice. Cheap work stays
inline; expensive small batches can activate workers. An existing pool bypasses cold admission.
Only the coordinator records timing, only while the pool is cold; workers never share mutable
timing state. The admission budget is a heuristic, not a latency guarantee or an optimal policy
for every filesystem. For a coordinator-read directory with at least
512 children and eagerly required metadata, 128-entry stat chunks use the same pool. At most one
chunk per worker is pending; worker jobs never wait on nested jobs. Metadata-free, lazy and
owned/archive-source listings keep their serial per-directory path.
Expand Down
9 changes: 8 additions & 1 deletion docs/native-profiling.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,21 @@ filter, or timing columns alone do not establish working counters. Use
The short duration above is a capability probe, not performance evidence.

Google Benchmark 1.9.5 pins libpfm 4.11.0.bcr.1. That BCR source names the retired
`netcologne.dl.sourceforge.net` mirror. The opt-in downloader configuration
`netcologne.dl.sourceforge.net` mirror. The downloader configuration
redirects exactly that archive to `downloads.sourceforge.net`, preserving the
pinned version, BCR patches and integrity check. No LLVM/toolchain dependency
changes are involved. It does not rewrite other archives or disable checksum
validation. If network access is unavailable, an independently verified copy of
`libpfm-4.11.0.tar.gz` can be supplied through Bazel's `--distdir=PATH`; do not
substitute an unverified system library.

Unconfigured `bazel query` traverses all `select` arms, including the optional
libpfm dependency, even when hardware counters are disabled. Queries therefore
use the same narrowly scoped download repair so compile-source indexing can
complete. This does not enable counters or change default/release builds,
compiler/linker selection or macOS runtime behavior. Source indexing still
fails closed on any unsuccessful query.

## Kernel and external perf capability

The `perf` executable is a separate host profiling tool, not a libpfm dependency.
Expand Down
Loading
Loading