Skip to content

Prepare immutable predicate guards once per expression - #1155

Merged
helly25 merged 16 commits into
mainfrom
perf/prepared-predicate-guards-current-20261109
Nov 9, 2026
Merged

helly25 merged 16 commits into
mainfrom
perf/prepared-predicate-guards-current-20261109

Conversation

@helly25

@helly25 helly25 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Prepare immutable predicate guards once per expression so each file visit does less repeated
setup. Linux large-listing diagnostics execute about 3% fewer userspace instructions;
macOS large-case process instruction counts fall by about 0.7-1.0%. Elapsed gains are smaller,
and the qualification retains adverse results and identical-baseline controls.

AG;DR

Implementation and dependencies

  • Depends on merged Reduce Linux raw filename output overhead #1154 (ad3be8a12e05ec5ca1c1c346f3c8b12390163a28). The PR was synchronized with main at 60977f06e3142b055b5b23c1bbbdd5a1ff284de5; current head 1325f0cc2fff9e6f4d902945ec18c136933dac1c additionally fixes two test declarations and retains the findings in TODO/performance-analysis notes. The synchronization preserves its deterministic archive fixture and all release-site mappings. The synchronization tree equals the pre-sync tree; required checks run in this PR context against the new pushed head.

  • Follow-up to Reduce Linux raw filename output overhead #1154, measured after integrating main's Adapt ordered matching pipeline to observed output volume #1152. The exact measured baseline is
    ed0d34be2b41dc48ad975403289199f99c41a66d; candidate C++ is
    374c2588f876d054c82e017b3d3ad0b4b993dc41. Later documentation commits do not relabel binaries.

  • Store five immutable predicate guards next to each prepared callback: content invalidation,
    metadata demand (including grep-template birth time), dry-run security preview, top and shard
    status. Prepared cursors borrow the flags by const reference. Reference cursors derive the
    same flags when reached.

  • Keep reached metadata loading, invalidation before/after actions, errors, short-circuiting
    and deferred controls in their original order. No CLI or measurement-contract change.

  • --explain for the measured listing expression reports the same 328 owned prepared-record
    bytes on both hosts and full/minimal CLIs. That counter excludes AST/backends/workers.

Correctness and production build identity

  • Nine existing suites executed (cached test results disabled) and passed on each OS:
    expression_execution, expression_program, metadata, evaluate, run, walk, parallel_match,
    rg, and presentation/render record_buffer.
  • The new birth-time guard case runs across Tree/Production/Bound/Prepared/Eager/Program:
    skipped predicates load no metadata, and reached metadata failures remain unknown without
    reading content or emitting output.
  • Full and minimal CLIs use their real clang_release builds; full adds xff_full.
    A configured query and frozen copy immediately follow each build. Source/tree IDs, binary
    hashes, configuration hashes and actual ThinLTO link actions are retained.

Measurement evidence and limits

  • Current listing screens: both shapes, 100/10k/100k files and workers 1/3/10, separate
    identical-baseline controls, 27 measured rounds retaining 21. All 6,048 calls per host passed
    independently generated output oracles. Warmups and excluded rounds remain in raw evidence.
  • Current Linux three-worker repeats add 2,016 validated calls. Large full/minimal ratios are
    0.99338/0.99213, controls 1.00803/1.01376. Neither screen nor repeat has a wholly slower paired
    interval. Linux large cases still take about 7-10% longer than GNU find.
  • Current macOS small repeats add 3,936 validated calls (81 rounds retaining 63), including
    the initially slower full/single-worker/100-file case. Its slower interval did not reproduce;
    no small-case speedup is claimed. Both original and repeat results are retained.
  • Separate diagnostics: Linux 144 perf invocations and macOS 240 /usr/bin/time -l calls.
    macOS paired large-case count intervals exclude parity. Linux userspace instructions and
    macOS process resource accounting have different scopes; no absolute cross-OS comparison
    or conversion of instruction savings into elapsed speedup.
  • Earlier const-reference source c0b861add3 versus 90f8c8c057 completed all five extended
    workloads with full CLI and four native workloads with minimal CLI: 9,720 validated calls
    per host, followed by 6,720 Linux and 2,688 macOS repeat calls. These older sources are
    explicit and are not pooled with current binaries. All initial slower cells are retained.
  • Linux fixtures are verified tmpfs with recorded affinity. macOS uses owned 2 GiB
    case-sensitive HFS+ RAM images, now released with ownership receipts. These measure
    memory-backed traversal, not SSD throughput.
  • The report and compressed qualification JSON are registered for the release site. Raw
    archives on drogon include exact source snapshots, frozen binaries, raw rounds, command and
    output proofs, contracts, link actions, manifests, storage receipts and logs. Manifest
    contents and retained archive SHA-256 were checked; the macOS copy was checked at destination.
  • Chart points use independently selected fastest-k means; paired intervals use the
    all-round paired geometric ratio and are not confidence intervals for that chart point.
  • Equal-weight aggregates and cell bootstrap intervals describe these observations. They do
    not establish universal speedups or confidence across independent machines. Local candidate
    data are not appended to published viewer history under a different contract.

Integration validation

  • The adopted branch has identical production C++ and Bazel configuration files to the measured candidate 374c2588f876; differences are documentation, release-site mappings, the archive test fixture inherited from Reduce Linux raw filename output overhead #1154, and two const-only declarations in the new test. Measured binaries retain their original source identifiers.

  • At synchronized head 60977f06e3, full clang_release / xff_full qualification passes all 343 configured test targets on each host, including standalone extension/API targets and required Linux FUSE tests. These invocations reuse valid Bazel test results where applicable; 88 Mac tests executed freshly.

  • Eleven focused suites execute freshly with --nocache_test_results for ASan and TSan on both hosts and MSan on Linux. All pass. Linux MSan uses the previously documented local LLD compatibility override; this is not a source or CI change.

  • At that synchronized head all 1,376 tracked file hashes match between the Mac worktree and Linux snapshot after qualification. The current tree has identical production C++/Bazel inputs to measured candidate 374c2588f876; integration tests do not relabel the retained measured binaries.

  • Changed-file formatting/policy hooks and affected-file clang-tidy use the existing task compile database. The initial 37-TU affected-header pass finished: 36 TUs passed and the new test had two const-correctness findings. Both are fixed; the corrected test passes clang-tidy in normal commit and push hooks. No lint process was canceled. The final test executes freshly in release/ASan/TSan on both hosts and Linux MSan; all pass. Final source hashes still match for all 1,376 tracked files.

  • Both current qualification archives were rehashed on drogon and match the report digests (Linux 81865ccce102c8691065502dd17e955311470047aae2e4030dddea4c95543b9c; Mac f5a9802ec77b55de3a0a5bacb7e78127f9984e7a0a8b09432b3b78dbfda46824).

  • The subsequent borrowed archive-policy experiment is held after a small macOS content-search penalty; it is excluded here. The output-batching prototype is also excluded.

  • The synchronized merge-base diff contains only this guard change, its test/report/site-map files, and canonical TODO/performance-analysis notes; inherited Reduce Linux raw filename output overhead #1154 files are absent. The final-head Test run is 42587197181 (the preceding synchronization run was 42585233223). The earlier run failed only on GitHub release-asset HTTP 504 errors before XFF compilation. Main had the same infrastructure failures; its failed jobs were retried after the exact URLs recovered. No checks are copied from another context and no safety/benchmark policy is weakened.

  • Final local integration logs, scripts, source manifests and a Git bundle are retained on drogon as /home/marcus/Documents/xff-pr1155-local-qualification-1325f0-20261109.tar.gz (794,116 bytes; SHA-256 a2416f6cfe8b4dacf4c0013eeb8f4ae98b05836eaad57a56c136b42a8a959d3b). All 45 manifest entries were verified after transfer. The bundle names its four public-main prerequisites. This capsule supplements the original measured binary/raw-round archives; it does not claim completed CI.

Final CI comparison review

  • The official comparison job 129078478461 in run 42587197181 passed. Both platform reports pool nine samples and retain seven, with executable SHA-256 verification receipts.
  • Linux matches all 360 comparison cells against the latest completed retained main baseline, ed883ecc52 (run 42564885478). Its median normalized change is +0.599% and five cells exceed the existing 15% informational alarm. This is not reported as a CI speedup.
  • Separately, the completed Linux shard jobs for the immediate parent ad3be8a12e permit a diagnostic comparison using the unchanged compatibility/pooling/mean helpers and the same reference-ratio formula. All 360 pairs match; median normalized change is +0.141%, no cell exceeds 15%, and the five older-baseline alerts range from -7.171% to +3.558%. That diagnostic was performed while the parent workflow was still running, without relabeling it successful or injecting it into official baseline selection. Parent run 42583205104, attempt 2, has now completed successfully; repeating the unchanged baseline-selection helpers on its final reports confirms the same 360 pairs, +0.141% median and zero 15% alarms. The original official PR reports remain unchanged. Both raw populations and both comparisons are retained; no cause is assigned to the older-baseline difference.
  • macOS has no compatible retained main baseline. Its CI samples and verified executable remain useful, but no normalized macOS CI improvement is claimed; the independent native qualification supplies the cross-platform adoption evidence.
  • GitHub's benchmark checkout is synthetic merge b39fd59933ec1def8bf2e38e81cefd8dd24b05ec, whose parents are exactly main ad3be8a12e and PR head 1325f0cc2f. Its tree 3da8b428391405516bed5390b3d5278c9e7624e1 equals the locally qualified final tree. Artifact source identifiers are preserved.
  • Final-head coverage passes all category policies: 331 selected tests pass, patch lines are 25/25 (100%) and patch branches 23/26 (88.46%). Existing 95%/85% patch floors remain unchanged. Main's post-Reduce Linux raw filename output overhead #1154 full Test workflow 42583205220 has also completed successfully after retrying dependency-download failures.
  • Final Test run 42587197181, attempt 1, completed successfully on exact head 1325f0cc2fff9e6f4d902945ec18c136933dac1c. All platform suites, sanitizers, lint, fuzz, coverage, benchmark measurements/comparison and the required done/pre-commit/trunk checks pass; extra adaptive shards were correctly skipped after initial agreement. GitHub reports CLEAN/MERGEABLE, no unresolved review threads, and both the local and remote heads contain current main ad3be8a12e. No required check is transplanted, bypassed or restarted.

Base automatically changed from perf/inline-raw-records-20261109 to main November 9, 2026 08:03
@helly25
helly25 marked this pull request as ready for review November 9, 2026 08:42
@helly25
helly25 merged commit 94e785f into main Nov 9, 2026
33 checks passed
@helly25
helly25 deleted the perf/prepared-predicate-guards-current-20261109 branch November 9, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant