[helly25]

Reduce ordinary directory-walk overhead by preparing deferred evaluation and shard matching only when used, and by checking each predicate descriptor once.

The change follows measured startup/per-entry work. It includes production-build comparisons on Linux and macOS, correctness checks, controls and the results that did not improve.

AG;DR

Implementation

• Bind the required immutable predicate descriptor once per shared predicate invocation. Preserve purity/action invalidation, metadata/birth-time requirements, dry-run security, and deferred control dispatch.
• Construct per-entry deferred memo state only when the expression has deferred nodes. Initial decisions are immutable and run-scoped; retained candidates still own their memo and outputs.
• Skip shard::Matcher::Make when neither shard grouping nor -shard-status is active. Profiles showed built-in RE2 shard patterns being compiled for plain -type f -print0. Optional observers preserve ordinary collections and -top; explicit shard consumers retain eager validation and existing behavior.
• Exclude the record-terminator capture experiment: removing its sampled stack store/load did not establish an elapsed-time gain. Preserve that experiment separately.

Validation and evidence

• Measured source 1ea02a41d7dcf25c043e1079c98a9e60d6d73290; baseline merged Fix historical backfill runtime compatibility and deadlines #1149, efffe2529f26c1c7ca61763fa5fbb17efd573f6f. Subsequent commits add qualification documentation and release-site mappings, and merge main through Fix historical Linux backfills on a compatible modern runtime #1150, without changing these C++ sources.
• Seven existing correctness suites pass on both platforms: run, expression execution/program, parallel matching, record buffer, shard matcher/grouping. Logs identify cached tests.
• Clean exact-source full production builds use //xff/cli:xff_full --config=clang_release --config=xff_full. Configured executable copies, source/configuration hashes and ThinLTO CppLink actions are retained. No builds ran during accepted timing collection on the same host.
• Linux uses verified tmpfs and recorded guest CPU affinity. macOS uses an explicitly approved owned 2 GiB case-sensitive HFS+ RAM image, checked by UUID/device/live image. macOS worker counts are requests, without pinning.
• Full extended-production-v1 grid: five tasks, every declared size, broad/deep and workers 1/3/10, 180 cells per platform including identical-binary controls. 9 measured rounds/7 keep, plus warmups: 5,400 independently checked participant executions per platform. Fixture, binary, tool and launcher checks passed before/after.
• Independent 27-round/21-keep listing screens: large-listing elapsed ratios 0.985856 Linux and 0.977096 macOS; controls 0.999689 and 1.001620. These support a modest listing improvement. Userspace instructions fall roughly 9% in separate validated Linux counters across 100/10k/100k sizes.
• Full-grid ratios are descriptive, not universal speedups: Linux 0.991092/control 0.998757; macOS 0.987598/control 1.005969. Full-grid control p95 factors are 1.0393 and 1.2213; several Mac cells are noisy.
• A descriptor-only predecessor had a repeatable Linux small-task slowdown and was not accepted alone. The final 54-round small repeat is close to flat overall (1.001746; control 0.993124), with two individual slower intervals retained. This does not claim improvement in every small case.
• Separately built and measured the minimal CI production target //xff/cli:xff --config=clang_release on both hosts (24 listing cells each, 27/21 rounds/keep). Linux large-listing ratio 0.985538/control 1.010545 agrees with full-feature gains. Mac minimal large ratio 0.968880/control 1.056338 is inconclusive, with control p95 factor 1.4369; all those controls and samples remain visible. No general minimal-Mac speedup claim.
• All 23 existing extended tooling tests pass on Linux. The approved RAM disk was released after final checks, with live ownership and absence checks recorded.
• The full report and compact per-cell qualification data are in docs/benchmarks/lazy-evaluation-shard-setup-20261108.{md,json}; raw samples/build evidence/viewers are retained locally and on drogon.

Remaining gap and limits

A separate output-checked minimal-CLI syscall diagnostic found exactly the same getdents64/write counts as GNU find for both 100k listing shapes. Normal minimal-CLI samples still show roughly 4-6 ms extra userspace CPU per 100k files. That is the next profiling target; the work does not establish that XFF always beats find.

Intervals describe paired within-cell variability, without correction for multiple comparisons or independent-machine confidence. Keep all cases and controls visible. This PR does not relabel extended measurements as core CI history or pretend continuous extended-history publication is already integrated. Historical Linux runtime recovery merged separately as PR #1150.

CI follow-up

The first CI run passed the renderer tests and all production correctness/benchmark jobs, but the configured release-site build rejected the new qualification report because its page mapping was missing. Add both the Markdown page and its JSON evidence to release-site.json. All 32 release-site regression tests and the complete configured documentation build pass locally; the generated report link resolves and its JSON bytes match the retained source. Synced the branch with main (#1150). The original run finished before pushing this follow-up.

Integration with adaptive ordered output (#1152)

Synchronized with main a600f8e307feef9a7238fa253e9ce1b682a38758 in merge commit df3558fd122f9b811ee781baaf09ae8340e71c1b. The only textual conflict was release-site.json; both report pages and evidence mappings are retained. The runtime changes merged automatically and were reviewed against main: descriptor binding, lazy deferred-entry state and conditional shard matching remain intact alongside #1152's output-credit/CPU-budget ownership.

Fresh combined-tree verification is complete on both hosts:

• All 343 configured full-feature release test targets pass on each host: 88 execute freshly on macOS and 89 on Linux; the remaining unchanged targets use their existing cache results. This count includes the standalone extension/API labels, not 343 distinct source files.
• Nine focused evaluator/deferred-state/worker/output/conformance suites execute and pass under ASan and TSan on both hosts, plus MSan on Linux. The Linux MSan diagnostic uses the documented newer absolute ld.lld override; CI retains its normal runner linker.
• Commit checks and the complete affected push scope pass, including clang-tidy. The initial default-root attempt misparsed mixed libc++ headers; rerunning with the compile database's matching task-owned Bazel root passes without source changes. GitHub SSH expired while hooks ran; transport was retried after the exact commit had passed those checks. The redundant retry lint also finishes successfully.
• Verify 781 source/build/tooling file hashes on drogon, with no mismatches. Re-fetch confirms both local and remote heads contain main a600f8e. The original dirty checkout is untouched.
• Both configured production link actions contain -flto=thin. Actual lean release images are 9e3f3cc84e87fdf135378063a2f43f0e885932d894e1c68525be7e865b3e4cb5 (macOS, 4,223,056 bytes) and 633d81bfbc206a6c2ac188d6081d787c69aff844a1a9c8e6efa3769312d39953 (Linux, 12,826,408 bytes). These add 720 and 2,464 bytes respectively against the measured main-equivalent images.

The original dated populations above remain historical qualification. An additional lean-CLI confirmation now compares the actual synchronized images against #1152's measured main-equivalent runtime, with an independently copied identical-main control:

• 76 cases per host: flat 10/100/1,000/10,000-file trees, many-directory 10,000-file trees, output-heavy fixtures, workers 1/3, listing/name/metadata/content/native-match/rg filename/rg line output.
• Three counterbalanced sessions; one discarded warmup and nine measured rounds per participant/case/session, retaining seven. Every invocation checks a complete independent output oracle and zero exit status. All 6,840 invocations pass per host; raw arithmetic and post-measurement image hashes are independently verified.
• Linux descriptive candidate/main geometric ratio 0.971864, identical-control ratio 1.001113; macOS 0.993965, control 0.999197. These are this fixture/host population's results, not universal speedups or reference-normalized CI results. Linux uses tmpfs and OS-reported distinct-core affinity; macOS uses the retained temporary fixtures without affinity, not the earlier HFS+ RAM-image population.
• Preserve the initial Linux three-worker heavy-output result of +4.16% elapsed. A separate six-case, 27-round/21-retained, three-session repeat does not reproduce it: the same heavy-output case is -0.48%, with control +0.66%. All 1,512 repeat invocations pass. Do not erase the original result or claim precisely zero cost: the repeat's 100-file output cell remains +2.32% (control +1.11%).
• Preserve all macOS cells too, including the largest descriptive filename-output increases of +1.89% / +1.54% for 10,000 files at one / three workers. Other cells improve; this does not claim every workload is faster.

Retained integration archives on drogon, independently hash-checked against local copies:

• /home/marcus/Documents/xff-pr1151-integration-linux-qualified-df3558f-20261109.tar.gz, SHA256 481f429427876dcac47501a4bbf3069a3291e677e4f4970e2906f88d315f8eb9 (9,175,753 bytes).
• /home/marcus/Documents/xff-pr1151-integration-macos-qualified-df3558f-20261109.tar.gz, SHA256 799d31b055dc81765c684128d27d6daa15d934342758fb1827ceecafb2035990 (3,534,663 bytes).

These retain the raw populations, slower observations, exact image bytes/hashes, commands, source identities, verification logs and drivers; they are project-host evidence, not public GitHub artifacts. No candidate source changed during qualification. Integration CI 42557985691 completed with one fixture-marker false failure; the following test-only correction is head 205428638215e4d9a49ea19b15bb1fb80c0b04cb, with replacement run 42561479105. No previous-head check is substituted for that run. #1154 remains a dependent follow-up and is not being rewritten concurrently.

Integration CI at df3558f and advisory review

Run 42557985691 tests merge 93f8792c2a5b6b15f50946668e1fb712b99f2b3c for head df3558f. Coverage passes: changed lines 29/29 (100%), branches 23/26 (88.46%). Pre-commit, Trunk, release-site, GCC, minimal, Markdown, clang-tidy, macOS tests, Ubuntu 24.04, ASan, TSan, MSan, both expression-measurement jobs and the tool-comparison aggregator passed. Ubuntu latest failed only the fixture-marker assertion described below, with the final aggregate reflecting that failure.

The current comparison artifact has two Linux reference-adjusted fuzzy warnings against main 52950f937a: broad 500 files/one worker +16.64%, deep 2,000/one worker +17.97%. Raw XFF times improve 4.19% / 4.36%, while find+fzf improves 17.86% / 18.92%. The typical normalized change over all 360 cells is approximately flat. macOS still has no compatible retained merged CI baseline, so its CI regression status remains unknown. Do not substitute the previous-head PR comment for these artifact results.

A scoped native paired confirmation uses those exact canonical fixture identities and scenario arguments, plus listing controls. It retains current-main, an independent same-main image copy, the candidate and the older CI-baseline image: three sessions, nine rounds/seven retained, 480 output-oracle-checked invocations per host. Fuzzy cases show median within-session elapsed changes versus current main of -6.14% / -3.97% on Linux (controls +0.86% / -0.45%) and -2.24% / -0.58% on macOS (controls -0.60% / +0.65%). The local check also does not reproduce a slowdown versus the older CI-baseline image. It uses regular-file capture rather than CI pipe capture and does not invalidate the CI observations or establish their cause. Both CI artifacts, raw confirmation populations, driver and independently checked arithmetic are retained in the supplemental integration capsule.

Deterministic correction for the Ubuntu archive fixture failure

The completed integration run passed 330/331 Ubuntu-latest test targets. archive_dive_test failed because its random temporary directory was tmp.xg3GOTvZn0: the expected archive-member refusal diagnostic legitimately contained that path, but the test treated any GOT substring as proof its forbidden child ran. The archive command itself correctly refused execution.

Commit 2054286382 changes only this shell test. Rename its fixture archive to GOT.tar so the collision is deterministic; assert absence of the complete child output (GOT plus the exact member path), while keeping exit-2/refusal checks and successful container-file execution. The seeded archive reproduces the old assertion failure on macOS. The corrected full archive-diving target then executes and passes all 44 cases on both macOS and drogon. Normal commit/push hooks, shfmt and shellcheck pass. No runtime, C++ source, toolchain or benchmark input changed, so the frozen runtime measurements above retain their stated provenance.

The actual current head is 205428638215e4d9a49ea19b15bb1fb80c0b04cb. Its own replacement Test run 42561479105 is fully successful, including the formerly failing Ubuntu-latest target, all platforms/sanitizers, clang-tidy, expression measurements, benchmark comparison and the final done gate. The earlier run was already terminal before the fix was pushed. No live CI job was canceled.

Final-head benchmark result

The current run uses merged #1152 (a600f8e, main benchmark run 42553697889) for Linux: 360/360 matched comparisons, typical reference-adjusted change -1.4%, no 15% slowdown alarms. Its retained Linux executable SHA exactly matches the locally qualified production image. The previous-run warnings above remain historical observations, not current-head alarms.

macOS still reports no compatible CI baseline. The main/PR contracts differ only in native accounting-helper SHA. Direct artifact inspection shows identical loaded sections/addresses and dynamic-binding/function payloads; differing debug-map sandbox paths in the symbol table change the file hash, UUID and signature metadata. The strict raw-file hash therefore prevents this comparison. Preserve unknown until a separately tested identity correction; do not simply drop the fingerprint or relabel historical reports. Native paired macOS qualification above remains separate evidence.

[github-actions]

Benchmark preview

Advisory: changes over 15% are warnings, not merge failures. Positive reference-adjusted change means slower; negative means faster.

linux

Performance overview

WARNING: 1 comparison(s) exceed the 15% slowdown threshold. Advisory only.

Merged baseline: efffe25; run 42518987368, attempt 1; mean of fastest 7/9 runs.

Compatible comparisons: 360/360; unavailable: 0.

Population	Comparisons	Typical change	Faster	Within threshold	Slower
All matched cases	360	-1.4%	0	359	1

Largest regressions:

Task	Tree	Workers	Files	Reference	Change
fuzzy-discovery-itm	deep	1	5,000	find+fzf	+25.7%
fuzzy-discovery-itm	deep	1	2,000	find+fzf	+14.0%
fuzzy-discovery-itm	deep	3	5,000	find+fzf	+11.6%
fuzzy-discovery-itm	broad	3	1,000	find+fzf	+10.1%
fuzzy-discovery-itm	broad	1	500	find+fzf	+9.9%

Typical change is an equal-weight geometric mean of reference-adjusted XFF/main factors. Negative is faster; positive is slower. These correlated cases are not a significance test.

Interactive preview | CI run

Tested merge 2af60e325e; PR #1151 head 720fedab08.

macos

Performance overview

No compatible merged baseline comparisons are available; regression status is unknown. Advisory only.

Merged baseline: unavailable.

Compatible comparisons: 0/360; unavailable: 360.

Population	Comparisons	Typical change	Faster	Within threshold	Slower
All matched cases	0	n/a	0	0	0

Typical change is an equal-weight geometric mean of reference-adjusted XFF/main factors. Negative is faster; positive is slower. These correlated cases are not a significance test.

Interactive preview | CI run

Tested merge 2af60e325e; PR #1151 head 720fedab08.