Prepare immutable predicate guards once per expression

TODO.md

@@ -63,6 +63,10 @@ Detailed scope, data-structure audit, benchmark matrix and decision gates:
       peer-path/expected-hash template compilation, hash spec resolution and diff option/regex
       preparation. Keep entry-dependent rendering and conditional observations dynamic; measure
       these families separately before deciding on additional run-owned records.
+      The [immutable predicate-guard follow-up](docs/benchmarks/prepared-predicate-guards-20261109.md)
+      has native full/minimal qualification on both hosts, including retained slower controls.
+      It preserves conditional metadata/effect handling and does not complete the remaining
+      operand families or the broader EP03/O15 requirements.
 - [ ] EP04: Compare contiguous opcode and bound-function programs; lower audited boolean control flow,
       reuse scratch and preserve source mapping, with explicit whole-expression fallback coverage.
       The switch/function pilot, branch-selectivity cases and storage-reuse tests are implemented;

docs/benchmarks/prepared-predicate-guards-20261109.md

@@ -0,0 +1,163 @@
+# Prepared immutable predicate guards (2026-11-09)
+
+## Outcome
+
+Prepare immutable predicate guards once with each expression callback. Reached predicates
+borrow those flags instead of repeatedly reading descriptor fields and inspecting a grep
+template for birth-time demand. Metadata loading, content invalidation, dry-run execution
+previews and deferred controls still happen in the original evaluation order.
+
+Large Linux listings execute about 3% fewer userspace instructions. macOS process instruction
+counts fall by about 0.7-1.0% for the large diagnostic cases. Elapsed improvements are smaller,
+and the measurements do not establish a speedup for every workload. The Linux gap to GNU find
+remains about 7-10% for the primary large three-worker cases.
+
+The measured candidate is `374c2588f876d054c82e017b3d3ad0b4b993dc41`, compared with
+`ed0d34be2b41dc48ad975403289199f99c41a66d`. That baseline integrates #1151 and #1154 with main
+after #1152. These source identities remain fixed even when documentation is committed later.
+The [qualification data](prepared-predicate-guards-20261109.json.gz) retain every cell,
+identical-baseline control, initial slower result, build identity and archive digest.
+
+## Current production binary qualification
+
+Both hosts built the full CLI with `--config=clang_release --config=xff_full` and the minimal
+CLI with `--config=clang_release`. Each build is immediately followed by a configured executable
+query and a frozen copy, before another build can change output links. Clean source/tree IDs,
+binary SHA-256, configuration hashes and actual ThinLTO link actions are retained. Build servers
+were stopped before measurements.
+
+Each CLI's listing screen covers broad/deep trees, 100/10,000/100,000 files and workers 1/3/10:
+18 candidate cells plus 18 separate identical-baseline controls. Every cell rotates XFF,
+baseline XFF and GNU find through one warmup and 27 measured rounds, retaining the fastest 21
+per participant. All 6,048 participant invocations per host passed their independent output
+oracle. Raw samples include warmups and rounds excluded by the estimator.
+
+The following table equally weights broad/deep trees at 100,000 files and three workers.
+Negative elapsed changes mean less time. Candidate/find above one means XFF takes longer.
+
+| Platform | CLI     | Elapsed change | Identical control change | Candidate/find |
+| -------- | ------- | -------------: | -----------------------: | -------------: |
+| Linux    | full    |         -0.60% |                   +0.34% |         1.0955 |
+| Linux    | minimal |         -0.80% |                   -0.08% |         1.0737 |
+| macOS    | full    |         -0.78% |                   -0.06% |         1.0146 |
+| macOS    | minimal |         -0.70% |                   -0.16% |         0.9948 |
+
+These point estimates describe these hosts and fixtures. Equal weighting is descriptive;
+within-cell paired intervals do not establish confidence across independent machines.
+
+The chart-style elapsed ratio uses independently selected fastest-round means. Paired
+intervals instead describe the geometric ratio across all measured round pairs, including
+rounds excluded by the chart estimator. They are a separate diagnostic, not confidence
+intervals for the reported fastest-round ratio. Instruction-count intervals likewise use
+all measured pairs and no fastest-round selection.
+
+### Independent repeats, including adverse results
+
+Linux repeated all six three-worker cases for both CLIs: 100/10,000/100,000 files and both
+shapes, with a fresh 27-round population and identical controls. All 2,016 participant calls
+passed. The large full/minimal ratios were 0.99338/0.99213, with controls 1.00803/1.01376.
+The full CLI's initially adverse middle-size point estimate, 1.01631, became 1.00159 on repeat;
+its repeat control was 1.01010. No Linux candidate interval in either screen or repeat was
+wholly slower than parity. This does not prove an absence of regressions elsewhere.
+
+On macOS, the initial full CLI's broad, 100-file, single-worker case was slower: ratio 1.02502,
+paired interval [1.01994, 1.30712]. A separate repeat included both shapes and workers 1/3
+at that size, for both CLIs, using 81 measured rounds and retaining 63. All 3,936 participant
+calls passed. No repeat candidate interval remained wholly slower than parity. Its overall
+full/minimal ratios were 0.99669/0.99814 and controls 0.99414/0.99711; those near-parity results
+do not establish a small-case speedup. The initial slower cell remains in the data.
+
+## Instruction diagnostics and memory
+
+Linux's separate `perf` diagnostic uses nine repetitions per binary, case and CLI, across four
+listing cases. All 144 calls passed their output oracle, with 100% counter scheduling coverage.
+Large broad/single-worker listing uses 3.00% fewer userspace instructions with the full CLI
+and 3.04% fewer with the minimal CLI. Deep/three-worker reductions are 2.88% and 2.91%.
+
+macOS's separate `/usr/bin/time -l` diagnostic rotates candidate, baseline and identical-baseline
+control through one warmup and nine measured rounds for the same four cases and both CLIs.
+All 240 calls passed. Paired geometric process-instruction ratios at 100,000 files are:
+
+| CLI     | Shape/workers | Candidate/baseline | 95% paired interval | Identical control |
+| ------- | ------------- | -----------------: | ------------------- | ----------------: |
+| full    | broad/1       |            0.99307 | [0.99122, 0.99521]  |           1.00145 |
+| full    | deep/3        |            0.99176 | [0.98942, 0.99496]  |           0.99892 |
+| minimal | broad/1       |            0.99051 | [0.98942, 0.99146]  |           0.99941 |
+| minimal | deep/3        |            0.99149 | [0.98954, 0.99318]  |           0.99970 |
+
+The intervals use all nine measured rounds, a paired geometric ratio, 1,000 bootstrap samples
+and seed 20261107. They describe within-cell count variation. macOS process resource accounting
+is not assumed equivalent to Linux's userspace event; absolute counts are not compared across
+OSes. Neither counter population is combined with accepted elapsed timings.
+
+For the measured `-type f -print0` expression, `--explain` reports 328 owned prepared-record bytes
+for baseline and candidate, on both hosts and both CLIs. This accounting excludes the source
+AST, regex backends, adapters, allocator headers and worker storage. No broader memory-reduction
+claim follows from it.
+
+## Earlier broad qualification and rejected variants
+
+Before integration with #1152, the same const-reference policy implementation at
+`c0b861add359058f54207c501a1cfcbfac7d5479` was compared with
+`90f8c8c0579b1132ea4d0bc714da9f855a77e012`. These are earlier source identities, not aliases
+for the current binaries. Full production qualification covers all five extended workloads:
+listing, native-case filtering, metadata/date output, content regex and TAR filtering.
+The minimal CLI covers the four native workloads because it lacks TAR support.
+
+Each host completed 180 full-CLI cells and 144 minimal-CLI cells, including separate controls,
+with nine measured rounds retaining seven. All 9,720 participant calls per host passed.
+Linux's descriptive full/minimal ratios were 0.99283/0.98520, with controls 0.99734/0.99522.
+Three full-CLI cells initially had wholly slower intervals. A fresh 20-case, 27-round population
+for each CLI included every adverse case and the opposite shapes, plus representative small
+workloads. All 6,720 calls passed; no repeat interval remained wholly slower than parity.
+
+macOS's earlier full/minimal grid ratios were 0.99847/0.99789, with controls 0.99601/0.99930.
+Six full and two minimal cells initially had wholly slower intervals. Repeats included all
+eight and their opposite shapes: 24 full cells and eight minimal cells, 2,688 calls total.
+None of those original slower intervals reproduced. A different minimal broad-content case
+was 1.98% slower, while its identical-baseline control was 1.80% slower; the retained evidence
+does not isolate that result to the change. No broad macOS elapsed improvement is claimed.
+
+The initial by-value experiment at `a425603e4333d9c4fd90532de2622805a43d60dc` is also retained.
+Its listing screens and repeats motivated borrowing the stored flags by const reference.
+Its initial adverse macOS middle-size cells and later small-case adverse result remain in the
+earlier archives and qualification data. Those populations are not pooled with final results.
+
+## Correctness and storage
+
+The current candidate executed and passed nine existing suites on each host under `clang_opt`,
+with cached test results disabled: expression execution, expression program, metadata,
+evaluation, run, walk, parallel matching, ripgrep mode and raw record buffering.
+The new parameterized birth-time case checks both reached and short-circuited grep templates
+across Tree/Production/Bound/Prepared/Eager/Program execution modes. A reached metadata failure
+must remain unknown, and a skipped predicate must perform no metadata or content read.
+Existing cases cover action invalidation, dry-run security, deferred controls and errors.
+
+Linux fixtures use verified tmpfs, with recorded guest CPU affinity. macOS fixtures use owned
+2 GiB case-sensitive HFS+ RAM images without CPU affinity. Both current images were released
+after fixture, binary, tool and native-launcher integrity checks. Creation/release receipts
+record the live device/image/UUID identity. The measurements describe memory-backed traversal,
+not SSD throughput; host context and process-resource observations remain in raw evidence.
+
+Every participant invocation checks an independently generated multiset oracle, duplicate
+counts, exit status and diagnostics. Final integrity checks passed for every completed
+population. No timing sample is accepted merely because its command succeeded.
+
+## Retained evidence
+
+Durable archives are on `drogon.boerger.online` under `/home/marcus/Documents/`:
+
+- `xff-prepared-guards-linux-qualified-20261109.tar.gz`
+- `xff-prepared-guards-macos-qualified-20261109.tar.gz`
+- `xff-predicate-policy-linux-evidence-20261109.tar.gz`
+- `xff-predicate-policy-macos-evidence-20261109.tar.gz`
+- `xff-predicate-macos-native-instructions-20261109.tar.gz`
+
+The qualification JSON records exact archive SHA-256 and sizes. Current archives include both
+complete measured source trees, frozen production binaries, actual link actions, configured
+queries, correctness logs, contracts, all raw rounds, output proofs, fixture manifests,
+storage receipts and collection/analysis helpers. Their manifests were verified against the
+archive contents before retention. The macOS copy was additionally checked on the destination.
+
+This change does not alter the benchmark measurement contract or add these local experiments
+to published viewer history. Continuous extended-production-v1 history remains separate work.

docs/performance-analysis.md

@@ -5210,3 +5210,25 @@ The capsule preserves original observations, repeated-baseline and independent-i
 controls, the contaminated earlier Mac session, failed prototypes, fixture recipes
 and exact image identities. No whole-process bound, active-I/O preemption or complete
 JC/source/temporary admission is inferred. Those parts of O18 remain open.
+
+### Prepared immutable predicate guards
+
+The [predicate-guard qualification](benchmarks/prepared-predicate-guards-20261109.md)
+prepares five immutable decisions next to each expression callback: content invalidation,
+metadata demand, security previews, top selection and shard-status selection. Reached
+predicates borrow these guards; conditional metadata reads, error propagation, short-circuiting
+and action invalidation remain in source order. The production record size for the measured
+listing expression remains 328 bytes, excluding the AST, backends and worker storage.
+
+Large Linux listings execute about 3% fewer userspace instructions. The macOS process
+instruction diagnostic falls about 0.7-1.0%; the two operating systems' counters have different
+scopes and are not compared directly. Elapsed gains are smaller: the large three-worker
+full/minimal Linux repeats give candidate/baseline ratios 0.99338/0.99213, while Linux still
+takes about 7-10% longer than GNU find. Preserve the initial slower cases, independent repeats
+and identical-baseline controls rather than infer a universal speedup from instruction counts.
+
+The report identifies measured candidate `374c2588f876` and baseline `ed0d34be2b41` explicitly,
+with full/minimal production ThinLTO build identities and durable raw archives for both hosts.
+Later synchronization does not relabel those binaries. This removes one invariant predicate
+setup cost; the remaining EP03/O15 operand families, expression-program decisions and wider
+pipeline/admission work remain open.

release-site.json

@@ -158,7 +158,8 @@
     "docs/benchmarks/lazy-evaluation-shard-setup-20261108.md": "docs/benchmarks/lazy-evaluation-shard-setup-20261108.html",
     "docs/benchmarks/linux-inline-raw-records-20261109.md": "docs/benchmarks/linux-inline-raw-records-20261109.html",
     "docs/benchmark-merged-pr-coverage.md": "docs/benchmark-merged-pr-coverage.html",
-    "docs/benchmarks/adaptive-ordered-output-20261109.md": "docs/benchmarks/adaptive-ordered-output-20261109.html"
+    "docs/benchmarks/adaptive-ordered-output-20261109.md": "docs/benchmarks/adaptive-ordered-output-20261109.html",
+    "docs/benchmarks/prepared-predicate-guards-20261109.md": "docs/benchmarks/prepared-predicate-guards-20261109.html"
   },
   "generated_html": {
     "XFF.md": {
@@ -206,6 +207,7 @@
     "docs/benchmarks/ordered-output-credit-requalification-20261108.json.gz": "docs/benchmarks/ordered-output-credit-requalification-20261108.json.gz",
     "docs/benchmarks/lazy-evaluation-shard-setup-20261108.json": "docs/benchmarks/lazy-evaluation-shard-setup-20261108.json",
     "docs/benchmarks/linux-inline-raw-records-20261109.json": "docs/benchmarks/linux-inline-raw-records-20261109.json",
-    "docs/benchmarks/adaptive-ordered-output-20261109.json.gz": "docs/benchmarks/adaptive-ordered-output-20261109.json.gz"
+    "docs/benchmarks/adaptive-ordered-output-20261109.json.gz": "docs/benchmarks/adaptive-ordered-output-20261109.json.gz",
+    "docs/benchmarks/prepared-predicate-guards-20261109.json.gz": "docs/benchmarks/prepared-predicate-guards-20261109.json.gz"
   }
 }

xff/engine/evaluate.cc

@@ -3878,6 +3878,31 @@ bool IsFuzzyOnlyExpression(const parser::Expr& expr) {
   return false;
 }
 
+// These guards depend only on the immutable source expression. Prepared nodes keep them
+// beside their callback; reference cursors derive the same guards when a predicate is reached.
+struct PredicatePolicy {
+  bool invalidates_content = false;
+  bool needs_metadata = false;
+  bool preview_execution = false;
+  bool selects_top = false;
+  bool selects_shard_status = false;
+};
+
+PredicatePolicy PolicyFor(const parser::Expr& expression) {
+  if (expression.kind != parser::Expr::Kind::kPredicate) {
+    return {};
+  }
+  const auto& descriptor = *expression.descriptor;
+  return {
+      .invalidates_content = !descriptor.pure || descriptor.kind == registry::Kind::kAction,
+      .needs_metadata = descriptor.needs_metadata
+                        || (expression.grep_template != nullptr && expression.grep_template->NeedsBirthTime()),
+      .preview_execution = descriptor.safety == registry::Safety::kSecurity,
+      .selects_top = descriptor.control == registry::Control::kTop,
+      .selects_shard_status = descriptor.control == registry::Control::kShardStatus,
+  };
+}
+
 struct TreeCursor {
   std::reference_wrapper<const parser::Expr> expression;
 
@@ -3890,6 +3915,8 @@ struct TreeCursor {
   bool Predicate(EvalContext& context) const { return EvaluatePredicate(Get(), context); }
 
   bool FuzzyOnly() const { return IsFuzzyOnlyExpression(Get()); }
+
+  PredicatePolicy Policy() const { return PolicyFor(Get()); }
 };
 
 std::optional<int> MinScore(std::optional<int> lhs, std::optional<int> rhs) {
@@ -4091,29 +4118,28 @@ struct BooleanResult {
 template<typename Result = EvaluationResult, typename Cursor>
 Result EvaluatePredicateResult(Cursor cursor, EvalContext& context) {
   const auto& expr = cursor.Get();
-  const auto& descriptor = *expr.descriptor;
-  const bool invalidate = !descriptor.pure || descriptor.kind == registry::Kind::kAction;
-  if (invalidate) {
+  const auto& policy = cursor.Policy();
+  if (policy.invalidates_content) {
     context.content.Invalidate();
   }
-  if (descriptor.needs_metadata || (expr.grep_template != nullptr && expr.grep_template->NeedsBirthTime())) {
+  if (policy.needs_metadata) {
     context.control.metadata_error = context.visit.EnsureMetadata();
     if (!context.control.metadata_error.ok()) {
       return {.unknown = true};
     }
   }
-  if (context.dry_run && descriptor.safety == registry::Safety::kSecurity) {
+  if (context.dry_run && policy.preview_execution) {
     PreviewExecution(expr, context);
     return {.unknown = true};
   }
-  if (descriptor.control == registry::Control::kTop) {
+  if (policy.selects_top) {
     if constexpr (std::is_same_v<Result, EvaluationResult>) {
       return EvaluateTop(expr, context);
     } else {
       return {};  // The boolean evaluator is selected only without deferred state.
     }
   }
-  if (descriptor.control == registry::Control::kShardStatus) {
+  if (policy.selects_shard_status) {
     if constexpr (std::is_same_v<Result, EvaluationResult>) {
       return EvaluateShardStatus(expr, context);
     } else {
@@ -4124,7 +4150,7 @@ Result EvaluatePredicateResult(Cursor cursor, EvalContext& context) {
   if (!context.control.output_error.ok() || !context.control.metadata_error.ok()) {
     return {.unknown = true};
   }
-  if (invalidate) {
+  if (policy.invalidates_content) {
     // An action may read fields before mutating this same file through its sink.
     context.content.Invalidate();
   }
@@ -4222,6 +4248,8 @@ struct BoundExpression::Cursor {
   bool Predicate(EvalContext& context) const { return node.evaluate(Get(), context); }
 
   bool FuzzyOnly() const { return node.fuzzy_only; }
+
+  PredicatePolicy Policy() const { return PolicyFor(Get()); }
 };
 
 absl::StatusOr<BoundExpression> BoundExpression::Prepare(const parser::Expr& expression) {
@@ -4283,6 +4311,7 @@ struct PreparedExpression::Data {
     ExpressionSourceId lhs;
     ExpressionSourceId rhs;
     OperandId operand{0};
+    PredicatePolicy policy;
     bool fuzzy_only = false;
   };
 
@@ -4303,6 +4332,7 @@ struct PreparedExpression::Data {
     Node node{
         .expression = source.expression,
         .evaluate = binding.prepared_eval,
+        .policy = PolicyFor(expr),
     };
     if (binding.slot_kind == WorkerSlotKind::kMatcher
         || (binding.prepare != nullptr && (!expr.args.empty() || binding.prepare_without_argument))) {
@@ -4364,6 +4394,8 @@ struct PreparedExpression::Cursor {
   }
 
   bool FuzzyOnly() const { return node.fuzzy_only; }
+
+  const PredicatePolicy& Policy() const { return node.policy; }
 };
 
 PreparedExpression::PreparedExpression(std::unique_ptr<Data> data) : data_(std::move(data)) {}