Plan content workers from remaining batch cost

XFF.md

@@ -564,7 +564,7 @@ See also: [Configuration](#topic-config), [Archives](#topic-archive), [Output](#
 <a id="flag-jobs"></a>
 
 - `-j N, -j=N, --jobs=N|all` - directory-read, eligible content-match, comparison and -exec workers _(global, find, xff, rg)_
-  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories using one command-owned, lazily started pool of up to `N-1` background workers plus the calling coordinator, reused for directory reads, required eager metadata, eligible content matching and file comparison. Comparison trees are coordinated sequentially within that same budget. The caller participates in directory reads, eligible content and comparison work. It reads an uncached current directory while workers read later siblings, and can claim a required queued read once. Directory read-ahead uses a bounded ordered window and a shared retained-storage allowance; oversized speculative listings are read on demand instead. Unneeded queued reads are canceled after pruning or stopping, but an in-flight filesystem call must finish. Workers reuse the same executor for independent work; small workloads stay on the coordinator. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
+  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories using one command-owned, lazily started pool of up to `N-1` background workers plus the calling coordinator, reused for directory reads, required eager metadata, eligible content matching and file comparison. Comparison trees are coordinated sequentially within that same budget. The caller participates in directory reads, eligible content and comparison work. It reads an uncached current directory while workers read later siblings, and can claim a required queued read once. Directory read-ahead uses a bounded ordered window and a shared retained-storage allowance; oversized speculative listings are read on demand instead. Unneeded queued reads are canceled after pruning or stopping, but an in-flight filesystem call must finish. Workers reuse the same executor for independent work; small workloads stay on the coordinator. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches match in bounded parallel batches, with output kept in traversal order. A short timed sample estimates the remaining batch's work; admission accounts for cold startup, warm dispatch and pool growth. Completed batches do not justify starting workers for a small tail. Small batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
   See also: [Output](#topic-output), [Statistics](#topic-stats)
 
 <a id="flag-sort"></a>

docs/benchmark-correctness-audit.md

@@ -388,13 +388,45 @@ Python worker's high floor. The forked launcher's inherited high-water mark must
 be corrected or explicitly excluded before using those reported peaks to choose
 arena sizes. This is separate from the worker scheduler experiment.
 
+### Remaining matcher-work experiment
+
+A regression against the caller-participation candidate processes 8,192 entries
+in sixteen-entry batches, then a 32-entry tail. The unchanged matcher starts one
+worker for that tail because already-completed work bypasses its cold probe.
+The regression fails before the change and passes after it; completed work cannot
+repay the new worker's cost on that tail.
+
+The separate follow-up removes the completed-entry counter. Four timed chunks
+evaluate up to sixteen entries exactly once; the middle two observations estimate
+only the remaining owned batch. A pure cost planner includes cold startup,
+dispatch, pool growth and minimum saving, retains caller-inclusive budgeting,
+and limits participants by both cost and independent remaining entries. Warm
+reuse is not automatically beneficial. Adaptive atomic-claim grain aims to
+amortize scheduling while retaining several balancing waves. Workers remain
+leaves, ordered results and private evaluator state remain unchanged, and tiny
+batches never require a timing observation or background dispatch.
+
+The allowances are experimental, not portable measured constants: 3.5 ms cold,
+50 us per added worker/dispatch, 100 us saving and a 32 us chunk target. The
+default Linux/x86 executor diagnostic in PR #1000 informs the cold cliff and
+the need to distinguish cold/warm/grain; it is not a file-cost calibration or
+native M5 Pro validation. Controlled slow evaluator tests exercise actual caller
+participation, exact-once results and worker reuse without touching a filesystem;
+they are not physical slow-storage acceptance. A trimmed small prefix still
+cannot prove uniform costs in an arbitrary batch. Directory/stat policies,
+candidate allocation profiles, repeated/reordered actual-workload timing and
+native cross-platform controls remain required. No performance acceptance is
+claimed by this implementation or its correctness tests.
+
 The complete audit remains open. Final shared-executor timing must be collected
 after every build has finished, using the complete prepared grid. Repeated
 controls must cover content, eager metadata and slow storage. Separate evidence
 in [PR #995](https://github.com/mboworks/xff/pull/995) now retains the exact-source
 Linux baseline, enumeration-only allocation profiles and coarse counters/stacks
-for 20k/50k/100k single-worker workloads. Those baseline diagnostics do not complete
-precise phase-level attribution or allocation controls for the other workloads,
-platforms or new scheduler candidates. Hosted macOS checks provide portability evidence for PR #991;
+for 20k/50k/100k single-worker workloads. Its later allocation controls also cover
+tree output, collection, lazy/eager metadata and content matching for both layouts
+at those sizes, with separate native RSS and exact-output checks. Those baseline
+diagnostics do not complete precise phase-level attribution, multiworker/candidate
+allocation controls or native cross-platform acceptance. Hosted macOS checks provide portability evidence for PR #991;
 they do not replace the specified native M5 Pro A/B and sampling controls for
 the shared-executor experiment.

docs/design-parallel.md

@@ -56,10 +56,10 @@ background workers under `--jobs=N`:
 - The coordinator applies traversal controls and writes the output sink in traversal order.
   Independent content predicates and rg content searches reuse that executor, with the caller also
   claiming chunks; all stateful expressions, mutations and execution actions retain the coordinator evaluator.
-- Content matching schedules chunks of four owned entries, so clustered expensive files can use
+- Content matching schedules adaptive chunks of owned entries, so clustered expensive files can use
   several workers. Decision-only batches hold at most 8,192 entries; matching-line output retains
   the smaller 256-entry bound. Cold admission is described under Parallelism control; tails below
-  16 run inline even after workers exist. Later larger batches can grow the executor up to the background-worker
+  six run inline even after workers exist. Later sufficiently costly batches can grow the executor up to the background-worker
   allowance. Completed results return in input order, including a trailing path-only print action.
   Cheap name/type matching stays inline to avoid scheduling overhead.
 - Eligibility comes from audited descriptor capabilities, not names: only independent tests
@@ -212,12 +212,51 @@ disappears, leaving the executor reusable by the next phase.
 Content matching retains private evaluator and regex state for each logical
 worker. Decision-only matching collects up to 8,192 entries before flushing;
 rendered-content batches remain bounded at 256 entries because their output can
-be large. A cold executor keeps cheap small batches on the coordinator until
-8,192 entries have accumulated. A timed 16-entry prefix admits slower work
-earlier when that prefix takes at least 500 microseconds. Once traversal or
-matching has started the executor, batches of at least 16 entries may reuse it.
-Each submitted matcher job drains entry chunks, and results are consumed in
-original entry order.
+be large. Matcher admission now estimates only the owned batch's remaining
+work; completed batches do not justify starting workers for a small tail.
+Batches below six entries stay on the caller. Larger batches evaluate up to sixteen
+entries as four timed chunks of one to four entries, retain those results once, and use
+the middle two elapsed observations for a remaining-work estimate. Discarding
+the fastest/slowest chunk reduces sensitivity to one already-paid cold/outlier
+entry; it does not prove that the rest of a heterogeneous batch has equal cost.
+
+The pure planner compares remaining serial cost against predicted caller-inclusive
+parallel cost. It accounts separately for cold startup, waking the executor,
+new worker creation and a minimum saving allowance. Already-started workers are
+reused, but cheap warm batches may still remain serial. Worker growth must improve
+the estimate; the full requested budget is not automatically activated. The count
+of independent remaining items limits participants, but even a few expensive
+items may repay dispatch; there is no minimum of sixteen items per worker. A convex
+cost model permits checking only boundaries and the two integers around its
+continuous minimum, avoiding a linear search over a large requested worker count.
+
+The experimental default allowances are 3.5 ms cold startup, 50 us per added
+background worker, 50 us dispatch and 100 us minimum predicted saving. These are
+planning allowances, not portable measurements or worst-case guarantees. The cold
+allowance includes the separately observed default Linux/x86 Abseil calibration
+cliff; native M5 Pro and actual matcher setup/dispatch need independent validation.
+No runtime flag selects these testable internal policies, and neither synthetic
+executor controls nor controlled slow evaluators constitute storage acceptance.
+
+After admission, each background worker gets one drain and the caller also drains.
+Atomic-claim grain targets 32 us of estimated useful work, capped at 128 entries
+and, where possible, at a quarter of each participant's average share to retain balancing waves.
+Slow entries reduce the grain toward one. Workers never recursively submit or
+wait on their own executor; results are still consumed in original entry order.
+These drains are bounded by the matcher batch, not an entire tree. They do not
+make queue fairness preemptive or interrupt a running filesystem call. Directory
+and eager-stat admission remain separate inherited policies, requiring their own
+remaining-cost, metadata and slow-storage controls before acceptance.
+
+This candidate forecasts only the owned matcher batch. It does not establish
+eligible work in an unvisited frontier or credit future batches merely because
+earlier ones were expensive. Cheap bounded batches can therefore stay serial
+even when a larger command has more work ahead. Command-wide known-frontier cost
+planning and heterogeneous/cached workload controls remain part of the audit;
+this matcher experiment is not a completed adaptive command scheduler.
+Private matcher/evaluator setup and contention with other phases are not
+individually calibrated by this cost model; a warm shared executor does not imply
+that a newly encountered matcher's private state is already prepared.
 
 Under `-j > 1` the serial `-exec ... ;` / `-execdir ... ;` form launches its child
 on a bounded runner (at most `N` outstanding) instead of running it synchronously.

xff/cli/globals.cc

@@ -915,7 +915,9 @@ constexpr std::array kGlobals = std::to_array<GlobalFlag>({
             "filesystem call must finish. Workers reuse the same executor for independent work; small workloads "
             "stay on the coordinator. Lazy and metadata-free listings do not start "
             "stat workers. Independent content predicates and rg searches match in bounded parallel batches, "
-            "with output kept in traversal order. Small initial batches, archive members, stateful expressions "
+            "with output kept in traversal order. A short timed sample estimates the remaining batch's work; "
+            "admission accounts for cold startup, warm dispatch and pool growth. Completed batches do not "
+            "justify starting workers for a small tail. Small batches, archive members, stateful expressions "
             "stay on the coordinator. Metadata-consuming output does not itself prevent eligible content "
             "filters from running in workers; reductions and rendering remain serial. Independent "
             "regular-file comparison pairs can also use bounded workers, preserving relative-path output "

xff/engine/BUILD.bazel

@@ -272,6 +272,22 @@ cc_fuzz_test(
     ],
 )
 
+cc_library(
+    name = "match_work_cc",
+    srcs = ["match_work.cc"],
+    hdrs = ["match_work.h"],
+)
+
+cc_test(
+    name = "match_work_test",
+    size = "small",
+    srcs = ["match_work_test.cc"],
+    deps = [
+        ":match_work_cc",
+        "@googletest//:gtest_main",
+    ],
+)
+
 # Drives one find run: walk the roots, evaluate the expression, (implicit) print.
 cc_library(
     name = "parallel_match_cc",
@@ -281,6 +297,7 @@ cc_library(
         ":collect_cc",
         ":evaluate_cc",
         ":expression_execution_cc",
+        ":match_work_cc",
         ":run_executor_cc",
         "//xff/parser:parser_cc",
         "//xff/registry:registry_cc",
@@ -299,10 +316,14 @@ cc_test(
     deps = [
         ":evaluate_cc",
         ":expression_execution_cc",
+        ":match_work_cc",
         ":parallel_match_cc",
         "//xff/matching/regex:regex_cc",
         "//xff/parser:parser_cc",
         "//xff/vfs:local_fs_cc",
+        "@abseil-cpp//absl/base:core_headers",
+        "@abseil-cpp//absl/synchronization",
+        "@abseil-cpp//absl/time",
         "@googletest//:gtest_main",
         "@mboworks_mbo//mbo/testing:status_cc",
     ],
@@ -600,7 +621,10 @@ cc_test(
         "//xff/parser:parser_cc",
         "//xff/vfs:local_fs_cc",
         "//xff/vfs:vfs_cc",
+        "@abseil-cpp//absl/base:core_headers",
         "@abseil-cpp//absl/status",
+        "@abseil-cpp//absl/synchronization",
+        "@abseil-cpp//absl/time",
         "@googletest//:gtest",
         "@googletest//:gtest_main",
         "@mboworks_mbo//mbo/testing:matchers_cc",