Adapt ordered matching pipeline to observed output volume

.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
     hooks:
       - id: check-added-large-files
         # Retain the exact lossless cross-platform performance evidence archives.
-        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107|transactional-rg-records-20261107|immutable-zip-locator-evaluation-20261107|queued-read-cancellation-20261107|archive-btree-historical-decision-20261107|compiled-field-stat-demand-20261107|unchanged-site-payloads-20261107|archive-read-buffer-clear-20261107|archive-read-buffer-requalification-20261108|language-list-reserve-20261107|language-list-reserve-requalification-20261108|ordered-output-credit-20261107)\.json\.gz$
+        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107|transactional-rg-records-20261107|immutable-zip-locator-evaluation-20261107|queued-read-cancellation-20261107|archive-btree-historical-decision-20261107|compiled-field-stat-demand-20261107|unchanged-site-payloads-20261107|archive-read-buffer-clear-20261107|archive-read-buffer-requalification-20261108|language-list-reserve-20261107|language-list-reserve-requalification-20261108|ordered-output-credit-20261107|adaptive-ordered-output-20261109)\.json\.gz$
       - id: check-merge-conflict
       - id: check-yaml
       # BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with

CHANGELOG.md

@@ -3,6 +3,8 @@
 
 # 0.11.0
 
+- Adapt ordered content-output scheduling to heavy batches, apply byte backpressure, and share the matcher/coordinator CPU allowance while preserving whole-file validation and cancellation.
+
 - Avoid redundant 64 KiB buffer clearing in legacy archive extraction.
 
 - Preserve filesystem-native name matching for dangling and cross-volume symlinks when case

TODO.md

@@ -352,7 +352,8 @@ Measured follow-ups (see the P07/P08 sections of `docs/performance-analysis.md`)
 
 - [ ] Add archive read admission or serial extraction into byte-bounded matcher queues before enabling member workers.
 - [ ] Investigate ZIP member restart/indexing cost separately from compressed TAR decoding.
-- [ ] Design a byte-bounded ordered output queue and shared CPU allowance before enabling an overlapping pipeline.
+- [ ] Qualify the byte-bounded ordered output queue and shared coordinator/matcher CPU allowance
+      before enabling overlap. Preserve the existing independent directory-I/O limit and JC scope.
 
 ## Rg integration review (PRs 916-919)
 
@@ -506,12 +507,25 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
       keep the specialized pools and retain [all variants and counterexamples](docs/benchmarks/shared-stage-workers-20261107.md).
       Future reuse designs must preserve independent root/mount allowances, leaf jobs, owner
       draining and JC gates, and remove the repeatable Linux comparison regression before adoption.
+- [ ] Qualify direct span reads in `content::ScanLines` separately from O18. Native
+      profiling attributes 43.94% of sampled baseline CPU to zeroing beneath its owning
+      64 KiB read loop. Reuse `ReadInto` only with initialized-range, binary/EOF/error
+      tests and native/memory/archive controls; retain O07's rejected variants rather
+      than assuming removal of zeroing is always a win. See the current O18 qualification
+      notes in [the performance analysis](docs/performance-analysis.md).
 - [ ] O18: Qualify byte-bounded overlap with backpressure, ordered publication and cancellation;
       avoid per-batch throwaway threads and nested waits on the same saturated pool.
       Retain the [ordered-output credit experiment](docs/benchmarks/ordered-output-credit-20261107.md).
-      All runtime refinements remain unselected after repeatable ordinary Linux output costs,
-      despite large-output/RSS gains. Solve synchronization costs and missing input/slot/source/
-      renderer-temporary admission independently; native output and stage overlap remain open.
+      Adaptive native/rg output, owned-text admission, byte backpressure and a shared matcher/
+      coordinator CPU allowance pass cross-platform qualification in
+      [the adaptive-output report](docs/benchmarks/adaptive-ordered-output-20261109.md).
+      Retain ordinary-path costs and anomalous Linux populations alongside large-output gains.
+      Keep full source/decoder/derived-cache/renderer-temporary admission and broader JC gates
+      open; output credit is not a whole-process cap. No public crossover setting is exposed.
+- [ ] Benchmark qualification: distinguish repeated execution of one image from independent
+      file-backed baseline copies. Retain both controls when investigating unexplained native
+      costs, and compare loaded code/data identities before repeating full populations for a
+      build-ID-only change. Preserve all observations rather than choosing a favorable cohort.
 - [ ] O18a: Redesign queued cancellation after both measured policies fail cross-platform
       qualification. The deterministic nine-to-four backend-call proof and exact serial/recycling
       tests pass, but acquire-load and explicit-stop screens expose ordinary Mac costs. Retain

XFF.md

@@ -565,7 +565,7 @@ See also: [--help=config](#topic-config), [--help=archive](#topic-archive), [--h
 <a id="flag-jobs"></a>
 
 - `-j N, -j=N, --jobs=N|all` - directory-read, eligible content-match, comparison and -exec workers _(global, find, xff, rg)_
-  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories on up to `N` worker threads; large directory listings can split required eager metadata reads across the same pool. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches also match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
+  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories on up to `N` worker threads; large directory listings can split required eager metadata reads across the same pool. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches also match in bounded parallel batches, with output kept in traversal order. For content output, matchers and the coordinator share an `N`-participant allowance; output-heavy batches may overlap collection, matching and publication. Saved output uses byte backpressure, with the current oversized record allowed to finish; this is not a process-memory limit. Small initial batches, archive members and stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
   See also: [--help=output](#topic-output), [--help=stats](#topic-stats)
 
 <a id="flag-sort"></a>