[helly25]

Reduce ostream overhead for sustained non-interactive listings with a lazily allocated 4 KiB buffer. Short output keeps the existing path; terminal output, pagers and terminal-taking expressions keep their direct behavior.

Independent large-listing repeats improve about 0.9% on Linux and 4.4-4.5% on macOS. The measured macOS cases finish ahead of GNU find; Linux still trails it. The report preserves mixed small results and all adverse observations.

AG;DR

Implementation

• Add a noncopyable BufferedListingOutput streambuf owned by a unique_ptr in RunMain.
• Admit it when a record crosses the remaining 16 KiB direct-output budget. This is a record boundary, not an exact first-16-KiB guarantee. Smaller output allocates no byte buffer.
• Batch small writes in 4 KiB; flush then bypass storage for writes at least that large. Flush and restore the original stream buffer on destruction, preserving error state. sync() also flushes the underlying buffer so stderr's tie retains diagnostic ordering.
• Keep TTYs, pagers, quiet output, terminal-taking expressions, failed streams, exception-enabled streams and unitbuf streams on the existing path. Consume stdin before installing the wrapper.
• Preserve the required streambuf raw-pointer signature within its annotated ABI override. No new host-I/O path, thread, flag, expression vocabulary or measurement-contract change.
• Add the detailed qualification report, complete per-cell summary data, performance-analysis findings and release-site mappings. The 4 KiB bound applies only to this added byte storage.

Exact source and build provenance

Measured candidate 268bb2ef0f8b6887ac3924c94c9fb44d10f172a9 versus prepared-guard baseline 374c2588f876d054c82e017b3d3ad0b4b993dc41. This new PR is based on main after #1155 merged. The initial PR runtime and Bazel inputs matched the measured candidate, apart from #1155's constness correction for two test locals. The coverage follow-up moves the byte-identical buffer definition from main.cc into a private CLI header and adds a dedicated test target; admission and buffering behavior remain unchanged. Measurements still identify the original source and binaries; qualification of this later integration layout is recorded separately. Frozen binaries are never relabeled as this later integration commit.

Both hosts built full and minimal production CLIs with clang_release (full adds xff_full). Each configured query/copy immediately followed its own build. Retained evidence includes source/tree identities, binary SHA-256/size, build configuration and actual ThinLTO actions. Bazel servers stopped before elapsed timings.

Validation and measurement

• Fourteen existing CLI suites actually executed and passed on each host under clang_opt/xff_full with cached results disabled: color, CSV, exact, exec, exit, pager unit/shell, parity, rg full/input/terminal/basic, sort and style. These are separate from timed production binaries.
• Actual changed-translation-unit clang-tidy passed on both hosts. Mac analysis-database SDK/hermetic-libc++ fixes changed only lint inputs, with original actions, failed attempts and final receipt retained. Normal commit/push hooks are also used; the fresh integration checkout has no compile database, so its local lint hook may skip while CI remains the full gate.
• Linux: 520 cells and 35,256 output-checked elapsed/warmup participant calls; macOS: 484 cells and 30,936 calls. Full CLI covers all five extended workloads; minimal covers four native workloads. Every invocation checks independent output multiset/duplicates, status and diagnostics.
• Separate screens, 81-round listing repeats, broader workloads, targeted 27-round repeats and final 81-round native-case confirmations remain independent populations. All original slower results remain. Initial adverse signals do not persist in the final targeted confirmations; this does not prove zero regressions elsewhere.
• Large 100,000-file / three-CPU-budget broad/deep repeat candidate/baseline ratios: Linux full/minimal 0.99146/0.99122; macOS 0.95615/0.95496. Identical controls: Linux 1.00161/0.99828, macOS 1.00042/0.99629. Candidate/find: Linux 1.08992/1.09534, macOS 0.96307/0.94839.
• Separate 144-invocation Linux counter diagnostic shows about 4.4-5.3% fewer large-listing userspace instructions, with a small 100-file instruction cost. No new macOS instruction-count claim and no demonstrated first-output latency gain.
• Chart points divide independently fastest-k means. Paired intervals use all-round geometric ratios; they are not confidence intervals for those chart points. Identical controls are shown without post-hoc division. No universal or independent-machine speedup claim.

Storage, retention and limitations

Linux fixtures use verified tmpfs and recorded affinity. macOS uses five independently owned 2 GiB case-sensitive HFS+ RAM images, all UUID-matched and released. This measures memory-backed traversal rather than SSD throughput. Hosts were not used for our builds or heavy analysis during accepted timings.

Final archives in /home/marcus/Documents/ on drogon.boerger.online, manifest-verified against contents:

• xff-lazy-listing-buffer-linux-20261109.tar.gz: 27,473,416 bytes, SHA-256 1cf694372a3f0e78e3e228a6477a435f07e4a8a2cb13d210d2555c721c035066, 2,013 entries.
• xff-lazy-listing-buffer-macos-20261109.tar.gz: 16,253,106 bytes, SHA-256 8714ff015878ead60cb0c53d6e6702da4a4c9c0fa0d9a4a670cd37790d0b2069, 1,644 entries; destination hash checked after copying.

Archives retain exact source/binaries, link actions, tests, lint, raw rounds, output oracles, helpers, fixture/tool identities and storage journals. The prior immediate stack-owned buffer remains retained as a held experiment; it is not pooled with final results. Dynamic-library references are recorded, not a complete operating-system image. Continuous extended-production-v1 viewer integration and missing merged-PR publication remain separate work.

Current integration and coverage qualification

Current head: fcf178c3041aef16f52f921d580d813a46dc3739, based on main 94e785f02f6d4546484e6e224351eba814deb979.

The private buffer header now has 16 committed memory-only stream-contract cases. Tests use production's ownership form: a const unique_ptr owning a mutable stream buffer. They cover pending-byte ordering, large and exact-capacity bypass, character overflow, nonpositive writes, downstream short writes and synchronization failures, destructor restoration and tied diagnostic ordering. No coverage exclusions or threshold changes were introduced.

The performance-analysis index links to the full existing report so GitHub's Markdown rendering request fits the service limit. The actual API render and replacement release-site CI checks passed. Original findings, measured images and all raw evidence remain preserved.

The complete 344-target release configuration passed on both hosts at the preceding integration heads; all results retain their exact source identities. The ownership-corrected test additionally passed fresh release tests and actual affected-file clang-tidy on both hosts before this push, and normal commit/push hooks passed with the Mac compile database present. Current-head qualification is complete on both hosts: 344-target release configuration, fresh execution of the 16-case affected unit target, actual main.cc and test translation-unit clang-tidy, 15 fresh focused suites under ASan/TSan on both hosts and Linux MSan. All 1,380 tracked file hashes match across hosts. Full/minimal clang_release images are frozen with actual ThinLTO link actions; build servers stopped before final-layout performance checks.

Local Mac sanitizer checks add the selected compiler runtime directory as an absolute link rpath because an existing rg test copies the executable to test its find invocation. The original dyld loader failure is retained. No source, production link setting, sanitizer instrumentation or test case is changed/skipped for this diagnostic setup.

Current-head CI coverage now passes: 57/57 patch lines (100%) and 40/46 patch branches (86.96%), with original 95%/85% patch floors unchanged. Release-site, pre-commit and Trunk checks also pass. The controlled final-layout and direct-parent comparisons are complete; all current-head PR CI has completed successfully, including done.

Exact final-head native confirmation

Two additional, separate populations each contain 48 cells and 5,376 output-checked calls per host: full/minimal shipping CLIs, broad/deep trees, 100/100,000 files and 1/3/10 worker budgets; one warmup plus 27 measured rounds, retaining the fastest 21. They include repeated original-image and independent ordinary file-copy controls. Images keep their original revision, SHA and link proof; no old measurement is relabeled or pooled into a new population.

The first compares final fcf178c304 with measured layout 268bb2ef0f. Mac is near parity in the large three-worker cases; Linux shows 0.29%/0.76% full/minimal large-case costs and three small minimal cases with all-round paired intervals wholly slower. These observations remain retained, including the independent copy controls.

The second directly compares final fcf178c304 with parent merge 94e785f02f. The Linux parent executable was frozen from a verified archive snapshot of 1325f0cc2f with the identical 3da8b428 merge tree and 1,376 verified files; the original source.kind and represented_merge proof remain unchanged and are included in the collector records. It is not described as a Git checkout.

Large three-worker broad/deep geometric ratios (final/parent):

Platform	CLI	Final/parent	Repeated parent control	Independent copy/parent
Linux	full	0.99157	1.00019	0.99773
Linux	minimal	0.99114	0.99799	0.99730
macOS	full	0.95097	0.99963	0.99191
macOS	minimal	0.95505	1.00544	1.00071

Every direct-parent cell and slower result is retained: Linux has 10/24 slower point estimates (maximum +1.36%), Mac 6/24 (maximum +0.86%). No direct-parent all-round paired interval is wholly slower; this does not prove zero regressions. Fastest-round point estimates and all-round paired intervals remain different estimators. Controls are displayed without post-hoc division. Both Mac RAM images were identity-verified and released; Linux's complete borrowed tmpfs fixture was verified before/after and preserved. No build, lint or archive work ran on a host during accepted timings.

Final readiness and retention

Current-head Test run 42610302444 completed successfully, including done, every platform/sanitizer/compiler check, coverage, both expression benchmarks and the final benchmark comparison. The actual checkout was synthetic merge 5318de2fbb277b6266750ac848e0a42e31d5354a; its parents are exactly base 94e785f02f and head fcf178c304, and its aab5fe6592a00c2ee0a182f4d9213d457a4d1551 tree matches both locally qualified source trees. Local and remote heads independently contain current main; there are no unresolved review threads.

Additional local/current-head native capsules in /home/marcus/Documents/ on drogon.boerger.online, each SHA-256 and every manifest file verified at that destination:

• xff-pr1156-local-native-macos-fcf178c304-20261109.tar.gz: 39,284,988 bytes, SHA-256 165ecd76395ff14c6c8c8c7e083728f206ebd4e4a58c69ac113a6e082822744c; 481 files verified.
• xff-pr1156-local-native-linux-fcf178c304-20261109.tar.gz: 107,194,728 bytes, SHA-256 d5a09a90cbed24ee4dc0c06c923dfa883ce5c8789926c7c9ee47f8a71cd0bfad; 462 files verified.

These capsules retain all source/binary/link identities, raw native populations, ordinary independent image copies, original failures, the corrected local harnesses and source bundle. They snapshot local/native qualification and earlier partial CI observations; final CI/merge evidence is retained separately. They do not reproduce a complete operating-system image or erase earlier adverse populations.