diff --git a/TODO.md b/TODO.md index 7e7dcdb193..2430d94832 100644 --- a/TODO.md +++ b/TODO.md @@ -530,6 +530,14 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms file-backed baseline copies. Retain both controls when investigating unexplained native costs, and compare loaded code/data identities before repeating full populations for a build-ID-only change. Preserve all observations rather than choosing a favorable cohort. +- [ ] Qualify delayed, bounded stdout buffering independently of broader O18 admission. + PR #1156 retains full/minimal native production measurements on both hosts and all + adverse controls in [the report](docs/benchmarks/bounded-stdout-buffer-20261109.md). + Complete the stream-contract coverage follow-up and current-head verification before + merging; a 4 KiB stdout buffer does not complete source/decoder/derived-cache budgets. +- [ ] Keep the performance-analysis index below the Markdown service request limit before + appending further findings. Retain full results and sources in linked qualification + reports; consider local rendering or independently renderable sections for further growth. - [ ] O18a: Redesign queued cancellation after both measured policies fail cross-platform qualification. The deterministic nine-to-four backend-call proof and exact serial/recycling tests pass, but acquire-load and explicit-stop screens expose ordinary Mac costs. Retain diff --git a/docs/benchmarks/bounded-stdout-buffer-20261109.json.gz b/docs/benchmarks/bounded-stdout-buffer-20261109.json.gz new file mode 100644 index 0000000000..72ac3d2742 Binary files /dev/null and b/docs/benchmarks/bounded-stdout-buffer-20261109.json.gz differ diff --git a/docs/benchmarks/bounded-stdout-buffer-20261109.md b/docs/benchmarks/bounded-stdout-buffer-20261109.md new file mode 100644 index 0000000000..4495b0cff8 --- /dev/null +++ b/docs/benchmarks/bounded-stdout-buffer-20261109.md @@ -0,0 +1,210 @@ +# Delayed, bounded listing output buffer (2026-11-09) + +## Outcome + +Batch sustained, non-interactive listing output through a lazily allocated 4 KiB stream +buffer. Initial records use the existing stream, so output below the 16 KiB admission +threshold allocates no additional byte buffer. This reduces repeated ostream work without +retaining the complete output or adding another worker. + +The independent 81-round listing repeat improves the large, three-CPU-budget cases about +0.9% on Linux and 4.4-4.5% on macOS. Those macOS cases finish about 4-5% ahead of GNU find; +Linux still takes about 9% longer. Small workloads remain mixed. Neither a universal speedup +nor parity with find across all sizes is established. + +The exact measured candidate is `268bb2ef0f8b6887ac3924c94c9fb44d10f172a9`, compared with +`374c2588f876d054c82e017b3d3ad0b4b993dc41`, the prepared-predicate-guard implementation +merged in #1155. Later integration keeps those source identities and binary hashes unchanged +in the evidence. The initial integration runtime and Bazel inputs matched the measured candidate, +apart from the parent PR's two const test locals. The coverage follow-up below relocates +the unchanged buffer definition to a private header and adds independent contract tests. +The measured source and binary identities continue to refer to the original layout. + +The [qualification data](bounded-stdout-buffer-20261109.json.gz) retain all 1,004 cells, +including controls, initial adverse results, every independent repeat, build identities and +archive digests. The two hosts completed 66,192 correctness-checked elapsed/warmup participant +invocations. Another 144 Linux counter invocations are kept separate. + +## Stream behavior + +`BufferedListingOutput` wraps the existing stdout stream buffer. Its inline byte array is +uninitialized until written; the object itself is allocated only when an output record +crosses the remaining 16 KiB direct-output budget. Admission happens at record boundaries, +so this is not a guarantee that exactly the first 16 KiB use the ordinary stream. + +Small writes are combined in the 4 KiB buffer. A write at least as large as that capacity +flushes pending bytes and goes directly to the original buffer. The destructor flushes and +restores the original buffer, preserving the stream's error state across `rdbuf()`, which +otherwise clears it. Explicit flushes also flush the underlying buffer; stderr's existing +tie to stdout therefore retains ordering before diagnostics. + +TTYs, quiet output, terminal-taking expressions, pagers, failed streams, exception-enabled +streams and `unitbuf` streams keep the direct path. Stdin is consumed before installing the +wrapper. Terminal-taking expressions include process execution and interactive confirmation. +The additional byte storage is bounded at 4 KiB; this is not a whole-process memory bound. + +## Production build and measurement identities + +Both hosts built full and minimal production CLIs with `--config=clang_release`; the full +CLI additionally uses `--config=xff_full`. Each build is followed immediately by the matching +configured executable query and a frozen copy. Another configuration is built only after +that copy. Clean source/tree IDs, SHA-256, byte sizes, configuration hashes and actual ThinLTO +link actions are retained. Bazel servers were stopped before accepted elapsed measurements. + +The full CLI covers listing, native-case filtering, metadata/date output, content regex and +TAR filtering. The minimal CLI covers the four native workloads. Native broad/deep trees contain +100, 10,000 and 100,000 files; TAR fixtures contain 100, 1,000 and 10,000 files. CPU budgets +are 1, 3 and 10. A budget does not imply that this +particular expression actually starts that many matcher threads. + +Every cell rotates candidate, frozen baseline and a traditional-tool reference through a +warmup and measured rounds. Separate identical-baseline cells expose within-session +variation. Every participant invocation checks the independently generated output multiset, +duplicate counts, exit status and diagnostics. Final fixture, executable, reference-tool and +native-launcher integrity checks passed. Raw evidence retains warmups and excluded rounds. + +### Primary large listing comparison + +The following equally weights broad/deep 100,000-file listings with a three-CPU budget in +the independent 81-round repeat. Negative elapsed changes mean less time. Candidate/find +above one means XFF takes longer. + +| Platform | CLI | Elapsed change | Identical control change | Candidate/find | +| -------- | ------- | -------------: | -----------------------: | -------------: | +| Linux | full | -0.85% | +0.16% | 1.08992 | +| Linux | minimal | -0.88% | -0.17% | 1.09534 | +| macOS | full | -4.38% | +0.04% | 0.96307 | +| macOS | minimal | -4.50% | -0.37% | 0.94839 | + +These are descriptive point estimates for these hosts and fixtures. Identical-baseline +controls are shown, not divided out after measurement. Interleaving reduces shared load +effects; it does not establish a machine-independent speedup. + +### Estimators and independent populations + +Chart-style points divide independently selected fastest-round means. Paired intervals use +the geometric candidate/baseline ratio across all measured round pairs, with 1,000 bootstrap +samples and seed 20261107. Those intervals describe a different estimator and are not +confidence intervals for chart-style points. Neither establishes confidence across machines. + +| Population | Rounds / retained | Linux cells / calls | macOS cells / calls | +| ------------------------------------- | -------------------- | ------------------: | ------------------: | +| Initial listing screen, both CLIs | 27 / 21 | 72 / 6,048 | 72 / 6,048 | +| Independent listing repeat, both CLIs | 81 / 63 | 40 / 9,840 | 40 / 9,840 | +| Five full / four minimal workloads | 9 / 7 | 324 / 9,720 | 324 / 9,720 | +| Targeted adverse-workload repeat | 27 / 21 | 68 / 5,712 | 40 / 3,360 | +| Final native-case confirmation | 81 / 63 | 16 / 3,936 | 8 / 1,968 | +| Total elapsed populations | separate populations | 520 / 35,256 | 484 / 30,936 | + +Calls include warmups, candidate and identical-baseline cells, and all three participants. +Initial screens and repeats remain separate populations; no averaging across repeats is +used to erase an adverse result. + +## Adverse results and confirmations + +Linux's initial listing screen has no candidate paired interval wholly slower than parity. +The broader nine-round screen has five: full deep listing at 10,000 files / one CPU, full +deep native-case filtering at 100,000 / one CPU, minimal broad content at 10,000 / three CPUs, +minimal broad native-case filtering at 100 / ten CPUs, and minimal deep native-case filtering +at 10,000 / three CPUs. + +The fresh 27-round workload repeat includes those cases and their opposite shapes. None of +the full CLI's intervals remains wholly slower. Two minimal native-case results do: +broad 100 / ten CPUs has chart ratio 1.01118 and paired interval [1.00474, 1.03308]; deep +10,000 / three CPUs has chart ratio 1.03108 and interval [1.00351, 1.05818]. A final 81-round +confirmation covers both shapes of both cases for both CLIs. None has a wholly slower +interval. Its full/minimal chart aggregates are 0.99440/1.00150, with controls 0.99735/0.99698. +The minimal aggregate and controls still show variation around parity; the earlier slower +populations remain evidence, not failed tests to discard. + +macOS's initial full listing screen has one adverse small case: deep 100 / three CPUs, +chart ratio 1.00959 and paired interval [1.00175, 1.03360]. The independent 81-round listing +repeat has no wholly slower interval in either CLI. The broader screen then identifies full +TAR filtering at 1,000 files / one CPU in both shapes: broad ratio 1.04258, interval +[1.00394, 1.10610], and deep ratio 1.02412, interval [1.00949, 1.04462]. Minimal metadata +output at broad 100 / ten CPUs has ratio 1.01522 and interval [1.00306, 1.02783]. + +Those three macOS broader-screen signals do not reproduce in their 27-round workload +repeat. A different full native-case cell, deep 100 / three CPUs, has ratio 1.00350 and +interval [1.00022, 1.02647]. The final 81-round confirmation covers both shapes for both +CLIs; none has a wholly slower interval. Full/minimal chart aggregates are 0.99560/0.99584, +with controls 0.99989/0.99515. These observations do not prove an absence of regressions +elsewhere or a general small-case improvement. + +## Instruction and latency diagnostics + +Linux's separate `perf` diagnostic checks 144 invocations over four cases for both CLIs. +At 100,000 files, candidate/baseline userspace instruction ratios are 0.94750/0.94688 for +full/minimal broad listings and 0.95581/0.95530 for deep listings: about 4.4-5.3% fewer +instructions. The 100-file broad case instead executes about 0.18%/0.29% more instructions. +Counter samples are not pooled with accepted elapsed samples; no new macOS instruction-count +claim is made for this change. + +An audit of Linux's initial elapsed population finds large three-budget first-stdout ratios +0.99829/1.00152 for full/minimal, with controls 0.98564/0.98497. There is no established +first-output latency improvement. The raw process CPU and RSS observations remain available; +zero CPU values at small sizes are preserved rather than replaced with invented ratios. + +## Correctness, lint and storage + +Fourteen existing CLI suites actually executed on each host with `clang_opt`, `xff_full` +and cached test results disabled: color, CSV, exact output, exec, exit status, pager unit and +shell tests, parity, rg full/input/terminal/basic, sort and style. Logs and result XML are +retained. These correctness suites are separate from the production ThinLTO binaries used +for timing and per-invocation output-oracle checks. + +Changed-translation-unit clang-tidy passed on both hosts, in addition to normal policy hooks. +On macOS the scoped analysis database initially mixed SDK and hermetic libc++ paths. The +retained fix uses physical external paths, the explicit SDK sysroot and `-nostdinc++` with +the lint wrapper's hermetic libc++ includes. It changes the analysis database only; original +compile actions, failed attempts and the final successful analysis remain. Baseline/candidate +compiler bytes match and both build logs identify LLVM 23.1.5. + +Linux fixtures are on verified tmpfs with recorded CPU affinity. Each macOS phase uses a +new owned 2 GiB case-sensitive HFS+ RAM image without affinity. All five images were released +after integrity checks, matching the live image, device and UUID to ownership receipts. +These measurements exercise memory-backed traversal, not SSD throughput. The retained tools, +host/resource observations and reference-library paths describe the environment; the archives +are not complete operating-system images. + +## Retained evidence and earlier variant + +Both final archives are on `drogon.boerger.online` under `/home/marcus/Documents/`: + +- `xff-lazy-listing-buffer-linux-20261109.tar.gz`: 27,473,416 bytes, SHA-256 + `1cf694372a3f0e78e3e228a6477a435f07e4a8a2cb13d210d2555c721c035066`. +- `xff-lazy-listing-buffer-macos-20261109.tar.gz`: 16,253,106 bytes, SHA-256 + `8714ff015878ead60cb0c53d6e6702da4a4c9c0fa0d9a4a670cd37790d0b2069`. + +The Linux/macOS manifests verify 2,013/1,644 entries against archive contents. The macOS +destination digest was independently checked after copying. Archives retain measured source, +frozen production binaries, link actions, commands, contracts, raw rounds, output proofs, +fixture manifests, storage journals, correctness/lint logs and collection/analysis helpers. +The complete baseline sources are also retained in the prepared-guard evidence archives +identified by the qualification data and the preceding predicate-guard report. + +The earlier immediate, stack-owned buffer at `feca20ad0da8dc13ab5dc23be864bc36a53e1d40` +was held after mixed Linux elapsed/latency results. Its 6,048 output-checked calls and 144 +counter calls remain in `xff-buffered-listing-linux-initial-20261109.tar.gz`, SHA-256 +`158ebf0a23d9a020de6538dd6d8a65ff5f565d6ff5abbfed3ea52dcdd12203e4`. +That experiment is not relabeled as the final candidate or pooled into its results. + +This implementation does not change the measurement contract or publish local experiments +as viewer history. Continuous extended-production-v1 history and filling missing merged-PR +viewer entries remain separate work. + +## CI coverage follow-up + +The first PR run exposed missing stream-contract tests: changed-line coverage was +41/57 and changed-branch coverage was 29/46. The buffer definition now lives in a +private CLI header with the same implementation bytes and unchanged admission policy. +A dedicated memory-only test target exercises short and exact-capacity writes, +large-write bypass, character overflow, nonpositive writes, downstream short writes +and synchronization failures, destructor restoration, and tied diagnostic ordering. +The test buffers remain mutable because their owning ostream writes through the +installed streambuf. These correctness tests do not supply performance measurements. Frozen measured +binaries and their original revision identities remain unchanged. + +The performance-analysis index entry links to this complete report so its GitHub +Markdown rendering request stays within the service's size limit. No findings or +raw qualification evidence are removed. diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md index 03650ba1c3..c0fa5b8386 100644 --- a/docs/performance-analysis.md +++ b/docs/performance-analysis.md @@ -5232,3 +5232,8 @@ with full/minimal production ThinLTO build identities and durable raw archives f Later synchronization does not relabel those binaries. This removes one invariant predicate setup cost; the remaining EP03/O15 operand families, expression-program decisions and wider pipeline/admission work remain open. + +### Delayed listing output buffer + +See the [qualification report](benchmarks/bounded-stdout-buffer-20261109.md) for the +4 KiB buffer, production comparisons on both hosts, adverse results and retained evidence. diff --git a/release-site.json b/release-site.json index ef3df53f3e..871363e5b5 100644 --- a/release-site.json +++ b/release-site.json @@ -159,7 +159,8 @@ "docs/benchmarks/linux-inline-raw-records-20261109.md": "docs/benchmarks/linux-inline-raw-records-20261109.html", "docs/benchmark-merged-pr-coverage.md": "docs/benchmark-merged-pr-coverage.html", "docs/benchmarks/adaptive-ordered-output-20261109.md": "docs/benchmarks/adaptive-ordered-output-20261109.html", - "docs/benchmarks/prepared-predicate-guards-20261109.md": "docs/benchmarks/prepared-predicate-guards-20261109.html" + "docs/benchmarks/prepared-predicate-guards-20261109.md": "docs/benchmarks/prepared-predicate-guards-20261109.html", + "docs/benchmarks/bounded-stdout-buffer-20261109.md": "docs/benchmarks/bounded-stdout-buffer-20261109.html" }, "generated_html": { "XFF.md": { @@ -208,6 +209,7 @@ "docs/benchmarks/lazy-evaluation-shard-setup-20261108.json": "docs/benchmarks/lazy-evaluation-shard-setup-20261108.json", "docs/benchmarks/linux-inline-raw-records-20261109.json": "docs/benchmarks/linux-inline-raw-records-20261109.json", "docs/benchmarks/adaptive-ordered-output-20261109.json.gz": "docs/benchmarks/adaptive-ordered-output-20261109.json.gz", - "docs/benchmarks/prepared-predicate-guards-20261109.json.gz": "docs/benchmarks/prepared-predicate-guards-20261109.json.gz" + "docs/benchmarks/prepared-predicate-guards-20261109.json.gz": "docs/benchmarks/prepared-predicate-guards-20261109.json.gz", + "docs/benchmarks/bounded-stdout-buffer-20261109.json.gz": "docs/benchmarks/bounded-stdout-buffer-20261109.json.gz" } } diff --git a/xff/cli/BUILD.bazel b/xff/cli/BUILD.bazel index 62f7cdf72a..4e94d003e5 100644 --- a/xff/cli/BUILD.bazel +++ b/xff/cli/BUILD.bazel @@ -48,6 +48,7 @@ cc_library( ], visibility = ["//xff/cli/testing:__pkg__"], deps = [ + ":buffered_listing_output_cc", ":config_validation_cc", ":diagnostics_cc", ":globals_cc", @@ -87,6 +88,23 @@ cc_library( ], ) +cc_library( + name = "buffered_listing_output_cc", + hdrs = ["buffered_listing_output.h"], +) + +cc_test( + name = "buffered_listing_output_test", + size = "small", + srcs = ["buffered_listing_output_test.cc"], + deps = [ + ":buffered_listing_output_cc", + "@googletest//:gtest", + "@googletest//:gtest_main", + "@mboworks_mbo//mbo/testing:matchers_cc", + ], +) + cc_library( name = "config_validation_cc", srcs = ["config_validation.cc"], diff --git a/xff/cli/buffered_listing_output.h b/xff/cli/buffered_listing_output.h new file mode 100644 index 0000000000..7f5771e2f2 --- /dev/null +++ b/xff/cli/buffered_listing_output.h @@ -0,0 +1,98 @@ +// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors +// SPDX-License-Identifier: Apache-2.0 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#ifndef XFF_CLI_BUFFERED_LISTING_OUTPUT_H_ +#define XFF_CLI_BUFFERED_LISTING_OUTPUT_H_ + +#include +#include +#include +#include +#include + +namespace xff::cli { + +// Batch non-interactive listing records before the existing stdout buffer. In particular, +// retain ostream flush semantics: stderr's tie flushes these bytes before diagnostics. +// Terminal-taking expressions and pagers keep their direct output behavior. +class BufferedListingOutput final : public std::streambuf { + public: + static constexpr std::size_t kCapacity = 4'096; + + explicit BufferedListingOutput(std::ostream& output) : output_(output), original_(*output.rdbuf()) { + setp(buffer_.begin(), buffer_.end()); + output_.rdbuf(this); + } + + ~BufferedListingOutput() override { + // Admission requires the ordinary non-throwing stream policy. Restore any failed + // state after rdbuf(), which otherwise clears it while restoring the original buffer. + output_.flush(); + const auto state = output_.rdstate(); + output_.rdbuf(&original_); + output_.clear(state); + } + + BufferedListingOutput(const BufferedListingOutput&) = delete; + BufferedListingOutput& operator=(const BufferedListingOutput&) = delete; + BufferedListingOutput(BufferedListingOutput&&) = delete; + BufferedListingOutput& operator=(BufferedListingOutput&&) = delete; + + protected: + int sync() override { return FlushBuffer() ? original_.pubsync() : -1; } + + int_type overflow(int_type value) override { + if (!FlushBuffer()) { + return traits_type::eof(); + } + if (!traits_type::eq_int_type(value, traits_type::eof())) { + *pptr() = traits_type::to_char_type(value); + pbump(1); + } + return traits_type::not_eof(value); + } + + // XFF_ABI_POINTER: std::streambuf requires this borrowed character-buffer override. + std::streamsize xsputn(const char* data, std::streamsize count) override { + if (count <= 0) { + return 0; + } + if (std::cmp_greater_equal(count, buffer_.size())) { + return FlushBuffer() ? original_.sputn(data, count) : 0; + } + if (count > epptr() - pptr() && !FlushBuffer()) { + return 0; + } + traits_type::copy(pptr(), data, static_cast(count)); + pbump(static_cast(count)); + return count; + } + + private: + bool FlushBuffer() { + const auto count = static_cast(pptr() - pbase()); + setp(buffer_.begin(), buffer_.end()); + return count == 0 || original_.sputn(buffer_.data(), count) == count; + } + + std::ostream& output_; + std::streambuf& original_; + // Default initialization intentionally leaves unused byte storage untouched. + std::array buffer_; +}; + +} // namespace xff::cli + +#endif // XFF_CLI_BUFFERED_LISTING_OUTPUT_H_ diff --git a/xff/cli/buffered_listing_output_test.cc b/xff/cli/buffered_listing_output_test.cc new file mode 100644 index 0000000000..c8c7ccf685 --- /dev/null +++ b/xff/cli/buffered_listing_output_test.cc @@ -0,0 +1,240 @@ +// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors +// SPDX-License-Identifier: Apache-2.0 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include "xff/cli/buffered_listing_output.h" + +#include +#include +#include +#include +#include +#include + +#include "gmock/gmock.h" +#include "gtest/gtest.h" +#include "mbo/testing/matchers.h" + +namespace xff::cli { +namespace { + +using ::mbo::testing::EqualsText; +using ::testing::Eq; +using ::testing::IsEmpty; +using ::testing::IsFalse; +using ::testing::IsTrue; + +// An in-memory downstream buffer with observable writes and injectable short writes. +// No host handle or filesystem participates in these stream contract tests. +struct RecordingBuffer final : std::streambuf { + std::string bytes; + std::streamsize write_limit = std::numeric_limits::max(); + int writes = 0; + int syncs = 0; + int sync_result = 0; + + protected: + // XFF_ABI_POINTER: std::streambuf requires this borrowed character-buffer override. + std::streamsize xsputn(const char* data, std::streamsize count) override { + ++writes; + const auto accepted = std::min(count, write_limit); + bytes.append(data, static_cast(accepted)); + return accepted; + } + + int sync() override { + ++syncs; + return sync_result; + } +}; + +struct BufferedListingOutputTest : ::testing::Test { + RecordingBuffer sink; + std::ostream output{&sink}; +}; + +TEST_F(BufferedListingOutputTest, SmallWritesAreHeldUntilExplicitFlush) { + const auto buffer = std::make_unique(output); + output << "first" << "second"; + EXPECT_THAT(sink.bytes, IsEmpty()); + output.flush(); + EXPECT_THAT(sink.bytes, EqualsText("firstsecond")); + EXPECT_THAT(sink.writes, Eq(1)); + EXPECT_THAT(sink.syncs, Eq(1)); + EXPECT_THAT(output.good(), IsTrue()); +} + +TEST_F(BufferedListingOutputTest, EmptyFlushDoesNotWriteButSynchronizesDownstream) { + const auto buffer = std::make_unique(output); + output.flush(); + EXPECT_THAT(sink.bytes, IsEmpty()); + EXPECT_THAT(sink.writes, Eq(0)); + EXPECT_THAT(sink.syncs, Eq(1)); +} + +TEST_F(BufferedListingOutputTest, DestructionFlushesPendingBytesAndRestoresOriginalBuffer) { + { + const auto buffer = std::make_unique(output); + output << "pending"; + EXPECT_THAT(sink.bytes, IsEmpty()); + } + EXPECT_THAT(sink.bytes, EqualsText("pending")); + EXPECT_THAT(output.rdbuf(), Eq(&sink)); + EXPECT_THAT(output.good(), IsTrue()); + output << " restored"; + EXPECT_THAT(sink.bytes, EqualsText("pending restored")); +} + +TEST_F(BufferedListingOutputTest, LargeWriteFlushesPendingBytesThenBypassesStorage) { + const std::string large(BufferedListingOutput::kCapacity + 1, 'x'); + const auto buffer = std::make_unique(output); + output << "prefix" << large; + EXPECT_THAT(sink.bytes, EqualsText("prefix" + large)); + EXPECT_THAT(sink.writes, Eq(2)); + EXPECT_THAT(output.good(), IsTrue()); +} + +TEST_F(BufferedListingOutputTest, ExactCapacityWriteAlsoBypassesStorage) { + const std::string large(BufferedListingOutput::kCapacity, 'x'); + const auto buffer = std::make_unique(output); + output << large; + EXPECT_THAT(sink.bytes, EqualsText(large)); + EXPECT_THAT(sink.writes, Eq(1)); +} + +TEST_F(BufferedListingOutputTest, SmallWriteCrossingCapacityFlushesWithoutLosingTail) { + const std::string prefix(BufferedListingOutput::kCapacity - 3, 'a'); + { + const auto buffer = std::make_unique(output); + output << prefix << "tail"; + EXPECT_THAT(sink.bytes, EqualsText(prefix)); + EXPECT_THAT(sink.writes, Eq(1)); + } + EXPECT_THAT(sink.bytes, EqualsText(prefix + "tail")); + EXPECT_THAT(output.good(), IsTrue()); +} + +TEST_F(BufferedListingOutputTest, CharacterOverflowFlushesFullStorageAndRetainsNewCharacter) { + const std::string prefix(BufferedListingOutput::kCapacity - 1, 'a'); + { + const auto buffer = std::make_unique(output); + output << prefix; + output.put('b'); + EXPECT_THAT(sink.bytes, IsEmpty()); + output.put('c'); + EXPECT_THAT(sink.bytes, EqualsText(prefix + "b")); + EXPECT_THAT(output.good(), IsTrue()); + } + EXPECT_THAT(sink.bytes, EqualsText(prefix + "bc")); +} + +TEST_F(BufferedListingOutputTest, NonpositiveWritesDoNotTouchDownstreamOrPendingBytes) { + const auto buffer = std::make_unique(output); + output << "pending"; + EXPECT_THAT(buffer->sputn("", 0), Eq(0)); + EXPECT_THAT(buffer->sputn("", -1), Eq(0)); + EXPECT_THAT(sink.writes, Eq(0)); + output.flush(); + EXPECT_THAT(sink.bytes, EqualsText("pending")); +} + +TEST_F(BufferedListingOutputTest, FailedFlushPreservesBadStateWhenRestoringOriginalBuffer) { + sink.write_limit = 2; + { + const auto buffer = std::make_unique(output); + output << "pending"; + output.flush(); + EXPECT_THAT(output.bad(), IsTrue()); + } + EXPECT_THAT(output.rdbuf(), Eq(&sink)); + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, EqualsText("pe")); + EXPECT_THAT(sink.writes, Eq(1)); + EXPECT_THAT(sink.syncs, Eq(0)); +} + +TEST_F(BufferedListingOutputTest, DestructionDetectsFailedFlush) { + sink.write_limit = 0; + { + const auto buffer = std::make_unique(output); + output << "pending"; + } + EXPECT_THAT(output.rdbuf(), Eq(&sink)); + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, IsEmpty()); +} + +TEST_F(BufferedListingOutputTest, FailedFlushPreventsLargeBypassWrite) { + const std::string large(BufferedListingOutput::kCapacity, 'x'); + sink.write_limit = 0; + const auto buffer = std::make_unique(output); + output << "pending" << large; + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, IsEmpty()); + EXPECT_THAT(sink.writes, Eq(1)); +} + +TEST_F(BufferedListingOutputTest, ShortLargeBypassWriteSetsBadState) { + const std::string large(BufferedListingOutput::kCapacity, 'x'); + sink.write_limit = 2; + const auto buffer = std::make_unique(output); + output << large; + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, EqualsText("xx")); +} + +TEST_F(BufferedListingOutputTest, FailedFlushPreventsSmallCrossingWrite) { + const std::string prefix(BufferedListingOutput::kCapacity - 1, 'a'); + sink.write_limit = 0; + const auto buffer = std::make_unique(output); + output << prefix << "tail"; + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, IsEmpty()); + EXPECT_THAT(sink.writes, Eq(1)); +} + +TEST_F(BufferedListingOutputTest, FailedCharacterOverflowReturnsFailureToOstream) { + const std::string prefix(BufferedListingOutput::kCapacity - 1, 'a'); + sink.write_limit = 0; + const auto buffer = std::make_unique(output); + output << prefix; + output.put('b'); + output.put('c'); + EXPECT_THAT(output.bad(), IsTrue()); + EXPECT_THAT(sink.bytes, IsEmpty()); + EXPECT_THAT(sink.writes, Eq(1)); +} + +TEST_F(BufferedListingOutputTest, DownstreamSynchronizationFailureSetsBadState) { + sink.sync_result = -1; + const auto buffer = std::make_unique(output); + output << "pending"; + output.flush(); + EXPECT_THAT(sink.bytes, EqualsText("pending")); + EXPECT_THAT(sink.syncs, Eq(1)); + EXPECT_THAT(output.bad(), IsTrue()); +} + +TEST_F(BufferedListingOutputTest, TiedDiagnosticStreamFlushesListingFirst) { + const auto buffer = std::make_unique(output); + std::ostream diagnostics(&sink); + diagnostics.tie(&output); + output << "listing"; + diagnostics << "diagnostic"; + EXPECT_THAT(sink.bytes, EqualsText("listingdiagnostic")); + EXPECT_THAT(output.bad(), IsFalse()); +} + +} // namespace +} // namespace xff::cli diff --git a/xff/cli/main.cc b/xff/cli/main.cc index d47bdca72d..8980e44913 100644 --- a/xff/cli/main.cc +++ b/xff/cli/main.cc @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -36,6 +37,7 @@ #include "absl/strings/str_join.h" #include "absl/types/span.h" #include "mbo/status/status_macros.h" +#include "xff/cli/buffered_listing_output.h" #include "xff/cli/config_validation.h" #include "xff/cli/diagnostics.h" #include "xff/cli/globals.h" @@ -957,8 +959,9 @@ int RunMain(std::string_view program, const std::vector& args, xff: std::cerr << "xff: " << listing_config.status().message() << "\n"; return 2; } - const xff::cli::PagerDecision listing_pager = xff::cli::DecidePager( - *listing_config, xff::cli::PagerOutput::kListing, stdout_is_tty, quiet || xff::parser::TakesTerminal(command)); + const bool takes_terminal = xff::parser::TakesTerminal(command); + const xff::cli::PagerDecision listing_pager = + xff::cli::DecidePager(*listing_config, xff::cli::PagerOutput::kListing, stdout_is_tty, quiet || takes_terminal); const xff::cli::PagerStream pager_stream(listing_pager); const xff::vfs::LocalFs host_fs; std::optional input; @@ -979,10 +982,25 @@ int RunMain(std::string_view program, const std::vector& args, xff: } const xff::cli::RgInputFs input_fs(host_fs, std::move(input)); const xff::vfs::FileSystem& fs = command.rg ? static_cast(input_fs) : host_fs; + // Keep initial output on the ordinary stream, and construct byte storage only + // after sustained output. Short runs allocate no buffer and keep a small stack frame. + std::unique_ptr buffered_listing; + std::size_t direct_output_budget = 4 * xff::cli::BufferedListingOutput::kCapacity; + const bool can_buffer = !quiet && !stdout_is_tty && !takes_terminal + && listing_pager.action == xff::cli::PagerAction::kDirect && std::cout.good() + && std::cout.exceptions() == std::ios_base::goodbit + && (std::cout.flags() & std::ios_base::unitbuf) == 0; const xff::engine::RunResult result = xff::engine::RunFind( command, fs, - [quiet](std::string_view record) { + [quiet, can_buffer, &direct_output_budget, &buffered_listing](std::string_view record) { if (!quiet) { // --quiet suppresses output; the match is still recorded via `matched` + if (can_buffer && !buffered_listing && std::cout.good()) { + if (record.size() >= direct_output_budget) { + buffered_listing = std::make_unique(std::cout); + } else { + direct_output_budget -= record.size(); + } + } std::cout.write(record.data(), static_cast(record.size())); } },