Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ repos:
rev: v5.0.0
hooks:
- id: check-added-large-files
# Retain the exact lossless cross-platform audit evidence archive.
exclude: ^docs/benchmarks/executor-copy-audit-20261101\.json\.gz$
- id: check-merge-conflict
- id: check-yaml
# BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

# 0.11.0

- Avoid temporary collection-name and comparison-category strings while retaining owned keys.

# 0.10.0

- Verify staged release executables generate Git-applicable comparison patches for binary and empty-file additions, removals, and updates, including reverse application.
Expand Down
6 changes: 6 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,12 @@ Measured follow-ups (see the P07/P08 sections of `docs/performance-analysis.md`)

## Performance optimization follow-up

- [ ] Qualify executor admission after the clock correction with separate cold/warm controls and
16/32/63/64-entry matching boundaries; check Linux and macOS workload regressions before
promoting shared executors, caller-inclusive budgets or bounded directory frontiers.
- [ ] Revisit recycling matcher batch buffers only when repeatable gains justify the additional API;
retain ownership and thread-safety tests plus benchmarks with identical bytes as negative controls.

- [ ] Benchmark collection arena block policies: 4 KiB -> 64 KiB (current), 64 KiB -> 1 MiB,
and 4 KiB -> 1 MiB. Compare allocation counts, CPU time, unused capacity and peak memory
for small/large collections on Linux and macOS before choosing new defaults.
Expand Down
Binary file not shown.
303 changes: 303 additions & 0 deletions docs/benchmarks/executor-copy-audit-20261101.md

Large diffs are not rendered by default.

45 changes: 45 additions & 0 deletions docs/performance-analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -2731,3 +2731,48 @@ results; all five macOS targets execute. Full CI remains required before merging
The macOS samples, binary hashes, invocation settings, output oracles and probe
source are retained in
[`abseil-steady-clock-macos-20261031.json.gz`](benchmarks/abseil-steady-clock-macos-20261031.json.gz).

## Executor and allocation audit after the clock correction (2026-11-01)

Keep main's runtime scheduling. The [complete executor/copy audit](benchmarks/executor-copy-audit-20261101.md)
compares all seven runtime/measurement PRs, their actual dependencies, manageable combinations and
isolated mechanism controls on macOS and Linux. The initial native matrix has 53 cases and three
sessions on each platform, with independent output oracles; supplemental native and in-memory
measurements have separate contracts. The [lossless evidence bundle](benchmarks/executor-copy-audit-20261101.json.gz)
retains all ten report populations, complete source patches, drivers, identities and provenance corrections.

Reject #991's current four-millisecond directory-admission policy and #1001's current remaining-work
cost model. A separate matching guard inherited by #992/#996 accumulates 8,192 entries and samples
16 entries against a 500-microsecond threshold. This guard survives excluding #991's directory policy.
Removing it restores near-baseline clustered-content and line-output performance on both platforms;
it does not establish that executor reuse is a universal improvement or restore main's cold-batch policy.

Command-wide execution and bounded read-ahead have useful Linux listing/metadata/comparison results,
but retain material workload and platform regressions. At three workers, the 65,536-file long-name
collection case regresses by 39.6% for #996 and 36.3% for #998 on macOS. The #992-only eager control
also has rg-selection counterexamples. Keep these runtime integrations conditional rather than
promoting a replacement policy from the favorable cells alone.

Bounded read-ahead has a substantial measured memory benefit on both platforms. In the
65,536-file metadata case at three workers, median whole-process peak RSS is:

| Platform | Main | #996 | #997 | #998 |
| :------- | --------: | --------: | -------: | -------: |
| Linux | 20.81 MiB | 20.06 MiB | 6.25 MiB | 6.39 MiB |
| macOS | 18.09 MiB | 17.34 MiB | 5.44 MiB | 5.47 MiB |

The #997 frontier reduces peak RSS by about 69% relative to #996 on both platforms;
#998 retains that improvement. This proves a benefit for this fixture, rather than a
worst-case bound for arbitrary directory width, path size, archives or storage behavior.

Retain C01/C03: avoid temporary collection-name and comparison-category strings while preserving
owned keys and caller-storage lifetimes. This is a narrow construction/ownership improvement, without
a universal wall-clock claim. Defer C02's matcher batch-buffer recycling API until repeatable gains
justify its additional lifetime transition. Byte-identical collector binaries provide a negative control;
small timing differences between those labels cannot be attributed to recycling.

Infrastructure is isolated in [PR #1035](https://github.com/mboworks/xff/pull/1035): the query downloader
repair and executor diagnostic use private test-only support. The shipping CLI is identical to baseline
on Linux; all non-debug Mach-O sections match on macOS. Its pre-sync full CI passed after one targeted
retry of a runner disk-capacity failure, with no source/head change. The subsequent main synchronization
has a separate current-head CI context. Runtime experiments remain unmerged.
1 change: 1 addition & 0 deletions release-site.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"CHANGELOG.md": "CHANGELOG.html",
"CONTRIBUTING.md": "CONTRIBUTING.html",
"docs/benchmarks/execution-resources.md": "docs/benchmarks/execution-resources.html",
"docs/benchmarks/executor-copy-audit-20261101.md": "docs/benchmarks/executor-copy-audit-20261101.html",
"docs/benchmarks/executor-dispatch-probe.md": "docs/benchmarks/executor-dispatch-probe.html",
"docs/execution-resources.md": "docs/execution-resources.html",
"docs/ci-performance.md": "docs/ci-performance.html",
Expand Down
1 change: 1 addition & 0 deletions xff/cli/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -1264,6 +1264,7 @@ bashtest(
bashtest(
name = "rg_test",
size = "small",
timeout = "moderate", # Repeated frontend subprocesses exceed 60 seconds under MSan.
srcs = ["rg_test.sh"],
data = ["//xff/cli/testing:xff"],
)
Expand Down
1 change: 1 addition & 0 deletions xff/engine/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ cc_test(
"//xff/vfs:vfs_cc",
"@abseil-cpp//absl/status:statusor",
"@googletest//:gtest_main",
"@mboworks_mbo//mbo/testing:matchers_cc",
"@mboworks_mbo//mbo/testing:status_cc",
],
)
Expand Down
2 changes: 1 addition & 1 deletion xff/engine/collect.cc
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ bool Collections::Add(std::string_view name, const Visit& visit) {
}
rows_ += 1;
bytes_ += entry_bytes;
by_name_[std::string(name)].push_back(
by_name_.try_emplace(name).first->second.push_back(
Entry{
.path = CopyText(visit.path),
.name = CopyText(visit.name),
Expand Down
17 changes: 17 additions & 0 deletions xff/engine/collect_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@

#include "gmock/gmock.h"
#include "gtest/gtest.h"
#include "mbo/testing/matchers.h"
#include "mbo/testing/status.h"
#include "xff/engine/walk.h"
#include "xff/parser/parser.h"
Expand All @@ -30,6 +31,7 @@
namespace xff::engine {
namespace {

using ::mbo::testing::EqualsText;
using ::testing::AllOf;
using ::testing::ElementsAre;
using ::testing::Eq;
Expand Down Expand Up @@ -80,6 +82,21 @@ TEST_F(CollectTest, AddStoresUnderTheNamedCollection) {
EXPECT_THAT(entry.metadata.size, Eq(42));
}

TEST_F(CollectTest, RepeatedLongNamesOwnTheirKeyBeyondCallerStorage) {
const vfs::Metadata metadata{.type = vfs::FileType::kRegular};
const std::string expected_name(100, 'n');
Collections collections;
{
std::string name = expected_name;
EXPECT_THAT(collections.Add(name, MakeVisit("./a", "a", ".", metadata)), IsTrue());
EXPECT_THAT(collections.Add(name, MakeVisit("./b", "b", ".", metadata)), IsTrue());
name.assign(100, 'x');
}
EXPECT_THAT(collections.Names(), ElementsAre(EqualsText(expected_name)));
EXPECT_THAT(collections.Entries(expected_name), SizeIs(2));
EXPECT_THAT(collections.Entries(expected_name).back().path, EqualsText("./b"));
}

TEST_F(CollectTest, AsVisitRebuildsTheEntryForASink) {
const vfs::Metadata md{.type = vfs::FileType::kRegular, .size = 7};
Collections collections;
Expand Down
4 changes: 2 additions & 2 deletions xff/engine/run.cc
Original file line number Diff line number Diff line change
Expand Up @@ -4574,7 +4574,7 @@ RunResult RunTreeCompare(
return RunResult{.errors = std::max(left_run_result.errors, right_result.errors)};
}

const auto record_category = [&](const std::string& path, std::string_view category) {
const auto record_category = [&](std::string_view path, std::string_view category) {
if (side_summaries.at(0).per_entry || side_summaries.at(1).per_entry) {
categories.emplace(path, category);
}
Expand Down Expand Up @@ -4624,7 +4624,7 @@ RunResult RunTreeCompare(
const std::size_t category = !pair.left ? 1 : !pair.right ? 0 : result->same ? 3 : 2;
const auto category_name = kComparisonCategories.at(category);
counts.Add(category, pair.left, pair.right);
record_category(std::string(pair.path), category_name);
record_category(pair.path, category_name);
different |= category != 3;
if (!selected.at(category)) {
continue;
Expand Down
Loading