diff --git a/CHANGELOG.md b/CHANGELOG.md index 31cf70f9c1d..3ddfc29733f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ # 0.11.0 +- Render lexical filename fields and path qualifiers from borrowed input slices instead of temporary path objects. + - Cache fuzzy normalization ceilings and reuse bounded per-worker scoring rows; retain valid ranked matches after long gaps. - Prepare simple case-sensitive name/path globs once and avoid copying owned native glob patterns; preserve native matching for complex and case-folded patterns. diff --git a/TODO.md b/TODO.md index 51c0882cc69..01a65d731cc 100644 --- a/TODO.md +++ b/TODO.md @@ -371,7 +371,7 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms - [x] O02: Qualify owned-pattern references and prepared simple ASCII glob operands (#1042). [Cross-platform measurements](docs/benchmarks/native-glob-preparation-20261101.md) preserve case-folded/complex fallbacks, native controls and preparation costs. -- [ ] O03: Qualify cached fuzzy self-score ceilings and bounded per-worker score rows on macOS +- [x] O03: Qualify cached fuzzy self-score ceilings and bounded per-worker score rows on macOS and Linux. Preserve ranking/negation/case semantics and measure allocation counts separately. [Selected policy and full evidence](docs/benchmarks/fuzzy-score-workspace-20261102.md): exact initial allocation beats the rejected 64-character minimum; full release suites and scoped lint @@ -381,6 +381,9 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms general fuzz input cap is too small for this case. - [ ] O04-O06: Independently qualify lexical field views, heterogeneous capture/definition lookup, and remaining read-buffer zeroing. Prefer existing values and interfaces before adding prepared wrappers. + [O04 evidence](docs/benchmarks/lexical-field-views-20261102.md) retains native dirname + spelling, exact output oracles and all slower controls. Qualify O05 key/reference and map + choices separately before selecting a representation. - [ ] O07-O12: Profile and qualify caller-owned stream buffers, typed reusable directory task slots, batch ownership, bounded result reuse, active-worker dispatch and direct production evaluation. Preserve existing stage policy while isolating task storage and lifecycle costs. diff --git a/docs/benchmarks/lexical-field-views-20261102-linux.json.gz b/docs/benchmarks/lexical-field-views-20261102-linux.json.gz new file mode 100644 index 00000000000..950d8b8cfd3 Binary files /dev/null and b/docs/benchmarks/lexical-field-views-20261102-linux.json.gz differ diff --git a/docs/benchmarks/lexical-field-views-20261102-macos.json.gz b/docs/benchmarks/lexical-field-views-20261102-macos.json.gz new file mode 100644 index 00000000000..25e1bae7bf7 Binary files /dev/null and b/docs/benchmarks/lexical-field-views-20261102-macos.json.gz differ diff --git a/docs/benchmarks/lexical-field-views-20261102-provenance.json.gz b/docs/benchmarks/lexical-field-views-20261102-provenance.json.gz new file mode 100644 index 00000000000..26c7cda132b Binary files /dev/null and b/docs/benchmarks/lexical-field-views-20261102-provenance.json.gz differ diff --git a/docs/benchmarks/lexical-field-views-20261102.md b/docs/benchmarks/lexical-field-views-20261102.md new file mode 100644 index 00000000000..e080d06a6e5 --- /dev/null +++ b/docs/benchmarks/lexical-field-views-20261102.md @@ -0,0 +1,92 @@ +# Borrowed lexical field components (2026-11-02) + +## Decision + +Use the existing `StringOrView` return type to borrow filename, stem, final extension, all suffixes and core slices directly from the input. Apply the same helpers to path qualifiers and language/MIME filename lookup. Keep directory rendering through native `parent_path`: repeated-separator normalization is observable and does not always correspond to one input slice. No new path wrapper, per-entry cache, thread, ownership hierarchy, flag or dependency is introduced. + +The input context owns or borrows a stable path for the duration of rendering. Compiled templates append borrowed values while that context and any owned transformed field value remain alive. Directory results remain owned when native path rendering constructs them. Backslashes remain ordinary filename bytes on both supported POSIX hosts. Embedded NULs remain length-delimited input, not C-string terminators. + +## Measurement contract + +Reference main `2b21790502fd6bf9b6e1afb53cc1444cf140f296` includes collection text sharing and prepared globs. Only `fields.cc` differs between runtime variants; both use the same test-only kernel source and build configuration. No unmerged fuzzy change is included. Source/binary identities and scripts are retained below. + +Release LLVM 23 + ThinLTO on the local M5 Pro (warm APFS, requested 1/3/10 workers without affinity) and drogon Ryzen 9950X KVM guest (tmpfs, child affinity 1/3/10). Builds and tests finish before timing. Three sessions, discarded native warmups, nine interleaved rounds, fastest seven. Every fresh command checks exact complete output against an independent filename/component/directory oracle. Counts 10/1,000/10,000/100,000 with one/three workers plus 100,000 with ten: 4,320 invocations across both hosts including warmups. Commands use explicit `--template`, `--exact`, `--sort=dir`, UTC and no config/pager. + +The in-memory kernel compiles templates before timing and renders 10/1,000/100,000 paths with short (8-byte) or long (160-byte) directory/filename components. It measures one field, six direct fields, six path qualifiers and an unchanged directory control. Nine repetitions retain seven, in three shuffled sessions. Path construction and template compilation are excluded from this steady-state kernel, so no preparation/startup improvement is claimed. Native commands include those real costs. + +Tables show median session times and median within-session ratios. Negative means faster. Absolute times across hosts are not interchangeable. + +## Native 100,000-file output + +| Host | Template | Workers | Baseline ms | Candidate ms | Change | +| ----- | ----------- | ------: | ----------: | -----------: | ------: | +| macos | name | 1 | 228.628 | 225.766 | -1.04% | +| macos | components | 1 | 264.324 | 242.042 | -8.54% | +| macos | qualified | 1 | 263.991 | 241.427 | -8.54% | +| macos | dir-control | 1 | 241.864 | 242.050 | +0.32% | +| macos | name | 3 | 104.010 | 104.073 | -0.41% | +| macos | components | 3 | 105.073 | 106.703 | +0.95% | +| macos | qualified | 3 | 104.237 | 105.056 | +0.79% | +| macos | dir-control | 3 | 104.911 | 104.453 | -0.90% | +| macos | name | 10 | 74.524 | 74.184 | -0.11% | +| macos | components | 10 | 82.081 | 76.808 | -6.42% | +| macos | qualified | 10 | 82.977 | 76.849 | -4.75% | +| macos | dir-control | 10 | 74.542 | 72.319 | -1.33% | +| linux | name | 1 | 102.180 | 100.281 | -1.75% | +| linux | components | 1 | 121.704 | 108.422 | -10.91% | +| linux | qualified | 1 | 119.912 | 108.021 | -9.59% | +| linux | dir-control | 1 | 103.248 | 104.082 | +0.02% | +| linux | name | 3 | 36.557 | 35.777 | -2.13% | +| linux | components | 3 | 49.582 | 40.223 | -18.88% | +| linux | qualified | 3 | 48.022 | 40.219 | -16.20% | +| linux | dir-control | 3 | 37.430 | 37.763 | +0.27% | +| linux | name | 10 | 21.361 | 20.333 | -5.71% | +| linux | components | 10 | 40.752 | 27.924 | -31.24% | +| linux | qualified | 10 | 39.747 | 28.795 | -26.01% | +| linux | dir-control | 10 | 23.296 | 23.781 | +2.08% | + +Linux multi-field output improves substantially. On macOS the one/ten-worker multi-field cases improve, while three-worker native results remain near baseline or slightly slower despite clear kernel gains. Traversal, coordination and output remain significant; do not turn a rendering-kernel percentage into a filesystem throughput claim. + +## In-memory rendering of 1,000 paths + +| Host | Template | Component bytes | Baseline us | Candidate us | Change | +| ----- | ----------- | --------------: | ----------: | -----------: | ------: | +| macos | name | 8 | 274.73 | 238.43 | -14.69% | +| macos | name | 160 | 432.85 | 280.87 | -35.66% | +| macos | components | 8 | 595.93 | 375.69 | -36.67% | +| macos | components | 160 | 1438.05 | 582.34 | -59.97% | +| macos | qualified | 8 | 620.72 | 400.64 | -34.88% | +| macos | qualified | 160 | 1440.45 | 584.73 | -59.59% | +| macos | dir-control | 8 | 283.60 | 281.77 | -0.64% | +| macos | dir-control | 160 | 507.00 | 510.62 | -0.15% | +| linux | name | 8 | 60.00 | 42.95 | -28.34% | +| linux | name | 160 | 117.48 | 93.27 | -20.05% | +| linux | components | 8 | 244.42 | 111.72 | -54.23% | +| linux | components | 160 | 515.83 | 381.12 | -26.11% | +| linux | qualified | 8 | 227.12 | 119.02 | -47.55% | +| linux | qualified | 160 | 544.03 | 378.62 | -30.40% | +| linux | dir-control | 8 | 62.68 | 64.07 | +2.43% | +| linux | dir-control | 160 | 149.16 | 150.91 | +1.12% | + +## Costs and counterexamples + +- macos: slowest paired native median +1.74% at (10000, 3, 'name'); all small and directory-control samples remain in the evidence. +- macos: build-output executable 4,069,152 -> 4,068,752 bytes (-400), before release stripping/compression. +- linux: slowest paired native median +2.08% at (100000, 10, 'dir-control'); all small and directory-control samples remain in the evidence. +- linux: build-output executable 12,306,928 -> 12,287,256 bytes (-19,672), before release stripping/compression. + +This removes temporary path/component construction, not final output storage. Long outputs still allocate the returned result string; directory fields retain their previous native path work. No claim of zero allocations for whole template rendering is made. There is no additional retained per-entry state. + +## Correctness and verification + +Differential tests compare every direct field and its path qualifier against native `std::filesystem::path` lexical results on both hosts: empty/root paths, repeated/trailing separators, dot/dot-dot, hidden names, trailing dots, compound suffixes, backslashes, virtual member names, embedded NULs, nonterminated slices and long components. Combined templates, owned definition inputs and chained transforms retain their lifetimes. Each C++ test source has its own test rule. The existing field and fuzz suites also pass. Complete release/extension suites pass 305/305 on macOS and Linux (with required working FUSE). +The added long-owned-parent lifetime case passes separately on both hosts. Scoped clang-tidy passes +the runtime, differential test and benchmark. Its benchmark-only unused loop token required a +`[[maybe_unused]]` annotation; the final executable code section is byte-identical to the measured +kernel on both hosts. Final normal hooks and published-head CI remain required. + +## Lossless raw evidence + +- [lexical-field-views-20261102-macos.json.gz](lexical-field-views-20261102-macos.json.gz): 115,070 bytes; SHA-256 `f3040e3bff4fb49a6c9d1b6716b9818e98f48c1742976fe5db56ec99be8b2bc7`. +- [lexical-field-views-20261102-linux.json.gz](lexical-field-views-20261102-linux.json.gz): 114,380 bytes; SHA-256 `6b2de8b3e8bd56c90770384a22aa8ad994afca87a7de6a300df852ae8daaf984`. +- [lexical-field-views-20261102-provenance.json.gz](lexical-field-views-20261102-provenance.json.gz): 47,600 bytes; SHA-256 `99581677c57cec0949ed9ad4140ed49edacca6dc28faed7e2d6090cb70326fc1`. diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md index 7c1c28bd3e7..8f71cbc60c8 100644 --- a/docs/performance-analysis.md +++ b/docs/performance-analysis.md @@ -2918,8 +2918,8 @@ virtual layers and generic executor hierarchies need an actual lifetime or exten | O21 | Reuse similarity representations | Repeated shingle generation and pairwise verification dominate large document sets | Existing near-duplicate design | Cache reference shingles; exact verification after indexing | | O22 | Reduce output construction churn | Numeric/cell/JSON strings, row insertion and aligned-table retention allocate | Existing output/scope contracts | Append into reusable row buffers; stream machine formats | -All elapsed improvements in this table except O01 are hypotheses, not measured gains. Allocation -sites are inspected code facts; an allocation's wall-time significance still requires profiling. +O01-O03 have qualified cross-platform reports below. Other entries remain hypotheses until their +individual report records a decision; inspected allocation sites alone do not establish elapsed gains. ### Small changes first @@ -3130,3 +3130,23 @@ Linux. These are component/whole-engine fixtures, not filesystem throughput clai and scoring input; bare/control cases stay near baseline. Scored kernels improve 29-54%, while unrelated bare kernels include 3-4% slower cases and preparation is slightly slower. Keep these counterexamples and all raw data in the report; the result is not a claim about every fuzzy model. + +## O04 borrowed lexical field components + +The [cross-platform field experiment](benchmarks/lexical-field-views-20261102.md) reuses +`StringOrView` for filename, stem, final/all suffixes and core fields, their path qualifiers and +language/MIME filename lookup. Native directory spelling remains unchanged; repeated separators +make parent rendering observably different from a raw slice. No new cache or path wrapper is needed. + +Differential field and fuzz tests pass on both hosts. The six-field in-memory kernel improves +26-60% depending on host/path length; 100,000-file native output improves about 9-11% with one +worker on Linux and 8.5% on macOS. Linux three/ten-worker cases improve further, while macOS +three-worker native results remain near baseline or slightly slower. Preserve small regressions, +the unchanged directory control and all source/binary identities; do not generalize kernel gains +into filesystem speedups. Complete release/extension suites and scoped lint remain required. + +O05 should also test the existing ownership boundary before changing every map declaration: +`Template::Segment` already owns its compiled key. Passing that key to field callbacks by const +string reference can avoid the temporary lookup string in definition/capture rendering without +adding another map, interner or lookup layer. General heterogeneous maps remain useful where the +caller genuinely has only a view; distinguish those callers from an avoidable view conversion. diff --git a/release-site.json b/release-site.json index ed86c768e53..32207bfbc73 100644 --- a/release-site.json +++ b/release-site.json @@ -61,7 +61,8 @@ "docs/performance-analysis.md": "docs/performance-analysis.html", "docs/design-expression-program.md": "docs/design-expression-program.html", "docs/expression-qualification-audit.md": "docs/expression-qualification-audit.html", - "docs/benchmarks/fuzzy-score-workspace-20261102.md": "docs/benchmarks/fuzzy-score-workspace-20261102.html" + "docs/benchmarks/fuzzy-score-workspace-20261102.md": "docs/benchmarks/fuzzy-score-workspace-20261102.html", + "docs/benchmarks/lexical-field-views-20261102.md": "docs/benchmarks/lexical-field-views-20261102.html" }, "generated_html": { "XFF.md": { diff --git a/xff/presentation/fields/BUILD.bazel b/xff/presentation/fields/BUILD.bazel index 90eb1506c31..5a49c422903 100644 --- a/xff/presentation/fields/BUILD.bazel +++ b/xff/presentation/fields/BUILD.bazel @@ -14,7 +14,7 @@ # limitations under the License. load("@rules_fuzzing//fuzzing:cc_defs.bzl", "cc_fuzz_test") -load("//xff:cc.bzl", "cc_library", "cc_test") +load("//xff:cc.bzl", "cc_binary", "cc_library", "cc_test") package(default_visibility = ["//visibility:private"]) @@ -59,6 +59,19 @@ cc_test( ], ) +cc_test( + name = "lexical_fields_test", + size = "small", + srcs = ["lexical_fields_test.cc"], + deps = [ + ":fields_cc", + "//xff/vfs:vfs_cc", + "@googletest//:gtest", + "@googletest//:gtest_main", + "@mboworks_mbo//mbo/testing:matchers_cc", + ], +) + # Compile and render arbitrary field templates against a deterministic context. The driver checks # the context-free extraction classification against the actual extraction API. cc_fuzz_test( @@ -76,3 +89,15 @@ cc_fuzz_test( "@abseil-cpp//absl/time", ], ) + +# Test-only compiled template kernel; inputs are in memory and never read the host filesystem. +cc_binary( + name = "fields_benchmark", + testonly = True, + srcs = ["fields_benchmark.cc"], + deps = [ + ":fields_cc", + "//xff/vfs:vfs_cc", + "@com_github_google_benchmark//:benchmark", + ], +) diff --git a/xff/presentation/fields/fields.cc b/xff/presentation/fields/fields.cc index f48f8e15617..cf7fe9b46e4 100644 --- a/xff/presentation/fields/fields.cc +++ b/xff/presentation/fields/fields.cc @@ -178,7 +178,7 @@ std::string HumanSize(std::uint64_t bytes) { // Per-field renderers. The signature is uniform so they can share one dispatch // table: (key, qualifier, ctx). `key` is the bound argument for dynamic fields // (a capture index, an {env.NAME} var, ...), unused (unnamed) by the builtins. -// `path`-derived fields build their own std::filesystem::path. +// Lexical filename fields borrow path slices; directory spelling keeps native path semantics. StringOrView PathField(std::string_view, std::string_view, const RenderContext& ctx) { return ctx.path; } @@ -215,6 +215,26 @@ StringOrView RelpathField(std::string_view, std::string_view, const RenderContex return path; // not root-prefixed (should not happen from the walk) } +// Both supported hosts use POSIX separators. These slices retain the caller's path lifetime, +// including embedded NULs and virtual archive member names; no filesystem lookup is involved. +std::string_view Filename(std::string_view path) { + const std::size_t slash = path.rfind('/'); + return slash == std::string_view::npos ? path : path.substr(slash + 1); +} + +std::string_view Extension(std::string_view filename) { + if (filename == "." || filename == "..") { + return {}; + } + const std::size_t dot = filename.rfind('.'); + return dot == std::string_view::npos || dot == 0 ? std::string_view{} : filename.substr(dot); +} + +std::string_view AllSuffixes(std::string_view filename) { + const std::size_t dot = filename.find('.', 1); + return dot == std::string_view::npos ? std::string_view{} : filename.substr(dot); +} + StringOrView DirField(std::string_view, std::string_view, const RenderContext& ctx) { std::string parent = stdfs::path(std::string(ctx.path)).parent_path().string(); if (parent.empty()) { @@ -224,46 +244,32 @@ StringOrView DirField(std::string_view, std::string_view, const RenderContext& c } StringOrView NameField(std::string_view, std::string_view, const RenderContext& ctx) { - return stdfs::path(std::string(ctx.path)).filename().string(); + return Filename(ctx.path); } StringOrView StemField(std::string_view, std::string_view, const RenderContext& ctx) { - return stdfs::path(std::string(ctx.path)).stem().string(); + const std::string_view filename = Filename(ctx.path); + return filename.substr(0, filename.size() - Extension(filename).size()); } StringOrView ExtField(std::string_view, std::string_view, const RenderContext& ctx) { - std::string ext = stdfs::path(std::string(ctx.path)).extension().string(); // includes the leading '.' - if (!ext.empty()) { - ext.erase(0, 1); - } - return {std::move(ext)}; + const std::string_view extension = Extension(Filename(ctx.path)); + return extension.empty() ? extension : extension.substr(1); } StringOrView SuffixesField(std::string_view, std::string_view, const RenderContext& ctx) { - std::string filename = stdfs::path(std::string(ctx.path)).filename().string(); - const std::string::size_type dot = filename.find('.', 1); // all extensions; a leading dot is not one - if (dot == std::string::npos) { - return ""; - } - filename.erase(0, dot); - return {std::move(filename)}; + return AllSuffixes(Filename(ctx.path)); } -// {suffix}: the LAST extension WITH its leading dot (pathlib's .suffix: ".gz"), -// complementing {ext} (no dot: "gz") and {suffixes} (all of them: ".tar.gz"). +// {suffix} includes the final extension's dot; {ext} omits it. StringOrView SuffixField(std::string_view, std::string_view, const RenderContext& ctx) { - return stdfs::path(std::string(ctx.path)).extension().string(); // ".gz", or "" when none + return Extension(Filename(ctx.path)); } -// {core}: the filename with ALL extensions removed (foo.tar.gz -> foo), the complement -// of {suffixes} ({core} + {suffixes} == {name}, as {stem} + {suffix} == {name}). +// {core} + {suffixes} reconstructs {name}, including a leading dot in the core. StringOrView CoreField(std::string_view, std::string_view, const RenderContext& ctx) { - std::string filename = stdfs::path(std::string(ctx.path)).filename().string(); - const std::string::size_type dot = filename.find('.', 1); // first extension; a leading dot is not one - if (dot != std::string::npos) { - filename.erase(dot); - } - return {std::move(filename)}; + const std::string_view filename = Filename(ctx.path); + return filename.substr(0, filename.size() - AllSuffixes(filename).size()); } StringOrView DepthField(std::string_view, std::string_view, const RenderContext& ctx) { @@ -354,12 +360,12 @@ StringOrView FuzzyField(std::string_view, std::string_view, const RenderContext& // "Python"), from its filename/extension via the language table; empty when unrecognized. Content // is not read, so it is cheap; composes with --summary group-by to tally files per language. StringOrView LanguageField(std::string_view, std::string_view, const RenderContext& ctx) { - const std::string name = stdfs::path(std::string(ctx.path)).filename().string(); + const std::string_view name = Filename(ctx.path); return language::LanguageForName(name); } std::optional LanguageInfo(const RenderContext& ctx) { - return language::InfoForName(stdfs::path(std::string(ctx.path)).filename().string()); + return language::InfoForName(Filename(ctx.path)); } StringOrView LanguageTypeField(std::string_view, std::string_view, const RenderContext& ctx) { @@ -383,12 +389,12 @@ StringOrView LanguageSourceField(std::string_view, std::string_view, const Rende } StringOrView MimeField(std::string_view, std::string_view, const RenderContext& ctx) { - const std::string name = stdfs::path(std::string(ctx.path)).filename().string(); + const std::string_view name = Filename(ctx.path); return mime::TypeForName(name); } mime::TypeInfo MimeInfo(const RenderContext& ctx) { - return mime::InfoForName(stdfs::path(std::string(ctx.path)).filename().string()); + return mime::InfoForName(Filename(ctx.path)); } StringOrView MimeCategoryField(std::string_view, std::string_view, const RenderContext& ctx) { @@ -909,34 +915,34 @@ bool IsPathComponent(std::string_view qualifier) { // so any path-valued field composes: {target:stem}, {path:name}, {def.B:dir}, ... // {core} = the filename with ALL extensions removed (foo.tar.gz -> foo), the // complement of {suffixes}; {path} (and any unknown keyword) is the identity. -std::string PathComponent(std::string_view value, std::string_view component) { - const stdfs::path path{std::string(value)}; +StringOrView PathComponent(std::string_view value, std::string_view component) { if (component == "dir") { - const std::string parent = path.parent_path().string(); - return parent.empty() ? "." : parent; + // parent_path can normalize repeated separators; keep the native spelling contract here. + std::string parent = stdfs::path(std::string(value)).parent_path().string(); + return parent.empty() ? StringOrView(".") : StringOrView(std::move(parent)); } + const std::string_view filename = Filename(value); if (component == "name" || component == "basename" || component == "file") { - return path.filename().string(); + return filename; } + const std::string_view extension = Extension(filename); if (component == "stem") { - return path.stem().string(); + return filename.substr(0, filename.size() - extension.size()); } if (component == "ext" || component == "extension") { - const std::string ext = path.extension().string(); // includes the leading '.' - return ext.empty() ? ext : ext.substr(1); + return extension.empty() ? extension : extension.substr(1); } if (component == "suffix") { - return path.extension().string(); // last extension WITH its dot + return extension; } - const std::string filename = path.filename().string(); - const std::string::size_type dot = filename.find('.', 1); // first extension; a leading dot is not one + const std::string_view suffixes = AllSuffixes(filename); if (component == "suffixes") { - return dot == std::string::npos ? "" : filename.substr(dot); + return suffixes; } if (component == "core") { - return dot == std::string::npos ? filename : filename.substr(0, dot); + return filename.substr(0, filename.size() - suffixes.size()); } - return std::string(value); // "path" (whole) or an unrecognised keyword -> identity + return value; // "path" (whole) or an unrecognised keyword -> identity } absl::Status ValidateRewriteChain(std::string_view spec) { @@ -1291,7 +1297,7 @@ std::string Template::Render(const RenderContext& context) const { segment.fn(segment.key, transform ? std::string_view{} : segment.qualifier, entry); const std::string_view value = field_value.view(); switch (segment.post) { - case Segment::PostProcess::kComponent: out.append(PathComponent(value, segment.qualifier)); break; + case Segment::PostProcess::kComponent: out.append(PathComponent(value, segment.qualifier).view()); break; case Segment::PostProcess::kNone: out.append(value); break; case Segment::PostProcess::kRewrite: out.append(segment.transform->Rewrite(value)); break; // A per-line extraction is a value stream. A terminal reducer (`;join(...)`) collapses it to diff --git a/xff/presentation/fields/fields_benchmark.cc b/xff/presentation/fields/fields_benchmark.cc new file mode 100644 index 00000000000..bfa212ec18b --- /dev/null +++ b/xff/presentation/fields/fields_benchmark.cc @@ -0,0 +1,91 @@ +// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors +// SPDX-License-Identifier: Apache-2.0 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include +#include +#include +#include +#include +#include +#include + +#include "benchmark/benchmark.h" +#include "xff/presentation/fields/fields.h" +#include "xff/vfs/entry.h" + +namespace { + +struct Scenario final { + std::string_view name; + std::string_view format; +}; + +void RenderFields(benchmark::State& state, Scenario scenario) { + const auto count = static_cast(state.range(0)); + const auto length = static_cast(state.range(1)); + std::vector paths; + paths.reserve(count); + for (std::size_t at = 0; at < count; ++at) { + paths.push_back( + "root/" + std::string(length, 'd') + "/" + std::string(length, 'f') + std::to_string(at) + ".tar.gz"); + } + const xff::fields::Template compiled = xff::fields::Template::Compile(scenario.format); + const xff::vfs::Metadata metadata; + for ([[maybe_unused]] const auto unused : state) { + for (const std::string& path : paths) { + std::string value = compiled.Render({.path = path, .metadata = metadata}); + benchmark::DoNotOptimize(value); + } + } + state.SetItemsProcessed(state.iterations() * static_cast(count)); +} + +// Repetitions retain the same fastest-seven protocol used by other expression benchmarks. +double FastestSeven(const std::vector& values) { + std::vector sorted(values); + std::sort(sorted.begin(), sorted.end()); + const std::size_t count = std::min(7, sorted.size()); + return count == 0 ? 0 + : std::accumulate(sorted.begin(), sorted.begin() + static_cast(count), 0.0) + / static_cast(count); +} + +} // namespace + +// XFF_ABI_POINTER: Google Benchmark requires the process argv ABI. +int main(int argc, char** argv) { + benchmark::Initialize(&argc, argv); + if (benchmark::ReportUnrecognizedArguments(argc, argv)) { + return 1; + } + static constexpr auto kScenarios = std::to_array({ + {.name = "name", .format = "{name}"}, + {.name = "components", .format = "{name}|{stem}|{ext}|{suffix}|{suffixes}|{core}"}, + {.name = "qualified", .format = "{path:name}|{path:stem}|{path:ext}|{path:suffix}|{path:suffixes}|{path:core}"}, + {.name = "dir-control", .format = "{dir}"}, + }); + for (const Scenario& scenario : kScenarios) { + benchmark::RegisterBenchmark("fields/" + std::string(scenario.name), RenderFields, scenario) + ->ArgsProduct({{10, 1'000, 100'000}, {8, 160}}) + ->UseRealTime() + ->Unit(benchmark::kMicrosecond) + ->Repetitions(9) + ->MinTime(0.01) + ->MinWarmUpTime(0.01) + ->ComputeStatistics("fastest7of9", FastestSeven); + } + benchmark::RunSpecifiedBenchmarks(); + benchmark::Shutdown(); +} diff --git a/xff/presentation/fields/lexical_fields_test.cc b/xff/presentation/fields/lexical_fields_test.cc new file mode 100644 index 00000000000..22f62e24c5b --- /dev/null +++ b/xff/presentation/fields/lexical_fields_test.cc @@ -0,0 +1,129 @@ +// SPDX-FileCopyrightText: Copyright (c) M. Boerger, the MBO Works authors +// SPDX-License-Identifier: Apache-2.0 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include +#include +#include +#include +#include +#include + +#include "gmock/gmock.h" +#include "gtest/gtest.h" +#include "mbo/testing/matchers.h" +#include "xff/presentation/fields/fields.h" +#include "xff/vfs/entry.h" + +namespace xff::fields { +namespace { + +using ::mbo::testing::EqualsText; +namespace stdfs = ::std::filesystem; + +struct LexicalFieldsTest : ::testing::Test {}; + +TEST_F(LexicalFieldsTest, BorrowedComponentsMatchNativeLexicalPaths) { + std::vector paths{ + "", + "/", + "//", + "///", + ".", + "..", + "...", + ".hidden", + ".hidden.cc", + "file.", + "file..", + "dir/", + "dir//", + "dir/.", + "dir/..", + "dir///file.tar.gz", + "//dir/file", + "/dir/file/", + "archive.tar!member/.file.cc", + "a\\b.cc", + std::string("dir/file\0.cc", 12), + std::string(200, 'd') + "/" + std::string(200, 'f') + ".tar.gz", + }; + static constexpr auto kParts = std::to_array({"", ".", "..", "a", ".a", "a.", "a.b.c", "a..b"}); + static constexpr auto kSeparators = std::to_array({"/", "//", "///"}); + for (const std::string_view left : kParts) { + for (const std::string_view separator : kSeparators) { + for (const std::string_view right : kParts) { + paths.push_back(std::string(left) + std::string(separator) + std::string(right)); + } + } + } + static constexpr auto kFields = + std::to_array({"name", "stem", "ext", "suffix", "suffixes", "core", "dir"}); + std::array direct; + std::array qualified; + for (std::size_t at = 0; at < kFields.size(); ++at) { + const std::string field(kFields.at(at)); + direct.at(at) = Template::Compile("{" + field + "}"); + qualified.at(at) = Template::Compile("{path:" + field + "}"); + } + const vfs::Metadata metadata; + for (const std::string& path : paths) { + SCOPED_TRACE(path); + const stdfs::path native(path); + const std::string filename = native.filename().string(); + const std::string suffix = native.extension().string(); + const std::size_t dot = filename.find('.', 1); + const std::array expected{ + filename, + native.stem().string(), + suffix.empty() ? suffix : suffix.substr(1), + suffix, + dot == std::string::npos ? "" : filename.substr(dot), + dot == std::string::npos ? filename : filename.substr(0, dot), + native.parent_path().empty() ? "." : native.parent_path().string(), + }; + // A nonterminated slice catches accidental reliance on the input's c_str boundary. + const std::string storage = "before" + path + "after"; + const RenderContext context{.path = std::string_view(storage).substr(6, path.size()), .metadata = metadata}; + for (std::size_t at = 0; at < kFields.size(); ++at) { + SCOPED_TRACE(kFields.at(at)); + EXPECT_THAT(direct.at(at).Render(context), EqualsText(expected.at(at))); + EXPECT_THAT(qualified.at(at).Render(context), EqualsText(expected.at(at))); + } + } +} + +TEST_F(LexicalFieldsTest, MultipleFieldsAndOwnedTransformInputsRetainTheirLifetimes) { + const Template compiled = + Template::Compile("{name}|{stem}|{ext}|{suffix}|{suffixes}|{core}|{path:stem}|{path:s|^|prefix/|;s|$|.txt|}"); + const vfs::Metadata metadata; + const RenderContext context{.path = "dir/archive.tar.gz", .metadata = metadata}; + EXPECT_THAT( + compiled.Render(context), + EqualsText("archive.tar.gz|archive.tar|gz|.gz|.tar.gz|archive|archive.tar|prefix/dir/archive.tar.gz.txt")); + const Template rewritten = Template::Compile("{def.input:name}|{def.input:stem}|{def.input:core}"); + const std::map definitions{{"input", "a/temporary.long.name"}}; + RenderContext defined = context; + defined.defines.set_ref(definitions); + EXPECT_THAT(rewritten.Render(defined), EqualsText("temporary.long.name|temporary.long|temporary")); + // DirField owns its normalized parent. A component view must be consumed before that owner dies. + const std::string parent = std::string(200, 'd') + ".tar.gz"; + const std::string path = "root/" + parent + "/file.txt"; + const RenderContext long_parent{.path = path, .metadata = metadata}; + const Template parent_components = Template::Compile("{dir:name}|{dir:stem}"); + EXPECT_THAT(parent_components.Render(long_parent), EqualsText(parent + "|" + std::string(200, 'd') + ".tar")); +} + +} // namespace +} // namespace xff::fields