Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

# 0.11.0

- Render lexical filename fields and path qualifiers from borrowed input slices instead of temporary path objects.

- Cache fuzzy normalization ceilings and reuse bounded per-worker scoring rows; retain valid ranked matches after long gaps.

- Prepare simple case-sensitive name/path globs once and avoid copying owned native glob patterns; preserve native matching for complex and case-folded patterns.
Expand Down
5 changes: 4 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
- [x] O02: Qualify owned-pattern references and prepared simple ASCII glob operands (#1042).
[Cross-platform measurements](docs/benchmarks/native-glob-preparation-20261101.md) preserve
case-folded/complex fallbacks, native controls and preparation costs.
- [ ] O03: Qualify cached fuzzy self-score ceilings and bounded per-worker score rows on macOS
- [x] O03: Qualify cached fuzzy self-score ceilings and bounded per-worker score rows on macOS
and Linux. Preserve ranking/negation/case semantics and measure allocation counts separately.
[Selected policy and full evidence](docs/benchmarks/fuzzy-score-workspace-20261102.md): exact
initial allocation beats the rejected 64-character minimum; full release suites and scoped lint
Expand All @@ -381,6 +381,9 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
general fuzz input cap is too small for this case.
- [ ] O04-O06: Independently qualify lexical field views, heterogeneous capture/definition lookup,
and remaining read-buffer zeroing. Prefer existing values and interfaces before adding prepared wrappers.
[O04 evidence](docs/benchmarks/lexical-field-views-20261102.md) retains native dirname
spelling, exact output oracles and all slower controls. Qualify O05 key/reference and map
choices separately before selecting a representation.
- [ ] O07-O12: Profile and qualify caller-owned stream buffers, typed reusable directory task slots,
batch ownership, bounded result reuse, active-worker dispatch and direct production evaluation.
Preserve existing stage policy while isolating task storage and lifecycle costs.
Expand Down
Binary file not shown.
Binary file not shown.
Binary file not shown.
92 changes: 92 additions & 0 deletions docs/benchmarks/lexical-field-views-20261102.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Borrowed lexical field components (2026-11-02)

## Decision

Use the existing `StringOrView` return type to borrow filename, stem, final extension, all suffixes and core slices directly from the input. Apply the same helpers to path qualifiers and language/MIME filename lookup. Keep directory rendering through native `parent_path`: repeated-separator normalization is observable and does not always correspond to one input slice. No new path wrapper, per-entry cache, thread, ownership hierarchy, flag or dependency is introduced.

The input context owns or borrows a stable path for the duration of rendering. Compiled templates append borrowed values while that context and any owned transformed field value remain alive. Directory results remain owned when native path rendering constructs them. Backslashes remain ordinary filename bytes on both supported POSIX hosts. Embedded NULs remain length-delimited input, not C-string terminators.

## Measurement contract

Reference main `2b21790502fd6bf9b6e1afb53cc1444cf140f296` includes collection text sharing and prepared globs. Only `fields.cc` differs between runtime variants; both use the same test-only kernel source and build configuration. No unmerged fuzzy change is included. Source/binary identities and scripts are retained below.

Release LLVM 23 + ThinLTO on the local M5 Pro (warm APFS, requested 1/3/10 workers without affinity) and drogon Ryzen 9950X KVM guest (tmpfs, child affinity 1/3/10). Builds and tests finish before timing. Three sessions, discarded native warmups, nine interleaved rounds, fastest seven. Every fresh command checks exact complete output against an independent filename/component/directory oracle. Counts 10/1,000/10,000/100,000 with one/three workers plus 100,000 with ten: 4,320 invocations across both hosts including warmups. Commands use explicit `--template`, `--exact`, `--sort=dir`, UTC and no config/pager.

The in-memory kernel compiles templates before timing and renders 10/1,000/100,000 paths with short (8-byte) or long (160-byte) directory/filename components. It measures one field, six direct fields, six path qualifiers and an unchanged directory control. Nine repetitions retain seven, in three shuffled sessions. Path construction and template compilation are excluded from this steady-state kernel, so no preparation/startup improvement is claimed. Native commands include those real costs.

Tables show median session times and median within-session ratios. Negative means faster. Absolute times across hosts are not interchangeable.

## Native 100,000-file output

| Host | Template | Workers | Baseline ms | Candidate ms | Change |
| ----- | ----------- | ------: | ----------: | -----------: | ------: |
| macos | name | 1 | 228.628 | 225.766 | -1.04% |
| macos | components | 1 | 264.324 | 242.042 | -8.54% |
| macos | qualified | 1 | 263.991 | 241.427 | -8.54% |
| macos | dir-control | 1 | 241.864 | 242.050 | +0.32% |
| macos | name | 3 | 104.010 | 104.073 | -0.41% |
| macos | components | 3 | 105.073 | 106.703 | +0.95% |
| macos | qualified | 3 | 104.237 | 105.056 | +0.79% |
| macos | dir-control | 3 | 104.911 | 104.453 | -0.90% |
| macos | name | 10 | 74.524 | 74.184 | -0.11% |
| macos | components | 10 | 82.081 | 76.808 | -6.42% |
| macos | qualified | 10 | 82.977 | 76.849 | -4.75% |
| macos | dir-control | 10 | 74.542 | 72.319 | -1.33% |
| linux | name | 1 | 102.180 | 100.281 | -1.75% |
| linux | components | 1 | 121.704 | 108.422 | -10.91% |
| linux | qualified | 1 | 119.912 | 108.021 | -9.59% |
| linux | dir-control | 1 | 103.248 | 104.082 | +0.02% |
| linux | name | 3 | 36.557 | 35.777 | -2.13% |
| linux | components | 3 | 49.582 | 40.223 | -18.88% |
| linux | qualified | 3 | 48.022 | 40.219 | -16.20% |
| linux | dir-control | 3 | 37.430 | 37.763 | +0.27% |
| linux | name | 10 | 21.361 | 20.333 | -5.71% |
| linux | components | 10 | 40.752 | 27.924 | -31.24% |
| linux | qualified | 10 | 39.747 | 28.795 | -26.01% |
| linux | dir-control | 10 | 23.296 | 23.781 | +2.08% |

Linux multi-field output improves substantially. On macOS the one/ten-worker multi-field cases improve, while three-worker native results remain near baseline or slightly slower despite clear kernel gains. Traversal, coordination and output remain significant; do not turn a rendering-kernel percentage into a filesystem throughput claim.

## In-memory rendering of 1,000 paths

| Host | Template | Component bytes | Baseline us | Candidate us | Change |
| ----- | ----------- | --------------: | ----------: | -----------: | ------: |
| macos | name | 8 | 274.73 | 238.43 | -14.69% |
| macos | name | 160 | 432.85 | 280.87 | -35.66% |
| macos | components | 8 | 595.93 | 375.69 | -36.67% |
| macos | components | 160 | 1438.05 | 582.34 | -59.97% |
| macos | qualified | 8 | 620.72 | 400.64 | -34.88% |
| macos | qualified | 160 | 1440.45 | 584.73 | -59.59% |
| macos | dir-control | 8 | 283.60 | 281.77 | -0.64% |
| macos | dir-control | 160 | 507.00 | 510.62 | -0.15% |
| linux | name | 8 | 60.00 | 42.95 | -28.34% |
| linux | name | 160 | 117.48 | 93.27 | -20.05% |
| linux | components | 8 | 244.42 | 111.72 | -54.23% |
| linux | components | 160 | 515.83 | 381.12 | -26.11% |
| linux | qualified | 8 | 227.12 | 119.02 | -47.55% |
| linux | qualified | 160 | 544.03 | 378.62 | -30.40% |
| linux | dir-control | 8 | 62.68 | 64.07 | +2.43% |
| linux | dir-control | 160 | 149.16 | 150.91 | +1.12% |

## Costs and counterexamples

- macos: slowest paired native median +1.74% at (10000, 3, 'name'); all small and directory-control samples remain in the evidence.
- macos: build-output executable 4,069,152 -> 4,068,752 bytes (-400), before release stripping/compression.
- linux: slowest paired native median +2.08% at (100000, 10, 'dir-control'); all small and directory-control samples remain in the evidence.
- linux: build-output executable 12,306,928 -> 12,287,256 bytes (-19,672), before release stripping/compression.

This removes temporary path/component construction, not final output storage. Long outputs still allocate the returned result string; directory fields retain their previous native path work. No claim of zero allocations for whole template rendering is made. There is no additional retained per-entry state.

## Correctness and verification

Differential tests compare every direct field and its path qualifier against native `std::filesystem::path` lexical results on both hosts: empty/root paths, repeated/trailing separators, dot/dot-dot, hidden names, trailing dots, compound suffixes, backslashes, virtual member names, embedded NULs, nonterminated slices and long components. Combined templates, owned definition inputs and chained transforms retain their lifetimes. Each C++ test source has its own test rule. The existing field and fuzz suites also pass. Complete release/extension suites pass 305/305 on macOS and Linux (with required working FUSE).
The added long-owned-parent lifetime case passes separately on both hosts. Scoped clang-tidy passes
the runtime, differential test and benchmark. Its benchmark-only unused loop token required a
`[[maybe_unused]]` annotation; the final executable code section is byte-identical to the measured
kernel on both hosts. Final normal hooks and published-head CI remain required.

## Lossless raw evidence

- [lexical-field-views-20261102-macos.json.gz](lexical-field-views-20261102-macos.json.gz): 115,070 bytes; SHA-256 `f3040e3bff4fb49a6c9d1b6716b9818e98f48c1742976fe5db56ec99be8b2bc7`.
- [lexical-field-views-20261102-linux.json.gz](lexical-field-views-20261102-linux.json.gz): 114,380 bytes; SHA-256 `6b2de8b3e8bd56c90770384a22aa8ad994afca87a7de6a300df852ae8daaf984`.
- [lexical-field-views-20261102-provenance.json.gz](lexical-field-views-20261102-provenance.json.gz): 47,600 bytes; SHA-256 `99581677c57cec0949ed9ad4140ed49edacca6dc28faed7e2d6090cb70326fc1`.
24 changes: 22 additions & 2 deletions docs/performance-analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -2918,8 +2918,8 @@ virtual layers and generic executor hierarchies need an actual lifetime or exten
| O21 | Reuse similarity representations | Repeated shingle generation and pairwise verification dominate large document sets | Existing near-duplicate design | Cache reference shingles; exact verification after indexing |
| O22 | Reduce output construction churn | Numeric/cell/JSON strings, row insertion and aligned-table retention allocate | Existing output/scope contracts | Append into reusable row buffers; stream machine formats |

All elapsed improvements in this table except O01 are hypotheses, not measured gains. Allocation
sites are inspected code facts; an allocation's wall-time significance still requires profiling.
O01-O03 have qualified cross-platform reports below. Other entries remain hypotheses until their
individual report records a decision; inspected allocation sites alone do not establish elapsed gains.

### Small changes first

Expand Down Expand Up @@ -3130,3 +3130,23 @@ Linux. These are component/whole-engine fixtures, not filesystem throughput clai
and scoring input; bare/control cases stay near baseline. Scored kernels improve 29-54%, while
unrelated bare kernels include 3-4% slower cases and preparation is slightly slower. Keep these
counterexamples and all raw data in the report; the result is not a claim about every fuzzy model.

## O04 borrowed lexical field components

The [cross-platform field experiment](benchmarks/lexical-field-views-20261102.md) reuses
`StringOrView` for filename, stem, final/all suffixes and core fields, their path qualifiers and
language/MIME filename lookup. Native directory spelling remains unchanged; repeated separators
make parent rendering observably different from a raw slice. No new cache or path wrapper is needed.

Differential field and fuzz tests pass on both hosts. The six-field in-memory kernel improves
26-60% depending on host/path length; 100,000-file native output improves about 9-11% with one
worker on Linux and 8.5% on macOS. Linux three/ten-worker cases improve further, while macOS
three-worker native results remain near baseline or slightly slower. Preserve small regressions,
the unchanged directory control and all source/binary identities; do not generalize kernel gains
into filesystem speedups. Complete release/extension suites and scoped lint remain required.

O05 should also test the existing ownership boundary before changing every map declaration:
`Template::Segment` already owns its compiled key. Passing that key to field callbacks by const
string reference can avoid the temporary lookup string in definition/capture rendering without
adding another map, interner or lookup layer. General heterogeneous maps remain useful where the
caller genuinely has only a view; distinguish those callers from an avoidable view conversion.
3 changes: 2 additions & 1 deletion release-site.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@
"docs/performance-analysis.md": "docs/performance-analysis.html",
"docs/design-expression-program.md": "docs/design-expression-program.html",
"docs/expression-qualification-audit.md": "docs/expression-qualification-audit.html",
"docs/benchmarks/fuzzy-score-workspace-20261102.md": "docs/benchmarks/fuzzy-score-workspace-20261102.html"
"docs/benchmarks/fuzzy-score-workspace-20261102.md": "docs/benchmarks/fuzzy-score-workspace-20261102.html",
"docs/benchmarks/lexical-field-views-20261102.md": "docs/benchmarks/lexical-field-views-20261102.html"
},
"generated_html": {
"XFF.md": {
Expand Down
27 changes: 26 additions & 1 deletion xff/presentation/fields/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# limitations under the License.

load("@rules_fuzzing//fuzzing:cc_defs.bzl", "cc_fuzz_test")
load("//xff:cc.bzl", "cc_library", "cc_test")
load("//xff:cc.bzl", "cc_binary", "cc_library", "cc_test")

package(default_visibility = ["//visibility:private"])

Expand Down Expand Up @@ -59,6 +59,19 @@ cc_test(
],
)

cc_test(
name = "lexical_fields_test",
size = "small",
srcs = ["lexical_fields_test.cc"],
deps = [
":fields_cc",
"//xff/vfs:vfs_cc",
"@googletest//:gtest",
"@googletest//:gtest_main",
"@mboworks_mbo//mbo/testing:matchers_cc",
],
)

# Compile and render arbitrary field templates against a deterministic context. The driver checks
# the context-free extraction classification against the actual extraction API.
cc_fuzz_test(
Expand All @@ -76,3 +89,15 @@ cc_fuzz_test(
"@abseil-cpp//absl/time",
],
)

# Test-only compiled template kernel; inputs are in memory and never read the host filesystem.
cc_binary(
name = "fields_benchmark",
testonly = True,
srcs = ["fields_benchmark.cc"],
deps = [
":fields_cc",
"//xff/vfs:vfs_cc",
"@com_github_google_benchmark//:benchmark",
],
)
Loading
Loading