Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ repos:
hooks:
- id: check-added-large-files
# Retain the exact lossless cross-platform performance evidence archives.
exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107|transactional-rg-records-20261107|immutable-zip-locator-evaluation-20261107|queued-read-cancellation-20261107|archive-btree-historical-decision-20261107|compiled-field-stat-demand-20261107|unchanged-site-payloads-20261107|archive-read-buffer-clear-20261107|archive-read-buffer-requalification-20261108|language-list-reserve-20261107|language-list-reserve-requalification-20261108|ordered-output-credit-20261107|adaptive-ordered-output-20261109)\.json\.gz$
exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107|transactional-rg-records-20261107|immutable-zip-locator-evaluation-20261107|queued-read-cancellation-20261107|archive-btree-historical-decision-20261107|compiled-field-stat-demand-20261107|unchanged-site-payloads-20261107|archive-read-buffer-clear-20261107|archive-read-buffer-requalification-20261108|language-list-reserve-20261107|language-list-reserve-requalification-20261108|ordered-output-credit-20261107|adaptive-ordered-output-20261109|direct-line-scan-spans-20261109-(linux|macos))\.json\.gz$
- id: check-merge-conflict
- id: check-yaml
# BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with
Expand Down
42 changes: 31 additions & 11 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,12 @@ regresses 3-11% across the two sessions. Keep this tradeoff visible when compari

## Toolchain follow-up

- [ ] Fix explicit Linux `--config=clang_opt --config=lto`: the configured action
combines Bazel split ThinLTO indexing with `-ffat-lto-objects`, and the compiler
does not emit `word.indexing.o`. The scanner qualification retains the failed
action and succeeds with shipping `xff_production_lto`. Keep shipping object
sharing intact and verify feature/configuration order before a separate fix.

- [ ] Complete the Clang/LLVM 23.1.2 upgrade: validate native macOS linking, Linux LTO, the
LLVM 22 MSan compiler/runtime override, matching formatting, full CI, benchmark results,
and stripped binary sizes before merging the separate follow-up to #936.
Expand Down Expand Up @@ -511,12 +517,15 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
keep the specialized pools and retain [all variants and counterexamples](docs/benchmarks/shared-stage-workers-20261107.md).
Future reuse designs must preserve independent root/mount allowances, leaf jobs, owner
draining and JC gates, and remove the repeatable Linux comparison regression before adoption.
- [ ] Qualify direct span reads in `content::ScanLines` separately from O18. Native
profiling attributes 43.94% of sampled baseline CPU to zeroing beneath its owning
64 KiB read loop. Reuse `ReadInto` only with initialized-range, binary/EOF/error
tests and native/memory/archive controls; retain O07's rejected variants rather
than assuming removal of zeroing is always a win. See the current O18 qualification
notes in [the performance analysis](docs/performance-analysis.md).
- [x] Qualify direct span reads in `content::ScanLines` separately from O18. Retain
the distinct 43.94% and current 40.81% sampled CPU zeroing profiles; neither
predicts elapsed gains. Compare stack and lazy overwrite heap storage with
native/memory/archive and incapable-stream controls, preserving initialized
prefixes, changing capabilities, EOF, binary detection and late errors.
Both-host complete native/engine/kernel/image-control qualification selects stack
storage, with measured legacy and high-worker costs retained. Keep O07's
rejected variants and every adverse result in
[the scanner report](docs/benchmarks/direct-line-scan-spans-20261109.md).
- [ ] O18: Qualify byte-bounded overlap with backpressure, ordered publication and cancellation;
avoid per-batch throwaway threads and nested waits on the same saturated pool.
Retain the [ordered-output credit experiment](docs/benchmarks/ordered-output-credit-20261107.md).
Expand All @@ -526,15 +535,19 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
Retain ordinary-path costs and anomalous Linux populations alongside large-output gains.
Keep full source/decoder/derived-cache/renderer-temporary admission and broader JC gates
open; output credit is not a whole-process cap. No public crossover setting is exposed.
- [ ] Benchmark qualification: independent CLI copies must preserve `xff`/`xff_full`
basenames, because invocation names select configuration. Retain earlier controls
with different names, disclose that limitation and revisit conclusions depending
only on those controls. The scanner qualification adds a matching-basename cohort.
- [ ] Benchmark qualification: distinguish repeated execution of one image from independent
file-backed baseline copies. Retain both controls when investigating unexplained native
costs, and compare loaded code/data identities before repeating full populations for a
build-ID-only change. Preserve all observations rather than choosing a favorable cohort.
- [ ] Qualify delayed, bounded stdout buffering independently of broader O18 admission.
PR #1156 retains full/minimal native production measurements on both hosts and all
adverse controls in [the report](docs/benchmarks/bounded-stdout-buffer-20261109.md).
Complete the stream-contract coverage follow-up and current-head verification before
merging; a 4 KiB stdout buffer does not complete source/decoder/derived-cache budgets.
- [x] Qualify delayed, bounded stdout buffering independently of broader O18 admission.
Merged PR #1156 retains full/minimal native production measurements on both hosts
and adverse controls in [the report](docs/benchmarks/bounded-stdout-buffer-20261109.md).
The 16-case stream-contract test, current-head checks and post-merge Test/Benchmarks
pass; a 4 KiB stdout buffer does not complete source/decoder/derived-cache budgets.
- [ ] Keep the performance-analysis index below the Markdown service request limit before
appending further findings. Retain full results and sources in linked qualification
reports; consider local rendering or independently renderable sections for further growth.
Expand Down Expand Up @@ -598,6 +611,13 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
separate hardware diagnostics are retained in
[the allocation decisions](docs/benchmarks/archive-directory-allocation-20261105.md);
microbenchmark gains alone do not qualify either runtime implementation.
- [ ] O19g3: Prepare the requested member's normalized name once per streamed open,
preserving null-header, normalization, error and duplicate-selection behavior. The
current `MemberSource::Open` repeats `NormalizeMemberName(member_)` while walking
preceding headers. Retain cold/first-member/missing-member controls and measure on
both hosts before adoption. The scanner qualification also exposes roughly eight
seconds for the broad 10,000-member, single-worker TAR invocation; reuse observed plain-TAR payload locators
as a separate, format-validated O19b experiment rather than bypass decoder checks.
- [x] O19i: Bound initial archive-directory result storage to reduce small-listing vector
reallocations, retaining tiny/root/chain costs and complete native controls. Full both-host
qualification and the held O19g variants are in
Expand Down
1 change: 1 addition & 0 deletions bazelmod/clang_tidy_targets.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ CLANG_TIDY_MANUAL_TARGETS = [
"//tools:read_source_benchmark",
"//tools:time_format_benchmark",
"//xff/cli:globals_benchmark",
"//xff/content:line_scan_benchmark",
"//xff/engine:read_accounting_test",
"//xff/engine:compare_benchmark",
"//xff/engine:executor_benchmark",
Expand Down
Binary file not shown.
Binary file not shown.
Loading
Loading