diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 537238c439..a060d61283 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -24,7 +24,7 @@ repos: hooks: - id: check-added-large-files # Retain the exact lossless cross-platform performance evidence archives. - exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107)\.json\.gz$ + exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107)\.json\.gz$ - id: check-merge-conflict - id: check-yaml # BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with diff --git a/TODO.md b/TODO.md index 3b49fd4a62..7ad4654e58 100644 --- a/TODO.md +++ b/TODO.md @@ -1438,20 +1438,25 @@ this as intentional accounting. Actual release assembly retains the dynamic mode branch, but its removal fails to establish a consistent cross-platform standalone benefit. Preserve both complete screens, copied-parent drift and exact source/image identities; do not adopt the private template. -- [ ] O27: Independently measure archive-index insertion changes: defer an owned ancestor key - until transparent lookup proves absence; compare lower_bound plus emplace_hint to the - repeated search, and compare direct mapped-value construction separately. Preserve - duplicate overwrite, synthesis flags and inode order. Construction/destruction benchmarks - must include and name actual public archive parsing/source costs, not hot-listing setup. +- [x] O27a: Evaluate deferred archive ancestor-key ownership. Complete 28-construction, + 82-command and 88-listing trials on both hosts did not establish an attributable + cross-platform whole-command benefit. Retain the baseline and the original raw evidence. +- [x] O27b: Screen direct mapped-node construction on current main. All 28 fixture oracles and + focused tests pass on both hosts, but the screen does not justify adoption or full timing. + Both decisions and the lossless evidence are in the [O27 report](docs/benchmarks/archive-index-insertion-20261107.md). +- [ ] O27c: Independently measure transparent lower_bound plus hinted ancestor insertion. + Preserve duplicate overwrite, synthesis flags, child counts and inode order. Public + construction/destruction benchmarks include parsing/source costs; hot-listing setup does not. - [x] O29: [Qualify borrowed archive member keys and bounded parent-length checks](docs/benchmarks/archive-member-keys-20261105.md). Both full 333-target release suites, relevant sanitizers, actual changed-TU lint, exact ordinary release images and 15/15 lines plus 2/2 branches pass. All 14 targeted operations improve in each of three sessions on both hosts; whole commands remain mostly modest. Retain the Linux large-deep-root listing cost around 5%, the conflicting Mac full/subset populations, and every earlier rejected variant. PR/current-base CI remains separate. -- [ ] O30: Move the owned OpenBytes input into its memory source instead of copying the complete +- [x] O30: Move the owned OpenBytes input into its memory source instead of copying the complete buffer before streaming. TAR/PHAR correctness and full focused suites pass on macOS/Linux; six complete macOS pairs and three Linux pairs show large-input wins, with about 64 MiB lower peak RSS at 64 MiB member payload. Full protocol and raw evidence: [O30 report](docs/benchmarks/openbytes-owned-input-20261106.md). - Sync to current main, create the PR, and complete changed-file lint plus all required CI before - marking this merged candidate complete. The initial noisy macOS screen remains in the artifact. + Implemented in PR #1122; exact post-merge main Test run 42276661524 passed both operating + systems, sanitizers, coverage and changed-file lint. The initial noisy macOS screen remains + in the artifact. diff --git a/docs/benchmarks/archive-index-insertion-20261107.json.gz b/docs/benchmarks/archive-index-insertion-20261107.json.gz new file mode 100644 index 0000000000..2db3e2af96 Binary files /dev/null and b/docs/benchmarks/archive-index-insertion-20261107.json.gz differ diff --git a/docs/benchmarks/archive-index-insertion-20261107.md b/docs/benchmarks/archive-index-insertion-20261107.md new file mode 100644 index 0000000000..ea1983999b --- /dev/null +++ b/docs/benchmarks/archive-index-insertion-20261107.md @@ -0,0 +1,100 @@ +# Archive index insertion qualification, 2026-11-07 + +Keep the existing archive index insertion code. Delaying ancestor-key ownership was +not selected after its complete earlier trials; direct mapped-node construction also +does not justify adoption in a new screen on current main. The remaining hinted +ancestor-insertion idea is separate and unmeasured. + +The [lossless evidence archive](archive-index-insertion-20261107.json.gz) retains the +new source patch, both participants' source and image identities, every screen +observation, the analysis drivers and build/test logs. It also embeds the original +Linux and Mac evidence archives for the earlier deferred-ownership experiment. +Their different parents and protocols remain separate; do not combine their numbers. +The combined archive is 4,525,832 bytes, SHA-256 +`ef0f035c7369a11f8191f9b46407b6682cfb93ce1190e70743172fd0ed360527`. +Its embedded earlier archives retain their original compressed bytes and hashes. + +## Direct node construction + +Both participants start at main `9bf1aad510d843d1d6338fbdf23107a30d339730` +(#1125). The candidate changes only `ArchiveFileSystem::Index`: + +- Construct a new member's mapped `Node` directly through `try_emplace`. +- On an existing key, retain the original overwrite and direct-child-count preservation. + A failed `try_emplace` does not move from its value argument. +- Construct a proven-missing synthesized ancestor directly through `try_emplace`. + +The original ancestor-key ownership, lookup count, `std::map`, inode order and source +ownership remain unchanged. This is a distinct mechanism from the rejected borrowed +ancestor lookup. It removes source-level default construction followed by assignment; +that fact alone does not prove that the optimizer emitted fewer costly instructions. + +Both hosts build with `--config=clang_release --config=xff_full`. Frozen images are +recorded before measurement. Linux uses one allowed logical CPU; Mac runs unpinned. +No build, sanitizer or lint runs alongside a timing population on the same host. + +The existing public construction benchmark checks all fixture paths, file types, +metadata and directory counts before timing. It measures complete `OpenSource` +construction and destruction, including archive parsing and member materialization. +Fixture generation and source ownership stay outside timing; this is not a private +index-only benchmark. Its 28 cases cover flat, short/long wide and deep parents, and +chains. Counts are 10, 100, 1,000, 10,000 and 100,000, with chains capped at 1,000. + +The predeclared screen uses one session, three repetitions per case and participant, +and the mean of the fastest two. Participants run baseline, candidate and a second +copy of the exact baseline image, with randomized case interleaving inside each +invocation. The table gives the median of case percentage changes within each shape; +it is not aggregate throughput. Negative is faster. All 28 individual results and +every raw repetition are retained. + +| Shape | Mac candidate | Mac baseline repeat | Linux candidate | Linux baseline repeat | +| :--------- | ------------: | ------------------: | --------------: | --------------------: | +| Flat | +0.70% | +0.99% | +2.34% | -1.64% | +| Wide short | -0.59% | -0.60% | +2.10% | -0.22% | +| Wide long | +0.73% | -0.32% | +0.70% | +0.02% | +| Deep short | -1.74% | -2.69% | +0.76% | -0.45% | +| Deep long | +0.43% | -0.27% | +1.11% | -0.76% | +| Chain | -5.56% | -5.82% | +0.60% | -0.38% | + +No Linux shape improves at this estimator. Mac's small apparent gains mostly track +its unchanged-image control. Its tiny cases also have large disagreement: wide-short +10 is +66.79% while the repeat baseline is -23.65%; deep-long 10 is +48.78% with a ++40.10% repeat. Retain those observations without assigning them an unmeasured cause. +At larger counts the candidate has no convincing independent gain on both hosts. + +The screen fails its predeclared requirement for an attributable cross-platform +improvement. Stop here: no full three-session timing population, full-root test suite, +new sanitizer qualification or runtime PR is claimed for this candidate. Both hosts +passed the three focused optimized archive targets (`archive_fs_test`, +`archive_directory_metadata_test`, `archive_openbytes_ownership_test`), and all screen +fixture oracles passed. These results justify retaining the baseline, not a claim +that the candidate can never win on any machine. + +## Earlier deferred ancestor ownership + +The earlier contains-first candidate delayed constructing an owned ancestor key until +a transparent lookup proved it missing. It completed all 28 public construction, +82 native command and 88 hot-listing cases on each host, in three balanced sessions +with nine repetitions retaining seven and an unchanged-parent image control. +Its own final review found modest lifecycle gains without an attributable +cross-platform whole-command benefit beyond control variation. It remained unselected. + +That experiment's exact parent, source variants, raw populations and narrower +validation claims are retained in its original embedded archives and plan. They are +not replaced by the new direct-construction screen. A duplicate borrowed-key prototype +was identified before any new build or measurement and was not rerun. + +## Remaining work + +A transparent `lower_bound` plus hinted insertion can avoid a second search for a +missing ancestor. It remains an independent candidate. Preserve duplicate replacement, +synthesized-directory state, child counts and inode order; returned sources and map +keys must retain their existing ownership. A future B-tree layout still needs its +own reference-invalidation and construction/RSS audit. + +O30's owned archive-input transfer is already implemented by #1122. Its exact merged +commit `0e95e191ba7d7ee3769217e7fbf82a825ff9b90d` passed +[post-merge main Test run 42276661524](https://github.com/mboworks/xff/actions/runs/42276661524), +including both operating systems, sanitizers, coverage and changed-file lint. Its +stale create-a-PR TODO is closed; the initial noisy Mac observations remain in O30's +existing raw evidence. diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md index 098291d7a9..a90ea21cbd 100644 --- a/docs/performance-analysis.md +++ b/docs/performance-analysis.md @@ -4640,19 +4640,25 @@ its percentages to another change or treat an eliminated branch as a measured im ### Further independent archive allocation candidates -O27 examines construction work that the existing hot-listing benchmark explicitly excludes. -The current index constructs an owned long ancestor key before a transparent lookup even when -that ancestor already exists. First defer that construction until absence is established. A -separate lower_bound/emplace_hint variant can avoid another ancestor search; separately compare -member insert_or_assign and known-missing-parent try_emplace to default mapped construction. -Keep duplicate overwrite, synthesized flags, ancestor break behavior and inode assignment order. - -Measure the real public archive construction lifecycle, including destruction, and name any -parsing/source-copy cost included in that path. Do not introduce a benchmark-only public/friend -API or pretend an untimed setup benchmark measures indexing. Preserve flat/wide/deep/chain -construction, complete command/listing controls, tiny cases, allocation/RSS limits and both hosts. -Do not retain map key views across a future B-tree insertion: relocated small-string keys can -invalidate them. These insertion prototypes remain uninstalled and unqualified. +[The O27 insertion report](benchmarks/archive-index-insertion-20261107.md) retains two +unselected mechanisms separately. Deferred ancestor-key ownership completed the original +28-construction, 82-command and 88-listing trials on both hosts, without an attributable +cross-platform whole-command gain. Direct mapped-node construction was then screened on main +`9bf1aad5`: all 28 public construction fixture oracles and three focused optimized archive +suite targets pass on each host, but no Linux shape median improves. Mac's small apparent +wins mostly track its unchanged-image control, with large tiny-case disagreement. Stop at +that screen rather than spending full qualification cycles or repeating until favorable. + +Both original source/image identities, every raw observation, exact invocations and the +previous experiment's complete archives remain available. The source-level elimination of +an initialization/assignment sequence is not evidence of fewer costly optimized instructions. +No production index, CLI, comparator, safety setting or ownership behavior changes. + +Transparent `lower_bound` plus hinted ancestor insertion remains an independent unmeasured +candidate. Preserve duplicate replacement, synthesized flags, child counts and inode order. +Measure public construction and destruction including parsing, not only hot listings whose +index setup is untimed. A future B-tree audit still must account for relocated small-string +map keys and escaping references. O29 examines borrowed normalized member-key views for immediate stat/access/readlink/cache lookups. Normalization only selects a contiguous input subview. Stored index keys and both @@ -4736,3 +4742,11 @@ and correctness evidence](benchmarks/shared-stage-workers-20261107.md). The nest mounted-walk test also demonstrates why a run-wide worker ceiling cannot silently replace independent useful stage allowances. O18 and the wider performance goal remain open; no production worker policy changes follow from these experiments. + +### O30 tracking reconciliation + +The owned `OpenBytes` transfer is implemented by PR #1122, merge `0e95e191ba`. +Its exact post-merge main Test run 42276661524 passed both operating systems, ASan, MSan, +TSan, GCC, coverage, fuzzing, changed-file lint and policy checks. Close the stale create-PR +instruction without removing the earlier noisy Mac populations or extending private benchmark +image claims to a different public integration image. diff --git a/release-site.json b/release-site.json index b237d45343..26f99e5557 100644 --- a/release-site.json +++ b/release-site.json @@ -133,7 +133,8 @@ "docs/benchmarks/archive-observation-mode-20261105.md": "docs/benchmarks/archive-observation-mode-20261105.html", "docs/benchmarks/cli-stream-sync-audit-20261105.md": "docs/benchmarks/cli-stream-sync-audit-20261105.html", "docs/benchmarks/archive-member-keys-20261105.md": "docs/benchmarks/archive-member-keys-20261105.html", - "docs/benchmarks/shared-stage-workers-20261107.md": "docs/benchmarks/shared-stage-workers-20261107.html" + "docs/benchmarks/shared-stage-workers-20261107.md": "docs/benchmarks/shared-stage-workers-20261107.html", + "docs/benchmarks/archive-index-insertion-20261107.md": "docs/benchmarks/archive-index-insertion-20261107.html" }, "generated_html": { "XFF.md": {