Record archive insertion qualification and reconcile tracking

.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
     hooks:
       - id: check-added-large-files
         # Retain the exact lossless cross-platform performance evidence archives.
-        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107)\.json\.gz$
+        exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag|forward-cursor-skipping|raw-record-reuse|similarity-prefix-view|similarity-reference-reuse|owned-word-moves|smaller-shingle-probes|native-zip-seek-evaluation|encoded-record-reuse|threshold-aware-similarity)-20261103|(column-row-reuse|word-set-insertion|field-context-owned-results|selected-summary-output|fixed-summary-metrics|direct-summary-cells|compact-comparison-entries|comparison-category-sequence|native-name-paths|uniform-native-case|compact-traversal-metadata|eager-native-paths|native-case-experiments)-20261104|(folded-ascii-globs|bulk-directory-metadata|comparison-retention-followups|comparison-storage-followups|archive-prefix-reuse|archive-directory-range|archive-directory-allocation|archive-directory-observations|native-symlink-observations|native-metadata-followups|symlink-target-demand|native-pgo|readlink-short-target|cctz-empty-format|readlink-input-path|readlink-owned-transfer|archive-lookup-trials|archive-member-keys)-20261105|archive-directory-child-capacity-20261106|shared-stage-workers-20261107|archive-index-insertion-20261107)\.json\.gz$
       - id: check-merge-conflict
       - id: check-yaml
       # BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with

TODO.md

@@ -1438,20 +1438,25 @@ this as intentional accounting.
       Actual release assembly retains the dynamic mode branch, but its removal fails to
       establish a consistent cross-platform standalone benefit. Preserve both complete screens,
       copied-parent drift and exact source/image identities; do not adopt the private template.
-- [ ] O27: Independently measure archive-index insertion changes: defer an owned ancestor key
-      until transparent lookup proves absence; compare lower_bound plus emplace_hint to the
-      repeated search, and compare direct mapped-value construction separately. Preserve
-      duplicate overwrite, synthesis flags and inode order. Construction/destruction benchmarks
-      must include and name actual public archive parsing/source costs, not hot-listing setup.
+- [x] O27a: Evaluate deferred archive ancestor-key ownership. Complete 28-construction,
+      82-command and 88-listing trials on both hosts did not establish an attributable
+      cross-platform whole-command benefit. Retain the baseline and the original raw evidence.
+- [x] O27b: Screen direct mapped-node construction on current main. All 28 fixture oracles and
+      focused tests pass on both hosts, but the screen does not justify adoption or full timing.
+      Both decisions and the lossless evidence are in the [O27 report](docs/benchmarks/archive-index-insertion-20261107.md).
+- [ ] O27c: Independently measure transparent lower_bound plus hinted ancestor insertion.
+      Preserve duplicate overwrite, synthesis flags, child counts and inode order. Public
+      construction/destruction benchmarks include parsing/source costs; hot-listing setup does not.
 - [x] O29: [Qualify borrowed archive member keys and bounded parent-length checks](docs/benchmarks/archive-member-keys-20261105.md).
       Both full 333-target release suites, relevant sanitizers, actual changed-TU lint, exact
       ordinary release images and 15/15 lines plus 2/2 branches pass. All 14 targeted operations
       improve in each of three sessions on both hosts; whole commands remain mostly modest.
       Retain the Linux large-deep-root listing cost around 5%, the conflicting Mac full/subset
       populations, and every earlier rejected variant. PR/current-base CI remains separate.
-- [ ] O30: Move the owned OpenBytes input into its memory source instead of copying the complete
+- [x] O30: Move the owned OpenBytes input into its memory source instead of copying the complete
       buffer before streaming. TAR/PHAR correctness and full focused suites pass on macOS/Linux;
       six complete macOS pairs and three Linux pairs show large-input wins, with about 64 MiB lower
       peak RSS at 64 MiB member payload. Full protocol and raw evidence: [O30 report](docs/benchmarks/openbytes-owned-input-20261106.md).
-      Sync to current main, create the PR, and complete changed-file lint plus all required CI before
-      marking this merged candidate complete. The initial noisy macOS screen remains in the artifact.
+      Implemented in PR #1122; exact post-merge main Test run 42276661524 passed both operating
+      systems, sanitizers, coverage and changed-file lint. The initial noisy macOS screen remains
+      in the artifact.

docs/benchmarks/archive-index-insertion-20261107.md

@@ -0,0 +1,100 @@
+# Archive index insertion qualification, 2026-11-07
+
+Keep the existing archive index insertion code. Delaying ancestor-key ownership was
+not selected after its complete earlier trials; direct mapped-node construction also
+does not justify adoption in a new screen on current main. The remaining hinted
+ancestor-insertion idea is separate and unmeasured.
+
+The [lossless evidence archive](archive-index-insertion-20261107.json.gz) retains the
+new source patch, both participants' source and image identities, every screen
+observation, the analysis drivers and build/test logs. It also embeds the original
+Linux and Mac evidence archives for the earlier deferred-ownership experiment.
+Their different parents and protocols remain separate; do not combine their numbers.
+The combined archive is 4,525,832 bytes, SHA-256
+`ef0f035c7369a11f8191f9b46407b6682cfb93ce1190e70743172fd0ed360527`.
+Its embedded earlier archives retain their original compressed bytes and hashes.
+
+## Direct node construction
+
+Both participants start at main `9bf1aad510d843d1d6338fbdf23107a30d339730`
+(#1125). The candidate changes only `ArchiveFileSystem::Index`:
+
+- Construct a new member's mapped `Node` directly through `try_emplace`.
+- On an existing key, retain the original overwrite and direct-child-count preservation.
+  A failed `try_emplace` does not move from its value argument.
+- Construct a proven-missing synthesized ancestor directly through `try_emplace`.
+
+The original ancestor-key ownership, lookup count, `std::map`, inode order and source
+ownership remain unchanged. This is a distinct mechanism from the rejected borrowed
+ancestor lookup. It removes source-level default construction followed by assignment;
+that fact alone does not prove that the optimizer emitted fewer costly instructions.
+
+Both hosts build with `--config=clang_release --config=xff_full`. Frozen images are
+recorded before measurement. Linux uses one allowed logical CPU; Mac runs unpinned.
+No build, sanitizer or lint runs alongside a timing population on the same host.
+
+The existing public construction benchmark checks all fixture paths, file types,
+metadata and directory counts before timing. It measures complete `OpenSource`
+construction and destruction, including archive parsing and member materialization.
+Fixture generation and source ownership stay outside timing; this is not a private
+index-only benchmark. Its 28 cases cover flat, short/long wide and deep parents, and
+chains. Counts are 10, 100, 1,000, 10,000 and 100,000, with chains capped at 1,000.
+
+The predeclared screen uses one session, three repetitions per case and participant,
+and the mean of the fastest two. Participants run baseline, candidate and a second
+copy of the exact baseline image, with randomized case interleaving inside each
+invocation. The table gives the median of case percentage changes within each shape;
+it is not aggregate throughput. Negative is faster. All 28 individual results and
+every raw repetition are retained.
+
+| Shape      | Mac candidate | Mac baseline repeat | Linux candidate | Linux baseline repeat |
+| :--------- | ------------: | ------------------: | --------------: | --------------------: |
+| Flat       |        +0.70% |              +0.99% |          +2.34% |                -1.64% |
+| Wide short |        -0.59% |              -0.60% |          +2.10% |                -0.22% |
+| Wide long  |        +0.73% |              -0.32% |          +0.70% |                +0.02% |
+| Deep short |        -1.74% |              -2.69% |          +0.76% |                -0.45% |
+| Deep long  |        +0.43% |              -0.27% |          +1.11% |                -0.76% |
+| Chain      |        -5.56% |              -5.82% |          +0.60% |                -0.38% |
+
+No Linux shape improves at this estimator. Mac's small apparent gains mostly track
+its unchanged-image control. Its tiny cases also have large disagreement: wide-short
+10 is +66.79% while the repeat baseline is -23.65%; deep-long 10 is +48.78% with a
++40.10% repeat. Retain those observations without assigning them an unmeasured cause.
+At larger counts the candidate has no convincing independent gain on both hosts.
+
+The screen fails its predeclared requirement for an attributable cross-platform
+improvement. Stop here: no full three-session timing population, full-root test suite,
+new sanitizer qualification or runtime PR is claimed for this candidate. Both hosts
+passed the three focused optimized archive targets (`archive_fs_test`,
+`archive_directory_metadata_test`, `archive_openbytes_ownership_test`), and all screen
+fixture oracles passed. These results justify retaining the baseline, not a claim
+that the candidate can never win on any machine.
+
+## Earlier deferred ancestor ownership
+
+The earlier contains-first candidate delayed constructing an owned ancestor key until
+a transparent lookup proved it missing. It completed all 28 public construction,
+82 native command and 88 hot-listing cases on each host, in three balanced sessions
+with nine repetitions retaining seven and an unchanged-parent image control.
+Its own final review found modest lifecycle gains without an attributable
+cross-platform whole-command benefit beyond control variation. It remained unselected.
+
+That experiment's exact parent, source variants, raw populations and narrower
+validation claims are retained in its original embedded archives and plan. They are
+not replaced by the new direct-construction screen. A duplicate borrowed-key prototype
+was identified before any new build or measurement and was not rerun.
+
+## Remaining work
+
+A transparent `lower_bound` plus hinted insertion can avoid a second search for a
+missing ancestor. It remains an independent candidate. Preserve duplicate replacement,
+synthesized-directory state, child counts and inode order; returned sources and map
+keys must retain their existing ownership. A future B-tree layout still needs its
+own reference-invalidation and construction/RSS audit.
+
+O30's owned archive-input transfer is already implemented by #1122. Its exact merged
+commit `0e95e191ba7d7ee3769217e7fbf82a825ff9b90d` passed
+[post-merge main Test run 42276661524](https://github.com/mboworks/xff/actions/runs/42276661524),
+including both operating systems, sanitizers, coverage and changed-file lint. Its
+stale create-a-PR TODO is closed; the initial noisy Mac observations remain in O30's
+existing raw evidence.

docs/performance-analysis.md

@@ -4640,19 +4640,25 @@ its percentages to another change or treat an eliminated branch as a measured im
 
 ### Further independent archive allocation candidates
 
-O27 examines construction work that the existing hot-listing benchmark explicitly excludes.
-The current index constructs an owned long ancestor key before a transparent lookup even when
-that ancestor already exists. First defer that construction until absence is established. A
-separate lower_bound/emplace_hint variant can avoid another ancestor search; separately compare
-member insert_or_assign and known-missing-parent try_emplace to default mapped construction.
-Keep duplicate overwrite, synthesized flags, ancestor break behavior and inode assignment order.
-
-Measure the real public archive construction lifecycle, including destruction, and name any
-parsing/source-copy cost included in that path. Do not introduce a benchmark-only public/friend
-API or pretend an untimed setup benchmark measures indexing. Preserve flat/wide/deep/chain
-construction, complete command/listing controls, tiny cases, allocation/RSS limits and both hosts.
-Do not retain map key views across a future B-tree insertion: relocated small-string keys can
-invalidate them. These insertion prototypes remain uninstalled and unqualified.
+[The O27 insertion report](benchmarks/archive-index-insertion-20261107.md) retains two
+unselected mechanisms separately. Deferred ancestor-key ownership completed the original
+28-construction, 82-command and 88-listing trials on both hosts, without an attributable
+cross-platform whole-command gain. Direct mapped-node construction was then screened on main
+`9bf1aad5`: all 28 public construction fixture oracles and three focused optimized archive
+suite targets pass on each host, but no Linux shape median improves. Mac's small apparent
+wins mostly track its unchanged-image control, with large tiny-case disagreement. Stop at
+that screen rather than spending full qualification cycles or repeating until favorable.
+
+Both original source/image identities, every raw observation, exact invocations and the
+previous experiment's complete archives remain available. The source-level elimination of
+an initialization/assignment sequence is not evidence of fewer costly optimized instructions.
+No production index, CLI, comparator, safety setting or ownership behavior changes.
+
+Transparent `lower_bound` plus hinted ancestor insertion remains an independent unmeasured
+candidate. Preserve duplicate replacement, synthesized flags, child counts and inode order.
+Measure public construction and destruction including parsing, not only hot listings whose
+index setup is untimed. A future B-tree audit still must account for relocated small-string
+map keys and escaping references.
 
 O29 examines borrowed normalized member-key views for immediate stat/access/readlink/cache
 lookups. Normalization only selects a contiguous input subview. Stored index keys and both
@@ -4736,3 +4742,11 @@ and correctness evidence](benchmarks/shared-stage-workers-20261107.md). The nest
 mounted-walk test also demonstrates why a run-wide worker ceiling cannot silently
 replace independent useful stage allowances. O18 and the wider performance goal
 remain open; no production worker policy changes follow from these experiments.
+
+### O30 tracking reconciliation
+
+The owned `OpenBytes` transfer is implemented by PR #1122, merge `0e95e191ba`.
+Its exact post-merge main Test run 42276661524 passed both operating systems, ASan, MSan,
+TSan, GCC, coverage, fuzzing, changed-file lint and policy checks. Close the stale create-PR
+instruction without removing the earlier noisy Mac populations or extending private benchmark
+image claims to a different public integration image.

release-site.json

@@ -133,7 +133,8 @@
     "docs/benchmarks/archive-observation-mode-20261105.md": "docs/benchmarks/archive-observation-mode-20261105.html",
     "docs/benchmarks/cli-stream-sync-audit-20261105.md": "docs/benchmarks/cli-stream-sync-audit-20261105.html",
     "docs/benchmarks/archive-member-keys-20261105.md": "docs/benchmarks/archive-member-keys-20261105.html",
-    "docs/benchmarks/shared-stage-workers-20261107.md": "docs/benchmarks/shared-stage-workers-20261107.html"
+    "docs/benchmarks/shared-stage-workers-20261107.md": "docs/benchmarks/shared-stage-workers-20261107.html",
+    "docs/benchmarks/archive-index-insertion-20261107.md": "docs/benchmarks/archive-index-insertion-20261107.html"
   },
   "generated_html": {
     "XFF.md": {