Skip to content
8 changes: 5 additions & 3 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -1453,9 +1453,11 @@ this as intentional accounting.
- [x] O27b: Screen direct mapped-node construction on current main. All 28 fixture oracles and
focused tests pass on both hosts, but the screen does not justify adoption or full timing.
Both decisions and the lossless evidence are in the [O27 report](docs/benchmarks/archive-index-insertion-20261107.md).
- [ ] O27c: Independently measure transparent lower_bound plus hinted ancestor insertion.
Preserve duplicate overwrite, synthesis flags, child counts and inode order. Public
construction/destruction benchmarks include parsing/source costs; hot-listing setup does not.
- [x] O27c: Screen transparent lower_bound plus hinted archive ancestor insertion on both hosts.
All 28 construction oracles and focused tests pass. Linux shape medians improve 2.89-4.35%,
but Mac flat/100,000 costs 13.97% with same-image repeat -0.79%; do not adopt the runtime.
Retain the isolated prototype and every raw case in the [hinted-insertion report](docs/benchmarks/archive-ancestor-hint-20261107.md).
All three requested O27 mechanisms are now evaluated; further index work needs a new hypothesis.
- [x] O29: [Qualify borrowed archive member keys and bounded parent-length checks](docs/benchmarks/archive-member-keys-20261105.md).
Both full 333-target release suites, relevant sanitizers, actual changed-TU lint, exact
ordinary release images and 15/15 lines plus 2/2 branches pass. All 14 targeted operations
Expand Down
Binary file not shown.
80 changes: 80 additions & 0 deletions docs/benchmarks/archive-ancestor-hint-20261107.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Archive ancestor insertion hints, 2026-11-07

Keep the production index unchanged. Reusing an ordered ancestor lookup for hinted
insertion improves the Linux constructor screen, but Mac results are mixed and include
a substantial unchanged-flat-path cost. This trial does not qualify portable adoption.

## Isolated mechanism

The candidate retains owned ancestor keys, default mapped-node construction and its
subsequent assignment. Member insertion, duplicate overwrite, direct-child counts,
synthesis flags and inode order stay unchanged. Only the missing-ancestor search
changes: `lower_bound` supplies the insertion position to `try_emplace`, avoiding a
second ordered lookup. This differs from the earlier rejected borrowed-key and direct
mapped-value-construction trials; neither change is included here.

A nonempty ancestor is a strict prefix of the already-stored member. Under the map's
`std::less<>` ordering, that member is greater than the ancestor, so `lower_bound`
cannot return `end()`. The lower bound plus the reverse comparator determines equality.
No other map mutation happens before the hinted insertion.

## Screen and provenance

The unchanged frozen baseline is main `9bf1aad5`, reused byte-for-byte from the preceding
O27 experiment. Candidate/baseline configurations, compiler version (Clang 23.1.4) and
recorded source hashes match except for the intended index implementation. Both use
`clang_release` and `xff_full`. The exact baseline image is repeated as a control.

Each host runs every one of the 28 public construction/destruction cases: flat,
wide-short/wide-long, deep-short/deep-long and chain archives, with 10 through 100,000
members where supported; the chain cases stop at 1,000. This includes real public
archive parsing/source/index/destruction work, not a hot private map lookup. Complete
fixture metadata/path/directory oracles run before timing. Every oracle passes.

This is one initial screen with three raw repetitions per case/participant; retain
the fastest two and average them. The table shows the median case change per shape,
not elapsed-time weighting. Linux is pinned to one allowed CPU; Mac is unpinned.
No host builds or sanitizer checks run concurrently with timing. Positive is slower.

| Shape | Mac candidate | Mac same-image repeat | Linux candidate | Linux same-image repeat |
| :--------- | ------------: | --------------------: | --------------: | ----------------------: |
| flat | +3.46% | -1.80% | -4.35% | +0.38% |
| wide-short | +10.53% | -1.09% | -3.20% | +0.59% |
| wide-long | +2.65% | +0.14% | -2.89% | +0.55% |
| deep-short | +1.98% | -4.83% | -3.93% | +0.35% |
| deep-long | -3.71% | -0.86% | -2.89% | +0.48% |
| chain | -4.40% | -0.74% | -4.28% | -0.88% |

The Mac flat/100,000 case costs 13.97% while the identical baseline repeat changes
-0.79%. The modified ancestor branch does not execute for flat members; no causal
explanation is claimed from source inspection alone. Keep that counterexample and
the noisy tiny cells, including Mac deep-short/10 at +30.66% (repeat -21.61%).
Linux's consistently favorable shape medians do not erase the Mac costs.

## Correctness and decision boundary

Both hosts pass optimized `archive_fs_test`, `archive_directory_metadata_test` and
`archive_openbytes_ownership_test`. The additional in-memory regression checks
synthesized ancestors alongside lexically adjacent member keys and their contents;
existing tests cover explicit/synthesized replacement, duplicate/direct-child counts,
inodes, metadata and retained ownership. Test source, Mac XML and the Linux complete
build/test log are retained. This is focused prototype validation, not a claim of a
full release/sanitizer/current-head CI run.

The first Mac attempt failed before any timing because copied baseline files lacked
executable permissions. Only their mode was corrected, failed setup logs remain,
and the complete corrected screen is separate. No source or measured image bytes
changed as a recovery, and no successful screen was repeated until favorable.

The predeclared next stage required a promising cross-platform screen. That condition
is not met, so no full three-session nine-repeat/retain-seven population, native
82-case CLI/88-case listing controls, sanitizer expansion or runtime PR follows.
Retain the candidate rather than hiding the negative result or changing production
code on a Linux-only improvement.

## Evidence

[`archive-ancestor-hint-20261107.json.gz`](archive-ancestor-hint-20261107.json.gz) is 115,195 bytes, SHA-256
`8359b99d815469270bbdb90aef7ec8dd11f9156ef37d6b7c5a879672badfaad7`. Its 53 exact source/raw/log records include both complete populations,
copy controls, failed setup and per-record byte counts/hashes. Measured executables
remain in the local/remote task directories with image identities recorded.
16 changes: 16 additions & 0 deletions docs/performance-analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -4820,3 +4820,19 @@ Empty valid output, too-small destinations, corrupt data and exact-size mismatch
retain committed coverage. Current integration PR checks remain authoritative. This
small ownership cleanup does not complete the broader immutable-catalog/first-parse
investigation or imply a user-visible command improvement.

### O27c: reject hinted archive ancestor insertion after both-host screen

[The isolated hinted-insertion report](benchmarks/archive-ancestor-hint-20261107.md)
retains owned ancestor keys and mapped-value construction, changing only the ordered
lookup and missing-node insertion hint. All 28 complete construction fixture oracles
and three optimized archive targets pass on both hosts. One three-repetition screen
retaining two improves Linux shape medians 2.89-4.35%, but Mac results are mixed:
wide-short +10.53% and flat/100,000 +13.97%, against same-image repeats -1.09% and -0.79%.
The ancestor branch does not execute for flat members; no causal explanation is claimed.

Stop at the predeclared screen rather than selecting the Linux result or repeating
until favorable. No full three-session population, native/listing expansion or new
full-root/sanitizer qualification is claimed. Preserve both complete populations,
source/image identities, the failed pre-timing executable-mode setup and raw controls.
O27a, O27b and O27c are now separately evaluated without changing the production index.
6 changes: 4 additions & 2 deletions release-site.json
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,8 @@
"docs/benchmarks/shared-stage-workers-20261107.md": "docs/benchmarks/shared-stage-workers-20261107.html",
"docs/benchmarks/archive-index-insertion-20261107.md": "docs/benchmarks/archive-index-insertion-20261107.html",
"docs/benchmarks/transactional-rg-records-20261107.md": "docs/benchmarks/transactional-rg-records-20261107.html",
"docs/benchmarks/language-json-storage-20261107.md": "docs/benchmarks/language-json-storage-20261107.html"
"docs/benchmarks/language-json-storage-20261107.md": "docs/benchmarks/language-json-storage-20261107.html",
"docs/benchmarks/archive-ancestor-hint-20261107.md": "docs/benchmarks/archive-ancestor-hint-20261107.html"
},
"generated_html": {
"XFF.md": {
Expand All @@ -160,6 +161,7 @@
],
"files": {
"docs/benchmarks/transactional-rg-records-20261107.json.gz": "docs/benchmarks/transactional-rg-records-20261107.json.gz",
"docs/benchmarks/language-json-storage-20261107.json.gz": "docs/benchmarks/language-json-storage-20261107.json.gz"
"docs/benchmarks/language-json-storage-20261107.json.gz": "docs/benchmarks/language-json-storage-20261107.json.gz",
"docs/benchmarks/archive-ancestor-hint-20261107.json.gz": "docs/benchmarks/archive-ancestor-hint-20261107.json.gz"
}
}
Loading