Skip to content

perf: prepare streamed archive member names once - #1158

Merged
helly25 merged 2 commits into
mainfrom
perf/prepare-streamed-member-name-20261109
Nov 9, 2026
Merged

helly25 merged 2 commits into
mainfrom
perf/prepare-streamed-member-name-20261109

Conversation

@helly25

@helly25 helly25 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Prepare an archive member's requested name once per streamed open instead of repeating that work for every preceding header. Preserve lookup semantics and add focused first-member and missing-member performance controls.

AG;DR

Implementation:

  • Hoist the normalized requested string_view into each MemberSource::Open. It borrows that source's owned member name only for the duration of the open; no allocation, lock, retained cache, or lifetime policy changes.
  • Share the narrow value-returning C pathname predicate with the existing opened-archive lookup. Preserve null versus empty names, first matching duplicates, matching directory/symlink rejection, missing-member errors, interior dot segments, case, and source ownership/reopening.
  • Add seven dedicated member-source tests under one new cc_test and extend the existing manual archive benchmark with 16 first/missing controls: 10/10,000 headers, normal/prefixed requests, and reused/fresh wrappers. Fresh means a new wrapper and archive cursor, not cold operating-system caches.
  • Publish complete Mac/Linux data and a qualification report through the existing release-site mapping. Track validated plain-TAR payload locators and host-wide benchmark ownership separately; this does not complete broader archive replay/indexing work.

Production measurements use the exact shipping clang_release/xff_full ThinLTO recipes, RAM-backed trees, one requested worker, an independent output oracle on every invocation, one discarded warmup, and nine measured rounds retaining seven. Main/candidate/independent byte-identical copy rotate once each per round. Both hosts pass all 340 native invocation oracles. The full 10,000-member archive searches change by:

Host Broad Deep
Mac -2.35% -2.09%
Linux -0.6% -1.5%

The unchanged 48-case archive-source kernel has a descriptive last-member geometric change of -3.32% Mac and -4.36% Linux; it is not a weighted deployment workload. Actual configured unstripped full CLI sizes change by zero bytes on Mac and +120 bytes on Linux. Frozen image identities and actual configured artifacts are retained; these are not tagged-release asset sizes.

Tradeoffs and limits:

  • Linux missing-member controls retain group costs of +2.60%/+2.54%, with an individual maximum of +3.45%. This error-path tradeoff is explicit; the change is not a universal speedup.
  • The largest native point regression is Mac broad/1,000 members at +1.42%, accompanied by +1.79% in the copied-main control. Ordinary native listings remain near parity.
  • Original slower/noisy Mac kernel observations are retained. A fixed 16-case confirmation includes every original candidate above +5% or copy outside +/-5%. The +6.74%/+5.63% legacy read points become -11.60%/+0.38%; a +17.09% listing point also changes sign, while another listing retains +2.35%. Neither cohort replaces the other. Copy movement is diagnostic, never a correction divisor or proof of a physical cause.
  • An initial Linux kernel run overlapped independently started benchmarking and remains classified as contended. The full fixed matrix was repeated with overlap checks. An initial native attempt captured zero samples because an MSan compiler appeared; a separate confirmation waited for 60 seconds clear and checked between rounds.
  • Whole Mac baseline hashes differ from the preceding scanner report only in debugger source-directory metadata, UUID/signature consequences. All eleven file-backed sections and symbol values match after debugger-directory normalization. This experiment consistently uses its own baseline; Linux main is byte-identical to the prior shipping image.

Development validation:

  • Both hosts: full configured release suite, 345 targets; valid Bazel reuse only for unchanged declared inputs.
  • Eight affected archive/read-source suites freshly pass ASan and TSan on both hosts, plus MSan on Linux.
  • Three actual archive translation units pass scoped clang-tidy on both hosts. The added benchmark TU also passes on both hosts; normal commit/push hooks remain enabled.
  • First/missing controls validate all expected contents/errors: 16 cases, three rotated blocks, nine repetitions per block, 1,296 iteration rows on each host.
  • All exported native retained means and kernel block means/ratios were independently recomputed from original per-iteration samples. Dataset hashes and source/build provenance are in the report.

These development checks are scoped to their recorded revisions. The signed runtime commit b961a4bf977691f8e76e31b7dcf582cede6596f7 has the same complete tree as qualified c041358fdb6cbf42849f1d88cee2c67af44bf23e after rebasing onto the squash merge of #1157. Later changes add only common benchmark controls, report/data, mappings and tracking. This PR receives its own current-head checks; no previous PR's statuses substitute for them. At the current pushed head 733c6208a6eb043d8caef1c9b4ee9e9f3b2cdb06, both hosts pass all 345 configured release targets (valid unchanged-input cache reuse) and eight freshly executed affected archive/read-source suites; tracked source hashes remain unchanged. Actual configured shipping CLI and new control-helper outputs from this exact head are byte-identical to their measured candidate images on both hosts. The real-API release-site build and link validation pass, and both published data downloads match their recorded hashes. Normal signed commit and pre-push hooks pass. This PR's own current-head workflow is Test 42680343313; all required checks, including the final done aggregate, have completed successfully.

Base: current main 91437fef41f86c7f048dd542e77f21103d4e1268; #1157 is merged. Its post-merge Test and Benchmarks workflows have both completed successfully after recovery from GitHub's November 9 Actions/Checks outage; no source workaround or protection bypass was used.

@helly25
helly25 marked this pull request as ready for review November 9, 2026 20:15
@helly25
helly25 merged commit 81f5d87 into main Nov 9, 2026
34 checks passed
@helly25
helly25 deleted the perf/prepare-streamed-member-name-20261109 branch November 9, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant