Skip to content

Reduce Linux raw filename output overhead - #1154

Merged
helly25 merged 11 commits into
mainfrom
perf/inline-raw-records-20261109
Nov 9, 2026
Merged

helly25 merged 11 commits into
mainfrom
perf/inline-raw-records-20261109

Conversation

@helly25

@helly25 helly25 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Reduce Linux filename-output work with a reusable inline buffer. Qualified production binaries show about 1-1.5% less elapsed time for 100,000-file listings at three workers, and separate counters show about 2.1% fewer userspace instructions. macOS keeps its previous implementation following its qualification.

Refresh the merged-PR coverage inventory and verify that recovered PRs #914, #986 and #1143 load charts on both platforms.

AG;DR

Scope and implementation

This follows merged #1151 (ed883ecc527bc551e778845d054afc27c9b6b024). Current head 1f37b3cc846efa0541396daf84d984f24ed565e5 contains that main commit locally and remotely. The synchronization preserves both release-site page/asset maps. No runtime edits followed the synchronization.

  • Linux RawRecordBuffer has 256 bytes of coordinator-owned inline storage. Initialize only the accepted path prefix and terminator; use the existing bounded string fallback for longer records.
  • Add storage once per buffer rather than per visited entry. Preserve embedded NULs, alternating terminators, oversized-capacity release and synchronous borrowed-consumer lifetime. Queued consumers still copy records.
  • Preserve the previous macOS layout/runtime path after its portable experiment failed to establish a benefit. Express retained capacity as the maximum of active capacities with one shared return, avoiding inactive-platform coverage lines.
  • Add tests immediately below/at/above inline capacity, embedded NULs, alternating terminators and reuse after long records. Preserve existing queued-copy and growth/release coverage.

No flags, benchmark recipe, sample population or viewer contract change. Extended-history integration remains separate in #1153; the missing benchmark-history backlog is not declared complete.

Current-head verification

  • All 343 configured full-feature release targets pass on each host, including the separate extension/API scope. Eleven focused suites execute freshly under Mac ASan/TSan and Linux ASan/TSan/MSan: CPU allowance, output credit, parallel match, rg, run, evaluate, expression execution/program, conformance, record buffer and renderer. Linux MSan uses the documented local linker override, without changing CI.
  • All 781 recorded source/build/tooling hashes match after qualification. Actual lean production link actions contain -flto=thin on both hosts. Mac image: a4d4f82703982e562aa1c8bdb71782c5bd67d4fc9101cb7b861f40d3dbd8b4ab, 4,223,056 bytes. Linux image: 455ee7cf849dafdb19b797165ed7e976018596df1df5ddf8929e818b0cde8578, 12,826,528 bytes.
  • Exact CI coverage configuration executes all 331 targets locally on Linux with FUSE required. The unchanged policy tool, run against original Linux LCOV and the actual synchronized Git diff, passes every category: 17/17 changed lines and 2/2 changed branches. The collector uses Bazel's installed embedded JDK because this host lacks system Java; retain the setup failure before that local correction. Local coverage does not replace required GitHub checks.
  • Normal commit/push hooks passed, including affected-file clang-tidy; no hooks or required checks were bypassed. Main Reduce unused evaluation and shard setup in ordinary walks #1151 Test 42564885437 and Benchmarks 42564885478 both succeeded. Final PR Test run 42566803779, attempt 5, completed successfully on this exact qualified head. All required checks pass, including final done.

Integrated production measurements

Measure the actual frozen lean executables against a main-equivalent #1151 image and an independent copy of that baseline. Stop build servers before timing and verify image hashes afterward. Every invocation checks complete output against an independent oracle, exit status and diagnostics; recompute all retained means.

Both hosts complete 76 ordinary cases, three sessions of nine measured rounds retaining seven, with discarded warmups: 6,840 checked invocations each. Descriptive candidate/control geometric ratios are Linux 0.996617 / 0.998538 and Mac 1.001438 / 0.998967. These population summaries do not establish universal speedups.

Keep every initially slower Linux observation. One selected confirmation includes all geometric candidate costs over 3% or costs positive in all three sessions: twelve cases, three sessions of 27 rounds retaining 21, 3,024 checked invocations. None remains positive in every confirmation session; the largest repeat point estimate is +0.54%, with copy control +0.46%. The original observations remain retained rather than being replaced by the repeat.

The separate Linux listing screen uses 100/100,000 files, broad/deep trees and workers 1/3/10, with three sessions of 27 rounds retaining 21. Rotate baseline/copy/candidate/GNU find; all 4,032 checked invocations pass. All twelve candidate point estimates improve. Large/three-worker geometric change is -1.40%, copied control +0.12%; candidate/find is 1.1005, so this does not claim to beat find. After an EDQUOT setup failure with no accepted timings, remove only the recorded private partial fixture and verify all retained extended-production-v1 fixture contents, metadata and population before reuse. Do not label this recipe as canonical CI data.

Separate post-timing counters cover 54 checked invocations with 100% scheduling: instructions fall 2.17% for broad/single-worker and 2.12% for deep/three-worker at 100,000 files; copy controls are effectively unchanged. Profiled durations never enter elapsed populations.

Final Mac code/data contents and addresses, dyld bindings and function metadata match merged main; only LC_UUID differs. Its integrated ordinary screen uses retained APFS temporary fixtures without affinity, separate from the original RAM-image experiment. No Mac speedup is claimed. Linux pins OS-reported distinct cores without claiming physical-host isolation.

Original experiment and retained evidence

The original report remains frozen at source 90f8c8c0579b1132ea4d0bc714da9f855a77e012, before #1152 integration, against #1151 source 1ea02a41d7dcf25c043e1079c98a9e60d6d73290. It separately records actual full/minimal release images, the five-task 90-case grid, all slower controls, 32 final-screen cells and 2,688 checked invocations per host, the rejected Mac portable experiment and RAM-image ownership/release receipts. Its full/minimal results are not relabeled as the integrated lean measurements above.

Public report and lossless qualification summaries: docs/benchmarks/linux-inline-raw-records-20261109.{md,json}. Durable integration archives on drogon retain actual executable bytes, raw samples, scripts, oracles, logs, source bundle and identities:

  • /home/marcus/Documents/xff-pr1154-integration-linux-qualified-1f37b3c-20261109.tar.gz: 13,354,890 bytes; SHA-256 38aa03e3bf71a6b97e3f5695b64226d2cce0426143ff654625805b9f0cf40400.
  • /home/marcus/Documents/xff-pr1154-integration-macos-qualified-1f37b3c-20261109.tar.gz: 5,376,859 bytes; SHA-256 aff6ab8f1063eb992ce3afcfdd4051fb5878d97d0c573f65b55516f95ebb1ab3.
  • /home/marcus/Documents/xff-pr1154-integration-coverage-complete-1f37b3c-20261109.tar.gz: 330,465 bytes; SHA-256 19c9baae644094076de5f5a184d82432e7a4d50cbd42e6f0025163dbbc29dc5a; original raw LCOV verified inside.

Transferred copies are independently hash-checked. Access to the project host is required. Original pre-integration archives remain documented in the frozen report. Mac command metadata was reconstructed from the unchanged executed script after collection overwrote its local command JSON; original raw suite logs and terminal success remain retained with explicit provenance.

Benchmark coverage inventory

Refresh the dated direct-main-merge inventory: 237 expected, 162 published on both platforms and 75 missing in the snapshot. Development-branch merges are excluded. Record existing recovery jobs without redispatching successful work. Chromium selects #914/#986/#1143 in both published CI datasets, loads their actual figure payloads and reports no page errors. Preserve their conservative binary verification state and historical workflow-driver mismatch; do not bypass it. This inventory update does not claim completion of the missing-run backlog or extended-history publication.

Final CI performance review

Both official aggregate reports pool nine measured samples and retain the fastest seven. Linux matches all 360 comparisons to current main run 42564885478: median normalized change -0.074%, with no 15% regression alarms. Mac reports no compatible historical main baseline, so no normalized CI regression claim is made there; retain the independent local qualification and identical loaded-code/data proof. Both reports contain executable provenance receipts. GitHub measured synthetic merge 7f88c94cfe1c949d229e46c77cc41a2d6d895c55; its verified parents are current main and the PR head, and its tree exactly matches the qualified PR tree. No recorded artifact commit was relabeled.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Benchmark preview

Advisory: changes over 15% are warnings, not merge failures. Positive reference-adjusted change means slower; negative means faster.

linux

Performance overview

WARNING: 1 comparison(s) exceed the 15% slowdown threshold. Advisory only.

Merged baseline: 52950f9; run 42534134015, attempt 1; mean of fastest 7/9 runs.

Compatible comparisons: 360/360; unavailable: 0.

Population Comparisons Typical change Faster Within threshold Slower
All matched cases 360 -1.6% 1 358 1

Largest regressions:

Task Tree Workers Files Reference Change
fuzzy-discovery-itm deep 1 2,000 find+fzf +18.4%
fuzzy-discovery-itm broad 1 500 find+fzf +9.1%
content-needle deep 3 50 rg +7.0%
fuzzy-discovery-itm deep 1 5,000 find+fzf +6.8%
fuzzy-discovery-hdn deep 1 2,000 find+fzf +6.0%

Typical change is an equal-weight geometric mean of reference-adjusted XFF/main factors. Negative is faster; positive is slower. These correlated cases are not a significance test.

Interactive preview | CI run

Tested merge a40008ca22; PR #1154 head b3bf7c8ed7.

macos

Performance overview

No compatible merged baseline comparisons are available; regression status is unknown. Advisory only.

Merged baseline: unavailable.

Compatible comparisons: 0/360; unavailable: 360.

Population Comparisons Typical change Faster Within threshold Slower
All matched cases 0 n/a 0 0 0

Typical change is an equal-weight geometric mean of reference-adjusted XFF/main factors. Negative is faster; positive is slower. These correlated cases are not a significance test.

Interactive preview | CI run

Tested merge a40008ca22; PR #1154 head b3bf7c8ed7.

@helly25
helly25 merged commit ad3be8a into main Nov 9, 2026
109 of 133 checks passed
@helly25
helly25 deleted the perf/inline-raw-records-20261109 branch November 9, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant