Skip to content

Avoid eager stats for lexical listing templates - #1134

Merged
helly25 merged 16 commits into
mainfrom
perf/template-stat-demand
Nov 7, 2026
Merged

helly25 merged 16 commits into
mainfrom
perf/template-stat-demand

Conversation

@helly25

@helly25 helly25 commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Lexical listing templates can render paths, names and supplied context without requesting unused file attributes. Compile their metadata requirements once and preserve eager metadata for columns and fields that need it. Both-host measurements show targeted gains with explicit control costs.

AG;DR

Implementation and correctness

  • Add conservative stat requirements beside the constexpr field registry's bound renderers. Literal segments, audited lexical fields and supplied context values are metadata-free; new/unclassified fields, size/time/identity, content, target and type remain conservative. Compilation retains the requirement in each segment; planning queries it once, with no per-entry flag/name scan or new ownership/worker wrapper.
  • Apply admission only to implicit --template output. Columns remain eager on every platform and worker count. Expression metadata, unknown/directory/followed-link traversal, safety, colors, reductions, collections, deferred evaluation and native-case requirements remain independent. Mixed {name}:{size} output still requests metadata.
  • Keep fuzzy-score collection independent of stat metadata. The initial full-suite failure exposed the old coupling; the final regression crosses batches over 1,000 entries, both one/three workers, and verifies a fuzzy score followed by an empty non-fuzzy OR result. Preserve aliases/transforms, lazy expression short circuit, denied metadata, unknown types and exact column stat expectations in committed tests.
  • No new flag, output spelling, operating-system/count/timer gate, VFS method, worker policy, synchronization primitive or benchmark dependency. Update design, CHANGELOG and canonical tracking together.

Evidence, selection and limits

docs/benchmarks/compiled-field-stat-demand-20261107.md and its lossless 6,954,149-byte capsule retain 629 individually verified records / 73,562,227 decoded bytes. SHA-256 b52cc8687dd213cda89b958b31ca912c4846d8f9d00ae32714227d7afdbbd8be. Report HTML and a supported raw-file download mapping are registered; the large-file exemption matches this exact filename. No historical measurement is removed.

The earlier combined template/column policy is rejected: Linux many-directory/10,000-file/three-worker TSV repeatedly costs about 24%, despite favorable cohort medians. Preserve initial screens, the correctness failure, corrected combined populations, all source/image identities and slower cells. Do not reopen it unchanged or pool its samples with the selected variant.

Template-only qualification uses exact main 1786340680499f50b08dee5b1ceffbc20fd7b533, with 80 in-memory production RunFind cases and 64 independently output-checked native CLI cases per host, three balanced sessions, nine raw repetitions and fastest seven retained. Include the unchanged image repeat and every column/plain/size control. Linux uses physical-core affinity; macOS has no affinity claim. Actual shipping compile/link/config/image receipts are archived; no peak-RSS, cold-cache or archive-specific speedup is claimed.

  • Native lexical-template cohort medians: Mac -4.34% / -1.06%, Linux -11.18% / -2.43% for one/three workers.
  • In-memory target medians: Mac -10.55% / -5.63%, Linux -17.41% / -12.04%.
  • Costs remain: Mac serial in-memory plain/size cohorts +3.24% / +4.04% (same-image repeats +0.31% / -0.68%); individual Linux native size controls +5.60% / +7.69%. Ordinary native control medians are about +0.3% to +1.5%. Mac parallel memory repeat variation limits precision. The former TSV case now has median +1.08%, with sessions +1.08/-7.50/+10.25%; it is not uniformly unchanged.
  • Experimental full CLI sizes are 112 bytes smaller on Mac and 2,608 bytes smaller on Linux; these are not compressed release downloads. Targeted gains justify this proposal with documented tradeoffs, not a universal improvement or attribution of every control difference.

Column pipeline causes, score-consumer-specific demand and conditional stat-bearing admission remain separate open work. No selective favorable rerun, normalization or subtraction hides costs.

Dependencies and validation

This is a separately reviewable child of #1133 (3c68a363f201e1f0f6222645cae0fa60568342a1), following #1129 → #1130 → #1131 → #1132 → #1133. Preserve each merge and its benchmark automation. Parent reports/tracking and the workflow correction are retained; no ready ancestor was modified. The six runtime/build/test input files were identical between the experimental main and this public parent, and all six final source hashes plus the frozen report/capsule remain byte-identical after normal publication hooks.

Development qualification passed all 323 documented core/registered-extension release targets on both hosts, seven focused ASan targets on both, and seven focused MSan/TSan targets on Linux. The unchanged-parent proof reproduces eight expected template failures (1,001 stats versus one); column expectations remain unchanged. Seven bounded optimized integration suites also passed in this successor worktree (all cached, zero freshly executed); this is development feedback, not public readiness.

All 629 evidence record lengths/hashes, site configuration/page/file mappings, exact allowlist positive and negative controls, canonical table alignment, diff checks and 32 release-site tests pass. Changed-file Trunk formatting/checks and normal signed commit/push hooks pass; SKIP=clang-tidy is supplied as requested, and no local clang-tidy executable ran. Signed head 4cebe20c36675fad9422645310154342783474c7 preserves the measured six source inputs exactly. This PR must pass its own actual pushed-head CI, including GCC, coverage, sanitizers, CI clang-tidy, final benchmark comparison and required done/pre-commit/trunk checks; private or ancestor results do not substitute. After retargeting to main, synchronize/push immediately and require its own new main-context validation before merging.

Current main integration

Signed synchronization a5fafd3c9cf8401030f5da49f41bcad31ad2b7c6 incorporates #1133 actual main squash 5b2e3847beb26d27ca569e559ee69e1e302d4448. Strict proof verifies that the real included old parent, qualified signed parent and actual squash have equal complete trees. The entire qualified child tree 141a4c6313dcef357a87629c18a7495199a11bf9 is preserved, including all runtime, test, report and capsule bytes. Normal signed commit and immediate push hooks passed; normal merge hooks intentionally skip redundant Trunk. No local clang-tidy, private timing or unchanged source test rerun was performed.

Own fresh current-main Test run 42383967956 completed successfully on signed head a5fafd3c9cf8401030f5da49f41bcad31ad2b7c6: all platform tests, GCC, sanitizers, coverage, actual CI clang-tidy, pre-commit, Trunk, final best-7-of-9 benchmark comparison and required done passed. All six initial measurement shards and the fourth macOS candidate passed; after-fourth agreement succeeded and both fifth candidates skipped normally. Exact predecessor-main Test 42383864867 also completed successfully. Earlier stacked CI and private qualification remain historical evidence and do not replace these checks. The user explicitly requested this PR after #1133; #1133 has now merged.

Base automatically changed from docs/archive-btree-historical-qualification to main November 7, 2026 14:55
@helly25
helly25 merged commit f8d7efd into main Nov 7, 2026
32 checks passed
@helly25
helly25 deleted the perf/template-stat-demand branch November 7, 2026 15:55
helly25 added a commit that referenced this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant