Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
277e987
Reuse bounded matcher output capacities across batches
helly25 Nov 2, 2026
be8b8f7
perf: limit warmed matcher participation to available chunks
helly25 Nov 2, 2026
634b85e
Merge current main after bounded matcher result reuse
helly25 Nov 2, 2026
bcfef58
docs: retain rejected direct evaluator worker qualification
helly25 Nov 2, 2026
cfc21a6
Merge current main after useful matcher worker qualification
helly25 Nov 2, 2026
374a309
perf: carry requested content values to ordered consumers
helly25 Nov 2, 2026
ed9763f
test: verify parallel columns and line histogram cache reuse
helly25 Nov 2, 2026
a4a2207
Merge current main after evaluator qualification report
helly25 Nov 2, 2026
563a5c1
Reuse immutable default catalogs across library runs
helly25 Nov 2, 2026
7acbf03
Synchronize catalog reuse with merged main
helly25 Nov 2, 2026
bc4294e
Prepare immutable file-diff configuration once per style
helly25 Nov 2, 2026
77abeac
Synchronize diff configuration with merged main
helly25 Nov 2, 2026
327ea78
Prepare compact hash-primary operands once per expression
helly25 Nov 2, 2026
4e77548
Merge main after prepared diff configuration lands
helly25 Nov 2, 2026
cc00b95
Aggregate ordinary comparison side totals directly
helly25 Nov 2, 2026
ae7ec80
Merge main after prepared hash operands land
helly25 Nov 2, 2026
d528de1
Merge remote-tracking branch 'origin/main' into perf/borrow-normalize…
helly25 Nov 2, 2026
0f22d63
Reclaim unused hosted Android SDK space before LLVM extraction
helly25 Nov 2, 2026
fdc2c22
Merge remote-tracking branch 'origin/ci/free-llvm-extraction-space' i…
helly25 Nov 2, 2026
00b5fa0
Clear removed Android SDK discovery variables for native Bazel builds
helly25 Nov 2, 2026
18d57ba
Merge remote-tracking branch 'origin/ci/free-llvm-extraction-space' i…
helly25 Nov 2, 2026
8f86055
perf: reduce byte-word shingle allocation and copying
helly25 Nov 2, 2026
214ed91
Merge remote-tracking branch 'origin/main' into perf/borrow-normalize…
helly25 Nov 2, 2026
1a210e6
test: allow instrumented byte-reference comparisons to finish
helly25 Nov 2, 2026
35f404a
Reuse caller-owned archive input storage across callbacks
helly25 Nov 2, 2026
6b5a14e
Borrow cached digests and reduce record construction churn
helly25 Nov 3, 2026
37c8033
Merge current main dependency updates into archive buffer work
helly25 Nov 3, 2026
492c1ef
Merge remote-tracking branch 'origin/perf/reuse-archive-input-buffer'…
helly25 Nov 3, 2026
9928ddb
Reuse caller-owned buffers for memory and archive member streams
helly25 Nov 3, 2026
676b325
Sync output optimization with main and allow cold benchmark builds
helly25 Nov 3, 2026
c6a3290
Merge remote-tracking branch 'origin/perf/borrow-output-values' into …
helly25 Nov 3, 2026
e6a60c7
Sync stream buffers with main and give full rg MSan coverage enough time
helly25 Nov 3, 2026
8c12718
Fill source results directly and reuse owned decoder blocks
helly25 Nov 3, 2026
b740b73
Bind hash field qualifiers once without growing template segments
helly25 Nov 3, 2026
6672247
Retain explicit header verification and refresh completed analysis
helly25 Nov 3, 2026
8c6b458
Sync source materialization with merged stream buffers
helly25 Nov 3, 2026
a6ca3d2
Merge remote-tracking branch 'origin/perf/direct-source-materializati…
helly25 Nov 3, 2026
d95860e
Sync hash field binding with merged materialization
helly25 Nov 3, 2026
38ac542
Remove obsolete hash parser dispatch after binding
helly25 Nov 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ repos:
hooks:
- id: check-added-large-files
# Retain the exact lossless cross-platform performance evidence archives.
exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization)-20261103)\.json\.gz$
exclude: ^docs/benchmarks/((executor-copy-audit|bounded-read-ahead)-20261101|(directory-task-slots|packed-visit-snapshots|bounded-match-results|useful-match-workers|direct-evaluator-workers|derived-content-handoff|immutable-default-catalogs|prepared-diff-configuration|prepared-hash-operands|aggregate-comparison-totals|byte-shingle-representations|archive-input-buffers)-20261102|(digest-and-record-storage|caller-owned-streams|direct-source-materialization|hash-field-binding|hash-field-binding-tag)-20261103)\.json\.gz$
- id: check-merge-conflict
- id: check-yaml
# BINARY_TYPES below: archive containers are binary but can START with text (a phar begins with
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

# 0.11.0

- Bind explicit hash-field qualifiers once while preserving dynamic defaults, copied templates and snapshot lifetimes.

- Fill bounded source ranges directly into final storage and reuse owned decoder blocks during materialization, preserving read limits, errors and budgets.

- Reuse caller-owned input storage for memory-backed and nested archive-member streams while preserving read errors, sparse content and shared budgets.
Expand Down
5 changes: 3 additions & 2 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -468,8 +468,9 @@ EP/JC/P work; candidates are not speedup claims until measured on both platforms
- [x] O22a: Qualify snapshot-owned digest observers against owning consumption, preserving cache
insertion stability, invalidation, local-snapshot fallback, read errors and derived handoff.
Keep [the separate mechanisms and controls](docs/benchmarks/digest-and-record-storage-20261103.md).
- [ ] O22b: Qualify invariant field-hash binding without enlarging every unrelated template segment
or hiding typed state in strings. Preserve dynamic defaults and cold/non-hash controls.
- [x] O22b: Qualify invariant field-hash binding without enlarging every unrelated template segment
or hiding typed state in strings. Preserve dynamic defaults and cold/non-hash controls in
[the binding report](docs/benchmarks/hash-field-binding-20261103.md), including rejected setup work.
- [x] O22c1: Qualify direct plain-record construction and bulk C-escaping on both hosts, retaining
tiny colored controls and shared display/tree consumers in the O22a report.
- [ ] O22c2: Qualify command-owned reusable record/row buffers and streaming machine formats.
Expand Down
Binary file not shown.
202 changes: 202 additions & 0 deletions docs/benchmarks/hash-field-binding-20261103.md

Large diffs are not rendered by default.

Binary file not shown.
59 changes: 59 additions & 0 deletions docs/benchmarks/hash-field-binding-tag-20261103.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Hash-field binding: direct tag refinement

The main-based coverage run for PR #1070 identified an obsolete hash parser whose address was
used to classify bound fields, although rendering no longer called it. The implementation now
uses the existing nonzero hash-binding tag and removes that parser. A segment is a field when
it has an ordinary renderer or a hash binding; literal and unknown-field behavior, content
counts, extraction, transforms, validation and dynamic defaults retain their contracts.

This refinement is measured against the exact pushed `d95860e8e4aa5ed57e442b8b7e91b8be88fee2d2`
implementation, separately from the [original qualification](hash-field-binding-20261103.md).
It adds no bytes to the segment layout, public API, dependencies, locks or content caches.

## Comparison

Each host runs 16 in-memory cases and 20 native cases in three independent sessions, with nine
interleaved repetitions and the fastest seven retained. Native cases discard warmups and check
complete output and exits against independent hashlib expectations. They cover 10, 1,000 and
10,000 files, long paths, ordinary listing, repeated fields and transformed hash fields with one
or three requested workers. Linux uses tmpfs and CPU affinity; macOS uses warm APFS. No release
timing overlaps builds, lint, tests or instrumentation on the same host. Negative changes mean
less elapsed time. These are paired changes from the already optimized implementation.

| Case | Linux | macOS |
| :------------------------------------------------------------ | ---------------: | ---------------: |
| Sixteen warm explicit hash fields | -7.00% | -7.98% |
| Sixteen warm default hash fields | -3.28% | -5.71% |
| Whole-engine repeated hash fields, 1,000 files / 1 worker | -3.23% | -0.87% |
| Whole-engine transformed hash fields, 1,000 files / 3 workers | -1.52% | +0.15% |
| Ordinary name-field kernel | +0.04% | +5.08% |
| Ordinary component-field kernel | +0.83% | +3.32% |
| Single warm explicit hash field | +13.13% | -1.87% |
| Native command change range, all controls retained | -2.01% to +3.51% | -0.55% to +0.75% |
| Unstripped full CLI byte change | -4,896 | -208 |

Small kernels have mixed costs; do not describe every row as faster. The single-field Linux
cell and ordinary-field macOS cells remain in the lossless results with absolute times.
Compilation changes span -1.59% to +1.35% on Linux and -6.69% to +5.27% on macOS.
The original optimization's populations remain separate, rather than pooling timings across
this refinement or subtracting unrelated percentage changes.

## Verification and evidence

The added regression distinguishes hash tags from literals, escaped hash text and unknown fields,
and checks rendering, content counts and extraction classification. Existing hash extraction,
all-catalog algorithms, changing defaults, copied/moved templates, snapshot lifetimes and fuzz
cases remain active. All 313 release suites pass on each host. Nine focused ASan suites pass on each host, and nine
MSan and nine TSan suites pass on Linux. Three explicit changed-header consumer checks pass per
host. Focused Linux coverage of changed production field lines reaches 86/88 (97.73%) and
67/70 branches (95.71%); this scoped check does not replace the full CI coverage gate.
Complete release, sanitizer and explicit-header checks are recorded in the
companion lossless archive; current-head GitHub checks remain authoritative for merge readiness.

[Lossless sources, manifests, scripts, raw samples and validation logs](hash-field-binding-tag-20261103.json.gz)
retain compiler/configuration identities, old/tag binary sizes and hashes, all successful output
oracles and the original coverage failure. The finite-catalog overflow guard remains intact;
coverage is fixed by deleting dead code, without excluding executable lines or weakening policy.

Archive size: 585,438 bytes. SHA-256:
`558057de3b7c3f6cb8f84632721ad0297deae2e8bd050c7edd2099ba2ef9ebd6`.
45 changes: 40 additions & 5 deletions docs/performance-analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -3543,10 +3543,9 @@ own warmup, nine repetitions and fastest-seven selection with rotating participa
The Linux case-isolated matrix remains separate; compare within a recorded protocol rather than
silently treating absolute measurements from different protocols as interchangeable.

O22b still needs typed field-hash binding. Compact IDs/default-use bits may fit existing segment
padding, but actual layout, renderer-call overhead, changing context defaults and cold/non-hash
controls must be qualified before using that space. Do not encode binding state in arbitrary
strings or add a view-rich allocation to every segment.
O22b's compact field-hash binding is qualified below, including layout, renderer-call overhead,
changing defaults and cold/non-hash controls. It uses existing segment padding without encoding
state in strings or adding a view-rich allocation to every segment.

O22c2 still needs command-owned reusable output buffers and streaming rows. Keep owning/stateless
APIs intact; a private stateful record writer could reuse capacity on the coordinator and lend a
Expand Down Expand Up @@ -3583,7 +3582,7 @@ in the raw artifact. The frozen formatter, source guards, independent full-outpu
all selected root/extension release tests and focused sanitizers remain part of qualification.
The initial nested fixture command missed `--archive-depth=2`; its warmup oracle failed before
accepting any native row. The corrected matrix uses explicit depth two. O19b seeking/index/replay
and admission, and O19d direct final-buffer materialization, remain open.
and admission remain open; O19d direct final-buffer materialization is qualified below.

### O19d: fill final range and materialization storage

Expand Down Expand Up @@ -3622,3 +3621,39 @@ reserved storage directly; its expected owned-read counter changes from one to z
parent lifetimes, content and shared-budget tests remain. No new public flag, external dependency,
worker, lock, metadata probe or host-I/O bypass is introduced. O19b seeking/index/replay/admission
remains open.

### O22b: bind hash fields without growing unrelated segments

A private two-byte binding fits beside the existing post-processing enum. Algorithm IDs derive
from the existing `hash::AlgorithmNames` catalog; encoding choices distinguish explicit values
from context defaults. This avoids qualifier parsing and default work for explicit choices while
preserving changed defaults on every render, copies/moves, invalid specifications, post-render
transforms and snapshot ownership/invalidation. No borrowed qualifier view is retained.

Compilation validates the binding once. Bare/transformed defaults bind directly, and rendering
uses the existing nonzero binding tag to classify hash fields; ordinary renderers retain their direct calls.
A coverage-driven refinement removes an obsolete parser that had served only as a function-identity
tag. The first prototype duplicated static
validation and read a separate tag for every field. It increased hash compilation costs by about
30-50% and is not selected; complete first/refined populations remain separate.

[Cross-platform qualification](benchmarks/hash-field-binding-20261103.md) includes warm/cold
explicit, partial and default specs, compilation, ordinary fields, whole-engine in-memory runs and
native listing/hash-field commands. Explicit cached kernels improve about 41-56% in independent
confirmation; larger in-memory commands improve about 2-5%. Native gains are smaller. The Linux
simple-name field kernel retains about a 2% cost, roughly one nanosecond per entry; cold and Mac
controls vary across populations. Preserve their absolute times and identical-baseline movement;
this is not a universal rendering speedup.

Tracked template storage is unchanged across measured shapes. This is conservative capacity
accounting, not an allocator or RSS claim. Full CLI footprint increases 1,440 bytes on Linux and
368 bytes on macOS. Three new tests cover changing defaults across copies/moves, all shared-catalog
algorithms and invalid dynamic defaults with independent explicit choices. Existing derivation,
extraction, snapshot, output and fuzz suites remain; complete release and focused sanitizer
qualification passes both hosts. O22c2 command-owned buffers and streaming rows remain open.

The [direct-tag confirmation](benchmarks/hash-field-binding-tag-20261103.md) retains old/tag sources
and three-session cross-platform measurements. Sixteen warm explicit fields improve another 7-8%,
while small ordinary kernels and one Linux single-hash cell retain costs. Native differences remain
small, and the full CLI decreases 4,896 bytes on Linux and 208 bytes on macOS relative to the prior
binding implementation. A structural regression preserves literal/hash/extraction distinctions.
4 changes: 3 additions & 1 deletion release-site.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,9 @@
"docs/benchmarks/archive-input-buffers-20261102.md": "docs/benchmarks/archive-input-buffers-20261102.html",
"docs/benchmarks/digest-and-record-storage-20261103.md": "docs/benchmarks/digest-and-record-storage-20261103.html",
"docs/benchmarks/caller-owned-streams-20261103.md": "docs/benchmarks/caller-owned-streams-20261103.html",
"docs/benchmarks/direct-source-materialization-20261103.md": "docs/benchmarks/direct-source-materialization-20261103.html"
"docs/benchmarks/direct-source-materialization-20261103.md": "docs/benchmarks/direct-source-materialization-20261103.html",
"docs/benchmarks/hash-field-binding-20261103.md": "docs/benchmarks/hash-field-binding-20261103.html",
"docs/benchmarks/hash-field-binding-tag-20261103.md": "docs/benchmarks/hash-field-binding-tag-20261103.html"
},
"generated_html": {
"XFF.md": {
Expand Down
Loading
Loading