perf: optionally seed phase 1 with segment-aware packed keys

[BenjaminDEMAILLE]

Summary

Adds an opt-in, segment-aware packed-prefix seed for external-memory phase 1. Each phase-1 subarray is first ordered by a fixed-depth u64 key; only equal-key runs invoke the complete LCP merge comparator. Keys stop at LimitProvider boundaries and reserve a boundary code on the side declared by the provider, so the optimization supports both standard generalized-SA ordering and STAR's spacer-as-largest convention.

PR #14, the standalone ruSTAR replay harness, has now merged. This branch is refreshed onto that main and contains only the feature, its policy/API, tests, and documentation changes relative to the merged harness.

Public API and fallback behavior

Activation is separate from comparator semantics:

let opts = ExtMemOpts::default()
    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly);

• PackedPrefixSeedPolicy::Disabled is the default.
• DenseAlphabetOnly requires symbols already encoded as 0..alphabet_size and never allocates a second text-sized buffer. This is the intended ruSTAR policy for A=0, C=1, G=2, T=3, N=4, spacer=5.
• remap(max_extra_bytes) permits an order-preserving ranked copy for a gapped byte alphabet only within the explicit allocation budget.
• LimitProvider::boundary_rank() declares semantic eligibility only. Built-in providers return ShorterFirst; ruSTAR's provider returns LongerFirst.
• Non-u8 symbols, finite contexts, providers without a representable boundary order, 256-symbol byte alphabets, insufficient remap budgets, and failed remap reservations automatically use the existing comparison sort.

All cheap policy/type/context/provider gates run before the one alphabet scan. The boundary rank is computed once and shared by the phase-1 tasks.

Independent ruSTAR measurements

Complete production-shaped fixture: GENCODE Human v50 GRCh38 primary assembly, all 698,597 prepared/deduplicated splice junctions, sjdbOverhang=100, ruSTAR padding and junction append, forward/reverse-complement layout, 1,397,582 segments, STAR boundary ordering, ACGT-start filter, u64, 8,192 partitions, and 32 pinned AMD EPYC 9555 physical cores. The text has 6,557,611,930 symbols and emits 6,176,694,310 suffixes.

One warm-up followed by three interleaved measured runs; values are medians:

Configuration	Build	Phase 1	Phase 4	User CPU	Peak RSS
released v0.7.0, direct LCP	198.924 s	49.038 s	144.923 s	5,153.25 s	8.719 GiB
v0.7.0 + geometric memo	171.205 s	49.034 s	117.021 s	4,684.95 s	8.737 GiB
packed seed, direct LCP	162.650 s	12.204 s	145.187 s	3,987.38 s	9.086 GiB
packed seed + geometric memo	134.618 s	12.179 s	117.265 s	3,507.54 s	9.095 GiB

The seed reduces phase 1 by 75.1%, improves the memoized ruSTAR configuration by 21.4%, and improves the full stack by 32.3% over un-memoized v0.7.0. Geometric memoization changes phase 4 by 19.25% without the seed and 19.23% with it, confirming that the mechanisms compose.

The additional 366–376 MiB peak RSS (about 4.1%) matches one (u64, u64) key record per selected suffix in each active phase-1 task. Dense ruSTAR input does not allocate a ranked-text copy.

Refreshed-head correctness and performance gate

After merging current main and the PR #14 harness cleanup, the focused chr21 genomic backbone plus every GENCODE-v50-derived splice flank was rerun at 32 physical cores:

• optimized + geometric memo: phase 1 0.896 s, build 6.732 s;
• provider-decline fallback: phase 1 2.548 s, build 7.651 s;
• both emitted 359,616,038 positions;
• the optimized output matched every stored v0.7.0 reference position exactly;
• the fallback produced the same ordered 128-bit hash, 6d13e13009418fd6efbe4a73d7721e2e.

Targeted coverage includes exact adjacent-LCP oracles below, at, and above the packed-key depth under both boundary conventions; simultaneous boundary ties; the 255/256-symbol reserved-code edge; dense and budgeted-remap alphabets; signed i8; finite contexts; randomized segmented/filter constructions; policy-disabled and provider-decline paths; and composition with geometric memoization.

Validation

• 89 debug and 89 release tests pass across all targets.
• Formatting and Clippy pass with warnings denied.
• Rustdoc and doctests pass with warnings denied.
• The Astro user documentation builds under Node 22.
• cargo package --locked --offline --allow-dirty verifies the packaged crate.
• The replay harness exposes independent --packed-prefix-seed, --no-packed-prefix-seed, and --no-boundary-rank controls.

The bon builder migration remains intentionally separate: it is a broader API decision and is not required for this optimization.

[BenjaminDEMAILLE]

One configuration finding worth folding into this PR, since it changes what a
caller should run rather than what this PR changes.

Every number above is with memoization off, which is the default. It is not
the best configuration for this workload, and the two changes compose. Same
fixture, same harness, work directory purged between runs, two interleaved
pairs:

	run 1	run 2	mean
this PR, memoization off	347.237 s	364.597 s	355.9 s
this PR, memoization on	275.711 s	249.168 s	262.5 s

So LcpMemoizationPolicy::geometric() takes another 26.2% off on top of the
packed-key seed, and phase 4 falls from 318.2 s to 237.4 s. Against the 0.7.0
baseline with memoization on, 346.653 s measured in the same session, the two
together are 262.5 s.

The two act on different phases, which is why they compose: the seed cuts
phase 1 (126 s to 28 s), memoization cuts phase 4. Nothing here argues for
changing the default; it argues that a ruSTAR-shaped caller should turn it on,
and that this PR does not make that choice for anyone.

[rob-p]

Thanks, Ben — I independently audited and benchmarked this at the current PR head (114cc8c) against caps-sa main (76d13ab). The core result is solid, and I think the packed phase-1 seed should land after a focused revision.

My complete fixture is the production-shaped ruSTAR construction from GENCODE Human v50: GRCh38 primary assembly, all 698,597 prepared/deduplicated junctions, sjdbOverhang=100, ruSTAR bin padding and Gsj append, forward/RC layout, 1,397,582 segments, STAR boundary ordering, ACGT-start filtering, u64, 8,192 partitions, and 32 pinned physical cores on an AMD EPYC 9555. It contains 6,557,611,930 text symbols and emits 6,176,694,310 suffixes.

After warm-up, three interleaved measured rounds gave these medians:

Configuration	Build	Phase 1	Phase 4	User CPU	Peak RSS
main, direct LCP	198.924 s	49.038 s	144.923 s	5,153.25 s	8.719 GiB
main + geometric memo	171.205 s	49.034 s	117.021 s	4,684.95 s	8.737 GiB
PR #15, direct LCP	162.650 s	12.204 s	145.187 s	3,987.38 s	9.086 GiB
PR #15 + geometric memo	134.618 s	12.179 s	117.265 s	3,507.54 s	9.095 GiB

So I measure:

• packed seed alone: -18.2% end-to-end and -75.1% in phase 1;
• packed seed in ruSTAR's current memoized configuration: -21.4%;
• full stack versus un-memoized main: -32.3%;
• geometric memoization changes phase 4 by -19.25% on main and -19.23% on this PR.

That last pair is the clearest evidence that the two mechanisms genuinely compose. I did not reproduce the exact -24.7% / -26.2% figures in the PR, but the difference is unsurprising across hardware and fixtures, and the independent effect remains far outside noise.

Controls were also positive: the focused chr21-backbone plus all v50 annotation flanks improved 21.7%, while chr21 with no GTF/junctions improved 27.7%. Every full run emitted the same count and ordered 128-bit stream hash (e81c8f9881e322148741a23c92ae2000). The focused PR+memo output also matched all 359,616,038 stored main positions exactly.

I found the algorithm itself careful and sound: exact-u8 gating protects signed symbols, keys respect segment limits, sentinel placement captures both boundary conventions, equal keys fall back to the complete comparator, cross-run LCPs are capped by both segment limits, and finite contexts decline the seed. In addition to the PR suite, negative-i8, finite-context, and 3,000 randomized segmented/filter constructions across dense/gapped alphabets, five partition counts, and memoization on/off all matched direct comparison.

I do want to change a few integration details before merging:

1. Keep LimitProvider::boundary_rank() as a semantic capability declaration, but separate it from activation. Built-in providers currently enable the optimization silently and callers have no configuration-level off switch.
2. Check the configured policy, finite context, exact symbol type, and boundary_rank() before scanning the alphabet. Compute the rank once and pass it to the workers.
3. Make ranked-text memory explicit. Dense ruSTAR input requires no copy, but a gapped byte alphabet currently allocates a text-sized Vec<u8>. For the initial public policy I propose a safe dense-only mode, with any future remapping mode carrying an explicit extra-memory budget.
4. Document the measured per-worker scratch. (u64, u64) is about 11.5 MiB per active worker here; I observed an additional 366-376 MiB, about +4.1% peak RSS, rather than +0.3%. That is entirely acceptable for this gain, but it should be represented accurately.
5. Add a direct naïve-LCP oracle test around key depths k-1, k, and k+1, under both boundary conventions and with boundary ties. The current property tests cover final order but not the seed-produced LCP array directly.
6. Remove or split the incomplete lcp_replay example: its docs advertise capture and --threads, while the implementation supports neither.
7. Add README/API/changelog coverage for the optional capability and its eligibility/fallback behavior.

I am going to push these revisions to the PR branch under the enabled maintainer-edit permission. My proposed caller-facing shape is an explicit PackedPrefixSeedPolicy, defaulting to Disabled, with ruSTAR selecting the dense-alphabet mode alongside LcpMemoizationPolicy::geometric(). I will keep field width/key depth internal rather than exposing unmeasured tuning knobs.

On bon: I evaluated it, but I do not plan to mix that migration into this PR. It is MSRV-compatible, but all ExtMemOpts fields currently have defaults and there are only six handwritten setters, so required-field typestate buys little. Preserving today's repeatable last-setter-wins behavior also needs Bon's experimental overwritable feature. I think it is worth a separate additive API prototype later, retaining Default, from_env(), and the existing consuming setters while measuring downstream compile cost.

[rob-p]

I pushed the maintainer revisions discussed above as 2b0f03f.

The important API/design changes are now concrete:

• packed-prefix seeding is a caller policy, not an implicit consequence of LimitProvider::boundary_rank();
• PackedPrefixSeedPolicy::Disabled is the default;
• DenseAlphabetOnly is allocation-free with respect to the input text and is the intended ruSTAR mode;
• remap(max_extra_bytes) supports gapped byte alphabets only within an explicit ranked-copy budget, and falls back if the budget or allocation is unavailable;
• boundary_rank() is now documented strictly as a semantic capability declaration;
• policy, unbounded-context, exact-u8, and provider-rank gates all run before the alphabet scan;
• signed symbols, finite contexts, unsupported boundary semantics, and ineligible alphabets retain the existing comparison sort.

I also strengthened the tests to use a direct adjacent-LCP oracle, including LCPs just below, at, and above the packed-key depth; added external-output, allocation-policy, environment-policy, and early-gating tests; removed the incomplete lcp_replay example; and documented the API, memory cost, fallback behavior, and full-genome measurements.

The revised code retains the measured benefit on the focused ruSTAR chr21 backbone plus all GENCODE-v50-derived splice flanks (32 physical cores, one warm-up and three interleaved measured runs):

Configuration	Median build	Median phase 1
current main	7.709 s	2.575 s
revised PR, seed enabled	6.025 s	0.895 s
revised PR, provider declines	7.676 s	2.559 s

That is a 21.8% total improvement and 65.2% phase-1 reduction on the focused annotated fixture. The declined result is within run noise of main, as intended. All runs emitted 359,616,038 positions with hash 6d13e13009418fd6efbe4a73d7721e2e; a streaming comparison also matched every position against the main reference exactly.

The focused fixture does not benefit from geometric memoization (6.080 s with both versus 6.025 s with the seed alone), but the complete GRCh38 + GENCODE v50 run does: 171.205 s with memoization alone versus 134.618 s with both. This supports the original conclusion that the optimizations compose, while also showing that memoization's incremental value is workload-dependent.

Local validation is green: 88 debug and release tests, all targets, Clippy with warnings denied, rustdoc with warnings denied, doctests, formatting, the Astro documentation build, exact-output comparison, signed-i8, randomized segmented, and finite-context fallback probes. CI is now running on the pushed commit.

I have intentionally left a bon migration out of this PR. It is a broader public-API refactor and should be evaluated separately after this feature's policy surface settles.