perf: optionally seed phase 1 with segment-aware packed keys

CHANGELOG.md

@@ -2,6 +2,39 @@
 
 Release notes for the [`caps-sa`](https://crates.io/crates/caps-sa) crate.
 
+## Unreleased
+
+### Added
+
+- Opt-in `PackedPrefixSeedPolicy` for seeding external-memory phase-1 sorts
+  with segment-aware fixed-depth `u64` prefix keys. The default is
+  `Disabled`; `DenseAlphabetOnly` never allocates a second text-sized buffer,
+  while `remap(max_extra_bytes)` explicitly bounds an order-preserving ranked
+  copy for gapped byte alphabets.
+- `LimitProvider::boundary_rank()` and `BoundaryRank` let a provider declare
+  whether segment ends sort below or above real symbols. This semantic
+  capability is separate from the `ExtMemOpts` activation policy and defaults
+  to `None` for custom providers.
+
+### Changed
+
+- Eligible packed-prefix builds resolve most phase-1 comparisons from one
+  segment-bounded key, use exact key-derived LCPs between runs, and invoke the
+  full comparator only inside equal-key groups. On the complete 6.56-billion-
+  symbol ruSTAR GRCh38 + GENCODE v50 fixture, the seed reduced phase 1 from
+  49.038 to 12.204 seconds and the memoized build from 171.205 to 134.618
+  seconds (21.4%) at 32 physical cores, with identical output.
+- Packed-prefix eligibility is checked before alphabet scanning. Non-`u8`
+  symbols, finite contexts, providers without a representable boundary order,
+  and over-budget remaps fall back to the existing comparison sort.
+
+### Memory
+
+- The packed seed holds one `(u64, I)` record per selected suffix in each
+  active phase-1 task. On the annotated GRCh38 `u64` run this added 366-376
+  MiB (about 4.1%) peak RSS. Callers must opt in so this bounded worker scratch
+  and any explicitly budgeted ranked-text copy are never imposed silently.
+
 ## [v0.7.0](https://github.com/COMBINE-lab/caps-sa/releases/tag/v0.7.0) — 2026-08-13
 
 ### Added

README.md

@@ -73,16 +73,46 @@ build_ext_mem(&text, &opts, |sa_pos| {
 })?;
 ```
 
+Byte-valued workloads can optionally seed each external-memory phase-1 sort
+with a fixed-depth packed prefix key. Pre-encoded dense alphabets use no
+text-sized copy:
+
+```rust
+use caps_sa::{ExtMemOpts, PackedPrefixSeedPolicy};
+
+let opts = ExtMemOpts::default()
+    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly);
+```
+
+The seed requires unbounded comparisons and a `LimitProvider` whose
+`boundary_rank()` describes how segment ends sort. `PlainText` and
+`SegmentedText` provide the standard shorter-first declaration; custom
+comparators such as STAR's longer-first convention declare their own. Other
+symbol types, finite contexts, unsupported boundary semantics, and ineligible
+alphabets fall back to the comparison sort.
+
+The policy is disabled by default because each active task additionally holds
+one `(u64, I)` key record per selected suffix in its subarray. A gapped byte
+alphabet can use `PackedPrefixSeedPolicy::remap(max_extra_bytes)`, which makes
+the possible text-sized ranked copy explicit and declines it when the budget
+is insufficient.
+
 Inputs with many repeated long contexts can opt into bounded geometric LCP
 memoization during the final partition merges:
 
 ```rust
-use caps_sa::LcpMemoizationPolicy;
+use caps_sa::{LcpMemoizationPolicy, PackedPrefixSeedPolicy};
 
 let opts = ExtMemOpts::default()
+    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly)
     .lcp_memoization(LcpMemoizationPolicy::geometric());
 ```
 
+The two policies act on different phases and compose on the ruSTAR workload:
+the current 32-core GRCh38 + GENCODE v50 A/B measured 171.205 seconds with
+memoization alone and 134.618 seconds with both enabled (21.4% faster), with
+identical output.
+
 The direct path remains the default: memoization pays only when the workload
 contains enough repeated long contexts. See the
 [user guide](https://combine-lab.github.io/caps-sa/concepts/geometric-memoization/)

bench/README.md

@@ -73,7 +73,8 @@ cargo build --release --example rustar_segmented_bench
 
 CAPS_SA_PROFILE=1 taskset -c 0-31 \
   target/release/examples/rustar_segmented_bench /path/to/fixture \
-  --threads 32 --repeat 3 --work-dir /path/to/fast/temp
+  --threads 32 --repeat 3 --work-dir /path/to/fast/temp \
+  --packed-prefix-seed
 ```
 
 Omit `taskset` on platforms where it is unavailable. Use `--in-mem` only for
@@ -82,6 +83,13 @@ checksums are useful regression signals, but hashes are not proofs of equality;
 release validation should compare emitted position streams exactly or use a
 direct suffix comparator on a smaller fixture.
 
+For a packed-prefix A/B, use `--packed-prefix-seed` for the candidate and
+`--no-packed-prefix-seed` for the comparison; these override the corresponding
+`ExtMemOpts::from_env()` policy. Combined with `--packed-prefix-seed`,
+`--no-boundary-rank` is a separate negative control that keeps the policy
+enabled but makes the provider semantically ineligible, verifying the
+automatic fallback path.
+
 ## Results
 
 Machine: 64-core x86_64 Linux node, 1 socket, AVX2 enabled.

docs/src/content/docs/concepts/algorithm.md

@@ -18,7 +18,7 @@ When a fallback scan *is* needed, it runs through the **SIMD LCP fast path** (se
 For inputs too large for a single merge-sort pass, caps-sa wraps the kernel in a **sample sort**. The in-memory path uses the conventional four stages; the external path fuses the first three to avoid a complete intermediate spill. With `p` subproblems:
 
 1. **Presample pivots.** Sort a small deterministic position sample and pick `p − 1` evenly-spaced pivots. These define `p` partition ranges that together cover the whole SA.
-2. **Sort + distribute.** Split positions into `p` subarrays. Sort each in an outer Rayon task, binary-search its pivot splits, and write each sorted slice directly to its final partition bucket.
+2. **Sort + distribute.** Split positions into `p` subarrays. Sort each in an outer Rayon task, binary-search its pivot splits, and write each sorted slice directly to its final partition bucket. Eligible byte texts may opt into a segment-bounded packed-prefix seed: keys decide short prefixes, while equal-key groups retain the complete LCP merge-sort.
 3. **Per-partition merge.** Load each partition's bucket, cascade 2-way LCP-enhanced merges over its sub-slices, and emit the resulting sorted positions through the caller's closure.
 
 Because the partitions are globally ordered, emitting them in turn yields the full SA in lexicographic order, and **peak RAM stays bounded at `~O(text + n/p)` per worker** regardless of input size.
@@ -27,7 +27,7 @@ Because the partitions are globally ordered, emitting them in turn yields the fu
 
 The external-memory path (`build_ext_mem`) is the default for production-scale genomes. Final partition buckets are **disk-spilling**. Positions are read back partition-by-partition only when that partition is merged, then streamed straight out—the suffix array is **never fully materialised in memory**.
 
-The bucket pool collapses the `p` logical partition buckets onto a small set of physical temp files (one per worker by default) to keep kernel-level write contention bounded. Tuning knobs—subproblem count, working directory, physical file count, and LCP memoization policy—are on [`ExtMemOpts`](/caps-sa/reference/api/#extmemopts).
+The bucket pool collapses the `p` logical partition buckets onto a small set of physical temp files (one per worker by default) to keep kernel-level write contention bounded. Tuning knobs—subproblem count, working directory, physical file count, packed-prefix seed, and LCP memoization policy—are on [`ExtMemOpts`](/caps-sa/reference/api/#extmemopts).
 
 :::note[Positioned I/O]
 The pooled bucket path uses positioned reads/writes (`pread`/`pwrite` on Unix, `seek_read`/`seek_write` on Windows) so many workers can share one file handle without a shared cursor. It is portable across Unix and Windows as of v0.6.1.

docs/src/content/docs/getting-started/quick-start.md

@@ -36,15 +36,35 @@ build_ext_mem(&text, &opts, |sa_pos| {
 })?;
 ```
 
+### Optional: seed phase 1 from packed prefixes
+
+Dense byte alphabets can opt into segment-aware fixed-depth keys for the
+external-memory phase-1 sorts:
+
+```rust
+use caps_sa::{ExtMemOpts, PackedPrefixSeedPolicy};
+
+let opts = ExtMemOpts::default()
+    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly);
+```
+
+The mode is disabled by default because it adds one key record per selected
+suffix in each active phase-1 task. It requires unbounded comparisons and a
+`LimitProvider` with a representable `boundary_rank()`; otherwise caps-sa
+falls back automatically. `DenseAlphabetOnly` never creates a text-sized copy.
+See the [library API](/caps-sa/reference/api/#packed-prefix-phase-1-seed) for
+gapped alphabets and custom boundary conventions.
+
 ### Optional: reuse repeated long contexts
 
 For inputs with many long repeated contexts, opt into geometric LCP
 memoization for the final partition merges:
 
 ```rust
-use caps_sa::{ExtMemOpts, LcpMemoizationPolicy};
+use caps_sa::{ExtMemOpts, LcpMemoizationPolicy, PackedPrefixSeedPolicy};
 
 let opts = ExtMemOpts::default()
+    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly)
     .lcp_memoization(LcpMemoizationPolicy::geometric());
 ```
 
@@ -56,6 +76,11 @@ repetitive inputs can be neutral or slightly slower. See
 [Geometric LCP memoization](/caps-sa/concepts/geometric-memoization/) before
 enabling it broadly.
 
+The policies compose: packed prefixes reduce phase 1, while geometric
+memoization reduces phase 4. On complete ruSTAR-shaped GRCh38 + GENCODE v50,
+the memoized build improved from 171.205 to 134.618 seconds when the packed
+seed was also enabled, with identical output.
+
 ## Library: sort only a subset
 
 When many positions should be excluded from the sort (e.g. `N`s or inter-sequence spacers in a genome), hand only the positions you want sorted to a `*_for_positions` entry point. The others never enter the sort:

docs/src/content/docs/reference/api.md

@@ -80,13 +80,64 @@ Sample-sort / external-memory tuning. `Default::default()` is tuned for genome-s
 | `work_dir` | `std::env::temp_dir()` | Directory for the temporary bucket files. |
 | `physical_file_count` | `0` → auto | Physical temp files in the bucket pool. `0` picks one per worker; the `p` logical partition buckets collapse onto this pool. |
 | `ordered_phase4_emit` | `false` | Opt into the bounded ordered emitter (lower transient residency on skewed inputs, slightly slower on balanced ones). |
+| `packed_prefix_seed` | `Disabled` | Optional fixed-depth packed-key seed for external-memory phase 1. `DenseAlphabetOnly` adds no text-sized copy. |
 | `lcp_memoization` | `Disabled` | Optional exact long-LCP reuse. `LcpMemoizationPolicy::geometric()` selects the GRCh38-tuned defaults. |
 
 `ExtMemOpts` is non-exhaustive. Construct it with `default()` or `from_env()`
 and use its builder methods rather than an external struct literal.
 
 The `CAPS_SA_N_PHYS` environment variable overrides `physical_file_count` for one-off runs.
 
+#### Packed-prefix phase-1 seed
+
+Packed-prefix seeding is an external-memory-only, opt-in acceleration for byte
+texts. The safe mode expects symbols already encoded as the dense range
+`0..alphabet_size`:
+
+```rust
+use caps_sa::{ExtMemOpts, PackedPrefixSeedPolicy};
+
+let opts = ExtMemOpts::default()
+    .packed_prefix_seed(PackedPrefixSeedPolicy::DenseAlphabetOnly);
+```
+
+Each selected suffix receives a segment-bounded `u64` key. The key decides
+order for prefixes that differ within its fixed depth; equal-key groups retain
+the complete LCP merge comparator. The mode requires:
+
+- symbol type exactly `u8`;
+- `max_context == usize::MAX`;
+- a `LimitProvider::boundary_rank()` declaration; and
+- an alphabet with room for one reserved boundary code.
+
+`PlainText` and `SegmentedText` declare `BoundaryRank::ShorterFirst`. A custom
+STAR-compatible provider whose `boundary_order` places an ending suffix above
+a continuing suffix declares `BoundaryRank::LongerFirst`:
+
+```rust
+fn boundary_rank(&self) -> Option<caps_sa::BoundaryRank> {
+    Some(caps_sa::BoundaryRank::LongerFirst)
+}
+```
+
+`boundary_rank()` describes comparator semantics; it does not activate the
+optimization. Unsupported builds fall back without changing output.
+
+For a gapped byte alphabet,
+`PackedPrefixSeedPolicy::remap(max_extra_bytes)` permits an order-preserving
+dense copy only if its exact text-length allocation fits the supplied budget.
+Allocation failure or an insufficient budget falls back to comparison sort.
+Dense inputs avoid the copy under both policies.
+
+Phase 1 additionally holds one `(u64, I)` record per selected suffix in each
+active subarray. For the 6.18-billion-position ruSTAR `u64` construction at
+8,192 partitions and 32 workers, that is about 11.5 MiB per active worker;
+measured peak RSS increased by 366–376 MiB (about 4.1%).
+
+`ExtMemOpts::from_env()` recognizes `CAPS_SA_PACKED_PREFIX_SEED` for the
+dense-only policy and `CAPS_SA_PACKED_PREFIX_REMAP_BYTES` for an explicit
+remap budget. The latter takes precedence when both are set.
+
 #### Geometric LCP memoization
 
 Memoization is opt-in and applies only to phase-4 partition merges:

docs/src/content/docs/reference/performance.md

@@ -1,6 +1,6 @@
 ---
 title: Performance
-description: Production-shaped caps-sa 0.7 measurements and the standard upstream comparison.
+description: Production-shaped caps-sa measurements and the standard upstream comparison.
 ---
 
 Unless stated otherwise, numbers are suffix-array construction time and output
@@ -35,6 +35,34 @@ On the focused chromosome-21 backbone plus every GENCODE-derived junction
 flank, stable measurements improved from 11.97–12.00 seconds to 7.77–7.80
 seconds. All 359,616,038 emitted positions matched the reference.
 
+## Optional packed-prefix phase-1 seed
+
+The opt-in packed-prefix seed was measured against the released v0.7.0
+baseline on the same complete fixture, with one warm-up followed by three
+interleaved measured runs. Values below are medians:
+
+| Configuration | Build | User CPU | Peak RSS | Phase 1 | Phase 4 |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Direct LCP | 198.924 s | 5,153.25 s | 9,142,300 KiB | 49.038 s | 144.923 s |
+| Geometric memo only | 171.205 s | 4,684.95 s | 9,161,688 KiB | 49.034 s | 117.021 s |
+| Packed seed only | 162.650 s | 3,987.38 s | 9,527,240 KiB | 12.204 s | 145.187 s |
+| Packed seed + geometric memo | **134.618 s** | **3,507.54 s** | 9,536,692 KiB | **12.179 s** | **117.265 s** |
+
+The seed reduces phase 1 by 75.1% and improves the memoized ruSTAR
+configuration by 21.4%. Memoization changes phase 4 by 19.25% without the seed
+and 19.23% with it, confirming that the policies compose rather than competing
+for the same work. Every run emitted 6,176,694,310 positions with ordered hash
+`e81c8f9881e322148741a23c92ae2000`.
+
+Peak RSS increased by 366–376 MiB (about 4.1%), matching the bounded
+`(u64, u64)` key records held by the 32 active phase-1 tasks. The dense ruSTAR
+alphabet required no ranked-text copy.
+
+On chr21 without annotations, the direct build improved from a 1.291-second
+median to 0.934 seconds (27.7%). On the chr21 backbone plus every annotation-
+derived flank it improved from 7.709 to 6.025 seconds (21.8%), and the latter
+matched all 359,616,038 reference positions exactly.
+
 ## Geometric memoization
 
 The table above has geometric memoization enabled on both sides; do not add its