perf: skip long periodic runs instead of scanning them (independent)

[BenjaminDEMAILLE]

Independent of every other open PR. Branches from main, touches no packed-key code, reviewable and revertable on its own.

@rob-p asked in #7 to extract run skipping into its own PR and called it the strongest part of the stack. This is that extraction, with the two corrections from the same review applied.

What

The LCP-enhanced merge resolves most steps without touching the text, but when it does compare two suffixes it scans their shared prefix. That is fine until the text contains a long periodic run, where two suffixes inside it agree for as far as the run continues and one comparison scans megabytes.

If text[s..e) has period q and two suffixes start at a < b inside it with (b - a) % q == 0, they agree until the later reaches e, so

lcp(a, b) >= e - b

is known in O(1) from the run's bounds with nothing scanned. Mismatched phase means they differ within q symbols, so the ordinary scan is already short. Scans are additionally bounded to stop at a run's start rather than traverse it.

A homopolymer detector would not have worked. In 60-column wrapped FASTA the longest run of a single byte is 60, because each line of Ns ends in a newline. The real structure is a period-61 repeat.

Corrections from your review, applied

The table is no longer consulted on every LCP call. You were right that this taxed the common path. Consulting it costs up to three binary searches before any comparison, and nearly every LCP call in real sequence mismatches within a few symbols and never reaches a run. It now probes with the ordinary bounded scan first and consults the table only after a match survives 256 symbols. On a filtered N-containing chr21 matching your control fixture (80,177,238 retained positions), with byte-identical output throughout:

  run table disabled          1.97 s
  table consulted eagerly     2.40 s   (+22%)
  probe first                 2.02 s   (+2.5%, within noise)

The satellite claim is corrected. Your measurement is right: period 171 is not detected. The docs now state the measured coverage — periods 1, 2, 61 and 64 detected with full coverage and 5.8-6.9x faster; 65 and 171 not detected and at parity — and say explicitly that alpha-satellite arrays fall outside the detector, along with the note that the sampling stage can miss a sufficiently localised run.

Measured

12 threads, external memory, output verified byte-identical to the in-memory suffix array:

  chr21 FASTA (6.6 Mb of N)     24.19 s -> 2.05 s

Neutral where there are no runs, which is the design goal. Interleaved against main on N-free DNA:

  main   3.920  4.419  3.490 s   mean 3.943
  this   4.180  3.854  3.584 s   mean 3.873

Detection is two-stage so that costs nothing: a sampling pass collects the periods that occur and the full scan runs only for those, so an N-free text gets an empty table and every query short-circuits on a slice-empty check.

Verification

71 tests (62 on main plus 9 here) in debug and release; fmt, Clippy with warnings denied, rustdoc. The run-aware LCP is checked against a byte-at-a-time oracle over sampled position pairs on homopolymers, wrapped-FASTA blocks and multi-period texts, plus detection shape — sorted, disjoint, and the period claim actually holding — and max_context behaviour inside a run.

🤖 Generated with Claude Code

[rob-p]

Thanks for separating this change and for incorporating the lazy lookup idea. I tested the implementation at PR head 87ccdb9b8b3144a33d6b48121f07db65160298ab, rebased onto current main after the accepted #9 work (ba29c3bf8c3d4b09cdca2f5bf76a554b6fdb18b7; conflict-resolved test commit dcd46e5).

Decision: close without merging

The inference is sound and the PR already implements the requested lazy policy: an ordinary bounded LCP scan must match 256 symbols before any run-table lookup occurs. The output was correct in the ruSTAR-shaped tests. However, on the real annotated-human workload the change is a clear regression, not a speedup.

Production-shaped fixture

This is the same fixture used for the issue review, built through ruSTAR's actual GENCODE parsing and junction construction:

• GENCODE Human v50 GRCh38 primary-assembly FASTA
• comprehensive primary-assembly GTF
• sjdbOverhang = 100
• every deduplicated splice-junction flank from the full annotation
• ruSTAR forward/reverse-complement layout and spacer-separated segments
• ruSTAR/STAR boundary ordering
• only A/C/G/T-starting suffixes retained
• external-memory u64 construction, 8,192 partitions, 32 pinned physical cores
• text: 6,557,611,930 symbols
• retained suffixes: 6,176,694,310
• junctions: 698,597; segments: 1,397,582

One candidate warm-up was followed by an immediately paired baseline/candidate measurement. build_seconds includes run detection and all construction work.

Variant	Build (s)	Phase 1 (s)	Phase 4 (s)	User CPU (s)	Peak RSS (KiB)	FS output blocks
current main (ba29c3b)	373.800	105.440	263.166	10,425.51	9,787,324	193,033,040
#10, lazy (dcd46e5)	408.053	115.453	272.918	11,026.45	10,245,028	193,033,480
delta	+34.253 (+9.16%)	+10.013	+9.752	+600.94	+457,704	+440

The candidate warm-up was consistent at 404.648 s (phase 1 114.134 s; phase 4 270.800 s). Thus the result is far outside the sub-second run-to-run variation previously measured for this baseline and is not a cold-run artifact.

Both full runs emitted exactly 6,176,694,310 positions with hash e81c8f9881e322148741a23c92ae2000. Temporary writes were effectively identical, as expected; this optimization changes comparison work rather than the spill layout.

The focused chr21-backbone + full-annotation-flank fixture showed the same direction:

Variant	Build (s)
current main	15.577
#10, lazy	17.009
delta	+1.433 (+9.20%)

The candidate matched the stored baseline position-for-position for all 359,616,038 emitted suffixes.

Why the synthetic win does not transfer to ruSTAR

The periodic-run inference is genuinely valuable when the sorted suffix population contains many aligned starts inside one very long run, as in the raw-FASTA synthetic benchmark. That is not ruSTAR's construction:

1. ruSTAR parses FASTA before caps-sa sees it, so FASTA line breaks are absent. A wrapped 60 N + newline period-61 pattern is not present in the production text.
2. The suffix filter excludes every N-starting suffix. The population that makes an all-N synthetic suffix sort pathological is therefore not sorted at all.
3. A retained suffix starts with A/C/G/T. Most comparisons differ before both sides enter the same long N tract in the same phase, so very few calls survive the 256-symbol probe and satisfy the skip condition.
4. The LimitProvider stops LCPs at chromosome/junction-flank segment boundaries. Comparisons cannot scan through spacers or beyond the short annotation-derived flank contexts.

Lazy table lookup removes the much larger eager-lookup penalty, but it cannot make this workload free: run detection still scans the 6.56-billion-symbol text, and every hot LCP call still goes through the run-aware wrapper/table-state branch. Here those costs are paid broadly while successful skips are too rare to recover them. This is visible in both phase 1 and phase 4 and in the +5.8% user-CPU increase.

Correctness and checks

• focused ruSTAR output: exact match across 359,616,038 positions
• full annotated-human output: same count and 128-bit stream hash
• randomized custom STAR boundary-order probe: pass
• negative-i8 symbol probe: pass
• PR run/LCP tests including periods 1 and 61: pass
• all 71 library tests: pass when serialized

The default parallel test invocation can exceed this host's file-descriptor limit because multiple external-memory tests independently open pools at once; all tests pass with --test-threads=1. That is unrelated to the run-skipping logic but worth addressing separately in CI/test setup.

So I am closing #10 without merging. I still like the inference as a targeted technique for unfiltered periodic inputs. A future version could be reconsidered behind an explicit/adaptive opt-in or a filter/segment-aware activation criterion, but it should not be on by default in the ruSTAR path based on these data.