Separate dense SV discovery from complete-input candidate support

CHANGELOG.md

@@ -4,6 +4,16 @@ Behavior changes that can alter results or break callers, by release. Patch
 releases that only fix bugs or add fixtures are omitted; see the
 [commit history](https://github.com/openvax/isovar/commits/master) for those.
 
+## 1.44.0
+
+- Add opt-in `isovar sv-rna --dense-support` / `dense_support=True`: scan
+  original alignments on disk, select event-first sequence/placement bundles
+  under the discovery cap, then count direct junction and exact full-path/ORF
+  witnesses across the complete eligible input. Original records, missing
+  qualities, cell/UMI ambiguity and signal ancestry remain auditable. Discovery
+  and ORF-start assessment limits remain explicit; complete candidate support
+  does not imply exhaustive discovery. Defaults are unchanged (#437).
+
 ## 1.39.11
 
 - Fragments with conflicting ref/alt and third-allele mate observations no

docs/dense-sv-support-spec.md

@@ -0,0 +1,58 @@
+# Dense SV discovery and support (#437)
+
+Add an opt-in `dense_support` API / `--dense-support` CLI mode. Scan the entire
+original alignment input into a temporary SQLite spool, retaining original SAM
+records and input-scoped segment identities. Group supplementary/alternative
+records before selection. Search the requested breakpoint/exon/explicit regions;
+prioritize placed event joins, then breakpoint clips, over ordinary context, then select deterministic
+sequence/placement representatives under `max_records`. Duplicate abundance and
+coordinate order must not hide a late rare event. Keep omitted discovery groups,
+failed segment builds, query/path/extension limits explicit.
+
+Run the existing reconstruction engine on the bounded discovery BAM. Stream the
+spooled original groups again to find direct junction/clip observations for the
+fixed candidate paths. Reuse the existing placement, exact-interval, read-filter,
+cell/UMI and signal-lineage policies. Count full-path and ORF witnesses from all
+available originals, including one-read cells and missing QUAL. Keep discovery
+votes separate from exhaustive direct/full-interval support; unrelated background
+must never count as event support. Retain original witness records and query
+intervals, and preserve ambiguous placements and junction-base alternatives.
+
+The complete-input support pass cannot make omitted hypotheses assessed, establish
+independent molecules, or demonstrate translation. Ordinary intron-competition
+assessment remains scoped to discovery records and must say so. Scratch disk scales
+with input size; retained support witnesses and the JSON output can scale with
+actual candidate support. The default bounded API and schema stay compatible.
+
+Verify rare late events, order/cap-independent support for a fixed candidate,
+original SAM/QUAL preservation, conflicting mates, alternative paths, filters,
+explicit omissions, CLI/export behavior, and the complete lint/test/CI gates.
+
+## Verification evidence
+
+The regression suite includes the original ATP5MG--KMT2A short-read fixture,
+whose pinned peptide stays `MAQFVRNLVEKTPALVNG`, and synthetic split-read cases
+with alternate sequences, reversed partner orientation, missing QUAL, conflicting
+mate labels, repeated records and a visible signal parent outside search regions.
+Original-read interval matching and the normal ORF exporter are reused.
+
+A local synthetic scale check (macOS, Python 3.11.12, pysam 0.24.1) placed
+13 two-piece event segments after repeated 90-base donor background records.
+With `max_records=2` and assembly disabled, every run selected two discovery
+records and counted all 13 event fragments. Timings measure dense reconstruction;
+peak RSS covers the Python process, including input creation and imports.
+
+| Background records | Reconstruction seconds | Peak RSS (MiB) |
+|---:|---:|---:|
+| 1,000 | 0.035 | 120.7 |
+| 100,000 | 2.066 | 155.3 |
+| 1,000,000 | 37.365 | 160.7 |
+
+These are synthetic background-scale checks, not throughput estimates for noisy
+long reads or witness-heavy events. They do not replace a rerun of the Sid pilot.
+The audit runner in draft PR #433 can opt into this mode when it is ready.
+
+Testing also reproduced the default collector's SAM-payload identity collision,
+filed as [#438](https://github.com/openvax/isovar/issues/438). Dense mode raises on
+a conflicting original instead of selecting an arbitrary payload. Identical
+repeat records still deduplicate, and interrupted/failed spools are cleaned up.

docs/sv-rna.md

@@ -464,3 +464,58 @@ Sources: [SAM](https://samtools.github.io/hts-specs/SAMv1.pdf) and
 [fusion reconstruction assessment](https://pmc.ncbi.nlm.nih.gov/articles/PMC6802306/),
 [NCBI translation tables](https://www.ncbi.nlm.nih.gov/Taxonomy/Utils/wprintgc.cgi),
 [Kozak 1986](https://pubmed.ncbi.nlm.nih.gov/3943125/).
+
+## Dense inputs: discovery and complete-input support
+
+`--dense-support` (Python: `dense_support=True`) scans the **entire original
+alignment file**, including records outside the nominated regions. It groups
+actual records by segment on disk, gives placed event junctions first priority, followed by breakpoint
+clips, and reserves representative bundles for distinct
+sequence/placement classes before spending the discovery budget on repeated
+copies. A late event need not fall within the first `--max-records` records.
+Supplementary and alternative records of a selected segment stay together;
+an SA tag alone still supplies no observed partner ([SAM specification](https://samtools.github.io/hts-specs/SAMv1.pdf)).
+
+```bash
+isovar sv-rna --input event.json --bam original-regional.bam --output dense.json \
+    --dense-support --sv-scratch-dir /path/to/scratch --max-records 50000
+```
+
+The temporary SQLite spool and discovery BAM require scratch space proportional
+to the whole input, and are removed on success or failure. Without
+`--sv-scratch-dir`, the system temporary directory is used. Only the bounded
+discovery records and retained candidate witnesses are held in memory;
+witness-heavy results and their JSON can still be large. Use an original
+regional BAM to avoid unnecessarily scanning a whole-genome alignment file.
+
+After bounded reconstruction, a separate pass visits all original eligible
+segments to count support for the fixed candidate paths:
+
+- `junctions[].direct_support` counts the complete input's observed junctions,
+  retaining alternative junction-base sequences. A shared junction does not
+  establish the complete candidate sequence.
+- `paths[].full_interval_support` requires one exact, placement-compatible
+  observation spanning the **entire path and its junction**. The exploratory
+  ORFs' `full_interval_support` uses the same rule for each ORF interval.
+  Partial context reads do not become complete witnesses.
+- Original SAM records, query intervals, missing QUAL, cell/UMI evidence and
+  producer-aware signal ancestry remain available. Visible conflicting mate
+  labels remain unresolved, including mates outside the discovery regions.
+- `sequence_evidence.voting_support` and seed counts remain discovery counts.
+  `voting_observations` identifies the retained voters; their labels and ancestry
+  consult the complete input, including distant visible mates and parents.
+  ORF start/inclusion assessments and competing annotated splices also remain
+  scoped to discovery records; their scope is recorded explicitly.
+- `support_acquisition` describes the complete-input pass and its exclusions
+  and segment-build notes. `discovery` reports selected and omitted regional
+  records. `discovery_record_limit`, query/path/extension limits and any
+  `support_segment_grouping_limit` remain explicit. ORF exports retain both
+  discovery and support-pass metadata.
+
+Complete support for a returned candidate does not make hypothesis discovery
+exhaustive. A `no_candidate_paths` result with discovery limits remains
+unassessed evidence, not absence of an event or expression. Counts remain
+input-scoped segments, fragments and labels, not independent molecules.
+
+Support-pass completion refers to the supplied alignment file. It cannot recover
+original reads omitted during upstream acquisition, filtering or processing.

isovar/__init__.py

@@ -10,7 +10,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = "1.42.1"
+__version__ = "1.44.0"
 
 
 from .allele_interpretations import reconcile_allele_interpretations

isovar/cli/isovar_sv_rna.py

@@ -44,6 +44,11 @@ def make_parser(prog="isovar sv-rna"):
                        help="Base/small-indel alternatives also need this fraction of the best (default: %(default)s)")
     group.add_argument("--max-records", type=int, default=SV_MAX_RECORDS,
                        help="Alignment records examined; reaching the limit is reported (default: %(default)s)")
+    group.add_argument("--dense-support", action="store_true",
+                       help="Scan the full original input on disk, prioritize diverse event seeds, and count "
+                            "candidate support across all eligible original records")
+    group.add_argument("--sv-scratch-dir", type=Path,
+                       help="Existing scratch directory for --dense-support (space scales with the input)")
     group.add_argument("--max-queries", type=int, default=SV_MAX_QUERIES,
                        help="Region/mate/supplementary lookups; reaching the limit is reported (default: %(default)s)")
     group.add_argument("--max-paths", type=int, default=SV_MAX_PATHS,
@@ -95,6 +100,7 @@ def run(args=None, prog=None):
                 min_alternative_fragments=options.min_alternative_fragments,
                 min_alternative_fraction=options.min_alternative_fraction,
                 min_local_variant_fraction=options.min_local_variant_fraction, max_records=options.max_records,
+                dense_support=options.dense_support, scratch_dir=options.sv_scratch_dir,
                 max_queries=options.max_queries, max_paths=options.max_paths, assemble=options.assemble,
                 breakpoint_window=options.breakpoint_window, max_breakpoint_shift=options.max_breakpoint_shift,
                 max_extension_segments=options.max_extension_segments,

isovar/sv_rna.py

@@ -1269,6 +1269,7 @@ def span(observation):
                            somatic_causation_proven=False),
         sequence_evidence=dict(
             voting_support=store.cell_umi.support(segments),
+            voting_observations=sorted(voters),
             missing_quality_reads=len({o.identity for o in voting if o.missing_qualities}),
             secondary_reads=len({o.identity for o in voting if o.secondary}),
             assembly_phase="hypothesis_not_proven_long_range_phase"),
@@ -1324,7 +1325,8 @@ def reconstruct_sv_rna(bam, *, event_id, reference_name, donor, acceptor, region
                        inclusion_min_splice_anchor_bases=SV_INCLUSION_MIN_SPLICE_ANCHOR_BASES,
                        inclusion_min_base_quality=SV_INCLUSION_MIN_BASE_QUALITY,
                        inclusion_min_mapping_quality=SV_INCLUSION_MIN_MAPPING_QUALITY,
-                       inclusion_mapq_255_is_unique=SV_INCLUSION_MAPQ_255_IS_UNIQUE):
+                       inclusion_mapq_255_is_unique=SV_INCLUSION_MAPQ_255_IS_UNIQUE,
+                       dense_support=False, scratch_dir=None):
     """Reconstruct RNA paths for one nominated, oriented DNA adjacency.
 
     Paths start at unannotated RNA junctions (and, when soft-clipped bases are
@@ -1402,6 +1404,14 @@ def reconstruct_sv_rna(bam, *, event_id, reference_name, donor, acceptor, region
         in those gates. False treats it as unavailable, per the SAM
         specification; excluded 255 records are then reported as
         ``mapq_255_excluded_from_inclusion``.
+    dense_support : bool
+        Scan the complete original input into a disk spool, select bounded,
+        event-first discovery representatives, then count candidate junction
+        and full-interval witnesses across every original eligible segment.
+        Discovery and intron-competition limits remain explicit.
+    scratch_dir : str or pathlib.Path, optional
+        Existing directory for the dense-mode temporary spool and discovery
+        BAM. Scratch space scales with the complete input, not max_records.
 
     Returns
     -------
@@ -1410,6 +1420,11 @@ def reconstruct_sv_rna(bam, *, event_id, reference_name, donor, acceptor, region
         frame and event-linkage evidence, original records and limitations.
         This is not the validated supplied-fusion result consumed by Vaxrank.
     """
+    options = locals().copy()
+    if not isinstance(dense_support, bool):
+        raise ValueError("dense_support must be True or False")
+    if scratch_dir is not None and not dense_support:
+        raise ValueError("scratch_dir requires dense_support")
     if not all((event_id, reference_name, sample_id, source)) or not isinstance(event_provenance, dict) \
             or not event_provenance:
         raise ValueError("SV reconstruction requires event, reference, sample, source and event provenance")
@@ -1444,6 +1459,13 @@ def reconstruct_sv_rna(bam, *, event_id, reference_name, donor, acceptor, region
     if len({(r.annotation, r.transcript_id) for r in references}) != len(references):
         raise ValueError("Duplicate reference transcript identity")
     collector = ReadCollector() if read_collector is None else read_collector
+    if dense_support:
+        from .sv_rna_dense import reconstruct_dense_sv_rna
+
+        options.pop("bam")
+        options.update(references=references, regions=tuple(regions), read_collector=collector,
+                       peptide_lengths=lengths)
+        return reconstruct_dense_sv_rna(bam, options)
     models = [_Model(r) for r in references]
     annotated = set().union(*(m.junctions() for m in models))
     reference_peptides = {m.protein[i:i + k] for m in models if m.protein