Reconcile ORF hypotheses and scoped RNA observations

CHANGELOG.md

@@ -7,6 +7,18 @@ between published tags; older pre-5.0 notes are retained below.
 
 For current interfaces, see the [consumer guide](docs/consumer-guide.md).
 
+## 5.87.0
+
+- Link normalized events, nucleotide ORF hypotheses, full protein products and
+  peptide occurrences without collapsing alternative translations or conflicting
+  source measurements. Add relational evidence views and explicit RNA count union
+  that requires known membership and measurement scope (#370).
+- Preserve those identities through long/wide CSV and TSV, original/additive
+  ranking and supporting-hypothesis selection. Distinguish known terminal flanks
+  from missing flanks in source observation IDs (#435).
+- Keep empty-source validation, genotype-aware filtering and output-warning
+  tests compatible with pandas 3 (#437).
+
 ## 5.86.2
 
 - Make dtype inference explicit when stacking sparse prediction frames and

docs/combined-sources.md

@@ -207,32 +207,32 @@ tool's RNA measurements; abundance is never automatically averaged or summed.
 Equal protein products do not establish equal nucleotide ORFs. Distinct
 `orf_id`, `coding_sequence`, transcript, frame or completeness annotations
 remain independent source observations even when the translated sequence is
-identical. Full ORF reconciliation is tracked in
-[#370](https://github.com/openvax/topiary/issues/370).
+identical. Use `reconcile_evidence` below to establish explicit ORF relationships
+while retaining those observations.
 
 `event_id` must be explicitly harmonized, including assembly/variant identity
 where appropriate. Native labels from different tools are not assumed equal.
 Without a common event ID, observations stay separate while sequence IDs still
 show exact sequence agreement. Local `pep_context` or `sequence` is not promoted
 to a full ORF. A peptide-only table does not establish its generating full ORF.
 
-To use another source's RNA measurement for an explicitly matching ORF, join it
-as a named feature. `join_annotations` refuses ambiguous annotation keys:
+To use another source's RNA measurement for an explicitly matching ORF, first
+reconcile the input and join the selected measurement as a named feature. `join_annotations` refuses ambiguous annotation keys:
 
 ```python
-from topiary import join_annotations
+from topiary import join_annotations, reconcile_evidence
 
-# combined contains both ORF/RNA rows and peptide rows with explicit
-# protein_sequence and event_id identifying their generating protein.
-keys = ["candidate_sample", "event_id", "protein_sequence_id"]
+combined = reconcile_evidence(combined)
+# Require an explicit ORF match; equal proteins alone are insufficient.
+keys = ["candidate_sample", "orf_hypothesis_id"]
 rna = combined.df.loc[
     combined.df.source_label.eq("exacto_normalized"),
     [*keys, "transcript_expression"],
 ]
 enriched = join_annotations(
     combined, rna, on=keys, prefix="exacto",
     provenance={"source": "exacto_normalized", "unit": "TPM",
-                "policy": "same sample, event and exact full sequence"},
+                "policy": "same sample and reconciled ORF; retain transcript-level unit"},
 )
 ranked = rank_candidates(
     enriched, "exacto_transcript_expression / affinity['netmhcpan'].value",
@@ -244,6 +244,74 @@ This expression illustrates a policy, not a calibrated biological model. The
 alternative ORF's abundance cannot attach just because its variant or gene
 agrees. Missing or ambiguous identity requires explicit resolution.
 
+## Reconcile ORFs and RNA observations (5.87.0+)
+
+`reconcile_evidence(combined)` returns a copy with stable links between biological
+entities. It neither chooses a caller nor changes the candidate universe.
+`evidence_views(combined)` exposes `events`, `orfs`, `proteins`, `occurrences`,
+`candidates`, `rna_observations`, `links` and `candidate_occurrences` as DataFrames; supply
+`source_labels=["lens"]` for a source-stratified view.
+
+| Input assertion | Identity rule |
+| --- | --- |
+| `event_id` or list-valued `event_ids` | Same sample, explicit `reference_name` and normalized event name; absent reference remains source-local |
+| `orf_id` | Caller-local ID; contradictory descriptors raise, absent descriptors can be supplied by another row with the same local ID |
+| `coding_sequence` and `transcript_path`, or `transcript_id` plus `orf_start`/`orf_end` | Cross-caller ORF agreement requires the same sample, reference and all supplied descriptors |
+| `reading_frame`, `orf_completeness`, start/stop flags, `linked_variants` | Retained in ORF identity; differing assertions remain alternative hypotheses |
+| `protein_sequence` | Exact full product, separate from nucleotide ORF identity |
+| `protein_hypothesis_sequence` | May be partial; never promoted to a full protein |
+| `peptide_start`/`peptide_end` | Zero-based half-open occurrence in the supplied protein; validated against its sequence |
+| Missing peptide coordinates | Source-local occurrence; no guessed equivalence from a shared peptide |
+
+ORF bounds are also zero-based half-open. `transcript_path` is an explicit ordered
+JSON path in the caller's normalized reference convention. Topiary does not lift
+over assemblies, normalize native genomic variant strings or infer a path from
+gene names. Incomplete records remain useful without establishing equivalence.
+Reference scope follows the distinction between sequence and annotation identity
+in [NCBI's feature documentation](https://www.ncbi.nlm.nih.gov/genbank/genomes_gff/).
+
+`rank_candidates` retains the chosen row's `orf_hypothesis_id` and
+`peptide_occurrence_id`. `candidate_observations` and the relational `links`
+retain alternative support. Rediscovery does not multiply candidate scores.
+Use an explicit duplicate policy or source filter when measurements disagree.
+`candidate_occurrences` links all same-sample peptide occurrences to existing
+pMHC queries, including sequence reports with no HLA assignment. Its
+`reported_candidate` flag distinguishes a reported peptide-HLA pair from a
+sequence match. These links transfer no scores, expression, specificity or
+eligibility between observations, and create no new candidates.
+
+RNA observations can be supplied as a list of records in `rna_observations`:
+
+```python
+measurement = {
+    "sample_name": "patient-01", "entity_type": "transcript",
+    "entity_id": "ENST-example", "quantity": "count", "unit": "reads",
+    "value": 2, "library_id": "rna-library-1", "read_set_id": "alignment-1",
+    "evidence_unit_ids": ["read-a", "read-b"], "method": "caller", "version": "1",
+}
+```
+
+`normalize_rna_observation` validates this shape. `entity_type` may be `gene`,
+`transcript`, `orf` or `variant`. A TPM measurement uses `quantity="abundance"`,
+`unit="TPM"` and no evidence-unit list. Unknown values remain null. Original
+quantifier metadata and extra fields survive; gene/transcript expression never
+becomes ORF expression automatically. Each caller's observations remain distinct.
+
+`union_rna_observations([measurement, other_measurement])` unions identified
+read, fragment, UMI or cell memberships only within one sample, library, read-set
+namespace, measured entity and unit. Shared members count once. Missing membership,
+different namespaces and TPM quantities raise: neither different file names nor
+different callers establish independent evidence. An explicitly empty membership
+is measured zero; an absent membership is unknown. Scalar expression/count
+columns already present in source tables stay untouched and are never implicitly
+converted into identified evidence sets.
+
+All identity columns, descriptors, alternative observations and metadata survive
+Topiary CSV/TSV save/reload in long and wide forms. Reconciliation can be repeated
+after reloading. The composed workflow test covers original and additive scoring,
+retained hypothesis selection, and count union without running a predictor during
+assembly.
+
 ## Add prediction features on demand
 
 ```python

tests/test_allele_set.py

@@ -159,12 +159,14 @@ def test_dependence_is_read_from_the_set():
 def test_an_allele_scoped_filter_keeps_the_genotype_row():
     """Same peptide-level rule as allele-free evidence (#183)."""
     df = _genotype_df()
+    original = df.copy(deep=True)
 
     kept = apply_filter(df, parse("affinity.value <= 500"))
 
     assert sorted(kept["kind"].unique()) == [
         "pMHC_affinity", "pMHC_presentation",
     ]
+    pd.testing.assert_frame_equal(df, original)
 
 
 def test_a_peptide_filtered_out_entirely_takes_its_genotype_row():

tests/test_args_outputs.py

@@ -4,7 +4,6 @@
 import warnings
 import pandas as pd
 import pytest
-from pandas.errors import SettingWithCopyWarning
 
 from .common import eq_
 
@@ -42,10 +41,15 @@ def test_write_outputs():
         print(df_from_file)
         eq_(len(df_expected), len(df_from_file))
         assert (df_expected == df_from_file).all().all()
+        assignment_warnings = tuple(
+            getattr(pd.errors, name) for name in ("SettingWithCopyWarning", "ChainedAssignmentError")
+            if hasattr(pd.errors, name)
+        )
         assert not any(
-            issubclass(warning.category, SettingWithCopyWarning)
+            issubclass(warning.category, assignment_warnings)
             for warning in caught_warnings
         )
+        pd.testing.assert_frame_equal(df, pd.DataFrame({"x": [1, 2, 3], "y": [10, 20, 30]}))
 
 
 def test_preview_shows_selected_renamed_columns_and_reports_omitted_rows(capsys):

tests/test_consumer_workflows.py

@@ -3153,3 +3153,66 @@ def test_cli_output_can_be_read_back_by_topiary(cli_output_request, tmp_path, ca
         affinity = reloaded[reloaded.kind == "pMHC_affinity"].set_index("peptide")
         assert affinity.value.dtype == "float64"
         assert affinity.value.to_dict() == {"SIINFEKL": 50.0, "GILGFVFTL": 500.0}
+
+
+@pytest.mark.parametrize("wide", [False, True])
+@pytest.mark.parametrize("extension", ["tsv", "csv"])
+def test_reconciled_hypotheses_survive_files_and_explicit_scoring(tmp_path, monkeypatch, wide, extension):
+    from topiary import (
+        combine_sources, evidence_views, reconcile_evidence, rank_candidates,
+        rescore_candidates, read_tsv, read_csv, union_rna_observations,
+    )
+    from .test_candidate_tables import Model, source
+    from .test_reconciliation import rna
+
+    def forbidden(*args, **kwargs):
+        raise AssertionError("Table-only reconciliation must not instantiate a predictor")
+
+    monkeypatch.setattr(TopiaryPredictor, "__init__", forbidden)
+    protein = "MAAASIINFEKL"
+    coding = "ATGGCTGCTGCTTCTATTATTAATTTTGAAAAACTG"
+    common = dict(reference_name="GRCh38", event_ids=["event-one", "event-two"],
+                  orf_id="caller-one-orf", coding_sequence=coding, transcript_id="t",
+                  orf_start=0, orf_end=len(coding), orf_completeness="start_to_stop")
+    first = source(**common)
+    first.df["event_ids"] = [common["event_ids"], common["event_ids"]]
+    first.df["protein_sequence"] = [protein, "MAAAGILGFVFTL"]
+    first.df["orf_id"] = ["orf-one", "orf-two"]
+    first.df["coding_sequence"] = [coding, "ATGGCTGCTGCTGGTATTCTGGGTTTTGTTTTTACTCTG"]
+    first.df["orf_end"] = first.df.coding_sequence.str.len()
+    first.df["peptide_start"] = [4, 4]
+    first.df["peptide_end"] = [12, 13]
+    first.df["rna_observations"] = [[rna()], [rna(entity_id="other-transcript")]]
+    other = first.df.iloc[:1].copy()
+    other["orf_id"] = "other-caller-name"
+    other["value"] = 75.
+    other["rna_observations"] = [[rna(evidence_unit_ids=["r2", "r3"], method="other")]]
+    orphan = pd.DataFrame([dict(common, protein_sequence=protein, transcript_expression=99.)])
+    result = reconcile_evidence(combine_sources({"one": first, "two": other, "orf-only": orphan}, sample_name="p"))
+    assert len(evidence_views(result)["orfs"]) == 2
+    assert len(evidence_views(result)["events"]) == 2
+    ranked = rank_candidates(result, "affinity.value", ascending=True, duplicates="best")
+    assert ranked.peptide.tolist() == ["SIINFEKL", "GILGFVFTL"]
+    assert ranked.iloc[0].orf_hypothesis_id == result.df.iloc[0].orf_hypothesis_id
+    assert ranked.iloc[0].peptide_occurrence_id == result.df.iloc[0].peptide_occurrence_id
+    assert len(rank_candidates(result, "affinity.value", ascending=True, duplicates="best",
+                               strata=["source_label", "candidate_mhc_class"])) == 3
+    # The same read is not independent support merely because two callers used it.
+    assert union_rna_observations([result.df.iloc[0].rna_observations[0],
+                                  result.df.iloc[2].rna_observations[0]])["value"] == 3
+    model = Model()
+    scored = rescore_candidates(result, model, prefix="fresh")
+    assert len(model.calls) == 2
+    assert scored.df.candidate_id.nunique() == 2
+    saved = scored.to_wide() if wide else scored
+    path = tmp_path / ("reconciled." + extension)
+    getattr(saved, "to_" + extension)(path)
+    restored = (read_tsv if extension == "tsv" else read_csv)(path).to_long()
+    after = reconcile_evidence(restored)
+    for name, before in evidence_views(result).items():
+        pd.testing.assert_frame_equal(evidence_views(after)[name], before, check_dtype=False)
+    assert rank_candidates(after, "affinity.value", ascending=True, duplicates="best").peptide.tolist() == ranked.peptide.tolist()
+    assert rank_candidates(after, "fresh__testmodel__pMHC_affinity__value", ascending=True,
+                           duplicates="best").peptide.tolist() == ["GILGFVFTL", "SIINFEKL"]
+    assert after.df.loc[after.df.source_label.eq("orf-only"), "candidate_id"].isna().all()
+    assert after.df.loc[after.df.source_label.eq("orf-only"), "value"].isna().all()