Combine source tables using original evidence by default, with explicit re-scoring #366

Status after Topiary 5.88.0

The two upstream implementation boundaries are shipped:

• Reconcile ORF hypotheses and source observations without collapsing alternate translations or RNA evidence #370 / PR Reconcile ORF hypotheses and scoped RNA observations #436: normalized ORF/event/occurrence reconciliation and scoped RNA evidence, released in 5.87.0.
• Add native Exacto fragment ingestion with evidence-preserving ranking workflow #365 / PR Import native Exacto evidence and optional fragments #438: native Exacto table/fragment ingestion, released in 5.88.0. Composed tests cover native Exacto plus LENS/pVACseq, original-evidence ranking, long/wide CSV/TSV, explicit rescoring and separate novelty-aware scanning.

Keep this broader issue open for the remaining selection/reporting contract, including:

• An auditable eligibility view with pass/fail/unknown and reasons, retaining excluded candidates as selection records.
• Persisted ranking configuration/provenance in exported rankings: expression, filter, selected methods/versions, strata and conflict/missingness policy.

A current rank_candidates workflow was run with two imported candidates and filter_by="value < 100": the original combined table retained both rows; the returned ranking contained one ranked row, no eligibility/reason columns, and empty DataFrame metadata. Its ordinary CSV export did not contain the scoring expression. This confirms that the new ingestion/reconciliation releases do not by themselves complete this reporting contract.

Broader Vaxrank consumer adoption remains separate in openvax/vaxrank#497; this session changed only Topiary.

User requirement

Combine Exacto, LENS, pVACseq and Topiary source tables using only the data those tables contain, preserving and using original measurements by default. Support explicit re-scoring later without replacing the original results. Emit pooled and source/category-stratified rankings with traceable provenance.

This is a table-first re-ranking workflow. Running every candidate through a common predictor is an optional enrichment, not a prerequisite for combining sources.

Default: combine and re-rank tables

• Import supported table/schema versions into a common representation, retaining source file/row identity, sample, peptide/proteoform context, antigen category and all original measurements/annotations, including original ranks where supplied.
• Retain each measurement's source, model/version, quantity and units when known. Preserve unknown metadata as unknown; normalizing names/units must not destroy the source value or its provenance.
• Combine and rank the reported candidate universe without predictor execution, model-weight downloads, raw DNA/RNA files, or reconstruction of protein fragments. Limited tables remain useful: an aggregated report contains only its selected candidates, and missing candidates/evidence cannot be reconstructed by merging.
• Use compatible original measurements by default. Record the ranking expression, selected measurement sources and missing/conflicting-evidence policy. Do not silently mix incompatible scores, fabricate values, or assume every candidate is tumor-specific. Keep unscorable candidates visible with the missing requirement stated; source-stratified lists must remain possible.
• Link duplicate candidate identities within the appropriate patient/sample context to all contributing observations. Preserve context-specific predictions and disagreements. Multiple discovery tools reporting a candidate must not multiply RNA counts or imply independent evidence.
• Export a combined candidate/evidence table, a pooled ranking where the chosen policy can score candidates, and source/category-stratified rankings. Keep MHC-I and MHC-II ranking spaces separate by default. Record eligibility pass/fail/unknown and reasons as an auditable selection view while retaining imported records.

Explicit: re-score existing candidates

• Let the user choose the candidate subset, predictor/model version and requested measurement kinds.
• Re-score exact reported peptide-HLA candidates from table data whenever the chosen model's required inputs are present. Require additional flanks, genotype or context only for models/measurements that actually need them; report unavailable requirements explicitly.
• Allow filling missing compatible predictions and explicitly requested fresh re-scoring. Neither operation overwrites imported values; new values carry their own method/version, input context and run provenance.
• Make original-only, new-only and explicitly combined ranking policies selectable. Re-scoring alone does not silently switch the active ranking policy or add new candidates.
• Preserve original and new evidence through save/reload, with reproducible ranking results.

Separate opt-in: expand candidates

Scanning new peptide windows/lengths or adding HLA combinations changes the candidate universe. Treat it as a distinct request from re-scoring. Use available context and recoverable mutation/junction metadata; identify generated versus imported candidates and leave unknown novelty explicit.

Verification

Add composed workflows to tests/test_consumer_workflows.py:

1. Read supported source fixtures, combine, de-duplicate with retained provenance, rank and export both pooled and stratified views with predictor entry points set to raise if invoked. Assert original values and ranks survive save/reload.
2. Repeat with sparse/disjoint evidence and conflicting duplicate observations. Assert explicit missingness and selection policy, not just column presence.
3. Explicitly re-score a selected subset. Assert only that subset is scored, candidate identities and original values remain unchanged, and choosing the new evidence changes ranking on a controlled fixture while original-only ranking remains identical.
4. Assert expanding candidates requires a separate operation and that mutation/junction evidence affects the generated windows.

Related work

• Add native Exacto fragment ingestion with evidence-preserving ranking workflow #365: native Exacto ingestion; its supported tables must participate in the default path without mandatory protein reconstruction or prediction.
• Reader-to-fragment conversion loses mutation metadata needed for novel-only prediction #364: preserve novelty metadata for optional context rescanning; this must not block combining/ranking existing table rows or exact-peptide re-scoring.
• Support mutant protein sequences from multiple sources (varcode, isovar, file import) #102: original common-fragment abstraction.

Prior local verification established reader/ranking components and a live shared-model example, but it is not a claim that this entire contract is implemented. This issue defines the remaining end-to-end acceptance criteria.