Report policy evidence coverage and common-set comparisons

CHANGELOG.md

@@ -7,6 +7,18 @@ between published tags; older pre-5.0 notes are retained below.
 
 For current interfaces, see the [consumer guide](docs/consumer-guide.md).
 
+## 5.93.0
+
+- Report policy coverage separately from score filling and acceptance, with
+  explicit occurrence and model-request denominators, missing/failure reasons,
+  and distinct raw-row counts that do not multiply allele-free projections (#457).
+- Compare complete policy membership and raw scores on the shared assessable
+  set; retain unknown criteria, effective scores, definitions and provenance.
+  Typed CSV/TSV preserves reports and their replay inputs.
+- Exercise the released Vaxrank `openvax-v1` bundle with a synthetic context
+  overlay: missing short-flank evidence changes the chosen vaccine window while
+  shared assessable scores remain unchanged. No default recipe changes.
+
 ## 5.92.0
 
 - Add all-candidate self matching within an explicit same-length Hamming radius

docs/combined-sources.md

@@ -272,6 +272,81 @@ The Vaxrank consumer test carries these records through native dataset save/load
 and verifies frozen scores, selected windows, peptide constructs and mRNA
 constructs, plus a changed criterion that changes the selected window.
 
+### Coverage and policy comparisons
+
+Coverage does not depend on whether a policy fills a missing score with zero,
+includes unknown eligibility, or rejects a candidate. Use retained evaluations:
+
+```python
+from topiary import policy_coverage, summarize_policy_coverage, compare_policy_evaluations
+
+coverage = policy_coverage(evaluated)
+summary = summarize_policy_coverage(coverage)
+reasons = summarize_policy_coverage(coverage, by=["level", "criterion", "reason"])
+comparison = compare_policy_evaluations(evaluated, another_evaluation)
+shared = comparison.df.query("assessment_set == 'both'")
+coverage.to_tsv("coverage.tsv")
+comparison.to_tsv("policy-comparison.tsv")
+```
+
+`policy_coverage` emits one score and each named criterion per occurrence. Native
+`pass`, `fail` and numeric `value` are all assessed; `unknown` becomes missing.
+Not-applicable and not-evaluated remain distinct. A measured zero is assessed;
+a zero from `score_fill` retains its missing raw score. The original status,
+reason, criterion value, raw/effective score and eligibility remain visible.
+
+By default the denominator contains evaluated groups. Supply `universe`, a
+DataFrame of unique identity keys that includes every evaluated group, to keep
+expected inputs that produced no evaluation. Absent groups are not evaluated.
+Keys default to the retained occurrence grouping; explicit `keys` must uniquely
+identify occurrences. Do not drop flanks, offsets, samples or sources when that
+would merge different occurrences.
+
+Returned prediction rows alone cannot reveal failed or unattempted requests.
+For model coverage, pass an explicit `prediction_requests` DataFrame: one row
+per expected occurrence/model/field assessment, with the identity keys plus
+canonical `kind`, `prediction_method_name`, `predictor_version`, and numeric
+`field` (for example `score`). A null version matches only an unstated version.
+Include `allele_set` for joint-genotype evidence. An optional diagnostic `status`
+(`missing`, `failed`, `not_applicable`, or `not_evaluated`), `reason`, and `detail`
+can preserve explanations such as `insufficient_c_terminal_context` or a backend
+failure when no row exists. Diagnostics cannot override observed measurements.
+Topiary does not infer a context/domain failure from an absent value.
+
+Model records describe **availability**, not which model contributed to a
+criterion. They match exact model/version, occurrence context and canonical MHC
+scope; allele-free measurements can project, while allele-credited and joint
+measurements retain their scope. A criterion can combine multiple models or
+annotations, and this report does not invent lineage for that expression.
+
+The summary separates requested assessment counts from unique occurrences,
+unique peptide/allele/genotype candidates, and the union of referenced raw
+prediction rows. Repeated source discoveries can describe one candidate;
+allele projections can reference the same raw row. Finite raw fields are counted
+separately as `n_observed_prediction_rows`, even if conflicting values make the
+assessment unknown. Requests with no output contribute to assessment counts,
+not raw-row counts. `assessed_fraction` explicitly uses **all assessments** as
+its denominator, including inapplicable and unevaluated requests. Group by
+`reason` or any occurrence key to inspect the cause or location of missingness.
+
+Comparison preserves both memberships, eligibility, raw/effective scores and
+criterion decisions. `assessment_set` separates `both`, `left_only`,
+`right_only`, and `neither`; a finite raw score plus no unknown referenced
+criterion is required for assessability. Unused and inapplicable criteria do
+not disqualify an otherwise assessed score. Prefiltered groups have no assessed
+score. `raw_score_delta` is right minus left **only on the shared assessable
+set**; it does not normalize different policy scales or add scientific weights.
+
+Reports retain definitions, digests, contexts, provenance and explicit reporting
+inputs in `extra["policy_coverage"]` or `extra["policy_comparison"]`. Save the
+underlying `evaluated.evidence` too: replay it with `replay_selection_policy`,
+then regenerate coverage using the saved `keys`, `universe` and
+`prediction_requests` (convert the latter two record lists to DataFrames when
+not null). CSV/TSV readers may add their normal per-row file `source` label;
+coverage counts and replay are unchanged. Reports run no predictors and change
+no selection. Vaxrank owns vaccine-window/construct choices and its frozen
+`openvax-v1` bundle.
+
 ## Identities and source evidence
 
 | Column | Meaning |

scripts/check_vaxrank_candidates.py

@@ -344,3 +344,63 @@ def construct(epitopes):
     assert restored.selection == saved.selection
     pd.testing.assert_frame_equal(replay.occurrences, changed.occurrences, check_exact=True)
     assert construct(restored.epitopes) == construct(changed_epitopes)
+
+
+def test_short_context_coverage_explains_vaxrank_window_change():
+    from dataclasses import replace
+    from topiary import (SelectionCriterion, TopiaryResult, evaluate_selection_policy,
+                         policy_coverage, summarize_policy_coverage, compare_policy_evaluations)
+    from tests.test_policy_coverage import short_context_inputs
+    from vaxrank.config.loader import load_vaxrank_config, extract_epitope_config_kwargs
+    from vaxrank.epitope_dsl import prediction_group_columns
+    from vaxrank.window_selection import WindowSelection, select_windows
+
+    bundle = load_vaxrank_config(config_path="builtin:openvax-v1")
+    cfg = EpitopeConfig(**extract_epitope_config_kwargs(bundle))
+    baseline = SelectionPolicy.from_dict(cfg.selection_policy)
+    overlay = replace(baseline, name="synthetic-context-overlay",
+        score_by=baseline.score_by + ' * criterion("processing")', criteria=baseline.criteria + (
+            SelectionCriterion("processing", "peptide_view(proteasome_cleavage.score)", "score"),))
+    evidence = short_context_inputs()
+    frame = evidence.df
+    keys = prediction_group_columns(frame)
+    measured = frame.loc[frame.kind.eq("pMHC_affinity")].to_dict("records")
+    rows = []
+    for row in measured:
+        prediction = Prediction(peptide=row["peptide"], allele=row["allele"], kind="pMHC_affinity",
+                                value=row["value"], score=row["score"], predictor_name="original", predictor_version="1")
+        rows.append(dict(peptide=row["peptide"], source=row["source_sequence"], offset=row["peptide_offset"],
+                         n_flank=row["n_flank"], c_flank=row["c_flank"], prediction_id=row["prediction_id"],
+                         mutant=prediction, source_class="mutation", overlaps_targetable=True,
+                         occurs_in_reference=False, occurs_in_non_CTA_reference=False, patient_alleles=[row["allele"]]))
+    epitopes = candidate_epitopes_from_rows(rows)
+    evaluations, choices = [], []
+    for policy in (baseline, overlay):
+        configured = EpitopeConfig(**dict(extract_epitope_config_kwargs(bundle), selection_policy=policy.to_dict()))
+        scored = attach_per_allele_scores(epitopes, configured, topiary_df=frame)
+        evaluated = evaluate_selection_policy(TopiaryResult(frame), policy, group_keys=keys)
+        evaluations.append(evaluated)
+        candidates = []
+        for row in measured:
+            sequence = row["source_sequence"]
+            antigen = VaccineAntigen(kind="mutation", amino_acids=sequence,
+                targetable_mask=TargetableMask((AminoAcidInterval(row["peptide_offset"], row["peptide_offset"] + 8),)),
+                tumor_specificity=TumorSpecificityAttestation(status="admitted", evidence_kind="synthetic_fixture",
+                    evidence_source="check_vaxrank_candidates", patient_specific=True, rationale_code="test_only"),
+                source_identifier=row["prediction_id"])
+            candidates.append(VaccinePeptide(antigen=antigen,
+                epitopes=[epitope for epitope in scored if epitope.prediction_id == row["prediction_id"]],
+                combined_score_expr="target_epitope_score", ranking_rules=("target_epitope_score",)))
+        chosen = select_windows(candidates, WindowSelection(self_weight=0., serum_weight=0.), preferred_length=13, limit=1)
+        choices.append(chosen[0].amino_acids)
+    assert choices == ["SIINFEKL", "AAASIINFEKLGGG"]
+    requested = evaluations[1].occurrences.query("allele != ''")[keys].assign(kind="proteasome_cleavage",
+        prediction_method_name="context-model", predictor_version="1", field="score")
+    requested["status"], requested["reason"] = ["missing", None], ["insufficient_c_terminal_context", None]
+    coverage = policy_coverage(evaluations[1], prediction_requests=requested)
+    summary = summarize_policy_coverage(coverage, by=["level", "reason"])
+    assert summary.query("reason == 'insufficient_c_terminal_context'").n_missing.item() == 1
+    comparison = compare_policy_evaluations(*evaluations).df
+    assert comparison.query("prediction_id == 'short'").assessment_set.item() == "left_only"
+    shared = comparison.query("assessment_set == 'both'")
+    assert shared.prediction_id.tolist() == ["long"] and shared.raw_score_delta.eq(0.).all()

tests/test_consumer_workflows.py

@@ -3523,3 +3523,43 @@ def forbidden(*args, **kwargs):
     assert restored.df.loc[restored.df.source_label.eq("exacto"), "candidate_id"].isna().all()
     assert restored.extra == combined.extra
     assert rank_candidates(restored, expression, **options).candidate_id.tolist() == pooled.candidate_id.tolist()
+
+
+def test_short_context_coverage_explains_changed_selection_and_replays(tmp_path, pandas_string_inference):
+    from dataclasses import replace
+    from topiary import (SelectionCriterion, SelectionPolicy, evaluate_selection_policy, policy_coverage,
+                         summarize_policy_coverage, compare_policy_evaluations, replay_selection_policy)
+    from .test_policy_coverage import short_context_inputs, KEYS
+    from .test_twin_conformance import DELIMITED_IO_TWINS
+
+    evidence = short_context_inputs()
+    baseline = SelectionPolicy("synthetic-baseline", 'criterion("binding")', score_fill=0., min_score=.01,
+        criteria=(SelectionCriterion("binding", "1 / affinity.value", "score"),))
+    overlay = replace(baseline, name="context-overlay", score_by=baseline.score_by + ' * criterion("processing")',
+        criteria=baseline.criteria + (SelectionCriterion("processing", "peptide_view(proteasome_cleavage.score)", "score"),))
+    left = evaluate_selection_policy(evidence, baseline, group_keys=KEYS)
+    right = evaluate_selection_policy(evidence, overlay, group_keys=KEYS)
+    assert left.selected.sort_values("score", ascending=False).prediction_id.iloc[0] == "short"
+    assert right.selected.prediction_id.tolist() == ["long"]
+    requested = right.occurrences.query("allele != ''")[KEYS].assign(
+        kind="proteasome_cleavage", prediction_method_name="context-model", predictor_version="1", field="score")
+    requested["status"] = ["missing", None]
+    requested["reason"] = ["insufficient_c_terminal_context", None]
+    coverage = policy_coverage(right, prediction_requests=requested)
+    summary = summarize_policy_coverage(coverage, by=["level", "reason"])
+    assert summary.query("reason == 'insufficient_c_terminal_context'").n_missing.item() == 1
+    model = summarize_policy_coverage(coverage).query("level == 'prediction'").iloc[0]
+    assert model.n_assessments == 2 and model.assessed_fraction == .5
+    comparison = compare_policy_evaluations(left, right)
+    assert comparison.df.query("prediction_id == 'short'").assessment_set.item() == "left_only"
+    shared = comparison.df.query("assessment_set == 'both'")
+    assert shared.prediction_id.tolist() == ["long"]
+    assert shared.raw_score_delta.eq(0.).all()  # selection changed through missingness
+    for suffix, writer, _, reader in DELIMITED_IO_TWINS:
+        path = tmp_path / ("context." + suffix)
+        writer(right.evidence, path)
+        replay = replay_selection_policy(reader(path))
+        repeated = policy_coverage(replay, prediction_requests=requested)
+        pd.testing.assert_frame_equal(repeated.df, coverage.df, check_exact=True)
+        pd.testing.assert_frame_equal(summarize_policy_coverage(repeated), summarize_policy_coverage(coverage))
+        pd.testing.assert_frame_equal(compare_policy_evaluations(left, replay).df, comparison.df, check_exact=True)