Preserve fixture provenance and native coordinates; add audited evidence (0.15.0)

MANIFEST.in

@@ -6,3 +6,6 @@ recursive-include tests/data/allele_resolution *.fasta *.vcf *.txt *.md
 recursive-include scripts *.py
 
 recursive-include tests/data/additional_svs *.bam *.bai *.md
+recursive-include tests/data/comprehensive *.json.gz *.md
+recursive-include tests/data/evidence-overlap *.bam *.md
+recursive-include tests/data/retrieval *.bam *.bai *.md

docs/changelog.md

@@ -5,9 +5,28 @@ your version with `osteosarc --version`. Pin both the package version and your
 snapshot (by name or download date) for reproducible analyses. Full release notes are on
 [GitHub](https://github.com/iskandr/osteosarc/releases).
 
-## Unreleased
-
+## 0.15.0 (2026-09-29)
+
+- `subset_bundle` and `test-data make --from` carve out members offline while
+  preserving original source headers, acquisition receipts and selection results (#90).
+- New `sv-regressions-v2` panel supplies the three Isovar fusions' retained sides;
+  newly requested fusion bundles use it. Published v1 panels and bundles stay unchanged (#96).
+- Historical retrieval cases have verified native coordinates, exact BAM contig names,
+  mitochondrial reference identity and a guard against incompatible merges (#99).
+- Reviewed Tempus evidence overlap is available on files and file selections, with
+  explicit preference required to exclude one product. Both original labels and
+  acquisition receipts remain intact; the true sample identity is still unresolved (#100).
 - Python 3.14 support: the tests run on Python 3.9 to 3.14.
+- Comprehensive vaccine/RNA test corpus: 51 vaccine-associated variants from the
+  union of membership sources, a complete per-BAM support matrix, four
+  DNA-supported SNVs with zero reported RNA alternate reads, a historical RNA
+  disagreement case, frameshifts and SV evidence. Its
+  openvax-v3 read specification is an unpublished candidate.
+- Shared read builds can checkpoint completed source selections and set their
+  supplementary-alignment depth and region limits explicitly in the spec.
+- Compact carry-forward preserves query positions for placed-unmapped records.
+  Comprehensive control audits verify source URLs and BAM/index checksums against
+  acquisition receipts; packaged RNA-support claims follow the actual recounts.
 
 ## 0.14.4 (2026-09-28)
 

docs/samples.md

@@ -1,5 +1,33 @@
 # Samples and files
 
+## Known overlapping RNA evidence
+
+The reprocessed Tempus ALMY2X4KMV and KCVBE1UI1P products overlap almost completely
+in the audited vaccine regions: 18,971 shared read names and 40,684 matching core
+alignment records out of 40,696 per file, including base qualities. They must not
+be counted as independent RNA evidence. This is a regional observation, not proof
+that the full libraries are identical or that either biological sample label is correct.
+See [the provenance investigation](https://github.com/iskandr/osteosarc/issues/100).
+
+Each `file.evidence_overlaps` lists reviewed relationships for its exact product
+URL. A selection's `files.overlap_groups` reports pairs present together, and
+`files.require_no_known_overlaps()` raises before they can be used as independent
+evidence. To keep one explicitly, call
+`files.without_known_overlaps(prefer=[chosen_url])`. The returned selection records
+every exclusion in `files.source["excluded_evidence_overlaps"]`; original labels,
+sample claims and receipts remain unchanged. No listed overlap means no reviewed
+overlap is known, not proof of independence.
+
+The published FASTQ catalogue already identifies KCVBE1UI1P as a duplicate delivery
+of ALMY2X4KMV RNA. Corresponding mate objects also have matching lengths and multipart
+ETags (server metadata, not whole-file SHA-256 hashes). The overlap registry preserves
+that source receipt and the object metadata. The existing `tempus-file-labels`
+curation disputes the published T1 timepoint: several observations point to T0, but
+do not establish the specimen conclusively. Resolving that still requires specimen
+documentation; neither the alias nor this audit silently changes the timepoint.
+
+## Catalogue
+
 - **Samples** are what was collected: tumor biopsies and resections at time points T0
   to T3, an organoid grown from the T1 tumor, and blood draws.
 - **Files** are what's in the public S3 bucket: aligned reads (BAMs), raw reads

docs/test-data.md

@@ -77,6 +77,54 @@ instead, use `osteosarc test-data export`.
 
 ## Shared test data
 
+To carve out a library's members without reacquiring its source BAMs:
+
+<!-- docs-check: skip (downloads the shared bundle and writes a new folder) -->
+```sh
+osteosarc test-data make isovar-subset --from openvax-v2 --member isovar/
+```
+
+The Python equivalent is `osteosarc.subset_bundle(bundle, members, destination)`;
+`members` contains exact names, whereas the CLI accepts prefixes. This keeps the
+original source headers, acquisition receipts, selection outcomes and record
+multiplicities, and pins the parent manifest checksum. Empty, unresolved and
+omitted members retain their status. A local bundle needs no network access.
+
+New fusion requests use `sv-regressions-v2`. Its GABBR1--SLC29A1 and OTUD7A--FMN1
+breakends retain right/right; PARD3B--CDKN2B retains left/right. The versioned v1
+panel and already published bundles retain their original metadata.
+
+### Historical retrieval cases
+
+The 49 `shared-v1` cases are small indexed-retrieval examples. They use five BAM
+reference dictionaries and **cannot all be merged and queried under one genome
+build or contig naming convention**. Their `variant_id` is the catalogue identity;
+it is not necessarily the coordinate in the BAM. Read the reviewed metadata with
+`osteosarc.retrieval.load_retrieval_cases()`, query each original BAM using its
+`query_region`, and use `assert_compatible_retrieval_cases(cases)` before a raw merge.
+Matching dictionaries establish coordinate compatibility, not sample independence.
+
+| Case | Catalogue position | Native BAM query | Nuclear assembly | Mitochondrial reference |
+| --- | --- | --- | --- | --- |
+| 44, MT-ND5 / Tempus | chrM:12994 | MT:12994 | GRCh37 | rCRS, 16,569 bases; validation uses hg38 chrM |
+| 45, MT-ND5 / CeGaT | chrM:12994 | chrM:12995 | GRCh37 | UCSC hg19 chrM, 16,571 bases |
+| 48, NR2F2 / CeGaT | chr15:96332299 | chr15:96875528 | GRCh37 | Not applicable |
+
+Positions in this table are one-based; `query_region` is zero-based and half-open.
+The new `native-retrieval-v1` metadata preserves the original cases, validation
+receipts and BAM checksums. `annotate_retrieval_manifest(manifest, directory)`
+reproduces it from an existing export, checking every asset and native indexed
+query. No BAM or historical manifest is silently rewritten or lifted over.
+
+### Comprehensive corpus
+
+For the comprehensive vaccine/RNA candidate, see the
+[frozen corpus and build instructions](https://github.com/iskandr/osteosarc/tree/main/tests/data/comprehensive).
+It includes the union of all vaccine membership sources (51 variants in the
+2026-09-28 snapshot), the full per-BAM support matrix, RNA-negative DNA-positive
+SNV controls, frameshifts and structural candidates. Its local openvax-v3 build
+preserves openvax-v2's library fixtures; it is not yet a published bundle.
+
 Isovar, Topiary, Varcode and Vaxrank can all test against one bundle of reads,
 openvax-v2, chosen once, here. It holds:
 

osteosarc/__init__.py

@@ -18,6 +18,7 @@
     export_bundle,
     generate_bundle,
     list_bundle,
+    subset_bundle,
     verify_bundle,
     verify_digest,
     verify_gzip_digests,
@@ -56,7 +57,7 @@
 from .sv_candidates import load_sv_candidates
 from .timeline import Event, Timeline
 
-__version__ = "0.14.4"
+__version__ = "0.15.0"
 
 # The public API. A few other helpers are importable from here because the OpenVax
 # libraries use them (digest, parse_variants, inspect_alignment, ...); everything
@@ -68,7 +69,7 @@
     # Reads and test data
     "extract_reads", "ReadFilter", "ReadSubset", "RecoveryPolicy",
     "bundle_file", "fetch_bundle", "list_bundle", "export_bundle", "check_fixtures", "verify_bundle",
-    "generate_bundle", "validate_recipe",
+    "generate_bundle", "subset_bundle", "validate_recipe",
     "load_panel",
     "load_sv_candidates",
     # Corrections

osteosarc/bundles.py

@@ -14,7 +14,7 @@
 
 from .cache import Cache, digest, file_lock, is_sha256, stable_id, write_json
 from .errors import IntegrityError, SchemaError
-from .fixtures import select_fixtures, validate_recipe
+from .fixtures import FixtureSelection, select_fixtures, validate_recipe
 from .models import File, Region
 from .reads import extract_reads
 from .records import RECORD_ENCODING, read_records, record_multiset
@@ -339,6 +339,46 @@ def list_bundle(bundle, *, cache=None):
     return verify_bundle(bundle, cache=cache)["members"]
 
 
+def subset_bundle(bundle, members, destination, *, recipe_id=None, cache=None,
+                  size_budget=DEFAULT_SIZE_BUDGET):
+    """Repack named members offline, preserving original headers and receipts.
+
+    Members are exact names, including empty, omitted and unresolved members.
+    Selection policies are not rerun: their recorded outcomes and reasons survive
+    unchanged. Only referenced sources, targets and record occurrences are kept.
+    The parent manifest checksum pins the provenance of this subset. A published
+    bundle name is downloaded on first use; a local folder needs no cache/network.
+    """
+    if isinstance(members, (str, bytes)):
+        raise TypeError("members must be an iterable of exact member names, not a string")
+    names = sorted(set(members))
+    root = _folder(bundle, cache)
+    manifest = verify_bundle(root)
+    if missing := set(names) - manifest["members"].keys():
+        raise KeyError(f"Unknown bundle members: {sorted(missing)}")
+    recipe = json.loads((root / "recipe.json").read_text())
+    recipe["id"] = recipe_id if recipe_id is not None else recipe["id"] + "-subset-" + stable_id(names)[:12]
+    recipe["members"] = {name: recipe["members"][name] for name in names}
+    for section, field in (("sources", "source"), ("targets", "target")):
+        needed = {m[field] for m in recipe["members"].values()}
+        recipe[section] = {key: value for key, value in recipe[section].items() if key in needed}
+    recipe = validate_recipe(recipe)
+    selected = {name: copy.deepcopy(manifest["members"][name]) for name in names}
+    active = {m["source"] for m in selected.values() if m["status"] not in ("omitted", "unresolved")}
+    headers, records = {}, {}
+    for sid in sorted(active):
+        source = manifest["sources"][sid]
+        headers[sid] = json.loads(safe_path(root, source["original_header"]).read_text())
+        records[sid] = list(read_records(safe_path(root, source["bam"])))
+    receipts = json.loads((root / "acquisition.json").read_text())
+    selection = FixtureSelection(recipe, selected, headers, records,
+                                 {sid: value for sid, value in receipts.items() if sid in recipe["sources"]})
+    return pack_bundle(selection, destination, header_policy=manifest["header_policy"],
+                       size_budget=size_budget,
+                       parent=dict(manifest_sha256=digest(root / "manifest.json"),
+                                   recipe_sha256=manifest["recipe_sha256"]))
+
+
 def export_bundle(bundle, to, *, members=None, format="bam", cache=None):
     """Write a bundle's members into a folder, each as a file named after it.
 

osteosarc/cli.py

@@ -225,6 +225,10 @@ def command(name, *, data=True, epilog=None):
                            "your newest, which must list each BAM unchanged; or give them with --source")
     make.add_argument("--source", action="append", default=[], metavar="ID=LOCAL_BAM",
                       help="With --recipe: use a BAM you already have for one of its sources")
+    make.add_argument("--from", dest="from_bundle", metavar="BUNDLE",
+                      help="Subset an existing bundle, keeping its original headers and acquisition receipts")
+    make.add_argument("--member", action="append", metavar="PREFIX",
+                      help="With --from: keep members whose names start with this prefix; repeat for several")
     make.add_argument("--header-policy", choices=("full", "compact"), default="full",
                       help="compact keeps only the header lines the records need")
     make.add_argument("--unplaced-mates", action="store_true",
@@ -610,13 +614,36 @@ def test_data(args, cache):
     """test-data make, list, verify, export and check."""
     from pathlib import Path
 
-    from .bundles import export_bundle, generate_bundle, list_bundle, sources_to_read, verify_bundle
+    from .bundles import (
+        export_bundle,
+        generate_bundle,
+        list_bundle,
+        sources_to_read,
+        subset_bundle,
+        verify_bundle,
+    )
     from .fixtures import validate_recipe
     from .shared import bundle_folder, check_fixtures, read_json
     from .views import plural, table
     action = args.test_data_command
     if action == "make":
-        if args.recipe:
+        if args.member and not args.from_bundle:
+            raise ValueError("--member goes with --from")
+        if args.from_bundle:
+            if (args.recipe or args.source or args.sources or args.variant or args.sv or args.assay
+                    or args.platform or args.unplaced_mates or args.header_policy != "full"):
+                raise ValueError("--from preserves the existing bundle's selections and header policy; "
+                                 "use only --member and --size-budget")
+            if not args.member:
+                raise ValueError("--from requires at least one --member prefix")
+            root = bundle_folder(args.from_bundle, cache=cache)
+            available = list_bundle(root)
+            for prefix in args.member:
+                if not any(name.startswith(prefix) for name in available):
+                    raise ValueError(f"No bundle member starts with {prefix!r}")
+            names = [name for name in available if any(name.startswith(p) for p in args.member)]
+            manifest = subset_bundle(root, names, args.output, size_budget=args.size_budget)
+        elif args.recipe:
             if args.sources or args.variant or args.sv or args.assay or args.platform or args.unplaced_mates:
                 raise ValueError("--recipe makes the bundle from the recipe alone, which says how each BAM is read; "
                                  "drop the BAMs, --variant, --sv and --unplaced-mates")