Stream depth probes and reconcile current sample metadata

MANIFEST.in

@@ -9,3 +9,4 @@ recursive-include tests/data/additional_svs *.bam *.bai *.md
 recursive-include tests/data/comprehensive *.json.gz *.md
 recursive-include tests/data/evidence-overlap *.bam *.md
 recursive-include tests/data/retrieval *.bam *.bai *.md
+recursive-include tests/data/otud4-review *.fasta *.vcf *.md

docs/changelog.md

@@ -5,6 +5,24 @@ your version with `osteosarc --version`. Pin both the package version and your
 snapshot (by name or download date) for reproducible analyses. Full release notes are on
 [GitHub](https://github.com/iskandr/osteosarc/releases).
 
+## Unreleased
+
+- Link FASTQs from published filename prefixes and the Tempus tumor/normal row
+  aliases, count only raw-read files, and generate working download commands.
+  FASTQ library metadata supplies assay/provider filters even for shared folders
+  such as CeGaT RNA delivered inside a WES directory.
+  Refresh the sample guide against all 31 samples in the October 7 metadata.
+  Data-page provider notes and mixed DNA/RNA analysis directories no longer
+  introduce false assay or provider conflicts.
+- Supplementary-depth probes stream record counts and cache small receipts instead
+  of entire BAM extracts. Existing verified probes and overflow results are reused;
+  the depth cap and selected fixture records are unchanged. Reused BAM counts are
+  checked against their original records, including compact acquisition receipts.
+- Recognize the reviewed 2026-10-05 upstream allele, ID, specimen and timeline
+  corrections while preserving historical snapshots and future drift checks.
+  A public CeGaT normal-DNA call supplies OTUD4's newly published literal allele;
+  the original Natera report and somatic status remain unverified.
+
 ## 0.15.4
 
 - Bounded extraction reads BAM pipes without constructing a pysam AlignmentFile

docs/corrections.md

@@ -90,8 +90,12 @@ that should still be there, since a record disappearing doesn't prove a fix.
 
 ## The corrections
 
-Checked against the website on 2026-09-24: 30 fixes apply and 5 are already fixed on
-the site; a snapshot from 2026-09-18 uses all 35. Each fix's evidence is in
+Reviewed again against the website on 2026-10-05. Fifteen additional layouts accept
+verified upstream changes, including renamed TRMO, literal MAP2/DCHS2/GAPVD1
+alleles, the removed split MAP2 entries, the expanded FAM157A protein caveat,
+corrected specimen descriptions, T0 Tempus tumor labels and ReyaGel's explicit end
+year. Historical snapshots retain their original corrections and unexpected
+future values still warn. Each fix's evidence is in
 `data.corrections` and the
 [source](https://github.com/iskandr/osteosarc/blob/main/osteosarc/curation.py). None
 only rewrites a value in an equivalent form: each fixes a wrong value, fills in a
@@ -114,8 +118,8 @@ missing one, or flags something to be careful with.
 | muc3a-grch38-placement | note | The Tempus call is on GRCh37, in a repeat that differs between builds, so its GRCh38 position stays unresolved. |
 | ush2a-transposed-duplicate | note | USH2A-chr1-215560752 is a typo of USH2A-chr1-215650752. The site has merged them; the kept entry's location label and sequence context are fixed. |
 | fam157a-withdrawn-protein | flag | NCBI withdrew the protein model behind FAM157A's insertion; the site now says so too. |
-| natera-alleles-unavailable | flag | COL3A1 and OTUD4 come from a Natera report that isn't public; a Tempus call now gives COL3A1's allele. |
-| otud4-source-unavailable | note | No public call gives OTUD4's allele, so the entry has none. |
+| natera-alleles-unavailable | flag | The original Natera report is not public. Independent Tempus and CeGaT calls supply matching COL3A1 and OTUD4 alleles; they do not verify that report or establish somatic status. |
+| otud4-source-unavailable | note | Historical entries remain unresolved. The reviewed 2026-10-05 entry matches a public CeGaT normal-DNA deletion call, with transcript and assembly checks; this does not verify the original Natera call or somatic status. |
 | transcript-DCHS2 | edit | NM_1142552 is missing two zeros: NM_001142552, which the site now uses. |
 | transcript-COL4A2 | edit | NM_001846. has a stray dot. |
 | transcript-GTF3C5 | edit | NM_00112283 is missing a digit: NM_001122823. |
@@ -153,10 +157,10 @@ cleared rather than read as zero.
 | specimen-T3-site | edit | T3 was an MSKCC resection, not a UCSF biopsy. |
 | pbmc-capture-dates | flag | Four blood samples are dated by cell capture; the blood was drawn two to four days earlier. |
 | events-duplicate-rows | flag | SQ3370 and Trabectedin each appear twice. |
-| tempus-timepoint | flag | The timeline dates the Tempus tests to T0 (2022), which the data supports, but the files are labeled T1 (2024-06). |
-| tempus-file-labels | flag | The Tempus files are labeled T1, but carry all three variants seen only at T0 and none of the 35 seen only at T1. The labels are left as published. |
+| tempus-timepoint | flag | The timeline dates the Tempus tests to T0 (2022), consistent with the data and the reviewed 2026-10-05 tumor-file labels. Historical file labels say T1 (2024-06). |
+| tempus-file-labels | flag | Historical Tempus files are labeled T1, but carry T0 evidence. The reviewed 2026-10-05 metadata labels tumor products T0 and leaves the later normal draw without a tumor timepoint. Published labels are retained; overlapping RNA products remain dependent evidence. |
 | apheresis-date | flag | The apheresis is 2024-05-14 on the timeline and 2024-05-15 in the ELISPOT records. |
-| reyagel-end-date | flag | ReyaGel's end date is 7/14, with no year, so it shows as one day; the only broken date of the sheet's 371. |
+| reyagel-end-date | flag | Historical ReyaGel's end date is 7/14 without a year. The reviewed 2026-10-05 sheet and event explicitly give 2026-07-14. |
 
 ## Other things to know
 

docs/reads.md

@@ -1,5 +1,12 @@
 # Reads
 
+For a depth check without retaining a BAM, use
+`data.count_reads(file, regions, max_records=1_000_000)`. The result has
+`records` and an acquisition `receipt`. It counts records in the indexed region
+union, including original duplicates, and does not fetch mates. Exceeding the cap
+raises `RecordLimitError`; complete counts and overflow results are cached for
+offline reuse. The standalone equivalent is `osteosarc.count_reads(...)`.
+
 Copy just the reads you need out of the remote BAMs into small local BAMs: test
 data for a unit test in seconds, without downloading any BAM whole. You need
 SAMtools 1.21 or newer on your PATH; reopening reads you already fetched

docs/samples.md

@@ -21,19 +21,21 @@ overlap is known, not proof of independence.
 The published FASTQ catalogue already identifies KCVBE1UI1P as a duplicate delivery
 of ALMY2X4KMV RNA. Corresponding mate objects also have matching lengths and multipart
 ETags (server metadata, not whole-file SHA-256 hashes). The overlap registry preserves
-that source receipt and the object metadata. The existing `tempus-file-labels`
-curation disputes the published T1 timepoint: several observations point to T0, but
-do not establish the specimen conclusively. Resolving that still requires specimen
-documentation; neither the alias nor this audit silently changes the timepoint.
+that source receipt and the object metadata. The reviewed October metadata now
+labels the Tempus tumor products T0 and records the matched normal separately as
+`blood_2024-09-21`. Historical snapshots retain the `tempus-file-labels` warning.
+The updated labels do not make the overlapping RNA products independent; the
+specimen-provenance investigation remains open.
 
 ## Catalogue
 
 - **Samples** are what was collected: tumor biopsies and resections at time points T0
-  to T3, an organoid grown from the T1 tumor, and blood draws.
+  to T3, organoids grown from the T1 and T2 tumors, and blood draws.
 - **Files** are what's in the public S3 bucket: aligned reads (BAMs), raw reads
   (FASTQs), variant calls, expression tables, scans and more, nearly 400,000 in all.
 
-Each sample has BAMs and FASTQ folders. Most files belong to no sample: imaging,
+Each sample lists its BAMs and FASTQ locations. A location can be a directory or
+a filename prefix. Most files belong to no sample: imaging,
 pathology slides, analyses and references. The examples open your most recent
 snapshot; see [get started](index.md#get-started).
 
@@ -57,23 +59,27 @@ data.samples.select(timepoint="T0", tissue="tumor")
 data.samples.select(assay="scrna-seq", platform="ont")
 ```
 
-The single-cell Oxford Nanopore samples in the 2026-09-25 snapshot:
+The single-cell Oxford Nanopore samples in the 2026-10-07 snapshot:
 
 ```text
 3 tumor samples
 
-sample    timepoint  date        where  sequencing                               BAMs  FASTQ folders
---------  ---------  ----------  -----  ---------------------------------------  ----  -------------
-T1_tumor  T1         2024-06-06  UCLA   rna-seq, wes, wgs, scrna-seq (ont,       14    14
-                                        pacbio)
-T2_tumor  T2         2025-01-28  UCLA   rna-seq, wgs, scrna-seq (ont), cite-seq  8     7
-T3_tumor  T3         2025-04-17  MSKCC  scrna-seq (ont), cite-seq                2     5
+sample    timepoint  date        where  sequencing                                  BAMs  FASTQ locations
+--------  ---------  ----------  -----  ------------------------------------------  ----  ---------------
+T1_tumor  T1         2024-06-06  UCLA   rna-seq, wes, wgs, scrna-seq (ont, pacbio)  12    10
+T2_tumor  T2         2025-01-28  UCLA   rna-seq, wgs, scrna-seq (ont), cite-seq     8     7
+T3_tumor  T3         2025-04-17  MSKCC  scrna-seq (ont), cite-seq                   2     5
 ```
 
 One sequencing run can have several BAMs, such as a vendor's and a reprocessed one.
 `data.samples["T1_tumor"]` shows one sample: where it came from, each BAM with its
 assay, provider, size and whether you've downloaded it, each FASTQ folder with its
-library, file count and size, and the calls that fetch them.
+library, FASTQ-only file count and size, and the calls that fetch them. Filename
+prefixes include the CeGaT, regenerated Natera and UCLA deliveries. The Tempus
+`DNA/FastQ/T` and `DNA/FastQ/N` row labels resolve to the full tumor and normal
+filename prefixes within their shared directory; they are not actual folders.
+`sample.fastq_folders` retains these published location labels, while the displayed
+locations and download commands use the resolved prefixes.
 
 ### What the fields mean
 
@@ -86,8 +92,14 @@ library, file count and size, and the calls that fetch them.
 | Platform | ont | The sequencing technology, when the site says |
 | Provider | BostonGene | Who produced the data |
 
-T0_tumor is the primary tumor resection on 2022-12-16, and T0_blood is blood from
-the same visit. T1_tumor and T1_organoid share a time point but not a tissue.
+T0_tumor is the primary tumor resection on 2022-12-16. T0_blood is a historical
+grouping of normal products, not proof that all blood was drawn that day: its
+current notes identify the BostonGene draw as 2023-01-27, while Natera normal
+collection details remain unconfirmed. The Personalis normal is recorded separately
+as blood_2024-06-11. T1_tumor and T1_organoid share a time point but not a tissue.
+T2_organoid comes from the 2025-01-28 UCLA biopsy; only its WGS is assigned to the
+sample because the delivered RNA contains substantial contamination from another
+person. The WGS notes also flag mouse reads and required preprocessing.
 T3_tumor_CD45neg is a CD45-negative fraction of T3_tumor. Later blood draws are
 named by date, such as blood_2025-06-26, and have no time point. A snapshot's date
 is when you downloaded the website's metadata, not a collection date.
@@ -107,10 +119,16 @@ and its corrections name the [fixes](corrections.md) that changed it.
 ### Where sequencing comes from
 
 A sample's sequencing combines the site's sample registry with its FASTQ table,
-which records each folder's assay. The four 2026 blood draws have nothing in the
-registry, but their FASTQ folders show single-cell gene expression, T-cell receptor
+which records each location's assay. The eleven 2026 blood entries in the October 7
+snapshot have no assays in the registry, but their FASTQ folders show single-cell
+gene expression, T-cell receptor
 and CITE-seq libraries.
 
+The FASTQ table also supplies each matched file's assay and provider for file
+filters. Its library-specific labels take precedence over a broader data-page
+directory, such as CeGaT RNA delivered inside a WES folder; conflicting claims
+about an exact file remain visible.
+
 The site's labels map to osteosarc's names like this; using a site label as a filter
 gives an error that names the right one:
 
@@ -119,6 +137,7 @@ gives an error that names the right one:
 | RNA, bulk RNA | rna-seq | | Bulk RNA sequencing |
 | WES | wes | | Whole-exome DNA sequencing |
 | WGS | wgs | | Whole-genome DNA sequencing |
+| Panel | panel | | Targeted DNA panel (Tempus xT) |
 | scRNA, scRNA_GEX | scrna-seq | gene expression | Single-cell RNA sequencing |
 | scRNA_TCR, scRNA_TCRgd, scRNA_BCR | scrna-seq | αβ TCR, γδ TCR, BCR | T- and B-cell receptor libraries from the same cells |
 | scRNA_ONT | scrna-seq, platform ont | long reads | Single-cell Oxford Nanopore RNA sequencing |

docs/testing.md

@@ -60,15 +60,27 @@ On 2026-09-18 this returned 3,788 reads, and reopened the same result offline.
 
 ## Last checked
 
-On a snapshot downloaded 2026-09-25 (UTC):
+On a snapshot downloaded 2026-10-07 (UTC), using the site's bucket inventory
+generated 2026-10-01:
 
 | Check | Result |
 | --- | --- |
-| Files | 395,541, including 843 alignments and 323 VCF or BCF files |
-| Variants | 181 on the variants page: 179 ready, 1 with a placeholder allele, 1 with none |
-| Corrections | 35: 30 applied, 5 already fixed on the site, none stale |
+| Files | 397,038 bucket objects; 397,043 catalogue entries including site tables; 913 alignments and 357 VCF or BCF files |
+| Variants | 183 on the variants page: 182 ready, 1 with a placeholder allele |
+| Corrections | 35: 13 applied, 22 already fixed on the site, none stale |
 | Unknown labels | None |
-| Timeline and samples | 790 events and 21 samples |
+| Timeline and samples | 807 events and 31 samples (24 blood, 5 tumor/fractions, 2 organoids) |
+
+The sample audit checks every registered BAM and FASTQ location, including the
+eleven entries represented as filename prefixes or Tempus row aliases rather
+than directories. The current tumor dates and collection sites agree with the
+timepoint summary. Regression excerpts and original source receipts are in
+`tests/data/sample-locations-2026-10-07.json`; older fixtures remain unchanged.
+Live S3 listings matched the inventory for all 317 FASTQ locations, and HEAD
+requests matched the sizes and modification dates of all 94 sample BAMs. One
+published FASTQ-table count differs: `rna-seq/tempus/TL-24-ALMY2X4KMV/RNA` lists
+two compressed FASTQs (1,238,829,178 bytes), but also contains `2022_R1.fastq`.
+The inventory and sample view retain all three files (4,470,300,364 bytes).
 
 The 2026-09-18 snapshot, from before the site renamed five variants and merged two
 USH2A entries, still applies all 35 corrections to its 182 entries. Every allele

osteosarc/__init__.py

@@ -47,9 +47,11 @@
 )
 from .read_receipts import read_receipt_files  # noqa: F401
 from .reads import (  # noqa: F401
+    ReadCount,
     ReadFilter,
     ReadSubset,
     assembly_from_header,
+    count_reads,
     extract_reads,
     inspect_alignment,
 )

osteosarc/catalog.py

@@ -10,6 +10,7 @@
 from .cache import stable_id
 from .curation import (
     ASSAYS,
+    FASTQ_PREFIX_ALIASES,
     label_assay,
     label_claims,
     normalize_provider,
@@ -109,7 +110,7 @@ def object_key(value, base=BUCKET):
     return value
 
 
-def build_files(listing, bams, metadata, vafs, path_claims=(), *, tables=None):
+def build_files(listing, bams, metadata, vafs, path_claims=(), *, tables=None, fastqs=()):
     """Retain every listed object, enriching exact paths before basename matches.
 
     Basename joins are used only when unique among alignment objects. Path
@@ -159,13 +160,25 @@ def build_files(listing, bams, metadata, vafs, path_claims=(), *, tables=None):
     by_path = defaultdict(list)
     for prefix, claim in path_claims:
         by_path[prefix.rstrip("/")].append(claim)
+    fastq_rows = [(object_key(row["s3_folder"], base), row) for row in fastqs if row.get("s3_folder")]
     files = []
     for key, object_metadata in objects.items():
         kind, format = file_type(key)
         info = dict(extra.get(key, {}))
         records = list(claims.get(key, ()))
+        if kind == "reads":
+            for location, row in fastq_rows:
+                if matches_fastq(key, location):
+                    assay, platform = ASSAYS.get(row.get("assay"), (None, None))
+                    by_path[key].append(SampleClaim(
+                        "fastqs", row.get("display_name", ""), normalize_timepoint(row.get("timepoint")),
+                        row.get("sample_date") or None, assay, platform,
+                        normalize_tissue(row.get("tissue")), normalize_provider(row.get("provider"))))
+                    info.setdefault("fastq_rows", []).append(dict(row))
         # The most specific data-page path wins: a file's own row overrides the
         # row for its directory, which can also hold other assays' files.
+        # A matched FASTQ library row likewise names the actual read files and
+        # overrides a broader directory's assay (e.g. CeGaT RNA filed under WES).
         parts = key.split("/")
         for depth in range(len(parts), 0, -1):
             if "/".join(parts[:depth]) in by_path:
@@ -232,7 +245,9 @@ def parse_data_paths(html):
     result = []
     for context, values in data_page_rows(html):
         title = context.lower()
-        assay = ("wgs" if "whole genome" in title or re.search(r"\bwgs\b", title) else
+        assay = (None if "variant calling" in title else
+                 "panel" if "panel" in values.get("Provider", "").lower() else
+                 "wgs" if "whole genome" in title or re.search(r"\bwgs\b", title) else
                  "wes" if "whole exome" in title or re.search(r"\bwes\b", title) else
                  "scrna-seq" if any(x in title for x in ("single cell", "single-cell")) else
                  "rna-seq" if "bulk rna" in title else None)
@@ -247,3 +262,26 @@ def parse_data_paths(html):
             assay=assay, platform=platform, tissue=tissue,
             provider=normalize_provider(values.get("Provider")))))
     return tuple(result)
+
+
+def fastq_prefix(location):
+    """Effective bucket prefix; retain the published location separately."""
+    location = location.rstrip("/")
+    return FASTQ_PREFIX_ALIASES.get(location, location)
+
+
+def matches_fastq(key, location):
+    """Match a FASTQ directory, exact file or filename prefix, never sidecars.
+
+    A filename prefix ends at a separator so P116686_1 cannot claim P116686_10.
+    This interprets published locations only; it does not infer sample identity.
+    """
+    if file_type(key) != ("reads", "fastq"):
+        return False
+    prefix = fastq_prefix(location)
+    if not prefix or not key.startswith(prefix):
+        return False
+    tail = key[len(prefix):]
+    if not tail or tail.startswith("/"):
+        return True
+    return "/" not in tail and (prefix.endswith(("_", "-", ".")) or tail[0] in "_.-")