Skip to content

Stream depth probes and reconcile current sample metadata - #102

Merged
iskandr merged 4 commits into
mainfrom
fix/comprehensive-read-disk-use
Oct 7, 2026
Merged

iskandr merged 4 commits into
mainfrom
fix/comprehensive-read-disk-use

Conversation

@iskandr

@iskandr iskandr commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Supplementary-depth probes now stream record counts and cache verified receipts instead of retaining large BAM extracts before fetching accepted regions again. Record caps, selected fixture records, timeout/overflow handling, and offline reuse remain intact. Verified legacy BAM probes are recounted before reuse. This branch includes current main's inventory modification-date safeguards, with regression coverage for new and legacy count receipts.

Reconcile the reviewed October curation changes while retaining historical layouts and future drift guards. This covers renamed/corrected alleles, specimen sites and dates, Tempus labels, and timeline updates. A public CeGaT normal-DNA call supplies the newly published OTUD4 literal allele; it does not establish somatic status or verify the unavailable original Natera report. Tempus RNA overlap warnings remain in place; #100 remains open.

The October 7 sample audit also found eleven FASTQ locations represented as filename prefixes or specimen row aliases rather than directories. Sample views now link those files, preserve the original location labels, count only FASTQs, and generate usable download commands. The Tempus /T and /N aliases select the corresponding full accession/tissue prefixes without mixing normal and tumor reads. Library-specific FASTQ metadata supplies assay/provider filters, including CeGaT RNA delivered in a WES directory. Explanatory provider labels and mixed DNA/RNA analysis directories no longer create false catalogue conflicts; conflicting exact-file claims are still retained.

The sample guide now describes all 31 current samples, including both organoids, the separately recorded Personalis and Tempus normal draws, eleven 2026 blood entries, T0 normal grouping caveats, and the excluded contaminated T2 organoid RNA. Checked source excerpts and receipts are included.

Validation:

  • PR CI passes on Python 3.9–3.14, both consumer configurations, and documentation.

  • 495 local tests pass, including count-stream integrity, inventory-date enforcement, historical/future curation, all eleven FASTQ location cases, sample sites/dates, and exact-file conflict preservation.

  • Ruff, strict documentation, wheel/sdist builds, package-content checks, and pip check pass. All sample-guide examples pass against the refreshed snapshot.

  • October 7 snapshot: 397,038 bucket objects, 31 samples, 807 timeline events, and 183 site variants. corrections --strict reports 22 fixed upstream, 13 applied, no stale rules or unknown labels.

  • Live S3 listings match every one of the 317 registered FASTQ locations; HEAD checks match sizes and modification dates for all 94 sample BAMs. No alignment or FASTQ metadata conflicts remain in this snapshot.

  • A fresh five-region live stream returned 962 records with no stored BAMs, matched the snapshot's modification date, and reused its count offline.

  • One upstream count discrepancy is documented: the Tempus ALMY2X4KMV RNA folder lists two compressed FASTQs but also holds 2022_R1.fastq; the inventory and sample view retain all three files.

The published openvax-v1/v2 bundles and frozen v3 catalogue are unchanged. Full v3 acquisition and delivery remain separate work.

Fixes #103.

@iskandr
iskandr force-pushed the fix/comprehensive-read-disk-use branch from 354a9dd to 27d2901 Compare October 5, 2026 14:50
@iskandr iskandr changed the title Stream supplementary depth probes without caching BAMs Stream supplementary depth probes and reconcile reviewed curation drift Oct 5, 2026
@iskandr iskandr changed the title Stream supplementary depth probes and reconcile reviewed curation drift Stream depth probes and reconcile current sample metadata Oct 7, 2026
@iskandr
iskandr marked this pull request as ready for review October 7, 2026 16:32
@iskandr
iskandr merged commit c52e9d8 into main Oct 7, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconcile stale sample and variant curation guards with the October metadata snapshot

1 participant