Repository navigation
Stream depth probes and reconcile current sample metadata - #102
Merged
Merged
Conversation
iskandr
force-pushed
the
fix/comprehensive-read-disk-use
branch
from
October 5, 2026 14:50
354a9dd to
27d2901
Compare
iskandr
marked this pull request as ready for review
October 7, 2026 16:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supplementary-depth probes now stream record counts and cache verified receipts instead of retaining large BAM extracts before fetching accepted regions again. Record caps, selected fixture records, timeout/overflow handling, and offline reuse remain intact. Verified legacy BAM probes are recounted before reuse. This branch includes current main's inventory modification-date safeguards, with regression coverage for new and legacy count receipts.
Reconcile the reviewed October curation changes while retaining historical layouts and future drift guards. This covers renamed/corrected alleles, specimen sites and dates, Tempus labels, and timeline updates. A public CeGaT normal-DNA call supplies the newly published OTUD4 literal allele; it does not establish somatic status or verify the unavailable original Natera report. Tempus RNA overlap warnings remain in place; #100 remains open.
The October 7 sample audit also found eleven FASTQ locations represented as filename prefixes or specimen row aliases rather than directories. Sample views now link those files, preserve the original location labels, count only FASTQs, and generate usable download commands. The Tempus
/Tand/Naliases select the corresponding full accession/tissue prefixes without mixing normal and tumor reads. Library-specific FASTQ metadata supplies assay/provider filters, including CeGaT RNA delivered in a WES directory. Explanatory provider labels and mixed DNA/RNA analysis directories no longer create false catalogue conflicts; conflicting exact-file claims are still retained.The sample guide now describes all 31 current samples, including both organoids, the separately recorded Personalis and Tempus normal draws, eleven 2026 blood entries, T0 normal grouping caveats, and the excluded contaminated T2 organoid RNA. Checked source excerpts and receipts are included.
Validation:
PR CI passes on Python 3.9–3.14, both consumer configurations, and documentation.
495 local tests pass, including count-stream integrity, inventory-date enforcement, historical/future curation, all eleven FASTQ location cases, sample sites/dates, and exact-file conflict preservation.
Ruff, strict documentation, wheel/sdist builds, package-content checks, and
pip checkpass. All sample-guide examples pass against the refreshed snapshot.October 7 snapshot: 397,038 bucket objects, 31 samples, 807 timeline events, and 183 site variants.
corrections --strictreports 22 fixed upstream, 13 applied, no stale rules or unknown labels.Live S3 listings match every one of the 317 registered FASTQ locations; HEAD checks match sizes and modification dates for all 94 sample BAMs. No alignment or FASTQ metadata conflicts remain in this snapshot.
A fresh five-region live stream returned 962 records with no stored BAMs, matched the snapshot's modification date, and reused its count offline.
One upstream count discrepancy is documented: the Tempus ALMY2X4KMV RNA folder lists two compressed FASTQs but also holds
2022_R1.fastq; the inventory and sample view retain all three files.The published openvax-v1/v2 bundles and frozen v3 catalogue are unchanged. Full v3 acquisition and delivery remain separate work.
Fixes #103.