fix(nextflow): give the Nextflow report every table the bash report reads, and run the CPSR read check there too

assets/output_contract.tsv

@@ -107,6 +107,7 @@ file	{sample}/cpic/{sample}_cpic_recommendations.txt	27	CPIC_LOOKUP	both	text	st
 file	{sample}/cpic/{sample}_phenotypes.tsv	27	CPIC_LOOKUP	both	tsv	stable	both	-
 file	{sample}/cpsr/{sample}.cpsr.grch38.classification.tsv.gz	17	CPSR	nextflow	tsv	stable	-	-
 file	{sample}/cpsr/{sample}.cpsr.grch38.html	17	CPSR	nextflow	html	stable	-	-
+file	{sample}/cpsr/{sample}_cpsr_cdna_check.tsv	17	CLINVAR_CDNA_CHECK	both	tsv	stable	-	-
 file	{sample}/cyrius/{sample}_cyp2d6.tsv	21	CYRIUS	nextflow	tsv	stable	-	-
 file	{sample}/cyrius/{sample}_cyp2d6_depth_check.tsv	21	CYRIUS	nextflow	tsv	stable	-	-
 file	{sample}/delly/{sample}_sv.vcf.gz	19	DELLY_BCF2VCF	nextflow	vcf	stable	nextflow	INFO/SVTYPE

bin/owngenome_live.py

@@ -678,7 +678,29 @@ def run_files(self, run):
         return out
 
     def run_file_allowed(self, run, rel):
-        return rel in self.run_files(run) or any(fnmatch.fnmatch(rel, g) for g in self.app.run_globs)
+        files = self.run_files(run)
+        if rel in files or any(fnmatch.fnmatch(rel, g) for g in self.app.run_globs):
+            return True
+        # The one HTML file directly in a folder the report links (cpsr/, which
+        # holds CPSR's own report): the folder request redirects to it (run_path).
+        folder = os.path.dirname(rel)
+        return bool(folder) and self.sole_html(run, folder, files) == os.path.basename(rel)
+
+    def sole_html(self, run, folder, files=None):
+        """The name of the only HTML file directly in a run folder the report links, inside the
+        run and the root; None when the folder is not linked or holds none or several."""
+        d = run["dir"]
+        if not d or folder not in (self.run_files(run) if files is None else files):
+            return None
+        p = os.path.join(d, folder)
+        try:
+            names = os.listdir(p) if os.path.isdir(p) else []
+        except OSError:
+            return None
+        top = os.path.realpath(d)
+        pages = [f for f in names if f.lower().endswith(".html") and os.path.isfile(os.path.join(p, f))
+                 and Live.inside(os.path.join(p, f), top) and Live.inside(os.path.join(p, f), self.root)]
+        return pages[0] if len(pages) == 1 else None
 
     def index_page(self):
         assets = self.cfg.assets
@@ -952,6 +974,14 @@ def run_path(self, run, rest):
             self.log("file", name, rel, 404)
             return self.deny(404)
         p = os.path.join(d, rel)
+        if os.path.isdir(p) and Live.inside(p, os.path.realpath(d)) and Live.inside(p, self.live.root):
+            # The site lists no folder: a linked folder with one HTML file in it
+            # (cpsr/ with CPSR's report) opens that file.
+            page = self.live.sole_html(run, rel)
+            if page:
+                self.log("file", name, rel, 302)
+                return self.send_body(302, "text/plain", "", [("Location", "/runs/" + urllib.parse.quote(name) + "/"
+                                                               + urllib.parse.quote(rel + "/" + page))])
         if not os.path.isfile(p) or not Live.inside(p, os.path.realpath(d)) or not Live.inside(p, self.live.root):
             self.log("file", name, rel, 404)
             return self.deny(404)

docs/17-cpsr.md

@@ -121,7 +121,7 @@ Run by hand like this, `--sample_id` must be 3 to 40 characters long. The script
 - `${SAMPLE}.cpsr.grch38.classification.tsv.gz` — Tab-separated variant classifications (gzipped; read it with `zcat`)
 - Every variant in the panel genes gets one of the five ACMG/AMP classes: Pathogenic, Likely pathogenic, VUS, Likely benign, Benign
 - In the TSV, `CLASSIFICATION` is the final class and `ASSERTION_AUTHORITY` says whose it is: ClinVar's, unless ClinVar has no record or a conflicted one, then CPSR's own (`CPSR_CLASSIFICATION`, which every variant has). PCGR 2.2.5 named the final column `FINAL_CLASSIFICATION`; the reports (`bin/collect_summary.py`) read either. The file names are the same in both versions.
-- `${SAMPLE}_cpsr_cdna_check.tsv` — step 6's cDNA read check on CPSR's P/LP rows (final class or CPSR's own), in the columns of step 6's `clinvar/${SAMPLE}_clinvar_cdna_check.tsv`. The rows are read from `VAR_ID` (or `GENOMIC_CHANGE`) and keep CPSR's contig names (`13`, not `chr13`); the BAM is read under either name. It needs the sample's BAM, `aligned/${SAMPLE}_sorted.bam` with its `.bai` (`ALIGN_DIR` picks another folder), and the GENCODE exons `setup.sh` installs; without either, every row is "not checked" with the reason. The Nextflow `CPSR` process does not run it: it gets no BAM, as `CLINVAR_SCREEN` does not.
+- `${SAMPLE}_cpsr_cdna_check.tsv` — step 6's cDNA read check on CPSR's P/LP rows (final class or CPSR's own), in the columns of step 6's `clinvar/${SAMPLE}_clinvar_cdna_check.tsv`. The rows are read from `VAR_ID` (or `GENOMIC_CHANGE`) and keep CPSR's contig names (`13`, not `chr13`); the BAM is read under either name. It needs the sample's BAM, `aligned/${SAMPLE}_sorted.bam` with its `.bai` (`ALIGN_DIR` picks another folder), and the GENCODE exons `setup.sh` installs; without either, every row is "not checked" with the reason. In Nextflow the `CLINVAR_CDNA_CHECK` process runs it after `CPSR` (source `cpsr`), with the samplesheet's BAM and `--gencode_exons`, and publishes the table to `cpsr/`; the HTML report puts it beside each P/LP row there too.
 
 ### What the reports show
 

docs/live-site.md

@@ -63,11 +63,11 @@ The reads cross the network to the browser that logged in: only the byte ranges
 
 ## Privacy and what is stored
 
-- The site serves only its own pages, the run's state, the report, the files the report links to by a relative link, and the genome viewer's files of the run (see above). Any other file is 404, and so is a link that leaves the run folder or a symlink out of `GENOME_DIR`. It never lists a folder.
+- The site serves only its own pages, the run's state, the report, the files the report links to by a relative link, and the genome viewer's files of the run (see above). Any other file is 404, and so is a link that leaves the run folder or a symlink out of `GENOME_DIR`. It never lists a folder: a link to a folder that holds one HTML file (the report's `cpsr/`, with CPSR's own report) opens that file, and any other folder link is 404.
 - Each message is cut to what a progress view needs before it is stored: the task, its process and status, times, CPU and memory figures. Nextflow's parameters, command line and task scripts are dropped. Three paths stay on the server for later views (the output folder, the CPU and memory caps, the work directory); none of them reaches a page.
 - Messages go to `<state dir>/runs/<run>/events.jsonl` (state dir: `GENOME_DIR/.owngenome-live`, or `OWNGENOME_LIVE_STATE_DIR`). The site writes nowhere else, so `GENOME_DIR` can be mounted read-only.
 - The access log, `<state dir>/access.log`, has one tab-separated row per page or file read and per login (the genome viewer's files excepted: one row per page open, see above): time, client, the first 8 characters of a hash of the session, action, run, file, status. It never holds a query string, a request body, a result or the password. It moves to `access.log.1` past `--access-log-max-mb` (10).
-- Every page carries a Content-Security-Policy that lets it talk to this server only, and the page loads no script from anywhere else. The run's own HTML (the report, CPSR, PharmCAT, MultiQC) is sent sandboxed: no script runs and nothing outside is fetched.
+- Every page carries a Content-Security-Policy that lets it talk to this server only, and the page loads no script from anywhere else. The run's own HTML (the report, CPSR, PharmCAT, MultiQC) is sent sandboxed: no script runs and nothing outside is fetched. CPSR's report draws its variant tables with scripts, so on the live site those tables stay empty: open the file from `<sample>/cpsr/` on disk to see them.
 
 ## The password and the limits
 

docs/nextflow.md

@@ -6,7 +6,7 @@ The pipeline is a [Nextflow](https://www.nextflow.io/) DSL2 pipeline, `main.nf`.
 - **a BAM or a CRAM** without a VCF: it is called;
 - **a VCF** from any caller (nf-core/sarek, DRAGEN, a provider), with an optional BAM or CRAM.
 
-Every BAM then goes through a sex check (indexcov), and the pipeline runs pharmacogenomics, variant annotation, clinical screening, BAM analyses, structural variant calling and reporting: 8 workflows, 85 processes in 47 module files under `modules/local/`. A VCF given in the samplesheet needs FILTER=PASS records and GRCh38 contig names with chr; see [FILTER=PASS required](#filterpass-required) and [Contig names and gVCF input](#contig-names-and-gvcf-input). Starting from a provider's VCF: [Starting from a Vendor VCF](vcf-first.md).
+Every BAM then goes through a sex check (indexcov), and the pipeline runs pharmacogenomics, variant annotation, clinical screening, BAM analyses, structural variant calling and reporting: 8 workflows under `workflows/`, built from the processes under `modules/local/` (one folder per tool). A VCF given in the samplesheet needs FILTER=PASS records and GRCh38 contig names with chr; see [FILTER=PASS required](#filterpass-required) and [Contig names and gVCF input](#contig-names-and-gvcf-input). Starting from a provider's VCF: [Starting from a Vendor VCF](vcf-first.md).
 
 > **Nextflow is the pipeline; the scripts are single steps.** Each numbered script in `scripts/` runs one step on its own and takes its image tags from the same `versions.env` and its helpers from `scripts/lib/common.sh`; `run-all.sh` is a launcher: it writes a one-row samplesheet and starts this pipeline with `-resume`, then runs the few script-only steps you ask for and the reports. CI runs the scripts and the pipeline on the same reads and fails when their results differ (see [Bash vs Nextflow parity](#bash-vs-nextflow-parity)). The Singularity profile is untested (see [Profiles](#profiles)).
 
@@ -207,21 +207,31 @@ results/
 │   ├── vep/                # VEP VCF, and the vcfanno-enriched VCF when a score file is set
 │   ├── slivar/             # Prioritized variants + compound hets
 │   ├── clinical/           # Clinically relevant variant subset
-│   ├── cpsr/               # Cancer predisposition report
+│   ├── cpsr/               # Cancer predisposition report: CPSR's HTML and classification table, and the cDNA
+│   │                       #   read check of its P/LP rows (<sample>_cpsr_cdna_check.tsv)
 │   ├── roh/                # Runs of homozygosity
 │   ├── prs/                # Polygenic risk scores
 │   ├── ancestry/           # Ancestry: projection onto the panel (with --ancestry_ref)
 │   ├── ancestry_fine/      # Fine-grained ancestry: fractions, painting BED and PNG (opt-in: ancestry_fine)
 │   ├── mito/               # Mitochondrial variant calls with their ClinVar and gnomAD annotation, the haplogroup
 │   │                       #   from them and haplocheck's contamination check
 │   ├── y_haplogroup/       # Y-chromosome haplogroup of male samples (Yleaf, opt-in: y_haplogroup)
+│   ├── y_str/              # <sample>_y_str.tsv, .csv and the calls VCF: Y-STR repeat counts of male samples (opt-in: y_str)
 │   ├── bloodtype/          # Blood group antigens predicted from the genome (opt-in: bloodtype)
 │   ├── lpa/                # LPA KIV-2 repeat count from read depth (opt-in: lpa)
 │   ├── chip/               # <sample>_chip.tsv, <sample>_chip_depth.tsv and Mutect2's filtered calls: clonal
 │   │                       #   haematopoiesis in 13 genes (opt-in: chip)
+│   ├── vntyper/            # <sample>_vntyper.tsv and VNtyper's run folder: the MUC1 VNTR screen (opt-in: vntyper)
+│   ├── viral/              # <sample>_viral.tsv, the coverage of each virus and of the human genome, and the viral reads BAM (opt-in: viral)
+│   ├── exomiser/           # <sample>_exomiser_top.tsv, the text summary, the HPO terms and Exomiser's own results:
+│   │                       #   variants ranked against the symptoms (--hpo, exomiser)
+│   ├── mei/                # <sample>_mei.tsv, <sample>_mei_sites.tsv, the depth table and MEGANE's VCF: mobile element insertions (opt-in: mei)
+│   ├── mosaic/             # <sample>_mocha.tsv and <sample>_mocha_summary.tsv: mosaic chromosomal changes (MoChA, opt-in: mocha)
+│   ├── locityper/          # <sample>_locityper.tsv and Locityper's work folder: genotypes of hard genes (opt-in: locityper)
 │   ├── archaic/            # Neanderthal and Denisovan share and segments (IBDmix, opt-in: archaic)
 │   ├── ancient/            # The nearest ancient groups (table and text), their info and the PCA plot (AADR, opt-in: ancient)
 │   ├── hla/                # HLA typing
+│   ├── hla_optitype/       # OptiType's HLA-A and HLA-B, and <sample>_hla_concordance.tsv against T1K (--hla_qc)
 │   ├── expansion_hunter/   # Repeat expansion calls, Stranger's annotation and, when Stranger flags RFC1, its motif count
 │   ├── telomere/           # Telomere length estimation
 │   ├── coverage/           # Coverage statistics (mosdepth)
@@ -347,7 +357,7 @@ Where a module and its script differ on purpose:
 | Mito variants (20) | extracts the chrM reads with `samtools view`; annotates the calls with `clinvar/clinvar_chr.vcf.gz` and the gnomAD chrM sites under `gnomad_mito/` when they are installed | with GATK `PrintReads`. Mutect2 applies its own read filters to either, so the calls are expected to match; CI does not compare them. Both mark possible NuMTs at the median autosomal depth from mosdepth (step 16b, or `MOSDEPTH` when `mosdepth` is in `--tools`). `MITO_VARIANTS_ANNOTATE` runs the same annotation (`bin/collect_summary.py mito-annotate`) with the files of `--mito_clinvar` and `--gnomad_mito`; without them the table names each file as not given and leaves its columns `.`. `run-all.sh` passes `--mito_clinvar` when `clinvar/clinvar_chr.vcf.gz` and its `.tbi` are installed, and `--gnomad_mito` when the gnomAD chrM sites are |
 | AnnotSV (05) | annotates step 15's duphold-filtered calls; without them, or when Manta's calls are newer, Manta's calls, and says so | `ANNOTSV` always reads `DUPHOLD_FILTER`'s output: `annotsv` in `--tools` needs `duphold` and `manta` |
 | Y haplogroup (37) | reads the sex step 16 (indexcov) infers | runs on the rows whose samplesheet sex is male, which `INDEXCOV` has checked against the reads |
-| HTML report (24) | renders every section from `bin/collect_summary.py`'s summary | `HTML_REPORT` runs the same code on the outputs of this run's steps, among them the cDNA read check of the ClinVar hits, the CYP2D6 depth check, the phased compound-het pairs, the RFC1 motif count, AnnotSV's class 4 and 5 table, the chrM annotation and step 35's HBA1/HBA2 count. The clinical filter and slivar cards show counts only, beside slivar's pairs by phase (the module gets their VCFs, not their summary tables). For every section, run `GENOME_DIR=<outdir> scripts/24-html-report.sh <sample>` on the Nextflow output |
+| HTML report (24) | renders every section from `bin/collect_summary.py`'s summary | `HTML_REPORT` runs the same code on the outputs of this run's steps, among them the cDNA read check of the ClinVar hits, the CYP2D6 depth check, the phased compound-het pairs, the RFC1 motif count, AnnotSV's class 4 and 5 table, the chrM annotation and step 35's HBA1/HBA2 count. It also gets the VEP VCF (each ClinVar hit's gnomAD AF and rsIDs), CPSR's classification table and the cDNA read check of its P/LP rows, `ALIGN_MINIMAP2`'s read count, and the clinical filter's and slivar's summary tables. The slivar card has no compound-het variant and gene counts: the Nextflow slivar module writes no `<sample>_compound_hets.tsv`. For every section, run `GENOME_DIR=<outdir> scripts/24-html-report.sh <sample>` on the Nextflow output |
 | CNVpytor (18) | mounts each resource file over the image's data folder | copies the files into the image's `site-packages`, so it needs a writable container |
 | Cyrius (21) | runs Cyrius from the install `setup.sh --cyrius` made | the same install, given as `--cyrius_install` (see [Cyrius, opt-in](#cyrius-opt-in)) |
 
@@ -433,7 +443,7 @@ Cyrius (CYP2D6 star alleles) is under a non-commercial licence and is not in the
 
 ### CI validation scope
 
-`nextflow.yml` lints the pipeline and stub-runs every process, on the pinned Nextflow release and the newest 26.10.x, with one samplesheet row per starting point; it also checks that `T1K_BUILD` ran once for all samples, and that a VCF+BAM-only samplesheet against a reference with no `.sr.mmi` beside it finishes without building an index or aligning anything. A job that fails prints which tasks were running or waiting and uploads the Nextflow logs as the artifact `nextflow-logs-<line>`. The E2E workflow runs the pipeline with real containers on a slice of the public HG002 genome: from the FASTQ pair through alignment, the sex check and DeepVariant to the default tools but cyrius (the fixture's BAM lacks the autosomal bins Cyrius normalises on), plus clinvar and hla_typing; through the VCF+BAM entry, over several cases, with clinvar, mosdepth, delly, manta, vcfanno, roh, pharmcat, cpic, pypgx, telomere_hunter, mito_haplogroup, survivor_merge, y_haplogroup, sample_qc, prs, cram_archive and html_report (the prs case's row names a gVCF too); and the sex-check stop (see [Testing](testing.md#the-e2e-job)). It then runs the parity check above. It cannot run the tools whose databases do not fit a CI runner (vep's offline cache, cpsr) or the ones it has no data for. Before trusting results from a new installation, run the pipeline on a known sample and compare key outputs (PharmCAT star alleles, ClinVar hit counts) against expected values.
+`nextflow.yml` lints the pipeline and stub-runs every process, on the pinned Nextflow release and the newest 26.10.x, with one samplesheet row per starting point; it also checks that `T1K_BUILD` ran once for all samples, and that a VCF+BAM-only samplesheet against a reference with no `.sr.mmi` beside it finishes without building an index or aligning anything. A job that fails prints which tasks were running or waiting and uploads the Nextflow logs as the artifact `nextflow-logs-<line>`. The E2E workflow runs the pipeline with real containers on a slice of the public HG002 genome: from the FASTQ pair through alignment, the sex check and DeepVariant to the default tools but cyrius (the fixture's BAM lacks the autosomal bins Cyrius normalises on), plus clinvar and hla_typing; through the VCF+BAM entry, over several cases, with clinvar, mosdepth, delly, manta, vcfanno, roh, pharmcat, cpic, pypgx, telomere_hunter, mito_haplogroup, survivor_merge, y_haplogroup, sample_qc, prs, cram_archive and html_report (the prs case's row names a gVCF too), and duphold, expansion_hunter and cnvpytor (case `nf-wiring-2-sv-str-tools`); and the sex-check stop (see [Testing](testing.md#the-e2e-job)). It then runs the parity check above. It cannot run the tools whose databases do not fit a CI runner (vep's offline cache, cpsr) or the ones it has no data for. Before trusting results from a new installation, run the pipeline on a known sample and compare key outputs (PharmCAT star alleles, ClinVar hit counts) against expected values.
 
 ---