diff --git a/docs/testing.md b/docs/testing.md index e3ef3253..b0d663e6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -102,7 +102,7 @@ The base leg runs every base case whenever any case runs, because every shard st The images the cases use are cached in four GitHub Actions cache entries, one per group. `scripts/ci/e2e-run.sh --image-groups` names each image's group, taken from its repository name without the tag, so a version bump changes the key of one group only; a group whose key missed restores the newest older entry of that group and pulls only what changed. A group keeps its smallest images up to 1500 MB of zstd output, and the larger ones are pulled on every run. Four groups of 1500 MB and the six data entries below stay under the repository's 10 GB cache limit even while main replaces one group: 5.9 GB of images, 1.5 GB more for the group being replaced, and about 2.4 GB of data. Images pinned by digest are not cached, because `docker load` gives them back without the digest, and they are left out of the group's key, so a digest bump misses no entry. Only the base leg saves, and only on main (or in a run by hand with `save_images`, on any branch), after it pulled every image the cases use and got through its cases, and only the groups whose key missed: a cache saved by a pull request can be read by that pull request alone and would only push main's entries out. The job summary lists every image with its size and what the cache did with it: restored on an exact key hit, restored from an older entry, saved, or pulled. Every merge in the e2e scope starts a run on main, so the cache follows main's images. GitHub drops an entry no run has read for 7 days, so after a quiet week the next run on main fills the cache again. Until this layout the cache was one entry, and by October 2026 it had grown to about 11 GB, over the limit, so every run missed it. -Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. The one builder download it lacks is the TRExplorer catalog of `build-sv-rule-regions.sh`, which is not on EBI: on a miss it comes from its own GitHub release. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. +Six downloads are cached the same way. Two come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case `mocha-1`), keyed on `MOCHA_PANEL_MANIFEST_SHA256` and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case `archaic-ancestry-1`), keyed on `ARCHAIC_1KG_SITES_MD5`. On a miss, each of those two is tried first at `E2E_DATA_MIRROR`, the GitHub release `e2e-data-1` that holds copies of the pinned files, and then at EBI; the pin checks both. The release also holds the four PGS Catalog score files, the HAPNEST panel and the table builders' downloads below, so an EBI outage costs none of them on a cold cache. It holds the TRExplorer catalog of `build-sv-rule-regions.sh` too (the v2.1 file its `TRX_SHA256` pins), so a miss tries the mirror first and the catalog's own GitHub release after it. Two are small and failed whole runs on 2026-11-04, because their case is in the base leg or a shard that cannot start without it: `hla.dat` of the two IPD-IMGT/HLA releases `bash-step-08-hla` installs, from raw.githubusercontent.com (which served it under 1 KB/s), with GENCODE's gene and exon files from EBI's FTP server (curl exit 7 and 28), and the four real PGS Catalog score files of `prs-scores-and-run-options-1-strand-sex`, from the same server. `scripts/ci/e2e-run.sh --data-key` names their keys: the releases and a hash of the code in `scripts/lib/common.sh` that installs them, and a hash of the score ids and their server. `scripts/ci/e2e-run.sh` hands each case its files just before it runs, at the paths `setup.sh` and `install_data_file` read them from, so they download nothing; the score files the case copies from the cache itself. On a miss it downloads them first, checked against their pins or the checksum files the servers publish beside them (`MD5SUMS`, `md5checksum.txt`, the `.md5` of each score), trying up to 6 times 30 s apart. The four score files are pinned by md5 in `PGS_MD5` of `e2e-run.sh` (the md5 the PGS Catalog served beside each on 2026-11-06), tried on the mirror first and then at EBI; a restored or downloaded score must match its pin as well as its `.md5`. A file gets at most `E2E_MIRROR_BUDGET` seconds on the mirror (900, or half the fill's budget when that is less), so a mirror that trickles leaves EBI its turn. One fill may take at most `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`): a source that trickles above `fetch`'s stall limit would otherwise run on to the leg's 240-minute limit, as EBI's 1000 Genomes server did twice on 2026-11-05. A fill still running then is stopped and its case fails as `FAIL (data fill over 2400s)` without running, since its own download would be as slow. A restored entry is checked again before use (`--data-check`): the IPD-IMGT/HLA and GENCODE entry against the sha256 list written when it was filled, the score files against their `.md5`; one that fails is downloaded again. `bash-step-08-hla` then runs with every HTTP and HTTPS request sent to a closed port, after a control shows that a request under that block fails at the closed port, so a warm cache cannot reach either host. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with `save_images`, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. The fifth is the PGS Catalog's synthetic HAPNEST panel (0.27 GB, cases `prs-2-ancestry` and `prs-scores-and-run-options-2-panel-chrx`) from EBI's FTP server, which failed cases with curl exit 7 on 2026-11-04. It is keyed on the sha256 `tests/e2e/lib.sh` pins (`HAPNEST_SHA256`). The cases handle it themselves (`setup_ancestry_panel` in `lib.sh`): with the panel in the cache, or on a miss from `E2E_DATA_MIRROR`, they check the sha256 and run `setup.sh --ancestry-panel` with every HTTP and HTTPS request sent to a closed port, so an EBI outage cannot fail them; when neither has it, `setup.sh` downloads the panel from EBI as a user's run does. The first case puts a panel that did not come from the cache in the cache for the save step. A red control in `prs-2-ancestry` runs `setup.sh` with the network blocked and no panel, which must fail. The sixth, entry `catalogs`, holds the pinned downloads of `scripts/ci/build-callability.sh`, `scripts/ci/build-cn-intervals.py` and `scripts/ci/build-sv-rule-regions.sh`, whose cases `callability-1-catalog`, `cn-intervals-1-catalog` and `followups-ci-3-sv-rule-regions` rebuild `assets/callability_genes.tsv`, the interval rows of `bin/cn_loci.tsv` and `assets/sv_rule_regions.tsv` and compare them with the committed files. Each builder lists its downloads with `--list`; the key is a hash of those lists, so a new pin is a new key. GENCODE's GTF among them comes from EBI's FTP server, which refused every connection for minutes on 2026-11-06 and failed `callability-1-catalog` on main (run 42259432681). The runner fills the entry as it does the others, mirror first and every file checked against its sha256, and each case points its builder's cache at its folder there; a builder that still has to download tries for about five minutes, 30 s apart. Every download, there and in `setup.sh`, gives up on a transfer that stalls (`FETCH_STALL`, see `fetch` in `scripts/lib/common.sh`), and the next try resumes it. How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware. diff --git a/nextflow.config b/nextflow.config index 5b95c3bf..314c00b6 100644 --- a/nextflow.config +++ b/nextflow.config @@ -163,7 +163,10 @@ nextflow.enable.moduleBinaries = true // VCF) and haplocheck, which only reads Mutect2's calls; the ANCESTRY_FINE_* // processes of step 26b, the CHIP_* processes of step 49 and the FAMILY_IBD_* // processes of step 38 (the shared segments, a leaf of the family workflow) -// feed only the report. main.nf's completion +// feed only the report. So do the cDNA read checks: CLINVAR_CDNA_CHECK +// (step 06) and CPSR_CDNA_CHECK (step 17, the same process under an alias) +// write a table that only the report joins; a sample whose check failed shows +// its ClinVar or CPSR hits without the read check. main.nf's completion // handler names every ignored task and writes failed_tasks.tsv, and run-all.sh // marks its step failed. The PARAPHASE processes are here too: PARAPHASE_CHECK // fails on a BAM that is not PacBio HiFi, and the rest of the run goes on. @@ -172,8 +175,11 @@ nextflow.enable.moduleBinaries = true // the stop. A withName setting wins over base.config's process-wide one. The // exit-code list is the one in conf/base.config's errorStrategy: change both // (tests/test_retry_codes.sh fails when they differ). +// scripts/ci/check-report-only.py (tests/test_contract_report_only.py) finds +// each workflow process whose channels reach main.nf only through the +// report's remainder joins and fails when no block here names it. process { - withName: 'TELOMERE_HUNTER|EXPANSION_HUNTER|STRANGER|STRANGER_RFC1|MITO_VARIANTS|MITO_VARIANTS_ANNOTATE|ANNOTSV|CPSR|Y_POSITIONS|Y_PILEUP|Y_HAPLOGROUP|Y_STR|BLOODTYPE_REGIONS|BLOODTYPE_DEPTH|BLOODTYPE|LPA_REGIONS|LPA_DEPTH|LPA|VNTYPER|LOCITYPER|VIRAL_READS|VIRAL_REPORT|EXOMISER_PREPARE|EXOMISER|EXOMISER_REPORT|MEI_DEPTH|MEI|MEI_REPORT|TRAITS_SITES|TRAITS_PHASE|TRAITS|MULTIQC|CONSUMER_EXPORT|PARAPHASE_CHECK|PARAPHASE|PARAPHASE_REPORT|ANCESTRY_FINE_.*|CHIP_.*|ARCHAIC_.*|FAMILY_IBD_.*|MOCHA_.*' { + withName: 'TELOMERE_HUNTER|EXPANSION_HUNTER|STRANGER|STRANGER_RFC1|MITO_VARIANTS|MITO_VARIANTS_ANNOTATE|ANNOTSV|CPSR|CPSR_CDNA_CHECK|CLINVAR_CDNA_CHECK|Y_POSITIONS|Y_PILEUP|Y_HAPLOGROUP|Y_STR|BLOODTYPE_REGIONS|BLOODTYPE_DEPTH|BLOODTYPE|LPA_REGIONS|LPA_DEPTH|LPA|VNTYPER|LOCITYPER|VIRAL_READS|VIRAL_REPORT|EXOMISER_PREPARE|EXOMISER|EXOMISER_REPORT|MEI_DEPTH|MEI|MEI_REPORT|TRAITS_SITES|TRAITS_PHASE|TRAITS|MULTIQC|CONSUMER_EXPORT|PARAPHASE_CHECK|PARAPHASE|PARAPHASE_REPORT|ANCESTRY_FINE_.*|CHIP_.*|ARCHAIC_.*|FAMILY_IBD_.*|MOCHA_.*' { errorStrategy = { task.exitStatus in [143,137,104,134,139,247] && task.attempt == 1 ? 'retry' : 'ignore' } } } diff --git a/scripts/ci/check-report-only.py b/scripts/ci/check-report-only.py new file mode 100755 index 00000000..a10fcee7 --- /dev/null +++ b/scripts/ci/check-report-only.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Every process whose output only the report reads has the report-only error strategy. + +nextflow.config gives report-only processes a withName errorStrategy that +retries a task killed for memory or time once and otherwise ignores the +failure, so one odd input cannot stop the run before DeepVariant and the +report. A process left out keeps conf/base.config's stop. This check finds +the processes whose output only the report reads and fails for each one no +such block names. + +A process counts as report-only when, in its workflows/.nf, every +reference to .out. is either `ch_versions = +ch_versions.mix(.out.versions)` or a plain `ch_x = .out.`, +that ch_x is used nowhere else in the file but in its own assignments and in +the emit block, and main.nf uses every such emit only in a line +`.join(.out.... remainder: true)`, the report's joins that keep +a sample whose step is missing. Anything else (a map on the channel, a second +consumer, a join without remainder) makes the process a feeder of something +else, and it is not asked for. The check errs toward silence: it never asks for +a process that feeds another step. + +KEEP_STOP names the report-only processes that keep the stop by choice. A name +there that the scan no longer finds as report-only is an error too, so the +list cannot go stale. + +Usage: check-report-only.py [--repo DIR] +Exit 0 when every report-only process is covered, 1 when one is not, 2 on a +usage error. Only the standard library. +""" +import argparse +import os +import re +import sys + +# Report-only by their channels, kept fatal until a decision says otherwise: +# ROH (step 17's runs of homozygosity) and CPIC_LOOKUP (the CPIC +# recommendations) have not been through the report-only review yet. +KEEP_STOP = {"ROH", "CPIC_LOOKUP"} + +IGNORE_BLOCK = re.compile( + r"withName:\s*'([^']*)'\s*\{\s*errorStrategy\s*=\s*\{[^}]*'ignore'[^}]*\}", re.S) + + +def strip_comments(text): + return "\n".join(line.split("//")[0] for line in text.splitlines()) + + +def report_only_patterns(config_text): + return [re.compile("^(?:%s)$" % m.group(1)) + for m in IGNORE_BLOCK.finditer(strip_comments(config_text))] + + +def report_only_processes(repo): + """Yield (workflow file, process, [emit, ...]) for each report-only process.""" + main = strip_comments(open(os.path.join(repo, "main.nf")).read()) + wdir = os.path.join(repo, "workflows") + for fname in sorted(os.listdir(wdir)): + if not fname.endswith(".nf"): + continue + text = strip_comments(open(os.path.join(wdir, fname)).read()) + wm = re.search(r"^workflow\s+(\w+)", text, re.M) + if not wm or "emit:" not in text: + continue + wf = wm.group(1) + emits = re.findall(r"^\s*(\w+)\s*=\s*(ch_\w+)\s*$", text.split("emit:")[-1], re.M) + for proc in sorted(set(re.findall(r"\b([A-Z][A-Z0-9_]+)\.out\.", text))): + used = [] + ok = True + for line in re.findall(r"^.*\b%s\.out\.\w+.*$" % proc, text, re.M): + if re.match(r"^\s*ch_versions\s*=\s*ch_versions\.mix\(\s*%s\.out\.versions\s*\)\s*$" + % proc, line): + continue + a = re.match(r"^\s*(ch_\w+)\s*=\s*%s\.out\.\w+\s*$" % proc, line) + if not a: + ok = False + break + ch = a.group(1) + others = [l for l in re.findall(r"^.*\b%s\b.*$" % ch, text, re.M) + if not re.match(r"^\s*(%s\s*=|\w+\s*=\s*%s\s*$)" % (ch, ch), l)] + names = [e for e, c in emits if c == ch] + if others or not names: + ok = False + break + for emit in names: + lines = re.findall(r"^.*\b%s\.out\.%s\b.*$" % (wf, emit), main, re.M) + if not lines or not all( + re.search(r"\.join\(\s*%s\.out\.%s\b.*remainder:\s*true" % (wf, emit), l) + for l in lines): + ok = False + used.append(emit) + if ok and used: + yield fname, proc, used + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--repo", default=os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__))))) + args = ap.parse_args() + try: + patterns = report_only_patterns(open(os.path.join(args.repo, "nextflow.config")).read()) + except OSError as e: + print("ERROR: %s" % e, file=sys.stderr) + return 2 + if not patterns: + print("ERROR: no report-only withName block in nextflow.config", file=sys.stderr) + return 1 + bad = 0 + found = set() + for fname, proc, emits in report_only_processes(args.repo): + found.add(proc) + covered = any(p.match(proc) for p in patterns) + if proc in KEEP_STOP: + print("keep-stop: %s (workflows/%s), by choice" % (proc, fname)) + elif covered: + print("ok: %s (workflows/%s)" % (proc, fname)) + else: + print("ERROR: %s (workflows/%s) feeds only the report (%s, joined with remainder: true " + "in main.nf) but no report-only withName block in nextflow.config names it, so " + "one failed task stops the whole run" % (proc, fname, ", ".join(emits)), + file=sys.stderr) + bad += 1 + for proc in sorted(KEEP_STOP - found): + print("ERROR: KEEP_STOP names %s, which the scan no longer finds as report-only; " + "drop it from the list" % proc, file=sys.stderr) + bad += 1 + if not found: + print("ERROR: the scan found no report-only process at all; the patterns no longer " + "match the code", file=sys.stderr) + bad += 1 + return 1 if bad else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_contract_report_only.py b/tests/test_contract_report_only.py new file mode 100644 index 00000000..7616733d --- /dev/null +++ b/tests/test_contract_report_only.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""scripts/ci/check-report-only.py: report-only processes carry the ignore strategy. + + 1. the check passes on this checkout and names the two cDNA read checks; + 2. on a copy whose report-only list drops CLINVAR_CDNA_CHECK (the state + before it was added) it fails and names that process; so it does when + CPSR_CDNA_CHECK is dropped, or MITO_DELETIONS's own block is removed; + 3. on a copy where a second consumer reads CLINVAR_CDNA_CHECK's channel, the + process feeds another step, so the check no longer asks for it; + 4. a KEEP_STOP name the scan no longer finds is refused: on a copy where + ROH's channel gets a second consumer, the check fails naming ROH. + +Run: python3 tests/test_contract_report_only.py +""" +import os +import re +import shutil +import subprocess +import sys +import tempfile + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +CHECK = os.path.join(REPO, "scripts", "ci", "check-report-only.py") +FAILS = 0 + + +def check(desc, ok, detail=""): + global FAILS + print("%s: %s" % ("ok" if ok else "FAIL", desc)) + if not ok: + FAILS += 1 + if detail: + print(detail) + + +def run(repo): + p = subprocess.run([sys.executable, "-I", CHECK, "--repo", repo], + capture_output=True, text=True) + return p.returncode, p.stdout + p.stderr + + +def copy(): + d = tempfile.mkdtemp(prefix="report-only-") + shutil.copy(os.path.join(REPO, "main.nf"), d) + shutil.copy(os.path.join(REPO, "nextflow.config"), d) + shutil.copytree(os.path.join(REPO, "workflows"), os.path.join(d, "workflows")) + return d + + +def edit(path, old, new, count=1): + text = open(path).read() + assert old in text, "planted edit not found in %s: %r" % (path, old) + open(path, "w").write(text.replace(old, new, count)) + + +rc, out = run(REPO) +check("checkout passes", rc == 0, out) +check("checkout names both cDNA checks", + "ok: CLINVAR_CDNA_CHECK" in out and "ok: CPSR_CDNA_CHECK" in out, out) + +for name, plant in [ + ("CLINVAR_CDNA_CHECK", lambda d: edit(os.path.join(d, "nextflow.config"), + "|CLINVAR_CDNA_CHECK|", "|")), + ("CPSR_CDNA_CHECK", lambda d: edit(os.path.join(d, "nextflow.config"), + "|CPSR_CDNA_CHECK|", "|")), + ("MITO_DELETIONS", lambda d: edit(os.path.join(d, "nextflow.config"), + "withName: 'MITO_DELETIONS'", "withName: 'NO_SUCH_PROCESS'")), +]: + d = copy() + plant(d) + rc, out = run(d) + check("dropping %s from the report-only list fails and names it" % name, + rc == 1 and re.search(r"ERROR: %s \(workflows/" % name, out) is not None, out) + shutil.rmtree(d) + +d = copy() +pgx = os.path.join(d, "workflows", "pgx.nf") +edit(os.path.join(d, "nextflow.config"), "|CLINVAR_CDNA_CHECK|", "|") +edit(pgx, " clinvar_cdna = ch_clinvar_cdna", + " clinvar_cdna_more = ch_clinvar_cdna.map { it }\n clinvar_cdna = ch_clinvar_cdna") +rc, out = run(d) +check("a second consumer makes CLINVAR_CDNA_CHECK a feeder, not asked for", + rc == 0 and "CLINVAR_CDNA_CHECK" not in out, out) +shutil.rmtree(d) + +d = copy() +clin = os.path.join(d, "workflows", "clinical.nf") +text = open(clin).read() +m = re.search(r"^\s*roh_regions\s*=\s*(ch_\w+)", text, re.M) +check("ROH has its emit in workflows/clinical.nf", m is not None) +if m: + edit(clin, m.group(0), " roh_more = %s.map { it }\n%s" % (m.group(1), m.group(0))) + rc, out = run(d) + check("a KEEP_STOP name the scan no longer finds fails", + rc == 1 and "KEEP_STOP names ROH" in out, out) +shutil.rmtree(d) + +print("FAILED: %d" % FAILS if FAILS else "all report-only checks passed") +sys.exit(1 if FAILS else 0)