From 5f29a65e0644e69f4e748d7139ff087ce149c438 Mon Sep 17 00:00:00 2001 From: linsalrob Date: Wed, 5 Aug 2026 16:52:11 +0930 Subject: [PATCH 1/2] creating a read the docs template --- .readthedocs.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 .readthedocs.yaml diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 0000000..575f578 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,23 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Set the OS, Python version, and other tools you might need +build: + os: ubuntu-24.04 + tools: + python: "3.13" + +# Build documentation in the "docs/" directory with Sphinx +sphinx: + configuration: docs/conf.py + +# Optionally, but recommended, +# declare the Python requirements required to build your documentation +# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html +# python: +# install: +# - requirements: docs/requirements.txt + From 0c1180002bb95c1dbc4d33567d9ad4a8e5304d7b Mon Sep 17 00:00:00 2001 From: linsalrob Date: Wed, 5 Aug 2026 17:53:24 +0930 Subject: [PATCH 2/2] Build comprehensive Read the Docs guide --- .readthedocs.yaml | 9 +-- docs/_static/.gitkeep | 1 + docs/cluster-support.md | 128 +++++++++++++++++++++++++++++++++++++++ docs/conf.py | 28 +++++++++ docs/configuration.md | 108 +++++++++++++++++++++++++++++++++ docs/contributing.md | 104 +++++++++++++++++++++++++++++++ docs/design.md | 60 ++++++++++++++++++ docs/index.md | 57 +++++++++++++++++ docs/installation.md | 101 +++++++++++++++++++++++++++++++ docs/methods.md | 79 ++++++++++++++---------- docs/outputs.md | 81 +++++++++++++++++++++++++ docs/overview.md | 56 +++++++++++++++++ docs/pipeline.md | 109 +++++++++++++++++++++++++++++++++ docs/profiles.md | 54 +++++++++++++++++ docs/quickstart.md | 131 ++++++++++++++++++++++++++++++++++++++++ docs/requirements.txt | 3 + docs/troubleshooting.md | 93 ++++++++++++++++++++++++++++ docs/utilities.md | 66 ++++++++++++++++++++ 18 files changed, 1229 insertions(+), 39 deletions(-) create mode 100644 docs/_static/.gitkeep create mode 100644 docs/cluster-support.md create mode 100644 docs/conf.py create mode 100644 docs/configuration.md create mode 100644 docs/contributing.md create mode 100644 docs/design.md create mode 100644 docs/index.md create mode 100644 docs/installation.md create mode 100644 docs/outputs.md create mode 100644 docs/overview.md create mode 100644 docs/pipeline.md create mode 100644 docs/profiles.md create mode 100644 docs/quickstart.md create mode 100644 docs/requirements.txt create mode 100644 docs/troubleshooting.md create mode 100644 docs/utilities.md diff --git a/.readthedocs.yaml b/.readthedocs.yaml index 575f578..3c1e133 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -14,10 +14,7 @@ build: sphinx: configuration: docs/conf.py -# Optionally, but recommended, -# declare the Python requirements required to build your documentation -# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html -# python: -# install: -# - requirements: docs/requirements.txt +python: + install: + - requirements: docs/requirements.txt diff --git a/docs/_static/.gitkeep b/docs/_static/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/docs/_static/.gitkeep @@ -0,0 +1 @@ + diff --git a/docs/cluster-support.md b/docs/cluster-support.md new file mode 100644 index 0000000..6b31582 --- /dev/null +++ b/docs/cluster-support.md @@ -0,0 +1,128 @@ +# Adding support for another cluster + +Atavide lite keeps separate scheduler profiles because resource policy, storage, software setup, and operational limits differ even between systems using the same scheduler. A successful port preserves the scientific stage contract while translating those system-specific details explicitly. + +## Begin with a cluster-support issue + +Search existing issues, then open the repository's **New cluster support** form. A complete request should contain: + +- cluster and organisation name; +- links to current official scheduler, storage, and software documentation; +- scheduler family and exact version when relevant; +- sequencing input model to support; +- all CPU, high-memory, GPU, long, debug, and transfer partitions/queues; +- maximum wall time, cores, usable memory, nodes, GPUs, arrays, running jobs, and submitted jobs; +- account/project, QoS, partition/queue, constraint, reservation, export, and GPU rules; +- shared scratch, node-local scratch, persistent storage, quotas, and purge periods; +- shell, module, Conda/Mamba, container, and GPU-runtime setup; +- submit, parsable job ID, array, dependency, status, accounting, cancellation, and task-launch syntax; and +- a small batch script known to run successfully. + +Ask HPC support for anything uncertain. Remove credentials, private account codes, sensitive paths, and research data. + +## Choose the closest source profile + +| Need | Starting points | +| --- | --- | +| Paired-end reads, shared scratch, Slurm | `pawsey_shortread` | +| Single/long reads, shared scratch, Slurm | `pawsey_minion` | +| Paired-end reads, node-local scratch, Slurm | `deepthought_shortread` | +| Single/long reads, node-local scratch, Slurm | `deepthought_minion` | +| Paired reads, PBS | `nci_pbs/fasta` | + +Copy the entire closest directory to a new, clearly named profile. Do not change an existing maintained profile merely to support a different system. + +```bash +cp -R pawsey_shortread examplecluster_shortread +``` + +## Porting checklist + +Review every file in the copy. At minimum translate: + +1. **Batch headers:** shebang, account/project, queue/partition, QoS, time, tasks, CPUs, memory, nodes, GPUs, arrays, output/error, exports, constraints, and exclusivity. +2. **Submission examples:** job-ID parsing, arrays, dependency conditions, throttling, and cancellation. +3. **Storage:** analysis paths, databases, shared and node-local scratch, copying, cleanup, persistent storage, and purge policy. +4. **Environment:** login shell, modules, Conda hooks/prefixes, containers, compilers, and GPU runtime. +5. **Resources:** make application thread counts agree with scheduler allocations and map memory-heavy, long, GPU, and transfer stages to valid queues. +6. **Control files:** retain the paired `R1_reads.txt` or single `reads.txt` contract and appropriate `DEFINITIONS.sh` examples. +7. **Documentation:** update every path, variable, command, queue, environment, and profile name in the copied README. + +Search for stale source-cluster terms after editing: + +```bash +rg -n 'PAWSEY|Pawsey|BGFS|PBS_JOBFS|/scratch|--partition|--qos|#PBS' \ + examplecluster_shortread +``` + +The presence of a match is not automatically wrong; each match must be understood. + +## Resource mapping by stage + +A cluster description is not complete until the main workload classes have destinations: + +| Workload | Questions to answer | +| --- | --- | +| fastp and light summaries | Ordinary CPU queue? Array/concurrency limit? | +| Host mapping and read mapping | CPU/thread count, memory, local I/O strategy? | +| MMseqs2 | High memory or standard memory? Temporary storage? Maximum duration? | +| MEGAHIT | Per-sample versus cross-assembly, memory, restart, long queue? | +| VAMB | CPU or GPU build? GPU type/count/runtime/account? | +| Database download and archival | Transfer nodes/queue? Outbound network? Persistent target? | + +Measure small representative jobs and revise defaults based on scheduler accounting. Requesting the maximum available resource for every stage wastes allocations and may delay scheduling. + +## Worked example: Pawsey Setonix + +Setonix demonstrates why a separate profile is needed. It uses Slurm, shares compute nodes by default, and recommends explicit node, task, CPU, and wall-time requests. Shared-node jobs request total node memory with `--mem`; complete-node jobs use `--exclusive`. + +General-purpose partitions relevant to atavide lite include: + +| Partition | Use | Maximum wall time | Resources per node | Selected limits | +| --- | --- | --- | --- | --- | +| `work` | CPU production | 24 h | 128 cores, 230 GiB usable memory | General production | +| `long` | Long CPU work | 96 h | 128 cores, 230 GiB | One node/job; 4 running and 96 submitted/user | +| `highmem` | Memory-heavy CPU work | 96 h | 128 cores, 980 GiB | One node/job; 2 running and 96 submitted/user | +| `gpu` | GPU production | 24 h | 64 cores, 230 GiB, 8 logical GPUs | GPU project account required | +| `gpu-highmem` | GPU with more host memory | 48 h | 64 cores, 460 GiB, 8 logical GPUs | GPU project account required | +| `copy` | Large transfers | 48 h | 32 cores, 115 GiB | 4 running and 500 submitted/user | +| `debug` | CPU development | 1 h | 128 cores, 230 GiB | 4 nodes/job; 1 running and 4 submitted/user | +| `gpu-dev` | GPU development | 4 h | 64 cores, 230 GiB, 8 logical GPUs | 2 nodes/job; 1 running and 4 submitted/user | + +These limits are a documentation snapshot, not pipeline constants. Confirm them using Pawsey's [Running Jobs on Setonix](https://pawsey.atlassian.net/wiki/spaces/US/pages/51929058/Running+Jobs+on+Setonix), [Job Scheduling](https://pawsey.atlassian.net/wiki/spaces/US/pages/51925964/Job+Scheduling), and current association data: + +```bash +sacctmgr show associations user="$USER" cluster=setonix +``` + +Ordinary production uses the default `normal` QoS. Pawsey documents `--qos=high` as a limited priority boost. GPU jobs charge an account formed by adding `-gpu` to the base project code. + +A representative shared-node header is: + +```bash +#!/bin/bash --login +#SBATCH --account= +#SBATCH --partition=work +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=58880M +#SBATCH --time=01:00:00 +``` + +Pawsey recommends integer memory values and currently favours `--mem` over `--mem-per-cpu`. Shared project scratch is under `/scratch/$PAWSEY_PROJECT/$USER` and is subject to a 21-day purge policy. Persistent data are staged through Acacia object storage, with large transfers assigned to `copy`. Environments and databases stored on scratch must be reproducibly recreatable. + +## Validate the new profile + +Validation should progress from harmless to representative: + +1. Check shell syntax for every batch script. +2. Check paths and stale identifiers with `rg`. +3. Submit a minimal job that prints hostname, storage, environment, and versions. +4. Submit one array task and one dependency chain. +5. Process a small non-sensitive sample through every stage. +6. Record scheduler time, peak memory, CPU efficiency, and GPU use. +7. Test restart behaviour after an intentionally stopped disposable job. +8. Ask a local HPC administrator or experienced user to review directives. + +If cluster access is unavailable, state that clearly in the pull request and open it as a draft. diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..a45798b --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,28 @@ +from pathlib import Path +import sys + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +project = "atavide lite" +author = "The atavide lite contributors" +copyright = "2026, The atavide lite contributors" + +extensions = ["myst_parser"] +source_suffix = {".rst": "restructuredtext", ".md": "markdown"} +master_doc = "index" +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] + +html_theme = "sphinx_rtd_theme" +html_static_path = ["_static"] +html_theme_options = { + "collapse_navigation": False, + "navigation_depth": 4, + "titles_only": False, +} + +myst_enable_extensions = ["colon_fence", "deflist", "fieldlist"] +myst_heading_anchors = 3 + +nitpicky = True diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..a9338eb --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,108 @@ +# Configuration and input preparation + +## Analysis directory layout + +Run the workflow from a dedicated analysis directory, not from the repository checkout. A typical paired-read analysis begins as: + +```text +analysis/ +├── DEFINITIONS.sh +├── R1_reads.txt +├── fastq/ +│ ├── sample_a_R1.fastq.gz +│ └── sample_a_R2.fastq.gz +└── slurm_output/ +``` + +Single/long-read analyses use `reads.txt`. Pipeline output directories are created alongside these control files. Keeping the repository and analysis separate prevents accidental commits of data and allows one code checkout to serve multiple analyses. + +## `DEFINITIONS.sh` + +Every scheduler job runs in the analysis directory and sources `DEFINITIONS.sh`. Copy an example from the selected profile and review every value. + +`SAMPLENAME` +: A short dataset identifier used in output names. Avoid whitespace and shell metacharacters. + +`FILEEND` +: The exact suffix removed to derive sample names. In a paired profile it normally includes the R1 marker, such as `_R1.fastq.gz`; in a single-read profile it may be `.fastq.gz`. + +`FAFILEEND` +: Used by profiles that derive names from converted FASTA files. It generally mirrors `FILEEND` with the FASTQ extension replaced by FASTA. Not every current profile defines it. + +`SOURCE` +: Directory containing the starting FASTQ files, commonly `fastq`. + +`HOSTFILE` +: FASTA or compressed FASTA containing the reference sequence(s) to separate from downstream reads. It may be omitted when host removal is disabled, depending on the chosen definitions example. + +`HOST` +: Output directory for reads mapped to the host reference. + +`HOSTREMOVED` +: Output directory for reads retained for downstream metagenomics analysis. When host removal is skipped, profiles commonly point this at the fastp output directory. + +Cluster profiles also rely on system variables such as `PAWSEY_PROJECT`, `USER`, `BGFS`, scheduler job/array IDs, or `PBS_JOBFS`. Confirm these are defined inside batch jobs rather than only in an interactive login shell. + +## Paired-read naming + +Paired scripts commonly derive R2 from R1 by replacing `_R1` with `_R2`. Each filename must therefore contain the expected marker exactly once. Rename `_1`/`_2` downloads if required: + +```bash +for file in fastq/*_1.fastq.gz; do + mv "$file" "${file/_1.fastq.gz/_R1.fastq.gz}" +done +for file in fastq/*_2.fastq.gz; do + mv "$file" "${file/_2.fastq.gz/_R2.fastq.gz}" +done +``` + +Preview rename operations on valuable data and follow local data-management practice before modifying originals. + +Create the sample list deterministically: + +```bash +find fastq -type f -name '*_R1*' -printf '%f\n' | sort > R1_reads.txt +``` + +Check counts and mates: + +```bash +wc -l R1_reads.txt +while IFS= read -r r1; do + r2=${r1/_R1/_R2} + test -f "fastq/$r2" || printf 'Missing mate: %s\n' "$r2" +done < R1_reads.txt +``` + +## Single/long-read naming + +Create one entry per file: + +```bash +find fastq -type f -name '*.fastq.gz' -printf '%f\n' | sort > reads.txt +``` + +Choose `FILEEND` so removing it yields a unique, meaningful sample name. Do not include reports, checksums, or intermediate FASTQ files in `reads.txt`. + +## Host-removal choices + +Host filtering is a scientific and governance decision. Select an appropriate reference assembly, record its accession and version, and understand whether removal is required before data leave controlled storage. A multi-FASTA can represent a non-human host, vector, symbiont, or other sequence collection. + +The pipeline separates matching and non-matching reads, but downstream stages currently use the non-host set. Review mapping statistics and a small sample of reads before discarding or restricting access to either output. + +## Database provenance + +Record the following with each analysis: + +- database name and release, such as UniRef100; +- download or build date and source URL; +- taxonomy dump date; +- Subsystems mapping/database version; +- host reference accessions and checksums; and +- commands used to create local MMseqs2 indices. + +Changing a database changes the analysis even when the scripts and input reads are identical. + +## Resource settings + +Resource headers are starting points. Before submission, review `--time`, `--cpus-per-task`, `--mem`, nodes, partition/queue, QoS, account/project, array limits, output paths, and GPU directives in every selected script. Ensure the application's own thread option matches the scheduler CPU allocation; requesting one CPU while starting many threads creates contention and misleading accounting. diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..93a264b --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,104 @@ +# Contributing + +Contributions are welcome from people of every background and experience level, including first-time open-source contributors. Bug reports, questions, documentation, tests, utilities, and cluster profiles all improve the project. + +Communicate respectfully and assume good intent. It is fine not to know the repository, Git, an HPC scheduler, or a bioinformatics tool yet. Ask for help when something is unclear, avoid sharing sensitive data or credentials, and follow institutional policy. + +## Start with an issue + +Search existing issues before starting. Use the bug or feature template for general work. For a new HPC system, use the **New cluster support** form and read [Adding support for another cluster](cluster-support.md). + +An issue gives maintainers a chance to confirm scope, point to related work, and identify scientific or operational constraints before implementation. + +Include evidence that lets another person understand the request: + +- repository commit/profile and relevant command; +- expected and observed behaviour; +- scheduler/cluster and redacted resource header; +- minimal logs or traceback; +- reproducible sample naming/layout without sequence data; and +- official documentation supporting cluster-specific values. + +Never post passwords, tokens, private project identifiers, controlled data, or sensitive paths. + +## Create a focused branch + +Fork the repository if necessary, clone it, and create a branch from the current default branch: + +```bash +git clone https://github.com//atavide_lite.git +cd atavide_lite +git switch -c docs/clear-topic-name +``` + +Use a cluster-specific branch such as `cluster/examplecluster_shortread` for a profile port. Keep unrelated changes on separate branches. + +## Make changes that remain inspectable + +- Preserve the visible stage boundaries and control-file contract unless the proposal intentionally changes them. +- Do not put usernames, account codes, or private filesystem paths into shared scripts. +- Keep application thread counts consistent with scheduler allocations. +- Update documentation when commands, outputs, dependencies, or profile behaviour change. +- Prefer a small, reviewable change over a large unrelated rewrite. +- Do not commit input data, generated outputs, databases, scheduler logs, or environments. + +## Using agentic AI + +Coding assistants can help inventory scripts and perform repetitive profile adaptation. Give the assistant the issue, official HPC documentation, and the closest existing profile. One useful request is: + +> Create `examplecluster_shortread` from `pawsey_shortread`. Use the cluster-support issue and official scheduler documentation to adapt every directive, resource request, storage path, environment step, and README command. Do not modify existing profiles. List anything that cannot be verified and run local checks without submitting a real job. + +Ask the assistant to inspect the complete source profile and show its diff. Then personally verify every queue, QoS, account, time, CPU, memory, GPU, path, module, and command. AI can invent plausible but invalid HPC details. Do not authorise it to submit jobs, access sensitive data, or expose credentials unless those actions are explicitly intended and understood. + +## Validate locally and on the cluster + +Before committing: + +1. Run `git diff --check` and review the entire diff. +2. Run the repository unit tests. +3. Parse or build documentation and check links. +4. Run `bash -n` on changed shell and scheduler scripts. +5. Search new profiles for stale cluster names, paths, accounts, partitions, and environment variables. +6. Check resource requests against current official policy. +7. Submit a harmless minimal job and then a small non-sensitive pipeline example when cluster access is available. + +Document what was tested and what remains unverified. + +## Commit and open a pull request + +Stage only intended paths: + +```bash +git status --short +git add docs/ examplecluster_shortread/ +git commit -m "Add ExampleCluster short-read profile" +git push -u origin cluster/examplecluster_shortread +``` + +Open a draft pull request through GitHub or the CLI: + +```bash +gh pr create --draft --fill +``` + +The pull request should link its issue and explain: + +- the problem and approach; +- the source profile, if any; +- scheduler, storage, environment, and scientific changes; +- documentation or evidence supporting decisions; +- checks and sample jobs completed; and +- remaining risks or unverified assumptions. + +Keep the PR in draft while implementation or target-cluster testing remains. Respond to review comments with follow-up commits so the review history remains useful. Maintainers decide when the change is ready to merge. + +## Documentation contributions + +Install the documentation dependencies, then build with warnings treated as errors: + +```bash +python -m pip install -r docs/requirements.txt +sphinx-build -W --keep-going -b html docs docs/_build/html +``` + +Read the rendered pages, not only the Markdown. Navigation, code blocks, tables, cross-references, and external links should remain clear on narrow screens and for readers unfamiliar with the project. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..d25a8ad --- /dev/null +++ b/docs/design.md @@ -0,0 +1,60 @@ +# Design and rationale + +## Explicit stages instead of one opaque run + +The original, more comprehensive atavide pipeline used Snakemake as its main workflow manager. That approach automated scheduling effectively, but operational experience with large and heterogeneous datasets showed a cost: failures at different stages were difficult to isolate, and adapting global resource settings to different clusters was cumbersome. + +`atavide_lite` therefore treats the HPC scheduler as the top-level orchestrator. Users submit visible scripts for each major stage and connect them with success dependencies such as Slurm's `afterok`. Snakemake is still used where its rule graph is helpful, including taxonomy summarisation, but it no longer hides the entire end-to-end execution. + +This is an intentional trade-off. Users issue more explicit commands, but gain direct control over restart points, resource requests, logs, and cluster-specific behaviour. + +## Failure containment and restartability + +Every pipeline stage produces durable outputs before downstream work starts. A failed MMseqs2 array task, for example, can be investigated and resubmitted without rerunning quality control or host removal. Scheduler output and error files preserve the job and array identifiers, making failures traceable to a stage and sample. + +Dependencies express ordering without requiring a long-lived process on a login node. Downstream jobs start only after the required upstream job succeeds. Where a downstream diagnostic should run even after failure, individual profiles may deliberately use `afterany`; users should inspect those choices before submission. + +## Separate profiles instead of universal scheduler abstraction + +The project maintains distinct directories because cluster differences extend beyond translating `#SBATCH` to `#PBS`. Profiles encode: + +- paired versus single-read filename handling; +- Slurm or PBS submission and dependency syntax; +- partition, queue, QoS, account, and GPU directives; +- job-array limits; +- shared, node-local, and persistent storage paths; +- scratch staging and cleanup; +- shell, modules, Conda/Mamba, and GPU runtime setup; and +- resource estimates for particular pipeline stages. + +A universal template would either expose all of these differences as configuration or conceal them behind generated code. Separate readable profiles make local policy visible and allow maintainers to optimise the systems they actually operate. + +## A small analysis contract + +The profiles share a lightweight contract rather than a large configuration framework: + +- `DEFINITIONS.sh` records dataset-specific paths and naming decisions. +- `R1_reads.txt` controls paired-end samples; `reads.txt` controls single/long-read samples. +- Scheduler array indices select entries from those files. +- Stable directory names connect outputs from one stage to inputs of the next. +- Job IDs returned by the scheduler create dependency chains. + +These plain-text control files become part of the analysis record and are easy to inspect, version, or regenerate. + +## Similar processing across sequencing technologies + +Paired short reads and long reads require different filename handling and may use different quality thresholds, but their downstream questions are similar. The project therefore keeps stage names and overall logic aligned where practical: quality control, optional host removal, conversion or ORF preparation, taxonomy, function, assembly, and binning. + +The separation is retained where it improves clarity. Paired reads are tracked as R1/R2 mates, while single or long reads use one sample file. Defaults should be reviewed for the technology and biological question; a convenient profile name is not a substitute for checking tool parameters. + +## Read-based and assembly-based branches + +Read classification and assembly answer different questions, so neither is treated as a replacement for the other. Read-level MMseqs2 results support abundance summaries. Assembly and VAMB support contig- and genome-level exploration. Because the branches diverge after cleaned reads are available, assembly work does not need to block read-based annotation unless a specific downstream dependency requires it. + +## Resource-aware data movement + +Large databases and temporary files make storage architecture part of the computation. Deepthought profiles can use fast node-local `$BGFS` space, while Pawsey profiles use shared `/scratch` and account for its purge policy. NCI jobs can use `$PBS_JOBFS`. Moving temporary I/O to suitable storage can materially improve runtime, but every profile must also copy required results back to durable, shared storage. + +## Conservative use of automation and AI + +Repetition makes profile adaptation suitable for coding assistants, but scheduler directives can be plausible and still wrong. The contribution process requires official cluster documentation, a tested minimal batch header, diff review, and human verification of every queue, account, resource, path, and module choice. Automation assists the port; it does not validate institutional policy. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..acd70b7 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,57 @@ +# atavide lite documentation + +`atavide_lite` is a modular metagenomics workflow for quality control, optional host-read removal, read-based taxonomic and functional profiling, assembly, and recovery of metagenome-assembled genomes (MAGs). It supports paired-end short reads and single-end or long reads on Slurm- and PBS-managed high-performance computing (HPC) systems. + +The project deliberately uses visible, independently submitted batch scripts instead of hiding the entire analysis inside one workflow invocation. That makes failures easier to locate, individual stages easier to repeat, and CPU, memory, wall-time, storage, and GPU requests easier to tune for each cluster. + +## Where to begin + +New users should follow this path: + +1. Read [Overview](overview.md) to decide whether the workflow fits the analysis. +2. Follow [Installation](installation.md) to clone the repository, build its utilities, and create the software environments. +3. Use [Choosing a profile](profiles.md) to select scripts for the read type and HPC system. +4. Complete [Quick start](quickstart.md) with a small, non-sensitive dataset. +5. Refer to [Configuration](configuration.md) and [Pipeline stages](pipeline.md) while scaling up. + +Cluster maintainers and contributors can start with [Contributing](contributing.md) and [Adding a cluster](cluster-support.md). + +```{toctree} +:maxdepth: 2 +:caption: Introduction + +overview +design +``` + +```{toctree} +:maxdepth: 2 +:caption: User guide + +installation +profiles +quickstart +configuration +pipeline +outputs +troubleshooting +``` + +```{toctree} +:maxdepth: 2 +:caption: Technical guide + +methods +utilities +cluster-support +contributing +``` + +## Project links + +- [Source repository](https://github.com/linsalrob/atavide_lite) +- [Issue tracker](https://github.com/linsalrob/atavide_lite/issues) +- [License](https://github.com/linsalrob/atavide_lite/blob/main/LICENSE) +- [Citation metadata](https://github.com/linsalrob/atavide_lite/blob/main/citation.cff) + +The documentation describes the repository as shipped. Cluster policies and third-party software change independently, so confirm scheduler limits and software versions with the relevant HPC provider before submitting expensive jobs. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..51b5ea7 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,101 @@ +# Installation + +## Prerequisites + +Use a Linux HPC environment with Bash, Git, a C compiler and `make`, and either Slurm or PBS for the supplied cluster profiles. Conda-compatible package management is recommended; the examples use Mamba because dependency resolution is usually faster. + +Before installing, confirm that the intended compute nodes can access: + +- the cloned repository; +- the working and database directories; +- the selected temporary filesystem; and +- the Conda environment or modules used by batch jobs. + +Do not perform computational analysis on a login node. Compilation and environment creation should also follow local HPC policy. + +## Clone the repository + +The profile scripts commonly expect the repository under `$HOME/GitHubs`, so that layout requires the fewest changes: + +```bash +mkdir -p "$HOME/GitHubs" +cd "$HOME/GitHubs" +git clone https://github.com/linsalrob/atavide_lite.git +cd atavide_lite +``` + +To record the exact code used in an analysis: + +```bash +git rev-parse HEAD +``` + +Store that commit ID with the analysis metadata. Pulling later changes during an active analysis can make stages inconsistent, so update deliberately. + +## Build the bundled utilities + +Compile the C utilities in `bin/`: + +```bash +cd "$HOME/GitHubs/atavide_lite/bin" +make all +``` + +This builds at least `fastq2fasta` and `fastg2gfa`. Confirm the build completed without compiler errors before submitting dependent jobs. + +## Create software environments + +The main environment contains tools used across the read-processing and annotation stages: + +```bash +mamba env create \ + --file "$HOME/GitHubs/atavide_lite/atavide_lite.yaml" +``` + +VAMB has a separate environment definition: + +```bash +mamba env create \ + --file "$HOME/GitHubs/atavide_lite/atavide_lite_vamb.yaml" +``` + +Important dependencies include fastp, minimap2, samtools, MMseqs2, MEGAHIT, VAMB, CheckM, TaxonKit/PyTaxonKit, Snakemake, GNU Parallel, pigz, rsync, and rclone. The environment files are the source of truth for packaged dependencies. + +```{note} +The Pawsey scripts intentionally place environments under `/scratch/$PAWSEY_PROJECT/$USER` and provide `pawsey_lib/check_atavide_lite_env.sh` because scratch content is temporary. Follow the Pawsey profile README instead of assuming a named environment in `$HOME`. +``` + +## Databases and references + +Pipeline stages may require: + +- a host reference FASTA for optional host removal; +- an MMseqs2 UniRef database, with current profiles favouring UniRef100 where provided; +- NCBI taxonomy dumps for TaxonKit; +- the atavide lite BV-BRC Subsystems SQLite mapping; and +- CheckM or PHROG data when those optional stages are used. + +Pawsey profiles include scheduler scripts for several database downloads. On other systems, place databases on storage visible to compute nodes and update the profile paths. Databases are large, change over time, and affect reproducibility; record their source, release/version, retrieval date, and local path. + +## Verify before using real data + +Run these checks from the repository: + +```bash +test -x bin/fastq2fasta +test -x bin/fastg2gfa +bash -n pawsey_shortread/fastp.slurm +``` + +Then submit a minimal scheduler job that prints the hostname and tool versions. Only after storage, environment activation, modules, accounting, and logging work should you run a small non-sensitive sample through the pipeline. + +## Updating + +Check the current branch and local changes before updating: + +```bash +git status --short --branch +git pull --ff-only +``` + +Rebuild `bin/` and review environment changes after an update. Do not overwrite locally adapted cluster scripts without first saving them on their own branch. diff --git a/docs/methods.md b/docs/methods.md index d8bc675..a3be556 100644 --- a/docs/methods.md +++ b/docs/methods.md @@ -1,43 +1,56 @@ -# Metagenomics Analysis With Atavide +# Scientific methods -To address the limitations of monolithic metagenomic workflows, we developed `atavide_lite`, -a modular, script-based pipeline that decouples computational processes to enhance robustness, -transparency, and adaptability across diverse high-performance computing (HPC) environments. +This page describes the scientific intent of the shipped workflow. Exact commands and defaults remain visible in each cluster profile and should be recorded for every analysis. -This design allows users to inspect, rerun, or replace individual steps without affecting the -overall pipeline logic or requiring global recomputation, features that are particularly -advantageous when dealing with heterogeneous sequencing technologies, large sample sets, and unstable -job scheduling systems. +## Quality control -The pipeline performs multiple steps that are standard for metagenomics processing. +Read-based analyses begin with adapter trimming and filtering using fastp. Profile defaults are tailored separately for paired short reads and single/long reads. Paired profiles commonly retain reads of at least 100 bp with no more than one uncalled base, while long-read examples may use different thresholds. Users should select thresholds appropriate to the library and preserve fastp reports. -_Quality control and host read removal_. Read-based analyses begin with adapter trimming and quality -filtering using fastp, with parameters optimized for Illumina or Oxford Nanopore reads. -Sequences are filtered to be longer than the minimum read length (100 bp) and we only -allow a single uncalled base. +## Host-read removal -_Functional and taxonomic profiling_. The cleaned reads are converted to FASTA format using -a compiled C utility. Taxonomic classification is performed using `mmseqs2 easy-taxonomy`, with -UniRef50 as the default database. A summarisation pipeline, implemented in `snakemake`, adds taxonomic -labels via `pytaxonkit`, allowing a more complete resolution of taxonomic lineages than native -`mmseqs2` outputs. +Quality-controlled reads are aligned to a user-selected host reference with minimap2. Samtools separates mapped and unmapped primary reads that pass platform/vendor checks and are not marked as PCR or optical duplicates. Paired profiles preserve R1/R2 identity through additional flag selection. Non-host reads continue through metagenomics analysis; host-mapped reads are stored separately but are not currently used downstream. -Taxonomy tables are generated at each rank for downstream comparative analyses. +The host reference can contain any genome or multi-FASTA sequence collection that the analysis should exclude. Its accession, version, and checksum are part of the method. -Functional annotation leverages an SQLite database derived from the BV-BRC Subsystems framework. -Subsystem assignments are joined with the taxonomic tables to yield both raw and normalised counts, -producing an integrated taxonomic-functional matrix. +## Read-based taxonomic profiling -_Assembly and metagenome-assembled genome (MAG) recovery_. _De novo_ assembly is carried -out for each sample independently using megahit, avoiding the memory constraints associated -with cross-assemblies. Contigs from all samples are concatenated and binned using `vamb`, which -employs a variational autoencoder for clustering based on tetranucleotide frequencies and -abundance profiles. Read mapping for abundance estimation is performed using `minimap2`. To -validate MAG quality, we apply `checkm`, assessing completeness and contamination for bins derived -from both unsplit and split clusters. +Cleaned reads are converted to FASTA using the repository's compiled converter and analysed with MMseqs2 `easy-taxonomy`. Current project workflows favour UniRef100 for improved hit coverage, although UniRef50 variants remain available for comparison and lower resource demand. -_Common user modifications._ For stratified analyses (e.g. ecological or host-based subsets), we -support grouped VAMB binning using a user-defined -TSV mapping of samples to groups, followed by independent execution of the concatenation, -mapping, and binning steps per group. +MMseqs2 uses k-mer prefiltering, ungapped filtering, vectorised alignment, and a lowest-common-ancestor procedure to assign taxonomy. Results depend on the MMseqs2 and UniRef releases and any changed parameters. +Taxonomic lineages are expanded using TaxonKit through PyTaxonKit and merged into per-rank tables. This additional step was chosen because it can recover more complete standard lineages than the native MMseqs2 representation alone. + +## Functional profiling + +UniRef identifiers from MMseqs2 results are joined to protein functions and the hierarchical BV-BRC Subsystems framework through a project SQLite mapping. Mapped reads are counted for each function/Subsystem. Outputs can include raw counts, counts normalised by mapped-read totals, and combined taxonomic-functional records. Where one protein maps to multiple functions, weighted counts may divide its contribution among assignments. + +## Assembly + +Cleaned non-host reads are assembled with MEGAHIT, which uses a succinct de Bruijn graph across multiple k-mer sizes and manages memory conservatively. Profiles commonly assemble samples independently to avoid the memory growth of complete cross-assembly, while specialised scripts support larger combined assemblies where appropriate resources are available. + +## MAG recovery + +Assemblies are concatenated with unique contig identifiers. Minimap2 maps reads from each sample back to the combined contigs to estimate differential abundance. VAMB combines abundance with tetranucleotide composition in a variational-autoencoder representation and clusters contigs into candidate MAGs. Depending on the VAMB version/profile, split and unsplit cluster sets may be produced. + +CheckM estimates completeness and contamination for candidate bins. These estimates support quality screening but should be interpreted with taxonomy, coverage, strain variation, and the intended downstream use. + +## Optional analyses + +Profiles may provide PHROG functional annotation for phage proteins, grouped VAMB runs for user-defined sample strata, read-fate/Sankey summaries, 16S screening, and selective removal of taxonomic lineages from downstream summaries. + +## Reproducible reporting + +A methods description should identify: + +- the atavide lite commit and profile; +- sequencing type and sample naming model; +- quality-control parameters and adapters; +- host reference and filtering decision; +- MMseqs2, UniRef, and taxonomy releases; +- Subsystems mapping release; +- MEGAHIT and VAMB parameters; +- split/unsplit/grouped binning choice; +- CheckM version/database; and +- any changes to supplied scheduler resources or commands. + +Use the repository's `citation.cff` to cite atavide lite and `references.bib` to identify the primary publications for integrated tools. diff --git a/docs/outputs.md b/docs/outputs.md new file mode 100644 index 0000000..adc41e8 --- /dev/null +++ b/docs/outputs.md @@ -0,0 +1,81 @@ +# Outputs and interpretation + +Exact filenames vary by profile and dataset configuration. This page describes the stable output concepts; the selected profile README and scripts define exact paths. + +## Scheduler logs + +`slurm_output/` and equivalent PBS locations contain standard output and error logs. Array filenames normally include both the parent job ID and array task ID. Preserve these logs at least until the analysis is validated because they record tool commands, versions, timings, warnings, and failure locations. + +A missing downstream output should first be investigated through scheduler state and the corresponding `.err` file, not by resubmitting the entire workflow. + +## Quality-controlled reads + +`fastq_fastp/` commonly contains cleaned FASTQ files, while `fastp_output/` contains JSON and HTML reports. Review: + +- reads before and after filtering; +- adapter trimming; +- quality by cycle; +- length distributions; +- duplication and GC patterns; and +- consistency between mates and samples. + +## Host and non-host reads + +Directories named by `HOST` and `HOSTREMOVED` contain the separated read sets. The non-host directory is the normal input to subsequent stages. Record mapping rates per sample and investigate outliers before annotation or assembly. + +Host-mapped reads may be sensitive. Apply the appropriate access, retention, and deletion policy even if the pipeline does not use them downstream. + +## MMseqs2 results + +The `mmseqs/` hierarchy generally contains one directory per sample with `easy-taxonomy` results, including LCA and report files. Treat these as intermediate evidence supporting summary tables. Preserve the command, database release, and relevant MMseqs2 version/parameters. + +## Taxonomy tables + +Taxonomy processing creates per-record lineage files and joined tables at standard ranks. Typical summary outputs include kingdom, phylum, class, order, family, genus, and species tables, often compressed TSV. + +Rows or columns depend on the joining script and workflow version. Before statistical analysis, confirm orientation, identifier uniqueness, unclassified handling, taxonomic naming conventions, and whether values are raw counts or another measure. + +## Functional and Subsystems tables + +Functional outputs join UniRef identifiers, protein descriptions, taxonomic information, and hierarchical Subsystems levels. Count summaries may include raw and normalised abundance. + +A protein can participate in multiple functional assignments. Some utilities distribute a read's contribution across mappings using weighted counts. Read the output header and utility documentation rather than assuming every count is an integer. + +## Assemblies + +MEGAHIT creates one output hierarchy per sample in the relevant assembly directory. Important products include final contigs and, where generated, graph formats. MEGAHIT working directories can be large; copy final products to durable storage before cleaning temporary data. + +Interpret assembly statistics together. A higher N50 alone does not establish a better metagenome assembly, and results depend strongly on depth, complexity, read quality, and host removal. + +## VAMB products + +VAMB-related directories include concatenated contigs, mapping products, abundance data, cluster assignments, and candidate bin FASTAs. Profiles may retain `clusters_split` and `clusters_unsplit`, group-specific directories, or legacy `bins/` paths. + +Contig names are the link between assemblies, mappings, clusters, and bins. Do not rename them after concatenation unless all dependent records are changed consistently. + +## CheckM results + +CheckM directories contain bin-level completeness and contamination estimates. Keep the exact bin set and CheckM database/version associated with every report. Comparing reports from split and unsplit VAMB outputs can guide which binning strategy to carry forward. + +## Sankey and read-fate outputs + +These consolidate how many reads remain or receive assignments after key stages. Use them to detect: + +- unexpected quality-control loss; +- unusually high host mapping; +- samples with few MMseqs2 assignments; +- mate-count inconsistencies; and +- stages that produced no counted output. + +## Reproducibility record + +For each completed analysis, archive or record: + +- repository commit ID and selected profile; +- `DEFINITIONS.sh`, read list, and grouping files; +- scheduler submission commands and relevant logs; +- software environment exports and module versions; +- scheduler resources actually used; +- host and annotation database identities/checksums; +- tool parameters changed from profile defaults; and +- final output checksums and storage locations. diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..c4232f6 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,56 @@ +# Overview + +## What problem does atavide lite solve? + +Metagenomics datasets routinely contain hundreds or thousands of samples. A typical analysis performs quality filtering, host screening, taxonomic and functional annotation, assembly, read mapping, binning, and MAG quality assessment. A failure in any sample or any stage can interrupt a monolithic workflow and make it difficult to see which command, input, dependency, or resource limit caused the problem. + +HPC portability is a second problem. Two clusters using the same scheduler can still differ in partition names, maximum wall times, array limits, memory accounting, local and shared scratch, filesystem purge policies, software modules, GPU directives, and project accounting. A resource request optimised for one system may be invalid or inefficient on another. + +`atavide_lite` responds by exposing the analysis as a sequence of small batch scripts. Each stage: + +- has a clear input and output; +- can be inspected before submission; +- writes scheduler logs that identify the failing sample or job; +- can be rerun without repeating successful upstream work; +- can use scheduler dependencies to preserve ordering; and +- can request resources appropriate to both the task and cluster. + +## Supported analyses + +The main profiles cover two input models: + +- **Paired-end short reads**, normally named with `_R1` and `_R2`, from platforms such as Illumina or MGI. +- **Single-end or long reads**, represented by one FASTQ file per sample, including Oxford Nanopore data. + +The workflow combines two complementary branches: + +- **Read-based analysis** retains per-read abundance information and produces taxonomic and BV-BRC Subsystems summaries. +- **Assembly-based analysis** creates contigs with MEGAHIT, maps reads back to contigs, groups contigs into MAGs with VAMB, and can assess bins with CheckM. + +Host removal is optional. The “host” can be any reference sequence or multi-FASTA collection that should be separated from downstream analysis; it is not restricted to human data. + +## What the project provides + +The repository contains: + +- cluster-specific Slurm profiles for Pawsey Setonix and Flinders Deepthought; +- PBS scripts for NCI Gadi; +- paired-read and single/long-read variants; +- adapter sequences; +- Conda/Mamba environment definitions; +- small compiled and Python utilities; +- Snakemake subworkflows for selected summarisation tasks; and +- documentation and profile-specific run books. + +The profile scripts are intended as readable operational recipes. They are not a promise that every default resource request is optimal for every dataset. Users should start small, inspect scheduler usage, and tune time, memory, CPU, and storage requests within their local policy. + +## Scope and responsibilities + +`atavide_lite` coordinates established bioinformatics tools; it does not replace understanding their outputs or limitations. Users remain responsible for: + +- ethical and authorised handling of sequence and host data; +- selecting appropriate references and databases; +- checking quality-control and host-removal results; +- validating taxonomic, functional, assembly, and MAG interpretations; +- complying with HPC allocation and storage policies; and +- citing the project and the tools used in their analysis. diff --git a/docs/pipeline.md b/docs/pipeline.md new file mode 100644 index 0000000..bb134f1 --- /dev/null +++ b/docs/pipeline.md @@ -0,0 +1,109 @@ +# Pipeline stages + +The pipeline contains a shared preparation phase followed by read-based and assembly-based branches. Script names vary slightly by profile; inspect the selected directory and its README before copying a command. + +## Dependency model + +For Slurm, `sbatch --parsable` returns a job ID that can be used in later submissions: + +```bash +FIRST_JOB=$(sbatch --parsable first.slurm) +SECOND_JOB=$(sbatch --parsable \ + --dependency="afterok:${FIRST_JOB}" second.slurm) +``` + +`afterok` starts the dependent job only if the prerequisite exits successfully. Array semantics and PBS dependency syntax differ. A dependency cannot verify scientific validity: inspect outputs before continuing when first using a stage or dataset. + +## 1. Quality control with fastp + +The fastp stage removes adapters and filters reads. Current profile defaults differ by input type. The paired short-read scripts generally allow at most one uncalled base and require a minimum length near 100 bp; single/ONT examples may allow two uncalled bases and use a shorter minimum in some profiles. + +These are project defaults, not universal biological rules. Short archived reads can all fail a 100 bp threshold. Review the exact `fastp` command, adapter FASTA, JSON/HTML report, retained-read counts, and length distribution before accepting the output. + +Typical outputs include cleaned FASTQ files under `fastq_fastp/` and reports under `fastp_output/`. + +## 2. Optional host-read separation + +`host_removal` uses minimap2 to align cleaned reads to `HOSTFILE` and samtools flags to separate mapped and unmapped primary reads that pass platform/vendor checks and are not marked as PCR/optical duplicates. Paired profiles additionally retain mate identity when writing R1 and R2 outputs. + +The reference can contain any sequence set that should be removed from downstream analysis. The mapped and unmapped reads are stored separately; downstream stages consume `HOSTREMOVED`. + +Validate: + +- reference identity, version, and indexing; +- mapping and retained-read percentages; +- mate pairing after filtering; +- whether unexpectedly high host content indicates a sample or reference problem; and +- data-governance requirements for retained host reads. + +To skip host alignment, configure downstream input to use the fastp output as described by the profile rather than inventing a dummy reference. + +## 3. FASTQ-to-FASTA conversion + +MMseqs2 `easy-taxonomy` consumes FASTA input. The repository's compiled `fastq2fasta` utility converts cleaned reads efficiently and can process a directory. Paired profiles preserve `/1` and `/2` identity in headers where required. + +Confirm that the binary was compiled, output files are non-empty, compression is as expected, and derived sample names match `FILEEND`/`FAFILEEND`. + +## 4. Read-based taxonomy with MMseqs2 + +MMseqs2 `easy-taxonomy` compares reads with a UniRef database and assigns taxonomy using a lowest-common-ancestor approach after its prefiltering and alignment stages. This is commonly the most computationally demanding read-based stage because both query volume and database size are large. + +Current Pawsey scripts provide UniRef50 and UniRef100 variants, while the current project methods favour UniRef100 for greater coverage. The database choice affects runtime, storage, hit rate, identifiers, and interpretation. Do not mix results from different UniRef releases without recording and accounting for the difference. + +MMseqs2 uses substantial temporary storage. Select a long or high-memory queue only when its limits and charging model fit the job, and place temporary files on storage designed for heavy I/O. + +## 5. Taxonomy completion and summaries + +The `summarise_taxonomy` component uses TaxonKit through PyTaxonKit to expand taxonomic lineages from MMseqs2 LCA outputs, then joins per-sample data into tables for ranks such as kingdom, phylum, class, order, family, genus, and species. + +This is a deliberate extra step. TaxonKit can provide more complete lineages with fewer gaps than relying only on the native MMseqs2 output. It requires a current NCBI taxonomy dump and a correctly set `TAXONKIT_DB`. + +The scheduler wrapper runs the Snakemake component and joining scripts. For standalone use, consult the repository's `summarise_taxonomy/README.md`. + +## 6. Functional annotation with BV-BRC Subsystems + +UniProt/UniRef identifiers from MMseqs2 are joined to an SQLite mapping derived from the BV-BRC Subsystems framework. Subsystems provide a hierarchical functional vocabulary. The workflow produces per-sample functional records and summary tables, including raw and normalised counts and taxonomic-functional combinations where supported. + +Normalisation makes samples with different mapped-read depths more comparable, but it does not remove every source of technical or compositional bias. Preserve raw counts, document the denominator, and select downstream statistical methods appropriate for compositional data. + +## 7. Optional PHROG annotation + +Some paired profiles include scripts to add PHROG phage-protein functions and count them. This requires the relevant annotation/mapping data and is not required for the core taxonomy or Subsystems results. + +## 8. Assembly with MEGAHIT + +MEGAHIT assembles cleaned, non-host reads with a succinct de Bruijn graph. Profiles may assemble each sample independently to avoid the extreme memory demand of complete cross-assembly; other scripts support larger combined assemblies where cluster resources permit. + +Assembly can run in parallel with read-based taxonomy once cleaned reads are ready. Review contig counts, total assembled bases, N50 alongside other statistics, minimum contig lengths, and warnings. An assembly completing successfully does not guarantee biological quality. + +## 9. Preparing contigs for VAMB + +VAMB jointly uses contig composition and differential abundance. Preparation generally includes: + +1. concatenating assemblies while retaining unique sample/contig identity; +2. building a minimap2 index for the combined contigs; and +3. mapping every sample's reads back to the contigs to create abundance inputs. + +The concatenation job must finish before mapping. All mapping array tasks must finish before VAMB runs. Check that sample names, BAM outputs, contig identifiers, and expected counts align. + +## 10. Binning with VAMB + +VAMB uses a variational autoencoder to combine k-mer composition and coverage information in a latent representation, then clusters contigs into candidate MAGs. Some profiles request GPUs and load a cluster-specific runtime such as ROCm; others may use different installations. + +GPU directives, account suffixes, module versions, and VAMB builds are especially cluster-specific. Confirm them with a minimal GPU job. VAMB may emit split and unsplit clusters; keep both until quality assessment determines which set is useful. + +## 11. MAG quality with CheckM + +CheckM estimates completeness and contamination for candidate bins. Profiles can run it separately for split and unsplit VAMB clusters. Its reference data and software environment must be installed and visible to compute nodes. + +Quality labels are screening criteria rather than proof that a bin represents a complete organism. Examine contamination, strain heterogeneity, taxonomy, coverage, and downstream biological consistency. + +## 12. Optional grouped VAMB + +Paired profiles can bin subsets independently using a two-column tab-separated mapping of group to sample prefix. The prefix must identify zero or one entry in `R1_reads.txt`; ambiguous substrings can silently select unintended samples. + +Grouped processing repeats concatenation, mapping, and VAMB for each defined group. Use it when coverage patterns are expected to be more informative within biological or technical strata, and retain the mapping file as analysis provenance. + +## 13. Read-fate and Sankey summaries + +Where supplied, read-fate and Sankey scripts combine counts from quality control, host removal, annotation, and assembly-oriented stages. These summaries are valuable pipeline sanity checks: large unexplained losses often reveal naming, input, threshold, or failed-stage problems. diff --git a/docs/profiles.md b/docs/profiles.md new file mode 100644 index 0000000..4a1aa5a --- /dev/null +++ b/docs/profiles.md @@ -0,0 +1,54 @@ +# Choosing a profile + +Select a profile using two independent questions: how are the reads represented, and which cluster's scheduler/storage model is closest? + +## Read type + +| Input | Control file | Closest profiles | +| --- | --- | --- | +| Paired-end short reads (`R1` and `R2`) | `R1_reads.txt` | `pawsey_shortread`, `deepthought_shortread`, `nci_pbs/fasta` | +| Single-end or long reads | `reads.txt` | `pawsey_minion`, `deepthought_minion` | + +Illumina single-end data uses the single-read control model even though the reads are short. Conversely, paired data downloaded with `_1.fastq.gz` and `_2.fastq.gz` may need renaming to `_R1.fastq.gz` and `_R2.fastq.gz` before using a paired profile. + +## Supplied cluster profiles + +### Pawsey Setonix + +`pawsey_shortread/` and `pawsey_minion/` use Slurm and shared project scratch under `/scratch/$PAWSEY_PROJECT/$USER`. They include environment checks because files on scratch are subject to purge. Large persistent data should be staged to and from Pawsey's Acacia object storage, and transfer work should follow Pawsey's current partition policy. + +Choose these profiles when using Setonix. They may also be the closest starting point for a different Slurm cluster with shared scratch, but they are not generic: project variables, account rules, partitions, GPU modules, paths, and purge assumptions must be reviewed. + +### Flinders Deepthought + +`deepthought_shortread/` and `deepthought_minion/` use Slurm. Some stages take advantage of fast node-local storage exposed through `$BGFS`. That storage is not a shared hand-off location, so scripts stage inputs in and copy required results back out. + +Choose these profiles on Deepthought or as the starting point for a cluster whose node-local temporary storage is central to performance. + +### NCI Gadi + +`nci_pbs/fasta/` contains PBS scripts for Gadi. It reflects PBS directives, project charging, queue selection, Gadi array constraints, shared `/g/data`, and compute-node-local `$PBS_JOBFS`. + +The NCI profile is less complete than the paired Slurm profiles and should be treated as an existing PBS implementation to inspect, not as a scheduler-neutral conversion layer. + +### Generic and component workflows + +`bash/atavide_lite.sh` demonstrates the main commands in one Bash script. It is useful for learning and adaptation, but the repository explicitly recommends running and checking stages one at a time. + +`summarise_taxonomy/` and `minion_taxonomy_and_function/` are component workflows. Use them when only those functions are needed or when integrating atavide lite outputs into a larger analysis. + +## Selection checklist + +Before running a profile, confirm all of the following: + +- filenames match the paired or single-read assumptions; +- scheduler directives are valid on the target cluster; +- account, project, partition/queue, and QoS values are permitted; +- requested time, CPUs, memory, nodes, GPUs, and array sizes are within limits; +- all hard-coded and environment-derived paths exist on compute nodes; +- temporary output is copied back before node-local storage disappears; +- the job shell initialises Conda and modules correctly; +- database versions and locations are correct; and +- profile README commands reference scripts that exist in that directory. + +If no supplied profile passes this checklist, follow [Adding support for another cluster](cluster-support.md) instead of modifying a maintained profile in place. diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..422f1c2 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,131 @@ +# Quick start + +This walkthrough uses paired-end data and Slurm-like commands to explain the shared workflow. Use the exact README and scheduler syntax in the selected profile. Start with one or two small, non-sensitive samples; do not begin with an entire production dataset. + +## 1. Install and choose a profile + +Complete [Installation](installation.md), then point `ATAVIDE_PROFILE` at the chosen directory. For Pawsey paired reads: + +```bash +export ATAVIDE_PROFILE="$HOME/GitHubs/atavide_lite/pawsey_shortread" +``` + +Avoid naming this variable `SOURCE`: `SOURCE` is a dataset setting inside `DEFINITIONS.sh`. + +## 2. Create an analysis directory + +Keep input data and analysis outputs outside the cloned repository: + +```bash +mkdir -p /path/to/analysis/fastq +cd /path/to/analysis +``` + +Place or link paired files into `fastq/` so every sample has one R1 and one R2 file, for example: + +```text +fastq/ +├── sample_a_R1.fastq.gz +├── sample_a_R2.fastq.gz +├── sample_b_R1.fastq.gz +└── sample_b_R2.fastq.gz +``` + +## 3. Configure the dataset + +Copy the closest definitions example: + +```bash +cp "$ATAVIDE_PROFILE/DEFINITIONS_human.sh" DEFINITIONS.sh +``` + +Edit it for the dataset. At minimum check `SAMPLENAME`, `FILEEND`, `SOURCE`, `HOSTFILE`, `HOST`, and `HOSTREMOVED`. For analysis without host removal, use a no-host example when supplied or set `HOSTREMOVED` to the quality-controlled read directory as documented by the profile. + +```{warning} +`DEFINITIONS.sh` is sourced as shell code by jobs. Do not insert untrusted commands, spaces around assignments, or secrets. Quote paths in new adaptations and avoid whitespace in sample identifiers. +``` + +## 4. Build the sample list + +For paired-end reads: + +```bash +find fastq -name '*_R1*' -printf '%f\n' | sort > R1_reads.txt +export NUM_R1_READS="$(wc -l < R1_reads.txt)" +``` + +Check the list and verify every R1 has a corresponding R2. For single/long reads, create `reads.txt` instead and export `NUM_READS`: + +```bash +find fastq -type f -name '*.fastq.gz' -printf '%f\n' | sort > reads.txt +export NUM_READS="$(wc -l < reads.txt)" +``` + +These counts are shell-session variables. Recreate them after logging in again or store a safe setup command with the analysis notes. + +## 5. Create log directories + +Profiles write scheduler logs to stage-specific paths. Create those used by the selected scripts before submission: + +```bash +mkdir -p \ + slurm_output/fastp_slurm \ + slurm_output/host_slurm \ + slurm_output/megahit_slurm \ + slurm_output/mmseqs_slurm \ + slurm_output/vamb_slurm +``` + +## 6. Prepare databases + +Confirm the host reference, UniRef/MMseqs2 database, taxonomy data, Subsystems mapping, and optional tool databases exist at the paths expected by the profile. On Pawsey, profile download scripts can be submitted separately. Wait for required databases before submitting dependent annotation jobs. + +## 7. Submit quality control + +For paired Slurm profiles: + +```bash +QC_JOB=$(sbatch --parsable \ + --array="1-${NUM_R1_READS}:1" \ + "$ATAVIDE_PROFILE/fastp.slurm") +echo "Quality-control job: $QC_JOB" +``` + +Inspect the first completed `.out` and `.err` files and fastp reports before continuing. + +## 8. Submit optional host removal + +```bash +HOST_JOB=$(sbatch --parsable \ + --array="1-${NUM_R1_READS}:1" \ + --dependency="afterok:${QC_JOB}" \ + "$ATAVIDE_PROFILE/host_removal.slurm") +``` + +If host removal is disabled, identify the cleaned-read job and directory that downstream scripts should use; do not blindly submit a nonexistent dependency. + +## 9. Start the two analysis branches + +Assembly can begin from cleaned reads without waiting for read-based taxonomy: + +```bash +ASSEMBLY_JOB=$(sbatch --parsable \ + --dependency="afterok:${HOST_JOB}" \ + "$ATAVIDE_PROFILE/megahit_hostremoved.slurm") +``` + +Prepare FASTA for MMseqs2: + +```bash +FASTA_JOB=$(sbatch --parsable \ + --dependency="afterok:${HOST_JOB}" \ + "$ATAVIDE_PROFILE/fastq2fasta.slurm") +``` + +Then submit the taxonomy script available in the selected profile. Pawsey paired profiles currently provide UniRef50 and UniRef100 variants; choose one deliberately and keep the resulting database identity with the analysis metadata. + +## 10. Monitor before scaling up + +Use the scheduler's status and accounting commands to check exit status, elapsed time, maximum memory, and CPU efficiency. Inspect outputs after every new stage on the small dataset. Only then increase array size or process the full dataset. + +Continue with [Pipeline stages](pipeline.md) for downstream taxonomy, function, assembly, VAMB, and CheckM dependencies. diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 0000000..8429acb --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,3 @@ +sphinx==8.2.3 +myst-parser==4.0.1 +sphinx-rtd-theme==3.0.2 diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..df632e3 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,93 @@ +# Troubleshooting + +## A practical diagnosis order + +When a job fails: + +1. Identify the job and array task in the scheduler. +2. Read the complete `.err` and `.out` files. +3. Find the last command printed or the line reported by the shell error trap. +4. Check whether expected inputs exist and are non-empty. +5. Confirm `DEFINITIONS.sh`, the sample list entry, and derived filename. +6. Check scheduler exit state, elapsed time, maximum memory, and node failure information. +7. Reproduce with one small sample or a harmless diagnostic job. +8. Fix the cause and resubmit only the failed stage and its blocked dependants. + +Do not immediately add memory or time. Incorrect paths, filenames, databases, or environment activation are more common and will fail again with a larger allocation. + +## Unbound variables + +Scripts use `set -u`, so a missing variable stops the job. Verify that: + +- `DEFINITIONS.sh` exists in the submission working directory; +- assignments have no spaces around `=`; +- required values such as `FILEEND` and `HOSTREMOVED` are present; +- cluster variables such as `PAWSEY_PROJECT`, `BGFS`, or `PBS_JOBFS` exist in batch jobs; and +- the job was submitted from the intended analysis directory. + +## No samples or wrong array indices + +Print the control file with line numbers: + +```bash +nl -ba R1_reads.txt +``` + +Slurm arrays in the supplied scripts are normally one-based because selection uses `head -n $SLURM_ARRAY_TASK_ID`. An array starting at zero can select no record. Remove blank lines and regenerate the list with `sort` for deterministic ordering. + +## Missing paired reads + +Paired scripts often replace `_R1` with `_R2`. Check capitalization, lane/run suffixes, and whether `FILEEND` removes the correct suffix. Confirm each R1-derived R2 path exists before submitting an array. + +## Conda activation fails in a batch job + +An environment working interactively does not prove the batch shell initialises it. Inspect the script's shebang and Conda hook. Confirm the environment path is visible from compute nodes and has not been purged. On Pawsey, run the supplied environment check/recreation workflow. + +Avoid editing global shell startup files as a first response; make environment setup explicit and reproducible inside the profile. + +## Tool not found or wrong version + +Print versions after activation in a small scheduler job: + +```bash +fastp --version +minimap2 --version +samtools --version +mmseqs version +``` + +Check for a site module overriding the Conda executable and verify GPU/runtime modules match the VAMB build. + +## Job exceeded wall time + +Determine whether the job made progress and whether its runtime scales with reads, database size, or assembly complexity. Use scheduler accounting from successful small jobs to estimate a new request. If the required time exceeds the current partition limit, choose an authorised long partition and adapt its node/memory constraints rather than requesting an invalid duration. + +## Out of memory + +Confirm the application actually received and used the requested CPUs and memory. MMseqs2, MEGAHIT, mapping, and cross-sample VAMB preparation can scale strongly with input size. Possible responses include reducing concurrency, splitting work, using per-sample assembly, selecting an authorised high-memory queue, moving temporary data to suitable storage, or increasing memory based on measured peak use. + +Do not assume `--mem-per-cpu` and `--mem` are interchangeable; follow cluster policy. + +## Files disappeared from scratch + +Scratch is temporary by design. Check the provider's purge policy and object/persistent storage. Recreate environments or databases from recorded definitions, then stage required inputs back. Build transfer and archival jobs into the workflow rather than relying on manual access to reset file age. + +## MMseqs2 database not found + +Check the expanded database path from inside a compute job, not only on the login node. Confirm all files that make up the MMseqs2 database are present and readable, and that the script's UniRef name matches the installed database. + +## TaxonKit errors or incomplete taxonomy + +Verify `TAXONKIT_DB` points to a directory containing `names.dmp`, `nodes.dmp`, `delnodes.dmp`, and `merged.dmp`. Record the dump date. Use a PyTaxonKit version compatible with current NCBI taxonomy formatting. + +## Empty output after fastp + +Read the fastp JSON/HTML report. The minimum-length or maximum-N threshold may exclude an older or unusual library. Change thresholds only after deciding they remain scientifically appropriate for downstream tools, and record the change. + +## Dependency will never run + +Inspect the prerequisite job's terminal state. A job submitted with `afterok` remains blocked or is cancelled when the prerequisite fails. After fixing and resubmitting the failed job, submit a new downstream dependency using the new job ID; changing files does not alter an already-recorded dependency. + +## Asking for help + +Open a bug report with the repository commit, profile, scheduler/cluster, exact submission command, relevant script header, redacted `DEFINITIONS.sh`, sample naming pattern, scheduler state/accounting, and the smallest relevant log excerpt. Remove credentials, private project codes, sensitive paths, and sequence data. diff --git a/docs/utilities.md b/docs/utilities.md new file mode 100644 index 0000000..e8e33f2 --- /dev/null +++ b/docs/utilities.md @@ -0,0 +1,66 @@ +# Components and utilities + +The cluster profiles coordinate tools in the repository root and several component directories. This page helps users locate the implementation behind a stage. + +## `bin/` + +`bin/` contains compiled C programs, Python utilities, Perl, and shell helpers. Run `make all` there before using compiled tools. + +Important utilities include: + +- `fastq2fasta`: compiled FASTQ-to-FASTA conversion used before MMseqs2; +- `fastg2gfa`: converts MEGAHIT graph output; +- `vamb_concatenate.py`: prepares uniquely named concatenated contigs for VAMB; +- `vamb_create_fasta.py` and related scripts: materialise cluster/bin FASTAs; +- `count_subsystems.py` and selective variants: build functional count summaries; +- `taxonomy_selected.py`: filter downstream taxonomic summaries by a selected lineage; +- `read_fate.py` and `sankey_plot.py`: summarise pipeline disposition of reads; +- MMseqs2 report conversion and taxonomy/function joining utilities; and +- `sqlite/` scripts for building the MMseqs2/Subsystems mapping database. + +Most Python utilities provide command-line help: + +```bash +python bin/count_subsystems.py --help +``` + +Run help before reusing a utility outside its profile, because input columns and output assumptions are specialised. + +## `summarise_taxonomy/` + +This Snakemake component enriches MMseqs2 LCA output with TaxonKit lineages, merges sample results, and builds rank-specific summaries. It requires NCBI taxonomy dump files and `TAXONKIT_DB`. + +The cluster wrapper usually runs this component. For standalone execution, the component README documents its expected `mmseqs//` layout and joining command. + +## `minion_taxonomy_and_function/` + +This component supports a protein/ORF-oriented long-read taxonomy and function workflow. It assumes basecalling, long-read quality control, and host removal have already occurred, then predicts ORFs, runs amino-acid MMseqs2 taxonomy, completes lineages, and adds BV-BRC Subsystems functions. + +It is more specialised than the main `pawsey_minion`/`deepthought_minion` script sequence. Read its README and Snakefiles before deciding which long-read route fits the analysis. + +## `adapters/` + +Adapter FASTA files are provided for paired short-read and Oxford Nanopore library preparation examples. Confirm the library kit and adapter set match the data. Kit names and chemistry change over time. + +## `pawsey_lib/` + +Pawsey helper scripts check or recreate temporary Conda environments expected by Pawsey profiles. They encode Setonix paths and should not be reused unchanged on another cluster. + +## `bash/` + +The single Bash workflow is an educational, linear representation of the major commands. It is helpful when understanding or porting the pipeline, but running stages independently preserves the project's failure isolation and scheduler-resource benefits. + +## Custom taxon exclusion + +After taxonomy and Subsystems processing, selected lineages can be removed to test whether a signal depends on them. For example: + +```bash +python bin/count_subsystems_selective.py \ + --directory mmseqs \ + --subsystems subsystems_no_pseudo \ + --name no_pseudo \ + --taxa f__Pseudomonadaceae \ + --verbose +``` + +Then create a filtered taxonomy hierarchy and summaries with `taxonomy_selected.py` and `join_taxonomies.py`. Check each script's current `--help`; option names may differ between versions. Preserve both original and filtered tables and document the selected lineage.