Build comprehensive Read the Docs documentation

.readthedocs.yaml

@@ -0,0 +1,19 @@
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version, and other tools you might need
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+python:
+  install:
+    - requirements: docs/requirements.txt

docs/_static/.gitkeep

@@ -0,0 +1 @@
+

docs/cluster-support.md

@@ -1,38 +1,103 @@
 # Adding support for another cluster
 
-Atavide lite keeps separate scheduler scripts for each supported cluster because scheduler policy, storage, software environments, and resource limits differ in ways that are difficult to hide behind one generic script. To request a port, open a **New cluster support** issue and provide enough information to adapt one of these starting points:
+Atavide lite keeps separate scheduler profiles because resource policy, storage, software setup, and operational limits differ even between systems using the same scheduler. A successful port preserves the scientific stage contract while translating those system-specific details explicitly.
 
-- `deepthought_shortread` or `pawsey_shortread` for paired-end reads.
-- `deepthought_minion` or `pawsey_minion` for single-end or long reads.
+## Begin with a cluster-support issue
 
-The request needs more than the scheduler name. It must identify suitable partitions or queues for ordinary CPU work, memory-heavy work, long assemblies, GPU VAMB jobs, and data transfers. It must also document resource limits, account and QoS rules, array/dependency syntax, shared and node-local temporary storage, environment setup, and a tested batch header. The issue form prompts for each of these details.
+Search existing issues, then open the repository's **New cluster support** form. A complete request should contain:
+
+- cluster and organisation name;
+- links to current official scheduler, storage, and software documentation;
+- scheduler family and exact version when relevant;
+- sequencing input model to support;
+- all CPU, high-memory, GPU, long, debug, and transfer partitions/queues;
+- maximum wall time, cores, usable memory, nodes, GPUs, arrays, running jobs, and submitted jobs;
+- account/project, QoS, partition/queue, constraint, reservation, export, and GPU rules;
+- shared scratch, node-local scratch, persistent storage, quotas, and purge periods;
+- shell, module, Conda/Mamba, container, and GPU-runtime setup;
+- submit, parsable job ID, array, dependency, status, accounting, cancellation, and task-launch syntax; and
+- a small batch script known to run successfully.
+
+Ask HPC support for anything uncertain. Remove credentials, private account codes, sensitive paths, and research data.
+
+## Choose the closest source profile
+
+| Need | Starting points |
+| --- | --- |
+| Paired-end reads, shared scratch, Slurm | `pawsey_shortread` |
+| Single/long reads, shared scratch, Slurm | `pawsey_minion` |
+| Paired-end reads, node-local scratch, Slurm | `deepthought_shortread` |
+| Single/long reads, node-local scratch, Slurm | `deepthought_minion` |
+| Paired reads, PBS | `nci_pbs/fasta` |
+
+Copy the entire closest directory to a new, clearly named profile. Do not change an existing maintained profile merely to support a different system.
+
+```bash
+cp -R pawsey_shortread examplecluster_shortread
+```
+
+## Porting checklist
+
+Review every file in the copy. At minimum translate:
+
+1. **Batch headers:** shebang, account/project, queue/partition, QoS, time, tasks, CPUs, memory, nodes, GPUs, arrays, output/error, exports, constraints, and exclusivity.
+2. **Submission examples:** job-ID parsing, arrays, dependency conditions, throttling, and cancellation.
+3. **Storage:** analysis paths, databases, shared and node-local scratch, copying, cleanup, persistent storage, and purge policy.
+4. **Environment:** login shell, modules, Conda hooks/prefixes, containers, compilers, and GPU runtime.
+5. **Resources:** make application thread counts agree with scheduler allocations and map memory-heavy, long, GPU, and transfer stages to valid queues.
+6. **Control files:** retain the paired `R1_reads.txt` or single `reads.txt` contract and appropriate `DEFINITIONS.sh` examples.
+7. **Documentation:** update every path, variable, command, queue, environment, and profile name in the copied README.
+
+Search for stale source-cluster terms after editing:
+
+```bash
+rg -n 'PAWSEY|Pawsey|BGFS|PBS_JOBFS|/scratch|--partition|--qos|#PBS' \
+  examplecluster_shortread
+```
+
+The presence of a match is not automatically wrong; each match must be understood.
+
+## Resource mapping by stage
+
+A cluster description is not complete until the main workload classes have destinations:
+
+| Workload | Questions to answer |
+| --- | --- |
+| fastp and light summaries | Ordinary CPU queue? Array/concurrency limit? |
+| Host mapping and read mapping | CPU/thread count, memory, local I/O strategy? |
+| MMseqs2 | High memory or standard memory? Temporary storage? Maximum duration? |
+| MEGAHIT | Per-sample versus cross-assembly, memory, restart, long queue? |
+| VAMB | CPU or GPU build? GPU type/count/runtime/account? |
+| Database download and archival | Transfer nodes/queue? Outbound network? Persistent target? |
+
+Measure small representative jobs and revise defaults based on scheduler accounting. Requesting the maximum available resource for every stage wastes allocations and may delay scheduling.
 
 ## Worked example: Pawsey Setonix
 
-Pawsey's Setonix illustrates why a dedicated profile is useful. Setonix uses Slurm and shares compute nodes by default. Pawsey recommends explicitly requesting nodes, tasks, CPUs per task, and wall time. Shared-node jobs should request total memory per node with `--mem`; jobs requiring a complete node should use `--exclusive`.
+Setonix demonstrates why a separate profile is needed. It uses Slurm, shares compute nodes by default, and recommends explicit node, task, CPU, and wall-time requests. Shared-node jobs request total node memory with `--mem`; complete-node jobs use `--exclusive`.
 
-The general-purpose partitions relevant to an atavide lite port are:
+General-purpose partitions relevant to atavide lite include:
 
-| Partition | Intended use | Maximum wall time | Resources per node | Important limits |
+| Partition | Use | Maximum wall time | Resources per node | Selected limits |
 | --- | --- | --- | --- | --- |
-| `work` | CPU production | 24 hours | 128 CPU cores, 230 GiB usable memory | General production queue |
-| `long` | Long CPU production | 96 hours | 128 CPU cores, 230 GiB usable memory | One node per job; four concurrent jobs and 96 submitted jobs per user |
-| `highmem` | Memory-heavy CPU production | 96 hours | 128 CPU cores, 980 GiB usable memory | One node per job; two concurrent jobs and 96 submitted jobs per user |
-| `gpu` | GPU production | 24 hours | 64 CPU cores, 230 GiB usable memory, eight logical GPUs | Uses the project account with a `-gpu` suffix |
-| `gpu-highmem` | GPU work needing more host memory | 48 hours | 64 CPU cores, 460 GiB usable memory, eight logical GPUs | Uses the project account with a `-gpu` suffix |
-| `copy` | Large data transfers | 48 hours | 32 CPU cores, 115 GiB usable memory | Four concurrent jobs and 500 submitted jobs per user |
-| `debug` | CPU development and debugging | 1 hour | 128 CPU cores, 230 GiB usable memory | Four nodes per job; one concurrent and four submitted jobs per user |
-| `gpu-dev` | GPU development and debugging | 4 hours | 64 CPU cores, 230 GiB usable memory, eight logical GPUs | Two nodes per job; one concurrent and four submitted jobs per user |
+| `work` | CPU production | 24 h | 128 cores, 230 GiB usable memory | General production |
+| `long` | Long CPU work | 96 h | 128 cores, 230 GiB | One node/job; 4 running and 96 submitted/user |
+| `highmem` | Memory-heavy CPU work | 96 h | 128 cores, 980 GiB | One node/job; 2 running and 96 submitted/user |
+| `gpu` | GPU production | 24 h | 64 cores, 230 GiB, 8 logical GPUs | GPU project account required |
+| `gpu-highmem` | GPU with more host memory | 48 h | 64 cores, 460 GiB, 8 logical GPUs | GPU project account required |
+| `copy` | Large transfers | 48 h | 32 cores, 115 GiB | 4 running and 500 submitted/user |
+| `debug` | CPU development | 1 h | 128 cores, 230 GiB | 4 nodes/job; 1 running and 4 submitted/user |
+| `gpu-dev` | GPU development | 4 h | 64 cores, 230 GiB, 8 logical GPUs | 2 nodes/job; 1 running and 4 submitted/user |
 
-These values are an example snapshot, not hard-coded pipeline assumptions. Check Pawsey's [Running Jobs on Setonix](https://pawsey.atlassian.net/wiki/spaces/US/pages/51929058/Running+Jobs+on+Setonix) and [Job Scheduling](https://pawsey.atlassian.net/wiki/spaces/US/pages/51925964/Job+Scheduling) documentation before changing scripts, because cluster policy can change. Current limits for a user's associations can also be inspected on Setonix with:
+These limits are a documentation snapshot, not pipeline constants. Confirm them using Pawsey's [Running Jobs on Setonix](https://pawsey.atlassian.net/wiki/spaces/US/pages/51929058/Running+Jobs+on+Setonix), [Job Scheduling](https://pawsey.atlassian.net/wiki/spaces/US/pages/51925964/Job+Scheduling), and current association data:
 
 ```bash
-sacctmgr show associations user=$USER cluster=setonix
+sacctmgr show associations user="$USER" cluster=setonix
 ```
 
-Setonix assigns the `normal` QoS to ordinary production jobs by default. A project may use `--qos=high` for a priority boost on up to 10% of its allocation; other QoS levels are controlled by allocation state or reserved for special cases. GPU submissions use an account formed by adding `-gpu` to the base project code.
+Ordinary production uses the default `normal` QoS. Pawsey documents `--qos=high` as a limited priority boost. GPU jobs charge an account formed by adding `-gpu` to the base project code.
 
-A representative shared-node CPU header is:
+A representative shared-node header is:
 
 ```bash
 #!/bin/bash --login
@@ -45,8 +110,19 @@ A representative shared-node CPU header is:
 #SBATCH --time=01:00:00
 ```
 
-Pawsey currently recommends `--mem` rather than `--mem-per-cpu`, integer memory values, and an explicit login shell so system modules initialise correctly. Resource requests in the actual atavide lite scripts vary by pipeline stage and must remain within the chosen partition's limits.
+Pawsey recommends integer memory values, an explicit login shell so system modules initialise correctly, and currently favours `--mem` over `--mem-per-cpu`. Resource requests in the actual atavide lite scripts vary by pipeline stage and must remain within the selected partition's limits. Shared project scratch is under `/scratch/$PAWSEY_PROJECT/$USER` and is subject to a 21-day purge policy. Persistent data are staged through Acacia object storage, with large transfers assigned to `copy`. Environments and databases stored on scratch must be reproducibly recreatable; the Pawsey profiles include checks and recreation steps for purged environments.
+
+## Validate the new profile
+
+Validation should progress from harmless to representative:
 
-Storage is another part of the port. Setonix profiles use shared scratch under `/scratch/$PAWSEY_PROJECT/$USER`; scratch files are subject to a 21-day purge policy. Project data that must persist is staged to or from Pawsey's Acacia object storage, with large transfers assigned to the `copy` partition. Conda environments and databases used by the Pawsey profiles are also placed under the project scratch hierarchy, so the scripts include checks and recreation steps for purged environments.
+1. Check shell syntax for every batch script.
+2. Check paths and stale identifiers with `rg`.
+3. Submit a minimal job that prints hostname, storage, environment, and versions.
+4. Submit one array task and one dependency chain.
+5. Process a small non-sensitive sample through every stage.
+6. Record scheduler time, peak memory, CPU efficiency, and GPU use.
+7. Test restart behaviour after an intentionally stopped disposable job.
+8. Ask a local HPC administrator or experienced user to review directives.
 
-This example supplies the information needed to decide how to translate scheduler headers and paths. A complete new-cluster request should additionally supply a tested batch script, exact job-array and dependency syntax, software/module versions, GPU runtime requirements, filesystem quotas, and cluster-specific operational restrictions.
+If cluster access is unavailable, state that clearly in the pull request and open it as a draft.

docs/conf.py

@@ -0,0 +1,28 @@
+from pathlib import Path
+import sys
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+project = "atavide lite"
+author = "The atavide lite contributors"
+copyright = "2026, The atavide lite contributors"
+
+extensions = ["myst_parser"]
+source_suffix = {".rst": "restructuredtext", ".md": "markdown"}
+root_doc = "index"
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]
+html_theme_options = {
+    "collapse_navigation": False,
+    "navigation_depth": 4,
+    "titles_only": False,
+}
+
+myst_enable_extensions = ["colon_fence", "deflist", "fieldlist"]
+myst_heading_anchors = 3
+
+nitpicky = True

docs/configuration.md

@@ -0,0 +1,108 @@
+# Configuration and input preparation
+
+## Analysis directory layout
+
+Run the workflow from a dedicated analysis directory, not from the repository checkout. A typical paired-read analysis begins as:
+
+```text
+analysis/
+├── DEFINITIONS.sh
+├── R1_reads.txt
+├── fastq/
+│   ├── sample_a_R1.fastq.gz
+│   └── sample_a_R2.fastq.gz
+└── slurm_output/
+```
+
+Single/long-read analyses use `reads.txt`. Pipeline output directories are created alongside these control files. Keeping the repository and analysis separate prevents accidental commits of data and allows one code checkout to serve multiple analyses.
+
+## `DEFINITIONS.sh`
+
+Every scheduler job runs in the analysis directory and sources `DEFINITIONS.sh`. Copy an example from the selected profile and review every value.
+
+`SAMPLENAME`
+: A short dataset identifier used in output names. Avoid whitespace and shell metacharacters.
+
+`FILEEND`
+: The exact suffix removed to derive sample names. In a paired profile it normally includes the R1 marker, such as `_R1.fastq.gz`; in a single-read profile it may be `.fastq.gz`.
+
+`FAFILEEND`
+: Used by profiles that derive names from converted FASTA files. It generally mirrors `FILEEND` with the FASTQ extension replaced by FASTA. Not every current profile defines it.
+
+`SOURCE`
+: Directory containing the starting FASTQ files, commonly `fastq`.
+
+`HOSTFILE`
+: FASTA or compressed FASTA containing the reference sequence(s) to separate from downstream reads. It may be omitted when host removal is disabled, depending on the chosen definitions example.
+
+`HOST`
+: Output directory for reads mapped to the host reference.
+
+`HOSTREMOVED`
+: Output directory for reads retained for downstream metagenomics analysis. When host removal is skipped, profiles commonly point this at the fastp output directory.
+
+Cluster profiles also rely on system variables such as `PAWSEY_PROJECT`, `USER`, `BGFS`, scheduler job/array IDs, or `PBS_JOBFS`. Confirm these are defined inside batch jobs rather than only in an interactive login shell.
+
+## Paired-read naming
+
+Paired scripts commonly derive R2 from R1 by replacing `_R1` with `_R2`. Each filename must therefore contain the expected marker exactly once. Rename `_1`/`_2` downloads if required:
+
+```bash
+for file in fastq/*_1.fastq.gz; do
+  mv "$file" "${file/_1.fastq.gz/_R1.fastq.gz}"
+done
+for file in fastq/*_2.fastq.gz; do
+  mv "$file" "${file/_2.fastq.gz/_R2.fastq.gz}"
+done
+```
+
+Preview rename operations on valuable data and follow local data-management practice before modifying originals.
+
+Create the sample list deterministically:
+
+```bash
+find fastq -type f -name '*_R1*' -printf '%f\n' | sort > R1_reads.txt
+```
+
+Check counts and mates:
+
+```bash
+wc -l R1_reads.txt
+while IFS= read -r r1; do
+  r2=${r1/_R1/_R2}
+  test -f "fastq/$r2" || printf 'Missing mate: %s\n' "$r2"
+done < R1_reads.txt
+```
+
+## Single/long-read naming
+
+Create one entry per file:
+
+```bash
+find fastq -type f -name '*.fastq.gz' -printf '%f\n' | sort > reads.txt
+```
+
+Choose `FILEEND` so removing it yields a unique, meaningful sample name. Do not include reports, checksums, or intermediate FASTQ files in `reads.txt`.
+
+## Host-removal choices
+
+Host filtering is a scientific and governance decision. Select an appropriate reference assembly, record its accession and version, and understand whether removal is required before data leave controlled storage. A multi-FASTA can represent a non-human host, vector, symbiont, or other sequence collection.
+
+The pipeline separates matching and non-matching reads, but downstream stages currently use the non-host set. Review mapping statistics and a small sample of reads before discarding or restricting access to either output.
+
+## Database provenance
+
+Record the following with each analysis:
+
+- database name and release, such as UniRef100;
+- download or build date and source URL;
+- taxonomy dump date;
+- Subsystems mapping/database version;
+- host reference accessions and checksums; and
+- commands used to create local MMseqs2 indices.
+
+Changing a database changes the analysis even when the scripts and input reads are identical.
+
+## Resource settings
+
+Resource headers are starting points. Before submission, review `--time`, `--cpus-per-task`, `--mem`, nodes, partition/queue, QoS, account/project, array limits, output paths, and GPU directives in every selected script. Ensure the application's own thread option matches the scheduler CPU allocation; requesting one CPU while starting many threads creates contention and misleading accounting.