Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
name: Bug report
about: Report a reproducible problem with atavide lite
title: "[Bug]: "
labels: bug
assignees: ""
---

## What happened?

<!-- Describe the problem and what you expected to happen instead. -->

## How can we reproduce it?

<!-- Include the commands you ran, the pipeline stage, and a minimal example if possible. -->

1.
2.
3.

## Error output

<!-- Paste the relevant scheduler .out/.err output or traceback. Remove credentials and sensitive data. -->

```text

```

## Environment

- atavide lite commit or version:
- Pipeline profile (for example, `pawsey_shortread`):
- Cluster and scheduler:
- Operating system:
- Conda environment or relevant tool versions:
- Input type (paired short reads or single/long reads):

## Additional context

<!-- Include anything else that may help, such as resource requests, job ID, or a link to cluster documentation. -->
161 changes: 161 additions & 0 deletions .github/ISSUE_TEMPLATE/cluster_support.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
name: New cluster support
description: Request an atavide lite profile for another HPC cluster
title: "[Cluster]: Support "
labels:
- enhancement
- cluster-support
body:
- type: markdown
attributes:
value: |
Thanks for helping us port atavide lite. Please ask your HPC support team for any details you do not know. A complete response should let us adapt one of `deepthought_minion`, `deepthought_shortread`, `pawsey_minion`, or `pawsey_shortread` without guessing cluster policy.

- type: input
id: cluster
attributes:
label: Cluster and organisation
description: Give the cluster name and the organisation that operates it.
placeholder: Setonix — Pawsey Supercomputing Research Centre
validations:
required: true

- type: input
id: docs
attributes:
label: Cluster documentation
description: Link to the scheduler, partition/queue, storage, and software-environment documentation.
placeholder: https://example.edu/hpc/running-jobs
validations:
required: true

- type: dropdown
id: scheduler
attributes:
label: Scheduler
description: Which workload manager accepts batch jobs?
options:
- Slurm
- PBS / PBS Pro / OpenPBS
- Other (describe below)
validations:
required: true

- type: dropdown
id: data_type
attributes:
label: Sequencing input to support
description: This selects the closest existing profile.
options:
- Paired-end short reads (adapt `deepthought_shortread` or `pawsey_shortread`)
- Single-end or long reads (adapt `deepthought_minion` or `pawsey_minion`)
- Both
validations:
required: true

- type: dropdown
id: closest_profile
attributes:
label: Closest existing profile
description: Select the implementation whose storage and input model is most similar.
options:
- pawsey_shortread
- pawsey_minion
- deepthought_shortread
- deepthought_minion
- Unsure
validations:
required: true

- type: textarea
id: partitions
attributes:
label: Partitions or queues
description: List every usable CPU, high-memory, GPU, long-running, debug, and data-transfer partition/queue. For each, give its purpose, maximum wall time, cores and usable memory per node, maximum nodes, and GPU type/count where relevant.
placeholder: |
| Name | Purpose | Max wall time | Cores/node | Memory/node | Max nodes | GPUs/node |
| --- | --- | --- | --- | --- | --- | --- |
| work | CPU production | 24 h | 128 | 230 GiB | ... | — |
validations:
required: true

- type: textarea
id: resource_mapping
attributes:
label: Pipeline resource mapping
description: Explain which partition/queue should run ordinary steps, MMseqs2 or large assemblies, GPU VAMB, downloads/transfers, and jobs that exceed the normal time limit. Include realistic time, CPU, memory, node, and GPU requests for each class.
placeholder: |
- Ordinary per-sample jobs: ...
- MMseqs2 / memory-heavy jobs: ...
- Assembly / long-running jobs: ...
- VAMB / GPU jobs: ...
- Downloads and data transfers: ...
validations:
required: true

- type: textarea
id: directives
attributes:
label: Required submission directives and policies
description: Provide exact `#SBATCH` or `#PBS` syntax for account/project, partition/queue, QoS, wall time, memory, CPUs/tasks/nodes, GPUs, job arrays, output/error paths, and any required export, reservation, constraint, or exclusive-node attributes. State which are mandatory and include a minimal working batch header.
render: shell
validations:
required: true

- type: textarea
id: qos_limits
attributes:
label: QoS, accounting, and queue limits
description: List allowed/default QoS values, project or account naming rules (including separate GPU accounts), charging rules, maximum running/submitted/array jobs, and any per-user or per-project restrictions. Write “none” only if confirmed by the cluster documentation.
validations:
required: true

- type: textarea
id: storage
attributes:
label: Storage and temporary files
description: Give paths or environment variables for home, software, shared scratch, node-local scratch, and long-term storage. Include quotas, purge periods, file-count limits, staging commands, and which paths are visible from compute nodes.
placeholder: Shared scratch is /scratch/$PROJECT/$USER, visible on compute nodes, and files are purged after ...
validations:
required: true

- type: textarea
id: software
attributes:
label: Software environment
description: Explain how jobs initialise the shell, load modules, activate Conda/Mamba environments, and access containers. List required compiler/GPU modules and versions, especially those needed by VAMB.
placeholder: Include the batch shebang, module commands, environment locations, and GPU runtime/module.
validations:
required: true

- type: textarea
id: commands
attributes:
label: Scheduler command differences
description: Give commands and syntax for submission, parsable job IDs, arrays and concurrency limits, dependencies, cancellation, status, accounting, and launching parallel tasks (`srun`, `mpirun`, or equivalent).
validations:
required: true

- type: textarea
id: sample
attributes:
label: Tested batch script and output
description: Paste a minimal batch script that has run successfully on this cluster, plus its scheduler submission command. Remove usernames, project IDs, credentials, and sensitive paths.
render: shell
validations:
required: true

- type: textarea
id: caveats
attributes:
label: Other cluster-specific requirements
description: Describe network restrictions, login-node rules, data transfer nodes, email directives, licensed software, environment variables, filesystem performance advice, maintenance windows, or other details that scripts and documentation must handle.

- type: checkboxes
id: confirmation
attributes:
label: Confirmation
options:
- label: I checked these values against current cluster documentation or with the cluster support team.
required: true
- label: I removed credentials, private project identifiers, and sensitive data from this issue.
required: true
23 changes: 23 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
name: Feature request
about: Suggest an improvement or new capability
title: "[Feature]: "
labels: enhancement
assignees: ""
---

## Problem or use case

<!-- What problem would this feature solve, and who would benefit? -->

## Proposed change

<!-- Describe the behaviour or interface you would like. -->

## Alternatives considered

<!-- How are you handling this today? Are there other possible approaches? -->

## Additional context

<!-- Add examples, links, screenshots, or related issues. For a new cluster profile, use the dedicated cluster-support form instead. -->
98 changes: 98 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Contributing to atavide lite

Thank you for helping improve atavide lite. Contributions are welcome from people of every background and experience level, including first-time open-source contributors. Clear questions, documentation fixes, bug reports, tests, and cluster profiles are all valuable contributions.

Please communicate respectfully and assume good intent. It is fine not to know the repository, Git, a scheduler, or a bioinformatics tool yet. Ask questions when something is unclear, avoid sharing sensitive research data or credentials, and follow your institution's policies when describing its systems.

## Start with an issue

Before changing code, search the existing issues for related work. For bugs and general improvements, use the relevant issue template. For support for a new HPC system, open the **New cluster support** form and give as much detail as possible about:

- Slurm, PBS, or the scheduler your cluster uses.
- CPU, high-memory, GPU, long-running, debug, and transfer partitions or queues.
- Maximum wall times, CPUs, memory, nodes, GPUs, job arrays, and queue limits.
- Required project/account, QoS, queue, GPU, export, or other submission directives.
- Shared and node-local scratch storage, long-term storage, quotas, and purge policies.
- Modules, Conda/Mamba, containers, GPU runtimes, and other environment setup.
- Submission, dependency, array, status, accounting, and cancellation commands.

Link to current cluster documentation and include a small batch script that has run successfully. Ask your HPC support team for missing details. Do not include passwords, tokens, private project identifiers, sensitive paths, or research data. The [cluster support guide](docs/cluster-support.md) uses Pawsey Setonix as an example of a complete system description.

Starting with an issue lets maintainers and other users confirm the intended profile, identify policy constraints, and avoid duplicating work.

## Make a branch

Fork the repository if you do not have write access, clone your fork, and create a focused branch from the current default branch:

```bash
git clone https://github.com/<your-user>/atavide_lite.git
cd atavide_lite
git switch -c cluster/examplecluster_shortread
```

Keep unrelated changes on separate branches. If the work takes time, regularly incorporate upstream changes using the Git workflow you are comfortable with.

## Adapt the closest profile

Choose a starting directory by input type and cluster behaviour:

- Use `deepthought_shortread` or `pawsey_shortread` for paired-end reads.
- Use `deepthought_minion` or `pawsey_minion` for single-end or long reads.
- Prefer the Pawsey profiles for systems with shared scratch similar to `/scratch`.
- Prefer the Deepthought profiles when node-local temporary storage is the closer model.

Copy the entire closest directory to a clearly named new directory, then update the copy. Do not change an existing cluster profile merely to make a different cluster work. For example:

```bash
cp -R pawsey_shortread examplecluster_shortread
```

Review every scheduler script, definition file, environment check, path, and README in the new directory. Translate scheduler directives, resource requests, arrays, dependencies, output paths, storage locations, software activation, modules, GPU setup, and submission examples. Keep the pipeline stages consistent with the source profile unless the issue describes a deliberate functional change.

## Using agentic AI

An agentic coding assistant can perform repetitive copying and scheduler-header changes, but it needs precise source material and human review. Give the assistant the cluster-support issue, links to official HPC documentation, and the name of the closest profile. A useful request is:

> Create `examplecluster_shortread` from `pawsey_shortread`. Use the attached cluster-support issue and official scheduler documentation to adapt every scheduler directive, resource request, storage path, environment setup step, and README command. Do not modify existing cluster profiles. List any policy or parameter you cannot verify, and run repository checks without submitting a real cluster job.

Ask the assistant to inspect the whole source directory before editing and to show a diff afterward. Then personally verify every generated resource request and command against current cluster documentation. AI output can contain plausible but invalid queue names, directives, paths, module versions, or limits. Never let an assistant submit jobs, expose credentials, or access sensitive data unless that action is explicitly authorised and understood. Test with a harmless minimal job before using real data or expensive allocations.

## Check the contribution

Before committing:

1. Search the new directory for old cluster names, account codes, paths, partitions, and environment variables.
2. Check that each requested wall time, CPU count, memory value, node count, GPU request, and array size is allowed in its selected queue.
3. Check shell syntax and run the repository's relevant automated tests.
4. Confirm that README commands use the new directory and scheduler.
5. Run one minimal job and then a small, non-sensitive pipeline example on the target cluster when access is available.
6. Review `git diff` and make sure the branch contains no generated data, scheduler output, credentials, or unrelated files.

If cluster access is unavailable, say exactly what was checked locally and what still needs testing. That does not prevent early discussion or a draft pull request.

## Commit and open a pull request

Stage only the intended files, write a concise commit message, and push your branch:

```bash
git status --short
git add examplecluster_shortread
git commit -m "Add ExampleCluster short-read profile"
git push -u origin cluster/examplecluster_shortread
```

Open a pull request against the upstream repository using GitHub's website or GitHub CLI:

```bash
gh pr create --draft --fill
```

In the pull request, link the cluster-support issue and explain:

- Which existing profile was used as the starting point.
- The scheduler, queues, storage, and software-environment changes made.
- Which documentation and cluster policies support the chosen values.
- Tests and sample jobs that passed.
- Anything that remains unverified or needs a maintainer or HPC administrator to review.

Use a draft pull request while work or cluster testing remains. Respond to review comments with follow-up commits rather than replacing discussion history. A maintainer can help decide when the contribution is ready to merge.
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
Atavide lite is a simple, yet complete workflow for metagenomics data analysis, including QC/QA, optional host
removal, annotation, assembly and cross-assembly, and individual read based annotations.

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for an inclusive guide to reporting issues,
adding support for another HPC cluster, using agentic AI carefully, and opening a pull request.

The motivation is based on the more complete [atavide pipeline](https://github.com/linsalrob/atavide) we built that
uses snakemake as a workflow manager. We found that solution to be effective, but routine failure at different steps
was hard to debug and follow. In addition, as we move between compute resources, we need to adjust the time, resources, and
Expand All @@ -26,7 +29,9 @@ that will run on most clusters, and then there are specific scripts for the
Flinders University's [deepthought](https://doi.org/10.25957/FLINDERS.HPC.DEEPTHOUGHT) cluster, and
the [National Computational Infrastructure](https://nci.org.au/) Gadi system. These are
the machines that we use everyday, and so we maintain those scripts to ensure that they work for us. If you would like
to ammend the scripts for your cluster, please submit a pull request, or open an issue, and we will try to address it.
to amend the scripts for your cluster, please submit a pull request or open the dedicated **New cluster support** issue
form. The [cluster support guide](docs/cluster-support.md) explains the information needed to adapt an existing profile
and uses Pawsey Setonix as a worked example.

In our experience, each cluster has enough [minor differences](#system-nuances) and unique ways to optimise data storage,
code execution, and system preferences, that it is more robust to maintain individual scripts for each cluster,
Expand Down Expand Up @@ -151,8 +156,15 @@ processes that use a lot of memory and create temporary files, such as `megahit`
## Pawsey Supercomputing Centre's [Setonix](https://pawsey.org.au/)

Setonix uses slurm for scheduling and has a `/scratch` drive that is used for temporary storage,
but is available from the compute nodes. However, data on `/scratch` is deleted after a short time of not being accessed (approximately 21 days).
Therefore, we need to move data in and out of `/scratch` via Pawsey's acacia file system, and we also need to recreate
but is available from the compute nodes. Setonix compute nodes are shared by default, so jobs should request an explicit
wall time, number of tasks, CPU count, and either memory for shared access or exclusive-node access. Its production partitions
include `work` (24 hours), `long` (96 hours), `highmem` (96 hours), `gpu` (24 hours), `gpu-highmem` (48 hours), and
`copy` (48 hours); the GPU partitions require a project account with a `-gpu` suffix. See the
[cluster support guide](docs/cluster-support.md#worked-example-pawsey-setonix) for resource and queue details and links
to Pawsey's current policies.

Data on `/scratch` is deleted after a short time of not being accessed (approximately 21 days).
Therefore, we need to move data in and out of `/scratch` via Pawsey's Acacia object storage, and we also need to recreate
our conda environments from time to time, so we provide easy scripts to do that.
Comment thread
linsalrob marked this conversation as resolved.

## NCI's [Gadi](https://nci.org.au/)
Expand Down Expand Up @@ -193,4 +205,3 @@ python ~/GitHubs/atavide_lite/summarise_taxonomy/scripts/join_taxonomies.py -t t

The first makes a copy of the per-sample taxonomy files originally in `taxonomy` and puts them in `taxonomy_no_pseudo` but without the `f__Pseudomonadaceae` reads.
The second makes the summary directory with kingdom, phylum, class, etc. summaries without the `f__Pseudomonadaceae` reads.

Loading