-
Notifications
You must be signed in to change notification settings - Fork 4
Add issue templates and cluster contribution guide #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| --- | ||
| name: Bug report | ||
| about: Report a reproducible problem with atavide lite | ||
| title: "[Bug]: " | ||
| labels: bug | ||
| assignees: "" | ||
| --- | ||
|
|
||
| ## What happened? | ||
|
|
||
| <!-- Describe the problem and what you expected to happen instead. --> | ||
|
|
||
| ## How can we reproduce it? | ||
|
|
||
| <!-- Include the commands you ran, the pipeline stage, and a minimal example if possible. --> | ||
|
|
||
| 1. | ||
| 2. | ||
| 3. | ||
|
|
||
| ## Error output | ||
|
|
||
| <!-- Paste the relevant scheduler .out/.err output or traceback. Remove credentials and sensitive data. --> | ||
|
|
||
| ```text | ||
|
|
||
| ``` | ||
|
|
||
| ## Environment | ||
|
|
||
| - atavide lite commit or version: | ||
| - Pipeline profile (for example, `pawsey_shortread`): | ||
| - Cluster and scheduler: | ||
| - Operating system: | ||
| - Conda environment or relevant tool versions: | ||
| - Input type (paired short reads or single/long reads): | ||
|
|
||
| ## Additional context | ||
|
|
||
| <!-- Include anything else that may help, such as resource requests, job ID, or a link to cluster documentation. --> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| name: New cluster support | ||
| description: Request an atavide lite profile for another HPC cluster | ||
| title: "[Cluster]: Support " | ||
| labels: | ||
| - enhancement | ||
| - cluster-support | ||
| body: | ||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| Thanks for helping us port atavide lite. Please ask your HPC support team for any details you do not know. A complete response should let us adapt one of `deepthought_minion`, `deepthought_shortread`, `pawsey_minion`, or `pawsey_shortread` without guessing cluster policy. | ||
|
|
||
| - type: input | ||
| id: cluster | ||
| attributes: | ||
| label: Cluster and organisation | ||
| description: Give the cluster name and the organisation that operates it. | ||
| placeholder: Setonix — Pawsey Supercomputing Research Centre | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: input | ||
| id: docs | ||
| attributes: | ||
| label: Cluster documentation | ||
| description: Link to the scheduler, partition/queue, storage, and software-environment documentation. | ||
| placeholder: https://example.edu/hpc/running-jobs | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: scheduler | ||
| attributes: | ||
| label: Scheduler | ||
| description: Which workload manager accepts batch jobs? | ||
| options: | ||
| - Slurm | ||
| - PBS / PBS Pro / OpenPBS | ||
| - Other (describe below) | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: data_type | ||
| attributes: | ||
| label: Sequencing input to support | ||
| description: This selects the closest existing profile. | ||
| options: | ||
| - Paired-end short reads (adapt `deepthought_shortread` or `pawsey_shortread`) | ||
| - Single-end or long reads (adapt `deepthought_minion` or `pawsey_minion`) | ||
| - Both | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: closest_profile | ||
| attributes: | ||
| label: Closest existing profile | ||
| description: Select the implementation whose storage and input model is most similar. | ||
| options: | ||
| - pawsey_shortread | ||
| - pawsey_minion | ||
| - deepthought_shortread | ||
| - deepthought_minion | ||
| - Unsure | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: partitions | ||
| attributes: | ||
| label: Partitions or queues | ||
| description: List every usable CPU, high-memory, GPU, long-running, debug, and data-transfer partition/queue. For each, give its purpose, maximum wall time, cores and usable memory per node, maximum nodes, and GPU type/count where relevant. | ||
| placeholder: | | ||
| | Name | Purpose | Max wall time | Cores/node | Memory/node | Max nodes | GPUs/node | | ||
| | --- | --- | --- | --- | --- | --- | --- | | ||
| | work | CPU production | 24 h | 128 | 230 GiB | ... | — | | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: resource_mapping | ||
| attributes: | ||
| label: Pipeline resource mapping | ||
| description: Explain which partition/queue should run ordinary steps, MMseqs2 or large assemblies, GPU VAMB, downloads/transfers, and jobs that exceed the normal time limit. Include realistic time, CPU, memory, node, and GPU requests for each class. | ||
| placeholder: | | ||
| - Ordinary per-sample jobs: ... | ||
| - MMseqs2 / memory-heavy jobs: ... | ||
| - Assembly / long-running jobs: ... | ||
| - VAMB / GPU jobs: ... | ||
| - Downloads and data transfers: ... | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: directives | ||
| attributes: | ||
| label: Required submission directives and policies | ||
| description: Provide exact `#SBATCH` or `#PBS` syntax for account/project, partition/queue, QoS, wall time, memory, CPUs/tasks/nodes, GPUs, job arrays, output/error paths, and any required export, reservation, constraint, or exclusive-node attributes. State which are mandatory and include a minimal working batch header. | ||
| render: shell | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: qos_limits | ||
| attributes: | ||
| label: QoS, accounting, and queue limits | ||
| description: List allowed/default QoS values, project or account naming rules (including separate GPU accounts), charging rules, maximum running/submitted/array jobs, and any per-user or per-project restrictions. Write “none” only if confirmed by the cluster documentation. | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: storage | ||
| attributes: | ||
| label: Storage and temporary files | ||
| description: Give paths or environment variables for home, software, shared scratch, node-local scratch, and long-term storage. Include quotas, purge periods, file-count limits, staging commands, and which paths are visible from compute nodes. | ||
| placeholder: Shared scratch is /scratch/$PROJECT/$USER, visible on compute nodes, and files are purged after ... | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: software | ||
| attributes: | ||
| label: Software environment | ||
| description: Explain how jobs initialise the shell, load modules, activate Conda/Mamba environments, and access containers. List required compiler/GPU modules and versions, especially those needed by VAMB. | ||
| placeholder: Include the batch shebang, module commands, environment locations, and GPU runtime/module. | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: commands | ||
| attributes: | ||
| label: Scheduler command differences | ||
| description: Give commands and syntax for submission, parsable job IDs, arrays and concurrency limits, dependencies, cancellation, status, accounting, and launching parallel tasks (`srun`, `mpirun`, or equivalent). | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: sample | ||
| attributes: | ||
| label: Tested batch script and output | ||
| description: Paste a minimal batch script that has run successfully on this cluster, plus its scheduler submission command. Remove usernames, project IDs, credentials, and sensitive paths. | ||
| render: shell | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: caveats | ||
| attributes: | ||
| label: Other cluster-specific requirements | ||
| description: Describe network restrictions, login-node rules, data transfer nodes, email directives, licensed software, environment variables, filesystem performance advice, maintenance windows, or other details that scripts and documentation must handle. | ||
|
|
||
| - type: checkboxes | ||
| id: confirmation | ||
| attributes: | ||
| label: Confirmation | ||
| options: | ||
| - label: I checked these values against current cluster documentation or with the cluster support team. | ||
| required: true | ||
| - label: I removed credentials, private project identifiers, and sensitive data from this issue. | ||
| required: true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| --- | ||
| name: Feature request | ||
| about: Suggest an improvement or new capability | ||
| title: "[Feature]: " | ||
| labels: enhancement | ||
| assignees: "" | ||
| --- | ||
|
|
||
| ## Problem or use case | ||
|
|
||
| <!-- What problem would this feature solve, and who would benefit? --> | ||
|
|
||
| ## Proposed change | ||
|
|
||
| <!-- Describe the behaviour or interface you would like. --> | ||
|
|
||
| ## Alternatives considered | ||
|
|
||
| <!-- How are you handling this today? Are there other possible approaches? --> | ||
|
|
||
| ## Additional context | ||
|
|
||
| <!-- Add examples, links, screenshots, or related issues. For a new cluster profile, use the dedicated cluster-support form instead. --> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # Contributing to atavide lite | ||
|
|
||
| Thank you for helping improve atavide lite. Contributions are welcome from people of every background and experience level, including first-time open-source contributors. Clear questions, documentation fixes, bug reports, tests, and cluster profiles are all valuable contributions. | ||
|
|
||
| Please communicate respectfully and assume good intent. It is fine not to know the repository, Git, a scheduler, or a bioinformatics tool yet. Ask questions when something is unclear, avoid sharing sensitive research data or credentials, and follow your institution's policies when describing its systems. | ||
|
|
||
| ## Start with an issue | ||
|
|
||
| Before changing code, search the existing issues for related work. For bugs and general improvements, use the relevant issue template. For support for a new HPC system, open the **New cluster support** form and give as much detail as possible about: | ||
|
|
||
| - Slurm, PBS, or the scheduler your cluster uses. | ||
| - CPU, high-memory, GPU, long-running, debug, and transfer partitions or queues. | ||
| - Maximum wall times, CPUs, memory, nodes, GPUs, job arrays, and queue limits. | ||
| - Required project/account, QoS, queue, GPU, export, or other submission directives. | ||
| - Shared and node-local scratch storage, long-term storage, quotas, and purge policies. | ||
| - Modules, Conda/Mamba, containers, GPU runtimes, and other environment setup. | ||
| - Submission, dependency, array, status, accounting, and cancellation commands. | ||
|
|
||
| Link to current cluster documentation and include a small batch script that has run successfully. Ask your HPC support team for missing details. Do not include passwords, tokens, private project identifiers, sensitive paths, or research data. The [cluster support guide](docs/cluster-support.md) uses Pawsey Setonix as an example of a complete system description. | ||
|
|
||
| Starting with an issue lets maintainers and other users confirm the intended profile, identify policy constraints, and avoid duplicating work. | ||
|
|
||
| ## Make a branch | ||
|
|
||
| Fork the repository if you do not have write access, clone your fork, and create a focused branch from the current default branch: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/<your-user>/atavide_lite.git | ||
| cd atavide_lite | ||
| git switch -c cluster/examplecluster_shortread | ||
| ``` | ||
|
|
||
| Keep unrelated changes on separate branches. If the work takes time, regularly incorporate upstream changes using the Git workflow you are comfortable with. | ||
|
|
||
| ## Adapt the closest profile | ||
|
|
||
| Choose a starting directory by input type and cluster behaviour: | ||
|
|
||
| - Use `deepthought_shortread` or `pawsey_shortread` for paired-end reads. | ||
| - Use `deepthought_minion` or `pawsey_minion` for single-end or long reads. | ||
| - Prefer the Pawsey profiles for systems with shared scratch similar to `/scratch`. | ||
| - Prefer the Deepthought profiles when node-local temporary storage is the closer model. | ||
|
|
||
| Copy the entire closest directory to a clearly named new directory, then update the copy. Do not change an existing cluster profile merely to make a different cluster work. For example: | ||
|
|
||
| ```bash | ||
| cp -R pawsey_shortread examplecluster_shortread | ||
| ``` | ||
|
|
||
| Review every scheduler script, definition file, environment check, path, and README in the new directory. Translate scheduler directives, resource requests, arrays, dependencies, output paths, storage locations, software activation, modules, GPU setup, and submission examples. Keep the pipeline stages consistent with the source profile unless the issue describes a deliberate functional change. | ||
|
|
||
| ## Using agentic AI | ||
|
|
||
| An agentic coding assistant can perform repetitive copying and scheduler-header changes, but it needs precise source material and human review. Give the assistant the cluster-support issue, links to official HPC documentation, and the name of the closest profile. A useful request is: | ||
|
|
||
| > Create `examplecluster_shortread` from `pawsey_shortread`. Use the attached cluster-support issue and official scheduler documentation to adapt every scheduler directive, resource request, storage path, environment setup step, and README command. Do not modify existing cluster profiles. List any policy or parameter you cannot verify, and run repository checks without submitting a real cluster job. | ||
|
|
||
| Ask the assistant to inspect the whole source directory before editing and to show a diff afterward. Then personally verify every generated resource request and command against current cluster documentation. AI output can contain plausible but invalid queue names, directives, paths, module versions, or limits. Never let an assistant submit jobs, expose credentials, or access sensitive data unless that action is explicitly authorised and understood. Test with a harmless minimal job before using real data or expensive allocations. | ||
|
|
||
| ## Check the contribution | ||
|
|
||
| Before committing: | ||
|
|
||
| 1. Search the new directory for old cluster names, account codes, paths, partitions, and environment variables. | ||
| 2. Check that each requested wall time, CPU count, memory value, node count, GPU request, and array size is allowed in its selected queue. | ||
| 3. Check shell syntax and run the repository's relevant automated tests. | ||
| 4. Confirm that README commands use the new directory and scheduler. | ||
| 5. Run one minimal job and then a small, non-sensitive pipeline example on the target cluster when access is available. | ||
| 6. Review `git diff` and make sure the branch contains no generated data, scheduler output, credentials, or unrelated files. | ||
|
|
||
| If cluster access is unavailable, say exactly what was checked locally and what still needs testing. That does not prevent early discussion or a draft pull request. | ||
|
|
||
| ## Commit and open a pull request | ||
|
|
||
| Stage only the intended files, write a concise commit message, and push your branch: | ||
|
|
||
| ```bash | ||
| git status --short | ||
| git add examplecluster_shortread | ||
| git commit -m "Add ExampleCluster short-read profile" | ||
| git push -u origin cluster/examplecluster_shortread | ||
| ``` | ||
|
|
||
| Open a pull request against the upstream repository using GitHub's website or GitHub CLI: | ||
|
|
||
| ```bash | ||
| gh pr create --draft --fill | ||
| ``` | ||
|
|
||
| In the pull request, link the cluster-support issue and explain: | ||
|
|
||
| - Which existing profile was used as the starting point. | ||
| - The scheduler, queues, storage, and software-environment changes made. | ||
| - Which documentation and cluster policies support the chosen values. | ||
| - Tests and sample jobs that passed. | ||
| - Anything that remains unverified or needs a maintainer or HPC administrator to review. | ||
|
|
||
| Use a draft pull request while work or cluster testing remains. Respond to review comments with follow-up commits rather than replacing discussion history. A maintainer can help decide when the contribution is ready to merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.