From cd9368b1db9b0e519c6668d2ddb7f205bd3e12e7 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 10:19:02 -0700 Subject: [PATCH 1/8] Workflow management summary page Fixes #914 --- docs/Software/Workflow_Managers.md | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 docs/Software/Workflow_Managers.md diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md new file mode 100644 index 000000000..605806cd1 --- /dev/null +++ b/docs/Software/Workflow_Managers.md @@ -0,0 +1,33 @@ +--- +created_at: 2026-06-24 +description: Summary of available workflow managers +tags: + - software +--- + +!!! tip "Webinar" + Check out the recording of our webinar entitled [Introduction to Containers and Workflows](https://www.youtube.com/watch?v=Dn0xIdPfPQ4). + + +Workflow managers help make research more replicable and coordinate tasks that are interdependent. +These tools can range from the simple, running the same process on a large number of inputs, to the complex, managing a series of analyses with filtering logic. + +## What are workflow managers? + +Workflow managers automate multi-step analysis pipelines so that you don't need to run each step or script independently. +They are commonly used in modeling and bioinformatics workflows, but can be applicable in a large range of settings. +Rather than manually checking the results of one task before starting the next, workflow managers can confirm that the first task completed successfully before starting the second task. +Instead of submitting one Slurm job, or job array, for the first task and checking to see when it is finished before starting the second task, a workflow manager allows you to start one task which will monitor the progress of the subtasks and start them when appropriate. + +## What are the options? + +On Mahuika we have several workflow managers installed: +- [Cylc](./Available_Applications/Cylc.md) + - Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains. +- [Nextflow](./Available_Applications/Nextflow.md) + - Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, AWS Batch, Azure Batch, Google Cloud Batch, and Kubernetes. Additionally, it supports many ways to manage your software dependencies, including Conda, Spack, Docker, Podman, Singularity, and more. +- [snakemake](./Available_Applications/snakemake.md) + - The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Snakemake is highly popular, with on average more than 7 new citations per week in 2021, and almost 400k downloads. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment. + +There are reasons for each workflow manager, but a first step for identifying the tool you want to use is to see if anyone in your research group or field is already using a tool and has a workflow you can adapt for your needs. + From 3c74c908976ea654fa18f611990e7c5492143284 Mon Sep 17 00:00:00 2001 From: Jen Reeve <31419037+jen-reeve@users.noreply.github.com> Date: Thu, 25 Jun 2026 05:34:02 +1200 Subject: [PATCH 2/8] Update docs/Software/Workflow_Managers.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Jen Reeve <31419037+jen-reeve@users.noreply.github.com> --- docs/Software/Workflow_Managers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index 605806cd1..3a595fe3e 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -22,11 +22,11 @@ Instead of submitting one Slurm job, or job array, for the first task and checki ## What are the options? On Mahuika we have several workflow managers installed: -- [Cylc](./Available_Applications/Cylc.md) +- [Cylc](../Available_Applications/Cylc.md) - Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains. -- [Nextflow](./Available_Applications/Nextflow.md) +- [Nextflow](../Available_Applications/Nextflow.md) - Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, AWS Batch, Azure Batch, Google Cloud Batch, and Kubernetes. Additionally, it supports many ways to manage your software dependencies, including Conda, Spack, Docker, Podman, Singularity, and more. -- [snakemake](./Available_Applications/snakemake.md) +- [Snakemake](../Available_Applications/snakemake.md) - The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Snakemake is highly popular, with on average more than 7 new citations per week in 2021, and almost 400k downloads. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment. There are reasons for each workflow manager, but a first step for identifying the tool you want to use is to see if anyone in your research group or field is already using a tool and has a workflow you can adapt for your needs. From e70374c76eecc7c0844c5509ab7655c075cae786 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 10:35:06 -0700 Subject: [PATCH 3/8] formatting --- docs/Software/Workflow_Managers.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index 3a595fe3e..9cccc5dbe 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -22,6 +22,7 @@ Instead of submitting one Slurm job, or job array, for the first task and checki ## What are the options? On Mahuika we have several workflow managers installed: + - [Cylc](../Available_Applications/Cylc.md) - Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains. - [Nextflow](../Available_Applications/Nextflow.md) From 4e7162419eb82ca59cceb03f58f3c94549baa167 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 10:37:35 -0700 Subject: [PATCH 4/8] adding table --- docs/Software/Workflow_Managers.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index 9cccc5dbe..171f89256 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -32,3 +32,11 @@ On Mahuika we have several workflow managers installed: There are reasons for each workflow manager, but a first step for identifying the tool you want to use is to see if anyone in your research group or field is already using a tool and has a workflow you can adapt for your needs. + +| Workflow manager | Workflow language | Script language | +| -------------------------------------------------- | ----------------- | --------------- | +| [Cylc](../Available_Applications/Cylc.md) | Python | | +| [Nextflow](./Available_Applications/Nextflow.md) | Groovy DSL | Any | +| [snakemake](./Available_Applications/snakemake.md) | Python | Any | + + From fa9e78a7eefdbbdbf8a0755e943d71a43ee8a3c2 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 10:40:25 -0700 Subject: [PATCH 5/8] Gemini lied --- docs/Software/Workflow_Managers.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index 171f89256..d56d410cb 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -23,11 +23,11 @@ Instead of submitting one Slurm job, or job array, for the first task and checki On Mahuika we have several workflow managers installed: -- [Cylc](../Available_Applications/Cylc.md) +- [Cylc](./Available_Applications/Cylc.md) - Cylc (pronounced silk) is a general purpose workflow engine that also manages cycling systems very efficiently. It is used in production weather, climate, and environmental forecasting on HPC, but is not specialized to those domains. -- [Nextflow](../Available_Applications/Nextflow.md) +- [Nextflow](./Available_Applications/Nextflow.md) - Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, AWS Batch, Azure Batch, Google Cloud Batch, and Kubernetes. Additionally, it supports many ways to manage your software dependencies, including Conda, Spack, Docker, Podman, Singularity, and more. -- [Snakemake](../Available_Applications/snakemake.md) +- [Snakemake](./Available_Applications/snakemake.md) - The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Snakemake is highly popular, with on average more than 7 new citations per week in 2021, and almost 400k downloads. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment. There are reasons for each workflow manager, but a first step for identifying the tool you want to use is to see if anyone in your research group or field is already using a tool and has a workflow you can adapt for your needs. @@ -35,7 +35,7 @@ There are reasons for each workflow manager, but a first step for identifying th | Workflow manager | Workflow language | Script language | | -------------------------------------------------- | ----------------- | --------------- | -| [Cylc](../Available_Applications/Cylc.md) | Python | | +| [Cylc](./Available_Applications/Cylc.md) | Python | | | [Nextflow](./Available_Applications/Nextflow.md) | Groovy DSL | Any | | [snakemake](./Available_Applications/snakemake.md) | Python | Any | From a57ff3122f190123db7873bc253f4b3736ca2567 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 10:48:55 -0700 Subject: [PATCH 6/8] more for the table --- docs/Software/Workflow_Managers.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index d56d410cb..a1967aed5 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -7,7 +7,6 @@ tags: !!! tip "Webinar" Check out the recording of our webinar entitled [Introduction to Containers and Workflows](https://www.youtube.com/watch?v=Dn0xIdPfPQ4). - Workflow managers help make research more replicable and coordinate tasks that are interdependent. These tools can range from the simple, running the same process on a large number of inputs, to the complex, managing a series of analyses with filtering logic. @@ -32,11 +31,8 @@ On Mahuika we have several workflow managers installed: There are reasons for each workflow manager, but a first step for identifying the tool you want to use is to see if anyone in your research group or field is already using a tool and has a workflow you can adapt for your needs. - -| Workflow manager | Workflow language | Script language | -| -------------------------------------------------- | ----------------- | --------------- | -| [Cylc](./Available_Applications/Cylc.md) | Python | | -| [Nextflow](./Available_Applications/Nextflow.md) | Groovy DSL | Any | -| [snakemake](./Available_Applications/snakemake.md) | Python | Any | - - +| Workflow manager | Workflow language | Script language | Slurm integration | Use GPU | Dynamic resource allocation | Failed job retry | +| -------------------------------------------------- | ----------------- | --------------- | ----------------- | -- | -- | -- | +| [Cylc](./Available_Applications/Cylc.md) | Python | bash | Built in | | | | +| [Nextflow](./Available_Applications/Nextflow.md) | Groovy DSL | Any | Built in | Yes | Yes | Yes | +| [snakemake](./Available_Applications/snakemake.md) | Python | Any | Plugin installed | | | | From c6e38078f078697d3f975909cdb406c6a83760c4 Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 11:09:02 -0700 Subject: [PATCH 7/8] using real words --- docs/Software/Workflow_Managers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index a1967aed5..e74332f1d 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -8,7 +8,7 @@ tags: !!! tip "Webinar" Check out the recording of our webinar entitled [Introduction to Containers and Workflows](https://www.youtube.com/watch?v=Dn0xIdPfPQ4). -Workflow managers help make research more replicable and coordinate tasks that are interdependent. +Workflow managers help make research more reproducible and coordinate tasks that are interdependent. These tools can range from the simple, running the same process on a large number of inputs, to the complex, managing a series of analyses with filtering logic. ## What are workflow managers? From 30520b9edf6036b9cab65b5c6ea55702211b45ca Mon Sep 17 00:00:00 2001 From: jen-reeve Date: Wed, 24 Jun 2026 11:09:49 -0700 Subject: [PATCH 8/8] contact support --- docs/Software/Workflow_Managers.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Software/Workflow_Managers.md b/docs/Software/Workflow_Managers.md index e74332f1d..a497c2282 100644 --- a/docs/Software/Workflow_Managers.md +++ b/docs/Software/Workflow_Managers.md @@ -36,3 +36,5 @@ There are reasons for each workflow manager, but a first step for identifying th | [Cylc](./Available_Applications/Cylc.md) | Python | bash | Built in | | | | | [Nextflow](./Available_Applications/Nextflow.md) | Groovy DSL | Any | Built in | Yes | Yes | Yes | | [snakemake](./Available_Applications/snakemake.md) | Python | Any | Plugin installed | | | | + +As always, feel free to {% include "partials/support_request.html" %} with any questions or for help implementing a workflow manager in your research.