nextstrain run: Analysis directory configuration

[victorlin]

Currently, the analysis directory is loosely coupled with the pathogen workflow that it is intended to run with.¹ That is, a user needs to invoke nextstrain run with the same pathogen and workflow that the analysis directory's contents were created for.

nextstrain run measles phylogenetic .

I think we should make this coupling more explicit. Analysis directories are likely to look similar with a config.yaml and supporting files. Because of the similarities across pathogen workflows, the contents might even run successfully under a different pathogen workflow, mixing result files from different pathogens.

¹ I'm speculating that each analysis directory is only created for one pathogen workflow. Maybe that's not the case and users would prefer to use the same config across different workflows? Even then, it seems weird to mix files from runs of different pathogen workflows, and I think a better pattern is to use separate analysis directories with identical config files.

Possible solution

Read the information from a file. Example contents:

pathogen: measles@main
workflow: phylogenetic

Options:

1. Introduce a new configuration file for the analysis directory, e.g. nextstrain.yaml or nextstrain-analysis.yaml.
2. Overload the existing config.yaml (workflow-specific, but standard).

Then the user can simply invoke nextstrain run without any arguments.

Prototypes

• b12e21a: nextstrain run [--pathogen] [--workflow] [<analysis-directory>] writes to nextstrain-analysis.yaml

[joverlee521]

Oh, interesting idea! I might also add on version or pathogen_version to the configuration file (thinking of the potential workflow versions work for PGCOE). I'm also tempted to add runtime/runtime_version for reproducibility, but don't want to derail this idea.

A couple questions on how this would flow:

1. Should nextstrain run measles phylogenetic . automatically create nextstrain.yaml if it doesn't already exist?
2. Would there be an error if pathogen and workflow in the command don't match the values in nextstrain.yaml?

[victorlin]

I might also add on version or pathogen_version to the configuration file

👍 pathogen can take any value that is currently accepted by the positional parameter, <pathogen-name>[@<version>]|<pathogen-path>.

I'm also tempted to add runtime/runtime_version for reproducibility, but don't want to derail this idea.

I also thought of this, but it clashes with #448 – should runtime be configured by the pathogen repo or analysis directory (or both)? We can consider it separately from the more well-defined pathogen/workflow.

1. Should nextstrain run measles phylogenetic . automatically create nextstrain.yaml if it doesn't already exist?
2. Would there be an error if pathogen and workflow in the command don't match the values in nextstrain.yaml?

pathogen and workflow are currently required positional arguments. We could move them to optional --pathogen and --workflow arguments, but then we'd need to consider (2). I think it'd be easier if we make one big shift from args to config entirely.

We could automatically create nextstrain.yaml with a new command like nextstrain init <analysis-directory> (basically "configure the pathogen workflow as a one-time setup for the analysis directory" in https://github.com/nextstrain/nextstrain-desktop/issues/1#issuecomment-5273025421.

[joverlee521]

I also thought of this, but it clashes with #448 – should runtime be configured by the pathogen repo or analysis directory (or both)? We can consider it separately from the more well-defined pathogen/workflow.

Ah right, that makes sense. Maybe this would a difference of minimum version (configured in repo) vs actual version used in workflow (configured in analysis dir)? But yeah, can consider this later...

I think it'd be easier if we make one big shift from args to config entirely.

Interesting, will think on this. It does make sense especially if we are focusing on use of nextstrain run within nextstrain-desktop.

[victorlin]

Pushed a prototype to https://github.com/nextstrain/cli/tree/victorlin/analysis-configuration.

analysis-configuration.mov

[victorlin]

We should call this something other than "analysis configuration" to avoid confusion with the workflow configuration file.

Some alternatives:

• analysis manifest
• analysis context
• analysis definition

Keep in mind that contents could look like this in the future:

pathogen: measles@v1
workflow: phylogenetic
runtime: docker
image: nextstrain/base:build-20260819T223854Z

[victorlin]

Another option is to put everything in one file which can continue to be referenced as "configuration".

pathogen: measles@v1
workflow: phylogenetic
runtime: docker
image: nextstrain/base:build-20260819T223854Z

builds:
  - genome/global
  - genome/north-america

additional_inputs:
  - name: ppx_restricted
    metadata: "s3://nextstrain-data/files/workflows/measles/metadata_restricted.tsv.zst"
    sequences: "s3://nextstrain-data/files/workflows/measles/sequences_restricted.fasta.zst"
…

[jameshadfield]

Another option is to put everything in one file which can continue to be referenced as "configuration".

I like this, although we might be wise to nest them, e.g. nextstrain_run.pathogen: measles@v1

[joverlee521]

Another option is to put everything in one file which can continue to be referenced as "configuration".

Will this be the same file that is passed to Snakemake as the configfile? Would we have to configure Snakemake to ignore the nextstrain_run params or do we add them to the workflow config schema as well so that changes in nextstrain_run params also trigger re-runs?

[victorlin]

Will this be the same file that is passed to Snakemake as the configfile?

It's either that, or we write another YAML file with nextstrain_run deleted. I think easier to work with one file.

Would we have to configure Snakemake to ignore the nextstrain_run params or do we add them to the workflow config schema as well so that changes in nextstrain_run params also trigger re-runs?

We should add them to the workflow config schema so that validation doesn't fail due to additionalProperties: false.

We may want to treat changes to nextstrain_run params differently per param. Changes to pathogen/workflow should be user error. Changes to runtime/image may warrant re-runs, but we can cross that road later.