Add PR-driven reference-data workflow-bundle pipeline (MetaPhlAn, mOTUs, SameStr) by mvdbeek · Pull Request #89 · galaxyproject/idc

mvdbeek · 2026-07-09T20:34:43Z

What

A PR-driven pipeline for building versioned reference databases with Galaxy data managers and publishing them to CVMFS — complementing the existing genome-indexing pipeline. First data managers: MetaPhlAn, mOTUs, and SameStr (chained on MetaPhlAn).

A contributor requests a version by adding one flat YAML file at data-managers/<data_manager>/<version>.yaml; the pipeline builds it as a data-manager-bundle workflow on test.galaxyproject.org (via planemo) and Jenkins imports the bundle onto CVMFS.

The three stages

Lint (.github/workflows/lint.yml, PR): scripts/request_models.py validates the request (version-pinned toolshed GUID, folder/table match, depends_on resolution) and gxformat2-validates the workflow it generates; runs the unit tests; and warns (informational) if the data already exists. Fork-safe, no secrets.
Build (.github/workflows/build.yml, on merge): scripts/generate_build.py turns a request into a gxformat2 workflow (each DM step __data_manager_mode: bundle) and planemo runs it on test.galaxyproject.org into history idc-<dm>-<version>. Needs secret TEST_API_KEY.
Import (.ci/import_reference_data.sh + splice in .ci/jenkins.sh): resolves each bundle from the build's workflow invocation and imports it with galaxy-import-data-bundle, recording record/<dm>/<version> for idempotency.

Highlights

gxformat2 validation of every generated workflow (strict schema + format2→native + semantic lint), run in CI.
Chained builds: SameStr's db_source|database is wired from the upstream MetaPhlAn bundle in a single invocation. If the upstream already exists in the data table, it's referenced directly instead of rebuilt.
"Does it already exist?" uses the authoritative, public tool data table API (GET /api/tool_data/<table>) — catching data available from any source (e.g. the byhand CVMFS) — at lint (warn), build (skip), and import (graceful skip).
22 offline unit tests covering validation, generation, bundle resolution, import, and existence checking.

Verified live on test.galaxyproject.org

Full MetaPhlAn build end-to-end → data_manager_json bundle with is_bundle=True and the correct table row; resolver reads it from the invocation's named output.
SameStr chain: downstream step correctly consumed the upstream MetaPhlAn bundle; reference-existing variant fed the existing table value with no rebuild.

Not yet exercised (need environments outside CI)

SameStr live green — blocked by a server-side routing gap on test.galaxyproject.org (data_manager_samestr has no eligible destination; the same fix MetaPhlAn already got).
CVMFS import — the jenkins.sh splice is additive and clearly marked UNTESTED; needs a real Jenkins/Stratum-0 run.
Large real-version builds.

Opened as a draft for review of the approach.

…generator) Introduce a contribution model where a reference-data version is requested via a single flat YAML file at data-managers/<data_manager>/<version>.yaml, built by a Galaxy data-manager *bundle* workflow (run on test.galaxyproject.org via planemo) and later imported onto CVMFS. Phase 1 - schema & validation: - scripts/request_models.py: pydantic schema + linter for request files (version-pinned toolshed GUID required, dir/table match, depends_on upstream must exist, reject versions already in published.yml). - .github/workflows/lint.yml: Stage 1 CI (fork-safe, no secrets). - config/tool_data_table_conf.xml: add metaphlan_database_versioned, samestr_db and motus_db_versioned tables. - published.yml: idempotency ledger (empty). - Seed requests for MetaPhlAn, mOTUs, and SameStr (chained on MetaPhlAn). Phase 2 - build generation: - scripts/generate_build.py: generate a gxformat2 bundle workflow + planemo job from a request. Build parameters are exposed as workflow inputs and connected to the tool params; chained requests (depends_on) become multi-step workflows that wire the upstream bundle's out_file into the downstream data manager. Validated offline via gxformat2 format2->native conversion.

- generate_build.py now declares each data manager's bundle as a named workflow output (<step>_bundle -> <step>/out_file), so the invocation surfaces bundle datasets directly for the Stage 3 CVMFS import. - Add validate_workflow(): strict gxformat2 schema (Format2StrictModel), format2->native conversion, and the core gxformat2 semantic linter; run by default on generation (--no-validate to skip). - Add --all to generate every request's workflow (shell-independent). - lint.yml: install gxformat2 and validate all generated build workflows in CI, catching wiring errors (e.g. a chained request with no CHAIN_WIRING entry).

- scripts/get_bundle_urls.py: resolve every reference-data bundle a build's workflow invocation produced from its named *_bundle outputs. A chained build (e.g. samestr, which also builds MetaPhlAn) yields several bundles and all are returned, unlike the legacy single-bundle .ci/get-bundle-url.py. Supports a saved invocation dict (offline), an invocation id, or a history-name fallback. - scripts/import_bundles.py: import each bundle with galaxy-import-data-bundle into <cvmfs-root>/data, with record/<dm>/<version> idempotency markers and a --dry-run for offline verification. Intended to run in the Jenkins CVMFS transaction (.ci/jenkins.sh). - tests/test_pipeline.py: offline pytest covering request validation, build generation + gxformat2 validation, bundle resolution, and import command assembly / idempotency. - lint.yml: run the pytest suite in CI; broaden path triggers to scripts/ and tests/.

- .github/workflows/build.yml: on merge to main, build each changed (and not-yet-published) request on test.galaxyproject.org via planemo (--no_wait), leaving bundles in history idc-<dm>-<version>. Needs secret TEST_API_KEY. - scripts/pending_requests.py: filter requests down to those not in published.yml (drives build.yml's selection; keeps the fragile logic out of shell heredocs). - .ci/import_reference_data.sh: Stage 3 driver - for each request, import its bundles onto CVMFS via import_bundles.py, keyed by history idc-<dm>-<version>. Runs inside the Jenkins CVMFS transaction. - get_bundle_urls.bundles_from_history: resolve by the history's *workflow invocation* named outputs (precise - exactly the bundles that build produced), falling back to a dataset scan only when there is no invocation. - import_bundles.py: add --history-name mode (the stable Stage 2 -> Stage 3 key). - tests: history->invocation resolution (+ fallback), pending-requests filter.

Add import_reference_data_bundles() + has_reference_data_requests(), and call them from do_import_remote so the CVMFS transaction opens for genome OR reference-data work. For each request under data-managers/, import_bundles.py resolves the build's bundles from its workflow invocation (history idc-<dm>-<version>) on REFERENCE_DATA_GALAXY_URL (test.galaxyproject.org) and imports each with galaxy-import-data-bundle, recording record/<dm>/<version>. Clearly marked UNTESTED against live Jenkins/Stratum 0: only the remote (non-docker) path is wired, and the genome pipeline is left untouched (its PUBLISH_GALAXY_URL is unchanged).

…flow Add a README section covering the flat request-YAML format (standalone and chained/depends_on), the three pipeline stages (lint -> build on test.galaxyproject.org -> Jenkins CVMFS import), how to onboard a new data manager, and local testing commands.

…table API The authoritative 'does this already exist?' check is the target Galaxy's data table: GET /api/tool_data/<table> (public, no key) lists what is actually available there from any source - including the byhand data.galaxyproject.org CVMFS - so we never rebuild/re-import data a Galaxy already has. - scripts/check_data_exists.py: query the data table and report whether a request's version is present (heuristic match over version / params / depends_on, handling each data manager's differing identity column). Modes: default (exit 1 if any exist), --warn (annotate), --print-new (filter). - lint.yml: informational --warn step (fork-safe, public endpoint). - build.yml: filter out requests whose data already exists before building. - import: import_bundles.py / bundles_from_history now skip gracefully when the build history is absent (i.e. the build skipped a request that already exists), instead of erroring - so import naturally follows the build's decision. - README: document the existence check across lint/build/import. Verified live: metaphlan mpa_vJan21 (already in the byhand table) is flagged; motus 3.1.0 and samestr (empty tables) are treated as new.

For a chained request (depends_on), if the upstream database already exists in the reference Galaxy's data table, the generated workflow now references that existing entry directly - a single downstream step whose data-table input is fed the existing entry's value - instead of adding an upstream build step. This avoids rebuilding a database that already exists (e.g. a MetaPhlAn DB already provided by the byhand CVMFS). - check_data_exists.py: resolve_existing_value() returns the data-table value of an existing entry for a version. - generate_build.py: build(..., reference_galaxy=...) references existing upstreams; without --reference-galaxy it still chains (rebuilds), so offline generation/CI validation is unchanged. - build.yml: pass --reference-galaxy https://test.galaxyproject.org. Verified live: the referencing samestr workflow ran as a single step and the samestr job received db_source.database = the existing MetaPhlAn table value (mpa_vJan21_CHOCOPhlAnSGB_202103-04042023) - no rebuild. (The job itself is blocked only by the same server-side data_manager_samestr routing gap.)

mvdbeek · 2026-07-09T20:44:17Z

@jmchilton @natefoo @bernt-matthias this is the slop version of what i think could work ? I'm not crazy about https://github.com/galaxyproject/idc/pull/89/changes#diff-02cd8fc04fe67ae8f0ed088e0460943a5b606f36236b774f3f7a9a8c6b6c48cfR52 but maybe this is a good enough start and we start building out our reference data coverage if we "like" this ?

jmchilton · 2026-07-10T18:59:52Z

So much system stuff that I don't have context for and I don't know how to review. But I scanned through it a couple times and read the PR description and I'm excited to see progress.

Stage 3 imports bundles from $REFERENCE_DATA_GALAXY_URL (test.galaxyproject.org), a distinct server from $PUBLISH_GALAXY_URL. Add $REFERENCE_DATA_API_KEY for it, falling back to $EPHEMERIS_API_KEY for backward compatibility, and pass it to import_bundles.py instead of reusing the publish-server key.

mvdbeek · 2026-07-11T17:25:01Z

@galaxybot deploy

The Stratum 0 (cvmfs0-psu0) was upgraded off the rh-python38 SCL; that path no longer exists and setup_remote_ephemeris died at venv creation. Default to the system /usr/bin/python3 (Python 3.9), env-overridable for future OS bumps.

galaxy2 (tools-iuc#8172) reconstructs MetaPhlAn markers via bowtie2-inspect; installed on test.galaxyproject.org via usegalaxy-tools#1596.

mvdbeek · 2026-07-11T18:01:41Z

@galaxybot deploy

The Stratum 0's /usr/bin/python3 is 3.9, too old for galaxy-import-data-bundle's deps (yacman>=1.0 needs 3.10+ syntax) - the import crashed on 'unsupported operand type(s) for |'. Use /cvmfs/test.galaxyproject.org/venv's Python 3.13, which is version-controlled and OS-bump-proof.

@galaxybot

'@galaxybot deploy reference-data' (or REFERENCE_DATA_ONLY=true) imports only data-managers/ bundles and skips the genome tool-data build+import, so a reference-data publish can be tested without also running pending genome imports.

mvdbeek · 2026-07-11T18:44:59Z

@galaxybot deploy reference-data

mvdbeek · 2026-07-11T18:50:27Z

@galaxybot deploy reference-data

mvdbeek · 2026-07-11T18:52:55Z

@galaxybot deploy reference-data

The Stratum 0's system python3 is 3.9 (too old for galaxy-maintenance-scripts' deps) and CVMFS-provided Pythons (/cvmfs/test.galaxyproject.org/venv) aren't mounted on every worker. setup_remote_python() installs uv into the remote workdir and uses it to fetch a pinned standalone CPython ($REMOTE_PYTHON_VERSION, default 3.13), setting $REMOTE_PYTHON. An explicit REMOTE_PYTHON in the env still wins and skips the bootstrap.

mvdbeek · 2026-07-11T19:03:35Z

@galaxybot deploy reference-data

stop_build_galaxy unconditionally sourced ./ansible-venv/bin/activate, but that venv only exists when the build Galaxy runs (genome DM tasks). An import-only run (reference-data-only, or 'Nothing to build') never starts it, so the teardown failed and marked the build FAILURE *after* a successful CVMFS publish. Guard on $BUILD_GALAXY_UP, matching trap_handler.

mvdbeek · 2026-07-11T20:08:40Z

Successfully published metaphlan mpa_vJan21_CHOCOPhlAnSGB_202103 🎉

Bumps the requested tool_id from motus_db_fetcher/3.1.0+galaxy0 to +galaxy1, the fix from the just-merged bgruening/galaxytools data_manager_motus_db_downloader update (toolshed revision 603939ba21ec). Depends on galaxyproject/usegalaxy-tools#1600 installing galaxy1 on test.galaxyproject.org first; do not deploy a motus build until that lands.

mvdbeek · 2026-07-12T16:51:31Z

@galaxybot deploy reference-data

mvdbeek · 2026-07-12T17:40:43Z

Next deploy also successful:

/var/spool/cvmfs/idc.galaxyproject.org/scratch/current
├── config
│   └── motus_db_versioned.loc
├── data
│   └── motus_database
│       └── db_from_2026-07-12T140418Z
│           └── db_mOTU
│               ├── README
│               ├── db_mOTU_DB_CEN.fasta
│               ├── db_mOTU_DB_CEN.fasta.amb
│               ├── db_mOTU_DB_CEN.fasta.ann
│               ├── db_mOTU_DB_CEN.fasta.annotations
│               ├── db_mOTU_DB_CEN.fasta.bwt
│               ├── db_mOTU_DB_CEN.fasta.pac
│               ├── db_mOTU_DB_CEN.fasta.sa
│               ├── db_mOTU_DB_NR.fasta
│               ├── db_mOTU_DB_NR.fasta.amb
│               ├── db_mOTU_DB_NR.fasta.ann
│               ├── db_mOTU_DB_NR.fasta.bwt
│               ├── db_mOTU_DB_NR.fasta.pac
│               ├── db_mOTU_DB_NR.fasta.sa
│               ├── db_mOTU_MAP_MGCs_to_mOTUs.tsv
│               ├── db_mOTU_MAP_MGCs_to_mOTUs_in-line.tsv
│               ├── db_mOTU_MAP_genes_to_MGCs.tsv
│               ├── db_mOTU_bam_header_CEN
│               ├── db_mOTU_bam_header_NR
│               ├── db_mOTU_genes_length_NR
│               ├── db_mOTU_padding_coordinates_CEN.tsv
│               ├── db_mOTU_padding_coordinates_NR.tsv
│               ├── db_mOTU_taxonomy_CAMI.tsv
│               ├── db_mOTU_taxonomy_meta-mOTUs.tsv
│               ├── db_mOTU_taxonomy_ref-mOTUs.tsv
│               ├── db_mOTU_taxonomy_ref-mOTUs_short_names.tsv
│               ├── db_mOTU_test
│               │   ├── test1.motus
│               │   └── test1_single.fastq
│               ├── db_mOTU_versions
│               └── public_profiles
│                   ├── mOTUs.profiles.gz
│                   └── mOTUs.profiles_environments.gz
└── record
    └── motus_db_versioned
        └── 3.1.0

bernt-matthias · 2026-07-13T09:21:43Z

+
+# A toolshed GUID looks like:
+#   toolshed.g2.bx.psu.edu/repos/<owner>/<repo>/<tool>/<version>
+TOOL_ID_PREFIXES = ("toolshed.g2.bx.psu.edu/repos/", "testtoolshed.g2.bx.psu.edu/repos/")


Do we need the testtoolshed?

bernt-matthias · 2026-07-13T09:28:37Z

This looks great. Was wondering if one should get the data_manager from the toolshed and validate data table names and maybe even parameter names / values. But I guess its fine to see this failing when we actually run it.

bernt-matthias · 2026-07-13T09:36:04Z

+CHAIN_WIRING = {
+    ("samestr_db", "metaphlan_database_versioned"): {
+        "tool_state": {"db_source": {"db_type": "metaphlan"}},
+        "connect_param": "db_source|database",
+    },
+    ("samestr_db", "motus_db_versioned"): {
+        "tool_state": {"db_source": {"db_type": "motus"}},
+        "connect_param": "db_source|motus_db",
+    },
+}


Wondering if it would be more accessible to contributors if this would be configured as yaml. Do we want to support that wiring can change over time? Then we could have this configuration in the yaml file.

bernt-matthias · 2026-07-13T09:43:17Z

Should we have a cron job running here that checks if all data in data_managers has made it in the reference Galaxy's data tables? As a sanity check that tells us that all components of the pipeline are running.

bernt-matthias · 2026-07-13T09:48:19Z

+## Contributing versioned reference data (workflow bundles)
+
+Beyond the genome-indexing pipeline above (`genomes.yml` + `data_managers.yml`),
+the IDC supports **versioned reference databases** built by data managers — e.g.


That means that the DM needs to support some versioning? Could we deal with unversioned reference data like BLAST NR databases by using the download date?

bernt-matthias · 2026-07-13T09:51:17Z

Could you add a config that would use the motus reference data, i.e. use the non-default branch of the conditional. Then we would have documented also the usage of params with an example.

bernt-matthias · 2026-07-13T09:53:36Z

+doi:
+checksum:


How and where is the checksum computed?

bernt-matthias · 2026-07-13T09:56:38Z

+
+### Avoiding rebuilds of existing data ("does this already exist?")
+
+Reference data that a Galaxy already has is never rebuilt or re-imported. The


The workflow assumes that the path reflects the name of the data table entry that is constructed. Do we make sure at any point that this is true?

bernt-matthias · 2026-07-13T10:02:17Z

At which point of the pipeline is this file used at the moment? Would it be better to use the one from CVMFS?

bernt-matthias · 2026-07-13T10:07:02Z

I do not understand yet when data is added to this file in this pipeline .. and maybe if we really need it, since we could check against a reference index.

bernt-matthias · 2026-07-13T10:09:43Z

This looks great. Added a few comments.

How do you implement the bot at the moment? This is just for demo, or?

For the motus, you get and entry with a date (db_from_2026-07-12T140418Z)? How can we refer to that in samestr?

mvdbeek · 2026-07-13T10:34:46Z

(db_from_2026-07-12T140418Z)

that name is generated via https://github.com/bgruening/galaxytools/blob/master/data_managers/data_manager_motus_db_downloader/data_manager/data_manager_fetch_motus_db.py#L98. i don't know if this really makes sense, especially since the data is coming forma zenodo snapshot. ping @paulzierep, you've added this in bgruening/galaxytools@f42263c

How can we refer to that in samestr?

dbkey should probably be settable in the data manager so we can set it instead of having to discover it ? fwiw if we do a pipeline build it's just a dataset input so it'll work automatically, but we probably want something stable ? dbkey is just such a weird concept for stuff that's not a single genome or not a genome at all.

mvdbeek added 9 commits July 9, 2026 19:56

Document reference-existing behavior for chained requests

7c176fc

mvdbeek added 2 commits July 11, 2026 10:49

Ignore .DS_Store and ephemeris tool_test_output artifacts

f884229

mvdbeek added 2 commits July 11, 2026 19:58

Pin samestr_db request to data_manager_samestr 1.2025.111+galaxy2

b1967c5

galaxy2 (tools-iuc#8172) reconstructs MetaPhlAn markers via bowtie2-inspect; installed on test.galaxyproject.org via usegalaxy-tools#1596.

mvdbeek added 2 commits July 11, 2026 20:10

mvdbeek mentioned this pull request Jul 12, 2026

Add SameStr workflow galaxyproject/iwc#1257

Open

12 tasks

bernt-matthias reviewed Jul 13, 2026

View reviewed changes

mvdbeek added 2 commits July 13, 2026 12:59

Restrict reference-data requests to production Tool Shed

1d9502e

Drop unused reference-data checksum field

26eb2d8


		### Avoiding rebuilds of existing data ("does this already exist?")

		Reference data that a Galaxy already has is never rebuilt or re-imported. The

		doi:
		checksum:

Uh oh!

Conversation

mvdbeek commented Jul 9, 2026

What

The three stages

Highlights

Verified live on test.galaxyproject.org

Not yet exercised (need environments outside CI)

Uh oh!

mvdbeek commented Jul 9, 2026

Uh oh!

jmchilton commented Jul 10, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 11, 2026

Uh oh!

mvdbeek commented Jul 12, 2026

Uh oh!

mvdbeek commented Jul 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bernt-matthias commented Jul 13, 2026

Uh oh!

mvdbeek commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants