Skip to content

Remote nextflow hooks#54

Merged
SkepticRaven merged 21 commits into
mainfrom
remote-nextflow-hooks
Aug 26, 2025
Merged

Remote nextflow hooks#54
SkepticRaven merged 21 commits into
mainfrom
remote-nextflow-hooks

Conversation

@SkepticRaven
Copy link
Copy Markdown
Contributor

@SkepticRaven SkepticRaven commented Jul 23, 2025

Adds:

  • Filtering of inputs to skip already processed data
  • rclone cloud remote hook (dropbox)
  • globus endpoint remote hook (JAX T0/T1/T2)
  • Exposes these new options to all workflow inputs
  • Some nf-tests for new remote processes

SkepticRaven added 17 commits July 8, 2025 08:58
@SkepticRaven
Changing rclone commands to properly use nextflow config vars
87a8b9c
@SkepticRaven
Adding globus-specific modules
3816540
@SkepticRaven
Adding filters for files
24a04c2 
Some syntax correction
@SkepticRaven
Starting the remote workflow logic changes.
6a85365
@SkepticRaven
Adjusting var names to read more clearly
ee58960
@SkepticRaven
Multimouse workflow used wrong property
a5da72d
@SkepticRaven
Reworking filtering structure and prepare data workflows to be more u…
5fc673d 
…niform.

Also attempting to restructure all to operate on file lists for retrieval.
@SkepticRaven
First version of failing test, but can get nf-tests running
42e522a
@SkepticRaven
Getting dropbox retrieval operating.
3e3f8d1
@SkepticRaven
Adjusting other rclone-based funcs to use adjusted variable
389ac85
@SkepticRaven
Restricting rclone to 1 transfer at a time
24a7ed5
@SkepticRaven
Adding globus remote test
510e74a
@SkepticRaven
Updating resource paths due to failing hard drive. Also making them s…
2841f69 
…impler to update in the future.
@SkepticRaven
Adding proper config to test globus transfers
a8fcd62
@SkepticRaven
Getting the globus test to actually ping globus. Currently goes into …
531049e 
…an infinite wait loop if files don't exist.
@SkepticRaven
Adding a catch for infinite task wait loop
d5582b8
@SkepticRaven
Adding an invalid file test for globus
a4e095a
@SkepticRaven SkepticRaven requested a review from bergsalex July 23, 2025 14:35
bergsalex
bergsalex approved these changes Jul 23, 2025
View reviewed changes
Comment thread support_code/globus-cli.def
Comment thread nextflow/tests/remote/globus_test.nf.test
Copy link
Copy Markdown
Collaborator

@bergsalex bergsalex Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 for nextflow tests!

Comment thread nextflow/tests/remote/rclone_test.nf.test
Comment thread nextflow/modules/utils.nf
Comment thread nextflow/modules/remote_io.nf
Comment thread main.nf Outdated
Comment thread nextflow/modules/remote_io.nf Outdated
Comment thread nextflow/modules/remote_io.nf Outdated
@SkepticRaven SkepticRaven merged commit e2ff6ef into main Aug 26, 2025
3 checks passed
@SkepticRaven SkepticRaven deleted the remote-nextflow-hooks branch August 26, 2025 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bergsalex bergsalex bergsalex approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SkepticRaven @bergsalex