Accelerate scverse/scanpy steps with rapids-singlecell

[Zethson]

Description of feature

Heyy,

the pipeline currently uses scanpy for several steps and might also use other scverse tools like decoupler or pertpy in the future for some steps like gene set enrichment or sample comparisons.

I suggest that we integrate github.com/scverse/rapids-singlecell to optionally GPU accelerate these steps. I saw that some GPU infrastructure already exists for scvi-tools related steps that I'm hoping to reuse. There's no RSC module yet so I'd start with that and then start integrating it into the pipeline.

Any thoughts, concerns, or helpful pointers?

CC @gburnett-nvidia @DCGenomics

[nictru]

Some time ago, there was already limited support for rapids in the pipeline. But some users reported strange results with rapids. I think it was related to an issue concerning neighbors or UMAP in rapids, and I think it has been resolved since, but I don't remember what exact issue it was. Anyway, at the time I removed rapids, to prevent users from getting flawed results (yes there might have been other ways - e.g. disable by default and add a warning), but somehow I opted for removing it entirely.

The old implementation actually did not have rapids-specific modules, but rather augmented the scanpy modules with rapids. You can see what I mean in #185. This appeared natural to me, as it prevents implementing the wrapper logic twice, but also it's not 100% nf-core-style. Not sure what the best strategy for this effort will be

[Zethson]

There's additional discussion here: https://nfcore.slack.com/archives/C07PPQR415W/p1778157151146789

I can vouch that rapids-singlecell does things properly and you don't need to worry about the correctness of the implementation or any other quirks. It's very mature.

[Zethson]

You had the following questions @nictru

1. How is the scanpy/RSC handling implemented? How can we keep the same featureset available for both CPU and GPU users with a minimal amount of redundant code?
2. Where are the supermodule boundaries?

I am hopeful that this will be easy in the future. Severin Dicks @Intron7 is currently implementing scverse-backends which will allow us to register a RSC backend into scanpy. The goal of this is to enable "write once, run anywhere where a backend is registered" pattern. In other words, if RSC is installed and at runtime a backend is set like:

sc.settings.backend = "cuda"

then the existing scanpy code will automatically dispatch to RSC and therefore the GPU. No additional code changes are required. Therefore, I think we'll just need some logic to select the right container in the processes (pure scanpy or scanpy + RSC) based on a new backend parameter (which we can also just set/pass based on --gpu). The Python scripts (or modules) should then just set the backend accordingly.
The rest should then pretty much magically work minus some weird quirks that we'll uncover later.
RSC and scanpy do not share all parameters but almost all parameters. scverse-backends takes care of this problem though so in almost all cases, we shouldn't have to worry about this.

scverse-backends is still in development so I think we'll need to push this out a few more weeks but I'll get back to this.

[nictru]

Sounds like an amazing solution, looking forward to this!