ENH: Consolidate remote modules into hierarchical category repos

[hjmjohnson]

Consolidate remote modules into hierarchical category repos

Reduce ~60 remote module repos to ~5 domain-based category repos plus ~13 independent repos. New itk_fetch_module_group() CMake function enables hierarchical sub-groups. Proof of concept (ITKRemoteAnalysis, 8 modules, 77/77 tests pass) in PR #6061. Supersedes #5134.

Problem

ITK has ~60 remote module repositories, each with its own git repo, CI, Python packaging, and boilerplate:

• ~177 lines of nearly identical config per module (~10,600 lines total duplicated boilerplate)
• CI workflow versions fragmented across 10 different ITKRemoteModuleBuildTestPackageAction tags
• A single CI infrastructure update requires 15+ coordinated PRs
• Every new filter (like SSIM in ENH: Add StructuralSimilarity Beta module (SSIM image filter) — stacked on #6085 #6034) requires creating an entirely new repository

Discussed in #6034 where @blowekamp, @dzenanz, and @hjmjohnson agreed consolidation is needed during the 6.0 beta phase.

Proposal

Group modules into 5 category repos by domain (per @thewtex and @dzenanz: no "Kitware" category, no standalone "Microscopy" — group by domain):

Category Repo	Candidate Modules	Count
ITKRemoteIO	AnalyzeObjectLabelMap, IOFDF, IOMeshMZ3, IOMeshSTL, IOMeshSWC, IOScanco, IOTransformDCMTK, MGHIO, IOOpenSlide	9
ITKRemoteFiltering	AnisotropicDiffusionLBR, FastBilateral, GenericLabelInterpolator, HigherOrderAccurateGradient, LabelErodeDilate, MorphologicalContourInterpolation, MultipleImageIterator, ParabolicMorphology, PhaseSymmetry, RLEImage, SimpleITKFilters, SmoothingRecursiveYvvGaussianFilter, SplitComponents	13
ITKRemoteMesh	BSplineGradient, Cuberille, FPFH, MeshNoise, MeshToPolyData, PrincipalComponentsAnalysis, SubdivisionQuadEdgeMeshFilter	7
ITKRemoteAnalysis	BoneEnhancement, BoneMorphometry, HASI, IsotropicWavelets, PerformanceBenchmarking, RANSAC, TextureFeatures, Thickness3D, (+ StructuralSimilarity from #6034)	8-9
ITKRemoteRegistration	FixedPointInverseDisplacementField, GrowCut, LesionSizingToolkit, MinimalPathExtraction, Montage, PolarTransform, SkullStrip, Strain, TwoProjectionRegistration, VariationalRegistration	10

Stay independent (large, complex, or external dependencies):

• RTK, CudaCommon (RTKConsortium)
• TubeTK (large, many examples/applications — @dzenanz)
• Ultrasound (complex multi-dependency, serves as blueprint — @dzenanz)
• WebAssemblyInterface (wasm toolchain)
• AdaptiveDenoising (ntustison — external maintainer)
• Cleaver (SCIInstitute — external dependencies, defer — @dzenanz)
• SCIFIO, Shape, TractographyTRX (external orgs)
• SphinxExamples (documentation)
• VkFFTBackend (GPU backend)
• TotalVariation, IOOMEZarrNGFF (external dependencies, defer — @dzenanz)
• BioCell (legacy, may deprecate)

CMake mechanism

New ITKGroup_Remote_<Category> sub-group options, all OFF by default:

ITKGroup_Remote = OFF              # master switch (existing)
  ITKGroup_Remote_IO = OFF         # NEW: fetches ITKRemoteIO repo when ON
  ITKGroup_Remote_Filtering = OFF
  ITKGroup_Remote_Mesh = OFF
  ITKGroup_Remote_Analysis = OFF
  ITKGroup_Remote_Registration = OFF

• Sub-group OFF: repo not cloned, modules not available
• Sub-group ON: repo cloned, all contained modules individually toggleable via Module_<name>
• ITKGroup_Remote=ON: all sub-groups enabled (backward compat)
• Modules with EXCLUDE_FROM_DEFAULT need explicit Module_<name>=ON even when sub-group is ON

~160 lines of CMake changes across 3 files. New itk_fetch_module_group() function. Existing itk_fetch_module() untouched (independent repos keep working).

Python packaging

Per-category wheels (itk-remote-io, itk-remote-filtering, etc.) replace individual itk-<name> packages. Old packages get a deprecation release pointing to the new category package.

New module workflow

New filters added as a subdirectory PR to the appropriate category repo. No new git repo, no new CI, no new PyPI package setup.

Example: SSIM (PR #6034) goes into ITKRemoteAnalysis as StructuralSimilarity/ subdirectory.

Implementation phases

• Phase 0: Community discussion — Discourse #7753, this issue
• Phase 1: CMake infrastructure — itk_fetch_module_group() in WIP: Combine many remote to few remote module groups #6061
• Phase 2: Proof of concept — ITKRemoteAnalysis (8 modules, 1662 commits, 77/77 tests pass)
• Phase 3: Push to GitHub, CI validation — hjmjohnson/ITKRemoteAnalysis, CI running in WIP: Combine many remote to few remote module groups #6061
• Phase 4: Transfer to ISC org once CI green
• Phase 5: Roll out remaining categories (IO, Filtering, Mesh, Registration)

Decisions

Question	Decision	Source
Category groupings	5 domain categories (IO, Filtering, Mesh, Analysis, Registration)	@thewtex, @dzenanz
"Kitware" category	No — merge by domain	@thewtex, @dzenanz
Microscopy category	No — IOOpenSlide goes to IO	@dzenanz
KitwareMedical modules	Merge into domain categories	@thewtex, @dzenanz
TubeTK	Stay independent (large, many examples)	@dzenanz
Ultrasound	Stay independent, serves as blueprint	@dzenanz
External-dep modules	Defer (TotalVariation, Cleaver, IOOMEZarrNGFF)	@dzenanz
EXCLUDE_FROM_DEFAULT	Respected: need explicit Module_<name>=ON	Confirmed by prototype
Backward compat	ITKGroup_Remote=ON enables all sub-groups	Confirmed by prototype
Supersedes	#5134	@dzenanz
AdaptiveDenoising (ntustison)	TBD — discuss with maintainer
PyPI: individual to category wheels	TBD
BioCell	TBD — may deprecate

References

• ENH: Add StructuralSimilarity Beta module (SSIM image filter) — stacked on #6085 #6034 — StructuralSimilarityImageFilter (triggered this discussion)
• ENH: Integrate FastBilateralImageFilter from remote module #5134 — Earlier consolidation discussion (superseded)
• WIP: Combine many remote to few remote module groups #6061 — WIP PR with itk_fetch_module_group() and Analysis proof of concept
• Discourse: https://discourse.itk.org/t/7753
• Prototype repo: https://github.com/hjmjohnson/ITKRemoteAnalysis

[thewtex]

I don't think "Kitware" is a topic category and the modules should be added to the appropriate topics.

[dzenanz]

This consolidation sure would make any future refactoring easier. It would also supersede #5134.

I don't think "Kitware" is a topic category and the modules should be added to the appropriate topics.

I agree with this.

[dzenanz]

IOOpenSlide should be part of IO group, not a group by itself.

TubeTK is quite big, with a lot of examples and applications. I would not group it with anything else.

The main general problem with grouping is that some modules have external dependencies (e.g. TotalVariation, IOOMEZarrNGFF) which are hard to update, and require special build-related code. Maybe we should leave those alone for now?

[dzenanz]

Cleaver has external dependencies. As such, I would leave it alone, at least for the initial refactoring/grouping/combining.

[dzenanz]

Ultrasound is already quite a complicated module. It may serve as a blueprint for these combined thematic modules.

[hjmjohnson]

Proof of concept complete

Draft PR #6061 implements the CMake infrastructure and demonstrates it with an ITKRemoteAnalysis category repo containing 8 modules. All 77 tests pass across all 8 modules, with 1661 commits and full git blame preserved.

Net result: −221 lines of CMake (106 new lines for itk_fetch_module_group(), 395 removed from 7 individual .remote.cmake files).

What works

• itk_fetch_module_group(Analysis ...) clones a multi-module repo into Modules/Remote/Analysis/
• 4-level glob in ITKModuleEnablement.cmake discovers modules at Remote/Analysis/TextureFeatures/itk-module.cmake
• ITKGroup_Remote_Analysis=ON requests all non-excluded modules; ITKGroup_Remote=ON propagates to all sub-groups
• Individual Module_<name>=ON/OFF toggles work within an enabled sub-group
• Git history fully preserved (1661 commits, blame works, old-to-new hash maps generated)
• 77/77 tests pass across all 8 modules

CMake changes required

File	Change	Lines
CMake/ITKModuleEnablement.cmake	4-level glob	+1
CMake/ITKModuleRemote.cmake	itk_fetch_module_group()	+106
CMake/ITKGroups.cmake	Sub-group request propagation	+50
Modules/Remote/ITKRemoteAnalysis.remote.cmake	Category group file	+17
7 individual .remote.cmake files	Removed	-395
Net		-221 lines

Decisions confirmed by the prototype

Question	Answer
Module enable without sub-group	Works: Module_X=ON requires sub-group enabled first (module won't be discovered otherwise)
EXCLUDE_FROM_DEFAULT modules	Respected: sub-group ON makes them available but not auto-built; need explicit Module_<name>=ON
Backward compat	ITKGroup_Remote=ON enables all sub-groups
Independent modules	Unaffected: itk_fetch_module() untouched

ITKRemoteAnalysis repo contents

• 8 modules: TextureFeatures, BoneMorphometry, BoneEnhancement, Thickness3D, IsotropicWavelets, RANSAC, PerformanceBenchmarking, StructuralSimilarity
• 1661 commits with preserved authorship and blame
• commit-maps/ directory with old-to-new hash mappings (1731 entries)
• COMPLIANCE.md per module (extracted from .remote.cmake grading reports)
• Reusable consolidation script: itk-consolidate-remotes skill

Next steps

1. Create InsightSoftwareConsortium/ITKRemoteAnalysis repo on GitHub, push
2. Update .remote.cmake GIT_REPOSITORY to point at GitHub URL
3. Get Discourse feedback on category groupings
4. Repeat for IO, Filtering, Mesh, Registration, Segmentation, Microscopy

[SimonRit]

Sounds like a good initiative. I don't see where RTK would fit (and it's complicated like Ultrasound), I agree it can be kept independent.

ITKCudaCommon however is small, very similar to (and based on) GPUCommon. I personally would be in favor of integrating it to the Core of ITK, it would be useful for machine learning applications to avoid CPU <-> GPU transfers since it supports __cuda_array_interface__. It might not be ready for this though, we're happy to do the required developments to this end.

[dzenanz]

While OpenCL is not dead, its further development has ceased. CUDA has outlived it. I would be fine with adding CudaCommon to the main repository, with implied expectation that you would continue being its primary maintainer.