Supporting multiple input .tck files

[jhadida]

This PR is an enhancement of the existing code to support multiple input .tck files for commands that currently accept .tck files:

• This is done by introducing a new file type with extension .lst (for "list"), containing one filename per line. These can be used either in place of .tck arguments, or in place of weights files (e.g. option -tck_weights_in).
• The changes are completely transparent AFAICT: the behaviour of the software remains unchanged when .lst files are not involved.

Implementation

The changes mainly consist in a re-write of the class Tractography::Reader, which now holds a std::vector of tracks filenames (and optionally of weights filenames too). The ReaderInterface is preserved: the operator() method implemented in Reader simply reads the data off the currently open file(s), and switches to the next file(s) when reaching EOF.

The main challenge in this implementation is to find a consensus between the headers of the multiple .tck files. The proposed implementation mimicks the strategy currently found in tckedit, namely:

• accumulate properties count and total_count;
• aggregate other properties and comments as long as they are the same (or not present) across headers;
• assign the value "variable" in case of conflict.

In addition, the proposed consensus fails if the datatypes varies across files.

Limitations

The intention behind the changes introduced is to avoid unnecessary merge operations, and to provide a seamless capability to handle multiple tracks sources in input. This does not apply to the output; .lst files will not be recognised as output arguments, and any output will still be written to a single file (as before).

This can be seen as a limitation, espectially when the output is meant to be used as an input to a subsequent command. For example, in the case of tcksift2 followed by tck2connectome, when the output weights are used to compute a structural connectome matrix. In that case, at present, the weights file must be manually split into as many files as present in the .lst file; and a new .lst file must be created, listing the previous split weights files, and fed to the -tck_weights_in option. This is relatively easy to implement in Python, and I would be happy to provide an example if needed.

Alternatively, a split-output behaviour could potentially be implemented within the Writer class, by specifying output names within an .lst file as well (although this might not be relevant for all commands, and implementing a distinction might be cumbersome).

Possible future work

The consensus implemented in tckedit should be replaced with the new implementation in tractography/file_base.

The changes introduced make the implementations of Tractography::Editing:Loader and Tractography::Mapping::Loader somewhat redundant. I think these loader-classes should be merged into a single implementation, in order to assess whether or not they are really needed, given the new capabilities of Tractography::Reader.

[jhadida]

Apparently the syntax test failed; not sure what this entails or how to fix it?

[thijsdhollander]

Apparently the syntax test failed; not sure what this entails or how to fix it?

This is the relevant info:

Checking syntax and memory alignment for Eigen 3.3 compatibility...
################################### core/file/utils.h
std::vector
################################### src/dwi/tractography/file_base.h
struct TrackFileInfo {std::vector
################################### src/dwi/tractography/file_base.cpp
std::vector
################################### src/dwi/tractography/file.h
std::vector
std::vector
std::vector
std::vector
Please check the following syntax requirements:
  - Add MEMALIGN() or NOMEMALIGN macro to any class declarations identified above;
  - Replace all occurrences of std::vector<> with MR::vector<> (or just vector<>);
  - Avoid use of std::make_shared();
  - Replace all instances of std::abs() with MR::abs() (or just abs());
  - Replace all instances of %zu in print() calls with PRI_SIZET.

Also, I'm not sure you were entirely aware, but we add new features first via the dev branch, which at the moment is quite a bit ahead of master. You should be able to edit the pull request to change the base branch to dev, but depending on what's changed in the mean time in dev vs your pull request, there might be some merging to look into.

[jhadida]

@thijsdhollander I created a new PR #1571 to merge with the dev branch.

[thijsdhollander]

You can actually change the base branch of the existing pull request to obtain more or less the same result. No worries though; the new pull request will work just as well. If everything's sitting there now, feel free to close this pull request to avoid confusion.

[thijsdhollander]

...I'll simply already go ahead and close this one; but feel free to re-open if I missed something.