Curated tiff directories in yaml file should be relative to yaml file

[matham]

Description

What is this PR

• Bug fix
• Addition of a new feature
• Other

Why is this PR needed?

Currently, after exporting the curated cuboids, if you move the exported data elsewhere (such as another computer) and then try to use the yaml for training, it'll error out because the paths in the yaml file for the tiff directories are absolute. This is annoying because you have to manually adjust the paths in all the yaml files every time you move them.

Here's a test I added to the branch, which when run on main results in the following error, which similarly happens if you try to use a yaml file after it has been moved from the location where data was originally exported:

tests\napari\test_curation.py F                                                                                                                                                                            [100%]

=================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________________________ test_curated_data_can_be_loaded ________________________________________________________________________________________

valid_curation_widget = <cellfinder.napari.curation.CurationWidget object at 0x00000289DB0F7B10>, tmp_path = WindowsPath('C:/msys64/tmp/pytest-of-CPLab/pytest-30/test_curated_data_can_be_loade0')
qtbot = <pytestqt.qtbot.QtBot object at 0x000002898AE3AA50>

    def test_curated_data_can_be_loaded(valid_curation_widget, tmp_path, qtbot):
        """
        Tests that the curated data can be loaded by training code. Both in its
        original exported location and after it has been moved elsewhere.
        """
        widget = valid_curation_widget
        widget.output_directory = tmp_path

        with patch("cellfinder.napari.curation.show_info") as show_info:
            widget.save_training_data(prompt_for_directory=False, block=False)

            assert not (tmp_path / "training.yaml").exists()

            qtbot.waitUntil(lambda: show_info.call_count, timeout=20000)

        # check that we can load the training data where it currently exists
        training_yaml = tmp_path / "training.yaml"
        src_tiffs = {f.resolve() for f in tmp_path.glob("**/*.tif")}

        assert training_yaml.exists()
        assert len(src_tiffs) == 4

        # check read tiffs are same
        yaml_contents = parse_yaml([str(training_yaml)])
        tiff_lists = get_tiff_files(yaml_contents)
        loaded_tiffs = {
            f for group in tiff_lists for im in group for f in im.img_files
        }
        for f in loaded_tiffs:
            assert Path(f).exists()

        assert {Path(f).resolve() for f in loaded_tiffs} == src_tiffs

        # now move the data and check that we can still load it
        items = os.listdir(tmp_path)
        new_place = tmp_path / "new_place"
        new_place.mkdir()
        for item in items:
            shutil.move(tmp_path / item, new_place)

        training_yaml = new_place / "training.yaml"
        moved_tiffs = {f.resolve() for f in new_place.glob("**/*.tif")}

        assert training_yaml.exists()
        assert {f.name for f in moved_tiffs} == {f.name for f in src_tiffs}

        # check read tiffs are same
        yaml_contents = parse_yaml([str(training_yaml)])
>       tiff_lists = get_tiff_files(yaml_contents)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests\napari\test_curation.py:353:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cellfinder\core\train\train_yaml.py:354: in get_tiff_files
    TiffDir(d["cube_dir"], channels, channels_metadata, d["type"])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cellfinder.core.tools.tiff.TiffDir object at 0x00000289DB4FC190>, tiff_dir = 'C:\\msys64\\tmp\\pytest-of-CPLab\\pytest-30\\test_curated_data_can_be_loade0\\cells', channels = [0, 1]
channels_metadata = [{'mean': 262.34832005565846, 'std': 205.0074719217545}, {'mean': 214.4578158959736, 'std': 145.9514501306045}], label = 'cell'

    def __init__(
        self,
        tiff_dir: str,
        channels: list[int],
        channels_metadata: list[dict],
        label: str | None = None,
    ):
        super(TiffDir, self).__init__(
            [
                join(tiff_dir, f)
>               for f in listdir(tiff_dir)
                         ^^^^^^^^^^^^^^^^^
                if f.lower().endswith("ch" + str(channels[0]) + ".tif")
            ],
            channels,
            channels_metadata,
            label,
        )
E       FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\msys64\\tmp\\pytest-of-CPLab\\pytest-30\\test_curated_data_can_be_loade0\\cells'

cellfinder\core\tools\tiff.py:74: FileNotFoundError

What does this PR do?

1. This branch saves the tiff directories in the yaml as being relative to the training.yaml file. I'm assuming that when people move training data they move it as a whole so the yaml file will always have the same relationship to the directories as when originally exported.
2. I also added a key to the yaml file pointing to the original data filename from which the cubes came from, if that is available. Napari keeps track of the filename from which an image layer comes from, if it is available. The key is not used anywhere, but is helpful if you want to know where the data came from.
3. I also adjusted test_async_save_done_only_after_cubes_written, because we should be checking for the done message to appear, not for the yaml file to be written because we can't assume training.yaml is the last file that gets saved.

References

None.

How has this PR been tested?

Added tests.

Is this a breaking change?

No.

Does this PR require an update to the documentation?

No.

Checklist:

• The code has been tested locally
• Tests have been added to cover all new functionality (unit & integration)
• The documentation has been updated to reflect any changes
• The code has been formatted with pre-commit