Merge proposed v1.3 changes into main - #136
Open
mgiammar wants to merge 11 commits into
Open
Conversation
* feat: inspect peaks, fourier dropout, frame correlations, and distributed match-template Squashed rebuild of the mdg_inspect_peaks PR (#124) on top of development_v1.3 to remove duplicated upstream history. Tree is identical to prior branch tip 84d1ce8. * feat: pull out mean/std normalization into z-score from refinement core * feat: implement peak inspection result saving/loading and refactor related components * refactor: streamline peak inspection methods * refactor: major restructuring of the per-frame inspect peaks manager * Enhance per-frame inspection and result handling - Introduced per-frame inspection capabilities in `inspect_peaks_result.py`, allowing independent scoring of each movie frame. - Updated `InspectionResult` to include a `frame_index` for tracking frame-specific results. - Modified `save_inspection_result` to handle per-frame data, including new axes for frame-specific results. - Refactored `FrameInspectionManager` to streamline per-frame processing and result collection, including a new method for saving results to `.npz` files. - Added tests for saving and loading inspection results, ensuring correct handling of both spatial and per-frame data formats. - Adjusted `PreprocessingFilters` to conditionally apply random Fourier dropout during filter combination. - Removed unused methods related to cross-correlation peak reduction and CSV output from `FrameInspectionManager`. * fix: filter warnings raised internally by mrcfile
…ation) (#126) * feat: only update statistics on valid correlation region Under the standard benchmark setup, this should be excluding ~25% of pixels during the conditional update based on MIP value. Benchmarking the code did not show a significant performance change, however. Statistics updates though are not consuming the majority of computation time. * feat: add correlation table to multi-GPU match template * fix: update correlation table for stability * feat: initial integration with zipFFT * test: update unit tests for zipFFT consistency * fix: minor bug fixes for zipfft match template * fix: changed to mitigate errors on feature integration * feat: integrate all stats updates together and simplify backend logic * lint: work towards passing linter * ci(dependabot): bump softprops/action-gh-release from 2 to 3 (#122) Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2 to 3. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](softprops/action-gh-release@v2...v3) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bugfix to send correlation mean/var to proper devices on multi-GPU refine template (#125) * feat: add CorrelationTable model and integrate into match template results * test: add basic test coverage for correlation table * fix: add tensordict dependency and update pytest filterwarnings * fix: import errors after merge conflict resolution * feat: add hdf5-backed version of MatchTemplateResult * feat: integrate into match template manager * test: add unit tests covering new result functionality * feat: (untested) initial work on hdf5 backed result formats * fix: failing CI from old file path attributes * (fix) resolve edge-case in unit test data setup * feat: add corr table for distributed backend, address review comments * feat: add option to skip re-download of example data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Josh Dickerson <jdickerson@mrc-lmb.cam.ac.uk>
* ci(dependabot): bump softprops/action-gh-release from 2 to 3 (#122) Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2 to 3. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](softprops/action-gh-release@v2...v3) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bugfix to send correlation mean/var to proper devices on multi-GPU refine template (#125) * (docs) migrate to zensical as build backend * (docs) API build for zensical * (docs) update paper ref and citation to published manuscript * (CI/CD) update deployment pipeline for documentation with zensical * ci(dependabot): bump codecov/codecov-action from 6 to 7 (#128) Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 6 to 7. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v6...v7) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * (fix): CI/CD disable pre-release install dependencies * Prior inspect peaks work (#124) * feat: inspect peaks, fourier dropout, frame correlations, and distributed match-template Squashed rebuild of the mdg_inspect_peaks PR (#124) on top of development_v1.3 to remove duplicated upstream history. Tree is identical to prior branch tip 84d1ce8. * feat: pull out mean/std normalization into z-score from refinement core * feat: implement peak inspection result saving/loading and refactor related components * refactor: streamline peak inspection methods * refactor: major restructuring of the per-frame inspect peaks manager * Enhance per-frame inspection and result handling - Introduced per-frame inspection capabilities in `inspect_peaks_result.py`, allowing independent scoring of each movie frame. - Updated `InspectionResult` to include a `frame_index` for tracking frame-specific results. - Modified `save_inspection_result` to handle per-frame data, including new axes for frame-specific results. - Refactored `FrameInspectionManager` to streamline per-frame processing and result collection, including a new method for saving results to `.npz` files. - Added tests for saving and loading inspection results, ensuring correct handling of both spatial and per-frame data formats. - Adjusted `PreprocessingFilters` to conditionally apply random Fourier dropout during filter combination. - Removed unused methods related to cross-correlation peak reduction and CSV output from `FrameInspectionManager`. * fix: filter warnings raised internally by mrcfile * Tracking possible detections via correlation table (and zipfft integration) (#126) * feat: only update statistics on valid correlation region Under the standard benchmark setup, this should be excluding ~25% of pixels during the conditional update based on MIP value. Benchmarking the code did not show a significant performance change, however. Statistics updates though are not consuming the majority of computation time. * feat: add correlation table to multi-GPU match template * fix: update correlation table for stability * feat: initial integration with zipFFT * test: update unit tests for zipFFT consistency * fix: minor bug fixes for zipfft match template * fix: changed to mitigate errors on feature integration * feat: integrate all stats updates together and simplify backend logic * lint: work towards passing linter * ci(dependabot): bump softprops/action-gh-release from 2 to 3 (#122) Bumps [softprops/action-gh-release](https://github.com/softprops/action-gh-release) from 2 to 3. - [Release notes](https://github.com/softprops/action-gh-release/releases) - [Changelog](https://github.com/softprops/action-gh-release/blob/master/CHANGELOG.md) - [Commits](softprops/action-gh-release@v2...v3) --- updated-dependencies: - dependency-name: softprops/action-gh-release dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bugfix to send correlation mean/var to proper devices on multi-GPU refine template (#125) * feat: add CorrelationTable model and integrate into match template results * test: add basic test coverage for correlation table * fix: add tensordict dependency and update pytest filterwarnings * fix: import errors after merge conflict resolution * feat: add hdf5-backed version of MatchTemplateResult * feat: integrate into match template manager * test: add unit tests covering new result functionality * feat: (untested) initial work on hdf5 backed result formats * fix: failing CI from old file path attributes * (fix) resolve edge-case in unit test data setup * feat: add corr table for distributed backend, address review comments * feat: add option to skip re-download of example data --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Josh Dickerson <jdickerson@mrc-lmb.cam.ac.uk> * (docs) add updates to data format sections * (docs) add information on zipFFT backend * (docs) restrict documentation dependencies * (docs) add documentation on inspect peaks/frames * (docs) remove python 3.10 as supported version * (lint) fix pylint errors towards allowing CI/CD --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Josh Dickerson <jdickerson@mrc-lmb.cam.ac.uk>
mgiammar
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PRs #130 and #133 can be merged after this since this PR includes fixes for failing CI/CD.