Feat/rust metadata#88
Open
mathleur wants to merge 61 commits into
Open
Conversation
* fix structural hash * WIP: add algo for node union * WIP: implement union * WIP: node union method * WIP: overall union method * WIP: fix tests * WIP: add hash computation when we start merging * WIP: add hash computation with children coordinates * WIP: add hash computation test for semi-similar qubes * WIP: new try merge function * WIP: second implementation of node_union * WIP: start internal_set_op * WIP: implement internal_set_operations * WIP: fix recursion of the union operation * WIP: add children to intermediate intersection nodes * WIP: working merge but with duplicates still * fix bug wrong values appended * start implementing merge * clean up a bit * WIP: add merging of coords * finish compression * add compression test * semi-working compression * fixed compression * fix compression test * fix compression to remove duplicates as well, and fix union to remove duplicates * clean up * add CI and QA * add QA format * ignore test scripts * clean up * format rust code * harmonise structural hashes in Qube and NodeRef * add pre-commit * clean up * fix build * start implementing cads rust adaptor with merge * implement append_datacube * compile * add tests * try cds constraints reading * failing union example * WIP: fix union to merge branches only in other well, but duplicates remain * WIP: see if node already exists when creating a new one * WIP: try to remove the same node idx, which seems to have duplicates in the qube * fix merge/union * WIP: try to fix merging of different dimension orders * change when CI is run * fix union * fix bug and make dss eg work * change when to run CI * implement union many * cads rust adaptor * to datacubes method (#64) * remove mars list file * try to separate floats from general coords * separate float from coords * clean up * add all_unique_dim_coords method and tests * clean up * doc code * doc code * rename funcs according to review * clear other qube in union * add test for mars_list * fix tests * fix mod test mars list adapter
* fix structural hash * WIP: add algo for node union * WIP: implement union * WIP: node union method * WIP: overall union method * WIP: fix tests * WIP: add hash computation when we start merging * WIP: add hash computation with children coordinates * WIP: add hash computation test for semi-similar qubes * WIP: new try merge function * WIP: second implementation of node_union * WIP: start internal_set_op * WIP: implement internal_set_operations * WIP: fix recursion of the union operation * WIP: add children to intermediate intersection nodes * WIP: working merge but with duplicates still * fix bug wrong values appended * start implementing merge * clean up a bit * WIP: add merging of coords * finish compression * add compression test * semi-working compression * fixed compression * fix compression test * fix compression to remove duplicates as well, and fix union to remove duplicates * clean up * add CI and QA * add QA format * ignore test scripts * clean up * format rust code * harmonise structural hashes in Qube and NodeRef * add pre-commit * clean up * fix build * start implementing cads rust adaptor with merge * implement append_datacube * compile * add tests * try cds constraints reading * failing union example * WIP: fix union to merge branches only in other well, but duplicates remain * WIP: see if node already exists when creating a new one * WIP: try to remove the same node idx, which seems to have duplicates in the qube * fix merge/union * WIP: try to fix merging of different dimension orders * change when CI is run * fix union * fix bug and make dss eg work * change when to run CI * implement union many * cads rust adaptor * to datacubes method (#64) * remove mars list file * try to separate floats from general coords * separate float from coords * clean up * add all_unique_dim_coords method and tests * clean up * doc code * doc code * start building python interface * add to and from ascii methods * add python bindings to qubed-meteo * WIP: python interface to mars_list adapter * WIP: python interface to mars_list adapter * fix python interface between rust modules * bindings for to_datacubes * refactor: separate rust crates from python wrapper crates * chore: clean rust warnings in core and meteo adapter * test: move python binding tests into wrapper crates * ci: unify checks and wheel builds with rust cache * ci: remove --locked from workspace test step * chore: remove legacy embedded python modules and stale ignore paths * ci: run python binding tests and optimize PyQube __len__ * fix: make union_many validate input types before mutating * clean up * fix tests --------- Co-authored-by: mathleur <mathilde.leuridan@ecmwf.int> Co-authored-by: Mathilde Leuridan <90444327+mathleur@users.noreply.github.com>
* fix structural hash * WIP: add algo for node union * WIP: implement union * WIP: node union method * WIP: overall union method * WIP: fix tests * WIP: add hash computation when we start merging * WIP: add hash computation with children coordinates * WIP: add hash computation test for semi-similar qubes * WIP: new try merge function * WIP: second implementation of node_union * WIP: start internal_set_op * WIP: implement internal_set_operations * WIP: fix recursion of the union operation * WIP: add children to intermediate intersection nodes * WIP: working merge but with duplicates still * fix bug wrong values appended * start implementing merge * clean up a bit * WIP: add merging of coords * finish compression * add compression test * semi-working compression * fixed compression * fix compression test * fix compression to remove duplicates as well, and fix union to remove duplicates * clean up * add CI and QA * add QA format * ignore test scripts * clean up * format rust code * harmonise structural hashes in Qube and NodeRef * add pre-commit * clean up * fix build * start implementing cads rust adaptor with merge * implement append_datacube * compile * add tests * try cds constraints reading * failing union example * WIP: fix union to merge branches only in other well, but duplicates remain * WIP: see if node already exists when creating a new one * WIP: try to remove the same node idx, which seems to have duplicates in the qube * fix merge/union * WIP: try to fix merging of different dimension orders * change when CI is run * fix union * fix bug and make dss eg work * change when to run CI * implement union many * cads rust adaptor * to datacubes method (#64) * remove mars list file * try to separate floats from general coords * separate float from coords * clean up * add all_unique_dim_coords method and tests * clean up * doc code * doc code * rename funcs according to review * clear other qube in union * add test for mars_list * fix tests * fix mod test mars list adapter * add to_dss_constraints encoder * add arena json serde * add fdb adapter * just parse all coords as string for now in fdb adapter * merge python interface branch * add python bindings for new serde and fdb adapter * fix CI * add docs * keep leading 0 when parsing strings * try rsfdb from github * try path * add fdb list adapter eg * add more complete docs * remove old docs * TODOs * TODOs * get_or_create_child method * keep native types in json serialisation * update fdb adapter to go from a json map * Stop tracking generated mdBook output; add to .gitignore * add docs action * patch rsfdb * patch rsfdb * run tests locally
* add basic fdb reader * float coordinates * add datetime coord * fix fdb reader on db * add coord types to json serde * fix when empty child * add qube examples * fix qa * add version to json * ignore files * rename vars Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Change datetime format to ISO-like without timezone * format * fix compile errors * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * add patch for private repos * test json version --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* add python bindings * drop and squeeze * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * select multiple vals * add python bindings for squeeze and drop * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add Python CI, update API docs, and fix Rust append behavior * Update qubed/src/merge.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update qubed/tests/test_union.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * cargo fmt --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix select * fix select * Update qubed/src/select.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update qubed/src/select.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * declare higher level python project (#83) * Improve error handling in node removal * cargo fmt --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contributor Declaration
By opening this pull request, I affirm the following: