Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
27d6de8
start to incorporate qubed latest
mathleur Feb 24, 2025
a13861a
start to incorporate qubed latest
mathleur Feb 24, 2025
c415f7b
add qubed tests
mathleur Feb 24, 2025
a411cc8
qubed tree slicing
mathleur Feb 25, 2025
422f4c1
add qubed slice operaiton
mathleur Feb 28, 2025
ec4285d
Merge branch 'develop' of github.com:ecmwf/polytope into feature/inte…
mathleur Mar 25, 2025
e46258e
include axes transformations and datacube axes in qubed slicing
mathleur Mar 25, 2025
6251edb
WIP
mathleur Mar 25, 2025
df39ace
WIP
mathleur Mar 26, 2025
ac32f00
make qubed test work
mathleur Mar 26, 2025
836f923
fix bugs
mathleur Apr 3, 2025
a4fd597
clean up
mathleur Apr 10, 2025
b4e52e5
WIP
mathleur Apr 16, 2025
a418337
fix slicing polytopes across several child nodes
mathleur Apr 17, 2025
3afaa2e
start adding grid axes
mathleur Apr 17, 2025
f2ada06
finish adding prototype of grid axes
mathleur Apr 17, 2025
6159237
clean up and start integrating into polytope fe
mathleur Apr 22, 2025
290bc82
integrate qubed more into a backend + slicer
mathleur Apr 22, 2025
21519f5
WIP extract data for qubed datacube
mathleur Apr 22, 2025
636f32a
use new gj iterator for qubed backend WIP
mathleur Apr 23, 2025
8acef9f
make prototype qubed integration work and compare against fdb
mathleur Apr 23, 2025
2e5c080
add the service part of the qubed like the pre-path
mathleur Apr 24, 2025
a8a06c9
clean up
mathleur Apr 25, 2025
1b4305a
start refactoring
mathleur Apr 25, 2025
d3548df
start incorporating transformations inside qubed
mathleur Apr 28, 2025
05948ac
add everything except grid transformations
mathleur Apr 28, 2025
931bc30
add support for cyclic axes on qubed datacube
mathleur Apr 29, 2025
e30f17f
remove unnecessary md5 hash assignment
mathleur Apr 29, 2025
c945f6b
TODO for recursive datacube construction from qubed
mathleur May 21, 2025
816800c
add axes dynamically method
mathleur May 21, 2025
746e101
WIP: try to make datacube construction recursive
mathleur May 21, 2025
e0b39fb
add recursive axis update with qubed
mathleur May 23, 2025
9dbc947
add where to add option/mapper updates once available in qubed
mathleur May 27, 2025
7ec2e42
update gitignore
mathleur Jul 1, 2025
3449db5
clean up
mathleur Jul 1, 2025
f4cbf31
refactor
mathleur Jul 1, 2025
d79f226
WIP refactor
mathleur Jul 1, 2025
dad3175
WIP refactor
mathleur Jul 2, 2025
40276f7
refactor + take necessary info from options to create qubed datacube
mathleur Jul 2, 2025
c3078aa
add axes dynamically only when options change
mathleur Jul 2, 2025
551b586
remove .DS_Store file
mathleur Jul 3, 2025
7d04753
WIP: try to integrate with new qubed develop
mathleur Jul 15, 2025
765cd32
refactor gitignore file
mathleur Jul 16, 2025
6d7f0c0
update gitignore
mathleur Jul 16, 2025
d9a4791
WIP: add right metadata to constructed qube and flatten metadata at g…
mathleur Jul 22, 2025
522d205
Merge branch 'feature/recursive_qubed_datacube_construction' into fea…
mathleur Jul 22, 2025
77dfe9f
Merge pull request #411 from ecmwf/feature/recursive_qubed_w_types_test
mathleur Jul 22, 2025
7d5b281
merge with new qubed develop
mathleur Jul 22, 2025
27d73e6
WIP: collect right metadata in return qube
mathleur Jul 22, 2025
a684046
keep right metadata during qubed slicing
mathleur Jul 23, 2025
036f44d
finish adding metadata for gj and pass it all the way to gj
mathleur Jul 23, 2025
1df4c84
add metadata as gj expects in qubed backend
mathleur Jul 31, 2025
4d21e3f
clean up
mathleur Jul 31, 2025
2fd4306
add result to Qube metadata
mathleur Jul 31, 2025
ee37224
flatten metadata to give to gj
mathleur Jul 31, 2025
394240b
handle when there are no sliced polytopes
mathleur Jul 31, 2025
a07212a
fix bug
mathleur Aug 4, 2025
35d3286
clean up
mathleur Aug 4, 2025
097a674
WIP: clean up
mathleur Aug 4, 2025
96605ea
avoid deepcopies
mathleur Aug 4, 2025
acc74c3
consistent metadata idxs
mathleur Aug 4, 2025
d09d016
optimise metadata indexing
mathleur Aug 4, 2025
29d5c13
fix compression of axes with qube slicer
mathleur Aug 4, 2025
a50cf2f
WIP:optimise
mathleur Aug 5, 2025
003815b
fix polytope-gj interface for qubed backend
mathleur Aug 6, 2025
4221363
think about refactoring gj/qube backend
mathleur Aug 6, 2025
601ade3
merge with develop
mathleur Aug 6, 2025
243d4f9
fix bug
mathleur Aug 7, 2025
2212768
clean up
mathleur Aug 7, 2025
3996212
Merge pull request #413 from ecmwf/feature/qubed_tree_metadata_merged…
mathleur Aug 8, 2025
6c4f306
Merge pull request #414 from ecmwf/feature/optimised_qubed_backend_w_…
mathleur Aug 8, 2025
d9dcb36
add idea
mathleur Aug 12, 2025
41093c8
construct first slicing tier and compress, add start of second tier s…
mathleur Aug 12, 2025
694f3a3
WIP slice grid axes
mathleur Aug 13, 2025
6426e73
add sliced_polys as metadata and use them to continue building rest o…
mathleur Aug 13, 2025
b53edd7
Merge branch 'feature/qubed_tree_w_gj_metadata' of github.com:ecmwf/p…
mathleur Aug 13, 2025
d4998e4
add the right polytopes to slice on the grid axes
mathleur Aug 18, 2025
c2becf1
fix two tier slicing
mathleur Aug 18, 2025
e156f21
modify compression to accomodate the sliced_polys
mathleur Aug 19, 2025
106b75d
adapt to new qube compression function
mathleur Aug 20, 2025
e8f4638
adapt to new qube compression function
mathleur Aug 20, 2025
9e3e71c
WIP: clean up
mathleur Aug 28, 2025
d6ab7c7
WIP: clean up
mathleur Aug 28, 2025
2780777
merge develop
mathleur Oct 7, 2025
eda8ece
WIP: working qubed backend
mathleur Oct 7, 2025
456c15f
expose QubedSlicer
mathleur Oct 9, 2025
9e9be52
fix other tests
mathleur Oct 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 10 additions & 20 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,9 @@ Cargo.lock
site
example_eo
example_mri
new_test_venv_polytope
new_venv_polytope
new_polytope_venv
polytope_venv
polytope_venv_latest
new_updated_numpy_venv
newest-polytope-venv
.mypy_cache
*.req
polytope_python.egg-info
serializedTree
new_polytope_venv
newest-polytope-venv
Expand All @@ -49,23 +45,17 @@ polytope_feature/datacube/quadtree/target
venv_python3_11
venv_gj_iterator
*.json
venv_python3_11
*.txt
tests/data
venv_gj_iterator
target
rust_deployment_venv
*.so
*.nc

test_w_qubed
rust_venv
new_rust_test_venv
*.so
rust_deployment_venv
*.lock
**/target
**/icon_grids
polytope_feature/_version.py
**/build
**.lock
**/target
**/*venv*/
test_w_qubed
performance_plots_qubed
_version.py
*.DS_Store
icon_grids
Expand Down
29 changes: 29 additions & 0 deletions polytope_feature/datacube/backends/datacube.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ def __init__(self, axis_options=None, compressed_axes_options=[], grid_online_pa
self.unwanted_path = {}
self.compressed_axes = compressed_axes_options
self.grid_md5_hash = None
self.datacube_transformations = []
self.grid_online_path = grid_online_path
self.grid_local_directory = grid_local_directory

Expand Down Expand Up @@ -92,6 +93,8 @@ def _create_axes(self, name, values, transformation_type_key, transformation_opt

if transformation not in self._axes[axis_name].transformations: # Avoids duplicates being stored
self._axes[axis_name].transformations.append(transformation)
if transformation not in self.datacube_transformations:
self.datacube_transformations.append(transformation)
else:
# Means we have an unsliceable axis since we couln't transform values to desired type
if self._axes is None or axis_name not in self._axes.keys():
Expand All @@ -111,6 +114,20 @@ def _check_and_add_axes(self, options, name, values):
if self._axes is None or name not in self._axes.keys():
DatacubeAxis.create_standard(name, values, self)

def _add_all_type_change_transformation_axes(self, options, name, values):
for transformation_type_key in options.transformations:
if transformation_type_key == "type_change":
self._create_axes(name, values, transformation_type_key, options)
else:
DatacubeAxis.create_standard(name, values, self)

def _check_and_readd_axes(self, options, name, values):
if options is not None:
self._add_all_transformation_axes(options, name, values)
else:
if self._axes is None or name not in self._axes.keys():
DatacubeAxis.create_standard(name, values, self)

def has_index(self, path: DatacubePath, axis, index):
"Given a path to a subset of the datacube, checks if the index exists on that sub-datacube axis"
path = self.fit_path(path)
Expand Down Expand Up @@ -163,6 +180,7 @@ def create(
alternative_axes=[],
grid_online_path="",
grid_local_directory="",
datacube_axes={},
context=None,
):
# TODO: get the configs as None for pre-determined value and change them to empty dictionary inside the function
Expand All @@ -189,6 +207,17 @@ def create(
return fdbdatacube
if type(datacube).__name__ == "MockDatacube":
return datacube
if type(datacube).__name__ == "Qube":
from ..datacube_axis import _str_to_axis
from .qubed import QubedDatacube

actual_datacube_axes = {}
for key, value in datacube_axes.items():
actual_datacube_axes[key] = _str_to_axis[value]
qubed_datacube = QubedDatacube(
datacube, actual_datacube_axes, config, axis_options, compressed_axes_options, alternative_axes, context
)
return qubed_datacube

def check_branching_axes(self, request):
pass
Expand Down
Loading
Loading