`ccdtools` RACMO dataset load causes UCX/MPI segfault (signal 11/7) on Gadi

[justinh2002]

Loading the RACMO dataset via ccdtools catalog.load_dataset('racmo2.4p1_monthly_11km_1979-2023, version='v1') causes a fatal UCX/MPI crash on Gadi, killing the Python process with exit code 139 (SIGSEGV) or 135 (SIGBUS).

[gadi-login-07:...:0] UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[gadi-login-07:...:2] Caught signal 7 (Bus error: Sent by the kernel)
[gadi-login-07:...:1] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
Bus error

steps to reproduce

import ccdtools as dp
catalog = dp.catalog.DataCatalog()
racmo = catalog.load_dataset('racmo2.4p1_monthly_11km_1979-2023', version='v1')

Other datasets (measures_bedmachine_antarctica, measures_insar_based_antarctica_ice_velocity_map, antarctic_geothermal_heat_flow_model_aq1) load without issue via the CCD. The crash appears specific to the RACMO dataset.

[lawrenceabird]

@justinh2002 - Is this error specific to login nodes? By default, this loads all the RACMO variables, so it's a large amount of data and it's not too surprising that it fails on a login node.

If there is a need/reason to run this on a login node (and not an ARE session/PBS job), perhaps we can look at different chunking or something. Is there a use case for doing this on a login node?

[justinh2002]

@lawrenceabird my bad, this error isn't specific to login nodes, I had tested it on an X-large ARE job (12 CPUs etc.?) and the jupyter kernel kept crashing, which is why I resorted to translating everything into script form and trying to debug it over on a login node.

[lawrenceabird]

Hi @justinh2002 - thanks for clarifying that!

Loading this dataset appears to be behaving as expected, it's just really memory hungry. It's loading a lot of data, so it requires a large memory instance. I've been able to successfully load this dataset using a custom ARE instance of 4 CPU and 128 GB memory.

We have similar large-memory dependence with other datasets (e.g. high-resolution REMA). I have created a subsequent issue to tackle this (and included a couple of sub-issues that I think would be a good place to start) here.

If you're happy to continue this discussion/tackle this on that issue, can we close this one?