Fix issues with writing vars with unlimited dimensions using NetCDF/HDF5

[jayeshkrishna]

Variables with unlimited dimensions require collective puts with NetCDF

• Use COLLECTIVE for all variables using NetCDF4 parallel I/O
• Also adding generic cleanup calls for HDF5 and NetCDF
• Explicitly close open files and I/O decomps in finalize (to free up resources)

Fixes #711

[jayeshkrishna]

Since I/O decomp freeing/finalizing requires MPI calls we need to explicitly clean it up in finalize - so that it happens before MPI finalize (Also see #698 where logic to automatically cleanup I/O decomps was added). This change fixes #711

[jayeshkrishna]

NetCDF and HDF5 cleanup calls,

• nc_finalize() was introduced in NetCDF 4.4.0 (SCORPIO requires NetCDF >= 4.4.0)
• h5close() is available on all recent versions

[Copilot]

Pull request overview

This PR targets robustness of SCORPIO’s NetCDF/HDF5 backends by (a) enforcing COLLECTIVE access for NetCDF4 parallel I/O to better support variables with unlimited dimensions, and (b) performing more aggressive cleanup during PIOc_finalize_impl() to avoid MPI-related teardown problems (Fixes #711).

Changes:

• Adjust file-finalization logic to close/cleanup remaining open files and delete I/O decompositions during PIOc_finalize_impl().
• Add NetCDF4-parallel “set COLLECTIVE access” behavior during open and var definition paths.
• Add explicit NetCDF/HDF5 library cleanup calls (nc_finalize(), H5close()) when the last IO system is finalized.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/clib/pio_internal.h	Updates internal helper signature for finalize-time file cleanup.
src/clib/core/util/pio_lists.cpp	Adds conditional behavior around waiting on hard-close vs. not waiting during finalize cleanup.
src/clib/core/pioc.cpp	Extends PIOc_finalize_impl() to close remaining files, delete decomps, and call NetCDF/HDF5 cleanup.
src/clib/core/pioc_support.cpp	Adds NetCDF4-parallel “set COLLECTIVE access” logic on file open paths.
src/clib/core/pio_nc.cpp	Small control-flow bracing change around nc_inq_varid() call.
src/clib/core/pio_getput_int.cpp	Adds an (currently unused) helper related to detecting unlimited dimensions; adds includes.

Suppressed comments (2)

src/clib/core/pioc_support.cpp:5141

• In the NCZarr open path this attempts nc_var_par_access(file->fh, NC_GLOBAL, ...) even when nc_open_par() was never called (e.g., empty NCZarr filename) or failed, and NC_GLOBAL is not a valid varid for nc_var_par_access(). This can generate spurious warnings and does not achieve the intended COLLECTIVE setting for variables.

        /* Set COLLECTIVE access for all variables.
         * FIXME: We might just need it for some variables (unlimited dims, chunked,
         * compressed etc). Revisit this call after measuring performance
         */
        ierr = nc_var_par_access(file->fh, NC_GLOBAL, NC_COLLECTIVE);
        if(ierr != PIO_NOERR){

src/clib/core/pioc.cpp:1766

• H5close() returns a status; ignoring it can hide failures to shut down the HDF5 library cleanly (often indicating leaked open objects). Consider checking the return value and emitting a warning on failure.

#ifdef _HDF5
    if(niosysid == 1){
      /* All data should already be flushed by now. Free any HDF5 internal structures/objects/memory */
      H5close();
    }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/clib/core/util/pio_lists.cpp:228

• When wait_on_hard_close is false (the default in non-async builds), this function deletes file descriptors from the global list without actually closing the underlying file handles. That can leak resources and undermines the intent of closing files during PIOc_finalize_impl(). Consider performing a synchronous hard close in the non-wait path before deleting from the list.

    if(file->iosystem->iosysid == iosysid){
      if(wait_on_hard_close){
        //ret = spio_hard_closefile(file->iosystem, file, true);
        ret = spio_wait_on_hard_close(file->iosystem, file);
        if(ret != PIO_NOERR){