@graziano-giuliani we raised the question: "Async IO introduced by Everett with queue for operations: does it work for parallel I/O?". The program was built with ./configure --enable-async-netcdf --enable-pnetcdf flags.
Answer: We discovered a bug related to async-netcdf feature with parallel writing. The program execution hangs until it is stopped by slurm due to time limit:
The actual bug
--enable-async-netcdf defers every nf90_put_var call to a per-rank background pthread (Share/mod_async_netcdf.F90:1979-2049 — the put_var_r4/r8/i4 worker functions that the thread executes call plain nf90_put_var, not _all). The main computation thread enqueues the write and immediately moves on to the next timestep, it never waits for that write to actually land on disk.
The one function that would force a rank to block until its pending async writes finish is flush_output_streams (Main/mpplib/mod_ncout.F90:4245-4249, which calls outstream_async_flush → async_netcdf_wait_all). I grepped the entire tree for callers:
Main/mpplib/mod_ncout.F90:41: public :: flush_output_streams
Main/mpplib/mod_ncout.F90:4245: subroutine flush_output_streams
And we found out that flush_output_streams has zero callers anywhere in the codebase. Nothing in RCM_run's loop per timestep ever invokes it.
Why the program hangs specifically with do_parallel_netcdf_out=.true.
Extending the record dimension on every new output timestep is a collective metadata operation in parallel HDF5: every rank must call it, in the same relative order. With writes silently deferred to independently-scheduled background threads and no barrier ever forcing ranks back into lockstep, different ranks' threads finish their previous write at different real times. When the next collective metadata call comes due, HDF5 blocks each rank until every other rank also reaches it. Since nothing guarantees they ever do so within the same window, ranks stall waiting on peers whose background thread hasn't caught up.
This is also why it doesn't manifest with do_parallel_netcdf_out=.false.: only iocpu ever writes there, so there's no cross-rank collective dependency to desynchronize in the first place. A single writer can be as asynchronous as it likes with respect to itself.
Summary
--enable-async-netcdf + do_parallel_netcdf_out=.true. is currently an unsupported combination: the synchronization primitive needed to make deferred per-rank writes safe under collective parallel HDF5 exists in the code but was never wired into the run loop. We needs to call flush_output_streams() inserted at the right point in RCM_run. It can be after each output write, before the next timestep that would trigger a collective metadata op. Until then, async-netcdf needs to be restricted to do_parallel_netcdf_out=.false. only.
Attachments
Here are the logs when the program was built and run. Also includes the slurm script and namelist files.
build_async_on.sh
io-bench-build-async-on-49180760.txt
async-on__pin_T_pout_T__nodes16.sbatch.txt
async-on__pin_T_pout_T__nodes16.out.txt
async-on__pin_T_pout_T__nodes16.err.txt
@graziano-giuliani we raised the question: "Async IO introduced by Everett with queue for operations: does it work for parallel I/O?". The program was built with
./configure --enable-async-netcdf --enable-pnetcdfflags.Answer: We discovered a bug related to async-netcdf feature with parallel writing. The program execution hangs until it is stopped by slurm due to time limit:
The actual bug
--enable-async-netcdfdefers everynf90_put_varcall to a per-rank background pthread (Share/mod_async_netcdf.F90:1979-2049— theput_var_r4/r8/i4worker functions that the thread executes call plainnf90_put_var, not_all). The main computation thread enqueues the write and immediately moves on to the next timestep, it never waits for that write to actually land on disk.The one function that would force a rank to block until its pending async writes finish is
flush_output_streams(Main/mpplib/mod_ncout.F90:4245-4249, which callsoutstream_async_flush→async_netcdf_wait_all). I grepped the entire tree for callers:And we found out that
flush_output_streamshas zero callers anywhere in the codebase. Nothing inRCM_run's loop per timestep ever invokes it.Why the program hangs specifically with
do_parallel_netcdf_out=.true.Extending the record dimension on every new output timestep is a collective metadata operation in parallel HDF5: every rank must call it, in the same relative order. With writes silently deferred to independently-scheduled background threads and no barrier ever forcing ranks back into lockstep, different ranks' threads finish their previous write at different real times. When the next collective metadata call comes due, HDF5 blocks each rank until every other rank also reaches it. Since nothing guarantees they ever do so within the same window, ranks stall waiting on peers whose background thread hasn't caught up.
This is also why it doesn't manifest with
do_parallel_netcdf_out=.false.: onlyiocpuever writes there, so there's no cross-rank collective dependency to desynchronize in the first place. A single writer can be as asynchronous as it likes with respect to itself.Summary
--enable-async-netcdf+do_parallel_netcdf_out=.true.is currently an unsupported combination: the synchronization primitive needed to make deferred per-rank writes safe under collective parallel HDF5 exists in the code but was never wired into the run loop. We needs to callflush_output_streams()inserted at the right point inRCM_run. It can be after each output write, before the next timestep that would trigger a collective metadata op. Until then,async-netcdfneeds to be restricted todo_parallel_netcdf_out=.false.only.Attachments
Here are the logs when the program was built and run. Also includes the slurm script and namelist files.
build_async_on.sh
io-bench-build-async-on-49180760.txt
async-on__pin_T_pout_T__nodes16.sbatch.txt
async-on__pin_T_pout_T__nodes16.out.txt
async-on__pin_T_pout_T__nodes16.err.txt