MPI fails with boolean reduce ops on darwin/mpich

[platipodium]

mpirun -np 2 ./pschism
Assertion failed in file src/mpi/coll/reduce_local/reduce_local.c at line 64: MPIR_Internal_op_dt_check(op, datatype)
0   libpmpi.12.dylib                    0x000000010596506c MPL_backtrace_show + 56
1   libpmpi.12.dylib                    0x0000000105890b78 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x00000001057a7e6c MPIR_Reduce_local + 528
3   libpmpi.12.dylib                    0x00000001057dc610 MPIDI_POSIX_mpi_release_gather_gather + 1044
4   libpmpi.12.dylib                    0x00000001057e9d98 MPIDI_POSIX_mpi_allreduce_release_gather + 912
5   libpmpi.12.dylib                    0x00000001057e9074 MPIDI_Allreduce_intra_composition_gamma + 432
6   libpmpi.12.dylib                    0x00000001057d0354 MPIR_Allreduce + 1564
7   libmpi.12.dylib                     0x0000000104770358 MPI_Allreduce + 2600
8   libmpifort.12.dylib                 0x00000001043d5144 mpi_allreduce_ + 144
9   pschism_FABM_OLDIO_PREC_EVAP_BLD_ST 0x0000000102b1cd60 levels0_ + 41376
10  pschism_FABM_OLDIO_PREC_EVAP_BLD_ST 0x00000001028b2914 schism_init_ + 518564
11  pschism_FABM_OLDIO_PREC_EVAP_BLD_ST 0x00000001027a0bac schism_main_ + 108
12  pschism_FABM_OLDIO_PREC_EVAP_BLD_ST 0x00000001027a0d50 MAIN__ + 252
13  pschism_FABM_OLDIO_PREC_EVAP_BLD_ST 0x00000001027a0da4 main + 56
14  dyld                                0x000000018e951d54 start + 7184

[platipodium]

Hmm, copilot suggested that the MPI_Allreduce for BOOLEAN is not portable and suggests to do the reduce on a portable integer

 !         Final extrapolation
           srwt_xchng(1)=istop==1
-          call mpi_allreduce(srwt_xchng,srwt_xchng_gb,1,MPI_LOGICAL,MPI_LAND,comm,ierr)
+          i_srwt_xchng=0
+          if (srwt_xchng(1)) i_srwt_xchng=1
+          call mpi_allreduce(i_srwt_xchng,i_srwt_xchng_gb,1,itype,MPI_MIN,comm,ierr)
+          srwt_xchng_gb(1)=i_srwt_xchng_gb==1
           if(srwt_xchng_gb(1)) then !all ranks ready
             if(myrank==0) write(16,*)'doing final extrapolation in levels1...'

[platipodium]

I tested this on my workstation and indeed it worked. It occurs in 5 places in misc_subs.F90, how do you think we can proceed, i.e., (1) implement this conversion, or (2) change the primary variable Xrwt_exchange from boolean to integer, or (3) forget about portability to this system or (4) some other idea?

[josephzhang8]

I'd stick to original. Event though the # of occurrence is not big at this time, there will be more in other codes. MPI manual states this is standard so I'd blame that MPI installation. BTW: Mpich is known to be problematic for many systems and that's why most use mvapich now.

[platipodium]

OK, let me try with other communicators on macOS. If Mpich/darwin is not supported, we should document it somewhere (well ... sort of it is documented by this issue :=)

[platipodium]

BTW, the compiler already warns on this issue (with both MPICH and OpenMPI), e.g.

1289 |           call mpi_allreduce(srwt_xchng,srwt_xchng_gb,1,MPI_LOGICAL,MPI_LAND,comm,ierr)
      |                                        1
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)).
/devel/schism/schism/src/Hydro/misc_subs.F90:349:23:

[platipodium]

The root cause seems to be that fortran logical ABI is not guaranteed to match MPI_LOGICAL, unlike integers, who are guaranteed in bits used and interpretation across implementations. So we risk misinterpretation of True/False, even if OpenMPI compiles and runs fine.

[jreniel]

The full fix is/was here: #208