Repository navigation
Multi-rank marker exchange with CUDA-aware MPI #698
Description
Particles have to move between ranks without going through the host.
- check
Particles.mpi_sort_markersand the send/receive buffers for host staging on CuPy - use device buffers with CUDA-aware MPI (
xp.mpi_is_cuda_aware(),cunumpy.mpi.synchronize_for_mpi, as feectools does since CUDA 2 - MPI with device buffers feectools#86) - a 2-rank GPU run without host/device transfers in the exchange
Part of #650 (see CUDA_STRATEGY.md).
Activity
CUDA 13 - Use maybempi for MPI instead of feectools.ddm.mpi (#701)
Marker exchange between ranks on the device (CUDA-aware MPI)
Marker exchange between ranks on the device (CUDA-aware MPI) (#712)
max-models commentedon Oct 14, 2026
Done in #712:
Particles.mpi_sort_markersexchanges the marker rows as device buffers with CUDA-aware MPI (_sendrecv_markers_device), with pinned host staging and a warning when MPI isn't CUDA-aware. The SPH ghost-box exchange (_sendrecv_markers_boxes) still goes through the host. That part belongs with the SPH port in #694.