Skip to content

draw_markers does not MPI-sort the markers without sorting boxes (off-rank markers in setup accumulations, segfault on 2 ranks) #764

Closed

Description

@spossann

Bug: Particles.draw_markers MPI-sorts the markers after loading only if sorting boxes exist:

# src/struphy/pic/base.py (devel a8865744, line 1408)
if self._initialized_sorting and sort:
    if self.mpi_comm is not None:
        self.mpi_sort_markers()
    self.do_sort()

_initialized_sorting is True only when SortingParameters(boxes_per_dim=...) is given. Without sorting boxes, on more than one rank, the markers drawn on the whole domain (sobol_standard, and possibly other loading modes) stay on the wrong rank until the first PushEta sorts them. Every accumulation before the time loop sees them. For example, the initial Poisson solve in VlasovAmpereOneSpecies.post_allocate and LinearVlasovAmpereOneSpecies.post_allocate then writes outside the local stencil arrays.

Observed (found while reviewing #734; Linux, Open MPI 4.1.6, 2 ranks, NumPy backend):

  • 80 of the 160 markers of each rank lie outside the rank's domain after draw_markers;
  • rank 1 segfaults in StencilVector.exchange_assembly_data (feectools/ddm/nonblocking_data_exchanger.py, start_exchange_assembly_data), called from Accumulator._accumulate in the initial Poisson solve;
  • with one mpi_sort_markers() after draw_markers, both e2e tests of [gpu-1] End-to-end GPU test of VlasovAmpereOneSpecies on 1 and 2 ranks #734 pass on 2 ranks (NumPy and fake CuPy).

Reproduce: mpirun -n 2 pytest --with-mpi src/struphy/models/tests/test_gpu_LinearVlasovAmpereOneSpecies.py::test_numpy_reference on the branch of #734 (any model run with SortingParameters() without boxes, on 2 ranks, should do).

The verification tests always use sorting boxes, which is why CI never hit this. On other platforms the out-of-bounds writes may corrupt memory without crashing.

Proposed fix: in draw_markers, MPI-sort whenever the communicator has more than one rank. This is needed for correctness, independently of the sorting boxes and of SortingParameters.do_sort, which defaults to False and is about sorting into boxes. Sort into boxes (do_sort()) only if they exist and sort is set. Add an MPI regression test: draw without boxes on 2 ranks and check that all markers are on their rank's domain.

🤖 Generated with Claude Code

Activity

  1. added 2 commits that reference this issue on Oct 13, 2026
  2. spossann commented on Oct 14, 2026

    @spossann
    MemberAuthor

    fixed in stack #762

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions