Skip to content

GPU stack 8/8: profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) - #789

Draft
max-models wants to merge 9 commits into
gpu-stack-7-particle-loading-sortingfrom
gpu-stack-8-profiling-cases
Draft

max-models wants to merge 9 commits into
gpu-stack-7-particle-loading-sortingfrom
gpu-stack-8-profiling-cases

Conversation

@max-models

@max-models max-models commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Part 8 of 8 of the split of #772 (cuda-aware-mpi-fixes) into stacked PRs. Requires struphy-hub/feectools#104 to be merged into feectools devel-tiny first (PR 1 bumps the submodule).

  • Poisson gaussian_cube: --steps (repeated solves with the matrices assembled once), --solver pcg|pcg_sr, --gpu-timing (one thread per rank, CUDA-event device times per region), --no-output. Submit scripts for 28 CPU ranks vs 1 GPU and for GPU scaling. examples/Poisson/sweep_poisson.py runs several cases.
  • Vlasov homogeneous_slab: particle push profiling case (16M markers by default, --test-run for the examples test), with post-processing and the same two submit scripts.
  • profiling/profiling_job.py: launcher detection and job setup for these cases.
  • profiling/sweep.py (run_sweep, variant_entries): runs several submit scripts as one sweep in one process (profiling_job.begin_sweep / finalize_all), so they share one packaging folder; a failing script is reported and the sweep goes on. sweep_poisson.py imported it, but it was never committed, so it is written new here. Checked with a dry run of sweep_poisson.py (30 submit commands) and a run on test scripts (variants, a failing exit code, an exception, registered cases).

Tests

Local, macOS without a GPU, in a fresh environment with feectools 1d8d47c and the pinned cunumpy, after merging devel (with #774 and #775). The PR passes test_cuda_parity, test_cuda_emulation, test_kernel_backends, test_sorting, test_accum_matrix_cupy, test_preconditioner_cupy, test_poisson and test_output on its own. The top of the stack also passes the mass-matrix, preconditioner, Derham-GPU, fused-gradient, Kronecker-block, Poisson, shear Alfvén, post-processing, polar fake-CuPy and all pic tests (1477 passed), the examples test for the two profiling cases, the sorting, Poisson and fused-gradient tests under mpirun -n 2, and the device MPI sort on the fake CuPy on 2 and 4 ranks. The Kronecker products match the stencil mass matrices to 5e-16 for M0, M1, M2, M3, Mv, M1n and M2n on a Cuboid, periodic and Dirichlet, on 1 and 2 ranks.
Not tested on a GPU here. The CUDA kernels and the device paths need a run on the cluster.

Stack (merge in order)

  1. Device ghost-exchange plans, fused block exchanges, single-reduction PCG (CUDA-aware MPI) feectools#104 (feectools, into devel-tiny)
# PR base
1 #782 update feectools (device ghost exchange, single-reduction PCG); pcg_sr solver option devel
2 #783 host/device conversions in post-processing; math.sqrt in Species #782
3 #784 CUDA mass-matrix assembly kernel kernel_3d_mat #783
4 #785 apply separable mass matrices as Kronecker products of 1D banded matrices #784
5 #786 Poisson/diffusion solve with fewer passes over the vectors #785
6 #787 marker rows prepared inside the push kernels #786
7 #788 chunked marker loading, row-based weight update, leaner MPI sorting; static field in PushVxB #787
8 #789 profiling cases Poisson cube and Vlasov slab (CPU node vs GPU, GPU scaling) (this PR) #788

🤖 Generated with Claude Code

max-models and others added 2 commits October 16, 2026 10:28
…U scaling

- Poisson gaussian_cube: --steps (repeated solves with the matrices assembled once), --solver
  pcg|pcg_sr, --gpu-timing (one thread per rank, CUDA-event times per region), --no-output;
  submit scripts for 28 CPU ranks vs 1 GPU and for GPU scaling; sweep_poisson.py runs several
  cases.
- Vlasov homogeneous_slab: particle push profiling case (default 16M markers, --test-run for the
  examples test) with post-processing and the same two submit scripts.
- profiling_job.py: launcher detection and job setup for these cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its two pushers have CUDA versions, so the case keeps the layout it was measured with
(STRUPHY_MARKERS_COLUMN_MAJOR defaults to 1 here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
max-models and others added 7 commits October 16, 2026 12:54
devel stores the marker buffer as (n_cols, n_markers) on both backends (#774), so the column-major
opt-in of the GPU stack is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…profiling-cases (feectools update, CachedLaunch)
…profiling-cases (pcg_sr in the shear Alfven solves)
examples/Poisson/sweep_poisson.py imports run_sweep and variant_entries from it, and
profiling_job.py refers to it (begin_sweep, finalize_all), but it was never committed on
cuda-aware-mpi-fixes, so the sweep script failed at import. variant_entries lists one entry per
key of a submit script's VARIANTS; run_sweep runs the scripts as __main__ in this process inside
profiling_job.begin_sweep (each submits its jobs and only registers its case), reports failing
scripts and goes on, and then calls profiling_job.finalize_all to package (and push) every run
as its job finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@max-models
max-models marked this pull request as draft October 22, 2026 06:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant