Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
96011ba
Move FEEC assembly kernels into kernel folders
max-models Oct 7, 2026
d422684
Merge field-diagnostics-on-device (#724) into feec-assembly-kernel-fo…
max-models Oct 7, 2026
570cb12
Merge field-diagnostics-on-device (#724) into feec-assembly-kernel-fo…
max-models Oct 7, 2026
21a7511
Merge field-diagnostics-on-device (#724) into feec-assembly-kernel-fo…
max-models Oct 7, 2026
8adde2c
Merge field-diagnostics-on-device (#724) into feec-assembly-kernel-fo…
max-models Oct 8, 2026
9b12837
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 8, 2026
00949e4
Merge field-diagnostics-on-device (#724) into feec-assembly-kernel-fo…
max-models Oct 8, 2026
5d1d1f9
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 9, 2026
c4bc45a
Merge parent into feec-assembly-kernel-folders
max-models Oct 9, 2026
5e1a35e
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 9, 2026
3fe50b3
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 10, 2026
90e2823
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 10, 2026
233c713
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 10, 2026
c39f740
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 10, 2026
75d046b
Merge field-diagnostics-on-device into feec-assembly-kernel-folders
max-models Oct 10, 2026
7757eb1
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 12, 2026
0734720
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 12, 2026
3dcc52e
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 12, 2026
e0f031d
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 12, 2026
44ab9bf
Merge branch 'field-diagnostics-on-device' into feec-assembly-kernel-…
max-models Oct 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion CUDA_STRATEGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Progress is tracked in struphy-hub/struphy#650.
- [x] **PR 19: Spline mappings on the GPU** ([#683](https://github.com/struphy-hub/struphy/pull/683)) — `kind_map` 0–2 (`IGAPolarCylinder`, `IGAPolarTorus`, `Tokamak`, GVEC, DESC): `DomainArgs` gets array views for `t1..3`, `ind1..3` and `cx/cy/cz` (shapes needed on the device), spline-mapped `Domain`s can be created on the CuPy backend (control points fitted on the host), and `spline_3d`, `spline_2d_straight`, `spline_2d_torus` get device versions in the `kind_map` switch, so every mapping runs on CuPy and `check_mapping_on_device` is gone. Parity arguments and device-helper tests add four spline mappings. Polar splines were host-only at this stage; their current CuPy support is described in [Polar splines on CuPy](#polar-splines-on-cupy-695). See also the [PR 19 implementation notes](#pr-19-implementation-notes).
- [x] **`linear_vlasov_ampere` on the GPU** (part of #688) — the first matrix accumulation: CUDA version of `linear_vlasov_ampere` (`EfieldWeightsCoupling`) on cunumpy's `Array6D` views (cunumpy 0.6.1), device versions of `fill_mat`, `fill_mat_vec`, `m_v_fill_b_v1_symm` and `outer`, and the matrix path of `Accumulator` checked on the CuPy backend (device data, ghost regions, transposed blocks, Schur operator) without a GPU through cunumpy's fake CuPy with emulated kernel launches (see the [implementation notes](#linear_vlasov_ampere-implementation-notes)).
- [ ] **Diagnostics on the device** (struphy-hub/struphy#697) — scalar quantities, marker binning, saved markers and spline evaluation for output run on the device; the scalars reach the host in one copy per update (`Scalars.to_host`), every other saved dataset in one counted copy per output step (`DataContainer.save_data`), and the time state is host bookkeeping. CUDA version of `eval_spline_mpi_tensor_product_fixed` (`SplineFunction.eval_tp_fixed_loc`), moved into its own folder. Marker diagnostics kernels (`pic/diagnostics/kernels`) and SPH kernel density plots are follow-ups (see [Diagnostics on the device](#diagnostics-on-the-device-697)).
- [x] **FEEC assembly kernels in kernel folders** (#411) — the 16 assembly kernels of `feec/mass.py`, `feec/boundary_mass.py` and `feec/basis_projection_ops.py` move from `feec/mass_kernels.py` and `feec/basis_projection_kernels.py` into one folder each under `feec/kernels/`; pure restructuring, ready for CUDA ports (see [FEEC assembly kernels](#feec-assembly-kernels-411)).
- [ ] **Next (order to be confirmed)**: one complete model (`VlasovAmpereOneSpecies`) on the GPU end to end, which needs the [feectools stack](#feectools) merged and released first; the guiding-center kernels (step 3), by hand (decided in #690, see [Hand-written CUDA](#hand-written-cuda-decision-for-690)).
- [ ] **feectools**: the FEEC side (stencil vectors and matrices, MPI exchange, GPU binding) in feectools, stacked PRs [#85](https://github.com/struphy-hub/feectools/pull/85) (merged into `cuda-development`), [#86](https://github.com/struphy-hub/feectools/pull/86), [#87](https://github.com/struphy-hub/feectools/pull/87), [#88](https://github.com/struphy-hub/feectools/pull/88), integrated by [#90](https://github.com/struphy-hub/feectools/pull/90); needed before the end-to-end model run, not for PR 18/19 (see [feectools](#feectools)).
- [x] **PR 13: Move the kernel infrastructure to cunumpy** — `Kernel`, `KernelCatalog`, `CudaKernel` and `Argument` are replaced by their cunumpy counterparts; each owner has a single `args_*` object on both backends, and `pusher_args.cuh` is generated (see [Moving to cunumpy](#moving-to-cunumpy-pr-13)). Kernels and device helpers only change their includes.
Expand Down Expand Up @@ -66,7 +67,7 @@ CUDA kernels can be added one by one. If the code runs on the GPU and needs a ke
| `src/struphy/utils/cuda_arguments.py` | `CUDA_STRUCTS`, `CUDA_OPTIONS`, `write_pusher_header()`, `write_local_projectors_header()` and `write_spline_header()` |
| `src/struphy/kernel_arguments/pusher_args.cuh`, `local_projectors_args.cuh`, `spline_args.cuh` | the C structs `MarkerArgs`, `DerhamArgs`, `DomainArgs`, `LocalProjectorsArgs` and `SplineArgs`, generated from the CUDA classes, using cunumpy array views |
| `src/struphy/geometry/base.py`, `src/struphy/pic/base.py`, `src/struphy/feec/psydac_derham.py` | `Domain.args_domain`, `Particles.args_markers` and `Derham.args_derham` are the pyccel class on NumPy and the CUDA class on CuPy, chosen once at construction; every kernel call goes through a `Kernel` |
| `src/struphy/*/kernels/` (`pic/pushing`, `pic/accumulation`, `pic/diagnostics`, `pic/sph`, `bsplines`, `geometry`, `feec`, `feec/local_projectors`) | every kernel called from Python with argument objects, one folder each: 44 pusher/evaluation (incl. `reflect`), 16 accumulation, 10 marker diagnostics, 4 SPH evaluation, 3 spline evaluation, 4 geometry, 1 FEEC utility and 8 local projector kernels. Each folder's `__init__.py` declares its `Kernel`; the code imports it (`from struphy.geometry.kernels.kernel_evaluate import kernel_evaluate`). Sixteen have CUDA versions (`linear_vlasov_ampere` the latest), with parity cases in `pic/tests/cuda_parity_cases.py`, among them the four geometry kernels (PR 18) |
| `src/struphy/*/kernels/` (`pic/pushing`, `pic/accumulation`, `pic/diagnostics`, `pic/sph`, `bsplines`, `geometry`, `feec`, `feec/local_projectors`) | every kernel called from Python with argument objects, one folder each: 44 pusher/evaluation (incl. `reflect`), 16 accumulation, 10 marker diagnostics, 4 SPH evaluation, 4 spline evaluation, 4 geometry, 1 FEEC utility, 16 FEEC assembly (#411) and 8 local projector kernels. Each folder's `__init__.py` declares its `Kernel`; the code imports it (`from struphy.geometry.kernels.kernel_evaluate import kernel_evaluate`). Eighteen have CUDA versions (the latest `linear_vlasov_ampere`, `vlasov_maxwell`, `bstar_parallel_3form` and `eval_spline_mpi_tensor_product_fixed`), with parity cases in `pic/tests/cuda_parity_cases.py`, among them the four geometry kernels (PR 18) |
| `src/struphy/geometry/evaluation_kernels.cuh`, `transform_kernels.cuh`, `spline_mappings_kernels.cuh`, `geometry/domains/<name>/<name>_cuda.cuh` | device versions of the mapping helpers for every mapping: `f`/`df` per analytic domain (`kind_map` 10–12, 20–22, 30–32, PR 18) and the spline mappings `spline_3d`, `spline_2d_straight`, `spline_2d_torus` (`kind_map` 0–2, PR 19), the `kind_map` switch, `det_df`, `df_inv`, `g`, `g_inv`, `select_metric_coeff`, `pull`, `push`, `tran` |
| `src/struphy/pic/tests/test_cuda_parity.py`, `test_cuda_emulation.py` | parity of every CUDA kernel with pyccel on the cases of `pic/tests/cuda_parity_cases.py` (`PARITY_CASES`): on a GPU (`assert_kernels_agree`), and without one by CPU emulation |
| `src/struphy/pic/tests/test_kernel_backends.py` | the generated-header tests, the check that each CUDA class mirrors its pyccel class, signature check of all kernels, owner class selection, and (on a GPU) struct layout and copy/pickle checks |
Expand Down Expand Up @@ -865,6 +866,23 @@ What a CuPy run copies to the host per output step, and where:
- **Follow-ups**: the marker diagnostics kernels in `pic/diagnostics/kernels/` (5D energies, magnetic moments,
canonical momenta, `thermal_energy`; used by the guiding-center and hybrid models) have no CUDA version; SPH kernel
density plots need the SPH kernels (step 6).

## FEEC assembly kernels (#411)

The assembly kernels of the weighted mass operators, the boundary mass operators and the basis projection
operators are entry kernels like the pushers, so they follow the folder rule: `feec/kernels/<name>/` with
`<name>_kernels.py` and an `__init__.py` that declares the `Kernel`. Moved, without numerical change:
`kernel_{1,2,3}d_mat`, `kernel_{1,2,3}d_vec`, `kernel_{1,2,3}d_eval`, `kernel_3d_matrixfree`, `kernel_3d_diag`,
`surface_kernel_3d_mat`, `surface_kernel_3d_vec` (from `feec/mass_kernels.py`) and
`assemble_dofs_for_weighted_basisfuns_{1,2,3}d` (from `feec/basis_projection_kernels.py`); both modules are
gone. The call sites import the kernels and pick the one for the dimension from a small dict
(`_MAT_KERNELS[ldim]`, ...) instead of `PyccelKernel(getattr(module, name))`.

None of them has a CUDA version yet. They are declared with `missing_cuda="fallback"`, which keeps the behaviour
of the former `PyccelKernel` calls on the CuPy backend (host copies of the arrays at every call, with a warning
once) instead of raising, so FEEC setups that worked on CuPy keep working. **They are ready for CUDA ports**: a
port adds `<name>_cuda.cu` to the folder and its parity cases to `pic/tests/cuda_parity_cases.py`, and drops the
fallback for that kernel. They take only arrays and scalars, no argument objects.
extraction/DOF operators, derivatives and transposes, vector arithmetic, all three projector solvers,
transposed and boundary-condition solves, mass matrices/preconditioners, callable and sampled weights,
and spline coefficient extraction/restart. Repeated Kronecker and preconditioner applications are checked
Expand Down
Loading
Loading