This is a fork of openEMS that adds a CUDA-accelerated batch FDTD engine (openems-cuda/) for running many small electromagnetic simulations on NVIDIA GPUs.
Status: Experimental. The batch engine is designed for parameter sweeps and inverse-design loops where hundreds or thousands of small FDTD simulations must run with identical grid topologies.
- Fused batched FDTD kernels — multiple simulations execute in a single CUDA kernel launch per Yee phase, reducing kernel-launch overhead.
- Ragged-grid support — simulations in a batch can have different grid dimensions; inactive thread blocks are skipped in-kernel.
- Per-simulation early stopping — each simulation converges independently based on its own energy threshold.
- CUDA Graphs — the per-step launch sequence (PML/Mur pre/post + Yee V/H + non-excitation Apply) is stream-captured into two
cudaGraphExec_ts and replayed each timestep, eliminating ~70% of per-step host-side dispatch. - Batched port monitor — every active sim's V + I integrals run in a single fused kernel per timestep instead of ~5 launches per sim.
- In-memory batch runner (
MemBatchRunner) — run parameter sweeps entirely on the GPU and return S-parameters / Z-parameters without writing intermediate files. - Field-snapshot batch runner (
BatchRunner) — run batches that write E/H fields to disk, with full parity to the per-simulation GPU path. - NF2FF support — near-field to far-field transformation is supported in fused batches.
- Numerical parity — fused batches match the per-simulation GPU engine bit-for-bit on the E-field; the batched port reduction reorders ~50 floating-point sums and shows ≤ 2.2 × 10⁻⁷ relative error on the I time-series (well below typical S-parameter tolerances).
- Python bindings —
pyopenems_cudamodule for scripting parameter sweeps from Python.
Measured on the 16-config dipole bench (sim/bench_16configs_parity.py, 16 sims × ~1700 timesteps × ~25 k cells, RTX 4050 Laptop):
| Configuration | Wall time | Throughput |
|---|---|---|
| Per-sim (max_concurrent=1) | 1.68 s | 9.5 sims/s |
| Fused, no graph, no batched port | 1.53 s | 10.5 sims/s |
| Fused + CUDA Graphs | 1.21 s | 13.2 sims/s |
| Fused + CUDA Graphs + batched port | 1.03 s | 15.5 sims/s |
End-to-end: ~1.5× faster than the previous fused path and ~3.5× faster than the per-sim baseline on this workload, with no change to user-facing API.
Set OPENEMS_CUDA_TIMING=1 to print a per-batch wall/GPU/host/energy/capture breakdown to stderr; set OPENEMS_CUDA_GRAPH=0 to disable the graph capture for A/B comparison.
openEMS/ # upstream openEMS + patches
├── openems-cuda/ # this extension
│ ├── include/ # C++ headers (Engine_CUDA, BatchRunner, etc.)
│ ├── src/ # CUDA kernels and host drivers
│ ├── extensions/ # GPU versions of PML, excitation, Mur ABC, lumped RLC
│ ├── python/ # pybind11 bindings + Python package
│ └── tests/ # parity and smoke tests
The following minimal patches are required so the batch engine can inject a GPU Engine into the standard FDTD_Operator workflow:
FDTD/operator.h— addSetEngine()accessorFDTD/extensions/operator_ext_excitation.h— addSetEngine()accessorFDTD/extensions/operator_ext_lumpedRLC.h— addSetEngine()accessorFDTD/extensions/operator_ext_mur_abc.h— addSetEngine()accessorFDTD/extensions/operator_ext_upml.h— addSetEngine()accessorCommon/processing.h— add helper for port-probe processing in batch mode
| Dependency | Minimum Version | Notes |
|---|---|---|
| CMake | 3.18 | 3.1 for openEMS, 3.18 for CUDA |
| CUDA Toolkit | 11.0+ | Tested on CUDA 12.x |
| CSXCAD | latest | from upstream openEMS dependencies |
| HDF5 | any | C + HL components |
| pybind11 | 2.6+ | for Python bindings |
| Boost | 1.60+ | openEMS dependency |
| VTK | optional | openEMS dependency |
| MPI | optional | openEMS dependency |
Follow the openEMS build guide to install CSXCAD, fparser, and other dependencies.
cd openEMS
mkdir build && cd build
cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$HOME/opt/openEMS \
-DCSXCAD_ROOT_DIR=$HOME/opt/openEMS
make -j$(nproc)
make installIf CUDA is not found or you want to skip the batch engine:
cmake .. -DCMAKE_DISABLE_FIND_PACKAGE_CUDA=TRUEcd openems-cuda/tests
python3 test_fused_parity_uniform.py # field-snapshot parity
python3 test_fused_parity_mem.py # in-memory S-parameter parity
python3 test_earlystop_parity.py # early-stopping parityAll tests should report 0.000e+00 relative error against the per-simulation GPU baseline.
from openems_cuda import CUDABatch
# Run a parameter sweep entirely on the GPU
batch = CUDABatch(
config_generator=my_generator, # yields openEMS config dicts
base_path="/tmp/sweep",
num_processes=1,
)
results = batch.run_mem(
max_concurrent=8, # number of sims fused per chunk
nprobes=1,
)For more examples see openems-cuda/python/openems_cuda/.
- Compute capability: SM 8.9 (Ada Lovelace, e.g. RTX 4050/4060) is the default target.
- Adjust for your GPU: edit
CMAKE_CUDA_ARCHITECTURESinopenems-cuda/CMakeLists.txtor pass-DCMAKE_CUDA_ARCHITECTURES=75for Turing,-DCMAKE_CUDA_ARCHITECTURES=86for Ampere, etc.
max_concurrent=8is a good starting point for an RTX 4050 with ~20 k-cell grids.- Larger grids or more memory-hungry extensions may require smaller chunks.
- Use the in-memory runner (
run_mem) instead of the disk-writing runner (run) for parameter sweeps — it avoids all I/O overhead. - The batched port monitor activates automatically when every engine in the batch has a port monitor and shares the same
record_interval. Mismatched intervals fall back transparently to the per-engine path. - The CUDA graph re-captures only when the active block map shape changes (sims dropping out via early-stopping). Capture cost is ~0.2 ms per re-capture and amortises over the next 100 timesteps.
openEMS is licensed under the GPL v3 or later. This fork inherits the same license. See the upstream repository for full details.
If you want to upstream the GPU batch engine, the minimal prerequisite is getting the SetEngine() accessors merged into openEMS core. After that, openems-cuda/ can be built as a standalone extension.