Repository navigation
Create the domain decomposition once the spline degree is known #628
Description
Each MPI process must own at least degree coefficients along each direction (at least degree + 1 along non-periodic directions of multipatch geometries), see psydac.fem.partitioning.partition_coefficients. However, the domain decomposition is created before the spline degree is known (in the Geometry constructors, or in discretize(domain, ncells=...)), hence nothing prevents a decomposition where some processes own too few cells. The problem is only detected later, when the spline space is created:
# Run with: mpirun -n 4 python example.py
import numpy as np
from mpi4py import MPI
from psydac.ddm.cart import DomainDecomposition
from psydac.fem.splines import SplineSpace
from psydac.fem.tensor import TensorFemSpace
ncells, degree = 8, 3 # 2 cells per process, i.e. 2 < 3 coefficients
W = SplineSpace(degree=degree, grid=np.linspace(0, 1, ncells + 1), periodic=True)
ddm = DomainDecomposition([ncells], [True], comm=MPI.COMM_WORLD)
V = TensorFemSpace(ddm, W) File "psydac/fem/partitioning.py", line 72, in partition_coefficients
assert all(e-s+1 >= V.degree)
AssertionError
PR #527 replaces this assertion with a ValueError which explains the problem and suggests possible remedies (fewer processes along that direction, e.g. with mpi_dims_mask, or more cells), but the user still has to find a valid configuration by hand.
The domain decomposition should rather be created when the spline degree is known, so that the process grid can be chosen accordingly (e.g. with the argument min_blocksizes of psydac.ddm.partition.compute_dims, which is currently unused by DomainDecomposition).
Constraints
- A domain decomposition is shared by all the spaces defined on a geometry, which may be created by separate calls to
discretizewith different degrees; onlydiscretize(derham, ...)knows all the degrees of a sequence at once - A geometry read from a file contains spline mappings, which are defined on a domain decomposition created when the file is read, before any FEM space exists
- The minimum number of cells per process depends on the degree, on the multiplicity (with multiplicity
m, a process owningccells ownsm * ccoefficients), on the periodicity, and on whether the geometry is multipatch
Possible strategies
- Explicit: pass the degree (or a minimum number of cells) when the geometry is created, e.g.
discretize(domain, ncells=..., degree=...)and in theGeometryconstructors;Geometry.from_filecould also use the degrees of the mappings stored in the file. Simple and predictable, but it extends the API, and users must give the maximum degree of all the spaces they will create - Lazy:
Geometrystores the number of cells, the periodicity, the communicator, andmpi_dims_mask, and creates the domain decomposition when the first space (or de Rham sequence) is discretized. Transparent for users, but spaces of higher degree created later may not fit, and it does not work for geometries read from a file - Sequence-level: create the domain decomposition in
discretize(derham, ...), which knows all the degrees, when the geometry was created from the number of cells only. It covers the common FEEC use case, but not standalone spaces
These strategies may also be combined, e.g. strategy 1 for geometry files and for explicit control, and strategy 2 or 3 as default for geometries created from the number of cells.
Discussion started in #527 (comment).
Activity