LibDDLA 0.0.7 is a library for dense distributed linear algebra.
Building LibDDLA requires C++17; applications using its public headers need C++11 or newer.
It provides ScaLAPACK-style APIs with 2D block-cyclic data distribution over an
MPI process grid. Every build includes the CPU backend (OpenBLAS/vendor
BLAS). CUDA or HIP can add GPU support to the same library, with backend
selection per handle at runtime. All functions live in the ddla namespace.
Version 0.0.7 is in preparation. It includes source-incompatible API changes; rebuild applications and follow the 0.0.7 migration guide. See the changelog for changes since 0.0.6.
- Build and prerequisites, runtime backend selection, and communication modes
- Data model and lifecycle and API overview
- Hermitian and tridiagonal eigensolvers, including partial and generalized problems
- Internal workspace interfaces for library developers
- Testing and 0.0.7 release checks
-
DTK HIP POTRF. Use only the factor in the selected
uplo; the strict opposite triangle may be overwritten. See the DTK compatibility notes for version-specific measurements and the regression-test policy. -
Backend selection. The CPU backend is always built. Enable
DDLA_USE_CUDAorDDLA_USE_HIPto add GPU support; the two GPU vendors are mutually exclusive. With neither enabled the build is CPU-only. There is no GPU-only configuration orDDLA_USE_CPUbuild option. CPU+GPU libraries select the backend perDdlaHandle_tat runtime. -
CPU and GPU coverage. Both backends provide the local BLAS/LAPACK wrappers, including
gemm,trmm,trtrm,gemmBatched,getrf,potrf,geqrfandlarft; distributed GEMM, LU, Cholesky, QR/QL, triangular multiplication/solve, and data movement; andrandom_generate. Public compute calls select the backend from their handle: the samepgetrf(handle, ...)call uses host BLAS/LAPACK and MPI on a CPU handle or the GPU implementation on a GPU handle. Scalar types are inferred from arguments or supplied explicitly, such aspgetrf<double>(handle, ...). Homogeneous variable-size batched GEMMs require a GPU build. The mixed real/complexgemmVbatchedand mixed-domaingemmVbatched2soverloads also support CPU execution with host metadata.DDLA_USE_CCLis not supported with a CPU-only build, since there is no GPU backend to communicate with. -
Supported scalar types are
float,double,std::complex<float>, andstd::complex<double>. The distributed Cholesky family (ppotrf/ppotrs/pposv) is instantiated for all four types. -
Submatrix support depends on the routine. Most factorization/solve routines accept descriptors larger than the logical sub-matrix and operate only on the leading block anchored at global (0,0) (the same convention as
pgemm); local extents are derived from the logical dimensions, not the descriptor's full size.pdamtakes an optionaln(default-1= the whole matrix).pgetrf,pgeadd,gatherandscattersupport explicit one-based global origins; theia/jaarguments ofppotrfand theia/ja/ib/jbarguments ofpposvmust currently be 1. -
Matrix storage is caller-owned. Use host pointers for CPU handles and device pointers for GPU handles. Individual routines may allocate temporary storage internally.
ddla::vector<T>provides optional ownership of an uninitialized buffer in the handle's memory space; the handle must outlive it. Public convenience routines arrange their temporary storage. Internal callers can check arguments once, query local host/backend workspace byte counts without allocation, and reuse caller-ownedstd::vector<char>/ddla::vector<char>buffers through unchecked execution. Scratch-free routines such as AXPY have check/run only. See internal workspace interfaces for examples and memory/resource lifetime requirements. -
Integer dimensions (ScaLAPACK-compatible). Global/local dimensions, strides, and indices use
int, matching ScaLAPACK's descriptor convention. Matrices whose element count exceedsINT_MAXare not supported; extending toint64_tglobal dimensions is a deliberate future work item and is not part of the current API. -
An installable CMake package config is provided: after
cmake --install(ormake install), downstream builds can usefind_package(LibDDLA)and linkddla::ddla. The package config locates MPI automatically and exports the real library target, including its SONAME, C++11 consumer requirement and link dependencies. Private backend macros are not required by public headers.
- CMake ≥ 3.18 (the source lists use
TARGET_DIRECTORYproperties) - C++17 compiler to build LibDDLA; C++11 or newer for public-header consumers
- MPI (Open MPI or MPICH)
- Every build: a BLAS library (e.g. OpenBLAS) providing the standard
Fortran
?gemm/?scal/?axpy/i?amax/?ger/?geru/?swapsymbols, the CBLAScblas_cdotc_sub/cblas_zdotc_subcomplex-dot interfaces, and the LAPACK?getrf/?potrf/?laswp/?geqrf/?geqlf/?larftones. OpenBLAS and MKL carry LAPACK in the same library. Point CMake at it with-DDDLA_CPU_BLAS_LIBRARY=/path/to/libopenblas.so, or leave it unset to fall back tofind_package(BLAS REQUIRED). All builds require both CBLAS DOT interfaces at configuration time; complex dot products always call the library, with no scalar fallback. - CUDA backend: CUDA Toolkit with cuBLAS, cuSOLVER, cuRAND
- HIP backend: ROCm / DTK with hipBLAS, hipSOLVER, hipRAND, and a CMake version ≥ 3.21 for first-class HIP language support
- Host LAPACK: required by every source build.
Use
-DDDLA_LAPACK_LIBRARY=/path/to/libopenblas.sowhen it is not already supplied byDDLA_CPU_BLAS_LIBRARYor found byfind_package(LAPACK). CMake also accepts an existing BLAS/LAPACK library path from theOPENBLASenvironment variable whenDDLA_CPU_BLAS_LIBRARYis unset. - CCL mode: NCCL or RCCL for direct inter-GPU collectives (enable
-DDDLA_USE_CCL=ON)
The current HIP link configuration includes RCCL in its backend library list, so RCCL must be available to HIP builds even when another communication mode is selected.
The build system does not enforce a minimum CUDA/ROCm/NCCL version; use toolchains that are compatible with C++17, the GPU hardware, and the required math/solver libraries.
CUDA dependency discovery prefers CUDA::cublas, CUDA::cublasLt,
CUDA::curand, CUDA::cusolver and CUDA::cudart from FindCUDAToolkit.
Explicit CUBLAS_LIBRARY, CUBLASLT_LIBRARY, CURAND_LIBRARY,
CUSOLVER_LIBRARY and CUDART_LIBRARY paths override those selections.
For a split NVHPC layout, DDLA_CUDA_MATH_LIBRARY_DIR can supply a fallback
math-library directory; standard CUDA searches remain enabled. Configuration
reports DDLA_CUDA_LIBRARY_PROVIDERS for inspection.
Architecture values (CMAKE_CUDA_ARCHITECTURES, CMAKE_HIP_ARCHITECTURES)
are hardware-specific — use the SM number for your NVIDIA GPU (e.g. "80" for
A100, "70" for V100) or the gfx target for your AMD GPU (e.g. "gfx90a" for
MI200 series).
cmake -S . -B build-cpu \
-DDDLA_CPU_BLAS_LIBRARY=/path/to/libopenblas.so \
-DBUILD_TESTS=ON \
-DCMAKE_INSTALL_PREFIX="$PWD/install"
cmake --build build-cpu -j
cmake --build build-cpu --target installThe implementation is organized under src/Eigensolver/one_stage/ and
src/Eigensolver/two_stage/. The public dispatcher and shared operations stay
in src/Eigensolver/.
pheevd reads DDLA_EIGEN_METHOD on every call:
- Unset, empty or
auto(default): select from the measured one-stage/two-stage policy. two_stage: force full → band → tridiagonal.one_stage: force direct Householder reduction to tridiagonal.
Local syevd (real) and heevd (complex) use the same handle-driven
interface as the other single-process solvers:
(handle, jobz, uplo, n, A, lda, w, info). A, w and the int* info are in the
handle's backend memory; GPU callers synchronize before copying device results to the host.
The private check, workspace query and unchecked execution can reuse scratch.
On a single-rank grid, automatic calls use LAPACK SYEVD/HEEVD on CPU or
cuSOLVER/hipSOLVER on GPU directly. The leading active block and UPLO are passed
to the local solver; eigenvalues still return in host memory, and eigenvectors
are copied to Z when necessary. Padding is preserved unless it overlaps active
Z. Exact in-place output needs no matrix copy; other overlapping A/Z layouts
use one compact temporary matrix. Explicit one-stage/two-stage choices retain
their algorithms, including on one rank. In automatic single-rank CPU mode,
full-vector problems of order 512 or more can use the existing one-/two-stage
paths. Real problems below order 4096 retain the local solver if the one-stage
profile is ineligible. Complex eigenvalues-only calls use the two-stage path
from order 2048. Smaller problems, partial-vector CPU calls and GPU calls keep
the local solver. These are routing policies, not a universal speed guarantee.
Automatic selection on multiple ranks is conservative: it considers the scalar type, matrix order,
eigenvalues/eigenvectors request, block/grid layout, cached host thread budget,
shared-memory topology and GPU support. Unmeasured profiles retain two-stage.
Nonempty two-stage tuning options, such as DDLA_EIG_NBW, keep the two-stage
route; common tridiagonal-solver and host-thread controls apply to either method.
Selection performs no matrix allocation and does not change either algorithm's
workspace ownership. See the measured-profile table below for the calibrated
limits; these are heuristics, not a promise that one method always wins.
| CUDA scalar type | Eigenvalues only (N) |
Eigenvectors (V) |
|---|---|---|
| float | two-stage | one-stage for 4096–16384 |
| double | two-stage | one-stage for 3072–16384 |
| complex float | one-stage for 1536–16384 | one-stage for 1024–16384 |
| complex double | one-stage for 1024–16384 | one-stage for 1024–16384 |
These one-stage ranges require one shared-memory domain, eight CUDA SM70 GPUs with NCCL (without host staging), a row-major 2×4 grid, square blocks of64, source coordinates zero, matching output blocks, and one configured host and recognized BLAS worker per rank. Other GPU cases retain two-stage, including HIP, host-staged transport, and orders above16384. CPU thresholds are not inferred from GPU timings: on one shared-memory node, CPU handles use one-stage for full eigenvectors of order below 2048 (any block size, row-major grid, recognized BLAS threads equal to the host budget on every rank; at n=2000 on 40–96 cores it measured 0.3–0.5 s against 0.8–1.0 s real and 0.4–0.9 s against 1.4–1.8 s complex) and two-stage otherwise. Manual method choices remain available everywhere.
For example, export DDLA_EIGEN_METHOD=one_stage selects the one-stage solver
on CPU or GPU. All MPI ranks must select the same method; invalid values and
inconsistent selections are rejected collectively. pheevd_one_stage and
pheevd_two_stage select their algorithm explicitly and ignore this variable.
Both read only the selected input triangle and destroy A. The one-stage solver
completes the other triangle, then uses both halves as workspace for full-matrix
GEMV/GEMM; uplo selects the input triangle, not a separate reduction method.
For partial eigenvectors, DDLA_EIG_TRIDIAGONAL=selected explicitly selects
bisection and distributed inverse iteration during the tridiagonal stage of
both methods. The default is dc (divide and conquer). All ranks must choose
the same policy. Selected applies only to vector calls with 0 < nev < n;
values-only, full-vector and local vendor solves retain their existing routes.
All n eigenvalues remain available. The selected stage stores only the
requested row-distributed vectors, requiring O(n*nev/P + n + P) storage per
rank, including linear tridiagonal scratch and distribution padding. The dense
reduction and backtransform retain their existing distributed storage.
The selected method has bounded iteration and explicitly reports convergence or representability failures; it does not silently allocate a full-vector fallback. Host BLAS work follows the handle's thread budget on CPU and GPU handles, using recognized OpenBLAS/MKL controls and restoring their previous setting. These controls are process-global, so do not overlap unrelated BLAS calls in the same process. Select the method explicitly according to the matrix and requested vector count; no universal speed advantage is assumed.
Initialize MPI with MPI_Init_thread(..., MPI_THREAD_FUNNELED, &provided) and
check that provided >= MPI_THREAD_FUNNELED. Call MPI-using LibDDLA routines
from the thread that initialized MPI. Internal CPU workers perform local
numerical work; communication and shared-window MPI calls stay on the caller.
The C++ API leaves MPI initialization and finalization to the application. The bindings request FUNNELED when they initialize MPI automatically. If MPI is already initialized, its thread level remains the application's responsibility.
The CPU eigensolver (pheevd) scales with threads as well as ranks. The host kernels that BLAS does not
cover (the stage-4 rotation replay, the stage-3 divide-and-conquer loops, the stage-2 chase team) run on
OpenMP threads (DDLA_USE_OPENMP, default ON when OpenMP is found; without it they run serially with
identical results). Rules for a threaded run:
- Set
OMP_NUM_THREADSto the threads per rank;DDLA_EIG_THREADStakes precedence. Without either positive override, each rank derives its budget from its allowed CPU affinity: each CPU contributes one divided by the number of node-local ranks allowed on that CPU, and the total is rounded down (minimum one). Disjoint ten-core bindings retain ten workers per rank; four ranks sharing forty allowed CPUs also receive ten. Partially overlapping masks share only their common CPUs. The budget is fixed whenddlaSetsets up the handle; later environment or affinity changes take effect after anotherddlaSet. - Each CPU vector
pstedcrank calls native LAPACK STEDC once, temporarily using the handle's host-thread budget for BLAS. The previous count is restored on return or exception. GPU local STEDC keeps CPU leaf BLAS serial while its independent small leaves run on host workers; vector updates run on the GPU. - The vendor thread setting is process-global. Run one solver call per process, with no overlapping solves or unrelated concurrent BLAS calls while the guard is active. The guard recognizes OpenBLAS/MKL get/set control pairs; unsupported libraries retain their configuration. The validation below covers OpenBLAS and does not establish a threading guarantee for other backends.
- Bind each MPI rank to its allocated cores. Under Slurm request
Ptasks withTCPUs per task (--ntasks=P --cpus-per-task=T) and usempirun -np P --map-by slot:PE=T --bind-to core. - Results are bitwise independent of the thread count for the library's own loops; a threaded BLAS may change the last bits of the GEMM-based stages, as it does between BLAS builds.
geam(matrix add / transpose-copy) on a CPU handle uses the same per-handle budget fixed byddlaSet; its result is bitwise independent of the thread count.gemmBatchedon a CPU handle uses the same budget: each OpenMP thread computes whole matrix products, so every core handles one multiplication at a time. For the duration of that loop the BLAS library's own thread count (OpenBLASopenblas_set_num_threads, MKLmkl_set_num_threads, found at run time) is held at one and then restored, so even a pthreads OpenBLAS does not start a thread pool inside each call. A single product or a one-thread budget runs serially and leaves the BLAS threads alone.- The CPU two-stage solver can use Householder reflectors or Givens rotations.
With square blocks and band width 64, its automatic Householder path supports
one shared-memory node, n >= 2048, at most 64 ranks and 1–64 host workers per rank.
The existing
DDLA_EIG_CPU_ACCUM=1/=0controls select the Givens path's accumulation or replay engine and bypass automatic Householder selection.
With pthreads OpenBLAS 0.3.29 on HPC3, the following CPU threading setup was validated with one MPI rank bound to eight cores. Use default eigensolver tuning for this example. The native test observed BLAS1 inside parallel STEDC leaves, restoration to BLAS8 after success and exception unwind, correct numerical results, and unchanged affinity. These settings are a tested configuration; performance depends on the matrix, process grid and hardware.
export OMP_NUM_THREADS=8 OPENBLAS_NUM_THREADS=8
export OPENBLAS_MAIN_FREE=1 OMP_PROC_BIND=false
export OMP_DYNAMIC=false OMP_WAIT_POLICY=PASSIVE
# In a Slurm allocation with one task and eight CPUs per task:
mpirun -np 1 --map-by slot:PE=8 --bind-to core ./your_cpu_eigensolverThe GPU eigensolver applies distributed Householder reflectors or Givens rotations
without forming the dense band-to-tridiagonal factor. The tested eight-V100
CUDA/NCCL profile can select Householder automatically; other GPU profiles retain
conservative fallbacks. See the pheevd_two_stage contract
for bandwidth, topology, thread-budget and CPU/GPU stage-2 selection rules.
For multiple ranks with separate, matching compact input/output layouts, it reuses A and Z between
stages and temporarily backs up A in distributed host memory. Padded or different layouts use separate
distributed workspaces. The Givens stage-2 path distributes band cores between nodes
and rotation history between ranks; native GPU Householder stage 2 currently
requires one shared-memory node. Eigenvalues and tridiagonal diagonals remain replicated. A and Z may overlap: the solver preserves
the reflectors needed by its final back-transform.
cmake -S . -B build-cuda \
-DDDLA_USE_CUDA=ON \
-DDDLA_CPU_BLAS_LIBRARY=/path/to/libopenblas.so \
-DCMAKE_CUDA_ARCHITECTURES="80" \
-DBUILD_TESTS=ON \
-DDDLA_USE_CCL=ON \
-DCMAKE_INSTALL_PREFIX="$PWD/install"
cmake --build build-cuda -j
cmake --build build-cuda --target installcmake -S . -B build-hip \
-DDDLA_USE_HIP=ON \
-DDDLA_CPU_BLAS_LIBRARY=/path/to/libopenblas.so \
-DCMAKE_HIP_ARCHITECTURES="gfx90a" \
-DBUILD_TESTS=ON \
-DDDLA_USE_CCL=ON \
-DCMAKE_INSTALL_PREFIX="$PWD/install"
cmake --build build-hip -j
cmake --build build-hip --target installEvery CUDA or HIP build contains both CPU and GPU specializations. No
additional CPU build switch is needed. Legacy caches with DDLA_USE_CPU=OFF
are migrated to CPU-enabled builds during configuration.
The private DdlaStream has the same layout for CPU and GPU handles. Streams,
BLAS/solver handles and CCL communicators occupy opaque void* slots; CPU
handles leave them null. src/ddla_stream.cpp provides backend-templated
resource setup and cleanup in one translation unit. Dual builds compile each
library source once and generate both backend specializations in that pass;
they do not build a second CPU copy of the shared sources.
The installed layout uses include/ddla/*.h, the platform library directory
(normally lib/libddla.so on Linux), and a CMake package under
lib/cmake/LibDDLA. Downstream builds can use:
find_package(LibDDLA REQUIRED)
target_link_libraries(myapp PRIVATE ddla::ddla)The installed ddla::ddla target supplies the include path, MPI dependency
and configured compile definitions. CPU support is unconditional; public headers
obtain GPU availability from <ddla/ddla_config.h> (DDLA_HAS_GPU), so
applications using only the public API do not need private vendor headers.
Every handle requires an explicit backend:
ddlaInit(handle, ddla::DdlaBackend::CPU) for host data or
ddlaInit(handle, ddla::DdlaBackend::GPU) for device data. Public computational
templates take scalar types only and always use the handle's backend; there
is no separate backend template argument to keep in sync. Rebuild C++ clients
after removing the backend template argument, for example using
gemm<double>(handle, ...) with a CPU handle for host data. C, Fortran and Python handle initialization must also select a backend
explicitly; no no-backend initializer remains. The
separate Cholesky cleanup removes the sign/location arguments: regenerate
and rebuild native bindings together with their Fortran/Python consumers.
| Option | Default | Description |
|---|---|---|
BUILD_TESTS |
OFF | Build test executables |
DDLA_USE_CUDA |
OFF | Add NVIDIA CUDA support to the CPU backend |
DDLA_USE_HIP |
OFF | Add AMD HIP/ROCm support to the CPU backend |
DDLA_CPU_BLAS_LIBRARY |
(empty) | Path to a BLAS library for the CPU backend; falls back to find_package(BLAS REQUIRED) if unset |
DDLA_USE_OPENMP |
ON | OpenMP threads for the host kernels of the CPU eigensolver (src/Eigensolver/eig_cpu_kernels.cpp, compiled with the host compiler in every build); OFF or no OpenMP: serial, same results |
DDLA_LAPACK_LIBRARY |
(empty) | Host LAPACK for all source builds; otherwise use DDLA_CPU_BLAS_LIBRARY, the OPENBLAS environment path, or LAPACK discovery |
DDLA_USE_DEBUG |
OFF | Enable DDLA_USE_DEBUG preprocessor macro |
DDLA_USE_CCL |
OFF | Build/link NCCL (CUDA) or RCCL (HIP), making CCL available for GPU handles |
DDLA_ENABLE_FORTRAN |
OFF | Build ddla_c, the ddla_fortran library and Fortran module ddla; requires a Fortran compiler |
DDLA_ENABLE_PYTHON |
OFF | Build ddla_c and stage the Python package; requires a Python 3 interpreter |
Host-staged MPI is available in every GPU build; it needs no build option.
Each handle stores a DdlaCommunication policy independently of its CPU/GPU
backend. CPU handles use MPI. GPU handles support MPI and HostStagedMPI,
plus CCL when built with NCCL/RCCL. GPU handles initially use CCL when
compiled in, otherwise MPI, preserving the previous default when no staging
option was specified. Select HostStagedMPI with the handle setter when MPI
does not support device buffers.
| Policy | GPU behaviour |
|---|---|
DdlaCommunication::MPI |
Synchronize the handle stream, then pass device buffers to MPI |
DdlaCommunication::HostStagedMPI |
Copy device data to host staging buffers, communicate with MPI, then copy results back |
DdlaCommunication::CCL |
NCCL/RCCL device communication, using communicators for the handle's grid |
GPU MPI requires a compatible GPU-aware MPI installation and environment.
ddlaCommunicationAvailable reports library/handle capability; it does not
probe device-pointer support, connectivity or performance. Select host-staged
MPI if that support has not been established. CPU handles reject the two GPU
policies. No rebuild or new handle is needed to switch supported GPU modes:
// After ddlaInit and ddlaSet, every rank in this handle's communicator calls:
if (ddla::ddlaCommunicationAvailable(handle, ddla::DdlaCommunication::HostStagedMPI)) {
ddla::DDLA_CHECK(ddla::ddlaSetCommunication(
handle, ddla::DdlaCommunication::HostStagedMPI));
}
auto current = ddla::ddlaGetCommunication(handle);Before ddlaSet, the setter only stores a valid policy locally. After grid
setup, switching is collective over the handle communicator: all ranks must
call in the same order with the same requested mode, outside an active
communication group and without concurrent operations on that handle. The
setter completes pending handle work before changing transport resources.
Invalid/mismatched requests and active groups report INVALID_VALUE;
unavailable modes report BACKEND_UNAVAILABLE. Rejection preserves the old
policy. Choose the transport before querying/preparing transport-dependent
private workspaces and resources; repeat that preparation after a change.
The selected policy persists across repeated ddlaSet grid reconfiguration.
Changing away from CCL drains the handle queues and destroys its CCL
communicators; switching back recreates them. Host-staging buffers retain
their high-water allocation for reuse. Repeating the same mode still validates
collectively but does not recreate resources. Different handles can retain
different policies in the same library.
The existing src/comm_traits.h primitives dispatch from the handle policy.
The C/Fortran interfaces expose ddla_set_communication,
ddla_get_communication and ddla_communication_available (C symbols have
ddla_c_ prefixes). The getter returns a status and writes the mode through
an integer output. Python exposes Communication, handle.communication,
handle.set_communication(mode) and handle.is_communication_available(mode).
Fortran uses ddla_init_backend(handle, backend). Python requires
Handle(backend=Backend.CPU) or Handle(backend=Backend.GPU); neither has
an implicit backend.
#include <ddla/ddla.h>
#include <mpi.h>
ddla::DdlaHandle_t handle = nullptr; // opaque pointer (DdlaStream*)
ddla::DDLA_CHECK(ddla::ddlaInit(handle, ddla::DdlaBackend::CPU)); // explicitly select CPU and allocate handle
ddla::DDLA_CHECK(ddla::ddlaSet(handle, MPI_COMM_WORLD, 'R')); // create resources and auto grid
// or: ddla::DDLA_CHECK(ddla::ddlaSet(handle, MPI_COMM_WORLD, nprows, npcols, 'R'));ddlaSetstores the process-grid dimensions and communicator on the handle. The automatic form ('R') derives a 2D grid from the number of MPI ranks.- Initialize MPI before setting up the handle. In a dual build,
ddlaInit(handle, ddla::DdlaBackend::CPU)selects the CPU backend; subsequent compute calls use that handle's backend automatically. - Destroy with
ddla::DDLA_CHECK(ddla::ddlaDestroy(handle))when done.
pgemm selects its communication algorithm automatically from the matrix
shape, transpose flags, block sizes, and process grid. The AB branch broadcasts
panels of A and B while keeping C in place. For a thin output, BC keeps A in
place and communicates panels of B and C; AC keeps B in place and communicates
panels of A and C. The latter branches sum partial products to their output
owners, so C retains its ordinary two-dimensional block-cyclic distribution.
All branches support real/complex scalars and N, T, and C operations.
Selection uses ScaLAPACK 2.2.2's communication-volume estimates, with a preference
for AB when estimated costs are close. There is no public branch override.
AB defaults to a 1024-wide K target for NN and one logical A descriptor block
when either operand is transposed/conjugated. A positive DDLA_PGEMM_KPANEL
overrides either default, including wider T/C batching; an unset, empty or
nonpositive value retains the operation-specific default. Existing descriptor
rounding, actual-stage and rank-share limits still bound the allocated panel.
GPU execution uses two one-share panel slots for overlap. Synchronous CPU
execution uses one slot and may combine up to four rank shares when the grid
and descriptor blocks permit. The requested width still caps both paths;
CPU grouping stays below a full operand except for unavoidable single-block
cases. Each panel shrinks with rank count up to descriptor-block rounding.
DDLA_PGEMM_KPANEL controls K-panel batching in AB only; AC and BC
use one output descriptor block per panel, bounding temporary storage.
gemmt (local) and pgemmt (distributed) compute only the selected triangle
of C = alpha * op(A) * op(B) + beta * C. The opposite triangle stays
untouched. A and B are general matrices; the result need not be symmetric or
Hermitian. C uses ordinary column-major storage, not packed triangular storage.
// Lower triangle only; op(A) is n-by-k and op(B) is k-by-n.
ddla::gemmt(h, 'L', 'N', 'N', n, k,
alpha, A, lda, B, ldb, beta, C, ldc);
ddla::pgemmt(h, 'L', 'N', 'N', n, k,
alpha, A_local, descA, B_local, descB, beta, C_local, descC);Both 'U' and 'L' and independent 'N'/'T'/'C' operand flags are supported.
CPU and GPU handles use the same interface with pointers in the matching memory
space. Complex alpha, beta and diagonal entries are unrestricted.
beta=0 reads no old C; alpha=0 or k=0 reads no A/B.
C must not overlap the inputs. The distributed routine supports rectangular
process grids and independent descriptor source coordinates, with a common
square block size and leading logical regions. Matrices remain distributed;
workspace consists of panels rather than replicated full matrices.
trtrm and ptrtrm accept independent input triangles. Upper times lower and
lower times upper produce a full, generally nonsymmetric matrix. Matching
triangles update only the corresponding triangle of C. All diagonal entries
are explicit; unused input triangles need not be initialized.
// Full C = alpha * U * L + beta * C.
ddla::trtrm(h, 'U', 'N', 'L', 'N', n,
alpha, U, ldu, L, ldl, beta, C, ldc);
ddla::ptrtrm(h, 'U', 'N', 'L', 'N', n,
alpha, U_local, descU, L_local, descL, beta, C_local, descC);Both routines use the flag order uplo_a, transa, uplo_b, transb, with
N, T, and C for unchanged, transpose, and conjugate transpose. The uplo
flags describe stored triangles; transposition flips the effective triangle.
The effective triangles determine whether one triangle or all of C is updated.
Distributed ptrtrm redistributes one panel at a time for transposed operands;
its descriptors require a common square block size and matching source
coordinates. Both routines expose only this four-flag interface.
benchmark_products cpu|gpu local|distributed s|d|c|z n [nb=128] [warmups=3] [repeats=9] compares these products and symmetric/Hermitian
products against GEMM/PGEMM. Run it under MPI; local mode requires one rank.
Set DDLA_BENCH_CASES=sym for symmetric/Hermitian cases or tri for
upper-times-lower and lower-times-upper cases; the default all runs both.
Set DDLA_BENCH_GRID=2x2 for an explicit distributed grid. Each sample times
the complete public call, including internal allocation and communication,
with backend synchronization and the slowest-rank elapsed time. Input setup
and correctness checks are outside the timed interval. Raw samples include
both full-output comparisons and independent scalar-reference error checks.
diag_logdet and pdiag_logdet compute the principal natural logarithm of
the product of the diagonal entries, without forming that product:
std::complex<double> logdet; // host output, also for real input
ddla::diag_logdet(handle, n, A, lda, logdet);
ddla::pdiag_logdet(handle, n, local_A, descA, logdet);The handle selects CPU or GPU input storage. Both support float, double and
their complex counterparts, ignore off-diagonal entries and preserve A.
The complex result retains negative/complex determinant phase in (-pi,pi].
Empty input returns zero; a zero diagonal entry returns (-infinity,0);
nonfinite diagonal entries throw without changing the output.
These functions perform no factorization or permutation correction. For an LU factor, they give the logarithm of the product of U's diagonal; the caller must additionally account for row-swap parity and wrap the phase. For a Cholesky factor, the positive-definite matrix's log determinant is twice the real part of the factor's diagonal log determinant. The distributed routine honors rectangular block sizes, source coordinates and padded leading dimensions. Every rank receives the result, using only four host-scalar totals; the matrix is never gathered or copied to the host. CPU reduction uses OpenMP. GPU reduction uses Thrust on the handle stream, with temporary reduction storage bounded by the local iteration range along the more distributed axis, plus block rounding.
Distributed routines describe matrix layouts with a plain length-9 int array
in ScaLAPACK's DESCINIT layout, plus the handle that owns the process grid.
Slot constants live in <ddla/ddla_desc.h>: DDLA_DTYPE_, DDLA_CTXT_,
DDLA_M_, DDLA_N_, DDLA_MB_, DDLA_NB_, DDLA_RSRC_, DDLA_CSRC_,
DDLA_LLD_, DDLA_DLEN_.
Important: desc[DDLA_CTXT_] is ignored. A BLACS context is an index into
a table private to the BLACS library that minted it, so the process grid,
communicators and stream always come from the DdlaHandle_t passed alongside
the descriptor. Every descriptor is validated against that grid, and a
mismatch (usually an LLD_A too small for the local row count) throws
std::invalid_argument.
Important: most factorization and solve routines require square blocks
(mb == nb).
int m = 4096, n = 64;
int desc[ddla::DDLA_DLEN_];
ddla::DDLA_CHECK(ddla::ddlaDescInit(desc, handle, m, n, 64, 64, 0, 0));
// Over-allocated local buffer? Raise the leading dimension afterwards:
// desc[ddla::DDLA_LLD_] += extra_rows;ddlaDescInit fills the descriptor the way DESCINIT does and derives the
tight LLD = max(1, LOCr(M_A)) from the handle's grid.
Free index-mapping helpers are in <ddla/ddla_desc.h>:
indxg2p, indxg2l, indxl2g, num_loc (the ScaLAPACK convention), plus
desc-level indx_g2l_r / indx_g2l_c / indx_l2g_r / indx_l2g_c, each
taking (const int* desc, const DdlaHandle_t& handle, index). Local extents
are not stored in a descriptor; derive them with
num_loc(desc[DDLA_M_], desc[DDLA_MB_], myprow, desc[DDLA_RSRC_], nprows)
after fetching the grid with ddlaGetGridDims / ddlaGetGridCoords.
Each MPI rank owns a contiguous local submatrix of size m_loc × n_loc stored
in host memory for a CPU handle or device memory for a GPU handle. The caller
allocates and frees this memory, with helper calls taking the handle first.
#include <ddla/ddla.h>
#include <mpi.h>
int main(int argc, char** argv) {
int provided = MPI_THREAD_SINGLE;
if (MPI_Init_thread(&argc, &argv, MPI_THREAD_FUNNELED, &provided) != MPI_SUCCESS)
return 1;
if (provided < MPI_THREAD_FUNNELED) {
MPI_Abort(MPI_COMM_WORLD, 1);
return 1;
}
ddla::DdlaHandle_t handle = nullptr;
ddla::DDLA_CHECK(ddla::ddlaInit(handle, ddla::DdlaBackend::CPU));
ddla::DDLA_CHECK(ddla::ddlaSet(handle, MPI_COMM_WORLD, 'R'));
int n = 4096, nrhs = 64;
int descA[ddla::DDLA_DLEN_], descB[ddla::DDLA_DLEN_];
// Square blocks (mb == nb): use block size 64 for both matrices.
ddla::DDLA_CHECK(ddla::ddlaDescInit(descA, handle, n, n, 64, 64, 0, 0));
ddla::DDLA_CHECK(ddla::ddlaDescInit(descB, handle, n, nrhs, 64, 64, 0, 0));
int nprows = 0, npcols = 0, myprow = -1, mypcol = -1;
ddla::ddlaGetGridDims(handle, nprows, npcols);
ddla::ddlaGetGridCoords(handle, myprow, mypcol);
const long m_locA = ddla::num_loc(n, 64, myprow, 0, nprows);
const long n_locA = ddla::num_loc(n, 64, mypcol, 0, npcols);
const long m_locB = ddla::num_loc(n, 64, myprow, 0, nprows);
const long n_locB = ddla::num_loc(nrhs, 64, mypcol, 0, npcols);
double *d_A = nullptr, *d_B = nullptr;
ddla::DDLA_CHECK(ddla::ddlaMalloc(handle, &d_A, m_locA * n_locA * sizeof(double)));
ddla::DDLA_CHECK(ddla::ddlaMalloc(handle, &d_B, m_locB * n_locB * sizeof(double)));
// ... fill matrices, call ddla routines ...
ddla::DDLA_CHECK(ddla::ddlaFree(handle, d_A));
ddla::DDLA_CHECK(ddla::ddlaFree(handle, d_B));
ddla::DDLA_CHECK(ddla::ddlaDestroy(handle));
MPI_Finalize();
return 0;
}The ddlaMalloc / ddlaFree / ddlaMemcpy / ddlaMemcpy2D helpers declared
in <ddla/ddla_handle_t.h> (pulled in by <ddla/ddla.h>) all take the handle
as their first argument and dispatch to the handle's backend (host memory for
CPU handles, device memory on the handle's stream for GPU handles). Zero-byte
requests are safe (a zero-byte ddlaMalloc sets *ptr = nullptr).
ddlaMemcpy(handle, dst, src, bytes) copies a contiguous range and
ddlaMemcpy2D(handle, dst, dpitch, src, spitch, width, height) copies a
pitched rectangle (pitches and width in bytes, as in cudaMemcpy2D). There is
no direction argument: a CPU handle copies host to host, and a GPU handle asks
the GPU runtime where each pointer lives (host memory or device memory, managed
memory counts as device) and chooses host to host, host to device, device to
host or device to device itself. A host-to-host copy is a plain memcpy that
runs immediately on the calling thread and is not ordered after the handle's
stream; the other directions are enqueued asynchronously on the handle's
stream, so call ddlaSynchronize(handle) before using a device-to-host
result.
ddlaMemcpyTri(handle, uplo, m, n, dst, ldd, src, lds, include_diagonal=true)
copies only a local column-major upper/lower triangle. Leading dimensions are
in elements; the other triangle and padding remain unchanged. The matching
ddlaMemsetTri(handle, uplo, m, n, ptr, lda, value=0, include_diagonal=true)
repeats a byte value, just like ddlaMemset (use zero for numerical zero).
Pass false to preserve the diagonal. Both accept trivially copyable element
types, including real and complex scalars. Large host operations use the
handle's OpenMP thread budget; GPU fills and device-to-device copies use one
kernel on the handle stream without scratch allocation. Host/device triangular
copies enqueue selected contiguous column segments, with the same pointer
direction and synchronization rules as ddlaMemcpy. GPU fills require device
or managed memory. These are local memory helpers, not distributed operations
or transpose/conjugation routines.
The same family covers the rest of what the library does with the device, so that no CUDA or HIP type appears in your code either:
ddlaMallocalso takes a typed pointer (double* p; ddlaMalloc(handle, &p, bytes)).ddlaMemset(handle, ptr, value, bytes)fills device memory (host memory on a CPU handle).ddlaMallocHost/ddlaFreeHostallocate page-locked host memory, the source and target of true asynchronous copies.ddlaEventCreate/ddlaEventDestroy/ddlaEventRecord/ddlaEventWait/ddlaEventSynchronizemark a point in the handle's queue and order work against it.ddlaGetLastError(handle)is the check after your own kernel launch,ddlaSetDevice(handle)selects the handle's device on a helper thread,ddlaGetDeviceCount/ddlaMemGetInfoquery the device.
All of them return a ddlaStatus_t; DDLA_CHECK(status) throws on anything but
DDLA_STATUS_SUCCESS, with the vendor error text from ddlaGetErrorString
for a device failure. On a CPU handle the event calls are no-ops.
Copies also take the handle first: ddlaMemcpy(handle, dst, src, bytes) and
ddlaMemcpy2D(handle, dst, dpitch, src, spitch, width, height). There is no
copy-direction argument. GPU copies involving device memory are queued on
the handle; call ddlaSynchronize(handle) before reading their results on
the host. Pitches and copy sizes are in bytes.
| Symbol | Description |
|---|---|
DdlaHandle_t |
Opaque handle type (DdlaStream*) |
ddlaInit |
Allocate a handle and select its backend |
ddlaSet |
Configure the grid (auto 'R' or explicit nprows×npcols) and create backend resources |
ddlaSetCommunication / ddlaGetCommunication / ddlaCommunicationAvailable |
Set/query a handle's transport and query its supported modes; switching after grid setup is collective |
ddlaGetBackend / ddlaBackendAvailable |
Query the handle's backend / compiled backend availability |
ddlaDestroy |
Tear down handle (idempotent) |
ddlaGetStream |
Return the default device stream from a handle (CPU: nullptr) |
ddlaMalloc / ddlaFree |
Allocate / free backend memory: ddlaMalloc(handle, &ptr, bytes), ddlaFree(handle, ptr) |
ddlaMemcpy / ddlaMemcpy2D |
Contiguous / pitched copy: ddlaMemcpy(handle, dst, src, bytes), ddlaMemcpy2D(handle, dst, dpitch, src, spitch, width, height); the direction is derived from the handle and the pointers |
ddlaMemset / ddlaMallocHost / ddlaFreeHost |
Fill device memory / allocate and free pinned host memory |
ddlaEventCreate / ddlaEventDestroy / ddlaEventRecord / ddlaEventWait / ddlaEventSynchronize |
Mark a point in the handle's queue and order work against it |
ddlaGetLastError / ddlaSetDevice / ddlaGetDeviceCount / ddlaMemGetInfo / ddlaGetErrorString |
Launch-error check, device selection and queries, status text |
ddlaSynchronize |
Wait for the handle's stream (needed after an async DeviceToHost copy) |
ddlaDescInit |
Fill a ScaLAPACK-layout int[9] descriptor (DESCINIT equivalent) |
int[9] descriptor |
2D block-cyclic matrix layout, slots DDLA_DTYPE_ … DDLA_LLD_ |
ddlaGetGridDims / ddlaGetGridCoords |
Process grid of a handle |
indxg2p / indxg2l / indxl2g / num_loc |
Free index-mapping functions |
indx_g2l_r / indx_g2l_c / indx_l2g_r / indx_l2g_c |
Desc-level index mapping (desc, handle, index) |
| Function | Description |
|---|---|
pgemm |
C = α·op(A)·op(B) + β·C; automatic AB/SUMMA, AC, or BC communication |
pgeadd |
C = α·op(A) + β·op(B) with required one-based ia,ja, ib,jb, ic,jc. Supports independent block sizes, within-block offsets, and owners for N/T/C. Matching layouts retain direct local addition and whole-slab redistribution, including transpose on square grids (m*n <= INT_MAX). Other layouts use block-intersection redistribution. |
pdam |
Add scalar to the diagonal of a distributed matrix (leading-block, optional n; default -1 = whole matrix) |
ptrmm |
In-place triangular multiply: B = α·op(A)·B or B = α·B·op(A) |
gather / scatter |
Move a submatrix between distributed storage and a column-major buffer on one root rank; explicit one-based origins, host buffers on CPU and device buffers on GPU |
Local calls perform no inter-rank communication. Their column-major matrices must live in the handle's memory space, and GPU work uses the handle's stream.
| Function | Description |
|---|---|
gemm |
C = α·op(A)·op(B) + β·C; also has a real-A × complex-B overload with complex C, α and β |
trmm |
Overwrite B with α·op(A)·B or α·B·op(A) |
trtrm |
C = α·op(A)·op(B) + β·C with independent triangle and transpose flags for A and B |
symm / hemm |
Symmetric / Hermitian matrix multiplication |
symmt / hemmt |
Symmetric / Hermitian A times general B (or B times A), updating only the selected triangle of C |
syrk / syr2k / herk / her2k |
Symmetric / Hermitian rank-k and rank-2k updates |
gemmBatched |
Fixed-size GEMMs over arrays of pointers; arrays and matrices use the handle's memory space |
trtrm(handle, uplo_a, transa, uplo_b, transb, n, alpha, A, lda, B, ldb, beta, C, ldc)
reads the U or L stored triangle of each operand; transa and transb select
N, T or C. Both diagonals are explicit. Matching effective triangles update
only that triangle of C; mixed effective triangles update all of C. The unused
triangle of C and leading-dimension padding are preserved. A and B may coincide,
but C must not overlap either input. The previous single-uplo interface is
replaced by passing uplo, 'N', uplo, 'N'.
symmt(handle, side, uploA, uploC, n, alpha, A, lda, B, ldb, beta, C, ldc)
computes the selected U or L triangle of C = alpha*A*B + beta*C
(side='L') or C = alpha*B*A + beta*C (side='R'). All matrices are
n-by-n. Only uploA of symmetric A is read; B is general. Complex symmetric
A is reflected by transpose and retains its complex diagonal. hemmt uses
conjugate reflection and a real Hermitian diagonal instead. The other triangle
and padding of C are preserved. C must not overlap A or B.
psymmt(handle, side, uploA, uploC, n, alpha, A, descA, B, descB, beta, C, descC)
is the distributed counterpart, with block-cyclic descriptors and the same
layout requirements as psymm; C uses square blocks. It shares phemmt's
triangle-only update schedule. Both routines support CPU and GPU backends and
all four scalar types. Zero alpha skips A/B; zero beta skips reading the selected
C entries; n=0 or alpha=0,beta=1 permits null data pointers.
| Function | Description |
|---|---|
pgetrf |
LU with partial (row) pivoting on an explicit one-based ia,ja submatrix |
pgetrf_bpiv |
Block LU with partial pivoting per block row (device pivot array) |
pgetrf_nopiv |
Multi-process LU without pivoting |
getrf_nopiv |
Local (single-process) LU without pivoting |
pgetrs |
Solve using pivoted LU factors: op(A)·X = B (side='L') or X·op(A) = B (side='R'), trans='N'/'T'/'C' |
pgetrs_nopiv |
Solve using no-pivot LU factors (same side/trans options) |
pgetrs_bpiv |
Solve using block-LU factors from pgetrf_bpiv (side, trans) |
pgesv |
Driver: LU + solve with pivoting (side, trans) |
pgesv_nopiv |
Driver: LU + solve without pivoting (side, trans) |
pgesv_bpiv |
Driver: block-LU + solve (side, trans) |
ptrtrs |
Distributed triangular solve (side × uplo × trans × diag) |
plapiv |
Apply pivot permutation to rows or columns, forward or backward |
pswap |
Swap rows or columns between two distributed matrices |
pgetrf uses a private blocked panel factorization, pgetf2; it is not a
public API. DDLA_LU_PANEL=inplace (default) or gather selects the panel
method, and all ranks must use the same setting.
pgetrf, pgetrs and public plapiv take pivot arrays in the selected
backend: host memory for CPU handles and device memory for GPU handles.
Allocate them with ddla::vector<int>(handle, local_rows) (local rows of the
full factor descriptor). Entries remain one-based global row indices,
distributed by process row and replicated across process columns. Use
ddlaMemcpy plus synchronization only when host code needs to inspect them.
pgesv owns its pivots in backend workspace. GPU panel/control logic still
uses small host metadata; no global matrix replica is introduced.
| Function | Description |
|---|---|
ppotrf |
Standard Cholesky factorization (all four scalar types) |
ppotrs |
Solve using Cholesky factor (side='L'/'R'; trans='N'/'C') |
pposv |
Driver: Cholesky + solve (side='L'/'R'; trans='N'/'C') |
potrf_bottom_right |
Local bottom-right Cholesky (all four scalar types) |
ppotrf_bottom_right |
Distributed bottom-right Cholesky (all four scalar types) |
| Function | Description |
|---|---|
pgeqrf(handle, m, n, ...) |
Distributed Householder QR, A = Q·R; all four scalar types, with host tau indexed by local column |
pgeqlf(handle, m, n, ...) |
Distributed Householder QL, A = Q·L; host tau uses local-column indices, including the final reflector columns of wide matrices |
pgelqf(handle, m, n, ...) |
Distributed Householder LQ, A = L·Q; host tau indexed by local row |
pgerqf(handle, m, n, ...) |
Distributed Householder RQ, A = R·Q; host tau indexed by local row, including the final reflector rows of tall matrices |
Panels use a gathered local geqrf by default (DDLA_QR_PANEL=gather),
or the distributed inplace variant. All ranks must use the same selector.
Local geqrf (QR/QL selected by uplo), gelqf and gerqf are also
available. Local reflector scalars reside in the handle-selected backend, while distributed reflector
scalars are host arrays. LQ/RQ use an additional distributed adjoint buffer;
they do not gather or replicate a full global matrix.
| Function | Description |
|---|---|
pgesvd |
Driver: A = U·diag(s)·V^H (ScaLAPACK PxGESVD semantics: economy U m×k and VT k×n, jobu/jobvt = 'V'/'N', host s descending and replicated; U's rows aligned with A's, VT's columns aligned with A's; handle-selected backend, all four scalar types; the bidiagonal vector stage uses row-distributed divide and conquer on CPU and GPU, with O(k²/P) factor storage per rank and bounded leaf solves; DDLA_SVD_DC_LEAF selects the leaf size, default 128, capped at 256; values-only jobs use O(k) dqds storage. The obsolete DDLA_SVD_BDSOLVER selector is ignored; replicated lapack/qr modes have been removed) |
pbdsdc |
Bidiagonal divide-and-conquer SVD B = U·diag(s)·V^T, upper/lower real bidiagonal, float/double, CPU/GPU. Host d, e, s are replicated; s is descending. Both U and V return contiguous row slabs over all communicator ranks with local leading dimension ld; no descriptor-output or replicated-factor variant. Matrix-sized storage is distributed; bounded panels and scalar vectors may be replicated. |
pgebrd |
Householder bidiagonalization A = Q·B·P^H (ScaLAPACK PxGEBRD; upper bidiagonal for m >= n, lower for m < n; LAPACK ?GEBRD storage; host d, e, tauq, taup of global length, replicated) |
punmbr |
Apply Q/Q^H from the left or P/P^H from the right (ScaLAPACK PxUNMBR; vect='Q' with side='L', vect='P' with side='R') |
pgesvd uses direct bidiagonal reduction by default. DDLA_SVD_SHAPE=direct
(also unset or empty) retains that path. Set DDLA_SVD_SHAPE=qr explicitly to
permit the QR-based tall/wide reduction when the aspect ratio is at least four
and the existing distributed-memory eligibility checks pass. Ineligible layouts
and single-rank CPU calls keep the direct path. QR can change both workspace
and performance; it is not selected automatically. All ranks must select the
same effective route. Invalid selector values are rejected collectively before
native workspace preparation or matrix changes. The checked route is frozen
for private sizing and raw execution; changing the environment does not change
an already checked plan.
Solve semantics. For side='L' the right-hand side B is n × nrhs and
the system is op(A)·X = B; for side='R' B is nrhs × n and the system is
X·op(A) = B. The LU-family solves (pgetrs, pgetrs_nopiv, pgetrs_bpiv,
pgesv, pgesv_nopiv, pgesv_bpiv) accept trans = 'N', 'T' or 'C'. The Cholesky solves
(ppotrs, pposv) accept trans = 'N' or 'C', which are equivalent for a
Hermitian matrix ('T' is not supported). plapiv applies the pivot
permutation to either rows (rowcol='R') or columns (rowcol='C'), in
forward (direc='F') or backward (direc='B') order. Cholesky factorization
reports a nonpositive pivot through info; the driver solves only when
factorization succeeds. The factor passed to ppotrs is read-only.
| Function | Description |
|---|---|
swap |
Swap two vectors (BLAS-1 ?swap; strides may be negative) |
getrf |
Local LU with partial pivoting, A = P·L·U (LAPACK ?getrf; ipiv and info live with the data) |
potrf |
Local Cholesky factorization, uplo = 'U'/'L' (LAPACK ?potrf; use the selected factor; DTK caveats apply to the opposite triangle) |
laswp |
Local row interchanges from a getrf pivot array (LAPACK ?laswp; any nonzero incx) |
geqrf(handle, uplo, m, n, A, lda, tau) |
Local QR / QL factorization, uplo = 'U' (A = Q·R, LAPACK ?geqrf) or 'L' (A = Q·L, LAPACK ?geqlf; cuSOLVER / hipSOLVER have no QL, so the GPU runs the vendor geqrf on the reversed matrix J A J); tau lives with the data |
larft(handle, direct, storev, n, k, V, ldv, tau, T, ldt) |
Local triangular factor T of a block reflector (LAPACK ?larft, storev = 'C'): H = I - V·T·V^H for k reflectors stored as geqrf / geqlf leave them, direct = 'F' (T upper, H = H(1)…H(k)) or 'B' (T lower, H = H(k)…H(1)); T is written whole (zeros in the other triangle). The GPU needs no vendor routine (cuSOLVER has Xlarft only from CUDA 12.4, hipSOLVER none): herk forms V^H V and one small kernel runs the column recurrence |
gemmVbatched(handle, ...) |
Batch of GEMMs with variable dimensions; homogeneous GPU calls and mixed real/complex CPU/GPU overloads |
gemmVbatched2s(handle, ...) |
Two-stage variable-batch GEMM with reusable temporary; homogeneous GPU calls and mixed-domain CPU/GPU overloads |
random_generate(handle, data, n) |
Fill a buffer with uniform random values |
swap, getrf, potrf, laswp, geqrf, larft, random_generate,
gemm, geam and write_matrix are scalar templates declared in
<ddla/ddla.h>. These compute and matrix-output calls take the handle first;
backend selection comes from that handle. For example,
write_matrix(handle, A, m, n, "matrix.txt") consumes host A on a CPU handle
or device A on a GPU handle. Use a CPU handle when writing a host reference
array, even in a GPU-enabled program. Supported instantiations cover float,
double, std::complex<float> and std::complex<double>, subject to each
routine's real/complex constraints.
Pointers use the memory domain documented by each routine. Local BLAS/LAPACK
buffers live with the handle's backend -- host memory on CPU, device memory on
GPU -- including local ipiv and info. GPU calls are asynchronous on the
handle's stream, so read info after ddlaSynchronize. Distributed routines
may document host metadata separately. gemmVbatched / gemmVbatched2s also
use the handle-driven scalar interface. Homogeneous overloads require a GPU
handle and reject CPU handles with std::invalid_argument. Mixed-domain
overloads support CPU and GPU; dimension and pointer arrays live in the
handle-selected memory domain.
Built with -DDDLA_ENABLE_FORTRAN=ON (needs a Fortran compiler, off by
default): ddla_c contains the C-ABI bridge (fortran/ddla_c_api.cpp), and
ddla_fortran supplies the Fortran module (fortran/ddla.f90, bind(C)
interfaces) and links to that bridge. Installing places both libraries next
to libddla and ddla.mod under include/ddla. The signature table in
fortran/gen_bindings.py generates the C-ABI, Fortran and Python ctypes files.
Enable DDLA_ENABLE_PYTHON to build the bridge and stage the Python package;
see Python setup and examples.
Routines keep their names under a LAPACK-style type prefix: ddla_s_*,
ddla_d_*, ddla_c_* and ddla_z_* take real(c_float), real(c_double),
complex(c_float) and complex(c_double) data (ddla_d_pgemm,
ddla_z_pgesv, ...). There are deliberately no generic names: most routines
pass their matrices as type(c_ptr), which a Fortran generic cannot resolve
by scalar type. Every binding returns integer(c_int) status, 0 = success
(ddla_check(status) aborts with the text from ddla_error_text(status));
a C++ exception from the library surfaces as DDLA_STATUS_EXCEPTION with its
what() text.
The ScaLAPACK block-cyclic data model carries over unchanged: ddla_descinit
fills the same integer(c_int) :: desc(9) array ScaLAPACK's DESCINIT would
(ddla_numloc is the LOCr/LOCc helper), the process grid comes from the
handle (ddla_init + ddla_set(handle, fcomm, nprows, npcols, 'R'), with
0, 0 requesting the automatic grid), and the local pieces are plain Fortran
arrays -- c_loc(array) on a CPU handle, ddla_malloc/ddla_memcpy device
buffers on a GPU handle. Complex scalars travel as the bind(C) derived
types ddla_complex_float / ddla_complex_double (the layout of
complex(c_float) / complex(c_double)).
use ddla
include 'mpif.h'
type(c_ptr) :: h
integer(c_int) :: st, desca(9)
st = ddla_init_backend(h, DDLA_BACKEND_CPU)
st = ddla_set(h, int(MPI_COMM_WORLD, c_int), 0, 0, 'R')
st = ddla_descinit(desca, h, n, n, mb, mb, 0, 0, -1)
st = ddla_d_pgesv(h, 'L', 'N', n, nrhs, pa, desca, pb, descb)Bound: the distributed family (pgemm, pgetrf + _nopiv/_bpiv,
pgetrs + _nopiv/_bpiv, pgesv + _nopiv/_bpiv, ppotrf, ppotrs,
pposv, ppotrf_bottom_right, potrf_bottom_right, pgeqrf, pgeadd,
pdam, ptrtrs, ptrmm, plapiv, pswap, gather, scatter), the local
BLAS/LAPACK wrappers (gemm, gemv, trsm, trmm, trmv, syrk, herk,
her2k, hemv, hemm, her2, geam, trtrm, scal, axpy, swap,
geru, gerc, iamax, nrm2, dotc, getrf, potrf, laswp, geqrf,
larft) and the lifecycle / descriptor / memory calls (ddla_init,
ddla_init_backend, ddla_set, ddla_destroy, ddla_grid_dims,
ddla_grid_coords, ddla_rank, ddla_size, ddla_descinit, ddla_numloc,
ddla_malloc, ddla_free, ddla_memcpy, ddla_memcpy2d, ddla_memset,
ddla_synchronize). Not bound: the pointer-array APIs (gemmBatched,
gemmVbatched, gemmVbatched2s), the utilities
random_generate / write_matrix, and the mixed real-by-complex gemm
overload -- call those from C++.
tests/test_fortran_api.f90 registers as the test_fortran_api_mpi ctest
when the option is on: handle lifecycle, descriptors, memory round trips, and
pgemm / pgesv / ppotrf+pdam+ppotrs / pgeqrf residual checks for
real and complex data on an explicitly selected backend (device buffers +
ddla_memcpy on a GPU handle).
Both local trmm and distributed ptrmm overwrite B, following BLAS/PBLAS:
trmm<T>(handle, side, uplo, trans, diag, m, n,
alpha, A, lda, B, ldb);
ptrmm<T>(handle, side, uplo, trans, diag, m, n,
alpha, A, descA, B, descB);A is read-only and must not overlap B. The unused triangle and implicit
unit diagonal are not referenced. With alpha == 0, the logical entries of
B become zero without reading A or the original B; padding is preserved.
The former local trmm(..., B, ldb, C, ldc) interface has been removed.
Callers that need to preserve B must first copy it to their destination,
then call in-place trmm on that destination. Rebuild C++ clients and migrate
C, Fortran, and Python calls to the same shorter argument list. The distributed
ptrmm signature is unchanged. Internal QR, SVD, triangular-product and
Householder consumers use their existing work buffers in place.
Build with -DBUILD_TESTS=ON. Test executables are MPI programs and must be
launched with mpirun. The total number of ranks must equal the product of
process-grid rows and columns (nprows × npcols), which defaults
automatically. You can override the grid with --grid:
# Run a single test on 4 ranks (auto grid)
mpirun -np 4 build-cuda/tests/test_random_generate
# Explicit 2×2 grid for a test that supports --grid
mpirun -np 4 build-cuda/tests/test_api_grid_ptrtrs --grid 2x2The files tests/test_cuda.sh and tests/test_hip.sh are cluster-specific
Slurm batch scripts. They load modules, build, and run the test suite on
particular machines; adapt their module/partition settings to your own cluster.
On Slurm, request the MPI tasks before launching tests and keep the launched
rank count within that allocation. CTest's PROCESSORS property counts each
test's ranks; limit concurrent tests to the allocated rank budget. Single-node
CPU performance measurements should use all physical cores (for example,
4 ranks × 10 cores on HPC3 p1, or 4 × 12 on its 48-core nodes). Correctness
CI can use a smaller allocation.
The HPC3 validation guide describes the CPU and four-/six-GPU jobs, their resource requests and result logs. The KSSG validation guide describes the manual HIP workflow.
To check an installation independently of the source build, run
tests/install_consumer/check_install.sh <build-dir> <new-scratch-dir> in the
validation allocation. It installs and relocates a temporary prefix, then uses
only find_package(LibDDLA) and ddla::ddla to build a separate C++ consumer.
The one-rank smoke checks the installed capability header and CPU computation;
a dual installation also links a GPU specialization without initializing a GPU.
See the consumer fixture for audit outputs.
include/ddla/ddla.h owns the documented public signatures. The scalar and
backend routing policies live in src/public_api_manifest.json. After adding
an interface and its backend implementation, update that policy and run:
python3 tools/generate_public_api.py
python3 tools/generate_public_api.py --check
python3 tests/test_public_api_generation.pyThe generator writes src/backend_api.h, src/public_dispatch.cpp and derived
manifest metadata. Keep exceptional routes explicit in the generator; do not
hand-edit its output. Python is needed for development checks, not for ordinary
library compilation or installed C++ clients.
LibDDLA/
├── include/ddla/ Public headers
│ ├── ddla.h Main API declarations (all Doxygen comments)
│ ├── ddla_handle_t.h Handle type, init/set/destroy, memory & stream helpers
│ ├── ddla_desc.h ScaLAPACK int[9] descriptor contract and index mapping helpers
│ ├── ddla_vector.h Owned, uninitialized storage in the handle's memory space
│ └── ddla_config.h.in Generated config header template (ddla_config.h at build time)
├── src/ Implementation (one routine per .cpp) and private headers,
│ grouped per algorithm: LU/ Cholesky/ LDLT/ QR/ Eigensolver/ SVD/
│ BLAS/ PBLAS/ (plus gpu/ and vbatched/); shared
│ infrastructure (comm_traits.h, ddla_connector.h, ...) at root
│ └── comm_traits.h Unified communication layer (commSend/commRecv/
│ commBcast/commAlltoallv)
├── fortran/ Fortran bindings: C-ABI shim, bind(C) module, signature-table generator
├── tests/ Integration tests (MPI executables)
├── benchmarks/ Benchmark data and supporting artifacts
├── install_scripts/ Backend-specific build/install scripts
├── .ci/ Cluster CI control files (hpc3: CUDA, kssg: HIP/DCU)
├── cmake/ CMake helper modules
├── CMakeLists.txt Top-level build
└── LICENSE LGPL-3.0 license text
Implementation headers are private under src/. BLAS and LAPACK entry
checks and execution helpers are declared in src/BLAS/blas_impl.h and
src/lapack_impl.h; callers use the public functions in <ddla/ddla.h>.
The stream implementation, workspace helpers and vendor aliases also stay
private. Operation sources are grouped under src/BLAS, src/PBLAS,
src/LU, src/Cholesky, src/LDLT, src/QR, src/Eigensolver,
src/SVD and src/vbatched.
CPU support is unconditional. The generated DDLA_HAS_GPU macro gates GPU
code in the public headers and library sources. Private DDLA_USE_CUDA /
DDLA_USE_HIP definitions select the vendor API in src/gpu/gpu.h; a GPU
library build supplies exactly one. CPU-only builds use no dummy GPU types
or no-op vendor calls. Consumers use the public headers and can select the
CPU backend of a GPU-enabled library without vendor definitions.
For an independent local tridiagonal solve, use
stedc<T>(handle, compz, n, d, e, Z, ldz, info) with real or complex
single/double precision output. compz='N' returns eigenvalues only;
'I' writes tridiagonal eigenvectors; 'V' replaces the input orthogonal
or unitary Z with Z times those eigenvectors. d, e, and info are
host data on both backends; Z is in handle-selected memory. d is
replaced by ascending eigenvalues, e is destroyed, and leading-dimension
padding is preserved. The call completes before returning and performs no
MPI communication. CPU uses native LAPACK STEDC; GPU uses hybrid divide and
conquer with host LAPACK leaves/secular roots and device eigenvector updates.
Values-only uses host LAPACK on either backend. GPU handles must first be
initialized with ddlaSet, as for local BLAS operations.
pstedc<T>(handle, jobz, n, nblk, d, e, w, z, ldz, info) has two modes:
jobz='N' computes eigenvalues only; jobz='V' also returns eigenvectors as
contiguous row slabs in backend memory. For communicator rank q among P ranks,
local_rows = n/P + (q < n%P) and the first global row is
q*(n/P) + min(q,n%P). Each rank stores all n columns with
ldz >= max(1,local_rows). Column j corresponds to w[j]. Leading-dimension
padding and columns beyond n are preserved; zero-row ranks may pass null z.
The replicated host vectors d and e are destroyed, and w receives ascending
eigenvalues on every rank. For 'N', z and ldz are ignored and z may be null.
The former descZ overload and its automatic row-to-2D redistribution have
been removed. Callers must allocate the row layout described above.
With one rank, vector pstedc uses local stedc on the selected backend.
With multiple ranks, every nonempty rank solves one contiguous local problem
through the backend-selected stedc implementation, then participates in the
row-distributed divide-and-conquer merges. The required positive nblk balances
contiguous descriptor blocks: n=1000, nblk=128 and two ranks give 512/488.
If blocks are too large to balance local factors within the existing row slabs,
the solver uses equal contiguous row cuts. Local output and factors reuse the
outer scratch slabs, and bounded host panels scatter only rows belonging to other ranks.
No full global eigenvector matrix is gathered or replicated. DDLA_EIG_DC_LEAF
is no longer used. The local GPU engine and eigenvalues-only tree keep internal
bounded leaf sizes; nblk does not set that numerical grain.
The Hermitian one-stage and two-stage drivers use the same row-distributed D&C solver. Each rank owns a contiguous share of rows and all columns, keeping eigenvector storage distributed as O(n²/P). One-stage redistributes bounded real panels into the output's 2D descriptor, widens locally for complex output, and applies its stored reflectors. Two-stage feeds the row slabs directly to its replay, Householder, or accumulated-Q back-transform. Both drivers return eigenvectors in the caller's 2D output descriptor and reuse caller-sized workspace across these phases. No dense secular factor is replicated.
The GPU Householder route supports native host or GPU band-to-tridiagonal reduction and a device back-transform. Reflector history is partitioned across MPI shared windows on one node; each GPU receives bounded panels. Narrow panels use register windows, wider panels use cooperative warps, and a GPU BLAS/WY reference supports comparisons.
Automatic Householder selection is scoped to the measured CUDA SM70/NCCL
profile: one node, eight ranks in a row-major 2×4 grid, one host worker per
rank and block size 64, with full eigenvectors. Double defaults to
Householder/band64 from n=4096; complex double from n=16384. Stage 2 uses CPU
below n=16384 and GPU from n=16384. An explicit DDLA_EIG_NBW=64 also
permits complex double from n=8192; explicit band16 selects Householder only
for double from n=32768. Other widths, scalar types, eigenvalues-only calls
and GPU profiles keep their existing policy. These thresholds describe this
scoped dispatch policy, not a performance guarantee on other hardware.
For CPU two-stage solves, the default band is a divisor of the block size. The selector first looks for the largest divisor from 32 through 64 (for example, 64 for blocks of 128 or 256, and 50 for blocks of 100), then the smallest divisor above 64 through 128, then the largest divisor from 16 through 31. Smaller blocks keep their own width. The distributed band limit can reduce the selected width.
Automatic Householder chasing on CPU requires order at least 2048, a band of at least 16 that divides the block size, and one shared-memory node with at most 128 ranks and 64 host workers per rank. It applies to eigenvalues-only solves and solves requesting all eigenvectors. Explicit engine choices and legacy chase settings take precedence.
test_pheevd_householder_gpu checks numerical and layout behavior. Private
comparison entry points are the backend-templated detail::pheevd_givens,
detail::pheevd_householder and detail::pheevd_householder_stage4, declared
in src/Eigensolver/two_stage/pheevd_impl.h; the reduction-backend argument
selects CPU or GPU stage 2 independently of matrix storage.
Panel extraction and redistribution use pgeadd followed by row/column
broadcasts. pgeadd accepts independent block sizes, owners, and submatrix
offsets for N/T/C. Aligned panel copies retain the direct packing path.
The former public transport_block interface has been removed.
pheevd, pheevd_one_stage and pheevd_two_stage require nev after
n, following ELPA's count-based convention. They return all n eigenvalues
in ascending order and the eigenvectors for the lowest nev eigenvalues:
ddla::pheevd<double>(handle, 'V', 'U', n, nev,
A, descA, eigenvalues, Z, descZ, info);Use 0 <= nev <= n. For vectors, descZ must cover n rows and nev
columns with the method's existing block/distribution requirements; an exact
n × nev descriptor or a larger parent descriptor is supported. Only the
leading output block is written. Padding and remaining parent columns are
preserved unless they overlap A, whose active n × n block is destroyed.
nev=0 skips vectors and ignores Z/descZ. jobz='N' returns only
values regardless of the valid nev count. Pass nev=n for full output.
The old signatures without nev have been removed; downstream callers must
update their calls and rebuild.
Reduction and the current distributed tridiagonal divide-and-conquer solver
still compute the full spectrum. Redistribution and both back-transforms
process only the requested vectors; this is not an eigenvalue-range solver.
ScaLAPACK's *syevx/*heevx expresses the lowest-vector selection using
RANGE='I', IL=1, IU=nev; its all-spectrum *syevd/*heevd interface has
no nev argument. LibDDLA retains its own handle/workspace interface.
Automatic thresholds calibrated for full eigenvectors are applied only to
nev=n. Partial-vector multi-rank calls retain the conservative two-stage
route; an explicit one-stage request remains available. Single-rank automatic
calls use the single-rank policy described above.
ddla::pheevd_chebyshev(handle, uplo, n, nev, nex, A, descA, w, Z, descZ,
tolerance, max_iterations, degree, warm_start,
iterations, converged, info);This separate scalar-only entry supports real/complex float/double and manages
workspace internally. It returns nev host eigenvalues, rather than the
all-n eigenvalue contract of pheevd, with an n × nev distributed output.
q=nev+nex must not exceed n; nev=0 does no matrix-data work. All controls
are required, and pheevd never selects this method automatically.
Only uplo of A is read, ignoring imaginary complex diagonals. A is destroyed
by binary scaling and full Hermitian completion. A needs square source-zero
blocks; Z may use any valid distribution covering its leading output block.
A/Z may alias: warm vectors are copied before A changes, and final vectors
are written after the final operator application. Cold output is never read.
The host w array and active Z must be disjoint; parent regions and padding
outside active blocks are preserved. Warm mode reads the first nev columns,
perturbs every warm column and generates the nex extra directions randomly.
The fixed-degree filter uses distributed QR and Rayleigh–Ritz. Convergence
requires the normwise residual
||Ax-λx||₂ / ((||A||∞+|λ|)||x||₂) <= tolerance, tested with each returned,
rounded eigenvalue. Adequate randomized overlap with the wanted subspace is
assumed: this residual criterion alone does not certify lowest eigenvalue
indices. iterations counts completed filter/Ritz sweeps; converged counts
requested pairs passing the residual criterion. info=0 means all requested
pairs passed, and info=1 returns the available approximations at the limit.
Other positive codes are 2 numerical/rank breakdown, 3 nonfinite input/warm
vectors, 4 irreversible scaling, 5 projected-solve failure, and 6 eigenvalues
unrepresentable at the requested tolerance. Invalid options throw collectively.
All matrix-sized storage stays distributed: three n × q basis matrices,
two q × q projected matrices, and maximum phase scratch. Descriptor/block
rounding and linear panel overhead remain when q is below the process count.
This opt-in method makes no unconditional convergence or performance promise.
The test_pheevd_chebyshev target reuses the private analytic fixture through
the public entry; CPU is default and --gpu explicitly selects GPU execution.
ddla::herm_complete(handle, 'U', n, A, lda); // local Hermitian
ddla::syrm_complete(handle, 'L', n, A, lda); // local symmetric
ddla::pherm_complete(handle, 'U', n, A, descA); // distributed Hermitian
ddla::psyrm_complete(handle, 'L', n, A, descA); // distributed symmetricU/L selects the source triangle. Completion is in place: the other triangle
is overwritten, while parent data and leading-dimension padding are preserved.
Hermitian completion conjugates and sets complex diagonal imaginary parts to
zero; symmetric completion does neither, including for complex matrices.
Local operations need no scratch or communication; GPU work uses the handle
stream. Distributed operations share the tile-exchange implementation and
require square descriptor blocks with RSRC=CSRC=0. All ranks participate;
no rank gathers the full matrix.
sygvd (real) and hegvd (complex) solve local A X = B X Lambda, with
positive-definite B. Their arguments are (handle, jobz, uplo, n, A, lda, B, ldb, w, info). Like syevd/heevd, GPU w and info are device arrays.
The implementation calls OpenBLAS/LAPACK or the selected GPU vendor solver.
phegvd(handle, jobz, uplo, n, nev, A, descA, B, descB, w, Z, descZ, info)
provides the distributed interface for all four scalar types. It returns
all n ascending eigenvalues in replicated host w and the lowest nev
eigenvectors, normalized by Z^H B_original Z = I. Set nev=n for all
vectors, or jobz='N'/nev=0 for values only. GPU matrices remain on-device.
The driver composes distributed Cholesky, two triangular solves, pheevd
and a triangular eigenvector back-transform. All matrix-sized storage stays
distributed. A is destroyed and B becomes its Cholesky factor; the selected
triangle is sufficient on input. A and Z may alias, while B must be
disjoint from both active matrices. A/B need matching square source-zero
blocks, and Z has the existing pheevd distribution requirements.
If the leading minor i of B is not positive definite, the driver returns
info=n+i without starting the reduction or changing A.
For repeated problems with a fixed positive-definite B, factor it once with
ppotrf and reuse its triangular factor through
phegst(handle, uplo, n, A, descA, factorB, descB). This type-1 reduction
supports all four scalar types. Lower storage produces L^-1 A L^-H from
B=L L^H; upper storage produces U^-H A U^-1 from B=U^H U.
It writes the complete leading reduced matrix with real complex diagonals.
The supplied factor, including its unused triangle, remains unchanged. A/factor
parent regions and padding outside the active blocks are preserved.
The factor must have nonzero diagonal entries, A/factor active spans must be
disjoint, and both descriptors need matching square source-zero blocks.
phegst manages its workspace internally and throws collectively for invalid
arguments. The caller supplies an already valid factor; the reduction does
not return a positive-definiteness status. After reduction, use pheevd
and a triangular backsolve for the eigenvectors. The reduction shares the
maintained dense completion and two raw triangular solves with phegvd.
phegst additionally completes the final triangle to canonical full Hermitian
storage; the generalized eigensolver keeps its existing subsequent phase order.
Numerical scratch is the domain-wise maximum of sequential completion/solve
requirements, with no packed-diagonal prefix or repeated shrinking-factor solve.
Coarse descriptor blocks and empty local partitions retain their normal ownership
imbalance.
When the active matrix fits in one descriptor block, its sole owner uses local
completion and two TRSM calls; all ranks require zero reduction workspace and
communication-cache bytes, including ranks with no active entries.
For HostStagedMPI, owning preparation reserves both communication-cache slot
high waters before matrix updates. Private workspace queries report required
host-cache bytes separately from numerical backend/host arenas; this
does not cover vendor first-use allocations or whole-call peak memory.
The opt-in pheevd_chebyshev owner and the distributed/shape pgesvd owner
also reserve their two HostStagedMPI cache slots during agreed preparation,
before raw matrix work. The private staging queries compose actual panel
origins and frozen routes, including QR, reflector application, projected
one-stage solves and real-row redistribution. Slot requirements are reported
separately from backend and host numerical arenas; raw workspace callers must
reserve them before entering the communication schedule.
The generalized phegvd owner reserves the factorization transport before B
is modified. After B succeeds it separately prepares completion/congruence,
then the selected eigen route and vector backsolve. A non-positive B still
returns before future eigensolver method options/resources are read. Private
fully prepared callers query phegvd_staging_bytes after creating all route
resources and call phegvd_prepare_staging before the first raw run. This
covers cold transport caches; no numerical warmup is needed. The staging
query is pure metadata and reports two slot requirements independently of
the numerical arena maxima. Early numerical returns can use smaller payloads.
test_staging_preparation --gpu --composite-only --transport staged checks
these phases and the complete prepared composition with the native observers,
including U/L, selected Local/One/Two routes and allocation-failure/retry
boundaries. test_workspace_phegvd --transport staged --major R also checks
cold first execution and repeats with all four scalar types; its root-only
staging queries make no allocation or communication. --grid RxC and
--major C support alternate allocated-rank layouts. The direct-MPI and
CCL staging query results are zero. Queried payload sizes and actual retained
vector capacities are different memory quantities, particularly when public
phase preparation grows an existing cache.
PBDSDC coefficient broadcasts depend on numerical deflation. Its staging query therefore supplies a reachable geometric supremum, using each problem's bounded coefficient panel and largest rank share; initial splits and deflation can make observed messages smaller. It does not use the entire backend workspace as a host-cache bound. Retained cache capacity and vendor first-use allocations still belong in whole-call peak-memory measurements. These reservations do not imply a global cold-allocation or performance guarantee.
The diagnostic test_staging_preparation --gpu --spectral-only --transport staged
checks cold reserved raw execution with actual native MPI payload observers.
--grid RxC must match the allocated ranks, and --major R|C selects the grid
ordering. Run with both diagnostic preloads, libpblas_alloc_failure.so and
libworkspace_trace.so, on MPI ranks. The suite distinguishes exact fixed
geometry from PBDSDC's data-dependent bound, and includes actual one-rank
cache-allocation failures with collective rejection and same-handle retries.
0.0.7 (in preparation) — defined in src/version.h. The shared library
SONAME tracks the major version. An unchanged major/SONAME does not imply API
or ABI compatibility for this release; rebuild downstream applications.
See the changelog and migration guide.
LGPL v3. See LICENSE.