Document the features merged for 1.1.0

docs/_toc.yml

@@ -43,6 +43,7 @@ parts:
         #- file: user_guide/tutorials/epie
         #- file: user_guide/tutorials/visualizations
         - file: user_guide/tutorials/core_loss
+        - file: user_guide/tutorials/phonon_loss
         #- file: user_guide/tutorials/custom_simulation_objects
         #- file: user_guide/tutorials/plasmons
     - file: user_guide/examples/examples

docs/abtem/changelog.md

@@ -1,14 +1,138 @@
 # Changelog
 
-## Upcoming: major features
+## Upcoming: 1.1.0
+
+The `dev` branch is at version `1.1.0`. The entries below are merged into `dev` but not yet released.
 
 Features:
-- Energy ensemble support across the codebase (PlaneWave, Probe, SMatrix, Bloch) ([PR #257](https://github.com/abTEM/abTEM/pull/257))
-- Significant improvements on simulating large potentials on GPU, alongside minor performance improvements ([PR #269](https://github.com/abTEM/abTEM/pull/269))
-- In addition to the Gaussian (G) distribution, now also implemented Lorentzian (L), Voigtian (convolution L * G) and 
-pseudo-Voigtian (L + G) source-size distributions and filters ([PR #270](https://github.com/abTEM/abTEM/pull/270))
+
+- Energy ensemble support across the codebase: `PlaneWave`, `Probe`, `SMatrix`, `BlochWaves` and `CTF`
+  accept a list of energies, and the resulting `EnergyAxis` propagates through indexing, angular sampling,
+  unit conversion and diffraction-spot indexing ([PR #257](https://github.com/abTEM/abTEM/pull/257))
+- C-PRISM: `SMatrix(upsample=True)` reduces every probe from the complete plane-wave expansion of the
+  aperture, so the interpolation factor only sets the number of multislice runs. Adds
+  `CompressedSMatrixArray` and `GridScan.commensurate` ([PR #318](https://github.com/abTEM/abTEM/pull/318))
+- Phonon-loss (thermal diffuse scattering) energy-loss workflow: `EnergyResolvedAtomsEnsemble`,
+  `phonon_loss_diffraction_patterns`, `momentum_resolved_spectrum` and `MomentumResolvedSpectrum`,
+  the `SpectralAnnularDetector` and `SpectralSlitDetector`, and detailed-balance thermal weighting that
+  splits the classical TDS signal into loss and gain sides
+  ([PR #324](https://github.com/abTEM/abTEM/pull/324), [PR #351](https://github.com/abTEM/abTEM/pull/351))
+- Linear-scaling PRISM-EELS for core-loss simulations: `SMatrix.transition_potential_scan`, with
+  single- and double-channel scattering and an optional windowed inelastic crop
+  ([PR #289](https://github.com/abTEM/abTEM/pull/289))
+- In addition to the Gaussian (G) distribution, now also implemented Lorentzian (L), Voigtian (convolution L * G) and
+  pseudo-Voigtian (L + G) source-size distributions and filters ([PR #270](https://github.com/abTEM/abTEM/pull/270))
+- The exact free-space propagator is now the default for Fourier multislice,
+  `FourierMultislice(order="exact")`, replacing the paraxial approximation. Spatial frequencies beyond
+  `k > 1 / lambda` are treated as evanescent rather than propagating; the paraxial propagator remains
+  available as `order=1` ([PR #298](https://github.com/abTEM/abTEM/pull/298))
+- Magnetic potentials and fields from collinear GPAW calculations: `gpaw_magnetic_fields` builds the
+  electrostatic potential, vector potential and magnetic field from the same calculator(s) in one call,
+  returning a `GPAWMagneticFields` bundle with `.tile()`, `.combined_potential()` and `.show()`;
+  `rotate_field` now defaults to `"auto"` ([PR #326](https://github.com/abTEM/abTEM/pull/326))
+- `GPAWParametrization` is now usable: it fits a Lobato-form IAM potential to the X-ray scattering factor
+  of an all-electron GPAW calculation, with working ionization support and a regularized fit
+  ([PR #329](https://github.com/abTEM/abTEM/pull/329))
+- `Potential(sampling="auto", slice_thickness="auto")` automatically finds a grid commensurate with the atomic
+  lattice — so translation-equivalent atoms discretize identically — near the usual default targets
+  ($0.05 \ \mathrm{Å}$, $1 \ \mathrm{Å}$), preferring FFT-fast grid sizes wherever that is compatible with
+  commensurability. New `grid.round-to-fast-fft` config key (`'auto'`/`True`/`False`), `Grid.round_to_fast_fft()`,
+  and `is_fast_fft_size`/`next_fast_fft_size` helpers in `abtem.core.fft`
+  ([PR #274](https://github.com/abTEM/abTEM/pull/274), [PR #347](https://github.com/abTEM/abTEM/pull/347))
+- Significant improvements on simulating large potentials on GPU, alongside minor performance improvements
+  ([PR #269](https://github.com/abTEM/abTEM/pull/269))
+    - The potential is now built in chunks of contiguous slices instead of all at once, keeping peak VRAM
+      bounded; new config key `potential.slice-chunk-size` (default `"auto"`)
+    - Opt-in multi-GPU via the new config key `dask.multi-gpu` (requires `dask-cuda`)
+    - `cupy.fft-cache-size`'s previous `0 MB` default silently disabled the cuFFT plan cache; changed to
+      `-1` (unlimited) here, then to the device-relative `auto` default below once unbounded retention
+      turned out to cost tens of GB on FFT-unfriendly grids
+
+Performance:
+
+- The projection integrator is shared by reference across ensemble members instead of being deep-copied
+  (and re-uploaded to the GPU) for each ([PR #350](https://github.com/abTEM/abTEM/pull/350))
+- Removed a redundant potential rebuild on every scan chunk ([PR #340](https://github.com/abTEM/abTEM/pull/340))
+
+Dependencies:
+
+- The `core-loss` extra is merged into a single `gpaw = ["hankel", "sympy"]` extra, and a new `all` extra
+  installs every optional runtime dependency ([PR #329](https://github.com/abTEM/abTEM/pull/329))
+
+Bugfixes:
+
+- `GPAWPotential` for the new-style GPAW calculator API (GPAW 26+), and `GPAWPotential.from_file` on
+  old-style restarted calculators ([PR #325](https://github.com/abTEM/abTEM/pull/325))
+- `GPAWPotential` single-calculator `frozen_phonons` ensemble building
+  ([PR #327](https://github.com/abTEM/abTEM/pull/327)), plus removal of dead and broken code from
+  `GPAWPotential` and `GPAWParametrization` ([PR #328](https://github.com/abTEM/abTEM/pull/328))
+- `FieldArray.tile()` for vector-valued fields, and unsupported `frozen_phonons`/`repetitions` on magnetic
+  fields now raise instead of being silently ignored ([PR #326](https://github.com/abTEM/abTEM/pull/326))
+- Silent corruption in eager multislice for potentials with two or more ensemble axes
+  ([PR #333](https://github.com/abTEM/abTEM/pull/333))
+- `numba` `TypingError` in quasi-dipole interpolation on some numba/numpy pairings
+  ([PR #332](https://github.com/abTEM/abTEM/pull/332))
+- Single-point `GridScan` failing when built lazily ([PR #342](https://github.com/abTEM/abTEM/pull/342))
+- `LinearAxis` losing its offset under dask ensemble chunk partitioning
+  ([PR #344](https://github.com/abTEM/abTEM/pull/344))
+- Nondeterministic atom loss in `orthogonalize_cell`, and a hardened Gram-Schmidt fallback
+  ([PR #345](https://github.com/abTEM/abTEM/pull/345))
+- Repeated axis labels and colorbar overlap in exploded spectrum panels, and silently returned zeros for
+  single-configuration TDS ([PR #351](https://github.com/abTEM/abTEM/pull/351))
+- Azimuthal convention in `prism_coefficients`, which reflected azimuthally dependent aberrations in a PRISM
+  reduction with a `CTF`, and exit planes not being remapped when slicing a `PotentialArray`
+  ([PR #318](https://github.com/abTEM/abTEM/pull/318))
+- `numpy` 2.5 test failures caused by an ASE deprecation warning
+  ([PR #343](https://github.com/abTEM/abTEM/pull/343))
+- `Probe.transition_potential_scan` raised for a scan split into more than one chunk (for example through
+  `max_batch`) unless the scattering sites were passed explicitly
+  ([PR #353](https://github.com/abTEM/abTEM/pull/353))
+- Colorbars did not span multi-row exploded plots, and neighbouring panels could abut closely enough for
+  their tick labels to collide ([PR #354](https://github.com/abTEM/abTEM/pull/354))
+- Multi-GPU hardening, and the configuration fix found while chasing it
+  ([PR #346](https://github.com/abTEM/abTEM/pull/346))
+    - **The client's configuration now reaches `distributed` workers.** *ab*TEM resolves configuration
+      inside each task, and worker processes start fresh and previously saw only the YAML defaults, so
+      any distributed computation with a non-default configuration silently used the defaults instead —
+      most consequentially `precision`, which meant `float64` runs were computed in `float32`.
+      Distributed results obtained with a non-default configuration are worth repeating. Applies to any
+      distributed client, CPU clusters included
+    - `to_zarr()` on a lazy result honours `dask.multi-gpu`; it previously ignored the flag and ran the
+      whole computation on a single device
+    - `cupy.fft-cache-size` defaults to `auto` — 25 % of each device's memory, resolved per device —
+      rather than unlimited. `-1` restores unlimited, `0 MB` disables the cache, and a size such as
+      `512 MB` sets a fixed bound. A single plan larger than the bound runs uncached with a warning
+      instead of raising
+    - New config keys `dask.multi-gpu-rmm-pool` and `dask.multi-gpu-devices`, for an RMM memory pool per
+      worker and for restricting the cluster to a subset of GPUs
+    - Automatically sized scan batches are halved on grid sizes that force cuFFT's Bluestein fallback,
+      which needs a much larger FFT workspace
+    - Warnings replace silent fallbacks: multi-GPU requested but declined (with the reason), a missing
+      `if __name__ == "__main__"` guard, and a grid size that forces the Bluestein fallback (naming the
+      next fast size)
+
+Documentation:
+
+- `sampling="auto"`/`slice_thickness="auto"` documented in detail in the potentials walkthrough, including a
+  worked example of the commensurability artifact they remove; cross-referenced from the convergence appendix
+  (manual commensurate sampling) and the performance-tips appendix (fast FFT sizes)
+- New tutorial on phonon-loss spectroscopy: energy-resolved frozen phonons, the TDS decomposition, the
+  momentum-resolved spectrum $S(q, E)$, the spectral detectors and detailed-balance thermal weighting
+- Energy ensembles documented in the wave-function walkthrough, with an energy series added to the
+  multislice walkthrough
+- PRISM-EELS added to the core-loss tutorial, compared against the equivalent multislice scan
+- The exact free-space propagator is documented in the multislice walkthrough and the real-space multislice
+  tutorial, which now selects the paraxial propagator explicitly where it compares algorithms at equal order
+- The installation page documents the optional pip extras (`gpaw`, `extra`, `all`) and why the GPU packages
+  are not among them
+- The configuration reference is synchronized with the new and changed config keys
+- The multiple-GPUs section of the parallelization walkthrough is expanded to cover the multi-GPU
+  hardening fixes above, and the FFT plan-cache documentation is corrected to match the shipped `auto`
+  default (it previously described a stale `0 MB` default that was never shipped)
+
+Planned for this release (not yet merged):
+
 - Support for skewed pixels (non-orthogonal x,y,z cell axes) ([PR #282](https://github.com/abTEM/abTEM/pull/282))
-- Linear-scaling PRISM-EELS for core-loss simulations ([PR #289](https://github.com/abTEM/abTEM/pull/289))
 - Radially variable detector sensitivity ([PR #283](https://github.com/abTEM/abTEM/pull/283))
 - Plasmons: fast `PhaseScramblePlasmons` for multislice and PRISM, and `MonteCarloPlasmons` for Bloch wave
 - CBED patterns for Bloch waves ([PR #254](https://github.com/abTEM/abTEM/pull/254))

docs/getting_started/install.md

@@ -1,3 +1,4 @@
+(getting_started:install)=
 # Installation
 
 There are many ways to install the *ab*TEM package, for example conda or pip:
@@ -26,6 +27,28 @@ pip install git+https://github.com/abTEM/abTEM
 
 ## Optional dependencies
 
+A few optional Python packages are bundled as pip *extras*, so they can be installed together with *ab*TEM:
+
+- `abtem[gpaw]` installs [hankel](https://hankel.readthedocs.io/) and [sympy](https://www.sympy.org/), the Python-side requirements of the DFT-based features — the form factors for core-loss EELS, and the potential parametrization fitted to an all-electron calculation. GPAW itself is *not* installed by this extra, see below.
+- `abtem[extra]` installs [bokeh](https://bokeh.org/), [ipycytoscape](https://ipycytoscape.readthedocs.io/) and the [Dask labextension](https://github.com/dask/dask-labextension), used for the task-graph visualizations and the diagnostics dashboard shown in the [parallelization walkthrough](walkthrough:parallelization).
+- `abtem[all]` installs both of the above.
+
+```{code-block}
+pip install abtem[all]
+```
+
+With conda, install the packages by name instead, for example:
+
+```{code-block}
+conda install -c conda-forge hankel sympy bokeh ipycytoscape
+```
+
+```{note}
+Before version `1.1.0`, the `gpaw` extra was called `core-loss` and installed only `sympy`.
+```
+
+The GPU packages ([CuPy](https://cupy.dev/) and, for several GPUs, `dask-cuda`) are deliberately left out of the extras: their wheels are specific to a CUDA version and have to match the toolkit installed on your machine, so they are installed separately as described below.
+
 ### GPAW (not available on Windows)
 
 Some features of *ab*TEM, such as calculating potentials from DFT, require a working installation
@@ -52,6 +75,8 @@ gpaw install-data <dir>
 ````
 `````
 
+The `hankel` and `sympy` packages that the DFT-based features need on top of GPAW itself come from the `abtem[gpaw]` extra above.
+
 ### GPU (only NVIDIA) 
 
 GPU calculations with *ab*TEM require a working installation of [CuPy](https://cupy.dev/) and compatible hardware.
@@ -76,6 +101,24 @@ where * should be substituted for the CUDA Toolkit version.
 ````
 `````
 
+To distribute a calculation across **several GPUs** on a node, also install
+[`dask-cuda`](https://docs.rapids.ai/api/dask-cuda/stable/install/) (NVIDIA GPUs,
+Linux only), matched to your CUDA/RAPIDS version. See
+{ref}`Multiple GPUs <walkthrough:parallelization:multigpu>` in the walkthrough.
+
+```{note}
+`dask-cuda` pins `dask` and `distributed` to the versions of its own RAPIDS
+release, which may not be the versions *ab*TEM was installed with. If pip
+insists on downgrading `dask` (or refuses to resolve the environment at all),
+install `dask-cuda` without its dependencies and keep the versions you already
+have:
+
+    pip install --no-deps dask-cuda
+
+Both packages track `dask` closely, so it is worth checking that a simple
+multi-GPU computation runs after installing this way.
+```
+
 ### Metal on Apple silicon (experimental)
 
 A subset of features in *ab*TEM can be accelerated on Apple silicon processors using their [Metal API](https://developer.apple.com/metal/). 

docs/reference/default_config.yaml

@@ -8,7 +8,7 @@ fft: fftw
 precision: float32
 diagnostics:
   # Show the progress bar. Options are 'true', 'false' or 'tqdm'
-  progress_bar: tqdm
+  progress_bar: "tqdm"
   # Show the progress of each task. Options are 'true' or 'false'
   task_progress: false
 dask:
@@ -18,10 +18,30 @@ dask:
   chunk-size: 128 MB
   # The target chunk size to use for dask arrays on the gpu
   chunk-size-gpu: 512 MB
+  # Automatically distribute gpu computations over all visible gpus by starting a
+  # dask-cuda cluster. Requires the optional dask-cuda package.
+  multi-gpu: false
+  # Optional RMM memory-pool size per multi-gpu worker (e.g. "20 GB").
+  # null disables the RMM pool.
+  multi-gpu-rmm-pool: null
+  # Optional subset of GPUs for the multi-gpu cluster, as a list of device
+  # indices (e.g. [0, 1]) or a comma-separated string (e.g. "0,1").
+  # null spans all visible GPUs.
+  multi-gpu-devices: null
 cupy:
-  # The size of the fft cache in MB used by cupy
+  # Maximum GPU memory (bytes) used by the cuFFT plan cache.
   # https://docs.cupy.dev/en/stable/user_guide/fft.html#fft-plan-cache
-  fft-cache-size: 0 MB
+  # auto  — 25% of the device's total memory, resolved per device: scales
+  #          with the card like the auto-sized batches whose plans it holds,
+  #          keeping live plans hot while capping stale-plan retention. A
+  #          ceiling, not a reservation — unused headroom costs no memory.
+  # 0 MB  — disable caching entirely (workspace freed after every FFT call;
+  #          use when VRAM is tight and large-grid OOMs occur).
+  # > 0   — bound cached workspace to this many bytes (e.g. "512 MB").
+  # -1    — unlimited (CuPy default; fastest but accumulates workspace,
+  #          which on Bluestein-fallback grid sizes can reach tens of GB).
+  # null  — same as -1 (no bound).
+  fft-cache-size: auto
 mkl:
   # The number of threads to use for mkl
   threads: 2
@@ -36,11 +56,24 @@ fftw:
   planning_timelimit: 60
   # Whether to allow falling back to not using wisdom if the cache fails
   allow_fallback: true
+grid:
+  # Round automatically derived gpts up to the next fast FFT length (all prime
+  # factors in {2, 3, 5, 7}). Fast lengths avoid the slow, memory-hungry
+  # Bluestein FFT fallback on GPU and speed up CPU FFTs as well. Rounding is
+  # upward only, so a grid derived from a numeric sampling never comes out
+  # coarser than requested. gpts given explicitly are never altered.
+  #   'auto'  round only grids abTEM derives on its own (sampling='auto')
+  #   true    additionally round gpts derived from a numeric sampling
+  #   false   never round
+  round-to-fast-fft: auto
 warnings:
   # Show the dask warning about the blockwise performance when the number are increased dramatically
   dask-blockwise-performance: false
   # Show a warning when the grid is overspecified
   overspecified-grid: true
+potential:
+  # Number of slices to build at once during multislice. "auto" = memory-budget-aware.
+  slice-chunk-size: "auto"
 antialias:
   # The antialias cutoff in reciprocal space
   cutoff: 0.6666666
@@ -60,4 +93,4 @@ visualize:
   # Scale the values of interactive plots automatically
   autoscale: false
   # Use tex rendering in plots
-  use_tex: true
+  use_tex: true

docs/user_guide/appendix/convergence.ipynb

@@ -522,7 +522,11 @@
     "\n",
     "Our default value of $0.05\\ \\mathrm{Å}$ is a good compromise for speed and reliability for most systems. For specimen models containing only light elements (with relatively weak nuclear potentials), slightly larger values may produce acceptable results, whereas simulations of heavier elements as well as thicker specimens (where small numerical errors can accumulate), or when very high precision is required, may require smaller real-space sampling.\n",
     "\n",
-    "To ensure the best uniformity, it is also often beneficial to use sampling grids that are commensurate with the atomic positions, which in this case means also implies fractions of $a/4$. We use a geometric sequence halving the sampling at each step, but starting from the $1/16$ sampling of the entire unit cell, $a/16, ..., a/512$."
+    "To ensure the best uniformity, it is also often beneficial to use sampling grids that are commensurate with the atomic positions, which in this case means also implies fractions of $a/4$. We use a geometric sequence halving the sampling at each step, but starting from the $1/16$ sampling of the entire unit cell, $a/16, ..., a/512$.\n",
+    "\n",
+    "```{note}\n",
+    "Since version `1.1.0`, `Potential(..., sampling=\"auto\")` finds a commensurate sampling like this automatically, targeting the default $0.05 \\ \\mathrm{Å}$ used here rather than a manually chosen fraction of $a$ — see the [walkthrough on potentials](walkthrough:potentials) for how it works and why commensurability matters beyond the resolution question this section is about.\n",
+    "```"
    ]
   },
   {