Conversation
- walkthrough/parallelization: expand the "Multiple GPUs" section with the new `dask.multi-gpu` config flag (automatic dask-cuda cluster over all GPUs), the bring-your-own-cluster path, the single-threaded-client requirement, and when multi-GPU actually helps; add a label for cross-referencing. - reference/default_config.yaml: add the `dask.multi-gpu` option (was out of sync with abtem.yaml). - getting_started/install: note the optional `dask-cuda` dependency and link to the walkthrough. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a tip to the "Multiple GPUs" section noting that more GPUs provide more aggregate device memory, so memory-bound simulations can scale super-linearly. Quote a large benchmark: ~2.0x on two GPUs and ~4.3x on four (beyond the 4x linear ideal), with the caveat that the exact speed-up depends on the calculation parameters and available hardware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Covers the abTEM PRs merged into dev since v1.0.10 that had no user-facing documentation, and corrects the pages a behaviour change had invalidated. - Exact free-space propagator (abTEM #298): the real-space multislice tutorial showed the old FourierMultislice signature and relied on the previous default; it now selects order=1 explicitly where it compares algorithms at equal order. The propagator is derived in the multislice walkthrough. - Energy ensembles (abTEM #257): a new section in the wave-function walkthrough, and an energy series in the multislice walkthrough. - PRISM-EELS (abTEM #289): a new section in the core-loss tutorial, compared against the equivalent multislice scan. - Phonon loss (abTEM #324): a new tutorial building S(q, E) from an energy-resolved frozen-phonon ensemble. - Optional pip extras (abTEM #329) on the installation page, including the core-loss -> gpaw rename. - Configuration reference synchronised with the new and changed keys, and the changelog expanded to cover all 21 merged PRs. The core-loss scans now pass sites explicitly, working around a regression in Probe.transition_potential_scan with max_batch and no sites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abTEM #353 fixes the site auto-extraction for chunked scans, so the core-loss scans no longer need sites= passed explicitly; verified bit-identical to the explicit-sites result, and the PRISM-EELS scan drops it too for the same reason. The full tutorial runs clean without it. abTEM #354 changes the layout of exploded plots (colorbar span and inter-panel padding), so the figures of the sections added for 1.1.0 are regenerated against it. Only images and execution counts change; every numeric output is identical. Both fixes are added to the changelog. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a section deriving the relativistic wavelength, the Helmholtz equation, the Fourier decomposition and the resulting exact propagator, of which the paraxial Fresnel propagator is the first-order term. This is the theory behind the order="exact" default of FourierMultislice. Also tightens the typography of the existing derivation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The "Requires optional libraries" boxes put their title on the directive
line of a {warning} directive, which takes no argument, so MyST read it as
content split across the first line and the body of a directive that also
has an options block. Use {admonition} with the title as its argument and
warning as a class instead; the rendered output is unchanged (the HTML
still carries the warning, dropdown and admonition classes).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…loses #14) Adds a section between the algorithm-object overview and the real-space examples: what LaplaceOperator computes and why (evaluating the transverse Laplacian on the real-space grid, rather than as a single reciprocal-space multiplication, is what lets propagation and transmission be combined for expansion_scope="full"), how derivative_accuracy trades stencil width for truncation error, and a plot comparing the finite-difference Laplacian against the exact eigenvalue across scattering angle for accuracy 2/4/6/8 -- verified numerically before writing the accompanying text (accuracy=2 crosses 1% relative error around 40 mrad, accuracy=4 around 113 mrad, accuracy=6 stays under it through 120 mrad, at 30 keV and this tutorial's sampling). Also answers the issue's "when is real-space preferable" question, now that FourierMultislice's propagator is exact by default: expansion_scope ="full" (interaction between propagation and transmission within a slice) and the coherent backscattered wave, which has no forward-only Fourier-space counterpart. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reference page states that it shows "the full default configuration file", but the copy had drifted: cupy.fft-cache-size was documented as "0 MB" when the shipped default was -1 (and is now "auto"), and the potential.slice-chunk-size, dask.multi-gpu-rmm-pool and dask.multi-gpu-devices keys were missing entirely. The file is now a verbatim copy of abtem/core/abtem.yaml, so it also picks up the per-key comments explaining the fft-cache-size settings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tion The "Using GPUs" section claimed that abTEM sets the CuPy FFT plan cache size to zero because the plans are not worth their memory. That was never the shipped default -- it was -1 (unlimited, CuPy's own default) and is now "auto", 25 % of the device's total memory resolved per device. The recommendation was inverted too: a flat bound measurably costs performance on fast-radix grids, which is why the default is device-relative. The paragraph now describes what the cache does, all four settings, and the uncached-oversized-plan fallback. The "Multiple GPUs" section gains what changed with the multi-GPU hardening work: - to_zarr() distributes like compute(); previously it silently ran the whole scan serially on one device, which is the failure that motivated the work. - The multiprocessing entry-point guard that dask-cuda requires in a script rather than a notebook. - dask.multi-gpu-rmm-pool and dask.multi-gpu-devices, which cover the two cases the section previously sent readers to a hand-built cluster for. - The client configuration reaching the workers, so that precision and device mean the same in a distributed run as in a local one. - A note that a declined multi-GPU request is now reported rather than silently ignored. The scaling tip keeps its argument -- aggregate device memory permits larger batches -- but replaces the earlier super-linear figures with the measured single-node numbers at equal precision: 51 minutes on one 40 GB A100 against 13 minutes on four, 3.86x. The earlier numbers predate the discovery that the client configuration never reached the workers, so a float64 client dispatched float32 work to them; better-than-linear speed-ups were that bug's signature and should not be published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Running out of memory?" listed only reduce-early, smaller batches and fewer workers, all of which are host-memory answers. Device memory is usually the tighter constraint, so the section now also covers the potential slice-chunk size, the cuFFT plan-cache bound, and the aggregate device memory that more GPUs provide. The "good numbers of gpts" discussion gains a note that the point is sharper on GPU: cuFFT has no kernel for a length with a prime factor above 7 and falls back to Bluestein, which needs a workspace several times the transform size, so an unlucky grid costs memory as well as time. abTEM now warns when it meets such a grid and names the next good size; the note quotes that warning and shows the helpers behind it. The float-precision section notes that the setting is resolved inside each task and now reaches distributed workers -- and that before 1.1 it did not, so double-precision results from a distributed run on an earlier version were actually computed in float32. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Installing dask-cuda pulls in the dask and distributed pins of its own RAPIDS release, which frequently conflict with the versions abTEM was installed with; --no-deps is the practical route, and the install page now says so. The page also gains a cross-reference target so the walkthrough can link to it. The changelog gains the multi-GPU feature entries and, separately, the behaviour changes: the client configuration now reaching distributed workers (which changes distributed results for anyone who relied on a non-default configuration), the bounded cuFFT plan cache, and the halved scan batches on Bluestein grids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review (TomaSusi): the tip gave 51 minutes against 13 minutes and called it 3.86x, but 51/13 = 3.92. The minute figures were rounded from the raw measurement, and rounding both of them down inflated the implied ratio. The underlying numbers are 3088.8 s and 799.4 s, whose ratio is 3.8639 -- so the 3.86x was right and the minutes were the lossy part. Quoting the seconds makes all three figures consistent and lets a reader verify the ratio, which is the point of putting a benchmark in the documentation at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PR was retargeted from main to v1.1.0. Only default_config.yaml conflicted: v1.1.0's copy is an exact mirror of abTEM dev's abtem.yaml, and ours is the same mirror taken from multi-gpu-hardening, so the resolution is our version -- it is dev's file plus exactly the keys #346 adds, and drops nothing v1.1.0 had. Restructured the changelog entries to fit the rewritten 1.1.0 section. They had been appended to the old "Upcoming: major features" list, which the merge moved wholesale into "Planned for this release (not yet merged)". #346 is indeed unmerged, so that section is the right home -- but as one entry with sub-bullets rather than three loose bullets and a dangling "Behavior changes" heading. The already-merged #269 entry is left alone, including its statement that cupy.fft-cache-size defaults to -1: that is still true of dev, and the #346 entry records the change to auto. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a "Choosing sampling and slice thickness automatically" section to the potentials walkthrough: what commensurability means and why it's not just a numerical nicety, plus a worked demonstration -- 11 SrTiO3 unit cells in a row, where a hand-picked 0.05 A sampling puts the same translation-equivalent Sr atom at 11 different fractional pixel offsets and its discretized potential value drifts by 21% along the chain, while sampling="auto" collapses that spread to floating-point noise (numbers verified against the actual abTEM dev build, not asserted). Also covers slice_thickness="auto" aligning slice boundaries to crystal planes, the CrystalPotential/AtomsEnsemble caveats, and the grid.round-to-fast-fft config option. Cross-referenced from two places this affects: - performance_tips.ipynb's existing "use good numbers of gpts" section, which recommended manually rounding gpts to small-prime products -- now largely automated by is_fast_fft_size/next_fast_fft_size/ Grid.round_to_fast_fft() - convergence.ipynb's real-space sampling section, which already recommended commensurate sampling manually; now points at the automatic version Config reference and changelog synced with the new grid.round-to-fast-fft key and PRs #274/#347. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The merge of v1.1.0 (which added the sampling="auto" rounding cells) dropped the closing bracket/brace/cell-type lines between the GPU Bluestein-note cell and the next inserted cell, corrupting the notebook's JSON and failing the check-notebook-widgets CI check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Document multi-GPU support
Pre-merge review of PR #20 found this listed under "Planned for this release (not yet merged)" even though abTEM PR #346 merged into dev before the doc-side multi-gpu-docs PR did -- a genuine race between the two, not a documentation error, but the changelog needed to catch up. Moved it into Bugfixes with content matching what the parallelization walkthrough's multi-GPU section already documents accurately (to_zarr() distribution, config reaching workers, VRAM/plan-cache handling, the new RMM-pool/device-subset config keys), and corrected the now-superseded claim that cupy.fft-cache-size defaults to -1 (it went -1 -> auto in the same PR). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pre-merge review of PR #20 (which now includes the merged multi-gpu-docs PR #19) found performance_tips.ipynb quoting an outdated UserWarning message: "e.g. by setting gpts explicitly instead of the sampling" is what abtem.core.fft._warn_slow_fft_size said before the autogrid work landed. The shipped message (checked against the actual abTEM dev warning, not just the source) also offers grid.round-to-fast-fft as a remedy -- verified verbatim and quoted correctly now. Also collapses three cells my own earlier commit added (re-importing and re-demonstrating is_fast_fft_size/next_fast_fft_size a second time, duplicating the GPU note's own demo of the same functions on the same 486/512/521 example) into one short paragraph pointing at the existing demo instead of repeating it. Changelog fix in the prior commit (multi-GPU hardening had merged, not "planned") was verified against abTEM PR #346's actual merge time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the abTEM PRs merged into
devsincev1.0.10that had no user-facing documentation, and corrects pages a behaviour change had invalidated.FourierMultislicesignature and relied on the previous default; it now selectsorder=1explicitly where it compares algorithms at equal order. The propagator is derived in the multislice walkthrough, and the derivation appendix gains a full higher-order derivation (relativistic wavelength, Helmholtz equation, Fourier decomposition, exact propagator).core-loss->gpawrename.dev; the docs no longer carry a workaround for either, and the affected figures were regenerated against the layout fix.apidocs/build-output directory left in the working tree, and fixed a MyST directive-parsing warning on three "Requires optional libraries" admonitions.Test plan
jb build .succeeds with 38 warnings (down from 2969), all pre-existing/upstream (deliberately-hidden notebooks, matplotlib intersphinx references pulled in via:inherited-members:, bokeh output mime types, one upstream docstring footnote)dev(commite62e7dbe) with no errorssites=workaround$ $delimiters, not$$ $$🤖 Generated with Claude Code