Skip to content

Julia backend experiment (do not merge) - #559

Closed
jtravs wants to merge 108 commits into
mainfrom
feature/julia-backend-experiment
Closed

jtravs wants to merge 108 commits into
mainfrom
feature/julia-backend-experiment

Conversation

@jtravs

@jtravs jtravs commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

An experiment. Do not merge. It asks how fast one compact Julia source can run solprop's focus routes next to the native backend:

  • on CPUs, and on NVIDIA, AMD and Apple GPUs;
  • in binary64 and binary32.

It carries #10's CPU prototype (closed) to current main and rewrites its engine. The benchmark harness's per-arm backends come from #543.

Handoff for the next worker: read julia/STATUS.md ("Next steps" lists the measured to-dos and the working setups), then julia/README.md. The ledger rows are in docs/dev/benchmarks.md, "Julia backend experiment".

What it is

backend="julia" steps the translated problem behind the same StepDriver seam as native. The engine is about 3,500 lines of Julia under julia/src (plus a 1,600-line precompile trace), and the adapter is about 2,400 lines of Python under src/solprop/backends/julia. For scale, native is about 31,000 lines of C++.

One source for every device and precision. Each state-sized pass is a scalar body launched by a small device layer:

  • On a CPU, Polyester tasks run contiguous SIMD loops.
  • On a GPU, KernelAbstractions kernels run each pass, with AcceleratedKernels for the reductions and scans.
  • The device is chosen by a package extension: CUDA.jl (julia/cuda), AMDGPU.jl (julia/rocm) or Metal.jl (julia/metal).
  • FFTs are AbstractFFTs plans (FFTW, cuFFT, rocFFT or MPSGraph). Products are BLAS (Accelerate by default on macOS) or the vendor's library.

Binary32 follows SEP 33. The state is held as a / s. The operator's exponents, the propagators and the Adams phi functions are formed in a wide type and rounded once to binary32:

  • The wide type is binary64 by default.
  • SOLPROP_JULIA_WIDE=df32 makes it a double-float instead: a value as two binary32s, about 48 bits (julia/src/dfloat.jl, the mlx backend's primitives).
  • An Apple GPU has no binary64, so Metal always uses df32 and refuses binary64 by name.
  • Binary64 runs keep their arithmetic exactly.

Routes:

  • Resolved and mode-averaged capillaries, radial, Cartesian and line free space; lockstep batches.
  • Kerr; field and envelope plasma, several channels; Raman, several components, varying along a gradient; tapers.
  • DOPRI5, TSIT5, the exponential Adams march (modal and free space) and RK4IP.
  • Vector polarisation, tensor Raman and the rest are refused by name; julia/STATUS.md lists them.

Correctness

tests/backends/test_julia.py checks against numpy_ref:

  • Seven focus windows in binary64: the same steps, fields within 1e-10, and every statistic.
  • Binary32 windows in both wide types: within 1e-4 of the reference's binary32 path.
  • Bit for bit: the helpers, trial tables, stage scalars and the Julia Adams march against their Python forms.
  • The double-float phi functions and propagator pair: against binary64, within 2^-36 and 8 binary32 ulps.

Julia runs bounds-checked in the tests. Passing on 2026-11-08:

  • the CPU suite (59 tests) on the M5 Max and on DMOG;
  • CUDA on an A40 (59);
  • Metal on the M5 Max GPU (the 13 binary32 and host tests; the binary64-only ones are skipped there);
  • subsets on an NVIDIA H200 and an AMD MI300X (33 each).

How fast

Each figure is Julia's time over native's, from python -m benchmarks ab, both arms in one interpreter. Below 1, Julia is faster.

machine geometric mean range what was measured
Apple M5 Max CPU, 6 threads 0.82 0.68 (line) to 1.02 (kerr_beam_192) whole runs, binary64, same steps, fields within 1e-11
Apple M5 Max CPU, 1 / 6 / 18 threads 0.83 0.84–1.23 / 0.68–0.90 / 0.38–0.86 2 % windows
Intel Xeon Gold 6240, 16 threads 1.13 0.78 to 1.26 2 % windows, at a6722029
NVIDIA A40, binary64 1.15 (1.97 on 2026-11-07) ctc_reduced and h2_raman_6ps 0.60 to kerr_beam_192 2.44 (about 1.24 after e3b07264) at 51a153de
NVIDIA A40, binary32 with df32 0.79 (1.72) h2_raman_6ps 0.25 to kerr_beam_192 2.49 Julia faster in five of seven
NVIDIA H100, binary64 1.25 kerr_beam_192 0.66 to line 1.77 at 6f686ea4; the H200 was 1.69 (2.39 before) at 67491bf9, before the source graphs and the copy fix
AMD MI300X, binary64 / binary32 df32 1.28 / 1.48 (MI325X 2.48) kerr_beam_192 0.89 to radial 1.75 at 1b17fc75, no graphs on ROCm yet
Apple M5 Max GPU, binary32 1.95 against native on 6 CPU threads Cartesian 0.33, radial 0.92, capillaries 3 to 6
Apple M5 Max GPU against mlx 1.25 at ecbcafbe Cartesian 0.71, radial and ctc_reduced 0.96 h2_raman_6ps 2.1, about 1.6 after 19215f03

On the M5 Max CPU, every variant gives about 0.8:

  • windows: 0.83;
  • whole runs: 0.82;
  • 30 s idle between workers: 0.79;
  • each backend run alone, back to back: 0.72 to 0.92 by case.

On one thread the two engines are close (0.84 to 1.23). The gap opens with threads, as native slows on the 12 second-tier cores (#333).

Absolute times on the Mac drift by up to 40 % over a day, so compare only interleaved ratios.

Other measurements on this branch (2026-11-07)

change where result
Accelerate in place of OpenBLAS for Julia M5 Max 0.94 overall; radial 0.59 (1 thread) and 0.82 (6), no case slower. Now the default on macOS
Contraction (muladd) in ordinary kernels, as SEP 25 permits M5 Max 0.99. Kept; every test passes unchanged
Flush-to-zero (SOLPROP_JULIA_FTZ=1, experiment only, #561) M5 Max binary32 1.00, binary64 0.99
Flush-to-zero Xeon, binary32 0.99. Binary32 there takes 2 to 3 times binary64's time, and subnormals are not the cause
df32 against binary64 wide, binary32 A40 0.99; free space 0.88 and 0.91 from forming the operator once on the host
Strided GEMM operands copied to scratch Metal h2_raman_6ps 0.76, radial 0.91
Metal.jl probes M5 Max GPU every FFT plan form the engine uses works, with no Bluestein cliff (10240 points as fast as 8192); float32 GEMM is full precision, not TF32
Metal wait costs M5 Max GPU a queued launch 4.8 us; a synchronize 160 us; a reduction read back 210 us; a 40 kB copy to the host 190 us

Evening of 2026-11-07: Metal tuning and mlx's features

Julia on the M5 Max GPU against mlx: 0.66 (1.25 that morning). Both binary32, pow2 transforms, three rounds, at 0f2c1842:

case mlx, s Julia, s ratio
rdw 0.423 0.169 0.40
argon_blueshift 0.373 0.244 0.65
h2_raman_6ps 0.921 0.653 0.73
ctc_reduced 0.949 0.251 0.26
radial_filament_256 0.190 0.188 0.98
kerr_beam_192 0.248 0.188 0.76
line 0.131 0.203 1.52

What made it, each change A/B'd on Metal:

  • Device copies as one kernel, views included (0.81). Metal.jl's copyto! into a view cost about 200 us.
  • One chunk of nodes a source on a GPU (0.84).
  • Contiguous projection rows (0.93).
  • Packed forward streams (0.92).
  • One readback a step, and the population statistics in one readback (0.96).
  • Packed table uploads, and one host copy a state.

On the A40 the same changes measured 0.97, then 0.95, then 0.94, each against the tree before it.

A40 against native: about 1.9 in binary64 and 1.5 to 1.6 in binary32 with df32. h2_raman_6ps is at par in binary64 and Julia is faster in df32 (0.58).

  • Fair profiles need nsys --cuda-graph-trace=node: native's sources run in CUDA graphs.
  • What is left is in julia/STATUS.md:
    • host time between launches, about twice native's;
    • the binary64 Adams predictor, whose phi functions spill to local memory;
    • Cartesian 2D FFTs on the frequency-first layout, 3.4 times slower than on a transverse-first one;
    • radial's binary64 CUTLASS GEMMs.

mlx's features: 11 of 13 closed, each tested against numpy_ref (binary64, fields within 1e-10, most about 1e-15):

  • lockstep batches, TSIT5 and modal RK4IP;
  • several Raman components, and tapers;
  • the window statistics and zdw;
  • resolved and radial envelopes, and the envelope's third harmonic;
  • Raman in free space, and several plasma channels.

Still open:

  • varying Raman;
  • Cartesian Adams;
  • envelope plasma;
  • the on-axis plasma statistics;
  • vector polarisation, deferred.

2026-11-08: mlx's features, and the GPU host

mlx's features are closed but for vector polarisation and tensor Raman (deferred). New against numpy_ref: varying Raman along a gradient, Adams in free space, envelope plasma (julia/src/envelope.jl, the oracle's cycle quadrature), the on-axis plasma statistics. Envelope plasma in binary32 converges on Metal with polynomial sin/cos and mlx's precise rate table (Metal's library is 2 to 3 ulp).

The A40 against native went from 1.80 to 1.15 in binary64 and from 1.56 to 0.79 in binary32 df32. Each change A/B'd against the tree before it:

change binary64 binary32 df32
Kernel bodies inlined (@inline f(t, m)): left to the inliner each sample was a device call with its arguments in local memory 0.70 0.62
Source evaluations as CUDA graphs, a gradient's stage scalars in a device vector 0.95 0.89
Four-step split of cuFFT's long binary64 transforms (one block a transform, 96 us at 5120 points; split, 23) 0.99
Tiled transposes for the Cartesian planes (CUDA only) 0.92 to 0.95 on free space
Large states copied straight between the device and the reader's buffer kerr_beam_192 0.56 0.60

Tried and reverted, each in the ledger: a whole-trial graph and an order-independent one (the order's variants, or phi functions to the largest width, cost more than the launches saved), and fused device reductions (ctc_reduced 10 to 20 % slower in binary32).

Where the A40's time goes now (Nsight, scripts/step_timeline.py, binary64, a step's median): on argon_blueshift the device's work is native's (375 against 377 us); Julia's step is 539 us against 466, all of it idle device time between the host's calls. The saved planes add about 0.5 ms of idle a plane beyond native's.

Cards: an H200 (1.69 before the graphs), an MI300X (1.28) and an H100 (1.25, with them) were rented on DigitalOcean, US$11.53 in all. On a full-rate card binary32 with df32 is slower than binary64 (rdw 0.111 against 0.058 s on the MI300X): it belongs to cards without fast binary64.

What is left

julia/STATUS.md, "Next steps", has the order and the setups. In short:

  • The GPU host: a plane's observations in one Julia call (about 1.1 ms of host time a saved plane between separate calls); the GEMMs' scalars uploaded once; graphs on ROCm.
  • line on Metal (1.5 against mlx) and Metal's start latency (about 26 s of kernel compilation; GPUCompiler no longer has a disk cache).
  • Binary32 on x86, two to three times binary64 on the Xeon, cause unknown.
  • Merging main (127 commits behind, including native's RK march), then a Julia RK march to compare.

Findings for main

No CI lane runs Julia; the heads are [skip ci].

🤖 Generated with Claude Code

https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R

jtravs and others added 30 commits November 6, 2026 13:02
Carries the prototype of #10 (Julia sources, JuliaCall adapter, bootstrap)
onto the current seam: no schema number (a source digest checked at load
instead), set_initial's resume, the Adams engine protocol's carrier, hold,
restore, observe and snapshot_nbytes, stage scalars from the driver's
bundle cache, and general-leaf and carrier-frame refusals. Julia 1.13.3,
with a fresh manifest. The benchmark harness takes julia arms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The boundary monitor's split-step reading and SEP 44's transverse edge go
through the driver's host reductions, so every focus case now takes the
same steps as native.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine is rewritten around a small device layer (launch.jl): every
state-sized operation is a scalar body launched over (t, m) by each!,
down columns by each_column!, or folded by reduce_each, on Arrays or a
GPU array type, of Float32 or Float64. Inverse FFTs are unnormalised and
their scale folded into the next coefficient; the DOPRI combination,
candidate and controller norms are one pass; FSAL keeps stage one in
place; FFTW plans follow native's policy (measure serial rows and planes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every CPU pass, transform and product runs in Polyester's pool: FFTs are
blocks of whole columns (or plane rows) with one single-threaded plan
each, GEMMs blocks of rows of single-threaded BLAS. FFTW and BLAS threads
beside the pool's spinning workers made 4-thread runs slower than one.
Every plan is measured (a threaded estimate of the Raman c2r was 4.4x
slower). Raman convolves the modal products once a source when there are
fewer than nodes, as native does: H2 Raman 25 s -> 3.7 s on four cores.
Heavy passes (the Adams phi functions) split at a smaller size, and skip
out-of-band bins, where everything is exactly zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SOLPROP_JULIA_DEVICE=cuda loads julia/cuda, CUDA.jl resolved together
with the engine, whose extension makes its arrays CuArrays; every pass
is the same body as on a CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… entries [skip ci]

The boundary monitor's power, the split's found power (formed inside the
commit from the transform the split already does) and the crossing's
occupancy are reduced by the engine; only short vectors cross. States
cross the bridge as column-major views with one owning copy. create()
compiles every entry point for the engine's types, so a run's first steps
no longer pay seconds of JIT inside the timed propagation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s [skip ci]

NumPy's OpenBLAS spun beside the engine's pool after each host product
and made a radial window four times slower; the engine leases every host
BLAS pool at one thread for its lifetime, as native does. Kernel bodies
capture zero values and scalars instead of element types, which a GPU
kernel cannot take as arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The transverse edge's node and column energies are the engine's sums;
the face tables act on the short vectors. SOLPROP_JULIA_CHECK refuses a
kernel body capturing a type, box or union on any device, so CPU runs
catch what a GPU would refuse. tests/backends/test_julia.py runs seven
focus windows against numpy_ref: the same steps and the field to 1e-10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
The series stops at a quarter ulp and multiplies by the divisors'
reciprocals, the large-argument recurrence uses one reciprocal: the
radial Adams predictor 0.16 s -> 0.10 s. A radial read's synthesis rows
are uploaded once, not at every read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…ion [skip ci]

On a GPU the plasma's passes are a sample a thread and its two running
integrals column-wise scans (SOLPROP_JULIA_PLASMA=samples takes the same
path on a CPU). create() runs each path once on the zero state, so a
GPU's kernels compile at construction rather than in the timed run.
A branch-free log for later vectorised passes (vmath.jl, not yet used).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…he bridge [skip ci]

The warm-up's Adams trials read the store through zero slots and its
occupancy wrote bucket zero, both outside their arrays under @inbounds:
intermittent wrong steps at four threads on four cores. Slots and buckets
now start valid, every Adams buffer is cleared after the warm-up, and the
tests run Julia with bounds checking. The edge, populations, occupancy
and each bridge entry's conversions are warmed at construction: a run's
first timed window now compiles 0.01-0.17 s instead of 0.6-3.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
Branch-free vlog/vexp vectorise in isolation (1 ulp), but the plasma's
table gather keeps its loops scalar, and the cached scalar exponential is
faster there: argon's plasma 1.03 s -> 1.44 s with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
The populations' column scans were 70 % of an A40 rdw window (20 ms per
call, one thread a column); they now take the current's per-sample form,
their column diagnostics array reductions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
… [skip ci]

thread_budget() reads a sysfs file per CPU; parallel_rows asked for it
before seeing that a pass was too small to thread, so every host
statistic at every saved plane paid it: 60 ms of a 0.36 s rdw window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…statistics, no copy [skip ci]

The host fills a buffer the engine copies into (and loads from) with the
pool's threads: a 226 MB Cartesian state's single-threaded first-touch
copy took 75-220 ms. The radial edge's synthesis is the pool's GEMM
(2.9 -> 0.44 ms a read), the crossing's buckets convert in place, and an
empty statistics spec copies nothing. Cartesian 192 window 1.72 -> 1.10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
precision="single" runs the engine with T=Float32: the state held as
a / s, s the launch's power-of-two scale; the cubic responses take s^2,
the plasma rate the physical field s E and the loss potential 1/s^2; the
controller's atol is divided by s. The operator and free space's separable
tables stay binary64, so propagators, the longitudinal root and the Adams
flow are formed in binary64 and rounded once. Tests compare with
numpy_ref's binary32 path to 1e-4. The harness's julia32 names the julia
backend in single precision. On x86 the binary32 window is slower than
binary64: the scaled field's tails make subnormals, and flush-to-zero is
not allowed (AGENTS.md); with it, the Kerr pass was 20x faster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…age [skip ci]

The Raman kernel's spectrum is held as spectrum * 2^-e with its factor
taking 2^e (SEP 33's block exponent): in binary32 its small entries were
subnormal, a 1.7 % field error on H2 Raman, now 1.5e-7 from numpy_ref's
binary32 path; binary32's tests cover the Raman window too. julia/README,
julia/STATUS, the manual page and a release-note fragment; the julia
package is a recorded coverage omission, like mlx. The prototype's
Julia-side tests and interop script are replaced by the pytest module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…GEMMs [skip ci]

Radially, where the nodes are as many as the modes, the source is
synthesised and projected on the band's rows: each node's spectrum is the
modes' times its synthesis row, its analytic signal one inverse transform,
and the response and both plasma streams join before one projection. A
quarter of the GEMM work and a third fewer transforms; a radial window at
16 Xeon threads 0.86 -> 0.63 s. Complex-by-real products run as real GEMMs
on interleaved views: Julia's mixed-type product is generic (slow on a
GPU) and, in binary32, wrong when run from several tasks at once (a 1.5
relative error; LinearAlgebra, Julia 1.13.3). Ledger section added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
timed_run evaluated simulation.metadata() inside its window; the first
evaluation builds the planes' bases, about 0.18 s on rdw, which every
arm's first repeat paid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
… the engine [skip ci]

The saved planes' analytic power and on-axis flux come from the engine's
state (poynting), reduced by the reference's own code; the populations
read the state the engine holds instead of a host copy. rdw at 16 threads:
statistics 55 ms -> 40 ms of the window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…ine [skip ci]

The stage tables' Chebyshev rows and the Adams basis are the engine's
transcriptions (julia/src/helpers.jl, SEP 29 section 6.6), and an Adams
trial's coefficient tables are formed in the engine from its nodes, in
the shared functions' operations and order: equal to the Python forms
bit for bit (tested). rdw at 16 threads on the Xeon: 0.273 s -> 0.214 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…uction [skip ci]

The analytic signals' buffers, their transform and the power weight are
bound once (again only when a varying guide's weight changes), and each
plane fills them without allocating: rdw's 12 planes took 13 to 83 ms,
collector pauses among them, and now take 10 to 11 ms. The plan is
measured at construction rather than inside the first plane. The fixed
provenance in metadata() is read from Julia once, not at every plane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
A varying problem's scalar tables (the operator's and prefactor's
expansion weights, the Kerr and plasma densities, the preionisation) are
bound once as PiecewiseChebyshev.packed gives them, and each trial's
scalars are formed in one engine call by the transcribed Clenshaw pass:
scalars_at's values bit for bit (tested on rdw). rdw at 16 threads:
stage preparation 17 ms -> 5 ms, the window 0.214 s -> 0.19 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
…ip ci]

The rate is formed in two passes: the samples at zero or the cap are
written and the others' indices gathered, then the gathered ones are
evaluated with no branch. The carrier takes the field across the table's
lower edge twice a cycle, a branch no predictor follows, and each miss
discarded a logarithm and an exponential in flight. The same operations
per sample (a NaN field stays live): argon's plasma on one thread 0.99 s
-> 0.88 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
julia/src/vmath.jl: branch-free binary64 exp and log, each within one ulp
of the correctly rounded value (Cody-Waite reduction, Taylor polynomials,
explicit fused steps; the role of native's vmath.hpp and SLEEF), which a
CPU vectorises under AVX2 as under AVX-512. A column's live rates run as
three passes (vectorised log, the table's lookup, vectorised exp), its
fractions and currents as vectorised passes of their own, behind function
barriers: argon's plasma on one thread 4.7 ms -> 2.9 ms a source. The GPU
and the observation populations keep Base's functions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGENPuYVwK6ASX5LHmbgEY
jtravs and others added 15 commits November 7, 2026 21:06
The free-space source forms the physical field once (the plasma reads it), then
the responses in place as a modal source's nodes take them: the Raman drive from
the field or its analytic signal, the envelope or field-grid Kerr with Raman.
Without Raman the arithmetic is unchanged. On a line, envelope and field grids
(with and without the third harmonic) take numpy_ref's steps, fields within 3e-16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
A plane's observation, statistics, boundary monitor and transverse edge each
asked for the state on the host, and each copy from a GPU waits for it (rdw on
Metal: a sixth of the window in copy_state!). Every write of the state bumps
engine.version; copy_state! keeps the last host copy and copies from the device
only for a new version. A CPU reads its own array as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
On a GPU in binary64 too, each element's free-space rate and prefactor are
formed once on the host and held as tables, as the double-float path holds them:
each source re-formed the prefactor, a root and a division, at an A40's 1/64
binary64 rate (the Cartesian window's elementwise kernels were half its 379 ms
of kernels against native's 90). A CPU keeps forming them where it reads them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
A mixture's ionising species each take their own rate table and constants
(the adapter packs them, the first unsuffixed); the engine runs the current and
population passes once a channel, the later ones adding their currents, and
returns each species' populations (a row a channel) for the shared statistics.
Each later channel's density and preionisation are stage rows after the taper.
One channel runs exactly as before. Air's two channels on a capillary match
numpy_ref: the field within 1e-10, every species statistic within 1e-9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…leads) [skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
… [skip ci]

The free-space Raman restructure formed the scaled field and then the
polarisation in two passes; on a Cartesian state that extra pass made the
192 x 192 window 25 % slower on an A40 (binary64). Without plasma or Raman the
field and its polarisation are one pass again, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…ip ci]

Varying Raman: each term's decay rate and each component's density as stage
tables; the kernel rebuilt when they change. Cartesian Adams: the free-space
rate table as the Adams operator. Envelope plasma: the cycle-averaged rate by
the oracle's angular quadrature (mlx's precise rate for binary32's refinement),
the drive and loss streams through the shared plasma path. Each tested against
numpy_ref (binary64 1e-10; binary32 1e-4 in both wide types) on DMOG CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…s on Metal [skip ci]

The on-axis statistics through the driver's shared form (a node on the axis
hands over its row, else the axis row is scanned). Envelope plasma in binary32:
sin and cos by polynomials on [0, pi/2] (Metal's library is 2-3 ulp, which near
the low cutoff exceeded the 2e-6 refinement), the cosine moment alone required
to converge (the sine moment enters nothing at a scalar envelope's zero ratio),
and a non-converged sample raises the oracle's ConvergenceError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…s on CUDA [skip ci]

cuFFT runs a binary64 transform of a few thousand points as one block (96 us at
5120 on an A40, any batch); LongFFT splits it n = n1 n2 (23 us). Cartesian
transverse transforms through a transverse-first copy: A40 0.92/0.89 binary64
(Metal slower, so CUDA only). Section timers on the free-space DOPRI stages;
julia/profile.py takes SOLPROP_PYTHON_PROFILE=1. 59 tests pass on the A40.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…p ci]

The GPU kernel called its body as a device function per sample (arguments through
local memory): 9x slower in binary64 and 20x in binary32 on an A40 than inlined.
@inline f(t, m) in each_kernel!, reductions through it: A40 binary64 0.70 over the
seven cases (fields identical); Metal neutral. SOLPROP_JULIA_KERNEL_TIMING=1 times
each GPU pass by its body. 58 of 59 CUDA tests (the compile-time window is marginal:
it passes alone), 13 Metal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…[skip ci]

The transverse planes' copies through a 32 x 32 local-memory tile (GPUArrays'
permutedims took 2.65 ms for a 384 x 36864 state on an A40); AK reductions reuse
their scratch (dozens of pool allocations a step on rdw). The FFT levers can be
asked for on any GPU (SOLPROP_JULIA_FOURSTEP=1, SOLPROP_JULIA_TRANSPOSE=1).
59 CUDA tests pass, 13 Metal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…edger [skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
… the device [skip ci]

A source's passes (kernels, cuFFT, cuBLAS) are captured once their buffers and host
values repeat, then replayed: host 4 us against 300-650 to launch them. The Kerr
factor and plasma densities/preionisations are read from a device vector uploaded
when they change, so a gradient replays one graph. Scans keep their AK workspaces
(graph-safe). A40 binary64 0.95, binary32 df32 0.89 against launched sources (fields
identical). A whole-trial graph was tried and reverted (1.60: order-dependent keys).
SOLPROP_JULIA_TIMING=host times sections without synchronising; julia/profile.py
takes SOLPROP_JULIA_ALLOCS. 59 CUDA tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
…skip ci]

A state above 16 MB goes from the device straight into the reader's buffer (a
binary32 one through a kept staging buffer), and a binary64 load straight up: the
held copy, a conversion and the reader's copy were three passes and two 226 MB
allocations on kerr_beam_192 (0.29 of its 0.42 s window). A40 kerr_beam_192 0.56
binary64, 0.60 binary32; small states unchanged. The A40 against native at
51a153d: 1.15 binary64, 0.79 binary32 df32. 59 CUDA tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
jtravs and others added 2 commits November 8, 2026 21:50
… uploads [skip ci]

Reductions form each element inside the reduction's first pass and finish on
the device (two kernels, one value read back), where AK's reduction finished its
last stage on the host: about 50 us of idle device a step on an A40. The stage
tables' and scalars' host staging is kept (no per-trial arrays; pageable on
purpose). Nsight step timelines against native (scripts/step_timeline.py) in the
ledger. A40 binary64 0.96 against e3b0726; 59 CPU and 59 CUDA tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
jtravs and others added 3 commits November 9, 2026 03:18
…ecorded [skip ci]

The fused device reductions of 1b17fc7 cost ctc_reduced 10-20 % in binary32 on an
A40 (and a hybrid did too) without a reliable gain over whole windows, so AK's
reductions with kept workspaces return; the per-trial host staging stays. An AMD
MI300X (ROCm, no graphs): native against Julia 1.28 binary64, 1.48 binary32 df32.
STATUS next steps updated. 59 CPU and 59 CUDA tests pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
… and STATUS [skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
… [skip ci]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAYtF8VxmP17Bx5wGFqR2R
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: backends Across backends, and numba_cpu do not merge experiment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant