Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ one; its arms are written `fftn_omp`.
| `GRID_PHASE_RADIX=R` | fftn, fftn_omp | the largest radix of the grid's phase rule (`build_settings::grid_phase_radix`; default 16): every codelet radix up to R, so 32 and 64 with `R=64` (experiment [X3](../dev/experiments/X3-big-radices.md)) |
| `GRID_MIN_BYTES`, `GRID_THREE_BYTES`, `GRID_BLOCK_BYTES`, `GRID_BLOCK`, `GRID_FUSE_BYTES` | fftn, fftn_omp | the planner's grid fields of the machine profile (`src/threading/schedule.hpp`): a row of more bytes than the first is a grid, of more than the second a grid of three phases; blocks widen while one fits the third; the fourth forces the block width; up to the fifth, phases with small radices read and write rows in their end passes (`never` disables the first two and makes the fifth unlimited) |
| `POW2_CHAIN_BYTES`, `POW2_RADIX8_BYTES` | fftn, fftn_omp | the machine profile's long power-of-two fields (`src/threading/schedule.hpp`, experiment [X32](../dev/experiments/X32-long-pow2.md)): a power-of-two c2c row stays a chain up to the first (`never`: at any size), and a power-of-two chain beyond `GRID_MIN_BYTES` takes radix 8 up to the second, radix 4 beyond |
| `GRID_TWO_ODD_PART` | fftn, fftn_omp | rows whose length has an odd part of at least this stay in two grid phases at any size (0: off; experiment X34) |
| `GRID_PAD_PERIOD` | fftn, fftn_omp | the L1 set period the grid pads its intermediate array's rows against (`machine_profile::grid_pad_period`; default 4 KiB where the L2 is shared by a cluster, as on Apple processors, none where each core has its own); `never` turns the padding off (experiment [X26b](../dev/experiments/X26b-padded-grid.md)) |

The backend builds and executes plans through the functions the public `plan` forwards
Expand Down
2 changes: 1 addition & 1 deletion bench/backends/fftn.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
// GRID_PHASE_RADIX=R the largest radix of the grid's phase rule (build_settings::
// grid_phase_radix; default 16): codelet radices up to R (experiment X3)
// GRID_MIN_BYTES, GRID_THREE_BYTES, GRID_BLOCK_BYTES, GRID_BLOCK, GRID_FUSE_BYTES,
// GRID_PAD_PERIOD, POW2_CHAIN_BYTES, POW2_RADIX8_BYTES
// GRID_PAD_PERIOD, POW2_CHAIN_BYTES, POW2_RADIX8_BYTES, GRID_TWO_ODD_PART
// the planner's grid fields of the machine profile (schedule.hpp)
// ISA is a public option; the others reach the library only through the private build
// settings, as a test's do: the library has no switch for them, so dev/KNOBS.toml does not
Expand Down
4 changes: 4 additions & 0 deletions bench/backends/fftn_tuning.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ constexpr std::array size_fields{
.key = "GRID_THREE_BYTES",
.member = &fftn::detail::machine_profile::grid_three_bytes,
},
size_field{
.key = "GRID_TWO_ODD_PART",
.member = &fftn::detail::machine_profile::grid_two_odd_part,
},
size_field{
.key = "GRID_FUSE_BYTES",
.member = &fftn::detail::machine_profile::grid_fuse_bytes,
Expand Down
7 changes: 7 additions & 0 deletions dev/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1187,6 +1187,12 @@ Xeon lost 1.06–1.50 with chains). The rules price radices in cache (X2) and ch
passes, whose 16 streams a power of two apart fall into one L2 set where the buffers lie on
transparent huge pages: X29's chains of 1–2 MiB on the EPYC were bimodal (0.7 to 1.8 of the
grid's time by process) for that reason, and chains of radices ≤ 8 are steady (X32).
Where a private L2 has less L3 per CPU behind it (the Xeon 6240), a row whose length has
an odd part (n over its largest power-of-two factor) of at least 15 stays in two phases at
any size (`grid_two_odd_part`; X34: two phases took 0.70–1.01 of three phases' cycles on
the 15 such rows tried, 71680 to 4 × 10^6 points, 100000 0.73; rows nearer a power of
two, odd part 3–9, lost up to 1.13 as two phases, and on the EPYC two phases lost
1.13–1.67 below 16 MiB).
Where a cluster shares L2, a row whose length is not a power of two stays a chain when the
power-of-two part of its length is at most L1/8 in bytes (1024 points in double on the
M5), at any size, or when the row is at most 2 · L1 (`chain_part_bytes`,
Expand Down Expand Up @@ -2660,6 +2666,7 @@ to change before a release.
| 2026-11-15 | A pass kernel takes its source and destination as arguments (`pass_fn(args, src, dst, first, last)`), like a leaf; `pass_args` holds only the fixed geometry, and the driver calls kernels on the plan's own arguments; one transform of a flat program runs from a contiguous `flat_body`, entered from `execute_core` with its slots as scalars; `plan::execute` is one library call (X28) | The per-stage, per-transform copy of `pass_args` cost 3.7–6 ns per stage on Zen 3 (much of it store-to-load forwarding into the kernel), 1.5–2.7 on the M5 Max: single rows of 24–120 points 0.69–0.88 of their time on the Mac; only grid stages with moving twiddles and lanes blocks still copy |
| 2026-11-15 | Where the L2 is shared (Apple), a c2r row out of place whose half-length transform is a two-phase grid pairs its bins in phase 1's gather (`c2r_pair_gather`, `c2r_paired_gather`), reading the input only; x86 keeps the separate pass (EPYC: 1.11); in place and three-phase grids keep the separate pairing pass; the r2c pairing stays separate (section 9.1) | X29 (Mac): 0.92–0.94 of the separate route's time at 32768–131072 and 46080 out of place, where the workspace block goes; 1.02–1.06 where the block stays (the gather reads the input twice) |
| 2026-11-15 | Where a cluster shares L2 (Apple), a row whose length is not a power of two stays a chain when its power-of-two part is at most L1/8 in bytes, or the row at most 2 L1 (`chain_part_bytes`, `chain_mixed_bytes`; section 8.5) | X29 (M5 Max): chains took 0.49–0.72 of the grid's time for every such length tried (144 KiB to 62.5 MiB), the grid's L1-conflict problem being one of powers of two; tracking tier 0.983 (46080 0.63, 100000 0.60), nothing slower |
| 2026-11-16 | With a private L2 and less than 2 MiB of L3 per CPU (Xeon 6240), a row whose length has an odd part of at least 15 stays a two-phase grid at any size (`grid_two_odd_part`; section 8.5) | X34 (Xeon): two phases took 0.70–1.01 of three phases' cycles on the 15 such rows tried, 71680 to 4 × 10^6 points; fbench (run 20261116-125601-0542) 0.86 over the 14 changed rows, 100000 0.71, 10^6 0.72, none slower than 0.996; chains lost 1.05–1.45 there; the EPYC (two phases 1.13–1.67 below 16 MiB) and the Mac change no plan |
| 2026-11-16 | With a private L2 and at least 2 MiB of L3 per CPU (Zen), a row whose length is not a power of two and has a power-of-two part of at most 1024 points stays a chain up to half of L3 (section 8.5) | X29 (EPYC 7763): 0.64–0.85 of the grid's time, steadily, at 46080 to 2 Mi points; power-of-two chains of 1–2 MiB were bimodal (0.7–1.8 by process) and keep the grid; the Xeon 6240 (0.7 MiB of L3 per CPU) keeps L2/2, where every chain lost 1.1–1.6 |
| 2026-11-16 | Where the L2 is shared by a cluster (Apple), an intermediate the grid already keeps in the workspace block is padded after each of phase 2's rows (`grid_pad`, profile `grid_pad_period`); off on x86. The copies' prefetch keeps X26's target gate: keying it to the core was rejected (experiment X26b) | Mac: double long rows 0.976 of main (2^18 0.94); Xeon 1.006 and EPYC 0.993, where counters showed no fewer L1 refills. The prefetch on an Intel core under `avx2` took 2^22 to 0.84 but float 100000 to 1.27; X26's cheaper copy loops were rejected again on the padded grid |
| 2026-11-16 | Pair leaves on x86 (experiment X28b): a contiguous c2c row of n = r1 m with C dividing m and about n / C registers runs the chain r1 × m as one leaf, the grouped pass's C × C transposes held in registers as the last pass's input; a table of lengths per target and precision (`src/isa/pairs_<target>.cpp`); the lanes strategy runs the same chain on its tile | Single rows of 20–120 points 0.42–0.69 of the chain's time on the Xeon 6240 (`avx512`), 20–48 at 0.51–0.76 on the EPYC 7763, float 0.39–0.78; batches of 24 and 48 points 0.43–0.70; Xeon FFTN ÷ FFTW over single rows 0.98 → 0.65 with the other two X28b arms; +0.75 % library; the bits are the forced chain's |
Expand Down
10 changes: 10 additions & 0 deletions dev/LESSONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -366,6 +366,16 @@ tried that and fbench saw nothing (0.99–1.01), because fbench's buffers sit at
multiples of 64 bytes, where the near-alias cases are rare: a benchmark that never places
buffers as malloc does cannot judge such a fix. [X32](experiments/X32-long-pow2.md).

3.16 **Where L3 is small and far, count the passes over memory before the cache fit of
each pass.** X12b's rule (three grid phases beyond L2, so that each phase's block stays
within L1) was set on powers of two. On the Xeon 6240 (1 MiB L2, 0.7 MiB of L3 per CPU)
rows with an odd part of 15 or more ran in 0.70–1.01 of the cycles as two phases, whose
blocks no longer fit L1, from 1.1 MiB to 64 MiB (100000 points 0.73, 10^6 0.70; L2
misses per point 1.3 → 0.8), while on the EPYC 7763 (512 KiB L2, 4 MiB of L3 per CPU) the
same two-phase plans took 1.13–1.67 of the cycles below 16 MiB. Rows near a power of two
(odd part 3–9) lost as two phases on both: their gathers' rows still collide in L1. One
blocking rule does not fit both kinds of uncore. [X34](experiments/X34-xeon-mixed.md).

## 4. Kernels and algorithms

4.1 **A whole-transform codelet wins only where it fills the register.** One 32-point
Expand Down
9 changes: 7 additions & 2 deletions dev/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,12 @@ X29's screen favoured them on the EPYC), Zen 5 unmeasured (the rule applies ther
fbench placing buffers as malloc does (16 bytes into a page) to judge X32's arm (b).
(b) Long real rows (r2c and c2r of 65536 and 2²⁰, 1.17–1.44): the r2c pairing fused into
the grid's last phase (needs block pairs, X29), the Xeon's r2c 65536 is memory (X25).
(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X29's chain rules key on L3 per
CPU and leave the Xeon alone. (d) Rules by core on `avx2` (X26's prefetch, X30's 512
(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X34 (pull request 21) keeps
rows with an odd part ≥ 15 in two grid phases where L3 per CPU is small (Xeon 100000 2.03 →
1.44 of FFTW, 0.95 of MKL; chains lose there). Left: 100000 still 1.44 (FFTW's
recursive four-pass plan); 46080 (1.30, a two-phase grid within L2); Granite Rapids in an
8-vCPU VM reports 63 MiB of L3 per CPU, so X29's chains apply there (100000 2.16 as a
chain): a key that covers it needs a core field or a measured choice. (d) Rules by core on `avx2` (X26's prefetch, X30's 512
override: Intel cores without AVX-512 want other choices than Zen): deferred by the
maintainer (2026-11-16) until a wider set of machines shows how far cores of one ISA
diverge; server chips and the Mac come first, fastest everywhere is the end goal. (e) Primes outside
Expand Down Expand Up @@ -465,6 +469,7 @@ running, done (with verdict).
| [X30b](experiments/X30b-twiddle-tables.md) | x86 chains at 1024–4096 (lead 3, after X30): 64-byte aligned plan tables (L), the half-size (wr, wi) twiddle storage only for passes with large tables (C), three-pass `avx2` double 4096 (R) | 1 | done: L adopted (aligned plan tables; EPYC x30 double 0.986, float 0.971; Xeon `avx512` 0.989, `avx2` 0.996; Mac 0.999); C rejected (EPYC x30 0.986 / 0.962 of L but float 256 × 64 1.04 in every variant, Xeon c2c 16 1.05, `avx512` double 1.001); R rejected (4096 three-pass 1.30–1.88) |
| [X31](experiments/X31-cloud-1d.md) | One-dimensional double after the push on recent server chips (Zen 5, Granite Rapids with MKL, Axion): where does FFTN stand against FFTW, and does any core diverge from the EPYC, Xeon and Mac enough for per-core rules (lead (d))? | 1 | done: nothing badly off; FFTN ÷ FFTW over the 46 one-dimensional double cases Zen 5 0.80, Granite Rapids 0.88 (MKL 0.85), Axion 0.74 (before the push 1.05, 1.12, 0.86); Axion beats the Mac's ratios on long rows (0.75 against 1.19), in place, real rows and planes, mostly because FFTW is slower there; Axion 8 points 1.24; Granite Rapids repeats the Xeon's long mixed rows worse (100000 2.16, 46080 1.60) and in place 65536 1.50; Zen 5 diverges in no group; USD 0.75 |
| [X32](experiments/X32-long-pow2.md) | Long powers of two (c2c ≥ 16384, lead (a)): copy-free chains of small radices (FFTL.jl's plan) against the grid, per machine; why power-of-two chains of 1–2 MiB are bimodal on the EPYC | 1 | done: chains of FFTL.jl's radices adopted on Apple (0.57–0.94 of main; ÷ FFTW 1.16 → 0.87) and Zen-like caches (EPYC 0.53–0.85); Xeon keeps the grid (chains 1.06–1.50); bimodality = radix-16 passes on transparent huge pages; scratch placement against 4K aliasing neutral in fbench, rejected |
| [X34](experiments/X34-xeon-mixed.md) | The Xeon's long rows of mixed length (100000 1.99, 46080 1.29 of FFTW): why, and does a chain rule keyed on L2, wider grid blocks for lengths with a large odd part, or other grid factors close it without a vendor switch? (lead (c)) | 1 | done: chains lose on the Xeon (1.05–1.45); wider blocks superseded; two phases for rows with an odd part ≥ 15 where L3 per CPU is small adopted (Xeon 0.86 over the 14 changed rows, 100000 0.71: 2.03 → 1.44 of FFTW, 0.95 of MKL); the EPYC (two phases 1.13–1.67) and the Mac unchanged |
| P1 | Cost-model accuracy: rank agreement with measurement across the catalogue per machine | 2 | planned |
| P2 | Measurement budget: how many candidates and how long before the choice stops improving | 2 | planned |
| [T1](experiments/T1-dispatch-wait-costs.md) | Dispatch, wake and wait costs per machine and runtime (libgomp, libomp, Kokkos, pool) | 3 | done: hand-off cost and claim runs adopted on Zen 3; dispatch and work costs wait for a per-lane dispatch term |
Expand Down
Loading
Loading