diff --git a/bench/README.md b/bench/README.md index 7f7e1a3..364de42 100644 --- a/bench/README.md +++ b/bench/README.md @@ -142,6 +142,7 @@ one; its arms are written `fftn_omp`. | `GRID_PHASE_RADIX=R` | fftn, fftn_omp | the largest radix of the grid's phase rule (`build_settings::grid_phase_radix`; default 16): every codelet radix up to R, so 32 and 64 with `R=64` (experiment [X3](../dev/experiments/X3-big-radices.md)) | | `GRID_MIN_BYTES`, `GRID_THREE_BYTES`, `GRID_BLOCK_BYTES`, `GRID_BLOCK`, `GRID_FUSE_BYTES` | fftn, fftn_omp | the planner's grid fields of the machine profile (`src/threading/schedule.hpp`): a row of more bytes than the first is a grid, of more than the second a grid of three phases; blocks widen while one fits the third; the fourth forces the block width; up to the fifth, phases with small radices read and write rows in their end passes (`never` disables the first two and makes the fifth unlimited) | | `POW2_CHAIN_BYTES`, `POW2_RADIX8_BYTES` | fftn, fftn_omp | the machine profile's long power-of-two fields (`src/threading/schedule.hpp`, experiment [X32](../dev/experiments/X32-long-pow2.md)): a power-of-two c2c row stays a chain up to the first (`never`: at any size), and a power-of-two chain beyond `GRID_MIN_BYTES` takes radix 8 up to the second, radix 4 beyond | +| `GRID_TWO_ODD_PART` | fftn, fftn_omp | rows whose length has an odd part of at least this stay in two grid phases at any size (0: off; experiment X34) | | `GRID_PAD_PERIOD` | fftn, fftn_omp | the L1 set period the grid pads its intermediate array's rows against (`machine_profile::grid_pad_period`; default 4 KiB where the L2 is shared by a cluster, as on Apple processors, none where each core has its own); `never` turns the padding off (experiment [X26b](../dev/experiments/X26b-padded-grid.md)) | The backend builds and executes plans through the functions the public `plan` forwards diff --git a/bench/backends/fftn.cpp b/bench/backends/fftn.cpp index e4ecd92..020042e 100644 --- a/bench/backends/fftn.cpp +++ b/bench/backends/fftn.cpp @@ -57,7 +57,7 @@ // GRID_PHASE_RADIX=R the largest radix of the grid's phase rule (build_settings:: // grid_phase_radix; default 16): codelet radices up to R (experiment X3) // GRID_MIN_BYTES, GRID_THREE_BYTES, GRID_BLOCK_BYTES, GRID_BLOCK, GRID_FUSE_BYTES, -// GRID_PAD_PERIOD, POW2_CHAIN_BYTES, POW2_RADIX8_BYTES +// GRID_PAD_PERIOD, POW2_CHAIN_BYTES, POW2_RADIX8_BYTES, GRID_TWO_ODD_PART // the planner's grid fields of the machine profile (schedule.hpp) // ISA is a public option; the others reach the library only through the private build // settings, as a test's do: the library has no switch for them, so dev/KNOBS.toml does not diff --git a/bench/backends/fftn_tuning.hpp b/bench/backends/fftn_tuning.hpp index af4fdd9..a126b6a 100644 --- a/bench/backends/fftn_tuning.hpp +++ b/bench/backends/fftn_tuning.hpp @@ -87,6 +87,10 @@ constexpr std::array size_fields{ .key = "GRID_THREE_BYTES", .member = &fftn::detail::machine_profile::grid_three_bytes, }, + size_field{ + .key = "GRID_TWO_ODD_PART", + .member = &fftn::detail::machine_profile::grid_two_odd_part, + }, size_field{ .key = "GRID_FUSE_BYTES", .member = &fftn::detail::machine_profile::grid_fuse_bytes, diff --git a/dev/DESIGN.md b/dev/DESIGN.md index d4a3935..4e3d96a 100644 --- a/dev/DESIGN.md +++ b/dev/DESIGN.md @@ -1187,6 +1187,12 @@ Xeon lost 1.06–1.50 with chains). The rules price radices in cache (X2) and ch passes, whose 16 streams a power of two apart fall into one L2 set where the buffers lie on transparent huge pages: X29's chains of 1–2 MiB on the EPYC were bimodal (0.7 to 1.8 of the grid's time by process) for that reason, and chains of radices ≤ 8 are steady (X32). +Where a private L2 has less L3 per CPU behind it (the Xeon 6240), a row whose length has +an odd part (n over its largest power-of-two factor) of at least 15 stays in two phases at +any size (`grid_two_odd_part`; X34: two phases took 0.70–1.01 of three phases' cycles on +the 15 such rows tried, 71680 to 4 × 10^6 points, 100000 0.73; rows nearer a power of +two, odd part 3–9, lost up to 1.13 as two phases, and on the EPYC two phases lost +1.13–1.67 below 16 MiB). Where a cluster shares L2, a row whose length is not a power of two stays a chain when the power-of-two part of its length is at most L1/8 in bytes (1024 points in double on the M5), at any size, or when the row is at most 2 · L1 (`chain_part_bytes`, @@ -2660,6 +2666,7 @@ to change before a release. | 2026-11-15 | A pass kernel takes its source and destination as arguments (`pass_fn(args, src, dst, first, last)`), like a leaf; `pass_args` holds only the fixed geometry, and the driver calls kernels on the plan's own arguments; one transform of a flat program runs from a contiguous `flat_body`, entered from `execute_core` with its slots as scalars; `plan::execute` is one library call (X28) | The per-stage, per-transform copy of `pass_args` cost 3.7–6 ns per stage on Zen 3 (much of it store-to-load forwarding into the kernel), 1.5–2.7 on the M5 Max: single rows of 24–120 points 0.69–0.88 of their time on the Mac; only grid stages with moving twiddles and lanes blocks still copy | | 2026-11-15 | Where the L2 is shared (Apple), a c2r row out of place whose half-length transform is a two-phase grid pairs its bins in phase 1's gather (`c2r_pair_gather`, `c2r_paired_gather`), reading the input only; x86 keeps the separate pass (EPYC: 1.11); in place and three-phase grids keep the separate pairing pass; the r2c pairing stays separate (section 9.1) | X29 (Mac): 0.92–0.94 of the separate route's time at 32768–131072 and 46080 out of place, where the workspace block goes; 1.02–1.06 where the block stays (the gather reads the input twice) | | 2026-11-15 | Where a cluster shares L2 (Apple), a row whose length is not a power of two stays a chain when its power-of-two part is at most L1/8 in bytes, or the row at most 2 L1 (`chain_part_bytes`, `chain_mixed_bytes`; section 8.5) | X29 (M5 Max): chains took 0.49–0.72 of the grid's time for every such length tried (144 KiB to 62.5 MiB), the grid's L1-conflict problem being one of powers of two; tracking tier 0.983 (46080 0.63, 100000 0.60), nothing slower | +| 2026-11-16 | With a private L2 and less than 2 MiB of L3 per CPU (Xeon 6240), a row whose length has an odd part of at least 15 stays a two-phase grid at any size (`grid_two_odd_part`; section 8.5) | X34 (Xeon): two phases took 0.70–1.01 of three phases' cycles on the 15 such rows tried, 71680 to 4 × 10^6 points; fbench (run 20261116-125601-0542) 0.86 over the 14 changed rows, 100000 0.71, 10^6 0.72, none slower than 0.996; chains lost 1.05–1.45 there; the EPYC (two phases 1.13–1.67 below 16 MiB) and the Mac change no plan | | 2026-11-16 | With a private L2 and at least 2 MiB of L3 per CPU (Zen), a row whose length is not a power of two and has a power-of-two part of at most 1024 points stays a chain up to half of L3 (section 8.5) | X29 (EPYC 7763): 0.64–0.85 of the grid's time, steadily, at 46080 to 2 Mi points; power-of-two chains of 1–2 MiB were bimodal (0.7–1.8 by process) and keep the grid; the Xeon 6240 (0.7 MiB of L3 per CPU) keeps L2/2, where every chain lost 1.1–1.6 | | 2026-11-16 | Where the L2 is shared by a cluster (Apple), an intermediate the grid already keeps in the workspace block is padded after each of phase 2's rows (`grid_pad`, profile `grid_pad_period`); off on x86. The copies' prefetch keeps X26's target gate: keying it to the core was rejected (experiment X26b) | Mac: double long rows 0.976 of main (2^18 0.94); Xeon 1.006 and EPYC 0.993, where counters showed no fewer L1 refills. The prefetch on an Intel core under `avx2` took 2^22 to 0.84 but float 100000 to 1.27; X26's cheaper copy loops were rejected again on the padded grid | | 2026-11-16 | Pair leaves on x86 (experiment X28b): a contiguous c2c row of n = r1 m with C dividing m and about n / C registers runs the chain r1 × m as one leaf, the grouped pass's C × C transposes held in registers as the last pass's input; a table of lengths per target and precision (`src/isa/pairs_.cpp`); the lanes strategy runs the same chain on its tile | Single rows of 20–120 points 0.42–0.69 of the chain's time on the Xeon 6240 (`avx512`), 20–48 at 0.51–0.76 on the EPYC 7763, float 0.39–0.78; batches of 24 and 48 points 0.43–0.70; Xeon FFTN ÷ FFTW over single rows 0.98 → 0.65 with the other two X28b arms; +0.75 % library; the bits are the forced chain's | diff --git a/dev/LESSONS.md b/dev/LESSONS.md index 6a153b6..aff05f7 100644 --- a/dev/LESSONS.md +++ b/dev/LESSONS.md @@ -366,6 +366,16 @@ tried that and fbench saw nothing (0.99–1.01), because fbench's buffers sit at multiples of 64 bytes, where the near-alias cases are rare: a benchmark that never places buffers as malloc does cannot judge such a fix. [X32](experiments/X32-long-pow2.md). +3.16 **Where L3 is small and far, count the passes over memory before the cache fit of +each pass.** X12b's rule (three grid phases beyond L2, so that each phase's block stays +within L1) was set on powers of two. On the Xeon 6240 (1 MiB L2, 0.7 MiB of L3 per CPU) +rows with an odd part of 15 or more ran in 0.70–1.01 of the cycles as two phases, whose +blocks no longer fit L1, from 1.1 MiB to 64 MiB (100000 points 0.73, 10^6 0.70; L2 +misses per point 1.3 → 0.8), while on the EPYC 7763 (512 KiB L2, 4 MiB of L3 per CPU) the +same two-phase plans took 1.13–1.67 of the cycles below 16 MiB. Rows near a power of two +(odd part 3–9) lost as two phases on both: their gathers' rows still collide in L1. One +blocking rule does not fit both kinds of uncore. [X34](experiments/X34-xeon-mixed.md). + ## 4. Kernels and algorithms 4.1 **A whole-transform codelet wins only where it fills the register.** One 32-point diff --git a/dev/PLAN.md b/dev/PLAN.md index 708cdf5..4870965 100644 --- a/dev/PLAN.md +++ b/dev/PLAN.md @@ -133,8 +133,12 @@ X29's screen favoured them on the EPYC), Zen 5 unmeasured (the rule applies ther fbench placing buffers as malloc does (16 bytes into a page) to judge X32's arm (b). (b) Long real rows (r2c and c2r of 65536 and 2²⁰, 1.17–1.44): the r2c pairing fused into the grid's last phase (needs block pairs, X29), the Xeon's r2c 65536 is memory (X25). -(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X29's chain rules key on L3 per -CPU and leave the Xeon alone. (d) Rules by core on `avx2` (X26's prefetch, X30's 512 +(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X34 (pull request 21) keeps +rows with an odd part ≥ 15 in two grid phases where L3 per CPU is small (Xeon 100000 2.03 → +1.44 of FFTW, 0.95 of MKL; chains lose there). Left: 100000 still 1.44 (FFTW's +recursive four-pass plan); 46080 (1.30, a two-phase grid within L2); Granite Rapids in an +8-vCPU VM reports 63 MiB of L3 per CPU, so X29's chains apply there (100000 2.16 as a +chain): a key that covers it needs a core field or a measured choice. (d) Rules by core on `avx2` (X26's prefetch, X30's 512 override: Intel cores without AVX-512 want other choices than Zen): deferred by the maintainer (2026-11-16) until a wider set of machines shows how far cores of one ISA diverge; server chips and the Mac come first, fastest everywhere is the end goal. (e) Primes outside @@ -465,6 +469,7 @@ running, done (with verdict). | [X30b](experiments/X30b-twiddle-tables.md) | x86 chains at 1024–4096 (lead 3, after X30): 64-byte aligned plan tables (L), the half-size (wr, wi) twiddle storage only for passes with large tables (C), three-pass `avx2` double 4096 (R) | 1 | done: L adopted (aligned plan tables; EPYC x30 double 0.986, float 0.971; Xeon `avx512` 0.989, `avx2` 0.996; Mac 0.999); C rejected (EPYC x30 0.986 / 0.962 of L but float 256 × 64 1.04 in every variant, Xeon c2c 16 1.05, `avx512` double 1.001); R rejected (4096 three-pass 1.30–1.88) | | [X31](experiments/X31-cloud-1d.md) | One-dimensional double after the push on recent server chips (Zen 5, Granite Rapids with MKL, Axion): where does FFTN stand against FFTW, and does any core diverge from the EPYC, Xeon and Mac enough for per-core rules (lead (d))? | 1 | done: nothing badly off; FFTN ÷ FFTW over the 46 one-dimensional double cases Zen 5 0.80, Granite Rapids 0.88 (MKL 0.85), Axion 0.74 (before the push 1.05, 1.12, 0.86); Axion beats the Mac's ratios on long rows (0.75 against 1.19), in place, real rows and planes, mostly because FFTW is slower there; Axion 8 points 1.24; Granite Rapids repeats the Xeon's long mixed rows worse (100000 2.16, 46080 1.60) and in place 65536 1.50; Zen 5 diverges in no group; USD 0.75 | | [X32](experiments/X32-long-pow2.md) | Long powers of two (c2c ≥ 16384, lead (a)): copy-free chains of small radices (FFTL.jl's plan) against the grid, per machine; why power-of-two chains of 1–2 MiB are bimodal on the EPYC | 1 | done: chains of FFTL.jl's radices adopted on Apple (0.57–0.94 of main; ÷ FFTW 1.16 → 0.87) and Zen-like caches (EPYC 0.53–0.85); Xeon keeps the grid (chains 1.06–1.50); bimodality = radix-16 passes on transparent huge pages; scratch placement against 4K aliasing neutral in fbench, rejected | +| [X34](experiments/X34-xeon-mixed.md) | The Xeon's long rows of mixed length (100000 1.99, 46080 1.29 of FFTW): why, and does a chain rule keyed on L2, wider grid blocks for lengths with a large odd part, or other grid factors close it without a vendor switch? (lead (c)) | 1 | done: chains lose on the Xeon (1.05–1.45); wider blocks superseded; two phases for rows with an odd part ≥ 15 where L3 per CPU is small adopted (Xeon 0.86 over the 14 changed rows, 100000 0.71: 2.03 → 1.44 of FFTW, 0.95 of MKL); the EPYC (two phases 1.13–1.67) and the Mac unchanged | | P1 | Cost-model accuracy: rank agreement with measurement across the catalogue per machine | 2 | planned | | P2 | Measurement budget: how many candidates and how long before the choice stops improving | 2 | planned | | [T1](experiments/T1-dispatch-wait-costs.md) | Dispatch, wake and wait costs per machine and runtime (libgomp, libomp, Kokkos, pool) | 3 | done: hand-off cost and claim runs adopted on Zen 3; dispatch and work costs wait for a per-lane dispatch term | diff --git a/dev/experiments/X34-xeon-mixed.md b/dev/experiments/X34-xeon-mixed.md new file mode 100644 index 0000000..0f77972 --- /dev/null +++ b/dev/experiments/X34-xeon-mixed.md @@ -0,0 +1,179 @@ +# X34: long rows of mixed length on the Xeon + +- **Status:** done +- **Owner:** x34 session, 2026-11-16 +- **Phase / task:** 1 (PLAN lead (c); follows X12b, X25, X26, X26b, X29) + +## Question + +Why is FFTN's c2c row of 100000 points 1.99 times FFTW's time on the Xeon 6240 (46080: +1.29; MKL 1.31 and less), when X29's chains took the same rows to 0.66–0.75 of `main` on +the EPYC and the Mac, and which rule closes it without a vendor switch? The answer sets the +chain-or-grid rule and the grid's block and factors for lengths that are not powers of two +on machines with little L3 per core. + +## Hypothesis + +X29's chains for such lengths key on 2 MiB of L3 per CPU (EPYC) or a shared L2 (Apple); the +Xeon (1 MiB private L2, 0.69 MiB of L3 per CPU) keeps `main`'s grid. The grids it builds +there are poor for lengths with a large odd part: + +- 100000 = 2⁵ · 5⁵ is a three-phase grid 40 × 50 × 50 with blocks of 4 columns. A block is + a power of two dividing both the first factor and the columns of phase 1, so the five + factors of 2 allow at most 4: every row a copy moves is one 64-byte register (X25: + `block_copy` 40 % of the cycles, 8.2M instructions per transform against FFTW's 2.4M). +- 46080 = 2¹⁰ · 45 (720 KiB, beyond L2/2 and within L2) is a two-phase grid 192 × 240, + whose longest factor (240) limits the block to 4 within the L1 budget. + +X29's Xeon screen showed chains losing beyond L2/2 for powers of two (c2c 65536 1.23, +2¹⁹ 1.17) but did not time c2c rows with a large odd part. Expected: either the chain +(fewer passes over L3 than the grid's three copies plus passes) or a grid whose blocks are +wider (a multiple of the register's columns that need not be a power of two, 20 at 100000; +three phases at 46080) takes these rows well below `main`'s time. + +## Change under test + +Chosen after the screen (step 1), at most three arms, each a commit against `main`: + +- (a) a chain-or-grid rule for lengths that are not powers of two on machines with a + private L2, keyed on the L2 (or on the measured crossover), not on the vendor or L3 per CPU; +- (b) the grid's block a multiple of the register's complex count dividing the first factor + and the columns of phase 1, not only a power of two (100000: 20 instead of 4); +- (c) whatever the screen and counters point to (factors or phase radices of lengths with a + large odd part). + +Bits: a changed plan changes the rounding of the rows it covers; both routes are tested +against the derived tolerances (the chain and grid suites force each). + +## Method + +- Machines: Xeon Gold 6240 (`avx512`, GCC 14.4, FFTW 3.3.11 with AVX-512, MKL 2026.1, Slurm + `-c 8` shared, `--serial`); EPYC 7763 (`avx2`, GCC 16.2, DMOG `-c 8` shared) and the Mac + M5 Max (`neon_fcma`, Homebrew Clang, NEON FFTW) for collateral checks of any arm that + changes a plan there. fbench interleaved rounds with the A/A control. +- Step 1, screen (Xeon): c2c rows 2^k × odd from X29's `x29-odd-part` group and the long + mixed rows (`bench/catalogues/x29.toml`), arms A = `main`, C = `STRATEGY=chain`, F = FFTW + (`PLAN_NOTE=1`), M = MKL; for 100000 and 46080 also forced grids (`GRID_BLOCK`, + `GRID_FACTORS`, `GRID_THREE_BYTES`). `run --pmu` counters where the time does not explain + itself. +- Step 2, each arm against A = `main` on the screened cases it changes and the standard + tracking tier, on every machine-target whose plans it changes. +- Metric: median paired ratios with 95 % intervals; weighted geomeans over the cases whose + plan an arm changes; the A/A band. +- Decision rule, written before running: an arm is adopted on a machine-target if over the + cases whose plan it changes the weighted geomean B ÷ A is below 1 − band with its interval + below 1, no such case is slower than 1 + max(band, 3 %), and no tracking case is slower than + 1 + max(band, 3 %) (re-run once if a case is just past it on a loaded machine). Release + toolchains decide (GCC on x86 Linux, Clang on macOS). The EPYC and the Mac keep `main`'s + plans unless the arm also passes this rule there. + +## Results + +### Step 1, the screen (Xeon) + +Run 20261116-113928-9d84 (Slurm 16011; arms A = `main` 4ebcee5, C = chain forced, F, M; +40 c2c rows of x29's odd-part and long groups; stopped after 6 rounds, load 1.1–1.5 other +cores; the run's A/A spread ±4 %). **Chains lose on the Xeon at every row the rules make a +grid**: C ÷ A 1.05–1.45 (46080 1.33, 61440 1.45, 100000 1.06, 10⁶ 1.05); the rows that +are chains in `main` (≤ 32768 points, and the odd 59049 and 78125) are unchanged. So X29's +chains are not the Xeon's answer, whatever the rule's key. FFTN ÷ FFTW in that run: 100000 +2.02 (A 7.6 ns per point, FFTW 3.8), 46080 1.08, 10⁶ 1.04, 2 · 10⁶ 0.97, the other grids +1.0–1.5. The plans: FFTN 100000 = grid 40 × 50 × 50 in blocks of 4 (phases 5 × 8, 5 × 10, +5 × 10); FFTW 25 × 20 × 10 × 20, recursive DIT with no copies; FFTN's chain 4 × 5 × 5 × 10 +× 10 × 10. + +A forced-grid screen (run 20261116-115712-7427, Slurm 16013) stopped at round 0 on a bad +option value (`GRID_THREE_BYTES=0`); its one round gave blocks of 20 at 100000 0.77 of A +and blocks of 16 at 46080 1.02. + +### Counters (Xeon, user mode, `run --pmu`, min of 2–3 processes) + +Campaign directory `~/awork/fftn-x34` on the Xeon (Slurm 16026, 16030, 16031; node load +13–30 from other users, so cycles, not times). c2c 100000 per transform: + +| plan | cycles | instructions | L2 misses | stall cycles | +|---|---|---|---|---| +| `main`: 40 × 50 × 50, blocks of 4 | 2.73M | 8.05M | 129k | 0.45M | +| 40 × 50 × 50, blocks of 20 (arm b) | 2.20M (0.80) | 5.16M | 124k | 0.47M | +| 200 × 500, blocks of 4 (two phases) | 1.92M (0.70) | 5.84M | 78k | 0.21M | +| 200 × 500, blocks of 20 | 1.88M (0.69) | 4.42M | 83k | 0.36M | +| chain 4 × 5 × 5 × 10 × 10 × 10 | 2.81M (1.03) | 2.20M | 343k | 1.72M | + +FFTW (X25): 1.33M cycles, 2.35M instructions. The chain has FFTW's instruction count but +stalls on memory; the three-phase grid's extra pass over memory, not its copy instructions +alone, is the rest. Two phases against three on every mixed row the Xeon makes a +three-phase grid (`GRID_THREE_BYTES=never`, `main`'s blocks), cycles ÷ `main`'s: + +| odd part ≥ 15 (the rule's rows) | ratio | odd part 3–9 | ratio | +|---|---|---|---| +| 71680, 86016 | 0.924, 0.915 | 73728, 81920, 98304 | 0.951, 0.967, 0.958 | +| 100000 | 0.730 | 294912 | 1.126 | +| 184320 | 1.014 | 327680 | 1.125 | +| 204800, 221184, 245760, 256000 | 0.898, 0.875, 0.888, 0.923 | 393216 | 1.082 | +| 737280, 819200, 983040 | 0.878, 0.835, 0.765 | | | +| 10⁶, 1024000, 2 · 10⁶, 4 · 10⁶ | 0.698, 0.880, 0.738, 0.895 | | | + +L2 misses per point fall from 1.3 to 0.8–1.1 with two phases. Rows nearer a power of two +lose at 294912–393216 (likely L1 set conflicts in the two-phase gathers, whose rows lie +nearly a multiple of 4 KiB apart there), hence the odd-part key. +The same screen on the EPYC 7763 (DMOG Slurm 510630, cycles and instructions): two phases +took **1.13–1.67** of the cycles on every such row below 16 MiB (only 2 · 10⁶ and 4 · 10⁶, +0.89 and 0.76, gained), so the rule stays off where X29's chains apply. Other counter +probes: 46080 and 64000 in three phases 1.24 and 1.09 of `main`'s two (kept), 51200 1.21. + +Arm (b) (blocks that are other multiples of the register's columns, commits 8cf4187 and +713b99f) was superseded before its timing run: its one gain, 100000 in three phases (0.80), +is below two phases with `main`'s blocks (0.70), and with two phases no measured row keeps +the narrowest block in three phases. It was removed (a9c4bf1). + +### Step 2, arm (c) against `main` (Xeon) + +Arm (c) = commit d33ea09 (`grid_two_odd_part` = 15 on the Xeon; later commits only format +and comment it). Run 20261116-125601-0542 (Slurm 16033 build, 16034 timing; 16 rounds, 6 +under load; A/A band ±1.95 %), x29's 23 mixed rows from 46080 to 2 · 10⁶, arms A = +`main` 29f9772, B = d33ea09, F = FFTW, M = MKL: + +- The 14 rows whose plan changes (odd part ≥ 15, beyond L2): geomean B ÷ A **0.861**, every + one at or below 1 (largest 184320 0.996; 100000 0.711 [0.696, 0.723], 10⁶ 0.724, 2 · 10⁶ + 0.742, 819200 0.814, 983040 0.818, 737280 0.874, 1024000 0.866, 245760 0.891, 204800 + 0.892, 221184 0.900, 256000 0.938, 86016 0.968, 71680 0.983). fbench's weighted geomean + over all 23 rows 0.915 [0.901, 0.934]. +- The 9 rows whose plan does not change (46080, 51200, 64000 within L2; odd part 3–9): + 0.989–1.032, all "no evidence" (327680 1.032 [0.971, 1.107], an unchanged plan). +- FFTN ÷ FFTW: 100000 2.03 → **1.44**, 10⁶ 1.01 → 0.73, 2 · 10⁶ 0.99 → 0.73; over the 23 + rows 1.19 → 1.09. FFTN ÷ MKL: 100000 1.34 → **0.95**, 10⁶ 1.27 → 0.92; over the 23 rows + 1.10 → 1.01. 46080 (a two-phase grid within L2, unchanged) stays 1.30 of FFTW and 1.01 of + MKL. + +Tracking tier (only 100000 changes plan there): run 20261116-131703-f74f (Slurm 16035 +build, 16036 timing; 59 rounds, 11 under load; band ±1.68 %): c2c 100000 0.706 [0.699, +0.722], every other case "no evidence", the largest r2c 1024² 1.030 [0.999, 1.056] (an +unchanged plan); geomean 0.995. FFTN ÷ FFTW over the tier 0.905 → 0.900, ÷ MKL 1.007 → +1.002; 100000 2.04 → 1.44 of FFTW and 1.33 → 0.94 of MKL. + +The EPYC and the Mac: `grid_two_odd_part` is 0 there (X29's large-L3 rule and the shared +L2 respectively; `Schedule.GridSizesFromCaches` checks it), so no plan changes and no run +was needed. + +## Verdict + +- **Diagnosis.** On the Xeon, 100000 was a three-phase grid in blocks of one register + (power-of-two part 32), with 3.4 times FFTW's instructions; the third phase's pass over + memory beyond a small, slow L3 costs more than the cache fit it buys. +- **Arm (a), chains for these lengths on the Xeon: rejected by the screen** (1.05–1.45 of the + grid at every grid row; the chain at 100000 has FFTW's instruction count but stalls on + memory). No rule keyed on L2 can make chains pay here. +- **Arm (b), blocks that are other multiples of the register's columns: dropped** + (100000 0.80 in cycles, superseded by arm c's 0.70; no row left that needs it). +- **Arm (c), two phases for rows with an odd part ≥ 15 where a private L2 has less than 2 MiB + of L3 per CPU: adopted on the Xeon** by the decision rule (0.861 over the 14 changed rows, + none slower; tracking tier 0.995, only 100000 changed, 0.706). The EPYC (two phases 1.13–1.67 of the cycles below 16 MiB) and the + Mac keep `main`'s plans (the field is 0 there). DESIGN 8.5 and the decision log, LESSONS + 3.16. +- **Unverified and open.** Granite Rapids (X31: 100000 2.16, 46080 1.60) in an 8-vCPU VM + reports 504 MiB of L3 for 8 CPUs, so X29's chains apply there, not this rule: whether it + wants the Xeon's plan needs a run with `CHAIN_PART_BYTES=0 GRID_TWO_ODD_PART=15` against + the default; if it does, cache sizes as a VM reports them cannot separate it from Zen, and + the key needs a core field or a measured choice (lead (d), planner 2.4). 46080 (1.30 of + FFTW, a two-phase grid within L2: three phases 1.24, blocks of 16 1.02) and 100000's + remaining 1.44 (FFTW's recursive 25 × 20 × 10 × 20 without copies) are not closed. diff --git a/src/core/plan_core.cpp b/src/core/plan_core.cpp index c0829e6..5c72b1a 100644 --- a/src/core/plan_core.cpp +++ b/src/core/plan_core.cpp @@ -354,8 +354,15 @@ grid_split plan_grid(std::size_t n, const side& in, const side& out, const build if (!settings.radices.empty() && forced == n) { return {}; } - grid_split g = choose_grid(n, lanes, settings.profile.grid_block_bytes / (4 * sizeof(T)), - n * 2 * sizeof(T) > settings.profile.grid_three_bytes, + // Three phases beyond grid_three_bytes, unless the odd part of n keeps two (X34). + std::size_t odd = n; + while (odd % 2 == 0) { + odd /= 2; + } + const bool three = + n * 2 * sizeof(T) > settings.profile.grid_three_bytes && + (settings.profile.grid_two_odd_part == 0 || odd < settings.profile.grid_two_odd_part); + grid_split g = choose_grid(n, lanes, settings.profile.grid_block_bytes / (4 * sizeof(T)), three, settings.profile.grid_block, settings.grid_factors); g.pad = grid_pad(g, 2 * sizeof(T), settings.profile.grid_pad_period); for (std::size_t i = 0; i < g.phases; ++i) { diff --git a/src/threading/schedule.cpp b/src/threading/schedule.cpp index 4651f99..a58f8df 100644 --- a/src/threading/schedule.cpp +++ b/src/threading/schedule.cpp @@ -93,8 +93,9 @@ void set_grid_sizes(machine_profile& p, const core_class& c) noexcept { // time in some processes and 0.7 in others (leg distances a multiple of the L2's way, // conflicting when the pages lie contiguous), and the Xeon 6240 (0.7 MiB of L3 per CPU) // lost with chains beyond L2/2 at every length. - if (private_l2 && c.l3.size != 0 && c.l3.shared_by != 0 && - c.l3.size / c.l3.shared_by >= std::size_t{2} * 1024 * 1024) { + const bool large_l3 = c.l3.size != 0 && c.l3.shared_by != 0 && + c.l3.size / c.l3.shared_by >= std::size_t{2} * 1024 * 1024; + if (private_l2 && large_l3) { p.chain_part_bytes = std::size_t{16} * 1024; p.chain_part_limit_bytes = c.l3.size / 2; // Experiment X32, EPYC 7763: power-of-two rows as chains of FFTL.jl's radix-8 sequence @@ -104,6 +105,12 @@ void set_grid_sizes(machine_profile& p, const core_class& c) noexcept { p.pow2_chain_bytes = 2 * c.l3.size; p.pow2_radix8_bytes = std::numeric_limits::max(); } + // Experiment X34: where those chains do not apply (Xeon 6240, 0.7 MiB of L3 per CPU), a + // row whose length has an odd part of at least 15 stays in two grid phases beyond L2 + // (15 rows of 71680 to 4 x 10^6 points: 0.70-1.01 of the three phases' cycles, 100000 0.73). + if (private_l2 && !large_l3) { + p.grid_two_odd_part = 15; + } // Experiment X29, M5 Max: with the grid from L1, rows whose length has a power-of-two part // of at most 1024 points took 0.49-0.72 of the grid's time as chains (to 4 Mi points), // and every other length that is not a power of two 0.56-0.63 up to twice L1. diff --git a/src/threading/schedule.hpp b/src/threading/schedule.hpp index af15590..351d933 100644 --- a/src/threading/schedule.hpp +++ b/src/threading/schedule.hpp @@ -102,6 +102,14 @@ struct machine_profile { // 20261114-132841-80e0: 2^20 1.46 against 1.35 times FFTW). default_machine_profile sets // it to a quarter of L2. std::size_t grid_fuse_bytes = std::size_t{4} * 1024 * 1024; + // A row whose length n has an odd part (n over its largest power-of-two factor) of at least + // grid_two_odd_part stays in two grid phases at any size (0: off; experiment X34). On the + // Xeon 6240 (private 1 MiB L2, 0.7 MiB of L3 per CPU) two phases took 0.70-1.01 of the + // three phases' cycles on the 15 such rows measured, 71680 to 4 x 10^6 points, while rows + // nearer a power of two (odd part 3-9: 294912, 327680, 393216) lost up to 1.13 as two + // phases. set_grid_sizes sets it where a private L2 has less than 2 MiB of L3 per CPU + // behind it, the machines where X29's chains for such rows do not apply. + std::size_t grid_two_odd_part = 0; std::size_t grid_block_bytes = std::size_t{64} * 1024; std::size_t grid_block = 0; // The set period (bytes) the grid pads the rows of its intermediate array against, so that diff --git a/tests/threading/schedule_test.cpp b/tests/threading/schedule_test.cpp index 2335396..3db1fae 100644 --- a/tests/threading/schedule_test.cpp +++ b/tests/threading/schedule_test.cpp @@ -467,6 +467,10 @@ TEST(Schedule, GridSizesFromCaches) { // With 4 MiB of L3 per CPU the EPYC keeps them chains to half of its L3 (X29). EXPECT_EQ(epyc.chain_part_bytes, 16 * kib); EXPECT_EQ(epyc.chain_part_limit_bytes, 16 * 1024 * kib); + // Without that L3 the Xeon keeps rows with an odd part of 15 or more in two phases (X34). + EXPECT_EQ(xeon.grid_two_odd_part, 15U); + EXPECT_EQ(epyc.grid_two_odd_part, 0U); + EXPECT_EQ(m5.grid_two_odd_part, 0U); // The c2r grid pairs in its first gather only where the L2 is shared (X29). EXPECT_TRUE(m5.c2r_paired_gather); EXPECT_FALSE(epyc.c2r_paired_gather);