From 32b668564198a02398d42f8a34cd6454ca064ab4 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 11:35:47 +0000 Subject: [PATCH 1/9] dev: X34 experiment file (the Xeon's long mixed rows, lead (c)) Co-Authored-By: Claude Opus 5.5 --- dev/PLAN.md | 1 + dev/experiments/X34-xeon-mixed.md | 76 +++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 dev/experiments/X34-xeon-mixed.md diff --git a/dev/PLAN.md b/dev/PLAN.md index b591222..2f9faf8 100644 --- a/dev/PLAN.md +++ b/dev/PLAN.md @@ -439,6 +439,7 @@ running, done (with verdict). | [X28b](experiments/X28b-short-leaves.md) | Single short rows after X28: do two-pass leaves held in registers (24, 48, 60 points and other short composites), leaves with constant offsets for rows of stride 1, and 128-bit leaves for 17, 19, 23 on x86 close what is left? (PLAN lead 4) | 1 | done: all adopted (pair leaves on x86: 20–120 points 0.42–0.69 of the chain on the Xeon's `avx512`, 0.51–0.76 on the EPYC; unit-stride leaves everywhere: 8–23 points 0.75–0.92; 128-bit 17–23 leaves 0.85–0.99); FFTN ÷ FFTW over single c2c rows Xeon 0.98 → 0.65, EPYC 0.94 → 0.73, Mac 0.80 → 0.76 | | [X30](experiments/X30-x86-pass-loops.md) | x86 chains at 128–4096 (lead 3): two-pass `avx2` double sequences with radix 32/64 (P), twiddles stored (wr, wi) with duplicating loads (T), 1024 = 16 × 64 (Q) | 1 | done: P adopted (EPYC x30 double 0.975: 256 0.905, 512 0.946, batches of 64 0.87–0.92, r2c/c2r 512–1024 0.90–0.95; nothing slower; Xeon `avx2` tracking 0.999, r2c 1024² 1.034 [0.997, 1.061] re-run pending); T rejected by single cases (geomeans 0.97–0.99, float 0.94, but short rows and small grouped passes up to 1.08–1.11); Q rejected (1024 batches 1.09) | | [X29](experiments/X29-real-footprint.md) | Real rows in the chain-or-grid rule: should r2c and c2r switch to the grid at another size than the c2c row of half their length, and is the grid's separate pairing pass worth fusing into its end phases? | 1–2 | done: real footprint rejected (no gain on any machine); c2r pairing in the grid's first gather adopted on Apple out of place in two phases (Mac 0.92–0.94; EPYC 1.11, not adopted); lengths with a power-of-two part ≤ 1024 points stay chains on Apple at any size (46080 0.65, 100000 0.62) and on Zen to half of L3 (0.64–0.85); power-of-two chains on the EPYC bimodal at 1–2 MiB, not adopted | +| [X34](experiments/X34-xeon-mixed.md) | The Xeon's long rows of mixed length (100000 1.99, 46080 1.29 of FFTW): why, and does a chain rule keyed on L2, wider grid blocks for lengths with a large odd part, or other grid factors close it without a vendor switch? (lead (c)) | 1 | running | | P1 | Cost-model accuracy: rank agreement with measurement across the catalogue per machine | 2 | planned | | P2 | Measurement budget: how many candidates and how long before the choice stops improving | 2 | planned | | [T1](experiments/T1-dispatch-wait-costs.md) | Dispatch, wake and wait costs per machine and runtime (libgomp, libomp, Kokkos, pool) | 3 | done: hand-off cost and claim runs adopted on Zen 3; dispatch and work costs wait for a per-lane dispatch term | diff --git a/dev/experiments/X34-xeon-mixed.md b/dev/experiments/X34-xeon-mixed.md new file mode 100644 index 0000000..5f9b5d7 --- /dev/null +++ b/dev/experiments/X34-xeon-mixed.md @@ -0,0 +1,76 @@ +# X34: long rows of mixed length on the Xeon + +- **Status:** running +- **Owner:** x34 session, 2026-11-16 +- **Phase / task:** 1 (PLAN lead (c); follows X12b, X25, X26, X26b, X29) + +## Question + +Why is FFTN's c2c row of 100000 points 1.99 times FFTW's time on the Xeon 6240 (46080: +1.29; MKL 1.31 and less), when X29's chains took the same rows to 0.66–0.75 of `main` on +the EPYC and the Mac, and which rule closes it without a vendor switch? The answer sets the +chain-or-grid rule and the grid's block and factors for lengths that are not powers of two +on machines with little L3 per core. + +## Hypothesis + +X29's chains for such lengths key on 2 MiB of L3 per CPU (EPYC) or a shared L2 (Apple); the +Xeon (1 MiB private L2, 0.69 MiB of L3 per CPU) keeps `main`'s grid. The grids it builds +there are poor for lengths with a large odd part: + +- 100000 = 2⁵ · 5⁵ is a three-phase grid 40 × 50 × 50 with blocks of 4 columns. A block is + a power of two dividing both the first factor and the columns of phase 1, so the five + factors of 2 allow at most 4: every row a copy moves is one 64-byte register (X25: + `block_copy` 40 % of the cycles, 8.2M instructions per transform against FFTW's 2.4M). +- 46080 = 2¹⁰ · 45 (720 KiB, beyond L2/2 and within L2) is a two-phase grid 192 × 240, + whose longest factor (240) limits the block to 4 within the L1 budget. + +X29's Xeon screen showed chains losing beyond L2/2 for powers of two (c2c 65536 1.23, +2¹⁹ 1.17) but did not time c2c rows with a large odd part. Expected: either the chain +(fewer passes over L3 than the grid's three copies plus passes) or a grid whose blocks are +wider (a multiple of the register's columns that need not be a power of two, 20 at 100000; +three phases at 46080) takes these rows well below `main`'s time. + +## Change under test + +Chosen after the screen (step 1), at most three arms, each a commit against `main`: + +- (a) a chain-or-grid rule for lengths that are not powers of two on machines with a + private L2, keyed on the L2 (or on the measured crossover), not on the vendor or L3 per CPU; +- (b) the grid's block a multiple of the register's complex count dividing the first factor + and the columns of phase 1, not only a power of two (100000: 20 instead of 4); +- (c) whatever the screen and counters point to (factors or phase radices of lengths with a + large odd part). + +Bits: a changed plan changes the rounding of the rows it covers; both routes are tested +against the derived tolerances (the chain and grid suites force each). + +## Method + +- Machines: Xeon Gold 6240 (`avx512`, GCC 14.4, FFTW 3.3.11 with AVX-512, MKL 2026.1, Slurm + `-c 8` shared, `--serial`); EPYC 7763 (`avx2`, GCC 16.2, DMOG `-c 8` shared) and the Mac + M5 Max (`neon_fcma`, Homebrew Clang, NEON FFTW) for collateral checks of any arm that + changes a plan there. fbench interleaved rounds with the A/A control. +- Step 1, screen (Xeon): c2c rows 2^k × odd from X29's `x29-odd-part` group and the long + mixed rows (`bench/catalogues/x29.toml`), arms A = `main`, C = `STRATEGY=chain`, F = FFTW + (`PLAN_NOTE=1`), M = MKL; for 100000 and 46080 also forced grids (`GRID_BLOCK`, + `GRID_FACTORS`, `GRID_THREE_BYTES`). `run --pmu` counters where the time does not explain + itself. +- Step 2, each arm against A = `main` on the screened cases it changes and the standard + tracking tier, on every machine-target whose plans it changes. +- Metric: median paired ratios with 95 % intervals; weighted geomeans over the cases whose + plan an arm changes; the A/A band. +- Decision rule, written before running: an arm is adopted on a machine-target if over the + cases whose plan it changes the weighted geomean B ÷ A is below 1 − band with its interval + below 1, no such case is slower than 1 + max(band, 3 %), and no tracking case is slower than + 1 + max(band, 3 %) (re-run once if a case is just past it on a loaded machine). Release + toolchains decide (GCC on x86 Linux, Clang on macOS). The EPYC and the Mac keep `main`'s + plans unless the arm also passes this rule there. + +## Results + +(pending) + +## Verdict + +(pending) From 8cf4187b233eda1331dfcaf6f98d43dbac99bde1 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 11:54:43 +0000 Subject: [PATCH 2/9] grid: blocks a multiple of the lanes, not only a power of two (X34 arm b) 100000 = 40 x 50 x 50 on avx512 takes blocks of 20 columns instead of 4. Co-Authored-By: Claude Opus 5.5 --- src/strategies/grid.cpp | 12 ++++++------ src/strategies/grid.hpp | 2 +- tests/core/grid_test.cpp | 7 +++++++ 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/src/strategies/grid.cpp b/src/strategies/grid.cpp index cc6ca1c..5c772b9 100644 --- a/src/strategies/grid.cpp +++ b/src/strategies/grid.cpp @@ -60,16 +60,16 @@ std::size_t first_columns(const grid_split& g) { return rest; } -// The block of a split: the widest power of two from `least` to max_grid_block dividing N1 -// and n / N1 with block x max Ni within the budget, or `least`. +// The block of a split: the widest multiple of `least` up to max_grid_block dividing N1 +// and n / N1 with block x max Ni within the budget, or `least`. Not only powers of two +// (experiment X34): 100000 = 40 x 50 x 50 has blocks of 20 instead of 4. std::size_t widest_block(const grid_split& g, std::size_t least, std::size_t budget) { const std::size_t longest = longest_factor(g); std::size_t block = least; - for (std::size_t b = 2 * least; b <= max_grid_block; b *= 2) { - if (g.factors[0] % b != 0 || first_columns(g) % b != 0 || b * longest > budget) { - break; + for (std::size_t b = 2 * least; b <= max_grid_block && b * longest <= budget; b += least) { + if (g.factors[0] % b == 0 && first_columns(g) % b == 0) { + block = b; } - block = b; } return block; } diff --git a/src/strategies/grid.hpp b/src/strategies/grid.hpp index ba2f072..512e351 100644 --- a/src/strategies/grid.hpp +++ b/src/strategies/grid.hpp @@ -83,7 +83,7 @@ struct grid_split { // factors have no prime above 13 (each phase is a chain of codelet passes), the block (a // multiple of lanes, at least 2) divides N1 and n / N1, and lanes divides N1. forced: the // factors in phase order (two or three), or empty to choose; forced_block: the block, or 0 -// to choose. The block of a split is the widest power of two from max(2, lanes) to +// to choose. The block of a split is the widest multiple of max(2, lanes) up to // max_grid_block that divides N1 and n / N1 and keeps block x max Ni within `budget` points // (the scratch lines a and b of one block in L1). The choice, among the splits into two or // (three_phases) into three: the widest block, then the shortest longest factor, then the diff --git a/tests/core/grid_test.cpp b/tests/core/grid_test.cpp index 34463fa..4d1f173 100644 --- a/tests/core/grid_test.cpp +++ b/tests/core/grid_test.cpp @@ -595,6 +595,13 @@ TEST(Grid, SplitRules) { EXPECT_EQ(g.factors[0], 8U); EXPECT_EQ(g.block, 8U); EXPECT_EQ(choose_grid(32, 8, 4096, false, 0, {}).phases, 0U); + // Not only a power of two (X34): 100000 = 2^5 5^5 with four lanes takes blocks of 20. + g = choose_grid(100000, 4, 512, true, 0, {}); + EXPECT_EQ(g.phases, 3U); + EXPECT_EQ(g.factors[0], 40U); + EXPECT_EQ(g.factors[1], 50U); + EXPECT_EQ(g.factors[2], 50U); + EXPECT_EQ(g.block, 20U); EXPECT_EQ(choose_grid(std::size_t{2} * 1031, 1, 4096, false, 0, {}).phases, 0U); // Forced factors and blocks are taken as given, or refused. g = choose_grid(1024, 2, 4096, false, 4, {32, 32}); From 713b99f61aafec0b1a4c84cee27929017bce4a79 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 12:09:02 +0000 Subject: [PATCH 3/9] grid: other multiples of the lanes only where powers of two leave the narrowest block (X34 arm b) Keeps every plan whose power-of-two block is wider than max(2, lanes) (EPYC 51200, 256000, the Mac's 51200 and 55296 would otherwise have changed, some to narrower blocks). Co-Authored-By: Claude Opus 5.5 --- src/strategies/grid.cpp | 79 +++++++++++++++++++++++----------------- src/strategies/grid.hpp | 6 ++- tests/core/grid_test.cpp | 6 ++- 3 files changed, 55 insertions(+), 36 deletions(-) diff --git a/src/strategies/grid.cpp b/src/strategies/grid.cpp index b383203..452bcc9 100644 --- a/src/strategies/grid.cpp +++ b/src/strategies/grid.cpp @@ -60,15 +60,17 @@ std::size_t first_columns(const grid_split& g) { return rest; } -// The block of a split: the widest multiple of `least` up to max_grid_block dividing N1 -// and n / N1 with block x max Ni within the budget, or `least`. Not only powers of two -// (experiment X34): 100000 = 40 x 50 x 50 has blocks of 20 instead of 4. -std::size_t widest_block(const grid_split& g, std::size_t least, std::size_t budget) { +// The block of a split: the widest power of two (any: multiple of `least`) from `least` to +// max_grid_block dividing N1 and n / N1 with block x max Ni within the budget, or `least`. +std::size_t widest_block(const grid_split& g, std::size_t least, std::size_t budget, bool any) { const std::size_t longest = longest_factor(g); std::size_t block = least; - for (std::size_t b = 2 * least; b <= max_grid_block && b * longest <= budget; b += least) { + for (std::size_t b = 2 * least; b <= max_grid_block && b * longest <= budget; + b = any ? b + least : 2 * b) { if (g.factors[0] % b == 0 && first_columns(g) % b == 0) { block = b; + } else if (!any) { + break; } } return block; @@ -200,11 +202,11 @@ grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, boo if (lanes == 0 || least % lanes != 0 || forced.size() == 1 || forced.size() > max_grid_phases) { return {}; } - const auto finish = [&](grid_split g) { + const auto finish = [&](grid_split g, bool any) { if (!runnable(g, lanes, least)) { return grid_split{}; } - g.block = forced_block != 0 ? forced_block : widest_block(g, least, budget); + g.block = forced_block != 0 ? forced_block : widest_block(g, least, budget, any); return g; }; if (!forced.empty()) { @@ -214,38 +216,49 @@ grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, boo g.factors.at(i) = forced[i]; product *= forced[i]; } - return product == n ? finish(g) : grid_split{}; + return product == n ? finish(g, false) : grid_split{}; } const std::vector ds = divisors(n); - grid_split two; - grid_split three; - for (const std::size_t n1 : ds) { - if (n1 < 2 || n1 == n) { - continue; - } - const grid_split g2 = finish({.phases = 2, .factors = {n1, n / n1, 0}}); - if (g2.phases != 0 && better(g2, two)) { - two = g2; - } - for (const std::size_t n2 : ds) { - if (n2 < 2 || (n / n1) % n2 != 0 || n / n1 == n2) { + const auto choose = [&](bool any) { + grid_split two; + grid_split three; + for (const std::size_t n1 : ds) { + if (n1 < 2 || n1 == n) { continue; } - const grid_split g3 = finish({.phases = 3, .factors = {n1, n2, n / n1 / n2}}); - if (g3.phases != 0 && better(g3, three)) { - three = g3; + const grid_split g2 = finish({.phases = 2, .factors = {n1, n / n1, 0}}, any); + if (g2.phases != 0 && better(g2, two)) { + two = g2; + } + for (const std::size_t n2 : ds) { + if (n2 < 2 || (n / n1) % n2 != 0 || n / n1 == n2) { + continue; + } + const grid_split g3 = finish({.phases = 3, .factors = {n1, n2, n / n1 / n2}}, any); + if (g3.phases != 0 && better(g3, three)) { + three = g3; + } } } - } - if (two.phases != 0 && (!three_phases || three.phases == 0)) { - return two; - } - if (three.phases != 0 && forced_block == 0) { - // Three phases are for rows far beyond L2, where X12 measured wide rows in the gathers - // and scatters worth more than keeping a block's chain within L1. - three.block = widest_block(three, least, 2 * budget); - } - return three; + if (two.phases != 0 && (!three_phases || three.phases == 0)) { + return two; + } + if (three.phases != 0 && forced_block == 0) { + // Three phases are for rows far beyond L2, where X12 measured wide rows in the gathers + // and scatters worth more than keeping a block's chain within L1. + three.block = widest_block(three, least, 2 * budget, any); + } + return three; + }; + // Blocks are powers of two. Where that leaves the narrowest block (a row of one register + // on avx512 in double), a length with a large odd part may take a block that is another + // multiple of `least` (experiment X34: 100000 = 40 x 50 x 50 in blocks of 20, not 4). + const grid_split pow2 = choose(false); + if (pow2.phases == 0 || pow2.block > least || forced_block != 0) { + return pow2; + } + const grid_split wider = choose(true); + return wider.block > pow2.block ? wider : pow2; } std::size_t grid_pad(const grid_split& g, std::size_t element_bytes, std::size_t period) { diff --git a/src/strategies/grid.hpp b/src/strategies/grid.hpp index b4ba6c6..380b4d2 100644 --- a/src/strategies/grid.hpp +++ b/src/strategies/grid.hpp @@ -83,13 +83,15 @@ struct grid_split { // factors have no prime above 13 (each phase is a chain of codelet passes), the block (a // multiple of lanes, at least 2) divides N1 and n / N1, and lanes divides N1. forced: the // factors in phase order (two or three), or empty to choose; forced_block: the block, or 0 -// to choose. The block of a split is the widest multiple of max(2, lanes) up to +// to choose. The block of a split is the widest power of two from max(2, lanes) to // max_grid_block that divides N1 and n / N1 and keeps block x max Ni within `budget` points // (the scratch lines a and b of one block in L1). The choice, among the splits into two or // (three_phases) into three: the widest block, then the shortest longest factor, then the // longest shortest factor, then the shortest first factor; three phases may fill twice the // budget (experiment X12). Two phases when three are not asked for or n has no such split, -// and three when n has no split into two. A pure function. +// and three when n has no split into two. Where that choice has the narrowest block, +// max(2, lanes), it is made again with blocks that are any multiple of it, and taken if its +// block is wider (X34: 100000 in blocks of 20 on avx512 in double). A pure function. [[nodiscard]] grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, bool three_phases, std::size_t forced_block, const std::vector& forced); diff --git a/tests/core/grid_test.cpp b/tests/core/grid_test.cpp index 4d1f173..486980b 100644 --- a/tests/core/grid_test.cpp +++ b/tests/core/grid_test.cpp @@ -596,12 +596,16 @@ TEST(Grid, SplitRules) { EXPECT_EQ(g.block, 8U); EXPECT_EQ(choose_grid(32, 8, 4096, false, 0, {}).phases, 0U); // Not only a power of two (X34): 100000 = 2^5 5^5 with four lanes takes blocks of 20. - g = choose_grid(100000, 4, 512, true, 0, {}); + g = choose_grid(100000, 4, 1024, true, 0, {}); EXPECT_EQ(g.phases, 3U); EXPECT_EQ(g.factors[0], 40U); EXPECT_EQ(g.factors[1], 50U); EXPECT_EQ(g.factors[2], 50U); EXPECT_EQ(g.block, 20U); + // Only where powers of two leave the narrowest block: 51200 keeps 32 x 40 x 40 in 32. + g = choose_grid(51200, 2, 1024, true, 0, {}); + EXPECT_EQ(g.factors[0], 32U); + EXPECT_EQ(g.block, 32U); EXPECT_EQ(choose_grid(std::size_t{2} * 1031, 1, 4096, false, 0, {}).phases, 0U); // Forced factors and blocks are taken as given, or refused. g = choose_grid(1024, 2, 4096, false, 4, {32, 32}); From a9c4bf142f9d3be4262e87f07a8f86568defe4cd Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 12:21:18 +0000 Subject: [PATCH 4/9] grid: drop X34 arm b (blocks of other multiples of the lanes) Superseded by two phases for these rows (arm c): 100000 in three phases with blocks of 20 took 0.80 of main's cycles on the Xeon, in two phases with main's blocks of 4 0.69-0.73; with two phases no measured row has the narrowest power-of-two block in three phases. Co-Authored-By: Claude Opus 5.5 --- src/strategies/grid.cpp | 81 +++++++++++++++++----------------------- src/strategies/grid.hpp | 4 +- tests/core/grid_test.cpp | 11 ------ 3 files changed, 35 insertions(+), 61 deletions(-) diff --git a/src/strategies/grid.cpp b/src/strategies/grid.cpp index 452bcc9..2d4c7ad 100644 --- a/src/strategies/grid.cpp +++ b/src/strategies/grid.cpp @@ -60,18 +60,16 @@ std::size_t first_columns(const grid_split& g) { return rest; } -// The block of a split: the widest power of two (any: multiple of `least`) from `least` to -// max_grid_block dividing N1 and n / N1 with block x max Ni within the budget, or `least`. -std::size_t widest_block(const grid_split& g, std::size_t least, std::size_t budget, bool any) { +// The block of a split: the widest power of two from `least` to max_grid_block dividing N1 +// and n / N1 with block x max Ni within the budget, or `least`. +std::size_t widest_block(const grid_split& g, std::size_t least, std::size_t budget) { const std::size_t longest = longest_factor(g); std::size_t block = least; - for (std::size_t b = 2 * least; b <= max_grid_block && b * longest <= budget; - b = any ? b + least : 2 * b) { - if (g.factors[0] % b == 0 && first_columns(g) % b == 0) { - block = b; - } else if (!any) { + for (std::size_t b = 2 * least; b <= max_grid_block; b *= 2) { + if (g.factors[0] % b != 0 || first_columns(g) % b != 0 || b * longest > budget) { break; } + block = b; } return block; } @@ -202,11 +200,11 @@ grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, boo if (lanes == 0 || least % lanes != 0 || forced.size() == 1 || forced.size() > max_grid_phases) { return {}; } - const auto finish = [&](grid_split g, bool any) { + const auto finish = [&](grid_split g) { if (!runnable(g, lanes, least)) { return grid_split{}; } - g.block = forced_block != 0 ? forced_block : widest_block(g, least, budget, any); + g.block = forced_block != 0 ? forced_block : widest_block(g, least, budget); return g; }; if (!forced.empty()) { @@ -216,49 +214,38 @@ grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, boo g.factors.at(i) = forced[i]; product *= forced[i]; } - return product == n ? finish(g, false) : grid_split{}; + return product == n ? finish(g) : grid_split{}; } const std::vector ds = divisors(n); - const auto choose = [&](bool any) { - grid_split two; - grid_split three; - for (const std::size_t n1 : ds) { - if (n1 < 2 || n1 == n) { + grid_split two; + grid_split three; + for (const std::size_t n1 : ds) { + if (n1 < 2 || n1 == n) { + continue; + } + const grid_split g2 = finish({.phases = 2, .factors = {n1, n / n1, 0}}); + if (g2.phases != 0 && better(g2, two)) { + two = g2; + } + for (const std::size_t n2 : ds) { + if (n2 < 2 || (n / n1) % n2 != 0 || n / n1 == n2) { continue; } - const grid_split g2 = finish({.phases = 2, .factors = {n1, n / n1, 0}}, any); - if (g2.phases != 0 && better(g2, two)) { - two = g2; - } - for (const std::size_t n2 : ds) { - if (n2 < 2 || (n / n1) % n2 != 0 || n / n1 == n2) { - continue; - } - const grid_split g3 = finish({.phases = 3, .factors = {n1, n2, n / n1 / n2}}, any); - if (g3.phases != 0 && better(g3, three)) { - three = g3; - } + const grid_split g3 = finish({.phases = 3, .factors = {n1, n2, n / n1 / n2}}); + if (g3.phases != 0 && better(g3, three)) { + three = g3; } } - if (two.phases != 0 && (!three_phases || three.phases == 0)) { - return two; - } - if (three.phases != 0 && forced_block == 0) { - // Three phases are for rows far beyond L2, where X12 measured wide rows in the gathers - // and scatters worth more than keeping a block's chain within L1. - three.block = widest_block(three, least, 2 * budget, any); - } - return three; - }; - // Blocks are powers of two. Where that leaves the narrowest block (a row of one register - // on avx512 in double), a length with a large odd part may take a block that is another - // multiple of `least` (experiment X34: 100000 = 40 x 50 x 50 in blocks of 20, not 4). - const grid_split pow2 = choose(false); - if (pow2.phases == 0 || pow2.block > least || forced_block != 0) { - return pow2; - } - const grid_split wider = choose(true); - return wider.block > pow2.block ? wider : pow2; + } + if (two.phases != 0 && (!three_phases || three.phases == 0)) { + return two; + } + if (three.phases != 0 && forced_block == 0) { + // Three phases are for rows far beyond L2, where X12 measured wide rows in the gathers + // and scatters worth more than keeping a block's chain within L1. + three.block = widest_block(three, least, 2 * budget); + } + return three; } std::size_t grid_pad(const grid_split& g, std::size_t element_bytes, std::size_t period) { diff --git a/src/strategies/grid.hpp b/src/strategies/grid.hpp index 380b4d2..470a6ac 100644 --- a/src/strategies/grid.hpp +++ b/src/strategies/grid.hpp @@ -89,9 +89,7 @@ struct grid_split { // (three_phases) into three: the widest block, then the shortest longest factor, then the // longest shortest factor, then the shortest first factor; three phases may fill twice the // budget (experiment X12). Two phases when three are not asked for or n has no such split, -// and three when n has no split into two. Where that choice has the narrowest block, -// max(2, lanes), it is made again with blocks that are any multiple of it, and taken if its -// block is wider (X34: 100000 in blocks of 20 on avx512 in double). A pure function. +// and three when n has no split into two. A pure function. [[nodiscard]] grid_split choose_grid(std::size_t n, std::size_t lanes, std::size_t budget, bool three_phases, std::size_t forced_block, const std::vector& forced); diff --git a/tests/core/grid_test.cpp b/tests/core/grid_test.cpp index 486980b..34463fa 100644 --- a/tests/core/grid_test.cpp +++ b/tests/core/grid_test.cpp @@ -595,17 +595,6 @@ TEST(Grid, SplitRules) { EXPECT_EQ(g.factors[0], 8U); EXPECT_EQ(g.block, 8U); EXPECT_EQ(choose_grid(32, 8, 4096, false, 0, {}).phases, 0U); - // Not only a power of two (X34): 100000 = 2^5 5^5 with four lanes takes blocks of 20. - g = choose_grid(100000, 4, 1024, true, 0, {}); - EXPECT_EQ(g.phases, 3U); - EXPECT_EQ(g.factors[0], 40U); - EXPECT_EQ(g.factors[1], 50U); - EXPECT_EQ(g.factors[2], 50U); - EXPECT_EQ(g.block, 20U); - // Only where powers of two leave the narrowest block: 51200 keeps 32 x 40 x 40 in 32. - g = choose_grid(51200, 2, 1024, true, 0, {}); - EXPECT_EQ(g.factors[0], 32U); - EXPECT_EQ(g.block, 32U); EXPECT_EQ(choose_grid(std::size_t{2} * 1031, 1, 4096, false, 0, {}).phases, 0U); // Forced factors and blocks are taken as given, or refused. g = choose_grid(1024, 2, 4096, false, 4, {32, 32}); From d33ea0967bf35213e66a39eb1def840e6faca2cf Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 12:24:54 +0000 Subject: [PATCH 5/9] planner: rows with an odd part of 15 or more stay in two grid phases where L3 per CPU is small (X34 arm c) On the Xeon 6240 (private 1 MiB L2, 0.7 MiB of L3 per CPU) two phases took 0.70-0.92 of the three phases' cycles on every mixed row measured from 71680 to 4 x 10^6 points with an odd part of 15 or more (100000 0.73, 10^6 0.70); rows nearer a power of two lost (odd part 3-9: up to 1.13). New profile field grid_two_odd_part (fbench GRID_TWO_ODD_PART), set to 15 where a private L2 has less than 2 MiB of L3 per CPU. Co-Authored-By: Claude Opus 5.5 --- bench/README.md | 1 + bench/backends/fftn.cpp | 2 +- bench/backends/fftn_tuning.hpp | 4 ++++ src/core/plan_core.cpp | 10 +++++++++- src/threading/schedule.cpp | 11 +++++++++-- src/threading/schedule.hpp | 8 ++++++++ tests/threading/schedule_test.cpp | 4 ++++ 7 files changed, 36 insertions(+), 4 deletions(-) diff --git a/bench/README.md b/bench/README.md index acb7de8..31c4a0c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -141,6 +141,7 @@ one; its arms are written `fftn_omp`. | `GRID_FACTORS=n1.n2[.n3]` | fftn, fftn_omp | the grid's factors in phase order (`build_settings::grid_factors`), for rows they multiply to; other rows plan as without it | | `GRID_PHASE_RADIX=R` | fftn, fftn_omp | the largest radix of the grid's phase rule (`build_settings::grid_phase_radix`; default 16): every codelet radix up to R, so 32 and 64 with `R=64` (experiment [X3](../dev/experiments/X3-big-radices.md)) | | `GRID_MIN_BYTES`, `GRID_THREE_BYTES`, `GRID_BLOCK_BYTES`, `GRID_BLOCK`, `GRID_FUSE_BYTES` | fftn, fftn_omp | the planner's grid fields of the machine profile (`src/threading/schedule.hpp`): a row of more bytes than the first is a grid, of more than the second a grid of three phases; blocks widen while one fits the third; the fourth forces the block width; up to the fifth, phases with small radices read and write rows in their end passes (`never` disables the first two and makes the fifth unlimited) | +| `GRID_TWO_ODD_PART` | fftn, fftn_omp | rows whose length has an odd part of at least this stay in two grid phases at any size (0: off; experiment X34) | | `GRID_PAD_PERIOD` | fftn, fftn_omp | the L1 set period the grid pads its intermediate array's rows against (`machine_profile::grid_pad_period`; default 4 KiB where the L2 is shared by a cluster, as on Apple processors, none where each core has its own); `never` turns the padding off (experiment [X26b](../dev/experiments/X26b-padded-grid.md)) | The backend builds and executes plans through the functions the public `plan` forwards diff --git a/bench/backends/fftn.cpp b/bench/backends/fftn.cpp index 681620e..e976f78 100644 --- a/bench/backends/fftn.cpp +++ b/bench/backends/fftn.cpp @@ -57,7 +57,7 @@ // GRID_PHASE_RADIX=R the largest radix of the grid's phase rule (build_settings:: // grid_phase_radix; default 16): codelet radices up to R (experiment X3) // GRID_MIN_BYTES, GRID_THREE_BYTES, GRID_BLOCK_BYTES, GRID_BLOCK, GRID_FUSE_BYTES, -// GRID_PAD_PERIOD +// GRID_PAD_PERIOD, GRID_TWO_ODD_PART // the planner's grid fields of the machine profile (schedule.hpp) // ISA is a public option; the others reach the library only through the private build // settings, as a test's do: the library has no switch for them, so dev/KNOBS.toml does not diff --git a/bench/backends/fftn_tuning.hpp b/bench/backends/fftn_tuning.hpp index e57e91f..02e1100 100644 --- a/bench/backends/fftn_tuning.hpp +++ b/bench/backends/fftn_tuning.hpp @@ -87,6 +87,10 @@ constexpr std::array size_fields{ .key = "GRID_THREE_BYTES", .member = &fftn::detail::machine_profile::grid_three_bytes, }, + size_field{ + .key = "GRID_TWO_ODD_PART", + .member = &fftn::detail::machine_profile::grid_two_odd_part, + }, size_field{ .key = "GRID_FUSE_BYTES", .member = &fftn::detail::machine_profile::grid_fuse_bytes, diff --git a/src/core/plan_core.cpp b/src/core/plan_core.cpp index dbb32da..d1aad8c 100644 --- a/src/core/plan_core.cpp +++ b/src/core/plan_core.cpp @@ -352,8 +352,16 @@ grid_split plan_grid(std::size_t n, const side& in, const side& out, const build if (!settings.radices.empty() && forced == n) { return {}; } + // Three phases beyond grid_three_bytes, unless the odd part of n keeps two (X34). + std::size_t odd = n; + while (odd % 2 == 0) { + odd /= 2; + } + const bool three = n * 2 * sizeof(T) > settings.profile.grid_three_bytes && + (settings.profile.grid_two_odd_part == 0 || + odd < settings.profile.grid_two_odd_part); grid_split g = choose_grid(n, lanes, settings.profile.grid_block_bytes / (4 * sizeof(T)), - n * 2 * sizeof(T) > settings.profile.grid_three_bytes, + three, settings.profile.grid_block, settings.grid_factors); g.pad = grid_pad(g, 2 * sizeof(T), settings.profile.grid_pad_period); for (std::size_t i = 0; i < g.phases; ++i) { diff --git a/src/threading/schedule.cpp b/src/threading/schedule.cpp index 44727b6..dd8e3d1 100644 --- a/src/threading/schedule.cpp +++ b/src/threading/schedule.cpp @@ -93,11 +93,18 @@ void set_grid_sizes(machine_profile& p, const core_class& c) noexcept { // time in some processes and 0.7 in others (leg distances a multiple of the L2's way, // conflicting when the pages lie contiguous), and the Xeon 6240 (0.7 MiB of L3 per CPU) // lost with chains beyond L2/2 at every length. - if (private_l2 && c.l3.size != 0 && c.l3.shared_by != 0 && - c.l3.size / c.l3.shared_by >= std::size_t{2} * 1024 * 1024) { + const bool large_l3 = c.l3.size != 0 && c.l3.shared_by != 0 && + c.l3.size / c.l3.shared_by >= std::size_t{2} * 1024 * 1024; + if (private_l2 && large_l3) { p.chain_part_bytes = std::size_t{16} * 1024; p.chain_part_limit_bytes = c.l3.size / 2; } + // Experiment X34: where those chains do not apply (Xeon 6240, 0.7 MiB of L3 per CPU), a + // row whose length has an odd part of at least 15 stays in two grid phases beyond L2 + // (71680 to 4 x 10^6 points: 0.70-0.92 of the three phases' cycles, 100000 0.73). + if (private_l2 && !large_l3) { + p.grid_two_odd_part = 15; + } // Experiment X29, M5 Max: with the grid from L1, rows whose length has a power-of-two part // of at most 1024 points took 0.49-0.72 of the grid's time as chains (to 4 Mi points), // and every other length that is not a power of two 0.56-0.63 up to twice L1. diff --git a/src/threading/schedule.hpp b/src/threading/schedule.hpp index 2ba7411..36ec42f 100644 --- a/src/threading/schedule.hpp +++ b/src/threading/schedule.hpp @@ -102,6 +102,14 @@ struct machine_profile { // 20261114-132841-80e0: 2^20 1.46 against 1.35 times FFTW). default_machine_profile sets // it to a quarter of L2. std::size_t grid_fuse_bytes = std::size_t{4} * 1024 * 1024; + // A row whose length n has an odd part (n over its largest power-of-two factor) of at least + // grid_two_odd_part stays in two grid phases at any size (0: off; experiment X34). On the + // Xeon 6240 (private 1 MiB L2, 0.7 MiB of L3 per CPU) two phases took 0.70-0.92 of the + // three phases' cycles on every such row measured, 71680 to 4 x 10^6 points, while rows + // nearer a power of two (odd part 3-9: 294912, 327680, 393216) lost up to 1.13 as two + // phases. set_grid_sizes sets it where a private L2 has less than 2 MiB of L3 per CPU + // behind it, the machines where X29's chains for such rows do not apply. + std::size_t grid_two_odd_part = 0; std::size_t grid_block_bytes = std::size_t{64} * 1024; std::size_t grid_block = 0; // The set period (bytes) the grid pads the rows of its intermediate array against, so that diff --git a/tests/threading/schedule_test.cpp b/tests/threading/schedule_test.cpp index f172f44..908c17e 100644 --- a/tests/threading/schedule_test.cpp +++ b/tests/threading/schedule_test.cpp @@ -467,6 +467,10 @@ TEST(Schedule, GridSizesFromCaches) { // With 4 MiB of L3 per CPU the EPYC keeps them chains to half of its L3 (X29). EXPECT_EQ(epyc.chain_part_bytes, 16 * kib); EXPECT_EQ(epyc.chain_part_limit_bytes, 16 * 1024 * kib); + // Without that L3 the Xeon keeps rows with an odd part of 15 or more in two phases (X34). + EXPECT_EQ(xeon.grid_two_odd_part, 15U); + EXPECT_EQ(epyc.grid_two_odd_part, 0U); + EXPECT_EQ(m5.grid_two_odd_part, 0U); // The c2r grid pairs in its first gather only where the L2 is shared (X29). EXPECT_TRUE(m5.c2r_paired_gather); EXPECT_FALSE(epyc.c2r_paired_gather); From 16c9b465bdad7c6ce3d56a4a12579da5e4ae3057 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 12:35:20 +0000 Subject: [PATCH 6/9] planner: clang-format (X34) Co-Authored-By: Claude Opus 5.5 --- src/core/plan_core.cpp | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/src/core/plan_core.cpp b/src/core/plan_core.cpp index d1aad8c..7f359d9 100644 --- a/src/core/plan_core.cpp +++ b/src/core/plan_core.cpp @@ -357,11 +357,10 @@ grid_split plan_grid(std::size_t n, const side& in, const side& out, const build while (odd % 2 == 0) { odd /= 2; } - const bool three = n * 2 * sizeof(T) > settings.profile.grid_three_bytes && - (settings.profile.grid_two_odd_part == 0 || - odd < settings.profile.grid_two_odd_part); - grid_split g = choose_grid(n, lanes, settings.profile.grid_block_bytes / (4 * sizeof(T)), - three, + const bool three = + n * 2 * sizeof(T) > settings.profile.grid_three_bytes && + (settings.profile.grid_two_odd_part == 0 || odd < settings.profile.grid_two_odd_part); + grid_split g = choose_grid(n, lanes, settings.profile.grid_block_bytes / (4 * sizeof(T)), three, settings.profile.grid_block, settings.grid_factors); g.pad = grid_pad(g, 2 * sizeof(T), settings.profile.grid_pad_period); for (std::size_t i = 0; i < g.phases; ++i) { From 7753b0a45115751b04ad2de616bf7cb52e5cb035 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 12:39:30 +0000 Subject: [PATCH 7/9] schedule: X34 comments give the measured range (0.70-1.01 over 15 rows) Co-Authored-By: Claude Opus 5.5 --- src/threading/schedule.cpp | 2 +- src/threading/schedule.hpp | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/threading/schedule.cpp b/src/threading/schedule.cpp index dd8e3d1..6ab91ca 100644 --- a/src/threading/schedule.cpp +++ b/src/threading/schedule.cpp @@ -101,7 +101,7 @@ void set_grid_sizes(machine_profile& p, const core_class& c) noexcept { } // Experiment X34: where those chains do not apply (Xeon 6240, 0.7 MiB of L3 per CPU), a // row whose length has an odd part of at least 15 stays in two grid phases beyond L2 - // (71680 to 4 x 10^6 points: 0.70-0.92 of the three phases' cycles, 100000 0.73). + // (15 rows of 71680 to 4 x 10^6 points: 0.70-1.01 of the three phases' cycles, 100000 0.73). if (private_l2 && !large_l3) { p.grid_two_odd_part = 15; } diff --git a/src/threading/schedule.hpp b/src/threading/schedule.hpp index 36ec42f..bd9a6e3 100644 --- a/src/threading/schedule.hpp +++ b/src/threading/schedule.hpp @@ -104,8 +104,8 @@ struct machine_profile { std::size_t grid_fuse_bytes = std::size_t{4} * 1024 * 1024; // A row whose length n has an odd part (n over its largest power-of-two factor) of at least // grid_two_odd_part stays in two grid phases at any size (0: off; experiment X34). On the - // Xeon 6240 (private 1 MiB L2, 0.7 MiB of L3 per CPU) two phases took 0.70-0.92 of the - // three phases' cycles on every such row measured, 71680 to 4 x 10^6 points, while rows + // Xeon 6240 (private 1 MiB L2, 0.7 MiB of L3 per CPU) two phases took 0.70-1.01 of the + // three phases' cycles on the 15 such rows measured, 71680 to 4 x 10^6 points, while rows // nearer a power of two (odd part 3-9: 294912, 327680, 393216) lost up to 1.13 as two // phases. set_grid_sizes sets it where a private L2 has less than 2 MiB of L3 per CPU // behind it, the machines where X29's chains for such rows do not apply. From 1fc042b388f341b820482e8007ad15048f2a849e Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 13:19:25 +0000 Subject: [PATCH 8/9] dev: X34 results, verdict, DESIGN 8.5 and decision log, LESSONS 3.14, PLAN Co-Authored-By: Claude Opus 5.5 --- dev/DESIGN.md | 6 ++ dev/LESSONS.md | 10 +++ dev/PLAN.md | 10 ++- dev/experiments/X34-xeon-mixed.md | 105 +++++++++++++++++++++++++++++- 4 files changed, 125 insertions(+), 6 deletions(-) diff --git a/dev/DESIGN.md b/dev/DESIGN.md index 2d7de77..b2932b5 100644 --- a/dev/DESIGN.md +++ b/dev/DESIGN.md @@ -1178,6 +1178,11 @@ stays a chain up to half of L3 (`chain_part_limit_bytes`; X29: 0.64–0.85 of th time on the EPYC 7763 at 46080 to 2 Mi points, steadily; power-of-two chains of 1–2 MiB were bimodal there, 0.7 to 1.8 of the grid's time from process to process, so powers of two keep L2/2; the Xeon 6240, with 0.7 MiB of L3 per CPU, keeps L2/2 for every length). +Where a private L2 has less L3 per CPU behind it (the Xeon 6240), a row whose length has +an odd part (n over its largest power-of-two factor) of at least 15 stays in two phases at +any size (`grid_two_odd_part`; X34: two phases took 0.70–1.01 of three phases' cycles on +the 15 such rows tried, 71680 to 4 × 10^6 points, 100000 0.73; rows nearer a power of two, odd part 3–9, lost up to +1.13 as two phases, and on the EPYC two phases lost 1.13–1.67 below 16 MiB). Where a cluster shares L2, a row whose length is not a power of two stays a chain when the power-of-two part of its length is at most L1/8 in bytes (1024 points in double on the M5), at any size, or when the row is at most 2 · L1 (`chain_part_bytes`, @@ -2651,6 +2656,7 @@ to change before a release. | 2026-11-15 | A pass kernel takes its source and destination as arguments (`pass_fn(args, src, dst, first, last)`), like a leaf; `pass_args` holds only the fixed geometry, and the driver calls kernels on the plan's own arguments; one transform of a flat program runs from a contiguous `flat_body`, entered from `execute_core` with its slots as scalars; `plan::execute` is one library call (X28) | The per-stage, per-transform copy of `pass_args` cost 3.7–6 ns per stage on Zen 3 (much of it store-to-load forwarding into the kernel), 1.5–2.7 on the M5 Max: single rows of 24–120 points 0.69–0.88 of their time on the Mac; only grid stages with moving twiddles and lanes blocks still copy | | 2026-11-15 | Where the L2 is shared (Apple), a c2r row out of place whose half-length transform is a two-phase grid pairs its bins in phase 1's gather (`c2r_pair_gather`, `c2r_paired_gather`), reading the input only; x86 keeps the separate pass (EPYC: 1.11); in place and three-phase grids keep the separate pairing pass; the r2c pairing stays separate (section 9.1) | X29 (Mac): 0.92–0.94 of the separate route's time at 32768–131072 and 46080 out of place, where the workspace block goes; 1.02–1.06 where the block stays (the gather reads the input twice) | | 2026-11-15 | Where a cluster shares L2 (Apple), a row whose length is not a power of two stays a chain when its power-of-two part is at most L1/8 in bytes, or the row at most 2 L1 (`chain_part_bytes`, `chain_mixed_bytes`; section 8.5) | X29 (M5 Max): chains took 0.49–0.72 of the grid's time for every such length tried (144 KiB to 62.5 MiB), the grid's L1-conflict problem being one of powers of two; tracking tier 0.983 (46080 0.63, 100000 0.60), nothing slower | +| 2026-11-16 | With a private L2 and less than 2 MiB of L3 per CPU (Xeon 6240), a row whose length has an odd part of at least 15 stays a two-phase grid at any size (`grid_two_odd_part`; section 8.5) | X34 (Xeon): two phases took 0.70–1.01 of three phases' cycles on the 15 such rows tried, 71680 to 4 × 10^6 points; fbench (run 20261116-125601-0542) 0.86 over the 14 changed rows, 100000 0.71, 10^6 0.72, none slower than 0.996; chains lost 1.05–1.45 there; the EPYC (two phases 1.13–1.67 below 16 MiB) and the Mac change no plan | | 2026-11-16 | With a private L2 and at least 2 MiB of L3 per CPU (Zen), a row whose length is not a power of two and has a power-of-two part of at most 1024 points stays a chain up to half of L3 (section 8.5) | X29 (EPYC 7763): 0.64–0.85 of the grid's time, steadily, at 46080 to 2 Mi points; power-of-two chains of 1–2 MiB were bimodal (0.7–1.8 by process) and keep the grid; the Xeon 6240 (0.7 MiB of L3 per CPU) keeps L2/2, where every chain lost 1.1–1.6 | | 2026-11-16 | Where the L2 is shared by a cluster (Apple), an intermediate the grid already keeps in the workspace block is padded after each of phase 2's rows (`grid_pad`, profile `grid_pad_period`); off on x86. The copies' prefetch keeps X26's target gate: keying it to the core was rejected (experiment X26b) | Mac: double long rows 0.976 of main (2^18 0.94); Xeon 1.006 and EPYC 0.993, where counters showed no fewer L1 refills. The prefetch on an Intel core under `avx2` took 2^22 to 0.84 but float 100000 to 1.27; X26's cheaper copy loops were rejected again on the padded grid | | 2026-11-16 | Pair leaves on x86 (experiment X28b): a contiguous c2c row of n = r1 m with C dividing m and about n / C registers runs the chain r1 × m as one leaf, the grouped pass's C × C transposes held in registers as the last pass's input; a table of lengths per target and precision (`src/isa/pairs_.cpp`); the lanes strategy runs the same chain on its tile | Single rows of 20–120 points 0.42–0.69 of the chain's time on the Xeon 6240 (`avx512`), 20–48 at 0.51–0.76 on the EPYC 7763, float 0.39–0.78; batches of 24 and 48 points 0.43–0.70; Xeon FFTN ÷ FFTW over single rows 0.98 → 0.65 with the other two X28b arms; +0.75 % library; the bits are the forced chain's | diff --git a/dev/LESSONS.md b/dev/LESSONS.md index 2e8cd81..cf08c14 100644 --- a/dev/LESSONS.md +++ b/dev/LESSONS.md @@ -340,6 +340,16 @@ in-place and three-phase rows 2–6 % slower, where the array stayed and the inp twice instead of once by a streaming in-place pass; on the EPYC even the out-of-place rows lost (1.11). [X29](experiments/X29-real-footprint.md). +3.14 **Where L3 is small and far, count the passes over memory before the cache fit of +each pass.** X12b's rule (three grid phases beyond L2, so that each phase's block stays +within L1) was set on powers of two. On the Xeon 6240 (1 MiB L2, 0.7 MiB of L3 per CPU) +rows with an odd part of 15 or more ran in 0.70–1.01 of the cycles as two phases, whose +blocks no longer fit L1, from 1.1 MiB to 64 MiB (100000 points 0.73, 10^6 0.70; L2 +misses per point 1.3 → 0.8), while on the EPYC 7763 (512 KiB L2, 4 MiB of L3 per CPU) the +same two-phase plans took 1.13–1.67 of the cycles below 16 MiB. Rows near a power of two +(odd part 3–9) lost as two phases on both: their gathers' rows still collide in L1. One +blocking rule does not fit both kinds of uncore. [X34](experiments/X34-xeon-mixed.md). + ## 4. Kernels and algorithms 4.1 **A whole-transform codelet wins only where it fills the register.** One 32-point diff --git a/dev/PLAN.md b/dev/PLAN.md index 948f080..34b694d 100644 --- a/dev/PLAN.md +++ b/dev/PLAN.md @@ -126,8 +126,12 @@ FFTL): a new idea is needed, since cheaper copies (X26), padding (X26b) and Zen 1–2 MiB (X29 d) failed; FFTL.jl's copy-free radix-4 Stockham chains are the reference. (b) Long real rows (r2c and c2r of 65536 and 2²⁰, 1.17–1.44): the r2c pairing fused into the grid's last phase (needs block pairs, X29), the Xeon's r2c 65536 is memory (X25). -(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X29's chain rules key on L3 per -CPU and leave the Xeon alone. (d) Rules by core on `avx2` (X26's prefetch, X30's 512 +(c) The Xeon's long mixed rows (100000 1.99, 46080 1.29): X34 (pull request 21) keeps +rows with an odd part ≥ 15 in two grid phases where L3 per CPU is small (Xeon 100000 2.03 → +1.44 of FFTW, 0.95 of MKL; chains lose there). Left: 100000 still 1.44 (FFTW's +recursive four-pass plan); 46080 (1.30, a two-phase grid within L2); Granite Rapids in an +8-vCPU VM reports 63 MiB of L3 per CPU, so X29's chains apply there (100000 2.16 as a +chain): a key that covers it needs a core field or a measured choice. (d) Rules by core on `avx2` (X26's prefetch, X30's 512 override: Intel cores without AVX-512 want other choices than Zen): deferred by the maintainer (2026-11-16) until a wider set of machines shows how far cores of one ISA diverge; server chips and the Mac come first, fastest everywhere is the end goal. (e) Primes outside @@ -457,7 +461,7 @@ running, done (with verdict). | [X29](experiments/X29-real-footprint.md) | Real rows in the chain-or-grid rule: should r2c and c2r switch to the grid at another size than the c2c row of half their length, and is the grid's separate pairing pass worth fusing into its end phases? | 1–2 | done: real footprint rejected (no gain on any machine); c2r pairing in the grid's first gather adopted on Apple out of place in two phases (Mac 0.92–0.94; EPYC 1.11, not adopted); lengths with a power-of-two part ≤ 1024 points stay chains on Apple at any size (46080 0.65, 100000 0.62) and on Zen to half of L3 (0.64–0.85); power-of-two chains on the EPYC bimodal at 1–2 MiB, not adopted | | [X30b](experiments/X30b-twiddle-tables.md) | x86 chains at 1024–4096 (lead 3, after X30): 64-byte aligned plan tables (L), the half-size (wr, wi) twiddle storage only for passes with large tables (C), three-pass `avx2` double 4096 (R) | 1 | done: L adopted (aligned plan tables; EPYC x30 double 0.986, float 0.971; Xeon `avx512` 0.989, `avx2` 0.996; Mac 0.999); C rejected (EPYC x30 0.986 / 0.962 of L but float 256 × 64 1.04 in every variant, Xeon c2c 16 1.05, `avx512` double 1.001); R rejected (4096 three-pass 1.30–1.88) | | [X31](experiments/X31-cloud-1d.md) | One-dimensional double after the push on recent server chips (Zen 5, Granite Rapids with MKL, Axion): where does FFTN stand against FFTW, and does any core diverge from the EPYC, Xeon and Mac enough for per-core rules (lead (d))? | 1 | done: nothing badly off; FFTN ÷ FFTW over the 46 one-dimensional double cases Zen 5 0.80, Granite Rapids 0.88 (MKL 0.85), Axion 0.74 (before the push 1.05, 1.12, 0.86); Axion beats the Mac's ratios on long rows (0.75 against 1.19), in place, real rows and planes, mostly because FFTW is slower there; Axion 8 points 1.24; Granite Rapids repeats the Xeon's long mixed rows worse (100000 2.16, 46080 1.60) and in place 65536 1.50; Zen 5 diverges in no group; USD 0.75 | -| [X34](experiments/X34-xeon-mixed.md) | The Xeon's long rows of mixed length (100000 1.99, 46080 1.29 of FFTW): why, and does a chain rule keyed on L2, wider grid blocks for lengths with a large odd part, or other grid factors close it without a vendor switch? (lead (c)) | 1 | running | +| [X34](experiments/X34-xeon-mixed.md) | The Xeon's long rows of mixed length (100000 1.99, 46080 1.29 of FFTW): why, and does a chain rule keyed on L2, wider grid blocks for lengths with a large odd part, or other grid factors close it without a vendor switch? (lead (c)) | 1 | done: chains lose on the Xeon (1.05–1.45); wider blocks superseded; two phases for rows with an odd part ≥ 15 where L3 per CPU is small adopted (Xeon 0.86 over the 14 changed rows, 100000 0.71: 2.03 → 1.44 of FFTW, 0.95 of MKL); the EPYC (two phases 1.13–1.67) and the Mac unchanged | | P1 | Cost-model accuracy: rank agreement with measurement across the catalogue per machine | 2 | planned | | P2 | Measurement budget: how many candidates and how long before the choice stops improving | 2 | planned | | [T1](experiments/T1-dispatch-wait-costs.md) | Dispatch, wake and wait costs per machine and runtime (libgomp, libomp, Kokkos, pool) | 3 | done: hand-off cost and claim runs adopted on Zen 3; dispatch and work costs wait for a per-lane dispatch term | diff --git a/dev/experiments/X34-xeon-mixed.md b/dev/experiments/X34-xeon-mixed.md index 5f9b5d7..f7a803b 100644 --- a/dev/experiments/X34-xeon-mixed.md +++ b/dev/experiments/X34-xeon-mixed.md @@ -1,6 +1,6 @@ # X34: long rows of mixed length on the Xeon -- **Status:** running +- **Status:** done - **Owner:** x34 session, 2026-11-16 - **Phase / task:** 1 (PLAN lead (c); follows X12b, X25, X26, X26b, X29) @@ -69,8 +69,107 @@ against the derived tolerances (the chain and grid suites force each). ## Results -(pending) +### Step 1, the screen (Xeon) + +Run 20261116-113928-9d84 (Slurm 16011; arms A = `main` 4ebcee5, C = chain forced, F, M; +40 c2c rows of x29's odd-part and long groups; stopped after 6 rounds, load 1.1–1.5 other +cores; the run's A/A spread ±4 %). **Chains lose on the Xeon at every row the rules make a +grid**: C ÷ A 1.05–1.45 (46080 1.33, 61440 1.45, 100000 1.06, 10⁶ 1.05); the rows that +are chains in `main` (≤ 32768 points, and the odd 59049 and 78125) are unchanged. So X29's +chains are not the Xeon's answer, whatever the rule's key. FFTN ÷ FFTW in that run: 100000 +2.02 (A 7.6 ns per point, FFTW 3.8), 46080 1.08, 10⁶ 1.04, 2 · 10⁶ 0.97, the other grids +1.0–1.5. The plans: FFTN 100000 = grid 40 × 50 × 50 in blocks of 4 (phases 5 × 8, 5 × 10, +5 × 10); FFTW 25 × 20 × 10 × 20, recursive DIT with no copies; FFTN's chain 4 × 5 × 5 × 10 +× 10 × 10. + +A forced-grid screen (run 20261116-115712-7427, Slurm 16013) stopped at round 0 on a bad +option value (`GRID_THREE_BYTES=0`); its one round gave blocks of 20 at 100000 0.77 of A +and blocks of 16 at 46080 1.02. + +### Counters (Xeon, user mode, `run --pmu`, min of 2–3 processes) + +Campaign directory `~/awork/fftn-x34` on the Xeon (Slurm 16026, 16030, 16031; node load +13–30 from other users, so cycles, not times). c2c 100000 per transform: + +| plan | cycles | instructions | L2 misses | stall cycles | +|---|---|---|---|---| +| `main`: 40 × 50 × 50, blocks of 4 | 2.73M | 8.05M | 129k | 0.45M | +| 40 × 50 × 50, blocks of 20 (arm b) | 2.20M (0.80) | 5.16M | 124k | 0.47M | +| 200 × 500, blocks of 4 (two phases) | 1.92M (0.70) | 5.84M | 78k | 0.21M | +| 200 × 500, blocks of 20 | 1.88M (0.69) | 4.42M | 83k | 0.36M | +| chain 4 × 5 × 5 × 10 × 10 × 10 | 2.81M (1.03) | 2.20M | 343k | 1.72M | + +FFTW (X25): 1.33M cycles, 2.35M instructions. The chain has FFTW's instruction count but +stalls on memory; the three-phase grid's extra pass over memory, not its copy instructions +alone, is the rest. Two phases against three on every mixed row the Xeon makes a +three-phase grid (`GRID_THREE_BYTES=never`, `main`'s blocks), cycles ÷ `main`'s: + +| odd part ≥ 15 (the rule's rows) | ratio | odd part 3–9 | ratio | +|---|---|---|---| +| 71680, 86016 | 0.924, 0.915 | 73728, 81920, 98304 | 0.951, 0.967, 0.958 | +| 100000 | 0.730 | 294912 | 1.126 | +| 184320 | 1.014 | 327680 | 1.125 | +| 204800, 221184, 245760, 256000 | 0.898, 0.875, 0.888, 0.923 | 393216 | 1.082 | +| 737280, 819200, 983040 | 0.878, 0.835, 0.765 | | | +| 10⁶, 1024000, 2 · 10⁶, 4 · 10⁶ | 0.698, 0.880, 0.738, 0.895 | | | + +L2 misses per point fall from 1.3 to 0.8–1.1 with two phases. Rows nearer a power of two +lose at 294912–393216 (likely L1 set conflicts in the two-phase gathers, whose rows lie +nearly a multiple of 4 KiB apart there), hence the odd-part key. +The same screen on the EPYC 7763 (DMOG Slurm 510630, cycles and instructions): two phases +took **1.13–1.67** of the cycles on every such row below 16 MiB (only 2 · 10⁶ and 4 · 10⁶, +0.89 and 0.76, gained), so the rule stays off where X29's chains apply. Other counter +probes: 46080 and 64000 in three phases 1.24 and 1.09 of `main`'s two (kept), 51200 1.21. + +Arm (b) (blocks that are other multiples of the register's columns, commits 8cf4187 and +713b99f) was superseded before its timing run: its one gain, 100000 in three phases (0.80), +is below two phases with `main`'s blocks (0.70), and with two phases no measured row keeps +the narrowest block in three phases. It was removed (a9c4bf1). + +### Step 2, arm (c) against `main` (Xeon) + +Arm (c) = commit d33ea09 (`grid_two_odd_part` = 15 on the Xeon; later commits only format +and comment it). Run 20261116-125601-0542 (Slurm 16033 build, 16034 timing; 16 rounds, 6 +under load; A/A band ±1.95 %), x29's 23 mixed rows from 46080 to 2 · 10⁶, arms A = +`main` 29f9772, B = d33ea09, F = FFTW, M = MKL: + +- The 14 rows whose plan changes (odd part ≥ 15, beyond L2): geomean B ÷ A **0.861**, every + one at or below 1 (largest 184320 0.996; 100000 0.711 [0.696, 0.723], 10⁶ 0.724, 2 · 10⁶ + 0.742, 819200 0.814, 983040 0.818, 737280 0.874, 1024000 0.866, 245760 0.891, 204800 + 0.892, 221184 0.900, 256000 0.938, 86016 0.968, 71680 0.983). fbench's weighted geomean + over all 23 rows 0.915 [0.901, 0.934]. +- The 9 rows whose plan does not change (46080, 51200, 64000 within L2; odd part 3–9): + 0.989–1.032, all "no evidence" (327680 1.032 [0.971, 1.107], an unchanged plan). +- FFTN ÷ FFTW: 100000 2.03 → **1.44**, 10⁶ 1.01 → 0.73, 2 · 10⁶ 0.99 → 0.73; over the 23 + rows 1.19 → 1.09. FFTN ÷ MKL: 100000 1.34 → **0.95**, 10⁶ 1.27 → 0.92; over the 23 rows + 1.10 → 1.01. 46080 (a two-phase grid within L2, unchanged) stays 1.30 of FFTW and 1.01 of + MKL. + +Tracking tier (only 100000 changes plan there): TRACKING. + +The EPYC and the Mac: `grid_two_odd_part` is 0 there (X29's large-L3 rule and the shared +L2 respectively; `Schedule.GridSizesFromCaches` checks it), so no plan changes and no run +was needed. ## Verdict -(pending) +- **Diagnosis.** On the Xeon, 100000 was a three-phase grid in blocks of one register + (power-of-two part 32), with 3.4 times FFTW's instructions; the third phase's pass over + memory beyond a small, slow L3 costs more than the cache fit it buys. +- **Arm (a), chains for these lengths on the Xeon: rejected by the screen** (1.05–1.45 of the + grid at every grid row; the chain at 100000 has FFTW's instruction count but stalls on + memory). No rule keyed on L2 can make chains pay here. +- **Arm (b), blocks that are other multiples of the register's columns: dropped** + (100000 0.80 in cycles, superseded by arm c's 0.70; no row left that needs it). +- **Arm (c), two phases for rows with an odd part ≥ 15 where a private L2 has less than 2 MiB + of L3 per CPU: adopted on the Xeon** by the decision rule (0.861 over the 14 changed rows, + none slower; TRACKING). The EPYC (two phases 1.13–1.67 of the cycles below 16 MiB) and the + Mac keep `main`'s plans (the field is 0 there). DESIGN 8.5 and the decision log, LESSONS + 3.14. +- **Unverified and open.** Granite Rapids (X31: 100000 2.16, 46080 1.60) in an 8-vCPU VM + reports 504 MiB of L3 for 8 CPUs, so X29's chains apply there, not this rule: whether it + wants the Xeon's plan needs a run with `CHAIN_PART_BYTES=0 GRID_TWO_ODD_PART=15` against + the default; if it does, cache sizes as a VM reports them cannot separate it from Zen, and + the key needs a core field or a measured choice (lead (d), planner 2.4). 46080 (1.30 of + FFTW, a two-phase grid within L2: three phases 1.24, blocks of 16 1.02) and 100000's + remaining 1.44 (FFTW's recursive 25 × 20 × 10 × 20 without copies) are not closed. From 76b4bd5f3f0a8b62832537431babb23df196d833 Mon Sep 17 00:00:00 2001 From: John Travers Date: Mon, 16 Nov 2026 14:18:36 +0000 Subject: [PATCH 9/9] dev: X34 tracking-tier confirmation (Xeon 0.995, 100000 0.706) Co-Authored-By: Claude Opus 5.5 --- dev/experiments/X34-xeon-mixed.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/dev/experiments/X34-xeon-mixed.md b/dev/experiments/X34-xeon-mixed.md index f7a803b..427d2aa 100644 --- a/dev/experiments/X34-xeon-mixed.md +++ b/dev/experiments/X34-xeon-mixed.md @@ -145,7 +145,11 @@ under load; A/A band ±1.95 %), x29's 23 mixed rows from 46080 to 2 · 10⁶, ar 1.10 → 1.01. 46080 (a two-phase grid within L2, unchanged) stays 1.30 of FFTW and 1.01 of MKL. -Tracking tier (only 100000 changes plan there): TRACKING. +Tracking tier (only 100000 changes plan there): run 20261116-131703-f74f (Slurm 16035 +build, 16036 timing; 59 rounds, 11 under load; band ±1.68 %): c2c 100000 0.706 [0.699, +0.722], every other case "no evidence", the largest r2c 1024² 1.030 [0.999, 1.056] (an +unchanged plan); geomean 0.995. FFTN ÷ FFTW over the tier 0.905 → 0.900, ÷ MKL 1.007 → +1.002; 100000 2.04 → 1.44 of FFTW and 1.33 → 0.94 of MKL. The EPYC and the Mac: `grid_two_odd_part` is 0 there (X29's large-L3 rule and the shared L2 respectively; `Schedule.GridSizesFromCaches` checks it), so no plan changes and no run @@ -163,7 +167,7 @@ was needed. (100000 0.80 in cycles, superseded by arm c's 0.70; no row left that needs it). - **Arm (c), two phases for rows with an odd part ≥ 15 where a private L2 has less than 2 MiB of L3 per CPU: adopted on the Xeon** by the decision rule (0.861 over the 14 changed rows, - none slower; TRACKING). The EPYC (two phases 1.13–1.67 of the cycles below 16 MiB) and the + none slower; tracking tier 0.995, only 100000 changed, 0.706). The EPYC (two phases 1.13–1.67 of the cycles below 16 MiB) and the Mac keep `main`'s plans (the field is 0 there). DESIGN 8.5 and the decision log, LESSONS 3.14. - **Unverified and open.** Granite Rapids (X31: 100000 2.16, 46080 1.60) in an 8-vCPU VM