Skip to content

perf: use a perfect-shuffle bitonic merge for the SSE union and xor - #909

Merged
lemire merged 1 commit into
masterfrom
sse-bitonic-merge-only
Oct 16, 2026
Merged

lemire merged 1 commit into
masterfrom
sse-bitonic-merge-only

Conversation

@lemire

@lemire lemire commented Oct 16, 2026

Copy link
Copy Markdown
Member

This is the first commit of #902 by @Dr-Emann, on its own, without the branchless block selection.

sse_merge used to rotate one register through the other in 8 dependent alignr/min/max rounds. This replaces it with the perfect-shuffle form of the bitonic merge: reverse the newly loaded block, then 4 rounds of lane-wise min/max followed by a 16-bit interleave. That is fewer uops (17 vs ~24), and the chain through vecMax is shorter because the reverse is applied to the newly loaded block rather than the carried one.

Benchmarks

Xeon Gold 6548N (Emerald Rapids), pinned to one core. The benchmark calls union_vector16 and xor_vector16 directly on 64 different input pairs, with four data shapes (interleaved, dense overlapping, clustered, disjoint) at 256, 1024 and 4096 elements per side. The numbers are speedups over the base of #902.

union xor
gcc 14.3 1.13–1.15x 1.14–1.19x
clang 21.1 1.10–1.14x 1.12–1.23x

Every case is faster. The second commit of #902 (branchless block selection) never improved on this one. clang turns take1 ? pa : pb back into a branch, so that version was up to 14% slower than the original code on large interleaved inputs.

On AVX-512 machines, fast_union_uint16 uses avx512_union_uint16, so the union speedup only reaches AVX2-only CPUs. xor_vector16 is used on any CPU with AVX2.

Testing

  • I checked the merge network against a reference sort on 20M random sorted pairs and on every 0-1 input pair (0-1 principle).
  • union_vector16 and xor_vector16 matched the scalar union_uint16 and xor_uint16 on 200k random cases.
  • The full test suite passes.

sse_merge rotated one register through the other with 8 dependent
alignr/min/max rounds. Replace it with the perfect-shuffle form of the
bitonic merge: reverse the incoming block, then 4 rounds of lane-wise
min/max and a 16-bit interleave. Fewer uops (17 vs ~24), no blends, and the
chain through vecMax drops from ~15 to ~8 cycles since the reverse is
applied to the freshly loaded block rather than the carry.
@lemire
lemire merged commit 33a28ce into master Oct 16, 2026
43 checks passed
@Dr-Emann
Dr-Emann deleted the sse-bitonic-merge-only branch October 16, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants