Repository navigation
perf: use a perfect-shuffle bitonic merge for the SSE union and xor - #909
Merged
Merged
Conversation
sse_merge rotated one register through the other with 8 dependent alignr/min/max rounds. Replace it with the perfect-shuffle form of the bitonic merge: reverse the incoming block, then 4 rounds of lane-wise min/max and a 16-bit interleave. Fewer uops (17 vs ~24), no blends, and the chain through vecMax drops from ~15 to ~8 cycles since the reverse is applied to the freshly loaded block rather than the carry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first commit of #902 by @Dr-Emann, on its own, without the branchless block selection.
sse_mergeused to rotate one register through the other in 8 dependent alignr/min/max rounds. This replaces it with the perfect-shuffle form of the bitonic merge: reverse the newly loaded block, then 4 rounds of lane-wise min/max followed by a 16-bit interleave. That is fewer uops (17 vs ~24), and the chain throughvecMaxis shorter because the reverse is applied to the newly loaded block rather than the carried one.Benchmarks
Xeon Gold 6548N (Emerald Rapids), pinned to one core. The benchmark calls
union_vector16andxor_vector16directly on 64 different input pairs, with four data shapes (interleaved, dense overlapping, clustered, disjoint) at 256, 1024 and 4096 elements per side. The numbers are speedups over the base of #902.Every case is faster. The second commit of #902 (branchless block selection) never improved on this one. clang turns
take1 ? pa : pbback into a branch, so that version was up to 14% slower than the original code on large interleaved inputs.On AVX-512 machines,
fast_union_uint16usesavx512_union_uint16, so the union speedup only reaches AVX2-only CPUs.xor_vector16is used on any CPU with AVX2.Testing
union_vector16andxor_vector16matched the scalarunion_uint16andxor_uint16on 200k random cases.