Use block reductions in TensorPrimitives.MinMaxCore and AggregateAnyAll - #134147
Draft
Mrnikbobjeff wants to merge 5 commits into
Draft
Mrnikbobjeff wants to merge 5 commits into
Mrnikbobjeff wants to merge 5 commits into
Conversation
MinMaxCore backs Min, Max, MinMagnitude, MaxMagnitude, MinNumber, MaxNumber, MinMagnitudeNumber, MaxMagnitudeNumber (and the Half path through HalfAsInt16AggregationOperator). Its vector loops used one accumulator, one vector per iteration and, for float/double, a NaN test with a branch on every vector. On AVX-512 hardware Vector.Min/Max<float> also lowers to vrangeps plus two dependent vfixupimmps, so the float loop was bound by a 3-instruction latency chain plus 5-6 instructions of NaN checking per vector. The vector paths (512/256/128, kept symmetric) now reduce the whole vectors in blocks of 32 vectors with two independent accumulators, then handle any remaining elements with one final vector that overlaps the last whole one, as before. NaN is decided once per block: the IsNaN masks of the block's vectors are OR-accumulated (branch-free) and, if any is set, a cold NoInlining helper rescans the block for the first NaN, so the first NaN of the input is still the one returned (blocks are visited in order and the running result is only combined after the check). The mask is gathered from the elements rather than from the result because the Number operators discard NaN; the previous behavior of returning the first NaN for those operators too is preserved. Min and Max over float and double take a fast path that reduces each block with Vector.MinNative/MaxNative (vminps/vmaxps, one instruction, no NaN or signed-zero guarantees) and restores the IEEE 754:2019 minimum/maximum rules once per block: the NaN mask above, and an OR (Min) or AND (Max) of the raw element bits. The native minimum never returns a value below every input, so when a block's minimum compares equal to zero and the block holds no NaN, no element is negative and any set sign bit is a -0; the maximum case is mirrored. The Magnitude and Number operators, Half and the integer types use the generic block reduction with the operator's own vector Invoke (for integers that is already two vpminsd with memory operands per iteration). Block results are combined with the operator's scalar Invoke, first block first, so the operators' tie rules are unchanged. The per-width methods are NoInlining so that each is its own JIT root with enough inlining budget for the block reduction and the horizontal aggregates; the old shape was inlined into the public entry point, whose small budget left HorizontalAggregate and the typeof helpers as real calls. Hot loop for float on AVX-512 (32 floats per iteration): 2 loads, 2 vminps, 1 vpternlogd, 2 vcmpneqps, korw, vpmovm2d, vorps, plus the loop overhead. Measured on a Ryzen 7 7800X3D (Zen 4, AVX-512), 65536 elements, package 10.0.12 vs this change, GElem/s: Min<float> 11.1-11.5 -> 27.0 Max<float> 11.4-11.5 -> 26.2-27.2 Min<double> 5.6-5.7 -> 13.5-14.1 Min<int> 31-35 -> 34-35 (load ceiling either way) Verified bit-exact against the unmodified library over a differential matrix (8 element types x 8 reductions x 29 lengths up to 65539 x random/all-equal/ all-zero/signed-zero/NaN-with-payload/two-NaN/infinity/magnitude-tie/MinValue cases, on the 512, 256 and 128-bit paths). The only difference is which of two distinct NaN payloads Half returns: the Half path never had a "first NaN" rule (it depends on the lane pairing of the vector operator, and the previous code returned the second NaN's payload in two thirds of those cases), and the block pairing changes it in about 5% of them. Tests cover lengths past 256, the first-NaN rule, signed zeros and all-zero inputs at block boundaries for all six span reductions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AggregateAnyAll backs the 36 IsXxAny / IsXxAll methods (IsNaNAny, IsFiniteAll,
IsNegativeAny, IsZeroAll, ...). Its vector loops tested and branched on every
vector: load, the operator's compare, a horizontal test (vptest, or for All a
compare against zero and vptest) and a conditional branch, plus the loop
overhead, so the loop was front-end bound at 8 instructions and two branches
per vector, and its 512-bit path was no faster than the 256-bit one.
The vector paths (512/256/128, kept symmetric) now fold the whole vectors in
blocks of 32 vectors into two independent accumulators with no branch on the
data and decide once per block, then fold the remaining whole vectors (fewer
than a block) the same way, and finally handle any remaining elements with one
vector that overlaps the last whole one, as before. The index is a nuint so that
the loads need no zero extension.
IAnyAllAggregator<T> gains Accumulate(accumulator, result) per width, the fold
of an operator result into a block accumulator, in which a lane with all bits
set means that the block settles the result. AnyAggregator ORs the result in;
AllAggregator ORs its complement in (a lane where the operator was false), so
both use a zero identity, an OR to merge the two accumulators and the same
block test (AnyWhereAllBitsSet). Accumulating the complement rather than AND-ing
the results also lets the JIT fold the negation into an operator that ends in
one, such as IsFinite. The per-vector ShouldEarlyExit overloads remain in use
for the final overlapping vector; the scalar path is unchanged. The results are
identical: a hit is still found, only its detection is deferred to the end of
its block, and every read stays within the span. The per-width methods are
NoInlining roots, as in MinMaxCore, so that the operator and the aggregator
stay inlined.
Hot loop for float on AVX2 (16 floats per iteration): 2 loads, 2 vcmpneqps,
2 vorps, add, cmp, jb; one vorps, vpcmpeqd x2, vptest, jne per 32 vectors. On
AVX-512 the JIT goes through a mask register per vector (vcmpneqps k1 +
vpmovm2d + vorps).
Measured on a Ryzen 7 7800X3D (Zen 4, AVX-512), 65536 elements, GElem/s:
BenchmarkDotNet, HPC AnyAllBenchmarks, package 10.0.12 vs this change, no hit
(the whole span is scanned):
IsNaNAny<float> 23.4 -> 32.5 (AVX-512 path), 23.9 -> 31.5 (AVX2 path)
IsFiniteAll<float> 22.3 -> 28.9, 12.6 -> 28.9
IsNegativeAny<float> 23.0 -> 30.5, 24.5 -> 35.6
IsNaNAny<double> 14.8 -> 17.8, 12.3 -> 18.0
Hit at N/2: IsNaNAny<float> 45 -> 65 (the block shape reads at most one extra
block past the hit); hit at N-1: the same as no hit.
Stopwatch A/B of 21 vectorized instantiations against the unmodified library,
no hit: 1.02x to 1.32x on the AVX-512 path, 1.04x to 2.05x on the AVX2 path
(IsFiniteAll<float> 14.3 -> 29.1, IsZeroAll<int> 21.9 -> 33.7, IsOddIntegerAny
<ushort> 29.3 -> 53.4). The one exception is IsSubnormalAny<float> on the
AVX-512 path, 20.8 -> 19.1: the old shape tested the compare's mask register
directly (kortestw), the block shape has to materialize it (vpmovm2d) to OR it
in, and with that operator's four instructions per vector the double-pumped
512-bit units are the bound on Zen 4; on the AVX2 path it is 17.5 -> 23.3.
Verified identical to the unmodified library over a differential matrix (all
36 methods x float/double/Half/int/long/byte/ushort x 33 lengths up to 65539 x
uniform fills, one deciding element at every vector, block and ragged boundary,
two deciding elements, random mixes: 253,555 cases, on the 512, 256 and 128-bit
paths). Tests place the deciding element at every block, vector and ragged
boundary of long inputs for IsNaNAny/All, IsFiniteAll/Any, IsNegativeAny,
IsPositiveAny and IsZeroAll/Any.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Min/Max/MinMagnitude/MaxMagnitude with two NaNs of different payloads at every pair of block-boundary positions over the long lengths: float and double must return the first NaN bit for bit; Half (which reaches the core as short and has no element-level NaN check) must return one of the two input NaNs rather than a canonical NaN or a value. The payloads sit above bit 13 so they survive the narrowing to Half, and the test asserts the two inputs really differ. Passes on the unmodified library as well; nothing in the suite covered this before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dotnet-policy-service
Bot
added
the
community-contribution
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/area-system-numerics |
….AggregateAnyAll loops
On AVX-512 hardware the JIT keeps a 512-bit comparison result in a mask
register, and OR-ing it into a vector accumulator first materializes it as a
vector (vpmovm2d), so the block loops of AggregateAnyAll cost three vector
instructions per vector. A select whose condition is the comparison lowers to a
masked blend that consumes the mask register as it is, and a select whose other
operand is the zero vector costs no extra instruction (the JIT even folds it into
a zero-masking move under the inverted comparison), whereas an all-ones operand
is rematerialized inside the loop (vpternlogd).
The 512-bit loops therefore accumulate by clearing lanes: the accumulators start
from all bits set, IAnyAllAggregator<T>.ClearSettled (which replaces the 512-bit
Accumulate) clears the lanes whose result settles the aggregation, the two
accumulators are combined with an AND and a block is decided by any lane being
zero. Any clears where the result is true (ConditionalSelect(result, Zero, acc))
and All where it is false (ConditionalSelect(result, acc, Zero)).
The 128- and 256-bit loops keep vector accumulators, but each is now a result
itself: the OR of Any results and the AND of All results, starting from the
result that equals DefaultResult, so that ShouldEarlyExit decides the block. This
removes the complement from All's accumulation ("acc | ~result"), which on
AVX-512VL hardware running 256-bit vectors folded into the predicate of an
integer comparison that the JIT can then not rewrite from its EVEX mask form
back to a VEX vector comparison, costing a vpmovm2d per vector.
The scalar loop moves to AggregateAnyAllScalar; results are unchanged.
Ryzen 7 7800X3D, best of three runs, N = 4096 (L1-resident), GElem/s, before ->
after, 512-bit path: IsNaNAny<float> 38.6 -> 42.7, IsNaNAll<float> 39.1 -> 45.6,
IsFiniteAll<float> 33.0 -> 39.9, IsFiniteAny<float> 32.0 -> 39.8,
IsInfinityAny<float> 30.6 -> 36.4, IsIntegerAll<float> 16.4 -> 20.8,
IsOddIntegerAny<ushort> 60.8 -> 72.5, IsNegativeAny<int> 34.5 -> 41.4,
IsNaNAny<double> 16.2 -> 17.7. The 256-bit path (with and without AVX-512VL) and
the 128-bit path are unchanged within noise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n TensorPrimitives.AggregateAnyAll Most of the IEEE 754 classifications are a single unsigned comparison of the element's bits against a constant: a NaN has an absolute bit pattern above infinity's, a finite value one below it, a negative value has bits above the largest pattern with the sign bit clear, and a normal or subnormal value falls in a range that a subtraction turns into a comparison with wrap-around. For such an operator an Any/All aggregation does not need the comparison per vector: it folds the keys of a block with one unsigned minimum or maximum per vector and compares the fold against the threshold once per block. Any looks for a key on the operator's side of the threshold and All for one on the other side, so the fold is the maximum when the operator is true above the threshold and the aggregation is Any, or true below it and the aggregation is All, and the minimum otherwise. IBooleanUnaryOperator<T> gains the threshold form as optional members with defaults: HasThresholdForm, TrueBelowThreshold, ThresholdBits (the threshold as the bits of an unsigned integer of the element size) and Key per width. The operators that take it are IsNaN, IsFinite and IsRealNumber (the absolute bits against infinity), IsNormal and IsSubnormal (the absolute bits minus the smallest normal value, or minus one, against the width of the range), IsZero (the absolute bits, or the bits for the integers, below one) and IsNegative and IsPositive (the bits against the sign bit), the last three for the primitive integers as well as for float and double. AggregateAnyAll dispatches such an operator to AggregateAnyAllThreshold, generic over the key type (the unsigned integer of the element size) and over the fold, MaxOperator<TKey> or MinOperator<TKey>: choosing the fold by type rather than by a branch on its direction leaves a single use of each key in the loop, which the JIT then folds into the fold instruction's memory operand. The 64-bit unsigned minimum and maximum are single instructions only with AVX-512, so double and the 64-bit integers use the threshold form only when Avx512F.VL is supported and keep the operator's comparison otherwise. The per-width loops have the block shape of the generic paths; the final overlapping vector is compared against the threshold as well. Hot loops on AVX-512 for IsNaNAny<float>: vandps with the load as its memory operand and vpmaxud per vector, vpcmpgtud and kortestw per block; for IsNegativeAny<int> a single vpmaxud with a memory operand per vector, and on SSE/AVX2 the same with 128- and 256-bit registers. Ryzen 7 7800X3D, best of three runs, N = 4096 (L1-resident), GElem/s, before the two commits -> after, 512-bit path: IsNaNAny<float> 39.8 -> 43.0, IsNaNAll<float> 40.1 -> 57.5, IsFiniteAll<float> 33.0 -> 43.2, IsFiniteAny<float> 32.4 -> 42.6, IsNormalAll<float> 28.3 -> 41.3, IsSubnormalAny<float> 19.5 -> 28.5 (the regression noted in the block-accumulation change is gone), IsNegativeAny<float> 38.9 -> 59.3, IsNegativeAny<int> 36.8 -> 65.9, IsNegativeAny<long> 17.9 -> 31.2, IsFiniteAll<double> 15.6 -> 21.4, IsNormalAll<double> 14.0 -> 20.3, IsZeroAll<int> 40.2 -> 44.1, IsZeroAny<byte> 140.9 -> 172.4. N = 65536 (L2-resident): IsNaNAny<float> 34.3 -> 36.4, IsFiniteAll<float> 31.9 -> 38.5, IsNormalAll<float> 27.7 -> 33.9, IsSubnormalAny<float> 19.0 -> 27.8. AVX2 path (AVX-512 disabled), N = 4096: IsFiniteAny<float> 24.2 -> 41.7, IsPositiveAll <float> 23.4 -> 42.8, IsNormalAll<float> 23.9 -> 40.1, IsZeroAll<int> 29.9 -> 43.1, IsNegativeAny<int> 41.0 -> 58.9; IsNaNAny<float> and IsZeroAny<float>, already two instructions per vector there, vary within +-10% between runs and sizes. 128-bit path: IsFiniteAny<float> 13.1 -> 26.8, IsPositiveAll<float> 13.2 -> 27.1, IsNegativeAny<int> 22.8 -> 32.7. Tests: long-length boundary tests for the operators that gained the threshold form, placing the values whose bits lie next to the thresholds (-0, the smallest and largest subnormal, the smallest normal, the largest finite value, infinity and the NaN whose bits follow it, each with both signs) at every block, vector and ragged boundary; and a theory over all 17 Any/All pairs that checks long spans filled with each special value, and spans in which one special value is placed at every boundary of a span filled with another, against the scalar predicate. The Is* tests pass at 512, 256 and 128 bits, with AVX-512, AVX2 and AVX disabled, and with hardware intrinsics disabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
|
Pushed the two follow-ups from the description as two commits on top of the PR:
BenchmarksRyzen 7 7800X3D (Zen 4, AVX-512, 1.
2. All affected methods, net11.0 build on this branch's own runtime (testhost: .NET 12.0.0-dev, X64 RyuJIT x86-64-v4), N = 4096 (L1-resident) and 65536 (L2-resident): 46 rows
Reading the tables:
Correctness: the results are unchanged. Long-length tests place the values whose bits sit next to each threshold (−0, the smallest and largest subnormal, the smallest normal, the largest finite value, infinity and the first NaN after it, with both signs) at every block, vector and ragged boundary, and a theory over all 17 Any/All pairs checks long spans filled with each special value, and spans with one special value placed at every boundary of another, against the scalar predicate. The |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #133969 (block reduction for
IndexOfMin/IndexOfMax/IndexOfMinMagnitude/IndexOfMaxMagnitude), applying the same loop shape to the two remaining reduction cores whose vector loops did one vector per iteration with a test and a branch on every vector. Both changes keep the observable results identical (verified bit for bit against the unmodified library, see below) and only change how many vectors are folded before a decision is made.MinMaxCore(Min, Max, MinMagnitude, MaxMagnitude, MinNumber, MaxNumber, MinMagnitudeNumber, MaxMagnitudeNumber, Half, the float overloads)The current loop keeps one accumulator and, for float and double, checks every vector for NaN before folding it. On AVX-512 hardware
Vector.Min<float>lowers tovrangepsplus two dependentvfixupimmps, so one accumulator serializes three ops per vector, and the NaN test adds a compare, a k-mask test and a branch on top;TensorPrimitives.Min<float>measures 11.4 GElem/s on a Ryzen 7 7800X3D for 65536 elements (its 512-bit path), a third of the load bandwidth.The core now reduces blocks of 32 vectors with two independent accumulators and decides once per block:
Invoke, NaN masks OR-accumulated from the elements (not from the operator's result, because theNumberoperators drop NaN and the current core returns the first NaN for them too), one test per block, then a coldNoInliningscan for the first NaN of that block, payload preserved. The overlapping final vector and the scalar path are unchanged.MinOperator/MaxOperatoron float and double:MinNative/MaxNativein the loop (a singlevminps/vmaxps), the NaN mask as above, and the raw bits OR-ed (Min) or AND-ed (Max) for the signed-zero rule.vminpsnever returns a value below every input, so when a block's minimum compares equal to zero and the block has no NaN, no element is negative and any set sign bit is a-0; the mirror argument gives+0for Max. This keeps IEEEminimum/maximumsemantics (-0 < +0, NaN propagates) with two instructions per vector on the loop-carried chain instead of three dependent ones.Integer types keep the operator's
Invoke(twovpminsdwith memory operands per iteration at the load ceiling).AggregateAnyAll(the 36IsXxAny/IsXxAllmethods)The current loop tests every vector's mask and branches. The loop is front-end bound (8 instructions and two branches per 32-byte load; its 512-bit path is no faster than 256-bit on Zen 4). The core now OR-accumulates the masks over blocks of 32 vectors with two accumulators (
Allaccumulates the complement, which lets the JIT fold the~and avoids a k-mask round trip that~IsZeroinside a vector AND would otherwise cost on AVX-512), tests once per block, and keeps the overlapping final vector and scalar path.IAnyAllAggregator<T>gained oneAccumulatemember per width. An early exit happens at most one block later than before, still within the span.Measurements
7800X3D (Zen 4, AVX-512), BenchmarkDotNet, N = 65536, GElem/s, package 10.0.12 versus this branch:
Min<float>Max<float>Min<double>Min<int>IsNaNAny<float>IsFiniteAll<float>IsNegativeAny<float>IsNaNAny<double>With the hit at N/2,
IsNaNAny<float>goes from 45.3 to 64.7 (elements per second over the full N). One small regression is known:IsSubnormalAny<float>on the 512-bit path, 20.8 to 19.1, because that operator's compare previously branched on its k-register directly and accumulation has to materialize it withvpmovm2d.Verification
mainbuild of the library, bit-exact, on the 512-, 256- and 128-bit paths: over 270,000 Min/Max cases (8 types, 8 reductions, 29 lengths from 1 to 65539, NaN payloads at block boundaries, two NaNs,-0/+0, magnitude ties,MinValue) and 253,555 Any/All cases (36 methods, 7 types, 33 lengths, the deciding element at every vector, block and ragged boundary). The only differences are which of two distinct NaN payloadsHalfMin/Max/MinMagnitude/MaxMagnitude returns (theHalfpath reaches the core asshort, so it has no element-level NaN check, and the old code returned the second NaN in two thirds of those pairs; the result is always one of the input NaNs in both).System.Numerics.Tensors.Testssuite: 6006 passed.Helpers.TensorLengthswith the best value, the first NaN,-0/+0and all-zero spans at every block boundary for Min/Max/MinMagnitude/MaxMagnitude/MinNumber/MaxNumber; long-length boundary tests for IsNaNAny/All, IsFiniteAll/Any, IsNegativeAny, IsPositiveAny, IsZeroAll/Any; and a two-distinct-NaN-payload test that pins "first NaN" for float and double and "one of the input NaNs" for Half.Possible follow-ups, not in this PR: operator-specific block folds for Any/All (an unsigned max of the absolute bit pattern makes IsNaNAny/IsFiniteAll reach the load ceiling), and k-mask accumulation on the 512-bit Any/All path.