Repository navigation
I2F from a 64-bit integer rounds once, as the GPU does (chain change) - #2045
Open
Know-It-All-Brother wants to merge 2 commits into
Open
Know-It-All-Brother wants to merge 2 commits into
Know-It-All-Brother wants to merge 2 commits into
Conversation
…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…r has 32-bit integers only, so it converted the two words to float apart and added them, which rounded twice and, for a negative value whose high word is -1 or -2, left the low word's rounding error in a result that nearly cancels (0xffffffff_fa543872 gave 0xccb57900 where a Jetson Orin gives 0xccb578f2). AI2f64 now builds the float's bits in integer arithmetic that Metal's fast math leaves alone: the 64-bit magnitude, its leading one, 24 bits of mantissa, a round bit and a sticky bit, to nearest even or the .RM / .RP / .RZ the instruction asks for. test_dmsh_ampere_orin.py's i2f_s64 went from 16 of 1,024 lanes differing to none and leaves KNOWN, the other translator tests are as on main, and a model of the same steps matched an exact reference on 200,000 random pairs in all four roundings, signed and unsigned; chain 6e3c16385059
Know-It-All-Brother
added a commit
to Know-It-All-Brother/nxnative
that referenced
this pull request
Nov 9, 2026
…nch's translator rebuilt from its own source; chain bc21ff4a5d62
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.
What and why
I2F from a 64-bit integer (
cvt.rn.f32.s64, opcode 0x312) computedfloat(hi) * 2^32 + float(lo), which rounds twice. For a negative value whose high word is -1 or -2, the two big terms nearly cancel and the low word's rounding error is left behind, so results were off by 1 ULP and up to 2^30 ULP. On a Jetson Orin, the 64-bit value with words0xfffffffe:0xaaaaaaaaconverts to0xcfaaaaab; we gave0xcfaaaaaa. The instruction's rounding field was ignored too.AI2f64now rounds once, with integer operations on the two 32-bit halves, since the IR has no 64-bit integers here. It takes the magnitude (carrying into the high word), finds the leading one, shifts it to the top keeping the dropped bits, and rounds to nearest even, or by the instruction's .RM / .RP / .RZ. Then it builds the float's bits and puts the sign back. Integer operations are exact under Metal's fast math. Destination F32 is still the only one supported, as before. It costs about 33 integer operations per 64-bit I2F, a rare instruction (the compiler's 32-bit division sequence is one place it comes up).Reference entry: "I2F from a 64-bit integer rounds once" in
docs/reference/shaders.md.How to verify
i2f_s64: 16 of 1,024 lanes differed on main (up to 1,073,741,824 ULP), and 0 differ here. The test takes it out ofKNOWN.The fixture only has the signed, round-to-nearest form, so the unsigned source and the directed roundings haven't run against the Orin. I checked a model of exactly these steps against an exact reference on 200,000 random values (signed and unsigned, all four roundings, including high words 0, -1, -2 and 0x80000000), and they all matched.
This overlaps #1800 only by position: #1800 changes the 32-bit I2F roundings in
AI2f, and this replacesAI2f64a few dozen lines further down.Checked
test_dmsh_ampere_orin.py; every othertest_dmsh_*gives the same result as on main (test_dmsh_ampere_mem.py,test_dmsh_meshstage.pyandtest_dmsh_pairs.pyfail on main the same way);test_nsp_chain.py,test_xlate_anycpu.py,test_docs.py.tool_lock.py --update-lockanddmsh_chain.py --headerwere run (chain 6e3c16385059).regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. I'll add a report once I have a fresh build.Docs
docs/reference/shaders.mdhas the entry, andpython3 dm/tools/docs_symptoms.pywas rundmsh_chain.py --headerwas run, anddmsh_chain.h, the translator dlls and the lock are part of the change