Skip to content

I2F from a 64-bit integer rounds once, as the GPU does (chain change) - #2045

Open
Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-i2f-s64
Open

Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-i2f-s64

Conversation

@Know-It-All-Brother

@Know-It-All-Brother Know-It-All-Brother commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.

What and why

I2F from a 64-bit integer (cvt.rn.f32.s64, opcode 0x312) computed float(hi) * 2^32 + float(lo), which rounds twice. For a negative value whose high word is -1 or -2, the two big terms nearly cancel and the low word's rounding error is left behind, so results were off by 1 ULP and up to 2^30 ULP. On a Jetson Orin, the 64-bit value with words 0xfffffffe:0xaaaaaaaa converts to 0xcfaaaaab; we gave 0xcfaaaaaa. The instruction's rounding field was ignored too.

AI2f64 now rounds once, with integer operations on the two 32-bit halves, since the IR has no 64-bit integers here. It takes the magnitude (carrying into the high word), finds the leading one, shifts it to the top keeping the dropped bits, and rounds to nearest even, or by the instruction's .RM / .RP / .RZ. Then it builds the float's bits and puts the sign back. Integer operations are exact under Metal's fast math. Destination F32 is still the only one supported, as before. It costs about 33 integer operations per 64-bit I2F, a rare instruction (the compiler's 32-bit division sequence is one place it comes up).

Reference entry: "I2F from a 64-bit integer rounds once" in docs/reference/shaders.md.

How to verify

DMSH_TEST_XLATE=dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate AMPERE_ORIN_ONLY=i2f_s64 python3 dm/gfx/tests/test_dmsh_ampere_orin.py

i2f_s64: 16 of 1,024 lanes differed on main (up to 1,073,741,824 ULP), and 0 differ here. The test takes it out of KNOWN.

The fixture only has the signed, round-to-nearest form, so the unsigned source and the directed roundings haven't run against the Orin. I checked a model of exactly these steps against an exact reference on 200,000 random values (signed and unsigned, all four roundings, including high words 0, -1, -2 and 0x80000000), and they all matched.

This overlaps #1800 only by position: #1800 changes the 32-bit I2F roundings in AI2f, and this replaces AI2f64 a few dozen lines further down.

Checked

  • Tests: test_dmsh_ampere_orin.py; every other test_dmsh_* gives the same result as on main (test_dmsh_ampere_mem.py, test_dmsh_meshstage.py and test_dmsh_pairs.py fail on main the same way); test_nsp_chain.py, test_xlate_anycpu.py, test_docs.py.
  • Translator: the dlls are rebuilt from this tree with .NET 10; tool_lock.py --update-lock and dmsh_chain.py --header were run (chain 6e3c16385059).
  • Devices and OS: MacBook Pro M5 Max, macOS 27.
  • regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. I'll add a report once I have a fresh build.
  • Games run: none.

Docs

  • docs/reference/shaders.md has the entry, and python3 dm/tools/docs_symptoms.py was run
  • A chain tool changed: dmsh_chain.py --header was run, and dmsh_chain.h, the translator dlls and the lock are part of the change
  • No game files, keys or data extracted from a game

…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…r has 32-bit integers only, so it converted the two words to float apart and added them, which rounded twice and, for a negative value whose high word is -1 or -2, left the low word's rounding error in a result that nearly cancels (0xffffffff_fa543872 gave 0xccb57900 where a Jetson Orin gives 0xccb578f2). AI2f64 now builds the float's bits in integer arithmetic that Metal's fast math leaves alone: the 64-bit magnitude, its leading one, 24 bits of mantissa, a round bit and a sticky bit, to nearest even or the .RM / .RP / .RZ the instruction asks for. test_dmsh_ampere_orin.py's i2f_s64 went from 16 of 1,024 lanes differing to none and leaves KNOWN, the other translator tests are as on main, and a model of the same steps matched an exact reference on 200,000 random pairs in all four roundings, signed and unsigned; chain 6e3c16385059
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders labels Nov 9, 2026
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…nch's translator rebuilt from its own source; chain bc21ff4a5d62
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant