Skip to content

F2I to a 64-bit integer: signed, every rounding, NaN and infinities as the GPU gives them (chain change) - #2059

Open
Know-It-All-Brother wants to merge 3 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-f2i-s64
Open

Know-It-All-Brother wants to merge 3 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-f2i-s64

Conversation

@Know-It-All-Brother

@Know-It-All-Brother Know-It-All-Brother commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.

What and why

AF2i64 handled only the unsigned truncation of an F32 to 64 bits, which the 32-bit division sequence uses, and did it with float arithmetic. So F2I.S64.TRUNC (opcode 0x311, from cvt.rzi.s64.f32) failed to translate. A float approach also can't give the GPU's NaN and infinity results, because Metal's fast math assumes they don't occur.

The new path uses integer operations on the float's bits only:

  • It decodes the sign, exponent and significand, with .FTZ flushing denormals and abs/negate applied to the sign.
  • It shifts the significand into the two-word result, rounding by .TRUNC, .FLOOR, .CEIL or .ROUND from the dropped bits.
  • It saturates out of range: signed gives 0x8000000000000000 / 0x7fffffffffffffff, and unsigned gives 0 below and all ones above.
  • NaN gives 0x8000000000000000, or 0 with .NTZ.

The Orin confirms NaN, +inf and −inf give exactly those signed values. The unsigned truncation now takes the same path with the same results in range. It costs about 45 integer operations per F2I to 64 bits, a rare instruction.

Reference entry: "F2I to a 64-bit integer, signed or rounded" in docs/reference/shaders.md.

How to verify

DMSH_TEST_XLATE=dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate AMPERE_ORIN_ONLY=f2i_s64 python3 dm/gfx/tests/test_dmsh_ampere_orin.py

The fixture's f2i_s64 program also loads a constant with HFMA2.MMA, which #2049 implements. So on this PR alone f2i_s64 stays in KNOWN, and the reason now names #2049. With #2049's one-line registration added, all 2,048 output words match the Orin. Whichever PR lands second should take f2i_s64 out of KNOWN.

Only the signed .TRUNC encoding is checked against the GPU. To cover the rest, I patched the program's F2I bits into 56 encodings (signed and unsigned, the four roundings, .FTZ, .NTZ, abs/negate) and ran each on this Mac's GPU over 5,120 inputs: every exponent, ties, and the fixture's rows. All 56 matched an exact model, and that model matches all of the Orin's rows.

Checked

  • Tests: test_dmsh_ampere_orin.py; every other test_dmsh_* gives the same result as on main (test_dmsh_ampere_mem.py, test_dmsh_meshstage.py and test_dmsh_pairs.py fail on main the same way); test_nsp_chain.py, test_xlate_anycpu.py, test_docs.py. lint_python.py shows no new findings.
  • Translator: the dlls are rebuilt from this tree with .NET 10; tool_lock.py --update-lock and dmsh_chain.py --header were run (chain 4d9030e22125).
  • Devices and OS: MacBook Pro M5 Max, macOS 27.
  • regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. Ocarina of Time's build already translates every shader, so it can't contain this instruction.
  • Games run: none.

Docs

  • docs/reference/shaders.md has the entry, and python3 dm/tools/docs_symptoms.py was run
  • A chain tool changed: dmsh_chain.py --header was run, and dmsh_chain.h, the translator dlls and the lock are part of the change
  • No game files, keys or data extracted from a game

…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…ules with its functions, so a rewrite that uses one (struct in PR seedds#1902's hfma_msl) runs as it does in a build instead of raising NameError
…igned, with every rounding: the translator had only the unsigned truncation the compiler's 32-bit division uses, done in float arithmetic, so F2I.S64 (cvt.rzi.s64.f32) failed to translate, and float arithmetic can't give the GPU's results for NaN or infinity under Metal's fast math. Both result words are now built from the float's bits with integer operations, the dropped fraction rounded as .ROUND, .FLOOR, .CEIL or .TRUNC asks, with the source's abs and negate, .FTZ and .NTZ; out-of-range values saturate and a NaN gives 0x8000000000000000 signed (0 unsigned or with .NTZ), as a Jetson Orin computes it. The Orin test's f2i_s64 went from failing to translate to 0 of 2,048 output words differing when HFMA2.MMA's immediate form (PR seedds#2049), which its program also uses, is present; without it the op stays in KNOWN for that reason. The same program with its F2I's bits changed (56 encodings over 5,120 inputs) matches a bit-exact model of those rules on this Mac's GPU, and the model matches every Orin row; chain 4d9030e22125
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders labels Nov 9, 2026
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…MMA here, f2i_s64 matches the Orin (PR seedds#2059), with this branch's translator rebuilt from its own source; chain 6a270099b402
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant