Repository navigation
F2I to a 64-bit integer: signed, every rounding, NaN and infinities as the GPU gives them (chain change) - #2059
Open
Know-It-All-Brother wants to merge 3 commits into
Open
Know-It-All-Brother wants to merge 3 commits into
Know-It-All-Brother wants to merge 3 commits into
Conversation
…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…ules with its functions, so a rewrite that uses one (struct in PR seedds#1902's hfma_msl) runs as it does in a build instead of raising NameError
…igned, with every rounding: the translator had only the unsigned truncation the compiler's 32-bit division uses, done in float arithmetic, so F2I.S64 (cvt.rzi.s64.f32) failed to translate, and float arithmetic can't give the GPU's results for NaN or infinity under Metal's fast math. Both result words are now built from the float's bits with integer operations, the dropped fraction rounded as .ROUND, .FLOOR, .CEIL or .TRUNC asks, with the source's abs and negate, .FTZ and .NTZ; out-of-range values saturate and a NaN gives 0x8000000000000000 signed (0 unsigned or with .NTZ), as a Jetson Orin computes it. The Orin test's f2i_s64 went from failing to translate to 0 of 2,048 output words differing when HFMA2.MMA's immediate form (PR seedds#2049), which its program also uses, is present; without it the op stays in KNOWN for that reason. The same program with its F2I's bits changed (56 encodings over 5,120 inputs) matches a bit-exact model of those rules on this Mac's GPU, and the model matches every Orin row; chain 4d9030e22125
Know-It-All-Brother
added a commit
to Know-It-All-Brother/nxnative
that referenced
this pull request
Nov 9, 2026
…MMA here, f2i_s64 matches the Orin (PR seedds#2059), with this branch's translator rebuilt from its own source; chain 6a270099b402
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.
What and why
AF2i64handled only the unsigned truncation of an F32 to 64 bits, which the 32-bit division sequence uses, and did it with float arithmetic. SoF2I.S64.TRUNC(opcode 0x311, fromcvt.rzi.s64.f32) failed to translate. A float approach also can't give the GPU's NaN and infinity results, because Metal's fast math assumes they don't occur.The new path uses integer operations on the float's bits only:
.FTZflushing denormals and abs/negate applied to the sign..TRUNC,.FLOOR,.CEILor.ROUNDfrom the dropped bits..NTZ.The Orin confirms NaN, +inf and −inf give exactly those signed values. The unsigned truncation now takes the same path with the same results in range. It costs about 45 integer operations per F2I to 64 bits, a rare instruction.
Reference entry: "F2I to a 64-bit integer, signed or rounded" in
docs/reference/shaders.md.How to verify
The fixture's
f2i_s64program also loads a constant with HFMA2.MMA, which #2049 implements. So on this PR alonef2i_s64stays inKNOWN, and the reason now names #2049. With #2049's one-line registration added, all 2,048 output words match the Orin. Whichever PR lands second should takef2i_s64out ofKNOWN.Only the signed
.TRUNCencoding is checked against the GPU. To cover the rest, I patched the program's F2I bits into 56 encodings (signed and unsigned, the four roundings,.FTZ,.NTZ, abs/negate) and ran each on this Mac's GPU over 5,120 inputs: every exponent, ties, and the fixture's rows. All 56 matched an exact model, and that model matches all of the Orin's rows.Checked
test_dmsh_ampere_orin.py; every othertest_dmsh_*gives the same result as on main (test_dmsh_ampere_mem.py,test_dmsh_meshstage.pyandtest_dmsh_pairs.pyfail on main the same way);test_nsp_chain.py,test_xlate_anycpu.py,test_docs.py.lint_python.pyshows no new findings.tool_lock.py --update-lockanddmsh_chain.py --headerwere run (chain 4d9030e22125).regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. Ocarina of Time's build already translates every shader, so it can't contain this instruction.Docs
docs/reference/shaders.mdhas the entry, andpython3 dm/tools/docs_symptoms.pywas rundmsh_chain.py --headerwas run, anddmsh_chain.h, the translator dlls and the lock are part of the change