Skip to content

FSETP.NE and FSET.NE are false with a NaN operand, as NVIDIA's are (chain change) - #2042

Open
Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-fsetp-ne
Open

Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-fsetp-ne

Conversation

@Know-It-All-Brother

@Know-It-All-Brother Know-It-All-Brother commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.

What and why

.NE is the ordered not-equal: it's false when either operand is NaN (.NEU is true then). The translator emits SPIR-V's OpFOrdNotEqual, which is right, but spirv-cross writes it as MSL's !=, which is unordered. So FSETP.NE held for a NaN operand, and a NaN guard let the NaN through. ptxas also builds setp.equ from the negated FSETP.NE, so that was wrong too.

The obvious rewrite, (a < b) || (a > b), doesn't work: Metal's fast math assumes no NaN and folds it straight back to !=. I checked that on the GPU. GetFPComparison now writes a != b && !isnan(a) && !isnan(b), and Metal keeps isnan under fast math, as the unordered comparisons already rely on.

The helper is shared, so Maxwell's FSETP and FSET get the fix too.

Reference entry: "FSETP.NE and FSET.NE held with a NaN operand" in docs/reference/shaders.md.

How to verify

DMSH_TEST_XLATE=dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate AMPERE_ORIN_ONLY=fsetp,fsetp_ftz,fset_bf,fset_u32,fsel_cmp python3 dm/gfx/tests/test_dmsh_ampere_orin.py
Op main this PR
fsetp (14 compares) 66 of 14,336 lanes differ, 47 of them with a NaN operand 16, all f32 denormal inputs (Metal flushes them)
fsetp_ftz 45 of 4,096 0

On the full 16,384-case run, 800 cases of each differed before, every one with a NaN operand. The test takes fsetp and fsetp_ftz out of KNOWN, so it fails on main's translator and passes here.

Checked

  • Tests: test_dmsh_ampere_orin.py, and every other test_dmsh_* gives the same result as on main (test_dmsh_ampere_mem.py, test_dmsh_meshstage.py and test_dmsh_pairs.py fail on main the same way). test_nsp_chain.py, test_xlate_anycpu.py, test_docs.py.
  • Translator: the dlls are rebuilt from this tree with .NET 10, tool_lock.py --update-lock and dmsh_chain.py --header were run (chain 230bb40d4e58).
  • Devices and OS: MacBook Pro M5 Max, macOS 27.
  • regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. I'll add a report once I have a fresh build. A scan of Ocarina of Time's shader code finds only a handful of FSETP.NE.
  • Games run: none. A scan of Ocarina of Time's shader code finds only a handful of FSETP.NE, so I don't expect a visible change there.

Docs

  • docs/reference/shaders.md has the entry, and python3 dm/tools/docs_symptoms.py was run
  • A chain tool changed: dmsh_chain.py --header was run, and dmsh_chain.h, the translator dlls and the lock are part of the change
  • No game files, keys or data extracted from a game

…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…e: the ordered not-equal went to SPIR-V as OpFOrdNotEqual, which spirv-cross writes as MSL's unordered !=, so a NaN guard let the NaN through. (a < b) || (a > b) does not help, since Metal's fast math folds it back to !=; GetFPComparison now writes a != b && !isnan(a) && !isnan(b), and Metal keeps isnan. The helper is shared, so Maxwell's FSETP and FSET change too. test_dmsh_ampere_orin.py's fsetp and fsetp_ftz match a Jetson Orin (800 of 16,384 cases each differed before); chain 230bb40d4e58
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders labels Nov 9, 2026
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…, with this branch's translator rebuilt from its own source; chain c5573a16e3a3
AmateurWizrd added a commit to AmateurWizrd/nxnative that referenced this pull request Nov 10, 2026
…he chain batch (chain 483568384b04)

Rebuilt from the in-repo source after seedds#2042, seedds#2047, seedds#2048 (translator), seedds#1846, seedds#1797 and 064a8cc (dmsh_build.py).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant