Skip to content

HADD2, HMUL2 and HFMA2 .FTZ flush f16 subnormals, as the GPU does (chain change) - #2048

Open
Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-half-ftz
Open

Know-It-All-Brother wants to merge 2 commits into
seedds:mainfrom
Know-It-All-Brother:ampere-half-ftz

Conversation

@Know-It-All-Brother

@Know-It-All-Brother Know-It-All-Brother commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of the Ampere accuracy series (#2040). It depends on #2041 for the test: the commits before this PR's own are #2041's and drop out once it lands, so this PR's own change is the last commit.

What and why

HADD2, HMUL2 and HFMA2 with .FTZ (bit 80) flush f16 subnormals, and we ignored the flag. That was fine for f32, which Metal flushes by itself, but Metal keeps f16 subnormals. On a Jetson Orin, an f16 subnormal times +inf is NaN with .FTZ (the subnormal becomes 0 first), and we gave +inf.

I modelled each candidate rule exactly and checked it against every lane the Orin computed. What matches is: both sources and the result are flushed, and the zero keeps its sign. A source lane with |x| < 2^-14 becomes ±0, and a result lane that is subnormal after rounding to half becomes ±0, so 0x3555 × 0x8608 is 0x8000 on the Orin. Flushing only the sources leaves 57 wrong lanes of hmul2_ftz, flushing only the result leaves 60, and flushing to an unsigned zero leaves 125 or more.

The fix:

  • New helpers AHalfFtz / AHalfFlush flush an unpacked half lane to its signed zero, and fold constants.
  • HADD2, HMUL2 and HFMA2 flush their sources with them, and AHalfResult clears subnormal result lanes when bit 80 is set. A .F32 result is left alone, since Metal flushes f32 itself.
  • HSETP2 and HSET2 use the same helper in place of their own copy, which flushed to +0 (a comparison can't see the sign). HMNMX2 .FTZ was dropped silently before and now flushes its sources too.

Without .FTZ, the MSL is byte-for-byte the same as before. With it, each source lane costs about 4 operations and each packed result about 7. In Ocarina of Time's shader code, about 7,700 half instructions carry .FTZ (1.8 % of HADD2, 40 % of HMUL2, 24 % of HFMA2), out of roughly 34 million instructions in all.

Reference entry: "Half-pair .FTZ flushes f16 subnormals" in docs/reference/shaders.md.

How to verify

DMSH_TEST_XLATE=dm/gfx/tools/dmsh_xlate_bin/dmsh_xlate AMPERE_ORIN_ONLY=f16x2 python3 dm/gfx/tests/test_dmsh_ampere_orin.py
Op (halves of 2,048) main this PR
hadd2_ftz 246: 215 NaN bit patterns and 31 others 215, all NaN bit patterns (Metal's own)
hmul2_ftz 449: 228 NaN bit patterns, 215 others and 6 NaN on one side 234, all NaN bit patterns

Both come out of KNOWN, and every other half op gives the same lanes as before.

The fixture's hfma2* ops use HFMA2.MMA, which #2049 implements. To check the HFMA2 path anyway, I ran a copy of the fixture with the opcode patched to HFMA2 (0x231): hfma2_ftz went from 93 differing lanes to 0, and hfma2 and hfma2_sat match. With #2049 and this PR together, the real hfma2_ftz program matches the Orin too (only NaN bit patterns left), so whichever lands second should take hfma2_ftz out of KNOWN. Not checked against the Orin: HMNMX2 and HSETP2/HSET2 with .FTZ (none in the fixture), and .FTZ with a .F32 result.

Checked

  • Tests: test_dmsh_ampere_orin.py; every other test_dmsh_* gives the same result as on main (test_dmsh_ampere_mem.py, test_dmsh_meshstage.py and test_dmsh_pairs.py fail on main the same way); test_nsp_chain.py, test_xlate_anycpu.py, test_docs.py.
  • Translator: the dlls are rebuilt from this tree with .NET 10; tool_lock.py --update-lock and dmsh_chain.py --header were run (chain e7e8f7dbff4d).
  • Devices and OS: MacBook Pro M5 Max, macOS 27.
  • regress.py: not run. My only local Switch 2 build's shader package comes from an older shader split than main's map, so main's chain can't build it. I'll add a report once I have a fresh build.
  • Games run: none.

Docs

  • docs/reference/shaders.md has the entry, and python3 dm/tools/docs_symptoms.py was run
  • A chain tool changed: dmsh_chain.py --header was run, and dmsh_chain.h, the translator dlls and the lock are part of the change
  • No game files, keys or data extracted from a game

…re_orin.py runs about 190 operations, each the GPU compiler's own machine code for one PTX operation, through the translator, spirv-cross, the chain's MSL rewrites and Metal on this Mac's GPU, and compares every output word with what a Jetson Orin (sm_87) computed from the same 1,024 input rows. The translator tests checked the MSL a few encodings give, not the values; this one finds the FSETP.NE, SHFL lane, F2FP, rounding-mode and signed-zero differences the tracking issue lists, which KNOWN names until each fix takes its entry out. dm/gfx/tools/ampere_orin/capture.py records the fixture on any Linux machine with an Ampere GPU and NVIDIA's driver, with no toolkit or compiler on either side
…U does: a source lane under 2^-14 reads as a zero of its own sign, and a result lane that's subnormal after rounding to half becomes a zero of its sign, so HMUL2.FTZ of the subnormal 0x03ff and +Inf is NaN, not Inf. The .FTZ bit (80) was dropped like the f32 arithmetic's, but Metal keeps f16 subnormals, where it flushes f32 ones itself. The source flush is shared with HSETP2 and HSET2, and HMNMX2's .FTZ, which was dropped too, now flushes its sources; without .FTZ the MSL is unchanged. Against the Jetson Orin's results, hadd2_ftz differed beyond NaN bits in 31 of 2,048 lanes and hmul2_ftz in 221, both in none now, and the other half-pair ops are unchanged; HFMA2.FTZ, checked with the fixture's HFMA2.MMA opcode changed to 0x231, went from 93 lanes to none; chain e7e8f7dbff4d
@github-actions github-actions Bot added nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) docs Documentation shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders labels Nov 9, 2026
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…th this branch's translator rebuilt from its own source; chain 30f346ec0153
Know-It-All-Brother added a commit to Know-It-All-Brother/nxnative that referenced this pull request Nov 9, 2026
…s#2049), with this branch's translator rebuilt from its own source; with seedds#2048's .FTZ fix here, hfma2_ftz matches the Orin; chain 178829b4ca05
AmateurWizrd added a commit to AmateurWizrd/nxnative that referenced this pull request Nov 10, 2026
…he chain batch (chain 483568384b04)

Rebuilt from the in-repo source after seedds#2042, seedds#2047, seedds#2048 (translator), seedds#1846, seedds#1797 and 064a8cc (dmsh_build.py).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) shader chain The shader translation chain (DMSH_CHAIN): a change rebuilds every game's shaders

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant