Skip to content

Far-field hard edges on HIP rest on a square root whose rounding isn't guaranteed, and the fuzz isn't in the tree #504

Closed

Description

@jtravs

What happens

The far-field pass promises that a hard aperture edge decides exactly as numpy_ref decides. That rests on each product and sum rounding on its own, and on a correctly rounded square root.

#497 fixes the first part on HIP: Clang fused the operations there, and on an AMD Instinct MI325X 16 of 588 exact edge decisions disagreed with the reference. The second part is observed, not guaranteed. __dsqrt_rn is __builtin_sqrt on HIP, and I found nothing in its documentation that promises correct rounding. With the fix, 0 of 1,938 decisions disagreed on that card, and 0 of 591 on an NVIDIA H200. far_field.hpp says this in a comment.

The fuzz that found the bug was a throwaway script, so nothing in the tree would catch this coming back.

Why it matters

A wrong decision moves a sample across a hard edge between backends, which breaks the exact hard-mask contract without raising anything. A new ROCm release could change it.

To do

  • Add the exact-edge fuzz to the gpu-marked tests beside tests/backends/test_native_far_field.py: columns placed on a shape's edge, compared with the reference's decisions. It belongs with the device tests, not CI, until CI builds no CUDA or HIP target, so device builds break unnoticed #151 gives CI a device lane.
  • Find out what AMD's device library promises for double-precision sqrt, and state it in the comment and in docs/dev/accuracy.md.

Done when

The fuzz runs in the device tests and fails on the base of #497, and the comment states a documented guarantee or says plainly that there's none.

Depends on #497. Related: #151, #367.

Activity

  1. added
    testingTests, test tiers and CI coverage
    area: gpuCUDA and HIP builds: qualification and performance
    on Nov 4, 2026
  2. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    The first item is in #520: test_columns_on_a_shape_s_edge_decide_on_a_device_as_the_reference_decides in tests/backends/test_native_far_field.py, 40 seeds across a circle's, a half-plane's and an ellipse's hard edge, 5 columns each, with the exact edge and a float either side. The same fuzz runs on CPU with 4 seeds, so CI holds it too.

    • NVIDIA H200: 117 of the 120 cases passed at 8cd170e2. The other 3 were one bad seed in the test, not a decision: it drew only bins the load masks. That's fixed, and each target must now give both answers. It hasn't been rerun on that card.
    • AMD Instinct MI325X: its pass is waiting on capacity. It also builds with SOLPROP_NATIVE_FAR_FIELD_HIP_STRICT emptied, to show the fuzz fails without Far-field aperture shapes, a custom absorbed_power channel, and the boundary items #497's fix, which this issue asks.

    By a separate estimate at double-extended precision, a fused sum would flip about half of the half-plane's exact decisions, so the fuzz should find it.

    The second item, what AMD's device library promises for sqrt, I haven't looked into, so this stays open.

  3. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    The first item now holds on HIP. This was an AMD Instinct MI300X (gfx942, ROCm 10.0.0), with the far field's clang fp contract(off) removed, as on #497's base:

    • 31 of the fuzz's 120 device cases fail: 12 of the circle's 40 seeds, 19 of the ellipse's, and none of the half-plane's.
    • test_a_column_at_a_hard_edge_is_decided_on_a_device_as_the_reference_decides fails too.
    • With the pragma in place, every case passes in that card's pytest -m gpu run.

    So the fuzz sees the bug it was written for. The ledger in #542 has this under #541's pass.

    The second item, what AMD's device library promises for double-precision sqrt, is still open.

  4. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    The second item is answered in #542, so #542 closes this when it merges.

    What AMD documents. HIP's math reference for ROCm 10.0.0 doesn't promise correct rounding:

    • __dsqrt_rn "Returns √x in round-to-nearest-even mode", at 0 ulp.
    • sqrt(double) is listed at 1 ulp.
    • Both figures are a maximum measured over x in [0, 100] against the C++ library, not a bound.

    What the far field actually calls. On a HIP device it calls Kokkos::sqrt, which is HIP's sqrt(double). In ROCm 7.2.4's headers that is __builtin_sqrt, and the device library's __ocml_sqrt_f64 is the same llvm.sqrt.f64. LLVM's LangRef defines that intrinsic this way: "For types specified by IEEE 754, the result matches a conforming libm implementation", unless the call carries afn. IEEE 754's square root is correctly rounded.

    What the compiler does with it. The AMDGPU backend has expanded llvm.sqrt.f64 that way since LLVM e3fd8f8 (2023, "Correctly expand f64 sqrt intrinsic"). Compiled for gfx942 under ROCm 7.2.4 (AMD Clang 22), a kernel's double sqrt is a reciprocal-root estimate with FMA refinement and a residual correction, not the bare v_sqrt_f64. solprop's HIP build sets no fast-math option.

    So the guarantee comes from the compiler's IR semantics, not from AMD's device-library documentation. It holds only while nothing adds afn (for example -ffast-math) to a HIP build. The comment in far_field.hpp and a new paragraph at the end of docs/dev/accuracy.md section 7 say this. With the fuzz in the device tests failing on #497's base on an MI300X (above), both items are done.

  5. jtravs commented on Nov 6, 2026

    @jtravs
    ContributorAuthor

    Both items landed with #542, merged as 1d785cb3: the hard-edge fuzz is in the device tests, and the comment in far_field.hpp plus the paragraph at the end of docs/dev/accuracy.md section 7 say what a HIP device's square root guarantees. #542's body named this issue alongside #529, but GitHub only closes the first issue after "Closes", so I'm closing it by hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: gpuCUDA and HIP builds: qualification and performancepriority: lowWorth doing, no hurrytestingTests, test tiers and CI coverage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions