[ROCm] Skip core_test reduction autotuner failures on Navi/RDNA GPUs (gfx11xx)#758
[ROCm] Skip core_test reduction autotuner failures on Navi/RDNA GPUs (gfx11xx)#758CSkmd wants to merge 42 commits into
Conversation
…tignore (#563) When jaxlib was built in debug more, an assertion in LLVM code that lazy-loads VHLO dialect could fire, since the code path could execute in a multi-threaded environment, and LLVM dialect repositories aren't thread safe to modify. This patch applies the same changes that upstream makes to fix this: jax-ml@48c8762 (this includes disabling a call to `jax_mlir_ext.enter_multi_threaded_execution(context)` in `mlir.py`. Presumably, the whole functionality related to `enter_multi_threaded_execution()` multithreaded checks isn't ready yet, and it was prematurely rolled into the production code. Manual testing
(forgot this skip in the previous PR)
Co-authored-by: Daniel Suo <danielsuo@gmail.com> Co-authored-by: Jake VanderPlas <jakevdp@google.com>
Signed-off-by: Yanyao Wang <wangyanyao@msn.com>
XLA's GPU reduction autotuner has no valid tile configurations for gfx1151 (RDNA3.5/STRIX-HALO) because it uses 32-thread wavefronts (RDNA) vs the 64-thread wavefronts (CDNA) the autotuner was tuned for. JIT compilation triggers autotuning which fails with: NOT_FOUND: No valid config found! Affected tests: - CoreTest::test_simple_jit - CoreTest::test_tracer_reprs - CoreTest::test_verbose_tracer_reprs Follows the same skip pattern as #720 (fused attention / TF32 skips). Detection: ROCm device + "Radeon" in device_kind. Tracking: ROCM-21429 Long-term fix: add valid XLA reduction autotuner tile configs for WarpSize==32 in xla/service/gpu/reduction_utils.cc.
PR AnalysisRoot causeXLA's reduction autotuner ( for both Affected tests
Skip conditionUses An alternative tighter approach is possible using +from jax._src import xla_bridge
...
- if (jtu.is_device_rocm()
- and "Radeon" in jax.local_devices()[0].device_kind):
- self.skipTest(
- "XLA reduction autotuner has no valid config for Navi/RDNA GPUs (gfx11xx)")
+ pv = xla_bridge.get_backend().platform_version
+ if jtu.is_device_rocm() and any(
+ arch in pv for arch in ('gfx1103', 'gfx1150', 'gfx1151')):
+ self.skipTest(
+ "XLA reduction autotuner has no valid tile config for gfx11 APUs "
+ "(ROCM-21429; short-term skip pending fix in openxla/xla)")Long-term fix
|
|
Paging either @mminutoli and @JehandadKhan (or anyone else who's worked on Navi). Do we want to allow this skip for now? I doubt we're going to fix the tiling problem soon. |
|
One thing is sure, if we want the skip it's not to be merged in Nevertheless, I think this is calling for a method @magaonka-amd, @draganmladjenovic thoughts? |
|
@CSkmd thanks for the patch . if you are blocked on CI you can skip it but not this branch. which release branch are you folks using ?, you need to make a PR to that branch and you can assign ticket to me , I'll take a look. |
9a24964 to
40c94c1
Compare
28ad382 to
e3a4195
Compare
729ed31 to
88e42dc
Compare
cb097f7 to
d347a4f
Compare
3f9e8b4 to
f25f2b4
Compare
Description
Short-term CI unblock for ROCM-21429.
Adds skip markers to three failing
core_test.pytests on ROCm Navi/RDNAdevices, following the pattern established in #720. The skip uses
"Radeon" in device_kindto detect RDNA hardware.This is a short-term fix to unblock CI. The proper fix requires changes to
openxla/xla— see the analysis comment and ROCM-21429 for details.Follows skip pattern from #720.
Changes
Affected tests:
CoreTest::test_simple_jitCoreTest::test_tracer_reprsCoreTest::test_verbose_tracer_reprs