ci: deselect PGLE/FDO profiling tests on ROCm (ROCM-25581)#788
Open
srinivamd wants to merge 1 commit into
Open
ci: deselect PGLE/FDO profiling tests on ROCm (ROCM-25581)#788srinivamd wants to merge 1 commit into
srinivamd wants to merge 1 commit into
Conversation
PGLE/FDO profiling has never worked on any ROCm GPU. The XLA ROCm profiling backend (device_tracer_rocm.cc) uses the legacy roctracer v2 API which was deliberately removed for gfx120X (rocm-systems PR #307). Even on gfx94X where v2 still works, the ROCm collector does not produce CUPTI-equivalent XPlane annotations needed by ConvertXplaneToProfiledInstructionsProto. These 5 tests were already deselected in ROCm/rocm-jax-internal ci/pytest_skips.ini (commit b0cd9da7, 2025-09-26) but that skip file is not applied by the run_pytest_rocm.sh CI runner. Fixes: ROCM-25581
This was referenced Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deselects 5 failing PGLE/FDO profiling tests from the ROCm CI runner. These tests have never passed on any ROCm GPU.
Fixes ROCM-25581
Problem
tests/pgle_test.pyPGLE tests fail on all ROCm GPUs because:gfx120X (RDNA4): rocm-systems PR #307 (merged 2025-09-22) removed Navi4x support from rocprofiler v2. XLA's ROCm profiler backend (
device_tracer_rocm.cc) still uses the legacy roctracer/v2 API — every HIP API callback is dropped with"No capturing function was found"(28,746 dropped callbacks in the CI log), producing empty FDO profiles.gfx94X/gfx950 (CDNA): Even where roctracer v2 works, the ROCm collector (
rocm_collector.cc) doesn't produce CUPTI-equivalent XPlane annotations.ConvertXplaneToProfiledInstructionsProtofails to extractcustomkernel cost entries → FDO profiles are empty or missing expected kernel names.These tests were already deselected in
ROCm/rocm-jax-internal ci/pytest_skips.ini(commitb0cd9da7, 2025-09-26) for ROCM-24266, butrun_pytest_rocm.shdoes not consume that skip file.Fix
Add
--deselectentries for the 5 failing tests to both the single-accelerator and multi-accelerator pytest invocations inci/run_pytest_rocm.sh, matching the existing pattern for other known-failing tests.Tests deselected
PgleTest::testAutoPgleRuntimeWarning: PGLE collected an empty tracePgleTest::testAutoPgleWithCommandBuffers0PgleTest::testAutoPgleWithCommandBuffers1PgleTest::testAutoPgleWithPersistentCachePgleTest::testPGLEProfilerGetFDOProfileAssertionError: b'custom' not foundLong-term fix
Port XLA's
device_tracer_rocm.ccfrom roctracer v1/v2 to rocprofiler-sdk v3. This is a prerequisite for PGLE/FDO to work on any ROCm GPU and is mandatory for gfx120X where v2 was removed.Related