ci: deselect PGLE/FDO profiling tests on ROCm (ROCM-25581) [v0.9.2]#790
Open
srinivamd wants to merge 1 commit into
Open
ci: deselect PGLE/FDO profiling tests on ROCm (ROCM-25581) [v0.9.2]#790srinivamd wants to merge 1 commit into
srinivamd wants to merge 1 commit into
Conversation
PGLE/FDO profiling has never worked on any ROCm GPU. The XLA ROCm profiling backend (device_tracer_rocm.cc) uses the legacy roctracer v2 API which was deliberately removed for gfx120X (rocm-systems PR #307). Even on gfx94X where v2 still works, the ROCm collector does not produce CUPTI-equivalent XPlane annotations needed by ConvertXplaneToProfiledInstructionsProto. These 5 tests were already deselected in ROCm/rocm-jax-internal ci/pytest_skips.ini (commit b0cd9da7, 2025-09-26) but that skip file is not applied by the run_pytest_rocm.sh CI runner. Fixes: ROCM-25581
srinivamd
requested review from
Ruturaj4,
charleshofer,
magaonka-amd and
mminutoli
magaonka-amd
approved these changes
Jun 8, 2026
magaonka-amd
left a comment
magaonka-amd
left a comment
There was a problem hiding this comment.
it would be much better to skip it in file, but for now I'l apporve this because I do have plans to visit these fails later this week and address them. dont want to keep QA blocking until then.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of #788 to
rocm-jaxlib-v0.9.2. Deselects 5 failing PGLE/FDO profiling tests from the ROCm CI runner. These tests have never passed on any ROCm GPU.Fixes ROCM-25581
Problem
tests/pgle_test.pyPGLE tests fail on all ROCm GPUs because:gfx120X (RDNA4): rocm-systems PR #307 removed Navi4x support from rocprofiler v2. XLA's ROCm profiler backend still uses the legacy API — every HIP callback is dropped, producing empty FDO profiles.
gfx94X/gfx950 (CDNA): The ROCm collector doesn't produce CUPTI-equivalent XPlane annotations needed by
ConvertXplaneToProfiledInstructionsProto.These tests were already deselected in
ROCm/rocm-jax-internal ci/pytest_skips.inibutrun_pytest_rocm.shdoes not consume that skip file.Tests deselected
PgleTest::testAutoPgleRuntimeWarning: PGLE collected an empty tracePgleTest::testAutoPgleWithCommandBuffers0PgleTest::testAutoPgleWithCommandBuffers1PgleTest::testAutoPgleWithPersistentCachePgleTest::testPGLEProfilerGetFDOProfileAssertionError: b'custom' not foundRelated
rocm-jaxlib-v0.9.1