Replace rsmi_init with amdsmi_init (via dlsym) in intra_node_comm by adam360x · Pull Request #3299 · ROCm/pytorch

adam360x · 2026-06-14T01:52:38Z

Summary

Replace direct rsmi_init(0) call with dlsym(RTLD_DEFAULT, "amdsmi_init") in intra_node_comm.cpp
Eliminates dangling rsmi_init undefined symbol from libtorch_hip.so
No cmake or link-time dependency changes

Problem

libtorch_hip.so calls rsmi_init() but does not list librocm_smi64.so as a NEEDED dependency — rsmi_init is left as an undefined symbol resolved at runtime. libamd_smi.so exports rsmi_init for backward compatibility, so when it gets loaded with RTLD_GLOBAL (via the ROCm SDK dependency chain), the dynamic linker interposed that rsmi_init over libamd_smi.so's own internal copy. This left AMDSMI's RSMI singleton uninitialized, resulting in zero devices or sentinel values (e.g. gfxffffffffffffffff) when users called amdsmi_init() after importing torch.

Reproducer: import torch before amdsmi_init() on navi31/navi48 with pip-installed amdsmi and ROCm 7.12/7.13 nightlies.

Root cause verification

Confirmed via isolated ctypes tests on a Radeon RX 7900 XT (gfx1100):

# BROKEN: system librocm_smi64 loaded globally poisons nightly libamd_smi
ctypes.CDLL("librocm_smi64.so.1", mode=RTLD_GLOBAL)
nightly = ctypes.CDLL("libamd_smi.so.26")
nightly.amdsmi_init(2)  # returns SUCCESS
# amdsmi_get_socket_handles -> 0 sockets (broken)

# FIXED: with RTLD_LOCAL, no interposition
ctypes.CDLL("librocm_smi64.so.1", mode=RTLD_LOCAL)
nightly = ctypes.CDLL("libamd_smi.so.26")
nightly.amdsmi_init(2)  # returns SUCCESS
# amdsmi_get_socket_handles -> 2 sockets (correct)

Fix

Use dlsym(RTLD_DEFAULT, "amdsmi_init") to call amdsmi_init at runtime instead of linking rsmi_init directly. This:

Removes the rsmi_init undefined symbol from libtorch_hip.so, eliminating the interposition vector
Avoids any link-time NEEDED dependency on libamd_smi.so
Gracefully fails if libamd_smi.so is not loaded

rsmi_is_P2P_accessible remains unchanged — amdsmi_init initializes the RSMI layer internally, so existing rsmi_* query calls continue to work.

Test results

Build 4: 103,130 passed, 3 failed, 30,322 skipped. All 3 failures (test_ddp_apply_optim_in_backward_ignored_params, test_host_memory_stats, test_cuda_graph_tensor_item_not_allowed) are pre-existing on develop (develop build #25 has 31 failures including the same tests).

(cherry picked from commit a66eeda) Fixes #ISSUE_NUMBER Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>

========================================== Triton build conditionalized on ROCM_VERSION Include the ROCm version in triton version (cherry picked from commit 7d33910) (cherry picked from commit 0412eb4) Update triton-rocm.txt to triton.txt (cherry picked from commit 0ce9f6e) Use ROCm/triton for install_triton.sh (cherry picked from commit 6e9714b) update triton commit Revert "Use ROCm/triton for install_triton.sh" This reverts commit 81b0cbc8435122030044049c661f252ee8aa7ae5. change triton repo Update triton.txt to use release/internal/3.3.x branch Use ROCm/triton Use ROCm/triton for install_triton.sh (cherry picked from commit 0036db5)

…on (#2482) Related to https://github.com/ROCm/builder/pull/90/files http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/305/ PyTorch wheel installs successfully when building torchvision/torchaudio (cherry picked from commit c1ee54d)

Fixes #ISSUE_NUMBER (cherry picked from commit 0ea0592)

…A helper functions ======================================================================================= Implementation of PyTorch ut parsing script - QA helper function (#1386) * Initial implementation of PyTorch ut parsing script * Extracted path variables * Use nested dict to save results * Fixes typo * Cleanup * Fixes several issues * Minor name change * Update run_pytorch_unit_tests.py * Added file banners * Supported running from API * Added more help info * Consistent naming * Format help text --------- Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> Co-authored-by: Jithun Nair <jithun.nair@amd.com> Print consolidated log file for pytorch unit test automation scripts (#1433) * Print consolidated log file for pytorch uts * Update run_entire_tests subprocess call as well * lint * Add ERROR string [SWDEV-466849] Enhancements for PyTorch UT helper scripts (#1491) * Check that >1 GPUs are visible when running TEST_CONFIG=distributed * Add EXECUTION_TIME to file-level and aggregate statistics PyTorch unit test helper scripts enhancements (#1517) * Fail earlier for distributed-on-1-GPU scenario * print cmd in consolidated log with prettier formatting * python->python3 Fixes https://ontrack-internal.amd.com/browse/SWDEV-477264 --------- Co-authored-by: blorange-amd <bo.li2@amd.com> Several issues fix of QA helper script (#1564) Fixes SWDEV-475071: https://ontrack-internal.amd.com/browse/SWDEV-475071 Removed args inside function (#1595) Fixes SWDEV-475071 (cherry picked from commit 041aa1b47978154de63edc6b7ffcdea218a847a3) QA script - Added multi gpu check with priority_tests (#1604) Fixes SWDEV-487907. Verified throwing exception for distributed is working correctly on single gpu with command: python .automation_scripts/run_pytorch_unit_tests.py --priority_test (cherry picked from commit 57cc742271cbf4547f9213710e57f6444bbc983e) (cherry picked from commit 6d5c3dc) (cherry picked from commit 2ee3aa2)

* Use triton commit same as that used for release/2.6 branch since both are triton version 3.2.0, so assuming they're compatible. Relates to: https://github.com/ROCm/rocAutomation/pull/660/files https://github.com/ROCm/builder/pull/70/files Validation http://ml-ci-internal.amd.com:8080/job/pytorch/job/manylinux_rocm_wheels/568/ --------- Co-authored-by: Jithun Nair <jithun.nair@amd.com> Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> (cherry picked from commit 14c1417) (cherry picked from commit c20a8f8)

* Add trailing comma for consistency in gfx architecture list Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> * ROCm: Enable tf32 testing on test_nn Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> --------- Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> (cherry picked from commit c113e14)

…-deps flags (#2121) Cherry-pick of #2103 Co-authored-by: Ethan Wee <Ethan.Wee@amd.com> (cherry picked from commit 1dea6e8)

Relates to: ROCm/builder#82 Validation: http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/98/ Using `registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16180_ubuntu24.04_py3.12_pytorch_lw_rocm7.0_IT_upgrade_numpy_452f3df6`: ``` root@d92befdbb2a6:/# pip list | egrep "numpy|pandas" numpy 2.1.2 pandas 2.2.3 root@d92befdbb2a6:/# python3 Python 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas >>> import torch >>> import numpy >>> exit() root@d92befdbb2a6:/data/pytorch-micro-benchmarking# HIP_VISIBLE_DEVICES=1 python3 micro_benchmarking_pytorch.py --network resnet50 INFO: running forward and backward for warmup. INFO: running the benchmark.. OK: finished running benchmark.. --------------------SUMMARY-------------------------- Microbenchmark for network : resnet50 Num devices: 1 Dtype: FP32 Mini batch size [img] : 64 Time per mini-batch : 0.11369450092315674 Throughput [img/sec] : 562.9120096428937 ``` --------- Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> (cherry picked from commit cf32479)

…2269) Fixes SWDEV-536456 Fixes error post-#2256: ``` 00:12:44.248 #22 155.3 ERROR: Ignored the following versions that require a different python version: 0.52.0 Requires-Python >=3.6,<3.9; 0.52.0rc3 Requires-Python >=3.6,<3.9; 0.61.0 Requires-Python >=3.10; 0.61.0rc1 Requires-Python >=3.10; 0.61.0rc2 Requires-Python >=3.10; 0.61.1rc1 Requires-Python >=3.10; 0.61.2 Requires-Python >=3.10; 3.3 Requires-Python >=3.10; 3.3rc0 Requires-Python >=3.10; 3.4 Requires-Python >=3.10; 3.4.1 Requires-Python >=3.10; 3.4.2 Requires-Python >=3.10; 3.4rc0 Requires-Python >=3.10; 3.5 Requires-Python >=3.11; 3.5rc0 Requires-Python >=3.11; 8.2.0 Requires-Python >=3.10; 8.2.1 Requires-Python >=3.10 00:12:44.248 #22 155.3 ERROR: Could not find a version that satisfies the requirement numba==0.61.2 (from versions: 0.1, 0.2, 0.3, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.12.1, 0.12.2, 0.13.0, 0.13.2, 0.13.3, 0.13.4, 0.14.0, 0.15.1, 0.16.0, 0.17.0, 0.18.1, 0.18.2, 0.19.1, 0.19.2, 0.20.0, 0.21.0, 0.22.0, 0.22.1, 0.23.0, 0.23.1, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.1, 0.29.0, 0.30.0, 0.30.1, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.1, 0.36.2, 0.37.0, 0.38.0, 0.38.1, 0.39.0, 0.40.0, 0.40.1, 0.41.0, 0.42.0, 0.42.1, 0.43.0, 0.43.1, 0.44.0, 0.44.1, 0.45.0, 0.45.1, 0.46.0, 0.47.0, 0.48.0, 0.49.0, 0.49.1rc1, 0.49.1, 0.50.0rc1, 0.50.0, 0.50.1, 0.51.0rc1, 0.51.0, 0.51.1, 0.51.2, 0.52.0rc2, 0.53.0rc1.post1, 0.53.0rc2, 0.53.0rc3, 0.53.0, 0.53.1, 0.54.0rc2, 0.54.0rc3, 0.54.0, 0.54.1rc1, 0.54.1, 0.55.0rc1, 0.55.0, 0.55.1, 0.55.2, 0.56.0rc1, 0.56.0, 0.56.2, 0.56.3, 0.56.4, 0.57.0rc1, 0.57.0, 0.57.1rc1, 0.57.1, 0.58.0rc1, 0.58.0rc2, 0.58.0, 0.58.1, 0.59.0rc1, 0.59.0, 0.59.1, 0.60.0rc1, 0.60.0) 00:12:44.248 #22 155.3 ERROR: No matching distribution found for numba==0.61.2 ``` Validation: * Docker image: http://rocm-ci.amd.com/job/mainline-framework-pytorch-internal-cs9-ci/132 * Wheels: http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/102/ From `registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16180_ubuntu22.04_py3.9_pytorch_lw_rocm7.0_IT_py3.9_a11d94ad`: ``` root@f43861a0a856:/# pip list | egrep "numpy|pandas" numpy 2.0.2 pandas 2.2.3 root@f43861a0a856:/# python Python 3.9.23 (main, Jun 4 2025, 08:55:38) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> import numpy >>> import pandas root@f43861a0a856:/data/pytorch-micro-benchmarking# HIP_VISIBLE_DEVICES=1 python3 micro_benchmarking_pytorch.py --network resnet50 INFO: running forward and backward for warmup. INFO: running the benchmark.. OK: finished running benchmark.. --------------------SUMMARY-------------------------- Microbenchmark for network : resnet50 Num devices: 1 Dtype: FP32 Mini batch size [img] : 64 Time per mini-batch : 0.11354223489761353 Throughput [img/sec] : 563.6669038416574 ``` (cherry picked from commit a0a9d81)

…cm7.0/7.1 (#2239) Revamped version of #2108 PR to: - enable complex data types for sparse matmul on ROCm - fix sparse addmm/baddbmm on ROCm - fix sparse hipification for ROCm - fix/enable sparse tests on ROCm (~50 tests total for non-fp16/bf16): - enable fp16/bf16 sparse path for rocm7.0 - enable fp16/bf16 sparse tests for rocm7.0/7.1 ``` test_sparse_csr.py::TestSparseCSRCUDA::test_bmm_cuda_* test_sparse.py::TestSparseCUDA::test_sparse_matmul_cuda_* test_sparse_csr.py::TestSparseCSRCUDA::test_mm_cuda_float64 test_sparse_csr.py::TestSparseCSRCUDA::test_addmm_all_sparse_csr_SparseCS* test_sparse_csr.py::TestSparseCSRCUDA::test_addmm_sizes_all_sparse_csr_* test_sparse_csr.py::TestSparseCSRCUDA::test_sparse_addmm_cuda_float16 ``` (cherry picked from commit cc2a69c)

#2326) Fixes https://ontrack-internal.amd.com/browse/SWDEV-541809 Upgrading tensorboard after numpy upgrade Ran in **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16381_ubuntu24.04_py3.12_pytorch_lw_rocm7.0_internal_testing_afe8b782** ``` 7 git checkout rocm7.0_IT_upgrade_tensorboard 8 pip install .ci/docker/requirements-ci.txt 9 pip install -r .ci/docker/requirements-ci.txt 10 PYTORCH_TEST_WITH_ROCM=1 python test/test_monitor.py TestMonitorTensorboard.test_event_handler root@ubb4-rack-22:/var/lib/jenkins/pytorch# PYTORCH_TEST_WITH_ROCM=1 python test/test_monitor.py TestMonitorTensorboard.test_event_handler /opt/venv/lib/python3.12/site-packages/google/protobuf/internal/well_known_types.py:91: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). _EPOCH_DATETIME_NAIVE = datetime.datetime.utcfromtimestamp(0) . ---------------------------------------------------------------------- Ran 1 test in 0.327s OK root@ubb4-rack-22:/var/lib/jenkins/pytorch# ``` (cherry picked from commit c7f61f4)

Tested locally successfully ``` root@rocm-framework-47:/var/lib/jenkins/pytorch# pip install -r requirements.txt Ignoring numpy: markers 'python_version == "3.9"' don't match your environment Requirement already satisfied: setuptools<80.0,>=70.1.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 2)) (79.0.1) Requirement already satisfied: cmake>=3.31.4 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 3)) (4.0.0) Requirement already satisfied: ninja==1.11.1.3 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 4)) (1.11.1.3) Requirement already satisfied: numpy==2.1.2 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 5)) (2.1.2) Requirement already satisfied: packaging==25.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 6)) (25.0) Requirement already satisfied: pyyaml==6.0.2 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 7)) (6.0.2) Requirement already satisfied: requests==2.32.4 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2.32.4) Requirement already satisfied: six==1.17.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 9)) (1.17.0) Requirement already satisfied: typing-extensions==4.14.1 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 10)) (4.14.1) Requirement already satisfied: expecttest==0.3.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (0.3.0) Requirement already satisfied: filelock==3.18.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (3.18.0) Requirement already satisfied: fsspec==2025.7.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (2025.7.0) Requirement already satisfied: hypothesis==5.35.1 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (5.35.1) Requirement already satisfied: jinja2==3.1.6 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (3.1.6) Requirement already satisfied: lintrunner==0.12.7 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (0.12.7) Requirement already satisfied: networkx==2.8.8 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (2.8.8) Requirement already satisfied: optree==0.13.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 18)) (0.13.0) Requirement already satisfied: psutil==7.0.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 19)) (7.0.0) Requirement already satisfied: sympy==1.13.3 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 20)) (1.13.3) Requirement already satisfied: wheel==0.45.1 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 22)) (0.45.1) Requirement already satisfied: build[uv] in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (1.3.0) Requirement already satisfied: charset_normalizer<4,>=2 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (3.4.3) Requirement already satisfied: idna<4,>=2.5 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2.5.0) Requirement already satisfied: certifi>=2017.4.17 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2025.8.3) Requirement already satisfied: attrs>=19.2.0 in /opt/venv/lib/python3.10/site-packages (from hypothesis==5.35.1->-r requirements.txt (line 11)) (25.3.0) Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /opt/venv/lib/python3.10/site-packages (from hypothesis==5.35.1->-r requirements.txt (line 11)) (2.4.0) Requirement already satisfied: MarkupSafe>=2.0 in /opt/venv/lib/python3.10/site-packages (from jinja2==3.1.6->-r requirements.txt (line 12)) (3.0.2) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/venv/lib/python3.10/site-packages (from sympy==1.13.3->-r requirements.txt (line 20)) (1.3.0) Requirement already satisfied: pyproject_hooks in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (1.2.0) Requirement already satisfied: tomli>=1.1.0 in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (2.2.1) Requirement already satisfied: uv>=0.1.18 in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (0.8.10) root@rocm-framework-47:/var/lib/jenkins/pytorch# pip install -r requirements-build.txt ``` (cherry picked from commit 6e6e454)

This also fixes a problem in gesvd driver when UV is not needed. (cherry picked from commit 4ce57ec) (cherry picked from commit 167b4c1)

(cherry picked from commit d6879fa) (cherry picked from commit 123a164)

Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> (cherry picked from commit 1ad5bb95d796283d5f56ac1edd16f1731d24a49d) (cherry picked from commit 519160d)

Fixes #ISSUE_NUMBER

- Need to use upstream/main for rocm/pytorch's develop branch. For release branches, `github.event.pull_request.base.ref` should work as is. - Need to remove any trailing space in PR TITTLE so branch name can be formed correctly Fixes #ISSUE_NUMBER

# Conflicts: # .ci/docker/requirements-ci.txt

[AUTOGENERATED] develop_IFU_20251104

# Conflicts: # .ci/docker/ci_commit_pins/triton.txt # requirements.txt

To keep triton version consistent with what is in rocm/triton's release/internal/3.5.x branch, we need to keep triton_version.txt at 3.5.0 and move triton hash to ToT of that branch.

[AUTOGENERATED] develop_IFU_20251118

[AUTOGENERATED] develop_IFU_20251124

# Conflicts: # .ci/docker/ci_commit_pins/triton.txt # .ci/docker/requirements-ci.txt # .ci/docker/triton_version.txt # .circleci/scripts/binary_populate_env.sh # .github/scripts/build_triton_wheel.py # test/test_sparse_csr.py

…umn (#3153) ## Summary - Only display tests where ROCm status is FAILED in the summary (CUDA status shown as a context column alongside). Previously both ROCm and CUDA failures were shown. - Add "Also Failing In" column that shows which other architectures have the same test tuple (test_file, test_class, test_name) failing, making it easy to distinguish all-ROCm issues from architecture-specific ones. - Includes count of failed tests in the section header. - Add job-level and test-level shard info to "LOG-BASED FAILURES (not in XML)" and "FAILED TESTS" section - Includes flaky tests in "LOG-BASED FAILURES (not in XML)" section for any tests that pass when run in new process ## Test plan - [x] Cross-arch detection confirmed: tests failing on all 3 archs show the other 2 in "Also Failing In"; single-arch failures show empty - [x] CSV and Markdown output both updated consistently Latest run https://github.com/ROCm/pytorch/actions/runs/24798004968 Run without this PR on the same commit: https://github.com/ROCm/pytorch/actions/runs/24796654604

Repro job without this PR's change: https://github.com/ROCm/pytorch/actions/runs/25342470426/job/74303089638 Validation run with this PR's change: https://github.com/ROCm/pytorch/actions/runs/25342235984 Current issue: existing testing is not able to pick up the CUDA artifacts because the CUDA job and artifact names changed from `test` to `test-osdc` for default and distributed shards. Repro inputs: `sha=b1b5b61ddb689ea65aab0915ecfac5cc459b92fb`, `arch=mi355`, `skip_rocm=false`, `csv_name=pr3199-pre-change-repro`. CUDA job names now use `test-osdc` for default and distributed shards, for example: `linux-jammy-cuda13.0-py3.10-gcc11 / test-osdc (default, 1, 5, ...)` `linux-jammy-cuda13.0-py3.10-gcc11 / test-osdc (distributed, 1, 3, ...)` CUDA artifact names now look like: `test-reports-test-osdc-default-1-5` `test-reports-test-osdc-distributed-1-3`

## Summary - Update MI355 parity report shard counts to match current CI artifacts. - Change default shards from 6 to 10 and distributed shards from 3 to 4. ## Validation * Combined parity workflow for `5b9a4786ea4b1a6170c6e5a4878269e7f591224b` on `mi300, mi355`: <https://github.com/ROCm/pytorch/actions/runs/25738157290> --------- Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>

## Motivation Old IFU_GITHUB_TOKEN [seems to have expired](https://github.com/ROCm/pytorch/actions/runs/25856299592/job/75974982737) ## Technical Details Replace with PARITY_GITHUB_TOKEN (meant specifically for this workflow) ## Test Plan Run parity.yml with this PR branch and see if it still gives credential error. ## Test Result "Download artifacts" step succeeded in https://github.com/ROCm/pytorch/actions/runs/25857211908/job/75978008711 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

## Summary - Select the CUDA test artifact kind from the jobs present for the target SHA. - Detect whether the target SHA uses test-osdc or legacy test CUDA jobs, then use the detected kind when building log keys and artifact prefixes. - Apply the same dynamic selection to CUDA inductor jobs. - Treat missing per-arch summary buckets as zero so mixed ROCm/CUDA coverage does not crash report generation. ## Validation - PR/ciflow case: dispatched `Parity Report` on this branch with `sha=386f38175e3aaee2dadb36b5c364deff0869664d` and `arch=mi355, mi300, mi200, navi31`. CUDA default/distributed and inductor selected `test`. - Run: https://github.com/ROCm/pytorch/actions/runs/25866762885 - Main branch case: dispatched `Parity Report` on this branch with `sha=f38b1ec280bafa2ad11f6e767558e73e9eb508a6`, `arch=mi300`, `skip_rocm=true`, and `exclude_distributed=true`. CUDA default and inductor selected `test-osdc`. - Run: https://github.com/ROCm/pytorch/actions/runs/25867046276 - Local syntax check: `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogs .automation_scripts/pytorch-unit-test-scripts/generate_summary.py`.

## Summary - Prefer the arch-specific MI200 workflows in `download_testlogs`: `rocm-mi200`, `periodic-rocm-mi200`, and `inductor-rocm-mi200`. - Match arch-specific MI200 test jobs with the `linux-jammy-rocm-py3.10-mi200` prefix for default, distributed, and inductor shards. - Keep `trunk-rocm-sandbox` as the fallback workflow for older SHAs that do not have the MI200-specific workflows, using the legacy `linux-jammy-rocm-py3.10` prefix in that fallback path. ## Motivation A parity run for `50d07a990e33f9822ae4d48bed2d7f06c96522d0` tried to collect MI200 distributed jobs with: `linux-jammy-rocm-py3.10 / test (distributed, ...)` The upstream jobs for this SHA are arch-specific and include `-mi200`, so the log lookup missed all three shards and XML artifact collection fell through to empty results. The script should look for the MI200-specific workflows first, then fall back to `trunk-rocm-sandbox` for older commits. ## Validation - `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogs` - Confirmed the fixed prefix matches upstream jobs for `50d07a990e33f9822ae4d48bed2d7f06c96522d0`: - `rocm-mi200`: 6 default shard matches - `periodic-rocm-mi200`: 3 distributed shard matches - `inductor-rocm-mi200`: 2 inductor shard matches - Dispatched `Parity Report` on this branch with `sha=50d07a990e33f9822ae4d48bed2d7f06c96522d0`, `arch=mi200`, and `skip_cuda=true` to validate collection end-to-end. - Initial run before fallback commit: https://github.com/ROCm/pytorch/actions/runs/25920564353 (success) - Current branch run after fallback commit: https://github.com/ROCm/pytorch/actions/runs/25920808611 (queued) Made with [Cursor](https://cursor.com)

## Summary - Raise the Python CSV parser field limit in `generate_summary.py` so large parity CSV diagnostic fields can be read. - Truncate oversized diagnostic text fields while loading rows so long failure/skip messages do not make summary generation or output unwieldy. - Preserve test identity, status, timing, and shard fields used by the parity report tables. ## Root Cause A parity run failed in the `summarize` job when Python's default CSV field limit rejected a generated-code assertion message larger than 131,072 bytes: https://github.com/ROCm/pytorch/actions/runs/26168276671/job/76979094769 The first offending row was `inductor.test_torchinductor_codegen_dynamic_shapes::DynamicShapesCodegenGPUTests::test_vmap_dot_decomposes_bmm_dynamic_shapes_cuda`, where `message_rocm` was 145,748 bytes. ## Test plan - `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/generate_summary.py` - Re-ran `generate_summary.py` locally against the artifact from the failed run: - Input: `20260520_all_tests_status_mi355.csv` from run `26168276671` - Output: summary CSV and markdown generated successfully instead of failing with `_csv.Error: field larger than field limit (131072)`. - Triggered `parity.yml` on this branch with the same upstream commit and arch as the failing run: - SHA: `27f2e80e30fb950bc455c777a5e8079e9657a157` - Arch: `mi355` - Validation run: https://github.com/ROCm/pytorch/actions/runs/26175417191 - Result: `setup-matrix`, `generate-parity (mi355)`, and `summarize` all completed successfully. - The summarize log shows `CSV written to 27f2e80e30fb950bc455c777a5e8079e9657a157_summary.csv` and `Markdown written to 27f2e80e30fb950bc455c777a5e8079e9657a157_summary.md`.

## Summary Adds a single step to the `summarize` job in `parity.yml` that uploads the generated `*_summary.md` (the same content already appended to `$GITHUB_STEP_SUMMARY`) as a standalone artifact named `parity-summary-md`, with N-day retention. The existing per-arch result artifacts have a 1-day retention, which makes it impossible to recover the summary content (e.g. `### FAILED TESTS`, `### LOG-BASED FAILURES`) after that window. This change lets external tooling — for example the in-progress upstream CI failure tracking — fetch the exact UI summary via `gh run download` long after the CSVs are gone, with only a standard PAT. No behavior change for any existing job. `if-no-files-found: ignore` keeps the step a no-op on early-exit runs (no CSVs produced). ## Test plan - [ ] Re-run `parity.yml` (or an autoparity manual dispatch) and verify the `parity-summary-md` artifact appears alongside the per-arch results artifacts. - [ ] `gh run download <run_id> -R ROCm/pytorch -n parity-summary-md` returns the expected `*_summary.md`. - [ ] On a run with no CSVs (forced early exit), confirm the workflow still succeeds and no artifact is uploaded. Signed-off-by: Garay-Fernandez <pgarayfe@amd.com>

## Summary - Adds a clickable **Job ID** column at the end of both the `FAILED TESTS` and `LOG-BASED FAILURES (not in XML)` tables in the parity summary markdown. Each cell renders as `[<job_id>](https://github.com/pytorch/pytorch/actions/runs/<wf>/job/<job_id>)`, dropping the reviewer one click away from the stacktrace. - Threads the upstream `pytorch/pytorch` CI job url through the existing pipeline — `download_testlogs` was already fetching that info, it just wasn't being preserved. No new API calls; no schema migrations; just persistence through `download_testlogs` → `summarize_xml_testreports.py` / `detect_log_failures.py` → `generate_summary.py`. - Backwards-compatible: every consumer reads the new fields via `.get(..., '')` / `os.path.isfile`, so older artifacts and CSVs render the column as empty cells instead of breaking. ### Example resulting row (FAILED TESTS, set2-disabled case) ``` | Arch | Test Config | Test File | Test Class | Test Name | Job-Level Shard (rocm) | Test-Level Shard (rocm) | Status (rocm) | Also Failing In | Job ID (rocm) | | mi300 | default | test_foo | TestBar | test_baz | 3/6 | 5/15 | FAILED | mi355 | [76905282313](https://github.com/pytorch/pytorch/actions/runs/26146653222/job/76905282313) | ``` ### Data flow - **FAILED TESTS** (XML-based): `_shorten_unzipped_dirs` keeps the trailing `_<jobid>` of the artifact name on each `test-<cfg>-N-N/` dir → `download_xml_files` writes one `_wf_run_id` file at the parent → `parse_xml_reports_as_dict` builds the url and stamps it on each test case → per-arch CSV carries `job_url_{set_name}` → `collect_failed_tests` propagates → markdown renders. - **LOG-BASED FAILURES**: `write_test_log_to_file` writes a companion `<filename>.job_url` file (full url from the job's `html_url`) → `scan_logs` reads it and stamps `job_url` on every failure / flaky row → `log_failures_<arch>.csv` / `flaky_tests_<arch>.csv` carry it → `load_log_failures` / `load_flaky_tests_as_log_failures` propagate → markdown renders. ## Test plan - Trigger a `parity.yml` run and confirm: - Per-arch test-report shard dirs are named `test-<cfg>-N-N_<jobid>` after `_shorten_unzipped_dirs`. - `_wf_run_id` file exists alongside the shard dirs in `rocm_xml/` and `cuda_xml/`. - `<filename>.job_url` companion files exist next to each `rocm*.txt` / `cuda*.txt` log file. - Inspect the per-arch CSV emitted by `summarize_xml_testreports.py` and confirm `job_url_<set1_name>` / `job_url_<set2_name>` columns are populated for failing rows. - Inspect `log_failures_<arch>.csv` / `flaky_tests_<arch>.csv` and confirm `job_url` column is populated. - Inspect the parity summary markdown artifact and click a `Job ID` cell in both tables → lands on the failing pytorch/pytorch job page with the stacktrace. - Re-run against a historical commit whose artifacts predate this change and confirm cells render as empty (no crash, no broken table). --------- Signed-off-by: Garay-Fernandez <pgarayfe@amd.com>

## Summary Add a clickable `HUD LINK` to the parity report summary so users can jump from the GitHub Actions run to the matching PyTorch HUD page. For PR runs, the link points to the PR HUD page with the exact SHA used for the report, e.g. `https://hud.pytorch.org/pytorch/pytorch/pull/<pr_id>?sha=<sha>`. For SHA-only runs, the link points to the commit HUD page. ## Validation - PR-only example: triggered `parity.yml` with `pr_id=184377`, `arch=mi355`, and no SHA. The report resolved SHA `291ff45ffe10a301a88d1a83e98b9ba9987dbbfa`, so the HUD link is `https://hud.pytorch.org/pytorch/pytorch/pull/184377?sha=291ff45ffe10a301a88d1a83e98b9ba9987dbbfa`. Run passed: https://github.com/ROCm/pytorch/actions/runs/26476256833 - SHA-only example: triggered `parity.yml` with `sha=fe1b0a2ae93e0efcfa0defeee2ed879cf68eaac6`, `arch=mi355`, and no PR ID. Run passed: https://github.com/ROCm/pytorch/actions/runs/26475135922

## Summary - Adds a `Run Time (s)` column to both parity summary tables (FAILED TESTS and LOG-BASED FAILURES), in the `.md` and `.csv` outputs. - **FAILED** tests use the per-test JUnit XML `time` (already in the per-arch CSV as `running_time_<set>`). - **LOG-BASED** failures (timeouts/crashes/kills, which produce no XML) use the failing **job's wall-clock**, computed in `detect_log_failures.py` from each log's first-to-last ISO timestamps and attached to every failure/flaky entry. Implements ROCm/frameworks-internal#16856. ## Changes - `detect_log_failures.py`: compute per-log job run time; add `run_time` to the failures and flaky CSV reports. - `generate_summary.py`: `Run Time (s)` column in FAILED + LOG-BASED tables; thread run time through the flaky loader. ## Test plan - [x] Offline end-to-end: synthetic timeout log -> `detect_log_failures` (1801s) -> `generate_summary` LOG-BASED table -> downstream parser - [x] FAILED table run time verified with a synthetic per-arch CSV - [x] Verify on a real parity run --------- Signed-off-by: pablo-garay <pgarayfe@amd.com>

rocm-repo-management-api · 2026-06-14T02:06:52Z

Jenkins build for df95aa5c0d12e2db9f9f91644fc4e3e323910e5f commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Detected error during Pytorch building:

      |             ^~~~~~~~~~~~~~~~~~
[7680/8176] Building CXX object caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/CUDASymmetricMemoryUtils.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7681/8176] Building CXX object caffe2/CMakeFiles/MaybeOwned_test.dir/__/aten/src/ATen/test/MaybeOwned_test.cpp.o
[7682/8176] Building CXX object caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp.o 
/opt/cache/bin/sccache /opt/cache/bin/c++ -DAT_PER_OPERATOR_HEADERS -DFLASHATTENTION_DISABLE_ALIBI -DFLASHATTENTION_DISABLE_SOFTCAP -DFLASH_NAMESPACE=pytorch_flash -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DHIPBLASLT_USE_ROCROLLER -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DROCM_VERSION=70204 -DTORCH_CUDA_BUILD_MAIN_LIB -DTORCH_HIP_VERSION=702 -DUNFUSE_FMA -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_LAYERNORM_FAST_RECIPROCAL -DUSE_MEM_EFF_ATTENTION -DUSE_NCCL -DUSE_PROF_API=1 -DUSE_ROCM -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_AMD__=1 -Dtorch_hip_EXPORTS -I/var/lib/jenkins/pytorch/build/aten/src -I/var/lib/jenkins/pytorch/aten/src -I/var/lib/jenkins/pytorch/build -I/var/lib/jenkins/pytorch -I/var/lib/jenkins/pytorch/nlohmann -I/var/lib/jenkins/pytorch/moodycamel -I/var/lib/jenkins/pytorch/aten/src/THH -I/var/lib/jenkins/pytorch/third_party/mslk/include -I/var/lib/jenkins/pytorch/aten/src/ATen/hip -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/composable_kernel/include -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/composable_kernel/library/include -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/composable_kernel/example/ck_tile/01_fmha -I/var/lib/jenkins/pytorch/build/caffe2/aten/src/ATen/composable_kernel -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/aiter/csrc/include -I/var/lib/jenkins/pytorch/third_party/fmt/include -I/var/lib/jenkins/pytorch/build/caffe2/aten/src -I/var/lib/jenkins/pytorch/aten/src/ATen/.. -I/var/lib/jenkins/pytorch/torch/include -I/var/lib/jenkins/pytorch/c10/hip/../.. -I/var/lib/jenkins/pytorch/c10/.. -I/var/lib/jenkins/pytorch/torch/csrc/api -I/var/lib/jenkins/pytorch/torch/csrc/api/include -I/var/lib/jenkins/pytorch/build/third_party/gloo/hip -isystem /opt/rocm-7.2.4/include -isystem /var/lib/jenkins/pytorch/build/third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/tensorpipe/third_party/libuv/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googletest/include -isystem /var/lib/jenkins/pytorch/third_party/protobuf/src -isystem /opt/conda/envs/py_3.12/include -isystem /var/lib/jenkins/pytorch/third_party/XNNPACK/include -isystem /var/lib/jenkins/pytorch/third_party/ittapi/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/eigen -isystem /opt/rocm/include -isystem /var/lib/jenkins/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /var/lib/jenkins/pytorch/third_party/ideep/include -isystem /var/lib/jenkins/pytorch/INTERFACE -isystem /var/lib/jenkins/pytorch/third_party/nlohmann/include -isystem /var/lib/jenkins/pytorch/third_party/concurrentqueue -isystem /opt/rocm-7.2.4/include/hiprand -isystem /opt/rocm-7.2.4/include/rocrand -isystem /opt/rocm/magma/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_MSLK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -fPIC -fdiagnostics-color=always -DMKL_HAS_SBGEMM -DMKL_HAS_SHGEMM -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -Wall -Wextra -Wdeprecated -Wunused -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wredundant-move -Wno-interference-size -Wno-maybe-uninitialized -fvisibility=hidden -fPIC -D__HIP_PLATFORM_AMD__=1 -DCUDA_HAS_FP16=1 -DUSE_ROCM -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DTORCH_HIP_VERSION=702 -Wno-shift-count-negative -Wno-shift-count-overflow -DCAFFE2_USE_MIOPEN -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_HIP -DHIPBLAS_V2 -DHIP_ENABLE_WARP_SYNC_BUILTINS -DHIPBLASLT_OUTER_VEC -DUSE_ROCM_CK_GEMM -DHIP_VERSION=7 -Wno-duplicate-decl-specifier -DUSE_MIOPEN -MD -MT caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp.o -MF caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp.o.d -o caffe2/CMakeFiles/torch_hip.dir/__/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp.o -c /var/lib/jenkins/pytorch/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
/var/lib/jenkins/pytorch/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp: In function ‘c10d::intra_node_comm::NvlMesh c10d::intra_node_comm::getNvlMesh(const std::vector<int>&)’:
/var/lib/jenkins/pytorch/torch/csrc/distributed/c10d/symm_mem/intra_node_comm.cpp:57:7: error: ‘amdsmi_processor_type_t’ was not declared in this scope; did you mean ‘amdsmi_processor_handle’?
   57 |       amdsmi_processor_type_t ptype;

rocm-repo-management-api · 2026-06-14T17:21:52Z

Jenkins build for 3315c812c921428a48ba172144c7a356a7f1d411 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Detected error during Pytorch building:

[8074/8176] Linking CXX shared library lib/libtorch.so
Warning: Unused direct dependencies:
	/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so
	/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so
[8075/8176] Linking CXX executable bin/accelerator_graph_test
FAILED: bin/accelerator_graph_test 
: && /opt/cache/bin/c++ -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_MSLK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic     -Wl,--dependency-file=caffe2/CMakeFiles/accelerator_graph_test.dir/link.d -Wl,--no-as-needed caffe2/CMakeFiles/accelerator_graph_test.dir/__/aten/src/ATen/test/accelerator_graph_test.cpp.o -o bin/accelerator_graph_test -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/var/lib/jenkins/pytorch/build/lib:/opt/rocm-7.2.4/lib:/opt/rocm/lib:  lib/libgtest_main.a  lib/libgtest.a  lib/libgmock.a  -lstdc++  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/conda/envs/py_3.12/lib/libmkl_intel_lp64.a  /opt/conda/envs/py_3.12/lib/libmkl_gnu_thread.a  /opt/conda/envs/py_3.12/lib/libmkl_core.a  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.a  -lm  /usr/lib/x86_64-linux-gnu/libdl.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so" -Wl,--as-needed  lib/libc10_hip.so  lib/libc10.so  /opt/rocm-7.2.4/lib/libMIOpen.so.1.0.70204  /opt/rocm/lib/libhiprtc.so.7.2.70204  -ldl  /opt/rocm-7.2.4/lib/libhipblas.so.3.2.70204  /opt/rocm-7.2.4/lib/libhipfft.so.0.1.70204  /opt/rocm-7.2.4/lib/libhiprand.so.1.1.70204  /opt/rocm-7.2.4/lib/librocrand.so.1.1.70204  /opt/rocm-7.2.4/lib/libhipsparse.so.4.2.0.70204  /opt/rocm-7.2.4/lib/libhipsolver.so.1.0.70204  /opt/rocm-7.2.4/lib/librocsolver.so.0.7.70204  /opt/rocm-7.2.4/lib/librocblas.so.5.2.70204  /opt/rocm/lib/libhipblaslt.so.1.2.70204  /opt/rocm/lib/libamdhip64.so.7.2.70204  /opt/rocm-7.2.4/lib/libhipsparselt.so.0.2.70204  lib/libgtest.a && /opt/conda/envs/py_3.12/lib/python3.12/site-packages/cmake/data/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/accelerator_graph_test && :
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `amdsmi_init'
collect2: error: ld returned 1 exit status
[8076/8176] Linking CXX executable bin/NamedTensor_test
FAILED: bin/NamedTensor_test

rocm-repo-management-api · 2026-06-14T18:51:51Z

Jenkins build for 9cf4004684351e9dad64cc3b329d8d7355cb710e commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

rocm-repo-management-api · 2026-06-15T02:21:27Z

Jenkins build for 7094c1a6333859e38c13263e84c6a5dcd11320e2 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

adam360x · 2026-06-15T13:16:45Z

Failing tests seem unrelated to PR

libtorch_hip.so referenced rsmi_init and rsmi_is_P2P_accessible as undefined symbols without listing librocm_smi64.so as a NEEDED dependency. When libamd_smi.so (which also exports these symbols for backward compatibility) was loaded with RTLD_GLOBAL, the dynamic linker interposed them over libamd_smi.so's internal copies. This caused AMDSMI's RSMI singleton to remain uninitialized, resulting in zero devices or sentinel values (e.g. gfxffffffffffffffff) when amdsmi_init() was called after torch. Replace all rsmi_* calls (rsmi_init, rsmi_is_P2P_accessible) with their AMDSMI equivalents (amdsmi_init, amdsmi_is_P2P_accessible), resolved at runtime via dlsym. This: - removes all rsmi_*/amdsmi_* undefined symbols from libtorch_hip.so - avoids any link-time NEEDED dependency on libamd_smi.so - allows libamd_smi.so to drop the rsmi_* exports entirely - removes the #include <rocm_smi/rocm_smi.h> dependency - gracefully degrades if libamd_smi.so is not loaded

rocm-repo-management-api · 2026-06-15T18:52:05Z

Jenkins build for edd012e63bcedf4261f65d44de95d87b8659a596 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

pragupta and others added 30 commits October 29, 2025 17:24

Add github workflows to automate IFU (#2688) (#2748)

b97cff1

(cherry picked from commit a66eeda) Fixes #ISSUE_NUMBER Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>

[rocm7.1_internal_testing] Add triton_kernels wheel generation (#2566)

6b3a141

Fixes #ISSUE_NUMBER (cherry picked from commit 0ea0592)

[AUTOGENERATED] [rocm6.5_internal_testing] Remove --no-index and --no…

d14e5a9

…-deps flags (#2121) Cherry-pick of #2103 Co-authored-by: Ethan Wee <Ethan.Wee@amd.com> (cherry picked from commit 1dea6e8)

Enable gesvda for ROCM >= 6.1 (#1339)

11ca2d0

This also fixes a problem in gesvd driver when UV is not needed. (cherry picked from commit 4ce57ec) (cherry picked from commit 167b4c1)

Remove ROCmloops specific test

629e824

(cherry picked from commit d6879fa) (cherry picked from commit 123a164)

Bump triton to 3.5.x and update related_commits

ab4714d

Revert to prev sccache by ROCm

2536631

Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> (cherry picked from commit 1ad5bb95d796283d5f56ac1edd16f1731d24a49d) (cherry picked from commit 519160d)

pytorch_ifu.yml: Change date format (#2776)

777e73c

Fixes #ISSUE_NUMBER

Merge remote-tracking branch 'upstream/main' into develop_IFU_20251104

223b9c5

# Conflicts: # .ci/docker/requirements-ci.txt

Fix merge conflict

b4c1e1e

Merge pull request #2784 from ROCm/develop_IFU_20251104

3d74218

[AUTOGENERATED] develop_IFU_20251104

Merge remote-tracking branch 'upstream/main' into develop_IFU_20251118

da5ac4a

# Conflicts: # .ci/docker/ci_commit_pins/triton.txt # requirements.txt

Fix conflicts and move triton ver to 3.5.0

a3c49a9

To keep triton version consistent with what is in rocm/triton's release/internal/3.5.x branch, we need to keep triton_version.txt at 3.5.0 and move triton hash to ToT of that branch.

Merge pull request #2812 from ROCm/develop_IFU_20251118

5ca076d

[AUTOGENERATED] develop_IFU_20251118

Merge remote-tracking branch 'upstream/main' into develop_IFU_20251124

ecdea86

Merge pull request #2827 from ROCm/develop_IFU_20251124

f742da3

[AUTOGENERATED] develop_IFU_20251124

Fix merge conflicts + bump triton to 3.6.x branch

7e17fb9

Remove stale opentelemetry-cpp submodule

4d67363

ethanwee1 and others added 13 commits April 23, 2026 09:35

Ensure one of pr_id/sha1 is provided to download_testlogs

f401954

Use default value (45 days for ROCm org) for artifact retention

7826f04

adam360x force-pushed the users/adam360x/fix-rsmi-init-interposition branch from df95aa5 to 3315c81 Compare June 14, 2026 17:07

adam360x force-pushed the users/adam360x/fix-rsmi-init-interposition branch from 3315c81 to 9cf4004 Compare June 14, 2026 18:43

adam360x force-pushed the users/adam360x/fix-rsmi-init-interposition branch from 9cf4004 to 7094c1a Compare June 15, 2026 02:05

adam360x changed the title ~~Replace legacy rsmi_* calls with AMDSMI API in intra_node_comm~~ Replace rsmi_init with amdsmi_init (via dlsym) in intra_node_comm Jun 15, 2026

adam360x force-pushed the users/adam360x/fix-rsmi-init-interposition branch from 7094c1a to edd012e Compare June 15, 2026 18:43

jithunnair-amd force-pushed the develop branch from ff872e3 to f27bbcf Compare June 29, 2026 20:57

jithunnair-amd requested review from jataylo, jeffdaily, jithunnair-amd and pruthvistony as code owners June 29, 2026 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace rsmi_init with amdsmi_init (via dlsym) in intra_node_comm#3299

Replace rsmi_init with amdsmi_init (via dlsym) in intra_node_comm#3299
adam360x wants to merge 62 commits into
developfrom
users/adam360x/fix-rsmi-init-interposition

adam360x commented Jun 14, 2026 •

edited

Loading

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

rocm-repo-management-api Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

adam360x commented Jun 15, 2026

Uh oh!

rocm-repo-management-api Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Uh oh!

Conversation

adam360x commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Root cause verification

Fix

Test results

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adam360x commented Jun 15, 2026

Uh oh!

rocm-repo-management-api Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

adam360x commented Jun 14, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Jun 14, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Jun 15, 2026 •

edited

Loading

rocm-repo-management-api Bot commented Jun 15, 2026 •

edited

Loading