[CI] Nightly: publish upstream pytorch CI image (built torch + source) to rocm/pytorch-private#3353
Open
ethanwee1 wants to merge 69 commits into
Open
[CI] Nightly: publish upstream pytorch CI image (built torch + source) to rocm/pytorch-private#3353ethanwee1 wants to merge 69 commits into
ethanwee1 wants to merge 69 commits into
Conversation
========================================== Triton build conditionalized on ROCM_VERSION Include the ROCm version in triton version (cherry picked from commit 7d33910) (cherry picked from commit 0412eb4) Update triton-rocm.txt to triton.txt (cherry picked from commit 0ce9f6e) Use ROCm/triton for install_triton.sh (cherry picked from commit 6e9714b) update triton commit Revert "Use ROCm/triton for install_triton.sh" This reverts commit 81b0cbc8435122030044049c661f252ee8aa7ae5. change triton repo Update triton.txt to use release/internal/3.3.x branch Use ROCm/triton Use ROCm/triton for install_triton.sh (cherry picked from commit 0036db5)
…on (#2482) Related to https://github.com/ROCm/builder/pull/90/files http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/305/ PyTorch wheel installs successfully when building torchvision/torchaudio (cherry picked from commit c1ee54d)
Fixes #ISSUE_NUMBER (cherry picked from commit 0ea0592)
…A helper functions ======================================================================================= Implementation of PyTorch ut parsing script - QA helper function (#1386) * Initial implementation of PyTorch ut parsing script * Extracted path variables * Use nested dict to save results * Fixes typo * Cleanup * Fixes several issues * Minor name change * Update run_pytorch_unit_tests.py * Added file banners * Supported running from API * Added more help info * Consistent naming * Format help text --------- Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> Co-authored-by: Jithun Nair <jithun.nair@amd.com> Print consolidated log file for pytorch unit test automation scripts (#1433) * Print consolidated log file for pytorch uts * Update run_entire_tests subprocess call as well * lint * Add ERROR string [SWDEV-466849] Enhancements for PyTorch UT helper scripts (#1491) * Check that >1 GPUs are visible when running TEST_CONFIG=distributed * Add EXECUTION_TIME to file-level and aggregate statistics PyTorch unit test helper scripts enhancements (#1517) * Fail earlier for distributed-on-1-GPU scenario * print cmd in consolidated log with prettier formatting * python->python3 Fixes https://ontrack-internal.amd.com/browse/SWDEV-477264 --------- Co-authored-by: blorange-amd <bo.li2@amd.com> Several issues fix of QA helper script (#1564) Fixes SWDEV-475071: https://ontrack-internal.amd.com/browse/SWDEV-475071 Removed args inside function (#1595) Fixes SWDEV-475071 (cherry picked from commit 041aa1b47978154de63edc6b7ffcdea218a847a3) QA script - Added multi gpu check with priority_tests (#1604) Fixes SWDEV-487907. Verified throwing exception for distributed is working correctly on single gpu with command: python .automation_scripts/run_pytorch_unit_tests.py --priority_test (cherry picked from commit 57cc742271cbf4547f9213710e57f6444bbc983e) (cherry picked from commit 6d5c3dc) (cherry picked from commit 2ee3aa2)
* Use triton commit same as that used for release/2.6 branch since both are triton version 3.2.0, so assuming they're compatible. Relates to: https://github.com/ROCm/rocAutomation/pull/660/files https://github.com/ROCm/builder/pull/70/files Validation http://ml-ci-internal.amd.com:8080/job/pytorch/job/manylinux_rocm_wheels/568/ --------- Co-authored-by: Jithun Nair <jithun.nair@amd.com> Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> (cherry picked from commit 14c1417) (cherry picked from commit c20a8f8)
* Add trailing comma for consistency in gfx architecture list Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> * ROCm: Enable tf32 testing on test_nn Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> --------- Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> (cherry picked from commit c113e14)
Relates to: ROCm/builder#82 Validation: http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/98/ Using `registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16180_ubuntu24.04_py3.12_pytorch_lw_rocm7.0_IT_upgrade_numpy_452f3df6`: ``` root@d92befdbb2a6:/# pip list | egrep "numpy|pandas" numpy 2.1.2 pandas 2.2.3 root@d92befdbb2a6:/# python3 Python 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pandas >>> import torch >>> import numpy >>> exit() root@d92befdbb2a6:/data/pytorch-micro-benchmarking# HIP_VISIBLE_DEVICES=1 python3 micro_benchmarking_pytorch.py --network resnet50 INFO: running forward and backward for warmup. INFO: running the benchmark.. OK: finished running benchmark.. --------------------SUMMARY-------------------------- Microbenchmark for network : resnet50 Num devices: 1 Dtype: FP32 Mini batch size [img] : 64 Time per mini-batch : 0.11369450092315674 Throughput [img/sec] : 562.9120096428937 ``` --------- Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> (cherry picked from commit cf32479)
…2269) Fixes SWDEV-536456 Fixes error post-#2256: ``` 00:12:44.248 #22 155.3 ERROR: Ignored the following versions that require a different python version: 0.52.0 Requires-Python >=3.6,<3.9; 0.52.0rc3 Requires-Python >=3.6,<3.9; 0.61.0 Requires-Python >=3.10; 0.61.0rc1 Requires-Python >=3.10; 0.61.0rc2 Requires-Python >=3.10; 0.61.1rc1 Requires-Python >=3.10; 0.61.2 Requires-Python >=3.10; 3.3 Requires-Python >=3.10; 3.3rc0 Requires-Python >=3.10; 3.4 Requires-Python >=3.10; 3.4.1 Requires-Python >=3.10; 3.4.2 Requires-Python >=3.10; 3.4rc0 Requires-Python >=3.10; 3.5 Requires-Python >=3.11; 3.5rc0 Requires-Python >=3.11; 8.2.0 Requires-Python >=3.10; 8.2.1 Requires-Python >=3.10 00:12:44.248 #22 155.3 ERROR: Could not find a version that satisfies the requirement numba==0.61.2 (from versions: 0.1, 0.2, 0.3, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.7.2, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.12.1, 0.12.2, 0.13.0, 0.13.2, 0.13.3, 0.13.4, 0.14.0, 0.15.1, 0.16.0, 0.17.0, 0.18.1, 0.18.2, 0.19.1, 0.19.2, 0.20.0, 0.21.0, 0.22.0, 0.22.1, 0.23.0, 0.23.1, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.1, 0.29.0, 0.30.0, 0.30.1, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.1, 0.36.2, 0.37.0, 0.38.0, 0.38.1, 0.39.0, 0.40.0, 0.40.1, 0.41.0, 0.42.0, 0.42.1, 0.43.0, 0.43.1, 0.44.0, 0.44.1, 0.45.0, 0.45.1, 0.46.0, 0.47.0, 0.48.0, 0.49.0, 0.49.1rc1, 0.49.1, 0.50.0rc1, 0.50.0, 0.50.1, 0.51.0rc1, 0.51.0, 0.51.1, 0.51.2, 0.52.0rc2, 0.53.0rc1.post1, 0.53.0rc2, 0.53.0rc3, 0.53.0, 0.53.1, 0.54.0rc2, 0.54.0rc3, 0.54.0, 0.54.1rc1, 0.54.1, 0.55.0rc1, 0.55.0, 0.55.1, 0.55.2, 0.56.0rc1, 0.56.0, 0.56.2, 0.56.3, 0.56.4, 0.57.0rc1, 0.57.0, 0.57.1rc1, 0.57.1, 0.58.0rc1, 0.58.0rc2, 0.58.0, 0.58.1, 0.59.0rc1, 0.59.0, 0.59.1, 0.60.0rc1, 0.60.0) 00:12:44.248 #22 155.3 ERROR: No matching distribution found for numba==0.61.2 ``` Validation: * Docker image: http://rocm-ci.amd.com/job/mainline-framework-pytorch-internal-cs9-ci/132 * Wheels: http://rocm-ci.amd.com/job/mainline-pytorch_internal-manylinux-wheels/102/ From `registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16180_ubuntu22.04_py3.9_pytorch_lw_rocm7.0_IT_py3.9_a11d94ad`: ``` root@f43861a0a856:/# pip list | egrep "numpy|pandas" numpy 2.0.2 pandas 2.2.3 root@f43861a0a856:/# python Python 3.9.23 (main, Jun 4 2025, 08:55:38) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> import numpy >>> import pandas root@f43861a0a856:/data/pytorch-micro-benchmarking# HIP_VISIBLE_DEVICES=1 python3 micro_benchmarking_pytorch.py --network resnet50 INFO: running forward and backward for warmup. INFO: running the benchmark.. OK: finished running benchmark.. --------------------SUMMARY-------------------------- Microbenchmark for network : resnet50 Num devices: 1 Dtype: FP32 Mini batch size [img] : 64 Time per mini-batch : 0.11354223489761353 Throughput [img/sec] : 563.6669038416574 ``` (cherry picked from commit a0a9d81)
…cm7.0/7.1 (#2239) Revamped version of #2108 PR to: - enable complex data types for sparse matmul on ROCm - fix sparse addmm/baddbmm on ROCm - fix sparse hipification for ROCm - fix/enable sparse tests on ROCm (~50 tests total for non-fp16/bf16): - enable fp16/bf16 sparse path for rocm7.0 - enable fp16/bf16 sparse tests for rocm7.0/7.1 ``` test_sparse_csr.py::TestSparseCSRCUDA::test_bmm_cuda_* test_sparse.py::TestSparseCUDA::test_sparse_matmul_cuda_* test_sparse_csr.py::TestSparseCSRCUDA::test_mm_cuda_float64 test_sparse_csr.py::TestSparseCSRCUDA::test_addmm_all_sparse_csr_SparseCS* test_sparse_csr.py::TestSparseCSRCUDA::test_addmm_sizes_all_sparse_csr_* test_sparse_csr.py::TestSparseCSRCUDA::test_sparse_addmm_cuda_float16 ``` (cherry picked from commit cc2a69c)
#2326) Fixes https://ontrack-internal.amd.com/browse/SWDEV-541809 Upgrading tensorboard after numpy upgrade Ran in **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16381_ubuntu24.04_py3.12_pytorch_lw_rocm7.0_internal_testing_afe8b782** ``` 7 git checkout rocm7.0_IT_upgrade_tensorboard 8 pip install .ci/docker/requirements-ci.txt 9 pip install -r .ci/docker/requirements-ci.txt 10 PYTORCH_TEST_WITH_ROCM=1 python test/test_monitor.py TestMonitorTensorboard.test_event_handler root@ubb4-rack-22:/var/lib/jenkins/pytorch# PYTORCH_TEST_WITH_ROCM=1 python test/test_monitor.py TestMonitorTensorboard.test_event_handler /opt/venv/lib/python3.12/site-packages/google/protobuf/internal/well_known_types.py:91: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). _EPOCH_DATETIME_NAIVE = datetime.datetime.utcfromtimestamp(0) . ---------------------------------------------------------------------- Ran 1 test in 0.327s OK root@ubb4-rack-22:/var/lib/jenkins/pytorch# ``` (cherry picked from commit c7f61f4)
Tested locally successfully ``` root@rocm-framework-47:/var/lib/jenkins/pytorch# pip install -r requirements.txt Ignoring numpy: markers 'python_version == "3.9"' don't match your environment Requirement already satisfied: setuptools<80.0,>=70.1.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 2)) (79.0.1) Requirement already satisfied: cmake>=3.31.4 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 3)) (4.0.0) Requirement already satisfied: ninja==1.11.1.3 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 4)) (1.11.1.3) Requirement already satisfied: numpy==2.1.2 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 5)) (2.1.2) Requirement already satisfied: packaging==25.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 6)) (25.0) Requirement already satisfied: pyyaml==6.0.2 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 7)) (6.0.2) Requirement already satisfied: requests==2.32.4 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2.32.4) Requirement already satisfied: six==1.17.0 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 9)) (1.17.0) Requirement already satisfied: typing-extensions==4.14.1 in /opt/venv/lib/python3.10/site-packages (from -r /var/lib/jenkins/pytorch/requirements-build.txt (line 10)) (4.14.1) Requirement already satisfied: expecttest==0.3.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (0.3.0) Requirement already satisfied: filelock==3.18.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (3.18.0) Requirement already satisfied: fsspec==2025.7.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (2025.7.0) Requirement already satisfied: hypothesis==5.35.1 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (5.35.1) Requirement already satisfied: jinja2==3.1.6 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (3.1.6) Requirement already satisfied: lintrunner==0.12.7 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (0.12.7) Requirement already satisfied: networkx==2.8.8 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (2.8.8) Requirement already satisfied: optree==0.13.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 18)) (0.13.0) Requirement already satisfied: psutil==7.0.0 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 19)) (7.0.0) Requirement already satisfied: sympy==1.13.3 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 20)) (1.13.3) Requirement already satisfied: wheel==0.45.1 in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 22)) (0.45.1) Requirement already satisfied: build[uv] in /opt/venv/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (1.3.0) Requirement already satisfied: charset_normalizer<4,>=2 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (3.4.3) Requirement already satisfied: idna<4,>=2.5 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2.5.0) Requirement already satisfied: certifi>=2017.4.17 in /opt/venv/lib/python3.10/site-packages (from requests==2.32.4->-r /var/lib/jenkins/pytorch/requirements-build.txt (line 8)) (2025.8.3) Requirement already satisfied: attrs>=19.2.0 in /opt/venv/lib/python3.10/site-packages (from hypothesis==5.35.1->-r requirements.txt (line 11)) (25.3.0) Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /opt/venv/lib/python3.10/site-packages (from hypothesis==5.35.1->-r requirements.txt (line 11)) (2.4.0) Requirement already satisfied: MarkupSafe>=2.0 in /opt/venv/lib/python3.10/site-packages (from jinja2==3.1.6->-r requirements.txt (line 12)) (3.0.2) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/venv/lib/python3.10/site-packages (from sympy==1.13.3->-r requirements.txt (line 20)) (1.3.0) Requirement already satisfied: pyproject_hooks in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (1.2.0) Requirement already satisfied: tomli>=1.1.0 in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (2.2.1) Requirement already satisfied: uv>=0.1.18 in /opt/venv/lib/python3.10/site-packages (from build[uv]->-r requirements.txt (line 7)) (0.8.10) root@rocm-framework-47:/var/lib/jenkins/pytorch# pip install -r requirements-build.txt ``` (cherry picked from commit 6e6e454)
Signed-off-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com> (cherry picked from commit 1ad5bb95d796283d5f56ac1edd16f1731d24a49d) (cherry picked from commit 519160d)
Fixes #ISSUE_NUMBER
- Need to use upstream/main for rocm/pytorch's develop branch. For release branches, `github.event.pull_request.base.ref` should work as is. - Need to remove any trailing space in PR TITTLE so branch name can be formed correctly Fixes #ISSUE_NUMBER
# Conflicts: # .ci/docker/requirements-ci.txt
[AUTOGENERATED] develop_IFU_20251104
# Conflicts: # .ci/docker/ci_commit_pins/triton.txt # requirements.txt
To keep triton version consistent with what is in rocm/triton's release/internal/3.5.x branch, we need to keep triton_version.txt at 3.5.0 and move triton hash to ToT of that branch.
[AUTOGENERATED] develop_IFU_20251118
[AUTOGENERATED] develop_IFU_20251124
# Conflicts: # .ci/docker/ci_commit_pins/triton.txt # .ci/docker/requirements-ci.txt # .ci/docker/triton_version.txt # .circleci/scripts/binary_populate_env.sh # .github/scripts/build_triton_wheel.py # test/test_sparse_csr.py
## Motivation Old IFU_GITHUB_TOKEN [seems to have expired](https://github.com/ROCm/pytorch/actions/runs/25856299592/job/75974982737) ## Technical Details Replace with PARITY_GITHUB_TOKEN (meant specifically for this workflow) ## Test Plan Run parity.yml with this PR branch and see if it still gives credential error. ## Test Result "Download artifacts" step succeeded in https://github.com/ROCm/pytorch/actions/runs/25857211908/job/75978008711 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
## Summary - Select the CUDA test artifact kind from the jobs present for the target SHA. - Detect whether the target SHA uses test-osdc or legacy test CUDA jobs, then use the detected kind when building log keys and artifact prefixes. - Apply the same dynamic selection to CUDA inductor jobs. - Treat missing per-arch summary buckets as zero so mixed ROCm/CUDA coverage does not crash report generation. ## Validation - PR/ciflow case: dispatched `Parity Report` on this branch with `sha=386f38175e3aaee2dadb36b5c364deff0869664d` and `arch=mi355, mi300, mi200, navi31`. CUDA default/distributed and inductor selected `test`. - Run: https://github.com/ROCm/pytorch/actions/runs/25866762885 - Main branch case: dispatched `Parity Report` on this branch with `sha=f38b1ec280bafa2ad11f6e767558e73e9eb508a6`, `arch=mi300`, `skip_rocm=true`, and `exclude_distributed=true`. CUDA default and inductor selected `test-osdc`. - Run: https://github.com/ROCm/pytorch/actions/runs/25867046276 - Local syntax check: `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogs .automation_scripts/pytorch-unit-test-scripts/generate_summary.py`.
## Summary - Prefer the arch-specific MI200 workflows in `download_testlogs`: `rocm-mi200`, `periodic-rocm-mi200`, and `inductor-rocm-mi200`. - Match arch-specific MI200 test jobs with the `linux-jammy-rocm-py3.10-mi200` prefix for default, distributed, and inductor shards. - Keep `trunk-rocm-sandbox` as the fallback workflow for older SHAs that do not have the MI200-specific workflows, using the legacy `linux-jammy-rocm-py3.10` prefix in that fallback path. ## Motivation A parity run for `50d07a990e33f9822ae4d48bed2d7f06c96522d0` tried to collect MI200 distributed jobs with: `linux-jammy-rocm-py3.10 / test (distributed, ...)` The upstream jobs for this SHA are arch-specific and include `-mi200`, so the log lookup missed all three shards and XML artifact collection fell through to empty results. The script should look for the MI200-specific workflows first, then fall back to `trunk-rocm-sandbox` for older commits. ## Validation - `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/download_testlogs` - Confirmed the fixed prefix matches upstream jobs for `50d07a990e33f9822ae4d48bed2d7f06c96522d0`: - `rocm-mi200`: 6 default shard matches - `periodic-rocm-mi200`: 3 distributed shard matches - `inductor-rocm-mi200`: 2 inductor shard matches - Dispatched `Parity Report` on this branch with `sha=50d07a990e33f9822ae4d48bed2d7f06c96522d0`, `arch=mi200`, and `skip_cuda=true` to validate collection end-to-end. - Initial run before fallback commit: https://github.com/ROCm/pytorch/actions/runs/25920564353 (success) - Current branch run after fallback commit: https://github.com/ROCm/pytorch/actions/runs/25920808611 (queued) Made with [Cursor](https://cursor.com)
## Summary - Raise the Python CSV parser field limit in `generate_summary.py` so large parity CSV diagnostic fields can be read. - Truncate oversized diagnostic text fields while loading rows so long failure/skip messages do not make summary generation or output unwieldy. - Preserve test identity, status, timing, and shard fields used by the parity report tables. ## Root Cause A parity run failed in the `summarize` job when Python's default CSV field limit rejected a generated-code assertion message larger than 131,072 bytes: https://github.com/ROCm/pytorch/actions/runs/26168276671/job/76979094769 The first offending row was `inductor.test_torchinductor_codegen_dynamic_shapes::DynamicShapesCodegenGPUTests::test_vmap_dot_decomposes_bmm_dynamic_shapes_cuda`, where `message_rocm` was 145,748 bytes. ## Test plan - `python3 -m py_compile .automation_scripts/pytorch-unit-test-scripts/generate_summary.py` - Re-ran `generate_summary.py` locally against the artifact from the failed run: - Input: `20260520_all_tests_status_mi355.csv` from run `26168276671` - Output: summary CSV and markdown generated successfully instead of failing with `_csv.Error: field larger than field limit (131072)`. - Triggered `parity.yml` on this branch with the same upstream commit and arch as the failing run: - SHA: `27f2e80e30fb950bc455c777a5e8079e9657a157` - Arch: `mi355` - Validation run: https://github.com/ROCm/pytorch/actions/runs/26175417191 - Result: `setup-matrix`, `generate-parity (mi355)`, and `summarize` all completed successfully. - The summarize log shows `CSV written to 27f2e80e30fb950bc455c777a5e8079e9657a157_summary.csv` and `Markdown written to 27f2e80e30fb950bc455c777a5e8079e9657a157_summary.md`.
## Summary
Adds a single step to the `summarize` job in `parity.yml` that uploads
the generated `*_summary.md` (the same content already appended to
`$GITHUB_STEP_SUMMARY`) as a standalone artifact named
`parity-summary-md`, with N-day retention.
The existing per-arch result artifacts have a 1-day retention, which
makes it impossible to recover the summary content (e.g. `### FAILED
TESTS`, `### LOG-BASED FAILURES`) after that window. This change lets
external tooling — for example the in-progress upstream CI failure
tracking — fetch the exact UI summary via `gh run download` long after
the CSVs are gone, with only a standard PAT.
No behavior change for any existing job. `if-no-files-found: ignore`
keeps the step a no-op on early-exit runs (no CSVs produced).
## Test plan
- [ ] Re-run `parity.yml` (or an autoparity manual dispatch) and verify
the `parity-summary-md` artifact appears alongside the per-arch
results artifacts.
- [ ] `gh run download <run_id> -R ROCm/pytorch -n parity-summary-md`
returns the expected `*_summary.md`.
- [ ] On a run with no CSVs (forced early exit), confirm the workflow
still succeeds and no artifact is uploaded.
Signed-off-by: Garay-Fernandez <pgarayfe@amd.com>
## Summary - Adds a clickable **Job ID** column at the end of both the `FAILED TESTS` and `LOG-BASED FAILURES (not in XML)` tables in the parity summary markdown. Each cell renders as `[<job_id>](https://github.com/pytorch/pytorch/actions/runs/<wf>/job/<job_id>)`, dropping the reviewer one click away from the stacktrace. - Threads the upstream `pytorch/pytorch` CI job url through the existing pipeline — `download_testlogs` was already fetching that info, it just wasn't being preserved. No new API calls; no schema migrations; just persistence through `download_testlogs` → `summarize_xml_testreports.py` / `detect_log_failures.py` → `generate_summary.py`. - Backwards-compatible: every consumer reads the new fields via `.get(..., '')` / `os.path.isfile`, so older artifacts and CSVs render the column as empty cells instead of breaking. ### Example resulting row (FAILED TESTS, set2-disabled case) ``` | Arch | Test Config | Test File | Test Class | Test Name | Job-Level Shard (rocm) | Test-Level Shard (rocm) | Status (rocm) | Also Failing In | Job ID (rocm) | | mi300 | default | test_foo | TestBar | test_baz | 3/6 | 5/15 | FAILED | mi355 | [76905282313](https://github.com/pytorch/pytorch/actions/runs/26146653222/job/76905282313) | ``` ### Data flow - **FAILED TESTS** (XML-based): `_shorten_unzipped_dirs` keeps the trailing `_<jobid>` of the artifact name on each `test-<cfg>-N-N/` dir → `download_xml_files` writes one `_wf_run_id` file at the parent → `parse_xml_reports_as_dict` builds the url and stamps it on each test case → per-arch CSV carries `job_url_{set_name}` → `collect_failed_tests` propagates → markdown renders. - **LOG-BASED FAILURES**: `write_test_log_to_file` writes a companion `<filename>.job_url` file (full url from the job's `html_url`) → `scan_logs` reads it and stamps `job_url` on every failure / flaky row → `log_failures_<arch>.csv` / `flaky_tests_<arch>.csv` carry it → `load_log_failures` / `load_flaky_tests_as_log_failures` propagate → markdown renders. ## Test plan - Trigger a `parity.yml` run and confirm: - Per-arch test-report shard dirs are named `test-<cfg>-N-N_<jobid>` after `_shorten_unzipped_dirs`. - `_wf_run_id` file exists alongside the shard dirs in `rocm_xml/` and `cuda_xml/`. - `<filename>.job_url` companion files exist next to each `rocm*.txt` / `cuda*.txt` log file. - Inspect the per-arch CSV emitted by `summarize_xml_testreports.py` and confirm `job_url_<set1_name>` / `job_url_<set2_name>` columns are populated for failing rows. - Inspect `log_failures_<arch>.csv` / `flaky_tests_<arch>.csv` and confirm `job_url` column is populated. - Inspect the parity summary markdown artifact and click a `Job ID` cell in both tables → lands on the failing pytorch/pytorch job page with the stacktrace. - Re-run against a historical commit whose artifacts predate this change and confirm cells render as empty (no crash, no broken table). --------- Signed-off-by: Garay-Fernandez <pgarayfe@amd.com>
## Summary Add a clickable `HUD LINK` to the parity report summary so users can jump from the GitHub Actions run to the matching PyTorch HUD page. For PR runs, the link points to the PR HUD page with the exact SHA used for the report, e.g. `https://hud.pytorch.org/pytorch/pytorch/pull/<pr_id>?sha=<sha>`. For SHA-only runs, the link points to the commit HUD page. ## Validation - PR-only example: triggered `parity.yml` with `pr_id=184377`, `arch=mi355`, and no SHA. The report resolved SHA `291ff45ffe10a301a88d1a83e98b9ba9987dbbfa`, so the HUD link is `https://hud.pytorch.org/pytorch/pytorch/pull/184377?sha=291ff45ffe10a301a88d1a83e98b9ba9987dbbfa`. Run passed: https://github.com/ROCm/pytorch/actions/runs/26476256833 - SHA-only example: triggered `parity.yml` with `sha=fe1b0a2ae93e0efcfa0defeee2ed879cf68eaac6`, `arch=mi355`, and no PR ID. Run passed: https://github.com/ROCm/pytorch/actions/runs/26475135922
## Summary - Adds a `Run Time (s)` column to both parity summary tables (FAILED TESTS and LOG-BASED FAILURES), in the `.md` and `.csv` outputs. - **FAILED** tests use the per-test JUnit XML `time` (already in the per-arch CSV as `running_time_<set>`). - **LOG-BASED** failures (timeouts/crashes/kills, which produce no XML) use the failing **job's wall-clock**, computed in `detect_log_failures.py` from each log's first-to-last ISO timestamps and attached to every failure/flaky entry. Implements ROCm/frameworks-internal#16856. ## Changes - `detect_log_failures.py`: compute per-log job run time; add `run_time` to the failures and flaky CSV reports. - `generate_summary.py`: `Run Time (s)` column in FAILED + LOG-BASED tables; thread run time through the flaky loader. ## Test plan - [x] Offline end-to-end: synthetic timeout log -> `detect_log_failures` (1801s) -> `generate_summary` LOG-BASED table -> downstream parser - [x] FAILED table run time verified with a synthetic per-arch CSV - [x] Verify on a real parity run --------- Signed-off-by: pablo-garay <pgarayfe@amd.com>
…ruth) (#3311) ## What Introduces `.automation_scripts/pytorch-unit-test-scripts/parity_job_config.json` — a single source of truth for `pytorch/pytorch` job-name matching used by the parity tooling. Per-arch upstream workflow names, job-name prefixes, shard counts, artifact substrings, fallbacks (plus the CUDA equivalents and the check-run / workflow gating regexes) were previously duplicated: - hardcoded as large dicts inside `download_testlogs`, and - independently re-encoded as regexes inside `parity-auto.yml`. This PR lands just the config file so the matching rules live in exactly one place. ## Why split this out Per review feedback on #3278 (separate concerns, and this file also being introduced/changed in #3231), the shared config is pulled into its own foundational PR: - **#3278** (download_testlogs re-run fixes) stacks on this and *consumes* the config for S3 artifact / log matching. - **#3231** (parity auto-trigger) stacks on this and reads the check-run / workflow regexes for upstream gating. Neither downstream PR carries its own copy of the file anymore — no more add/add duplication. ## Merge order Land this first. #3278 and #3231 are rebased onto it and drop their copies of `parity_job_config.json`. ## Validation This is a data-only file; it is exercised end-to-end by the downloader in #3278, which loads this exact config. A stacked #3278 parity run for `346976bc` (`mi350`) reads this file over the new path and produces a fully-populated report: - https://github.com/ROCm/pytorch/actions/runs/27716114811 — passed Made with [Cursor](https://cursor.com)
…m/pytorch-private
|
Jenkins build for 57c0e74ea7e20ea8d131e540248e589ab15ac756 commit finished as NOT_BUILT |
|
Jenkins build for 52480d283ece84ca1ce2499cd67b5a9b631512c5 commit finished as NOT_BUILT |
|
Jenkins build for 52480d283ece84ca1ce2499cd67b5a9b631512c5 commit finished as FAILURE |
|
Jenkins build for d6d638c8a1e652faaebeaae82eb7bcacfe1170fa commit finished as FAILURE |
jithunnair-amd
left a comment
jithunnair-amd
left a comment
Collaborator
There was a problem hiding this comment.
@ethanwee1 I was hoping to port @amdfaa 's scripts to ROCm/pytorch and then use them in this workflow, instead of rehashing a lot of the same logic. From your PR description, it seems that should be feasible?
Also left some comments for specific aspects.
| sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /usr/local/.ghcup \ | ||
| /opt/hostedtoolcache/CodeQL "$AGENT_TOOLSDIRECTORY" || true | ||
| sudo docker image prune -af || true | ||
| df -h / /mnt |
Collaborator
There was a problem hiding this comment.
Is this really necessary? Looking at the example workflows mentioned in the PR description:
Total reclaimed space: 1.832GB
Filesystem Size Used Avail Use% Mounted on
/dev/root 145G 30G 116G 21% /
/dev/root 145G 30G 116G 21% /
So the diskspace cleanup only gained 1.8GB, even without which we would have sufficient diskspace to handle a 24GB CI docker image and more...
| echo '{"data-root":"/mnt/docker"}' | sudo tee /etc/docker/daemon.json | ||
| sudo systemctl start docker | ||
| docker info --format 'Docker Root Dir: {{.DockerRootDir}}' | ||
| df -h / /mnt |
Collaborator
There was a problem hiding this comment.
Same question as for previous step: is this really necessary?
| UPSTREAM_REPO: pytorch/pytorch | ||
| UPSTREAM_BRANCH: main | ||
| REGISTRY: docker.io | ||
| IMAGE_NAME: rocm/pytorch-private |
Collaborator
There was a problem hiding this comment.
Suggested change
| IMAGE_NAME: rocm/pytorch-private | |
| IMAGE_NAME: rocm/pytorch-nightly |
We want this workflow to revive the docker images that were being pushed to rocm/pytorch-nightly
|
Jenkins build for 3a30693b2fa228b05f5e01cde56e28c110042af8 commit finished as FAILURE |
jithunnair-amd
requested review from
jataylo,
jeffdaily and
pruthvistony
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a nightly workflow that publishes a Docker image to
docker.io/rocm/pytorch-nightlyequal to the upstreampytorch/pytorchROCm CI image at the latest trunk commit, with the CI-builttorchwheel installed and the source checked out at that commit.This also ports @amdfaa's PyTorch ROCm CI debug helper into
ROCm/pytorchat.github/scripts/pytorch_rocm_ci_debug_env.py, then has the workflow call that helper instead of carrying the resolver/install logic inline.How
No upstream credentials are needed — both upstream artifacts are public:
pytorch/pytorchmaincommit with a successful run of the source ROCm workflow (defaulttrunk.yml) and a presentartifacts.zip.docker-image-name,build-environment,.ci/dockertree sha, GHCR CI image ref, and public S3 artifact URL.pytorch/pytorchat the resolved commit into/var/lib/jenkins/pytorch, downloads/unzipsartifacts.zip, and installsdist/*.whl.rocm/pytorch-nightly:<push_tag>plus a datednightly-<date>-<shortsha>tag.The workflow is now intentionally thin: checkout scripts, install PyYAML, Docker login, and call:
Triggers & config
schedule: daily at 07:00 UTC.workflow_dispatchinputs:source_workflow(defaulttrunk.yml),commit(default: auto),push_tag(defaultnightly).secrets.DOCKERTOKEN || secrets.DCKRPATwithsecrets.DOCKERUSERNAME./mntdata-root move steps; the current hosted runner has enough root disk in the review examples.Validation
Validated end-to-end on
ethanwee1/pytorchActions (workflow_dispatch, run28110203714) using the ported helper: resolve → anonymous GHCR pull → container → in-container clone + wheel install →docker commit→ login → push succeeded.Resolved/pushed:
c004e2dc8082e016df1893e3f42a831ea3f12c75ghcr.io/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-eef9f7a31fd78532d8ccb7e86581665f90e84747linux-jammy-rocm-py3.10-mi350nightly-forktest-ported,nightly-20260624-c004e2dsha256:4c2751eb818609ce1d7aab1cc685ab54c07ad3b82ea325fe797546b1c802561e