Remove Python index generation(manage.py) from CI as it is moved over to S3-triggered Lambda#4900
Open
araravik-psd wants to merge 3 commits intomainfrom
Open
Remove Python index generation(manage.py) from CI as it is moved over to S3-triggered Lambda#4900araravik-psd wants to merge 3 commits intomainfrom
araravik-psd wants to merge 3 commits intomainfrom
Conversation
HereThereBeDragons
approved these changes
Apr 28, 2026
Contributor
HereThereBeDragons
left a comment
HereThereBeDragons
left a comment
There was a problem hiding this comment.
you might want to update the readmes in
- build_tools/third_party/s3_management/README.md
- build_tools/packaging/how_to_do_release.md
| run: | | ||
| pip install boto3 packaging | ||
| python ./build_tools/third_party/s3_management/manage.py ${CUSTOM_PREFIX} | ||
| --recursive --exclude "*" --include "*torch*${{ inputs.rocm_version }}*${{ env.cp_version }}*" No newline at end of file |
Contributor
There was a problem hiding this comment.
new line missing?
Contributor
Author
There was a problem hiding this comment.
Running the pre-commit script seems to be removing the new line at the end of the file
marbre
reviewed
Apr 28, 2026
Comment on lines
-303
to
-309
| - name: (Re-)Generate Python package release index for staging | ||
| if: ${{ github.repository_owner == 'ROCm' }} | ||
| env: | ||
| # Environment variables to be set for `manage.py` | ||
| CUSTOM_PREFIX: "${{ inputs.s3_staging_subdir }}/${{ inputs.amdgpu_family }}" | ||
| run: | | ||
| python ./build_tools/third_party/s3_management/manage.py ${{ env.CUSTOM_PREFIX }} |
Member
There was a problem hiding this comment.
We recently hit a CI failure where the index wasn't yet up to date (as we queue requests before generating server side). Eventually we should add a small sleep 45s or so here. @ScottTodd wdyt?
Member
There was a problem hiding this comment.
I'd rather not introduce a fixed artificial delay in workflows. Consider a few options and their tradeoffs:
- In build jobs: upload packages in a build job then wait
- Test jobs can trust the package index to already exist
- Test jobs can't start running/queuing until that wait completes
- In test jobs: attempt to install, retry/wait if failed or index does not yet exist
- Each test jobs needs to know that the index might not yet exist
- Test jobs can start running/queuing immediately
| run: | | ||
| pip install boto3 packaging | ||
| python ./build_tools/third_party/s3_management/manage.py ${CUSTOM_PREFIX} | ||
| --recursive --exclude "*" --include "*torch*${{ inputs.rocm_version }}*${{ env.cp_version }}*" No newline at end of file |
Comment on lines
-297
to
-304
| - name: (Re-)Generate Python package release index for staging | ||
| if: ${{ github.repository_owner == 'ROCm' }} | ||
| env: | ||
| # Environment variable to be set for `manage.py` | ||
| CUSTOM_PREFIX: "${{ env.S3_STAGING_SUBDIR }}/${{ matrix.target_bundle.amdgpu_family }}" | ||
| run: | | ||
| pip install boto3 packaging | ||
| python ./build_tools/third_party/s3_management/manage.py ${{ env.CUSTOM_PREFIX }} |
Member
|
@araravik-psd Please rebase to mitigate the conflicts and fix the pre-commit errors. This shouldn't sit so long. |
araravik-psd
force-pushed
the
users/arravikum/manage-filtering
branch
from
3603f3b to
4188d02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes manual index regeneration from CI workflows and shifts it to an S3 event-driven Lambda that invokes manage.py on object create/delete.
Key Changes:
Eliminates duplicate index generation across workflows
Avoids race conditions on shared prefixes
Removes CI dependency setup (boto3, packaging)
Centralizes and simplifies index updates
Scope
Covers both staging and release prefixes. Presently deployed in dev, nightly and prereleases for python buckets.
References
Resolves: #3321
Manage.py Refactor PR: #3737