Skip to content

Add multi-arch (flat) upload mode for Python packages and tarballs#5058

Open
araravik-psd wants to merge 4 commits intomainfrom
users/arravikum/upload_multi_arch
Open

Add multi-arch (flat) upload mode for Python packages and tarballs#5058
araravik-psd wants to merge 4 commits intomainfrom
users/arravikum/upload_multi_arch

Conversation

@araravik-psd
Copy link
Copy Markdown
Contributor

This PR mainly adds

  • Add --multi-arch flag to support flat S3 upload layout
  • Apply layout changes to both Python packages and tarballs via prefix selection
  • Preserve existing arch-based behavior by default
  1. Multi-arch upload mode (flat layout)

Introduced --multi-arch flag
Changes S3 key construction:

Default:

<prefix>/<arch>/<package>

Multi-arch:

<prefix>/<package>

  1. Prefix selection for both wheels and tarballs

Updated --use-release-buckets handling:
Default → v3/whl/, v3/tarball/
Multi-arch → v4/whl/, v4/tarball/
Ensures both Python packages and tarballs are uploaded to the correct layout

  1. Tarball uploads

Upload logic remains flat:

<prefix>/<tarball>

Destination changes based on selected prefix (standard vs multi-arch)

Testing:

https://gist.github.com/araravik-psd/904784e6f08af6c1fd79aab6b5d16da5

Copy link
Copy Markdown
Member

@marbre marbre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving as I assume the functionality has been validated manually. @HereThereBeDragons please look more closely.

Solely looking at the changes, the functionality added seems mostly covered with build_tools/github_actions/publish_rocm_to_release_buckets.py. It should at least be possible to reuse code _therock_utils.storage_location and _therock_utils.storage_backend in the script. This script is also lacking tests and I am only approving as this is time sensitive. Before landing I expect to

  1. Evaluate if reusing the functions pointed to above is reasonable. If it is, file an issue for a future refactoring.
  2. If it is not, file an issue to add tests specifically as this is lacking any test so far.

Issues should be linked and PR description can be tidied up a bit further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: TODO

Development

Successfully merging this pull request may close these issues.

3 participants