Trusted-publishing or twine

[andyfaff]

At the moment scipy uploads nightly wheels using anaconda-client. I was wondering if we could start using either the trusted-publishing approach, like that used for releases to pypi, or twine instead?
At the moment we're transitioning to a separate scipy-release repo and our current policy is fairly strict on the actions we'll use in it. It would be great if we could use the trusted publishing approach because then we can simply divert the wheel to either pypi (for a release) or to nightly wheels.

[agriyakhetarpal]

I think Anaconda’s platform doesn't yet have native OIDC integration for publishing to Anaconda-hosted indices, and this would also need support in anaconda-client. It's not listed here either: https://repos.openssf.org/trusted-publishers-for-all-package-repositories.html#support.

Same for Twine. The index follows PEP 503 for downloads, but if I remember correctly, the upload API at api.anaconda.org/upload is proprietary and not PyPI-like; so anaconda-client is required to upload to it and there's no /legacy/ upload endpoint.

I guess the best security measures one can take right now would be the usual: keep tokens fairly short-lived and regularly rotate them (but even that wouldn't mimic the short-livedness of tokens generated by a Trusted Publishing provider), and use a GitHub Environment to guard secrets and the workflows that can use them.

[matthewfeickert]

Yeah, I think for the reasons that @agriyakhetarpal mentioned plus all of those discussed in pypi/warehouse#15838 that this won't be viable unless Anaconda, Inc. and the PyPI security team worked together to figure out how Anaconda.org could become a verified platform.

[andyfaff]

guess the best security measures one can take right now would be the usual: keep tokens fairly short-lived and regularly rotate them (but even that wouldn't mimic the short-livedness of tokens generated by a Trusted Publishing provider), and use a GitHub Environment to guard secrets and the workflows that can use them.

It's more to simplify the process. Rather than using two different approaches for PyPI release and nightlies, just use the one.
One personal thought I had was to investigate the use of testpypi for nightlies, that would be a single approach. However, that would be too disruptive.

One recent drawback with the anaconda/anaconda-client approach is due to the wheels we make for win_arm64.
We use micromamba+anaconda to upload everything else, but pip installed anaconda-client for win_arm64. This is because micromamba doesn't support win_arm64.
anaconda-client requires the use of the cryptography package, which has just stopped releasing wheels for the win_arm64 platform for various reasons. We thus have to pin cryptography to an older version.

[di]

See https://docs.pypi.org/trusted-publishers/internals/#how-do-i-become-a-trusted-publishing-provider for details on adding another Trusted Publishing provider.

It's not clear to me how pypi/warehouse#15838 is related (is the "Anaconda platform" a GitLab instance?)

[agriyakhetarpal]

guess the best security measures one can take right now would be the usual: keep tokens fairly short-lived and regularly rotate them (but even that wouldn't mimic the short-livedness of tokens generated by a Trusted Publishing provider), and use a GitHub Environment to guard secrets and the workflows that can use them.

It's more to simplify the process. Rather than using two different approaches for PyPI release and nightlies, just use the one. One personal thought I had was to investigate the use of testpypi for nightlies, that would be a single approach. However, that would be too disruptive.

One recent drawback with the anaconda/anaconda-client approach is due to the wheels we make for win_arm64. We use micromamba+anaconda to upload everything else, but pip installed anaconda-client for win_arm64. This is because micromamba doesn't support win_arm64. anaconda-client requires the use of the cryptography package, which has just stopped releasing wheels for the win_arm64 platform for various reasons. We thus have to pin cryptography to an older version.

@andyfaff hmm, would it be possible for you to split the win_arm64 build component and the upload component, such that the upload step downloads the built win_arm64 wheels and a Linux runner with a micromamba-installed anaconda-client uploads them? That would circumvent the cryptography wheel requirement. In general, keeping builds and uploads separate is a good strategy, of course.

I would indeed not recommend using TestPyPI at this time, since they make no guarantees about the perenniality of what gets uploaded to the index (see the note under https://packaging.python.org/en/latest/guides/using-testpypi/#registering-your-account and this comment pypi/warehouse#15226 (comment)).

Yeah, I think for the reasons that @agriyakhetarpal mentioned plus all of those discussed in pypi/warehouse#15838 that this won't be viable unless Anaconda, Inc. and the PyPI security team worked together to figure out how Anaconda.org could become a verified platform.

See docs.pypi.org/trusted-publishers/internals#how-do-i-become-a-trusted-publishing-provider for details on adding another Trusted Publishing provider.

It's not clear to me how pypi/warehouse#15838 is related (is the "Anaconda platform" a GitLab instance?)

@matthewfeickert @di I believe this is a different discussion from PyPI Trusted Publishers and it's about trusted publishing in general. Since Anaconda.org is a package repository/index separate from PyPI, and it is not a code forge or CI provider for building and uploading wheels to PyPI, the conversation is about the need for the Anaconda.org platform (and its developers), i.e., the combination of their index/backend server and their client, to implement OIDC support that can be linked with OIDC identity providers, enabling them to support Trusted Publishing in a manner similar to PyPI, npm, and other package repositories have done (as linked in https://repos.openssf.org/trusted-publishers-for-all-package-repositories.html#support that I mentioned above).

[andyfaff]

An update on this. scipy-release took the route on uploading all the wheels in a single job, which simplifies things considerably.

[agriyakhetarpal]

This still needs support from the Anaconda side; that would be OIDC support in the index, and in anaconda-client, which would interact with said index (and the former would also need a way to register trusted publishers and mint short-lived tokens).

One option, also outside the scope of upload-nightly-action here, could be for channels on prefix.dev to support wheels just like Anaconda.org indices do. There is already trusted publishing via pixi publish for conda packages (see https://pixi.prefix.dev/latest/deployment/prefix/#trusted-publishing), so it would be nice if wheel uploads were also supported, since that would further strengthen the case for trusted publishing for those wheel uploads.