[BUG] Runners starting very slowly (Power and Z)

[alex]

In the last day or so, we're seeing ppc64le actions runners being very slow to start (many hours). Example jobs:

• https://github.com/pyca/infra/actions/runs/22802613974
• https://github.com/pyca/cryptography/actions/runs/22806809243/job/66157163863
• https://github.com/pyca/cryptography/actions/runs/22808533614/job/66161522697

[anup-kodlekere]

@alex over the past few days we've encountered payload delivery issues from GitHub due to their recent incidents with webhook and actions services as seen on their status page

i see that the workflow runs mentioned in the issue are more recent, which could suggest we had high utilization from previous runs due to queue-buildup.

we will monitor this situation closely and post an update here.

[anup-kodlekere]

I believe the service is fully operational again and the runner provisioning time should be back down. Let us know if you're still facing this issue or if we're good to close this.

[alex]

Times today have looked better, thanks.

[anup-kodlekere]

Great! We'll keep this issue open through this week, just in case. Thank you for your patience!

[pmur]

I think I am running into something similar? It's been about 12 hours, and no jobs have run on libdfp/libdfp#278.

[rahulssv-ibm]

Hi @pmur,
Has your organization made the changes mentioned in this doc?https://github.com/IBM/actionspz/blob/main/docs/onboarding.md#3-configure-organization--repository-settings

If these changes aren't made, the jobs won't be picked up. We can see your jobs reaching our servers and getting configured, but they are being blocked by the permissions required for public repos to run jobs.

Can you please confirm if this has been set up? Please note that only org admins can change this permission.

[pmur]

Can you please confirm if this has been set up? Please note that only org admins can change this permission.

Ah, the organizational level option wasn't set. Thanks. I must have missed that change. It's running again, though the s390x runner seems backed up.

[dale-fu]

Are you able to restart any existing jobs prior to the changes? Hopefully that will help.

[alex]

Follow up here, this morning we're seeing jobs taking >1hr to start, e.g.:

• https://github.com/pyca/cryptography/actions/runs/23289823056/job/67722359295
• https://github.com/pyca/cryptography/actions/runs/23290244025/job/67723712451
• https://github.com/pyca/cryptography/actions/runs/23292322855/job/67730796513
• https://github.com/pyca/cryptography/actions/runs/23292544670/job/67731574055

[Fryguy]

Back to slow again ☹️

• https://github.com/IBM/actions-ibmcloud-cli/actions/runs/23302049046/job/67765579413?pr=75
• https://github.com/IBM/actions-ibmcloud-cli/actions/runs/23302049046/job/67765579479?pr=75

[mtarsel]

Thanks for bringing this up! We are aware of this issue and have a temporary fix that should be rolled out in production very soon

@arunchakraborty @dale-fu please comment here once the service is restarted