Attempt to fix CI timeouts#1811
Conversation
For some reason, calls to skopeo copy sometimes hang indefinitely. Avoid this issue by specifying --command-timeout with reasonable values (longer for call which must fetch over the network). Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces timeouts to various skopeo copy commands and adds a retry loop to wait for the Docker registry to become ready before logging in, aiming to prevent CI hangs. The reviewer suggested wrapping the openssl s_client readiness check with a timeout command to prevent the check itself from hanging indefinitely if the registry is unresponsive during the TLS handshake.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…oceeding To try and avoid potential races, ensure that the local docker registry is ready before finishing the setup. Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
|
It does seem like an issue on the registry or maybe the node side of things. Updating the registry image to Test passes fine locally: $ bats --filter "custom image that needs an authentication file" test/system/101-create.bats
test suite: Set up
101-create.bats
✓ create: With a custom image that needs an authentication file
test suite: Tear down
1 test, 0 failuresEDIT: seems like the test was only failing due to a short timeout, removed that commit and looks like we're good now. I guess I need your thoughts on the registry image, and maybe you know who to contact about the rawhide node failures? @debarshiray |
Fallout from the following: * 61e2c97 * d5daa71 * ecd1ced * 7a5f3ba containers#1811 containers#1815
For some reason, calls to 'skopeo copy' to populate the local temporary Docker registry are sometimes hanging indefinitely. Everything was fine on the 28th of May [1]. The problem started with Fedora Rawhide on the 30th of May [2], and spread to Fedora 44 on the 4th of June [3]. Avoid the indefinite hangs and make skopeo(1) fail sooner by using reasonable timeouts - longer for remote network I/O and shorter for local. [1] containers#1799 [2] containers#1798 [3] containers#1769 containers#1811 containers#1815 Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
debarshiray
left a comment
debarshiray
left a comment
There was a problem hiding this comment.
Thanks for working on this @Rolv-Apneseth !
It looks like the underlying problem with skopeo copy hanging indefinitely went away because system-test-fedora-44-commands-options is no longer timing out, and we now have a problem with Fedora Rawhide that's potentially due to broken composes or some other bug because Software Factory is failing to set up the jobs. Anyway, I think your changes are still worth pursuing.
| declare -Ag IMAGES=([arch]="quay.io/toolbx/arch-toolbox" \ | ||
| [busybox]="quay.io/toolbox_tests/busybox" \ | ||
| [docker-reg]="quay.io/toolbox_tests/registry" \ | ||
| [docker-reg]="ghcr.io/distribution/distribution:3" \ |
There was a problem hiding this comment.
Do you think we should update the image? We probably should.
As far as I understand, we started using quay.io/toolbox_tests because Docker Hub has relatively restrictive rate limits, and we were occasionally hitting them. @martymichal will correct me if I am wrong.
If GitHub's Container Registry is generous enough, then maybe we can switch to it?
Does the GitHub Container Registry have a front-end or catalogue, like Docker Hub does? Something like hub.docker.com/_/registry for docker.io/library/registry. I am looking for a canonical source of documentation, metadata and the sources for the image.
I know this sounds stupid, and I just can't figure it out. If I type ghcr.io in Firefox, it takes me to github.com/features/actions. That page has a lot of things but nothing about the OCI images.
There was a problem hiding this comment.
Well, I thought it was the updated image that had resolved the issue, but I guess it was just lucky timing haha. Maybe we don't need to update it at all.
But those are good questions about the github container registry. As far as I know they don't have a single front-end/catalogue for all the images they host. But, following the "Packages" linked on the repo, I did find this: https://github.com/distribution/distribution/pkgs/container/distribution
As for the rate limits, I don't see anything mentioned here. I can't find anything else about it online either though.
The |
... before proceeding to avoid potential races. Some changes by Debarshi Ray. containers#1811 containers#1817 Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
| if timeout 5 openssl s_client \ | ||
| -connect "${DOCKER_REG_URI}" \ | ||
| -CAfile "${DOCKER_REG_CERTS_DIR}/domain.crt" \ | ||
| </dev/null >/dev/null 2>&1; then |
There was a problem hiding this comment.
I was wondering if we could retain the standard error stream from openssl(1) and the exit code, because they could be illuminating when dealing with weird failures and timeout(1) seems to take the trouble to have different exit codes. However, it gets funky because Bats runs everything with set -e.
After teaching myself enough about set -e, I was able to tweak your change to do that. I put it up at #1817 to not clobber your original version and to run the CI on it. It needs more testing. Let me know if I missed something.
There was a problem hiding this comment.
Sure, we'll continue that over there then. Maybe this PR is OK to be closed?
2 participants
Attempt to fix CI timeouts with suggestions from #1805.
--command-timeouttoskopeo copycalls so they can't hang forever@debarshiray I chose to go with 10m timeout for the one over the network and 60s for the ones happening locally (in case there's some really slow IO). Did also take the
openssl s_clientapproach like you suggested too. WDYT?