Skip to content

Attempt to fix CI timeouts #1811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Rolv-Apneseth wants to merge 3 commits into
base: main
Choose a base branch
from
+29 −4
Open

Attempt to fix CI timeouts #1811

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
75b2832
test/system: add --command-timeout to skopeo copy calls
Rolv-Apneseth Jun 25, 2026
1c876b8
test/system: ensure that the local docker registry is ready before pr…
Rolv-Apneseth Jun 25, 2026
2c1b83c
fixup!: try updating registry image
Rolv-Apneseth Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 29 additions & 4 deletions test/system/libs/helpers.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ readonly TOOLBX_TEST_SYSTEM_TAGS="${TOOLBX_TEST_SYSTEM_TAGS:-$TOOLBX_TEST_SYSTEM
# Images
declare -Ag IMAGES=([arch]="quay.io/toolbx/arch-toolbox" \
[busybox]="quay.io/toolbox_tests/busybox" \
[docker-reg]="quay.io/toolbox_tests/registry" \
[docker-reg]="ghcr.io/distribution/distribution:3" \

@debarshiray debarshiray Jul 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think we should update the image? We probably should.

As far as I understand, we started using quay.io/toolbox_tests because Docker Hub has relatively restrictive rate limits, and we were occasionally hitting them. @martymichal will correct me if I am wrong.

If GitHub's Container Registry is generous enough, then maybe we can switch to it?

Does the GitHub Container Registry have a front-end or catalogue, like Docker Hub does? Something like hub.docker.com/_/registry for docker.io/library/registry. I am looking for a canonical source of documentation, metadata and the sources for the image.

I know this sounds stupid, and I just can't figure it out. If I type ghcr.io in Firefox, it takes me to github.com/features/actions. That page has a lot of things but nothing about the OCI images.

@Rolv-Apneseth Rolv-Apneseth Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I thought it was the updated image that had resolved the issue, but I guess it was just lucky timing haha. Maybe we don't need to update it at all.

But those are good questions about the github container registry. As far as I know they don't have a single front-end/catalogue for all the images they host. But, following the "Packages" linked on the repo, I did find this: https://github.com/distribution/distribution/pkgs/container/distribution

As for the rate limits, I don't see anything mentioned here. I can't find anything else about it online either though.

[fedora]="registry.fedoraproject.org/fedora-toolbox" \
[rhel]="registry.access.redhat.com/ubi8/toolbox" \
[ubuntu]="quay.io/toolbx/ubuntu-toolbox")
Expand Down Expand Up @@ -116,7 +116,9 @@ function _pull_and_cache_distro_image() {
local -i ret_val

for ((j = 0; j < num_of_retries; j++)); do
error_message="$( (skopeo copy --dest-compress \
error_message="$( (skopeo copy \
--command-timeout 10m \
--dest-compress \
"docker://${image}" \
"dir:${IMAGE_CACHE_DIR}/${image_archive}" >/dev/null) 2>&1)"
ret_val="$?"
Expand Down Expand Up @@ -203,6 +205,23 @@ function _setup_docker_registry() {
"${IMAGES[docker-reg]}"
assert_success

# Ensure the registry is ready
local registry_ready=false
local -i retries
for ((retries = 0; retries < 30; retries++)); do
if timeout 5 openssl s_client \
-connect "${DOCKER_REG_URI}" \
-CAfile "${DOCKER_REG_CERTS_DIR}/domain.crt" \
</dev/null >/dev/null 2>&1; then

@debarshiray debarshiray Jul 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering if we could retain the standard error stream from openssl(1) and the exit code, because they could be illuminating when dealing with weird failures and timeout(1) seems to take the trouble to have different exit codes. However, it gets funky because Bats runs everything with set -e.

After teaching myself enough about set -e, I was able to tweak your change to do that. I put it up at #1817 to not clobber your original version and to run the CI on it. It needs more testing. Let me know if I missed something.

@Rolv-Apneseth Rolv-Apneseth Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we'll continue that over there then. Maybe this PR is OK to be closed?

registry_ready=true
break
fi
sleep 1
done
if ! $registry_ready; then
fail "Docker registry at ${DOCKER_REG_URI} did not reach a ready state"
Comment thread
Rolv-Apneseth marked this conversation as resolved.
fi

run podman login \
--authfile "${BATS_SUITE_TMPDIR}/authfile.json" \
--username user \
Expand All @@ -211,7 +230,9 @@ function _setup_docker_registry() {
assert_success

# Add fedora-toolbox:34 image to the registry
run skopeo copy --dest-authfile "${BATS_SUITE_TMPDIR}/authfile.json" \
run skopeo copy \
--command-timeout 60s \
Comment thread
Rolv-Apneseth marked this conversation as resolved.
--dest-authfile "${BATS_SUITE_TMPDIR}/authfile.json" \
dir:"${IMAGE_CACHE_DIR}"/fedora-toolbox-34 \
docker://"${DOCKER_REG_URI}"/fedora-toolbox:34
assert_success
Expand Down Expand Up @@ -331,7 +352,10 @@ function pull_distro_image() {
fi

# https://github.com/containers/skopeo/issues/547 for the options for containers-storage
run skopeo copy "dir:${IMAGE_CACHE_DIR}/${image_archive}" "containers-storage:[overlay@$TOOLBX_ROOTLESS_STORAGE_PATH+$TOOLBX_ROOTLESS_STORAGE_PATH]${image}"
run skopeo copy \
Comment thread
Rolv-Apneseth marked this conversation as resolved.
--command-timeout 60s \
"dir:${IMAGE_CACHE_DIR}/${image_archive}" \
"containers-storage:[overlay@$TOOLBX_ROOTLESS_STORAGE_PATH+$TOOLBX_ROOTLESS_STORAGE_PATH]${image}"

# shellcheck disable=SC2154
if [ "$status" -ne 0 ]; then
Expand Down Expand Up @@ -364,6 +388,7 @@ function pull_default_image_and_copy() {

# https://github.com/containers/skopeo/issues/547 for the options for containers-storage
run skopeo copy \
--command-timeout 60s \
"containers-storage:[overlay@$TOOLBX_ROOTLESS_STORAGE_PATH+$TOOLBX_ROOTLESS_STORAGE_PATH]$image" \
"containers-storage:[overlay@$TOOLBX_ROOTLESS_STORAGE_PATH+$TOOLBX_ROOTLESS_STORAGE_PATH]$image-copy"

Expand Down
Loading