fix(ci): wait for the showcase-runtime rollout before running its tests - #5340
Open
gustavolira wants to merge 2 commits into
Open
fix(ci): wait for the showcase-runtime rollout before running its tests#5340gustavolira wants to merge 2 commits into
gustavolira wants to merge 2 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release-1.10 #5340 +/- ##
=================================================
+ Coverage 41.03% 69.60% +28.57%
=================================================
Files 121 111 -10
Lines 2220 4702 +2482
Branches 562 537 -25
=================================================
+ Hits 911 3273 +2362
- Misses 1303 1428 +125
+ Partials 6 1 -5
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
The showcase-runtime phase started Playwright as soon as `helm upgrade`
returned, without waiting for the deployment to become ready. The first
test in the showcase-runtime-db dependency project scales the deployment
to 0, so when the initial rollout is still running that scale-down kills
the install-dynamic-plugins init container mid-install.
That script holds a lock file on the dynamic-plugins-root PVC and
releases it from an atexit handler, which a hard kill skips. The lock
survives on the PVC and every pod created afterwards blocks forever on:
======= Waiting for lock release (file: /dynamic-plugins-root/install-dynamic-plugins.lock)...
Each Azure DB test then burns its full 10 minute timeout waiting for a
pod that can never become ready. Ten attempts consume 100 minutes, the
job hits the Prow timeout inside this phase, and showcase-sanity-plugins
never runs.
The race only opens wide enough to fire when the showcase-runtime
namespace also has to extract the catalog index and pull OCI plugins,
which happens when CATALOG_INDEX_IMAGE is set. That is empty on the
daily nightly, so it passes, but it is always set on the RC and GA
verification runs triggered through Gangway with --catalog-index-image.
Those runs are the reason the release branch nightlies exist, and they
have not reached the sanity plugin check at all.
Swap testing::run_tests for testing::check_and_test in both the OCP Helm
and OCP Operator runtime phases. It gates on /healthcheck before starting
the tests, so the initial plugin install always completes before anything
scales the deployment down, and it collects pod logs when tests fail.
Also align restartDeployment's internal budgets with the 10 minute
timeout its callers set: 5m + 10s + 10m could not fit, so Playwright
killed the worker mid-restart, the catch block never logged pod
conditions or events, and the retry scaled the deployment down again on
top of a rollout that was still starting. 2m + 10s + 7m fits.
Verified on an OCP 4.22 cluster by running the full handle_ocp_nightly
pipeline with CATALOG_INDEX_IMAGE set, the configuration that reproduces
the hang:
showcase 35 passed
showcase-rbac 17 passed
showcase-runtime 19 passed (22.9m, was 100m and never finished)
showcase-sanity-plugins 9 passed (had never run)
Zero "Waiting for lock release" lines, against 648 in the failing run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gustavolira
force-pushed
the
fix/e2e-runtime-readiness-gate
branch
from
September 3, 2026 20:08
5bf135d to
5818217
Compare
Member
Author
|
/test ? |
Member
Author
|
/test e2e-ocp-helm-nightly |
Contributor
|
The container image build workflow finished with status: |
Contributor
|
The container image build workflow finished with status: |
The full reasoning lives in the commit message and the PR body; the inline comments only need to say why the call is shaped this way.
|
Member
Author
|
/test ? |
Member
Author
|
/test e2e-ocp-helm-nightly |
|
@gustavolira: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Triggering the
release-1.10OCP Helm nightly for an RC verification never reaches the sanity plugin check. The job dies insideshowcase-runtimeafter burning its whole budget there.The phase starts Playwright as soon as
helm upgradereturns, with no readiness gate and no--wait. The first test of theshowcase-runtime-dbdependency project scales the deployment to 0, so while the initial rollout is still running that scale-down killsinstall-dynamic-pluginsmid-install. The script holds a lock on thedynamic-plugins-rootPVC and releases it from anatexithandler, which a hard kill skips. The lock outlives the container and every later pod blocks forever on:Each Azure DB test then burns its full 10 minute timeout on a pod that can never become ready. Ten attempts consume 100 minutes and the job hits the Prow timeout.
Why it surfaced now
Within
release-1.10, the race needs the init container to still be running when the first test scales the deployment down. The window widens when theshowcase-runtimenamespace also has to extract the catalog index and pull OCI plugins, which happens whenCATALOG_INDEX_IMAGEis set andhelm::get_image_paramsinjects it into that namespace.CATALOG_INDEX_IMAGEis empty by default on this branch, and the last clean periodic run (2074358014747348992, Jul 7) had no catalog index and passed withshowcase-runtime19 passed in 20.7m. It is always set on the RC and GA verification runs triggered through Gangway with--catalog-index-image, and both such runs on PR #5335 hung, with 648Waiting for lock releaselines.mainis not affected, but it is also not a useful comparison: it moved the runtime deployment into TypeScript (runtime-deploy.ts/ensureRuntimeDeployed()), which runshelm upgrade ... --wait --timeout 10mand therefore cannot start tests against a rollout in progress. This branch still has the bash flow.Fix
Swap
testing::run_testsfortesting::check_and_testin the OCP Helm and OCP Operator runtime phases. It gates on/healthcheckbefore starting the tests, so the initial plugin install always completes before anything scales the deployment down, and it collects pod logs when tests fail. Every other phase on this branch already uses it.Also align
restartDeployment's internal budgets with the 10 minute timeout its callers set. 5m + 10s + 10m could not fit, so Playwright killed the worker mid-restart, thecatchblock never logged pod conditions or events, and the retry scaled the deployment down again on top of a rollout that was still starting. 2m + 10s + 7m fits.Verification
Ran the full
handle_ocp_nightlypipeline on an OCP 4.22 cluster withCATALOG_INDEX_IMAGEset, the configuration that reproduces the hang:showcaseshowcase-rbacshowcase-runtimeshowcase-sanity-pluginsWaiting for lock releaselines: 648 before, 0 after.Follow-up
install-dynamic-plugins.pyhas no staleness check on that lock —wait_for_lock_releaseis an unboundedwhile True. This PR closes the window that opens it, but any hard kill (grace period expiry, eviction, OOM) can still orphan it. I have a fix for that and will open it separately, since it touches the product image rather than CI.Longer term, this branch could adopt main's approach instead, where the runtime deploy owns its own
--wait.🤖 Generated with Claude Code