Skip to content

chore(ci): give the incus-ci memory gate 45 min of patience #192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Oddly merged 1 commit into from
Aug 10, 2026
Merged

chore(ci): give the incus-ci memory gate 45 min of patience #192

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/molecule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
description: JSON array of Elastic major versions
timeout:
type: number
default: 45
default: 60
max-parallel:
type: number
default: 10
Expand Down
10 changes: 6 additions & 4 deletions molecule/shared/create.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,14 +130,16 @@
REMOTE_SCRIPT
changed_when: true
register: _launch_result
# 30 retries × 30s = 15 min. Deliberately shorter than the workflow
# timeout (45 min for the standard molecule.yml call, 120 min for
# 90 retries × 30s = 45 min. Deliberately shorter than the workflow
# timeout (60 min for the standard molecule.yml call, 180 min for
# full_stack) so the task fails loudly with the last "No capacity:
# …" stdout instead of being silently cancelled by the workflow
# timeout — that made the queue-starvation cases very hard to
# diagnose (see PR investigating the elasticsearch_roles_calculation
# / elasticstack_default post-outage hangs).
retries: 30
# / elasticstack_default post-outage hangs). Bumped from 30 after a
# sustained multi-PR CI storm evicted every heavy scenario on the
# first attempt.
retries: 90
Comment on lines +133 to +142

@coderabbitai coderabbitai Bot Aug 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Make retries safe after a partial launch.

If an earlier incus launch succeeds and a later launch fails, the next retry performs the capacity check before the cleanup loop at Lines 117-119. The partial containers remain counted in committed_mb, while _needed_mb still includes the full batch. The retry can therefore fail the capacity gate on every attempt and consume all 90 retries.

Delete the expected containers or roll back partial launches while holding the lock before recalculating capacity.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@molecule/shared/create.yml` around lines 133 - 142, Update the retry handling
around the capacity check and cleanup loop in the Incus launch flow so partial
containers from a failed batch are deleted or their committed capacity is rolled
back while holding the lock before recalculating capacity. Ensure the next retry
excludes resources from earlier successful launches, allowing the capacity gate
to evaluate the remaining batch correctly.

delay: 30
until: _launch_result.rc == 0

Expand Down
Loading