From 62d03228b4dcdf82727c6c9a8bb1fde63b277627 Mon Sep 17 00:00:00 2001 From: oddly Date: Sat, 8 Aug 2026 12:21:47 +0200 Subject: [PATCH] chore(ci): give the incus-ci memory gate 45 min of patience instead of 15 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pre-launch capacity check in molecule/shared/create.yml has been retrying 30x30s (15 min) since the runner pool was rebuilt after the lab outage. That was fine for one PR at a time; it stops working the moment more than a couple of heavy PRs are open, because heavy scenarios (beats_security, es_kibana, upgrade_multi_node, elasticsearch_diagnostics, elasticstack_default) each need 8–20 GB and lose the memory race against a swarm of 1–2 GB jobs coming through faster. A recent 5-PR merge sequence lit up five of those with matching "No capacity: X committed + Y needed > Z available" stdouts. Bump to 90x30s (45 min) and lift the standard molecule.yml default timeout from 45 to 60 min so the gate still fails loudly with the last-attempt stdout before GitHub Actions cancels the job. The existing per-caller overrides (elasticsearch=30, modules=20) stay where they are — those jobs are small and don't starve. --- .github/workflows/molecule.yml | 2 +- molecule/shared/create.yml | 10 ++++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/molecule.yml b/.github/workflows/molecule.yml index c21c6d87..71049694 100644 --- a/.github/workflows/molecule.yml +++ b/.github/workflows/molecule.yml @@ -17,7 +17,7 @@ on: description: JSON array of Elastic major versions timeout: type: number - default: 45 + default: 60 max-parallel: type: number default: 10 diff --git a/molecule/shared/create.yml b/molecule/shared/create.yml index 28b4900b..cf9fbc90 100644 --- a/molecule/shared/create.yml +++ b/molecule/shared/create.yml @@ -130,14 +130,16 @@ REMOTE_SCRIPT changed_when: true register: _launch_result - # 30 retries × 30s = 15 min. Deliberately shorter than the workflow - # timeout (45 min for the standard molecule.yml call, 120 min for + # 90 retries × 30s = 45 min. Deliberately shorter than the workflow + # timeout (60 min for the standard molecule.yml call, 180 min for # full_stack) so the task fails loudly with the last "No capacity: # …" stdout instead of being silently cancelled by the workflow # timeout — that made the queue-starvation cases very hard to # diagnose (see PR investigating the elasticsearch_roles_calculation - # / elasticstack_default post-outage hangs). - retries: 30 + # / elasticstack_default post-outage hangs). Bumped from 30 after a + # sustained multi-PR CI storm evicted every heavy scenario on the + # first attempt. + retries: 90 delay: 30 until: _launch_result.rc == 0