fix(elasticsearch): use configurable retry budget on the post-restart health wait

[Oddly]

Companion fix to #198. The pre-node-down and post-restart health waits in the rolling upgrade are meant to be symmetric, but the post-restart wait had retries: 30 / delay: 30 hardcoded while the pre-node-down wait uses elasticsearch_upgrade_health_retries (default 100) and elasticsearch_upgrade_health_delay (default 30).

On incus-ci debian12 the mixed-version 8→9 peer discovery routinely takes >15 min under load, which is exactly the 30×30 budget. That is the attempts: 30, number_of_nodes: 1, status: yellow timeout signature seen on the last few PRs (including #198 itself, where the node-count guard was correct but the retry cap kept it under water on debian12).

Summary by CodeRabbit

• Bug Fixes
  • Improved rolling upgrades by honoring the configured retry count and delay when waiting for cluster health after an upgrade.
  • Allows upgrade checks to better accommodate environments with varying recovery times.

[coderabbitai]

📝 Walkthrough

Walkthrough

The Elasticsearch rolling upgrade health check now uses configurable retry and delay values instead of fixed values.

Changes

Elasticsearch rolling upgrade

Layer / File(s)	Summary
Configure post-upgrade health check roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml	The post-upgrade health check uses elasticsearch_upgrade_health_retries and elasticsearch_upgrade_health_delay.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the change to use configurable retry settings for the post-restart health wait.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/upgrade-retries-30-to-60

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml (1)

330-335: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Require green cluster health for rolling upgrades. elasticsearch_upgrade_wait_status accepts yellow, and both health gates can proceed while the cluster is yellow. Restrict this setting and both gates to green, then update the argument specification and documentation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml` around lines 330
- 335, Restrict elasticsearch_upgrade_wait_status to green only, and update both
rolling-upgrade health gates to require green cluster health rather than
accepting yellow. Align the argument specification and related documentation
with this green-only behavior, including any status list or validation symbols
used by the upgrade tasks.

Source: Path instructions

🧹 Nitpick comments (1)

roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml (1)

334-335: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Update the variable descriptions for the new usage.

These variables now control both the pre-node-down and post-restart health waits. The descriptions in roles/elasticsearch/defaults/main.yml Lines 95-100 and roles/elasticsearch/meta/argument_specs.yml Lines 162-169 describe only the pre-node-down wait. Update them to document both uses.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml` around lines 334
- 335, Update the descriptions for elasticsearch_upgrade_health_retries and
elasticsearch_upgrade_health_delay in the defaults and argument specification
definitions to state that they control health waits both before taking a node
down and after restarting it.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml`:
- Around line 330-335: Restrict elasticsearch_upgrade_wait_status to green only,
and update both rolling-upgrade health gates to require green cluster health
rather than accepting yellow. Align the argument specification and related
documentation with this green-only behavior, including any status list or
validation symbols used by the upgrade tasks.

---

Nitpick comments:
In `@roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml`:
- Around line 334-335: Update the descriptions for
elasticsearch_upgrade_health_retries and elasticsearch_upgrade_health_delay in
the defaults and argument specification definitions to state that they control
health waits both before taking a node down and after restarting it.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5fa94a4c-ce43-4f9e-b35c-3e078374c905

📥 Commits

Reviewing files that changed from the base of the PR and between 2c5175b and 1f8d441.

📒 Files selected for processing (1)

• roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml