fix(elasticsearch): gate rolling restarts on cluster health only, not on shard relocations - #187
Merged
Merged
Conversation
|
Warning Review limit reached
Next review available in: 51 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe rolling restart waits now require only the configured cluster health status. The fake API can report relocating shards, and an integration contract verifies rolling restart requests while relocation is ongoing. ChangesRolling restart relocation handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant RollingRestart as Rolling restart task
participant FakeAPI as Fake Elasticsearch API
participant NodeA as Elasticsearch node A
participant NodeB as Elasticsearch node B
RollingRestart->>FakeAPI: Read green health with relocating shards
RollingRestart->>NodeA: Send cluster-settings PUT
RollingRestart->>NodeB: Send cluster-settings PUT
FakeAPI-->>RollingRestart: Return configured green health
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/rolling_restart_contract.yml`:
- Around line 226-233: Update the until condition in the “Wait for fake API”
task to require HTTP status 200 and both relocating_shards and
initializing_shards equal 2, using the registered _fake_api_health response
fields. Apply the same condition to the corresponding repeated wait block.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0a8812e4-494c-47fc-bc85-6976de0bd698
📒 Files selected for processing (3)
roles/elasticsearch/tasks/restart_and_verify_elasticsearch_rolling_node.ymltests/fakes/fake_es_rolling_api.pytests/integration/rolling_restart_contract.yml
… on shard relocations
Oddly
force-pushed
the
fix/rolling-restart-gate-blocks-on-rebalance
branch
from
August 8, 2026 08:40
fcb03a7 to
369c768
Compare
Oddly
added a commit
that referenced
this pull request
Aug 8, 2026
…ew nits The PR body promised a contract test at tests/integration/node_maintenance_contract.yml but the file was never in the diff. This adds it (unblocked via the same .gitignore whitelist the other contract tests use), plus a couple of nits the review flagged: - node_maintenance_contract.yml: three plays against the fake ES API. Play 1 asserts the drain sequence hits every expected endpoint (voting exclusion for the correct nodename, allocation=primaries, ML upgrade mode ON, boost applied at 500mb, flush), then that the restore reverses each one (allocation reset, ML OFF, voting exclusion DELETE, baseline restored to 200mb from elasticsearch_cluster_settings, boosted-but-baseline-less keys explicitly nulled). Play 2 pins elasticsearch_ml_enabled: false and asserts no _ml/set_upgrade_mode call fires — catches the ML gating consistency fix below. - Docs example: elasticsearch_maintenance_wait_for_node now matches the internal _cat/nodes?h=name lookup by using elasticsearch_nodename | default(inventory_hostname). The old value silently spun 60x10s when node.name diverged from inventory_hostname and swallowed the fail. - ML gating: both node_maintenance_start and _end now guard the ML upgrade_mode call with when: elasticsearch_ml_enabled | bool, matching elasticsearch-rolling-upgrade.yml. - Voting exclusion: added a comment explaining why the maintenance entry points use it even though rolling-upgrade doesn't — the reboot/OS-patch window can be long, and the exclusion is always cleared in node_maintenance_end. Duplication with elasticsearch-rolling-upgrade.yml stays as-is — the sibling PRs #187 and #188 change the same shape of code but in the rolling-upgrade file, so unifying now would collide messily. Leaving the two implementations aligned by convention (both gate on status only, both take an operator-choosable wait_status) and opening a follow-up to fold rolling-upgrade onto these entry points after the current PR train settles.
…contract The 'cleanup via surviving peer on failure' play used a block/rescue to absorb the simulated es1 rejoin failure. Rescue runs across every host in the play, so when the include short-circuited es1 with a real failure, es2 hit the sanity fail() task with no rescue path (the rescue was flagged run_once: true and delegated back to es1) and the whole play failed. The 'Verify cleanup used surviving peer' play right after this already asserts against the fake API log that the DELETE landed on port 19201 (es2), so the sanity fail() was redundant. Just ignore_errors on the include and let the log assertion do the work.
…y, drop the sanity assert Take 3 on this — ignore_errors: true on include_tasks doesn't propagate into the included tasks reliably (the role's explicit ansible.builtin.fail with startup diagnostics still counted as a hard play failure). Go back to block/rescue, but keep the fix that started this thread: no sanity fail() task after the include. The following play already asserts against the fake API log that cleanup went via the surviving peer — that's the actual contract we want to prove, and it doesn't need a redundant intra-block trap that lit up es2 with no rescue path of its own.
Oddly
added a commit
that referenced
this pull request
Aug 10, 2026
…ew nits The PR body promised a contract test at tests/integration/node_maintenance_contract.yml but the file was never in the diff. This adds it (unblocked via the same .gitignore whitelist the other contract tests use), plus a couple of nits the review flagged: - node_maintenance_contract.yml: three plays against the fake ES API. Play 1 asserts the drain sequence hits every expected endpoint (voting exclusion for the correct nodename, allocation=primaries, ML upgrade mode ON, boost applied at 500mb, flush), then that the restore reverses each one (allocation reset, ML OFF, voting exclusion DELETE, baseline restored to 200mb from elasticsearch_cluster_settings, boosted-but-baseline-less keys explicitly nulled). Play 2 pins elasticsearch_ml_enabled: false and asserts no _ml/set_upgrade_mode call fires — catches the ML gating consistency fix below. - Docs example: elasticsearch_maintenance_wait_for_node now matches the internal _cat/nodes?h=name lookup by using elasticsearch_nodename | default(inventory_hostname). The old value silently spun 60x10s when node.name diverged from inventory_hostname and swallowed the fail. - ML gating: both node_maintenance_start and _end now guard the ML upgrade_mode call with when: elasticsearch_ml_enabled | bool, matching elasticsearch-rolling-upgrade.yml. - Voting exclusion: added a comment explaining why the maintenance entry points use it even though rolling-upgrade doesn't — the reboot/OS-patch window can be long, and the exclusion is always cleared in node_maintenance_end. Duplication with elasticsearch-rolling-upgrade.yml stays as-is — the sibling PRs #187 and #188 change the same shape of code but in the rolling-upgrade file, so unifying now would collide messily. Leaving the two implementations aligned by convention (both gate on status only, both take an operator-choosable wait_status) and opening a follow-up to fold rolling-upgrade onto these entry points after the current PR train settles.
Oddly
added a commit
that referenced
this pull request
Aug 10, 2026
…ML doesn't see the JSON colons The search() patterns held "method": "POST" etc. as bare plain scalars, which YAML treated as key: value mappings mid-item and rejected the file with 'did not find expected key'. Fixed by wrapping each list item in double quotes and escaping the inner double quotes. The file wasn't picked up by contracts CI until PR #187 landed; that's why the parse error only surfaced on the rebase pass.
Oddly
added a commit
that referenced
this pull request
Aug 11, 2026
…rchestration (#190) * feat(elasticsearch): add node maintenance entry points for external orchestration * fix(elasticsearch): make node maintenance entry points independent of the elasticstack role defaults * test(elasticsearch): add node_maintenance_contract.yml + address review nits The PR body promised a contract test at tests/integration/node_maintenance_contract.yml but the file was never in the diff. This adds it (unblocked via the same .gitignore whitelist the other contract tests use), plus a couple of nits the review flagged: - node_maintenance_contract.yml: three plays against the fake ES API. Play 1 asserts the drain sequence hits every expected endpoint (voting exclusion for the correct nodename, allocation=primaries, ML upgrade mode ON, boost applied at 500mb, flush), then that the restore reverses each one (allocation reset, ML OFF, voting exclusion DELETE, baseline restored to 200mb from elasticsearch_cluster_settings, boosted-but-baseline-less keys explicitly nulled). Play 2 pins elasticsearch_ml_enabled: false and asserts no _ml/set_upgrade_mode call fires — catches the ML gating consistency fix below. - Docs example: elasticsearch_maintenance_wait_for_node now matches the internal _cat/nodes?h=name lookup by using elasticsearch_nodename | default(inventory_hostname). The old value silently spun 60x10s when node.name diverged from inventory_hostname and swallowed the fail. - ML gating: both node_maintenance_start and _end now guard the ML upgrade_mode call with when: elasticsearch_ml_enabled | bool, matching elasticsearch-rolling-upgrade.yml. - Voting exclusion: added a comment explaining why the maintenance entry points use it even though rolling-upgrade doesn't — the reboot/OS-patch window can be long, and the exclusion is always cleared in node_maintenance_end. Duplication with elasticsearch-rolling-upgrade.yml stays as-is — the sibling PRs #187 and #188 change the same shape of code but in the rolling-upgrade file, so unifying now would collide messily. Leaving the two implementations aligned by convention (both gate on status only, both take an operator-choosable wait_status) and opening a follow-up to fold rolling-upgrade onto these entry points after the current PR train settles. * test(elasticsearch): quote node-maintenance contract assertions so YAML doesn't see the JSON colons The search() patterns held "method": "POST" etc. as bare plain scalars, which YAML treated as key: value mappings mid-item and rejected the file with 'did not find expected key'. Fixed by wrapping each list item in double quotes and escaping the inner double quotes. The file wasn't picked up by contracts CI until PR #187 landed; that's why the parse error only surfaced on the rebase pass. * test(elasticsearch): fix node_maintenance_contract regex order to match the fake API log The fake API emits {"body":…, "method":…, "path":…, "port":…} alphabetically per request, so content that lives in the body (primaries, 500mb, null values) shows up in the log line BEFORE the "method"/"path" markers. The old assertions expected the opposite ordering and could not match under regex '.' semantics (which don't cross newlines). Verified against a copy of the actual fake API log dump from the failed run. --------- Co-authored-by: Sam Crauwels <scrauwels@rinis.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The config-triggered rolling restart gated on
relocating_shards == 0andinitializing_shards == 0on top of the health status. A green cluster can relocate shards indefinitely (the balancer keepscluster_concurrent_rebalancemoves in flight until its backlog is drained), so the gate turned into wait-for-the-balancer and timed out into the rescue path on busy clusters — measured over two hours of waiting on a green, fully redundant 7-node cluster.Green already guarantees every primary and replica is allocated; a relocation aborted by a restart is retried I/O, not lost redundancy. All three gates (pre-restart, post-restart, cleanup via surviving peer) now check only
elasticsearch_config_restart_wait_status.Covered in
tests/integration/rolling_restart_contract.yml: the fake API gained a--relocatingflag and a section that asserts the second node is still processed while the cluster reports green with relocating shards. Fails without the fix on the pre-restart gate.Part of the discussion in #185.
Summary by CodeRabbit
Bug Fixes
Tests