fix(elasticsearch): gate the rolling-upgrade health wait on expected node count - #198
Conversation
…node count
The two 'wait for cluster health' points in elasticsearch-rolling-upgrade.yml
(before taking a node down, and after it comes back up) only checked
'.status'. Because elasticsearch_api_host defaults to localhost, both
waits query the local node — and immediately after a restart the local
node briefly reports 'number_of_nodes: 1' while peer discovery is still
running. In a 2-node mixed-version window the resulting shape is
'status: yellow, unassigned_shards: 1' for the full retry budget (up to
15 min), even though the cluster is fine and just needs another few
seconds to reform.
Both until clauses now also require number_of_nodes to have caught up
to the expected size ('groups[…] | length', with a single-host default
so single-node scenarios still pass). No behavioural change once the
cluster is actually reformed — this only removes the transient false
positive where we accepted an under-strength cluster reading.
Reproduces as the repeat upgrade_multi_node (debian12/rockylinux9,
elasticsearch_upgrade_8to9) failure that has been showing up on every
non-trivial PR since the wait_status: green default landed.
|
Warning Review limit reached
Next review available in: 56 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
The two
wait for cluster healthpoints inelasticsearch-rolling-upgrade.yml(before taking a node down, and after it comes back up) only checked.status. Becauseelasticsearch_api_hostdefaults tolocalhost, both waits query the local node — and immediately after a restart the local node briefly reportsnumber_of_nodes: 1while peer discovery is still running. In a 2-node mixed-version window the resulting cluster shape isstatus: yellow, unassigned_shards: 1for the full retry budget (up to 15 min), even though the cluster is fine and just needs another few seconds to reform.Both
untilclauses now also requirenumber_of_nodesto have caught up to the expected size (groups[…] | length, with a single-host default so single-node scenarios still pass). No behavioural change once the cluster is actually reformed — this only removes the transient false positive where we accepted an under-strength cluster reading.Reproduces as the repeat
upgrade_multi_node (debian12/rockylinux9, elasticsearch_upgrade_8to9)failure that showed up on every non-trivial PR since thewait_status: greendefault landed.