elasticsearch: persistent cluster settings are applied after the config-triggered rolling restart

[Oddly]

Problem

elasticsearch_cluster_settings is applied at the very end of the role, after any restart triggered by a config change has already completed.

In roles/elasticsearch/tasks/main.yml:

• line 532 — import_tasks: elasticsearch-security.yml, which ends with meta: flush_handlers (elasticsearch-security.yml:625). This is where Restart Elasticsearch rolling runs.
• line 651 — include_tasks: elasticsearch-cluster-settings.yml

So the rolling restart runs with whatever persistent settings the cluster happens to have at that moment, not with the ones declared in elasticsearch_cluster_settings.

Impact

This matters most for the settings that govern how fast a cluster recovers after a node comes back:

• indices.recovery.max_bytes_per_sec (ES default 40mb)
• cluster.routing.allocation.node_concurrent_incoming_recoveries (ES default 2)

On a first run, or any time those settings have drifted out of the cluster, every node in the rolling restart recovers at the ES defaults even though the playbook declares something higher.

It also interacts badly with the health gate in restart_and_verify_elasticsearch_rolling_node.yml, which waits for:

- status in _elasticsearch_health_statuses
- relocating_shards == 0
- initializing_shards == 0

A cluster can be green while the balancer is still relocating shards after the restarted node rejoins, so this gate waits for the rebalance to finish. The budget is elasticsearch_config_restart_health_retries (50) x elasticsearch_config_restart_health_delay (30s) = 25 minutes per node. At the default 40mb throttle a rebalance on a large data node can exceed that, and the run then falls into the rescue block and fails with:

Elasticsearch failed to restart on <host>.
Recent log output: ...

plus 50 lines of journal, while the node restarted perfectly fine and the cluster was green the whole time.

Suggested fix

Apply the persistent cluster settings before any restart handler can run, e.g. move the elasticsearch-cluster-settings.yml include ahead of elasticsearch-security.yml.

Secondary: the rescue message assumes a startup failure. When the failing task is the post-restart health gate, reporting the last cluster health response (status, relocating_shards, initializing_shards) would point at the real cause instead of the journal.

[Oddly]

Measured this on a 7-node cluster (3 masters, 2 hot, 2 warm, ~1950 shards) during a run of the config-triggered rolling restart.

The health gate sat on relocating_shards != 0 for over two hours while the cluster was green and fully redundant the entire time. _cluster/allocation/explain on one of the relocating shards:

can_rebalance: throttled
decider: concurrent_rebalance -> THROTTLE
"reached the limit of concurrently rebalancing shards [2],
 cluster setting [cluster.routing.allocation.cluster_concurrent_rebalance=2]"

There is a queue of rebalance moves and Elasticsearch keeps exactly cluster_concurrent_rebalance of them in flight. Every time one finishes the next starts, so relocating_shards never reaches 0 until the whole backlog is drained. The gate is not waiting for the cluster to be safe, it is waiting for the balancer to be finished.

The two extra conditions do not add anything over the status check:

• green already means every primary and every replica is allocated and active, so redundancy is guaranteed.
• With the cluster green, an initializing shard can only be a relocation target. A replica that still has to initialize makes the cluster yellow, not green.
• During a relocation the source shard stays active until the target is ready. Restarting a node in that window aborts the relocation and it is retried later. That costs I/O, it does not cost redundancy. It is also largely moot because the restart flow sets cluster.routing.allocation.enable=primaries for the duration, which pauses replica movement anyway.

Elastic's own rolling restart procedure goes no further than waiting for _cat/health to report green.

Narrowed all three health gates in restart_and_verify_elasticsearch_rolling_node.yml (pre-restart, post-restart, and the always cleanup via the surviving peer) to the status check only. elasticsearch_config_restart_wait_status stays the single knob.

Covered in tests/integration/rolling_restart_contract.yml: the fake API takes a --relocating argument and reports green with relocating and initializing shards, and the contract asserts the second node is still reached. Without the change the pre-restart gate fails on {"status": "green", "relocating_shards": 2, "initializing_shards": 2}.

Both original points in this issue are still open: the cluster-settings ordering, and the rescue message that reports a startup failure when the health gate is what actually timed out.