[neuyilan]

A worker group stops scaling (both up and down) indefinitely when a single worker Pod gets stuck in a terminal/terminating state (e.g. a Failed Pod whose node becomes NotReady, so the kubelet never confirms termination and the Pod keeps a deletionTimestamp that never clears).

Root cause - two gates, both tripped by the same stuck Pod:

1. Scale-expectation gate (expectations/scale_expectations.go): isPodScaled() has a timeout fallback for Create but none for Delete. A Pod requested for deletion that never disappears keeps its Delete expectation unsatisfied forever; IsSatisfied() short-circuits on it, freezing all scaling.
2. Unhealthy-deletion early-return (raycluster_controller.go): even past gate 1, a Failed Pod is unconditionally judged shouldDelete=true, counted into numDeletedUnhealthyWorkerPods, triggering the early return before workersToDelete / replica-diff logic runs. The stuck Pod re-triggers this every reconcile.

Fix:

• scale_expectations.go: give the Delete branch a timeout fallback, but only after Kubernetes has set deletionTimestamp, so an expectation is never satisfied prematurely before deletion actually starts.
• raycluster_controller.go: split out worker Pods that already have a deletionTimestamp (already terminating) so they no longer re-trigger unhealthy deletion or the early-return, and are skipped when draining workersToDelete. Terminating Pods still reserve their replica index so new Pods do not reuse it.
• Observability: when Pods are stuck terminating, log their names (terminatingPodNames), not just a count, and log which Pod/action is blocking a group's reconcile.

Tests:

• TestIsPodScaledDeleteTimeout / TestIsPodScaledDeleteDoesNotTimeoutBeforeDeletion.
• TestReconcilePodsStuckDeleteContinuesScaling / TestReconcilePodsStuckDeleteDoesNotReuseReplicaIndex.

Why are these changes needed?

Related issue number

Labels

• If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
• If this PR contains breaking changes, I have added the breaking-change label.

AI assistance

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Manual test instructions

[Copilot]

🟡 Changes recommended

The multi-slice E2E test still requires immediate reuse of a deleted Pod’s index, making its assertion timing-dependent when reconciliation reserves that index while the Pod is terminating.

1 open issue

• Make replacement index assertions tolerate terminating Pods · New

What changed in this PR

This PR prevents stuck terminating worker Pods from indefinitely blocking RayCluster scaling, while preserving their replica indices until they disappear.

Changes:

• Allow delete expectations to expire after a timeout only once Pod deletion has started.
• Exclude terminating Pods from active reconciliation while reserving their indices, and log blocked expectations and terminating Pod names.
• Add regression tests for continued scaling, index reservation, and delete-timeout safeguards; remove sleep-based timeout testing.

File	Description
ray-operator/​controllers/​ray/​raycluster_controller.go	Separate active and terminating workers and reserve occupied replica indices.
ray-operator/​controllers/​ray/​raycluster_controller_unit_test.go	Test scaling progress and index reservation with stuck terminating Pods.
ray-operator/​controllers/​ray/​expectations/​scale_expectations.go	Add guarded delete-expectation expiry and blocking diagnostics.
ray-operator/​controllers/​ray/​expectations/​scale_expectations_test.go	Test delete-timeout safeguards and use deterministic timestamps.

_{Review effort: Balanced}