Skip to content

fix(lock): stop renewing a lease that cannot be extended - #4911

Merged
MontaGhanmy merged 1 commit into
masterfrom
fix/lock-extend-lease
Sep 17, 2026
Merged

MontaGhanmy merged 1 commit into
masterfrom
fix/lock-extend-lease

Conversation

@MontaGhanmy

@MontaGhanmy MontaGhanmy commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Return Redis renewal errors and an internal errLockLost from Extend() instead of discarding the result. Log the first failed renewal and stop further renewal attempts.
  • Example: process A acquires a 20-second lease, then pauses before its first renewal. The lease expires at 20 seconds and process B acquires the same lock at 21 seconds with a new token. When A resumes at 25 seconds, Redis rejects A's old token. Previously A kept trying every tick; now the renewer logs the failure and exits.
  • Stopping renewal does not cancel the protected operation. RAG workspace creation and Swift migration can still continue after losing their lease.
  • Temporary Redis errors also stop renewal in this change. For example, if a renewal fails at about 6.7 seconds and Redis recovers at 8 seconds, no retry occurs at about 13.3 seconds and the original lease expires at 20 seconds. Recovery from transient errors remains a limitation.

@MontaGhanmy MontaGhanmy self-assigned this Sep 11, 2026
Return renewal failures from Extend and stop the renewer after logging
its first failure. Keep the ownership-loss error internal.

Cover ownership loss and Redis errors with deterministic virtual-time
tests. Shutdown on Unlock is handled in a separate change.
@shepilov
shepilov force-pushed the fix/lock-extend-lease branch from d364f39 to e549d73 Compare September 17, 2026 07:47
@MontaGhanmy
MontaGhanmy merged commit 46fd052 into master Sep 17, 2026
4 checks passed
@MontaGhanmy
MontaGhanmy deleted the fix/lock-extend-lease branch September 17, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants