Skip to content

Conversation

@JesperTerkelsen
Copy link
Member

Summary

Add retry logic with automatic rebase to component-deploy workflow to prevent race conditions when multiple environments deploy concurrently.

Problem

When dev and staging deployments run in parallel (or any concurrent deployments), they can encounter race conditions when pushing to kube-manifests:

  1. Both jobs check out kube-manifests at roughly the same time
  2. Both make changes and commit locally
  3. Both try to push - one succeeds, one fails with "remote has moved ahead"

Example failure: https://github.com/monta-app/service-admin-panel/actions/runs/21146966256/job/60814766132

Solution

Replace the ad-m/github-push-action with custom retry logic that:

  • Attempts to push up to 5 times
  • On push failure, waits 5 seconds
  • Fetches latest changes and rebases automatically
  • Retries the push
  • Exits with error only if rebase fails (conflicts) or max retries reached

Changes

.github/workflows/component-deploy.yml:

  • Replaced ad-m/github-push-action@master step with custom bash script
  • Added retry loop with configurable max attempts (5) and delay (5s)
  • Automatic fetch and rebase on push failure
  • Clear logging for each attempt and outcome

Benefits

  • Eliminates race conditions in concurrent deployments
  • No manual intervention needed when pushes collide
  • Maintains deployment reliability across all environments
  • Clear logging shows retry attempts and outcomes

Testing

This should be tested with concurrent deployments (dev + staging) to verify the retry logic works as expected.

🤖 Generated with Claude Code

When multiple deployments (dev/staging/production) run concurrently, they
can encounter race conditions when pushing to kube-manifests. This adds
automatic retry logic with rebase to handle concurrent pushes gracefully.

Changes:
- Replace github-push-action with custom retry logic
- Automatically fetch and rebase on push failure
- Retry up to 5 times with 5-second delay between attempts
- Exit with error if rebase fails (conflicts) or max retries reached

Fixes race condition seen in concurrent dev/staging deployments.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace custom retry script with the same Artur-Davtyan/git-rebase-push-action@v1
used in component-deploy-v2.yml. This provides:
- Automatic commit and push with retry logic
- Built-in rebase handling on conflicts
- Consistent approach across both workflows
- Simpler, more maintainable code
The git-rebase-push-action needs the yq_command parameter to re-apply
file changes after rebasing during retry attempts. This ensures the
updates to values.yaml and config.yaml persist through rebase operations.
@JesperTerkelsen JesperTerkelsen merged commit 02cc646 into main Jan 19, 2026
1 check passed
@JesperTerkelsen JesperTerkelsen deleted the fix/component-deploy-push-retry branch January 19, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants