Skip to content

feat(shared): retry with exponential backoff before dead-lettering - #6

Merged
devinder-dev merged 1 commit into
mainfrom
phase-4-retry-backoff
Jun 21, 2026
Merged

devinder-dev merged 1 commit into
mainfrom
phase-4-retry-backoff

Conversation

@devinder-dev

Copy link
Copy Markdown
Owner

Summary

  • Failed message handlers no longer dead-letter on the first failure. Each queue gets a companion <queue>.retry queue; on failure the message is republished there with a per-message TTL (exponential backoff: 1s, 2s, 4s) and an x-retry-count header, then dead-letters back into the original queue once that TTL expires for redelivery
  • After 3 failed attempts, falls through to the existing permanent <queue>.dlq exactly as before
  • No service index.ts files changed - entirely contained in packages/shared/src/mq.ts, same shape as the Phase 2 reconnect fix
  • Bonus fix found via this phase's own testing: a message redelivered via the retry queue arrives with msg.fields.routingKey set to the queue name (not the original event type), which broke schema lookup on the 2nd attempt. Fixed by carrying the true routing key forward in a header.

Test plan

  • bun run typecheck green across all services
  • Snyk code scan on packages/shared - 0 issues
  • Manual (throwaway test consumer, not committed): handler fails twice then succeeds on attempt 3 - retried at correct 1s/2s intervals, acked cleanly, never reached the DLQ
  • Manual: handler that always fails - retried exactly 3 times (4 total attempts at t=0/1/3/7s matching backoff), then landed in the permanent DLQ with no 4th retry
  • Manual: re-ran the full real order -> kitchen -> notification happy path afterward - zero retry/error log lines, fully unaffected

A handler failure used to dead-letter on the first attempt, too
aggressive for transient errors (a Redis blip during the idempotency
check, a momentary DB hiccup). Failed messages now go to a per-queue
retry queue with an exponential per-message TTL (1s/2s/4s) that
dead-letters them back into the original queue for redelivery, up to
3 attempts, before falling through to the existing permanent DLQ.

Also fixes a routing-key bug this surfaced: a message redelivered via
the retry queue arrives through the default exchange with
msg.fields.routingKey set to the queue name, not the original event's
routing key. The true routing key now travels in a header instead.
@devinder-dev
devinder-dev merged commit 3dc0b8b into main Jun 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant