Skip to content

fix(rabbitmq): recover consumer from channel death and broker cancel - #1

Merged
gitpancake merged 1 commit into
mainfrom
fix/rabbitmq-consumer-channel-recovery
Jul 20, 2026
Merged

fix(rabbitmq): recover consumer from channel death and broker cancel#1
gitpancake merged 1 commit into
mainfrom
fix/rabbitmq-consumer-channel-recovery

Conversation

@gitpancake

Copy link
Copy Markdown
Owner

Problem

image-engine sat online for ~27h with zero consumers on flash-engine.flash-received and a 17,240 message backlog. Process healthy, TCP to RabbitMQ ESTABLISHED, no log output, images_processed_total = 1 over two days. dataimpulse proxy and Pinata were both fine — the consumer had silently unsubscribed and could not recover without a full restart.

Cause

FlashcastrConsumer only recovered from connection-level close. Three paths left it subscribed to nothing:

  • channel close with a live connection — handler only logged. A broker-side channel error unsubscribed the consumer permanently.
  • broker-side consumer cancel — amqplib delivers null to the consume callback; swallowed by the if (!msg) return guard.
  • hung AMQP handshakeconnect() had no deadline. A socket that opens but never completes the handshake parks the reconnect loop forever with reconnecting stuck true.

Fix

  • channel close and broker cancel both trigger reconnect()
  • connectWithTimeout() races an explicit 20s handshake deadline (amqplib's timeout only covers TCP connect), and closes a late-arriving connection rather than leaking it
  • reconnect() calls teardown() first — detaches listeners before closing so close events don't re-enter the loop
  • new closing flag suppresses recovery during intentional shutdown so close() doesn't race the reconnect loop

Publisher was checked and left alone: it is lazy-connect, nulls its channel on close, and reconnects on next publish.

Verification

  • consumer.ts typechecks clean standalone
  • Production restart confirmed the diagnosis: consumer re-registered, backlog drained at ~254 msg/min, ipfs_failures_total 0, DLQ 0 — messages genuinely processed, not nacked
  • ⚠️ no unit testlibs/rabbitmq has no specs or vitest config; adding one means bootstrapping test infra for the lib
  • ⚠️ full tsc project build and nx build are broken on main already (stale dist/ TS6305, unresolvable @nx/next/plugin) — pre-existing, unrelated to this change

The recovery paths only fire on real channel death, so this deploys unexercised. Worth watching consumer count for a day.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HioDdCpmbbk6vrmkLt9Psg

FlashcastrConsumer only recovered from connection-level `close`. Three paths
left it subscribed to nothing while the process stayed healthy:

- channel `close` with a live connection only logged, so a broker-side channel
  error permanently unsubscribed the consumer
- amqplib delivers `null` to the consume callback on broker-side cancellation;
  that was swallowed by the `if (!msg) return` guard
- `connect()` had no handshake deadline, so a socket that opened but never
  finished the AMQP handshake parked the reconnect loop forever with
  `reconnecting` stuck true

Symptom in production: image-engine online for 2 days, TCP to RabbitMQ
ESTABLISHED, zero consumers on flash-engine.flash-received, 17,240 messages
backed up, and no log output for ~27h. Only a full process restart recovered it.

Reconnect now tears down stale handles first, and `closing` suppresses recovery
during intentional shutdown so close() doesn't race the reconnect loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HioDdCpmbbk6vrmkLt9Psg
@gitpancake
gitpancake merged commit 6b9f8ef into main Jul 20, 2026
1 check failed
@gitpancake
gitpancake deleted the fix/rabbitmq-consumer-channel-recovery branch July 20, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant