fix(rabbitmq): recover consumer from channel death and broker cancel - #1
Merged
Merged
Conversation
FlashcastrConsumer only recovered from connection-level `close`. Three paths left it subscribed to nothing while the process stayed healthy: - channel `close` with a live connection only logged, so a broker-side channel error permanently unsubscribed the consumer - amqplib delivers `null` to the consume callback on broker-side cancellation; that was swallowed by the `if (!msg) return` guard - `connect()` had no handshake deadline, so a socket that opened but never finished the AMQP handshake parked the reconnect loop forever with `reconnecting` stuck true Symptom in production: image-engine online for 2 days, TCP to RabbitMQ ESTABLISHED, zero consumers on flash-engine.flash-received, 17,240 messages backed up, and no log output for ~27h. Only a full process restart recovered it. Reconnect now tears down stale handles first, and `closing` suppresses recovery during intentional shutdown so close() doesn't race the reconnect loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HioDdCpmbbk6vrmkLt9Psg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
image-enginesat online for ~27h with zero consumers onflash-engine.flash-receivedand a 17,240 message backlog. Process healthy, TCP to RabbitMQESTABLISHED, no log output,images_processed_total = 1over two days. dataimpulse proxy and Pinata were both fine — the consumer had silently unsubscribed and could not recover without a full restart.Cause
FlashcastrConsumeronly recovered from connection-levelclose. Three paths left it subscribed to nothing:closewith a live connection — handler only logged. A broker-side channel error unsubscribed the consumer permanently.nullto the consume callback; swallowed by theif (!msg) returnguard.connect()had no deadline. A socket that opens but never completes the handshake parks the reconnect loop forever withreconnectingstucktrue.Fix
closeand broker cancel both triggerreconnect()connectWithTimeout()races an explicit 20s handshake deadline (amqplib'stimeoutonly covers TCP connect), and closes a late-arriving connection rather than leaking itreconnect()callsteardown()first — detaches listeners before closing so close events don't re-enter the loopclosingflag suppresses recovery during intentional shutdown soclose()doesn't race the reconnect loopPublisher was checked and left alone: it is lazy-connect, nulls its channel on close, and reconnects on next publish.
Verification
consumer.tstypechecks clean standaloneipfs_failures_total0, DLQ 0 — messages genuinely processed, not nackedlibs/rabbitmqhas no specs or vitest config; adding one means bootstrapping test infra for the libtscproject build andnx buildare broken onmainalready (staledist/TS6305, unresolvable@nx/next/plugin) — pre-existing, unrelated to this changeThe recovery paths only fire on real channel death, so this deploys unexercised. Worth watching consumer count for a day.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HioDdCpmbbk6vrmkLt9Psg