(moving our Slack convo here)
We had a situation where one message took a very long time to process and essentially got stuck in our queue. To solve for this, we would like the ability to impose a hard timeout for message processing. If a message fails to process within the timeout period, it should be sent to the configured DLQ.
I tried solving this a few different ways in our subclass of RunTaskWithMultiprocessing as well as a custom processing strategy but they were all pretty hacky/duplicative and it felt best to support directly within Arroyo. Also worth noting, the work our message processing is doing (heavy C++ use) completely hogs the GIL from what I can tell, so traditional signals and thread interrupts don't seem to work. From what I understand our use-case of Arroyo is unique (low-volume long-running jobs) but being able to set a timeout seems useful for others as well.
(moving our Slack convo here)
We had a situation where one message took a very long time to process and essentially got stuck in our queue. To solve for this, we would like the ability to impose a hard timeout for message processing. If a message fails to process within the timeout period, it should be sent to the configured DLQ.
I tried solving this a few different ways in our subclass of
RunTaskWithMultiprocessingas well as a custom processing strategy but they were all pretty hacky/duplicative and it felt best to support directly within Arroyo. Also worth noting, the work our message processing is doing (heavy C++ use) completely hogs the GIL from what I can tell, so traditional signals and thread interrupts don't seem to work. From what I understand our use-case of Arroyo is unique (low-volume long-running jobs) but being able to set a timeout seems useful for others as well.