A chaos proxy that actually speaks AMQP. Like Toxiproxy, except instead of cutting cables like a caveman, it reads the RabbitMQ protocol in flight and loses that one ack you were counting on.
Your consumer works great. It gets a message, does the thing, sends an ack, everyone's happy. You've even tested reconnects with Toxiproxy. Ship it.
Then one Tuesday in production, an ack gets lost somewhere between your consumer and the broker - the connection is fine, mind you, just that one frame never made it - and RabbitMQ does exactly what the spec says: it redelivers.
Here's the thing: you can't reproduce that bug with a TCP-level proxy. Toxiproxy sees a byte stream. It can slow it down or cut it, but it has no idea where one AMQP frame ends and the next begins, so it can't lose just the ack while leaving the connection healthy. The bugs that actually hurt RabbitMQ users live one layer above TCP:
- a consumer that double-processes because an ack got lost and the message came back,
- a publisher that silently drops messages because it never handled a missing confirm,
- clients that lock up on consumer cancels or unroutable basic.returns.
amoq sits between your app and the broker, parses the frame stream just enough to know "this is a basic.ack on channel 3", and then - with the probability of your choosing - pretends it never saw it. The rest of the bytes pass through untouched.
pip install amoq # not on PyPI yet - `pip install -e .` for now
# proxy localhost:5673 -> localhost:5672, losing 10 % of consumer acks
amoq --listen 5673 --upstream localhost:5672 --drop basic.ack:0.1
# add 200 ms ± 100 ms of lag to every frame, because the network is a lie
amoq --listen 5673 --upstream localhost:5672 --delay 200:100
# combine: flaky acks AND a laggy broker.
amoq --listen 5673 --upstream localhost:5672 --drop basic.ack:0.2 --delay 50:20
# tell every 20th publisher their message was unroutable (it wasn't)
amoq --listen 5673 --upstream localhost:5672 --inject basic.return:0.05Point your app at port 5673 and find out what it does when the world stops being polite.
| Toxic | Spec | What it does |
|---|---|---|
--drop |
method:probability[:direction] |
Drops matching method frames (e.g. basic.ack:0.1, basic.nack:1.0:c2s) |
--delay |
ms[:jitter_ms][:direction] |
Delays every frame |
--inject |
kind:probability[:trigger] |
Injects a crafted frame the broker never sent |
Direction is c2s (client -> broker), s2c (broker -> client) or both (the default, for maximum mayhem).
--drop refuses to touch heartbeats and connection.* handshake frames - killing those doesn't simulate an interesting failure, it just kills the connection, and you don't need a protocol parser for that.
Three kinds of frames amoq can make up out of thin air:
--inject basic.return:0.05- after abasic.publish, the client gets a312 NO_ROUTEreturn echoing the real exchange and routing key. The publish still reaches the broker; the point is to find out whether your publisher handles returns at all, or silently believes everything it sends arrives.--inject basic.cancel:0.05- after a delivered message (content and all), the consumer gets a cancel notification with its real consumer tag, exactly like when a queue is deleted under a live consumer. Does your app resubscribe, or does it sit there consuming silence forever?--inject channel.close:0.05- kills the channel after abasic.deliver(or another trigger:channel.close:0.05:basic.publish). The client sees a541 closed by amoq; amoq also closes the channel on the broker side, so unacked messages requeue and the channel id stays reusable. Both close-ok replies are swallowed. It's a full-service assassination.
Things amoq should also be able to do to you:
- Inject frames, not just drop them: a surprise
channel.close,basic.cancel, orbasic.return - Confirm-mode awareness: target publisher confirms specifically (
s2c basic.ackafterconfirm.select) - Scenario files: "lose every 3rd ack for 30 s, then heal" as declarative YAML
- Failover simulation: kill the connection mid-transaction, reconnect to a second upstream
- Heartbeat tampering
- Live control API - change toxics without restarting, à la Toxiproxy
make install # editable install + ruff
make check # lint + tests
make rmq-up # throwaway RabbitMQ in docker, for playing along at home
make run # proxy on :5673 losing 10 % of acksEarly prototype, honestly labeled: the frame parser, the drop/delay toxics and all three injections work end to end against a real broker. Everything still unchecked in the roadmap doesn't exist yet.