We Trusted Auto-Ack. The Queue Agreed. Our Costs Didn't.

Sanskriti Posted on May 30 We Trusted Auto-Ack. The Queue Agreed. Our Costs Didn't. # eventdriven # java # distributedsystems # architecture Most async bugs announce themselves. This one didn't. No failed jobs. No customer complaints. No error logs. Just infrastructure costs climbing steadily with no obvious cause. It took correlating message IDs across logs to finally see it: the same message being processed two, sometimes three times per delivery. The culprit was a race condition hiding inside an acknowledgment pattern. What Happened A consumer picked up a message and started doing work. That work took time. Before it finished, the queue's retry timeout fired, assumed failure, and redelivered the message to a second consumer. Now two workers were doing identical work concurrently, both completing successfully, both silently doubling the cost. The system looked healthy by every normal metric. It just wasn't. The Fix One configuration change. Python # The problem channel . basic_consume ( queue = ' jobs ' , on_message_callback = process , auto_ack = True ) # The fix def process ( ch , method , properties , body ): do_the_work ( body ) ch . basic_ack ( delivery_tag = method . delivery_tag ) channel . basic_consume ( queue = ' jobs ' , on_message_callback = process , auto_ack = False ) Enter fullscreen mode Exit fullscreen mode Java (Spring AMQP) // The problem @RabbitListener ( queues = "jobs" , ackMode = "AUTO" ) public void process ( String message ) { doTheWork ( message ); } // The fix @RabbitListener ( queues = "jobs" , ackMode = "MANUAL" ) public void process ( String message , Channel channel , @Header ( AmqpHeaders . DELIVERY_TAG ) long tag ) throws IOException { doTheWork ( message ); channel . basicAck ( tag , false ); } Enter fullscreen mode Exit fullscreen mode Acknowledge after the work completes, not when the message arrives. The Real Blindspots This pattern shows up in any async system. Three things that hide it. Auto-ack tells the queue you are do