Message Queues: Delivery Guarantees Demystified

Hey everyone, welcome to the forty-fifth issue of The Main Thread.

Every distributed system eventually grows a message queue. We have a service that produces work and a service that consumes it, and we don’t want the producer to wait for the consumer, or to lose the work if the consumer is down. So we put a broker in between - RabbitMQ, Kafka, SQS, whatever - and now our two services are decoupled, buffered, and independently scalable. It feels like a clear win.

Then the queue’s documentation starts using three phrases that sound like marketing and turn out to be the whole ballgame: at-least-once, exactly-once, and at-most-once delivery. Most of us engineers nod along with a vague sense that exactly-once is the good one and the others are compromises. That vague sense is exactly where the production issues come from. The customer charged twice. The email sent five times. The order disappeared. Every one of those is a delivery-guarantee misunderstanding.

More than marketing, these are precise statements about what happens to a message when something fails; and in a distributed system something is always failing. A network packet drops, a broker restarts, and an ack got lost on the way back. The delivery guarantee is a contract for how the system behaves in exactly those moments, and the three guarantees are three different answers to one question: when we are not sure whether a message was processed, do we risk losing it or risk doing it twice?

This article is the precise guide. What each guarantee actually means, why “exactly-once” is the most misunderstood phrase in distributed systems, how the real systems achieve what looks like exactly-once, and how to choose between the brokers that offer these trade-offs. It connects directly to two earlier articles: idempotency and the saga pattern, because as we will see, those are tools that make any of this safe.

The Three Guarantees, Precisely

A delivery guarantee describes message behaviour under failure and retry. In the happy path, all three guarantees look the same where the message arrives and gets processed. The differences only appear when something goes wrong, and they all hinge on one decision: when do we acknowledge the message - before or after processing it?

We should walk through the failure carefully, because the whole topic lives in this one sequence. A consumer receives a message, processes it, and sends an ack back to the broker so the broker can stop tracking it. Three things can fail: the consumer can crash before processing, during processing, or after processing but before the ack reaches the broker. The last case where the work done but ack lost is the one that makes this hard, because the broker genuinely can’t tell the difference between “the consumer did the work and ack was lost” and “the consumer crashed before doing the work“. From the broker’s side, those two situations are identical. It has to guess.

At-most once

Ack first, then process. The consumer acknowledges the message as soon as it receives it, then does the work. If it crashes mid-process, the message is already acked and gone; the work is never done. Messages can be lost, but never duplicated. This is fire-and-forget. We should use it when loss is cheaper than duplication: metrics, telemetry, log lines, a “user is typing” indicator. Dropping one data point is fine; processing one twice might not be.

At-least once

Process first, then ack. The consumer does the work, and then acknowledges. If it crashes after processing but before the ack, the broker assumes failure, and redelivers the message, so the work happens again. Messages are never lost, but can be duplicated. This is the default for almost any broker, and the correct default, because losing messages silently is usually worse than processing them twice. The price of this is that our consumer must be ready to see the same message more than once. Hold that thought; it’s the crux.

Exactly-once

Each messages takes effect once and once, no loss, no duplication. This is what everyone wants and what the next section is entirely about, because the short version is: exactly-once delivery is impossible, and exactly-once is achieved somewhere other than where people think it is.

Guarantee	Ack timing	Can lose?	Can duplicate?	Use when
At-most-once	Ack before processing	Yes	No	Loss is acceptable (metrics, telemetry)
At-least-once	Ack after processing	No	Yes	Loss is unacceptable (the common default)
Exactly-once	(see below)	No	No	We need both, and will work for it

The decision between at-most-once and at-least-once is therefore not about quality. It is a direct question: for this message, is it worse to drop it or do it again? The answer to this question per message type decides our guarantee.

Why Exactly Once is So Hard

❝

Exactly-once delivery over an unreliable network is provably impossible.

If the above statement surprises you, keep reading. Every system advertising “exactly-once” is doing something more subtle that the words suggest.

The impossibility is the Two Generals Problem. Two parties communicating over a channel that can drop messages can never become certain they agree, because the last message is always unconfirmed. Confirming it requires another message, which is also unconfirmed, forever. Map it into a queue: the broker sends a message and waits for an ack. If it gets no ack, it cannot know whether the message was lost, the processing failed, or only the ack was lost. Its only two options are resend (risking a duplicate; at-least-once) or don’t resend (risking a loss; at-most-once). There is no third option where it magically knows the truth. No amount of cleverness defeats this; it’s a property of unreliable channels, not of insufficiently good engineering.

So if delivery can’t be exactly-once, what does “exactly-once” mean when Kafka and others advertise it? The resolution is a distinction that, once we internalize it, dissolves most of the confusion:

❝

Exactly-once delivery is impossible. Exactly-once processing is achievable. The trick is to allow at-least-once delivery (so nothing is lost) and then ensure that duplicate deliveries have no additional effect.

We don’t stop the duplicates from arriving. We make the duplicates harmless. The message may be delivered two or five times, but the observable effect on the system happens exactly once. That’s the entire game, and there are two roads to it.

Road 1: idempotent consumers

If processing the same message twice produces the same result as processing it once, then duplicates are free. This is the idempotency issue applied directly: the consumer carries the burden, the broker stays simple (at-least-once), and correctness comes from the operation being safe to repeat. This is the more general and more portable road, because it works with any at-least-once broker.

Road 2: atomic consume-process-produce (transactional)

Make the act of consuming a message, doing the work, and committing the result a single atomic transaction, so that either all of it happens or none of it does - and crucially, the offset commit (the record “I have consumed up to here“) happens in the same transaction as the side effect. If the work commits, the offset commits; if it gets crashes, neither does, and on restart it re-reads from the last committed offset with no partial effect. This is what Kafka’s exactly-once semantics (EOS) actually implement, and it’s narrower than it sounds. It works cleanly when the side effect is writing back into Kafka, and gets harder the moment the side effect is an external system (a payment, an email) that is not part of the transaction.

The honest summary is that “exactly-once“ is real, but it is not a delivery guarantee we turn on. It’s an end-to-end processing property we engineer, by combining at-least-once delivery with either idempotency or transactional offset commits. Anyone who tells you their broker gives exactly-once “for free” is either using road 2 within a narrow boundary or hasn’t hit the failure case yet.

Achieving Effectively-Once in Practice

Both roads are worth seeing in code, because the difference between “we have exactly-once“ and “we have a duplicate-charge incident” is a few lines of discipline.

Road 1: the idempotent consumer

Here, we give every message a stable, producer-assigned ID, and have the consumer record which IDs it has already processed. The dedup check and side effect must commit together, or we have just moved the race.

async def handle_message(msg):
    # Stable ID assigned by the PRODUCER, identical across 
    # redeliveries. (A broker-assigned delivery ID changes 
    # on redelivery — useless for dedup.)
    message_id = msg.headers["message_id"]

    async with db.transaction():
        # Dedup check and side effect in ONE transaction.
        already = await db.fetch_one(
            "SELECT 1 FROM processed_messages WHERE message_id = %s",
            message_id,
        )
        if already:
            return  # duplicate

        await do_the_actual_work(msg) # the side effect
        await db.execute(
            "INSERT INTO processed_messages (message_id) VALUES (%s)",
            message_id,
        )
    # Ack only after the transaction commits. 
    # If we crash before the ack, redelivery hits the 
    # dedup check and does nothing. Effectively-once.

This design handles two failure modes. If the consumer crashes after the transaction commits but before the ack, redelivery finds the ID already recorded and no-ops - correct. If it crashes mid-transaction, nothing committed, redelivery does the work fresh - also correct. The whole guarantee rests on the dedup record and the side effect sharing one transaction; split them and we have a window where we have done the work but not recorded it (double-effect on redelivery) or recorded it but not done it (lost work). This is the same idempotency-key pattern the saga article used for compensating transactions.

For external side effects that can’t share the database transaction like charging the card, calling the third party, etc., we push the idempotency to the boundary: pass an idempotency key to the downstream API so it dedupes. Stripe, for instance, takes an idempotency-key header and guarantees the charge happens once per key no matter how many times we retry. We have delegated effectively-once to the system that owns the side effect, which is the only one that can truly enforce it.

Road 2: transactional offset commit

When the side effect is writing back to broker (the classic stream-processing shape: read a topic, transform, write another topic), bind the output write and the input offset commit into one transaction.

producer.init_transactions()
for batch in consumer:
    producer.begin_transaction()
    try:
        for record in batch:
            producer.send("output-topic", transform(record))
        # The offsets are committed AS PART OF the producer transaction,
        # output and "I consumed this" succeed or fail together.
        producer.send_offsets_to_transaction(batch.offsets, consumer.group_id)
        producer.commit_transaction()
    except Exception:
        producer.abort_transaction() # nothing visible downstream; reprocess

Kafka also gives us an idempotent producer (a producer id plus a per-partition sequence number) that dedupes producer-retries, so a producer resending after a network blip doesn’t write the record twice. That solves the producer half of duplication; transactions solve the consume-process-produce half. Neither extends to side effects outside Kafka; for those, we are back on road 1.

We must pick the road where our side-effect lives: inside our own database → idempotent consumer with a shared transaction; inside Kafka → transactional offsets; inside someone else’s API → idempotency key at the boundary. There is no road that makes external effects exactly-once without one of these.

Consumer Groups and Scaling

One producer, one consumer is the toy version. Real systems need many consumers sharing the load, and there are two fundamentally different patterns for that, and brokers differ sharply in which they favour.

Competing consumer groups (work queue)

Here, multiple consumers read from the same queue; each message goes to exactly one of them. We add more consumers to process faster. This is RabbitMQ’s bread and butter and SQS’ default model: the broker hands each message to whichever consumer is free, load spreads automatically, and parallelism is as simple as starting more workers. The catch is that ordering is sacrificed: if two consumers grab two messages, they finish in nondeterministic order, so a strict sequence is not preserved across the group.

Partitioned consumers (consumer group)

This is Kafka’s model, straight from the Kafka paper. This topic is split into partitions, and within a consumer group, each partition is assigned to exactly one consumer. Parallelism equals the number of partitions: ten partitions can feed to ten consumers, but an eleventh consumer in the group sits idle with nothing to read. The good thing is that ordering is preserved within each partition (one consumer, one partition, in order) while we still scale across partitions. Each consumer tracks its position with a committed offset, so on restart it resumes exactly where it left off.

# Kafka-style: partition count is the parallelism ceiling.
# 10 partitions -> at most 10 active consumers in the group.
consumer.subscribe(["orders"], group_id="fulfillment")
for record in consumer:
    process(record)
    consumer.commit(record.offset) # "I've consumed up to here"

There are two operational realities of consumer groups

1. Rebalancing

When a consumer joins or dies, the group reassigns partitions. During the rebalance, consumption pauses briefly and in-flight work may be reprocessed (redelivery; at-least-once again), which is one more reason the consumer must tolerate duplicates. Frequent rebalances (from flapping consumers or long processing pauses that trip the heartbeat) are a common, self-inflicted performance problem.

2. Fan-out vs work-share

Sometimes we want every consumer to see every message (a price update that three different services each react to), not one-of-N. That is pub/sub fan-out, achieved with separate consumer groups (in Kafka, each group gets the full stream) or separate queues bound to an exchange (RabbitMQ). Confusing “share the work” with “everyone gets a copy” is a frequent design error; they are different topologies.

❝

Mental Model

Competing consumers maximize throughput and forfeit order; partitioned consumers preserve order within a partition at the cost of capping parallelism at the partition count.

Ordering and Partitioning

“Does the queue preserve order?” is the wrong question. The right question is over what scope, because total ordering and horizontal scaling are fundamentally in tension.

A single queue or a single partition can guarantee a total order. But it does so by funnelling everything through one consumer, which means we cannot scale past what one consumer can handle. The moment we add a second consumer for throughput, we have given up total order, because two consumers working in parallel finish in non-deterministic order. We can have a global ordering or horizontal scaling, not both. Most systems don’t actually need global ordering; they need ordering per entity.

This is what partition keys buy us. We don’t need all events in order; we need all events for a given order id, at a given user, or a given account, in order, and we don’t care about relative order across different entities. So, we hash the entities to a partition (events for order #123 always land on the same partition, preserving their sequence), while different entities spread across partitions for parallelism. We choose a partition key with high cardinality and even traffic, whose values matches our ordering boundary, and a hot key will overload its partition just as hot key overloads a shard. The same principle, the same trap.

# Order is guaranteed only WITHIN a partition. Key by the entity
# whose sequence we must preserve.
producer.send("orders", key=order_id, value=event)
# All events for one order_id -> same partition -> processed in order.
# Different orders -> different partitions -> parallel, order-independent.

The subtle ordering trap that catches people even with a single partition: retries reorder messages. If a producer sends message A, it fails, and meanwhile message B is sent and it succeeds, then A is retried and landed after B; out of order, on the same partition. This is why Kafka’s idempotent producer matters beyond dedup: it preserves order under retries by tracking sequence numbers, so a retried message slots back into its correct position rather than tailing whatever arrived during the retry. If we run a producer with multiple in-flight requests and retry without idempotence, we can reorder a single partition. This is a bug that’s nearly impossible to reproduce on demand.

The takeaway is: decide the ordering scope explicitly (none/per-key/total), pick a partition key that matches it, and turn on producer idempotence so retries don’t betray the order we carefully arranged.

Dead Letter Queues and Poison Messages

At-least-once delivery has a dark corner: what about a message that fails every time? A malformed payload, a reference to a deleted record, a bug that throws on specific input. Under naive at-least-once, the broker redelivers it, the consumer fails it, the broker redelivers it again - forever. This is a poison message, and left unhandled it does two ugly things: it spins an infinite retry loop burning resources, and in ordered partition, it cause head-of-line blocking, where one stuck message halts every message behind it. One bad record freezes the whole partition.

The solution is Dead Letter Queue (DLQ): after a message has failed some bounded number of times, we stop retrying it and move it to a separate queue for later inspection, so the main flow can continue.

MAX_DELIVERIES = 5

async def handle_with_dlq(msg):
    delivery_count = int(msg.headers.get("x-delivery-count", 0))
    try:
        await process(msg)
        await msg.ack()
    except TransientError:
        # Probably recoverable (downstream blip): let it be redelivered,
        # ideally with backoff so we don't hammer a struggling dependency.
        await msg.nack(requeue=True)
    except Exception as e:
        if delivery_count + 1 >= MAX_DELIVERIES:
            # Poison: stop retrying, move aside, UNBLOCK the queue.
            await dlq.publish(msg, reason=str(e), failed_at=now())
            await msg.ack()   # ack so the broker stops redelivering it
        else:
            await msg.nack(requeue=True)

The design points that separate a real DLQ from a footgun:

Distinguish transient from permanent failures

A downstream timeout (transient) deserves retry with backoff; a schema-validation error (permanent) should go straight to DLQ because retrying it 5 times just delays the inevitable and blocks the queue. Treating all errors identically wastes the retry budget on the unrecoverable.

Backoff between retries

Immediate redelivery of a transient failure hammers the dependency that’s already struggling. This is the same thundering-herd reasoning behind exponential backoff and jitter from the saga and latency articles earlier.

An unmonitored DLQ is a data-loss bug

Messages in the DLQ are work that didn’t happen. We should have an alert on DLQ depth, and have a redrive path: a deliberate, often manual, re-injection of fixed messages back into the main queue after we have addressed the root cause. SQS and others build redrive in; if our broker doesn’t, we need to build it ourselves.

DLQ entries need context

We must store the failure reason, the timestamp, the delivery count, and the original message. A DLQ full of payloads with no diagnostics is a pile we can’t act on.

The DLQ is where at-least-once meets reality: It is the pressure-release valve that keeps one bad message from taking down the whole pipeline, and the audit log of work our system couldn’t complete.

Choosing a Broker: Kafka vs RabbitMQ vs SQS

The three guarantees are available, in some form, from all major brokers; but the brokers are built around different philosophies, and the right choice falls out of what we actually need. The deepest divide is smart broker / dumb consumer vs dumb broker / smart consumer.

RabbitMQ (smart broker)

A traditional message broker (AMQP) where the broker does the routing work: exchanges route messages to queues by flexible rules, messages are pushed to consumers, and a message is deleted once acked. It excels at task queues, RPC, and complex routing: when we need messages dispatches to the right worker by sophisticated rules and consumed once. Its model is competing consumers, so it scales work-sharing trivially. Its weaknesses: it’s not built for long retention or replay (acked messages are gone), and it tops out at lower throughput than a log-based system. We should use it when the job is “distribute tasks to workers with routing flexibility”.

Kafka (dumb broker, smart consumer)

A distributed, partitioned, retained log. The broker just appends messages to partitions and keeps them for a retention period; consumers track their own offsets and can replay history. It excels at high throughput, event streaming, replay, per-partition ordering, and stream processing with exactly once (EOS) transactions. The price of this operational complexity (partitions, consumer groups, offset management, more to run and tune) and that parallelism is capped by partition count. We should use it when the job is “a durable, replayable stream of events that many independent consumers process, possibly reprocessing history”.

SQS (managed default)

AWS’ fully managed queue, near-zero ops. There are two flavors: Standard (at-least-once, best-effort ordering, effectively unlimited throughput) and FIFO (strict ordering and built-in dedup within a 5-minute window, at much lower throughput). It has DLQ and redrive built in. Its limits are: no replay (messages are deleted after consumption), no rich routing, and a visibility time-out model rather than partitions. We should reach for it when we are on AWS, want at-least-once work distribution without operating anything, and don’t need replay or per-key ordering at scale.

	RabbitMQ	Kafka	SQS
Model	Smart broker, push, routing	Retained partitioned log, pull	Managed queue, pull
Consumer pattern	Competing consumers	Partitioned consumer groups	Competing consumers
Ordering	Per-queue (lost across consumers)	Per-partition	FIFO queues only
Replay / retention	No (deleted on ack)	Yes (retention window)	No (deleted on consume)
Throughput	Moderate	Very high	High (Standard) / low (FIFO)
Exactly-once	Via idempotent consumers	EOS within Kafka + idempotency	FIFO dedup window + idempotency
Ops burden	Moderate (self-hosted)	High (self-hosted)	None (managed)
Best for	Task queues, RPC, routing	Event streams, replay, high volume	AWS-native, simple, hands-off

The decisions is mostly 4 questions:

Do we need replay? (only Kafka, really).

Do we need per-key ordering at scale? (Kafka partitions or SQS FIFO).

Do we need rich routing to workers? (RabbitMQ)

Do we want to operate nothing? (SQS).

Takeaway

The three delivery guarantees are not a quality ladder with exactly-once at the top. They are three answers to one unavoidable question: when we can't tell whether a message was processed, do we risk losing it or doing it twice?, and the network forces us to pick. At-most-once risks loss. At-least-once risks duplication. There is no option that risks neither, because the Two Generals Problem says certainty over an unreliable channel is impossible.

Which means the real work is building consumers that don't care how many times they see a message. That's idempotency, and it's the load-bearing concept under everything here: dedup keys, transactional offset commits, idempotency keys at external boundaries, all of it is "make the repeat harmless." The brokers differ in throughput, ordering, replay, and routing, and those differences should drive our choice. But the safety of the whole system comes from our side of the wire, not theirs.

Distributed systems don't hand us transactions across boundaries; they hand us primitives: at-least-once delivery, idempotency keys, dead letter queues, partitioned ordering, compensating actions, that we compose into something that behaves correctly enough for the business. The message queue is one more place where the infrastructure gives us "at least once" and our discipline turns it into "exactly once, in effect".

What's the most expensive duplicate (or lost message) you've debugged? I am collecting stories: the double charge, the email that went out five times, the order that silently vanished into an unacked void. The failure modes are remarkably consistent across stacks, which is exactly why they're worth knowing in advance.

Hit reply. I read everything.

Namaste!

If this clicked, forward it to whoever just added a queue to your architecture: the delivery-guarantee conversation is much cheaper to have before the duplicate-charge incident than after. And if you want more deep dives like this, subscribe to The Main Thread: practical distributed systems engineering, one essay per week.