Hey everyone, welcome to the fortieth issue of The Main Thread. In this issue, we will learn about how to implement distributed transactions without two-phase commit.
First things first: ACID across services doesn’t work. But there’s something that does.
You must have learned about transactions in your first database course. Begin, do work, commit. If anything fails, roll back, and the world looks like nothing happened. Atomic, Consistent, Isolated, Durable. The database promises it, and the database delivers.
Then you split your monolith into services. Now your “transaction” spans the order service, the payment service, the inventory service, and the notification service. Each one has its own database. The atomicity guarantee that the database gave you for free doesn’t extend across this boundary. There is no BEGIN that wraps all four; there is no COMMIT that makes them all stick or all unwind. The clean abstraction is no more there.
The textbook answer is two-phase commit. The textbook is, in this case, lying to you. 2PC across microservices is a pattern that sounds elegant and falls apart in production for reasons we will cover. The pattern that actually works - the one Garcia-Molina and Salem described in 1987 for long-lived database transactions and the microservice community rediscovered almost three decades later - is the saga.
Saga trades the strong guarantees of ACID for a different model: a sequence of local transactions, each of which is committed independently, with explicit compensating transactions that undo the effects of earlier steps if a later step fails. It is not a free lunch. It moves complexity from the infrastructure into the application code. But for the kind of cross-service workflow that defines modern systems, it’s the only pattern that consistently survives contact with reality.
This article is the pragmatic guide to sagas - what they are, how to implement them in both styles, what compensating transactions actually look like, and the cases where you shouldn’t use them at all.
Why Distributed Transactions are Fundamentally Problematic
Before we talk about sagas, it’s worth being precise about why the obvious alternative fails. Two-phase commit has well-understood pathologies that get worse the more services you add.
The protocol works in two phases. In the prepare phase, a coordinator asks every participant: “are you ready to commit?” Each partition does its work, locks the relevant resources, and replies “yes” or “no”. If everyone says yes, the coordinator sends the commit message in the second phase. If anyone says no, the coordinator sends an abort. In the happy path, this gives us atomicity across heterogeneous resources.
The unhappy paths are where it falls apart.
It’s a blocking protocol. Once a participant has voted “yes” in the prepare phase, it must hold its locks until the coordinator decides. If the coordinator crashes between phase 1 and phase 2, the participant is stuck: it can’t commit (it didn’t get the commit message) and it can’t abort (the coordinator might have already told someone to commit). Locks held, throughput dead.
The coordinator is a single point of failure. We can replicate the coordinator with consensus, but now we have added a Paxos/Raft cluster to the critical path of every cross-service operation. Latency goes up; the operational footprint grows.
It doesn’t scale across organizational boundaries. 2PC requires every participant to implement a specific protocol and trust the coordinator. This works inside a single database supporting XA transactions. It doesn’t work when one of our “participant” is Stripe or Shopify or any external service that won’t expose a prepare/commit interface.
Performance is bad. Every cross-service transaction now requires multiple round-trips, distributed locking, and coordinated commit. Throughput is bounded by the slowest participant in the slowest phase. For high-volume systems, this is a non-starter.
Service availability becomes coupled. If any participant is down during a transaction, the whole transaction blocks. A 99.9% available service composed of five 99.9% available participants with 2PC is now ~99.5% available end-to-end. Service composition multiplies failure rates instead of dividing them.
The deeper problem is that 2PC pretends a network of services is one big database. It isn’t, and the impedance mismatch is paid for in availability and performance. The CAP theorem is doing its usual work in the background: 2PC chooses consistency over availability under partition. For most business flows, that’s the wrong trade-off.
The saga pattern starts from a different premise: accept that we can’t get atomicity across services, and design a workflow that is correct under the consistency model we actually have.
What a Saga Actually is
A saga is a sequence of local transactions where each transaction publishes events or messages that trigger the next. If any step fails, the saga executes a series of compensating transactions that undo the effects of the previously-completed steps in reverse order.
The original 1987 paper by Garcia-Molina and Salem was about long-running database transactions: instead of holding locks for hours, break the work into shorter local transactions and define compensating actions for each. The microservices version applies the same pattern across service boundaries.
For example: an order placement workflow:
T1: Create order C1: Cancel order
T2: Reserve inventory C2: Release inventory
T3: Charge payment C3: Refund payment
T4: Schedule shipment C4: Cancel shipment
T5: Send confirmation (no compensation — email is fine)Each Ti is a local transaction in some service. Each Ci is the semantic undo. If T1, T2, and T3 succeed but T4 fails, the saga runs C3 (refund), C2 (release inventory), and C1 (cancel order). The end state is as if the saga never started, even though the data doesn’t know anything about that.
The following three properties define the model:
Each step commits independently. No locks are held across step boundaries. Other transactions can see the partial results of an in-progress saga. This is a central trade-off where we give up isolation to get availability at scale.
Compensations are semantic. We don’t roll back the database to prior point in time. We write a new transaction that reverses the business effect. Cancel the order. Refund the payment. Restock the inventory. The database history shows both actions; the net effect is undone.
Compensations must be guaranteed to succeed. If T3 succeeded and T4 failed, we must be able to run C3 successfully. If C3 itself can fail in non-recoverable ways, our saga has no safe terminal state. This is one of the design constraints that shapes everything else.
The saga isn’t a transaction in the database sense. It is a workflow that approximates atomicity by being able to clean up after itself. The approximation is good enough for most business processes, and far better than 2PC’s blocking protocol when things go wrong.
Choreography vs Orchestration
There are two architectural styles for implementing sagas, and the choice between them is the most consequential decision we will make.
Choreography: Event-driven, peer-to-peer
In a choreographed saga, there is no central coordinator. Each service listens for events from other services and decides what to do. The saga’s logic is distributed across the participants - the workflow is implicit in the chain of event subscriptions.
Order Service:
receives: PlaceOrderCommand
emits: OrderCreated
Inventory Service:
receives: OrderCreated
emits: InventoryReserved (or InventoryReservationFailed)
Payment Service:
receives: InventoryReserved
emits: PaymentCharged (or PaymentFailed)
Shipping Service:
receives: PaymentCharged
emits: ShipmentScheduled (or ShipmentFailed)
Order Service:
receives: ShipmentScheduled → marks order complete
receives: PaymentFailed → triggers compensation
receives: ShipmentFailed → triggers compensation# Inventory service — responding to events
@event_handler("OrderCreated")
async def reserve_inventory(event):
try:
with db.transaction():
for item in event.items:
db.execute(
"UPDATE inventory SET reserved = reserved + %s "
"WHERE product_id = %s AND available - reserved >= %s",
(item.qty, item.product_id, item.qty)
)
await publish("InventoryReserved", {
"order_id": event.order_id,
"items": event.items,
})
except InsufficientStock:
await publish("InventoryReservationFailed", {
"order_id": event.order_id,
"reason": "insufficient_stock",
})
@event_handler("PaymentFailed")
async def release_inventory_on_payment_failure(event):
# Compensation
with db.transaction():
for item in event.items:
db.execute(
"UPDATE inventory SET reserved = reserved - %s WHERE product_id = %s",
(item.qty, item.product_id)
)
await publish("InventoryReleased", {"order_id": event.order_id})Strengths
No single point of failure. Each service is independent.
Loose coupling at the implementation level. Adding a new participant means subscribing to existing events; no central workflow needs editing.
Naturally fits an event-driven architecture we may already have.
Weaknesses
The workflow is implicit. To understand “what happens when an order is placed”, we have to read every service’s event handlers and reconstruct the choreography in our head. This gets unmanageable past four or five steps.
Cyclic dependencies are easy to introduce by accident. Service A subscribes Service B’s event; Service B subscribes to Service C’s; Service C’s subscribes to Service A’s. The workflow is impossible to reason about without a graph.
Monitoring is hard. There is no single artifact representing “the saga”. Tracing a single saga end-to-end requires correlation IDs and good distributed tracing, both of which teams don’t have until they need them.
Compensation flows are scattered across services. The “what if step 4 fails” logic isn’t in one place.
Orchestration: A central coordinator
In an orchestrated saga, a dedicated orchestrator service tells each participant what to do. The participants don’t know about each other; they know about the orchestrator. The workflow lives in the orchestrator’s code as a sequence of explicit steps.
async def place_order_saga(order_id: str, items: list, customer_id: str):
state = SagaState(order_id=order_id)
try:
await order_service.create(order_id, items, customer_id)
state.mark("order_created")
await inventory_service.reserve(order_id, items)
state.mark("inventory_reserved")
payment_id = await payment_service.charge(customer_id, total(items))
state.mark("payment_charged", payment_id=payment_id)
await shipping_service.schedule(order_id)
state.mark("shipment_scheduled")
await notification_service.send_confirmation(order_id)
state.mark("complete")
except SagaStepFailed as e:
await compensate(state)
raise
async def compensate(state: SagaState):
if state.has("shipment_scheduled"):
await shipping_service.cancel(state.order_id)
if state.has("payment_charged"):
await payment_service.refund(state.payment_id)
if state.has("inventory_reserved"):
await inventory_service.release(state.order_id)
if state.has("order_created"):
await order_service.cancel(state.order_id)This is the simplified shape; in practice, we’d never run an orchestrator inline like this. We’d use a workflow engine (Temporal, AWS Step Functions, Cadence, Camunda) that handles persistence, retries, and recovery. The point is that the workflow is one piece of code.
Strengths
The workflow is explicit. Reading the orchestrator code tells us the entire saga.
Compensation is centralized. The failure-handling logic lives next to the success logic.
Monitoring is straightforward. The orchestrator emits state transitions we can dashboard, alert on, and replay.
Easy to evolve. Adding a step or reordering steps is a code change in one place.
Weaknesses
The orchestrator becomes a critical service. If it’s down, no new sagas start; in-flight sagas may pause.
Risk of orchestrator becoming a smart-pipes fat-orchestrator monolith. Business logic that should live in the participating services migrate into workflow definition.
Tighter coupling. Adding a step usually means modifying the orchestrator and the participant.
Which to choose: the conventional wisdom, and the wisdom that holds up in practice, is that the orchestration is the better default for anything more complex than a 2-3 step linear flow. The “workflow as readable code” property is enormous for debugging, onboarding, and evolution. Choreography wins when the workflow is genuinely event-driven (each step is meaningful business event others care about anyway) and when the number of steps is small.
If you find yourself debugging a choreographed saga by drawing the event graph on whiteboard, you have outgrown choreography. Switch.
Designing Compensating Transactions
The compensating transaction is the central design challenge of sagas, and it is where most people get tripped up. The naive view is “compensation is the inverse of the action“. The reality is that compensation is rarely a clean inverse, because the world has moved on between the original action and the compensation.
Compensations must be semantic
We cannot, in general, restore the database to a prior state. A user might have read the data in between. A downstream system might have acted on it. The shipment might already have been picked.
Instead, we write a compensating transaction that reverses the business effect in a way the domain accepts. Few examples:
Original action | Naive "undo" | Real compensation |
|---|---|---|
|
|
|
|
|
|
| (impossible) |
|
|
|
|
|
| (depends on consumer) |
The “Real compensation” column above matters because in real-world production systems, the database history is part of the truth. A canceled order should look like a canceled order, not like an order that never existed.
Compensations must be idempotent
Compensations run in failure paths. Failure paths get retried. If the compensation is not idempotent, retrying it during a network blip will run it twice and refund the customer twice.
async def refund_payment(saga_id: str, payment_id: str, amount: float):
# Idempotency key derived from the saga, not generated each call
idempotency_key = f"refund:{saga_id}:{payment_id}"
existing = await db.fetch_one(
"SELECT result FROM refunds WHERE idempotency_key = %s",
idempotency_key
)
if existing:
return existing.result
result = await stripe.refunds.create(
payment_intent=payment_id,
amount=amount,
idempotency_key=idempotency_key,
)
await db.execute(
"INSERT INTO refunds (idempotency_key, result) VALUES (%s, %s)",
idempotency_key, result
)
return resultThis is the same idempotent pattern from the earlier issue: applied to the compensation side, where it matters at least as much.
Compensation must be guaranteed to eventually succeed
If a compensation can fail in non-recoverable ways, the saga has no safe terminal state. The system gets stuck in half-undone state, and a human has to intervene.
A few patterns to handle this:
Retry with exponential backoff and jitter, indefinitely. If the downstream system is down, eventually it comes back. We should not give up after
Nretries where “stop” leaves data inconsistent.Dead-letter queues for human inspection. When automated retries exhaust their reasonable budget, we must surface the failure. It can’t silently fail into a log file.
Pivot transactions. Some sagas are designed so that, after a certain point, they cannot be rolled back, they can only roll forward. The “pivot” is the step beyond which compensation is not possible. Past that pivot, every subsequent step must be retried until it succeeds.
The classic pivot example: a money transfer where we have already deducted from the source account and credited the destination. Compensating “credit destination” is fine, but if the destination owner has already withdrawn the money, compensation becomes a debt-collection problem. Many real systems treat the credit as the pivot: once it happens, the workflow rolls forward through any remaining steps no matter what.
Compensation must reckon with what others did with the data
This is the hardest constraint. Between T3 (charge payment) and the failure of T4 (schedule shipment), some other process may have looked at the order, marked it as paid in a customer-facing dashboard, and triggered downstream events. The compensation undoes the payment, but the side effect of others having seen the paid state cannot be undone.
There is no general fix. The patterns the literature has converged on (Caitie McCaffrey's "Dist Sys Programming with Sagas," Pat Helland's broader writing on this) are:
Semantic locks. Mark records as “in-progress” while a saga is running. Other transactions can see the record but skip the in-progress state. Releases when the saga commits or compensates.
Commutative updates. Design state changes so they commute. “Insert a payment record” commutes with “insert a refund record” regardless of the order they are processed in; the net is correct.
Pessimistic view. Don’t expose intermediate state. Show the user “your order is processing” until the entire saga commits, even though the order exists in the database with various intermediate statuses.
Re-read values. When a compensation runs, re-read the current state instead of assuming the state at the time of original action. The product price may have changed; refund based on what was actually charged, not what the original action recorded.
These are not workarounds. They are the actual design discipline of building correct sagas. If we don’t have one of these in place for each non-trivial step, the saga can’t leave the system in an externally-visible inconsistent state, and we will get a customer complaint about it.
Failure Handling and Partial Rollbacks
Real sagas have to handle a richer space of failures than “step succeed” or “step failed“.
Step timed out, status unknown. We sent the request to the payment service, the connection dropped before we got a response. Did the charge go through? We don’t know. We should treat it as: re-issue the same request with the same idempotency key. Either the original succeeded (we will get the same response back) or it failed (we will process it now).
Step succeeded but the orchestrator crashed before recording it. Our orchestrator state shows “in progress on T3”, but T3 actually completed. When the orchestrator recovers, it will retry T3. The participant’s idempotency key, which was derived from the saga ID, makes the retry safe. This is why participant-side idempotency is mandatory.
Compensation itself failed. We should retry it. If retries are exhausted, we can escalate to a dead-letter mechanism. We mustn’t “give up and mark the saga done” - the system is in inconsistent state and we need to know.
The participant is permanently broken. A 4xx response indicating that the requested action is impossible (refund a payment that’s already been refunded, cancel and order that doesn’t exist). We should treat it as a success as the desired end state is achieved.
It would look something like this:
async def execute_step(step_name, action, ctx):
"""Execute a saga step with built-in idempotency and retry."""
if ctx.is_completed(step_name):
return ctx.get_result(step_name)
attempt = 0
while True:
try:
result = await action(ctx)
ctx.mark_completed(step_name, result)
return result
except RetriableError as e:
attempt += 1
if attempt > MAX_RETRIES:
raise SagaStepFailed(step_name, e)
await asyncio.sleep(backoff(attempt))
except NonRetriableError as e:
raise SagaStepFailed(step_name, e)
except AlreadyAppliedError:
ctx.mark_completed(step_name, None)
return NoneThe ctx.is_completed check is the recovery mechanism: when the orchestrator restarts and replays the saga, it skips steps it already completed. The orchestrator’s state is durable (it has to be), and the saga moves forward from where it left off.
This is exactly what workflow engines like Temporal or Cadence give us out of the box. They persist every step transition; on restart, they replay the workflow function and re-execute only the steps that didn’t complete. We can write what looks like a normal Python function and get exactly-once semantics on each step. The cost is a workflow runtime in our stack and a learning curve. The benefit is that the failure handling above becomes infrastructure, not application code.
When to Use Sagas vs. Accept Eventual Consistency
Sagas exist on a spectrum between “fully transactional” (ACID) and “fully eventually consistent“ (no coordination at all). They are not the right answer for every cross-service workflow. Sometimes the right answer is to accept that the system is eventually consistent and design the user experience accordingly.
We should use a saga when:
The workflow has clear failure semantics requiring rollback. “If we charged the customer but couldn’t ship, we must refund“ is a saga. The compensations are well-defined and the consistency requirement is strong.
The workflow involves external systems with their own transactional boundaries. Calling Stripe, then Shopify, then our warehouse - each is its own boundary. We need explicit compensations because there is no shared transaction to roll back.
The number of steps is bounded and finite. Sagas with 3-7 steps are normal. Sagas with 30 steps are a sign we have modeled the workflow too tightly.
We shouldn’t use a saga when:
Eventual consistency is genuinely fine. “The user updated their profile, the search index will reflect it within a minute” doesn’t need a saga. It needs a propagation mechanism (events, change-data-capture) and a UX that doesn’t promise instant search updates.
The “compensations” are functionally meaningless. If undoing the work doesn’t restore any meaningful invariant, sagas are bureaucratic overhead.
One service can naturally own the whole workflow. Sometimes a “distributed transaction” reveals that we have drawn our service boundaries wrong, and what should be one service has been decomposed too aggressively. Re-monolithize the workflow back into a single service with a single database; we should use a real transaction. This is unfashionable advice and frequently the right one.
The workflow is genuinely stateless and idempotent end-to-end. If every step is naturally idempotent and order doesn’t matter, we may not need a saga; we may just need at-least-once delivery and the steps will converge.
The saga pattern’s complexity tax is real. Compensating transactions, idempotency, the workflow engine, the failure-handling matrix - all of it is code our team has to write, debug, and operate. If a simpler model would do, we must use a simpler model.
Takeaways
The saga pattern is the answer to a question many engineers don't realize they are asking: "How do I get something resembling transactional behavior across services that don't share a transaction?" The answer is that we don't. We build a workflow that can clean up after itself, and we accept that the cleanup is application code instead of infrastructure.
This is not a worse architecture than ACID; it's a different architecture, suited to a different problem. ACID transactions excel inside a single database. Sagas excel across systems with independent failure modes, including external systems we don't control. Trying to bring 2PC across that boundary fights the network and loses; sagas embrace what the network is and design around it.
The real skill with sagas is writing the compensations, knowing where the pivot is, and being clear about which intermediate states are externally visible.
The deeper lesson connects to the earlier issues in this series: idempotency, event sourcing, and now sagas are facets of the same underlying truth. Distributed systems don't give us transactions, but they give us primitives like append-only logs, idempotency keys, retry mechanisms, compensating actions, that we can compose into something that behaves transactionally enough for the business. The art is in the composition.
ACID across services is a fairy tale. Sagas are how grown-ups ship distributed systems.
What's the most painful saga you've debugged in production? I am collecting stories: orphaned compensations, infinite retry loops, sagas that ran for days before someone noticed. The patterns are educational and the failure modes are surprisingly consistent across companies.
Hit reply. I read everything.
Namaste!
If this clicked, forward it to your team. Sagas are one of those patterns that look academic until the day you really need them. And if you want more deep dives like this, subscribe to The Main Thread - practical distributed systems engineering, one essay per week.
