One Payment. Two Regions. Three Events. Designing an Idempotent Payment Architecture with Cosmos DB
One Payment. Two Regions. Three Events. Designing an Idempotent Payment Architecture with Cosmos DB
A customer clicks Pay once.
The payment provider processes $250 successfully.
But inside our distributed system, something very different happens:
Event #1 reaches Region A. Event #2 appears because the acknowledgement was lost. Event #3 is produced during a retry.
Meanwhile, Region B is also active.
Now ask the architectural question:
How do we guarantee that one business payment produces one business outcome—even when the infrastructure delivers the event multiple times?
This is where Event-Driven Architecture becomes more than simply:
API → Service Bus → Consumer → Cosmos DB
The Failure Scenario
Imagine this flow:
Customer
│
▼
Payment API
│
│ PaymentSucceeded
▼
Azure Service Bus
│
├───────────────┐
▼ ▼
Region A Region B
Consumer Consumer
│ │
└───────┬───────┘
▼
Cosmos DB Region A receives:
PaymentSucceeded(PaymentId=P10001)
It updates Cosmos DB successfully.
But before the consumer completes the Service Bus message, the process crashes.
From the application's perspective:
The database write succeeded.
From the message broker's perspective:
Processing was never acknowledged.
So the message can be delivered again.
Azure Service Bus Peek Lock provides at-least-once delivery, which means redelivery is an expected condition that consumers must be designed to handle. Microsoft specifically recommends idempotent processing for this reason.
The second consumer receives the same payment.
Without protection, we could execute:
Payment balance + $250
Invoice update
Reward points
Receipt generation
Accounting event
Notification twice.The problem isn't duplicate messages.
The problem is duplicate business effects.
Idempotency Must Be a Business Boundary
I would not make the message broker the only idempotency mechanism.
Instead, give every business operation a stable identity:
PaymentId = P10001
EventId = EVT-84721
Operation = PaymentCompleted
AggregateId = ORDER-50071Then the consumer effectively asks:
Have I already applied
PaymentCompleted
for PaymentId P10001?If yes:
Acknowledge the event.
Do not execute the business operation again.
If no:
Apply the state transition and record that the event was processed.
That changes the architecture from:
Receive → Updateto:
Receive
↓
Validate Event Identity
↓
Validate Current Business State
↓
Conditional State Transition
↓
Record Processed Event
↓
Publish Next Event
↓
AcknowledgeBut What If Two Regions Execute at the Same Time?
This is where Cosmos DB architecture matters.
Every Cosmos DB item has an _etag, and Cosmos DB supports optimistic concurrency control using conditional writes. If another process has already changed the document, a stale conditional update can be rejected instead of silently overwriting the newer state.
So instead of:
UPDATE Payment
SET Status = 'Completed'think:
Change:
Pending
↓
Completed
ONLY IF
Version = ExpectedVersion
AND
Payment has not already been completedThe business state itself becomes part of your concurrency protection.
Multi-Region Changes the Discussion
Active-active infrastructure does not mean every region should blindly update the same business entity simultaneously.
Cosmos DB multi-region writes can commit locally and reconcile writes across regions, so concurrent updates require an intentional conflict strategy. Microsoft also recommends minimizing conflicts rather than sending the same write independently to multiple regions.
For critical financial aggregates, I often prefer the principle:
Active-active compute. Controlled ownership of the business aggregate.
Both regions can serve traffic.
But a specific PaymentId should have a predictable ownership/idempotency strategy.
For example:
Partition Key
│
▼
PaymentId / AccountId
│
▼
Deterministic Business OwnershipMulti-region availability should not become multi-region ambiguity.
The Other Failure Architects Must Solve
There is another dangerous gap:
1. Cosmos DB updated successfully
2. Application crashes
3. PaymentCompleted event was never publishedNow your database says:
PAID
But downstream services never know about it.
This is the classic dual-write problem.
A Transactional Outbox changes the model.
Cosmos Transaction
│
├── Update Payment
│
└── Store PaymentCompleted Event
│
COMMIT
│
▼
Cosmos Change Feed
│
▼
Event Publisher
│
▼
Azure Service BusAzure's documented Cosmos DB outbox approach combines transactional batches, the Cosmos DB change feed, and a message broker so the business state and event can be persisted together before asynchronous publishing. Cosmos DB transactional batches operate within the same logical partition.
My Architecture Rule
For critical event-driven workflows, I design around several assumptions:
Messages will be duplicated. Consumers will crash. Networks will timeout. Regions can disagree temporarily. Retries will happen. Events will eventually need replay.
So the architecture must make all of those conditions safe.
My preferred defense is:
**Idempotency Key
- Business State Machine
- Optimistic Concurrency
- Transactional Outbox
- Broker Duplicate Detection
- DLQ
- Replay
- Reconciliation
- Full Observability**
Service Bus duplicate detection is useful, but it should be defense in depth, not the only protection. Microsoft's guidance explicitly notes that duplicate detection does not replace idempotent receive-side processing.
The Architectural Goal
The goal isn't:
“Make sure the event is delivered only once.”
In distributed systems, that is often the wrong thing to depend on.
The better goal is:
“No matter how many times the event arrives, the business outcome happens only once.”
That is the difference between simply implementing Event-Driven Architecture…
…and architecting a resilient financial system.
#SoftwareArchitecture #EventDrivenArchitecture #Azure #CosmosDB #AzureServiceBus #DistributedSystems #Microservices #CloudArchitecture #Idempotency #SystemDesign #FinTech #SolutionArchitecture
Comments
Post a Comment