During system design, we often come across very common problems. It is always good to know their solution in advance so that during an interview we can focus on other aspects of design.
In this series, I will write about common design problems and their solutions. Today I want to discuss the dual-write problem.
The Dual Write Problem
Suppose you work for a company that provides digital wallet services to users. A user is coming to your platform and initiating a request to pay a Wi-Fi bill using wallet balance. Now within our system, we want to deduct money from the wallet balance and put one event in Kafka so we can asynchronously call the API exposed by Wi-Fi provider to pay the bill.
There can be one scenario where you deduct money from the user’s wallet but before you can put the event in Kafka your server crashes. Our system is now inconsistent. This is dual-write problem where we want to write the same data to two different places. We can not use transactions for this task that spans two independent locations (database and messaging platform).
Solutions:
1. Transactional Outbox Pattern
In this pattern, we store two separate copies of data in a single transaction in our database. It requires below steps:
In one transaction we will deduct money from the wallet balance and store the event we want to put in Kafka in another database table (outbox table)
Have a separate process to push data from the outbox table to Kafka. Once the event is pushed successfully into Kafka remove that event from outbox table/mark it as processed.
One problem that can happen during this step is the process can send the event to Kafka but crashes before it can delete the event from outbox table. When we restart the process there can be duplicate events in Kafka but I will cover this in a separate “event deduplication” blog in the same series.
2. Change Data Capture
In the previous pattern, we need one additional service just to push events from the outbox table to Kafka. If your database supports you can use database inbuilt “change data capture” feature that detects row-level changes and sends the change as a message to the configured location.
Please subscribe if you are interested in learning about system design.