Solving the dual write problem with the outbox pattern

In an architecture based on microservices, there must be a way for different services to communicate with each other to signal that something has happened in service A which requires further action in service B.

The most basic form of communication is through synchronous API calls. For instance, consider the following example architecture:

Here, the UserService updates its local database to insert the new user (1) and makes a synchronous call to the EmailService (2), which is responsible for sending a welcome email to the user. The same pattern would apply to any services that need to react to the creation of new users.

Though simple and straightforward, this approach has a few important drawbacks:

• If B (the service called) is down at that moment, it will not be able to receive the message, forcing A to retry the operation until it eventually succeeds.
• If B is degraded and experiencing delays for any reason, A will wait for a long time before considering the operation successful and moving on.
• Each service needs to be aware of how to reach all other services it depends on.

To overcome these issues, communication between services is often handled asynchronously through messages brokered by an event bus (or message broker), with the idea of decoupling producers (which don’t need to wait for messages to be consumed) and consumers (which can process messages at their own pace). These messages, also called events, encode the information to be transmitted (the payload) plus other metadata like the time of creation and a unique identifier.

The dual write problem

So, you’ve got asynchronous messaging and everything works fine in the happy path. You’re done and ready for a well-deserved vacation, right?

Not quite. Usually, events are fired in response to a change of state in the source microservice, which typically means a change in its local relational database. Going back to the previous example, here’s how an async architecture would look:

In this case, once the event is delivered to the event bus, UserService doesn’t care about who will pick it up and can consider the operation committed. Multiple services can tap into the same event without the original service even being aware of their existence.

Now, what happens if the call to the event bus fails for any reason? You end up in an inconsistent state because the local database gets updated but the corresponding event is not emitted. If you switch the order of operations, things could get even worse, potentially publishing an update that was rolled back in the source database for some reason.

This is the so-called dual write problem: you want to reflect the same update in two different systems in a transactional way — either both calls get committed, or neither does.

Outbox pattern

The outbox pattern is a design pattern used in distributed systems to ensure reliable event publishing and data consistency by atomically updating a database and publishing events to an external system. Instead of directly publishing events to an event bus, the service first persists the event data to an outbox table within its own database, in the same transaction as the actual update. A separate process then reads from this outbox table and publishes the events to the message broker with eventual consistency guarantees.

Now, creating the user and the corresponding event happens at the same time (1), with no risk of getting into an inconsistent state. The outbox consumer then acts as a digital mailman, eventually picking up the event (2) and delivering it to the event bus (3), from where multiple consumers can retrieve it at their leisure (4).

Implementing the outbox pattern in practice

There are several ways to implement the outbox pattern, depending on your technology stack. For example, you can use a polling process that periodically checks the outbox table for new events and publishes them, or leverage change data capture (CDC) tools like Debezium to stream outbox events to your message broker in near real-time.

Here is a minimal example Postgres definition for an outbox table:

CREATE TABLE outbox_events (
    id BIGSERIAL PRIMARY KEY,
    event_id UUID NOT NULL UNIQUE,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    metadata JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
);

The table structure includes:

• id: Auto-incrementing primary key for internal database operations
• event_id: Unique identifier for the event (e.g. a UUID) to ensure idempotency
• event_type: Type of event (e.g. UserCreated)
• payload: JSON payload containing the event data
• metadata: Optional JSON metadata (timestamps, correlation IDs, etc.)
• created_at: When the event was created

A more complex implementation could include status fields to track whether the event was processed and when, error message and retries count to track failed processings, etc.

Trade-offs and considerations

While the outbox pattern solves the dual write problem and increases reliability, it also introduces some complexity. Specifically, it forces you to:

• Handle the possibility of duplicate event delivery and ensure idempotency in downstream consumers (though this is something you’d generally want to do anyway).
• Monitor the outbox processing pipeline and make sure it’s running correctly.
• In case of CDC implementation, endure extra effort on the database’s side to make it reliable. For instance, using Debezium with Postgres requires to enable logical replication which can cause unexpected issues when you stop consuming from a replication slot and also forces you to manage things like replication slot fail-over, which is not always strightforward,

Additionally, there may be a slight delay (eventual consistency, remember?) between the database update and the event being visible to other services, which introduces temporary inconsistencies between states in different services. This is especially true in case the outbox consumer is down or is being restarted.

Conclusion

The outbox pattern is a powerful tool for achieving reliable, consistent event publishing in microservices architectures. By leveraging atomic database transactions and decoupling event emission from business logic, you can avoid the pitfalls of the dual write problem and build more robust distributed systems. As with any architectural pattern, it’s important to weigh the trade-offs and choose the right implementation for your needs.