Transactional Outbox: From Idea to Open Source

Misha Merkushin leads the Ruby Platform team at SberMarket, which is responsible for internal libraries and improving the architecture of Ruby microservices. This article covers the Outbox architectural pattern and a tool the team built, developed iteratively in-house, and only recently released. It solves the problem of reliable, consistent message delivery from an application, guaranteeing that messages go out only after the database transaction commits successfully.

• About the Outbox pattern
• How the team got to Outbox: a Rake task, Sidekiq + Schked, and job overlap
• Moving orders from async HTTP to Kafka and scaling the solution
• New architecture: polling and processing in separate pools

Looking for reliable delivery

The team had a classic architecture: a monolith acting as the storefront for shoppers, plus a separate microservice for the back office that serves order pickers. Shoppers filled a cart in the storefront and placed orders, which were then sent to the back office over a REST API using asynchronous Sidekiq jobs.

Even after giving up synchronous distributed transactions, the team ran into a problem with lost orders, which inevitably degraded the customer experience. Sidekiq does have retries, which let a job be attempted again when a network failure occurs, but that does not solve the core problem.

The core problem is how the background job is scheduled. If the job is queued before the transaction commits in the database, there is a risk of sending raw data to the back office, because there is no guarantee that the transaction itself will commit successfully. On the other hand, if the job is queued right after the transaction commits, the application process may die before the job is queued — when it exceeds its memory limit, for example, and the process is killed. The order is then lost and the customer is left with nothing.

Using the Outbox pattern

The transactional outbox pattern is an architectural pattern used in distributed systems to guarantee reliable message delivery. It persists messages to a data store (usually an outbox table in the database) before they are finally handed to the message broker. This guarantees that data is not lost: either everything commits, or a failure rolls the whole thing back.

1. When the application receives a request to create an order, it opens a database transaction in which the order is stored. Inside the transaction, a message for another service is created and stored in the database in a dedicated outbox table. The transaction is then committed.
2. A separate process then periodically polls this table for new rows.
3. That process picks up messages from the outbox table and starts processing them — in this case, sending them to the message broker.

An outbox table usually contains the fields:

• primary key
• payload (the message body)
• and statuses that help tell what is currently happening with the message.

If there is a network problem or the broker is unavailable, the attempt is retried, so the message is not lost.

The pattern guarantees at-least-once message delivery: messages are delivered at least once, but may be delivered more than once, which can produce duplicates. That trade-off buys reliability in an unstable network where messages can go missing. In code, it looks roughly like this:

Order.transaction do
  order.complete!
  OrderOutboxItem.create(order)
end

When to use Outbox

Picture a large distributed system of a hundred services. The services exchange messages with each other, receiving change events to implement their own business logic. Reliability requirements for message delivery between services are largely driven by that business logic. Two examples make the point.

A courier is delivering an order. The system tracks the courier's coordinates and sends them to the customer, so that accurate time estimates can be shown. The data flow is huge, because the courier's position changes constantly. But losing a few messages changes nothing.

In contrast to the first, here is another example:

Accepting an order for pizza delivery from a restaurant. To deliver a pizza, it first has to start being cooked. The sooner that signal arrives, the better:

• If a new order arrives late, the customer does not get the pizza on time.
• If the message is lost, the customer does not get a pizza at all. The company loses both the customer and the money.

This article focuses on the second case, where the requirements for reliable data delivery are high and a lost message seriously breaks the business logic.

How the team got to Outbox

At SberMarket there are a number of services written in Ruby. Working on the project, the team needed a solution that was easy to use and, at the same time, scalable, to keep data flowing reliably between services. The work went through several stages, each with new problems to solve and new difficulties to overcome. It started from basic approaches and grew more complex over time. Walking through these stages helps build a deeper understanding of the outbox pattern and avoids some of the traps the team hit during development.

Streaming store settings

A store is the key entity of any marketplace. SberMarket is a delivery service, so the team works with all kinds of stores, each with a set of parameters. For example:

• a store may or may not be available;
• a store may operate during certain hours.

In technical terms, that means store setting changes from one microservice are sent to dozens of others. The SberMarket system has thousands of stores, and their settings are updated and refined regularly. The traffic is small but constant. The important part is that setting changes must reach consumers without loss, while a delay of up to 10 minutes is acceptable. That is why the team chose the Outbox pattern for this task, giving it reliable and safe data transfer.

Rake task

To validate the concept in practice, the team took the path of least resistance and used a Rake task activated on a Cron schedule. The idea was for the task to run every minute, collect data about recently placed orders as well as orders that had not been sent earlier for whatever reason (network errors, for example), and push them to the message broker's queue.

task :publish_store do
  StoreOutboxItem.pending.find_each(&:publish!)
end

For a PoC the scheme works, but:

• application boot takes a long time;
• the scheme does not scale, because everything runs in a single thread.

Sidekiq + Schked

Starting a fresh Ruby process every time is slow. Better to use the Schked job scheduler: schedule a regular task to send messages to the broker once a minute, with the job processed by Sidekiq.

every "1m" do
  StoreOutboxItemsPublishJob.enqueue
end

class StoreOutboxItemsPublishJob < ApplicationJob
  def perform
    StoreOutboxItem.pending.find_each(&:publish!)
  end
end

As a result, jobs start instantly, without the delays of environment initialization. But even with that improvement, scaling the system remains hard. One job may start while the previous one has not finished yet, which can break the order of order processing.

To avoid chaos, the team runs Sidekiq in single-threaded mode so jobs execute strictly one after another. That solves the parallel execution problem, but at the same time keeps Sidekiq from processing jobs in bulk and at speed.

Since the scaling problem was still unsolved, it was time to look at Kafka. A Kafka topic is the equivalent of a message queue. The topic itself is split into partitions, which is how the message queue is parallelized and throughput is scaled. When implementing message sending logic, the application decides which partition a message goes into, so events related to the same order can be routed to one partition. A single process consuming messages from one partition gives parallel processing with ordering preserved.

The idea was to parallelize event sending across partitions and do it in different threads. To make that work, the partition has to be computed in advance when the outbox item is saved.

Accordingly, Schked enqueues a Sidekiq job to send messages to a specific partition.

every "1m" do
  PARTITIONS_COUNT.times do |partition|
    StoreOutboxItemsPublishJob.enqueue(partition)
  end
end

That gives parallel sending across different partitions.

Overlapping jobs

As noted earlier, the current job scheme has a flaw: jobs can accumulate and overlap. The system can end up unable to process all events within the allotted time. To prevent that and get smoother, more reliable job handling, a popular tool helps: the sidekiq-uniq-jobs gem. It blocks a new job from starting if a previous job with the same unique identifier is still running.

class StoreOutboxItemsPublishJob < ApplicationJob
  sidekiq_options lock: :until_executed

  def perform(partition)
    StoreOutboxItem.publish!
  end
end

The scheme has a lot going for it:

• It relies on standard tools, which simplifies support and deployment.
• Good observability, since all the metrics are there by default.
• Scales well. When more parallelism is needed, add partitions to the Kafka topic and raise the Sidekiq queue concurrency.

Despite these advantages, the system stays complex, with many potential points of failure.

Everything worked more or less until the day it broke... An experienced Rubyist would point at the weakest link right away 🙂. One fine day, sidekiq-uniq-jobs got stuck. The next challenge was to reduce the number of failure points, clearing the way for solving more large-scale problems.

Moving orders from async HTTP to Kafka

So there were two systems — the storefront and the back office — actively exchanging order information at about 100 messages per second.

Historically, data exchange between the systems used a combination of Sidekiq and HTTP. That approach held up while the data volume stayed moderate. As load grew, problems with message ordering, scaling difficulty, and message loss became obvious. On top of that, the system was not extensible: adding new consumers required changes to the monolith.

The Ruby Platform team decided to move the order synchronization mechanism to Kafka, since delays in exchanging order information were unacceptable. Continuing with the Schked/Sidekiq/Uniq-jobs scheme clearly could not provide the required reliability and performance.

That stack would only take the team so far, so the decision was to build a custom outbox daemon, reducing the number of failure points to one. The daemons had to start independently, be replicable, and be interchangeable — so that one failing does not affect the others.

The new daemon's concept was simple and effective: a separate process runs and handles messages for all Kafka partitions in multithreaded mode. Coordination between processes happens over Redis using the RedLock algorithm, locking a specific partition and avoiding conflicts during data processing.

In the new architecture, one Ruby process serves the whole system, which simplifies maintenance and operations. Scaling happens by adding processes and threads, which lets the system adapt to growing load.

Safe migration to Kafka

The original task was to replace the Sidekiq+HTTP transport with Kafka, with maximum stability of the transition as the priority. The decision was to run the Kafka transport in parallel and migrate gradually from one solution to the other. That required the outbox pattern to support multiple transports.

The plan was:

1. First, the existing Sidekiq+HTTP implementation was modified slightly to work with Outbox.
2. Then order sending was duplicated through Kafka.

Comparative performance metrics (for example, which transport processed orders faster) confirmed that the new solution was better. The old synchronization system was then turned off, leaving only the new Kafka transport in production.

This approach let the team build and debug the new synchronization process without risking the running system. Evolutionary migration with minimal risk and no downtime is a strategy that applies not only to swapping transport protocols but also to breaking monolithic applications into smaller, manageable services. The experience shows that a gradual, controlled transition is an effective way to introduce changes into critical systems.

Observability and Tracing

The previous Sidekiq-based version of outbox message processing had built-in metrics such as yabeda-schked and yabeda-sidekiq, which gave good system observability without extra effort. Building a custom outbox message daemon from scratch, however, meant implementing all the observability metrics by hand, including metrics and distributed tracing.

Once the new outbox daemon went live, it became clear that its job was not limited to sending messages. To match the observability level the team had with Sidekiq, a metrics and tracing system had to be integrated, so that performance could be tracked, bottlenecks identified, and problems addressed quickly.

During development and rollout of the new daemon, particular attention went to these aspects:

• Performance metrics. Setting up collection of the number of messages processed, processing time, errors, and other key indicators that show how efficiently the system runs.
• Distributed tracing. To understand the execution flow and interaction between the different system components, and to simplify debugging and improve performance, distributed tracing was integrated. This made it possible to follow each request through all the microservices.

So while the new daemon delivered higher stability and scalability, it also required extra observability work to keep service quality high.

Inbox Pattern

Now look at the situation from the message receiver's side, which faces similar challenges. Working with Kafka, for example, there are no built-in guarantees of exactly-once delivery semantics. To solve that, the Transactional Outbox concept extends to a Transactional Inbox, where the mechanism is mirrored and applied on the consumer side.

To learn more about exactly-once in Kafka, the author recommends this talk by his colleague Artem Kuleshov.

When a message arrives, it is stored in a dedicated inbox table in the database. With Kafka, this is done by the inbox-kafka-consumer, which records the offset of processed messages inside a transaction.

A separate inbox process picks events from the database and triggers the business logic for each one. It is similar to the outbox process, except that instead of sending a message to the broker it runs the business logic for the incoming message.

This scheme is essentially a mirror of the outbox pattern, but here the consumer is what creates records in the database. The event polling mechanism from the database stays the same as in the outbox pattern. During processing, though, instead of sending a message to the broker the way outbox does, it runs the business logic meant to handle the received message. This gives exactly-once delivery semantics: inbox records are created with the same unique identifier (UUID) used for the outbox records, and a unique index on that column in the database enforces it.

This scheme pays off when extra guarantees are needed that a message will be processed correctly by the consumer.

Scaling

After implementing the inbox pattern, the next problem was making it scalable. In this architecture, receiving (consuming) messages is conceptually separate from actually processing them.

Consuming messages from Kafka is fairly fast, since it amounts to storing the received data in the database. Processing messages, which runs the business logic, takes considerably longer. That speed difference lets consumption and processing scale independently. For example, if the consumer has enough throughput to write every incoming message but the business logic cannot keep up with the stream, processing can be scaled on its own by adding threads that run the business logic.

Adding processing threads beyond the number of partitions is pointless, because only one message from one partition can be processed at a time. The extra threads would sit idle waiting for data. That is because scaling here is based on processing messages from different partitions in parallel.

So there is a limit: the scalability of message processing is bounded by the number of partitions in the Kafka topic. That becomes a problem when the message processing logic is resource-hungry and adding Kafka partitions is difficult for some reason.

Virtual partitions

Previously, scaling the outbox/inbox daemon meant a series of awkward steps: first increase the number of partitions in Kafka, restart the message producer, and then restart the outbox daemon itself. The approach was clunky and involved extra manual work.

To fix this, the team introduced the concept of a "bucket". When an outbox/inbox record is created, it now stores a bucket number instead of a partition number. The number of buckets is bounded and computed the same way as a partition number — as the remainder of dividing the event key. Several buckets, however, can be mapped to one Kafka partition.

That approach lets a large number of buckets be created up front while the number of Kafka partitions is adjusted dynamically, without restarting the whole system. The outbox/inbox daemon can now be scaled by changing configuration alone, which simplifies operations considerably.

The work solved the problem of streaming completed orders with high reliability and minimal latency. The following improvements were delivered:

• A custom outbox/inbox daemon: this gave the team full control over the message processing pipeline and made it possible to implement the necessary optimizations.
• Scaling outbox by the number of Kafka partitions: system throughput can be adjusted flexibly by changing the partition count.
• A reliable inbox pattern: every message is processed exactly once, which is critical for business processes that cannot tolerate data loss.
• Independent scaling through virtual partitions (buckets): the system scales without changing Kafka configuration or restarting producers. The result is a flexible, scalable, and fault-tolerant message processing system that handles heavy loads and delivers important data reliably.

Rolling out the solution

The Ruby Platform team's global goal was to create and standardize a universal message handling solution that all Ruby services could use. A well-designed solution would let product teams spend less time on technical problems and focus on product development and business logic.

The toolkit had to work with different Ruby services, adapt to different loads, and come with a full set of monitoring tools. On those terms, the outbox-based solution had reached a certain level of maturity:

1. It had been adopted successfully in several Ruby services.
2. It ran steadily.
3. It had good observability and scaling capabilities. But at wider adoption the team hit some difficulties. Despite independent scaling, tuning the daemon to a specific load meant calculating the ratio between several parameters:

• Number of Kafka partitions.
• Number of buckets.
• Number of daemon processes.
• Number of threads in each process.

That configuration was too complex for users, forcing them to understand the technical details of the implementation. Users often set too few partitions and too many daemon processes, which lowered performance. For mass adoption, the configuration had to be simplified and scaling made more intuitive and straightforward.

New architecture

A quick recap of how the daemon works: each thread sequentially polls the database for new messages in the buckets assigned to it. Once a message is received, it is processed (processing), which runs the business logic. Polling is fast, while processing a message can take a long time.

Originally, the unit of scaling for business logic in the daemon was a thread that did both polling and processing of messages. Scaling therefore increased the intensity of both operations, even when that was not needed. If the business logic started to slow down, for example, the thread count went up, which in turn increased how often the database was polled — even when the bottleneck was in processing rather than receiving messages. The result was over-scaling.

More threads meant proportionally more database queries. The database was polled for new messages constantly, because lowering the polling rate would slow down processing. The new architecture solved all of these problems by splitting the daemon into two thread pools: a polling pool and a processing pool.

Divide and rule: polling and processing in separate pools

The new architecture separates responsibility for polling and processing between different thread pools, using Redis as an intermediate message queue.

Polling pool:

• Threads in this pool fetch new messages from the database for each partition.
• When new messages are found, they are pushed to the Redis queue with the LPUSH command.
• To keep data consistent and preserve processing order, a lock is held on the partition while polling.

Processing pool:

• Threads in this pool take messages from the Redis queue with the BRPOP command.
• Each thread processes messages from a specific bucket, taking a lock for the duration of processing to prevent concurrent access and preserve message order within the bucket. Every daemon now consists of two thread pools — polling pool and processing pool. By default, each polling thread is paired with two processing threads, and the ratio is configurable.

Advantages of the new architecture:

• Simpler scaling: since polling is a lightweight operation, the polling pool can be much smaller than the processing pool. Scaling the system now comes down to adding daemon replicas, each with its own polling and processing pools.
• Better performance: separating polling from processing uses system resources more efficiently. Polling threads are not blocked while messages are processed, and processing threads do not sit idle waiting for new data.
• Flexible tuning: the ratio of threads in the polling and processing pools can be adjusted easily for the load profile and performance requirements.
• Automated scaling with Kubernetes HPA: using Redis as a buffer between the polling and processing pools makes autoscaling (HPA) in Kubernetes easy to set up. The Redis queue size is an accurate indicator of load on the processing pool. A growing queue means processing is falling behind, and HPA can raise the number of daemon replicas automatically. An empty queue means HPA can lower the replica count, optimizing resource usage. The result is a message processing system that is more flexible, more scalable, and simpler to configure.

From idea to an open-source solution

This story is a clear example of how a tool evolves, with each stage driven by new challenges and growing performance and reliability requirements.

The main takeaway: almost any problem can be solved in Ruby. Many open-source projects travel the path from solving one specific business problem to becoming a universal tool proven in production across many services.

This article covered three stages in the development of the Inbox/Outbox tool, built by the Ruby Platform team to handle message processing. Each stage aimed at making the tool more reliable, more scalable, and easier to use.

The final solution has been proven on real projects, both in large monolithic applications and in distributed systems of dozens of services. Confident in its stability and efficiency, the team decided to share it with the community and publish it as open source.

• sbmt-outbox — Outbox/Inbox Ruby gem
• sbmt-kafka_producer — Outbox Kafka transport
• sbmt-kafka_consumer — Inbox Kafka transport
• example applications — example microservice implementations