Premium Video

This video is available to premium subscribers only

Unlock Full Access

In this chapter, we'll look at what message queues are, how they work, and how to use them to build systems that handle load and failures well.

Suppose you're building a video platform. When a user uploads a video, the server needs to transcode it into multiple resolutions, generate thumbnails, and run a content check. Transcoding alone can take several minutes.

If the upload request waits for all of this work to finish, the user is stuck on a loading screen, and the request might time out before processing is complete.

There are other problems as well:

The underlying problem is that slow, resource-intensive work is tied directly to the user's request. Instead, we want to accept the upload quickly, record the tasks that need to be done, and let a separate set of workers process them independently.

This is exactly where a message queue comes in.

A message queue, such as Amazon SQS or RabbitMQ, is a buffer that sits between two kinds of services:

The producer sends a message to the queue describing a task, such as "transcode video 456". The queue stores the message until a consumer is ready. A consumer then takes the message from the queue, does the work, and tells the queue it is finished.

The system that stores the messages and hands them out is often called the broker. It accepts messages from producers, keeps them safe, delivers them to consumers, and tracks which ones have been acknowledged.

One Message, One Consumer

The key property of a queue is that each message is processed by one consumer. If you have ten workers reading from the same queue, each message goes to only one of them.

This is what separates a queue from pub/sub, where every subscriber receives its own copy of each message. A queue is for handing out work. Pub/sub is for telling several independent services that something happened.

Back to the Video Example

With a queue in place, the upload service stores the file, sends a message to the queue, and immediately returns a response. The user sees that their video is processing, and the heavy work happens in the background.

Let's take a closer look at what happens to a message as it moves through a queue.

Step 1: The Producer Sends a Message

The producer sends a message to the queue. The queue stores it durably, typically on disk and often replicated across multiple nodes, before confirming that it has been received.

Once the producer gets this confirmation, it can move on, knowing the message has been safely stored.

Step 2: A Consumer Receives the Message

Next, a consumer receives the message. But the queue doesn't remove it right away. Instead, it temporarily prevents other consumers from receiving the same message.

In Amazon SQS, this is handled through a visibility timeout. For a set period, say 30 seconds, the message stays invisible to other consumers while the first consumer processes it.

Step 3: The Consumer Acknowledges

Once processing is complete, the consumer acknowledges the message. In SQS, this means explicitly deleting it from the queue.

What If the Consumer Crashes?

Since the consumer never acknowledged the message, the queue still has it:

In both cases, another consumer can pick up the message and try again. This is how acknowledgments prevent messages from being lost.

The Catch: Duplicates

This safety net has a side effect. If a consumer finishes the work but crashes before acknowledging, the message is delivered again and the work runs twice. Consumers need to be safe to run more than once. We'll cover how in the Delivery Semantics chapter.

With a message queue, scaling is fairly simple. You add more consumers to the same queue, and each worker picks up available messages, spreading the workload across them. This is known as the competing consumers pattern.

Scaling on Queue Depth

Many systems automatically scale their workers based on queue depth, which is the number of messages waiting to be processed.

Adding workers only helps up to a point. If every worker depends on the same database or third-party API, that dependency eventually becomes the limit, and more workers just add more pressure on it.

Prefetch

There's also an important setting called prefetch. Instead of fetching one message at a time, a consumer can fetch several messages at once to reduce network round trips.

However, if one worker grabs a hundred messages while others sit idle, the workload becomes uneven. In RabbitMQ, you control this with the prefetch count. For long-running tasks like video transcoding, keeping this value small usually spreads the work more evenly. For quick, lightweight tasks, a larger value can improve throughput.

Message queues also help protect your system from sudden spikes in traffic.

Imagine a flash sale where incoming orders jump from 100 per second to 5,000 per second for ten minutes. The order service can accept these requests quickly, but the downstream fulfillment system can only process 1,000 orders per second.

This is often called load leveling. It works because it's acceptable for fulfillment to happen a few minutes later. If the user needed the result right away, a queue would not remove that requirement.

The Limits of Buffering

Queues have their limits. If messages consistently arrive faster than consumers can process them, the backlog will keep growing. Users will wait longer for their tasks to complete, and eventually the queue could run out of storage.

That's why it's important to monitor two key metrics:

The age of the oldest message is often more useful than queue depth alone. A queue with just 100 messages could still indicate a serious problem if the oldest one has been waiting for an hour.

Sometimes a message keeps failing no matter how many times it's processed. Maybe the video file is corrupted, or the message references a user account that no longer exists.

Each time a worker fails to process the message, it becomes available in the queue again, and another worker picks it up. Without a retry limit, this cycle can continue forever, wasting resources and delaying other messages.

A message that repeatedly fails processing is called a poison message.

A common solution is to limit the number of delivery attempts. In Amazon SQS, this is configured with a setting called maxReceiveCount. Once a message has been received that many times without being deleted, SQS moves it to a separate queue called a dead-letter queue, or DLQ.

This keeps the problem message from interfering with normal processing. Engineers can then investigate why it failed, fix the underlying issue, and reprocess the message when it's safe to do so.

Another important question is whether messages are processed in the order they were sent. The word "queue" suggests first in, first out, but in practice ordering isn't always guaranteed.

Even if messages are delivered in order, multiple consumers may process them at different speeds:

Retries can also change the processing order, since a failed message might be retried after newer messages have already been processed.

For many workloads, this isn't a problem. If two different videos need to be transcoded, it usually doesn't matter which one finishes first.

But some operations depend on strict ordering. Suppose one message updates an order's shipping address, and the next message tells the system to ship the order. If the shipping message is processed first, the package could be sent to the wrong address.

Ordering Within a Group

A common solution is to preserve ordering within related groups of messages, rather than across the entire queue.

Amazon SQS FIFO queues support this with a message group ID. Messages with the same group ID, such as those belonging to a particular order, are delivered in order. Messages from different groups can be processed in parallel.

This gives you ordering where it matters without forcing the whole queue through a single worker. Strict ordering across every message in a busy queue rarely scales well.

The messages themselves also need some careful design.

Keep them small. Message queues are designed to handle large numbers of small messages, and most have a limit on message size. If a task involves a large file, store it in object storage like Amazon S3 and include only a reference to it in the message.

For example, our transcoding message should contain the video ID and its storage location, not the video file itself:

It's also useful to include a few extra fields:

Versioning lets you change the message format over time without breaking consumers that still expect an older version.

Message queues are particularly useful when tasks are slow, resource-intensive, or don't need to finish before the user receives a response. Common examples include:

They're also useful when one service produces work faster than another can process it, or when a downstream service is unreliable and you want failed tasks to be retried rather than lost.

However, message queues aren't always the right choice. If the caller needs an immediate result, such as verifying a password during login, a direct request is usually simpler and more appropriate.

A message queue allows producers to hand off work without waiting for it to finish, while consumers process messages independently at their own pace.

But to use message queues reliably, you also need to: