Operating a Notification System with BullMQ, Part 1: What Is BullMQ?
Introduction
Every service needs to send messages to users, whether through AlimTalk, email, or push notifications. But implementing notification delivery as a simple HTTP call quickly runs into limits.
- Multiple domains call the same external API. Delivery paths become scattered across the API server, admin system, and batch jobs.
- External APIs have rate limits. A constraint such as “100 requests per five seconds” must be enforced somewhere.
- Many messages are not sent immediately. Timing branches such as “send guidance on D+1 after verification” or “send another notice three days after payment failure” are necessary. Some requirements, however, are difficult to represent with a state value—for example, a one-off message that an administrator schedules for a particular user at a particular time and that does not map naturally to the domain model.
- Bulk delivery always includes some failures. The system must track which messages failed and need to be retried.
Each of these problems can be solved with a different tool. We can consolidate delivery into one module, put a rate limit inside a function, create time-based branches with a scheduler, and record failures in a delivery-history table. It is possible.
But when all four problems must be solved within one system—and must work consistently in a distributed environment consisting of an API server and multiple Worker instances—we need a separate infrastructure abstraction. A queue is one tool we can choose at that point. This article explains why I chose BullMQ and how I use it in practice.
What Is BullMQ?
BullMQ is a Redis-backed job-queue library for Node.js. A Producer adds a Job to a queue, and a separate Worker process takes the Job from the queue and processes it.
There are only three core concepts:
- Queue — A logical channel in which Jobs are stored. Queues are separated by Redis key prefixes.
- Job — A unit of work to process. It contains a payload and options such as
delayandattempts. - Worker — A process that handles Jobs. It subscribes to a queue and processes Jobs asynchronously.
// Producer
await queue.add('send-alimtalk', { userId, template }, { delay: 60_000 });
// Worker
new Worker('notification', async (job) => {
await alimtalkService.send(job.data);
}, { connection, limiter: { max: 20, duration: 1000 } });How Does It Work?
The key to BullMQ is that Redis stores the entire state of the queue. The Jobs created by Producers, the list of Jobs waiting to be processed, the Jobs waiting for delayed delivery, and the locks on Jobs currently being processed all live in Redis. Workers continuously inspect Redis and take Jobs to process.
This architecture enables two things.
Producers and Workers are completely decoupled. A Producer is responsible only for creating a Job and putting it in Redis. A Worker is responsible only for taking a Job from Redis and processing it. They do not need to be in the same process or even on the same server. They only need to use the same Redis instance.
State remains consistent even when multiple Worker instances are running. A Job being processed is marked with a Redis lock so that another Worker cannot pick it up at the same time. The rate-limiter counter is also stored in Redis, so all Workers enforce one combined limit. “All shared state lives in Redis” is the principle behind BullMQ’s distributed behavior.
A Job passes through several states within a queue.
stateDiagram-v2
accTitle: Core BullMQ job states and transitions
accDescr: A job enters waiting immediately or delayed until its scheduled time, becomes active when a worker picks it up, and ends as completed or failed unless a retry returns it to waiting.
[*] --> Waiting: enqueue
[*] --> Delayed: enqueue with delay
Delayed --> Waiting: scheduled time reached
Waiting --> Active: worker picks up job
Active --> Waiting: error and retries remain
Active --> Completed: successful processing
Active --> Failed: error and retries exhausted
Completed --> [*]
Failed --> [*]
wait means waiting to be processed, delayed means waiting for its scheduled time, active means processing, and completed and failed are terminal states. These states are represented by Redis data structures such as lists, sorted sets, and hashes. State transitions are performed atomically through Lua scripts, which prevents multiple Workers from picking up the same Job simultaneously.
Architecture
- The lifecycle followed by a Job added through
Queue.add() - A Job in
delayeddoes not move directly toactivewhen its time arrives. Internally, it is first promoted towaitorprioritized, then entersactive.
stateDiagram-v2
accTitle: Lifecycle of a BullMQ job added with Queue.add
accDescr: A newly added job is routed by delay and priority into delayed, waiting, or prioritized storage, becomes active when a worker can process it, and then completes, fails, or returns for another attempt.
[*] --> Added: job added
Added --> delayChoice
state delayChoice <<choice>>
delayChoice --> Delayed: delay > 0
delayChoice --> priorityChoice: delay = 0
Delayed --> priorityChoice: scheduled time reached
state priorityChoice <<choice>>
priorityChoice --> Waiting: priority = 0
priorityChoice --> Prioritized: priority > 0
Waiting --> Active: worker available and within limits
Prioritized --> Active: worker available and within limits
Active --> Waiting: moveToWait or dynamic rate limit
Active --> Delayed: moveToDelayed or backoff retry
Active --> outcomeChoice: processing finishes
state outcomeChoice <<choice>>
outcomeChoice --> Completed: no error
outcomeChoice --> Failed: unrecoverable error or attempts exhausted
Completed --> [*]
Failed --> [*]
- The lifecycle of a Job added through
FlowProducer.add() waiting-childrenmeans “waiting for someone else to finish processing.”- When the final child completes, the parent is promoted to
wait,prioritized, ordelayed.
stateDiagram-v2
accTitle: Lifecycle of a BullMQ parent job added with FlowProducer.add
accDescr: A parent job waits for its children when dependencies exist, then follows the normal delay, priority, active, completion, failure, and retry states after the children finish.
[*] --> Added: job added
Added --> childrenChoice
state childrenChoice <<choice>>
childrenChoice --> WaitingChildren: child jobs exist
childrenChoice --> delayChoice: no child jobs
WaitingChildren --> delayChoice: all children completed
WaitingChildren --> Failed: child failed with failParentOnFailure
state delayChoice <<choice>>
delayChoice --> Delayed: delay > 0
delayChoice --> priorityChoice: delay = 0
Delayed --> priorityChoice: scheduled time reached
state priorityChoice <<choice>>
priorityChoice --> Waiting: priority = 0
priorityChoice --> Prioritized: priority > 0
Waiting --> Active: worker available and within limits
Prioritized --> Active: worker available and within limits
Active --> WaitingChildren: moveToWaitingChildren
Active --> Waiting: moveToWait or dynamic rate limit
Active --> Delayed: moveToDelayed or backoff retry
Active --> outcomeChoice: processing finishes
state outcomeChoice <<choice>>
outcomeChoice --> Completed: no error
outcomeChoice --> Failed: unrecoverable error or attempts exhausted
Completed --> [*]
Failed --> [*]
In BullMQ, a Job is a state machine.
The moment Queue.add() is called, the Job enters its lifecycle and moves through several states before ending in completion or failure. If a failed Job is retried, it begins a new lifecycle. The important point is that BullMQ does not manage “state” as a simple enum. It manages state by placing Job IDs in different Redis data structures.
| State | Data structure | Description |
|---|---|---|
| wait | LIST(bull:queue:wait) | Natural representation of a FIFO queue; atomically pops and moves a Job to active with BRPOPLPUSH |
| prioritized | ZSET | Uses the priority value as the score for automatic ordering |
| delayed | ZSET | Uses the execution timestamp as the score for time-based ordering |
| active | LIST | Set of Jobs being processed, used to detect stalled Jobs if a Worker dies |
| completed/failed | ZSET | Uses completion time as the score and supports cleanup of old Jobs with removeOnComplete |
| waiting-children | ZSET | Special state for parent-child dependencies |
Core BullMQ Features
1. Delayed Jobs
When you add a Job with the delay option, a Worker cannot pick it up until the specified time has passed.
// Send three days later
await queue.add('reapply-guide', payload, { delay: 3 * 24 * 60 * 60 * 1000 });Internally, BullMQ stores the Job in a Redis sorted set with its timestamp as the score. A separate scheduler moves it from delayed to wait.
2. Rate Limiter
The limiter option on a Worker enforces the number of Jobs processed per unit of time.
new Worker(name, handler, {
limiter: { max: 20, duration: 1000 }, // 20 per second
});The counter is stored in Redis, and the limit applies to the combined total even when multiple Worker instances are running. This makes it possible to comply precisely with an external API’s rate limit.
3. Deduplication
When you specify a jobId, a Job with that same ID can enter the queue only once.
BullMQ stores Job data in a Redis Hash with the key format bull:{queueName}:{jobId}. If you do not specify a jobId, BullMQ calls INCR on the bull:{queueName}:id counter to issue monotonically increasing integers: 1, 2, 3, and so on.
When you specify a jobId, the addJob.lua script atomically performs the following sequence:
1.
EXISTS bull:{queueName}:{jobId}— Does the Hash key exist?
2. It exists → do not push it to the queue (wait/delayed/prioritized); return the existingjobIdand emit aduplicatedevent.
3. It does not exist → create the Hash withHSETand push thejobIdto the queue.
This also matters when the Job has already completed successfully.
You can specify a jobId as follows:
await queue.add(name, data, { jobId: `payment-request-${paymentId}` });4. Retry and Backoff
When processing throws an error, BullMQ retries automatically. You can configure attempts and a backoff strategy such as fixed or exponential.
await queue.add(name, data, {
attempts: 3,
backoff: { type: 'exponential', delay: 5000 },
});5. Concurrency
This setting specifies how many Jobs a Worker can process simultaneously.
For I/O-heavy work such as calling external APIs, increasing concurrency is a good way to improve throughput.
new Worker(name, handler, { concurrency: 50 });6. Flows
FlowProducer can create a graph of Jobs with parent-child relationships. A parent Job remains in waiting-children until every child completes, and only then is it processed.
import { FlowProducer } from 'bullmq';
const flow = new FlowProducer({ connection });
await flow.add({
name: 'aggregate-results',
queueName: 'order',
data: { batchId: 'b-2026-01-15' },
children: [
{ name: 'item', queueName: 'item-queue', data: { chunk: 1 } },
{ name: 'order', queueName: 'order-queue', data: { chunk: 2 } },
{ name: 'notification', queueName: 'notification-queue', data: { chunk: 3 } },
],
});This model is natural when a large operation must be divided into chunks and the results aggregated after every chunk finishes, or when representing a multistage pipeline such as “preprocessing → delivery → post-processing.” If even one child fails, the parent is not processed, allowing a workflow with explicit dependencies to be managed safely.
BullMQ Is a Job Queue: How It Differs from Event-Driven Architecture
Because processing is asynchronous, something queue-like sits in the middle, and coupling is reduced, BullMQ may look like an event-driven system. Once you use it seriously, however, it becomes clear that BullMQ follows a different model.
Does the Producer Know the Consumer?
In a job queue, the Producer explicitly names the queue and the Job in code.
await queue.add('send-alimtalk', { userId, template });The Producer code explicitly knows that “someone exists to process a Job named send-alimtalk.” The Producer is entirely responsible for deciding which queue and payload to use. It delegates the work directly.
The event model is different.
eventBus.emit('user.signed-up', { userId });The Publisher emits only the fact that “a user signed up.” Who listens and what they do are not the Publisher’s concern. An analytics system may listen, a notification system may listen, or nobody may listen. Adding a new Listener does not change the Publisher code. This is the essence of event-driven architecture.
If your goal is truly event-driven architecture, RabbitMQ or Kafka may be a better choice.
Note: BullMQ Events
What BullMQ calls an “event” is a callback mechanism for observing internal changes in queue and Worker state. You can receive lifecycle events such as a Job completing, failing, becoming stalled, or updating its progress.
When Should You Choose BullMQ?
BullMQ may be the better choice in situations like these:
- Delivery triggers are distributed across multiple domains.
- You must comply strictly with an external API’s rate limit.
- You want to delegate work to horizontally scalable servers.
- You already operate Redis and use Node.js.
- You want the powerful retry logic provided at the library level, such as retrying only failed messages.
BullMQ Overview for the Notification Server
1. Consolidate Distributed Delivery Paths into One Queue
Delivery triggers occur everywhere: the API server, admin system, batch jobs, and more.
Put all of them into the same notification queue and delegate delivery to a separate Worker service.
flowchart LR
accTitle: Notification triggers converge on one BullMQ queue
accDescr: The API server, admin system, and batch jobs enqueue notification work, multiple workers share that queue, and each worker calls the same external delivery API.
api["API server"] --> queue[("notification queue")]
admin["Admin system"] --> queue
batch["Batch jobs"] --> queue
queue --> worker1["Worker 1"]
queue --> worker2["Worker 2"]
queue --> worker3["Worker 3"]
worker1 --> external["External API"]
worker2 --> external
worker3 --> external
Manage the delivery logic—template mapping, phone-number normalization, and external API calls—in one place.
If throughput is insufficient, add more Worker instances.
Multiple Workers subscribed to the same queue automatically divide the work. Because the rate limiter shares a Redis-backed counter, the total number of calls to the external API remains consistently within the limit whether you have one instance or ten.
2. Control the External API Rate Limit with the Worker Limiter
If the external API allows “100 requests per five seconds,” configure the Worker limiter as { max: 20, duration: 1000 }.
Twenty requests per second accumulate to 100 requests over five seconds, keeping the system within the limit.
Even when traffic spikes, requests merely accumulate in the queue and are smoothed out; the external API remains unaffected.
3. On a Paid Plan, Use Batch Processing for Bulk Requests
import { WorkerPro } from '@taskforcesh/bullmq-pro';
const worker = new WorkerPro('alimtalk-queue', async (job: JobPro) => {
const batch = job.getBatch(); // Multiple Jobs arrive as an array
for (const batchedJob of batch) {
await sendAlimtalk(batchedJob.data);
}
}, {
connection,
batch: {
size: 10, // Up to 10 Jobs in one batch
minSize: 5, // Wait until at least 5 Jobs have accumulated
timeout: 30000, // Process after 30 seconds even if fewer than 5 arrive
}
});If the external API you use has a strict rate limit, consider sending bulk requests through Batch.
Be sure to evaluate tradeoffs such as partial failures and a larger retry unit.
Next
We have learned what BullMQ is and used its features to sketch a simple Notification server design.
The next article will examine the incidents and tradeoffs I encountered while operating it.