Running a Notification System with BullMQ, Part 2 - Rate Limits and the Design They Forced

Introduction

Honestly, I did not think much about rate limits at first. It never occurred to me that we would end up sending that volume of alimtalk messages.

The third-party provider’s sending quota was 100 messages per 5 seconds. I copied that straight into BullMQ’s limiter option.

new Worker(name, handler, {
  limiter: { max: 100, duration: 5000 },
});

The counter lives in Redis, so no matter how many worker instances run, they add up and respect the limit. That is exactly what Part 1 described. Our average was 20 req/s, comfortably inside the quota on paper.

Then 429 Too Many Requests started showing up, sporadically.

The incident — 429 while staying inside the limit

My first thought was that the limiter was not working. But our own counters never exceeded 100 per 5 seconds. Adding worker instances did not change the total. The limiter was doing its job.

The problem was that our five seconds and the provider’s five seconds were not the same five seconds.

The fixed-window alignment problem

Both our limiter and the provider’s quota are fixed-window counters, and the start times of the two windows are not aligned.

Our window:      [---0~5s---][---5~10s---]
Send pattern:          ████  ████          ← clustered at the boundary
Provider window:     [---2~7s---]          ← observes 200 here → 429

We send 100 messages late in our window, then another 100 as soon as the next one opens. By our accounting each window holds exactly 100, so nothing is violated. But a provider window straddling that boundary observes 200.

This is the classic fixed-window trap. “We stayed within the limit” is only true inside our own window; an observer with a different window can see up to twice as much.

This is a known problem. In System Design Interview – An Insider’s Guide, chapter 4 (“Design a Rate Limiter”) names exactly this situation as the weakness of the fixed window counter. With a limit of five per minute, five requests between 2:00:30 and 2:01:00 plus five between 2:01:00 and 2:01:30 means ten requests in the one-minute span from 2:00:30 to 2:01:30 — twice the allowance, in the book’s words.

The book also lists the alternatives. A sliding window log is very accurate but keeps timestamps even for rejected requests, so it costs memory. A sliding window counter is memory-efficient but works from an estimate that assumes requests in the previous interval were evenly distributed. A token bucket absorbs bursts but has two parameters — bucket size and refill rate — that need tuning.

Reading through them, the question became how much accuracy we actually needed.

Slice the window smaller

- limiter: { max: 100, duration: 5000 }
+ limiter: { max: 20, duration: 1000 }

Average throughput is unchanged at 20 req/s. But with a one-second window, the worst case an outside observer can see drops from 200 to 40 even when sends cluster at a boundary.

To be honest, this is still a fixed window. It does not remove the underlying problem; it shrinks the problem until it fits inside the safety margin. We could have gone to a token bucket or a sliding window, but weighing operational complexity against the benefit, we stopped at slicing the window.

We could accept that trade-off because the requirement was “sustain 20 req/s,” not “consume exactly 100 per window.” If we had needed to spend the quota down to the last message, stopping here would not have been an option.

Failures happen anyway

No amount of prevention removes transient failures from an external API, so we configured failed jobs to be retained for a period.

await queue.add(name, data, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 5000 },
  removeOnFail: { age: 24 * 3600 }, // keep failed jobs for 24 hours
});

With removeOnFail expressed as an age, failed jobs do not disappear immediately, which makes it much easier to pick out only the failures and retry them later.

That policy paid off in a separate incident. An infrastructure change to the NAT Gateway blocked every alimtalk send at once, and the retained failed jobs were what made recovery possible afterwards.

We were not the only ones spending the quota

Even after slicing the window, the 429s did not disappear entirely. The remaining cause sat outside our code.

Another team was using the same third-party account.

Here the property from Part 1 comes back as a liability. BullMQ’s rate limiter is per queue. Workers subscribed to the same queue share a counter, but nothing beyond that is visible to them. The other team’s application does not look at our Redis.

flowchart LR
    accTitle: Exceeding a quota shared across teams
    accDescr: Our worker respects its own limiter exactly, but calls from another team using the same account add up, so the third party observes more requests than the quota allows.
    W["Our worker
20/s (compliant)"] --> API["Third-party account
quota 20/s"]
    T["Another team's app
outside our control"] --> API

Read only our logs and the limit looks perfectly respected. But the provider counts per account, so from its side we are over. Neither codebase has a bug, and the 429s still come.

Unify the worker instead of dividing the quota

The first fix that came to mind was splitting the account quota between teams — we take 12, they take 8, and each side puts its share into its own limiter.

But that only moves the problem. An allocation rests on the assumption that both limiters honour their share exactly, and the reason that assumption breaks is the alignment problem from the previous section. Our window and the other team’s window are not aligned either, so 12 and 8 respected separately can still be observed as more at the account level. On top of that we have no way to verify that the other side stays within its share, and no way to attribute a 429 when one appears.

There was no reason to have two counters in the first place. Unify the point that calls the external API.

The structure from chapter 10 turned out to be directly useful here. The book splits a notification system into notification servers and workers. Notification servers accept a send request, validate recipient details, pull templates and user settings from cache and database to build the message, and put it on a queue. Workers do nothing but take it off the queue and hand it to the third party.

In other words, rendering and sending live on different servers. What goes onto the queue is not “send this template to this user” but a finished payload.

// Notification server — renders only. It never calls the external API.
const template = await templateRepository.find(templateCode);
const payload = renderAlimtalk(template, user, variables); // templating, phone normalization
assertSendable(user);                                      // consent and validity checks
await alimtalkQueue.add('send', payload);

// Sending worker — calls the external API only. The single place a limiter belongs.
new Worker('alimtalk', async (job) => alimtalkClient.send(job.data), {
  connection,
  limiter: { max: 20, duration: 1000 },
});

Split the rendering servers, share only the sending worker

There is a catch. A single notification server would have to know everything needed to render every message — but each team queries a different database. For our notification server to compose the other team’s messages, we would need their domain schema and database access, which is far too much to pay for the sake of one rate limit.

So we kept the unified part as small as possible. Each team keeps its own notification server; only the sending worker is shared.

Each team’s notification server reads its own database and builds payloads from its own templates. A finished payload is just an instruction to send, requiring no domain knowledge, so from that point on it can share a queue. The worker behind the queue does not need to know which team a payload came from.

flowchart LR
    accTitle: Per-team notification servers with a shared sending worker
    accDescr: Each team has its own notification server reading its own database, and the finished payloads they produce converge on one alimtalk queue. Only the shared sending worker calls the external API.
    DB1[("Our DB")] --> N1["Our notification server
validate + render"]
    DB2[("Other team DB")] --> N2["Their notification server
validate + render"]
    N1 --> Q["alimtalk queue
finished payloads"]
    N2 --> Q
    Q --> W["Shared sending worker
limiter 20/s"]
    W --> API["Third-party account
quota 20/s"]

This is the real value of separating rendering from sending. Without that split, “let’s share the sending worker” would have meant “let’s share domain logic,” which was never going to work. Because the boundary sits at the point where the payload is finished, each team keeps its own domain and only the quota is held in common.

The load on the account is expressed by a single Redis counter again. No allocation, no agreement, no total-sum check: one limiter holds the account quota as it is.

The separation buys two more things for rate limiting.

The number the limiter counts matches the number of external calls one to one. If the worker does nothing but call the external API, one job is one request. Leave rendering inside the worker and template lookups and personalization fold into processing time, which makes it much harder to tell what is holding throughput back when tuning concurrency and the limiter together.

Failures separate by kind. A wrong template code or a malformed phone number gives the same result however many times it is retried, whereas a 429 succeeds if you wait and send again. With rendering and sending in one worker, both share the same attempts. Failures that retrying cannot help eat into the retry budget, leaving less of it for the 429s that actually need it. Finish validation up front and most failures reaching the queue are send failures, so the retry policy can be tuned for that one kind.

The synchronization problem chapter 4 names for distributed environments is exactly this. The web tier is stateless, so requests can land on different rate limiters, and limiters that do not know each other’s counters make the limit meaningless. The book’s answer is a centralized store such as Redis rather than sticky sessions.

BullMQ’s limiter is already built that way; we were simply using that centralized store inside our own team only. Sharing the queue is what extends it past the organizational boundary. Rather than splitting the quota, we made a single place where it is enforced.

The book separates notification servers from workers for decoupling and buffering, but the resulting property — exactly one caller of the third party — is precisely what a rate limit needs.

It is not free, of course. The other team now depends on our sending worker, so that worker becomes a failure point for both sides, and someone has to own the payload schema and the queue’s operation. If that team cannot reach our Redis at all, the approach does not apply.

If the accounts can be separated at all

Everything above assumes the account has to be shared. That assumption is worth questioning first.

If the third-party account can be split per team, that is the cleanest answer. Separate accounts mean separate quotas, so there is no reason to share a counter at all. Each team puts its own limiter on its own account, attribution for a 429 is obvious, and no coupling between organizations appears.

The reasons accounts cannot be split are usually contractual rather than technical: sender profiles tied to the account, per-account pricing that raises unit cost when divided, or templates already approved under that one account. Absent those constraints, separating accounts deserves consideration before sharing a queue.

That leaves three options, to be considered top down.

Approach	Works when	Cost
Separate accounts	contracts and pricing allow it	none — take this if you can
Shared sending worker	both sides reach the same Redis	cross-team coupling, shared failure point
Divided quota	neither of the above is possible	still breachable via alignment, unverifiable

We could not split the account, so we took the second. Given that a divided quota has to be set below an even split and cannot be verified, the third is the option to keep for last.

One clarification is worth making. Chapter 10 also mentions rate limiting, but it is a different kind of limit. There it means capping how often a single user receives notifications, because sending too often makes people turn notifications off entirely. That is a user-fatigue concern, not an external quota one. The rate limit in this post belongs to chapter 4.

Summary

There were two reasons we saw 429s while respecting the limit, and they were different in kind.

• Fixed-window alignment — when our window and the other side’s window start at different times, the other side can observe up to twice our number. We sliced the window to shrink that gap into the safety margin, and stopped there knowing it was not a fundamental fix.
• A shared account — the limiter is per queue, so it cannot count calls made outside our queue. The account could not be split, so we separated rendering from sending, kept a notification server per team, and shared only the sending worker. With one external call site, account load is expressed by a single counter again.

Both converge on the same point. A limiter constrains the number we observe, not the number the external API observes. Most of the design work was finding where those two numbers diverge.

Coming next

Part 3 covers delivery history. The failed jobs kept by removeOnFail in this post are only temporary storage inside the queue — they disappear once the retention age passes. What was sent, when, to whom, and with what result is a record that has to be kept separately.

References

• BullMQ Rate limiting
• BullMQ auto-removal of jobs
• Alex Xu, System Design Interview – An Insider’s Guide, ch. 4 “Design a Rate Limiter” and ch. 10 “Design a Notification System”