Rate Limiters Are Not Your Enemy: How Token Buckets Work and How Automation Should Behave Under Them

# webdev# api# tutorial# devtools
Rate Limiters Are Not Your Enemy: How Token Buckets Work and How Automation Should Behave Under ThemNguyen Tu

Rate limiters are having a quiet moment — the topic sits at #12 on daily.dev Popular this morning —...

Rate limiters are having a quiet moment — the topic sits at #12 on daily.dev Popular this morning — and for good reason: it is how most serious automation actually moves. Every pipeline we operate eventually meets a 429. A scraper that was happily pulling pages at full speed suddenly gets told to slow down, and the difference between a pipeline that survives the encounter and one that gets its IP banned for a week is almost never the request rate itself. It is whether the client understood the limiter on the other side.

We build pipelines that move data between systems for a living — lead ingestion, enrichment, scoring, and unattended workflows — and rate limiting is the part of that work nobody demos but everybody pays for. This piece is the mental model we use: how the common limiters actually work, what the response headers are really telling you, and how to design a client that treats a limiter as a protocol instead of an obstacle.

Our how we work rule applies here too: automate aggressively, but keep the blast radius small. In rate-limit terms, that means a single runaway job should never be able to spend the whole account's budget, and a single 429 should never cascade into a retry storm.

1. A limiter is a budget, not a wall

The most useful reframe for automation builders is this: a rate limiter does not exist to stop you. It exists to give every client a fair share of a finite resource, and to keep one client's burst from becoming everyone else's outage. When your scraper gets a 429, the server is not saying "go away." It is saying "you have spent your budget for this window; come back when it refills."

That framing changes the design. A client that treats 429 as an error retries blindly and burns its budget faster. A client that treats 429 as budget information slows down, waits for the refill, and finishes the job — just later.

HTTP gives you the vocabulary for this. Status 429 (Too Many Requests) is defined in RFC 6585, and the Retry-After header tells you how long to wait before trying again. Many APIs also send X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers, which tell you the size of the budget, how much of it is left, and when it refills. A client that reads those headers is negotiating. A client that ignores them is guessing.

2. The four limiters you will actually meet

Most limiters you encounter are one of four shapes. Knowing which one you are facing tells you how to behave.

Fixed window. The server counts requests in a calendar window — say, 100 requests per minute — and resets the counter at the window boundary. This is the simplest limiter to implement and the easiest to game: a client can fire 100 requests in the last second of one window and 100 more in the first second of the next, doubling its effective rate at the boundary. If you see your remaining budget jump back to full at a predictable second, you are probably looking at a fixed window. The correct behaviour is to spread your requests across the window, not to burst at its edges.

Sliding window. The server counts requests in a rolling window — the last 60 seconds, not the current calendar minute. This kills the boundary-burst trick, because there is no boundary to exploit. The cost is memory: the server has to remember timestamps, not just a counter. From the client's side, a sliding window feels like a fixed window that never quite refills all at once; your budget comes back gradually, one expired request at a time.

Token bucket. This is the one in the daily.dev post, and the most common design for serious APIs. Imagine a bucket that holds a fixed number of tokens. Every request costs one token. Tokens refill at a steady rate — say, 10 per second — up to the bucket's capacity. A client that has been quiet accumulates a full bucket and can burst; a client that has been hammering the API runs dry and has to wait for refills. The bucket gives you two knobs the other designs do not: capacity (how big a burst is allowed) and refill rate (the sustained rate). A bucket with capacity 100 and refill 10/second allows a burst of 100, then a steady 10/second forever. That is why token buckets feel forgiving at first and strict later: the burst budget is real, but it is finite, and once it is gone, the refill rate is the law.

Leaky bucket. The mirror image: requests enter a queue and leak out at a fixed rate. Bursts get queued rather than rejected, which smooths traffic but adds latency — and when the queue fills, requests are dropped. Leaky buckets are more common on the server's processing side than as a client-facing limiter, but the distinction matters when you are designing your own: token bucket limits how many requests you accept; leaky bucket limits how fast you process them.

Limiter Burst allowed? Refill behaviour What it feels like to a client
Fixed window Yes, at window edges All at once, on the boundary Budget jumps back to full at a predictable second
Sliding window Limited Gradually, as old requests expire Budget creeps back, never all at once
Token bucket Yes, up to capacity Steady rate, up to capacity Generous after idle, strict under sustained load
Leaky bucket Queued, not rejected Fixed processing rate Requests slow down before they fail

3. Read the headers before you retry

The single highest-value habit in rate-limit handling is reading the response before deciding what to do next. A well-behaved client checks, in order:

  1. The status code. 429 means "slow down." 403 with a rate-limit message (GitHub's secondary rate limits do this) means the same thing with different wording — read the body, not just the code.
  2. Retry-After. If the server tells you to wait 60 seconds, wait 60 seconds. Not 5, not 30. Servers that bother to send Retry-After usually enforce it, and retrying early is the fastest way to turn a temporary slowdown into a longer ban.
  3. The remaining budget. If X-RateLimit-Remaining is 0 and X-RateLimit-Reset is 40 seconds away, there is no point sending another request for 40 seconds. Sleep until the reset, then resume. If remaining is high, the 429 you just got was probably a secondary limit — a concurrency limit, a per-endpoint limit, or an abuse heuristic — and the fix is different from waiting for the main budget.
  4. The shape of the failure. A 429 on every endpoint means the account budget is gone. A 429 on one endpoint while others work means a per-route limit. A 429 that arrives with a CAPTCHA challenge means the server has decided you look like a bot, and no amount of waiting will fix that — that is a fingerprint and behaviour problem, not a budget problem. Our Shopee security-headers research covers that side of the problem in depth.

The code version of this is short, and it belongs in every client you write:

import time

def call_with_limit_awareness(http_call, max_attempts=5):
    for attempt in range(max_attempts):
        resp = http_call()
        if resp.status_code != 429:
            return resp
        # Respect the server's own number first.
        retry_after = resp.headers.get("Retry-After")
        if retry_after is not None:
            time.sleep(float(retry_after))
            continue
        # No Retry-After: back off exponentially, with jitter,
        # so a fleet of clients does not retry in lockstep.
        delay = min(60.0, (2 ** attempt)) + random.uniform(0, 1)
        time.sleep(delay)
    return resp
Enter fullscreen mode Exit fullscreen mode

Two details in that snippet do most of the work. First, Retry-After wins over any backoff formula you invented, because the server knows its own limiter and you do not. Second, the jitter matters as much as the backoff: without it, every client that hit the limit at the same second retries at the same second, and the limiter gets a second burst exactly when it is most sensitive. Our event-driven pipelines piece makes the same argument for poison-message retries — jitter is what turns a herd into a queue.

4. Budget your own tokens, client-side

Waiting for the server to tell you to slow down is reactive. The better design is to never hit the limiter in the first place, by running your own token bucket on the client side, sized conservatively below the server's published limits.

A minimal client-side bucket in Python:

import threading
import time

class TokenBucket:
    def __init__(self, capacity, refill_per_second):
        self.capacity = capacity
        self.tokens = capacity
        self.refill = refill_per_second
        self.updated_at = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            elapsed = now - self.updated_at
            self.tokens = min(self.capacity,
                              self.tokens + elapsed * self.refill)
            self.updated_at = now
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            wait = (1 - self.tokens) / self.refill
            time.sleep(wait)
            self.tokens = 0
            self.updated_at = time.monotonic()
            return True
Enter fullscreen mode Exit fullscreen mode

Three rules make a client-side bucket actually work in production:

  1. Size it below the documented limit, not at it. If the API documents 100 requests per minute, run your bucket at 80. The margin absorbs clock skew, undocumented secondary limits, and the requests your other jobs are making against the same key that you forgot about.
  2. One bucket per key, not per job. If three jobs share an API key and each runs its own bucket at 80/minute, you are making 240/minute against a 100/minute limit. The bucket must live where the key lives — a shared service, a Redis key, a single coordinator — not inside each job's memory.
  3. Treat the bucket as a queue, not a gate. When the bucket is empty, the correct behaviour for a scraper is usually to wait, not to fail. Failing on an empty bucket turns a rate limit into a data gap; waiting turns it into a slower run. The exception is interactive work, where a human is waiting — there, fail fast and say why, with the reset time in the message.

5. Distributed limiters and the limits of client-side politeness

Client-side buckets work until you have more than one machine, more than one key, or a limiter you cannot see. At that point the limiter has to live on the server side of your own system, and the usual tool is a shared store — Redis is the default — holding token counts that every worker checks before calling out. The design questions are the same ones as any distributed system: what happens when the store is briefly unavailable (fail open and risk the ban, or fail closed and stall the pipeline?), and how do you keep the check itself from becoming the bottleneck?

There is also a category of limiter no client-side discipline can fix: abuse heuristics. A server that decides your traffic pattern looks automated — identical request intervals, no mouse movement, a datacenter IP, a fingerprint that matches a known bot — will block you regardless of your rate. That is not a budget problem; it is a trust problem, and the fix is behavioural (human-like timing, residential egress, real browser fingerprints), not arithmetic. Knowing which kind of block you are facing is the whole diagnostic skill: check the status code, the headers, and the body, in that order, before changing anything.

6. The pre-flight checklist

Before a new scraper or automation client ships, it should pass five checks:

  1. Headers: does it read Retry-After and the X-RateLimit-* headers, and does Retry-After override its own backoff?
  2. Budget: does it run a client-side bucket sized below the documented limit, shared across every job using the same key?
  3. Backoff: are retries bounded, exponential, and jittered — never immediate, never infinite?
  4. Isolation: can one runaway job spend the whole account's budget, or is each job's share capped?
  5. Diagnosis: when a run is blocked, does the log say which limiter fired, what the remaining budget was, and when the reset is — or does it just say "429"?

A client that passes all five will still get rate-limited. That is the point. Rate limits are not failures to avoid; they are the protocol the internet uses to share finite resources. The clients that survive them are not the ones that never see a 429. They are the ones that treat the limiter as information, budget against it, and keep working at the rate the server can actually sustain.

FAQ

Is a token bucket the same as a fixed window?
No. A fixed window resets its whole budget at a boundary, which allows boundary bursts. A token bucket refills steadily, so the budget comes back gradually, and a burst is capped by the bucket's capacity rather than by the clock.

Should I just retry immediately when I get a 429?
No. Immediate retries are the most common way to turn a short slowdown into a long ban. Respect Retry-After when present; otherwise back off exponentially with jitter, and cap the number of attempts.

Why does my client get limited even though it is under the documented rate?
Usually one of three reasons: a secondary limit (per-endpoint, per-minute, or concurrency) that is not the headline number; another job sharing the same key; or an abuse heuristic that is not rate-based at all. Check the response headers and body before assuming the headline limit is wrong.

Do I need a distributed limiter, or is a local bucket enough?
A local bucket is enough for one process against one key. As soon as multiple jobs or machines share a key, the bucket has to be shared too, or each job's "safe" rate adds up to an unsafe total.

References

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.