Leaky bucket

A bucket with a hole in the bottom. Water pours in at whatever rate it likes, and leaves through the hole at a steady one. Pour faster than it drains and the bucket fills; keep going and it overflows.

The water is requests. The hole is your processing rate.

The leaky bucket holds arriving requests in a queue and processes them at a fixed rate, so what reaches the server is smooth however uneven the arrivals were.

Watching it fill and drain

Capacity of five requests, processed two per second, starting empty:

Second	Arrive	Queued	Processed	In queue after	Discarded
1	4	4	2	2	0
2	4	3, so 5 in queue	2	3	1
3	0	0	2	1	0
4	0	0	1	0	0

Second two is where the capacity bites. Four requests arrive against a queue already holding two, and only three fit. The eighth request is discarded outright.

Two properties fall out of that:

• The output rate never varies. Two per second, whether eight requests arrived or none.
• The queue is first in, first out. The request that arrived earliest is processed first.

A token bucket says spend your tokens whenever you like. A leaky bucket says get in line, I'll process you at my pace.

JunoWatching it fill and drain The difference from every algorithm before this: a request that arrives when things are busy is not refused, it waits.

Requests only get discarded when the queue itself is full, so the bucket absorbs a burst rather than rejecting it.

JunoWatching it fill and drain Waiting is a real cost and it lands on the client. A request queued behind four others at two per second waits two seconds before anything starts.

From the caller's side that is indistinguishable from a slow server.

So a queued request still needs a timeout. Without a ceiling on waiting time, a burst becomes a pile of clients who have all given up and are still holding connections.

JunoWatching it fill and drain Measured behaviour, for a bucket of 20 draining 5 per second: 25 requests arriving at once accepted 20 and discarded 5, with the twentieth waiting 4 seconds before it was processed.

That last number is the one to design around. A full queue plus a slow drain leaves the requests at the back waiting a long time, and 4 seconds is past the point where most clients have retried or given up.

Capacity is not free merely because those requests were not refused.

Which suggests sizing capacity from acceptable latency rather than from memory. At 5 per second, a 2 second worst case means a queue of 10, not 20.

What it is good and bad at

Strengths	Weaknesses
Smooths bursts into a constant output rate	No flexibility for a legitimate spike
The queue prevents overload downstream	Prioritises system stability over user experience
Simple to understand and implement	Clients wait even when the server could have coped
Predictable load, whatever arrives	Latency grows with queue depth

The weaknesses are all the same tradeoff seen from different angles. A leaky bucket protects the server by making clients wait, and it does that even when the server had capacity to spare.

That suits network bandwidth management, video streaming and steady server request handling, where a predictable rate is worth more than a fast response to a burst.

JunoWhat it is good and bad at Every row in that table comes from the same decision: the output rate is fixed and does not adapt.

That's the strength when you need predictable load, and the weakness when a client is waiting for something the server could have handled immediately.

JunoWhat it is good and bad at Reach for this when what you're protecting cannot absorb a spike: a downstream API with its own limit, a database that degrades under concurrency, a payment provider that charges per call.

Reach for a token bucket when the thing you're protecting can handle a burst and you want clients to feel responsive. The question is what's behind the limiter, not what's in front of it.

JunoWhat it is good and bad at Strict first in, first out is the part worth questioning. Under sustained overload it produces head-of-line blocking, where one slow request delays everything behind it however cheap those were.

It also means a client that flooded the queue keeps priority over one that arrived later with a single request, so a leaky bucket alone gives no fairness between clients. Per-client buckets fix that and multiply your state.

Worth naming the resemblance too: a leaky bucket is a bounded work queue with a rate limit on the worker.

If you already run a job queue with concurrency limits, you have most of one, and the question becomes whether the limiter belongs there instead of in the request path.

Try it

A leaky bucket with capacity 5, draining 2 requests per second, starting empty.

1. Six requests arrive at once. What happens to each?
2. How long does the sixth-arriving request wait before being processed, if it makes it in?
3. The same six arrive, but spread one per second. What happens?
4. Which algorithm would suit an endpoint that calls a payment provider charging per request, and why?

Compare your answers

#	Answer	Why
1	Five queue, one is discarded	The queue holds five and nothing has drained yet, since the burst is instantaneous
2	It never makes it in	The sixth is the one discarded. The fifth is the last accepted, and at two per second it waits two and a half seconds
3	All six processed, none discarded, none waiting	One per second is slower than the drain at two per second, so the queue never builds
4	Leaky bucket	The constraint is downstream and costs money per call, so a predictable outgoing rate is what you want. A token bucket would turn a burst of requests into a burst of charges

Question three is the one worth holding onto: all of these algorithms do nothing at all to traffic inside the limit. They only become visible at the edges.

Where this goes next

Five algorithms, and every one of them ends a request that goes too far with a refusal or a wait imposed silently.

Throttling makes the waiting deliberate and visible, slowing a client down as they approach a limit instead of refusing them at it, then combines both into one stack.