Requirements
- Limit each client (user id, API key or IP) to N requests per window, with different rules per endpoint or plan.
- Work across many API servers: the limit is global, not per server.
- Add very little latency (a millisecond or two) and stay available.
- Tell clients when they're limited: HTTP 429 with
Retry-AfterandX-RateLimit-Remainingheaders.
Where it runs
Usually in the API gateway or as middleware in each service, backed by a shared fast store (Redis) so all servers see the same counters. Rules ("free plan: 100/min on /search") live in config and are cached in memory by each limiter.
Algorithm and storage
Token bucket and sliding-window counter both need just a couple of numbers per client, so they fit easily in Redis. The key point is atomicity: two servers doing read-then-write concurrently can both see 99 and both allow request 100. Do the check-and-update in one atomic step, with a Lua script or atomic INCR.
-- fixed-window counter, run atomically as a Redis Lua script-- KEYS[1] = "rl:{user}:{endpoint}:{minute}", ARGV[1] = limitlocal n = redis.call('INCR', KEYS[1])if n == 1 then redis.call('EXPIRE', KEYS[1], 60) endif n > tonumber(ARGV[1]) then return 0 endreturn 1
A fixed window allows up to 2× the limit around a window boundary. The sliding window counter fixes most of that: estimate = current count + previous count × (fraction of the previous window still inside the last 60 seconds).
Scale and failure
- Shard counters across Redis nodes by client key; each client's counters live together.
- If Redis is unreachable, decide in advance: fail open (allow traffic, protect availability) or fail closed (reject, protect the backend). Most public APIs fail open.
- To cut latency, servers can keep local counters and sync to Redis in small batches, accepting slight overshoot.
- Rate limit early: at the edge or gateway, before expensive work happens.
See it animated
Interview questions
Try answering out loud first, then open the model answer.
Q1Why can't each server just keep its own counter?beginner
With 10 servers behind a load balancer, a client could get 10× the limit, and the effective limit would change whenever you scale. A shared store gives one global count. (Local counters are fine as an approximation if you divide the limit by the number of servers.)
Q2How do you prevent race conditions in a distributed limiter?intermediate
Make the read-modify-write atomic in the store: Redis INCR is atomic, and a Lua script runs as one unit, so the check and update can't interleave between servers.
Q3Fail open or fail closed?intermediate
Depends on what the limiter protects. If it mainly enforces fair usage, fail open so a Redis outage doesn't become a full outage. If it protects a fragile backend or stops abuse such as login brute-forcing, fail closed or fall back to conservative local limits.
Q4Which algorithm would you choose?intermediate
Token bucket when short bursts are acceptable and you want a steady average (most API limits). Sliding window counter when you need a smooth limit without boundary bursts at low memory cost. Sliding window log only when exactness matters and volumes are small.
Quick check
0 of 3 answered