Rate Limiting: Token Bucket vs Sliding Window

Token bucket refills tokens at a steady rate and allows short bursts up to bucket capacity — good for normal API traffic where a dashboard may fan out a few requests at once. Sliding window counts requests in a rolling interval — stricter, fewer boundary spikes than a naive fixed window, less burst room.

Decision order: user-facing APIs that tolerate small bursts → token bucket. Login, OTP, SMS, or anti-abuse endpoints → sliding window (or a tight fixed window). Fixed window counters are simpler but can double-allow at window edges. Always return 429 with Retry-After so well-behaved clients back off.

The token bucket holds up to a maximum capacity (burst). Tokens refill continuously at a configured rate (sustained limit). Each allowed request consumes one token. If the bucket is empty, reject with 429. Example: capacity 20 and refill 100/minute allows a short burst of 20, then roughly 1.67 requests/second thereafter.

Burst-friendly behavior fits APIs where a page load may hit several endpoints at once. Keep the bucket size small enough that a burst cannot overwhelm the slowest downstream dependency. Many managed gateways and API products offer token-bucket-style quotas; always confirm the vendor's docs for your product — algorithms and defaults differ.

A sliding window counts requests inside a rolling duration. For 100 requests per minute, count timestamps in the last 60 seconds — not only the current clock minute. On each request, drop entries older than the window; if the count is at the limit, reject.

That avoids the classic fixed-window edge spike (100 at 00:59 and 100 at 01:00 under a "100/min" rule). Cost: memory for timestamps (Redis sorted sets with ZADD / ZREMRANGEBYSCORE / ZCARD are a common approach). Approximate sliding-window counters trade a little precision for less memory. Prefer sliding or tight windows on login and messaging endpoints where bursts are a security risk.

Put rate limiting at the edge — API gateway, NGINX, or early middleware — before business logic spends real work. Share counters in Redis (or the gateway's store) so every app instance sees the same budget. Return 429 with Retry-After and, when useful, X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset.

Tier by plan and by route: generous reads, strict /login. Monitor 429 rates — spikes can mean abuse, a client bug, or limits that are too low. For multi-region systems, decide whether limits are global or per-region; global counters add latency and consistency questions. Gateway modules (for example NGINX limit_req) implement specific algorithms — read the module docs rather than assuming "sliding window" or "token bucket" universally.

Trigger. The non-atomic sliding-window sketch above — ZREMRANGEBYSCORE then ZCARD then a conditional ZADD, as three separate round trips — runs under a burst of 20 concurrent requests hitting the same key within milliseconds of each other. Symptom. The endpoint's configured limit is 100 requests per minute; monitoring shows 119 requests succeeded in that window. Nobody changed the limit, and no single request did anything wrong.

Root mechanism. Between the ZCARD read and the ZADD write, many of those 20 concurrent requests observe the same pre-increment count — say, 99, which is still under the limit of 100 — and all of them pass the check before any of them has actually recorded its own request. Each one independently thinks it's the request that brings the count to 100; none of them see each other's writes in time. Recovery: move the read-check-write sequence into a single Redis Lua script (EVAL), which Redis executes atomically — no other client's commands can interleave in the middle of it, closing the exact gap that let 20 requests all pass the same stale count. Verification: replay the same 20-concurrent-request burst against the Lua-script version and confirm exactly 1 (or however many the limit actually allows past the current count) succeed, with the rest receiving 429. Prevention: treat any check-then-act sequence spanning more than one round trip to shared state as a race condition by default, not an edge case — the fix is almost always collapsing it into one atomic operation, not adding a delay and hoping.