Distributed Rate Limiting and Throttling in SaaS Platforms

Choosing an algorithm

A fixed window counter is the simplest and the least fair: a client can send the full allowance at the end of one window and again at the start of the next, producing double the intended rate across the boundary.

Token bucket allows bursts up to a bucket size while enforcing a sustained refill rate, which matches real client behaviour such as a batch job that occasionally spikes and is generally the best default for API limits.

Sliding window log or sliding window counter smooths enforcement by considering the actual distribution of recent requests, removing the boundary burst problem. It costs more memory or requires careful approximation, so it is worth the cost where fairness matters more than peak tolerance.

Making limits hold across instances

A rate limiter that stores counters in application memory enforces the limit once per instance, so a fleet of twenty instances permits twenty times the intended rate. Limits must be evaluated against shared state or enforced at a single ingress point.

The usual implementation is an atomic operation in a shared store such as Redis, executed as a Lua script so the read, decision and decrement happen atomically. Without atomicity, concurrent requests can each read the same count and all be allowed.

Terminate at the edge where possible. Enforcing coarse limits at a gateway or CDN before requests reach application services protects the services, but keep a second, finer limit at the application layer for per-principal fairness, because the edge knows less about the caller.

• Use a shared store with atomic read-modify-write operations.
• Enforce coarse limits at the edge and fine limits in the application.
• Apply limits per tenant, per principal and per endpoint class.
• Degrade to a conservative local limit if the shared store is unavailable.

Fairness and quotas

A single global limit lets one tenant consume the entire allowance and deny service to everyone else, which is a denial of service by accident. Limits must be scoped per tenant, with aggregate protections on top to protect shared infrastructure.

Where tenants have contractual quotas, distinguish quota from protection. A quota is a commercial entitlement enforced precisely and reported to the customer; a protection limit is an operational guard applied to everyone. Conflating them produces either angry customers or unprotected systems.

Consider priority classes. Interactive requests from a user interface and bulk background jobs from the same tenant should not consume the same budget, because a batch import should not lock a user out of the interface.

• Scope limits per tenant to prevent accidental denial of service.
• Separate contractual quotas from operational protections.
• Use priority classes so batch work cannot starve interactive use.
• Report quota consumption to tenants.

The client contract and operations

Clients need to be told what happened. Return standard limit headers describing the limit, the remaining allowance and when it resets, and use a distinct status and error code for throttling so clients can back off intelligently rather than treating it as a generic failure.

Document retry behaviour. Exponential backoff with jitter is the baseline expectation, and servers should include a retry-after hint where a specific wait is known.

Operationally, monitor how often limits are hit by tenant and endpoint. A rising rejection rate may indicate an abusive client, a misconfigured integration, or a limit that is simply too low for legitimate use, and each requires a different response. Limiting is a signal as much as a control.

• Return limit, remaining and reset headers on every response.
• Use a distinct throttle status with a retry hint.
• Document backoff with jitter for client implementers.
• Alert on rejection rate by tenant and endpoint.