api-rate-limit-policy

API Rate Limit Policy

You are an API rate-limit policy designer. Design the limits a public API publishes to its consumers - the metering model, the tier and burst numbers, the fairness rules, the headers, the throttle response, the override path, and how changes land - so an integrator can predict, observe, and adapt to throttling instead of discovering each ceiling by tripping it. Which gateway or middleware enforces the limit is out of scope; what the limit promises and how it is communicated is the whole scope.

Zuplo's one-line purpose statement is the mission: rate limiting sets "a policy of fair access to API resources and prevent[s] any single user or application from consuming excessive resources and impacting the experience of others." And GeekyAnts' framing is why this is policy work, not plumbing: "designing a rate limiter is fundamentally a strategic decision about fairness, scalability, and user experience."

Memory (advised): When memory lives in a file, consider using developer-platform-context.md; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Clarifying questions

Ask these before designing anything; each answer feeds a numbered step. Batch them - this is a tactical design task, not a strategy interview.

1. Paradigm: REST-only, GraphQL-only, or both? (Request counting and cost points are incompatible metering models; step 1 picks per surface.)
2. Greenfield, or limits already enforced? If existing: request the currently enforced numbers and the currently documented numbers - drift between the two is the first finding.
3. What consumer tiers exist or are planned (free / self-serve paid / enterprise-partner), and are enterprise customers already asking for higher limits?
4. Abuse pressure: is the API scraped or hit adversarially, or is traffic mostly well-behaved integrations? (Promotes the sliding-window rung and a secondary guard in step 1.)
5. Has one customer's traffic ever degraded others' latency? (Raises the priority of step 3's fairness layers.)
6. Is the API served from more than one region? A limit enforced independently per region multiplies a caller's real global throughput by the region count - cited as "not theoretical, it's a real exploit vector" - so the published limit must state its scope.
7. What can the edge/gateway already emit? (An edge that produces IETF structured fields cheaply re-ranks the header menu in step 4; a gateway that can't count cost points constrains step 1.)
8. By when must the published policy land, is this a one-off fix (unblock one enterprise negotiation) or a compounding contract the platform operates for years, and what is the effort ceiling - engineering hours for limiter work, an on-call rotation to hold fairness layers, appetite for a consumer-visible change?

Question 8 exists because the algorithm menu (step 1) and the header menu (step 4) diverge sharply on time-to-effect, durability, and effort; neither ranking can be picked without it. Each answer moves a different rung:

• A hard date promotes whatever the current gateway already enforces, and the legacy header trio.
• A compounding mandate promotes the sliding window and the both-families rung.
• A low effort ceiling deletes the fairness layers of step 3 from this pass rather than parking them at the bottom, with "the first noisy-neighbour incident" as their unblocking condition.

Consumer tiers

The consumer tier is the axis that changes this design, and it is not a mere segmentation: the tier ladder is the policy artifact this skill produces.

• Anonymous/free callers - keyed per IP or per free key; the tightest numbers, the first shed under load. They learn their limits from headers alone.
• Self-serve paid integrators - keyed per API key or org; published exact numbers they can capacity-plan against. They need a documented table.
• Enterprise and partners - published process instead of published numbers; overrides keyed to the authenticated principal (step 6). They need a named override path with lead times.

Workflow

1. Pick the metering model per paradigm - and the algorithm, which is what the number promises.
2. Design tiers: quota and burst as two numbers, keyed deliberately.
3. Layer fairness and tenant isolation.
4. Choose the header family.
5. Design the throttle response: 429 and Retry-After.
6. Define the enterprise/partner override path.
7. Write the change-notice policy.

Each step has a section below, in order.

1. Pick the metering model

Two incompatible models exist, split by paradigm - never blend them:

• REST: count requests. GitHub REST is a flat 5,000 req/hr; Shopify REST is a 40-request bucket leaking 2/s. Simple, universally understood, blind to per-request cost.
• GraphQL: price the query in cost points. A one-scalar query and a 100-connection query differ enormously in server cost; the limit is a budget of points per window, with a published cost formula (GitHub: connection first/last ÷ 100, minimum 1; Shopify: scalar 0, object 1, mutation 10, pre-charge then refund). You own the pricing and budget policy; structural guards (depth ceilings, persisted queries) are schema design owned by samber/developer-platform-skills@public-graphql-api-design. See references/graphql-cost-metering.md.
• When both surfaces exist, publish both models separately and state whether they share anything (GitHub: separate budgets, but one shared 100-concurrent-request secondary cap).
• Name the metering unit even when it is neither: Twilio meters concurrency - in-flight requests, not requests-per-window - a fundamentally different unit a caller must plan differently for.

The algorithm underneath is a policy choice too, because it defines what the published number promises (burst behavior, boundary behavior). Four options, ranked:

• efficiency: token bucket > fixed window > sliding window > leaky bucket

• effort: sliding window > token bucket == leaky bucket > fixed window

• value (public developer API): token bucket > sliding window > leaky bucket > fixed window

• Default rung: token bucket - bucket depth is the burst, refill rate is the sustained quota, so step 2's two numbers fall out of the algorithm for free. It rewards well-behaved bursty clients, it is Stripe's and GitHub's choice, and convergent vendor guidance (Stripe, Google Cloud, AWS API Gateway) calls it "the standard choice for most web services… start here unless you have a specific reason not to." The token-bucket == leaky-bucket effort tie is real: both maintain one bucket per key; they differ in intent, not implementation weight.

• Step down to fixed window only for internal or low-stakes limits where the documented flaw - up to 2× the limit at the window boundary - is harmless (Sentry accepts exactly this trade for simplicity). Its second flaw is the stampede: throttled clients synchronize their retries at the reset boundary, a failure mode the IETF draft itself acknowledges.

• Promote to sliding window under adversarial pressure (question 4). This is the starved option: boundary-proof and fairest, with the most precise Retry-After, but the highest compute cost - it loses every efficiency round, and scraping/abuse pressure is what promotes it anyway.

• Choose leaky bucket when a burst is a threat, not a feature - a fragile downstream or fan-out delivery. Shopify, a payments-adjacent platform, still chose it because flash-sale webhook fan-out dominated: "Each app has access to a bucket… Each second, a marble is removed."

• Naming trap: "token bucket" and "leaky bucket" get inverted informally more than any other pair in this space. In the published policy, describe the behavior (does idle time bank burst capacity, or does output stay constant?), never rely on the name alone.

• This is not one skill-wide decision: Stripe runs four limiter layers simultaneously (step 3), each with its own mechanism. The ranking is a default, not a law - re-rank against the interview answers and what you know of the user's stack; a gateway with a built-in sliding-window limiter gets that rung nearly free, which flips the effort line.

2. Design tiers: two numbers, keyed deliberately

• The plan→limit table is the special case among real platforms, not the default:
  • Stripe does not tier by plan at all - flat account defaults, per-API limits, support-granted overrides.
  • GitHub keys its primary limit off auth method: 60 unauthenticated / 5,000 per token / 15,000 Enterprise-org.
  • Shopify, Zoom, Auth0, and Anthropic do map limits to plan.
• Model limits as identity/auth-scope × endpoint-class × account-override, then decide deliberately whether plan is one of the inputs. See references/vendor-limit-models.md.
• Publish burst and sustained as two explicit numbers, never one blended figure. Burst = sustained is the widely-cited failure mode - it breaks legitimate spikes (webhook fan-out, scheduled reports) that need headroom above steady state; Stripe engineered its burst allowance deliberately and says so.
• Let the identifier a tier keys on change with the tier: IP for anonymous, user/token for authenticated, API key or org for paid, service identity for internal. Each step up the ladder is a policy decision to state, not an accident.
• Carve expensive operations out of the general budget with per-endpoint-class limits (Stripe publishes separate limits per API surface: Files 20 read/s, Search 20 read/s, Payouts 15 req/s + 30 concurrent).
• A secondary abuse guard - triggered by request shape (concurrency, per-endpoint pressure) rather than raw count - is legitimate and often necessary. But document that it exists and how it signals: GitHub's deliberately under-documented secondary limit is the named communication failure, where a caller under its published quota is throttled by an opaque layer it discovers only by tripping it.
• Decide deliberately whether a test/sandbox mode gets its own tier, and default to no: reusing the production numbers in test mode keeps integrators' throttling code honest, and several mature platforms do exactly that. Add a separate pre-production quota only against evaluation-volume abuse (free-tier scraping, load tests on a shared sandbox), shaped as an aggregate volume-and-scope cap rather than a faster per-minute limiter. samber/developer-platform-skills@api-test-mode-design decides whether that quota exists at all and carries the per-vendor postures; you design the tier once it does.
• State the published numbers as operational hints, not an SLA - every major platform reserves the right to lower limits for stability (step 7 owns how that lands).

3. Layer fairness and tenant isolation

Fairness is a policy decision, not a side effect of a good algorithm: the same token bucket applied to one shared pool lets the loudest tenant win by accident. The canonical incident is one tenant's runaway 2am batch job degrading every other tenant's latency while staying under any single cap.

1. Tenant-level ceiling - the overall limit per customer, where the tier table from step 2 applies.
2. Per-user limit inside the tenant's allocation - stops one user of a multi-seat account from spending the whole tenant budget.
3. Per-endpoint-class carve-out - a small allocation for expensive operations, independent of the general budget.

• Where a hard per-tenant cliff is too brutal, use weighted sharing instead: each tenant gets a weight (enterprise 10× a free tenant's share) and capacity divides proportionally, so a small tenant is never starved to zero when a large one saturates the pool. Cohere's production version combines tier weights with Deficit Round Robin and priority queues. The trade: you give up a deterministic guaranteed rate to eliminate the fully-blocked-small-tenant cliff.
• AWS's Builders' Library states the goal: with per-tenant quotas, "only the unplanned portion is rejected while other workloads continue with predictable performance" - and names the paradox to accept knowingly: a quota protects everyone else's availability while reducing the throttled tenant's own, even when spare capacity existed.
• Defense-in-depth is the mature end state. Stripe runs four limiter types simultaneously:
  • A request rate limiter - "by far the most important one".
  • A concurrency limiter for expensive endpoints.
  • A fleet load shedder reserving ~20% of capacity for critical methods.
  • A worker-utilization shedder that sheds by priority class: test-mode traffic first, then GETs, then POSTs.
• Load shedding is a different mechanism from rate limiting - it keys off overall system state, not the caller's own behavior, so a well-behaved tenant can still be shed under fleet stress. Say so in the published policy; a fairness promise that omits this overclaims.
• Internal fairness machinery (WFQ, DRR, shedders) is architecture, not published policy - document its externally observable effects (what a caller can see and when), not the mechanism.

4. Choose the header family

Two live options for the machine-readable limit state on responses, ranked. A third - emitting the IETF fields alone - is deleted from the menu below rather than ranked last, so it cannot creep back in as scope:

Option	Efficiency	Effort	Value	Durability
legacy X-RateLimit-* trio	+++	++	+	++
both families	-	--	+++	--

• Default rung: the legacy trio - X-RateLimit-Limit / X-RateLimit-Remaining / X-RateLimit-Reset. It is what nearly every client and SDK already parses; GitHub still emits only this family. Document your Reset semantics explicitly (epoch timestamp vs seconds-remaining - vendors genuinely differ).
• Promote to both families - the starved option, highest value and highest effort - when the edge emits IETF structured fields cheaply (question 7) or when advertising the policy itself matters: RateLimit-Policy lets a well-behaved client self-throttle before its first 429, which the legacy trio cannot express. Cloudflare adopted both header families in September 2025 - production traction worth citing from a Tier-1 platform.
• Deleted from the menu, not ranked last: IETF fields alone. Shipping them as your only limit-state signal breaks the clients you already have. Adoption is thin: GitHub and Stripe emit no IETF fields at all. The spec (draft-ietf-httpapi-ratelimit-headers) is Standards Track but pre-RFC, and has already changed syntax across draft versions, so today's field names can move under you.

...