HTTP 429 Too Many Requests: Rate Limits Explained

Aug 05, 2026 · HTTP Status Codes

The first time I got paged for a 429 storm, the site wasn't down. CPU was fine, memory was fine, the database was bored. What had happened was subtler: a scraper had discovered our faceted search and was requesting every filter combination at 40 requests a second, our rate limiter had correctly started returning the 429 status code — and it was throttling everyone behind the same office NAT, including our biggest customer. The limiter worked exactly as configured. The configuration was the incident.

That's 429 in a nutshell. It's not a failure; it's a policy decision made in code. Which means every 429 investigation is really two questions: is the limit right, and is it hitting the right people?

What the 429 status code means in practice

A server returns 429 Too Many Requests when a client has exceeded whatever request budget the server enforces — requests per second per IP, API calls per hour per token, login attempts per account. Unlike a 5xx, nothing is broken. The server looked at your request, counted it against a quota, and declined. It's the only common status code whose entire meaning is "slow down."

The enforcement can live anywhere in the request path: the CDN edge (Cloudflare rate limiting rules), the reverse proxy (nginx's limit_req module), an API gateway, or application code checking a counter in Redis. Where it lives determines where you'll find the evidence — a 429 issued at the CDN never appears in your origin logs at all.

Who's actually hitting your rate limit

Before touching any config, identify the traffic. The breakdown is usually one of these:

• Scrapers and bots. The most common trigger. Look for high request rates spread across many URLs from one IP or one subnet, often with rotating or dishonest user agents.
• One legitimate client with a bug. A retry loop without backoff is a self-inflicted denial of service — a mobile app that retries instantly on any error will hit a rate limit and then hammer it harder. The signature is one API endpoint, one client version, requests spaced milliseconds apart.
• Many users behind one IP. Corporate NATs, university networks, and mobile carriers put thousands of humans behind a handful of addresses. Per-IP limits punish them collectively. If your 429s cluster on a few IPs with diverse, human-looking behavior, this is your problem.
• Search engine crawlers. Googlebot crawls in bursts, and an aggressive per-IP rule can catch it. This is the case that quietly damages you weeks later.

Your access logs answer this quickly. Group 429 responses by IP and user agent — something like awk '$9 == 429 {print $1}' access.log | sort | uniq -c | sort -rn | head against a standard nginx log — and the shape of the problem falls out in seconds.

Reading Retry-After and the rate-limit headers

A well-behaved 429 tells the client when to come back. The Retry-After header carries either seconds (Retry-After: 120) or an HTTP date. Many APIs add the de facto standard trio: X-RateLimit-Limit (your quota), X-RateLimit-Remaining (what's left), and X-RateLimit-Reset (when the window resets). If you operate an API, send these headers — clients that can see the budget stop guessing, and guessing is what creates retry storms. If you consume an API, honor them: sleep for the Retry-After value plus a little jitter, and never retry a 429 instantly.

Why Googlebot and 429 don't mix

Google explicitly treats 429 as a signal that your server is overwhelmed, and it responds by slowing its crawl. Serve Googlebot enough 429s and your crawl rate drops; keep it up for weeks and pages start going stale in the index, with URLs eventually dropping out — the same trajectory as persistent 5xx errors. The insidious part is that nothing looks broken. Your site is up, users are happy, and meanwhile your crawl budget is being spent on rejections.

Check Search Console's crawl stats for a rising 4xx share, and check your logs for 429s served to verified Googlebot addresses. If you find them, either raise the limit for verified crawlers or exempt them from per-IP rules entirely — Googlebot respects its own pacing based on how your server responds, so throttling it with 429s is redundant punishment. And if your server genuinely can't handle crawl load, that's a capacity conversation, not a rate-limiting one — closer to the territory of a 503.

Fixing 429s you're serving

1. Size limits from data, not vibes. Measure your real traffic's per-IP percentiles, then set the limit above the 99th percentile of legitimate behavior. In nginx, limit_req_zone $binary_remote_addr zone=main:10m rate=10r/s; with a sensible burst= parameter absorbs normal spikes; the burst buffer is the difference between smoothing traffic and rejecting it.
2. Key on something better than IP where you can. API token, session, or account-based limits don't collectively punish everyone behind a NAT.
3. Always send Retry-After. It converts angry retry loops into polite scheduled ones.
4. Allowlist verified crawlers — verify by reverse DNS, not user-agent string, since scrapers lie about being Googlebot constantly.
5. Handle abuse as abuse. If one client ignores 429s and keeps hammering, escalate from rate limiting to blocking at the firewall or CDN. Rate limits are for pacing, not for stopping determined bad actors.

Fixing 429s you're receiving

When your code is the one being throttled by someone else's API, the fix is client-side discipline: exponential backoff with jitter (wait 1s, 2s, 4s, 8s, each plus a random offset so a fleet of clients doesn't retry in lockstep), respect for Retry-After when present, and request budgeting so you never approach the documented quota in the first place. Caching responses you'd otherwise re-fetch is usually the biggest single win — most 429 trouble with third-party APIs turns out to be requesting the same data repeatedly.

Last thing: 429s are easy to miss because the site looks healthy while they happen. A monitoring setup that only checks the homepage won't catch a rate limiter misfiring on deeper pages, and it certainly won't catch Googlebot being throttled. Run a scheduled crawl of your site so you see every URL's status code the way a crawler sees it — that's how a misconfigured rate limit shows up before your search traffic does the telling.