What Is Crawl Budget? How Googlebot Decides What to Fetch

Jul 29, 2026 · SEO Glossary

Pull a week of server logs and filter for Googlebot's user agent (verify the IPs against Google's published ranges — plenty of scrapers impersonate it). On a 50,000-URL e-commerce site I audited, Googlebot made about 38,000 fetches in that week. Sounds healthy, until you group by URL pattern: 41% of those fetches went to faceted filter combinations and session-parameter URLs that were never going to be indexed. The product pages that actually mattered were being re-crawled every eleven days instead of daily. That gap is what crawl budget is about.

What is crawl budget? It's the number of URLs Googlebot will crawl on your site within a given timeframe — the practical output of two forces Google balances continuously: how much your server can handle, and how much Google wants your content.

The two halves: crawl capacity and crawl demand

Crawl capacity limit is the ceiling. Googlebot watches your server's behavior — response times, 5xx errors, timeouts — and throttles itself to avoid degrading your site. If your responses slow from 200ms to 900ms, crawl rate drops. If you start returning 503s, it drops sharply. This is measurable in Search Console's Crawl Stats: watch average response time against total crawl requests and you'll often see them move inversely.

Crawl demand is the appetite. Popular URLs get re-crawled more often. Content that changes frequently gets revisited frequently; a page unchanged for two years gets checked rarely. New sites and unpopular sections generate little demand regardless of how much capacity exists.

Your effective budget is whichever half is lower. A fast server hosting content nobody links to has spare capacity and no demand. A popular site on a struggling server has demand it can't serve.

Who actually needs to care

Google's own guidance is blunt: sites under roughly a million pages that update content weekly generally don't need to think about crawl budget. If you run a 200-page business site and your posts get indexed within a day or two, this is not your bottleneck — your time is better spent on content and links.

You do need to care if any of these describe you:

• Large inventory (100k+ URLs), especially e-commerce with faceted navigation multiplying URL combinations
• New pages taking weeks to be discovered, or updated pages showing stale versions in search for long periods
• Search Console showing large counts under "Discovered – currently not indexed"
• A news or listings site where freshness is the product

What burns crawl budget for nothing

The waste is rarely one big leak; it's four or five medium ones stacking:

• Parameter and facet URLs. ?color=blue&sort=price&page=3 and its ten thousand siblings. The single largest sink on most commerce sites.
• Redirect chains. Each hop is a separate fetch. A three-hop chain triples the cost of reaching one page.
• Soft 404s. Empty category pages returning 200 keep getting re-crawled because nothing tells Google they're gone.
• Infinite spaces. Calendar widgets with a "next month" link forever, or search-results pages that generate URLs endlessly.
• Duplicate hostnames and protocols. If http, https, www and non-www all resolve with 200s, every URL exists four times.

Reading the Crawl Stats report

Search Console → Settings → Crawl Stats is the closest thing to log analysis without logs. The breakdowns tell you where fetches go:

• By response: a healthy site is dominated by 200s. A high share of 301s or 404s means Googlebot is spending fetches on URLs you've already moved or removed — usually stale internal links.
• By purpose: "discovery" vs "refresh." Heavy discovery on a stable site suggests it keeps finding junk URLs.
• By file type: if a third of fetches are JS and CSS assets, rendering costs are eating into page fetches.

Reclaiming wasted crawls

1. Block the infinite spaces in robots.txt. A Disallow: /*?sort= style rule stops the fetches entirely — the one thing robots.txt genuinely does well. (Don't block URLs you want removed from the index; blocked URLs can't be re-crawled to see a noindex.)
2. Flatten redirect chains. Point internal links at final destinations and collapse multi-hop redirects into single 301s.
3. Return real 404/410s for gone content instead of soft 404s.
4. Fix server speed. Faster responses raise the capacity ceiling directly — the same fetch window fits more URLs.
5. Keep sitemaps honest. A sitemap containing 30% redirected or dead URLs teaches Google to trust it less.

Finding these issues is a crawl problem before it's a fix problem: you need the full list of redirect chains, dead internal links, and parameter URL patterns on your site. A crawler that walks your site the way Googlebot does produces exactly that inventory, and pairing it with the related concept of crawlability covers the other half of the question — not how much Google crawls, but whether it can reach the pages at all.

One persistent myth

Crawl budget is not a rankings lever. Getting crawled more doesn't rank you higher; plenty of junk gets crawled daily and never ranks. The budget matters only insofar as important pages are being discovered late or refreshed rarely. Fix that, and further crawl optimization returns nothing.