What Is Web Crawling? How Bots Actually Read Your Site

Jul 29, 2026 · SEO Glossary

Somewhere in Google's infrastructure right now, a scheduler is deciding whether your homepage is worth fetching again today. If yes, a machine opens an HTTP connection, sends GET / HTTP/1.1 with a Googlebot user-agent, stores whatever bytes come back, extracts every <a href> from the HTML, and feeds those URLs back into the same scheduler. Repeat a few billion times a day and you have a search engine's view of the web. That loop is the answer to what is web crawling — and once you can picture it, most indexing mysteries stop being mysterious.

What Is Web Crawling, Step by Step?

A crawler (spider, bot — same thing) maintains a frontier: a prioritized queue of URLs it knows about. The loop runs:

1. Select the next URL from the frontier, based on priority and politeness constraints.
2. Check permissions — fetch and honor the site's robots.txt before requesting anything else.
3. Fetch the URL, recording status code, headers, and body.
4. Parse the response: extract links, canonical tags, meta robots directives, structured data.
5. Enqueue newly discovered URLs, deduplicated against everything seen before.

Crawling is only step one of search: the fetched page then goes to indexing (parsing, quality evaluation, storage) and later ranking. Crawled-but-not-indexed is a normal, common state — appearing in Search Console as exactly that phrase.

Discovery: How Bots Learn Your URLs Exist

A page no crawler has heard of cannot be crawled, and discovery happens through exactly three channels: links from already-known pages (the main one — which is why internal linking is a crawling concern, not just a UX one), XML sitemaps you submit, and past crawl history (Googlebot revisits URLs for years after you delete them, which is why old URLs keep showing up in logs). The classic failure is the orphan page: published, live, in the CMS, but linked from nowhere — invisible to link-following discovery and reachable only if a sitemap mentions it.

Politeness, Budgets, and Why Bots Don't Hammer You

A crawler that fetched as fast as it could would take small sites offline, so real crawlers self-throttle: limited concurrent connections per host, delays between requests, and automatic backoff when the server slows or starts returning 5xx errors. Google adjusts its crawl rate continuously based on your server's response times — one reason chronic slowness has a compounding cost: fewer pages fetched per day, slower discovery of updates.

For large sites, this scheduling becomes a resource-allocation problem with a name — crawl budget: roughly, how many URLs a crawler will fetch from your site per unit time. Faceted navigation, infinite calendar pages, session-ID parameters, and redirect chains all burn budget on garbage. Under about 10,000 well-linked URLs you can mostly ignore the concept; at 500,000 it's a first-order concern.

The JavaScript Twist

The classic crawl loop reads HTML as served. But if your content is assembled in the browser by a JavaScript framework, the fetched HTML is an empty shell — so modern search crawlers added a second phase. Google queues fetched pages for rendering in a headless Chromium, executes the scripts, and re-extracts content and links from the rendered DOM. It works, with costs: rendering is far more expensive than fetching, so it happens later (sometimes much later), and links that only exist post-render enter the frontier on that delay. Two commands show you the gap: curl https://example.com/page gives the crawler's first-pass view, and DevTools' "View page source" versus Elements panel comparison shows what only exists after rendering. If the content you care about is missing from the first view, you're betting your indexing speed on the render queue — server-side rendering removes the bet.

Crawling Beyond Google

Search engines are the famous crawlers, but the same loop powers archive projects, AI training-data collectors (announcing themselves with their own user-agent strings), SEO link-index bots, and site audit tools. When WebsiteChecker.Tech crawls your site, it runs precisely this discover-fetch-parse loop over your domain and reports what a search engine would encounter: broken links, redirect chains, orphan pages, duplicate titles, noindex flags. The practical value of understanding crawling is that an audit crawl becomes a preview — anything that trips the audit bot will trip Googlebot the same way.

Reading the Evidence: Logs and Search Console

Two data sources tell you what crawlers actually did, as opposed to what you hope they did:

• Server access logs. Filter for verified crawler IPs and you get ground truth: which URLs Googlebot fetched, when, how often, and what status codes it got. Patterns worth hunting: heavy fetching of parameter URLs (budget waste), repeated 404s on long-deleted paths (fix with 410s or leave them — they're harmless if genuinely gone), and sections never visited at all (discovery problem).
• Search Console's Crawl Stats report (under Settings) summarizes 90 days of Googlebot activity: requests per day, average response time, breakdown by response code and file type. A rising average response time with falling request counts is the budget-throttling feedback loop in action, visible on one chart.

The mental model to keep: crawling is a cautious, budgeted, permission-checking loop that only knows what links and sitemaps tell it. Make every important page reachable, fast, and unambiguous, and the loop does the rest on its own.