What Is Robots.txt? Crawl Control at the Front Door

Jul 28, 2026 · SEO Glossary

Tail your access logs during a Googlebot visit and you'll notice it fetches /robots.txt before crawling anything else — and refetches it periodically, generally caching it for up to 24 hours. That little text file is the first conversation your server has with every legitimate crawler, and its rules decide which URLs the bot may request at all.

What is robots.txt, then? A plain-text file at the root of your host (exactly https://example.com/robots.txt — no other location counts) that implements the Robots Exclusion Protocol, formally standardized as RFC 9309 in 2022 after nearly 30 years as a de facto convention. It grants and denies crawling, per user agent, per URL path. What it does not do — and this misconception causes real damage — is control indexing. More on that below.

The Syntax That Covers 95% of Cases

Rules are grouped by user agent. A group starts with one or more User-agent lines and contains Disallow and Allow rules:

User-agent: *

Disallow: /cart/

Disallow: /search?

Allow: /search?q=featured

Sitemap: https://example.com/sitemap.xml

Matching details that actually matter in practice:

• Paths are prefix matches: Disallow: /private blocks /private/, /private.html, and /privateers/ — that last one surprises people.
• Two wildcards exist: * matches any character sequence, $ anchors the end. Disallow: /*.pdf$ blocks all PDFs.
• When Allow and Disallow both match, the more specific (longer) rule wins for Google.
• A crawler obeys the most specific matching user-agent group only — if there's a User-agent: Googlebot group, Googlebot ignores the * group entirely. Forgetting to repeat shared rules in the specific group is a classic bug.
• Disallow: with nothing after it means "allow everything." Disallow: / means "crawl nothing." One character between routine and catastrophe.
• Google reads only the first 500 KiB of the file — relevant if you generate enormous rule sets.

How Googlebot Behaves Around the File

This is where robots.txt gets interesting, because the failure modes differ wildly by HTTP status:

• 200: rules parsed and applied, cached typically up to 24 hours. Changes you deploy now take effect on the next refetch, not instantly.
• 404 (or any 4xx): treated as "no restrictions" — Google crawls everything. A missing file is safe; sites without one are crawled normally.
• 5xx or timeout: the dangerous one. Google treats an erroring robots.txt as "assume total disallow" for the first 12 hours and stops crawling the site. If errors persist up to 30 days, it uses its last cached copy; beyond that it behaves as if there are no restrictions. A misconfigured server that 500s on /robots.txt can therefore quietly halt your crawling while every content page works fine.

You can see all of this in Search Console's robots.txt report (under Settings), which shows the fetched versions, fetch errors, and lets you inspect what Google's parser extracted.

Worth knowing which directives don't work, because old advice keeps them circulating. Crawl-delay is ignored by Google entirely (Bing respects a version of it); Google's crawl rate is managed automatically based on your server's responsiveness. Noindex inside robots.txt was never official and Google stopped honoring it in 2019 — files still relying on it are silently doing nothing. Meanwhile the file has picked up a new modern job: AI crawlers like GPTBot, ClaudeBot, and Google-Extended (which controls AI training use, not search indexing) each honor their own user-agent group, so robots.txt is now where you decide which machine audiences may read your content — search, AI training, or both.

What Is Robots.txt Not? The Indexing Myth

Blocking a URL in robots.txt does not remove it from Google's index. If other pages link to the blocked URL, Google can index the URL itself — showing it in results with the infamous "No information is available for this page" or a title reconstructed from anchor text. Worse: because the page can't be crawled, Google can't see a noindex tag on it. Robots-blocking a page you want deindexed actively prevents the deindexing.

The correct tool depends on the goal: to remove from the index, allow crawling and serve noindex (or a 404/410, or use the removals tool for urgency). To keep private things private, use authentication — robots.txt is a publicly readable file, and listing /admin-backup-2024/ in it is an announcement, not a protection. Robots.txt is for one job: keeping crawlers from spending requests on URL spaces that waste them — faceted filter combinations, infinite calendars, cart and session URLs, internal search results. That's crawl budget management, and it's the file's real value on large sites.

Auditing Your Own File

Three checks catch most problems. First, fetch it yourself and read it — you'd be surprised how many production files still contain a staging-era Disallow: / or block /wp-content/ (which blocks CSS and images Google needs for rendering; Google explicitly asks you not to block assets). Second, cross-check against your XML sitemap: URLs that are simultaneously in the sitemap and robots-blocked are contradictory instructions, and Search Console flags them. Third, crawl your site and compare what a crawler can reach against what you intended — a good audit lists every URL blocked by robots.txt so you can scan the list for things that shouldn't be on it. The file is only ~20 lines on most sites; the cost of misreading one of them is what makes it worth the periodic five-minute review.