What Is a Disallow Directive in robots.txt?

Jul 27, 2026 · SEO Glossary

Two lines of plain text can stop Googlebot from fetching your entire site:

User-agent: *

Disallow: /

That trailing slash is the whole instruction. It matches every path on the host, and every compliant crawler will stop requesting your pages. Sites have shipped exactly this to production — usually a leftover from staging — and watched their crawl stats flatline. Understanding precisely how this directive matches URLs is the difference between surgical crawl control and a self-inflicted outage.

What is a disallow directive? It's a rule in your robots.txt file that tells crawlers not to request URLs whose paths begin with a given string. It controls fetching, nothing else — a distinction that matters more than the syntax.

The matching rules, exactly

A disallow value is a path prefix, matched from the start of the URL path. The rules:

• Disallow: /admin blocks /admin, /admin/, /admin/users, and also /administrator — it's a prefix match on the string, not on path segments. If you mean the directory, write Disallow: /admin/.
• Disallow: with an empty value blocks nothing (it's how you write "allow everything" explicitly).
• Matching is case-sensitive: Disallow: /Private/ does not block /private/.
• Two wildcards exist in the de facto standard Google follows: * matches any sequence of characters, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf; without the $ it would also block /report.pdf-download/.
• Query strings are part of the matchable URL: Disallow: /*?sessionid= blocks any URL containing that parameter.

A few housekeeping facts about the file the directive lives in: comments start with # and run to end of line, so you can annotate why a rule exists (do this — robots files outlive the people who wrote them). The file must live at the host's root (/robots.txt) and applies only to that exact host and protocol — a rule on www.example.com says nothing about shop.example.com. And Crawl-delay, which you'll see in older files, is ignored by Google entirely; Bing and Yandex honor it, but for Googlebot the only rate control is Search Console's crawl settings and your server's actual behavior.

Which rule wins when several match

Rules are grouped under User-agent lines, and a crawler obeys only the most specific group that matches it — Googlebot reading a file with both User-agent: googlebot and User-agent: * groups uses only the googlebot group. The catch-all group is ignored entirely for it, which surprises people who put shared rules only under *.

Within a group, when both an Allow and a Disallow match a URL, the longest (most specific) rule wins. This enables carve-outs:

Disallow: /downloads/

Allow: /downloads/brochure.pdf

The Allow is longer, so the brochure stays fetchable while the rest of the directory is blocked. On a tie, Google takes the less restrictive option.

What the disallow directive does not do

Three non-effects, each a common misconception:

1. It doesn't remove pages from the index. An already-indexed URL that gets disallowed stays in results as a bare link — Google just can't refresh it. Actual removal is a different job with different tools, covered under de-indexing.
2. It doesn't hide anything from people. robots.txt is a public file; listing Disallow: /secret-admin-panel/ advertises the path to anyone who looks. Access control is authentication's job.
3. It doesn't bind bad actors. Compliance is voluntary. Googlebot and Bingbot comply; scrapers mostly don't.

Patterns that break sites

• Blocking rendering assets. Disallow: /wp-includes/ or Disallow: /assets/ prevents Googlebot from fetching the CSS and JS it needs to render your pages. Rendered-blank pages rank like blank pages.
• Blocking a path you want de-indexed. If the page can't be fetched, its noindex tag can never be seen. De-index first, block after (if at all).
• The overbroad prefix. Disallow: /search intended for internal search also blocks /search-engine-optimization-guide/. Prefix matching is literal; end directories with a slash and use $ when you mean an exact URL.
• Case mismatches on Windows-hosted legacy sites where both /Products/ and /products/ resolve.

What disallow is genuinely good at

Its proper job is crawl traffic management — keeping bots out of URL spaces that waste fetches, which on large sites protects crawl budget for pages that matter:

• Faceted and sorted parameter URLs: Disallow: /*?sort=, Disallow: /*&filter=
• Internal search results: Disallow: /search/
• Cart, checkout, and session paths that generate infinite unique URLs
• Duplicate print or API views of existing pages

Testing before you ship

Every robots.txt change deserves a test, because the blast radius of a wrong rule is the whole site. Search Console's robots.txt report shows the file Google currently holds, when it last fetched it, and any parse errors. For rule-level checks, test specific URLs against your rules before deploying — and after deploying, watch crawl stats for unintended drops. A site crawler that respects your robots rules gives you the pre-flight view: run it against the new rules and see exactly which URLs become unreachable, before Googlebot finds out the hard way.