What Is a User-Agent? The Header That Names Every Bot

Jul 29, 2026 · SEO Glossary

Grep any server access log and every line ends with a string like Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). That's a User-Agent, and it's how you know that particular request came from Google's crawler rather than a person — or at least, how the request claims it did. If you've ever wondered what is a user-agent and why every browser string bizarrely begins with "Mozilla", this is the field guide.

What Is a User-Agent in HTTP Terms?

The User-Agent is a request header — one line of metadata a client sends with every HTTP request — identifying the software making the call:

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

It's free text. Nothing validates it, nothing enforces honesty, and any client can send anything: curl -A "MyBot/1.0" sets it in one flag. That single fact — self-declared, unverifiable — drives everything else in this article.

About that "Mozilla/5.0" prefix on every browser: it's thirty years of compatibility sediment. Sites in the 90s served better pages to Netscape ("Mozilla"), so Internet Explorer claimed to be Mozilla-compatible, then everyone claimed to be everyone, and today Chrome's UA nominally mentions Mozilla, AppleWebKit, KHTML, Gecko, and Safari. It's also frozen history: since 2023 Chrome ships a reduced UA that pins the OS version and patch numbers to static values, with the detail moved to opt-in Client Hints headers — so don't build anything that parses UA strings for precise versions.

The Strings Worth Recognizing on Sight

Since AI crawlers arrived in force, the roster of bots hitting a typical site has roughly doubled, and deciding who's welcome is now a routine policy question rather than an edge case.

Where SEO Meets the Header: robots.txt Groups

Crawl rules are scoped by user-agent token — a short product name, not the full string. In robots.txt:

User-agent: Googlebot
Disallow: /internal/

User-agent: GPTBot
Disallow: /

Two rules trip people up. First, a crawler obeys the most specific group that matches it and ignores User-agent: * entirely once a named group matches — so adding a Googlebot group means restating any general rules you still want applied to it. Second, matching is against Google's documented tokens, not substrings of the full UA; check the official crawler list when in doubt.

There's a subtler layering issue too: several of Google's products share the Googlebot infrastructure but honor different tokens. Google-Extended, for instance, never appears in your access logs as its own UA — requests arrive under the normal Googlebot string, and the token exists purely as a robots.txt switch controlling AI-training use of the fetched content. So you can't audit your Google-Extended policy from logs at all; the only place it's visible is the robots.txt file itself. When a client asks "are we blocking AI training?", the answer comes from reading their directives, not their traffic.

Spoofing: Why You Verify Instead of Trust

Because the header is self-declared, scrapers routinely send Googlebot's string to slip past bot protections and rate limits. If your firewall whitelists on UA alone, you've whitelisted anyone who can type. Verification is straightforward and Google documents it: reverse-DNS the requesting IP and confirm the hostname ends in googlebot.com or google.com, then forward-resolve that hostname and confirm it returns the original IP:

host 66.249.66.1 → crawl-66-249-66-1.googlebot.com → host crawl-66-249-66-1.googlebot.com → 66.249.66.1

Google also publishes JSON lists of its crawler IP ranges, which is the practical option for firewall rules at scale. Flip side: never serve different content to a verified crawler UA than to users — that's cloaking, and it's one of the few offenses that still draws manual penalties.

Debugging With User-Agents, Developer Edition

• Reproduce crawler-specific bugs. DevTools → Network conditions panel lets you override Chrome's UA to Googlebot Smartphone and reload; curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; ...)" does it from a terminal. If the page differs — some bot-protection layers serve challenge pages to crawler UAs, silently blocking indexing — you've found a problem worth escalating.
• Read your logs by agent. Segmenting requests by UA shows what Googlebot actually fetches versus what you think it fetches — the ground truth for diagnosing web crawling behavior on your site.
• Know your auditor's UA. Every legitimate crawler identifies itself; when you run a site audit, the crawler announces its own user-agent, so you can whitelist it in your WAF and distinguish audit traffic from real bots in the logs.
• Don't UA-sniff in application code. Serving different markup by parsed UA breaks in unexpected ways as strings evolve and freeze. Feature detection in JavaScript and responsive CSS solved this problem years ago.