Here is a robots.txt that appears in some form on a very large number of sites:
User-agent: *
Disallow: /admin/
Disallow: /private/
User-agent: Googlebot
Disallow: /nogoogle/
Read it the way anyone would: everyone is kept out of /admin/ and /private/, and Googlebot is additionally kept out of /nogoogle/.
That is not what it says. Googlebot is free to crawl /admin/ and /private/, and Bingbot is not.
You do not have to take my word for it, which is the point of this article. Google open-sourced the actual production matcher — the same parsing and matching code Googlebot runs — and there is a faithful Python port. So every claim below is a program you can run, not an interpretation of a specification.
Setting up the proof
The library is google/robotstxt, released under Apache 2.0. The Python port preserves the original behaviour and the full original test suite:
pip install gpyrobotstxt
The entire interface you need is one method:
from gpyrobotstxt.robots_cc import RobotsMatcher
def allowed(robots: str, agent: str, path: str) -> bool:
return RobotsMatcher().allowed_by_robots(
robots.encode(), [agent], "https://example.com" + path)
One API detail that will waste your afternoon otherwise: pass the product token, not a full User-Agent header string. "Googlebot" matches; "Googlebot/2.1" matches nothing at all. If you are feeding it raw request headers, run them through matcher.extract_user_agent() first.
Proof 1: groups are selected, not merged
Back to the file from the top. Run it:
robots = """
User-agent: *
Disallow: /admin/
Disallow: /private/
User-agent: Googlebot
Disallow: /nogoogle/
"""
for agent, path in [("Googlebot", "/admin/"),
("Googlebot", "/private/"),
("Googlebot", "/nogoogle/"),
("Bingbot", "/admin/")]:
verdict = "ALLOW" if allowed(robots, agent, path) else "BLOCK"
print(f"{agent:<12} {path:<14} {verdict}")
Output:
Googlebot /admin/ ALLOW
Googlebot /private/ ALLOW
Googlebot /nogoogle/ BLOCK
Bingbot /admin/ BLOCK
The mechanism is in RFC 9309 and it is unambiguous once you know to look for it: a crawler selects the single most specific group whose User-agent matches, and obeys only that group. Groups are not merged. There is no inheritance from the wildcard group.
So the moment you write User-agent: Googlebot anywhere in the file, Googlebot stops reading your User-agent: * rules entirely. Every restriction you wrote for "everyone" now applies to everyone except the crawler you most wanted to control.
This is not an obscure edge case. The pattern arises naturally: someone adds a Google-specific rule years after the wildcard block was written, the file still looks correct, and nothing visibly breaks. The wildcard rules are still there. They are simply no longer addressed to Google.
The fix is to repeat the shared rules inside the specific group:
User-agent: *
Disallow: /admin/
Disallow: /private/
User-agent: Googlebot
Disallow: /admin/
Disallow: /private/
Disallow: /nogoogle/
Verbose and redundant, and correct. robots.txt has no include mechanism, so duplication is the only way to express "these rules and also that one."
Proof 2: matching is by prefix, not by path segment
robots = """
User-agent: *
Disallow: /admin
"""
Googlebot /admin/ BLOCK
Googlebot /administrator-guide BLOCK <-- surprising
Googlebot /admin-login-help BLOCK <-- surprising
A Disallow value is a string prefix, not a directory. /admin matches every URL path that begins with those six characters, which includes a good deal of content nobody meant to hide.
One trailing slash fixes it. Disallow: /admin/ matches only the directory.
Proof 3: paths are case-sensitive
robots = """
User-agent: *
Disallow: /Admin/
"""
Googlebot /Admin/ BLOCK
Googlebot /admin/ ALLOW
The host part of a URL is case-insensitive. The path is not, and neither is robots.txt matching against it. A rule written in the wrong case protects nothing, silently.
Proof 4: longest match wins, and Allow breaks ties
The one piece of good news here — this behaviour is genuinely useful once you trust it:
robots = """
User-agent: *
Disallow: /files/
Allow: /files/public/
Disallow: /files/public/drafts/
"""
Googlebot /files/secret.pdf BLOCK
Googlebot /files/public/report.pdf ALLOW
Googlebot /files/public/drafts/wip.pdf BLOCK
Specificity is measured in characters matched, not in rule order. The most specific rule wins regardless of where it sits in the file, and where two rules match equally, Allow wins. That means you can carve exceptions out of exceptions, and you never have to reason about ordering.
Proof 5: unsupported directives fail silently
robots = """
User-agent: *
Noindex: /old-campaign/
Disallow: /private/
"""
Googlebot /old-campaign/ ALLOW
Googlebot /private/ BLOCK
Google stopped honouring Noindex: in robots.txt in September 2019. The line still parses without complaint, produces no warning anywhere, and does absolutely nothing. Every one of these still in the wild is a page someone believes is hidden.
Which leads to the thing that causes the most confusion in this whole area.
The blocking paradox
Disallow prevents crawling. It does not prevent indexing.
If other pages link to a URL you have blocked, Google can still index it from those links alone — which produces the listing with a URL, no title, and a note that no information is available. The page is in the index. It was never fetched.
Now the part that catches people. Suppose you want that page gone, so you add a noindex meta tag. Google must fetch the page to read that tag. Your robots.txt forbids fetching the page. The block prevents the removal. The page stays indexed indefinitely, and the harder you block it the more permanent that becomes.
The two tools do different jobs:
| Goal | Use |
|---|---|
| Save crawl budget on worthless URLs | Disallow |
| Keep a page out of search results | noindex, and allow crawling |
| Both | noindex first; add Disallow only after it has dropped out |
| Actual secrecy | Authentication. robots.txt is a public file listing your private paths |
That last row deserves saying plainly: robots.txt is world-readable and is the first thing an attacker fetches. Listing /admin/backup-2019/ in it is an advertisement, not a defence.
A linter you can run
Everything above is mechanical, which means it can be checked automatically. Here is a linter that runs your robots.txt through Google's own matcher and reports what it actually does.
The approach: rather than parsing the file and reasoning about the rules, it probes the matcher with paths a site would obviously not want blocked, and reports any that come back BLOCK.
#!/usr/bin/env python3
"""
robotslint - check a robots.txt against Google's own production matcher.
pip install gpyrobotstxt
./robotslint.py robots.txt [--agent Googlebot] [--urls urls.txt]
"""
import argparse
import re
import sys
from dataclasses import dataclass
from gpyrobotstxt.robots_cc import RobotsMatcher
HOST = "https://example.com"
# Paths a site almost never intends to block. Used to detect overbroad rules.
COLLATERAL_PROBES = [
("/administrator-guide", "documentation page"),
("/admissions", "content page"),
("/apidocs", "documentation"),
("/search-engine-optimisation-guide", "content page"),
("/services", "content page"),
("/user-guide", "documentation"),
("/products", "content page"),
]
ASSET_PROBES = [
("/themes/custom/app/css/style.css", "CSS"),
("/themes/custom/app/js/app.js", "JavaScript"),
("/core/misc/drupal.js", "JavaScript"),
("/sites/default/files/image.jpg", "image"),
]
@dataclass
class Finding:
level: str # ERROR | WARN | INFO
code: str
line: int | None
message: str
def __str__(self):
loc = f"line {self.line}" if self.line else "file"
return f" [{self.level:<5}] {self.code:<22} {loc:<9} {self.message}"
class RobotsLinter:
def __init__(self, raw: bytes, agent: str = "Googlebot"):
self.raw = raw
self.agent = agent
self.lines = raw.decode("utf-8", errors="replace").splitlines()
self.text = "\n".join(self.lines)
self.matcher = RobotsMatcher()
self.findings: list[Finding] = []
def allowed(self, path: str) -> bool:
return self.matcher.allowed_by_robots(self.raw, [self.agent], HOST + path)
# ---------- static checks ----------
def check_unsupported_directives(self):
for i, line in enumerate(self.lines, 1):
key = line.split(":")[0].strip().lower()
if key in ("noindex", "nofollow"):
self.findings.append(Finding(
"ERROR", "unsupported-directive", i,
f"'{key}:' is not a robots.txt directive. Ignored since "
f"Sept 2019. Use a meta robots tag or X-Robots-Tag header."))
elif key in ("crawl-delay", "host", "request-rate"):
self.findings.append(Finding(
"INFO", "nonstandard-directive", i,
f"'{key}:' is not in RFC 9309. Google ignores it; some "
f"other crawlers honour it."))
def check_rules_before_useragent(self):
seen_agent = False
for i, line in enumerate(self.lines, 1):
s = line.strip()
if not s or s.startswith("#"):
continue
key = s.split(":")[0].strip().lower()
if key == "user-agent":
seen_agent = True
elif key in ("allow", "disallow") and not seen_agent:
self.findings.append(Finding(
"ERROR", "orphan-rule", i,
"Allow/Disallow appears before any User-agent line, so it "
"belongs to no group and is silently ignored."))
def check_duplicate_groups(self):
agents: dict[str, list[int]] = {}
for i, line in enumerate(self.lines, 1):
s = line.strip()
if s.lower().startswith("user-agent:"):
name = s.split(":", 1)[1].strip().lower()
agents.setdefault(name, []).append(i)
for name, lns in agents.items():
if len(lns) > 1 and not self._contiguous(lns):
self.findings.append(Finding(
"WARN", "split-group", lns[1],
f"'{name}' declared in separate groups (lines {lns}). Rules "
f"across non-adjacent groups are NOT merged."))
def _contiguous(self, lns):
for a, b in zip(lns, lns[1:]):
between = [l.strip() for l in self.lines[a:b - 1]]
if any(l and not l.startswith("#") for l in between):
return False
return True
def check_missing_sitemap(self):
if not re.search(r"^\s*sitemap\s*:", self.text, re.I | re.M):
self.findings.append(Finding(
"INFO", "no-sitemap", None,
"No Sitemap: directive. Free, host-independent URL discovery."))
# ---------- behavioural checks, via the real matcher ----------
def check_blocks_everything(self):
if not self.allowed("/"):
self.findings.append(Finding(
"ERROR", "blocks-everything", None,
f"The site root is blocked for {self.agent}."))
def check_overbroad_prefixes(self):
for path, why in COLLATERAL_PROBES:
if not self.allowed(path):
rule = self._which_rule_blocks(path)
self.findings.append(Finding(
"ERROR", "overbroad-prefix", None,
f"{path} ({why}) is blocked"
+ (f" by '{rule}'" if rule else "")
+ ". Matching is on prefix, not path segment - "
"add a trailing slash to scope the rule."))
def check_blocked_assets(self):
blocked = [(p, k) for p, k in ASSET_PROBES if not self.allowed(p)]
if blocked:
kinds = sorted({k for _, k in blocked})
self.findings.append(Finding(
"WARN", "blocked-render-assets", None,
f"Blocked {', '.join(kinds)} - Google renders pages before "
f"indexing them. e.g. {blocked[0][0]}"))
def _which_rule_blocks(self, path: str) -> str | None:
best = None
for line in self.lines:
s = line.strip()
if s.lower().startswith("disallow:"):
val = s.split(":", 1)[1].strip()
if val and path.startswith(val.rstrip("*")):
if best is None or len(val) > len(best):
best = val
return f"Disallow: {best}" if best else None
def run(self):
for check in (self.check_unsupported_directives,
self.check_rules_before_useragent,
self.check_duplicate_groups,
self.check_blocks_everything,
self.check_overbroad_prefixes,
self.check_blocked_assets,
self.check_missing_sitemap):
check()
order = {"ERROR": 0, "WARN": 1, "INFO": 2}
self.findings.sort(key=lambda f: (order[f.level], f.line or 0))
return self.findings
def main():
ap = argparse.ArgumentParser()
ap.add_argument("robots")
ap.add_argument("--agent", default="Googlebot")
ap.add_argument("--urls", help="file of URL paths, one per line")
args = ap.parse_args()
raw = open(args.robots, "rb").read()
linter = RobotsLinter(raw, args.agent)
findings = linter.run()
print(f"\n{args.robots} (as {args.agent})")
print("-" * 72)
for f in findings or []:
print(f)
if not findings:
print(" no findings")
if args.urls:
print("\n crawl decisions\n " + "-" * 70)
for line in open(args.urls):
path = line.strip()
if path and not path.startswith("#"):
ok = linter.allowed(path if path.startswith("/") else "/" + path)
print(f" {'ALLOW' if ok else 'BLOCK':<6} {path}")
print()
return 1 if any(f.level == "ERROR" for f in findings) else 0
if __name__ == "__main__":
sys.exit(main())
Against a file with the trailing-slash problem:
$ ./robotslint.py robots.txt
robots.txt (as Googlebot)
------------------------------------------------------------------------
[ERROR] overbroad-prefix file /administrator-guide (documentation page)
is blocked by 'Disallow: /admin'. Matching is on prefix, not path
segment - add a trailing slash to scope the rule.
[ERROR] overbroad-prefix file /apidocs (documentation) is blocked by
'Disallow: /api'. ...
[INFO ] no-sitemap file No Sitemap: directive.
It exits non-zero when there is an ERROR-level finding, so it drops straight into CI. A robots.txt is a deployed artefact like any other, and there is no reason it should be the one file nobody tests.
Two caveats about the library
It matches, it does not crawl. The library implements parsing and matching only. Behaviour a crawler layers on top is not included — Googlebot-Image falling back to the Googlebot group when no Googlebot-Image group exists is a Googlebot behaviour, not a matcher behaviour, so do not expect the library to model it.
URLs must be RFC 3986 form. The library does not fully normalise them. Feed it the URL as it would actually be requested.
What to go and check
Three things, in descending order of how likely they are to be wrong on a site you maintain:
- Does your robots.txt contain a bot-specific group? If yes, that bot is ignoring every wildcard rule in the file. This is the one worth checking first.
- Does every
Disallowdirectory rule end in a slash? If not, run the linter and see what else it is catching. - Is anything you want de-indexed blocked in robots.txt? If so, the block is what is keeping it indexed. Allow crawling, let the
noindexbe read, and re-block later if you still want to.
The wider lesson from having written a crawler and then pointed the real matcher at these files: robots.txt looks like configuration but behaves like code, with prefix matching, group selection and specificity rules that are precise and almost entirely counter-intuitive. It has just never had the tooling that code gets. It costs about a hundred lines to fix that.