Drupal ships a sensible robots.txt. It blocks the admin paths, the login and registration forms, the search pages, and the node creation forms. For a single-language site it does the job.
I run a site in Danish and Swedish. On that site, almost none of those rules do anything.
The proof
Google open sourced the matcher that Googlebot uses, and there is a Python port of it. So this is not an interpretation of the specification. It is the actual matching code, run against Drupal's actual default file.
from gpyrobotstxt.robots_cc import RobotsMatcher
raw = open('robots.txt', 'rb').read()
def allowed(path):
return RobotsMatcher().allowed_by_robots(
raw, ['Googlebot'], 'https://example.dk' + path)
The result:
path verdict
/admin/ BLOCK
/da/admin/ ALLOW <-- crawlable
/sv/admin/ ALLOW <-- crawlable
/user/login BLOCK
/da/user/login ALLOW <-- crawlable
/sv/user/login ALLOW <-- crawlable
/search/ BLOCK
/da/search/ ALLOW <-- crawlable
/sv/search/ ALLOW <-- crawlable
/node/add/ BLOCK
/da/node/add/ ALLOW <-- crawlable
/media/oembed BLOCK
/da/media/oembed BLOCK
/sv/media/oembed BLOCK
Look at the last three lines. The oembed path is blocked in every language. Every other path is blocked only in the default one.
Why this happens
A Disallow value is a prefix match against the URL path, starting at the root. Disallow: /admin/ matches any path that begins with those seven characters.
Drupal's language negotiation puts the language code at the front of the path. So the admin page in Danish is /da/admin/, which does not begin with /admin/. It begins with /da/. The rule does not match, and the crawler is free to go.
This applies to every rule in the file. Adding a language prefix shifts the whole path, and every prefix rule written from the root stops matching.
Core knows, and fixed one path
This is the part I find interesting. The default file contains these two lines:
Disallow: /media/oembed
Disallow: /*/media/oembed
The second one uses a wildcard, so it matches /da/media/oembed, /sv/media/oembed, and any other language prefix. It works correctly on a multilingual site.
That rule was added to core in 2022, in an issue that explicitly discussed covering both single and multilingual sites. So the language prefix problem was understood, and a fix was applied.
To exactly one path. The oembed rules got the wildcard treatment. The admin paths, the user paths, the search path and the node paths did not, and still have not.
I am not sure this is wrong, to be fair. Adding Disallow: /*/admin/ would also block /anything/admin/, which on some sites is real content. A wildcard is a blunt instrument, and core cannot know your URL structure. But the result is a default file that quietly stops covering most of what it claims to cover the moment you enable a second language.
How much does this actually matter?
Less than the word "crawlable" suggests, and more than nothing. Worth being precise rather than alarming.
The admin paths are not a security problem. /da/admin/ requires authentication. A crawler gets a redirect to the login form. No content leaks. What you lose is crawl budget, spent on pages that will never produce anything useful.
The user paths are a real issue. /da/user/register and /da/user/password are publicly accessible and return real pages. They can be indexed. Core blocks them in the default language for a reason, and that reason does not stop applying in Danish.
The search path is the one I would fix first. Search result pages are public, they generate an unbounded URL space, and every query string is a new URL. This is a classic crawl trap. On a two-language site you now have two of them, and the default file catches neither.
So the summary is: mostly wasted crawl budget, some thin pages that can end up indexed, and one genuine trap. Not a disaster. Worth twenty minutes.
The fix
Write the rules out per language. It is repetitive and it is unambiguous.
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/logout
# Danish
Disallow: /da/admin/
Disallow: /da/comment/reply/
Disallow: /da/filter/tips
Disallow: /da/node/add/
Disallow: /da/search/
Disallow: /da/user/register
Disallow: /da/user/password
Disallow: /da/user/login
Disallow: /da/user/logout
# Swedish
Disallow: /sv/admin/
Disallow: /sv/comment/reply/
Disallow: /sv/filter/tips
Disallow: /sv/node/add/
Disallow: /sv/search/
Disallow: /sv/user/register
Disallow: /sv/user/password
Disallow: /sv/user/login
Disallow: /sv/user/logout
The wildcard form is shorter and I would use it only where you are confident about your URL structure:
Disallow: /*/user/login
Disallow: /*/search/
That matches one path segment followed by the rest, so it covers every current and future language without editing. It also matches anything else in that shape. If you have a content type that produces /blog/search/, the wildcard blocks it too.
My rule: wildcards for paths that are unmistakably Drupal internals, such as /*/user/login. Explicit language prefixes for anything that could collide with real content, such as /search/.
Keeping the file
Composer scaffolding overwrites robots.txt on every build, which is how careful edits vanish. Turn that off in composer.json:
"extra": {
"drupal-scaffold": {
"file-mapping": {
"[web-root]/robots.txt": false
}
}
}
The tradeoff is that you no longer receive core's updates to the file, so put a note in the file saying where it came from and when you last compared it.
On a multisite install, all sites share one robots.txt at the docroot. The RobotsTxt module serves a per-site file from configuration instead. Note that its default content lags behind core, so start from core's current file rather than the module's default.
Three other things the default cannot know
While you have the file open, the language gap is not the only limit of a static file written by someone who has never seen your site.
Your own crawl traps. Core blocks Drupal's paths. It says nothing about views with exposed filters, faceted search, or deep pagination. A URL like /products?color=red&size=l&sort=price&page=7 is where a Drupal site actually generates infinite URL space, and no default can anticipate it.
Files it ships with that you do not have. The file blocks /README.txt and /web.config. A Composer-managed install may have neither. Harmless, but it is a sign that most people have never read their own robots.txt.
Your sitemap. There is no Sitemap: line in the default. Adding one is free URL discovery and it is the easiest improvement in this whole article.
Sitemap: https://example.dk/sitemap.xml
Check your own site
Do not read the file and reason about it. Prefix matching is not intuitive, and reading is how this gap survived in the first place. Run your real robots.txt through the real matcher, with your real language prefixes.
pip install gpyrobotstxt
Then test the paths that matter to you. I wrote a linter around this that reports what Google's own parser decides, which is the only answer that counts.
If you have one language, the default file is fine and you can stop reading. If you have two, you probably have twice as many crawlable admin paths as you think.