I allowed every crawler, and half of them still backed off

My robots.txt blocks nothing. Every page carries a meta robots tag permitting indexing. Google crawls the site happily and indexes almost everything within a few days of publication.

And when an AI assistant tried to read one of my articles, it was refused.

Not throttled, not rate-limited — refused, with a robots-level rejection, on a URL that no robots directive anywhere on my site excludes. The homepage fetched fine. Article pages did not. Google saw no problem at all.

The cause turned out to be a host I was not thinking about, combined with a rule from the Robots Exclusion Protocol that I had written an article about three weeks earlier. This is what happened, and more usefully, how to check whether it is happening to you.

First, the thing most people get wrong about AI crawlers

Before the diagnosis, the distinction that makes any of this matter. "AI crawlers" is not one category, and the decision to allow or block them is not one decision. There are three kinds, and they do entirely different things:

Blocking all three because you object to the first is the common and expensive error. Training and retrieval are separate systems operated under separate user-agents precisely so you can make separate decisions about them.

Google-Extended is the cleanest illustration: it is a robots.txt token with no crawler behind it. It exists solely so you can opt out of Gemini training, and it has no effect whatsoever on Googlebot, indexing, or ranking. Blocking it costs you nothing in search.

All of which was moot on my site, because I had blocked none of them and something was still saying no.

The investigation

The evidence was asymmetric in a specific way, and the asymmetry is what points at the answer:

• The homepage fetched successfully.
• Article URLs on the same host were refused at the robots layer.
• Googlebot crawled and indexed both without complaint.
• robots.txt itself contained no exclusions at all.

Any explanation has to account for all four, which rules out most of the obvious candidates. A robots.txt rule would show up in the file. A meta noindex would not produce a crawl refusal. A firewall block would not politely identify itself as a robots decision.

Start with the file itself, on every origin it might be requested from:

curl -sI https://www.linkhub.dk/robots.txt | head -1
curl -sI https://linkhub.dk/robots.txt     | head -1
curl -sI http://www.linkhub.dk/robots.txt  | head -1
curl -sI http://linkhub.dk/robots.txt      | head -1

That is four requests because robots.txt is scoped per origin — the combination of scheme, host and port. https://linkhub.dk/ and https://www.linkhub.dk/ are two different origins as far as the protocol is concerned, and a crawler that lands on one may not reuse the robots.txt it fetched from the other. It has to ask again, at that host.

On my site, the www version returned 200. The non-www version did not.

Why that is worse than it sounds

Here is the rule that turns a minor misconfiguration into a site-wide crawl block, and it is the one I had written up in the polite crawler article without ever thinking to check my own site against it.

RFC 9309 specifies what a crawler should do when robots.txt cannot be fetched, and the two failure cases are treated in opposite ways:

The asymmetry is deliberate and correct. A missing robots.txt means the site never wrote one. A broken robots.txt means the site is having a bad day — and hammering a struggling server while it is down is precisely the behaviour the protocol exists to prevent. So a well-behaved crawler fails closed.

Which produces the outcome I hit. A crawler that reaches the non-www origin — through a link, a redirect, a canonical tag, or simply because someone typed the domain without the www — asks for robots.txt, gets a failure, and correctly concludes it has no permission to crawl anything on that host. It never sees my permissive file, because that file is on a different origin.

The better-behaved the crawler, the harder it fails. This is the part worth sitting with. Nothing here is a bug in the crawler. Backing off is the specified, polite, correct response to an ambiguous signal, and it is exactly the behaviour I argued for in writing. The crawlers doing the right thing are the ones that vanished.

Why Googlebot never noticed

Google's tolerance is what let this sit undetected, and it is worth understanding because it means Search Console will not warn you.

Googlebot caches robots.txt for around 24 hours, so a transient failure is usually invisible to it. It has years of accumulated signal about which host is canonical for a site, and it consolidates www and non-www aggressively. It retries generously. And it is crawling constantly, so a working fetch of the canonical host keeps the cached rules alive.

A crawler encountering the site for the first time has none of that. No cache, no history, no accumulated view of which host is real. It asks once, gets a failure, and correctly stops.

So the site looks perfectly healthy in Search Console while being closed to everyone else — which is why "Google indexes us fine" is not evidence that your crawl configuration is correct. It is evidence that Google, specifically, has enough history to route around it.

The fix

Small, and there are two acceptable versions.

Redirect the whole non-www origin, including robots.txt. The redirect is followed, the canonical robots.txt is fetched, everything resolves:

server {
    listen 443 ssl;
    server_name linkhub.dk;
    # certificates must cover this name too, or the request
    # fails at TLS before the redirect is ever sent
    return 301 https://www.linkhub.dk$request_uri;
}

The comment matters. A redirect that cannot complete its TLS handshake is a connection failure, which lands you right back in the fail-closed case.

Or serve a real robots.txt on both origins. Slightly more duplication, no redirect chain, and it works even if the redirect breaks later.

Either way, verify all four combinations afterwards, and add them to whatever you use for uptime monitoring. A robots.txt that starts returning 500 is a silent, total crawl outage that no other check will catch — your pages are up, your monitoring is green, and every compliant crawler has quietly stopped.

Check your own

Four things, in the order I would do them:

1. The four curl commands above. Every scheme and host combination. Look for anything that is not a 200 or a clean redirect chain ending in one.

2. Your logs, for who is actually arriving.

   grep -EiO 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|CCBot|Bytespider' \
     /var/log/nginx/access.log | sort | uniq -c | sort -rn
   

   An empty result on a site that gets traffic is a finding, not a relief.

3. Your CDN or WAF. Cloudflare and others ship AI-crawler blocking as a toggle, sometimes enabled by default on newer plans. Your robots.txt has no visibility into it and neither do you unless you look.
4. Your robots.txt against a real matcher, not by reading it. I wrote a linter that uses Google's own parser for exactly this, because the matching rules are less intuitive than they look.

So, what should you actually allow?

Having fixed the accident, the deliberate decision is still there to make, and it is genuinely contested. The honest version of both cases:

For blocking training crawlers: the traditional bargain with search engines was reciprocal — Googlebot took your content and sent readers back. An answer engine takes your content and answers the question, so the visit never happens. If you object to that, blocking the training user-agents is a real lever with no cost to your search visibility.

For allowing retrieval crawlers: AI-mediated discovery is a growing channel, and blocking the retrieval bots removes you from it completely rather than partially. Recent first-party log studies find user-initiated fetches — someone asking an assistant about a specific page — making up a substantial share of AI bot traffic. That is not scraping; that is a reader.

I should declare an interest of sorts: this article exists because an AI assistant could not read my site, so I am not a neutral party on whether that is a problem. Make your own call.

What is not defensible is the position I was accidentally in — permitting everything in writing while a misconfigured origin quietly refused the polite half of the internet. If you are going to block, block deliberately. If you are going to allow, check that allowing actually works.