Writing a polite web crawler

Every other piece of a search engine operates on data you already have. The crawler is the one component that reaches out and spends somebody else's money — their bandwidth, their CPU, their database connections — without asking first.

That asymmetry is the whole design constraint. A badly written crawler is indistinguishable from a denial of service attack, and the people running the servers you are hitting cannot tell the difference either. They will just block you. The good news is that politeness is not a tax on throughput: almost every rule below either costs you nothing or actively makes the crawler faster.

Say who you are

Start with the cheapest possible courtesy. Send a User-Agent string that identifies the crawler and gives a human a way to reach you:

User-Agent: LinkhubBot/1.0 (+https://www.linkhub.dk/bot)

Put a real page at that URL explaining what the bot does, how often it crawls, and how to get excluded. This costs an hour and it is the difference between a sysadmin emailing you and a sysadmin adding your IP range to a firewall rule.

Do not impersonate a browser. Sending Chrome's User-Agent to sneak past bot filters is a decision you make once and then live with when someone works it out. It also means anyone debugging their access logs has no idea what hit them.

robots.txt, properly

The Robots Exclusion Protocol was folklore for twenty-five years before being standardised as RFC 9309 in 2022. Most crawler bugs live in the gap between what people assume it says and what it actually says.

The mechanics: fetch /robots.txt from the host root before touching anything else on that host. Find the group whose User-agent line matches your bot name most specifically, falling back to the * group. Only that one group applies — you do not merge rules across groups, which is the single most common parsing mistake.

Within a group, the longest matching path wins, and Allow beats Disallow on ties. So given:

User-agent: *
Disallow: /admin/
Allow: /admin/public/

the path /admin/public/faq is permitted, because the Allow rule matches more characters. Both directives support * as a wildcard and $ to anchor the end of a path.

Failure handling is specified and worth getting right, because the intuitive behaviour is backwards:

A missing robots.txt means the site never wrote one. A broken robots.txt means the site is having a bad day, and hammering it while it is down is exactly the wrong response. Keep a cached copy and fall back to it during an outage rather than assuming permission you no longer have.

Cache robots.txt for around 24 hours. Refetching it before every request is its own kind of rudeness, and never refetching means you ignore an exclusion someone added specifically to stop you.

Two directives are outside the RFC but still worth honouring. Crawl-delay tells you the minimum seconds between requests; Google ignores it, but it is an explicit statement of what a server can handle, so take it. And Sitemap points at a sitemap index — free, structured, pre-authorised URL discovery that saves you crawling the site to find out what is on it.

Rate limiting is per host, not global

This is the idea that makes a crawler both fast and well-behaved, and it is worth stating plainly: politeness is a per-host constraint, throughput is an across-host property.

One request per second to a single server is considerate. One request per second to each of four hundred servers is 400 requests per second of aggregate throughput, and every individual server sees a well-mannered visitor. The architecture that follows is a queue per host, with a scheduler that picks whichever host is next eligible.

Fixed delays are a blunt instrument, though. A one-second gap is generous for a CDN-backed site and brutal for a Raspberry Pi in someone's cupboard. Scale the delay to the observed response time instead — if a server takes 800ms to answer, it is telling you something about its capacity:

class HostSchedule
{
    private array $nextAllowed = [];  // host => unix timestamp
    private array $delay       = [];  // host => seconds, from Crawl-delay

    public function __construct(private float $defaultDelay = 1.0) {}

    public function readyAt(string $host): float
    {
        return $this->nextAllowed[$host] ?? 0.0;
    }

    public function recordFetch(string $host, float $responseTime): void
    {
        // A server that takes two seconds to answer should not be
        // asked again in fifty milliseconds.
        $base     = $this->delay[$host] ?? $this->defaultDelay;
        $adaptive = max($base, $responseTime * 10);

        $this->nextAllowed[$host] = microtime(true) + $adaptive;
    }

    public function penalise(string $host, float $seconds): void
    {
        $this->nextAllowed[$host] = microtime(true) + $seconds;
    }
}

The multiplier of ten is arbitrary but defensible: it caps your share of the server's request-handling capacity at roughly ten percent. Also hold yourself to one open connection per host. Six parallel connections is what a browser does for a single page load by a single user, not what a background process does indefinitely.

Listen to what the server tells you

HTTP has a full vocabulary for "slow down" and "go away". A crawler that ignores it is not just impolite, it is wasting its own cycles on requests that will never succeed.

The 429 case deserves emphasis. It is the server explicitly saying you are going too fast, and it usually arrives with a Retry-After header giving you the exact number to wait. Ignoring that is how a temporary throttle becomes a permanent ban.

While you are being efficient, be efficient in the direction that helps both sides. Store the ETag and Last-Modified values from every fetch and send them back as If-None-Match and If-Modified-Since on the next visit. An unchanged page then costs a 304 with an empty body instead of a full transfer. Send Accept-Encoding: gzip too. On a recrawl-heavy workload these two things together can cut bandwidth by an order of magnitude.

Deduplication starts with the URL

The web will happily serve you the same page under a dozen addresses. Normalise before you decide whether a URL is new:

• Lowercase the scheme and host, leave the path alone (paths are case-sensitive)
• Strip the default port — :80 for http, :443 for https
• Resolve . and .. segments
• Drop the fragment entirely; it never reaches the server
• Sort query parameters, and strip tracking junk: utm_*, fbclid, gclid, session IDs
• Decide a policy on trailing slashes and apply it consistently

Then honour <link rel="canonical"> when you parse the page. A site telling you which URL it considers authoritative is doing your deduplication for you.

URL normalisation still will not catch a page served under genuinely different addresses, so hash the extracted text — after stripping navigation and boilerplate — and check the fingerprint before indexing. For near-duplicates, where only a date or a byline differs, you need something fuzzier like SimHash over shingles. That is worth doing before content reaches your tokenizer, because duplicate documents quietly corrupt the collection statistics that BM25 depends on: a page duplicated fifty times drags down the IDF of every term it contains.

Traps

Some of the infinite URL spaces on the web are deliberate. Most are accidental, and they will consume your crawler just as effectively.

The classic is a calendar widget with a "next month" link, which generates valid, distinct, entirely worthless URLs until the heat death of the universe. Faceted navigation is the modern version: five filters with six options each is 46,000 URL combinations for maybe two hundred actual products. Session IDs baked into paths generate a fresh URL space per visit. And a misconfigured symlink can produce /a/b/a/b/a/b/… indefinitely.

Defences, roughly in order of how much they buy you:

• A budget per host. Cap total pages fetched per domain per crawl cycle. This one control bounds the damage from every trap simultaneously, including the ones you have not thought of.
• Maximum depth. Anything more than ten or so links from the homepage is rarely worth having.
• Repeated path segments. If the same segment appears three times in one path, it is a loop.
• URL length limits. Traps grow URLs monotonically. A 2,000-character URL is a symptom.
• Parameter counting. More than about five query parameters usually means faceted navigation.
• Yield monitoring. Track new-content-per-fetch by host. A host producing thousands of pages of near-identical content has trapped you, whatever the URLs look like.

Set a response size cap while you are at it — a few megabytes — so one pathological file cannot exhaust memory.

Assume you will be interrupted

A crawl of any real size will outlive the process running it. Keep the frontier — the queue of URLs to visit — in durable storage rather than memory, and checkpoint often enough that a crash costs minutes rather than days. Persisting per-host state matters just as much: restarting with an empty rate limiter means every host in your queue gets slammed simultaneously the moment you come back up.

The same goes for robots.txt caches. Losing them on restart means refetching thousands of them at once, which is a small stampede of your own making.

The underlying principle

Everything above reduces to one idea: your crawler should be invisible in someone's access logs. Not absent — visible, identified, and easy to contact — but never the reason anyone looks at their logs in the first place.

That is also, conveniently, the configuration that gets you the most data. Crawlers that get blocked stop collecting. Crawlers nobody minds keep running for years, and the index you build from them is the thing everything else in this series operates on.