One 42 MB Sitemap Froze the Whole Crawl Queue for 60 Hours

My search engine has a worker that discovers sitemaps: for each host that's due, it fetches the declared sitemap, walks any child sitemaps, and records what it found. For 60-odd hours in September, that worker made no progress at all. Every cycle it picked up the same host, crashed, and started over on the same host again. Every other host waiting for a sitemap check sat behind it.

One host, always first

The host was apps.apple.com. Its declared sitemap points to child .xml.gz files, and at least one of them decompresses to roughly 42 MB of XML.

That's legal. The sitemaps.org protocol caps an uncompressed sitemap at 50 MB, and my SitemapFetcher enforces exactly that limit. A 42 MB file passes the check. The problem was the command running it: app:sitemap-discover ran under PHP's plain 128M memory_limit, and 42 MB of XML plus parsing plus the Symfony framework's own overhead doesn't fit in 128M.

So the process died, and the queue order made sure it died on the same host every time. findNeedingSitemapCheck() sorts hosts by sitemap_checked_at with NULLS FIRST, so hosts that have never been checked come first. Apple's host had never been checked, because every attempt crashed before it could be marked as checked. It stayed at the front of the line indefinitely.

Why the error handling didn't help

The command already had a catch (\Throwable $e) around the risky part, and the code after the fetch called markSitemapChecked() whether a sitemap was found or not. On paper, a failure should have been recorded and the queue should have moved on.

But a PHP out-of-memory fatal isn't a Throwable. There's no exception to catch. The engine stops the process, and nothing after that point runs: no catch block, no finally, no bookkeeping. The code that would have marked the host as attempted was always going to run after the call that killed the process.

That's the bug in one sentence: the "I tried this" marker was written after the thing that could kill the process, so the one failure it most needed to cover was the one it could never record.

The fix, in two parts

Commit 73f1fa5, on 2026-09-18, does two things.

The first is the obvious one: ini_set('memory_limit', '256M') at the top of the command. The indexer already bumps its own limit for large payloads, so there was a precedent in the same codebase. That alone would have unstuck Apple.

The second matters more. A new migration adds a host.sitemap_check_failed boolean, and the command now marks the host as checked and failed before calling discover():

$this->hosts->markSitemapChecked($hostId, failed: true);

If the check completes, success overwrites it with failed = false. If the process dies mid-fetch, the host is already stamped. Its sitemap_checked_at has moved, it's no longer at the front of the NULLS FIRST ordering, and the next cycle moves on to someone else. The worst case changes from "the whole queue freezes" to "one host is wrongly flagged as failed".

It also makes the flag readable. A true that's still there later means "died mid-check", not "never got to it", which is a distinction the old schema couldn't express.

After the deploy, apps.apple.com processed cleanly with sitemap_check_failed = false, and the worker completed its first full batch in more than 60 hours. A test now covers the pre-mark ordering, so a refactor that moves the mark back after the fetch will fail loudly.

The part that's easy to miss

Raising the memory limit fixes this host. It doesn't fix the next one. Somewhere out there is a sitemap that will be too big, or too slow, or malformed in a way that crashes the parser at the C level, and no memory limit protects against all of them. What protects the queue is the order of operations: record the attempt first, then do the dangerous thing.

That pattern isn't specific to sitemaps or PHP. Any worker that pulls "the next item" from an ordered queue, and only records progress after succeeding, can be pinned in place by one item that kills the process. Any exception the code can catch is recoverable. A fatal error isn't. Until this incident, my queue had never met an item that turned one into the other, so the gap had never shown.