In August, two background workers in my search engine had been crashing with PHP out-of-memory errors for about 20 hours before anyone noticed. Nothing looked wrong from the outside: docker ps showed the containers up, with a restart count of zero. What gave it away was that the whole fleet looked unusually loaded. The two causes were different, but both came down to the same mistake: holding a whole dataset in memory when it only ever needed to be processed a piece at a time.
Why Docker never saw it
Each worker ran inside a small shell loop, worker.sh, which runs the PHP command and, if it fails, runs it again. A PHP fatal error ended the PHP process, the loop caught the failure and started the next cycle, and the container's own main process never exited. Docker's restart count only counts container restarts, so it stayed at zero for the whole 20 hours. A retry loop inside a container hides crashes from the one number people usually check.
The eviction worker: a criterion that woke up
The first worker deletes low-value pages from the index. One of its criteria had a freshness check: only evict pages indexed before the last completed PageRank pass, written as a comparison against a value read from a one-row state table.
That table had been empty for a month, since a database migration. In SQL, comparing anything with NULL doesn't give false; PostgreSQL's documentation says ordinary comparison operators yield null, meaning "unknown", when either side is null. A WHERE clause keeps only the rows for which the condition is true, so this criterion had matched zero rows, silently, every run for a month.
Fixing the state table, for an unrelated reason, switched the criterion back on, against the full corpus of 117 million rows at once. Together with the other criteria, one run's candidate list came to more than 500,000 IDs. And the command fetched them like this:
$candidateIds = $conn->fetchFirstColumn('SELECT id FROM evict_ids ORDER BY id');$chunks = array_chunk($candidateIds, self::CHUNK);
The whole list in one PHP array, chunked afterwards. With per-row logging across the hundred or so resulting chunks, that was enough to exceed the worker's 160 MB memory limit, a limit that had been raised from 128 MB only the day before for a different reason.
The fix: keyset pagination
The fix reads the candidates one page at a time and remembers where the last page ended:
SELECT id FROM evict_ids WHERE id > ? ORDER BY id LIMIT ?
Each page starts after the last ID of the previous one. Peak memory is now one chunk's worth of IDs, whatever the total. This is keyset pagination, and it's the right tool here rather than LIMIT ... OFFSET ..., because PostgreSQL's documentation notes that rows skipped by an OFFSET still have to be computed inside the server: every later page would redo the work of all the pages before it. With a WHERE id > ? condition on an ordered key, each page starts where the previous one stopped.
The regression test seeds one chunk plus 137 candidates, so it crosses a page boundary instead of passing with a trivially small list.
This was the third version of that loop. An earlier one had swept the whole ID range in fixed-width chunks, and because the ID sequence only ever grows, it had to probe around 150,000 mostly empty ranges per run. That was replaced by fetching the real candidate list, which fixed the query count and set up the memory problem. Keyset pagination keeps both properties: only real candidates, and bounded memory.
The sitemap worker: a cache nobody cleared
The second worker discovers URLs from sitemaps, host by host, through Doctrine. Nothing in its loop ever cleared Doctrine's EntityManager or the repository's own caches, so every entity it touched stayed in memory for the whole run. A batch that happened to include a run of large sites, each with the maximum of 5,000 URLs, pushed up to 250,000 entities through it, past its 256 MB limit.
Doctrine's documentation on batch processing starts with exactly this: the main problem with bulk operations is usually not to run out of memory, and its examples call $em->clear() after each batch to detach all objects. The fix calls clear() and the repository's cache reset after every host, the same pairing the indexer already used between batches. The regression test checks that Doctrine's unit of work is empty after a two-host run.
One deployment lesson on the way
The sitemap fix gave the command a new constructor dependency. Copying just the changed file to the server, which had worked for earlier single-file fixes, wasn't enough: Symfony's compiled production container didn't know about the new argument until the cache was cleared, and the first deploy crashed on exactly that.
What generalises
Batch jobs should never assume the size of their input, even when today's input is small. Both workers were fine at their usual volumes and fell over at an unusual one. Stream by key, clear what you've finished with, and don't let a retry loop be the only witness when a worker falls over.
Leave a comment
Comments are reviewed before they appear.