Why I Didn't Use Symfony Messenger for a Shared Crawl Queue

In July my search engine needed a queue that several servers could share: crawlers on one machine putting fetched pages in, indexers on others taking them out. The project already depended on Symfony Messenger, and a Messenger transport DSN pointing at the database was even configured on one of the servers. Reusing it looked like the obvious move. The design document for the queue concluded otherwise, for two reasons that held up when checked, and built a small table of its own. Reading the reasons again for this article, one of them turned out to be stated more broadly than the facts support, and the precise version is more useful.

Reason one: the transport wasn't there

The DSN looked like half the work was done: MESSENGER_TRANSPORT_DSN=doctrine://... on one server. But the doctrine:// transport isn't part of the core symfony/messenger package. It's implemented by a separate package, symfony/doctrine-messenger, and that package wasn't installed. The setting pointed at a transport that didn't exist in the project. The design document had first described Messenger as "already half-configured"; checking showed that overstated it.

That's not a reason against Messenger by itself, since installing a package is easy. It mattered because it removed the "it's basically free" argument and forced the real question: does Messenger's model fit the work?

Reason two: how messages are claimed

The design document put the second reason as "Messenger's consumption model is one-message-at-a-time". That's too broad. Messenger has batch handlers: a handler that implements BatchHandlerInterface collects messages until a batch is full and then processes them together, and the project's own Messenger version, 7.3, includes them.

What the batch handler doesn't change is how messages are taken off the queue. In the Doctrine transport, each fetch selects one row (setMaxResults(1)) and locks it with SELECT ... FOR UPDATE, using SKIP LOCKED where the database supports it. Collecting a batch of 50 messages means 50 separate claims.

The indexer had just been rebuilt around the opposite pattern. It claimed a whole batch of work in one statement, then loaded every claimed page's database row with one bulk query, specifically to avoid one round trip per item. That change had just halved the number of queries per indexed page. A queue that claims one message per round trip would have put back the per-item round trips at the front of the pipeline.

What was built instead

A plain crawl_queue table, claimed with the same pattern the project already used for crawl leases: one atomic conditional UPDATE that sets a claimed_until timestamp and a claimed_by value on rows that are unclaimed or whose claim has expired, followed by a select of the rows that worker won. Many rows, one statement. Expired claims make crashed workers' items available again without any extra cleanup, and the claim logic was already tested in the crawler, not invented for the queue.

The same step also measured what the queue would hold. On the live crawl queue, a sample of 97 pages had a median raw HTML size of 132 KB, a 90th percentile of 536 KB, a 99th of 1.64 MB and a mean of 234 KB. Those numbers sized the compressed content column before any traffic went through it. The first migration added only the table, not wired to anything, so it carried no risk on its own.

When Messenger would have been the right call

None of this is a verdict on Messenger. For work that arrives as independent messages and is handled one at a time, or in batches where claiming each message separately is cheap, it's the better choice: retries, failure transports and workers come with it. The question that decided it here was narrower: how many round trips does it take to claim N items, and does that matter for this pipeline? For an indexer that had just been optimised around claiming in bulk, it did.

The general habit is to check the reasons, not just the conclusion. "Messenger is one message at a time" was too broad and would have misled a later reader into ruling out batch handlers. "The Doctrine transport claims one message per round trip" is precise, checkable in its source, and is what the decision rested on.