In August, two background workers in my search engine had been crashing with PHP out-of-memory errors for about 20 hours before anyone noticed. Nothing looked wrong from the outside: docker ps showed the containers up, with a restart count of zero. What gave it away was that the whole fleet looked unusually loaded.
"Is one big query faster than many small ones?" is one of the most-viewed database questions there is, and the usual answer is "batch it". In July I batched the writes of my search engine's indexer, which cut the number of MySQL queries per indexed page roughly in half.
Until September, this site had no structured data at all: no JSON-LD, no microdata, not even Open Graph tags. I noticed because the search-appearance report in Search Console came back completely empty in two exports in a row. I added Open Graph tags and a block of JSON-LD to every article page, including publisher and mainEntityOfPage.
My search engine parsed every crawled page with PHP 8.4's new HTML5 parser, Dom\HTMLDocument::createFromString(). In July, someone pointed out that a Danish result showed its title as "Bes�g i hansestaden L�beck" instead of "Besøg i hansestaden Lübeck".
The script that publishes articles on this site tests every article on a local copy of the site first, at https://linkhub.ddev.site. That local site runs in DDEV, and its HTTPS certificate is signed by a local certificate authority that mkcert created. On the Ubuntu install where the pipeline started, that just worked.
CaRMen's deploys start in GitHub Actions and end on a server, and something has to cross that gap. Here it's an SSH key, stored as a secret in the repository, that the deploy job uses to connect to the server and run the deploy. A key like that is exactly the kind of secret that leaks: it sits in a CI system, it's used automatically, and nobody watches each use. So the key can't open a shell.
CaRMen's public demo lives at /demo/ on the project's landing site. The landing site is its own nginx server block on the host; the demo is a separate application in Docker, listening on a local port.
My search engine's crawler backed off from hosts that kept failing: each consecutive failure doubled the wait before the next attempt, up to a cap of one day. During one long working session in July, one test of that logic failed in every run, and it kept being written off as a "pre-existing, unrelated" failure.
CaRMen's deploys run from GitHub Actions: once the tests pass on the main branch, a job builds and publishes the Docker images, and a deploy job connects to the server over SSH and tells it to switch to the new version. Within a day of setting that up, the deploy job started failing some of the time with SSH errors, while the images it was meant to deploy had been published just fine.
In early July the admin statistics page of my search engine started reporting zero: zero pages crawled in the last hour, zero in the last 24 hours, on a pipeline that was visibly busy. At the same time, re-crawls were being scheduled two hours early. Both had the same cause, and none of the data was wrong. The database and the application disagreed about what time it was.