The monitoring stack for my search engine ran Prometheus and Loki in Docker on one of the project's servers, with their ports published: 9090 for Prometheus and 3100 for Loki. The compose file still said "LAN only" next to both ports, from when the stack lived on a local network.
Articles on this site are drafted with an AI agent, Claude, working from my git history and design notes, and every draft goes through a verification pass before it's scheduled. On 15 October, an unattended pipeline run drafted and scheduled seven articles. I then checked them against the commits they were based on, and six of the seven contained a factual error.
In August, two background workers in my search engine had been crashing with PHP out-of-memory errors for about 20 hours before anyone noticed. Nothing looked wrong from the outside: docker ps showed the containers up, with a restart count of zero. What gave it away was that the whole fleet looked unusually loaded.
"Is one big query faster than many small ones?" is one of the most-viewed database questions there is, and the usual answer is "batch it". In July I batched the writes of my search engine's indexer, which cut the number of MySQL queries per indexed page roughly in half.
Until September, this site had no structured data at all: no JSON-LD, no microdata, not even Open Graph tags. I noticed because the search-appearance report in Search Console came back completely empty in two exports in a row. I added Open Graph tags and a block of JSON-LD to every article page, including publisher and mainEntityOfPage.
My search engine parsed every crawled page with PHP 8.4's new HTML5 parser, Dom\HTMLDocument::createFromString(). In July, someone pointed out that a Danish result showed its title as "Bes�g i hansestaden L�beck" instead of "Besøg i hansestaden Lübeck".
The script that publishes articles on this site tests every article on a local copy of the site first, at https://linkhub.ddev.site. That local site runs in DDEV, and its HTTPS certificate is signed by a local certificate authority that mkcert created. On the Ubuntu install where the pipeline started, that just worked.
CaRMen's deploys start in GitHub Actions and end on a server, and something has to cross that gap. Here it's an SSH key, stored as a secret in the repository, that the deploy job uses to connect to the server and run the deploy. A key like that is exactly the kind of secret that leaks: it sits in a CI system, it's used automatically, and nobody watches each use. So the key can't open a shell.
CaRMen's public demo lives at /demo/ on the project's landing site. The landing site is its own nginx server block on the host; the demo is a separate application in Docker, listening on a local port.
My search engine's crawler backed off from hosts that kept failing: each consecutive failure doubled the wait before the next attempt, up to a cap of one day. During one long working session in July, one test of that logic failed in every run, and it kept being written off as a "pre-existing, unrelated" failure.