I'm blogging about Emacs, Drupal, php and exciting subjects

· Lars Nielsen

cAdvisor Saw No Containers: Docker's containerd Snapshotter and a Missing mount-id File

The per-container panels in my search engine's Grafana had shown almost nothing. Memory per container, CPU per container: mostly empty, apart from a single series for the whole machine. The comments in the monitoring config blamed a known limitation, that filesystem metrics didn't work with Docker's newer storage backend. That was wrong in an important way.

Read article cAdvisor Saw No Containers: Docker's containerd Snapshotter and a Missing mount-id File
· Lars Nielsen

Caddy Wouldn't Start: Docker Compose Passed an Empty Variable and the Caddyfile Default Never Applied

My search engine's public server put Caddy in front of everything to get Let's Encrypt certificates automatically. While setting it up, I used Let's Encrypt's staging service, to avoid burning the production rate limit on failed attempts, and switched to production by removing the setting from .env. Caddy then refused to start.

Read article Caddy Wouldn't Start: Docker Compose Passed an Empty Variable and the Caddyfile Default Never Applied
· Lars Nielsen

A Backup Script Where a Failed Dump Can't Look Like a Backup: pipefail, Temp Files and Per-Label Retention

CaRMen, my CRM, has a backup script that's 22 lines long. It runs every night from cron and before every deploy, and the deploy script restores from it automatically when a release changes the schema and then fails its health check. A script that a rollback depends on has to be right in the boring details. Three of them are worth spelling out.

Read article A Backup Script Where a Failed Dump Can't Look Like a Backup: pipefail, Temp Files and Per-Label Retention
· Lars Nielsen

Three Empty Grafana Panels, Three Different Causes: MySQL Auth Plugins, a Missing URL Scheme and an Exporter That Doesn't Exist

My search engine's monitoring had a Grafana dashboard with a panel per database: MySQL, PostgreSQL and OpenSearch. At different times over the summer, each of those panels was empty. An empty panel doesn't say why, and in all three cases the answer was different, and usually more than one thing at once.

Read article Three Empty Grafana Panels, Three Different Causes: MySQL Auth Plugins, a Missing URL Scheme and an Exporter That Doesn't Exist
· Lars Nielsen

A Shell wait Loop That Spun for Nine Hours: Exit Status Above 128 Doesn't Mean Interrupted

One morning in August, a worker container in my search engine had been using about 86% of a CPU core for nine hours and producing no output at all. docker top showed only one busy process: the shell script that runs the worker, not the PHP program it starts. The script was stuck in a loop around wait.

Read article A Shell wait Loop That Spun for Nine Hours: Exit Status Above 128 Doesn't Mean Interrupted
· Lars Nielsen

Linux From Scratch to a KDE Plasma Desktop: The Whole Build, in Order

Between August and September I built a Linux From Scratch system on my ThinkPad, scripted end to end, and took it from an empty partition to a KDE Plasma desktop that I now do my work on. Along the way I wrote up the problems one at a time, as separate articles. This is the map: the build in the order it happened, a sentence or two on each problem, and a link to the full story.

Read article Linux From Scratch to a KDE Plasma Desktop: The Whole Build, in Order
· Lars Nielsen

Postgres Table Bloat From Updating Every Row: fillfactor, HOT Updates and the Index That Rules Them Out

In September, my search engine had a job that kept getting slower. Once per PageRank cycle, it wrote the new authority score into every row of the url table, about 150 million rows. Right after a database upgrade, that step took 64.5 minutes. Two weeks later, one run was still going after 13.5 hours.

Read article Postgres Table Bloat From Updating Every Row: fillfactor, HOT Updates and the Index That Rules Them Out
· Lars Nielsen

Slow Queries After a PostgreSQL Upgrade: pg_upgrade and the Planner Statistics It Leaves Behind

"Queries got slow after upgrading PostgreSQL" is a common complaint, and one of the usual causes is a step that's easy to skip because the upgrade itself already reported success. When I upgraded my search engine's database from PostgreSQL 15 to 16, the upgrade notes record that step done before any traffic was allowed back: rebuild the planner's statistics.

Read article Slow Queries After a PostgreSQL Upgrade: pg_upgrade and the Planner Statistics It Leaves Behind
Subscribe to Linkhub