For about a day in August, the Postgres server behind my search engine kept dying. Not slowing down, not timing out: the Linux OOM killer was terminating it, roughly every 20 to 90 minutes at the worst point, and each kill meant a crash-recovery cycle of about 90 seconds while Postgres replayed its write-ahead log. The cause turned out to be a setting I had made bigger on purpose, because bigger had sounded better.
The box and the setting
The database server had 23 GB of RAM and ran three things: PostgreSQL, OpenSearch and Redis, each in its own Docker container with a memory cap. Postgres was started with these settings in the compose file:
shared_buffers=16GBeffective_cache_size=20GBmem_limit: 21Gon the container
Sixteen gigabytes of shared_buffers is 70% of that machine's memory. The PostgreSQL documentation suggests 25% as a reasonable starting value on a dedicated database server, and says that because Postgres also relies on the operating system's cache, more than 40% is unlikely to work better than a smaller amount. This server wasn't even dedicated: OpenSearch and Redis lived on it too. The only reason the number hadn't caused trouble yet was that the corpus had been small, so Postgres hadn't used much of what it was allowed to.
What changed overnight
Then the crawl scaled up. The corpus grew about sevenfold over one night, and the memory Postgres was really using climbed from about 3 GB to about 8 GB in roughly 24 hours, heading for the 16 GB it had been promised. At the same time, about 18 crawler and indexer processes on two other machines held connections to it, and every connection's query work needs memory of its own on top of the shared pool. The box had no swap at all, so there was no slow path: when memory ran out, the kernel's OOM killer picked a process to kill, and it kept picking Postgres.
I confirmed this from dmesg rather than guessing. The kernel log showed Postgres killed six or more times and OpenSearch's JVM once. The search-sync job, which copies new documents into the search index, stalled for a full day as a side effect.
The suspect that wasn't
OpenSearch looked like the obvious culprit, because its container sat close to its memory cap. So I checked it before touching anything. Its JVM heap was 580 MB used out of 4096 MB, about 14%. The rest of what made its container look full was page cache from its index files: memory the kernel hands out freely and takes back under pressure. It looked full and wasn't under pressure at all.
The first mitigation I tried was lighter: stopping the database server's own crawler and indexer, to take some load off. It helped, but not enough. The pressure came from Postgres's own footprint, not mainly from the crawling on that box.
The fix: give the memory back
The real fix was to stop promising Postgres memory the machine couldn't spare:
shared_buffers: 16 GB to 8 GB, about 35% of RAM- the Postgres container cap: 21 GB to 12 GB
effective_cache_size: 20 GB to 16 GB- an 8 GB swap file on the host, persisted in
/etc/fstab
With that, the caps for Postgres, OpenSearch and Redis added up to about 19.5 GB against 23 GB of physical memory. That was the first time the numbers on paper fit inside the real machine. I recreated only the Postgres container, checked that the other machines could still connect, and looked at free memory: about 13 GB available, against the 1 to 4 GB it had been sitting at constantly before.
Two of those four changes are less important than they look. Lowering effective_cache_size didn't free anything: the documentation is explicit that it has no effect on the memory Postgres allocates and doesn't reserve any kernel cache; it's only a hint the query planner uses to estimate whether an index will be cached. Of the four changes, it's the one that did nothing for the OOM kills. And the swap file isn't there for performance. It's there so the next spike degrades into slowness instead of an instant kill.
What I took from it
A large shared_buffers value is a promise, and it only gets tested as the data grows into it. Sixteen gigabytes was harmless for months because nothing used it. The day the working set grew, the promise came due on a machine that had also promised memory to two other services.
The order of the investigation mattered as much as the fix. The container that looked full was fine, and the one that looked reasonable was the problem. Checking the heap and dmesg before changing anything saved me from shrinking OpenSearch and leaving the actual cause in place.
This was one of several resource fights on that fleet; Part 2 of the search engine story covers how one server became four in the first place.
Leave a comment
Comments are reviewed before they appear.