A CPU isolation setting sat unchanged for weeks, explicitly documented as deliberately left alone to avoid destabilizing a live system. When it finally got measured instead of just trusted, the setting turned out to be backwards — giving unrestricted access to the exact processes it was supposed to be isolated from, while pinning the processes causing the contention to almost nothing.
What the setting was supposed to do
A database and a search index shared a machine with a crawler and an indexer, all real, CPU-hungry processes competing for the same physical cores. The intent behind pinning processes to specific CPU cores is straightforward: give critical, latency-sensitive services (the database, the search index) guaranteed access to some cores free of contention from less latency-sensitive batch work (the crawler, the indexer), so a burst of crawling activity can't starve a live query.
What the actual configuration did instead
The database and search index were pinned to all eight cores on the machine — no restriction at all, full access to everything. The crawler and indexer, meanwhile, were restricted to just two of those same eight cores. On paper this reads like isolation: the batch processes are contained to a smaller slice. In practice, it's the opposite of what isolation requires. The database and search index had no cores reserved exclusively for them — the crawler and indexer could always contend with them on any of the eight cores, including the two the batch processes were confined to. Meanwhile the crawler and indexer, confined to just two cores, could never use the other six even when those cores sat completely idle. The processes meant to be protected from contention had zero cores free of it. The processes meant to be contained were the only ones actually contained.
Why this sat unfixed for weeks anyway
This wasn't an oversight nobody had looked at. It was a known, deliberate tradeoff, documented explicitly at the time it was set: leave the working configuration alone rather than risk destabilizing a live, already-tuned production system by changing CPU allocation without being certain of the consequences. That's a reasonable position to take when the cost of getting a change wrong is a production incident and nobody has the numbers yet to be confident about what a change would actually do.
What actually changed the calculus
Getting those numbers didn't require guessing or a synthetic benchmark — real usage history was already being collected. Three days of actual production load, including a recent period covering some of the heaviest sustained activity the machine had seen, showed the database peaking at roughly 3.17 cores under smoothed measurement and up to about 4.51 cores in short bursts; the search index peaked far lower, under one core. Against an 8-core machine, giving the database and search index six dedicated cores between them left comfortable headroom over even the measured worst case, while the crawler and indexer — real, but bursty, batch-style workloads — could reasonably share the remaining two.
The fix, applied without downtime
The database and search index moved to a defined, exclusive range of six cores; the crawler and indexer kept the remaining two, this time with the database and search index no longer able to contend with them there. The change was applied live, directly against the running containers, without restarting either service — verified immediately afterward with a real query against the database and a cluster health check against the search index, both confirming nothing had been disrupted by the change itself.
Why "leave it alone until it's measured" was the right call twice over
Leaving the original configuration in place for weeks wasn't a failure to notice a problem — it was refusing to change a live, working system based on a hunch, however reasonable that hunch was. Fixing it also wasn't guesswork the second time around: the specific new split wasn't chosen because it seemed more balanced, it was chosen because three days of real load data showed real headroom under it. The setting was backwards for weeks not because nobody suspected it, but because suspecting a misconfiguration and having the actual numbers to safely correct it without risking the same live system are two different thresholds, and only one of them justifies touching something that's currently working.
Leave a comment
Comments are reviewed before they appear.