One ALTER DATABASE Silently Rebuilt an Entire 22-Table Schema in the Wrong Place

Fixing one database incident accidentally created a second, unrelated one — and the second one hid the evidence needed to see whether the first one was actually fixed. One misordered database setting, changed to avoid a minor inconvenience, ended with an entire application's schema rebuilt from scratch in the wrong place.

The setting, and the reason for changing it

A graph-database extension installs its own schema alongside an application's normal tables. Every database connection needs to know where to look for which objects, and the ordinary way to avoid typing a schema prefix on every graph-related query is to add that schema to the database's default search path. That's what happened here: ALTER DATABASE corpus SET search_path = ag_catalog, "$user", public — the extension's schema listed first, the application's own public schema second.

Why the order was exactly backwards

A search path isn't just a convenience list — it's the order Postgres checks when a query refers to a table by name alone, without a schema prefix. Putting the extension's schema first means every unqualified table reference from any tool, for any purpose, checks the extension's schema before ever considering the application's own tables. The application's tables should always resolve first, with the extension's schema reached deliberately — a per-session setting or an explicit prefix — only when doing graph-specific work. The database-wide default did the opposite, silently, for every single connection made after the change.

What that broke, immediately, without any error

A separate migration — the one actually meant to fix an unrelated duplicate-row problem discovered earlier — got launched through a fresh container while this backwards search path was still in effect. The migration tool's runner looks for its own bookkeeping table to figure out which migrations have already run. With the extension's schema first and no bookkeeping table there yet, the tool concluded nothing had ever been migrated on this database — and quietly replayed every migration in the project's history from a blank slate. Sixty-nine of them, run again, from the beginning.

Where sixty-nine migrations actually landed

An unqualified CREATE TABLE lands in whichever schema comes first in the search path that the connecting user is allowed to create in. With the extension's schema first, that's exactly what happened: the application's entire structure — every one of 22 tables, including project-specific generated columns — got recreated, empty, inside the extension's own schema. Same table names. Same column types. Same constraint names, down to the letter. A structurally perfect, completely empty duplicate of the real application schema, sitting one schema over from where anyone would think to look for it.

The migration that succeeded at nothing

The dedup migration this whole sequence was originally meant to run then executed against that empty duplicate. It found no duplicate rows there, because there were no rows there at all. It reported success. Every log line, every exit code, every ordinary signal of "did this work" said yes — while the real, still-broken data sat completely untouched in the actual application schema, one schema away, still waiting for the fix that had just reported having already been applied.

Why the two incidents together were harder than either alone

A separate reindex operation, running concurrently and resolving tables by their internal identifier rather than by name, kept scanning the real, still-duplicated data the entire time — surfacing the same pre-existing duplicate names on every retry. Read on its own, that looked exactly like new duplicates being created continuously, which is a much scarier signal than "one already-known problem, still present." Confirming what had actually happened required checking a plain row count on the application's real table (reading zero when it should have read in the millions), ruling out a stats artifact with a forced full-table scan, and then finding the same table structure duplicated inside the extension's schema before the actual shape of the mistake became clear.

The lesson that outlasts this specific bug

A database-wide default that changes which schema a name refers to has a blast radius much larger than the one query or one tool it was changed for. Any other tool that infers state by checking whether some unqualified object already exists — a migration runner deciding whether to run, or anything with similar logic — inherits that changed default automatically and silently, with no way to know the assumption it's relying on just moved. A session-scoped setting, changed only for the connections that actually need it, doesn't have that property. A database-wide one does, and the tools most likely to be surprised by it are exactly the ones that never expected to need an opinion about which schema comes first.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
Please share this article on your favorite website or platform.