Canopy Host’s build pipeline started failing intermittently with “currently has a deploy lock in place” — a genuine Dokku error, not a red herring. The frustrating part: fixes seemed to work, then stop working, then work again, with no code change in between.

First wrong turn. I assumed the lock-file path our force-unlock script targeted was wrong. It was — /home/dokku/<app>/.deploy.lock doesn’t exist; the real lock lives at $DOKKU_LIB_ROOT/data/apps/<app>/.deploy.lock, a genuine flock acquired by Dokku’s own acquire_app_deploy_lock(). Fixing the path helped, but the symptom kept coming back.

Second wrong turn. I found a real concurrency gap: the build worker’s claim query had no per-environment exclusivity, so two builds for the same app could genuinely race for the same lock. I fixed that too (a single UPDATE ... WHERE id IN (SELECT ... FOR UPDATE SKIP LOCKED) with NOT EXISTS guards against a same-environment build already active or an older one still pending). Still not enough — the symptom persisted, unpredictably.

The actual cause was dumber than either fix: ps aux showed a second tsx ./lib/worker.js process, parent and child, running since two days earlier — completely absent from pm2 jlist. It had never been killed when the PM2-managed instance restarted (which happens on every deploy to the API itself), and both processes had been polling and claiming from the same builds table the entire time. Which one handled a given deploy was effectively random. Half the “fixes” that day really did work — for the requests the up-to-date process happened to claim.

The fix that actually mattered: a Postgres advisory lock (pg_try_advisory_lock), held on a dedicated reserved connection for the process’s lifetime, checked at worker startup. It self-heals on a crash — Postgres releases a session-scoped advisory lock the instant the holding connection closes, verified in tests via pg_terminate_backend() rather than trusting a killed process to clean up after itself. A second instance now refuses to start instead of silently coexisting.

Two lessons that generalize past this one incident: when a fix “sometimes doesn’t take,” check for a process-count mismatch before assuming the bug is in the code you’re staring at — ps aux vs. your process manager’s own view is a five-second check that can save a day. And a lock’s existence being correct doesn’t mean the thing acquiring it is the only thing that can.