Canopy Host started as one build-worker process talking to one hardcoded Dokku host. After fixing a real incident caused by an orphaned second process (see the previous post), the emergency fix — a hard singleton lock that refused to let a second worker start at all — directly conflicted with the actual goal: running more worker processes for throughput, and eventually more physical hosts for capacity.

The constraint that shaped everything: this box has no staging environment. Every change ships straight to production. That rules out a big-bang rewrite; each phase had to be independently safe to ship and easy to verify before the next one started.

Phase 1 — replace the lock with observability. The singleton lock’s job was really “detect an orphan,” not “prevent concurrency” — concurrency was already handled correctly by the claim query’s SKIP LOCKED logic. So it became a heartbeat registry instead: each worker process registers on start, heartbeats on its normal poll loop, and deregisters cleanly on shutdown. A stale heartbeat gets logged and flagged — never auto-killed. On a box with no safe place to rehearse an auto-remediation script, a human deciding what to do with a stale process is the right default, not a shortcut.

Phase 2 — host-assignment plumbing, shipped inert. Before any environment could be scheduled across multiple hosts, something had to track which host an environment actually deployed to. I added the table and the column, then threaded an optional host parameter through every one of the 100+ SSH call sites in the deploy orchestration layer — while making sure every existing environment still resolved to the exact same single host as before. The entire phase shipped with zero observable behavior change, verified against the existing test suite before the next phase touched it at all. Boring on purpose: the goal was a foundation nobody would need to revisit, not a demo.

Phase 3 — actually assign new environments somewhere. With the plumbing inert and proven, adding a real capacity-aware scheduler (pick the healthiest host with the fewest live environments) was a small, contained change on top of an already-safe foundation.

The pattern underneath all three phases: separate “does this change behavior” from “does this add capability,” and never ship both in the same step when you don’t have a staging environment to catch a mistake before a customer does.