Giovanni Beijl
Back to blog
•
18 min read
Forest, and the problem with pretending one machine is not enough
A technical account of Forest, the runtime behind Canopy Host: its workload model, Talos and Kubernetes layer, capacity mistakes, migration failures, and what production changed.

There is a particular kind of infrastructure lie that shows up very early.

You have one machine. It is doing the job. Then you draw the future architecture, and the machine has already become a cluster, three control planes, two workers, a load balancer, a service mesh, a database that needs a diagram, and probably a page about resilience written before anything has failed.

I understand why this happens. A server can feel temporary. Kubernetes diagrams make a thing look serious. Nobody wants to explain later that the production system was, in fact, one machine.

But a one-machine deployment is not automatically a temporary or embarrassing version of a real system. Sometimes it is the correct system. It has less surface area, fewer failure modes, less money tied up in idle capacity, and a much better chance of being understood by the person who has to repair it at 02:00.

Forest is the runtime and infrastructure layer behind Canopy Host. It exists because the previous deployment model tied an environment too closely to a specific Dokku host and its containers. That model was useful until it was not. An application could be running, but its build history, its intended image, the host that happened to own it, and the question of how to move it somewhere else were all too entangled.

The obvious replacement was not “let’s use Kubernetes.” Kubernetes is a very good part of the answer, but it is not the application model I wanted Canopy Host users to operate. Forest is the layer between Host’s deployment intent and Kubernetes’ actual work. It compiles a Canopy workload, delivers it to a Grove, reconciles it into Kubernetes resources, and reports back what happened.

That sounds clean when written down. The actual thing has had stale build artifacts, a migration that appeared complete while the old app still served public traffic, a swallowed exception that left a migration frozen for about 32 hours, an Alembic revision mismatch, a controller that called a broken rollout healthy, and a scheduler that had already told us the truth while Forest replaced it with a worse error.

So this is not a product announcement. It is the first proper explanation of what Forest is, what it does today, and what it taught me after it got out of the repository.

The smallest real Grove

A Grove is one Forest execution cluster. In the current implementation it is a Talos Linux Kubernetes cluster, provisioned as virtual machines with libvirt/KVM on physical machines Canopy operates. A Grove has a class, such as build or runtime, and an environment class, such as production, staging, or test.

The important part is less poetic: a Grove is where a workload is allowed to run.

The first production runtime Grove, prod-runtime-1, was deliberately brought up with one control-plane VM and one worker VM. It is production-class because it accepts production-labelled workload intent and has the production trust configuration. It is not highly available. One control plane is not an etcd quorum. Calling it HA would be dishonest.

It is also not unhealthy merely because it is not HA.

Forest now keeps those questions separate. The Host-side Grove classification has independent axes for health, topology, and availability. A Grove with one ready control-plane node can be healthy, single-node, and non-ha. It can still receive work. A three-control-plane Grove can be ha. A Grove with a stale heartbeat is unknown, because Host not seeing it is not proof that the workloads stopped serving.

That distinction came after an earlier implementation made a single-node Grove effectively unusable. It required two ready control-plane nodes before a Grove could become active. We had defined a valid deployment shape and then coded it out of existence. That is the sort of mistake that happens when “healthy” quietly means “resembles the biggest diagram I have seen.”

The current topology does not make a one-node Grove a different product. The application still becomes a Workload. It still has a generation, an immutable digest-qualified image, processes, resource requests, probes, routes, volumes, release tasks, and cron tasks. The physical shape underneath can change without turning the deployment into a different thing.

Canopy Host
  environment + build artifact + current configuration
                    |
                    | compile a numbered Workload generation
                    v
Grove
  forest-controller
     |              |\
     |              | \ creates and observes
     v              v  v
  Workload CR   Deployments, Services, Routes, Certificates, PVCs, Jobs
     |
     v
  Kubernetes scheduler -> worker node(s) -> running process pods

Host owns intent and durable deployment history. The Grove owns the local act of making that intent real and observing Kubernetes. Kubernetes schedules pods and restarts containers. Forest does not pretend it has invented a better scheduler. It gives that scheduler a coherent input and makes the result intelligible to Host.

A deployment is a generation, not a container name

Host compiles an environment into a WorkloadSpec. The image must be digest-qualified. Tags are rejected in several places because a moving tag means the thing you thought you deployed is not necessarily the thing that is running. That is a boring rule, but boring is good here.

The deploy path compiles a spec against a candidate next generation first. Only when compilation succeeds does it atomically increment the workload generation and insert the immutable generation record. It then puts the workload to the selected Grove.

const candidateGeneration = workload.generation + 1;
const compiled = compileWorkloadSpec(compileInput);

await db.transaction(async (tx) => {
  const bumped = await workloadService.bumpGeneration(workload.id, {
    desiredImage: compiled.spec.image,
  }, tx);
  await workloadGenerationService.record({
    workloadId: workload.id,
    generation: bumped.generation,
    spec: compiled.spec,
    specHash,
    image: compiled.spec.image,
  }, tx);
});

That order is there because the opposite order failed in a very specific way. Bumping a counter before compilation meant a failed compile could consume a generation. Worse, a Grove could poll in the gap after the counter changed but before the matching immutable row existed. The polling route defensively skipped missing generation rows, which was sensible by itself. Together, the two operations could create a generation that the Grove would never receive.

One generation is not a huge amount of information. It is enough to make deployments and rollbacks comprehensible. Forest’s rollback does not rewind the counter. It takes the old generation’s exact image digest, recompiles current intent around it, and creates a new generation. The history is linear even when the code moves backwards.

The controller in a Grove turns the custom Workload resource into the Kubernetes objects that actually matter. Processes become Deployments and Services. Routes become HTTPRoutes, or TLSRoutes for passthrough traffic. Custom hostnames can get their own Gateway and Certificate. Volumes become PVCs. Release tasks are Jobs that gate a rollout. Cron declarations become CronJobs. The controller has deliberately narrow RBAC for these workload-owned objects. The Grove’s shared Gateway classes, Envoy fleets, issuers, and other platform plumbing are provisioned by infrastructure code, not mutated by an application controller.

That boundary has saved us from a few tempting bad ideas. A controller should be allowed to produce a workload’s route. It should not be able to redesign the Grove because a route exists.

Why Talos and Kubernetes, but not raw Kubernetes

Kubernetes is doing the things I do not want to hand-roll: scheduling, resource accounting, container lifecycle, desired-state reconciliation, namespaced isolation, volume attachment, and a reasonably serious API for controllers.

Talos is the operating system on the Grove nodes. It removes the normal server shape where a machine slowly accumulates package installs, SSH exceptions, and hand-applied fixes that exist only in somebody’s shell history. Talos has no SSH daemon and no package manager in the normal sense. Its configuration is delivered through its API. That is not magic, and it does not eliminate operations. It does mean node configuration has a much smaller place to hide.

OpenTofu defines the Grove itself: its libvirt network, VM disks, control-plane nodes, workers, Talos bootstrap, and platform components. Cilium provides pod networking and the default-deny baseline. Gateway API and Envoy Gateway handle traffic. cert-manager owns certificates. CloudNativePG runs Forest-managed Postgres services. Build and runtime Groves are separate capacity classes, so a build does not compete directly with live request-serving capacity.

Forest does not expose this entire pile as the user interface. A Canopy Host user should not need to decide whether their route is an HTTPRoute, understand the controller’s owner references, or inspect a kubeconfig before every deploy. They tell Host what should run. Host compiles that to the Forest contract.

That does not mean hiding the useful facts when things fail. We got that wrong.

Capacity is requests, not optimism

Each Forest process has a CPU and memory request. Those are carried through into the pod template. The Grove capacity reporter reads Kubernetes nodes and pods, reports allocatable CPU and memory, and calculates requested CPU and memory using Kubernetes’ effective-request rule. This matters because init containers are not simply added to normal containers. A pod reserves the larger of the sum of normal-container requests or the largest init-container request, plus pod overhead.

Forest also reports the largest CPU and memory block that a normal, untolerated pod could schedule onto. That is not the same as total free memory across a cluster. A three-gigabyte pod needs three gigabytes on one eligible worker. Two gigabytes free on one worker and two on another does not help it.

Host uses the reported totals to classify a Grove as approaching capacity at 85% requested CPU or memory, alongside object-count and etcd thresholds. It does not autoscale the Grove. Capacity changes are deliberate infrastructure changes.

This is a slightly less glamorous arrangement than a slide labelled autoscaling. It has the advantage of existing.

one physical box
  -> right-size the existing worker VM
  -> add a worker VM to the Grove
  -> add more worker capacity where the physical host permits it
  -> provision another Grove on another physical machine

The workload remains a Workload throughout.

Vertical scaling remains legitimate. The fact that a system can add workers does not mean adding RAM to the worker that already owns the useful capacity is architecturally impure.

This was not theoretical. prod-runtime-1 ran into CPU pressure first. One worker had 4 vCPU and 8 GiB of RAM and had reached 89% of its CPU request capacity. A rolling update needed a surge pod and there was no place to put it. We added a second modest worker, 2 vCPU and 4 GiB, rather than immediately duplicating the first worker and pretending this host had unlimited room.

Two days later, Cortex API needed a 3 GiB surge pod. The two workers had 10.6 GiB of schedulable memory in total. Nine real migrated workloads already requested roughly 7 GiB, and the largest free block on any one worker was 1.95 GiB. The physical machine still had around 43 GiB of free RAM. Forest had created a much smaller artificial boundary inside a machine that had capacity left.

The fix was not a new cluster. We raised the first worker from 8 GiB to 16 GiB. CPU stayed where it was because the host had 12 physical cores and 28 vCPU allocated across its VMs. That was the actual constraint. More worker memory left about 31 GiB on the physical host. It was a capacity fix, not an architecture initiative.

There is a limited window where vertical scaling is the least complicated correct answer. We should take it seriously instead of treating it as failure to have invented a distributed system early enough.

The scheduler already knew

The memory incident had a second failure, and it was worse in some ways.

Kubernetes had put the new pod in Pending and recorded exactly why:

0/3 nodes are available: 1 node(s) had untolerated taint(s),
2 Insufficient memory

Forest did not surface that. It waited through the Deployment progress deadline and showed a generic progression timeout. The platform had the useful operational truth, then replaced it with a vaguer sentence.

The old status path concentrated on Deployment conditions. An unschedulable pod does not necessarily produce a useful Deployment failure immediately. It is sitting there, unscheduled, while the progress clock runs. The scheduler verdict lives on the pod’s PodScheduled=False condition.

The controller now lists only the pods belonging to the Workload during reconciliation. After a two-minute grace period, it adds a PodsSchedulable condition containing the scheduler’s reason and message. It ignores a pending pod that is already scheduled and just pulling an image. It ignores a terminating pod. With several stuck pods, it reports the oldest one and how many are stuck.

if cond := deriveSchedulingCondition(workloadPods.Items, time.Now()); cond != nil {
    newStatus.Conditions = append(newStatus.Conditions, *cond)
}

The delay is intentional. A transient scheduling delay is normal. Alerting on every scheduler cycle produces another kind of lie, where everything is always alarming and therefore nothing is.

Build Jobs already had a similar path. Their reporter forwards an Unschedulable verdict and message to Host after the same grace period, deduplicates unchanged reports, and sends one clearing update when the pod becomes schedulable. The workload controller had not inherited that lesson. It does now.

I think the rule is simple: remove complexity without removing truth. Forest should absorb Kubernetes object plumbing. It should not turn an actionable scheduler reason into “deployment did not progress” because the latter looks friendlier in a UI.

The same incident found a separate status bug. During a surge rollout, the old pod could still be ready while the new pod existed but had never passed readiness. Forest saw one updated replica and one ready replica, both equal to the desired count of one, and called the workload healthy. It had counted two different pods as proof of the same thing.

The status check now also requires total replicas to be back at the desired count and gates Running on AvailableReplicas, not merely ReadyReplicas. An external health check reaching the old pod is not proof that the new generation landed. That was a bad abstraction boundary and production found it quickly.

Migration is not a database field

The Dokku to Forest migration work had a few different failures that look similar if reduced to “migration was hard.” They were not the same problem.

First, assigning a grove_id and clearing dokku_host_id changed Host’s future intent. It did not destroy the old Dokku app. During the host-mcp migration, the Forest workload was healthy while the old Dokku container and nginx vhost kept serving mcp.canopy.pm for more than twelve hours. It was stale code, including fixes that existed in Forest but were unreachable to public traffic.

We added a scoped teardown path:

canopy admin environment-dokku-teardown --env <environment-id>

It rechecks that the environment has a Grove, that Host no longer routes it through Dokku, and that the recorded old application still exists before destroying it. It is deliberately not a generic orphan reclaimer. This is a migration with a real environment that still owns the old name, so the safety predicate is different.

The actual cutover also found an ACME chicken-and-egg problem. The old Dokku vhost redirected HTTP to HTTPS. Forest’s ingress sync would not publish the new vhost until it had a certificate. The ACME HTTP-01 check could not get through while the old vhost answered port 80. Removing the old vhost briefly made the domain unreachable, then a temporary plain-HTTP challenge passthrough allowed the certificate to issue and the normal sync process to take over.

The reusable Dokku teardown exists now. That certificate bootstrap gap is still a known gap for the next competing-vhost migration. Saying it is fixed because we got through one cutover would be the usual kind of post-incident fiction.

Then there was an accidental database migration. A local bun run db:migrate used DATABASE_URL, which pointed at production rather than the isolated test database. The migration was additive and no data was damaged, but an in-flight older build still had the previous migration bundle. Its migration step then hit an Alembic revision mismatch until a newer build caught up. That was not a Forest controller bug. It was still part of operating the runtime, because deployment artifacts and database schema must agree about time.

The later Cortex API migration froze for about 32 hours for another reason. rebuildEnvironment() was meant to tolerate MigrationInProgressError, but it caught every exception. A failed deploy could then silently do nothing. The fix rethrows unexpected exceptions, captures them, and the deploy route now checks a frozen migration state synchronously and returns a real 409.

The lesson was not “never catch exceptions.” One sibling path processes pushes across multiple environments and catches errors so one failure does not abort unrelated environments. Context matters. The lesson was that a broad catch in the path that owns a deployment cannot turn an unknown failure into silence.

Groves, workers, and the unfinished parts

Forest can already run multiple Groves. The current fleet includes separate runtime and build Groves, with physical machines connected by a WireGuard operator-access mesh. Build artifacts are content-addressed and can be promoted to the platform registry so a runtime Grove can pull the exact digest built elsewhere. Build dependency spikes and build isolation should not eat the same worker capacity that serves an API.

Within a Grove, worker counts and worker sizing are OpenTofu inputs. The module can make multiple control-plane and worker VMs. A three-control-plane topology is the current threshold Forest uses for HA availability because it can retain etcd quorum after losing one control-plane node. The local runtime Grove exercises a three-control-plane shape. The production runtime Grove does not claim that property today.

The Forest CLI is mostly for operating this reality, not acting as a second customer interface. It can inspect Groves and workload state, report capacity, emit a kubeconfig through the operator access path, drain workloads, and perform an explicitly confirmed rebuild into another Grove.

canopy admin groves capacity
canopy admin groves workloads <grove-id>
canopy admin groves drain-workloads <grove-id> --batch 10
canopy admin groves rebuild <grove-id> --into <target-grove-id> --confirm

There are limits to say plainly. Forest has no autoscaling today. It does not automatically add workers, control planes, or regions. It does not provide automatic multi-region failover. A Grove can be reconstructed from infrastructure code and its workloads can reconverge from declared intent, but a full deliberate destructive rehearsal of the production Grove has not happened. The controller release process has also been more manual than I want. Different Groves have carried different controller image digests, and that drift has made protocol changes harder to ship safely.

The infrastructure module has a comment about where a future cross-machine load-balancer pair would belong. It does not mean it exists. The Grove bootstrap module is named in the infrastructure layout and is not written. Those are exactly the sort of things that become accidental roadmap claims in an infrastructure post, so I am leaving them here as unfinished work instead.

What I would keep

Forest has made me more convinced that the smallest supported architecture matters. Not as a demo environment. As a real operating stance.

There is no prize for making a small deployment inherit the control-plane ceremony of a large one. More nodes are good when they solve a failure-domain, capacity, or maintenance constraint. A larger worker is good when it solves the constraint without inventing several others.

The opposite mistake is also real. A one-node Grove is only honest if its schedulable capacity corresponds to the physical capacity we have chosen to give it. The memory incident was a reminder that the host having free RAM is irrelevant if the worker VMs cannot offer it to Kubernetes. A diagram can say “multiple workers” while a 3 GiB pod has nowhere to go. The scheduler does not care how nice the diagram looks.

Forest is still evolving. There are things in it I would now design differently from the start. There are things that became less abstract after an outage. The useful direction has stayed fairly stable: applications should be deployable, understandable, recoverable, and able to grow when a real constraint asks them to.

That does not require turning every Grove into a tiny enterprise Kubernetes installation.

It requires being honest about what is running, what is not, and why.

Technical claims deliberately excluded

  • Forest does not automatically scale workers, Groves, or regions from load.
  • Forest does not currently provide automatic multi-region failover.
  • prod-runtime-1 is not HA. It has one control plane and must not be described as quorum-tolerant.
  • A cross-machine Grove load-balancer pair is discussed in infrastructure comments, but is not implemented.
  • The grove-bootstrap infrastructure module is planned in the layout and does not exist yet.
  • Production-Grove total-loss recovery has not had a full deliberate destructive rehearsal.