The control plane managed cloud hides.

The premise

The managed button is somebody else's control plane

A managed Kubernetes service, a one-click database, a control plane you never see: all of it is real infrastructure, built by someone, running underneath the button. Click-to-provision is not the absence of the hard work. It is the hard work, already done, wearing a nice interface.

A game studio hired me to build their entire hosting product (the full story is here). It ran on plain rented servers: bare machines with nothing set up on them, and no managed control plane underneath. So I built the control plane by hand.

What that meant

A workflow that survives its own crash

Adding a new server is not one step. It is several: create the machine, wait for it to boot, join it to the pool already running, wait for it to report ready. Any one of them can fail on its own schedule.

I wrote that sequence as a durable workflow in a small Go control plane, so a crash mid-provision does not strand a half-built machine that nobody notices. On restart, the workflow resumes from where it stopped instead of starting over or giving up.

It needed to. One upstream bug crashed a machine's first boot before its own setup script ever ran, on roughly three attempts in ten. The workflow's answer is not clever. It destroys the failed machine and creates a fresh one, up to a fixed number of tries. That dropped the first-attempt failure rate from about 30 percent to a few percent, which is the difference between a fleet that mostly builds itself and one an operator has to babysit.

Where the state lives

Never store the map inside the territory

Your infrastructure has a description: a record of what exists and where. That record, its Terraform state, I keep off the servers entirely, in separate storage the same system cannot reach. Never on a machine it could wipe.

The reason is simple. If the map of your servers lives on the servers themselves, the failure that takes one down can take your only record of it along with it. Keep the map somewhere the servers cannot reach, and a bad day stays a bad day instead of an unrecoverable one.

The same instinct runs the networking. Every private tunnel into the fleet is defined as code and created by the pipeline, never typed by hand into a console and left to become an orphan nobody remembers making.

Portable by design

Region is a variable, not an assumption

Every layer of this, the provisioning workflow, the networking, the k3s fleet, is written so a region is something you pass in, not something baked into one provider's own tooling. Move to a different provider and the workflow, the retries, and the state model do not change. Only the arguments do.

That portability was never a feature request. It is insurance against the exact vendor dependency I have written about elsewhere on this site: whoever controls your control plane controls your options the day a provider's price or capacity changes.