K3s starter kit on Hetzner cloud

What it is

• OpenTofu for the machines, with terraform-hcloud-kube-hetzner as a pinned dependency. Not a fork of it. That project solves cluster creation itself, and none of it is rewritten here.
• Flux for delivery, with the controller manifests committed rather than bootstrapped. No personal access token, and nothing gets written back to your repo.
• SOPS and age for secrets. Exactly one secret in the whole system is applied by hand and cannot be encrypted.
• Gateway API for routing, where the platform owns its Gateway and tenants own their Routes.
• VictoriaMetrics, VictoriaLogs and Tempo behind an OpenTelemetry collector. The collector is what makes traces worth having, because it stamps k8s.pod.name, k8s.namespace.name and k8s.node.name onto every span. An application knows what it is. It does not know where the scheduler put it.
• A namespace per tenant, generated with its quota, its limits, its network policy, its disruption budget, its backup job, its pod security level and its RBAC.

What it costs to run

€122.45 a month gross, including 25% Danish VAT, for the three nodes and the load balancer above. The kit’s default shape is three control planes and two agents so there is real quorum, which is €171.17.

The number that matters more is what the platform uses before any of your own code runs: 4,526 Mi of memory and 194m of CPU across 43 pods, two thirds of it observability, against 8 Mi for the static site it was serving. CPU is a rounding error at about 2% of allocatable. Memory is what you are paying for, and the same 4.5 GiB carries both the second tenant and the tenth.

The per-namespace breakdown, the 161,507 active time series, and the two control-plane histograms that are 27% of them with nobody querying them, are in the repository’s verification notes.

Where k3s is genuinely stronger

The FreeBSD end is not bare. Jails are the isolation primitive, bhyve runs a full guest kernel when a workload needs one, and bastille or cbsd will manage either. What is missing is the layer above them: nothing schedules across the three machines, and nothing reconciles them back to a written-down state. One caveat if you are planning around bhyve on this provider, though. It needs hardware virtualisation, and Hetzner Cloud exposes none, which is why installing FreeBSD there runs under TCG emulation. On these machines you get jails, not virtual machines.

I also run a production app on one FreeBSD machine for about €7 a month, and I have done the arithmetic on doing the same with three. Put those machines on the same instance type and the price difference disappears: three cpx32 is €133.08, more than the cluster, because the cluster’s control plane is a smaller machine. The saving is not in the operating system. It is that the FreeBSD machines fit in 4 GB and the k3s machines do not. So the cost argument against Kubernetes is really an observability argument in disguise. If you would build metrics, logs and traces anyway, you have already paid.

Docker Swarm fits in the middle: one team running their own apps, which talk to each other freely. If one thing needs isolating, put it outside the swarm. If several do, that is where Kubernetes starts to be the better choice, because the isolation is generated with each namespace and checked when it is applied, rather than remembered.