Self-hoster shares lessons from running bare-metal Kubernetes, seeks lighter alternative

Over the past year, a self-hoster built and ran a bare-metal Kubernetes deployment using Proxmox to prep base images, Ansible to bring up virtual machines, install dependencies like containerd or cri-o, and kubeadm/kubectl to form the cluster. Three separate clusters were each placed in their own VLAN, but together they consumed more than 20% of the hardware's CPU and RAM just to keep Kubernetes running, without doing any actual useful work. One deployment needed a dedicated Layer 3 network for custom routing, but insufficient depth with the Calico networking plugin meant the workaround was simply isolating that cluster on its own VLAN rather than solving the routing properly.

A missed certificate expiration caused the main cluster to fail hard, and there wasn't time to fix it, leading to a full week of downtime. The experience led to the conclusion that high availability, while nice, matters less than simplicity, and now a lighter-weight alternative that offers close-to-Kubernetes functionality without the full overhead is being sought.