- Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes".
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
by MisterKent
9 subcomments
- Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup.
Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
- > Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
- You might like https://uncloud.run/
- The problem with using k8s on homelabs, is that a lot of the applications you would usually deploy, are not designed for it; having to manage a bunch of persistent volumes because most of your applications use sqlite is not very practical, and if the backend is sqlite, then you are probably running only one pod, so no real HA (if the pod goes down k8s will start a new one though), if you have to go through hoops to deploy an application that's not designed for it decreases its value.
Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.
- IMO, kubernetes is overkill for a small non-homogeneous home cluster.
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
- I still don't know really what Kubernetes is for or why so many people outside specific environments are using it, but it's cool that you're using Ruby.
- Unless you’re running multiple home servers, I can not reason any reason to use k8s. I’ll push any business to k8s but I would never bother with 1-3 home labs. Surprised this is even a hacker news topic of interest.
- Meanwhile I’m busy moving the other direction. More K8S.
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
- I've recently reached for pyinfra of all things and found it straightforward enough... just an epsilon above a pile of bash scripts.
- I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..
- The deterministic IPv6 approach is fascinating. Have you run into any issues with ISP-provided IPv6 prefix delegation changing over time, or are you strictly using ULAs (Unique Local Addresses) internally to bypass that headache?
- Most of the stuff hyperscalers use have no place in homelabs, unless you're training for a job application.
So duh.
Hacker News