I don't want to go into too much detail here about what happened.

I have successfully created a k8s cluster with a stacked ETCD database, using 3 nodes as a control plane, and the rest as workers.

Using docker for containers.

And it worked... sort of. I'm not sure why, but from time to time, one or more nodes just stop responding. It took a setup of the central logging system to figure out that this was random. The node fucked up USB drivers, and switched to read only mode... So, the node was "up" and pingable. It just would not write to USB, and you could not log in.

Rebooting was my only option. I can't be really sure if this was caused by k8s, in the end. It could be that Ubuntu 20.10 is not fully baked, or something. There was a kernel update since I installed K3s, and I have not seen this happen yet.

I have also experienced quite high loads using K8s on Rpi 4. It was ok-ish in the 60% to 70% load, but when I hit it with Rook + Ceph, and started some cpu intensive service nodes, it just got bogged down. Also, Rook + Ceph gave me so many issues (related to the usb switching to read only) that I gave up on K8s altogether.

Last update: June 9, 2022