Hardware
Hardware for my Raspberry Pi 4 K3s Kubernetes cluster
This is the list of hardware I’m going to use. You can achieve the same with much less hardware, but this is what I have. The minimal setup I would go for is 3x Raspberry Pi with SSD and USB disk.
- 8x Raspberry Pi 4 (4 GB Ram version, more ram the better)
- 2x ROCK Pi 4C (4 GB Ram version) I have chosen this as possible alternative for Raspberry Pi 4 you can read more about it here: Rock Pi 4C review. These are not yet used in my cluster, but I’m planning to use them in the future.
- 9x 32 GB USB Disk (Samsung USB 3.1 32 GB Fit Plus)
- 9x 64 GB USB Disk (Samsung USB 3.1 64 GB Fit Plus) - This might be unnecessary, since I’m using 3x SSDs for storage for pods now.
- 3x SSD* 512GB cheap SSD
- Mikrotik 10 port Router mostly because of storage. Storage is network driven, and the faster the connection you have the better.
- 1x Power Supply 12V 20A (300W) Changed from USB power hubs to 12V power supply. For one Raspberry Pi 4 you need around 15W (5V x 3A). More information can be found in dedicated part of the guide to my chassis here Cluster chassis
Regarding the USB disks – The bigger ones (64 GB) were used as persistent storage for pods in my older iteration of the cluster. I was experimenting with GlusterFS storage, some CEPH + ROOK and ended with Longhorn. However, thanks to new power solution in my cluster I have switched to SSDs for storage. 3x SSDs on 3 Nodes (1 SSD per node) with help of Longhorn will give me all the storage and reliability I need for now.
The smaller ones are boot USB disks. This way, I hope it will live much longer than an SD card 🙂 I know they are almost the same technology, but experience so far has been better than with SD cards. Best solution would be to use USB to SSD or M.2 for OS, but that adds cost.
Speed benchmarks
Speed benchmarks between USB drive and SD card.
SD Card (Kingston Canvas Select Plus micro SDHC 16 GB Class 10 UHS-I):
Category Test Result
HDParm Disk Read 41.25 MB/s
HDParm Cached Disk Read 37.70 MB/s
DD Disk Write 9.9 MB/s
FIO 4k random read 2910 IOPS (11642 KB/s)
FIO 4k random write 657 IOPS (2630 KB/s)
IOZone 4k read 10288 KB/s
IOZone 4k write 2340 KB/s
IOZone 4k random read 7832 KB/s
IOZone 4k random write 2404 KB/s
Score: 963USB Flash Drive 64 GB (Samsung USB 3.1 64 GB Fit Plus
Category Test Result
HDParm Disk Read 69.28 MB/s
HDParm Cached Disk Read 70.57 MB/s
DD Disk Write 24.4 MB/s
FIO 4k random read 2960 IOPS (11841 KB/s)
FIO 4k random write 2932 IOPS (11731 KB/s)
IOZone 4k read 13793 KB/s
IOZone 4k write 13249 KB/s
IOZone 4k random read 10692 KB/s
IOZone 4k random write 11654 KB/s
Score: 2524NEW: SSD disk to 3.1USB (Apacer AS350 Panther 512GB):
Category Test Result
HDParm Disk Read 36.34 MB/s
HDParm Cached Disk Read 47.24 MB/s
DD Disk Write 107 MB/s
FIO 4k random read 21833 IOPS (87334 KB/s)
FIO 4k random write 9338 IOPS (37355 KB/s)
IOZone 4k read 29870 KB/s
IOZone 4k write 23508 KB/s
IOZone 4k random read 21132 KB/s
IOZone 4k random write 28707 KB/s
Score: 7152As you can see, this is not a miracle speedup, but it is better than an SD card, and hopefully a USB flash drive can survive a bit longer. Of course, if you would use any actual SSD drive the score would be something above 6000. Well, the score for SSD is indeed above 6000, its 7152 so that's nice. Or, you can boot from a network, which also avoids SD cards, but for that you need a beefier PXE boot server. I wonder what kind of performance we could get using iSCSI from NAS. I'll add that to my bucket list :)
Script used for testing: https://github.com/TheRemote/PiBenchmarks