This is the list of hardware I’m going to use (you can achieve this with one worker, or two)
- 9x Raspberry Pi 4 ( 4GB Ram version ) – 1pc = 68.90,-Eur
- 9x 32GB USB Disk (Samsung USB 3.1 32 GB Fit Plus) – 1pc = 10.90,-Eur
- 9x 64GB USB Disk ( Samsung USB 3.1 64 GB Fit Plus) – 1pc = 19.50,-Eur
- 1x WiFi router TP-Link TL-WR902AC – 1pc = 34.90,- Eur ( Later I changed from WiFi to LAN cables and Mikrotik 10 port Router, ostly because of storage. Storage is network driven, and the faster the connection you have the better. However, for basic testing WiFi will do just fine)
- 2x I-TEC USB 3.0 Charging HUB 16port + Power Adapter 90 W – 1pc = 54.90,-Eur ( One is not enough (One is not enough to power 9x Rpi, since we also power USB disks, and for just 1x Rpi you need around 15W (5V x 3A) [ all together 9x Rpi can draw up to 135W ]. Therefore,I’m adding one more of the same USB hub to help, but if possible, go with a PoE solution if you can get a cheap PoE switch.*).
In total: 1 036.15,- Eur ( shit, that much?!? I have bought these components over time… but fuck me, for that I could get HP G6 or G7 2U server, second hand 😀 )
Regarding the USB disks – The bigger ones 64GB) are to be used as persistent storage for pods. I was experimenting with GlusterFS storage, some CEPH + ROOK and ended with Longhorn.
The smaller ones are boot USB disks. This way, I hope it will live much longer than an SD card 🙂
Speed benchmarks between USB drive and SD card.
SD Card (Kingston Canvas Select Plus micro SDHC 16GB Class 10 UHS-I):
Category Test Result HDParm Disk Read 41.25 MB/s HDParm Cached Disk Read 37.70 MB/s DD Disk Write 9.9 MB/s FIO 4k random read 2910 IOPS (11642 KB/s) FIO 4k random write 657 IOPS (2630 KB/s) IOZone 4k read 10288 KB/s IOZone 4k write 2340 KB/s IOZone 4k random read 7832 KB/s IOZone 4k random write 2404 KB/s Score: 963
USB Flash Drive 64GB (Samsung USB 3.1 64 GB Fit Plus):
Category Test Result HDParm Disk Read 33.92 MB/s HDParm Cached Disk Read 25.00 MB/s DD Disk Write 16.2 MB/s FIO 4k random read 1713 IOPS (6855 KB/s) FIO 4k random write 1722 IOPS (6890 KB/s) IOZone 4k read 12017 KB/s IOZone 4k write 8285 KB/s IOZone 4k random read 10110 KB/s IOZone 4k random write 9038 KB/s Score: 1917
USB Flash Drive 256GB (Samsung USB 3.1 256 GB Fit Plus):
Category Test Result HDParm Disk Read 69.28 MB/s HDParm Cached Disk Read 70.57 MB/s DD Disk Write 24.4 MB/s FIO 4k random read 2960 IOPS (11841 KB/s) FIO 4k random write 2932 IOPS (11731 KB/s) IOZone 4k read 13793 KB/s IOZone 4k write 13249 KB/s IOZone 4k random read 10692 KB/s IOZone 4k random write 11654 KB/s Score: 2524
As you can see, this is not a miracle speedup, but it is better than an SD card, and hopefully a USB flash drive can survive a bit longer. Of course, if you would use any actual SSD drive the score would be something above 6000. Or, you can boot from a network, which also avoids SD cards, but for that you need a beefier PXE boot server.
Script used for testing: https://github.com/TheRemote/PiBenchmarks