Skip to content

Storage

Storage is certainly something you gonna need. Stateless application/containers are fun and all, but in the end if you want to do something useful you need to store some data. Even if that data is as simple as configuration file.

Here we are getting to point where the Raspberry Pi is going to fail us. Simply put there is no native CIS ( Container Storage Interface ) that is officially supported for arm64. And looking at the rest that exists, how are they even used in production since most of them are barely in beta, and only few are in stable releases…

Seriously how is Amazon or Google handling persistent storage on theirs Kubernetes clusters ? Maybe they just use external SAN like 3Par and present LUNs to each node… how than they deal with multiple containers trying to write to same file ?

Options

  • Rook + Ceph – This one you can possibly get to work, but with unofficial arm64 builds thanks to https://github.com/raspbernetes/multi-arch-images. I believe this is stable on normal servers and even production ready, but did not survive two reboots of my K8s cluster on Raspberry Pi 4. Combining with heavy load and not steep but vertical learning curve of Ceph, I would not recommend this for Raspberry home cluster.

  • Longhorn – Another native Kubernetes storage, now finally with support for arm64! https://github.com/longhorn/longhorn/issues/6 I ended up with this one as only viable solution.

  • GlusterFS + Heketi – GlusterFS works fine on Raspberry Pi, I tested it… Heketi is dead though.. so not really native support. However we can use GlusterFS to mount on each node a folder, and tell Kubernetes to use local FS as storage… this would make the data available on every node and in case pod will switch to another the persistent data will be there waiting for him. Slight issue with GlusterFS though. Its not recommended for “live” files… aka. databases, which sux… But to be hones I have seen MySQL running on GlusterFS cluster in production 😀

  • NFS – Funny enough this one works just fine, you can create claims and manage it from Kubernetes ( My first cluster was using NFS as persistent storage and it worked fine. ) But this is not clustered and its single point of failure, which in turn is against the exercise we are trying to do here.

Longhorn

Fairly new Kubernetes native file-system for arm64, at the time of writing, but man it just worked! After the ordeal with Rook + Ceph it was such a breeze to set it up ! My cluster is in 3 week stability testing phase now and nothing broke.

As you know from our node setup, I’ want to use separate USB flash-drive 64GB on each node to be volume for storage. Longhorn is making this simple for us, all you need to do is mount the disk under /var/lib/longhorn, this is a default data path. I forgot to do that beforehand and had to change it later, its super simple so no worries. I ended up with my storage under /storage.

There is an issue with Raspberry Pi / Ubuntu. The names for disks are assigned almost at random. So even if you have USB disks in the same slots on multiple nodes they can be named at random /dev/sda or /dev/sdb and there is no easy way to enforce the naming, without messing with udev rules a lot.

Identifying disks for storage.

We are going to use Ansible again a lot and will add new variable with disk name that will be used for storage into /etc/ansible/hosts.

We are going to use lsblk -f command on every node and look for disk labels

ubuntu@control01:~$ ansible cube -b -m shell -a "lsblk -f"
control01 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
sdb
├─sdb1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sdb2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d  220.3G     2% /
cube02 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.2G    13% /
sdb
└─sdb1 exfat  1.0   Samsung USB 64A5-F009
control02 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   23.3G    16% /
sdb
cube01 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.3G    12% /
sdb
control03 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   23.1G    16% /
sdb
cube03 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.2G    13% /
sdb
cube04 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.1G    13% /
sdb
cube05 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sda2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.2G    13% /
sdb
cube06 | CHANGED | rc=0 >>
NAME   FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
sdb
├─sdb1 vfat   FAT32 system-boot 2EC5-A982                             100.8M    60% /boot/firmware
└─sdb2 ext4   1.0   writable    c21fdada-1423-4a06-be66-0b9c02860d1d   24.2G    13% /

As you can see each node have two disks sda and sdb but assigned at boot. Every disk that splits to <name>1 and <name>2 and have /boot/firmware and / as mount points are our OS disks. The other one is our “storage” 🙂 Not all my disks are wiped though, so lets take care of that. You can see disks with: sdb1 exfat 1.0 Samsung USB 64A5-F009 which is default FAT partition for USB disk for windows and might be there by default….

Edit /etc/ansible/hosts and add new variable ( I have chosen name var_disk ) with the disk to wipe. TAKE YOUR TIME AND LOOK TWICE ! wipefs command we gonna use, will not wipe OS disk but might wipe any other that is not mounted.

[control]
control01  ansible_connection=local var_hostname=control01 var_disk=sda
control02  ansible_connection=ssh   var_hostname=control02 var_disk=sdb
control03  ansible_connection=ssh   var_hostname=control03 var_disk=sdb

[workers]
cube01  ansible_connection=ssh  var_hostname=cube01 var_disk=sdb
cube02  ansible_connection=ssh  var_hostname=cube02 var_disk=sdb
cube03  ansible_connection=ssh  var_hostname=cube03 var_disk=sdb
cube04  ansible_connection=ssh  var_hostname=cube04 var_disk=sdb
cube05  ansible_connection=ssh  var_hostname=cube05 var_disk=sdb
cube06  ansible_connection=ssh  var_hostname=cube06 var_disk=sda

[cube:children]
control
workers

Wipe

Wipe them! Wipe them all with wipefs -a, its the only way to be sure !

ansible cube -b -m shell -a "wipefs -a /dev/{{ var_disk }}"
The FS was wiped from /dev/sdb on cube2 node. You can check with the same command than before. All disks are ready. Remove the variable from /etc/ansible/hosts as we can’t guarantee that the names will be the same at next boot ( they should but who knows… ).

Filesystem and mount

We need to mount the storage disks, but before we need some file-system on them. Generally ext4 is recommended.

ansible cube -b -m shell -a "mkfs.ext4 /dev/{{ var_disk }}"
ansible cube -m shell -a "mkdir /storage" -b
ansible cube -b -m shell -a "mount /dev/{{ var_disk }} /storage"

We also need to add this disks into /etc/fstab so after reboot they mount automagically. For that we need UUID of the disks.

ubuntu@control01:~$ ansible cube -b -m shell -a "blkid -s UUID -o value /dev/{{ var_disk }}"
control03 | CHANGED | rc=0 >>
a0aa3b5c-743a-4948-a490-d38ee1ab2211
control01 | CHANGED | rc=0 >>
4690b93a-0466-401a-9900-beff65cca358
cube02 | CHANGED | rc=0 >>
5fc0389b-c8bf-4229-94db-99d7362496b6
cube01 | CHANGED | rc=0 >>
9792876f-ce6b-4a40-83da-d92aae49a51d
control02 | CHANGED | rc=0 >>
accdfe09-a0f8-43e3-9663-c095e9929fc9
cube03 | CHANGED | rc=0 >>
87296139-c7f8-4e5b-8ffa-f651fd445d76
cube04 | CHANGED | rc=0 >>
7ad7354b-8846-45f6-b0e1-ef0dc71d99fe
cube05 | CHANGED | rc=0 >>
40ee7dc2-6cc2-4520-bd84-367d0e2565be
cube06 | CHANGED | rc=0 >>
c416b477-7766-47a8-b9af-e8ae2768c55f

Add there to /etc/ansible/hosts with another custom variable, for example:

[control]
control01  ansible_connection=local var_hostname=control01 var_disk=sda var_uuid=4690b93a-0466-401a-9900-beff65cca358
control02  ansible_connection=ssh   var_hostname=control02 var_disk=sda var_uuid=accdfe09-a0f8-43e3-9663-c095e9929fc9
control03  ansible_connection=ssh   var_hostname=control03 var_disk=sda var_uuid=a0aa3b5c-743a-4948-a490-d38ee1ab2211

[workers]
cube01  ansible_connection=ssh  var_hostname=cube01 var_disk=sdb var_uuid=9792876f-ce6b-4a40-83da-d92aae49a51d
cube02  ansible_connection=ssh  var_hostname=cube02 var_disk=sda var_uuid=5fc0389b-c8bf-4229-94db-99d7362496b6
cube03  ansible_connection=ssh  var_hostname=cube03 var_disk=sda var_uuid=87296139-c7f8-4e5b-8ffa-f651fd445d76
cube04  ansible_connection=ssh  var_hostname=cube04 var_disk=sdb var_uuid=7ad7354b-8846-45f6-b0e1-ef0dc71d99fe
cube05  ansible_connection=ssh  var_hostname=cube05 var_disk=sdb var_uuid=40ee7dc2-6cc2-4520-bd84-367d0e2565be
cube06  ansible_connection=ssh  var_hostname=cube06 var_disk=sdb var_uuid=c416b477-7766-47a8-b9af-e8ae2768c55f

[cube:children]
control
workers

Using Ansible we add line into /etc/fstab ( yea I know, should have used lineinfile module... )

ansible cube -b -m shell -a "echo 'UUID={{ var_uuid }}  /storage       ext4    defaults        0       2' >> /etc/fstab"
#Check
ansible cube -b -m shell -a "grep UUID /etc/fstab"
#Make sure mount have no issues.
ansible cube -b -m shell -a "mount -a"

Longhorn requirements

Install open-iscsi on each node

ansible cube -b -m apt -a "name=open-iscsi state=present"

Install Longhorn

#Switch to home dir
cd
#Clone Longhorn, I did specified v1.1.0 since this was the first not yet released
#arm64 supporting version. I think its out now and you don't need -b v1.1.0
git clone -b v1.1.0 https://github.com/longhorn/longhorn.git
#Ah look Helm we installed before, use it !
kubectl create namespace longhorn-system
helm install longhorn ./longhorn/chart/ --namespace longhorn-system --kubeconfig /etc/rancher/k3s/k3s.yaml --set defaultSettings.defaultDataPath="/storage"

Give it some time, it should deploy, maybe some pods do restarts but in the end it should look something like this. More or less, this is already with storage created and attached to docker-registry. Everything under namespace longhorn-system should be 1/1 Running.

root@control01:/home/ubuntu# kubectl -n longhorn-system get pod
NAME                                        READY   STATUS    RESTARTS   AGE
csi-attacher-5dcdcd5984-6txrq               1/1     Running   0          6d
csi-attacher-5dcdcd5984-n9c5b               1/1     Running   0          6d
csi-attacher-5dcdcd5984-pm5pz               1/1     Running   0          6d
csi-provisioner-5c9dfb6446-6cpn4            1/1     Running   0          6d
csi-provisioner-5c9dfb6446-wcwsl            1/1     Running   0          6d
csi-provisioner-5c9dfb6446-xsnhx            1/1     Running   0          6d
csi-resizer-54d484bf8-2f8mh                 1/1     Running   0          6d
csi-resizer-54d484bf8-fvrr7                 1/1     Running   0          6d
csi-resizer-54d484bf8-pv7pp                 1/1     Running   0          6d
csi-snapshotter-96bfff7c9-cj7rt             1/1     Running   0          6d
csi-snapshotter-96bfff7c9-hv4gk             1/1     Running   0          6d
csi-snapshotter-96bfff7c9-jwbjc             1/1     Running   0          6d
engine-image-ei-d73fbea2-2kvjx              1/1     Running   0          6d
engine-image-ei-d73fbea2-6s268              1/1     Running   0          6d
engine-image-ei-d73fbea2-97gkp              1/1     Running   0          6d
engine-image-ei-d73fbea2-dhlg2              1/1     Running   0          6d
engine-image-ei-d73fbea2-hbczq              1/1     Running   0          6d
engine-image-ei-d73fbea2-jdldc              1/1     Running   1          6d
engine-image-ei-d73fbea2-m5c7r              1/1     Running   0          6d
engine-image-ei-d73fbea2-mtgd9              1/1     Running   0          6d
engine-image-ei-d73fbea2-vhjb5              1/1     Running   0          6d
instance-manager-e-222f0b35                 1/1     Running   0          6d
instance-manager-e-4c86d798                 1/1     Running   0          6d
instance-manager-e-59711753                 1/1     Running   0          6d
instance-manager-e-787819f8                 1/1     Running   0          6d
instance-manager-e-ada009c8                 1/1     Running   0          6d
instance-manager-e-b1661214                 1/1     Running   0          6d
instance-manager-e-d52fe797                 1/1     Running   0          6d
instance-manager-e-e1e5db80                 1/1     Running   0          6d
instance-manager-e-eef19185                 1/1     Running   0          6d
instance-manager-r-169dc3b5                 1/1     Running   0          6d
instance-manager-r-1b1bee2d                 1/1     Running   0          6d
instance-manager-r-2ea20bcf                 1/1     Running   0          6d
instance-manager-r-36d070bf                 1/1     Running   0          6d
instance-manager-r-527ae0a4                 1/1     Running   0          6d
instance-manager-r-9e702e8d                 1/1     Running   0          6d
instance-manager-r-bfeba198                 1/1     Running   0          6d
instance-manager-r-d0d7ca7b                 1/1     Running   0          6d
instance-manager-r-e0405b80                 1/1     Running   0          6d
longhorn-csi-plugin-2727k                   2/2     Running   0          6d
longhorn-csi-plugin-44hwr                   2/2     Running   0          6d
longhorn-csi-plugin-9fvsz                   2/2     Running   0          6d
longhorn-csi-plugin-b6kv2                   2/2     Running   0          6d
longhorn-csi-plugin-k8996                   2/2     Running   0          6d
longhorn-csi-plugin-m4c98                   2/2     Running   0          6d
longhorn-csi-plugin-qvdnl                   2/2     Running   0          6d
longhorn-csi-plugin-th6kh                   2/2     Running   0          6d
longhorn-csi-plugin-vlpwg                   2/2     Running   0          6d
longhorn-driver-deployer-75fcf87c46-8nsh7   1/1     Running   0          6d
longhorn-manager-56cs4                      1/1     Running   3          6d
longhorn-manager-84csp                      1/1     Running   3          6d
longhorn-manager-8mcq9                      1/1     Running   3          6d
longhorn-manager-c8fc4                      1/1     Running   4          6d
longhorn-manager-gdf6b                      1/1     Running   3          6d
longhorn-manager-mfzwf                      1/1     Running   4          6d
longhorn-manager-n8c95                      1/1     Running   3          6d
longhorn-manager-pzsvl                      1/1     Running   3          6d
longhorn-manager-zp2bj                      1/1     Running   4          6d
longhorn-ui-758c86dfbc-hlfdm                1/1     Running   0          6d

Look also at services. longhorn-frontend is management UI for storage, similar to what Rook + Ceph have, very useful!

Later on we will assign in its own LoadBalancer IP.

root@control01:/home/ubuntu# kubectl -n longhorn-system get svc
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
csi-attacher        ClusterIP   10.43.125.73    <none>        12345/TCP   6d
csi-provisioner     ClusterIP   10.43.118.73    <none>        12345/TCP   6d
csi-resizer         ClusterIP   10.43.245.224   <none>        12345/TCP   6d
csi-snapshotter     ClusterIP   10.43.230.3     <none>        12345/TCP   6d
longhorn-backend    ClusterIP   10.43.118.82    <none>        9500/TCP    6d
longhorn-frontend   ClusterIP   10.43.204.227   <none>        80/TCP      6d

UI

Important

You don't need to do this step, we added /storage during installation already. But I keep this here for further reference. Continue on "Make Longhorn the default storageclass"

Either directly from node, or ssh tunneling the fronted port to your workstation, you can access the UI. There is no log in and we are not going to use one, since my cluster is not accessible from Internet, it can be as it is. I'm sure there is a setup to make it protected.

This is how it looks like, with already one volume claimed ( You should have 0 in the left dial )

Longhorn UI

Add /storage

Important

You don't need to do this step, we added /storage during installation already. But I keep this here for further reference. Continue on "Make Longhorn the default storageclass"

We need to add our mounted storage as disk for Longhorn. Navigate to Node via the web UI. You need to to this for each node. Click on the Operation -> Edit node and disks.

Longhorn disk settings

You will have already node populated with default storage /var/lib/longhorn and some random name.

First click on Add Disk and fill in new disk location and make sure you switch Scheduling to Enable.

Longhorn add disk

Second on the disk with path /var/lib/longhorn switch Scheduling to Disable. and than click on trash can to remove it.

Do the same for each node.

Make Longhorn the default storageclass

Almost done ! I'd like to make Longhorn the default storage provider. So when using Helm that have already pre-set chart to use default storage provider, it will chose Longhorn.

By default it would look like this after fresh k3s install.

root@control01:/home/ubuntu# kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)          rancher.io/local-path   Delete          WaitForFirstConsumer   false                  6d1h
longhorn (default)   driver.longhorn.io      Delete          Immediate              true                   6d1h

Execute

kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Result

root@control01:/home/ubuntu# kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path          rancher.io/local-path   Delete          WaitForFirstConsumer   false                  6d1h
longhorn (default)   driver.longhorn.io      Delete          Immediate              true                   6d1h

Now Longhorn is the default storage class.

Thats all for now, I'll get into how to create PV and PVC for deployments when we are going to install docker-registry.


Last update: February 8, 2021

Comments