As I continue to mess around with various ways of installing and running Kubernetes in my home lab using Rancher I keep coming up with different ways to solve similar problems. Each time I set it up using different host OSs I learn a bit more which my primary goal. The latest iteration uses CentOS 8 and allows for iSCSI based persistent storage to work properly. I want to use CentOS 8 because it includes a newer kernel required for doing buildx based multi-arch builds. In this post, I’d like to go through the process of setting up CentOS 8 with Docker and what utilities to install to support NFS and iSCSI based persistent storage so that it works properly with Rancher.

In my home setup I am combining Rancher 2.x running in a VM on Proxmox with a FreeNAS based system for persistent storage. The host(s) running the Kubernetes components installed by Rancher are also running on Proxmox using disk images created using the process described here. The FreeNAS system is exporting both NFS as well as iSCSI based storage. The NFS based storage uses “external-storage” while the iSCSI setup was using the “freenas-iscsi-provisioner.” Using these tools allows Kubernetes to create storage that can be handed over to Pods that ask for it. Since setting them up initially both projects have been deprecated in favor of https://github.com/democratic-csi/democratic-csi. Although the projects I use to expose NFS and iSCSI storage have been deprecated how to setup the worker system should remain the same regardless.

Starting with the base CentOS 8 template generated by the Packer process previously discussed, I install the following pieces which are available in the base OS:

• nfs-utils
• iscsi-initiator-utils

These two software packages will install the necessary underlying components to allow the OS to mount NFS shares as well as discover and access iSCSI based shares. I then installed Docker using the process defined at https://linuxconfig.org/how-to-install-docker-in-rhel-8. Last, you will want to ensure firewalld has been removed, Docker has been enabled (using systemctl enable docker), selinux is disabled and the system rebooted. It is possible to install Rancher with selinux enabled if you prefer, refer to their documentation on how to do so.

With these prerequisites taken care of I moved on to installing Kubernetes onto the system. In Rancher with an existing cluster this is done by editing the cluster, selecting the desired options and then copy/pasting the command shown, which will look similar to this:

In my setup I wanted the new machine to take over all duties so I could delete the old one.

Once this is complete you need to edit how the Kubelet is run. This is because Rancher runs everything from Docker containers and the container that runs Kubelet is based on Debian which has different paths for iscsiadm. The general process is defined at https://github.com/rancher/rancher/issues/15118 but I found I had to add additional mounts to get it all working. To edit the how Kubelet is run edit the cluster and then click the Edit as YAML button. Find where kubelet is defined and add the following just under it:

extra_binds:
  - '/etc/iscsi:/etc/iscsi'
  - '/sbin/iscsiadm:/sbin/iscsiadm'
  - '/usr/lib64/libopeniscsiusr.so.0.2.0:/usr/lib/libopeniscsiusr.so.0.2.0'

It should look like this when finished:

When done click save and Kubelet will be restarted. Once finished Kubernetes should have no troubles accessing NFS and iSCSI based shares.