Home VPN WorkPlace Web Sites/Apps Small Business Other Apps Contact Guides

Raspberry Pi Cluster Running Ubuntu Server 20.04

This cluster utilizes NFS to store all of the file systems onto a single drive.

For easy setup & maintenance, I am utilizing small SD Cards to boot the kernel instead of PXE. Trust me life is so much easier with a pack of cheap SD Cards. As they are only used to boot, speed and longevity is not an issue.

Index

Environment

Example IP Structure

Node Name IP Role
1 pnode01 192.168.4.1 Storage Host, NFS Server, DHCP Server
2 pnode02 192.168.4.2 NFS Client
3 pnode03 192.168.4.3 NFS Client
4 pnode04 192.168.4.4 NFS Client
5 pnode05 192.168.4.5 NFS Client
6 pnode06 192.168.4.6 NFS Client
7 pnode07 192.168.4.7 NFS Client

Harware Example

Initial SD Card Setup

Using Raspberry Pi Imager, write 2 SD Cards with Ubuntu Server. 32bit or 64bit will do. 1st SD Card will be to utilize USB storage and the 2nd will be to utilize NFS. We will create an image of the 2nd to make easy duplicates for Nodes 03+.

Setup Node 01 (pnode01)

Insert and boot the first Pi (pnode01). Wait for the Pi to fully start the first time. You will know this because it will display certificate information then ask to login.

Login Username & Password will both be ubuntu. If you try to login with these credentials and they do not work, wait a little more, first boot is not completed.

Now plug in your USB storage drive. USB3 is way better then USB2.

Use the following guide to setup the usb storage as Ext4 with a partition label of 'osstorage' and temporarily mount to /osstorage

Manage Drives

Copy OS to USB Storage.
sudo cp -axv /. /osstorage/.
sudo cp -axv /dev/. /osstorage/dev/.

Now power off the Pi and remove the SD Card
sudo poweroff

Now insert the SD Card of pnode01 into your computer. Open the small visible partition and edit the file 'cmdline.txt'.
Alter root=LABEL=writable to root=LABEL=osstorage
root=LABEL=writable
root=LABEL=osstorage
Now save & eject the SD Card. Now when you boot your Pi (pnode01), it will be working from the USB storage drive.
You can verify this as the green SD indicator light at the rear of the Raspberry Pi will no longer blink after bootup.

IP Addressing Options.

Setup NFS on pnode01

Install required package
sudo apt install nfs-kernel-server

Now create directory structure
sudo mkdir /nfsroot
Create a directory for each node.
sudo mkdir /nfsroot/pnode02
sudo mkdir /nfsroot/pnode03
sudo mkdir /nfsroot/pnode04
sudo mkdir /nfsroot/pnode05
sudo mkdir /nfsroot/pnode06
sudo mkdir /nfsroot/pnode07
Create a shared directory for use by all nodes.
sudo mkdir /nfspublic

Now we configure the NFS Exports
sudo nano /etc/exports
Your config will need to look something like this below.
/nfsroot/pnode02 192.168.4.2(rw,no_root_squash,async,insecure,no_subtree_check) /nfsroot/pnode03 192.168.4.3(rw,no_root_squash,async,insecure,no_subtree_check) /nfsroot/pnode04 192.168.4.4(rw,no_root_squash,async,insecure,no_subtree_check) /nfsroot/pnode05 192.168.4.5(rw,no_root_squash,async,insecure,no_subtree_check) /nfsroot/pnode06 192.168.4.6(rw,no_root_squash,async,insecure,no_subtree_check) /nfsroot/pnode07 192.168.4.7(rw,no_root_squash,async,insecure,no_subtree_check) /nfspublic 192.168.4.*(rw,no_root_squash,async,insecure,no_subtree_check)
Now save and export configuration.
sudo exportfs -rv

Configure first NFS node pnode02

Lets start by unloading a couple of packages that will have a problem with NFS.
sudo apt remove cryptsetup
sudo apt remove mdadm
sudo snap remove lxd

Now we can add NFS support
sudo apt install nfs-common
Next to mount the NFS share that we need to set as OS Storage
sudo mkdir /osroot
sudo mount -t nfs -onolock 192.168.4.1:/nfsroot/pnode02 /osroot
And to copy OS
sudo cp -axv /. /osroot/.
sudo cp -axv /dev/. /osroot/dev/.
Now you can power off and remove the SD Card.
sudo poweroff

Now place the SD Card from pnode02 into your computer.

Using Computer Manager -> Disk Management, you can remove the linux partition. (Do not delete the small boot partition.)

Like the first SD Card, you will need to locate the cmdline.txt file and edit the following.

Replace the content of cmdline.txt with the following.
console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.4.1:/nfsroot/pnode02,vers=3 rw ip=dhcp rootwait elevator=deadline
Now save and close cmdline.txt

Now you can duplicate the boot SD Card as needed. Remember to update the nfsroot= for each SD Card.


Using a utility such as Win32 Disk Imager, create an image of the SD Card. With Win32 Disk Imager, check Read Only Allocated Partitions.

Duplicate first NFS Node

Log into pnode01 (Node with USB Storage) and duplicate the NFS node.
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode03
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode04
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode05
Etc as needed.

Setup Shared Directory /nfspublic

On each NFS node edit the /etc/fstab file and add the following
sudo nano /etc/fstab
192.168.4.1:/nfspublic /nfspublic nfs _netdev,noauto,x-systemd.automount,nolock,soft,intr,nfsvers=3,tcp,bg,rw 0 0
I find this configuration works best to quickly load the NSF resource for services that may utilize the NSF such as Apache.

You should now have all of your Nodes working

You can now configure each node as you desire.


First you should update the hostnames of each node.
sudo nano /etc/hostname

Configure The RTC Module if you have one.

I run the RTC Module on pnode01 and then host a Network Time Service to supply the other nodes.

Edit Boot Settings and add the following to 'usercfg.txt' found on Fat partition of Micro SD Card (You will have to remove the SD Card and edit it on your computer.)
dtoverlay=i2c-rtc,ds1307
Now Sync the module
sudo timedatectl set-local-rtc false
Check if time sync is already active
sudo systemctl is-active systemd-timesyncd
Next create the service
sudo nano /etc/systemd/system/hwclock-sync.service
[Unit] Description=Time Synchronisation from RTC Source After=systemd-modules-load.service RequiresMountsFor=/dev/rtc Conflicts=shutdown.target [Service] Type=oneshot ExecStart=/sbin/hwclock -s TimeoutSec=0 [Install] WantedBy=time-sync.target
Install TimeSyncD
sudo apt install systemd-timesyncd
Enable Service
sudo systemctl enable hwclock-sync systemd-timesyncd
sudo systemctl start hwclock-sync systemd-timesyncd
Check Time
timedatectl

Backup Node

To Backup a node to a single file maintaining permissions use:
sudo tar -cvpzf /nfsroot/pi4_pnode04_20200810.tar.gz /nfsroot/pnode04
To be continued later today. Sorry :), ran out of time.
CAMM Services Application Design
Brantford PC
Norfolk County PC
R.D. Cookson Disposal Ltd.
Web Site Design Brantford, Ontario
Web Site & Application Development Norfolk County, Ontario
Various front loader and roll off bin rentals in Simcoe, Ontario
3D Marketing and Exhibit Specialists, Greater Toronto Area
Robert Jones Marine Technical Services