Boot from iSCSI / LUN

[EddyG]

Below a description how to boot from iSCSI on a Synology NAS or other NAS with iSCSI/LUN.
THIS IS EXPERIMENTAL!!! If you do not exactly know what is being explained then STOP.
There might be typo's. I will fix them here when I find them.
Everything is based on a fixed IPaddress for the Raspberry Pi, make sure that you have ssh access.

Step 1. Take a normal SD-card and put the latest Raspbian Buster Lite on it.
Update and upgrade, you should be on: Linux raspberry 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux
or higher, currently (25-7-2019) on Linux raspberry 4.19.58-v7+ #1245 SMP Fri Jul 12 17:25:51 BST 2019 armv7l GNU/Linux
Step 2. Create a iSCSI target and LUN on your Synology. Mark your IQN. I made a LUN of 16Gb.
Step 3. Install iSCSI on the Raspberry Pi, make sure it is running and discover the Target. Try to login and find the device with dmesg.
Create an empty file iscsi.initramfs. Actually there are 2 ways: One is empty file and everything concerning iSCSI on the commandline in /boot/cmdline.txt or Second, everything concerning iSCSI in this file and not in the /boot/cmdline.txt. I choose the first option (empty file).

Code: Select all

sudo apt-get install open-iscsi
sudo systemctl restart iscsi.service
sudo iscsiadm -m discovery -t sendtargets -p <IP of Synology>
sudo iscsiadm --portal <IP of Synology> -T <IQN of Target on Synology> --mode node --login
dmesg
sudo touch /etc/iscsi/iscsi.initramfs

Step 4. Suppose you found with dmesg that the device is "/dev/sdb". I call it /dev/sdX . Give that device a filesystem ext4, and mark the UUID. Please be careful that you have the right device. Mount that device.

Code: Select all

sudo mkfs.ext4 -m0 /dev/sdX
sudo blkid /dev/sdX   ---->   Mark this /dev/sdX UUID
sudo mkdir /mnt/iscsi
sudo mount /dev/sdX /mnt/iscsi

Step 5. Copy the filesystem from the SD-card to the new LUN, and skip a view dirs. I mount my USB sticks on /media so I skip that one too. Wait some time until everything is copied. Then make some empty dirs.

Code: Select all

sudo rsync -avhP --exclude /boot --exclude /proc --exclude /run --exclude /sys --exclude /mnt --exclude /media / /mnt/iscsi/
sudo mkdir /mnt/iscsi/{proc,run,sys,boot,mnt,media}

Step 6. Goto /boot and copy some files and place the script below also on /boot. This is needed to switch between normal SD-card boot and LUN boot.

Code: Select all

sudo cp /boot/cmdline.txt /boot/cmdline.txt.org
sudo cp /boot/cmdline.txt /boot/cmdline.txt.lun
sudo cp /boot/config.txt /boot/config.txt.org
sudo cp /boot/config.txt /boot/config.txt.lun

Script: nano /boot/switch_boot_setup.sh and make the file executable.

Spoiler: show

#!/bin/bash

if [[ ! $(whoami) =~ "root" ]]; then
echo "**********************************"
echo "*** This should be run as root ***"
echo "**********************************"
exit 1
fi

function ask_yes_or_no() {
read -p "$1 ([y]es or [N]o): "
case $(echo $REPLY | tr '[A-Z]' '[a-z]') in
y|yes) echo "yes" ;;
*) echo "no" ;;
esac
}

if [ "$1" = "lun" ]; then
echo "Switch to 'lun' boot setup."
sudo cp /boot/cmdline.txt.lun /boot/cmdline.txt
sudo cp /boot/config.txt.lun /boot/config.txt

if [[ "yes" == $(ask_yes_or_no "Would you like to sync files first?") ]]; then
echo "Wait...."
sudo /root/scripts/sync_sd-card_2_lun.sh
else
echo "File Synchronisation Skipped."
fi
if [[ "yes" == $(ask_yes_or_no "REBOOT now?") ]]; then
if [[ "yes" == $(ask_yes_or_no "Are you *really* sure?") ]]; then
sudo reboot
exit 0
else
echo "Reboot Skipped."
fi
else
echo "Reboot Skipped."
fi

elif [ "$1" = "org" ]; then
echo "Switch to 'org' boot setup."
sudo cp /boot/cmdline.txt.org /boot/cmdline.txt
sudo cp /boot/config.txt.org /boot/config.txt

if [[ "yes" == $(ask_yes_or_no "Would you like to sync files first?") ]]; then
echo "Wait...."
sudo /root/scripts/sync_lun_2_sd-card.sh
else
echo "File Synchronisation Skipped."
fi
if [[ "yes" == $(ask_yes_or_no "REBOOT now?") ]]; then
if [[ "yes" == $(ask_yes_or_no "Are you *really* sure?") ]]; then
sudo reboot
exit 0
else
echo "Reboot Skipped."
fi
else
echo "Reboot Skipped."
fi

else
echo "Parameter should be 'lun' or 'org'."
echo "e.g. /boot/switch_boot_setup.sh lun"
fi

Step 7. Leave the *.org files untouched and edit the *.lun files and create an initramfs file (takes a while). The redirect is for looking what is inside the initramfs file. A file /boot/initrd.img-4.19.yy-xxx should be created. Mark that filename.
You might have to install initramfs-tools with:

Code: Select all

sudo apt-get install initramfs-tools

Code: Select all

cd /boot
sudo update-initramfs -v -k `uname -r` -c > /home/pi/initramfs.txt

Edit config.txt.lun and add at the end of the file with the name of the file just created:

Code: Select all

initramfs initrd.img-4.19.yy-xxx followkernel

Find the INITIATOR iqn from /etc/iscsi/initiatorname.iscsi
Edit cmdline.txt.lun. Replace the complete content with (everything on 1 line!!!):

Code: Select all

dwc_otg.lpm_enable=0 ip=<Raspberry IPaddress>:<Synology IPaddress>:<Gateway IP>:255.255.255.0:<Raspberry hostname>:eth0:off
ISCSI_INITIATOR=<IQN from /etc/iscsi/initiatorname.iscsi>
ISCSI_TARGET_NAME=<IQN from Synology> ISCSI_TARGET_IP=<Synology IPaddress>
ISCSI_TARGET_PORT=3260 ISCSI_TARGET_GROUP=1 rw rootfs=ext4 root=UUID=<UUID from the /dev/sdX device>
fsck.repair=yes elevator=deadline rootwait panic=15

Step 8. Fix fstab on the LUN device. Remove the line with mmcblk0p2 (not the mmcblk0p1). If that line is not present, delete the line with

Code: Select all

PARTUUID=xxxxxxxxxx  /               ext4    defaults,noatime  0       1

Most important is that there should be no line with a root (/) mount, because that is been replace with the line below.
Add a new line for the /dev/sdX device. Edit: nano /mnt/iscsi/etc/fstab

Code: Select all

UUID=<UUID from the /dev/sdX device>       /       ext4    defaults        1       1

Step 9. Execute and reboot:

Code: Select all

sudo /boot/switch_boot_setup.sh lun
sudo reboot

Step 10. After the reboot you should be on the LUN. Verify with:

Code: Select all

sudo df -H

There should be a line like:
/dev/sdX         17G  2.2G   15G  14% /

If you want to go back to the original SD-card boot, execute:

Code: Select all

sudo /boot/switch_boot_setup.sh org
sudo reboot

I hope I did not miss something, else this post will be edited.
If everything is stable I will post some scripts to keep the LUN and SD-card synchronized (almost done).

Enjoy.

Step 11.
Normal upgrading op packages can be done the normal way followed by sync of the files with the appropriate script.
When needed here is a logrotate file for both sd-card and lun the same file:

Code: Select all

sudo nano /etc/logrotate.d/sync_lun_and_sd-card

Content:

Code: Select all

/var/log/sync_lun_2_sd-card.log {
        rotate 1
        weekly
        compress
        missingok
}
/var/log/sync_sd-card_2_lun.log {
        rotate 1
        weekly
        compress
        missingok
}

Step 12. Upgrading the kernel.
Make sure that switching back and forth between lun and org works fine.
Also make sure that synchronising between lun and org is working fine too. There should be no errors.
Now switch back to org, synchronize and reboot to org.

Code: Select all

sudo /boot/switch_boot_setup.sh org

If you like, make a exact copy of your SD-card and make a snaphot of the LUN, just in case something goes wrong.
Now issue the following commands in exactly this order.

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo reboot

Ignore possible errors about 'update-initramfs', those will be handled after the reboot.
Once up again issue this command

Code: Select all

cd /boot
sudo update-initramfs -v -k `uname -r` -c > /home/pi/initramfs.txt

Mark the new 'initrd.img-4.xx.yy-zz' filename. Edit the last line of config.txt.lun with this new filename and synchronize the files to lun.

Code: Select all

sudo /root/scripts/sync_sd-card_2_lun.sh

Now switch back to the lun and reboot.

Code: Select all

sudo /boot/switch_boot_setup.sh lun

Check after the reboot if you are on the new kernel with.

Code: Select all

uname -a

Remove the OLDER 'initrd.img-4.xx.yy-zz' file from /boot

[waaren]

Below a description how to boot from iSCSI on a Synology NAS or other NAS with iSCSI/LUN.
THIS IS EXPERIMENTAL!!! If you do not exactly know what is being explained then STOP.
Enjoy.

Thanks !! Will test on a raspberry 4 ([EDIT]when my new one arrives because first one broke down ) and get back with the results

[EddyG]

The synchronisation scripts. The scripts are being placed in /root/scripts and started by cron or manual (as root). Most scriptnames start with sync* for a reason. For the scripts is bc used for timing, you can install it or edit the scripts.

Code: Select all

sudo apt-get install bc

First the setup for the script in the LUN environment.
sudo nano /root/scripts/sync_lun_2_sd-card.sh

Spoiler: show

#!/bin/bash

SCRIPTNAME=`basename "$0"`
SCRIPTNAME=${SCRIPTNAME%.*}
LOGFILE=/var/log/$SCRIPTNAME.log
ERRORLOG=/var/log/$SCRIPTNAME.log.ERROR
#PARAM="-a --update --verbose --delete --dry-run"
PARAM="-a --update --verbose --delete"
EXCLUDEFILE="/root/scripts/lun_sd-card_exclude"
HOSTNAME=$(hostname)
WRONGMOUNT="/dev/sdb"
MOUNTDEVICE="/dev/mmcblk0p2"
MOUNTPOINT="/mnt/$SCRIPTNAME"
SOURCE="/"

#=== End of parameters ============================

# Start Timestamp
STARTTIME=$(date +"%s.%N")

# Slow down the process to avoid to much cpu cycles:
#PID=$$
#/usr/bin/cpulimit --pid $PID --background --monitor-forks --limit 10 --lazy --quiet > /dev/null 2>&1

echo -n "LUN backup start at " >> $LOGFILE
echo $(date) >> $LOGFILE

if [ ! -f "$EXCLUDEFILE" ]; then
echo -n "$EXCLUDEFILE is NOT present. - " >> $ERRORLOG
date >> $ERRORLOG
exit 9
fi

if ! grep -qs $WRONGMOUNT /proc/mounts ; then
echo "Wrong script!!! Copy from LUN to SD-card???"
echo -n "Wrong script!!! Copy from LUN to SD-card??? - " >> $ERRORLOG
date >> $ERRORLOG
exit 8
fi

if [ ! -d $MOUNTPOINT ] ; then
mkdir $MOUNTPOINT
fi
if grep -qs $MOUNTDEVICE /proc/mounts ; then
echo "$MOUNTPOINT already mounted! UNMOUNT first!!!"
echo -n "$MOUNTPOINT already mounted! UNMOUNT first!!! - " >> $ERRORLOG
date >> $ERRORLOG
exit 7
else
sudo mount $MOUNTDEVICE $MOUNTPOINT
fi

if grep -qs $MOUNTPOINT /proc/mounts ; then
rsync $PARAM --exclude-from $EXCLUDEFILE $SOURCE $MOUNTPOINT >> $LOGFILE
echo -n "Synced with SD-card at " >> $LOGFILE
date >> $LOGFILE
else
echo -n "SD-card is NOT mounted. - " >> $ERRORLOG
date >> $ERRORLOG
fi

sudo umount $MOUNTPOINT

# End timestamp
ENDTIME=$(date +"%s.%N")

# Convert nanoseconds to milliseconds
# crudely by taking first 3 decimal places
TIMEDIFF=`echo "$ENDTIME - $STARTTIME" | bc | awk -F"." '{print $1"."substr($2,1,3)}'`
MIN=`echo "$TIMEDIFF / 60" | bc`
SEC=`echo "$TIMEDIFF % 60" | bc`

echo "$MIN minutes and $SEC seconds elapsed." >> $LOGFILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOGFILE

exit 0

File for both LUN and SD-card boot: sudo nano /root/scripts/lun_sd-card_exclude

Spoiler: show

- *~
- *.swp
- *.tmp
- *.temp
- *.bak
- *.pid
- *.sock
- /boot/
- /etc/fstab
- /etc/cron.d/sync*
- /lost+found
- /media/
- /mnt/
- /proc/
- /root/scripts/sync*
- /run/
- /sys/
- /tmp/
- /var/lock/
- /var/run/
- /var/tmp/
- /var/swap
- /var/log/sync*
- /var/lib/samba/private/

File for both LUN and SD-card boot: sudo nano /etc/logrotate.d/sync_lun_2_sd-card

Spoiler: show

/var/log/sync_lun_2_sd-card.log {
rotate 1
weekly
compress
missingok
}

Script for the SD-card enviroment: sudo nano /root/scripts/sync_sd-card_2_lun.sh
Before you run this script change the <IQN from Synology> to the correct IQN!

Spoiler: show

#!/bin/bash

SCRIPTNAME=`basename "$0"`
SCRIPTNAME=${SCRIPTNAME%.*}
LOGFILE=/var/log/$SCRIPTNAME.log
ERRORLOG=/var/log/$SCRIPTNAME.log.ERROR
#PARAM="-a --update --verbose --delete --dry-run"
PARAM="-a --update --verbose --delete"
EXCLUDEFILE="/root/scripts/lun_sd-card_exclude"
HOSTNAME=$(hostname)
REMOTEHOST="192.168.2.248"
WRONGMOUNT="/dev/root"
MOUNTDEVICE="/dev/sdb"
MOUNTPOINT="/mnt/$SCRIPTNAME"
CLIENTIQN="<IQN from Synology>"
SOURCE="/"

#=== End of parameters ============================

# Start Timestamp
STARTTIME=$(date +"%s.%N")

# Slow down the process to avoid to much cpu cycles:
#PID=$$
#/usr/bin/cpulimit --pid $PID --background --monitor-forks --limit 10 --lazy --quiet > /dev/null 2>&1

echo -n "LUN backup start at " >> $LOGFILE
echo $(date) >> $LOGFILE

if [ ! -f "$EXCLUDEFILE" ]; then
echo "$EXCLUDEFILE is NOT present. - " >> $ERRORLOG
date >> $ERRORLOG
exit 9
fi

if ! grep -qs $WRONGMOUNT /proc/mounts ; then
echo "Wrong script!!! Copy from LUN to SD-card???"
echo -n "Wrong script!!! Copy from LUN to SD-card??? - " >> $ERRORLOG
date >> $ERRORLOG
exit 8
fi

if [ ! -d $MOUNTPOINT ] ; then
mkdir $MOUNTPOINT
fi
if grep -qs $MOUNTPOINT /proc/mounts ; then
echo "$MOUNTPOINT already mounted! UNMOUNT first!!!"
echo -n "$MOUNTPOINT already mounted! UNMOUNT first!!! - " >> $ERRORLOG
date >> $ERRORLOG
exit 7
else
sudo iscsiadm --portal $REMOTEHOST -T $CLIENTIQN --mode node --login
lastline=$( { /usr/bin/iscsiadm -m session -P3 2>/dev/null || echo 0; } | tail -n 1 )
findstring="State: running"
if [[ "$lastline" == *"$findstring"* ]] ; then
DEVICE=$(echo $lastline | awk '{print $4}')
MOUNTDEVICE="/dev/$DEVICE"
sudo mount $MOUNTDEVICE $MOUNTPOINT
echo "Device $MOUNTDEVICE mounted on $MOUNTPOINT" >> $LOGFILE
else
echo -n "LUN is NOT present. - " >> $ERRORLOG
date >> $ERRORLOG
exit 6
fi
fi

if grep -qs $MOUNTPOINT /proc/mounts ; then
rsync $PARAM --exclude-from $EXCLUDEFILE $SOURCE $MOUNTPOINT >> $LOGFILE
echo -n "Synced with LUN at " >> $LOGFILE
date >> $LOGFILE
else
echo -n "SD-card is NOT mounted." >> $ERRORLOG
date >> $ERRORLOG
fi

sudo umount $MOUNTPOINT
sudo iscsiadm --portal $REMOTEHOST -T $CLIENTIQN --mode node --logout

# End timestamp
ENDTIME=$(date +"%s.%N")

# Convert nanoseconds to milliseconds
# crudely by taking first 3 decimal places
TIMEDIFF=`echo "$ENDTIME - $STARTTIME" | bc | awk -F"." '{print $1"."substr($2,1,3)}'`
MIN=`echo "$TIMEDIFF / 60" | bc`
SEC=`echo "$TIMEDIFF % 60" | bc`

echo "$MIN minutes and $SEC seconds elapsed." >> $LOGFILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOGFILE

exit 0

I use the cron every hour when I develop. When everything is stable I synchronize every day. I my case aprox. 10 Mb.
I use Synology to make a snapshot of the LUN every hour. So if anything goes wrong I am back with a previous version within 2 min.
The logging of milliseconds duration is my default way of timing how long a script runs. This scripts run over 10 seconds so it seems overkill.

[mcmikev]

Thanks for the good explanation !!

Saw one typo . In the script it is cmdline.txt.lun however the file made is cmdline.lun so be sure to correct that one otherwise no boot !!

[mcmikev]

Also change the IQN in the last script to your own IQN. Maybe create and $IQN for and use that in the script?

[mcmikev]

Also get error when running sync lun 2 sd script
probably need bc installed

/root/scripts/sync_lun_2_sd-card.sh: 1: /root/scripts/sync_lun_2_sd-card.sh: bc: not found
/root/scripts/sync_lun_2_sd-card.sh: 1: /root/scripts/sync_lun_2_sd-card.sh: bc: not found
/root/scripts/sync_lun_2_sd-card.sh: 1: /root/scripts/sync_lun_2_sd-card.sh: bc: not found

[mcmikev]

I am almost finished but get error when booted from sd-card and want to sync to lun

I get error :
root@xxx:~/scripts# sh sync_sd-card_2_lun.sh
Wrong script!!! Copy from LUN to SD-card???

Maybe because I have no /dev/mmcblk0p2 in my /proc/mounts
WRONGMOUNT="/dev/mmcblk0p2"

[EddyG]

Thanks. I changed the text/scripts in the above posts.
Yes, bc needs to be installed, or the script could be edit to give just full seconds. But then again bc is always handy.
The correction should be WRONGMOUNT="/dev/root/" At least for default Buster. (Its "none" for Stretch)
It is just a check that you are on the right setup, that check could be deleted.

[mcmikev]

The line 55 from the sync sd to lun keeps giving me an error
First i remove one [ ] bracket from that line, then I get other error

if [[ "$lastline" == *"$findstring"* ]] ; then

error i get
sync_sd-card_2_lun.sh: 55: [: Attached scsi disk sda State: running: unexpected operator

[EddyG]

Strange I do not have that error.
Will look into that.
What if you use:
if [[ $lastline == *$findstring* ]] ; then

[mcmikev]

I run the script as root (sudo su) and then sh sync.sh script. Is that the right way?

[EddyG]

No this should be run with bash. See first line.
Just do a /root/scripts/sync_sd-card_2_lun.sh (as root) or ./sync_sd-card_2_lun.sh when you are in /root/scripts

[mcmikev]

okay got it running now. Nice thank you

The clientIQN line is not the right iqn. This must be the iqn of the synology nas or other iscsi target

[EddyG]

Oeps, will change that. Too hot in the Netherlands today I guess.

[mcmikev]

zeker weten !! Nice day to play with these kind of scripts and solution

[EddyG]

Added upgrade of the kernel in the first post.
A normal upgrade of packages does not need that procedure.
It just needs a synchronize of files directly after an update/upgrade.
I just did an upgrade of the kernel from 4.19.57-v7+ to 4.19.58-v7+ and that worked fine.

[EddyG]

Yesterday I upgraded Buster to Linux 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux
All went fine within 15 minutes.
It is also quite easy to setup a new raspberry. It is simple.
Clone a LUN, change the IQN on the cmdline.txt and the scripts. Also change the IP-address in the cmdline.txt, and boot again.
After the boot change the fixed IP-address in /etc/dhcpd.conf and the hostname in /etc/hostname.
Synchronize with the script and you are done.

[EddyG]

Upgraded Buster to 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019
All went fine, but discovered some typo's and small bugs.
Changed those in the text/code in the first post.

[mcmikev]

had to reinstall my rpi3 and updated to buster while I'm at it.

So new install with Buster. Following the steps but get stuck on this line

sudo update-initramfs -v -k `uname -r` -c > /home/pi/initramfs.txt
That gives an error and seems te be related that there is no update-initramfs on buster???
sudo: update-initramfs: command not found

Any ideas?

[mcmikev]

Found it. You need to install
sudo apt install initramfs-tools