[TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

[Siewert308SW]

After 2 corrupt sd-cards i have found an alternative until MSD bootmode is final for the Pi3
At the moment MSD is only available in the branch=next firmware.
Which gave me issues as BT was going down all the time.
For me this solution works great and still able to run /root from my external HDD.
It's not my solution, as it is existing already i only gave it my twist.

Downside you still need an SD card to boot.
The advantage is that all data is written to your external HDD.
And it saves me a third upcoming corrupt sd card

Right now I use a "Seagate Expansion Portable 1TB" without USB hub.
Looks like the Pi3 with stock 5v 2.5A provides enough power to feed this disk.
I had to enable max_usb_current=1 so instead of 0.6 - 1.2A is sent through the USB port.
Which is only available on the Pi2+ and Pi3 as far as i know.

Below you'll find a description as how I configured it.
Assume that you have some expertise on Raspberry as i can't describe all the details

-------------------------------------------------- ------------

Step 1: Backing up or new SD Card stabbing
Make a backup of your existing setup.
Or in my case i used another 2gb sd card.

Step 2: Jessie download and transfer
Download Jessie Lite image in case of a headless set-up and drop it if you use Windows on your SD card using win32diskmanager.
Insert the SD card into your Pi and Pi on the boat.

Step 3: Pi Configuring
Please log in via ssh and configure the Pi first

sudo Raspi-config

- Expand filesystem and reboot

Login again and configure the rest
- Set locales
- Set Time Zone
- Set new password
- Ect ect

reboot again

Step 4: Static IP for Pi
If you want to have a static IP you want to Pi follow this step.
Otherwise, you can skip this step.

Log back in via ssh:

sudo nano /etc/dhcpcd.conf

The following static IP settings are focused on my situation.
Find out what your gateway ect and adjust if necessary.

Code: Select all

interface eth0
static ip_address=192.168.178.110/24
static routers=192.168.178.1
static domain_name_servers=8.8.8.8

Copy /Past the above setting in dhcpcd, conf
Finish with ctrl o followed by enter.
And to exit ctrl x
Reboot the Pi to let these setting take effect.

Step 5: Max USB current
Now we go to if you are going to provide your USB ports more Amps incase of a unpowered external HDD.
Note that max_usb_current=1 is only available for Pi2 + and Pi3 as far as I know
Log back in via SSH.

sudo nano /boot/config.txt

Copy /Past the following line at the bottom in the config.txt

Code: Select all

max_usb_current=1

Finish with ctrl o followed by enter.
To exit ctrl x
Reboot the Pi to let this setting take effect.

Step 6: OS update and install required packages
Now we will update your OS and already install some needed packages.
Log back in via SSH.

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo apt-get install rsync

Optionally, if you want to use the beta kernel:

sudo apt-get install RPI update -y
sudo RPI update

installed everything? Reboot your Pi again

Step 7: Prepping your External Disk
We are going to prepare the external drive so that it can carry the /root.

In SSH enter 'sudo blkid "and take notice of what you see there.
It will be easier later on to find your just inserted external HDD.

Now insert the external HDD and enter again "sudo blkid"
And see which disk just has been added.
In my example, I take "sda" in my examples below.

We will now give the HDD a full ext4 partition.

in SSH: sudo parted /dev/sda
With the above command you start parted and all processing is now done on your external HDD

In parted do the following:

mktable msdos

Here you get a message that all data will be removed from your HDD.
Agree if you want to proceed.
--------------
Then:

mkpart primary ext4 0% 100%

It creates an ext4 partition along the entire length of your external HDD
--------------

Now you can review using the command "print"
Will be something like:

Code: Select all

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary  ext4

--------------

To exit parted you enter the command "quit"
Only thing remaining is to tune the ext4 partition.
You do this by running the following command in SSH

sudo mkfs.ext4 /dev/sda1

Step 8: /root transfer to the external drive
Now we are ready to copy /root from sd-card to your external HDD partition.
In SHH, run it out below.

sudo mkdir /mnt/usbdrive
sudo mount /dev/sda1 /mnt/usbdrive/
sudo rsync -ax --progress / /mnt/usbdrive

May take some time before all the files are transferred

Step 9: Renew Certificates
Now we are going to renew the certificates on the HDD.
Otherwise you will no longer be able to log in via SSH.
In SSH, run the following commands:

cd /mnt/usbdrive
sudo mount --bind /dev dev
sudo mount --bind /sys sys
sudo mount --bind /proc proc
sudo chroot /mnt/usbdrive

rm /etc/ssh/ssh_host*
dpkg-reconfigure openssh-server
exit

sudo umount dev
sudo umount sys
sudo umount proc

Step 10: cmdline and fstab adjust
We are almost there.
We are going to adjust the boot cmdline and fstab to let your Pi know to seek on your external HDD.

Perform "sudo blkid" and locate your external HDD.
There will be something along the lines of:

Code: Select all

/dev/sda1: UUID="eda23afa-a07c-4ca5-8b00-f14b6f09095e" TYPE="ext4" PARTUUID="5905022c-01"

Remember the UUID hash and PARTUUID hash

The UUID hash is the name of your disk and PARTUUID hash your partition.
Normally, you could refer to the /root for example to /dev/sda
But we do not want to because if you ever going to use a USB stick later on it may screw everything up.
Resulting in a non booting Pi as it can't find /dev/sda because your OS has given it to your stick.
Instead, we use UUID and PARTUUID so no matter what you put into the Pi it will find your HDD using the hash

SSH in the following commands:

sudo sed -i "s,root=/dev/mmcblk0p2,root=PARTUUID=5905022c-01," /boot/cmdline.txt
sudo sed -i "s,/dev/mmcblk0p2,UUID=eda23afa-a07c-4ca5-8b00-f14b6f09095e," /etc/fstab

Now you can optionally verify cmdline.txt and fstab if the above has been applied.
Should look like the following:

- /boot/cmdline.txt

Code: Select all

dwc_otg.lpm_enable=0 console=tty1 root=PARTUUID=5905022c-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

- /etc/fstab

Code: Select all

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
UUID=eda23afa-a07c-4ca5-8b00-f14b6f09095e  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

Step 11: Almost done! [/ b]
Now we are ready and the external HDD contains everything from your sd-card /root.
If you've done everything properly your Pi will now start from the SD card and run from your external HDD.

SSH for now following commands.

cd ~
sudo umount /mnt/usbdrive
sudo rmdir /mnt/usbdrive
sudo reboot

If everything goes well then your Pi now reboots from the SD card and is running from your external HDD.

Step 12: Optional remove /root partition from your SD card [/ b]
You can now optionally remove /root partition from your SD card as we no longer need this.
SSH

sudo parted /dev/mmcblk0
rm 2 (2 stands for partition 2 verify it first using "print")
print
quit
sudo reboot

Step 13: Optional Domoticz install [/ b]
If want to have a fresh Domoticz installation.
Then you can SSH using "sudo curl -L install.domoticz.com | bash"
But you did above tut with your production Domoticz sd card then it should have been started already.

Last words [/ b]
It only remains me to say that hopefully someone overhere has a benefith by using this tut.
I'm still a noob in raspberry field and not so good it writing explanations.
Google was my best friend and using various found tuts on how to mount USB sticks, MSD mode to combine Pi3 to above tut.

Goodluck

[Justintime]

Thanks for the Tutorial. Ordered a 8gb SSD.
A corrupted SD card has never happened to me. But always in my mind it can.

[lost]

Another solution would be to use industrial grade SD (or PI foundation to make a on board connector for eUSB=USB keys made for embedded applications) like these:
http://www.smartm.com/salesLiterature/r ... erview.pdf

Other manufacturers maybe be virtium/micron/swissbit, they are all on the embedded storage market.

From previous document you can see a huge difference in TBW per GB capacity, between SLC and MLC based devices.

To sum-up, single level cell flash memory only store 1 state per cell = charge loaded or empty = 0/1 ; multi level ones makes uses of a comparator to be able to make difference between several levels of cell charge load, thus able to store several bits in a cell. If you have a 4 level comparator you can discriminate 00/01/10/11, thus store 2 bits in one place.

But of course, because of the erase (sector based) before write (even a single bit in a sector!) nature of flash memory, MLC in not so good in TBW: More need to erase (because more bits in a single cell), more complex management, cell wearing = more charge drain over storage time = more need to rewrite periodically in the background...

=> SLC given for 6TBW per GB capacity ; MLC for 0.2! So 30 times less!

This means that for a 16GB SD, even industrial grade, the manufacturer says lifetime writable amount of data is 96TB for SLC based devices and 3.2TB for MLC based ones.

At the moment, only setting /tmp and /var/tmp as a ram tmpfs, I have around 0.8GB written per day reported by ext4 session stats. So I would expect 4096 days for a MLC based device = 11 years.

But this is also dependant on write profiles (for very small files, this may be much less ; For big files, you can expect to reach manufacturer values).

And how do consumer grade devices compares? All are for sure MLC based and TBW are rarely published. Firmwares may also use less sophisticated wear leveling techniques. But as TBW is always related to capacity, using a 32GB device if you only need a 8GB one will multiply lifetime by 4 between 2 devices based on the same flash underlying memory technology.

Anyway, if there is a way to buy industrial grade SLC based devices designed for embedded market, that would be for sure a nice solution to this problem. Otherwise, my advice is to buy consumer grade devices from good brands and much larger than your needs.

Farnell sells swissbit products, S450 series are SLC based and S45 are MLC.
A 8GB S450 costs 108€, same price as a 64GB S45. So bit cost between SLC and MLC is multiplied by 8!

A 16GB S45 costs 39€. But underlying flash is specified for 3k erase cycles while S450 it's 100k, 33 times more! Close to the 30 found from comparing Smart devices TBW hereupper.

S45 series may indeed be a good cost/durability compromise for home use as sophisticated firmware will allow the device to wear evenly. Don't expect this from consumer grade devices that may wear very quickly on heavy used "hot spots".

See hereunder 16 and 32GB versions:
http://fr.farnell.com/swissbit/sfsd016g ... dp/2667833
http://fr.farnell.com/swissbit/sfsd032g ... dp/2667834

S46 series should also come soon: That's pSLC = MLC that can be used as SLC, specified at 20k erase cycles so fitting in between on the endurance side but using MLC chips that cost less. They should sell at the double bit cost compared to MLC/S45 when they arrive on market. So a 16GB should be in the 50/60€ price range with a nice durability.

That's the price of confidence!

[EdwinK]

[deleted]

[EdwinK]

Tried this, and it looks like all works.

But a bit hesitant about step 12. What is the root partition

Model: SD USD (sd/mmc)
Disk /dev/mmcblk0: 15.8GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 4194kB 48.0MB 43.8MB primary fat32 lba
2 48.2MB 15.8GB 15.8GB primary ext4

[EdwinK]

And now:

GNU Parted 3.2
Using /dev/sda1
Welcome to GNU Parted! Type 'help' to view a list of commands.
mkpart primary ext4 0% 100%
Error: Partition(s) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57,
58, 59, 60, 61, 62, 63, 64 on /dev/sda1 have been written, but we have been
unable to inform the kernel of the change, probably because it/they are in use.
As a result, the old partition(s) will remain in use. You should reboot now
before making further changes.
Ignore/Cancel?

[rebooted Pi]

(parted) print
Model: Unknown (unknown)
Disk /dev/sda1: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags

(parted)

[tgg3]

Thanks much for this tutorial!

I just used this tutorial and it worked great for me with one exception. The sed command for /etc/fstab didn't work, so I just edited the file by hand. It might be best to show the desired cmdline.txt and fstab files and edit them by hand.

[kimhav]

After 2 corrupt sd-cards i have found an alternative until MSD bootmode is final for the Pi3

Lots of steps. The far easiest it really to utilise the NOOBS Lite installation (instructions here) which will boot the system to GUI installer where you choose to install on the disk and you're up and running within 15 minutes. After the initial installation you just to the normal tweaking of Rpi and Domoticz.

[JuGuSm]

Thank you very much for this tutorial. It works perfectly for me but I didn't have the same original cmdline.txt and fstab file as you. But no problem I have been able to adapt it to my case.

Also, it would be perfect if it should be possible to boot directly on the external drive and completely remove the SD card... any suggestion?

[Siewert308SW]

Thank you very much for this tutorial. It works perfectly for me but I didn't have the same original cmdline.txt and fstab file as you. But no problem I have been able to adapt it to my case.

Also, it would be perfect if it should be possible to boot directly on the external drive and completely remove the SD card... any suggestion?

Ofcourse that's possible but you'll need additional software like berryboot if i'm right.
Further more you can take a look here but be carefull and read it carefully.
But for me personally ik like it this way.
If something happens with OS i can amend the cmdline.txt and instantly boot back up from the sdcard.
As for the boot on the SD, it aint written only read so a corrupted SD is out of the question.
Running and the same sdcard and hdd since 2016.

[EdwinK]

But for me personally ik like it this way.

I did too. Had it working, but then something happend, and I lost access to the harddrive. Going to try it once again one day.

[Siewert308SW]

But for me personally ik like it this way.

I did too. Had it working, but then something happend, and I lost access to the harddrive. Going to try it once again one day.

Strange... Only thing i can think of is that you didn't use the UUID but folder like /dev/sda in cmdline which can change and causing a hdd lost.
Otherwise i can't think of anything else.
Upcoming weekend i will try (if time) review the OP and update where it's needed.
Maybe a bash script to automate the setup will be written soon.

[EdwinK]

Yes, I think that was the problem, but it was some time ago, so I can't remember it I would welcome a setup script.

[brain]

hi i now have a read only system and i can't change anything, does anyone know what to do
thanx