Read only Raspberry Pi OS to prevent corruption Topic is solved

[ben53252642]

Hi Folks,

As a long time Domoticz user and someone who has setup others with Domoticz running on Raspberry Pi's I have observed a general issue with this setup:

Eventually the SD Card ALWAYS gets corrupted, it's only a matter of time. My father for instance, he had two SD cards become corrupted, eventually he lost faith in Domoticz and now has a Vera Edge running his Z-Wave system.

I would like to see the development of a 100% read only Raspberry Pi SD card image (write can be enabled for updates and then disabled again) and the option of writing the Domoticz database to an external storage (USB stick for example) or external MariaDB server.

This setup also has the advantage of being immune to sudden power loss should the user decide to power cycle the device by removing the power cable.

I know how to make such a OS image, wanting feedback on the idea, particularly in relation to options with the Domoticz DB.

[Egregius]

Isn't it better to provide a good step-by-step guide how to prevent SD card corruption?
And make a list of what are the causes of this?
I mean, mine is running 2 years and 3 months now. Not such a big deal to me if it crashes after that time.

[Logread]

I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive

Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).

[Jan Jansen]

Take a look in:
https://www.raspberrypi.org/forums/view ... 29&t=44177
https://wiki.pieper.eu/raspberry-pi-dra ... -usb-stick (Dutch version)

[freijn]

nice link Jan Jansen

Any idea / clue why he mentions :

Een (verse) installatie van Wheezy Raspbian image

?

Why can't this be done from a already running and installed machine ?

[ben53252642]

Take a look in:
https://www.raspberrypi.org/forums/view ... 29&t=44177
https://wiki.pieper.eu/raspberry-pi-dra ... -usb-stick (Dutch version)

According to this, Raspberry Pi 3 has an option to boot DIRECTLY from a USB device without an SD card needing to be present.

https://www.raspberrypi.org/documentati ... des/msd.md

As far as I'm concerned this development basically solves the problem. I'd just get a couple of cheap 32gb SSD's in USB cases and put them in a mdadm mirror raid.

[EddyG]

The best solution is: get a NAS which supports LUN's
Look at http://www.berryterminal.com/doku.php/berryboot and you have a Raspberry Pi that boots (readonly) on the SD-card and runs the OS from the LUN. That 'drive' is mostly a lot faster then any SD-card, so you get better performance.

[Jan Jansen]

The best solution is: get a NAS which supports LUN's

That is right, but in that case the NAS must be oneline 24/7. I use the USB solution and have not had any problems for a long time.

Jan

[EddyG]

24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.

[ben53252642]

24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.

In my environment I have a Synology NAS with RAID 6 which can do LUNs, didn't know about the option to boot from a LUN.

Even found a guide on how to do it!
http://www.berryterminal.com/doku.php/s ... sing_iscsi

Combined with watchdog and monit setup, Indeed this would appear to be a very good option particularly with the ability to schedule automatic snapshots.

[niceandeasy]

Mine runs its / partition from a Synology using NFS. /boot is only used read-only, during booting the Raspberry.
Rasbian and Domoticz 'survive' Synology DSM upgrades. Everything just freezes during DSM upgrade. But if the ethernet connection drops, the Raspberry will need to be rebooted. Maybe because it is configured using DHCP?
Anyway, I find it to be very, very reliable.

[pvm]

Mind automatic updates of your NAS and the related reboots of your NAS, this gave me issues in Domoticz not being available (synology nas). I changed back to new SD card and ramdrive as mentioned a few posts before this one

[Egregius]

Booting from NAS is rather pointless, than you could also run domoticz directly on the nas. Running on my nas is not a good option here as it consumes 65W, calculate the cost of that... Besides that I also tried RPi with NFS on nas but that gave a lot more problems.

I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive

Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).

I do something similar:

/etc/fstab: You may easily set a larger memory size, memory will only be occupied depending on the file sizes.

Code: Select all

tmpfs   /temp        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0
tmpfs   /var/log        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0

sudo crontab -e

Code: Select all

@reboot /home/pi/atstartup.sh >/dev/null 2>&1
0 * * * * /home/pi/cronhourly.sh >/dev/null 2>&1
1 0 * * * /home/pi/crondaily.sh >/dev/null 2>&1
2 0 * * 1 /home/pi/cronweekly.sh >/dev/null 2>&1

/home/pi/atstartup.sh: make sure some folders are writable and backups are restored to tmpfs drive

Code: Select all

#!/bin/bash
echo "performance" | tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
chmod 777 /var/log
touch /var/log/domoticz.log
chmod 666 /var/log/domoticz.log
cp /home/pi/zwcfg_0xe9238f6e.xml /var/log/zwcfg_0xe9238f6e.xml
cp /home/pi/domoticz.db /var/log/domoticz.db
cp /home/pi/options.xml /home/pi/domoticz/Config/options.xml
ln -sf /var/log/domoticz.db /home/pi/domoticz
chmod 666 /var/log/zwcfg_0xe9238f6e.xml
echo 0 | tee /sys/class/leds/led0/brightness
echo 0 | tee /sys/class/leds/led1/brightness
/opt/vc/bin/tvservice -o
sudo service domoticz.sh start

/home/pi/crondaily.sh: make a daily backup of the tmpfs files

Code: Select all

#!/bin/bash
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
cp /var/log/domoticz.db /home/pi/domoticz.db

/home/pi/cronweekly.sh: cleanup the logfiles

Code: Select all

#!/bin/bash
sudo rm /var/log/*.gz
sudo rm /var/log/*.1
sudo rm /var/log/SBFSPOT/*
sudo truncate -s 0 /var/log/*
sudo truncate -s 0 /var/log/lighttpd/*

/home/pi/options.xml: keep some zwave settings and set /var/log as log folder

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<!-- To be effective, this file should be placed in the user data folder specified in the Options::Create method -->
<Options xmlns='http://code.google.com/p/open-zwave/'>
  <Option name="logging" value="true" />
  <Option name="UserPath" value="/var/log/" />
  <Option name="Associate" value="true" />
  <Option name="NotifyTransactions" value="false" />
  <Option name="DriverMaxAttempts" value="5" />
  <Option name="SaveConfiguration" value="true" />
  <Option name="RetryTimeout" value="5000" /><!-- Default 40000 --><!-- OK: 15000 -->
  <Option name="PerformReturnRoutes" value="true" /><!-- added -->
  <Option name="AssumeAwake" value="false" /><!-- added -->
  <Option name="SaveLogLevel" value="3" /><!-- added -->
  <Option name="QueueLogLevel" value="3" /><!-- added -->
  <Option name="DumpTriggerLevel" value="3" /><!-- added -->
  <Option name="NetworkKey" value="0x09, 0x05, 0x03, 0x04, 0x01, 0x06, 0x07, 0x01, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10" />
  <Option name="RefreshAllUserCodes" value="false" />
  <Option name="ThreadTerminateTimeout" value="5000" /><!-- 5000 -->
</Options>

And finally in /etc/init.d/domoticz.sh: make a backup of the database and zwave config while stopping domoticz.

Code: Select all

do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
	cp /var/log/domoticz.db /home/pi/domoticz.db
	cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
        return "$RETVAL"
}

I'm pretty sure that with this you avoid most of the writes to the sd card.
There's one big downside of it: if you cut the power you'll lose data of that day. In my case I don't care about that. The temperatures I care about are pushed to a MySQL database on a VPS. As long as you can send the shutdown command there's nothing to worry about.

[EddyG]

Booting from NAS is rather pointless, than you could also run domoticz directly on the nas.

Not always. I have a RaZberry connected and positioned in the middle of the house. Also have a PiAware in the top of the house for better reception.
Also a P1 connection on a slave Domoticz for metering my Power consumption at the ground floor.

[Number8]

I have been running one master Domoticz maybe for 3 years now and a slave Domoticz for 2 years with no issues so far. Temporary files are stored on an external USB key. However, in case of a failure, despite backups I feel that restoring a functioning new system will a PITA especially because, if I understand well, we have to reinstall the exact same domoticz version to make sure the database will be compatible (I'm running under beta). I haven't seen so far a repository of all beta version.

The best solution is: get a NAS which supports LUN's
Look at http://www.berryterminal.com/doku.php/berryboot and you have a Raspberry Pi that boots (readonly) on the SD-card and runs the OS from the LUN. That 'drive' is mostly a lot faster then any SD-card, so you get better performance.

So I'm very happy to discover here the berryterminal solution, since I have a Synology NAS running 24x7. It will be one of my summer tasks.

Booting from NAS is rather pointless, than you could also run domoticz directly on the nas.

Certainly not, the current RaspPI locations are just incompatible with the NAS. And I will add a new one in the well, shall I put the NAS in it?

[hansrune]

Have used overlayfs as described on https://www.domoticz.com/wiki/Setting_u ... spberry_Pi to protect the setup. Recommend to add a small UPS

[Number8]

Have used overlayfs as described on https://www.domoticz.com/wiki/Setting_u ... spberry_Pi to protect the setup. Recommend to add a small UPS

As far as I'm concerned all RaspPI, routers, switches etc. are powered by a UPS.

[EdddieN]

You could always use one of these with a sata drive? https://www.element14.com/community/doc ... desktop-pc

[ben53252642]

My own Domoticz setup runs inside a KVM virtual machine with a BTRFS filesystem that I can backup and restore in seconds. I have the USB ports mapped through from the host, it's fantastic because I have completely flexibility to upgrade or change the physical hardware.

I've also run Domoticz on Raspberry Pi though and as mentioned setup Domoticz for others running on Raspberry Pi. There are a lot of good suggestions here for making the setup more reliable.

[dervogt]

I have migrated my root filesystem to an external USB drive with 500GB once my setup was up and running. This avoids the issues with symlinks or other small tricks you can do to "save" your SDCard. As the BOOT filesystem practically is read only and once the pi has booted into the OS, the SDCard practically is out of the game.

I have not gone the extra step of switching over to also boot from the USB drive, I personally am happy with the SDCard boot and then having my external HDD takeover the rest.