Page 1 of 3
Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 15:58
by ben53252642
Hi Folks,
As a long time Domoticz user and someone who has setup others with Domoticz running on Raspberry Pi's I have observed a general issue with this setup:
Eventually the SD Card ALWAYS gets corrupted, it's only a matter of time. My father for instance, he had two SD cards become corrupted, eventually he lost faith in Domoticz and now has a Vera Edge running his Z-Wave system.
I would like to see the development of a 100% read only Raspberry Pi SD card image (write can be enabled for updates and then disabled again) and the option of writing the Domoticz database to an external storage (USB stick for example) or external MariaDB server.
This setup also has the advantage of being immune to sudden power loss should the user decide to power cycle the device by removing the power cable.
I know how to make such a OS image, wanting feedback on the idea, particularly in relation to options with the Domoticz DB.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 16:17
by Egregius
Isn't it better to provide a good step-by-step guide how to prevent SD card corruption?
And make a list of what are the causes of this?
I mean, mine is running 2 years and 3 months now. Not such a big deal to me if it crashes after that time.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 17:45
by Logread
I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive
Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 19:18
by Jan Jansen
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 19:31
by freijn
nice link Jan Jansen
Any idea / clue why he mentions :
Een (verse) installatie van Wheezy Raspbian image
?
Why can't this be done from a already running and installed machine ?
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 19:33
by ben53252642
According to this, Raspberry Pi 3 has an option to boot DIRECTLY from a USB device without an SD card needing to be present.
https://www.raspberrypi.org/documentati ... des/msd.md
As far as I'm concerned this development basically solves the problem. I'd just get a couple of cheap 32gb SSD's in USB cases and put them in a mdadm mirror raid.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 19:57
by EddyG
The best solution is: get a NAS which supports LUN's
Look at
http://www.berryterminal.com/doku.php/berryboot and you have a Raspberry Pi that boots (readonly) on the SD-card and runs the OS from the LUN. That 'drive' is mostly a lot faster then any SD-card, so you get better performance.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 20:07
by Jan Jansen
EddyG wrote:The best solution is: get a NAS which supports LUN's
That is right, but in that case the NAS must be oneline 24/7. I use the USB solution and have not had any problems for a long time.
Jan
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 20:17
by EddyG
24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 20:21
by ben53252642
EddyG wrote:24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.
In my environment I have a Synology NAS with RAID 6 which can do LUNs, didn't know about the option to boot from a LUN.
Even found a guide on how to do it!
http://www.berryterminal.com/doku.php/s ... sing_iscsi
Combined with watchdog and monit setup, Indeed this would appear to be a very good option particularly with the ability to schedule automatic snapshots.

Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 21:39
by niceandeasy
Mine runs its / partition from a Synology using NFS. /boot is only used read-only, during booting the Raspberry.
Rasbian and Domoticz 'survive' Synology DSM upgrades. Everything just freezes during DSM upgrade. But if the ethernet connection drops, the Raspberry will need to be rebooted. Maybe because it is configured using DHCP?
Anyway, I find it to be very, very reliable.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Tuesday 04 July 2017 21:44
by pvm
Mind automatic updates of your NAS and the related reboots of your NAS, this gave me issues in Domoticz not being available (synology nas). I changed back to new SD card and ramdrive as mentioned a few posts before this one
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Wednesday 05 July 2017 6:29
by Egregius
Booting from NAS is rather pointless, than you could also run domoticz directly on the nas. Running on my nas is not a good option here as it consumes 65W, calculate the cost of that... Besides that I also tried RPi with NFS on nas but that gave a lot more problems.
Logread wrote:I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive
Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).
I do something similar:
/etc/fstab: You may easily set a larger memory size, memory will only be occupied depending on the file sizes.
Code: Select all
tmpfs /temp tmpfs defaults,noatime,nosuid,mode=777,size=300m 0 0
tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=777,size=300m 0 0
sudo crontab -e
Code: Select all
@reboot /home/pi/atstartup.sh >/dev/null 2>&1
0 * * * * /home/pi/cronhourly.sh >/dev/null 2>&1
1 0 * * * /home/pi/crondaily.sh >/dev/null 2>&1
2 0 * * 1 /home/pi/cronweekly.sh >/dev/null 2>&1
/home/pi/atstartup.sh: make sure some folders are writable and backups are restored to tmpfs drive
Code: Select all
#!/bin/bash
echo "performance" | tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
chmod 777 /var/log
touch /var/log/domoticz.log
chmod 666 /var/log/domoticz.log
cp /home/pi/zwcfg_0xe9238f6e.xml /var/log/zwcfg_0xe9238f6e.xml
cp /home/pi/domoticz.db /var/log/domoticz.db
cp /home/pi/options.xml /home/pi/domoticz/Config/options.xml
ln -sf /var/log/domoticz.db /home/pi/domoticz
chmod 666 /var/log/zwcfg_0xe9238f6e.xml
echo 0 | tee /sys/class/leds/led0/brightness
echo 0 | tee /sys/class/leds/led1/brightness
/opt/vc/bin/tvservice -o
sudo service domoticz.sh start
/home/pi/crondaily.sh: make a daily backup of the tmpfs files
Code: Select all
#!/bin/bash
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
cp /var/log/domoticz.db /home/pi/domoticz.db
/home/pi/cronweekly.sh: cleanup the logfiles
Code: Select all
#!/bin/bash
sudo rm /var/log/*.gz
sudo rm /var/log/*.1
sudo rm /var/log/SBFSPOT/*
sudo truncate -s 0 /var/log/*
sudo truncate -s 0 /var/log/lighttpd/*
/home/pi/options.xml: keep some zwave settings and set /var/log as log folder
Code: Select all
<?xml version="1.0" encoding="utf-8"?>
<!-- To be effective, this file should be placed in the user data folder specified in the Options::Create method -->
<Options xmlns='http://code.google.com/p/open-zwave/'>
<Option name="logging" value="true" />
<Option name="UserPath" value="/var/log/" />
<Option name="Associate" value="true" />
<Option name="NotifyTransactions" value="false" />
<Option name="DriverMaxAttempts" value="5" />
<Option name="SaveConfiguration" value="true" />
<Option name="RetryTimeout" value="5000" /><!-- Default 40000 --><!-- OK: 15000 -->
<Option name="PerformReturnRoutes" value="true" /><!-- added -->
<Option name="AssumeAwake" value="false" /><!-- added -->
<Option name="SaveLogLevel" value="3" /><!-- added -->
<Option name="QueueLogLevel" value="3" /><!-- added -->
<Option name="DumpTriggerLevel" value="3" /><!-- added -->
<Option name="NetworkKey" value="0x09, 0x05, 0x03, 0x04, 0x01, 0x06, 0x07, 0x01, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10" />
<Option name="RefreshAllUserCodes" value="false" />
<Option name="ThreadTerminateTimeout" value="5000" /><!-- 5000 -->
</Options>
And finally in /etc/init.d/domoticz.sh: make a backup of the database and zwave config while stopping domoticz.
Code: Select all
do_stop()
{
# Return
# 0 if daemon has been stopped
# 1 if daemon was already stopped
# 2 if daemon could not be stopped
# other if a failure occurred
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2
# Wait for children to finish too if this is a daemon that forks
# and if the daemon is only ever run from this initscript.
# If the above conditions are not satisfied then add some other code
# that waits for the process to drop all resources that could be
# needed by services started subsequently. A last resort is to
# sleep for some time.
start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
[ "$?" = 2 ] && return 2
# Many daemons don't delete their pidfiles when they exit.
rm -f $PIDFILE
cp /var/log/domoticz.db /home/pi/domoticz.db
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
return "$RETVAL"
}
I'm pretty sure that with this you avoid most of the writes to the sd card.
There's one big downside of it: if you cut the power you'll lose data of that day. In my case I don't care about that. The temperatures I care about are pushed to a MySQL database on a VPS. As long as you can send the shutdown command there's nothing to worry about.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Wednesday 05 July 2017 9:02
by EddyG
Egregius wrote:Booting from NAS is rather pointless, than you could also run domoticz directly on the nas.
Not always. I have a RaZberry connected and positioned in the middle of the house. Also have a PiAware in the top of the house for better reception.
Also a P1 connection on a slave Domoticz for metering my Power consumption at the ground floor.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Thursday 06 July 2017 19:01
by Number8
I have been running one master Domoticz maybe for 3 years now and a slave Domoticz for 2 years with no issues so far. Temporary files are stored on an external USB key. However, in case of a failure, despite backups I feel that restoring a functioning new system will a PITA especially because, if I understand well, we have to reinstall the exact same domoticz version to make sure the database will be compatible (I'm running under beta). I haven't seen so far a repository of all beta version.
The best solution is: get a NAS which supports LUN's
Look at
http://www.berryterminal.com/doku.php/berryboot and you have a Raspberry Pi that boots (readonly) on the SD-card and runs the OS from the LUN. That 'drive' is mostly a lot faster then any SD-card, so you get better performance.
So I'm very happy to discover here the berryterminal solution, since I have a Synology NAS running 24x7. It will be one of my summer tasks.
Booting from NAS is rather pointless, than you could also run domoticz directly on the nas.
Certainly not, the current RaspPI locations are just incompatible with the NAS. And I will add a new one in the well, shall I put the NAS in it?

Re: Read only Raspberry Pi OS to prevent corruption
Posted: Thursday 06 July 2017 23:53
by hansrune
Have used overlayfs as described on
https://www.domoticz.com/wiki/Setting_u ... spberry_Pi to protect the setup. Recommend to add a small UPS
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Friday 07 July 2017 9:46
by Number8
As far as I'm concerned all RaspPI, routers, switches etc. are powered by a UPS.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Friday 07 July 2017 9:55
by EdddieN
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Saturday 08 July 2017 3:06
by ben53252642
My own Domoticz setup runs inside a KVM virtual machine with a BTRFS filesystem that I can backup and restore in seconds. I have the USB ports mapped through from the host, it's fantastic because I have completely flexibility to upgrade or change the physical hardware.
I've also run Domoticz on Raspberry Pi though and as mentioned setup Domoticz for others running on Raspberry Pi. There are a lot of good suggestions here for making the setup more reliable.
Re: Read only Raspberry Pi OS to prevent corruption
Posted: Monday 10 July 2017 15:24
by dervogt
I have migrated my root filesystem to an external USB drive with 500GB once my setup was up and running. This avoids the issues with symlinks or other small tricks you can do to "save" your SDCard. As the BOOT filesystem practically is read only and once the pi has booted into the OS, the SDCard practically is out of the game.
I have not gone the extra step of switching over to also boot from the USB drive, I personally am happy with the SDCard boot and then having my external HDD takeover the rest.