Read only Raspberry Pi OS to prevent corruption Topic is solved

[oliviers]

Hello,
Like Egregius says, no big deal for me if the SD card gets corrupted after 2 or 3 years ... I would just swap it. Only 5-6€ ....
Oliviers

[Egregius]

I have migrated my root filesystem to an external USB drive with 500GB once my setup was up and running. This avoids the issues with symlinks or other small tricks you can do to "save" your SDCard. As the BOOT filesystem practically is read only and once the pi has booted into the OS, the SDCard practically is out of the game.

I have not gone the extra step of switching over to also boot from the USB drive, I personally am happy with the SDCard boot and then having my external HDD takeover the rest.

Your harddrive probably consumes more than your pi

[dervogt]

Your harddrive probably consumes more than your pi

Probably this is right, but coming from an Octacore AMD HTPC with a S-ATA RAID, two DVB-S cards, major PCIx VGA and another digital PCIx SoundCard, this still is an energetic improvement

[nigels0]

Booting from NAS is rather pointless, than you could also run domoticz directly on the nas. Running on my nas is not a good option here as it consumes 65W, calculate the cost of that... Besides that I also tried RPi with NFS on nas but that gave a lot more problems.

I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive

Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).

I do something similar:

/etc/fstab: You may easily set a larger memory size, memory will only be occupied depending on the file sizes.

Code: Select all

tmpfs   /temp        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0
tmpfs   /var/log        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0

sudo crontab -e

Code: Select all

@reboot /home/pi/atstartup.sh >/dev/null 2>&1
0 * * * * /home/pi/cronhourly.sh >/dev/null 2>&1
1 0 * * * /home/pi/crondaily.sh >/dev/null 2>&1
2 0 * * 1 /home/pi/cronweekly.sh >/dev/null 2>&1

/home/pi/atstartup.sh: make sure some folders are writable and backups are restored to tmpfs drive

Code: Select all

#!/bin/bash
echo "performance" | tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
chmod 777 /var/log
touch /var/log/domoticz.log
chmod 666 /var/log/domoticz.log
cp /home/pi/zwcfg_0xe9238f6e.xml /var/log/zwcfg_0xe9238f6e.xml
cp /home/pi/domoticz.db /var/log/domoticz.db
cp /home/pi/options.xml /home/pi/domoticz/Config/options.xml
ln -sf /var/log/domoticz.db /home/pi/domoticz
chmod 666 /var/log/zwcfg_0xe9238f6e.xml
echo 0 | tee /sys/class/leds/led0/brightness
echo 0 | tee /sys/class/leds/led1/brightness
/opt/vc/bin/tvservice -o
sudo service domoticz.sh start

/home/pi/crondaily.sh: make a daily backup of the tmpfs files

Code: Select all

#!/bin/bash
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
cp /var/log/domoticz.db /home/pi/domoticz.db

/home/pi/cronweekly.sh: cleanup the logfiles

Code: Select all

#!/bin/bash
sudo rm /var/log/*.gz
sudo rm /var/log/*.1
sudo rm /var/log/SBFSPOT/*
sudo truncate -s 0 /var/log/*
sudo truncate -s 0 /var/log/lighttpd/*

/home/pi/options.xml: keep some zwave settings and set /var/log as log folder

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<!-- To be effective, this file should be placed in the user data folder specified in the Options::Create method -->
<Options xmlns='http://code.google.com/p/open-zwave/'>
  <Option name="logging" value="true" />
  <Option name="UserPath" value="/var/log/" />
  <Option name="Associate" value="true" />
  <Option name="NotifyTransactions" value="false" />
  <Option name="DriverMaxAttempts" value="5" />
  <Option name="SaveConfiguration" value="true" />
  <Option name="RetryTimeout" value="5000" /><!-- Default 40000 --><!-- OK: 15000 -->
  <Option name="PerformReturnRoutes" value="true" /><!-- added -->
  <Option name="AssumeAwake" value="false" /><!-- added -->
  <Option name="SaveLogLevel" value="3" /><!-- added -->
  <Option name="QueueLogLevel" value="3" /><!-- added -->
  <Option name="DumpTriggerLevel" value="3" /><!-- added -->
  <Option name="NetworkKey" value="0x09, 0x05, 0x03, 0x04, 0x01, 0x06, 0x07, 0x01, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10" />
  <Option name="RefreshAllUserCodes" value="false" />
  <Option name="ThreadTerminateTimeout" value="5000" /><!-- 5000 -->
</Options>

And finally in /etc/init.d/domoticz.sh: make a backup of the database and zwave config while stopping domoticz.

Code: Select all

do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
	cp /var/log/domoticz.db /home/pi/domoticz.db
	cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
        return "$RETVAL"
}

I'm pretty sure that with this you avoid most of the writes to the sd card.
There's one big downside of it: if you cut the power you'll lose data of that day. In my case I don't care about that. The temperatures I care about are pushed to a MySQL database on a VPS. As long as you can send the shutdown command there's nothing to worry about.

Hi,

I Think I've got this working - I don't use z-wave, so commented anything about the zwcfg files out - but your list is missing cronhourly.sh (it's mentioned in the crontab entries)

[Egregius]

I shall look into that when I get back from vacation. Out of my head I thinks there's nothing domoticz related in it. More stuff like disk usage etc.

[nigels0]

No worries - nice set of scripts though! anything that stops the SD cards corrupting is welcome.

[dressie]

24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.

In my environment I have a Synology NAS with RAID 6 which can do LUNs, didn't know about the option to boot from a LUN.

Even found a guide on how to do it!
http://www.berryterminal.com/doku.php/s ... sing_iscsi

Combined with watchdog and monit setup, Indeed this would appear to be a very good option particularly with the ability to schedule automatic snapshots.

I just got a new Pi3 and used this set up. I'm testing it right now with a minimum filled Domoticz (main Domoticz is still running on a Pi2 with SD card). So far it looks good. It's fast, stable and you don't have to worry about the loss of data. I'm going to test it another few days and will move the database and the Zwave stick and 433Mhz over to the new setup. Will keep you up to date.

[Egregius]

but your list is missing cronhourly.sh (it's mentioned in the crontab entries)

Here it is, as I tought it's about checking disk usage:

Code: Select all

#!/bin/bash
df -H | grep -vE '^Filesystem|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 80 ]; then
    msg="Running out of space \"$partition ($usep%)\" on $(hostname)"
    curl -s --data-urlencode "text=$msg" --data "silent=false" http://127.0.0.1/secure/telegram.php
  fi
	if [ $usep -ge 50 ] && [ $partition = "tmpfs" ]; then
		sudo rm /var/log/*.gz
		sudo rm /var/log/*.1
		sudo truncate -s 0 /var/log/*
		sudo truncate -s 0 /var/log/lighttpd/*
	fi
done
sudo ntpdate be.pool.ntp.org

[Number8]

I just got a new Pi3 and used this set up. I'm testing it right now with a minimum filled Domoticz (main Domoticz is still running on a Pi2 with SD card). So far it looks good. It's fast, stable and you don't have to worry about the loss of data. I'm going to test it another few days and will move the database and the Zwave stick and 433Mhz over to the new setup. Will keep you up to date

Using wired Ethernet, Debian Lite has been installed without problem. Afterwards I tried to install using wifi and it was unsuccessful. It keeps trying to download the distribution forever. The wifi dongle is recognized by Berryterminal and connected to Internet. Once installation has been done using wired Ethernet, it is not possible to switch to wifi afterwards. I then discovered that the distribution cannot be upgraded, you have to stick to the one that has been installed. All these things considered I stopped the experiment.

[dressie]

Using wired Ethernet, Debian Lite has been installed without problem. Afterwards I tried to install using wifi and it was unsuccessful. It keeps trying to download the distribution forever. The wifi dongle is recognized by Berryterminal and connected to Internet. Once installation has been done using wired Ethernet, it is not possible to switch to wifi afterwards. I then discovered that the distribution cannot be upgraded, you have to stick to the one that has been installed. All these things considered I stopped the experiment.

Mine will stay wired no matter what, just curious what you ment by "cannot be upgraded". Are you referring to the basic 'sudo apt-get update' and 'sudo apt-get upgrade' ?

[Number8]

Mine will stay wired no matter what, just curious what you ment by "cannot be upgraded". Are you referring to the basic 'sudo apt-get update' and 'sudo apt-get upgrade' ?

Yes

[EddyG]

That is right, mine is on: Linux raspberrypi 4.9.26v7-aufs #1 SMP Tue May 9 20:14:03 CEST 2017 armv7l GNU/Linux
And sudo apt-mark showhold shows:
raspberrypi-bootloader
raspberrypi-kernel

That's because the system is using aufs (https://en.wikipedia.org/wiki/Aufs)

[dressie]

So it's stable, but eventually the system will become outdated. Time for some pros and cons .

Verstuurd vanaf mijn SM-G920F met Tapatalk

[Number8]

So it's stable, but eventually the system will become outdated. Time for some pros and cons .

Indeed

[EddyG]

There is a lot on GitHub, so you could build your own image with new(er) system.
And I suppose that the creator will update berryboot over time.

[lrybak]

/home/pi/crondaily.sh: make a daily backup of the tmpfs files

Code: Select all

#!/bin/bash
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
cp /var/log/domoticz.db /home/pi/domoticz.db

Did you consider to use bultin .backup command instead of hot copying db file?

Code: Select all

sqlite3 /var/log/domoticz.db ".backup /home/pi/domoticz.db"

backup cli command uses SQLite Online Backup API. In the same way domoticz handles hourly/daily/weekly backups.

[Egregius]

Nope, didn't know that.
Anyway, my SD card crashed anyway after 2 years. Moved to a NUC with proxmox now. No need to worry about backups anymore.

[lrybak]

Moved to a NUC with proxmox now

How're you deal with gpio/i2c and stuff like that?

No need to worry about backups anymore.

Don't be surprised, HDD/SSD can crash as well

[poudenes]

Hi Folks,

As a long time Domoticz user and someone who has setup others with Domoticz running on Raspberry Pi's I have observed a general issue with this setup:

Eventually the SD Card ALWAYS gets corrupted, it's only a matter of time. My father for instance, he had two SD cards become corrupted, eventually he lost faith in Domoticz and now has a Vera Edge running his Z-Wave system.

I would like to see the development of a 100% read only Raspberry Pi SD card image (write can be enabled for updates and then disabled again) and the option of writing the Domoticz database to an external storage (USB stick for example) or external MariaDB server.

This setup also has the advantage of being immune to sudden power loss should the user decide to power cycle the device by removing the power cable.

I know how to make such a OS image, wanting feedback on the idea, particularly in relation to options with the Domoticz DB.

I did a setup where i configured RPI3 to start from a USB stick directly without using a SD card at all.
A USB Stick is build to get read/write lot of times. SDCard is build to store

Here how i did it:

Code: Select all

Start from SDcard.
Goto RPi3 using ssh command as usual and give this command:

echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt

Reboot from SDcard and after ssh login check if the change is activated:

$ vcgencmd otp_dump | grep 17:
17:3020000a

When 17:3020000a appears then the change is done
Put USB Stick in RPi3 with Jessy/Strech and remove the SDcard and boot from USB Stick

[Egregius]

Moved to a NUC with proxmox now

How're you deal with gpio/i2c and stuff like that?

No need to worry about backups anymore.

Don't be surprised, HDD/SSD can crash as well

I just set a pastrough for my zwave stick and the rfxcom.
Hdd can fail, but with proxmox it's very easy to make complete backups to nas. I even created the image on another proxmox host while I was waiting for the delivery of the nuc