Automated/unattended backup of Raspberry/Domoticz

In this subforum you can show projects you have made, or you are busy with. Please create your own topic.

Moderator: leecollings

Post Reply
westd001
Posts: 22
Joined: Friday 28 August 2015 21:41
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:
Contact westd001

Automated/unattended backup of Raspberry/Domoticz

Post by westd001 »

Hi All,

Maybe this is interesting for some Domoticz/Raspberry users who want to make a backup of there system.
Last weeks I changed some existing scripts and added some other functionality to make an automated backup of my Rasspberry which runs Domoticz.

There are three scripts I share :
-One to make a backup on the same SD card
-One to copy the backup to another memory card in a USB card reader on the same PI
-One to copy the backup to my HiDrive (Strato) cloud storage via rsync

Every Sunday morning a fresh backup is made, copied and uploaded in my cloud via crontab.

Credits go to SYSMATT and many others who answered several questions on stackoverflow.

I am not an expert in bash coding so I think there will be some improvements possible and I know there are other solutions to transfer the data in a more secure way but its fine for me right now.

PS: Make a user variable with the name "Backupfile" as a string, change the parameters according your setup and be sure the necessary directory's exist on your PI. Store the password of HiDrive in a file called /home/pi/HiDrive/HiDrivePass and chmod this file so only the root user can read this.

Backup script

Code: Select all

#!/bin/bash 
# 20140804-1708 Matthew E Hoskins / Twitter: @sysmatt (c) GNU GPL
# Edited bij John Westdorp 2017-03-18

#set -x  # debugging

# print debug text on screen or in a log file
printscreen=FALSE

#Domoticz Parameters
domoticzIp=192.168.x.xx
domoticzPort=xxxx

# clear the logfile otherwise it grows and grows and.....
truncate -s 0 /var/log/backup_restore/templogFile_backup.txt

#The logFile is located /var/log/backup_restore
function printscreen_or_log {
    if [ "$printscreen" == "TRUE" ]
    then
       echo "$(date "+%FT%T"): Backup_script: ${1}"
    else
       echo "$(date "+%FT%T"): Backup_script: ${1}" >> "/var/log/backup_restore/templogFile_backup.txt"
    fi
}

bomb () {
	printscreen_or_log "BOMB: ${1}"
	exit 1
}

# Start backup
# Default destination directory
DSTDIR=/home/pi/backups
DEFAULT_DFILE="${DSTDIR}/pi.`uname -n`.`date +%Y%m%d%H%M%S`.backup.tar.gz"

# If we get a filename on the command line, use it instead of the default.
DFILE="${1:-$DEFAULT_DFILE}"
DEST_DIR_SANITY=`dirname "${DFILE}"`

# Make sure the destination exists
[ -d "${DEST_DIR_SANITY}" ] || bomb "Destination directory ${DEST_DIR_SANITY} does not exist"

# Store the filename in a user variable
curl -s $domoticzIp":"$domoticzPort"/json.htm?type=command&param=updateuservariable&vname=Backupfile&vtype=2&vvalue="${DFILE}

# Be sure to start backup ? (unattendant backup via CRON job, comment out read command)
printscreen_or_log "******************** Backup script is started ********************"
#read -p "PRESS ENTER to start backup, Press CTRL-c to abort" 

# check state of domoticz and stop the proces if running so files and the database will be closed 
status=`curl -s -i -H "Accept: application/json" "http://"$domoticzIp":"$domoticzPort"/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
if [ $status ]
then
   printscreen_or_log "Domoticz is running , it will be stopped before starting backup."
   sudo service domoticz.sh stop
   sleep 30
else
   printscreen_or_log "Domoticz isn't running, start backup immediately"
fi

# Run the tar archive
printscreen_or_log "Start making backup, this will take a while......"
tar --exclude="${DFILE}" --one-file-system -cvzf "${DFILE}" / /boot

printscreen_or_log "Created archive: ${DFILE}"

# Start Domoticz again, first check the status
status=`curl -s -i -H "Accept: application/json" "http://"$domoticzIp":"$domoticzPort"/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
if [ $status ]
then
   printscreen_or_log "Backup is finished, Domoticz has already been started"
else
   printscreen_or_log "Backup is finished, Domoticz was offline, attempting a restart."
   sudo service domoticz.sh stop
   sleep 30
   sudo service domoticz.sh start
fi

printscreen_or_log "******************** Backup script is finished ********************"

# END
Restore SD card

Code: Select all

#!/bin/bash 
# 20140804-1708 Matthew E Hoskins / Twitter: @sysmatt (c) GNU GPL
# Edited bij John Westdorp 2017-05-21

# print debug text on screen or in a log file
printscreen=FALSE

#Domoticz Parameters
domoticzIp=192.168.x.xx
domoticzPort=xxxx
backup_file_name_IDX=xx

# clear the logfile otherwise it grows and grows and.....
truncate -s 0 /var/log/backup_restore/templogFile_restore.txt

#The logFile is located /var/log/backup_restore
function printscreen_or_log {
    if [ "$printscreen" == "TRUE" ]
    then
       echo "$(date "+%FT%T"): Backup_script: ${1}"
    else
       echo "$(date "+%FT%T"): Backup_script: ${1}" >> "/var/log/backup_restore/templogFile_restore.txt"
    fi
}

printscreen_or_log "******************** Restore script is started ********************"

# get the SD CARD DEVICE name of the USB memory card to restore the backup image on it or supply the device name as argument ie /dev/sda
export USBKEYS=($(                         # Declaration of *array* 'USBKEYS'
    grep -Hv ^0$ /sys/block/*/removable |  # search for *not 0* in `removable` flag of all devices
    sed s/removable:.*$/device\\/uevent/ | # replace `removable` by `device/uevent` on each line of previous answer
    xargs grep -H ^DRIVER=sd |             # search for devices drived by `SD`
    sed s/device.uevent.*$/size/ |         # replace `device/uevent` by 'size'
    xargs grep -Hv ^0$ |                   # search for devices having NOT 0 size
    cut -d / -f 4                          # return only 4th part `/` separated
))

#for dev in ${USBKEYS[@]} ;do               # for each devices in USBKEY...
#    printscreen_or_log $dev \"$(r                        # echo device name and content of model file
#        sed -e s/\ *$//g </sys/block/$dev/device/model
#        )\" ;
#  done

# I have only one USB memory card in my Raspberry so there is only one device name in the array
DEFAULT_SDCARD_DEVICE=/dev/${USBKEYS[0]}                        

# get the backup_file_name from the uservariable or supply the file name as argument
DEFAULT_PI_BACKUP_SLASH=$(curl -s "http://$domoticzIp:$domoticzPort/json.htm?type=command&param=getuservariable&idx=$backup_file_name_IDX"  | grep "Value")
DEFAULT_PI_BACKUP_SLASH="${DEFAULT_PI_BACKUP_SLASH##*Value\" : \"}"
DEFAULT_PI_BACKUP_SLASH=${DEFAULT_PI_BACKUP_SLASH//\",/}

# If we get a device name and/or a filename on the command line, use it instead of the defaults.
SDCARD_DEVICE="${1:-$DEFAULT_SDCARD_DEVICE}"
PI_BACKUP_SLASH="${2:-$DEFAULT_PI_BACKUP_SLASH}"
shift; shift;
# Any remaining args are added to TAR command line

printscreen_or_log "SD card device ==>${SDCARD_DEVICE}<=="
printscreen_or_log "Backup_file_name ==>${PI_BACKUP_SLASH}<=="

# Extra operations which can be done in ENV Vars
# PI_OVERWRITE_ETC_HOSTNAME="new-hostname"
# PI_OPENVPN_CONFIG_FILE="/some/source/client.conf"

# Temp random mountpoint under /tmp
MOUNTPOINT="/tmp/pi.sd.${RANDOM}${$}"

bomb (){
	printscreen_or_log "BOMB: ${1}"
	exit 1
}

[ -x "/sbin/mkfs.vfat" ] || bomb "/sbin/mkfs.vfat Missing"

printscreen_or_log ""
printscreen_or_log "=== ${SDCARD_DEVICE} Current Partition Table - To be destroyed! ==="
parted --script ${SDCARD_DEVICE} "print"
printscreen_or_log ""
# Be sure to write image ? (unattendant writing via CRON job, comment out read command)
#read -p "PRESS ENTER to DESTROY ${SDCARD_DEVICE}, Press CTRL-c to abort" 

parted --script "${SDCARD_DEVICE}"  "mklabel msdos"
parted --script "${SDCARD_DEVICE}"  "mkpart primary fat16 1MiB 64MB"
parted --script "${SDCARD_DEVICE}"  "mkpart primary ext4 64MB -1s"
parted --script "${SDCARD_DEVICE}"  print
mkfs.vfat  "${SDCARD_DEVICE}1"
mkfs.ext4 -F -j "${SDCARD_DEVICE}2" 
mkdir -p "${MOUNTPOINT}"
mount "${SDCARD_DEVICE}2" "${MOUNTPOINT}"
mkdir -p "${MOUNTPOINT}/boot"
mount "${SDCARD_DEVICE}1" "${MOUNTPOINT}/boot"
df -h "${MOUNTPOINT}"
df -h "${MOUNTPOINT}/boot"
# unattendant writing via CRON job, comment out read command
#read -p "Press ENTER To begin image restore"
printscreen_or_log "Image restore started at device: ${SDCARD_DEVICE}, this will take a while......"
tar $@ -xvzf  "${PI_BACKUP_SLASH}" -C "${MOUNTPOINT}"

# Begin NOOBS fixup / Recommended by @KevinSidwar / 20140829-1744-MEH
ORIG_SLASHDEV_FOUND=`cat "${MOUNTPOINT}/etc/fstab"| grep " / "     |grep -v "^#" |awk '{ print $1; }'`
ORIG_BOOTDEV_FOUND=`cat  "${MOUNTPOINT}/etc/fstab"| grep " /boot " |grep -v "^#" |awk '{ print $1; }'`

# Hardcoded
NEW_SLASHDEV="/dev/mmcblk0p2"
NEW_BOOTDEV="/dev/mmcblk0p1"

# Defaults just in case the above search of fstab is a miss
ORIG_SLASHDEV="${ORIG_SLASHDEV_FOUND:-$NEW_SLASHDEV}"
ORIG_BOOTDEV="${ORIG_BOOTDEV_FOUND:-$NEW_BOOTDEV}"

printscreen_or_log "ORIG_SLASHDEV[${ORIG_SLASHDEV}] NEW_SLASHDEV[${NEW_SLASHDEV}] ORIG_BOOTDEV[${ORIG_BOOTDEV}] NEW_BOOTDEV[${NEW_BOOTDEV}]"

FILE_FSTAB="${MOUNTPOINT}/etc/fstab"
FILE_CMDLINE="${MOUNTPOINT}/boot/cmdline.txt"
FILE_OS_CONFIG_JSON="${MOUNTPOINT}/boot/os_config.json"

if [  "${ORIG_SLASHDEV}" != "${NEW_SLASHDEV}"  -o   "${ORIG_BOOTDEV}" != "${NEW_BOOTDEV}"  ]
then
	printscreen_or_log "CONFIGURATION CONVERSION NECESSARY - Devices do not match, modifying files. "

	THISFILE="${FILE_FSTAB}"
	if [ -e "${THISFILE}" ] 
	then
		printscreen_or_log "Fixing ${THISFILE}"
		cp -f "${THISFILE}" "${THISFILE}.old"
		cat "${THISFILE}.old" |sed -e "s#${ORIG_SLASHDEV}#%NEW_SLASHDEV%# ; s#${ORIG_BOOTDEV}#%NEW_BOOTDEV%#"  > "${THISFILE}.i"
		cat "${THISFILE}.i"   |sed -e "s#%NEW_SLASHDEV%#${NEW_SLASHDEV}# ; s#%NEW_BOOTDEV%#${NEW_BOOTDEV}#"    >"${THISFILE}"
		rm -f "${THISFILE}.i"
	fi

	THISFILE="${FILE_CMDLINE}"
	if [ -e "${THISFILE}" ] 
	then
		printscreen_or_log "Fixing ${THISFILE}"
		cp -f "${THISFILE}" "${THISFILE}.old"
		cat "${THISFILE}.old" |sed -e "s#${ORIG_SLASHDEV}#%NEW_SLASHDEV%# ; s#${ORIG_BOOTDEV}#%NEW_BOOTDEV%#"  > "${THISFILE}.i"
		cat "${THISFILE}.i"   |sed -e "s#%NEW_SLASHDEV%#${NEW_SLASHDEV}# ; s#%NEW_BOOTDEV%#${NEW_BOOTDEV}#"    >"${THISFILE}"
		rm -f "${THISFILE}.i"
	fi

	THISFILE="${FILE_OS_CONFIG_JSON}"
	if [ -e "${THISFILE}" ] 
	then
		printscreen_or_log "Fixing ${THISFILE}"
		cp -f "${THISFILE}" "${THISFILE}.old"
		cat "${THISFILE}.old" |sed -e "s#${ORIG_SLASHDEV}#%NEW_SLASHDEV%# ; s#${ORIG_BOOTDEV}#%NEW_BOOTDEV%#"  > "${THISFILE}.i"
		cat "${THISFILE}.i"   |sed -e "s#%NEW_SLASHDEV%#${NEW_SLASHDEV}# ; s#%NEW_BOOTDEV%#${NEW_BOOTDEV}#"    >"${THISFILE}"
		rm -f "${THISFILE}.i"
	fi

fi
# End Noobs fixup

if [ -n "${PI_OVERWRITE_ETC_HOSTNAME}" ] 
then
	# Write a new /etc/hostname
	printscreen_or_log "Writing new /etc/hostname with [${PI_OVERWRITE_ETC_HOSTNAME}] to ${MOUNTPOINT}/etc/hostname"
	printscreen_or_log "${PI_OVERWRITE_ETC_HOSTNAME}" > "${MOUNTPOINT}/etc/hostname"
fi

if [ -e "${PI_OPENVPN_CONFIG_FILE}" ] 
then
	printscreen_or_log "Writing OpenVPN config [${PI_OPENVPN_CONFIG_FILE}] to ${MOUNTPOINT}/etc/openvpn/"
	rm -f ${MOUNTPOINT}/etc/openvpn/*.conf
	cp -f "${PI_OPENVPN_CONFIG_FILE}" "${MOUNTPOINT}/etc/openvpn/"
fi

printscreen_or_log =DONE=
printscreen_or_log "The SD is mounted at: ${MOUNTPOINT}"
printscreen_or_log "Now would be a good time to make modifications in another shell session..."

# unattendant writing via CRON job, comment out read command
#read -p "Press ENTER To unmount or CTRL-c to exit leaving mounted."

printscreen_or_log "Unmounting. This may take a moment..."
set -x
umount "${MOUNTPOINT}/boot"
umount "${MOUNTPOINT}"
printscreen_or_log "******************** Restore script is finished ********************"

# END
Copy to HiDrive

Code: Select all

#!/bin/bash 
# 20170604-001 John Westdorp (c) GNU GPL

# print debug text on screen or in a log file
printscreen=FALSE

#Domoticz Parameters
domoticzIp=192.168.x.xx
domoticzPort=xxxx
backup_file_name_IDX=xx

#Variables
HIDRIVE_HOSTNAME="rsync.hidrive.strato.com"
HIDRIVE_USERNAME="xxxxxxxx"
HIDRIVE_PASSWORD=$(cat /home/pi/HiDrive/HiDrivePass)

# Source and destination directory
SRCDIR="/home/pi/backups"
DESTDIR="/users/xxxxxxx/Raspberry_backups"

# clear the logfile otherwise it grows and grows and.....
truncate -s 0 /var/log/backup_restore/templogFile_copy_HiDrive.txt

#The logFile is located /var/log/backup_restore
function printscreen_or_log {
    if [ "$printscreen" == "TRUE" ]
    then
       echo "$(date "+%FT%T"): Backup_script: ${1}"
    else
       echo "$(date "+%FT%T"): Backup_script: ${1}" >> "/var/log/backup_restore/templogFile_copy_HiDrive.txt"
    fi
}

# Start copying
printscreen_or_log "******************** Copy backups to HiDrive script is started ********************"

# Be sure to start copying (unattendant copying via CRON job, comment out read command)
#read -p "PRESS ENTER to start backup, Press CTRL-c to abort" 

# get the latest backup_file_name from the uservariable
PI_BACKUP=$(curl -s "http://$domoticzIp:$domoticzPort/json.htm?type=command&param=getuservariable&idx=$backup_file_name_IDX"  | grep "Value")
PI_BACKUP=${PI_BACKUP##*Value\" : \"}
PI_BACKUP=${PI_BACKUP//\",/}

printscreen_or_log "Backup_file ==>${PI_BACKUP}<== will be copied to HiDrive"
	
if [ -f $PI_BACKUP ]
then
   # Run the copy
   printscreen_or_log "Start copying, this will take a while......"

   sshpass -p $HIDRIVE_PASSWORD rsync -rltDvze "ssh" $SRCDIR $HIDRIVE_USERNAME@$HIDRIVE_HOSTNAME:$DESTDIR

   printscreen_or_log "Backup_file ==>${PI_BACKUP}<== is succesfully copied to HiDrive"

   # remove the file after copying otherwise the next backup will also contain the backup file etc. etc
   rm $PI_BACKUP
   printscreen_or_log "Backup_file ==>${PI_BACKUP}<== is deleted from ==>$SRCDIR<=="
else
    printscreen_or_log "Error: Can't copy backup_file ==>${PI_BACKUP}<== , file not found"
fi

printscreen_or_log "******************** Backup script is finished ********************"

# END
Have fun with it

JJ
Top
Post Reply

Return to “Show your projects”

Who is online

Users browsing this forum: No registered users and 1 guest