Script to backup to a Synology NAS

All kinds of 'OS' scripts

Moderator: leecollings

Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Script to backup to a Synology NAS

Post by Johan1974 »

Hello I used the script

Code: Select all

#!/bin/bash
#
# 20201028 Modified to replace CR/LF (windows) to LF (Linux)
#
# LOCAL/FTP/SCP/MAIL PARAMETERS
#
SERVER="192.168.1.11"                                      # IP of Synology NAS, used for ftp
USERNAME="Domoticz"                                        # FTP username of Network disk used for ftp
PASSWORD="12345"                                           # FTP password of Network disk used for ftp
DESTDIR=" /tmp/"                                           # used for temorarily storage
DESTDIRNAS="/homes/domoticz/backup/"                       # Path to your Synology NAS backup folder
DOMO_IP="192.168.1.12"                                     # Domoticz IP
DOMO_PORT="8080"                                           # Domoticz port

### END OF USER CONFIGURABLE PARAMETERS

TIMESTAMP=`/bin/date +%Y%m%d%H%M%S`
BACKUPFILE="domoticz_$TIMESTAMP.db" # backups will be named "domoticz_YYYYMMDDHHMMSS.db.gz"
BACKUPFILEGZ="$BACKUPFILE".gz

#
### Create backup and ZIP it
#
/usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
gzip -9 /tmp/$BACKUPFILE
tar -zcvf /tmp/domoticz_scripts_$TIMESTAMP.tar.gz /home/pi/domoticz/scripts/
tar -zcvf /tmp/telegram_scripts_$TIMESTAMP.tar.gz /home/pi/tg/scripts/

#
### Send to Network disk through FTP
#
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
curl -s --disable-epsv -v -T"/tmp/domoticz_scripts_$TIMESTAMP.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
curl -s --disable-epsv -v -T"/tmp/telegram_scripts_$TIMESTAMP.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"

#
### Remove temp backup file
#
/bin/rm /tmp/$BACKUPFILEGZ
/bin/rm /tmp/domoticz_scripts_$TIMESTAMP.tar.gz
/bin/rm /tmp/telegram_scripts_$TIMESTAMP.tar.gz

#eoj
## Done!

I save this script at: /home/pi/domoticz/scripts whit the name domoticz_backup.sh

When I test the script with:
sudo /home/pi/domoticz/scripts/domoticz_backup.sh

I see the next error:
sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory

What is going wrong.??
User avatar
cyberclwn
Posts: 103
Joined: Thursday 20 August 2015 22:53
Target OS: Raspberry Pi / ODroid
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by cyberclwn »

Hi,

does "/bin/bash" exist?

Code: Select all

ls -ls /bin/bash
And is domoticz_backup.sh executable?

Code: Select all

ls -ls /home/pi/domoticz/scripts/domoticz_backup.sh
Check if there is a x in the line ... rwx
if not, change it with

Code: Select all

chmod +x /home/pi/domoticz/scripts/domoticz_backup.sh
3xPi 2B (Domoticz "live", Domoticz "sandbox", PhotoFrame)
RFXCom433(E), KaKu, Oregon Scientific, Keyes 2-relay, Logitech Media Server, MiLight, Smartwares heating controller(2x), IR Send/Receive, Keyes PIR, XH-M131 DuskSensor, DHT22/11
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

Yes,

Code: Select all

 844 -rwxr-xr-x 1 root root 863400 Jan 15  2017 /bin/bash 

Code: Select all

 4 -rwxr-xr-x 1 pi pi 1604 Oct 27 21:57 /home/pi/domoticz/scripts/domoticz_backup.sh 
Last edited by Johan1974 on Tuesday 27 October 2020 22:17, edited 1 time in total.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Johan1974 wrote: Tuesday 27 October 2020 22:10 Yes,
@Johan1974

How should we read your "Yes" ?
Is everything OK now or does it mean the settings were already as suggested but the command still does not work?
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

Sorry, I'm not clear.

I have CMOD de file domoticz_backup to 777
When I use:

Code: Select all

 ls -ls /home/pi/domoticz/scripts/domoticz_backup.sh
the answer is

Code: Select all

 4 -rwxrwxrwx 1 pi pi 1609 Oct 28 09:50 /home/pi/domoticz/scripts/domoticz_backup.sh
When I use

Code: Select all

 sudo sh /home/pi/domoticz/scripts/domoticz_backup.sh
the answer is

Code: Select all

 tar: Removing leading `/' from member names
tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: Removing leading `/' from member names
tar: /home/pi/tg/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
'!b.gzCan't open '/tmp/domoticz_20201028100108
curl: try 'curl --help' or 'curl --manual' for more information
* Illegal characters found in URL
* Closing connection -1
* Illegal characters found in URL
* Closing connection -1
/bin/rm: cannot remove ‘/tmp/domoticz_20201028100108\r.db.gz\r\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/domoticz_scripts_20201028100108\r.tar.gz\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/telegram_scripts_20201028100108\r.tar.gz\r’: No such file or directory
The files domoticz_20201028100108.db.gz and telegram_scripts_20201028100108.tar.gz are made in the tmp directory on the raspberry. But not upload to the NAS
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Johan1974 wrote: Wednesday 28 October 2020 10:08 tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
Did you edit the file with a windows editor ? (asking this because of the \r in your log)
Google for "windows editor unix line breaks"
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

That's right, and I can remember making this mistake more than once .. :(
I started over from the beginning, with a new domoticz_backup.sh file. I make script executable with

Code: Select all

  sudo chmod +x /home/pi/domoticz/scripts/domoticz_backup.sh 
Then I run the script with:

Code: Select all

 sudo /home/pi/domoticz/scripts/domoticz_backup.sh 
the answer is:

Code: Select all

 sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory 


when I use

Code: Select all

 sudo sh /home/pi/domoticz/scripts/domoticz_backup.sh 

the answer is:

Code: Select all

 tar: Removing leading `/' from member names
tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: Removing leading `/' from member names
tar: /home/pi/tg/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
'!b.gzCan't open '/tmp/domoticz_20201028104943
curl: try 'curl --help' or 'curl --manual' for more information
* Illegal characters found in URL
* Closing connection -1
* Illegal characters found in URL
* Closing connection -1
/bin/rm: cannot remove ‘/tmp/domoticz_20201028104943\r.db.gz\r\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/domoticz_scripts_20201028104943\r.tar.gz\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/telegram_scripts_20201028104943\r.tar.gz\r’: No such file or directory
I do something wrong:(
what is the difference between sudo and sudo sh
I read the manual on https://www.domoticz.com/wiki/Daily_b ... ernal_hdd and follow the instructions
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Johan1974 wrote: Wednesday 28 October 2020 10:58 what is the difference between sudo and sudo sh
I read the manual on https://www.domoticz.com/wiki/Daily_b ... ernal_hdd and follow the instructions
sudo does use your default shell (probably bash ?) and sudo sh force the use of the sh shell
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Script to backup to a Synology NAS

Post by Egregius »

Why not use a script on your nas with rsync of the complete domoticz folder? Much easier.
If /tmp is on your SD card it'll ruin it.
If /tmp is in tmpfs you can hit memory limits.

Also, using rsync for the complete domoticz folder gives you the advantage of making versions possible and you'll have a backup of your scripts etc.
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

Okay, so I use to run the script:

Code: Select all

  sudo /home/pi/domoticz/scripts/domoticz_backup.sh  
that is also what I read in the manual on the website.
but then nothing happens, only the following message:

Code: Select all

  sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory 
When asked by @cyberclwn whether bin / bash exists I get the answer :

Code: Select all

  844 -rwxr-xr-x 1 root root 863400 Jan 15  2017 /bin/bash  
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

Egregius wrote: Wednesday 28 October 2020 11:51 Why not use a script on your nas with rsync of the complete domoticz folder? Much easier.
If /tmp is on your SD card it'll ruin it.
If /tmp is in tmpfs you can hit memory limits.

Also, using rsync for the complete domoticz folder gives you the advantage of making versions possible and you'll have a backup of your scripts etc.
There are more roads that lead to Rome. :) :)
I think it is important to have a back up because I already have a lot of data in domoticz.
Can I find a manual of this script.?

I still hope to find a solution to my first problem
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Johan1974 wrote: Wednesday 28 October 2020 12:34 I still hope to find a solution to my first problem
@Johan1974,

I modified your file in the opening post of this topic. It still contained CR/LF and with that the shell will not recognize the
#!/bin/bash (shebang) and does not know what to do with it.
Please copy and save it again and make sure no windows editor is touching it again. :)

And I fully agree with @Egregius. rsync is much better suited for the job.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Johan1974
Posts: 35
Joined: Monday 04 December 2017 20:56
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Script to backup to a Synology NAS

Post by Johan1974 »

I started all over again and didn't use a windows editor this time.
And yes! It works now! It's very easy to use an editor, but I understand I have to do everything with Putty and ssh..
Thank you very much.

I am still interested in the script @Egregius uses. A total backup is always better.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Johan1974 wrote: Wednesday 28 October 2020 14:20 I started all over again and didn't use a windows editor this time.
And yes! It works now! It's very easy to use an editor, but I understand I have to do everything with Putty and ssh..
II use Notepad++ on my Windows desktop and -laptop and that editor is perfectly OK with Unix files and it is opensource.
Also not unimportant; it is completely free :D
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Script to backup to a Synology NAS

Post by EddyG »

I use rsnapshot to take locally snapshots of a few dirs of my Raspberry to a local mounted USB-stick (Sandisk 32GB)
The snapshot is taken every 4 Hours for the duration of 4 weeks, kind of son father grandfather backup.
Every 5 minutes I backup only the Domoticz database to my Synology NAS and keep 864 versions eg. 3 days. You could extend that.
The database is saved with date and time and database version in the name. Logging is done in /var/log
Below my script. sqlite3 and bc should be installed and Synology should support nfs.
Run it as root from cron every 5 minutes. On a Pi 3B+ is takes about 6-10 seconds.
All my Raspberry's are backuped this way hence the $HOSTNAME in the path.

Code: Select all

#!/usr/bin/env /bin/bash

SCRIPTNAME=`basename "$0"`
SCRIPTNAME=${SCRIPTNAME%.*}
LOGFILE=/var/log/$SCRIPTNAME.log
ERRORLOG=/var/log/$SCRIPTNAME_ERROR.log
HOSTNAME=$(hostname)
DOMO_IP="127.0.0.1"      # Domoticz IP
DOMO_PORT="8080"         # Domoticz port
REMOTEHOST="192.168.2.xxx"
REMOTEPATH="/volume1/RaspberryPi/Domoticz_DB/$HOSTNAME"
SOURCEFILE="domoticz.db"
SOURCEPATH="/home/pi/domoticz/"
MOUNTPOINT="/mnt/$SCRIPTNAME"
### END OF USER CONFIGURABLE PARAMETERS

# Start Timestamp
STARTTIME=$(date +"%s.%N")

# Place in the log file information concerning the execution of this script
echo -n "Domoticz database backup start at " >> $LOGFILE
echo $(date) >> $LOGFILE

if [ ! -d $MOUNTPOINT ]; then
    sudo mkdir $MOUNTPOINT
fi

if ! mountpoint -q $MOUNTPOINT; then
    sudo mount -t nfs $REMOTEHOST:$REMOTEPATH $MOUNTPOINT
fi
if ! /bin/mountpoint -q $MOUNTPOINT; then
    touch $ERRORLOG
    chmod 0666 $ERRORLOG
    echo "Mount $MOUNTPOINT ERROR!!!" >> $ERRORLOG
    exit 99
fi

#Delete backups older than 3 days = 864 (288 per day is saved every 5 minutes)
cd $MOUNTPOINT
ls -tr | head -n -864 | xargs --no-run-if-empty rm

### Create backup
TIMESTAMP=`/bin/date +%Y%m%d%H%M%S`
DB_VERSION=`sqlite3 $SOURCEPATH$SOURCEFILE "Select nValue from Preferences WHERE Key = 'DB_Version'"`
BACKUPFILE=$HOSTNAME"_"$TIMESTAMP"_"$DB_VERSION".db" # backups will be named "domoticz_YYYYMMDDHHMMSS_ZZZ.db"

/usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > $MOUNTPOINT/$BACKUPFILE

sync
sudo /bin/umount -l $MOUNTPOINT

# End timestamp
ENDTIME=$(date +"%s.%N")

# Convert nanoseconds to milliseconds
# crudely by taking first 3 decimal places
TIMEDIFF=`echo "$ENDTIME - $STARTTIME" | bc | awk -F"." '{print $1"."substr($2,1,3)}'`
SEC=`echo "$TIMEDIFF % 60" | bc`

echo "Total of $SEC seconds elapsed." >> $LOGFILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOGFILE

### Done!
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Script to backup to a Synology NAS

Post by Egregius »

Johan1974 wrote: Wednesday 28 October 2020 14:20 I am still interested in the script @Egregius uses. A total backup is always better.
This is executed at my nas (only part of the script copied because it backups multiple servers)
It'll create a versioned backup for every day it has run. Because of the --link-dest option only changed files occupy space. Files that didn't change are actually symlinks. Don't worry about that, linux takes care of that perfectly.
The great advantage of this is that you can go back in time to any version you want. And that you don't stress your raspberry. The NAS can do perfectmy what it does best: keeping backups.

Code: Select all

#!/bin/sh
NOW=$(date +"%Y-%m-%d")
BPATH=/volume1/homes/guy/backup
RUNLOG=/volume1/homes/guy/log/$NOW.txt
NAME="Domoticz"
echo  ------------------- START $NAME -- $(date) | tee -a $RUNLOG
SOURCE="[email protected]:/domoticz/"
DESTINATION="$BPATH/$NAME/$NOW"
mkdir -p "$DESTINATION"
rsync -aP -e "ssh -i /root/.ssh/home" --stats --delete-after --links --ignore-errors --link-dest="../__prev/" "$SOURCE" "$DESTINATION" | tee -a $RUNLOG
rm -f "$BPATH/$NAME/__prev"
ln -s "$BPATH/$NAME/$NOW" "$BPATH/$NAME/__prev"
echo  ------------------- END  $NAME -- $(date) | tee -a $RUNLOG
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Script to backup to a Synology NAS

Post by EddyG »

Nice script working from the other side. :)
Just copying or rsyncing is a little bit dangerous on an open database.
A more save way is to use "backupdatabase.php" it has some provisions in the domoticz program for looking for busy and locks, and it Optimizes and Vacuums the database before a copy is made. The copy is made by the sqlite3 library.
User avatar
Egregius
Posts: 2589
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: v2024.7
Location: Beitem, BE
Contact:

Re: Script to backup to a Synology NAS

Post by Egregius »

Never had an issue with it, restored multiple times. Works flawless.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: Script to backup to a Synology NAS

Post by waaren »

Egregius wrote: Thursday 29 October 2020 7:08 Never had an issue with it, restored multiple times. Works flawless.
Lucky you :) And to be honest I do the same...

BUT

The SQLite developers say this:

How To Corrupt An SQLite Database File

1.2. Backup or restore while a transaction is active
Systems that run automatic backups in the background might try to make a backup copy of an SQLite database file while it is in the middle of a transaction. The backup copy then might contain some old and some new content, and thus be corrupt.

The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library. Failing that, it is safe to make a copy of an SQLite database file as long as there are no transactions in progress by any process. If the previous transaction failed, then it is important that any rollback journal (the *-journal file) or write-ahead log (the *-wal file) be copied together with the database file itself.
_____________________________________

So why take any chances? It is safer, quite easy and very quick to do something like

Code: Select all

sudo sqlite3 /opt/domoticz/domoticz/db  '.backup myBackup.db'
and copy the myBackup to your target location.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Script to backup to a Synology NAS

Post by EddyG »

My script has to be changed. See here: viewtopic.php?f=6&t=34253&p=260598#p260598
I my case (busy system) sometimes the methode of "backupdatabase.php" leads to a system freeze or halt.
My problem is solved by change that line in

Code: Select all

/usr/bin/sqlite3 $SOURCEPATH$SOURCEFILE ".backup $MOUNTPOINT/$BACKUPFILE"
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest