Page 1 of 2

Script to backup to a Synology NAS

Posted: Tuesday 27 October 2020 20:56
by Johan1974
Hello I used the script

Code: Select all

#!/bin/bash
#
# 20201028 Modified to replace CR/LF (windows) to LF (Linux)
#
# LOCAL/FTP/SCP/MAIL PARAMETERS
#
SERVER="192.168.1.11"                                      # IP of Synology NAS, used for ftp
USERNAME="Domoticz"                                        # FTP username of Network disk used for ftp
PASSWORD="12345"                                           # FTP password of Network disk used for ftp
DESTDIR=" /tmp/"                                           # used for temorarily storage
DESTDIRNAS="/homes/domoticz/backup/"                       # Path to your Synology NAS backup folder
DOMO_IP="192.168.1.12"                                     # Domoticz IP
DOMO_PORT="8080"                                           # Domoticz port

### END OF USER CONFIGURABLE PARAMETERS

TIMESTAMP=`/bin/date +%Y%m%d%H%M%S`
BACKUPFILE="domoticz_$TIMESTAMP.db" # backups will be named "domoticz_YYYYMMDDHHMMSS.db.gz"
BACKUPFILEGZ="$BACKUPFILE".gz

#
### Create backup and ZIP it
#
/usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
gzip -9 /tmp/$BACKUPFILE
tar -zcvf /tmp/domoticz_scripts_$TIMESTAMP.tar.gz /home/pi/domoticz/scripts/
tar -zcvf /tmp/telegram_scripts_$TIMESTAMP.tar.gz /home/pi/tg/scripts/

#
### Send to Network disk through FTP
#
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
curl -s --disable-epsv -v -T"/tmp/domoticz_scripts_$TIMESTAMP.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
curl -s --disable-epsv -v -T"/tmp/telegram_scripts_$TIMESTAMP.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"

#
### Remove temp backup file
#
/bin/rm /tmp/$BACKUPFILEGZ
/bin/rm /tmp/domoticz_scripts_$TIMESTAMP.tar.gz
/bin/rm /tmp/telegram_scripts_$TIMESTAMP.tar.gz

#eoj
## Done!

I save this script at: /home/pi/domoticz/scripts whit the name domoticz_backup.sh

When I test the script with:
sudo /home/pi/domoticz/scripts/domoticz_backup.sh

I see the next error:
sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory

What is going wrong.??

Re: Script to backup to a Synology NAS

Posted: Tuesday 27 October 2020 21:23
by cyberclwn
Hi,

does "/bin/bash" exist?

Code: Select all

ls -ls /bin/bash
And is domoticz_backup.sh executable?

Code: Select all

ls -ls /home/pi/domoticz/scripts/domoticz_backup.sh
Check if there is a x in the line ... rwx
if not, change it with

Code: Select all

chmod +x /home/pi/domoticz/scripts/domoticz_backup.sh

Re: Script to backup to a Synology NAS

Posted: Tuesday 27 October 2020 22:10
by Johan1974
Yes,

Code: Select all

 844 -rwxr-xr-x 1 root root 863400 Jan 15  2017 /bin/bash 

Code: Select all

 4 -rwxr-xr-x 1 pi pi 1604 Oct 27 21:57 /home/pi/domoticz/scripts/domoticz_backup.sh 

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 0:32
by waaren
Johan1974 wrote: Tuesday 27 October 2020 22:10 Yes,
@Johan1974

How should we read your "Yes" ?
Is everything OK now or does it mean the settings were already as suggested but the command still does not work?

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 10:08
by Johan1974
Sorry, I'm not clear.

I have CMOD de file domoticz_backup to 777
When I use:

Code: Select all

 ls -ls /home/pi/domoticz/scripts/domoticz_backup.sh
the answer is

Code: Select all

 4 -rwxrwxrwx 1 pi pi 1609 Oct 28 09:50 /home/pi/domoticz/scripts/domoticz_backup.sh
When I use

Code: Select all

 sudo sh /home/pi/domoticz/scripts/domoticz_backup.sh
the answer is

Code: Select all

 tar: Removing leading `/' from member names
tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: Removing leading `/' from member names
tar: /home/pi/tg/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
'!b.gzCan't open '/tmp/domoticz_20201028100108
curl: try 'curl --help' or 'curl --manual' for more information
* Illegal characters found in URL
* Closing connection -1
* Illegal characters found in URL
* Closing connection -1
/bin/rm: cannot remove ‘/tmp/domoticz_20201028100108\r.db.gz\r\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/domoticz_scripts_20201028100108\r.tar.gz\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/telegram_scripts_20201028100108\r.tar.gz\r’: No such file or directory
The files domoticz_20201028100108.db.gz and telegram_scripts_20201028100108.tar.gz are made in the tmp directory on the raspberry. But not upload to the NAS

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 10:26
by waaren
Johan1974 wrote: Wednesday 28 October 2020 10:08 tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
Did you edit the file with a windows editor ? (asking this because of the \r in your log)
Google for "windows editor unix line breaks"

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 10:58
by Johan1974
That's right, and I can remember making this mistake more than once .. :(
I started over from the beginning, with a new domoticz_backup.sh file. I make script executable with

Code: Select all

  sudo chmod +x /home/pi/domoticz/scripts/domoticz_backup.sh 
Then I run the script with:

Code: Select all

 sudo /home/pi/domoticz/scripts/domoticz_backup.sh 
the answer is:

Code: Select all

 sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory 


when I use

Code: Select all

 sudo sh /home/pi/domoticz/scripts/domoticz_backup.sh 

the answer is:

Code: Select all

 tar: Removing leading `/' from member names
tar: /home/pi/domoticz/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
tar: Removing leading `/' from member names
tar: /home/pi/tg/scripts/\r: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
'!b.gzCan't open '/tmp/domoticz_20201028104943
curl: try 'curl --help' or 'curl --manual' for more information
* Illegal characters found in URL
* Closing connection -1
* Illegal characters found in URL
* Closing connection -1
/bin/rm: cannot remove ‘/tmp/domoticz_20201028104943\r.db.gz\r\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/domoticz_scripts_20201028104943\r.tar.gz\r’: No such file or directory
/bin/rm: cannot remove ‘/tmp/telegram_scripts_20201028104943\r.tar.gz\r’: No such file or directory
I do something wrong:(
what is the difference between sudo and sudo sh
I read the manual on https://www.domoticz.com/wiki/Daily_b ... ernal_hdd and follow the instructions

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 11:40
by waaren
Johan1974 wrote: Wednesday 28 October 2020 10:58 what is the difference between sudo and sudo sh
I read the manual on https://www.domoticz.com/wiki/Daily_b ... ernal_hdd and follow the instructions
sudo does use your default shell (probably bash ?) and sudo sh force the use of the sh shell

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 11:51
by Egregius
Why not use a script on your nas with rsync of the complete domoticz folder? Much easier.
If /tmp is on your SD card it'll ruin it.
If /tmp is in tmpfs you can hit memory limits.

Also, using rsync for the complete domoticz folder gives you the advantage of making versions possible and you'll have a backup of your scripts etc.

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 11:53
by Johan1974
Okay, so I use to run the script:

Code: Select all

  sudo /home/pi/domoticz/scripts/domoticz_backup.sh  
that is also what I read in the manual on the website.
but then nothing happens, only the following message:

Code: Select all

  sudo: unable to execute /home/pi/domoticz/scripts/domoticz_backup.sh: No such file or directory 
When asked by @cyberclwn whether bin / bash exists I get the answer :

Code: Select all

  844 -rwxr-xr-x 1 root root 863400 Jan 15  2017 /bin/bash  

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 12:34
by Johan1974
Egregius wrote: Wednesday 28 October 2020 11:51 Why not use a script on your nas with rsync of the complete domoticz folder? Much easier.
If /tmp is on your SD card it'll ruin it.
If /tmp is in tmpfs you can hit memory limits.

Also, using rsync for the complete domoticz folder gives you the advantage of making versions possible and you'll have a backup of your scripts etc.
There are more roads that lead to Rome. :) :)
I think it is important to have a back up because I already have a lot of data in domoticz.
Can I find a manual of this script.?

I still hope to find a solution to my first problem

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 13:05
by waaren
Johan1974 wrote: Wednesday 28 October 2020 12:34 I still hope to find a solution to my first problem
@Johan1974,

I modified your file in the opening post of this topic. It still contained CR/LF and with that the shell will not recognize the
#!/bin/bash (shebang) and does not know what to do with it.
Please copy and save it again and make sure no windows editor is touching it again. :)

And I fully agree with @Egregius. rsync is much better suited for the job.

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 14:20
by Johan1974
I started all over again and didn't use a windows editor this time.
And yes! It works now! It's very easy to use an editor, but I understand I have to do everything with Putty and ssh..
Thank you very much.

I am still interested in the script @Egregius uses. A total backup is always better.

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 14:30
by waaren
Johan1974 wrote: Wednesday 28 October 2020 14:20 I started all over again and didn't use a windows editor this time.
And yes! It works now! It's very easy to use an editor, but I understand I have to do everything with Putty and ssh..
II use Notepad++ on my Windows desktop and -laptop and that editor is perfectly OK with Unix files and it is opensource.
Also not unimportant; it is completely free :D

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 16:08
by EddyG
I use rsnapshot to take locally snapshots of a few dirs of my Raspberry to a local mounted USB-stick (Sandisk 32GB)
The snapshot is taken every 4 Hours for the duration of 4 weeks, kind of son father grandfather backup.
Every 5 minutes I backup only the Domoticz database to my Synology NAS and keep 864 versions eg. 3 days. You could extend that.
The database is saved with date and time and database version in the name. Logging is done in /var/log
Below my script. sqlite3 and bc should be installed and Synology should support nfs.
Run it as root from cron every 5 minutes. On a Pi 3B+ is takes about 6-10 seconds.
All my Raspberry's are backuped this way hence the $HOSTNAME in the path.

Code: Select all

#!/usr/bin/env /bin/bash

SCRIPTNAME=`basename "$0"`
SCRIPTNAME=${SCRIPTNAME%.*}
LOGFILE=/var/log/$SCRIPTNAME.log
ERRORLOG=/var/log/$SCRIPTNAME_ERROR.log
HOSTNAME=$(hostname)
DOMO_IP="127.0.0.1"      # Domoticz IP
DOMO_PORT="8080"         # Domoticz port
REMOTEHOST="192.168.2.xxx"
REMOTEPATH="/volume1/RaspberryPi/Domoticz_DB/$HOSTNAME"
SOURCEFILE="domoticz.db"
SOURCEPATH="/home/pi/domoticz/"
MOUNTPOINT="/mnt/$SCRIPTNAME"
### END OF USER CONFIGURABLE PARAMETERS

# Start Timestamp
STARTTIME=$(date +"%s.%N")

# Place in the log file information concerning the execution of this script
echo -n "Domoticz database backup start at " >> $LOGFILE
echo $(date) >> $LOGFILE

if [ ! -d $MOUNTPOINT ]; then
    sudo mkdir $MOUNTPOINT
fi

if ! mountpoint -q $MOUNTPOINT; then
    sudo mount -t nfs $REMOTEHOST:$REMOTEPATH $MOUNTPOINT
fi
if ! /bin/mountpoint -q $MOUNTPOINT; then
    touch $ERRORLOG
    chmod 0666 $ERRORLOG
    echo "Mount $MOUNTPOINT ERROR!!!" >> $ERRORLOG
    exit 99
fi

#Delete backups older than 3 days = 864 (288 per day is saved every 5 minutes)
cd $MOUNTPOINT
ls -tr | head -n -864 | xargs --no-run-if-empty rm

### Create backup
TIMESTAMP=`/bin/date +%Y%m%d%H%M%S`
DB_VERSION=`sqlite3 $SOURCEPATH$SOURCEFILE "Select nValue from Preferences WHERE Key = 'DB_Version'"`
BACKUPFILE=$HOSTNAME"_"$TIMESTAMP"_"$DB_VERSION".db" # backups will be named "domoticz_YYYYMMDDHHMMSS_ZZZ.db"

/usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > $MOUNTPOINT/$BACKUPFILE

sync
sudo /bin/umount -l $MOUNTPOINT

# End timestamp
ENDTIME=$(date +"%s.%N")

# Convert nanoseconds to milliseconds
# crudely by taking first 3 decimal places
TIMEDIFF=`echo "$ENDTIME - $STARTTIME" | bc | awk -F"." '{print $1"."substr($2,1,3)}'`
SEC=`echo "$TIMEDIFF % 60" | bc`

echo "Total of $SEC seconds elapsed." >> $LOGFILE
echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" >> $LOGFILE

### Done!

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 17:57
by Egregius
Johan1974 wrote: Wednesday 28 October 2020 14:20 I am still interested in the script @Egregius uses. A total backup is always better.
This is executed at my nas (only part of the script copied because it backups multiple servers)
It'll create a versioned backup for every day it has run. Because of the --link-dest option only changed files occupy space. Files that didn't change are actually symlinks. Don't worry about that, linux takes care of that perfectly.
The great advantage of this is that you can go back in time to any version you want. And that you don't stress your raspberry. The NAS can do perfectmy what it does best: keeping backups.

Code: Select all

#!/bin/sh
NOW=$(date +"%Y-%m-%d")
BPATH=/volume1/homes/guy/backup
RUNLOG=/volume1/homes/guy/log/$NOW.txt
NAME="Domoticz"
echo  ------------------- START $NAME -- $(date) | tee -a $RUNLOG
SOURCE="[email protected]:/domoticz/"
DESTINATION="$BPATH/$NAME/$NOW"
mkdir -p "$DESTINATION"
rsync -aP -e "ssh -i /root/.ssh/home" --stats --delete-after --links --ignore-errors --link-dest="../__prev/" "$SOURCE" "$DESTINATION" | tee -a $RUNLOG
rm -f "$BPATH/$NAME/__prev"
ln -s "$BPATH/$NAME/$NOW" "$BPATH/$NAME/__prev"
echo  ------------------- END  $NAME -- $(date) | tee -a $RUNLOG

Re: Script to backup to a Synology NAS

Posted: Wednesday 28 October 2020 18:47
by EddyG
Nice script working from the other side. :)
Just copying or rsyncing is a little bit dangerous on an open database.
A more save way is to use "backupdatabase.php" it has some provisions in the domoticz program for looking for busy and locks, and it Optimizes and Vacuums the database before a copy is made. The copy is made by the sqlite3 library.

Re: Script to backup to a Synology NAS

Posted: Thursday 29 October 2020 7:08
by Egregius
Never had an issue with it, restored multiple times. Works flawless.

Re: Script to backup to a Synology NAS

Posted: Thursday 29 October 2020 7:50
by waaren
Egregius wrote: Thursday 29 October 2020 7:08 Never had an issue with it, restored multiple times. Works flawless.
Lucky you :) And to be honest I do the same...

BUT

The SQLite developers say this:

How To Corrupt An SQLite Database File

1.2. Backup or restore while a transaction is active
Systems that run automatic backups in the background might try to make a backup copy of an SQLite database file while it is in the middle of a transaction. The backup copy then might contain some old and some new content, and thus be corrupt.

The best approach to make reliable backup copies of an SQLite database is to make use of the backup API that is part of the SQLite library. Failing that, it is safe to make a copy of an SQLite database file as long as there are no transactions in progress by any process. If the previous transaction failed, then it is important that any rollback journal (the *-journal file) or write-ahead log (the *-wal file) be copied together with the database file itself.
_____________________________________

So why take any chances? It is safer, quite easy and very quick to do something like

Code: Select all

sudo sqlite3 /opt/domoticz/domoticz/db  '.backup myBackup.db'
and copy the myBackup to your target location.

Re: Script to backup to a Synology NAS

Posted: Thursday 12 November 2020 8:51
by EddyG
My script has to be changed. See here: viewtopic.php?f=6&t=34253&p=260598#p260598
I my case (busy system) sometimes the methode of "backupdatabase.php" leads to a system freeze or halt.
My problem is solved by change that line in

Code: Select all

/usr/bin/sqlite3 $SOURCEPATH$SOURCEFILE ".backup $MOUNTPOINT/$BACKUPFILE"