backup every 12 hours on qnap ftp

[guantolento]

good evenig. actually i use the domoticz on my rpi2, i have some script LUA made with the built in domoticz and some script in BASH in another folder.
i want to create a script to save on an external ftp a full backup. in the wiki i can see the procedure to create the script, but i don't understand what folder i need to backup. What do you recommend ? a full SD backup or some single folder ? thanks a lot.

[Egregius]

I only backup the complete domoticz folder to a versioned system with rsync.

[guantolento]

How you can save in this way? ???what do you setting in domoticz? ????

Inviato dal mio GT-I9301I utilizzando Tapatalk

[emme]

I only backup the complete domoticz folder to a versioned system with rsync.

I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example )

[Egregius]

rsync doesn't seem to have issues with that.

Code: Select all

SOURCE="/volume1/@appstore/domoticz/"
DESTINATION="/volume1/homes/guy/backup/domoticz"
LOGFILE=/volume1/web/secure/backups/logs/domoticz.$NOW.txt
echo ------------------------------------ START domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
rsync -aP --exclude-from '/volume1/homes/guy/backup/excludedfiles.txt' --stats --delete-after --ignore-errors --links --link-dest="$DESTINATION/__prev/" "$SOURCE" "$DESTINATION/$NOW" | tee -a $LOGFILE
rm -f "$DESTINATION/__prev"
ln -s "$NOW" "$DESTINATION/__prev"
echo ------------------------------------ END  domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE

Creates a folder in DESTINATION for each day the script has run.
Folder size is only the updated files as the rest are symlinks.

I do have an exclusion list wich hold this:

Code: Select all

var/domoticz.db-shm
var/domoticz.db-wal
var/domoticz.log
var/domoticz.pid
var/ozwcp.poll.XXXXXX.xml
var/ozwcp.topo.XXXXXX
var/zwscene.xml
pi/domoticz/domoticz.db-shm
pi/domoticz/domoticz.db-wal
pi/domoticz/domoticz.log
pi/domoticz/Config/OZW_Log.txt
pi/domoticz/Config/zwscene.xml

2 paths in it because I also backup a Domoticz Pi from a friend

[sincze]

I can confirm rsync is working.... nicely
or you can run a .sh ...
- stop domoticz.
- backup
- start domoticz.

[Egregius]

No need to stop domoticz, backup is complete with rsync without stopping it.

[SweetPants]

No need to stop domoticz, backup is complete with rsync without stopping it.

isn't that dangerous with an open database?

[Egregius]

I restored at least 10 times without issues. And IF one got corrupt I'l pick one from the day before

[guantolento]

Merry christmas,
actually i take this action:
- disable the automatic backup from the web gui of domoticz
- create a bash script to backup every day at 13:00 "domoticz" folder
- create a bash script to backup every day at 13:30 "script" folder
- create a bash script to backup every day at 14:00 domoticz database
This script save to an ftp QNAP-GL01, the datastore of this is on 2 hard disk in raid 1 (mirror). The datastore of QNAP-GL01 is replicated on QNAP-GL02 by a process rsync every 12 hours. Someone can tell me if i need to save another folder from the RPI2 ???? thanks a lot.

[guantolento]

I only backup the complete domoticz folder to a versioned system with rsync.

I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example )

i don't have this problem, i save to my FTP

[litfiba]

Hi guys,

A simple script I use to bachup DB or entire Domoticz folder.. Daily, weekly, Monthly.. I keep only last three backup of each..

3 3 * * * /home/pi/domoticz/scripts/script_backup.sh J DB
3 4 * * 0 /home/pi/domoticz/scripts/script_backup.sh S DB
3 5 1 * * /home/pi/domoticz/scripts/script_backup.sh M DB
3 1 * * 1 /home/pi/domoticz/scripts/script_backup.sh S SYSTEM
3 2 2 * * /home/pi/domoticz/scripts/script_backup.sh M SYSTEM

#!/bin/bash
# LOCAL/FTP PARAMETERS
SERVER="192.168.0.xxx" # IP of Synology NAS, used for ftp
USERNAME="xxx" # FTP username of Network disk used for ftp
PASSWORD="xxx" # FTP password of Network disk used for ftp
DESTDIRNAS="/Public/domoticz/" # Path to your NAS backup folder
DOMO_IP="192.168.0.yyy" # Domoticz IP
DOMO_PORT="8080" # Domoticz port
DOMO_USER="yyy"
DOMO_PASS="yyy"
### END OF USER CONFIGURABLE PARAMETERS
param=$1 # 1st parameter will be for example 'J' for day, 'S' for week or 'M' for month
mode=$2 # 2nd parameter could be 'DB' for DB only backup or 'SYSTEM' for entire domoticz bacup
BACKUPFILE=$mode"_"$param"_1"
BACKUPFILEGZ="$BACKUPFILE".gz
##delete last - rename hold one
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS" -X "DELE "$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_2.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_1.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_2.gz"
### Create backup and ZIP it
#service domoticz.sh stop # not needed
case "$mode" in "DB") # case of DB only backup
/usr/bin/curl -s http://$DOMO_USER:$DOMO_PASS@$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
#service domoticz.sh start # not needed
gzip -9 /tmp/$BACKUPFILE
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILEGZ
;;
"SYSTEM") # case of entire domoticz backup
tar -zcvf /tmp/$BACKUPFILE.tar.gz /home/pi/domoticz/
#service domoticz.sh start # not needed
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILE.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILE.tar.gz
;;
esac
### Done!

[jammoju]

Hi litfiba, (or anyone else who can help)

I was hoping you could help and give some further explination as to how to get your script working for me.

I am very new to this so sorry if its a dumb question. I am having trouble trying to automate the backup via FTP to my QNAP NAS.

This is what i get when i try to run the script

* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> EPSV
* Connect data stream passively
< 500 EPSV not understood.
* Failed EPSV attempt. Disabling EPSV
> PASV
< 227 Entering Passive Mode (192,168,1,3,4,132).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1156
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> SIZE Backup
< 550 Backup: not a regular file.
> RETR Backup
< 550 Backup: Not a regular file
* RETR response: 550
* Remembering we are in dir "/Domoticz/"
* Connection #0 to host 192.168.1.3 left intact
curl: (78) RETR response: 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_2.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_2.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_1.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_1.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> PASV
* Connect data stream passively
< 227 Entering Passive Mode (192,168,1,3,4,133).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1157
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> STOR Backup
< 550 Backup: Not a regular file
* Failed FTP upload: 550
* Remembering we are in dir "/Domoticz/"
* Uploaded unaligned file size (0 out of 472 bytes)
* Connection #0 to host 192.168.1.3 left intact

As I said im very new to this but im trying to learn so any help is greatly appreciated.

Thank you...

[ropske]

rsync doesn't seem to have issues with that.

Code: Select all

SOURCE="/volume1/@appstore/domoticz/"
DESTINATION="/volume1/homes/guy/backup/domoticz"
LOGFILE=/volume1/web/secure/backups/logs/domoticz.$NOW.txt
echo ------------------------------------ START domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
rsync -aP --exclude-from '/volume1/homes/guy/backup/excludedfiles.txt' --stats --delete-after --ignore-errors --links --link-dest="$DESTINATION/__prev/" "$SOURCE" "$DESTINATION/$NOW" | tee -a $LOGFILE
rm -f "$DESTINATION/__prev"
ln -s "$NOW" "$DESTINATION/__prev"
echo ------------------------------------ END  domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE

Creates a folder in DESTINATION for each day the script has run.
Folder size is only the updated files as the rest are symlinks.

I do have an exclusion list wich hold this:

Code: Select all

var/domoticz.db-shm
var/domoticz.db-wal
var/domoticz.log
var/domoticz.pid
var/ozwcp.poll.XXXXXX.xml
var/ozwcp.topo.XXXXXX
var/zwscene.xml
pi/domoticz/domoticz.db-shm
pi/domoticz/domoticz.db-wal
pi/domoticz/domoticz.log
pi/domoticz/Config/OZW_Log.txt
pi/domoticz/Config/zwscene.xml

2 paths in it because I also backup a Domoticz Pi from a friend

I'v used your file,and all files are copied to Domoticz_Backup/Pi3_versions/domoticz/ on my nas
but i get this error(see below):

Number of files: 1,736 (reg: 1,577, dir: 159)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 45,274,332 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.057 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 41,240
Total bytes received: 5,333

sent 41,240 bytes received 5,333 bytes 3,449.85 bytes/sec
total size is 45,274,332 speedup is 972.12
ln: failed to create symbolic link ‘/mnt/nas/Domoticz_Backup/Pi3_versions/__prev’ -> ‘’: No such file or directory
------------------------------------ END domoticz -- 2018-03-26 20:19:37 UTC+2



Can you point me what i'm doing wrong? :s

[ropske]

and this is my code:
btw, the LOGFILE, is this created by this script? or must it be there?

Code: Select all

#!/bin/bash
SOURCE="/home/pi/domoticz"
DESTINATION="/mnt/nas/Domoticz_Backup/Pi3_versions"
LOGFILE=/var/www/html/secure/backups/logs/domoticz.$NOW.txt
echo ------------------------------------ START domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
rsync -aP --exclude-from '/var/www/html/secure/backups/excludedfiles.txt' --stats --delete-after --ignore-errors --links --link-dest="$DESTINATION/__prev/" "$SOURCE" "$DESTINATION/$NOW" | tee -a $LOGFILE
rm -f "$DESTINATION/__prev"
ln -s "$NOW" "$DESTINATION/__prev"
echo ------------------------------------ END  domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE

[ropske]

erm, with copying your files Egrerius, my domoticz isn't working anymore


domoticz offline: manifest fetch failed

[ropske]

these are some of the errors that are coming when i check developer mode:

HTML1300: Er heeft navigatie plaatsgevonden.
192.168.1.175:8080 (1,1)

AppCache maken met manifest: http://192.168.1.175:8080/html5.appcache.

CSS3121: De mediaquery -ms-viewport is afgeschaft.

2 SCRIPT1002: SCRIPT1002: Syntax error
main.js (1,1)

SCRIPT5022: Bootstrap's JavaScript requires jQuery
bootstrap.min.js (5,30)

SCRIPT5007: Object expected
bootbox.min.js (1,1978)

14 SCRIPT5009: 'jQuery' is not defined
jquery-ui.min.js (5,72)

SCRIPT5009: '$' is not defined
192.168.1.175:8080 (253,1)

5 HTML1500: De tag kan geen zichzelf sluitende tag zijn. Gebruik een expliciete afsluitende tag.
192.168.1.175:8080 (647,6)

SCRIPT5009: '$' is not defined

[Egregius]

The script isn't complete qs $NOW isn't defined.
The logfile is created automatically but the path must exist.
I'll check tomorrow for a complete script.

[ropske]

thanks mate

i meanwhile i got my original domoticz running again


i don't know what happened with copying your 'older' domoticz versions

[Egregius]

Oh, that's why it went offline, not the backup script.
Don't know, maybe other platform or file permissions.