Option to disable hourly backups

[marin849]

My database is quite large, around 80 MB. Every time the database is backuped Domoticz is unresponsive for 1-2 minutes and it adds 2 GB of writes to the SD-card every day!

The backup is done monthly, daily and hourly. Every hour is quite often and I don't care if I would lose some hours of data in the event of a failure.

I propose that the hourly backup is removed or that there is a checkbox in settings where "hourly backups" can be enabled or disabled.

[Solderbro]

+1
I suggest to place a seconds or minute config option, where 0 means backups off. This amount of writes will destroy any today sd card very rapidly, because they don't have a wear level controller and the raspbian image is written with 4k filesystem blocks.

domoticz@domosys:~$ sudo tune2fs -l /dev/sda1 | grep -i 'block size'
Block size: 4096

Solderbro

[waaren]

My database is quite large, around 80 MB. Every time the database is backuped Domoticz is unresponsive for 1-2 minutes and it adds 2 GB of writes to the SD-card every day!
The backup is done monthly, daily and hourly. Every hour is quite often and I don't care if I would lose some hours of data in the event of a failure.
I propose that the hourly backup is removed or that there is a checkbox in settings where "hourly backups" can be enabled or disabled.

In fact when you switch on the automatic backup in domoticz the backup is always done hourly. The daily, monthly, Yearly backups are just copies from the hourly backup. The backup only contains domoticz.db so it's quite easy to make this backup using a cron job. Just use a sudo cp <domoticz dir>/domoticz.db <target dir>/<target name> If you have this cron job and tested it successfully you can switch off the automatic backup in domoticz.

Please also have a look at this post where an explanation is given on the main reason why the domoticz database can be (but not need to be) quite large and how it can be made smaller.

[niki_lauda]

My database is quite large, around 80 MB. Every time the database is backuped Domoticz is unresponsive for 1-2 minutes and it adds 2 GB of writes to the SD-card every day!

The backup is done monthly, daily and hourly. Every hour is quite often and I don't care if I would lose some hours of data in the event of a failure.

I propose that the hourly backup is removed or that there is a checkbox in settings where "hourly backups" can be enabled or disabled.

Made a script which backups on a daily bases. Based on this script. https://www.raspberrypi.org/forums/view ... 20#p862755
Change the /tmp/$BACKUPFILE to the destination

and don't remove the backup file
Change the last lines
### Remove temp backup file
###/bin/rm /tmp/$BACKUPFILEGZ
### Done!

[EddyG]

And if you use that script, do not gzip it, because it has a big impact on cpu usage on such a large database.
Compact your database every week, it makes the database somewhat smaller.
Just copying the database by script is the most efficient way, like @waaren suggested. Is takes only a few seconds.
My database of only 2 Mb takes only 2-3 seconds to be copied over the network to my NAS every 5 min.

[marin849]

Thanks for the input.
About a year or so ago when I always compiled Domoticz from source I just commented out the hourly backup in the source code. It worked fine and I got daily and monthly backups anyway.

Is it always reliable to use a simple copy operation on the active database file? I mean, it will be written to before the copy is finished.

I have 412 devices in Domoticz so the database is quite big even after shrinking. But it is working very well, the temporary database files is symlinked to /tmp which is in ram and does not write to the sd card directly.

[Egregius]

I just use a rsync script that is executed from nas to copy the whole domoticz folder. Works flawless and used it multiple times.
Even use it to backup a remote domoticz over the internet. Never had issues to restore it.

Here's part of the script, it has many other similar blocks to backup webserver, domoticz, domoticz webserver, a 2nd domoticz in the house, 3 raspberries wit RPIcam, a remote domoticz. The script is executed each time my Synology nas starts. The nas is set to start automatically each Friday to have at least a backup each week but is also started regularly to do other stuff.

In this example the first block does a backup of the domoticz folder. The second block a backup of the apache www folder. For the second block the timestamp of the last changed file is verified so no needless backup is done. For the domoticz folder this isn't needed because there the database is always changed anyway.
The backups are stored using symlinks to created versioned backups. I can always return to any previous state in time.
Because only the changed files occupy real disk space the size of the total backup is quite limited. The 284 versions between May 2018 and now only use 2.3GB instead of the 9.1GB it should be without using the symlinks.

Code: Select all

#!/bin/sh
NOW=$(date +"%Y-%m-%d")
BPATH=/volume1/homes/guy/backup
RUNLOG=/volume1/homes/guy/backup/__Logs/$NOW.txt
EXCLUDED=/volume1/homes/guy/backup/excludedfiles.txt

#------ BEGIN RPi's -------
NAME="domoticz"
echo  ------------------- START $NAME -- $(date) | tee -a $RUNLOG
SOURCE="root@domoticz:/domoticz/"
DESTINATION="$BPATH/$NAME/$NOW"
mkdir -p "$DESTINATION"
rsync -aP --exclude-from $EXCLUDED -e "ssh -i /root/.ssh/home" --stats --delete-after --links --ignore-errors --link-dest="../__prev/" "$SOURCE" "$DESTINATION" | tee -a $RUNLOG
rm -f "$BPATH/$NAME/__prev"
ln -s "$BPATH/$NAME/$NOW" "$BPATH/$NAME/__prev"
echo  ------------------- END  $NAME -- $(date) | tee -a $RUNLOG

NAME="apache"
echo  ------------------- START $NAME -- $(date) | tee -a $RUNLOG
SOURCE="root@domoticz:/var/www/"
DESTINATION="$BPATH/$NAME/$NOW"
LAST=$(ssh root@domoticz -i /root/.ssh/home "find /var/www/ -type f ! -name '*.cache' -printf '%T@\n' | sort -n | tail -1 | cut -f1- -d\" \"")
PREV=$(cat "$BPATH/timestamp_$NAME.txt")
echo $LAST>"$BPATH/timestamp_$NAME.txt"
if [ "$LAST" != "$PREV" ]
then
	mkdir -p "$DESTINATION"
	rsync -aP --exclude-from $EXCLUDED -e "ssh -i /root/.ssh/home" --stats --delete-after --links --ignore-errors --link-dest="../__prev/" "$SOURCE" "$DESTINATION" | tee -a $RUNLOG
	rm -f "$BPATH/$NAME/__prev"
	ln -s "$BPATH/$NAME/$NOW" "$BPATH/$NAME/__prev"
else
	echo Nothing to do | tee -a $RUNLOG
fi
echo  ------------------- END  $NAME -- $(date) | tee -a $RUNLOG

[pipiche]

I don't think the scripting is the right way as you might get some inconsistency and this is why the DB backup done by Domoticz is right.

However I do agree and share the fact that the hourly backup is quiet impacting the all system as during that time Domoticz is not responding at all (as the DB is most-likely locked).

In such I would be pleased to have only daily backup (some time during the night) and would be glad to be able to disable the hourly backup.

Is that something the Domoticz team would consider to implement ?

[waaren]

I don't think the scripting is the right way as you might get some inconsistency

If you use

Code: Select all

myDate=$(date "+%Y%m%d%H%M%S")
sqlite3 <domoticz dir>/domoticz.db ".backup domoticz.$myDate"

you will not have a higher risk on inconsistencies compared to the domoticz internal backup command.

[pipiche]

I don't think so. but using the Domoticz API yes: http://$DOMO_IP:$DOMO_PORT/backupdatabase.php

[waaren]

I don't think so. but using the Domoticz API yes: http://$DOMO_IP:$DOMO_PORT/backupdatabase.php

Do you have any arguments why you think the sqlite native .backup command would not be safe ?

[pipiche]

From my end Sqlite3 is not Multiprocess capable. Maybe when doing the .backup (as it is read ) will work especially as I think Domoticz use the WAL mode.

[wkossen]

this is how i do it...

Code: Select all

#!/bin/bash
#########################
### Author: Willem Kossen  #
#########################
#
# Make Backup of Domoticz Database
#
DOMO_IP=192.168.x.x
DOMO_PORT=8080
BACKUP_NAME=BackupDomoDatabase
DATE=`eval date +%Y%m%d`
BACKUP_NAME+="_$DATE.db"
BACKUP_PATH=/mnt/NAS/Backups/
curl  http://$DOMO_IP:$DOMO_PORT/backupdatabase.php >$BACKUP_PATH/$BACKUP_NAME

add your IP or DNS name, check the port. change the names if you like, add times if you want. You could crontab that to run as infrequent as you like...

[pipiche]

I think that one is much better as you trigger the Dz api. Doing that in the middle of the night with 20s of Dz blocked is ok


Envoyé de mon iPhone en utilisant Tapatalk

[sincze]

this is how i do it...

Code: Select all

#!/bin/bash

curl  http://$DOMO_IP:$DOMO_PORT/backupdatabase.php >$BACKUP_PATH/$BACKUP_NAME

I indeed sometimes heard the famous words "LIGHTS NOT TURNING ON!", only just to realize this had something to do with the hourly backup of a 90 MB file. I noticed it myself only 2 days ago at 06.00 and 08.00 when visiting the little boys room.
Instantly started digging the forum and implemented this script to see if it also fixes an api issue I have.
I fire that API at domoticz at 00:00 , sometimes it succeeds but many times I receives a time-out

The 90 MB DB is since 2014. I just spun up a homeassistant instance and that reached 32 GB within 10 days of running!
So I won't complain about the optimization Domoticz already has.

[waltervl]

The hourly backup can be disabled by removing the hourly folder in domoticz/backups. Then you only get the daily and monthly backup.

Also check the log history settings especially the Lights/switches log setting as those can become very big if it has a long retention period eg 30 days.

[sincze]

The hourly backup can be disabled by removing the hourly folder in domoticz/backups. Then you only get the daily and monthly backup.

Also check the log history settings especially the Lights/switches log setting as those can become very big if it has a long retention period eg 30 days.

You really are the impersonation of a walking Domoticz wiki

[joro75]

The hourly backup can be disabled by removing the hourly folder in domoticz/backups. Then you only get the daily and monthly backup.

I have tried to remove the 'hourly' directory in domoticz/backups/, however that does NOT disable the hourly backup. The directory is automatically recreated again by Domoticz. I also checked the source code implementation, and that indeed confirms that if the 'hourly' directory is not existing, it will be recreated.

What is working is:

1. Remove the 'hourly' directory from the domoticz/backups directory
2. Create a file named 'hourly' in the same directory. (For example by using: `touch hourly`)

With this configuration, Domoticz will detect that the 'hourly' directory is not available, and will try to create it, which doesn't succeed as a file with the same name is already present. Also creating the backup file itself will cause an error, which is nicely detected by Domoticz, and is shown as an error in the Log. No further problems or side effects are present. The 'daily' and 'montly' backups are still created, and are not disabled. It is possible to disable the 'daily' and 'monthly' backups with the same trick.

[waltervl]

Thanks for the feedback. I was just telling what some other user told somewhere on the forum how he disabled the hourly backup. Appearrantly he did not tell it correctly.

[solarboy]

It still would be nice as an option in settings instead of forum search>discover solution>login via ssh/vnc>delete folder (as root)>create file (as root) multiplied by however many users would like this option. Probably a very simple bit of coding for someone inside of the code.