Daily backup to external hdd - Suggestion.

[Duivleo]

Hello,

I am not quite sure where to direct this topic to; but most likely the moderator knows to whom to send this mail to.

I am using the Taskplanner of Synology DAS to make a daily back-up of the Domoticz.db to a different drive, as per Wiki "Daily backup to external hdd".

It works fine; except for the deleting of files older then x days.

The script in the wiki-article states something like:
## Delete files older then 7 days
/usr/bin/find /volume1/Back-ups/Domoticz-backup/ -name '*.db' -type f -mtime +7 -delete

The term "-mtime" should be replaced with "-atime", since -mtime refers to the date the file was last modified. The backups are normally not modified, but only made. By using -atime, the "made date" is used and the script works fine and files older then 7 days are deleted.

Hope this will help.
Regards,
Leo

[erem]

i had something similar to

Code: Select all

/usr/bin/find /volume1/Back-ups/Domoticz-backup/ -name '*.db' -type f -mtime +7 -delete

in my backup.
However, if your backup fails for 7 consecutive days you lose all backups if this statement runs.
i now use

Code: Select all

RETAIN=4
RET=$(($RETAIN-1))
echo "Successful backup, retaining last $RET backup files"
ls -t $DIR/*.gz | tail -n +$RETAIN | xargs -r rm

to retain the last x backups

just my $.02

[madpatrick]

I'm using rsnapshot. https://rsnapshot.org/download.html
Not sure if this is working on your distro

[waltervl]

The term "-mtime" should be replaced with "-atime", since -mtime refers to the date the file was last modified. The backups are normally not modified, but only made. By using -atime, the "made date" is used and the script works fine and files older then 7 days are deleted.

Are you saying it did not work with -mtime? Because atime is the time last accessed. And the backups are also not accessed...
What does ls -l in the folder say? Because that is showing the mtime.

[Duivleo]

The script as detailed in the WIKI was officially for a Pi. I have adapted it to run on a Synology NAS which I use.
Here, the -mtime did not work, but no fault-messages appeared when using Putty (via the NAS). See also later.

That made me to Google a bit more.
I found this:
https://www.hostinger.com/tutorials/how ... -in-linux/
QUOTE:
Searching by Time
If you want to search for files based on when they were accessed and modification time footprints. Linux keeps track of the files using these three timestamps.

Access Time (-atime) – when the file was either read or written into.
Modification Time (-mtime) – when the file was modified.
Change Time (-ctime) – when the file’s meta-data was updated.
This option has to be used with a number that specifies how many days passed since the file was accessed, modified or changed.
UNQUOTE.

Accoording to me: -atime refers to time when EITHER the file was assesed OR when data was written into it.
For me "data was written into it" is at the very start, i.e. at creation of the file.
Changinging contained data for me qualifies as 'modification'.

As via Putty no error messages were received when using -mtime, it appeared to me that -mtime is a valid option to use
but that, in my case, no files satisfied that very criteria of being modified.

After I changed -mtime to -atime, the script worked perfectly well.

Please do not get me wrong: I am not critisizing the author of the wiki; I am merely trying to improve that what
gave me headaches since I could not pinpoint the reason of the script not working.

Hopefully others will benefit from this.
For further details I refer to:
https://www.synology-forum.nl/synology- ... icseen#new
(The -l option I did not not use; I am not so profound with Linux, sorry.)

Regards,
Leo

[waltervl]

No problem Leo, I just wanted to be sure before I change something on the wiki.
Thank you for the explanation! I will change the wiki later this day.

[waaren]

The term "-mtime" should be replaced with "-atime", since -mtime refers to the date the file was last modified. The backups are normally not modified, but only made. By using -atime, the "made date" is used and the script works fine and files older then 7 days are deleted.

Are you saying it did not work with -mtime? Because atime is the time last accessed. And the backups are also not accessed...
What does ls -l in the folder say? Because that is showing the mtime.

I just tested this on my NAS (DSM 6.2) and the find command does work exactly the same as on my Raspberry (buster) and NUC (bullseye)

Code: Select all

/usr/bin/find --version
find (GNU findutils) 4.4.2
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Built using GNU gnulib version e5573b1bad88bfabcda181b9e0125fb0c52b7d3b
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION FTS() CBO(level=0)

tested with directory findTest

Code: Select all

ls -l findTest/*
-rwxrwxrwx 1 waaren users   151 Oct 27  2016 findTest/AddTime.sh
-rwxrwxrwx 1 waaren users  1828 Jul 11  2017 findTest/appenddomoticzlog.sh
-rwxrwxrwx 1 waaren users  2562 May  8  2020 findTest/Backup.sh
-rwxr-xr-x 1 root   root   3661 Sep 18  2019 findTest/CheckAndrewFilesBackup.sh
-rwxrwxrwx 1 waaren users  1459 Sep 25  2018 findTest/CheckDomoticzMemory.sh
-rwxrwxrwx 1 waaren users  1254 Feb 17  2017 findTest/CheckDomoticzPidfile.sh
-rwxrwxrwx 1 waaren users  2079 Oct  7  2017 findTest/CheckDot.sh
-rwxrwxrwx 1 waaren users   442 Oct  3  2017 findTest/CheckFirewall.sh
-rwxrwxrwx 1 waaren users  3034 May 24  2018 findTest/CheckMobiles.sh
-rwxrwxrwx 1 waaren users   557 Dec  5  2016 findTest/CheckMyIP.sh
-rwxrwxrwx 1 waaren users  1807 Oct  5  2018 findTest/CheckMySQL.sh
-rwxr-xr-x 1 root   root   3230 Jan  7  2019 findTest/CheckNASBackupServer.sh
-rwxrwxrwx 1 waaren users   807 Apr  5  2017 findTest/CheckNASBackup.sh
-rwxrwxrwx 1 waaren users   806 Dec  7  2016 findTest/CheckNASConfigBackup.sh
-rwxrwxrwx 1 waaren users  3202 Dec 18  2018 findTest/CheckNASFilesBackup.sh
-rwxr-xr-x 1 root   root    825 Oct 21  2018 findTest/CheckProcesses.sh
-rwxrwxrwx 1 waaren users   570 Oct 30  2016 findTest/Cleanup.sh
-rwxrwxrwx 1 waaren users  1256 Oct 18  2018 findTest/CreateDomoticzBinaryBackup.sh
-rwxrwxrwx 1 waaren users  2425 Feb 11  2017 findTest/CreateExcelInput.sh
-rwxrwxrwx 1 waaren users   793 Sep  9  2017 findTest/DomoticzLogCheck.sh
-rwxr-xr-x 1 root   root   3626 Dec  8  2018 findTest/ff.sh
-rwxrwxrwx 1 waaren users   457 Dec  5  2016 findTest/GetDomoticzlog.sh
-rwxrwxrwx 1 waaren users  4304 Sep  5  2017 findTest/GetEvoTemp.sh
-rwxrwxrwx 1 waaren users   575 Jul 11  2017 findTest/GetMyIP.sh
-rwxrwxrwx 1 waaren users   890 Feb 26  2017 findTest/GetPlexHistory.sh
-rwxrwxrwx 1 waaren users   212 Oct 27  2016 findTest/GrepDomoticzLog.sh
-rwxrwxrwx 1 waaren users  2078 Sep  8 09:21 findTest/HouseKeeping.sh
-rwxrwxrwx 1 waaren users  3689 Apr 28  2020 findTest/Init.sh
-rwxr--r-x 1 root   root    955 Jul 15  2019 findTest/iscsiAlive.sh
-rwxr-xr-x 1 waaren users   166 Oct 27  2016 findTest/LogScriptUsage.sh
-rwxrwxrwx 1 waaren users   544 Sep  9  2017 findTest/MonitReload.sh
-rwxrwxrwx 1 waaren users   850 Feb 10  2017 findTest/PreventConflict.sh
-rwxrwxrwx 1 waaren users  4749 Feb 23  2017 findTest/pushover.sh
-rwxrwxrwx 1 waaren users  1708 Dec  1  2017 findTest/ReadAfvalwijzer.sh
-rwxrwxrwx 1 waaren users    94 Oct 27  2016 findTest/ReadDomoticzLog.sh
-rwxrwxrwx 1 waaren users  1204 Feb 10  2017 findTest/ReadHomeWizard.sh
-rwxrwxrwx 1 waaren users   928 May 28  2017 findTest/ReadPublicHoliday.sh
-rwxrwxrwx 1 waaren users  2942 Oct 27  2016 findTest/ReadWriteEnvoy.sh
-rwxrwxrwx 1 waaren users   853 Apr  9  2017 findTest/ResetDomoticz.sh
-rwxrwxrwx 1 waaren users  2041 Apr  7  2020 findTest/RootfsMySQL.sh
-rwxr--r-x 1 waaren waaren 2162 Sep 18  2018 findTest/rsync.sh
-rwxrwxrwx 1 waaren users   415 Mar  8  2017 findTest/SafeDomoticzlogIfCrashed.sh
-rwxrwxrwx 1 waaren users   243 Dec 12  2016 findTest/setsleep.sh
-rwxr-xr-x 1 root   root    931 Oct  7  2017 findTest/ShutdownPI.sh
-rwxrwxrwx 1 waaren users   294 Dec  4  2016 findTest/SonosBridge.sh
-rwxrwxrwx 1 waaren users  1134 Jan 20  2017 findTest/StartDomoticz.sh
-rwxrwxrwx 1 waaren users  1109 Feb 13  2017 findTest/start-stop-CheckDomoticzPidfile.sh
-rwxrwxrwx 1 waaren users  1902 Mar 28  2017 findTest/start-stop-Fail2ban.sh
-rwxrwxrwx 1 waaren users  2206 Jan 21  2018 findTest/start-stop-Habridge.sh
-rwxr-xr-x 1 root   root   1944 Jan 21  2017 findTest/start-stop-Plex.sh
-rwxrwxrwx 1 waaren users  2437 Jun 15  2020 findTest/TempAndVolumeInfo2Domoticz.sh
-rwxr-xr-x 1 root   root   4282 Apr 28  2020 findTest/TempAndVolumeInfo2PI-1_Domoticz.sh
-rw-r--r-- 1 root   root   3479 Mar 23  2018 findTest/TempAndVolumeInfo2PI-3_Domoticz.sh
-rwxrwxrwx 1 waaren users   585 Dec 12  2016 findTest/turn_SonosRadio_on.sh
-rwxrwxrwx 1 waaren users   586 Dec 20  2016 findTest/turn_tv_off.sh
-rwxrwxrwx 1 waaren users   732 Feb 10  2017 findTest/Write2MySQL.sh

Change modtime to now for some files

Code: Select all

touch -m findTest/C*
ls -l findTest/C*
-rwxr-xr-x 1 root   root  3661 Feb 14 18:11 findTest/CheckAndrewFilesBackup.sh
-rwxrwxrwx 1 waaren users 1459 Feb 14 18:11 findTest/CheckDomoticzMemory.sh
-rwxrwxrwx 1 waaren users 1254 Feb 14 18:11 findTest/CheckDomoticzPidfile.sh
-rwxrwxrwx 1 waaren users 2079 Feb 14 18:11 findTest/CheckDot.sh
-rwxrwxrwx 1 waaren users  442 Feb 14 18:11 findTest/CheckFirewall.sh
-rwxrwxrwx 1 waaren users 3034 Feb 14 18:11 findTest/CheckMobiles.sh
-rwxrwxrwx 1 waaren users  557 Feb 14 18:11 findTest/CheckMyIP.sh
-rwxrwxrwx 1 waaren users 1807 Feb 14 18:11 findTest/CheckMySQL.sh
-rwxr-xr-x 1 root   root  3230 Feb 14 18:11 findTest/CheckNASBackupServer.sh
-rwxrwxrwx 1 waaren users  807 Feb 14 18:11 findTest/CheckNASBackup.sh
-rwxrwxrwx 1 waaren users  806 Feb 14 18:11 findTest/CheckNASConfigBackup.sh
-rwxrwxrwx 1 waaren users 3202 Feb 14 18:11 findTest/CheckNASFilesBackup.sh
-rwxr-xr-x 1 root   root   825 Feb 14 18:11 findTest/CheckProcesses.sh
-rwxrwxrwx 1 waaren users  570 Feb 14 18:11 findTest/Cleanup.sh
-rwxrwxrwx 1 waaren users 1256 Feb 14 18:11 findTest/CreateDomoticzBinaryBackup.sh
-rwxrwxrwx 1 waaren users 2425 Feb 14 18:11 findTest/CreateExcelInput.sh

Number of files in findTest before find -delete

Code: Select all

ls -l findTest/* | wc -l
56

find -delete command

Code: Select all

find /volume1/Synology/Domotica/findTest -name "*" -mtime +5 -delete

Number of files in findTest after find -delete

Code: Select all

ls -l findTest/* | wc -l
16

[waltervl]

The term "-mtime" should be replaced with "-atime", since -mtime refers to the date the file was last modified. The backups are normally not modified, but only made. By using -atime, the "made date" is used and the script works fine and files older then 7 days are deleted.

Are you saying it did not work with -mtime? Because atime is the time last accessed. And the backups are also not accessed...
What does ls -l in the folder say? Because that is showing the mtime.

I just tested this on my NAS (DSM 6.2) and the find command does work exactly the same as on my Raspberry (buster) and NUC (bullseye)

Thank you for checking this out on the Synology. I deleted the warning again on the wiki.

[JanJaap]

Hey,

I've been trying this script to run on my RPi 3B but it drives me crazy. I tweaked it a bit to get some logging on command line:

Code: Select all

    ### END OF USER CONFIGURABLE PARAMETERS
    TIMESTAMP=`/bin/date +%Y%m%d%H%M`
	echo "timestamp is $TIMESTAMP"
    BACKUPFILE="domoticz_$TIMESTAMP.db" # backups will be named "domoticz_YYYYMMDDHHMMSS.db.gz"
    BACKUPFILEGZ="$BACKUPFILE\.gz"
	echo "BACKUPFILE is $BACKUPFILE"
	echo "BACKUPFILEGZ is $BACKUPFILEGZ"
    ### create backup, ZIP it and start Domoticz again
    echo create backup, ZIP it and start Domoticz again
    #/usr/bin/curl -s http://$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
    gzip -9 /tmp/$BACKUPFILE
    ### Send to Network disk through FTP
    echo Send to Network disk through FTP
    curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/media/hdd/Domoticz_backup/"				
    ### Remove temp backup file
    /bin/rm /tmp/$BACKUPFILEGZ
    ### Done!
    echo Done!

and it results in the following logging:

Code: Select all

start
timestamp is 202311112125
.dbKUPFILE is domoticz_202311112125
.db\.gzILEGZ is domoticz_202311112125
create backup, ZIP it and start Domoticz again
: No such file or directory12125
Send to Network disk through FTP
'!b\.gzan't open '/tmp/domoticz_202311112125
curl: try 'curl --help' or 'curl --manual' for more information
/bin/rm: cannot remove '/tmp/domoticz_202311112125'$'\r''.db\.gz'$'\r\r': No such file or directory
Done!

it looks like there is some hidden character at the end of the timestamp creation that puts the pointer back to front , so the line

Code: Select all

echo "BACKUPFILE is $BACKUPFILE"

the file extension is printed in the front of the logline.

Anyone some ideas?