Script to restart Domoticz if it crashes Topic is solved

[Derik]

yep i use only a cable and no wifi..
Are tere the solutions to?

[Egregius]

I mean the name of the network interface. You can find it with ifconfig. Useally that's eth0 or wlan0

[jannl]

Why stop the network interface? I use tons of unix/linux systems, including some pi's, I never stop/start a network interface unless I am working on it.

Verstuurd vanaf mijn SM-G930F met Tapatalk

[Egregius]

Because his device looses network connection. It's better to first try if a restart of network avoids the problem than rebooting.
http://www.domoticz.com/forum/viewtopic ... 20#p152494

[jannl]

But what is the cause of the network loss?

Verstuurd vanaf mijn SM-G930F met Tapatalk

[Toulon7559]

Really a pain-in-the-ass if a 'countermeasure' is not reliable.
Besides a 'user' cron with more lines, in the 'sudo' cron I put a simple, single line for periodic reboot, but apparently even that does not work.
Even worse, sometimes I see that after the ordered reboot the applications/scripts do not start ( 'obviously' that occurs most of the times at the 'night-reboot' of 04:11).

Code: Select all

11 4/12/20 * * * sudo reboot

Presently don't see another practical solution than a 'habit' to remotely/manually reboot when opening respectively closing my PC-sessions:
stupid/rude, but at least a frequent kick to the Raspberries for clean start.

For other part not really a crash of Domoticz, but more a matter of one or more lua-script(s) which stop(s) running.
Stop occurring semi-randomly, sometimes at 00:00 and sometimes elsewhere.
Nuisance if you want continuity e.g. for power-meaurement or meteo-dataregistration (as described before).
Can usually be solved by the manual reboot, but nevertheless gaps in registration, because you are never in time to detect&close the gap.
Looked at Power Supply as possible source for corruption, but that seems OK.

Looking for other solutions:
1) Manual reboot always works through PuttySSH by manual CLI-command sudo reboot.
Is there a way to 'automate' such function as remote input from another computer?
2) Not use cron, but use a separate, dedicated script to order reboot (hoping that that 'reboot'-script stays running).
That 'reboot'-script acting on a flag from the suspect script which indicates heartbeat-present. Time-out on heartbeat = action = reboot-command
Any idea how to fit a 'flag' in the suspect script, which can be read as heartbeat-signal by the 'reboot'-script?
Simple&Rude Setup1 seems to be
a) the suspect script transmits it's time to a Dummy,
b) if the script stops, no further transmission and the time in the Dummy freezes
c) 'reboot'-script reads the Dummy and compares against current systemtime,
d) Defined difference = Time-out detection = order reboot.

Myself have been looking/experimenting on mechanisms directly getting&checking validity-time of data through JSON-calls (which Setup2 is essentally the same as Setup1, but without the Dummy as intermediary), but not yet found/made a reliable script: for status see here.
Setup1 and Setup2 could both be run locally at the Raspberry (for 'close-loop' reboot) or can be remotely run on another computer (for 'remote' reboot).

[Toulon7559]

Reading back info from a website as an indicator whether "the link & upload & the uploading computer is operational" has it's limitations, because sometimes the website or the receiving application extrapolates data, and then in the read-out you get info including validity time, which apparently is valid, although the related upload has stopped.

Perhaps another approach ...

My setup periodically uploads info to several websites.
- WUnderground and AWEKAS (for meteo-data) report after een time-out by mailnotification that upload is missing:
- WUnderground has a fixed time-out of 2 hours
- AWEKAS is even better with a time-out adjustable by the user.
- At PVOutput.org I found under the title Register Notification Service something which could be applicable, because you can make a http-call to see whether/which alert is applicable, but that function misses an alert for time-out.

Is there a way to link such incoming notification to one of the crash-scripts in this thread (or to another script that can remotely trigger a reboot)?
Obviously such mechanism only operates if the processor & Domoticz is still running, but only the subject uploading script has stopped.
If the processor really crashed, nothing helps but brute, manual forced reboot etc.

[NetEye]

Hello,
When I have use one of the on my RPi 3, how do I check if it works?
Installation of it went well !
Test it by a command or something?
regards
Jan

[Derik]

Sudo service domoticz.sh stop

Status
Start
Restart

[ben53252642]

I wrote an alternate version:

Code: Select all

#!/bin/bash
while true; do

domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
if [[ $domoticzcheck != "OK" ]]; then
        echo "Domoticz not running, checking again in 30 seconds."
        sleep 30
        domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
        if [[ $domoticzcheck != "OK" ]]; then
                echo "Watchguard is restarting Domoticz"
                service domoticz stop
                service domoticz start
        fi
else
echo "Domoticz is running..."
fi

sleep 10
done

Notes:
1) The curl command is set to allow unsigned SSL certificates.
2) You need change the service start and stop section of the script to the right name of your Domoticz service.
Eg if you type: "service domoticz stop" and that works on your system then great you got the right name. Some systems may have custom names that people manually specified or could be: "service domoticz.sh stop" as an example.

[Derik]

I wrote an alternate version:

Code: Select all

#!/bin/bash
while true; do

domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
if [[ $domoticzcheck != "OK" ]]; then
        echo "Domoticz not running, checking again in 30 seconds."
        sleep 30
        domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
        if [[ $domoticzcheck != "OK" ]]; then
                echo "Watchguard is restarting Domoticz"
                service domoticz stop
                service domoticz start
        fi
else
echo "Domoticz is running..."
fi

sleep 10
done

Note:
The curl command is set to allow unsigned SSL certificates.

mmm please can you tell me the different s between the other?
Where can i place this?
Need a crontab?

[ben53252642]

The original version may fail to detect if Domoticz is really running in some circumstances due to is simply doing a port check.

The version in my above post makes a request to the Domoticz JSON API and only accepts Domoticz as running if it receives a response which confirms that it "is" actually running AND responding.

If it detects that Domoticz is not running, it waits 30 seconds, checks a second time and if the second attempt fails it then restarts Domoticz.

To use the script:

1) sudo apt-get install screen

2) nano /etc/init.d/watchguard

3) paste the contents below, editing the start section as needed to point to where you saved the watchguard.sh file

Code: Select all

#!/bin/sh
### BEGIN INIT INFO
# Provides: watchguard
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 5
# Default-Stop:
# Description: Starts Watchguard
### END INIT INFO

case "$1" in
'start')
        sudo /usr/bin/screen -S watchguard -d -m sudo /root/scripts/watchguard/watchguard.sh
        ;;
'stop')
        sudo pkill -f "watchguard"
        ;;
*)
        echo "Usage: $0 { start | stop }"
        ;;
esac
exit 0

4) chmod 755 /etc/init.d/watchguard

5) update-rc.d watchguard defaults

6) reboot the system, then type: screen -x

If you see the watchguard running then you have successfully installed it.

If you need to stop the watchguard for example while updating Domoticz then use:

service watchguard stop

[Toulon7559]

Earlier I reported on some 'difficulties' with the crash-recovery-script.

Sometimes a planned reboot completely fails, and that is easy to see, because Domoticz stops, and the Raspberry needs manual help for restart.
But sometimes the reboot-failure is only 'half' as demonstrated in this graph.

graph_with_gap

pitemp2_180218.png (41.37 KiB) Viewed 5910 times

What puzzles, is that the effect is not consistent:
- 3 similar graphs with this kind of layout are generated at one and same Raspberry
Each graph originates from an RRD sh-file which in turn is called by a related lua-script located in /home/pi/domoticz/scripts/lua.
- one graph usually happily continues after the planned periodic reboot
- the other 2 graphs (of which this is one) stop at a reboot and revive at the next reboot
The gap is not predictable and randomly happens.
- the only functional difference is that the 'continuing' graph is coming from 1 rrd-database-file, and that the other 2 graphs come from another rrd-database-file. Those 2 rrd-database-files are maintained by 2 different, but very comparable lua-scripts.

Seems a 'soft, unpredictable' problem, but those are the most difficult to solve .........

[pratty]

Hi all, I am very new here but have been playing with domoticz for a number of months now.

I noticed a lot of people are stuck on where and how to implement a script to check domoticz.

I have written a little guide on how I have set it up.

Thanks to the handy work of so many people on this forum, it makes life very easy when issues are encountered with domoticz and the domoticz community is around to help.

The script i use is also very simple. I modified the earlier scripts that checks the domoticz state, but I only check if it is NOT equal to active running, therefore, any other state other than "active running" domoticz is restarted. I also get it to restart the homebridge service, as i have found that sometimes it stops responding after domoticz crashes. it obviously isn't a total solution but has been working well for me so far.

if you want to set it up this is how i have implemented it on my pi 3.

The script should work with a standard raspberry pi installation using raspbian.

go into domoticz scripts folder

Code: Select all

cd domoticz/scripts

then make a file called restart_domoticz.sh

Code: Select all

nano restart_domoticz.sh

copy and paste the script below

Code: Select all

#!/bin/bash

IsDomoticzRunning=`sudo systemctl status domoticz.service`

if [[ $IsDomoticzRunning != *"active (running)"* ]]
        then
                echo 'Domoticz is not running.Restarting Domoticz.'
                sudo systemctl stop domoticz.service
                sudo systemctl restart homebridge.service
                echo "Domoticz & Homebridge restarted."
                
	else
                echo "Domoticz is running. Nothing to do."
 fi
 

save the script by pressing CTRL x and then y and enter

then add execute permission to the script

Code: Select all

chmod +x restart_domoticz.sh

if all went well you should now be able to test your script by typing

Code: Select all

./restart_domoticz.sh

then all that is left to do is add to a cronjob to run every minute.

Code: Select all

crontab -e

then add the cronjob at the bottom of the file

Code: Select all

*/1 * * * * ./home/pi/domoticz/scripts/restart_domoticz.sh

save the file by pressing CTRL x and then y and enter.

A big thankyou to everyone who contributed to this idea to keeps domoticz running if or when it fails.

Hopefully, this can help everyone who is having trouble implementing these amazing ideas from the community.

[Egregius]

A running service doesn’t mean it’s running

[ben53252642]

A running service doesn’t mean it’s running

Absolutely correct! You must use an alternate method that tests if the service is actually running.

I wrote a simple bash script here:
viewtopic.php?f=63&t=3016&sid=a4e9c3646 ... 40#p170196

Alternatively use the Monit setup:
https://www.domoticz.com/wiki/Monitorin ... gure_Monit

[ayasystems]

Sometimes domoticz is not working, www server of domoticz is not working but systemctl status domoticz shows it as working so you can not check by this way...

Check doing a curl is a better option but you can not use this
domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/j ... getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')

If you are ussing HTTPS you can not send user passsoword as you are doing... Your script needs pass the info over parameters

Regards!

[ben53252642]

Sometimes domoticz is not working, www server of domoticz is not working but systemctl status domoticz shows it as working so you can not check by this way...

Check doing a curl is a better option but you can not use this
domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/j ... getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')

If you are ussing HTTPS you can not send user passsoword as you are doing... Your script needs pass the info over parameters

Regards!

It works fine over HTTPS, make sure your Domoticz is set to use basic auth, I've got at least a dozen scripts working this way.

[lost]

Check doing a curl is a better option but you can not use this
domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:[email protected]/j ... getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')

Every time Domoticz web server was down but service status still looking OK, a check with httping was enough. As I'm doing the check on the PI running Domoticz itself, I only httping usual HTTP port (not secured one) for port default 8080 on localhost.

Take care that when Domoticz does not httping anymore, only restarting the Domoticz service is not always successful: Most of the times, web server remains attached to 8080/443 ports & binding fails at restart. As this does not happen very often (once a month maybe), when this happen the monitoring script does a PI reboot.

This is runned every X minutes by a cron. Not at home for now, but if someone finds usefull I could send the script that proved successful (with it's httping timeouts/retry setup before giving up to reboot) for me.

You'll have to install httping before (apt-get install httping).

[RapTile]

<..>
This is runned every X minutes by a cron. Not at home for now, but if someone finds usefull I could send the script that proved successful (with it's httping timeouts/retry setup before giving up to reboot) for me.
<..>

I would like to see this script ( if you are still active on the forum )