Script to restart Domoticz if it crashes Topic is solved

[erem]

Friend
here is my take on this for Raspberry pi and Ubntu/debian systems
I do not run on synology, sorry

all documented in the code

if you need positive confirmation, redirect the echo commands to /var/log/syslog

hope someone benefits.

Code: Select all

#!/bin/bash
#****************************************************************************
# program    : check-domoticz-active.sh
# programmer : RM
# date       : 2019-03-21
#
# install in cron with crontab -e to run every 5 minutes
#
#  m h dom mon dow   command
#  */5 * * * *       /home/pi/domoticz/scripts/check-domoticz-active.sh >/dev/null 2>&1
#
#
# prerequisites
# jq installed (apt-get install jq)
# make sure the cron entry (example above)has the full path to the script!!
#
# revision
# 0.0.1      2019-03-21   initial release
#
#
#****************************************************************************

i=0

while [ $i -lt 3 ]
do
	# check if domoticz responds to a json query
	DOMOTICZ=`curl -s --connect-timeout 2 --max-time 5 "http://127.0.0.1:8080/json.htm?type=devices&rid=1"`
	STATUS=`echo $DOMOTICZ | jq -r '.status'`
	if [ "$STATUS" == "OK" ] ; then
		echo "Domoticz responded"
		break			# all ok
	else
		i=$(( $i + 1 ))
		echo "Domoticz did not respond on try $i "
	fi
	sleep 5
done

# if we do not have an OK, domoticz did not respond, stop and start it
if [ "$STATUS" != "OK" ] ; then
	echo "Stopping domoticz"
	sudo systemctl stop domoticz.service
	sleep 10
	echo "Starting domoticz"
	NOW=$(date +"%Y-%m-%d %H:%M:%S")
	sudo systemctl start domoticz.service
	Exitcode=$? 	# save systemctl exit code
	if [ $Exitcode != 0 ] ; then
		echo "$NOW - sudo systemctl start domoticz.service failed with exit code $Exitcode"
	else
		echo "$NOW - sudo systemctl start domoticz.service completed with exit code $Exitcode"
	fi
fi

# end of source

[Egregius]

Add domoticz at the end of the 127.0.0.1 line.

[Carthman]

This one works for the pi, needed to make more changes then expected.

Code: Select all

#!/bin/bash
dt=$(date '+%d/%m/%Y %H:%M:%S')
CONFIG=/home/pi/domoticz/scripts/bash/domoticz_state_checker.txt
STATUS=`curl -s --connect-timeout 2 --max-time 5 "Accept: application/json" "http://192.168.1.109:8080/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
if [ $STATUS ] ; then
   exit
else
   sleep 5
   STATUS2=`curl -s --connect-timeout 2 --max-time 5 "Accept: application/json" "http://192.168.1.109:8080/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
   if [ $STATUS2] ; then
      exit
   else
      sleep 5
      STATUS3=`curl -s --connect-timeout 2 --max-time 5 "Accept: application/json" "http://192.168.1.109:8080/json.htm?type=devices&rid=1" | grep "status"| awk -F: '{print $2}'|sed 's/,//'| sed 's/\"//g'`
      if [ $STATUS3 ] ; then
         exit
      else
         NOW=$(date +"%Y-%m-%d_%H%M%S")
         sudo service domoticz.sh stop
         sleep 8
		 sudo killall domoticz 
         sleep 8
         sudo service domoticz.sh start
		 echo   "$dt, Domoticz was offline. Restarted Domoticz...." >> "$CONFIG"
      fi
   fi
fi

Hi,

So I used this version for my pi
I can launch the script without errors, but is it not supposed to write in a log something ?
Or maybe it will just write if a restart occured ?

And finally you just have to add this script in the crontab ?

Many thanks !

For information this version of the script works fine for me !
I saw in the logfile that Domoticz have restarded one time, it was completely transparent for me, I didn't notice anything !!!

Thanks again for your help !

[erem]

@NvBgm

to test: stop domoticz -> sudo systemctl stop domoticz.service

then manually run thescript, domoticz should restart.

then install in cron and forget about it.

[benbidouille]

Friend
here is my take on this for Raspberry pi and Ubntu/debian systems
I do not run on synology, sorry

all documented in the code

if you need positive confirmation, redirect the echo commands to /var/log/syslog

hope someone benefits.

Thanks a lot erem

To make it working I had to modify your code from

Code: Select all

	if [ "$STATUS" == "OK" ] ; then
		echo "Domoticz responded"
		break			# all ok

to

Code: Select all

	if [ "$STATUS" = "OK" ] ; then
		echo "Domoticz responded"
		break			# all ok

as otherwise was never found OK.
Is now under test, looks good Thanks

[molnaratti]

Friend
here is my take on this for Raspberry pi and Ubntu/debian systems
I do not run on synology, sorry

all documented in the code

if you need positive confirmation, redirect the echo commands to /var/log/syslog

hope someone benefits.

Code: Select all

#!/bin/bash
#****************************************************************************
# program    : check-domoticz-active.sh
# programmer : RM
# date       : 2019-03-21
#
# install in cron with crontab -e to run every 5 minutes
#
#  m h dom mon dow   command
#  */5 * * * *       /home/pi/domoticz/scripts/check-domoticz-active.sh >/dev/null 2>&1
#
#
# prerequisites
# jq installed (apt-get install jq)
# make sure the cron entry (example above)has the full path to the script!!
#
# revision
# 0.0.1      2019-03-21   initial release
#
#
#****************************************************************************

i=0

while [ $i -lt 3 ]
do
	# check if domoticz responds to a json query
	DOMOTICZ=`curl -s --connect-timeout 2 --max-time 5 "http://127.0.0.1:8080/json.htm?type=devices&rid=1"`
	STATUS=`echo $DOMOTICZ | jq -r '.status'`
	if [ "$STATUS" == "OK" ] ; then
		echo "Domoticz responded"
		break			# all ok
	else
		i=$(( $i + 1 ))
		echo "Domoticz did not respond on try $i "
	fi
	sleep 5
done

# if we do not have an OK, domoticz did not respond, stop and start it
if [ "$STATUS" != "OK" ] ; then
	echo "Stopping domoticz"
	sudo systemctl stop domoticz.service
	sleep 10
	echo "Starting domoticz"
	NOW=$(date +"%Y-%m-%d %H:%M:%S")
	sudo systemctl start domoticz.service
	Exitcode=$? 	# save systemctl exit code
	if [ $Exitcode != 0 ] ; then
		echo "$NOW - sudo systemctl start domoticz.service failed with exit code $Exitcode"
	else
		echo "$NOW - sudo systemctl start domoticz.service completed with exit code $Exitcode"
	fi
fi

# end of source

sh /home/pi/domoticz/scripts/check-domoticz-active.sh
parse error: Invalid numeric literal at line 1, column 60

What's the problem? Thank you

[erem]

try running ./check-domoticz-active.sh from the /home/pi/domoticz/scripts directory

it may be the way you call the script using sh

[Alexis64]

DOMOTICZ=`curl -s --connect-timeout 2 --max-time 5 "http://User:[email protected]:8080/ ... getversion"`

In my case I had to add thr user name and password for Domoticz otherwize it comes with unautorized message.

[Carthman]

Hello,

I plan to upgrade to new version 2020.2, but with a complete new fresh install on a different model of Raspberry.

Does this script will work on new version ?
Is it still relevant to use it or new version has some native scripts to check the state of Domoticz ?

Thanks !

[erem]

The script i published run unchanged on my rpi with buster and domoticz version 2020.2.

[Carthman]

Thanks for you reply !

[Toulon7559]

Avoiding a crash is better than 'repair/restart' after a crash.

Perhaps kicking an open door, but the following 'remedy' may help as well to avoid or to decrease crashes.

One reason for a crash may be an overload of the processing:
a possible cause for significant peak load occurs if start & run for scripts is almost at same time.
The indicator CPU_Usage in Domoticz may be an indicator for your configuration.

Rather easy to remedy such situation, by scheduling of the scripts:
- some, but limited software has a functional demand to run with shortest interval and/or have to start at the 'obvious' times of full hour or every 5 minutes or less
- a script at high level may have that requirement, but seldomly all (sub)functions inside that script have that same requirement, and those (sub)functions could be skipped every so many times that the script is running, reducing the execution time
Example: read-out of data and saving in a database may require high speed/ low latency to get good resolution in time, but related upload and generation of graphs generally can be a slow(er) process.
Practise:
=> run scripts at lowest rate possible
=> inside the scripts, set (sub)functions at an even lower rate of timing
==> a script not every run takes the full, required time
=> do not run all scripts at the 'obvious' times of full hour or at every 5 minutes, but (if functionally possible) start/run scripts at 'odd' times
===> such distribution in time probably reduces the actual, cumulative load of processing

The implementation in a script is not more difficult than a construct (for a lua-script) like

Code: Select all

if (minutes % Interval == Offset) then
       < do function >
end

Fiddling with the settings for interval and offset puts the processing of the function at the desired place in time.

[EDsteve]

Code: Select all

#!/bin/bash
#****************************************************************************
# program    : check-domoticz-active.sh
# programmer : RM
# date       : 2019-03-21
#
# install in cron with crontab -e to run every 5 minutes
#
#  m h dom mon dow   command
#  */5 * * * *       /home/pi/domoticz/scripts/check-domoticz-active.sh >/dev/null 2>&1
#
#
# prerequisites
# jq installed (apt-get install jq)
# make sure the cron entry (example above)has the full path to the script!!
#
# revision
# 0.0.1      2019-03-21   initial release
#
#
#****************************************************************************

i=0

while [ $i -lt 3 ]
do
	# check if domoticz responds to a json query
	DOMOTICZ=`curl -s --connect-timeout 2 --max-time 5 "http://127.0.0.1:8080/json.htm?type=devices&rid=1"`
	STATUS=`echo $DOMOTICZ | jq -r '.status'`
	if [ "$STATUS" == "OK" ] ; then
		echo "Domoticz responded"
		break			# all ok
	else
		i=$(( $i + 1 ))
		echo "Domoticz did not respond on try $i "
	fi
	sleep 5
done

# if we do not have an OK, domoticz did not respond, stop and start it
if [ "$STATUS" != "OK" ] ; then
	echo "Stopping domoticz"
	sudo systemctl stop domoticz.service
	sleep 10
	echo "Starting domoticz"
	NOW=$(date +"%Y-%m-%d %H:%M:%S")
	sudo systemctl start domoticz.service
	Exitcode=$? 	# save systemctl exit code
	if [ $Exitcode != 0 ] ; then
		echo "$NOW - sudo systemctl start domoticz.service failed with exit code $Exitcode"
	else
		echo "$NOW - sudo systemctl start domoticz.service completed with exit code $Exitcode"
	fi
fi

# end of source

I tried that script but get this error:

Code: Select all

pi@domoticz:~/domoticz/scripts $ ./check-domoticz-active.sh
parse error: Invalid numeric literal at line 1, column 60

I understand "line 1". But what does "column 60" mean? As you can see i am not a programmer
Is there anything i do wrong?

[waaren]

Code: Select all

pi@domoticz:~/domoticz/scripts $ ./check-domoticz-active.sh
parse error: Invalid numeric literal at line 1, column 60

I understand "line 1". But what does "column 60" mean? As you can see i am not a programmer
Is there anything i do wrong?

The error you see comes from jq. It tells you something is wrong with the response from domoticz.
Did you enable password less access from local network?

Below modified version will show some more information and continues when jq errors

Code: Select all

#!/bin/bash
#****************************************************************************
# program	: check-domoticz-active.sh
# programmer : RM
# date	   : 2019-03-21
#
# install in cron with crontab -e to run every 5 minutes
#
#  m h dom mon dow   command
#  */5 * * * *	   /home/pi/domoticz/scripts/check-domoticz-active.sh >/dev/null 2>&1
#
#
# prerequisites
# jq installed (apt-get install jq)
# make sure the cron entry (example above)has the full path to the script!!
# make sure domoticz can be accessed without password from local system [Setup][Settings][System] "Local Networks (no username/password): 127.0.0.1 "
#
# revision
# 0.0.1	  2019-03-21   initial release
# 0.0.2	  2020-12-14   Use status JSON
#
#
#****************************************************************************

domoticzPort=8080
i=0

while [ $i -lt 3 ]
do
	# check if domoticz responds to a json query
	DOMOTICZ=$(curl -s --connect-timeout 2 --max-time 5 "http://127.0.0.1:$domoticzPort/json.htm?type=command&param=getversion")

	STATUS=$(echo $DOMOTICZ | jq -r '.status' || true )
	

	if [ "$STATUS" == "OK" ] ; then
		VERSION=$(echo $DOMOTICZ | jq -r '.version' || true)
		echo "Domoticz responded and is on version "$VERSION
		break			# all ok
	else
		i=$(( $i + 1 ))
		echo "Domoticz did not respond on try $i "
		echo curl response was ${DOMOTICZ:-Empty}
	fi
	sleep 5
done

# if we do not have an OK, domoticz did not respond, stop and start it
if [ "$STATUS" != "OK" ] ; then
	echo "Stopping domoticz"
	sudo systemctl stop domoticz.service
	sleep 10
	echo "Starting domoticz"
	NOW=$(date +"%Y-%m-%d %H:%M:%S")
	sudo systemctl start domoticz.service
	Exitcode=$? 	# save systemctl exit code
	if [ $Exitcode != 0 ] ; then
		echo "$NOW - sudo systemctl start domoticz.service failed with exit code $Exitcode"
	else
		echo "$NOW - sudo systemctl start domoticz.service completed with exit code $Exitcode"
	fi
fi

# end of source

[EDsteve]

Hi.

and thanks for your fast reply. Yes. I had passwords turned off for local networks. But i didn't add the 127.0.0.* IP range. The problem sits mostly between the chair and monitor

So for everybody who made the same mistake:
In Domoticz. Setup -> Settings

image_2020-12-16_162400.png (11.45 KiB) Viewed 7048 times

Script works nice. Thanks for that.

[anno]

I did run the script, without a problem.
But crontab won't run the script. I use ubuntu.

[jvdz]

Did you add the. execute attribute to the bash file?

[Carthman]

Is it still relevant to use this script with the last version 2023.2 ?

My last record on le logfile is from 2020, so maybe the last versions from 2020 are more stable ?

[waltervl]

It is up to you if you still need this script. I have Monit running to check Domoticz but also there no need for action the last versions.
Some Python plugins also could crash Domoticz (especially when they are new) so it is very dependent of your environment.

[lost]

Is it still relevant to use this script with the last version 2023.2 ?
My last record on le logfile is from 2020, so maybe the last versions from 2020 are more stable ?

I have a similar script but same feedback as you: Did not triggered a restart since several years, lasts were probably for v4.10717 I kept quite long, until 2020.2 (this was the OZW 1.4->1.6 transition with a few roadbumps expected).

Since then Domoticz is a swiss clock... I don't run any python plugin for now (did not go zigbee for now that does not have native implementation as zwave has... or had ).

IMO, keep it, even if you fond it now useless: Rare issues always occurs during vacations, when this may not be possible to remotely restart domoticz if needed!

Just take care that this script would not have been able to recover all issues I had in the past: If the service stops, that's already natively restarted so nothing to do. But I had issues with service still up, events handled... but web-server/UI side was unresponsive: Probably the kind of issue this script manages through a check using http/json api (I used httping on my side that proved being a valid test). But when this issue occured, sometimes restarting domoticz service was not enough as there was a bind error: HTTP(S) ports were not free by service stop & start was then failing leading to infinite restart attempts loop. So, if service restart was KO, there was a full system restart to handle any such case with unclean stop that may block restart.

Code: Select all

#!/bin/bash
# Check domoticz (from a crontab) is up a restart whole PI if needed...

domoticzUrl=localhost:8080
BN=`basename $0`

WHOCOUNT=$(who | wc -l)
if [ ${WHOCOUNT} -ne 0 ]
then
  logger $BN: Someone is logged on, no check.
  exit 0
fi

httping -c 5 -i 0 -t 1 --ts -v -Wsqg $domoticzUrl
STATUS=$?
if [ ${STATUS} -ne 0 ]
then
  logger $BN: Domoticz httping-ed KO, retry after 1mn... 
  sleep 1m
  # Retry once
  httping -c 5 -i 0 -t 1 --ts -v -Wsqg $domoticzUrl
  STATUS=$?
  if [ ${STATUS} -ne 0 ]
  then
    logger $BN: Still KO. Get last logs and try service restart then wait...
    tail -n 20 /tmp/domoticz.txt | logger
    service domoticz restart
    sleep 1m
    logger $BN: Check after service restart...
    httping -c 5 -i 0 -t 1 --ts -v -Wsqg $domoticzUrl
    STATUS=$?
    if [ ${STATUS} -ne 0 ]
    then
      logger $BN: Still KO. Get last logs and REBOOT...
      tail -n 20 /tmp/domoticz.txt | logger
      /sbin/shutdown --no-wall -r now
      STATUS=$?
    else
      logger $BN: Service restart OK !
    fi
  else
    logger $BN: Retry OK !
  fi
else
  logger $BN: Domoticz ALIVE !
fi

wait
logger $BN: DONE, status= ${STATUS} !!!
exit ${STATUS}

This runs as a root cron every 30mn:

Code: Select all

0,30  *  *   *   *     /root/checkDomoticz.sh > /dev/null 2>&1

Also, as this script may trigger a restart, it skips checks if someone is logged: Once bitten... I experienced an unexpected restart, domoticz intentionally stopped, just in the middle of a Debian version upgrade (not a full reinstall/new SD image, but the usual upgrade Debian way by modifying /etc/apt/sources.list and apt commands to update/upgrade skipping new pkgs/full upgrade then reboot on new version).
Of course, the PI was not able to start anymore...