2024.7 stops working every few days

[BartSr]

domoticz:
2024.7 built 16178
Raspberry Pi 3b+ bullseye

crash at night several times this week
ping device is oke
only solution : restart service
then next error:
2024-08-23 17:04:11.683 Error: Raspberry hardware (12) thread seems to have ended unexpectedly
has this todo with crash at night?

how to?

[Kedi]

Updated yesterday to 16182, crashed again today. Automatic back-up is disabled. Attached the crash log, but i do not know if there is any useable information in there.

Your problem comes probably from the GoodWe plugin.
Disable the plugin for some days and see if that solves the problem.
If so contact the plugin creator.

[BartSr]

Code: Select all

Domoticz Version: 2024.7 (build 16187)
Build Hash: 6b9cbb21c
Compile Date: 2024-08-22 08:08:34
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]

Raspberry Pi 3b+ SDcard 32Gb

Further to regular stops last week:

I found that somewhere files are growing.
Yesterday I noticed SD card 32Gb 100% full causing failures.
Then restored an image. 32Gb SD card now 65%.
Today already 76% Database is steady around 60Mb.

installed on SD card:

Bullseye OS
docker compose for zwave2mqtt and zigbee2mqtt
native (no docker) domoticz install
native (no docker) mosquitto
wireguard
Any suggestion where to look for growing files?

I used command
sudo du -sm /* 2>/dev/null | sort -V
from that I learnt that /VAR is growing

is there a linux command which lists all files into a textfilw sorted on size?

[Kedi]

Try:

Code: Select all

sudo du -ah /var | sort -rh | head -10 > textfile.txt

[BartSr]

@ Kedi
thanks a lot. by changing /var to
/var/lib
/var/lib/docker I can find the 'growing file'

It looks like this file (xxx/yyy-json.log) is the one growing fast (ca.11%/day) :

Code: Select all

5,1G	/var/lib/docker
4,2G	/var/lib/docker/containers
4,1G	/var/lib/docker/containers/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7-json.log
4,1G	/var/lib/docker/containers/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7

is there a way to find out which logfile it is?
zwave2mqtt and zigbee2mqtt are the only ones running in docker.

[Kedi]

If you do a

Code: Select all

sudo docker ps

You will get an overview, and the first part 'CONTAINER ID' is the first part of your log filename, eg. 896db812b0......

[BartSr]

thanks a lot.
I found it's the zwave container.
will it be save to remove the logfile?
meanwhile I found that in zwavejs debug was set to on and results to store in a file so I presume it's that huge file.
I will report the results.
Bart

[Kedi]

It is simple.
1. Do a 'sudo su'
2. Stop the container
3. Remove the log file(s)
4. Start the container.
5. Exit the 'su'

A good manual on docker log files can be found here: https://signoz.io/guides/docker-clear-logs/

[BartSr]

@Kedi
Thanks for your help.
Once I stopped the containers and restarted them it appeared there was no longer a log file. (I set log in ZwaveUI to off).
HD space used 48%. Before stopping both containers HD space used 78% SD card 32Gb.

Lessons learnt: only set log to on if there's a specific reason otherwise this may cause crash of domoticz due to less HD space.
Before I used SD card 16Gb but changed into 32Gb because of too less space left on HD.

Maybe it's a good idea to have notification once HD space used > 80%

-Bart

[FlyingDomotic]

Maybe it's a good idea to have notification once HD space used > 80%

It's not a good idea, it's an excellent idea!

Go to "Hardware" tab, and add a "Motherboard sensor" device. You'll have new devices with CPU/Memory/Disk space. It may then be judicious to make some notifications on them;-)

[Hiu80]

Updated yesterday to 16182, crashed again today. Automatic back-up is disabled. Attached the crash log, but i do not know if there is any useable information in there.

Your problem comes probably from the GoodWe plugin.
Disable the plugin for some days and see if that solves the problem.
If so contact the plugin creator.

Thanks for the heads up. Some time ago i updated the plugin, but somehow used an older version of it. It is now properly updated, and will check if is still crashes.

[microkid]

Any update on this issue? Domoticz is crashing almost every day at random times. Still nothing in the log files.

[waltervl]

Any update on this issue? Domoticz is crashing almost every day at random times. Still nothing in the log files.

Did you update to latest beta? There were some changes to at least show more logging before crashing. Also reports of no crashing. But all of this is readable in this topic....

[SESTH]

thanks a lot.
I found it's the zwave container.
will it be save to remove the logfile?
meanwhile I found that in zwavejs debug was set to on and results to store in a file so I presume it's that huge file.
I will report the results.
Bart

Log rotation should help.

[BartSr]

@sesth
thanks for pointing me to this brilliant explanation re the logfiles.

[Rik60]

Hi all,

I have a problem with Domoticz that it is crashing one/ two times a week. The only thing changed was a upgrade to version 2024.7 (i updated from 2024.6 to 2024.7 when it was released). Domoticz is running at a pi3B. The raspberry is running normal. A Domoticz service restart is enough to have Domoticz online again. Attached is a Domoticz_crash_log file. I can't find why Domoticz is crashing. Thanks for your help.

[FlyingDomotic]

Gulty could be:

Code: Select all

Thread 33 (Thread 0x6e2ff200 (LWP 13719) "Zigbee"):
#0  0x768e2824 in __GI___wait4 (pid=23406, stat_loc=0x6e2fc578, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00689f72 in dumpstack_gdb(bool) ()
#2  0x0068a3c8 in signal_handler(int, siginfo_t*, void*) ()
#3  <signal handler called>
#4  raise (sig=11) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  <signal handler called>
#6  0x759fe1b4 in ?? () from /lib/arm-linux-gnueabihf/libpython3.9.so
#7  0x759fe2d4 in ?? () from /lib/arm-linux-gnueabihf/libpython3.9.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Signal 11 (SIGSEGV, segmentation violation) means that the program accessed a memory location that was not assigned to it. This may be a bug...

[Rik60]

Scrolling back in the history, maybe the Reolink plugin is the problem. I installed this plugin in june, almost the sametime i upgrade to 2024.7. Sinds then Domoticz was reacting strange, sometimes blockly events were party working and errors in the logging. So i have removed the Reolink plugin and will see what happens.