2024.7 stops working every few days

[BartSr]

domoticz:
2024.7 built 16178
Raspberry Pi 3b+ bullseye

crash at night several times this week
ping device is oke
only solution : restart service
then next error:
2024-08-23 17:04:11.683 Error: Raspberry hardware (12) thread seems to have ended unexpectedly
has this todo with crash at night?

how to?

[Kedi]

Updated yesterday to 16182, crashed again today. Automatic back-up is disabled. Attached the crash log, but i do not know if there is any useable information in there.

Your problem comes probably from the GoodWe plugin.
Disable the plugin for some days and see if that solves the problem.
If so contact the plugin creator.

[BartSr]

Code: Select all

Domoticz Version: 2024.7 (build 16187)
Build Hash: 6b9cbb21c
Compile Date: 2024-08-22 08:08:34
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]

Raspberry Pi 3b+ SDcard 32Gb

Further to regular stops last week:

I found that somewhere files are growing.
Yesterday I noticed SD card 32Gb 100% full causing failures.
Then restored an image. 32Gb SD card now 65%.
Today already 76% Database is steady around 60Mb.

installed on SD card:

Bullseye OS
docker compose for zwave2mqtt and zigbee2mqtt
native (no docker) domoticz install
native (no docker) mosquitto
wireguard
Any suggestion where to look for growing files?

I used command
sudo du -sm /* 2>/dev/null | sort -V
from that I learnt that /VAR is growing

is there a linux command which lists all files into a textfilw sorted on size?

[Kedi]

Try:

Code: Select all

sudo du -ah /var | sort -rh | head -10 > textfile.txt

[BartSr]

@ Kedi
thanks a lot. by changing /var to
/var/lib
/var/lib/docker I can find the 'growing file'

It looks like this file (xxx/yyy-json.log) is the one growing fast (ca.11%/day) :

Code: Select all

5,1G	/var/lib/docker
4,2G	/var/lib/docker/containers
4,1G	/var/lib/docker/containers/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7-json.log
4,1G	/var/lib/docker/containers/896db812b0572298a3a703e65079aaa647a407b657d152772feb6a66d01da6c7

is there a way to find out which logfile it is?
zwave2mqtt and zigbee2mqtt are the only ones running in docker.

[Kedi]

If you do a

Code: Select all

sudo docker ps

You will get an overview, and the first part 'CONTAINER ID' is the first part of your log filename, eg. 896db812b0......

[BartSr]

thanks a lot.
I found it's the zwave container.
will it be save to remove the logfile?
meanwhile I found that in zwavejs debug was set to on and results to store in a file so I presume it's that huge file.
I will report the results.
Bart

[Kedi]

It is simple.
1. Do a 'sudo su'
2. Stop the container
3. Remove the log file(s)
4. Start the container.
5. Exit the 'su'

A good manual on docker log files can be found here: https://signoz.io/guides/docker-clear-logs/

[BartSr]

@Kedi
Thanks for your help.
Once I stopped the containers and restarted them it appeared there was no longer a log file. (I set log in ZwaveUI to off).
HD space used 48%. Before stopping both containers HD space used 78% SD card 32Gb.

Lessons learnt: only set log to on if there's a specific reason otherwise this may cause crash of domoticz due to less HD space.
Before I used SD card 16Gb but changed into 32Gb because of too less space left on HD.

Maybe it's a good idea to have notification once HD space used > 80%

-Bart

[FlyingDomotic]

Maybe it's a good idea to have notification once HD space used > 80%

It's not a good idea, it's an excellent idea!

Go to "Hardware" tab, and add a "Motherboard sensor" device. You'll have new devices with CPU/Memory/Disk space. It may then be judicious to make some notifications on them;-)

[Hiu80]

Updated yesterday to 16182, crashed again today. Automatic back-up is disabled. Attached the crash log, but i do not know if there is any useable information in there.

Your problem comes probably from the GoodWe plugin.
Disable the plugin for some days and see if that solves the problem.
If so contact the plugin creator.

Thanks for the heads up. Some time ago i updated the plugin, but somehow used an older version of it. It is now properly updated, and will check if is still crashes.

[microkid]

Any update on this issue? Domoticz is crashing almost every day at random times. Still nothing in the log files.

[waltervl]

Any update on this issue? Domoticz is crashing almost every day at random times. Still nothing in the log files.

Did you update to latest beta? There were some changes to at least show more logging before crashing. Also reports of no crashing. But all of this is readable in this topic....

[SESTH]

thanks a lot.
I found it's the zwave container.
will it be save to remove the logfile?
meanwhile I found that in zwavejs debug was set to on and results to store in a file so I presume it's that huge file.
I will report the results.
Bart

Log rotation should help.

[BartSr]

@sesth
thanks for pointing me to this brilliant explanation re the logfiles.

[Rik60]

Hi all,

I have a problem with Domoticz that it is crashing one/ two times a week. The only thing changed was a upgrade to version 2024.7 (i updated from 2024.6 to 2024.7 when it was released). Domoticz is running at a pi3B. The raspberry is running normal. A Domoticz service restart is enough to have Domoticz online again. Attached is a Domoticz_crash_log file. I can't find why Domoticz is crashing. Thanks for your help.

[FlyingDomotic]

Gulty could be:

Code: Select all

Thread 33 (Thread 0x6e2ff200 (LWP 13719) "Zigbee"):
#0  0x768e2824 in __GI___wait4 (pid=23406, stat_loc=0x6e2fc578, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00689f72 in dumpstack_gdb(bool) ()
#2  0x0068a3c8 in signal_handler(int, siginfo_t*, void*) ()
#3  <signal handler called>
#4  raise (sig=11) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  <signal handler called>
#6  0x759fe1b4 in ?? () from /lib/arm-linux-gnueabihf/libpython3.9.so
#7  0x759fe2d4 in ?? () from /lib/arm-linux-gnueabihf/libpython3.9.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Signal 11 (SIGSEGV, segmentation violation) means that the program accessed a memory location that was not assigned to it. This may be a bug...

[Rik60]

Scrolling back in the history, maybe the Reolink plugin is the problem. I installed this plugin in june, almost the sametime i upgrade to 2024.7. Sinds then Domoticz was reacting strange, sometimes blockly events were party working and errors in the logging. So i have removed the Reolink plugin and will see what happens.

[Syndomo]

Adding my 2 cents.

I switched to 2024.7 about a week ago, and have some issues I never had before.

SYSTEM:
- Raspberry Pi4 on Bookworm, running only Domoticz
- external SSD (no SD card)
- ZWave dongle
- SunFounder UPS
I don't observe high processor loads for Domoticz (typically 5-10%), no suspicious (increase of) memory use either.

HOME AUTOMATION:
I have 8 Daikin units as LAN/HTTP hardware in Domoticz.
They have a polling rate of 20 seconds (where the standard value is 300). This worked fine until I switched to 2024.7.
(Please don't start discussions about why such a high frequency is needed - though I'd be glad to elaborate on the details of my Daikin home automation it isn't the subject of this topic.)

Every minute a script checks when the latest temperature update came in for each unit. Considering the polling frequency it should definitely be less than 60 seconds ago. If not a notification is sent to me via mail.

Besides those 8 Daikin units I have
- 23 Z-Wave devices (ZWave-js-ui via MQTT). No problems whatsoever with those.
- a P1 meter being polled over HTTP every 10 seconds via a lua handler

SYMPTOM:
- sometimes I get an error in the log stating "Error: <Daikin unit xxx> hardware (17) thread seems to have ended unexpectedly".
I had one case where I had 4 of those in less than 2 minutes.
- sometimes I get a notification (see above), but I had cases where I got such notification without there being the above thread error in the log !?
- Domoticz web interface inaccessible (happened once until now). I don't know if Domoticz was still running or not. I had to stop/start Domoticz via ssh after which it happily (?) carried on.

FREQUENCY:
- On 29/01 I had several unexpected hardware thread problems (all for the same Daikin unit), and 7 notification messages in a time span of about 5 hours. In the end I couldn't access Domoticz via the web interface anymore.
- On 1/2 I had one unexpected hardware thread error in the log, but also 5 notification messages WITHOUT any thread error in the logging. Again all for the same Daikin unit, but not the same one as on 29/01.
- It's not related to midnight or hour (so not related to database backup), it happens randomly.
- It's not related to system load, I mean I have about 1000 dzVents script lines executing each minute but that takes only a couple of seconds, and an unexpected thread ending mostly happens when no script is running.
- When it happens, it re-occurs for the same Daikin unit.
- But after a Raspberry reboot it happened for another Daikin unit (and after that again and again for that one).

NEXT STEPS ?
Until now this hasn't been a blocking issue, but something must be wrong somewhere.
I hope this information can help to ring a bell about where to search for the potential cause.
If I can provide more information I'd be happy to assist.

[Syndomo]

Follow-up: I seem to have got the situation back under control by setting the Daikin LAN/HTTP polling interval back to 300 seconds instead of 20 seconds for all units.

HOWEVER
- in Domoticz 2024.4 it worked well with the short polling interval of 20 seconds
- in Domoticz 2024.7 the situation got unstable...