Page 1 of 1

ZWave seems to crash every day at the same hour

Posted: Sunday 05 September 2021 11:33
by sloeber70
Hi,

For 4 days-nights, the ZWave module seems to crash or to be unreachable for Domoticz. I get these lines in the logs :

2021-09-04 04:22:01.353 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:02.353 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:03.354 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:04.354 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:05.768 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:06.768 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:07.769 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:08.769 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:10.182 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:11.186 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:12.186 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:13.187 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:14.598 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:16.006 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:18.237 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:19.238 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:20.649 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:21.649 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:23.062 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:24.477 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:26.709 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:27.709 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:29.120 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:30.121 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:31.532 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:32.947 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:33.947 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:35.362 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:36.363 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:37.780 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:39.195 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:40.613 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:41.614 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:43.032 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:44.032 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:45.447 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:46.856 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:48.271 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:49.271 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:50.682 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:51.683 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:53.094 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:54.509 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:55.923 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:56.924 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:58.342 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:59.342 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:23:00.104 Status: dzVents: Info: +++ DEAD SENSORS LIGHTS CHECK +++ : ------ Start internal script: Dead l

This happens for all ZWave nodes in the database. When I do a /sbin/reboot... everything is working again.
Anyone who has the same issue ?

Issue occurs at 02:00:xx randomly seconds. At that time there is nothing running in crontab, the only thing I can think about is the hourly db backup, but that one runs every hour without any problem...

Kind regards,

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 05 September 2021 12:12
by Treve
Happy Birthday!

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 05 September 2021 16:18
by sloeber70
thx

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 19 September 2021 9:33
by sloeber70
Update:

The problem remains even after a reboot from a crontab job.
When I reboot manually in terminal, problem is solved.
Everything runs fine untill around 2:00 am.... Then ZWave crashes again and i need to do a manual reboot.

Help......

Domoticz latest version
Aeotec ZWave USB Gen5
RTL433 RFX.COM

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 26 September 2021 18:34
by sloeber70
No solution found yet...

Aeotec ZWave stick is not dead since it works during 22 hours a day.
USB : no issues found

What could it be.....

Should I do a soft reset, do I lose my ZWave settings then ? Backup first ?

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 26 September 2021 19:38
by rrozema
Try disabling the nightly heal:
hardware -> zwave -> settings button -> select the controller
then go to the bottom of the screen and find option "3. Enable Nightly Heal Network (04:00 am): " set it to Disabled
and then click the 'apply configuration to this device' button.

Re: ZWave seems to crash every day at the same hour

Posted: Tuesday 28 September 2021 11:09
by sloeber70
Hi,

thx for the reply.... but this setting was disabled...

re-enabled - disabled again...

OK, let's see what happens...

kind regards,

Re: ZWave seems to crash every day at the same hour

Posted: Thursday 30 September 2021 9:20
by sloeber70
Hi,

The problem remains after altering the Network Heal setting.

Double checked the setting.

Hmmm.... Really odd....

Thx for the tip anyway....

Kind regards

Re: ZWave seems to crash every day at the same hour

Posted: Thursday 30 September 2021 10:23
by rrozema
The weird thing is that your controller apparently does work: it still receives messages from the nodes, the timeout messages are messages themselves...

Just guessing here, but can it be you have some mains-powered node in the vicinity of your controller (probably one that you already have for a long time) that loses power around that time? I'm thinking that the problem could be that around 2:00 your network routing doesn't work any more. And a routing device that loses power could theoretically cause such a network routing failure.


If you've tried everything -which I think you did-, you can try running a manual network heal to make sure your mesh has sufficient redundant routes.. Please do be patient though: running the network heal can -depending on the number of nodes in your mesh- take up to more than an hour. Be aware that (parts of) your system can become unresponsive for prolonged periods during the scan, but in the end it should improve.

Re: ZWave seems to crash every day at the same hour

Posted: Saturday 02 October 2021 22:35
by sloeber70
Hi,

Hmmm... maybe I have found the root cause...

What I did not mention earlier, since I thought it was irrelevant to the problem :
I am running a Plex Server on the same PI 4B.

I've checked the CPU usage when both engines are running and they never are higher then 40 % CPU total use . But what could be the issue is that I run a daily plex maintenance routine at 2:00 am and that can last till 5:00 am in the schedule. Looking at the graphs for CPU usage at that timeframe tells me that the CPU is at 100% at 02:00am and there is another high level around 04:30 am every night. Together with a lot of disk usage for that period, this could explain the issue.

What I did : deleted the maintenace schedule for Plex to be sure, disabled the PLEX server during the night...
If problem is solved, then I need to move the Plex Server to another PI....

Keeping my fingers crossed ;-)

Kind regards,

Re: ZWave seems to crash every day at the same hour

Posted: Sunday 03 October 2021 11:02
by sloeber70
Hi,

The problem is solved.... The Plex Media Server is causing the troubles. Too high CPU and intensive disk I/O was slowing down the system so that it could not receive any ZWave signals durint the maintenance periods.

OK, need to move the Plex Server, thx for the tips, it made me "Think out of the Box"...

Kind regards,

Re: ZWave seems to crash every day at the same hour

Posted: Monday 04 October 2021 10:36
by rrozema
Ok, great news that you found it.