ZWave seems to crash every day at the same hour Topic is solved

[sloeber70]

Hi,

For 4 days-nights, the ZWave module seems to crash or to be unreachable for Domoticz. I get these lines in the logs :

2021-09-04 04:22:01.353 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:02.353 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:03.354 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:04.354 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:05.768 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:06.768 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:07.769 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:08.769 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:10.182 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:11.186 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:12.186 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:13.187 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 8 (0x08)
2021-09-04 04:22:14.598 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:16.006 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:18.237 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:19.238 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:20.649 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:21.649 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 9 (0x09)
2021-09-04 04:22:23.062 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:24.477 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:26.709 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:27.709 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:29.120 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:30.121 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 10 (0x0a)
2021-09-04 04:22:31.532 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:32.947 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:33.947 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:35.362 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:36.363 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:37.780 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 11 (0x0b)
2021-09-04 04:22:39.195 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:40.613 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:41.614 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:43.032 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:44.032 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:45.447 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 12 (0x0c)
2021-09-04 04:22:46.856 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:48.271 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:49.271 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:50.682 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:51.683 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:53.094 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 15 (0x0f)
2021-09-04 04:22:54.509 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:55.923 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:56.924 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:58.342 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:22:59.342 Status: ZwaveUSB: Received timeout notification from HomeID: 3704789617, NodeID: 16 (0x10)
2021-09-04 04:23:00.104 Status: dzVents: Info: +++ DEAD SENSORS LIGHTS CHECK +++ : ------ Start internal script: Dead l

This happens for all ZWave nodes in the database. When I do a /sbin/reboot... everything is working again.
Anyone who has the same issue ?

Issue occurs at 02:00:xx randomly seconds. At that time there is nothing running in crontab, the only thing I can think about is the hourly db backup, but that one runs every hour without any problem...

Kind regards,

[Treve]

Happy Birthday!

[sloeber70]

thx

[sloeber70]

Update:

The problem remains even after a reboot from a crontab job.
When I reboot manually in terminal, problem is solved.
Everything runs fine untill around 2:00 am.... Then ZWave crashes again and i need to do a manual reboot.

Help......

Domoticz latest version
Aeotec ZWave USB Gen5
RTL433 RFX.COM

[sloeber70]

No solution found yet...

Aeotec ZWave stick is not dead since it works during 22 hours a day.
USB : no issues found

What could it be.....

Should I do a soft reset, do I lose my ZWave settings then ? Backup first ?

[rrozema]

Try disabling the nightly heal:
hardware -> zwave -> settings button -> select the controller
then go to the bottom of the screen and find option "3. Enable Nightly Heal Network (04:00 am): " set it to Disabled
and then click the 'apply configuration to this device' button.

[sloeber70]

Hi,

thx for the reply.... but this setting was disabled...

re-enabled - disabled again...

OK, let's see what happens...

kind regards,

[sloeber70]

Hi,

The problem remains after altering the Network Heal setting.

Double checked the setting.

Hmmm.... Really odd....

Thx for the tip anyway....

Kind regards

[rrozema]

The weird thing is that your controller apparently does work: it still receives messages from the nodes, the timeout messages are messages themselves...

Just guessing here, but can it be you have some mains-powered node in the vicinity of your controller (probably one that you already have for a long time) that loses power around that time? I'm thinking that the problem could be that around 2:00 your network routing doesn't work any more. And a routing device that loses power could theoretically cause such a network routing failure.


If you've tried everything -which I think you did-, you can try running a manual network heal to make sure your mesh has sufficient redundant routes.. Please do be patient though: running the network heal can -depending on the number of nodes in your mesh- take up to more than an hour. Be aware that (parts of) your system can become unresponsive for prolonged periods during the scan, but in the end it should improve.

[sloeber70]

Hi,

Hmmm... maybe I have found the root cause...

What I did not mention earlier, since I thought it was irrelevant to the problem :
I am running a Plex Server on the same PI 4B.

I've checked the CPU usage when both engines are running and they never are higher then 40 % CPU total use . But what could be the issue is that I run a daily plex maintenance routine at 2:00 am and that can last till 5:00 am in the schedule. Looking at the graphs for CPU usage at that timeframe tells me that the CPU is at 100% at 02:00am and there is another high level around 04:30 am every night. Together with a lot of disk usage for that period, this could explain the issue.

What I did : deleted the maintenace schedule for Plex to be sure, disabled the PLEX server during the night...
If problem is solved, then I need to move the Plex Server to another PI....

Keeping my fingers crossed

Kind regards,

[sloeber70]

Hi,

The problem is solved.... The Plex Media Server is causing the troubles. Too high CPU and intensive disk I/O was slowing down the system so that it could not receive any ZWave signals durint the maintenance periods.

OK, need to move the Plex Server, thx for the tips, it made me "Think out of the Box"...

Kind regards,

[rrozema]

Ok, great news that you found it.